# tutorials

topic: admin
tutorial: monitoring
title: Galaxy Monitoring with Telegraf and Grafana
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "monitoring", "git-gat"]
content_hash: 00236cea2c80441a0c5a5d5bccd3a0b0
last_modified: 2026-02-26T21:12:14.149350
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/monitoring/tutorial.html

content:
Monitoring is an incredibly important part of server monitoring and maintenance. Being able to observe trends and identify hot spots by collecting metrics gives you a significant ability to respond to any issues that arise in production. Monitoring is quite easy to get started with, it can be as simple as writing a quick shell script in order to start collecting metrics.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="monitoring" %}

This tutorial explicitly assumes you are starting with a setup like that created in the *Galaxy installation with Ansible* tutorial

# Data Flow

The monitoring setup we (UseGalaxy.\*) use involves compute nodes collecting metrics and forwarding the data to a central collector. On each node we have a data collector running (Telegraf) which collects data from numerous sources before aggregating it and forwarding it onwards. On another, sometimes public, node we accept and collect all of these metrics (InfluxDB) and provide an interface to visualise and query that data (Grafana).

For UseGalaxy.eu, we pre-install Telegraf on our compute node images. Whenever we launch new compute nodes, they automatically start sending data to our central collection point. Telegraf additionally provides a nice feature of aggregating metrics it collects, batching them up before sending them off to InfluxDB. Telegraf has built in data collectors, or can run command line scripts and tools which output specifically formatted data.

Telegraf communicates with InfluxDB using a JSON based HTTP API, making it easy to write custom tooling to send data there, if need be. Grafana provides the visualisation component, the most interesting component for most consumers of the metrics and monitoring.


# Infrastructure

Setting up the infrastructure is quite simple thanks to the automation provided by Ansible. We will first setup a playbook for a "monitoring" machine, which will collect and visualize our data using InfluxDB and Grafana. We will then expand the Galaxy playbook to include Telegraf on the machine to monitor both it and Galaxy itself. In this tutorial we will do everything on the same machine. In practice you can separate these services to different machines, if you wish. The only requirement is that Telegraf run on the machine from which you wish to collect data.

## InfluxDB

[InfluxDB](https://www.influxdata.com/) provides the data storage for monitoring. It is a {TSDB}, so it has been designed specifically for storing time-series data like monitoring and metrics. There are other TSBD options for storing data but we have had good experiences with this one. TSBDs commonly feature some form of automatic data expiration after a set period of time. In InfluxDB these are known as "retention policies". Outside of this feature, it is a [relatively normal database](https://docs.influxdata.com/influxdb/v1.7/concepts/crosswalk/).

The available Ansible roles for InfluxDB unfortunately do not support configuring databases or users or retention policies. Ansible itself contains [several modules](https://docs.ansible.com/ansible/2.9/modules/list_of_database_modules.html#influxdb) you can use to write your own roles, but nothing generic. UseGalaxy.eu wrote [their own role](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/roles/hxr.influxdb/tasks/main.yml) for setting up their InfluxDB database, but it is not reusable enough for it to be used here yet. If you plan to automate your entire setup, this tutorial can perhaps provide inspiration for writing your own Ansible role. However, in this case it is sufficient to manually create your users and retention policies as a one-off task.

> <hands-on-title>Setting up InfluxDB</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -42,3 +42,5 @@
>       version: 1.8.0
>     - name: usegalaxy_eu.flower
>       version: 2.1.0
>    +- src: usegalaxy_eu.influxdb
>    +  version: v6.0.7
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Create a new playbook, `monitoring.yml` with the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/monitoring.yml
>    @@ -0,0 +1,4 @@
>    +- hosts: monitoring
>    +  become: true
>    +  roles:
>    +    - usegalaxy_eu.influxdb
>    {% endraw %}
>    ```
>    {: data-commit="Setup the monitoring playbook"}
>
>    During this tutorial we will install everything on the same host, but often one keeps the monitoring infrastructure (Grafana, InfluxDB, Sentry) on a separate host.
>
> 4. Edit the inventory file (`hosts`) an add a group for monitoring like:
>
>    {% raw %}
>    ```diff
>    --- a/hosts
>    +++ b/hosts
>    @@ -4,3 +4,5 @@ gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>     galaxyservers
>     [pulsarservers]
>     gat-0.oz.galaxy.training ansible_user=ubuntu
>    +[monitoring]
>    +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    {% endraw %}
>    ```
>    {: data-commit="Add the monitoring host"}
>
>    **Ensure that the hostname is the full hostname of your machine.**
>
> 4. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook monitoring.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

This will setup an InfluxDB server listening on port `:8086`. The service is currently unauthenticated but it is only listening on `localhost` so it is less of a concern. The service can be authenticated and SSL configured quite easily but that is outside the scope of this tutorial.

You can access the InfluxDB service by running the command `influx`.

```sql
$ influx
Connected to http://localhost:8086 version 1.7.7
InfluxDB shell version: 1.7.7
> show databases
name: databases
name
----
_internal
> use _internal
Using database _internal
> show measurements
name: measurements
name
----
cq
database
httpd
queryExecutor
runtime
shard
subscriber
tsm1_cache
tsm1_engine
tsm1_filestore
tsm1_wal
write
```

The `influx` command provides command line access to InfluxDB in a similar fashion to `psql` for Postgresql. It provides commands like `show databases` and others, but we will not use this interface very often. Telegraf will automatically try to create any database needed, and no interaction is required to setup Grafana to talk to the database.

## Telegraf

We use [Telegraf](https://github.com/influxdata/telegraf) for monitoring as it is incredibly easy to get started with, and it natively integrates with InfluxDB.

### Data Input

Telegraf has extensive documentation on how to configure different types of monitoring, and it supports [a huge array of inputs](https://github.com/influxdata/telegraf#input-plugins). If Telegraf doesn't support a specific data source you wish to query, you can write a bash script to query this data, and have Telegraf [execute it regularly](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec). Telegraf supports several text formats here but the easiest to manage is the InfluxDB line protocol format. The InfluxDB format looks like:


```
weather,country=germany,city=freiburg temperature=25,wind=0 1453832006274169688
weather,country=usa,city=state-college temperature=33,wind=10 1453832006274169688
```

The first portion is the metric name, in both of these cases `weather`. This is just a unique key for a metric which describes the data being collected. Telegraf commonly collects metrics like `cpu`, or `disk`, or `mem`. After the metric name, up to the first space are tags. Tags are used to store categorical data usually, something that is [discrete](https://en.wikipedia.org/wiki/Continuous_or_discrete_variable) and [enumerable](https://en.wikipedia.org/wiki/Enumeration). If you're interested in the weather you would collect some information from a particular city, and then *tag* these values with information about where they're collected. After the space, are the various values or data points that are of interest. For weather this would be information like temperature, wind speed or direction, percentage cloud coverage, etc. After the final space is a timestamp, formatted as [Unix epoch](https://en.wikipedia.org/wiki/Unix_time) with nanosecond precision.

To apply this example to real world data, we'll look at the some example output of the [`disk` plugin](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/disk), which measures information about disks mounted to a server:


```
disk,device=devtmpfs,fstype=devtmpfs,host=stats.galaxyproject.eu,mode=rw,path=/dev free=952795136i,inodes_free=232283i,inodes_total=232616i,inodes_used=333i,total=952795136i,used=0i,used_percent=0 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/dev/shm free=963690496i,inodes_free=235275i,inodes_total=235276i,inodes_used=1i,total=963690496i,used=0i,used_percent=0 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/run free=862007296i,inodes_free=234817i,inodes_total=235276i,inodes_used=459i,total=963690496i,used=101683200i,used_percent=10.551437460684472 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=ro,path=/sys/fs/cgroup free=963690496i,inodes_free=235260i,inodes_total=235276i,inodes_used=16i,total=963690496i,used=0i,used_percent=0 1564485157000000000
disk,device=vda2,fstype=xfs,host=stats.galaxyproject.eu,mode=rw,path=/ free=6652604416i,inodes_free=6134444i,inodes_total=6290368i,inodes_used=155924i,total=12872298496i,used=6219694080i,used_percent=48.31844197780791 1564485157000000000
disk,device=vdb,fstype=xfs,host=stats.galaxyproject.eu,mode=rw,path=/vdb free=2013384704i,inodes_free=1044728i,inodes_total=1048576i,inodes_used=3848i,total=2136997888i,used=123613184i,used_percent=5.78443173454367 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/run/user/0 free=192741376i,inodes_free=235275i,inodes_total=235276i,inodes_used=1i,total=192741376i,used=0i,used_percent=0 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/run/user/1000 free=192741376i,inodes_free=235275i,inodes_total=235276i,inodes_used=1i,total=192741376i,used=0i,used_percent=0 1564485157000000000

```

Reformatting one of the lines to be a bit easier to read:

```
disk,
	device=vdb,
	fstype=xfs,
	host=stats.galaxyproject.eu,
	mode=rw,
	path=/vdb

	free=2013384704i,
	inodes_free=1044728i,
	inodes_total=1048576i,
	inodes_used=3848i,
	total=2136997888i,
	used=123613184i,
	used_percent=5.78443173454367

	1564485157000000000
```

The plugin generates a line of output per disk. It is tagged with the `fstype`, the type of the filesystem (e.g. `ext3`/`ext4`/`xfs`/`autofs`/etc.), the mode by which the filesystem was mounted (`rw` or `ro`), and the path to which the filesystem was mounted. Then various numbers are collected, the `i` suffix meaning an [integer](https://en.wikipedia.org/wiki/Integer). Those without a suffix are [floats](https://en.wikipedia.org/wiki/Floating-point_arithmetic). In this example, you can see that it is an `xfs` formatted device, available as vdb, mounted read-write on the path /vdb. Telegraf automatically tags values with the name of the host from which the data was collected.

The values collected usually have quite self-explanatory names. Here there are `2013384704` bytes free on the disk, 1044728 [inodes](https://en.wikipedia.org/wiki/Inode), so many bytes total and used, and finally 5.784% disk usage.

### `gxadmin`

In the Galaxy world, the InfluxDB line protocol format and [exec](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec) plugin are commonly seen together in conjunction with [gxadmin](https://github.com/usegalaxy-eu/gxadmin) to run various database queries, and store the results into InfluxDB.

`gxadmin` provides various commands to inspect the database like `gxadmin query queue-detail`. It was written such that many of the queries can be automatically formatted for consumption by InfluxDB:

```
$ gxadmin iquery queue-overview --short-tool-id
queue-overview,tool_id=__SET_METADATA__,tool_version=1.0.1,state=new,handler=handler_main_9,destination_id=unknown,job_runner_name=unknown count=3
queue-overview,tool_id=iuc/mothur_pre_cluster/mothur_pre_cluster,tool_version=1.39.5.0,state=new,handler=handler_main_2,destination_id=unknown,job_runner_name=unknown count=1
```

If Telegraf runs this command in an exec block, the data will then be available for graphing. Here the only numeric value we're collecting is the *count* of how many jobs of that type are currently running.

```
queue-overview,
	tool_id=__SET_METADATA__,
	tool_version=1.0.1,
	state=new,
	handler=handler_main_9,
	destination_id=unknown,
	job_runner_name=unknown

	count=3
```

We capture information about what tool is running, the job state, and where it is running. This will allow us to produce nice graphs of the current queue status, how many jobs are new or queued or running.


### Configuring Telegraf

Setting up Telegraf is again very simple. We just add a single role to our playbook and set some variables.


> <hands-on-title>Dependencies</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -44,3 +44,7 @@
>       version: 2.1.0
>     - src: usegalaxy_eu.influxdb
>       version: v6.0.7
>    +# Monitoring
>    +- name: dj-wasabi.telegraf
>    +  src: https://github.com/dj-wasabi/ansible-telegraf
>    +  version: 6f6fdf7f5ead491560783d52528b79e9e088bd5b
>    {% endraw %}
>    ```
>    {: data-commit="Add Telegraf requirement"}
>
> 2. Install the requirements
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add an entry to the *end* of your `galaxy.yml` playbook under `roles:`
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -49,6 +49,7 @@
>         - usegalaxy_eu.rabbitmqserver
>         - galaxyproject.gxadmin
>         - galaxyproject.cvmfs
>    +    - dj-wasabi.telegraf
>       post_tasks:
>         - name: Setup gxadmin cleanup task
>           ansible.builtin.cron:
>    {% endraw %}
>    ```
>    {: data-commit="Add telegraf to the monitoring playbook"}
>
> 4. Create and edit `group_vars/all.yml` and add the following variables:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/all.yml
>    +++ b/group_vars/all.yml
>    @@ -20,3 +20,28 @@ galaxy_job_metrics_plugins:
>       - type: env
>       - type: cgroup
>       - type: hostname
>    +
>    +# Telegraf
>    +telegraf_agent_package_state: latest
>    +
>    +# Configure the output to point to an InfluxDB
>    +# running on localhost, and # place data in the
>    +# database "telegraf" which will be created if need be.
>    +telegraf_agent_output:
>    +  - type: influxdb
>    +    config:
>    +    - urls = ["http://127.0.0.1:8086"]
>    +    - database = "telegraf"
>    +
>    +# The default plugins, applied to any telegraf-configured host
>    +telegraf_plugins_default:
>    +  - plugin: cpu
>    +  - plugin: disk
>    +  - plugin: kernel
>    +  - plugin: processes
>    +  - plugin: diskio
>    +  - plugin: mem
>    +  - plugin: system
>    +  - plugin: swap
>    +  - plugin: net
>    +  - plugin: netstat
>    {% endraw %}
>    ```
>    {: data-commit="Setup telegraf's variables"}
>
>    This configures telegraf to output to the configured influxdb server in the `telegraf` database. A number of plugins are enabled as `defaults` like cpu or disk or memory, all of which is generically interesting to observe, across every host.
>
>    Any host that we setup Telegraf on, will have this base configuration. If you are setting up multiple hosts to be monitored, you will need to put the full hostname (or IP address) where InfluxDB will be available in the `telegraf_agent_output`, rather than `127.0.0.1`. That way every host with Telegraf will send data to the correct location.
>
> 5. Now with the generic configuration applied to all of our hosts, we will apply some specific configuration to the Galaxy server.
>
>    Open your `group_vars/galaxyservers.yml` file, and add the following variables at the end not under any category:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -331,3 +331,12 @@ flower_ui_users:
>
>     flower_environment_variables:
>       GALAXY_CONFIG_FILE: "{{ galaxy_config_file }}"
>    +
>    +# Telegraf
>    +telegraf_plugins_extra:
>    +  listen_galaxy_routes:
>    +    plugin: "statsd"
>    +    config:
>    +      - service_address = ":8125"
>    +      - metric_separator = "."
>    +      - allowed_pending_messages = 10000
>    {% endraw %}
>    ```
>    {: data-commit="Add extra monitoring for Galaxy"}
>
>    We have configured the `statsd` plugin for telegraf, as we will use it to receive Galaxy timing data. [StatsD](https://github.com/statsd/statsd) was an earlier time series database and had an associated line protocol with a different format. Telegraf supports data sent in this format, allowing us to reuse the long-present Galaxy support for this with our newer Telegraf/InfluxDB setup. Telegraf parses the data and converts it into a format that InfluxDB can understand.
>
> 5. Lastly, we need to enable Galaxy to send data to Telegraf:
>
>    In `group_vars/galaxyservers.yml`, edit the `galaxy_config` block, and add `statsd_host: localhost` and `statsd_influxdb: true` under the `galaxy` subsection. It should look like:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -116,6 +116,9 @@ galaxy_config:
>         celery_conf:
>           result_backend: "redis://localhost:6379/0"
>         enable_celery_tasks: true
>    +    # Monitoring
>    +    statsd_host: localhost
>    +    statsd_influxdb: true
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add extra monitoring for Galaxy"}
>
> 6. Run the `galaxy.yml` playbook
>
{: .hands_on}

> ```bash
> 3.sh
> ```
> {: data-test="true"}
{: .hidden}



# Monitoring with Grafana

## Grafana

[Grafana](https://grafana.com/) provides a visual interface to our metrics. It includes a nice query builder that provides a uniform experience across multiple backend databases, along with many attractive graphing and other visualization options. Each page in the Grafana webserver display is called a "dashboard." Dashboards can each have multiple visualizations and graphs, all responding to the data collected by InfluxDB. Another benefit of using Grafana is that many of the UseGalaxy.\* servers share their dashboards publicly, and you can easily copy these and use them on your own server.

There are some nice examples of dashboards available from the public Galaxies, we recommend that you peruse them to get an idea of the possibilities:

- [UseGalaxy.eu](https://stats.galaxyproject.eu/)
- [UseGalaxy.org.au](https://stats.genome.edu.au/)

> <hands-on-title>Setting up Grafana</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -48,3 +48,5 @@
>     - name: dj-wasabi.telegraf
>       src: https://github.com/dj-wasabi/ansible-telegraf
>       version: 6f6fdf7f5ead491560783d52528b79e9e088bd5b
>    +- src: cloudalchemy.grafana
>    +  version: 0.14.2
>    {% endraw %}
>    ```
>    {: data-commit="Add grafana requirement"}
>
> 2. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add `cloudalchemy.grafana` to your `monitoring.yml` playbook:
>
>    {% raw %}
>    ```diff
>    --- a/monitoring.yml
>    +++ b/monitoring.yml
>    @@ -2,3 +2,4 @@
>       become: true
>       roles:
>         - usegalaxy_eu.influxdb
>    +    - cloudalchemy.grafana
>    {% endraw %}
>    ```
>    {: data-commit="Add grafana to monitoring playbook"}
>
> 4. Edit the file `group_vars/monitoring.yml` and set the following variables:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/monitoring.yml
>    @@ -0,0 +1,17 @@
>    +grafana_url: "https://{{ inventory_hostname }}/grafana/"
>    +
>    +grafana_security:
>    +    # Please change at least the password to something more suitable
>    +    admin_user: admin
>    +    admin_password: password
>    +
>    +# These datasources will be automatically included into Grafana
>    +grafana_datasources:
>    + - name: Galaxy
>    +   type: influxdb
>    +   access: proxy
>    +   url: http://127.0.0.1:8086
>    +   isDefault: true
>    +   version: 1
>    +   editable: false
>    +   database: telegraf
>    {% endraw %}
>    ```
>    {: data-commit="Configure Grafana"}
>
> 5. Run the monitoring playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook monitoring.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 5. Update the nginx configuration in `templates/nginx/galaxy.j2` to include the following at the end, before the last curly brace
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -109,4 +109,9 @@ server {
>     		proxy_set_header X-Forwarded-Host $host;
>     		proxy_set_header X-Forwarded-Proto $scheme;
>     	}
>    +
>    +	location /grafana/ {
>    +		proxy_pass http://127.0.0.1:3000/;
>    +		proxy_set_header Host $http_host;
>    +	}
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Setup nginx location for grafana"}
>
>    Since we will setup everything on the same host, we will re-use the Nginx server we setup for Galaxy. If you had planned to run the Grafana and InfluxDB servers on a separate host, you would need to setup Nginx for this host separately.
>
> 5. Run the Galaxy playbook which includes Nginx:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

This has now deployed Grafana on your domain under `/grafana/`, with the username and password you set. The datasource, from which Grafana obtains data, is preconfigured. The Grafana web application will now be available, but currently there is no data available to it. We will return to Grafana shortly in the tutorial to configure dashboards once data is present.

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}

The stats have been collecting in InfluxDB for a few minutes, so now we will now configure Grafana with dashboards and alerting rules.

## Importing a dashboard

For any public Grafana dashboard, you can copy the dashboard for your own use. This is a nice feature of Grafana that has really helped it spread in the Galaxy community, any cool thing one of us builds, everyone else can copy and build upon.

> <hands-on-title>Import a dashboard</hands-on-title>
>
> 1. [Visit UseGalaxy.eu's Node Detail dashboard](https://stats.galaxyproject.eu/d/000000023/node-detail-infrastructure?orgId=1)
>
> 2. Look for the sharing icon at the top and click it
>
> 3. Under the "Export" tab, click "Save to file"
>
> 4. On your own Grafana server, on the home page, hover over the `+` icon and use "Import" from the menu.
>
> 5. Click "Upload .json file" and select the json dashboard you downloaded
>
> 6. Click "Import".
>
{: .hands_on}

With this, your first dashboard should be live! You should see some data from your Galaxy instance, like CPU/load/memory/etc. This can give you a nice `htop` like view into your systems, all collected in one easy dashboard. At the top you will see a box labelled "Host" with a dropdown. If you have more systems, you can click here to select between different machines.

## Setting up a Galaxy dashboard

Importing dashboards is a good start, but it's more interesting to create our own that's personalised to our needs.

> <hands-on-title>Create a dashboard</hands-on-title>
>
> 1. Again find the `+` icon in Grafana and create a dashboard. This will bring you to a new screen
>
> 2. Click *Add Query*, and you will be dropped into the Grafana query builder
>
>    ![Grafana query builder interface](../../images/grafana-query-builder.png)
>
>    This is the query builder interface. The interface somewhat resembles a SQL query, selecting data *from* a database, *where* it meets some condition, *select*ing some specific data, and *grouping by* time period. If this isn't immediately clear how it behaves, hopefully it will become more clear once you have built some queries.
>
> 3. Let's build a query:
>    - From:
>      - *"select measurement"*: `galaxy.`
>    - Select:
>      - *"field(value)"*: `field(mean)`
>    - Group by:
>      - *"fill(null)"*: `fill(none)`
>      - add new (+): `tag(path)`
>    - Alias by: `[[tag_path]]`
>
> 4. At the top of the page it probably says "Last 6 hours", click this to change to "Last 30 minutes"
>
> 5. Remember to save the dashboard (using the {% icon galaxy-save %} at the top), and give it a name like "Galaxy"
>
> 6. You can hit <kbd>Escape</kbd> to exit out of the graph editor, if need be.
>
{: .hands_on}

This will track how long it takes the interface to respond on various web routes and API routes. The collection of individual points is a bit hard to interpret the "feeling" of, so it's common to add a query like the [95th percentile](https://en.wikipedia.org/wiki/Percentile) of requests. This is a value that is calculated from all of the data points collected. The 95th percentile means that 95% of requests are responded to more quickly than this value.

> <hands-on-title>Add a second query to an existing graph</hands-on-title>
>
> 1. The top of the graph is probably labelled "Panel Title", unless you changed it. Click this to access a dropdown and click "Edit"
>
> 2. On the right side you will find a button **Add Query**, click it.
>
> 3. Let's build a query:
>    - From:
>      - *"select measurement"*: `galaxy.`
>    - Select:
>      - *"field(value)"*: `field(mean)`
>      - add new (+): Selectors → percentile
>    - Group by:
>      - *"fill(null)"*: `fill(none)`
>    - Alias by: `percentile`
>
> 4. Remember to save the dashboard
>
> 5. You can hit <kbd>Escape</kbd> to exit out of the graph editor
>
> 6. At any time you can drag the bottom right corner of the graph to resize it as needed.
>
{: .hands_on}

We should now have a graph that gives us not only individual data points, but also a more easily consumable overall representation. For many metrics there are so many data points individually collected that it can be overwhelming and so it is often useful to come up with an aggregate representation that summarises the points into a more easily consumed value. We will touch upon this again under the monitoring section.

## Styling

There is a significant amount of visual styling that one can do to the graphs to make data more or less prominent as you need.

![Grafana graph configuration sections](../../images/grafana-graph-sections.png "Graph configuration sections, from top to bottom: data, visualisation, configuration, and alerting")

We will update the panel we've added to highlight the important information and downplay less important facets, as well as configuring it to have a nicer title than "Panel Title"

> <hands-on-title>Styling the graph</hands-on-title>
>
> 1. Again edit the one graph we've added in our dashboard
>
> 2. On the left side, select the second icon, Visualisation
>
> 3. In the first section, we can edit some display attributes.
>
>    We are primarily interested in the 95th percentile line. Here we can change the display from points to lines or bars, change the tooltip that displays when we hover over the graph, among other facets.
>
>    - Click **Add series override**, click in the box that appears and type "percentile", to find the series we created early. Select it when it appears. We can now apply custom styling to this series.
>    - Click the "+" after the series override, and find **"Color → change"**, and choose a colour that you like.
>    - Click the "+" again, and find **"Line width → 5"** to make the line more visible. This is the most important facet of the graph for us, so we can emphasise it through colour and size.
>
> 3. In the second section, we can edit information about the Axes:
>
>    - Left Y
>      - *"Unit"*: Time → millisecond (ms)
>
>    This will cause the axis to display nicely at any scale. Before Grafana only knew it was a number, now it knows the type of number. If you have a request that takes 30 seconds, it will display as "30s" in Grafana, rather than "30000 ms".
>
>    This section is also useful for setting the scale if your data is better viewed with a logarithmic scale, or forcing specific axis bounds. Commonly on graphs that show things like "disk usage percentage", it can be useful to set the axis bounds to 0 and 100 to ensure that full context is available.
>
> 4. In the next section, "Legend" allows us to format the legend in more interesting ways.
>
>    - Options
>      - *"Show"*: yes
>      - *"As table"*: yes
>      - *"To right"*: yes
>    - Values
>      - *"Min"*: yes
>      - *"Max"*: yes
>      - *"Avg"*: yes
>
>    Clicking on the headers of the table that has appeared, you can force it to be sorted by one column or another.
>
> 5. On the left side, select the third icon, "General"
>
>    - *"Title"*: `Galaxy Request Times`
>
> 6. Save the dashboard
>
{: .hands_on}

Your graph should look something like the following:

![Final graph](../../images/grafana-graph-final.png "The final graph should look something like this, but probably different based on the requests to your server. For this image the author made repeated, automated requests to Galaxy to produce more data points than might normally be present.")

## Monitoring

Collecting all of this data is interesting to visualise but as an administrator you surely have more interesting things to do than to watch graphs all day. Many organisations like to display these dashboards on large monitors, but again this assumes someone is watching it. Everyone has better things to do with their time! So we will setup monitoring on the most important aspects of our system.

Doing monitoring effectively, without causing undue burden to the administrators (extraneous alerts that are not actionable), or the users (unexpected/unnoticed downtime), is a very complex topic. Recommended reading here includes the [monitoring chapter](https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/) of the Google SRE book {% cite Beyer:2016:SRE:3006357 %} which can provide some general guidance on the topic and what metrics may be interesting or annoying to alert upon.

> <comment-title>No generic advice</comment-title>
> We cannot easily provide generic and applicable recommendations, that work across every system and every scale. Some of these performance bounds or features you will need to discover yourself, either adding new metrics in support of this, or changing monitoring thresholds to match the values you need.
{: .comment }

We will add an example alert, to make you familiar with the process. This is not an alert that will probably be useful in production.

> <hands-on-title>Add an alert to your graph</hands-on-title>
>
> 1. Again edit the `Galaxy Request Times` graph
>
> 2. On the left side, select the last icon, Alerting
>
> 3. Click **Create Alert**
>
>    Alerts consist of a *Rule*, with some name, evaluated every N seconds, for a period of time. The *for* can be an important parameter, which you can read more about [in the Grafana documentation](https://grafana.com/docs/alerting/rules/).
>
>    Under some *conditions*, this alert will activate. We will change the conditions of the alert here:
>
>    - *"When"*: `avg()`
>    - *"OF"*: `query(B, 1m, now)`, here we select query B, the 95th percentile track, and the average over 1 minute
>    - *"IS ABOVE"*: `50`
>
>    We will not configure a notification channel in this tutorial. In practice, it is useful to do to ensure that the relevant people are notified automatically. If you and your team use [one of the many services](https://grafana.com/docs/alerting/notifications/), then you can configure this following their documentation.
>
> 4. Save the dashboard
>
{: .hands_on}

# Telegraf & `gxadmin`

Via this setup using `systemd` we collect metrics about Galaxy request times. To get statistics about other Galaxy-specific metrics such as the job queue status, we need to use `gxadmin` to query the Galaxy database and configure Telegraf to consume this data. In this section we will setup gxadmin, and to configure Telegraf to have permissions to run it.


## Installing gxadmin

It's simple to install gxadmin. Here's how you do it, if you haven't done it already:

> <hands-on-title>Installing gxadmin and configuring Telegraf</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    ```yml
>    - src: galaxyproject.gxadmin
>      version: 0.0.12
>    ```
>
> 2. Install the role with `ansible-galaxy install -p roles -r requirements.yml`
>
> 3. Add the role to your `galaxy.yml` playbook, it should come before the Telegraf role.
>
{: .hands_on}

You can run the playbook now, or wait until you have configured Telegraf below:

## Configuring Telegraf for gxadmin

> <hands-on-title>Configuring Telegraf</hands-on-title>
>
> 1. Edit the `group_vars/galaxyservers.yml`, we need to add some additional permissions to permit Telegraf to run `gxadmin`:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/dbservers.yml
>    +++ b/group_vars/dbservers.yml
>    @@ -2,9 +2,16 @@
>     # PostgreSQL
>     postgresql_objects_users:
>       - name: "{{ galaxy_user_name }}"
>    +  - name: telegraf
>     postgresql_objects_databases:
>       - name: "{{ galaxy_db_name }}"
>         owner: "{{ galaxy_user_name }}"
>    +postgresql_objects_privileges:
>    +  - database: galaxy
>    +    roles: telegraf
>    +    privs: SELECT
>    +    objs: ALL_IN_SCHEMA
>    +
>
>     # PostgreSQL Backups
>     postgresql_backup_dir: /data/backups
>    {% endraw %}
>    ```
>    {: data-commit="Configure database permissions for Telegraf"}
>
> 2. Again edit the `group_vars/galaxyservers.yml`, we need to configure Telegraf to run `gxadmin`
>
>    Under `telegraf_plugins_extra`, where we already have set a Galaxy StatsD listener, add a stanza to monitor the Galaxy queue
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -343,3 +343,10 @@ telegraf_plugins_extra:
>           - service_address = ":8125"
>           - metric_separator = "."
>           - allowed_pending_messages = 10000
>    +  monitor_galaxy_queue:
>    +    plugin: "exec"
>    +    config:
>    +      - commands = ["/usr/bin/env PGDATABASE=galaxy /usr/local/bin/gxadmin iquery queue-overview --short-tool-id"]
>    +      - timeout = "10s"
>    +      - data_format = "influx"
>    +      - interval = "15s"
>    {% endraw %}
>    ```
>    {: data-commit="Add extra monitoring for Galaxy"}
>
>    This one is slightly more complex in the configuration. The command block does several things:
>
>    - it wraps the command with `env` which allows setting environment variables for a single command
>    - It sets the `PGDATABASE` to the Galaxy database, by default the `psql` will try and connect to a database with the same name of the user. So the `telegraf` user will attempt to connect to a (non-existent) `telegraf` database.
>    - Then it calls the gxadmin command `queue-overview`. By using `iquery` instead of `query`, the output is automatically converted to InfluxDB line protocol.
>    - The command is run every 15 seconds, and has a timeout of 10 seconds. If the command fails to finish in 10 seconds, it will be killed.
>
> 3. Run the Galaxy playbook
>
{: .hands_on}

With this, Telegraf will start monitoring the Galaxy queue by calling the query every few seconds to check the status every 15 seconds. This monitoring *will* miss jobs that complete within the 15 second interval, but for most servers this is not an issue. Most jobs are running for more than 15 seconds, and if not, it still gives an accurate point-in-time view.

We'll now create a graph for this, just like the one on [stats.galaxyproject.eu](https://stats.galaxyproject.eu)

> <hands-on-title>Building the queue graph</hands-on-title>
>
> 1. Click the **new graph** button at the top of Grafana's interface, and *Add a Query*
>
> 2. Let's build a query:
>    - From:
>      - *"select measurement"*: `queue-overview`
>    - Select:
>      - *"field(value)"*: `field(count)`
>      - add new (+): Aggregations → sum
>    - Group by:
>      - *"time(__interval)"*: `time(15s)`, because we set the interval to 15s
>      - add new (+): `tag(tool_id)`
>      - add new (+): `tag(tool_version)`
>    - Alias by: `[[tag_tool_id]]/[[tag_tool_version]]`
>
> 3. In the second tab on the left, Visualisation:
>
>    Under the first section:
>
>    - Draw Modes:
>      - *"Bars"*: `no`
>      - *"Lines"*: `yes`
>      - *"Points"*: `no`
>    - Mode Options:
>      - *"Staircase"*: `yes`
>    - Stacking & Null Value
>      - *"Stack"*: `yes`
>      - *"Null Value"*: `null as zero`
>
>    Below, under the *Legend* section,
>
>    - Options:
>      - *"Show"*: `yes`
>      - *"As table"*: `yes`
>      - *"To Right"*: `yes`
>    - Values:
>      - *"Max"*: `yes`
>      - *"Avg"*: `yes`
>      - *"Current"*: `yes`
>    - Hide Series:
>      - *"With only nulls"*: `yes`
>      - *"With only zeros"*: `yes`
>
> 4. In the third tab, General settings:
>
>    - *"Title"*: `Galaxy Queue Overview`
>
> 5. You can hit <kbd>Escape</kbd> to exit out of the graph editor, and remember to save your dashboard.
>
{: .hands_on}

Run some tools in Galaxy, try to generate a large number of jobs. It is relatively easy to upload a dataset, and then run the "Secure Hash / Message Digest" or another tool repeatedly, running it over every dataset in your history, repeating until you've generated a few dozen datasets. If you have a slower tool like `bwa` installed, this can be an option too.


![Final Graph](../../images/grafana-final-graph.png "Final graphs generated from this tutorial.")

You can also import a [copy of the dashboard]({{ site.baseurl }}{{ page.dir }}dashboard.json).

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=13 %}

# Conclusion

Monitoring with Telegraf, InfluxDB, and Grafana can provide an easy solution to monitor your infrastructure. The UseGalaxy.\* servers use this stack and it has proven to be effective in production situations, with large Galaxy servers. The base monitoring done with Telegraf is easy to setup and extend on a per-site basis simply by adding scripts or commands to your servers which generate InfluxDB line protocol formatted output. Grafana provides an ideal visualisation solution as it encourages sharing, and allows you to import whatever dashboards have been developed by UseGalaxy.\*, and then to extend them to your own needs.


{% snippet topics/admin/faqs/git-gat-path.md tutorial="monitoring" %}


topic: admin
tutorial: sentry
title: Monitoring Galaxy and Pulsar with Sentry
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/sentry/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "git-gat"]
content_hash: 513d75a90f44b49181395fd5e54c8d41
last_modified: 2026-02-26T21:12:14.152445
workflows_json: []
content:
# Overview

Sentry is an error tracking software that helps admins and developers monitor and diagnose issues in their applications. It provides real-time alerts for errors and allows users to capture context information about each error, such as stack traces and user feedback. It is often possible to find and fix errors before users report them. Galaxy and Pulsar can log issues and failing tool runs to Sentry.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="sentry" %}

We're going to set up a local Sentry instance using docker-compose and connect Galaxy and Pulsar to that Sentry instance. Alternatively, you can use the hosted Sentry at https://sentry.io/.

# Installing and Configuring

To proceed from here it is expected that:

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have set up a working Galaxy instance as described in the [ansible-galaxy](../ansible-galaxy/tutorial.html) tutorial.
>
{: .comment}

# Installing and Configuring

First we need to add our new Ansible role to `requirements.yml`:

> <hands-on-title>Set up Sentry with Ansible</hands-on-title>
>
> 1. In your working directory, add the roles to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -53,3 +53,6 @@
>     # Training Infrastructure as a Service
>     - src: galaxyproject.tiaas2
>       version: 2.1.5
>    +# Sentry
>    +- name: mvdbeek.sentry_selfhosted
>    +  src: https://github.com/mvdbeek/ansible-role-sentry/archive/main.tar.gz
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement" data-ref="add-req"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the roles with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true" data-ref="req-install"}
>    {: .code-in}
>
> 3. Create a new playbook, `sentry.yml` with the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/sentry.yml
>    @@ -0,0 +1,7 @@
>    +- hosts: sentryservers
>    +  become: true
>    +  pre_tasks:
>    +    - pip:
>    +        name: docker-compose
>    +  roles:
>    +    - mvdbeek.sentry_selfhosted
>    {% endraw %}
>    ```
>    {: data-commit="Setup the sentry playbook"}
>
>    During this tutorial we will install everything on the same host, but often one keeps the monitoring infrastructure (Grafana, InfluxDB, Sentry) on a separate host.
>
> 4. Edit the inventory file (`hosts`) an add a group for Sentry like:
>
>    {% raw %}
>    ```diff
>    --- a/hosts
>    +++ b/hosts
>    @@ -6,3 +6,6 @@ galaxyservers
>     gat-0.oz.galaxy.training ansible_user=ubuntu
>     [monitoring]
>     gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    +
>    +[sentryservers]
>    +gat-0.eu.training.galaxyproject.eu ansible_connection=local ansible_user=ubuntu
>    {% endraw %}
>    ```
>    {: data-commit="Add the monitoring host"}
>
>    **Ensure that the hostname is the full hostname of your machine.**
>
>    Sentry requires its own (sub)domain. For the admin training we have set up the sentry.gat-N.eu.galaxy.training subdomain. If you run this tutorial outside of the training and you cannot obtain a domain or subdomain for sentry you can use the free [Duck DNS](https://www.duckdns.org/) service to map an IP address to a domain name.
>
> 5. Edit the file `group_vars/sentryservers.yml` and set the following variables:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/all.yml
>    +++ b/group_vars/all.yml
>    @@ -45,3 +45,6 @@ telegraf_plugins_default:
>       - plugin: swap
>       - plugin: net
>       - plugin: netstat
>    +
>    +# Used in galaxyservers, nginx and sentry
>    +sentry_domain: "sentry.{{ inventory_hostname }}"
>    diff --git a/group_vars/galaxyservers.yml b/group_vars/galaxyservers.yml
>    index 1bdac54..94b3fc3 100644
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -211,6 +211,7 @@ certbot_post_renewal: |
>         docker restart rabbit_hole || true
>     certbot_domains:
>      - "{{ inventory_hostname }}"
>    + - "{{ sentry_domain }}"
>     certbot_agree_tos: --agree-tos
>
>     # NGINX
>    diff --git a/group_vars/sentryservers.yml b/group_vars/sentryservers.yml
>    new file mode 100644
>    index 0000000..36298e3
>    --- /dev/null
>    +++ b/group_vars/sentryservers.yml
>    @@ -0,0 +1,6 @@
>    +sentry_version: 23.3.1
>    +sentry_url: "https://{{ sentry_domain }}"
>    +sentry_docker_compose_project_folder: /srv/sentry
>    +sentry_superusers:
>    +  - email:  admin@example.com
>    +    password: "{{ vault_sentry_password }}"
>    diff --git a/sentry.yml b/sentry.yml
>    index 285910b..ab32a34 100644
>    --- a/sentry.yml
>    +++ b/sentry.yml
>    @@ -1,7 +1,6 @@
>     - hosts: sentryservers
>    +  vars_files:
>    +    - group_vars/secret.yml
>       become: true
>    -  pre_tasks:
>    -    - pip:
>    -        name: docker-compose
>       roles:
>         - mvdbeek.sentry_selfhosted
>    {% endraw %}
>    ```
>    {: data-commit="Configure Sentry"}
>
> 9. We will add an associated admin password to the vault, do that now:
>
>    ><code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_sentry_password: 'some-super-secret-password'
>    ```
>
>    <!-- Ignore this, just for the gat-automation. Vaults are ugly to work with :(
>
>    +vault_sentry_password: password
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/secret.yml
>    +++ b/group_vars/secret.yml
>    @@ -1,22 +1,23 @@
>     $ANSIBLE_VAULT;1.1;AES256
>    -66653366633665383231303635643739396466653465626163633662623230643534346666303434
>    -3531356339646666623364383435306363373534373833370a333964306130653236373438373264
>    -34366433396135303064643932313135643064373034323865363939333565623238386161333465
>    -3932623162353034370a653663333363383938393936623663343639376430653366646662353830
>    -34383964326161383966306361633162653366353162633564623738353137333936313232363764
>    -38383962396637636136316366633936316365643038333030333932336365326335373737663834
>    -39363639303764343430336435313663313762353335366562656232646334356438343535303032
>    -35636131323163323036383163363964643237313137333131303737346662373233663562343031
>    -39396531306539623661376166313534303462623532323334393736316634616262313633626637
>    -37393066663239303564356232366466333334316565363631363230626665333039386133383933
>    -31656234363064646334623938383637393534313638643635396662643163346633346237353664
>    -34666161373764663166653737323736373261363565666633653831363833393264333666356633
>    -30373062333638366666316132623736386336383933326539663065653538386363323766313664
>    -63323737396264306439303366343834326164393763376663316366353766663131303966393039
>    -61343732313865613761623733356465336263633963376161663931653864383162393839643834
>    -34373239386461666462643162613536316166656135323332316163336635393035653164363138
>    -34336265383033663436306436663536383238366665653239313661666431666339323364646633
>    -64366361653034393962316562373835623631356339313062393933633634653735656636373031
>    -65376162633866623733623961303664663931636130373936326461623465363932373165356563
>    -36323838323865326464346337393164343765613333663830636234666663303535616532323936
>    -3230
>    +31383961303162656266343866376462343961326163356238636565386630363164353330643862
>    +3631663631383136356436363635373536323536353534650a393833343465396339373731663238
>    +38383763326230346561633061356634643566663863383838363765326162386133636330656136
>    +3935643333353365660a393932313965373166313838373739306234306634313139656334343839
>    +38393636396164373266646534326436316333313638366234323564323936643863633533366537
>    +66393535333263383837386538636530303137623761333161383765643938396232633863353838
>    +66363464613466353833663833613339666634333734343163326162393031363530643363396238
>    +36306562333763333435646432613565323537373032643335336639353762646662343865663266
>    +33646334623838333230343861353632626461363436343963636664393965613736306162396235
>    +64373239373739613164623165653832653139646433373531323562373534393563363962373634
>    +30393161303439376138653066646230303464353336656431663137366630323830316561356137
>    +65363663663165323666343335626238666334346664373938333839323161353131643336343738
>    +31646330386239363165643866316431386631643139613065663835323036353631646365333530
>    +37306161656562316665643666363132633766386330383864663937316233376261343264633133
>    +39356438623533376539393662326364666434386330626437646235386261323632396661656661
>    +31613232316237626562646330626564333566633266653762356338646236353137343863633634
>    +31623563393933656635323165326638376634383730393131373963353262366337383632666666
>    +64643264333463373334343433373564326632393637626237353336373837373438303231636633
>    +31666636316533313336613532643437646362636431306262623361326532613238303732396266
>    +62346131663135313935393439343433333439623335623131653066316561646631616361303632
>    +66633935356333643539323461333734376466316262656330396662346232333131313864343636
>    +6634613432653361653133383862383338353432666136303465
>    {% endraw %}
>    ```
>    {: data-commit="Add sentry passwords to the vault"}
>
>    -->
>
> 6. Add the nginx routes
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/nginx/sentry.j2
>    @@ -0,0 +1,20 @@
>    +server {
>    +	# Listen on port 443
>    +	listen        *:443 ssl;
>    +	# The virtualhost is our domain name
>    +	server_name   "{{ sentry_domain }}";
>    +
>    +	# Our log files will go here.
>    +	access_log  syslog:server=unix:/dev/log;
>    +	error_log   syslog:server=unix:/dev/log;
>    +
>    +	location / {
>    +		# This is the backend to send the requests to.
>    +		proxy_pass "http://localhost:9000";
>    +
>    +		proxy_set_header Host $http_host;
>    +		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>    +		proxy_set_header X-Forwarded-Proto $scheme;
>    +		proxy_set_header Upgrade $http_upgrade;
>    +	}
>    +}
>    {% endraw %}
>    ```
>    {: data-commit="Add nginx server"}
>
> 6. And make sure the sentry nginx configuration is deployed
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -220,6 +220,7 @@ nginx_servers:
>       - redirect-ssl
>     nginx_ssl_servers:
>       - galaxy
>    +  - sentry
>     nginx_enable_default_server: false
>     nginx_conf_http:
>       client_max_body_size: 1g
>    {% endraw %}
>    ```
>    {: data-commit="Deploy nginx server"}
>
> 7. Run the sentry playbook to deploy sentry and the galaxy playbook to update the nginx configuration.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook sentry.yml galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 8. Generate a project for Galaxy in Sentry
>  Go to the domain you configured for your Sentry instance. You need to log in with the username and admin you've set up in `group_vars/sentryservers.yml`. Click "continue" on the next page. Click "Projects", "Create Project", "Python", select "I'll create my own alerts later", and set "galaxy" as the Project Name. You'll see your project dsn that will look like `https://b0022427ee5345a8ad4cb072c73e62f4@sentry.gat-N.eu.galaxy.training/2`. We will need this string to let Galaxy know where to send data to. To avoid requesting an additional certificate for communication between Galaxy and Sentry we've set up communication via localhost:9000, so you can manually change the @ portion to localhost:9000.
>
> 9. We will add the galaxy project dsn to the vault. Edit your `group_vars/secret.yml` and add the sentry dsn.
>
>    ><code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_galaxy_sentry_dsn: 'http://b0022427ee5345a8ad4cb072c73e62f4@localhost:9000/2'
>    ```
>
> 9. Edit `group_vars/galaxyservers.yml` to reference the new vault secret:
>
>    This will let Galaxy know that captured logs should be sent to our Sentry instance.
>    We will also enable sending performance metrics to Sentry by setting the `sentry_traces_sample_rate` to `0.5`. This will send half of all transactions to Sentry. In a production environment you would reduce this to a smaller percentage of transactions.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -119,6 +119,8 @@ galaxy_config:
>         # Monitoring
>         statsd_host: localhost
>         statsd_influxdb: true
>    +    sentry_dsn: "{{ vault_galaxy_sentry_dsn }}"
>    +    sentry_traces_sample_rate: 0.5
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Configure Galaxy to report to Sentry"}
>
> 10. Run the galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on }

## Generate an error

Galaxy has a built in route that intentionally generates and error.
Just visit: [/error](https://my.gat.galaxy.training/?path=/error)

> <hands-on-title>Open the Galaxy Project in Sentry</hands-on-title>
> 1. Go to your Sentry instance and click on issues. You should see a couple of issues,
> one them should be the "Fake error" exception we generated by visiting [https://galaxy.example.org/error](https://my.gat.galaxy.training/?path=/error).
{: .hands_on }

## Sending tool error reports to Sentry

In addition to sending logging errors to Sentry you can also collect failing tool runs in Sentry. For this we will set up the error reporting configuration file and reference it in galaxy.yml. The `user_submission` parameter controls whether all reports will be collected in Sentry (when set to `false`) or only those that have been reported manually (when set to `true`). For testing purposes we'll also add a tool that will fail running so we can test that submitting tool errors to Sentry works as expected.

> <hands-on-title>Update Galaxy config to send tool error reports</hands-on-title>
>
> 1. Create the  `files/galaxy/config/error_reports.yml` file.
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/config/error_reports.yml
>    @@ -0,0 +1,2 @@
>    +- type: sentry
>    +  user_submission: false
>    {% endraw %}
>    ```
>    {: data-commit="Configure error reporting"}
>
> 2. Create a testing tool in `files/galaxy/tools/job_properties.xml`.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/tools/job_properties.xml
>    @@ -0,0 +1,65 @@
>    +<tool id="job_properties" name="Test Job Properties" version="1.0.0">
>    +    <stdio>
>    +        <exit_code range="127" level="fatal" description="Failing exit code." />
>    +    </stdio>
>    +    <version_command>echo 'v1.1'</version_command>
>    +    <command><![CDATA[
>    +#if $thebool
>    +    echo 'The bool is true' &&
>    +    echo 'The bool is really true' 1>&2 &&
>    +    echo 'This is a line of text.' > '$out_file1' &&
>    +    cp '$out_file1' '$one' &&
>    +    cp '$out_file1' '$two' &&
>    +    sleep $sleepsecs
>    +#else
>    +    echo 'The bool is not true' &&
>    +    echo 'The bool is very not true' 1>&2 &&
>    +    echo 'This is a different line of text.' > '$out_file1' &&
>    +    sleep $sleepsecs &&
>    +    sh -c 'exit 2'
>    +#end if
>    +#if $failbool
>    +    ## use ';' to concatenate commands so that the next one is run independently
>    +    ## of the exit code of the previous one
>    +    ; exit 127
>    +#end if
>    +    ]]></command>
>    +    <inputs>
>    +        <param name="sleepsecs" type="integer" value="0" label="Sleep this many seconds"/>
>    +        <param name="thebool" type="boolean" label="The boolean property" />
>    +        <param name="failbool" type="boolean" label="The failure property" checked="false" />
>    +    </inputs>
>    +    <outputs>
>    +        <data name="out_file1" format="txt" />
>    +        <collection name="list_output" type="list" label="A list output">
>    +            <data name="one" format="txt" />
>    +                <has_line line="The bool is true" />
>    +            </assert_stdout>
>    +            <assert_stderr>
>    +                <has_line line="The bool is really true" />
>    +            </assert_stderr>
>    +            <assert_command_version>
>    +                <has_text text="v1.1" />
>    +            </assert_command_version>
>    +        </test>
>    +        <test expect_exit_code="2">
>    +            <param name="thebool" value="false" />
>    +            <output name="out_file1" file="simple_line_alternative.txt" />
>    +            <assert_command>
>    +                <has_text text="very not" />
>    +            </assert_command>
>    +            <assert_stdout>
>    +                <has_line line="The bool is not true" />
>    +            </assert_stdout>
>    +            <assert_stderr>
>    +                <has_line line="The bool is very not true" />
>    +            </assert_stderr>
>    +        </test>
>    +        <test expect_exit_code="127" expect_failure="true">
>    +            <param name="thebool" value="true" />
>    +            <param name="failbool" value="true" />
>    +        </test>
>    +    </tests>
>    +    <help>
>    +    </help>
>    +</tool>
>    {% endraw %}
>    ```
>    {: data-commit="Configure a tool"}
>
> 3. Edit `group_vars/galaxyservers.yml` to reference the `error_reports.yml` file and the new testing tool.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -121,6 +121,7 @@ galaxy_config:
>         statsd_influxdb: true
>         sentry_dsn: "{{ vault_galaxy_sentry_dsn }}"
>         sentry_traces_sample_rate: 0.5
>    +    error_report_file: "{{ galaxy_config_dir }}/error_reports_file.yml"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    @@ -173,6 +174,8 @@ galaxy_config_files:
>         dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>       - src: files/galaxy/config/tpv_rules_local.yml
>         dest: "{{ tpv_mutable_dir }}/tpv_rules_local.yml"
>    +  - src: files/galaxy/config/error_reports.yml
>    +    dest: "{{ galaxy_config.galaxy.error_report_file }}"
>
>     galaxy_config_templates:
>       - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>    @@ -194,6 +197,7 @@ tpv_privsep: true
>
>     galaxy_local_tools:
>     - testing.xml
>    +- job_properties.xml
>
>     # Certbot
>     certbot_auto_renew_hour: "{{ 23 |random(seed=inventory_hostname)  }}"
>    {% endraw %}
>    ```
>    {: data-commit="Deploy files, error reporting"}
>
> 4. Run the galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on }

## Generating a tool error

To generate a tool error, run the job properties testing tool and set the `failbool` parameter to true.

> <hands-on-title>Open the Galaxy Project in Sentry</hands-on-title>
> 1. Go to your Sentry instance and click on issues. You should see an issue for the tool run error.
{: .hands_on }

## Reporting errors from the Pulsar server

It is also possible to report errors from the Pulsar server. You can either use the Galaxy project we created before in Sentry, or we can create a new project for Pulsar. We recommend creating a separate Pulsar project. Since the Pulsar server runs on a remote VM for this to work you need a valid certificate for the Sentry domain and you cannot use localhost.

> <hands-on-title>Add Sentry connection to Pulsar</hands-on-title>
> 1. Create a new dsn by creating a new pulsar project in Sentry.
> 2. We will add the project dsn to the vault. Edit your `group_vars/secret.yml` and add the sentry dsn.
>
>    ><code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_pulsar_sentry_dsn: 'https://f2a8a00d30224c2c9800a8f79194a32a@{{ groups['sentryservers'][0] }}/3'
>    ```
> 3. Add the sentry dsn to the pulsar group variables.
>    {% raw %}
>    ```diff
>    --- a/group_vars/pulsarservers.yml
>    +++ b/group_vars/pulsarservers.yml
>    @@ -44,6 +44,7 @@ pulsar_yaml_config:
>           - type: conda
>             auto_init: true
>             auto_install: true
>    +  sentry_dsn: "{{ vault_pulsar_sentry_dsn }}"
>
>     # Pulsar should use the same job metrics plugins as Galaxy. This will automatically set `job_metrics_config_file` in
>     # `pulsar_yaml_config` and create `{{ pulsar_config_dir }}/job_metrics_conf.yml`.
>    {% endraw %}
>    ```
>    {: data-commit="Configure pulsar for error reporting"}
>
> 4. Run the pulsar playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on }

Pulsar should now be set up to report errors to Sentry.

{% snippet topics/admin/faqs/missed-something.md step=12 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="sentry" %}


topic: admin
tutorial: job-destinations
title: Mapping Jobs to Destinations using TPV
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["jobs", "git-gat"]
content_hash: 3ecaaa53bd407a6dda014e2835b63380
last_modified: 2026-02-26T21:12:14.147784
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/job-destinations/tutorial.html

content:
This tutorial heavily builds on the [Connecting Galaxy to a compute cluster]({% link topics/admin/tutorials/connect-to-compute-cluster/tutorial.md %}) and it's expected you have completed this tutorial first.

Now that you have a working scheduler, we will start configuring which jobs are sent to which destinations.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="job-destinations" %}

# Mapping jobs to destinations

In order to run jobs in Galaxy, you need to assign them to a resource manager that can handle the task. This involves specifying the appropriate amount of memory and CPU cores. For production installations, the jobs
must be routed to a resource manager like SLURM, HTCondor, or Pulsar. Some tools may need specific resources such as GPUs or multi-core machines to work efficiently.

Sometimes, your available resources are spread out across multiple locations and resource managers. In such cases, you need a way to route your jobs to the appropriate location. Galaxy offers several methods for
routing jobs, ranging from simple static mappings to custom Python functions via [dynamic job destinations](https://docs.galaxyproject.org/en/latest/admin/jobs.html#dynamic-destination-mapping).

Recently, the Galaxy project has introduced a library named [{TPV}](https://total-perspective-vortex.readthedocs.io/) to simplify this process. {TPV} provides a admin-friendly YAML configuration that works for most scenarios.
For more complex cases, TPV also allows you to embed Python code into the configuration YAML file and implement fine-grained control over jobs. 

Lastly, TPV offers a shared global database of default resource requirements (more below). By leveraging this database, admins don't have to figure out the requirements for each tool
separately.

## Writing a testing tool

To demonstrate a real-life scenario and {TPV}'s role in it, let's plan on setting up a configuration where the VM designated for training jobs doesn't run real jobs and hence doesn't get overloaded. To start, we'll create a "testing" tool that we'll use in our configuration. This testing tool can run quickly, and without overloading our small machines.

> <hands-on-title>Deploying a Tool</hands-on-title>
>
> 1. Create the directory `files/galaxy/tools/` if it doesn't exist and edit a new file in `files/galaxy/tools/testing.xml` with the following contents:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/tools/testing.xml
>    @@ -0,0 +1,11 @@
>    +<tool id="testing" name="Testing Tool">
>    +    <command>
>    +        <![CDATA[echo "Running with '\${GALAXY_SLOTS:-1}' threads" > "$output1"]]>
>    +    </command>
>    +    <inputs>
>    +        <param name="input1" type="data" format="txt" label="Input Dataset"/>
>    +    </inputs>
>    +    <outputs>
>    +        <data name="output1" format="txt" />
>    +    </outputs>
>    +</tool>
>    {% endraw %}
>    ```
>    {: data-commit="Add testing tool"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Add the tool to the Galaxy group variables under the new item `galaxy_local_tools` :
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -162,6 +162,9 @@ galaxy_config_templates:
>     galaxy_extra_dirs:
>       - /data
>
>    +galaxy_local_tools:
>    +- testing.xml
>    +
>     # Certbot
>     certbot_auto_renew_hour: "{{ 23 |random(seed=inventory_hostname)  }}"
>     certbot_auto_renew_minute: "{{ 59 |random(seed=inventory_hostname)  }}"
>    {% endraw %}
>    ```
>    {: data-commit="Deploy testing tool"}
>
> 3. Run the Galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. Reload Galaxy in your browser and the new tool should now appear in the tool panel. If you have not already created a dataset in your history, upload a random text dataset. Once you have a dataset, click the tool's name in the tool panel, then click Execute.
>
>    > <question-title></question-title>
>    >
>    > What is the tool's output?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > Running with '1' threads
>    > > ```
>    > >
>    > {: .solution }
>    >
>    {: .question}
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

Of course, this tool doesn't actually *use* the allocated number of cores. In a real tool, you would call the tools's underlying command with whatever flag that tool provides to control the number of threads or processes it starts, such as `samtools sort -@ \${GALAXY_SLOTS:-1}`.

## Safeguard: TPV Linting

If we want to change something in production, it is always a good idea to have a safeguard in place. In our case, we would like to check the {TPV} configuration files for syntax errors, so nothing will break when we deploy broken yaml files or change them quickly on the server. TPV-lint-and-copy works with two separate locations:
- one where you can safely edit your files
- and the actual production config directory that galaxy reads.

Once you are done with your changes, you can run the script and it will automatically lint and copy over the files, if they are correct *and* mentioned in your job_conf.yml file, or in your `group_vars/galaxyservers.yml` inline `job_conf`.
And of course, Galaxy has an Ansible Role for that.

> <hands-on-title>Adding automated TPV-lind-and-copy-script</hands-on-title>
>
> 1. Add the role to your `requirements.yml`.
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -27,3 +27,6 @@
>       version: 0.0.3
>     - src: galaxyproject.slurm
>       version: 1.0.2
>    +# TPV Linting
>    +- name: usegalaxy_eu.tpv_auto_lint
>    +  version: 0.4.3
>    {% endraw %}
>    ```
>    {: data-commit="Add tpv-auto-lint to requirements"}
>
> 2. Install the missing role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Change your `group_vars/galaxyservers.yml`. We need to create a new directory where the TPV configs will be stored after linting, and add that directory name as variable for the role. The default name is 'TPV_DO_NOT_TOUCH' for extra safety 😉. If you want a different name, you need to change the `tpv_config_dir_name` variable, too. We also need to create a directory, `tpv_mutable_dir` (a role default variable), where TPV configs are copied before linting.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -145,6 +145,8 @@ galaxy_config:
>               - job-handlers
>               - workflow-schedulers
>
>    +galaxy_job_config_file: "{{ galaxy_config_dir }}/galaxy.yml"
>    +
>     galaxy_config_files_public:
>       - src: files/galaxy/welcome.html
>         dest: "{{ galaxy_mutable_config_dir }}/welcome.html"
>    @@ -161,6 +163,11 @@ galaxy_config_templates:
>
>     galaxy_extra_dirs:
>       - /data
>    +  - "{{ galaxy_config_dir }}/{{ tpv_config_dir_name }}"
>    +
>    +galaxy_extra_privsep_dirs:
>    +  - "{{ tpv_mutable_dir }}"
>    +tpv_privsep: true
>
>     galaxy_local_tools:
>     - testing.xml
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV config dir"}
> 4. Add the role to your `galaxy.yml` playbook.
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -38,6 +38,7 @@
>         - galaxyproject.slurm
>         - usegalaxy_eu.apptainer
>         - galaxyproject.galaxy
>    +    - usegalaxy_eu.tpv_auto_lint
>         - role: galaxyproject.miniconda
>           become: true
>           become_user: "{{ galaxy_user_name }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add tpv-auto-lint role"}
> 5. At this point we won't run the modified playbook just yet. Because TPV itself has not yet been installed,
>    the tpv_auto_lint role would fail at this point. So first, we'll have to install and configure TPV itself before the
>    linter can work.
>
{: .hands_on}
## Configuring TPV

We want our tool to run with more than one core. To do this, we need to instruct Slurm to allocate more cores for this job. First however, we need to configure Galaxy to use TPV.

> <hands-on-title>Adding TPV to your job configuration</hands-on-title>
>
> 1. Edit `group_vars/galaxyservers.yml` and add the following destination under your `job_config` section to route all jobs to TPV.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -27,38 +27,15 @@ galaxy_job_config:
>       handling:
>         assign: ['db-skip-locked']
>       execution:
>    -    default: slurm
>    +    default: tpv_dispatcher
>         environments:
>           local_env:
>             runner: local_runner
>             tmp_dir: true
>    -      slurm:
>    -        runner: slurm
>    -        singularity_enabled: true
>    -        # Enabling access to the reference data on CVMFS in the container
>    -        singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    -        env:
>    -        - name: LC_ALL
>    -          value: C
>    -        - name: APPTAINER_CACHEDIR
>    -          value: /tmp/singularity
>    -        - name: APPTAINER_TMPDIR
>    -          value: /tmp
>    -      singularity:
>    -        runner: local_runner
>    -        singularity_enabled: true
>    -        # Enabling access to the reference data on CVMFS in the container
>    -        singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    -        env:
>    -        # Ensuring a consistent collation environment is good for reproducibility.
>    -        - name: LC_ALL
>    -          value: C
>    -        # The cache directory holds the docker containers that get converted
>    -        - name: APPTAINER_CACHEDIR
>    -          value: /tmp/singularity
>    -        # Apptainer uses a temporary directory to build the squashfs filesystem
>    -        - name: APPTAINER_TMPDIR
>    -          value: /tmp
>    +      tpv_dispatcher:
>    +        runner: dynamic_tpv
>    +        tpv_config_files:
>    +          - "{{ tpv_config_dir }}/tpv_rules_local.yml"
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>           environment: local_env
>    @@ -154,6 +131,8 @@ galaxy_config_files_public:
>     galaxy_config_files:
>       - src: files/galaxy/themes.yml
>         dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>    +  - src: files/galaxy/config/tpv_rules_local.yml
>    +    dest: "{{ tpv_mutable_dir }}/tpv_rules_local.yml"
>
>     galaxy_config_templates:
>       - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV to job config"}
>
>  Note that we set the default execution environment to the tpv_dispatcher, added the tpv_dispatcher environment using the dynamic_tpv runner, and removed all other destinations.
>  Adding TPV as a runner will cause Galaxy to automatically install the `total-perspective-vortex` package on startup as a conditional dependency.
>  Finally, we added a new config file named `tpv_rules_local.yml`, which we will create next.
>
> 2. Create a new file named `tpv_rules_local.yml` in the `files/galaxy/config/` folder of your ansible playbook, so that it is copied to the config folder on the target.
>    The file should contain the following content:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -0,0 +1,32 @@
>    +tools:
>    +  .*testing.*:
>    +    cores: 2
>    +    mem: cores * 4
>    +
>    +destinations:
>    +  local_env:
>    +    runner: local_runner
>    +    max_accepted_cores: 1
>    +    params:
>    +      tmp_dir: true
>    +  singularity:
>    +    runner: local_runner
>    +    max_accepted_cores: 1
>    +    params:
>    +      singularity_enabled: true
>    +      # Enabling access to the reference data on CVMFS in the container
>    +      singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    +    env:
>    +      # Ensuring a consistent collation environment is good for reproducibility.
>    +      LC_ALL: C
>    +      # The cache directory holds the docker containers that get converted
>    +      APPTAINER_CACHEDIR: /tmp/singularity
>    +      # Singularity uses a temporary directory to build the squashfs filesystem
>    +      APPTAINER_TMPDIR: /tmp
>    +  slurm:
>    +    inherits: singularity
>    +    runner: slurm
>    +    max_accepted_cores: 16
>    +    params:
>    +      native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    +
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV local rules"}
>
>    In this TPV config, we have specified that the testing tool should use `2` cores. Memory has been defined as an expression and should be 4 times as much as cores, which, in this case, is 8GB.
>    Note that the tool id is matched via a regular expression against the full tool id. For example, a full tool id for hisat may look like: `toolshed.g2.bx.psu.edu/repos/iuc/hisat2/hisat2/2.1.0+galaxy7`
>    This enables complex matching, including matching against specific versions of tools.
>
>    Destinations must also be defined in TPV itself. Importantly, note that any destinations defined in the job conf are ignored by TPV. Therefore, we have moved all destinations from the job conf to TPV. In addition, we have removed some
>    redundancy by using the "inherits" clause in the `slurm` destination. This means that slurm will inherit all of the settings defined for singularity, but selectively override some settings. We have additionally
>    defined the `native_specification` param for SLURM, which is what SLURM uses to allocate resources per job. Note the use of the `{cores}` and `{mem}`
>    parameter within the native specification, which TPV will replace at runtime with the value of cores and memory assigned to the tool.
>
>    Finally, we have also defined a new property named `max_accepted_cores`, which is the maximum amount of cores this destination will accept. Since the testing tool requests 2 cores, but only the `slurm`
>    destination is able to accept jobs greater than 1 core, TPV will automatically route the job to the best matching destination, in this case, slurm.
>
> 3. Run the Galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. Click the rerun button on the last history item, or click **Testing Tool** in the tool panel, and then click the tool's Run Tool button.
>
>    > <question-title></question-title>
>    >
>    > What is the tool's output?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > Running with '2' threads
>    > > ```
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}


### Configuring defaults

Now that we've configured the resource requirements for a single tool, let's see how we can configure defaults for all tools, and reuse those defaults to reduce repetition.


> <hands-on-title>Configuring defaults and inheritance</hands-on-title>
>
> 1. Edit your `files/galaxy/config/tpv_rules_local.yml` and add the following settings.
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -1,4 +1,11 @@
>    +global:
>    +  default_inherits: default
>    +
>     tools:
>    +  default:
>    +    abstract: true
>    +    cores: 1
>    +    mem: cores * 4
>       .*testing.*:
>         cores: 2
>         mem: cores * 4
>    @@ -29,4 +36,3 @@ destinations:
>         max_accepted_cores: 16
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    -
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV default inherits"}
>
>    We have defined a `global` section specifying that all tools and destinations should inherit from a specified `default`. We have then defined a tool named `default`, whose properties
>    are implicitly inherited by all tools at runtime. This means that our `testing` tool will also inherit from this default tool, but it explicitly overrides cores.
>    We can also explicitly specify an `inherits` clause if we wish to extend a specific tool or destination, as previously shown in the destinations section.
>
> 2. Run the Galaxy playbook. When the new `tpv_rules_local.yml` is copied, TPV will automatically pickup the changes without requiring a restart of Galaxy.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

### TPV reference documentation

Please see [TPV's dedicated documentation](https://total-perspective-vortex.readthedocs.io/en/latest/) for more information.


# Configuring the TPV shared database

The Galaxy Project maintains a [shared database of TPV rules](https://github.com/galaxyproject/tpv-shared-database) so that admins do not have to independently rediscover ideal resource allocations for specific tools. These rules are based
on settings that have worked well in the usegalaxy.* federation. The rule file can simply be imported directly, with local overrides applied on top.

> <hands-on-title>Adding the TPV shared database to job_conf</hands-on-title>
>
> 1. Edit `group_vars/galaxyservers.yml` and add the location of the TPV shared rule file to the `tpv_dispatcher` destination.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -35,6 +35,7 @@ galaxy_job_config:
>           tpv_dispatcher:
>             runner: dynamic_tpv
>             tpv_config_files:
>    +          - https://gxy.io/tpv/db.yml
>               - "{{ tpv_config_dir }}/tpv_rules_local.yml"
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>    {% endraw %}
>    ```
>    {: data-commit="Importing TPV shared database via job conf"}
>
>    Note how TPV allows the file to be imported directly via its http url. As many local and remote rule files as necessary can be combined, with rule files specified later overriding
>    any previously specified rule files. The TPV shared database does not define destinations, only cores and mem settings, as well as any required environment vars.
>    Take a look at the shared database of rules and note that some tools have very large recommended memory settings, which may or may not be available within your local cluster.
>    Nevertheless, you may still wish to execute these tools with memory adjusted to suit your cluster's capabilities. 
>
> 2. Edit your `files/galaxy/config/tpv_rules_local.yml` and make the following changes.
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -33,6 +33,9 @@ destinations:
>       slurm:
>         inherits: singularity
>         runner: slurm
>    -    max_accepted_cores: 16
>    +    max_accepted_cores: 24
>    +    max_accepted_mem: 256
>    +    max_cores: 2
>    +    max_mem: 8
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    {% endraw %}
>    ```
>    {: data-commit="TPV clamp max cores and mem"}
>
>    These changes indicate that the destination will accept jobs that are up to `max_accepted_cores: 24` and `max_accepted_mem: 256`. If the tool requests resources that exceed these limits, the tool will be rejected
>    by the destination. However, once accepted, the resources will be forcibly clamped down to 2 and 8 at most because of the `max_cores` and `max_mem` clauses. (E.g. a tool requesting 24 cores would only be submitted with 16 cores at maximum.) Therefore, a trick that can be used here to support
>    job resource requirements in the shared database that are much larger than your destination can actually support, is to combine `max_accepted_cores/mem/gpus` with `max_cores/mem/gpus` to accept the job and then
>    clamp it down to a supported range. This allows even the largest resource requirement in the shared database to be accomodated.
>
>    > <comment-title>Clamping in practice</comment-title>
>    > For the purposes of this tutorial, we've clamped down from 16 cores to 2 cores, and mem from 256 to 8, which is unlikely to work in practice. In production, you will probably need to manually test
>    > any tools that exceed your cluster's capabilities, and decide whether you want those tools to run in the first place.
>    {: .comment}
>
> 3. Run the Galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

# Basic access controls

You may wish to apply some basic restrictions on which users are allowed to run specific tools. {TPV} accomodates user and role specific rules. In addition, TPV supports tagging of tools, users, roles and destinations. These tags
can be matched up so that only desired combinations are compatible with each other. While these mechanisms are detailed in the TPV documentation, we will choose a different problem that highlights some other capabilities
- restricting a tool so that only an admin can execute that tool.

> <hands-on-title>Using conditionals to restrict a tool to admins only</hands-on-title>
>
> 1. Edit your `files/galaxy/config/tpv_rules_local.yml` and add the following rule.
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -9,6 +9,15 @@ tools:
>       .*testing.*:
>         cores: 2
>         mem: cores * 4
>    +    rules:
>    +      - id: admin_only_testing_tool
>    +        if: |
>    +          # Only allow the tool to be executed if the user is an admin
>    +          admin_users = app.config.admin_users
>    +          # last line in block must evaluate to a value - which determines whether the TPV if conditional matches or not
>    +          not user or user.email not in admin_users
>    +        fail: Unauthorized. Only admins can execute this tool.
>    +
>
>     destinations:
>       local_env:
>    {% endraw %}
>    ```
>    {: data-commit="TPV admin only tool"}
>
> Note the use of the `if` rule, which allows for conditional actions to be taken in TPV. An `if` block is evaluated as a multi-line python block, and can execute arbitrary code, but the last line of the block must evaluate to a value. That value
> determines whether the `if` condtional is matched or not. If the conditional is matched, the `fail` clause is executed in this case, and the message specified in the `fail` clause is displayed to the user.
> It is similarly possible to conditionally add job parameters, modify cores/mem/gpus and take other complex actions.
>
> 2. Run the Galaxy playbook to update the TPV rules.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Try running the tool as both an admin user and a non-admin user, non-admins should not be able to run it. You can start a private browsing session to test as a non-admin, anonymous user. Anonymous users were enabled in your Galaxy configuration.
>
{: .hands_on}

> ```bash
> 3.sh
> ```
> {: data-test="true"}
{: .hidden}

# Job Resource Selectors

Certain tools can benefit from allowing users to select appropriate job resource parameters, instead of having admins decide resource allocations beforehand. For example, a user might know that a particular set of parameters and inputs to a certain tool needs a larger memory allocation than the standard amount given to that tool.
Galaxy provides functionality to have extra form elements in the tool execution form to specify these additional job resource parameters.
This of course assumes that your users are well behaved enough not to choose the maximum whenever available, although such concerns can be mitigated somewhat by the use of concurrency limits on larger memory destinations.

Such form elements can be added to tools without modifying each tool's configuration file through the use of the **job resource parameters configuration file**

> <hands-on-title>Configuring a Resource Selector</hands-on-title>
>
> 1. Create and open `templates/galaxy/config/job_resource_params_conf.xml.j2`
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/galaxy/config/job_resource_params_conf.xml.j2
>    @@ -0,0 +1,7 @@
>    +<parameters>
>    +    <param label="Cores" name="cores" type="select" help="Number of cores to run job on.">
>    +        <option value="1">1 (default)</option>
>    +        <option value="2">2</option>
>    +    </param>
>    +  <param label="Time" name="time" type="integer" size="3" min="1" max="24" value="1" help="Maximum job time in hours, 'walltime' value (1-24). Leave blank to use default value." />
>    +</parameters>
>    {% endraw %}
>    ```
>    {: data-commit="Add job resource params configuration"}
>
>    This defines two resource fields, a select box where users can choose between 1 and 2 cores, and a text entry field where users can input an integer value from 1-24 to set the walltime for a job.
>
> 2. As usual, we need to instruct Galaxy of where to find this file:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -37,9 +37,17 @@ galaxy_job_config:
>             tpv_config_files:
>               - https://gxy.io/tpv/db.yml
>               - "{{ tpv_config_dir }}/tpv_rules_local.yml"
>    +  resources:
>    +    default: default
>    +    groups:
>    +      default: []
>    +      testing: [cores, time]
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>           environment: local_env
>    +    - id: testing
>    +      environment: tpv_dispatcher
>    +      resources: testing
>
>     galaxy_config:
>       galaxy:
>    @@ -59,6 +67,7 @@ galaxy_config:
>         object_store_store_by: uuid
>         id_secret: "{{ vault_id_secret }}"
>         job_config: "{{ galaxy_job_config }}" # Use the variable we defined above
>    +    job_resource_params_file: "{{ galaxy_config_dir }}/job_resource_params_conf.xml"
>         # SQL Performance
>         slow_query_log_threshold: 5
>         enable_per_request_sql_debugging: true
>    @@ -140,6 +149,8 @@ galaxy_config_templates:
>         dest: "{{ galaxy_config.galaxy.container_resolvers_config_file }}"
>       - src: templates/galaxy/config/dependency_resolvers_conf.xml
>         dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
>    +  - src: templates/galaxy/config/job_resource_params_conf.xml.j2
>    +    dest: "{{ galaxy_config.galaxy.job_resource_params_file }}"
>
>     galaxy_extra_dirs:
>       - /data
>    {% endraw %}
>    ```
>    {: data-commit="Configure resources in job conf"}
>
>    We have added a resources section. The group ID will be used to map a tool to job resource parameters, and the text value of the `<group>` tag is a comma-separated list of `name`s from `job_resource_params_conf.xml` to include on the form of any tool that is mapped to the defined `<group>`.
>
>    We have also listed the `testing` tool as a tool which uses resource parameters. When listed here, the resource parameter selection form will be displayed.
>
>    Finally, we have specified that the `job_resource_params_conf.xml.j2` should be copied across.
>
{: .hands_on}

This will set everything up to use the function. We have:

- A set of "job resources" defined which will let the user select the number of cores and walltime.
- A job configuration which says:
    - that our testing tool should allow selection of the cores and time parameters
    - directs it to TPV's `tpv_dispatcher` destination

This is a lot but we're still missing the last piece for it to work:

## Configuring TPV to process resource parameters

Lastly, we need to write a rule in TPV that will read the value of the job resource parameter form fields and decide how to submit the job.

> <hands-on-title>Processing job resource parameters in TPV</hands-on-title>
>
> 1. Create and edit `files/galaxy/config/tpv_rules_local.yml`. Create it with the following contents:
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -6,6 +6,8 @@ tools:
>         abstract: true
>         cores: 1
>         mem: cores * 4
>    +    params:
>    +      walltime: 8
>       .*testing.*:
>         cores: 2
>         mem: cores * 4
>    @@ -17,7 +19,13 @@ tools:
>               # last line in block must evaluate to a value - which determines whether the TPV if conditional matches or not
>               not user or user.email not in admin_users
>             fail: Unauthorized. Only admins can execute this tool.
>    -
>    +      - id: resource_params_defined
>    +        if: |
>    +          param_dict = job.get_param_values(app)
>    +          param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
>    +        cores: int(job.get_param_values(app)['__job_resource']['cores'])
>    +        params:
>    +           walltime: "{int(job.get_param_values(app)['__job_resource']['time'])}"
>
>     destinations:
>       local_env:
>    @@ -47,4 +55,4 @@ destinations:
>         max_cores: 2
>         max_mem: 8
>         params:
>    -      native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    +      native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)} --time={entity.params['walltime']}:00:00
>    {% endraw %}
>    ```
>    {: data-commit="process resource params in TPV"}
>
>    We define a conditional rule and check that the job_resource_params have in fact been defined by the user. If defined, we override the cores value with the one specified by the user. We also define a walltime param, setting it to 8 hours by default.
>    We also override walltime if the user has specified it.
>    It is important to note that **you are responsible for parameter validation, including the job resource selector**. This function only handles the job resource parameter fields, but it could do many other things - examine inputs, job queues, other tool parameters, etc.
>
>    Finally, we pass the walltime as part of the native specification.
>
>    > <comment-title>Rules for TPV code evaluation</comment-title>
>    > Notice how the `walltime` parameter is wrapped in braces, whereas the `cores` value isn't, yet they are both expressions. The reason for this difference is that when TPV evaluates an expression, all string fields (e.g. env, params) are evaluated as Python f-strings,
>    > while all non-string fields (integer fields like `cores` and `gpus`, float fields like `mem`, boolean fields like `if`) are evaluated as Python code-blocks. This rule enables the YAML file to be much more readable, but requires you to keep this simple rule in mind.
>    > Note also that Python code-blocks can be multi-line, and that the final line must evaluate to a value that can be assigned to the field.
>    {: .comment}
>
> 2. Run the Galaxy playbook to update the TPV rules.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Run the **Testing Tool** with various resource parameter selections
>
>    - Use default job resource parameters
>    - Specify job resource parameters:
>      - 1 core
>      - 2 cores
>      - Some value for walltime from 1-24
>
{: .hands_on}

The cores parameter can be verified from the output of the tool. The walltime can be verified with `scontrol`:

> <code-in-title>Bash</code-in-title>
> Your job number may be different.
> ```
> scontrol show job 24
> ```
{: .code-in}

> <code-out-title></code-out-title>
> Your output may look slightly different. Note that the `TimeLimit` for this job (which I gave a 12 hour time limit) was set to `12:00:00`.
> ```console
> JobId=24 JobName=g24_multi_anonymous_10_0_2_2
>    UserId=galaxy(999) GroupId=galaxy(999)
>    Priority=4294901747 Nice=0 Account=(null) QOS=(null)
>    JobState=COMPLETED Reason=None Dependency=(null)
>    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>    RunTime=00:00:05 TimeLimit=12:00:00 TimeMin=N/A
>    SubmitTime=2016-11-05T22:01:09 EligibleTime=2016-11-05T22:01:09
>    StartTime=2016-11-05T22:01:09 EndTime=2016-11-05T22:01:14
>    PreemptTime=None SuspendTime=None SecsPreSuspend=0
>    Partition=debug AllocNode:Sid=gat2016:1860
>    ReqNodeList=(null) ExcNodeList=(null)
>    NodeList=localhost
>    BatchHost=localhost
>    NumNodes=1 NumCPUs=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>    TRES=cpu=1,node=1
>    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
>    MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
>    Features=(null) Gres=(null) Reservation=(null)
>    Shared=OK Contiguous=0 Licenses=(null) Network=(null)
>    Command=(null)
>    WorkDir=/srv/galaxy/server/database/jobs/000/24
>    StdErr=/srv/galaxy/server/database/jobs/000/24/galaxy_24.e
>    StdIn=StdIn=/dev/null
>    StdOut=/srv/galaxy/server/database/jobs/000/24/galaxy_24.o
>    Power= SICP=0
> ```
{: .code-out}

{% snippet topics/admin/faqs/missed-something.md step=10 %}


# More on TPV

The goal of this tutorial is to provide a quick overview of some of the basic capabilities of TPV. However, there are numerous features that we have not convered such as:
a. Custom code blocks - execute arbitrary python code blocks, access additional context variables etc.
a. User and Role Handling - Add scheduling constraints based on the user's email or role
b. Metascheduling support - Perform advanced querying and filtering prior to choosing an appropriate destination
c. Job resubmissions - Resubmit a job if it fails for some reason
d. Linting, formatting and dry-run - Automatically format tpv rule files, catch potential syntax errors and perform a dry-run to check where a tool would get scheduled.

These features are covered in detail in the [TPV documentation](https://total-perspective-vortex.readthedocs.io/en/latest/).

## Further Reading

- The [sample dynamic tool destination config file](https://github.com/galaxyproject/galaxy/blob/dev/config/tool_destinations.yml.sample) fully describes the configuration language
- [Dynamic destination documentation](https://docs.galaxyproject.org/en/latest/admin/jobs.html)
- Job resource parameters are not as well documented as they could be, but the [sample configuration file](https://github.com/galaxyproject/usegalaxy-playbook/blob/master/env/test/files/galaxy/config/job_resource_params_conf.xml) shows some of the possibilities.
- [usegalaxy.org's job_conf.yml](https://github.com/galaxyproject/usegalaxy-playbook/blob/master/env/common/templates/galaxy/config/job_conf.yml.j2) is publicly available for reference.
- [usegalaxy.eu's job_conf.xml](https://github.com/usegalaxy-eu/infrastructure-playbook/search?l=YAML&q=galaxy_jobconf) is likewise (see the `group_vars/galaxy.yml` result)

{% snippet topics/admin/faqs/git-gat-path.md tutorial="job-destinations" %}


topic: admin
tutorial: surf-research-cloud-pulsar
title: Pulsar usage on SURF Research Cloud
difficulty: introductory
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"none\"", "type: \"none\"", "type: \"none\"", "type: \"none\""]
tags_json: []
content_hash: 768eeda092786f2f7c27ace700692541
last_modified: 2026-02-26T21:12:14.165111
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/surf-research-cloud-pulsar/tutorial.html

content:
Using Pulsar via the {SRC} allows researchers to start Pulsar instances on-demand to expand their computational resources and even access GPUs to help and analyze their data in a secure environment following the {GDPR}.

There are two main use cases we envision this role being useful for:

Saving costs on SRC
:  Maybe you're already running Galaxy in SRC, but you don't want to run a GPU node because it is very expensive. By using the SRC Pulsar Catalog Item, you can launch a node to do computations and then shut it down when you're done, saving money. Pulsar instances can be started and stopped on demand, depending on personal cases and requirements, giving you a lot of freedom!

Accessing a GPU from a local (in UMC/hospital Galaxy)
:  If you do not have a GPU easily available within your institute, it may be attractive to send jobs securely to SRC, by launching a Pulsar node in SRC and attaching it to your institute's Galaxy instance.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prerequisites

The instance provides secure authentication, where users must have a SURF Research account prior to this tutorial, have set the {SRAM} authentication method, and connect an SSH key to their accounts. In case you are not familiar with {SRC} and need help in setting up your accounts, please follow the instructions on the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/SURF+Research+Cloud)

Inside the SRC members should have access to all publicly available catalog items. If you are not able to create a catalog item, please [contact SURF servicedesk](mailto:servicedesk@surf.nl).

This tutorial assumes you are member of a {CO} in {SRAM} that has access to {SRC} and a wallet with budget in SRC with enough sources to create Galaxy and Pulsar catalog items. (For more information please refer to the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/Budgets%2C+wallets%2C+contracts).

You should have previous experience working with data inside Galaxy.

> <tip-title>SSH Access Required</tip-title>
> SSH access is required to reconfigure your Galaxy instance. Please make sure you set an SSH key in your SRAM account if you are planning to use Galaxy in SRC for this tutorial.
{: .tip}

> <hands-on-title>Log In to SRC</hands-on-title>
> 1. Log in to [SURF Research Cloud](https://portal.live.surfresearchcloud.nl/)
>
>    ![screenshot of SRC login portal with a bright yellow login button](images/surf1.png)
{: .hands_on}

# Launching Pulsar in SRC

> <hands-on-title>Launching Pulsar</hands-on-title>
> 1. In the **Workspaces Tab** on the bottom half of the screen, you'll find a **Plus Button** at right to add a new workspace
>
>    ![A small plus button is hovered over which says Add. Below galaxy-test is shown running with a yellow Access button](images/surf3.png)
>
> 2. Clicking that will let you choose any of the *Catalog Items* from SRC. They've got a wide selection but we're only interested in the two Pulsar Catalog Items
>
>    ![the two pulsar components in SRC: Galaxy Pulsar GPU Node (CUDA) and Galaxy Pulsar Node are shown. The second is expanded showing an author, Helena Rasche, and a description: let's you run galaxy jobs on another node](images/surf4.png)
>
>    > <warning-title>GPUs are Expensive</warning-title>
>    > The GPU nodes are *expensive*. In fact it
>    > was the motivating reason for building this catalog item: to enable you to
>    > launch a node, run some computations, and shut it down, saving you money.
>    {: .warning}
>
> 3. Creating a "workspace" from a catalog item (a template) is easy, most of the options are fixed for you, you just need to choose the size of the item. Pick an appropriate size for whatever computations you need to do.
>
>    ![workspace creation screen, choose the cloud provider is locked to SURF HPC, flavour is locked to 22.04, and the size options are available from 1GB/8CPU to 60C/750GB Hi-mem](images/surf5.png)
>
> 6. Pick a name, it can be anything, it does not matter. Check the expiration date to ensure it is just enough time for your computation and no more.
>
>    > <tip-title>Expiration Date</tip-title>
>    > The standard life-time of the VM is 5 days. If you need it for longer, this option can be changed once the machine is running.
>    > Note, that once the machine is expired and deleted it cannot be restored! Plan accordingly and migrate your data in time to prevent data loss!
>    >
>    > This is an incredibly useful feature as it saves you from forgetting to destroy a VM. Especially for GPU nodes it can help you ensure that they disappear after your computation is complete.
>    {:.tip}
>
>    ![almost there! some final details reads this page, asking for name and description. both have been filled out with 'pulsar'. a yellow submit button waits at the bottom](images/surf6.png)
>
> 7. Click submit when you are happy with the configuration.
>
{: .hands_on}

Once done, the workspace will be created for you. You'll need to wait ~5 minutes usually. Go for a beverage ☕️

![workspace list showing a workspace named pulsar being created.](images/surf7.png)

## Using Pulsar on SRC

> <hands-on-title>Access the Information Page</hands-on-title>
>
> 1. Once the workspace is up, you'll see an **Access** link:
>
>    ![A small plus button is hovered over which says Add. Below galaxy-test is shown running with a yellow Access button](images/surf3.png)
>
> 2. Click that will show you a Pulsar information page. This page is running on your pulsar node itself, and is restricted to ensure only authorised members can access the contents. It includes some configuration you will need to copy to your Galaxy node in order to make use of the Pulsar node.
>
>    ![pulsar configuration information page showing an about with admins and metadata like workspace fqdn. Configuration for galaxy is shown below including XML changes](images/surf8.png)
{: .hands_on}

This information page should have more than enough information to connect this Pulsar instance to your Galaxy server. You will need to reference information from this page in the following steps:

> <hands-on-title>Configuring Galaxy to use SRC Pulsar</hands-on-title>
> 1. Collect the requirements for accessing the Galaxy machine. You will need:
>
>    - your username from the first step
>    - your SSH key that is associated with your SRAM account
>
> 2. SSH into your *Galaxy* machine (not pulsar!).
>
>    ```
>    ssh -i path/to/your/sram-ssh-key username@galaxy.src-winter-school.src.surf-hosted.nl
>    ```
>
> 3. You will need to `sudo su` to do anything useful. Do that now.
> 4. `cd /srv/galaxy/` to move into the directory Galaxy configuration is stored.
> 5. The configuration is discussed fully in the Pulsar information, but it will be briefly covered here as well. Generally there are a few steps that must be followed:
>
>    - A runner must be registered
>    - A destination/environment must be added with the pulsar details
>    - Some tools should be redirected to this Pulsar
>
>    Here is an example of what those changes *might* look like in your Galaxy node.  In this example our pulsar node was called `p20` but that will be different for you.
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>
>    ```diff
>     runners:
>       local:
>         load: galaxy.jobs.runners.local:LocalJobRunner
>         workers: 4
>       condor:
>         load: galaxy.jobs.runners.condor:CondorJobRunner
>    +  pulsar:
>    +    load: galaxy.jobs.runners.pulsar:PulsarRESTJobRunner
>
>     execution:
>       default: docker_dispatch
>       environments:
>         local_destination:
>           runner: local
>         # ... probably some more environments here.
>
>    +    remote_p20:
>    +       runner: pulsar
>    +       url: https://p20.src-sensitive-i.src.surf-hosted.nl
>    +       private_token: ySgfM1rnGIsiVN8Xlfk
>    +       dependency_resolution: remote
>    +       manager: _default_
>    +       # Uncomment the following to enable interactive tools:
>    +       docker_enabled: true
>    +       docker_set_user: null
>    +       docker_memory: "8G"
>    +       singularity_enabled: false
>    +       tmp_dir: true
>    +       outputs_to_working_directory: false
>    +       container_resolvers:
>    +       - type: explicit
>    +       require_container: True
>    +       container_monitor_command: /mnt/pulsar/venv/bin/galaxy-container-monitor
>    +       container_monitor_result: callback
>    +       container_monitor_get_ip_method: command:echo p20.src-sensitive-i.src.surf-hosted.nl
>
>
>     tools:
>     - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>       environment: local_env
>     - id: Cut1
>       environment: condor_1x1
>    +- id: interactive_tool_jupyter_notebook
>    +  environment: remote_p20
>    +- id: interactive_tool_rstudio
>    +  environment: remote_p20
>    ```
{: .hands_on}

With the above, the minimal configuration is done, the Galaxy server will be aware of the Pulsar node, and two tools will be sent there: the Jupyter and RStudio Interactive Tools.

While you will simply copy-paste the runner and environment, you will need to identify yourself which tools should go to this Pulsar node.
You can find the tool ID from the dropdown at the top right, just to the left of "Execute" at the top:

![url bar and tool interface for the Cut1 tool](./images/id2.png)

> <tip-title>An Easy Configuration Option: Send Every Job to Pulsar</tip-title>
>
> If you are running jobs for a limited period of time, you might consider making this pulsar node the default destination. Remember to use the `remote_...` name of your pulsar node, based on what you copied. Not `remote_p20`.
>
> ```diff
>  execution:
> -  default: docker_dispatch
> +  default: remote_p20
>    environments:
>      local_destination:
>        runner: local
> ```
{: .tip}

With that, you're done, and for the length of time your node is running, your chosen tools (or everything) will be executed on that Pulsar node with more memory and CPU than the Galaxy host, and maybe a GPU as well!

> <hands-on-title>Launch a Job on Pulsar</hands-on-title>
> 1. Login to your Galaxy
> 2. Run one of the tools you have decided to send to Pulsar
> 3. On the pulsar machine, you can check that it runs by following the logs:
>
>    ```bash
>    sudo journalctl -fu pulsar
>    ```
>
{: .hands_on}

Congratulations on launching Pulsar in SRC! 🌌


topic: admin
tutorial: wireguard
title: Deploying Wireguard for private mesh networking
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/wireguard/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["wireguard", "networking"]
content_hash: 07b4430d9c7c033dfe08767cce2bc094
last_modified: 2026-02-26T21:12:14.168826
workflows_json: []
content:
In this tutorial we will briefly cover what [Wireguard](https://www.wireguard.com/) is and how you can leverage it for your needs. This will not make you an expert on Wireguard but will give you the tools you need in order to setup a local Wireguard network.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# What is Wireguard?

Wireguard is a VPN like OpenVPN or IPSec, but instead of the hub and spoke model of those where all traffic must go through a central node, Wireguard creates a mesh network where machines can all talk individually to each other. Wireguard also uses modern encryption only, ensuring your data stays safe.

## Is it right for me?

If you have machines that need to talk to each other privately, and you don't
have a better way to do it like a local network team, then yes, it's a great
solution to private, secure, fast networking.
It has [excellent performance](https://www.wireguard.com/performance/)
despite the encryption, and is built directly into the kernel.

By using wireguard, you can let services listen only on the wireguard interface, and thus only known and trusted machines can access those services.

# Setting up the infrastructure

> <hands-on-title>Configuration files</hands-on-title>
>
> 1. Create a `ansible.cfg` file (next to your playbook) to [configure settings](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html) like the inventory file (and save ourselves some typing!), or the Python interpreter to use:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/ansible.cfg
>    @@ -0,0 +1,6 @@
>    +[defaults]
>    +interpreter_python = /usr/bin/python3
>    +inventory = hosts
>    +retry_files_enabled = false
>    +[ssh_connection]
>    +pipelining = true
>    {% endraw %}
>    ```
>    {: data-commit="Add ansible.cfg"}
>
>    > <tip-title>CentOS7</tip-title>
>    > As mentioned in the "Ubuntu or Debian, CentOS or RHEL?" comment above, if you are using CentOS7 do not set `interpreter_python` in `ansible.cfg` .
>    {: .tip}
>
>    Pipelining will make [Ansible run faster](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html#ansible-pipelining) by significantly reducing the number of new SSH connections that must be opened.
>
> 2. Create the `hosts` inventory file if you have not done so yet, defining an `[wireguard]` group with every host you want to be part of the cluster. For each machine, also set a variable `wireguard_ip` with an address from `192.168.0.0/24`
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat hosts
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > Your hostname is probably different:
>    > >
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,5 @@
>    > > +[wireguard]
>    > > +1-wg.galaxy.training wireguard_ip=192.168.0.1
>    > > +2-wg.galaxy.training wireguard_ip=192.168.0.2
>    > > +3-wg.galaxy.training wireguard_ip=192.168.0.3
>    > > +4-wg.galaxy.training wireguard_ip=192.168.0.4
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add hosts"}
>    > {: .code-out}
>    {: .code-2col}
{: .hands_on}

Wireguard can use any of the [private network](https://en.wikipedia.org/wiki/Private_network) blocks, here we use `192.168.0.0/16` for familiarity, Tailscale uses `100.0.0.0/8`, which is the [Carrier Grade NAT](https://en.wikipedia.org/wiki/Carrier-grade_NAT) range, and behaves somewhat differently to classic Private Network prefixes.

## Writing the playbook

First lets set up the playbook and install Wireguard, without configuring it.

> <hands-on-title>Installing Wireguard</hands-on-title>
>
> 1. Create and open `wg.yml` which will be our playbook. Add the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/wg.yml
>    @@ -0,0 +1,16 @@
>    +---
>    +- hosts: all
>    +  become: yes
>    +  vars:
>    +    wireguard_mask_bits: 24
>    +    wireguard_port: 51871
>    +  tasks:
>    +    - name: update packages
>    +      apt:
>    +        update_cache: yes
>    +        cache_valid_time: 3600
>    +
>    +    - name: Install wireguard
>    +      apt:
>    +        name: wireguard
>    +        state: present
>    {% endraw %}
>    ```
>    {: data-commit="Add initial playbook, install wireguard"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook wg.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
{:.hands_on}

Now we can start configuring it. Wireguard relies on each node having a private/public keypair which is used for encrypting communications between the nodes. By default this uses modern elliptic cryptography which is quite simple and provably secure, wireguard only has ~4k LOC compared to 400k+ LOC for most VPN solutions. Let's generate those keys now:

> <hands-on-title>Generating a keypair</hands-on-title>
>
> 1. Edit `wg.yml` and add
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -14,3 +14,8 @@
>           apt:
>             name: wireguard
>             state: present
>    +
>    +    - name: Generate Wireguard keypair
>    +      shell: wg genkey | tee /etc/wireguard/privatekey | wg pubkey | tee /etc/wireguard/publickey
>    +      args:
>    +        creates: /etc/wireguard/privatekey
>    {% endraw %}
>    ```
>    {: data-commit="Generate the keys"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook wg.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Go check out what the keys look like!
{:.hands_on}

Ok, that's got wireguard setup, but it still isn't running. As of 2018 or so, systemd added built in support for wireguard and we'll use that to setup the network devices, and the network itself.

> <hands-on-title>Setting up the network</hands-on-title>
>
> 1. Edit `wg.yml` and add the following. We want to register the contents of these keys, as they'll be used to configure our networks later.
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -19,3 +19,13 @@
>           shell: wg genkey | tee /etc/wireguard/privatekey | wg pubkey | tee /etc/wireguard/publickey
>           args:
>             creates: /etc/wireguard/privatekey
>    +
>    +    - name: register private key
>    +      shell: cat /etc/wireguard/privatekey
>    +      register: wireguard_private_key
>    +      changed_when: false
>    +
>    +    - name: register public key
>    +      shell: cat /etc/wireguard/publickey
>    +      register: wireguard_public_key
>    +      changed_when: false
>    {% endraw %}
>    ```
>    {: data-commit="Register the keys"}
>
> 2. Again editing `wg.yml`, we'll setup the network device:
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -29,3 +29,28 @@
>           shell: cat /etc/wireguard/publickey
>           register: wireguard_public_key
>           changed_when: false
>    +
>    +    - name: Setup wg0 device
>    +      copy:
>    +        content: |
>    +          [NetDev]
>    +          Name=wg0
>    +          Kind=wireguard
>    +          Description=WireGuard tunnel wg0
>    +          [WireGuard]
>    +          ListenPort={{ wireguard_port }}
>    +          PrivateKey={{ wireguard_private_key.stdout }}
>    +          {% for peer in groups['all'] %}
>    +          {% if peer != inventory_hostname %}
>    +          [WireGuardPeer]
>    +          PublicKey={{ hostvars[peer].wireguard_public_key.stdout }}
>    +          AllowedIPs={{ hostvars[peer].wireguard_ip }}/32
>    +          Endpoint={{ hostvars[peer].inventory_hostname }}:{{ wireguard_port }}
>    +          PersistentKeepalive=25
>    +          {% endif %}
>    +          {% endfor %}
>    +        dest: /etc/systemd/network/99-wg0.netdev
>    +        owner: root
>    +        group: systemd-network
>    +        mode: 0640
>    +      notify: systemd network restart
>    {% endraw %}
>    ```
>    {: data-commit="Setup the netdev"}
>
>    Here we do a number of things:
>    - We configure a [NetDev](https://freedesktop.org/software/systemd/man/systemd.netdev.html), a virtual network device. Systemd supports many types but we'll use wireguard.
>    - Next we setup wireguard, specifying a `ListenPort` which is used for all wireguard communication, and specifies our PrivateKey which is used by the tunnel.
>    - For every peer (all other instances than ourselves), we setup a `WireGuardPeer` with that peer's public key, and which IPs are allowed to connect.
>
> 2. Again editing `wg.yml`, we'll setup the final bit, the network service.
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -54,3 +54,16 @@
>             group: systemd-network
>             mode: 0640
>           notify: systemd network restart
>    +
>    +    - name: Setup wg0 network
>    +      copy:
>    +        content: |
>    +          [Match]
>    +          Name=wg0
>    +          [Network]
>    +          Address={{ wireguard_ip }}/{{ wireguard_mask_bits }}
>    +        dest: /etc/systemd/network/99-wg0.network
>    +        owner: root
>    +        group: systemd-network
>    +        mode: 0640
>    +      notify: systemd network restart
>    {% endraw %}
>    ```
>    {: data-commit="Setup the network"}
>
>    Here we just setup the network, declare the interface name, and the address for it.
>
> 2. And last let's restart the appropriate services.
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -67,3 +67,10 @@
>             group: systemd-network
>             mode: 0640
>           notify: systemd network restart
>    +
>    +  handlers:
>    +    - name: systemd network restart
>    +      service:
>    +        name: systemd-networkd
>    +        state: restarted
>    +        enabled: yes
>    {% endraw %}
>    ```
>    {: data-commit="Restart the network"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook wg.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Go check out the network! Try pinging each of the wireguard IPs to see if you can reach each of the machines.
{:.hands_on}


topic: admin
tutorial: reference-genomes
title: Reference Data with Data Managers
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 9da3c160bc82373f6a5838617fcc49db
last_modified: 2026-02-26T21:12:14.151845
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/reference-genomes/tutorial.html

content:
# Overview

## The problem

The Galaxy server administrator needed to know how to update each type of reference data, how to run the indexers, where to get the data from, and how to update Galaxy's *Tool Data Table* and *location* configuration files.

## Data managers to the rescue

Data Managers are a special class of Galaxy tool which allows for the download and/or creation of data that is stored within Tool Data Tables and their underlying flat location (e.g. .loc) files. These tools handle the creation of indices and the addition of entries/lines to the data table / .loc file via the Galaxy admin interface.

Data Managers can be defined locally in the data manager and tool data table configuration files or installed through the Tool Shed. When Data Managers are installed from the Tool Shed, their configuration is added to the shed versions of the data manager configuration and tool data table configuration files.

They are a flexible framework for adding reference data to Galaxy (not just genomic data). They are workflow compatible and can run via the Galaxy API. Data managers automatically update the appropriate location files when new data are installed.

For more background details on data managers, check out:
- [The data managers documentation on the hub](https://galaxyproject.org/admin/tools/data-managers/)
- [How to define a data manager for a tool](https://galaxyproject.org/admin/tools/data-managers/how-to/define/)
- There is also discussion of this topic in [the Reference Genomes in Galaxy slides]({% link topics/admin/tutorials/reference-genomes/slides.html %})

> <comment-title>Pre-built data are available</comment-title>
>
> The usegalaxy.\* servers and Galaxy Community have a large amount of reference data online and available for use by your Galaxy server. For instructions on how to access and use these data, see [the Reference Data with CVMFS tutorial]({% link topics/admin/tutorials/cvmfs/tutorial.md %}).
>
> If your data are not available as part of the CVMFS repository, Galaxy Data Managers can be used to locally install and build reference data.
>
{: .comment}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="reference-genomes" %}

# Galaxy Reference Data Components

As discussed in the overview, Galaxy Data Managers handle the population of numerous Galaxy components. From the highest level to the lowest:

- Addition of tool data table entries into the shed version of the tool data table configuration file, `shed_tool_data_table_conf.xml`. Multiple data tables with the same `name` attribute are automatically merged by Galaxy.
- Population of the corresponding location (`.loc`) file when new reference data are installed. The `.loc` file used corresponds to the Data Manager and its specific version that was executed to install the data. The path to the location file can be found in the Data Manager's entry in the tool data table configuration file (above).
- Generation of the reference data on disk, underneath the directory specified by `tool_data_path` in `galaxy.yml`.

# Installing and Using Data Managers with the Galaxy UI

## Install a the Fetch Genome Data Manager from the Tool Shed

> <comment-title>Galaxy Configuration</comment-title>
>
> No special configuration for Galaxy beyond creating an admin user is necessary in order to use Data Managers. However, you may want to be aware of the `enable_data_manager_user_view` option, which enables unprivileged user to browse data installed via Data Managers. This option is not required, and users will be able to *use* data installed via Data Managers regardless.
>
> Data are installed in the path specified by `tool_data_path` in `galaxy.yml`. If you prefer to keep hand-managed reference data separate from DM-managed reference data, you can set `galaxy_data_manager_data_path` instead.
>
{: .comment}

> <tip-title>Intall with Ephemeris</tip-title>
> This hands-on exercise installs data managers and reference data through the Galaxy UI, but you are encouraged to install tools in a deterministic, recordable way through the use of Ephemeris, which is described in [the Installing and Using Data Managers from the command line with Ephemeris section](#installing-and-using-data-managers-from-the-command-line-with-ephemeris) below.
{: .tip}

We will install a data manager that can fetch the various genome sequences from multiple sources.

> <hands-on-title>Install the Fetch Genome Data Manager</hands-on-title>
>
> 1. Access the **Admin** menu from the top bar (you need to be logged-in with an email specified in the `admin_users` setting)
> 2. Click **Install and Uninstall**, which can be found on the left, under **Tool Management**
> 3. Enter `fetch_genome` in the search interface
> 4. Click on the first hit, having `devteam` as owner
> 5. Click the **Install** button for the latest revision
{: .hands_on}

View in the file system where the various elements land. Have a look in the configuration files located in config directory.

> <question-title></question-title>
>
> What did this tool installation change?
>
> > <solution-title></solution-title>
> >
> > - The data manager and its data tables are added to the Galaxy-managed "shed" versions of the data manager config (`/srv/galaxy/var/config/shed_data_manager_conf.xml`) and data table config (`/srv/galaxy/var/config/shed_tool_data_table_conf.xml`)
> > - The data manager tool is installed along side other Galaxy tools in the shed tools directory
> >
> > > <code-in-title>Bash</code-in-title>
> > > Let's investigate the data manager config file.
> > > ```bash
> > > cat /srv/galaxy/var/config/shed_data_manager_conf.xml
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```xml
> > > <data_managers><data_manager id="fetch_genome_all_fasta_dbkeys" guid="toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager/fetch_genome_all_fasta_dbkeys/0.0.1" shed_conf_file="/srv/galaxy/var/config/shed_tool_conf.xml">
> > >         <tool file="toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/data_manager_fetch_genome_dbkeys_all_fasta/data_manager/data_manager_fetch_genome_all_fasta_dbkeys.xml" guid="toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4"><tool_shed>toolshed.g2.bx.psu.edu</tool_shed><repository_name>data_manager_fetch_genome_dbkeys_all_fasta</repository_name><repository_owner>devteam</repository_owner><installed_changeset_revision>4d3eff1bc421</installed_changeset_revision><id>toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4</id><version>0.0.4</version></tool><data_table name="all_fasta">
> > >             <output>
> > >                 <column name="value"/>
> > >                 <column name="dbkey"/>
> > >                 <column name="name"/>
> > >                 <column name="path" output_ref="out_file">
> > >                     <move type="file">
> > >                         <source>${path}</source>
> > >                         <target base="${GALAXY_DATA_MANAGER_DATA_PATH}">${dbkey}/seq/${path}</target>
> > >                     </move>
> > >                     <value_translation>${GALAXY_DATA_MANAGER_DATA_PATH}/${dbkey}/seq/${path}</value_translation>
> > >                     <value_translation type="function">abspath</value_translation>
> > >                 </column>
> > >             </output>
> > >         </data_table>
> > >         <data_table name="__dbkeys__">
> > >             <output>
> > >                 <column name="value"/>
> > >                 <column name="name"/>
> > >                 <column name="len_path" output_ref="out_file">
> > >                     <move type="file">
> > >                         <source>${len_path}</source>
> > >                         <target base="${GALAXY_DATA_MANAGER_DATA_PATH}">${value}/len/${len_path}</target>
> > >                     </move>
> > >                     <value_translation>${GALAXY_DATA_MANAGER_DATA_PATH}/${value}/len/${len_path}</value_translation>
> > >                     <value_translation type="function">abspath</value_translation>
> > >                 </column>
> > >             </output>
> > >         </data_table>
> > >     </data_manager>
> > > ```
> > {: .code-out.code-max-300}
> >
> > > <code-in-title>Bash</code-in-title>
> > > Let's also investigate the tool data table config file.
> > > ```bash
> > > cat /srv/galaxy/var/config/shed_tool_data_table_conf.xml
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```xml
> > > <?xml version="1.0" ?>
> > > <tables>
> > >     <table name="all_fasta" comment_char="#">
> > >         <columns>value, dbkey, name, path</columns>
> > >         <file path="/srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc"/>
> > >         <tool_shed_repository>
> > >             <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
> > >             <repository_name>data_manager_fetch_genome_dbkeys_all_fasta</repository_name>
> > >             <repository_owner>devteam</repository_owner>
> > >             <installed_changeset_revision>4d3eff1bc421</installed_changeset_revision>
> > >         </tool_shed_repository>
> > >     </table>
> > >     <table name="__dbkeys__" comment_char="#">
> > >         <columns>value, name, len_path</columns>
> > >         <file path="/srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/dbkeys.loc"/>
> > >         <tool_shed_repository>
> > >             <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
> > >             <repository_name>data_manager_fetch_genome_dbkeys_all_fasta</repository_name>
> > >             <repository_owner>devteam</repository_owner>
> > >             <installed_changeset_revision>4d3eff1bc421</installed_changeset_revision>
> > >         </tool_shed_repository>
> > >     </table>
> > > </tables>
> > > ```
> > {: .code-out.code-max-300}
> >
> {: .solution }
>
{: .question}

## Download and install a reference genome sequence

Next, we will install some reference data. Specifically, we will grab sacCer2 (version 2 of the *Saccharomyces cerevisiae* genome).

> <hands-on-title>Download and install sacCer2</hands-on-title>
>
> 1. Access the Admin menu from the top bar
> 2. [Click **Local Data**](https://my.gat.galaxy.training/?path=/admin/data_manager), which can be found on the left, under **Server**
>
>    You should see something like this:
>
>    ![nearly empty data manager tool list in Galaxy](../../images/dm-admin-local-data.png)
>
>    If you instead see the message "You do not currently have any Data Managers installed," you may need to restart your Galaxy server.
>
> 3. [Click **all_fasta**](https://my.gat.galaxy.training/?path=/admin/data_manager/table/all_fasta) under **View Tool Data Table Entries**
>
>    You should see the current contents of `tool-data/all_fasta.loc`, which will be empty.
>
> 4. Click **Local Data** to return to the previous page
>
> 5. Click {% tool [Create DBKey and Reference Genome - fetching](toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4) %} under **Installed Data Managers**.
>    - {% icon param-select %} *"Use existing dbkey or create a new one"*: `Existing`
>    - {% icon param-select %} *"DBKEY to assign to data"*: `sacCer2`
>    - {% icon param-text %} *"Name of sequence"*: `S. cerevisiae June 2008 (SGD/sacCer2)`
>    - {% icon param-select %} *"Choose the source for the reference genome"*: `UCSC`
>
>    > <details-title>What do these fields mean?</details-title>
>    >
>    > The data manager tool form fields correspond to the columns in the tool data table, as referenced in `shed_tool_data_table_conf.xml`.
>    >
>    > Tool Field                | Tool Data Table Entry         | Purpose
>    > ---                       | -----                         | ---
>    > *ID for sequence*         | `value` column in `all_fasta` | Identifier of specific build of the genome, if it differs from the DBKEY. Used for variant builds such as the `hg19female` build of `hg19`.
>    > *DBKEY to assign to data* | `dbkey` column in `all_fasta` | Identifier of the genome, the UCSC build ID for UCSC builds like `hg19`.
>    > *Name of sequence*        | `name` column in `all_fasta`  | Display name of the genome, displayed in Galaxy tool dropdowns.
>    {: .details}
>
> 5. Click **Execute**. In your history, you will see a new dataset for the data manager run. When the job has finished, go back to the Data Manager view on the Galaxy Admin page (Click **Local Data**).
> 6. Click **all_fasta** under *View Tool Data Table Entries*
>
>    You should see that *sacCer2* has been added to all_fasta.
>
>    ![populated all_fasta data table which now includes sacCer2 in the genome list](../../images/dm-all-fasta-populated.png)
>
{: .hands_on}

View in the file system where the changes to the location file and the reference genome fasta file.

> <question-title></question-title>
>
> What did this data manager execution change?
>
> > <solution-title></solution-title>
> >
> > - An entry for the *sacCer2* genome has been added to the `all_fasta.loc` file for the `data_manager_fetch_genome_all_fasta_dbkey` data manager. The path to the location file can be found in "shed" versions of the data manager config (`/srv/galaxy/var/config/shed_data_manager_conf.xml`) and data table config (`/srv/galaxy/var/config/shed_tool_data_table_conf.xml`)
> > - The data manager tool is installed along side other Galaxy tools in the shed tools directory
> >
> > > <code-in-title>Bash</code-in-title>
> > > Get the path to the `all_fasta.loc` file:
> > > ```bash
> > > grep 'data_manager_fetch_genome_dbkeys_all_fasta.*all_fasta.loc' /srv/galaxy/var/config/shed_tool_data_table_conf.xml
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```xml
> > >         <file path="/srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc"/>
> > > ```
> > {: .code-out}
> >
> > > <code-in-title>Bash</code-in-title>
> > > Let's investigate the contents of the location file identified above (the path to yours may be different if you installed a different version of the data manager):
> > > ```bash
> > > cat /srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```bash
> > > #This file lists the locations and dbkeys of all the fasta files
> > > #under the "genome" directory (a directory that contains a directory
> > > #for each build). The script extract_fasta.py will generate the file
> > > #all_fasta.loc. This file has the format (white space characters are
> > > #TAB characters):
> > > #
> > > #<unique_build_id>  <dbkey>     <display_name>  <file_path>
> > > #
> > > #So, all_fasta.loc could look something like this:
> > > #
> > > #apiMel3    apiMel3 Honeybee (Apis mellifera): apiMel3      /path/to/genome/apiMel3/apiMel3.fa
> > > #hg19canon  hg19        Human (Homo sapiens): hg19 Canonical        /path/to/genome/hg19/hg19canon.fa
> > > #hg19full   hg19        Human (Homo sapiens): hg19 Full         /path/to/genome/hg19/hg19full.fa
> > > #
> > > #Your all_fasta.loc file should contain an entry for each individual
> > > #fasta file. So there will be multiple fasta files for each build,
> > > #such as with hg19 above.
> > > #
> > > sacCer2	sacCer2	S. cerevisiae June 2008 (SGD/sacCer2)	/srv/galaxy/var/tool-data/sacCer2/seq/sacCer2.fa
> > > ```
> > {: .code-out}
> >
> > > <code-in-title>Bash</code-in-title>
> > > Finally, verify that we have some sequences in the genome fasta file:
> > > ```bash
> > > grep '^>' /srv/galaxy/var/tool-data/sacCer2/seq/sacCer2.fa
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```
> > > >2micron
> > > >chrI
> > > >chrII
> > > >chrIII
> > > >chrIV
> > > >chrIX
> > > >chrM
> > > >chrV
> > > >chrVI
> > > >chrVII
> > > >chrVIII
> > > >chrX
> > > >chrXI
> > > >chrXII
> > > >chrXIII
> > > >chrXIV
> > > >chrXV
> > > >chrXVI
> > > ```
> > {: .code-out}
> >
> {: .solution }
>
{: .question}

## Download and install the BWA data manager

Having the genome is a prerequisite for our ultimate goal, which is to use the sacCer2 genome as a *reference genome* for the BWA tool. BWA, like many tools, needs an *index* of the reference genome, and has its own format for that index. Thankfully, the BWA/BWA-MEM data manager will build that index for us.

In this part we will repeat the same process as when we installed the Fetch Genome data manager, except that we will install the BWA/BWA-MEM data manager this time.

> <hands-on-title>Install the BWA/BWA-MEM Data Manager </hands-on-title>
>
> 1. Access the **Admin** menu from the top bar
> 2. Click **Install and Uninstall**, which can be found on the left, under **Tool Management**
> 3. Enter `bwa_mem_index` in the search interface
> 4. Click on the first hit, having `devteam` as owner
> 5. Click the **Install** button for the latest revision
{: .hands_on}

## Build the BWA index for a reference genome

In this part we will actually build the BWA index for sacCer2. It will automatically be added to our list of available reference genomes in the BWA tool.

> <hands-on-title>Build the sacCer2 BWA index</hands-on-title>
>
> 1. Access the Admin menu from the top bar
> 2. Click **Local Data**, which can be found on the left, under **Server**
> 3. Click **{% tool [BWA-MEM index - builder](toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/bwa_mem_index_builder_data_manager/0.0.5) %}** under **Installed Data Managers**.
>    - {% icon param-select %} *"Source Fasta Sequence"*: `S. cerevisiae June 2008 (SGD/sacCer2)`
> 4. Click **Execute**.
> 5. Verify that the new BWA index for sacCer2 has been built and the .loc file has been filled in. From the **Local Data** page in the Admin section, click on **bwa_mem_indexes** under *View Tool Data Table Entries*
>
>    *S. cerevisiae* sacCer2 should now appear in the list!
{: .hands_on}

> <question-title></question-title>
>
> What changes were made by the **BWA-MEM index builder**?
>
> > <solution-title></solution-title>
> >
> > - An entry for the *sacCer2* BWA/BWA-MEM index has been added to the `bwa_mem_index.loc` file for the `data_manager_bwa_mem_index_builder` data manager.
> > - A symlink to the sacCer2 genome fasta file installed by the Fetch Genome data manager above was created in the sacCer2 BWA index directory, `/srv/galaxy/var/tool-data/sacCer2/bwa_mem_index/sacCer2/`.
> > - The BWA/BWA-MEM index for the sacCer2 genome has been built.
> >
> > > <code-in-title>Bash</code-in-title>
> > >
> > > ```bash
> > > ls -l /srv/galaxy/var/tool-data/sacCer2/bwa_mem_index/sacCer2
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```console
> > > total 20800
> > > lrwxrwxrwx 1 galaxy galaxy       20 Dec  9 18:29 sacCer2.fa -> ../../seq/sacCer2.fa
> > > -rw-r--r-- 1 galaxy galaxy       14 Dec  9 18:29 sacCer2.fa.amb
> > > -rw-r--r-- 1 galaxy galaxy      591 Dec  9 18:29 sacCer2.fa.ann
> > > -rw-r--r-- 1 galaxy galaxy 12163076 Dec  9 18:29 sacCer2.fa.bwt
> > > -rw-r--r-- 1 galaxy galaxy  3040750 Dec  9 18:29 sacCer2.fa.pac
> > > -rw-r--r-- 1 galaxy galaxy  6081552 Dec  9 18:29 sacCer2.fa.sa
> > > ```
> > {: .code-out}
> >
> {: .solution }
>
{: .question}

# Installing and Using Data Managers from the command line with Ephemeris

The same process described in the previous section can also be performed from the command line, e.g. in a CI/CD
pipeline, using Ephemeris. For a more in-depth look at Ephemeris, especially in the tool installation context, please see [the Galaxy Tool Management with Ephemeris tutorial]({% link topics/admin/tutorials/tool-management/tutorial.md %}).

In order to accomplish this, you will need:

- The URL of your Galaxy server
- The API key for your account, which must be an admin

{% snippet faqs/galaxy/preferences_admin_api_key.md admin=true %}

## Install a the Fetch Genome Data Manager from the Tool Shed

> <hands-on-title>Install the Fetch Genome Data Manager with Ephemeris</hands-on-title>
>
> 1. Re-activate the virtualenv you created for the [ephemeris tool management tutorial]({% link topics/admin/tutorials/tool-management/tutorial.md %}).
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > . ~/ephemeris_venv/bin/activate
>    > ```
>    {: .code-in}
>
>    > <tip-title>missing?</tip-title>
>    >
>    > then you might need to re-run the steps:
>    >
>    > ```bash
>    > python3 -m venv ~/ephemeris_venv
>    > . ~/ephemeris_venv/bin/activate
>    > pip install ephemeris
>    > ```
>    {: .tip}
>
> 2. Install the `data_manager_fetch_genome_dbkeys_all_fasta` data manager tool owned by `devteam`.
>
>    > <code-in-title>Bash</code-in-title>
>    > Be sure to adjust the value of `-g` appropriately for your Galaxy server, and replace the value of `-a` with your API key.
>    > ```console
>    > shed-tools install -g https://galaxy.example.org -a <api-key> --name data_manager_fetch_genome_dbkeys_all_fasta --owner devteam
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_x9xeu8ro
>    > (1/1) Installing repository data_manager_fetch_genome_dbkeys_all_fasta from devteam to section "None" at revision 4d3eff1bc421 (TRT: 0:00:00.401143)
>    > 	repository data_manager_fetch_genome_dbkeys_all_fasta installed successfully (in 0:00:25.530604) at revision 4d3eff1bc421
>    > Installed repositories (1): [('data_manager_fetch_genome_dbkeys_all_fasta', '4d3eff1bc421')]
>    > Skipped repositories (0): []
>    > Errored repositories (0): []
>    > All repositories have been installed.
>    > Total run time: 0:00:25.932659
>    > ```
>    {: .code-out}
>
{: .hands-on}

## Download and install a reference genome sequence

> <hands-on-title>Download and install sacCer3 with Ephemeris</hands-on-title>
>
> 1. Create a config file for `run-data-managers` named `fetch-sacCer3.yml`:
>
>    {% raw %}
>    ```yaml
>    data_managers:
>      - id: toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4
>        params:
>          - 'dbkey_source|dbkey': '{{ item.dbkey }}'
>          - 'sequence_name': '{{ item.name }}'
>          - 'reference_source|reference_source_selector': 'ucsc'
>          - 'reference_source|requested_dbkey': '{{ item.dbkey }}'
>        items:
>          - dbkey: sacCer3
>            name: 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)'
>        data_table_reload:
>          - all_fasta
>          - __dbkeys__
>    ```
>    {% endraw %}
>
>    > <details-title>What do these config file compontents mean?</details-title>
>    >
>    > The `run-data-managers` config file options correspond to the options in the data manager tool XML file. To locate the tool XML file for the Fetch Genomes data manager, you can search for it in [the Tool Shed](https://toolshed.g2.bx.psu.edu) the same way you did when installing it via the UI. You can also open the data manager's tool form in the UI as if to run it, and, using the drop-down menu at the top right of the tool form, click "See in Tool Shed". From the Tool Shed, you can click the **Development repository** link and browse to the [tool XML file, `data_manager/data_manager_fetch_genome_all_fasta_dbkeys.xml`](https://github.com/galaxyproject/tools-iuc/blob/8487d2c73793be0afa5b34388b122e686ac8a094/data_managers/data_manager_fetch_genome_dbkeys_all_fasta/data_manager/data_manager_fetch_genome_all_fasta_dbkeys.xml).
>    >
>    > `run-data-managers` config file component          | Purpose
>    > ---                                                | ---
>    > `id`                                               | Data manager full (shed) tool ID, this can be found in `shed_data_manager_conf.xml`
>    > `params`                                           | Data manager tool params, these correspond to `<param>` tags in the tool XML file. Nested paramaters are specified using a pipe character (`|`).
>    > param `dbkey_source|dbkey`                         | Value of `<param name="dbkey" ...>` in `<conditional name="dbkey_source">`.
>    > param `sequence_name`                              | Value of `<param name="sequence_name" ...>`.
>    > param `reference_source|reference_source_selector` | Value of `<param name="reference_source_selector" ...>` in `<conditional name="reference_source">`.
>    > param `reference_source|requested_dbkey`           | Value of `<param name="requested_dbkey" ...>` in `<conditional name="reference_source">`.
>    > `items`                                            | A list of variables to template in to `params`, referenced in param fields with {% raw %}`{{ item }}`{% endraw %}. In the case of genomes, for example, you can run this DM with multiple genomes, or you could give multiple URLs.
>    > `data_table_reload`                                | Names of the data tables you want to reload after your DMs are finished running. This can be important for subsequent data managers.
>    >
>    {: .details}
>
>
> 2. Run the Genome Fetch DM with `run-data-managers`:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```console
>    > run-data-managers -g https://galaxy.example.org -a <api-key> --config fetch-sacCer3.yml
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_f6klyy7v
>    > Running data managers that populate the following source data tables: ['all_fasta']
>    > Dispatched job 1. Running DM: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4" with parameters: {'dbkey_source|dbkey': 'sacCer3', 'dbkey_source|dbkey_name': 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)', 'reference_source|reference_source_selector': 'ucsc', 'reference_source|requested_dbkey': 'sacCer3'}
>    > Job 1 finished with state ok.
>    > Running data managers that index sequences.
>    > Finished running data managers. Results:
>    > Successful jobs: 1
>    > Skipped jobs: 0
>    > Failed jobs: 0
>    > ```
>    {: .code-out}
>
> 3. In the Galaxy UI, access the Admin menu from the top bar
> 4. Click **Local Data**, which can be found on the left, under **Server**
> 5. Click **all_fasta** under *View Tool Data Table Entries*
>
>    You should see that *sacCer3* has been added to all_fasta.
>
>    ![populated all_fasta data table with sacCer3](../../images/dm-all-fasta-populated-2.png)
>
>    > <question-title></question-title>
>    >
>    > Can we view the changes from the command line? Where are they located?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The `all_fasta.loc` file contains the entry for our new genome. If you need a reminder on how to locate it, see the *What did this data manager execution change?* question in the UI section above.
>    > >
>    > > > <code-in-title>Bash</code-in-title>
>    > > > ```bash
>    > > > cat /srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc
>    > > > ```
>    > > {: .code-in}
>    > >
>    > > > <code-out-title>Bash</code-out-title>
>    > > > ```bash
>    > > > #This file lists the locations and dbkeys of all the fasta files
>    > > > #under the "genome" directory (a directory that contains a directory
>    > > > #for each build). The script extract_fasta.py will generate the file
>    > > > #all_fasta.loc. This file has the format (white space characters are
>    > > > #TAB characters):
>    > > > #
>    > > > #<unique_build_id>  <dbkey>     <display_name>  <file_path>
>    > > > #
>    > > > #So, all_fasta.loc could look something like this:
>    > > > #
>    > > > #apiMel3    apiMel3 Honeybee (Apis mellifera): apiMel3      /path/to/genome/apiMel3/apiMel3.fa
>    > > > #hg19canon  hg19        Human (Homo sapiens): hg19 Canonical        /path/to/genome/hg19/hg19canon.fa
>    > > > #hg19full   hg19        Human (Homo sapiens): hg19 Full         /path/to/genome/hg19/hg19full.fa
>    > > > #
>    > > > #Your all_fasta.loc file should contain an entry for each individual
>    > > > #fasta file. So there will be multiple fasta files for each build,
>    > > > #such as with hg19 above.
>    > > > #
>    > > > sacCer2	sacCer2	S. cerevisiae June 2008 (SGD/sacCer2)	/srv/galaxy/var/tool-data/sacCer2/seq/sacCer2.fa
>    > > > sacCer3	sacCer3	S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)	/srv/galaxy/var/tool-data/sacCer3/seq/sacCer3.fa
>    > > > ```
>    > > {: .code-out}
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands-on}

> <warning-title>run-data-managers is not idempotent!</warning-title>
> Unlike `shed-tools install`, the Ephemeris `run-data-managers` utility is not idempotent. If run a second time on the same set of inputs, you will end up with two entries in your `all_fasta` data table, with the data from the second run overwriting the data from the first run.
>
> Please see [Galaxy issue #15188](https://github.com/galaxyproject/galaxy/issues/15188) for details.
>
{: .warning}

## Download and install the BWA data manager

> <hands-on-title>Install the BWA/BWA-MEM Data Manager </hands-on-title>
>
> 1. Install the `bwa_mem_index_builder_data_manager` data manager tool owned by `devteam`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```console
>    > shed-tools install -g https://galaxy.example.org -a <api-key> --name data_manager_bwa_mem_index_builder --owner devteam
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_2dyujjvi
>    > (1/1) Installing repository data_manager_bwa_mem_index_builder from devteam to section "None" at revision 63d5652be07a (TRT: 0:00:00.204350)
>    > 	repository data_manager_bwa_mem_index_builder installed successfully (in 0:00:05.599382) at revision 63d5652be07a
>    > Installed repositories (1): [('data_manager_bwa_mem_index_builder', '63d5652be07a')]
>    > Skipped repositories (0): []
>    > Errored repositories (0): []
>    > All repositories have been installed.
>    > Total run time: 0:00:05.804217
>    > ```
>    {: .code-out}
>
{: .hands-on}

## Build the BWA index for a reference genome

> <hands-on-title>Build the sacCer3 BWA index</hands-on-title>
>
> 1. Create a config file for `run-data-managers` named `build-sacCer3-bwa.yml`:
>
>    {% raw %}
>    ```yaml
>    data_managers:
>      - id: toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/bwa_mem_index_builder_data_manager/0.0.5
>        params:
>          - 'all_fasta_source': '{{ item.dbkey }}'
>          - 'sequence_name': '{{ item.name }}'
>        items:
>          - dbkey: sacCer3
>            name: 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)'
>        data_table_reload:
>          - bwa_mem_indexes
>    ```
>    {% endraw %}
>
> 2. Run the BWA-MEM index builder DM with `run-data-managers`:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```console
>    > run-data-managers -g https://galaxy.example.org -a <api-key> --config build-sacCer3-bwa.yml
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_esecdef4
>    > Running data managers that populate the following source data tables: ['all_fasta']
>    > Running data managers that index sequences.
>    > Dispatched job 2. Running DM: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/bwa_mem_index_builder_data_manager/0.0.5" with parameters: {'all_fasta_source': 'sacCer3', 'sequence_name': 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)'}
>    > Job 2 finished with state ok.
>    > Finished running data managers. Results:
>    > Successful jobs: 1
>    > Skipped jobs: 0
>    > Failed jobs: 0
>    > ```
>    {: .code-out}
>
> 3. In the Galaxy UI, access the Admin menu from the top bar
> 4. Click **Local Data**, which can be found on the left, under **Server**
> 5. Click **bwa_mem_indexes** under *View Tool Data Table Entries*
>
>    You should see that *sacCer3* has been added to bwa_mem_indexes. You can also verify this via the command line using the methods you've learned above.
>
{: .hands-on}

# Verify that BWA can access the new reference data

Finally, we will verify that the BWA tool can see the new genome indexes.

> <hands-on-title>Configure Galaxy</hands-on-title>
>
> 1. Install the BWA and BWA-MEM tools, if needed. If installing with Ephemeris, the repo name for the `--name` parameter is `bwa` and the owner is `devteam`.
>
>    {% snippet topics/admin/faqs/install_tool.md query="bwa" name="Map with BWA" section="Mapping" %}
>
> 2. Click the "Home" icon at the top to return to the Galaxy analysis interface
>
> 3. Load the {% tool [Map with BWA](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.5) %} tool and verify that the sacCer2 genome (if you completed the "Galaxy UI" section) and/or sacCer3 genome (if you completed the "command line with Ephemeris" section)  appears in the {% icon param-select %} *"Using reference genome"* option.
{: .hands_on}

How cool is that? No editing `.loc` files, no making sure you've got TABs instead of spaces. Fully auto!

{% snippet topics/admin/faqs/git-gat-path.md tutorial="reference-genomes" %}


topic: admin
tutorial: connect-to-compute-cluster
title: Connecting Galaxy to a compute cluster
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["jobs", "ansible", "git-gat"]
content_hash: 10c4c8575f77de6fff41ac098a886c12
last_modified: 2026-02-26T21:12:14.139045
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/connect-to-compute-cluster/tutorial.html

content:
The tools that are added to Galaxy can have a wide variance in the compute resources that they require and work efficiently on.
To account for this, Galaxy's job configuration needs to be tuned to run these tools properly. In addition, site-specific variables must
be taken into consideration when choosing where to run jobs and what parameters to run them with.

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="connect-to-compute-cluster" %}

## Installing Slurm

> <comment-title>Ansible Best Practices</comment-title>
> If you've set up your Galaxy server using the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, you will have created a `galaxyservers` group in your inventory file, `hosts`, and placed your variables in `group_vars/galaxyservers.yml`. Although for the purposes of this tutorial, the Galaxy server and Slurm controller/node are one and the same, in a real world deployment they are very likely to be different hosts. We will continue to use the `galaxyservers` group for simplicity, but in your own deployment you should consider creating some additional groups for Slurm controller(s), Slurm nodes, and Slurm clients.
{: .comment}

> <tip-title>Do you need a DRM?</tip-title>
> If you have a smaller server, do you still need a DRM? Yes! You should definitely run Slurm or a similar option. If you don't, as soon as you restart Galaxy with local runners, any running jobs will be killed. Even with a handful of users, it is a good idea to keep 1-2 CPU cores/4GB RAM reserved for Galaxy.
{: .tip}

> <hands-on-title>Installing Slurm</hands-on-title>
>
> 1. Edit your `requirements.yml` and include the following contents:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -22,3 +22,8 @@
>     # Singularity/Apptainer
>     - src: usegalaxy_eu.apptainer
>       version: 0.0.3
>    +# SLURM as our DRM
>    +- src: galaxyproject.repos
>    +  version: 0.0.3
>    +- src: galaxyproject.slurm
>    +  version: 1.0.2
>    {% endraw %}
>    ```
>    {: data-commit="Add requirements"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>    The `galaxyproject.repos` role adds the [Galaxy Packages for Enterprise Linux (GPEL)](https://depot.galaxyproject.org/yum/) repository for RedHat/CentOS, which provides both Slurm and Slurm-DRMAA (neither are available in standard repositories or EPEL). For Ubuntu versions 18.04 or newer, it adds the [Slurm-DRMAA PPA](https://launchpad.net/~natefoo/+archive/ubuntu/slurm-drmaa) (Slurm-DRMAA was removed from Debian/Ubuntu in buster/bionic).
>
> 2. In the same directory, run:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add `galaxyproject.repos`, `galaxyproject.slurm` to the *beginning* of your roles section in your `galaxy.yml` playbook:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -34,6 +34,8 @@
>             dest: /libraries/
>       roles:
>         - galaxyproject.tusd
>    +    - galaxyproject.repos
>    +    - galaxyproject.slurm
>         - usegalaxy_eu.apptainer
>         - galaxyproject.galaxy
>         - role: galaxyproject.miniconda
>    {% endraw %}
>    ```
>    {: data-commit="Add the repos and slurm roles"}
>
> 4. Add the slurm variables to your `group_vars/galaxyservers.yml`:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -199,6 +199,17 @@ nginx_ssl_role: usegalaxy_eu.certbot
>     nginx_conf_ssl_certificate: /etc/ssl/certs/fullchain.pem
>     nginx_conf_ssl_certificate_key: /etc/ssl/user/privkey-www-data.pem
>
>    +# Slurm
>    +slurm_roles: ['controller', 'exec'] # Which roles should the machine play? exec are execution hosts.
>    +slurm_nodes:
>    +- name: localhost # Name of our host
>    +  CPUs: 2         # Here you would need to figure out how many cores your machine has. For this training we will use 2 but in real life, look at `htop` or similar.
>    +  RealMemory: 8192  # Adjust based on available memory. For this training 8192 is sufficient.
>    +slurm_config:
>    +  SlurmdParameters: config_overrides   # Ignore errors if the host actually has cores != 2
>    +  SelectType: select/cons_tres
>    +  SelectTypeParameters: CR_CPU_Memory  # Allocate individual cores/memory instead of entire node
>    +
>     # TUS
>     galaxy_tusd_port: 1080
>     galaxy_tus_upload_store: /data/tus
>    {% endraw %}
>    ```
>    {: data-commit="Add slurm configuration"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

Note that the above Slurm config options are only those that are useful for this training exercise. In production, you would want to use a more appropriate configuration specific to your cluster (and setting `SlurmdParameters` to `config_overrides` is not recommended).

Installed with Slurm is MUNGE (MUNGE Uid 'N Gid Emporium...) which authenticates users between cluster hosts. You would normally need to ensure the same Munge key is distributed across all cluster hosts (in `/etc/munge/munge.key`) - A great task for Ansible. However, the installation of the munge package has created a random key for you, and you will not need to distribute this since you'll run jobs only on a single host.

You can now check that all of the daemons are running with the command `systemctl status munge slurmd slurmctld`

```console
$ sudo systemctl status munge slurmd slurmctld
● munge.service - MUNGE authentication service
  Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; vendor preset: disabled)
   Active: active (running) since Sa 2019-01-26 22:38:13 CET; 28min ago
     Docs: man:munged(8)
 Main PID: 22930 (munged)
    Tasks: 4
   Memory: 128.0K
   CGroup: /system.slice/munge.service
           └─22930 /usr/sbin/munged

Jan 26 22:38:13 helena-test.novalocal systemd[1]: Starting MUNGE authentication service...
Jan 26 22:38:13 helena-test.novalocal systemd[1]: Started MUNGE authentication service.

● slurmd.service - Slurm node daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sa 2019-01-26 23:04:21 CET; 2min 25s ago
  Process: 15051 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 15054 (slurmd)
    Tasks: 1
   Memory: 628.0K
   CGroup: /system.slice/slurmd.service
           └─15054 /usr/sbin/slurmd

Jan 26 23:04:21 helena-test.novalocal systemd[1]: Starting Slurm node daemon...
Jan 26 23:04:21 helena-test.novalocal systemd[1]: PID file /var/run/slurmd.pid not readable (yet?) after start.
Jan 26 23:04:21 helena-test.novalocal systemd[1]: Started Slurm node daemon.

● slurmctld.service - Slurm controller daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Sa 2019-01-26 23:04:20 CET; 2min 26s ago
  Process: 15040 ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 15042 (slurmctld)
    Tasks: 7
   Memory: 1.1M
   CGroup: /system.slice/slurmctld.service
           └─15042 /usr/sbin/slurmctld

Jan 26 23:04:20 helena-test.novalocal systemd[1]: Starting Slurm controller daemon...
Jan 26 23:04:20 helena-test.novalocal systemd[1]: PID file /var/run/slurmctld.pid not readable (yet?) after start.
Jan 26 23:04:20 helena-test.novalocal systemd[1]: Started Slurm controller daemon.
```

Running the playbook, the Slurm configuration, `/etc/slurm/slurm.conf` (or `/etc/slurm-llnl/slurm.conf` on Debian-based distributions) was created for you automatically. All of the variables were set by default. If you need to override the configuration yourself, Slurm provides [an online tool](https://slurm.schedmd.com/configurator.html) which will help you configure it.

## Using Slurm

You should now be able to see that your Slurm cluster is operational with the `sinfo` command. This shows the state of nodes and partitions (synonymous with queues in other DRMs). The "node-oriented view" provided with the `-N` flag is particularly useful:

```console
$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug*       up   infinite      1   idle localhost
$ sinfo -Nel
Fri Nov  4 16:51:24 2016
NODELIST   NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT FEATURES REASON
localhost      1    debug*        idle    1    2:1:1      1        0      1   (null) none
```

If your node state is not `idle`, something has gone wrong. If your node state ends with an asterisk \*, the Slurm controller is attempting to contact the Slurm execution daemon but has not yet been successful (the \* next to the partition name is normal, it indicates the default partition).

We want to ensure that Slurm is actually able to run jobs. There are two ways this can be done:

- `srun`: Run an interactive job (e.g. a shell, or a specific program with its stdin, stdout, and stderr all connected to your terminal.
- `sbatch`: Run a batch job, with stdin closed and stdout/stderr redirected to a file.

Galaxy runs `sbatch` jobs but we can use both `srun` and `sbatch` to test:


> <hands-on-title>Running commands with `srun`</hands-on-title>
>
> 1. Use [`srun`](https://slurm.schedmd.com/srun.html) to run the command `uname -a`
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > srun uname -a
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > Your output may look slightly different:
>    > ```console
>    > $ srun uname -a
>    > Linux gat-1.oz.training.galaxyproject.eu 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>    > ```
>    {: .code-out}
>
{: .hands_on}

Although it looks like this command ran as if I had not used `srun`, it was in fact routed through Slurm.

> <hands-on-title>Running commands with `sbatch`</hands-on-title>
>
> 1. Create a test job script somewhere, such as in `~/sbatch-test.sh`. It should be a batch script which runs `uname -a`, `uptime`, and sleeps for 30 seconds.
>
>    > <question-title></question-title>
>    >
>    > What does your shell script look like?
>    >
>    > > <solution-title></solution-title>
>    > > ```bash
>    > > #!/bin/bash
>    > > uname -a
>    > > uptime
>    > > sleep 30
>    > > ```
>    > {: .solution }
>    {: .question}
>
> 2. Make the script executable:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > chmod +x ~/sbatch-test.sh
>    > ```
>    {: .code-in}
>
> 3. Use [`sbatch`](https://slurm.schedmd.com/sbatch.html) to submit the job script
>
>    > <question-title></question-title>
>    >
>    > What command did you run?
>    >
>    > > <solution-title></solution-title>
>    > > ```console
>    > > $ sbatch ~/sbatch-test.sh
>    > > ```
>    > {: .solution }
>    {: .question}
>
> 4. Use [`squeue`](https://slurm.schedmd.com/squeue.html) to check the queue
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > squeue
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > Your output may look slightly different:
>    > ```console
>    >JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
>    >    3     debug sbatch-t   ubuntu  R       0:22      1 localhost
>    > ```
>    {: .code-out}
>
{: .hands_on}

If you've made it this far, your Slurm installation is working!

## Slurm-DRMAA

Above Slurm in the stack is slurm-drmaa, a library that provides a translational interface from the Slurm API to the generalized DRMAA API in C.

> <hands-on-title>Installing Slurm-DRMAA</hands-on-title>
>
> 1. Add a `post_task` to your playbook to install `slurm-drmaa1` (Debian/Ubuntu) or `slurm-drmaa` (RedHat/CentOS).
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -52,3 +52,6 @@
>             minute: "0"
>             hour: "0"
>             job: "SHELL=/bin/bash source {{ galaxy_venv_dir }}/bin/activate &&  GALAXY_LOG_DIR=/tmp/gxadmin/ GALAXY_ROOT={{ galaxy_root }}/server GALAXY_CONFIG_FILE={{ galaxy_config_file }} /usr/local/bin/gxadmin galaxy cleanup 60"
>    +    - name: Install slurm-drmaa
>    +      package:
>    +        name: slurm-drmaa1
>    {% endraw %}
>    ```
>    {: data-commit="Add post task to install slurm-drmaa"}
>
> 2. Run the playbook (`ansible-playbook galaxy.yml`)
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

Moving one level further up the stack, we find DRMAA Python. This is a Galaxy framework *conditional dependency*. Conditional dependencies are only installed if, during startup, a configuration option is set that requires that dependency. The `galaxyproject.galaxy` Ansible role will install these conditional dependencies, automatically.

# Galaxy and Slurm

At the top of the stack sits Galaxy. Galaxy must now be configured to use the cluster we've just set up. The DRMAA Python documentation (and Galaxy's own documentation) instruct that you should set the `$DRMAA_LIBRARY_PATH` environment variable so that DRMAA Python can find `libdrmaa.so` (aka slurm-drmaa). Because Galaxy runs under systemd, the environment that Galaxy starts under is controlled by the `environment` option in systemd service unit that the ansible role manages. The galaxy task should thus be updated to refer to the path to slurm-drmaa, which is `/usr/lib/slurm-drmaa/lib/libdrmaa.so.1`:


> <hands-on-title>Making Galaxy aware of DRMAA</hands-on-title>
>
> 3. We need to configure the Slurm job runner. First, we instruct Galaxy's job handlers to load the Slurm job runner plugin, and set the Slurm job submission parameters. A job runner plugin definition must have a unique name as its key, and a `load` attribute. Since we already have a good default destination that uses singularity, we will simply modify that to use the slurm runner. Galaxy will do the equivalent of submitting a job as `sbatch /path/to/job_script.sh`. <!-- Note that we also need to set a default destination now that more than one destination is defined. --> In a job environment, the key is a unique identifier for that environment and the `runner` attribute must match the key of a defined runner plugin:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -21,14 +21,29 @@ galaxy_job_config:
>         local_runner:
>           load: galaxy.jobs.runners.local:LocalJobRunner
>           workers: 4
>    +    slurm:
>    +      load: galaxy.jobs.runners.slurm:SlurmJobRunner
>    +      drmaa_library_path: /usr/lib/slurm-drmaa/lib/libdrmaa.so.1
>       handling:
>         assign: ['db-skip-locked']
>       execution:
>    -    default: singularity
>    +    default: slurm
>         environments:
>           local_env:
>             runner: local_runner
>             tmp_dir: true
>    +      slurm:
>    +        runner: slurm
>    +        singularity_enabled: true
>    +        # Enabling access to the reference data on CVMFS in the container
>    +        singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    +        env:
>    +        - name: LC_ALL
>    +          value: C
>    +        - name: APPTAINER_CACHEDIR
>    +          value: /tmp/singularity
>    +        - name: APPTAINER_TMPDIR
>    +          value: /tmp
>           singularity:
>             runner: local_runner
>             singularity_enabled: true
>    {% endraw %}
>    ```
>    {: data-commit="Configure slurm destination"}
>
> 4. Run your Galaxy playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 5. Watch the logs to check that everything loads correctly
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > journalctl -f -u 'galaxy-*'
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > Your output may look slightly different:
>    > ```console
>    > Jan 12 15:46:01 gat-1.oz.training.galaxyproject.eu gunicorn[1821134]: galaxy.jobs.runners DEBUG 2021-01-12 15:46:01,109 [p:1821134,w:0,m:1] [MainThread] Starting 4 SlurmRunner workers
>    > Jan 12 15:46:01 gat-1.oz.training.galaxyproject.eu gunicorn[1821134]: galaxy.jobs DEBUG 2021-01-12 15:46:01,110 [p:1821134,w:0,m:1] [MainThread] Loaded job runner 'galaxy.jobs.runners.slurm:SlurmJobRunner' as 'slurm'
>    > ```
>    {: .code-out}
>
{: .hands_on}


## Running a Job

You should now be able to run a Galaxy job through Slurm. The simplest way to test is using the digest tool to calculate an md5sum.

> <hands-on-title>Testing a Slurm Job</hands-on-title>
>
> 1. If you're not still following the log files with `journalctl`, do so now.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > journalctl -f -u 'galaxy-*'
>    > ```
>    {: .code-in}
>
> 2. Click the upload button at the top of the tool panel (on the left side of the Galaxy UI).
> 3. In the resulting modal dialog, click the "Paste/Fetch data" button.
> 4. Type some random characters into the text field that has just appeared.
> 5. Click "Start" and then "Close"
> 5. Open [*Secure Hash / Message Digest*](https://my.gat.galaxy.training/?path=/%3Ftool_id=secure_hash_message_digest) tool
>
>    > <code-out-title></code-out-title>
>    > Your output may look slightly different. In your `journalctl` terminal window you should see the following messages:
>    >
>    > ```
>    > galaxy.jobs.mapper DEBUG 2020-02-10 09:37:17,946 [p:9859,w:0,m:2] [JobHandlerQueue.monitor_thread] (1) Mapped job to destination id: slurm
>    > galaxy.jobs.handler DEBUG 2020-02-10 09:37:17,976 [p:9859,w:0,m:2] [JobHandlerQueue.monitor_thread] (1) Dispatching to slurm runner
>    > galaxy.jobs DEBUG 2020-02-10 09:37:18,016 [p:9859,w:0,m:2] [JobHandlerQueue.monitor_thread] (1) Persisting job destination (destination id: slurm)
>    > galaxy.jobs DEBUG 2020-02-10 09:37:18,021 [p:9859,w:0,m:2] [JobHandlerQueue.monitor_thread] (1) Working directory for job is: /srv/galaxy/jobs/000/1
>    > galaxy.jobs.runners DEBUG 2020-02-10 09:37:18,358 [p:9859,w:0,m:2] [JobHandlerQueue.monitor_thread] Job [1] queued (380.809 ms)
>    > galaxy.jobs.handler INFO 2020-02-10 09:37:18,372 [p:9859,w:0,m:2] [JobHandlerQueue.monitor_thread] (1) Job dispatched
>    > galaxy.jobs.command_factory INFO 2020-02-10 09:37:18,564 [p:9859,w:0,m:2] [SlurmRunner.work_thread-0] Built script [/srv/galaxy/jobs/000/1/tool_script.sh] for tool command [python '/srv/galaxy/server/tools/data_source/upload.py' /srv/galaxy/server /srv/galaxy/jobs/000/1/registry.xml /srv/galaxy/jobs/000/1/upload_params.json 1:/srv/galaxy/jobs/000/1/working/dataset_1_files:/data/000/dataset_1.dat]
>    > ...
>    > galaxy.jobs.runners.drmaa DEBUG 2020-02-10 09:37:18,645 [p:9859,w:0,m:2] [SlurmRunner.work_thread-0] (1) submitting file /srv/galaxy/jobs/000/1/galaxy_1.sh
>    > galaxy.jobs.runners.drmaa INFO 2020-02-10 09:37:18,654 [p:9859,w:0,m:2] [SlurmRunner.work_thread-0] (1) queued as 4
>    > galaxy.jobs DEBUG 2020-02-10 09:37:18,654 [p:9859,w:0,m:2] [SlurmRunner.work_thread-0] (1) Persisting job destination (destination id: slurm)
>    > ```
>    >
>    > At this point the job has been accepted by Slurm and is awaiting scheduling on a node. Once it's been sent to a node and starts running, Galaxy logs this event:
>    >
>    > ```
>    > galaxy.jobs.runners.drmaa DEBUG 2020-02-10 09:37:19,537 [p:9859,w:0,m:2] [SlurmRunner.monitor_thread] (1/4) state change: job is running
>    > ```
>    >
>    > Finally, when the job is complete, Galaxy performs its job finalization process:
>    >
>    > ```
>    > galaxy.jobs.runners.drmaa DEBUG 2020-02-10 09:37:24,700 [p:9859,w:0,m:2] [SlurmRunner.monitor_thread] (1/4) state change: job finished normally
>    > galaxy.model.metadata DEBUG 2020-02-10 09:37:24,788 [p:9859,w:0,m:2] [SlurmRunner.work_thread-1] loading metadata from file for: HistoryDatasetAssociation 1
>    > galaxy.jobs INFO 2020-02-10 09:37:24,883 [p:9859,w:0,m:2] [SlurmRunner.work_thread-1] Collecting metrics for Job 1 in /srv/galaxy/jobs/000/1
>    > galaxy.jobs DEBUG 2020-02-10 09:37:24,917 [p:9859,w:0,m:2] [SlurmRunner.work_thread-1] job_wrapper.finish for job 1 executed (154.514 ms)
>    > ```
>    >
>    > Note a few useful bits in the output:
>    > - `Persisting job destination (destination id: slurm)`: Galaxy has selected the `slurm` destination we defined
>    > - `submitting file /srv/galaxy/server/database/jobs/000/2/galaxy_2.sh`: This is the path to the script that is submitted to Slurm as it would be with `sbatch`
>    > - `(1) queued as 4`: Galaxy job id "1" is Slurm job id "4", this can also be seen with the `(1/4)` in other output lines.
>    > - If `job <id> ended` is reached, the job should show as done in the UI
>    {: .code-out}
>
{: .hands_on}

> ```bash
> 1-test-cat1.sh
> ```
> {: data-test="true"}
{: .hidden}

Slurm allows us to query the exit state of jobs for a time period of the value of Slurm's `MinJobAge` option, which defaults to 300 (seconds, == 5 minutes):

> <code-in-title>Bash</code-in-title>
> Your job number is potentially different.
> ```
> scontrol show job 4
> ```
{: .code-in}

> <code-out-title></code-out-title>
> Your output may also look slightly different:
> ```console
> JobId=4 JobName=g1_upload1_admin_example_org
>    UserId=galaxy(999) GroupId=galaxy(999) MCS_label=N/A
>    Priority=4294901757 Nice=0 Account=(null) QOS=(null)
>    JobState=COMPLETED Reason=None Dependency=(null)
>    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>    RunTime=00:00:05 TimeLimit=UNLIMITED TimeMin=N/A
>    SubmitTime=2020-02-10T09:37:18 EligibleTime=2020-02-10T09:37:18
>    StartTime=2020-02-10T09:37:19 EndTime=2020-02-10T09:37:24 Deadline=N/A
>    PreemptTime=None SuspendTime=None SecsPreSuspend=0
>    LastSchedEval=2020-02-10T09:37:19
>    Partition=debug AllocNode:Sid=gcc-1:9453
>    ReqNodeList=(null) ExcNodeList=(null)
>    NodeList=localhost
>    BatchHost=localhost
>    NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>    TRES=cpu=1,mem=1M,node=1,billing=1
>    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
>    MinCPUsNode=1 MinMemoryNode=1M MinTmpDiskNode=0
>    Features=(null) DelayBoot=00:00:00
>    Gres=(null) Reservation=(null)
>    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
>    Command=(null)
>    WorkDir=/srv/galaxy/jobs/000/1
>    StdErr=/srv/galaxy/jobs/000/1/galaxy_1.e
>    StdIn=StdIn=/dev/null
>    StdOut=/srv/galaxy/jobs/000/1/galaxy_1.o
>    Power=
> ```
{: .code-out}

After the job has been purged from the active jobs database, a bit of information (but not as much as `scontrol` provides) can be retrieved from Slurm's logs. However, it's a good idea to set up Slurm's accounting database to keep old job information in a queryable format.

> <tip-title>Which directories need to be shared on a cluster?</tip-title>
> The following directories need to be accesible via the same path on both the head node and compute nodes:
> - `galaxy_shed_tools_dir`
> - `galaxy_tool_dependency_dir`
> - `galaxy_file_path`
> - `galaxy_job_working_directory`
> - `galaxy_server_dir`
> - `galaxy_venv_dir`
{: .tip}

# Recording Job Metrics

Job metrics record properties of the jobs that are executed, information that can help you plan for trainings or plan capacity for further expansions of your Galaxy server. These properties include details such as the number of slots (cores) assigned to a job, the amount of memory available, details about the node on which the job executed, environment variables that were set at execution time, and more.

Galaxy collects and records very few job metrics by default, enabling more metrics plugins is recommended for any cluster-enabled Galaxy deployment. The metrics are stored in the Galaxy database, which can be queried externally to generate reports and debug job problems.

Some work has been done to try to analyse job runtime metrics to optimise cluster allocation based on job inputs, and enhance job submission ({% cite Tyryshkina_2019 %}). More work will be done in this area.

> <comment-title>Note</comment-title>
>
> Job metrics are only visible to Galaxy *admin users*, unless you set `expose_potentially_sensitive_job_metrics: true`, like UseGalaxy.eu does. EU's intention with this is to empower users and make everything as transparent as possible.
>
> This is the only option controlling which metrics general users see. Admins see all metrics collected, and by default general users see none.
> However most of the metrics exposed by this setting are quite safe (e.g. cgroups information on resource consumption, walltime, etc.)
>
{: .comment}

## Setting up Galaxy

By default, Galaxy enables the `core` metrics:

![screenshot of galaxy metrics](../../images/job-metrics-basic.png)

These include very basic submission parameters. We want more information!

> <hands-on-title>Setting up the job metrics plugin configuration</hands-on-title>
>
> 1. Edit the **global** (for all hosts) group variables file, `group_vars/all.yml`:
>
>    > <details-title>Why are we editing "all" instead of "galaxyservers" vars?</details-title>
>    > Both Galaxy and Pulsar use job metrics plugins, and when we configure Pulsar later, we will want it to have the same metrics plugin configuration as Galaxy. Putting this variable in `all.yml` will allow us to refer to it later when setting the corresponding variable for Pulsar.
>    {: .details}
>
>    The variable we'll set is named `galaxy_job_metrics_plugins`:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/all.yml
>    +++ b/group_vars/all.yml
>    @@ -10,3 +10,13 @@ galaxy_db_name: galaxy
>     cvmfs_role: client
>     galaxy_cvmfs_repos_enabled: config-repo
>     cvmfs_quota_limit: 500
>    +
>    +# Galaxy vars that will be reused by Pulsar
>    +galaxy_job_metrics_plugins:
>    +  - type: core
>    +  - type: cpuinfo
>    +  - type: meminfo
>    +  - type: uname
>    +  - type: env
>    +  - type: cgroup
>    +  - type: hostname
>    {% endraw %}
>    ```
>    {: data-commit="Configure job metrics plugins"}
>
> 2. Run your Galaxy playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

Currently, the job metrics plugin configuration is stored in a separate configuration file from Galaxy's main configuration file (`galaxy.yml`). By setting `galaxy_job_metrics_plugins`, we instructed the `galaxyproject.galaxy` role to create this file, and update the option (`job_metrics_config_file`) in `galaxy.yml` that sets the path to this file. You can inspect the contents of the new config file on your Galaxy server:

> <code-in-title>Bash</code-in-title>
> ```bash
> cat /srv/galaxy/config/job_metrics_conf.yml
> ```
{: .code-in}

> <code-out-title>Bash</code-out-title>
> ```yaml
> ---
> ##
> ## This file is managed by Ansible.  ALL CHANGES WILL BE OVERWRITTEN.
> ##
> -   type: core
> -   type: cpuinfo
> -   type: meminfo
> -   type: uname
> -   type: env
> -   type: cgroup
> -   type: hostname
> ```
{: .code-out.code-max-300}


## Generating Metrics

With this, the job metrics collection and recording should be set up. Now when you run a job, you will see many more metrics:

> <hands-on-title>Generate some metrics</hands-on-title>
>
> 1. Run a job (any tool is fine, even upload)
>
> 2. View the information of the output dataset ({% icon galaxy-info %})
>
{: .hands_on}

![advanced metrics](../../images/job-metrics-advanced.png)


## What should I collect?

There is not a good rule we can tell you, just choose what you think is useful or will be. Numeric parameters are "cheaper" than the text parameters (like uname to store), eventually you may find yourself wanting to remove old job metrics if you decide to collect the environment variables or similar.


## Accessing the data

You can access the data via BioBlend ([`JobsClient.get_metrics`](https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend.galaxy.jobs.JobsClient.get_metrics)), or via SQL with [`gxadmin`](https://galaxyproject.github.io/gxadmin/#/README.query?id=query-tool-metrics).


{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=9 %}

## Further Reading

- [Galaxy's cluster documentation](https://docs.galaxyproject.org/en/latest/admin/cluster.html) describes in detail alternative cluster configurations.
- [The job_conf documentation](https://docs.galaxyproject.org/en/latest/admin/jobs.html) fully describes the syntax of the job configuration file.
- The [Distributed Resource Management Application API (DRMAA)](https://www.drmaa.org/) page contains the DRMAA specification as well as documentation for various implementations. It also includes a list of DRMs supporting DRMAA.
- The [Slurm documentation](http://slurm.schedmd.com/) is extensive and covers all the features and myriad of ways in which you can configure slurm.
- [PSNC slurm-drmaa](http://apps.man.poznan.pl/trac/slurm-drmaa)'s page includes documentation and the SVN repository, which has a few minor fixes since the last released version. PSNC also wrote the initial implementations of the DRMAA libraries for PBSPro and LSF, so all three are similar.
- [Our own fork of slurm-drmaa](http://github.com/natefoo/slurm-drmaa) includes support for Slurms `-M`/`--clusters` multi-cluster functionality and newer versions of Slurm.
- [Slurm Accounting documentation](http://slurm.schedmd.com/accounting.html) explains how to set up SlurmDBD.

{% snippet topics/admin/faqs/git-gat-path.md tutorial="connect-to-compute-cluster" %}


topic: admin
tutorial: reports
title: Galaxy Monitoring with Reports
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/reports/tutorial.html
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "monitoring", "git-gat", "broken"]
content_hash: 1b28c83ec4a1055bda3bda2d8382f8be
last_modified: 2026-02-26T21:12:14.152262
workflows_json: []
content:
The reports application gives some pre-configured analytics screens. These are very easy to setup and can help with debugging issues in Galaxy.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="reports" %}

# Setting up Reports

The reports application is included with the Galaxy codebase and this tutorial assumes you've already done all of the setup required for Galaxy, systemd, uWSGI, and NGINX.

> <hands-on-title>Setup Reports</hands-on-title>
>
>
> 1. First we add a basic configuration of the Reports app to the playbook templates. Create `templates/galaxy/config/` folder, if it doesn't exist, and create `templates/galaxy/config/reports.yml` with the following contents:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/galaxy/config/reports.yml
>    @@ -0,0 +1,4 @@
>    +reports:
>    +    database_connection: "{{ galaxy_config.galaxy.database_connection }}"
>    +    file_path: "{{ galaxy_config.galaxy.file_path }}"
>    +    template_cache_path: "{{ galaxy_mutable_data_dir }}/compiled_templates/reports/"
>    {% endraw %}
>    ```
>    {: data-commit="Setup reports config file"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. In your `galaxyservers` group variables file, tell the playbook to deploy the reports configuration file, and gravity to manage reports:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -151,6 +151,11 @@ galaxy_config:
>             pools:
>               - job-handlers
>               - workflow-schedulers
>    +    reports:
>    +      enable: true
>    +      url_prefix: /reports
>    +      bind: "unix:{{ galaxy_mutable_config_dir }}/reports.sock"
>    +      config_file: "{{ galaxy_config_dir }}/reports.yml"
>
>     galaxy_job_config_file: "{{ galaxy_config_dir }}/galaxy.yml"
>
>    @@ -171,6 +176,8 @@ galaxy_config_templates:
>         dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
>       - src: templates/galaxy/config/job_resource_params_conf.xml.j2
>         dest: "{{ galaxy_config.galaxy.job_resource_params_file }}"
>    +  - src: templates/galaxy/config/reports.yml
>    +    dest: "{{ galaxy_config.gravity.reports.config_file }}"
>
>     galaxy_extra_dirs:
>       - /data
>    {% endraw %}
>    ```
>    {: data-commit="Enable gravity to manage reports"}
>
>
> 4. Then we need to tell NGINX it should serve our Reports app under `<server_url>/reports` url. Edit your `templates/nginx/galaxy.j2` file, and within the server block, add a block for proxying the reports application. It should look like:
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -103,4 +103,10 @@ server {
>     		proxy_set_header Upgrade $http_upgrade;
>     		proxy_set_header Connection "upgrade";
>     	}
>    +
>    +	location /reports/ {
>    +		proxy_pass http://{{ galaxy_config.gravity.reports.bind }}:/;
>    +		proxy_set_header X-Forwarded-Host $host;
>    +		proxy_set_header X-Forwarded-Proto $scheme;
>    +	}
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Configure a location block for Reports in NGINX"}
>
> 5. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 6. The reports application should be available, under [`/reports`](https://my.gat.galaxy.training/?path=/reports)
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

> <comment-title>Insecure!</comment-title>
> But notice that your Reports server is not secured! Check out the [External Authentication]({% link topics/admin/tutorials/external-auth/tutorial.md %}) tutorial for information on securing Reports.
{: .comment}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=15 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="reports" %}


topic: admin
tutorial: beacon
title: Deploying a Beacon v1 in Galaxy
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/beacon/tutorial.html
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ga4gh", "beacon", "git-gat"]
content_hash: ae61857e2daafb329ed5d10b84505781
last_modified: 2026-02-26T21:12:14.134064
workflows_json: []
content:
This tutorial will guide you through setting up a [GA4GH Beacon](https://beacon-project.io/)!

> The Beacon Project is developed under a Global Alliance for Genomics and Health (GA4GH) Iniciative for the federated discovery of genomic data in biomedical research and clinical applications. 
{: .quote}

> <warning-title>Beacon v1</warning-title>
> This deploys an older Beacon v1 which was a simpler system.
> The Beacon v1 is more or less deprecated, with users being pushed to Beacon v2 which gives much richer answers, and offers better querying syntax.
{: .warning}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Installing and Configuring

> <hands-on-title>Setting up a Beacon with Ansible</hands-on-title>
>
> 1. Setup the hosts
>
>    {% raw %}
>    ```diff
>    --- a/hosts
>    +++ b/hosts
>    @@ -9,3 +9,12 @@ gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>
>     [sentryservers]
>     gat-0.eu.training.galaxyproject.eu ansible_connection=local ansible_user=ubuntu
>    +
>    +[beacon]
>    +[beacon:children]
>    +beacon_import
>    +beacon_server
>    +[beacon_server]
>    +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    +[beacon_import]
>    +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    {% endraw %}
>    ```
>    {: data-commit="Add hosts"}
>
>    > <tip-title>What are these :children?</tip-title>
>    > Here we use some of the more advanced features of Ansible's Inventory system.
>    > We declare a host group called 'beacon' with no hosts of its own.
>    > 
>    > Then we declare that this beacon group has two children: beacon_import, and beacon_server. We can then define host groups for those two entries with as many different hosts as we need. This makes it very easy to scale up configuration.
>    > 
>    > Here we will be using that feature to declare some 'beacon variables', which will be shared between the beacon_server and beacon_importer. Because they're children of 'beacon', they'll inherit any group variables defined for `group_vars/beacon.yml`.
>    {: .tip}
>
> 1. Setup the requirements
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -59,3 +59,8 @@
>     # Our FTP Server
>     - src: galaxyproject.proftpd
>       version: 0.3.1
>    +# Beacon support
>    +- name: paprikant.beacon
>    +  src: https://github.com/Paprikant/ansible-role-beacon
>    +- name: paprikant.beacon-importer
>    +  src: https://github.com/Paprikant/ansible-role-beacon_importer
>    {% endraw %}
>    ```
>    {: data-commit="Add requirements"}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Install the docker role as well.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy collection install community.docker
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <tip-title>Why doesn't this use the requirements.yml?</tip-title>
>    > Ansible over time added the feature of 'collections' in addition to roles. It's a nice and convenient method for grouping roles together. Unfortunately this means we need to update a significant portion of the GAT learning path to use this, which we have not done yet, as it will take a bit of time to do. Collections come with the unpleasant additional aspect that they will be installed into a different path than roles, so we'll need to add a `collections` folder as well, or maybe `deps/collections` and `deps/roles` folders. When this happens we'll have to additionally modify the ansible.cfg to look for collections in this additional path.
>    {: .tip}
>
> 3. Create the vars file
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/beacon.yml
>    @@ -0,0 +1,31 @@
>    +---
>    +postgres_data_dir: /data/beacon/postgresql/data
>    +postgres_init_dir: /data/beacon/postgresql/init
>    +bp_external_binding: 5050 # The default
>    +postgres_user: "{{ beacon_db_user }}"
>    +postgres_pass: "{{ beacon_db_password }}"
>    +postgres_external_binding: "{{ beacon_db_port }}"
>    +# Database Configuration
>    +beacon_db_user: beacon
>    +beacon_db_host: "{{ groups['beacon_server'][0] }}"
>    +beacon_db_password: "{{ vault_beacon_db_password }}"
>    +beacon_db_port: 9001
>    +#galaxy_api_key: This we will set in secrets.
>    +# Information about your beacon (consider filling this out.
>    +beacon_info_title: GA4GH Beacon
>    +beacon_info_beacon_id: your.galaxy.beacon
>    +beacon_info_description: Beacon service hosting datasets from all over the Galaxy
>    +beacon_info_url: https://{{ groups['beacon_server'][0] }}/beacon/
>    +beacon_info_service_group: galaxy-eu
>    +beacon_info_org_id: usegalaxy.aq
>    +beacon_info_org_name: Some Galaxy
>    +beacon_info_org_description: Galaxy community
>    +beacon_info_org_address: 123 Main Street, ZA
>    +beacon_info_org_welcome_url: https://galaxyproject.org/
>    +beacon_info_org_contact_url: https://galaxyproject.org/
>    +beacon_info_org_logo_url: https://galaxyproject.org/images/galaxy-logos/galaxy_project_logo_square.png
>    +beacon_info_org_info: More information about the organisation than just the description can go here.
>    +# Script Configuration
>    +galaxy_api_url: "https://{{ groups['galaxyservers'][0] }}"
>    +script_user: beacon
>    +script_dir: /home/beacon/script
>    {% endraw %}
>    ```
>    {: data-commit="Add relevant group variables"}
>
>    > <tip-title>groups?</tip-title>
>    > Here we again use some advanced features of Ansible's inventory system. Ansible knows the name of every hostname in the inventory. Now that we want to point the beacon configuration, either at the database which should be on `beacon_server`, or at Galaxy in `galaxyservers`, we ask the `groups` variable for what the inventory looks like. We use `[0]` to pull out the first hostname we find for both of those groups.
>    {: .tip}
>
> 1. Edit your `group_vars/secret.yml` and define some random passwords:
>
>    - The API key for your account, which must be an admin
>    - A password for the beacon database
>
>    {% snippet faqs/galaxy/preferences_admin_api_key.md admin=true %}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```diff
>    --- a/group_vars/secret.yml
>    +++ b/group_vars/secret.yml
>    @@ -1,13 +1,15 @@
>    +galaxy_api_key: your-galaxy-api-key
>    +vault_beacon_db_password: some-super-secret-password
>    ```
>
>    <!-- Ignore this, just for the gat-automation. Vaults are ugly to work with :(
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/secret.yml
>    +++ b/group_vars/secret.yml
>    @@ -1,23 +1,26 @@
>     $ANSIBLE_VAULT;1.1;AES256
>    -31383961303162656266343866376462343961326163356238636565386630363164353330643862
>    -3631663631383136356436363635373536323536353534650a393833343465396339373731663238
>    -38383763326230346561633061356634643566663863383838363765326162386133636330656136
>    -3935643333353365660a393932313965373166313838373739306234306634313139656334343839
>    -38393636396164373266646534326436316333313638366234323564323936643863633533366537
>    -66393535333263383837386538636530303137623761333161383765643938396232633863353838
>    -66363464613466353833663833613339666634333734343163326162393031363530643363396238
>    -36306562333763333435646432613565323537373032643335336639353762646662343865663266
>    -33646334623838333230343861353632626461363436343963636664393965613736306162396235
>    -64373239373739613164623165653832653139646433373531323562373534393563363962373634
>    -30393161303439376138653066646230303464353336656431663137366630323830316561356137
>    -65363663663165323666343335626238666334346664373938333839323161353131643336343738
>    -31646330386239363165643866316431386631643139613065663835323036353631646365333530
>    -37306161656562316665643666363132633766386330383864663937316233376261343264633133
>    -39356438623533376539393662326364666434386330626437646235386261323632396661656661
>    -31613232316237626562646330626564333566633266653762356338646236353137343863633634
>    -31623563393933656635323165326638376634383730393131373963353262366337383632666666
>    -64643264333463373334343433373564326632393637626237353336373837373438303231636633
>    -31666636316533313336613532643437646362636431306262623361326532613238303732396266
>    -62346131663135313935393439343433333439623335623131653066316561646631616361303632
>    -66633935356333643539323461333734376466316262656330396662346232333131313864343636
>    -6634613432653361653133383862383338353432666136303465
>    +35326661346439306337366130643630636230643766333533343364613135326265643638663832
>    +3235653731653438383234386233393939376235666163610a613436623232623036316538356338
>    +39313965393163376666316138353631633537636438336537656466366662393539346634653134
>    +6539316435666632300a383730303262333263363161316533613132653464316536653439323331
>    +61376430336336663031393064396462653130336534623664643362663432613965343237326335
>    +37666237623961663763336362636166666233343438336633313765363539646433383736373433
>    +39303735616131663239313337343637663930373638356564356531323066363131396536333030
>    +64326264633936346533643661336133336633363031316236633065313635313762363461366363
>    +62353061653130366562666633616466646436636361316265313131383234613430656461656466
>    +62393264393532323836333536363062653331393738383861656462656539316661336132663739
>    +31316632376436656639323530616432356131663763333961343764616137376133643061383761
>    +66646461613866653662306239386331396364653132656536623938653134653931363366623231
>    +39626362633136353636383539363966363937616463306466643033333038343162366239336332
>    +30393062613161623535623130616635333330356462623363303231653635613263306635663466
>    +39623433383031393935323362353830346466353961636461386566333230316431623532623361
>    +38636233346335623234396233623662363165616339633536343436626633663236653365653065
>    +37363633393463386233643437633662663135613133616561346361633663393163343636663163
>    +36353333393132656265656432616239356462343439633939303339333633383935346233326664
>    +30303963303365313436346334666630373634323639313635333332656532393965346334363338
>    +63633962383732643063316366636334613236343638353233313962326435616661623230643531
>    +31366230643263393063653634383465303636643333363732336438623063333664326362363561
>    +63343538356536626232623838373138336130663435373830376635366566666532303830633834
>    +66353834356566636333393137643836336130653138633561613530363037393438653363623131
>    +34643433643965643366373335316133323830386635653036613061316231326561346631623865
>    +66343936666137376339313638313864353335343538643763366661333933356566
>    {% endraw %}
>    ```
>    {: data-commit="Add beacon/galaxy passwords to the vault"}
>
>    -->
>
>
> 3. Add the beacon-server playbook
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/beacon-server.yml
>    @@ -0,0 +1,9 @@
>    +---
>    +- name: Beacon Server
>    +  hosts: beacon_server
>    +  become: true
>    +  become_user: root
>    +  vars_files:
>    +    - group_vars/secret.yml
>    +  roles:
>    +    - paprikant.beacon
>    {% endraw %}
>    ```
>    {: data-commit="Add beacon server playbook"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook beacon-server.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in }
>
>    TODO: Check that it works
>
{: .hands_on}

## Setting up the Importer

Now that our beacon is running, we need to get data from Galaxy to the Beacon

> <hands-on-title>Setting up the Beacon Importer</hands-on-title>
>
> 1. Add the beacon-import playbook
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/beacon-import.yml
>    @@ -0,0 +1,9 @@
>    +---
>    +- name: Beacon Importer
>    +  hosts: beacon_import
>    +  become: true
>    +  become_user: root
>    +  vars_files:
>    +    - group_vars/secret.yml
>    +  roles:
>    +    - paprikant.beacon-importer
>    {% endraw %}
>    ```
>    {: data-commit="Add beacon importer playbook"}
>
> 1. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook beacon-import.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in }
>
> 1. Add the nginx routes
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -117,4 +117,14 @@ server {
>
>     	{{ tiaas_nginx_routes }}
>
>    +	location /beacon {
>    +		proxy_pass http://{{ groups['beacon_server'][0] }}:5050;
>    +		proxy_http_version 1.1;
>    +		proxy_set_header Upgrade $http_upgrade;
>    +		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>    +		proxy_set_header X-Real-IP $remote_addr;
>    +		proxy_set_header Connection "upgrade";
>    +		proxy_set_header Host $host;
>    +	}
>    +
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Add beacon routes to nginx"}
>
> 1. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in }
{: .hands_on}

Congratulations, you've set up a Beacon v1 for Galaxy! Go check it out at `/beacon/` on your server.

{% snippet topics/admin/faqs/missed-something.md step=17 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="beacon" %}


topic: imaging
tutorial: biapy
title: Execute a BiaPy workflow in Galaxy
url: https://training.galaxyproject.org/training-material/topics/imaging/tutorials/biapy/tutorial.html
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
content_hash: 81991a1f2c88e4a5a25e84e4f5938fdd
last_modified: 2026-02-26T21:12:16.707030
zenodo_link: https://zenodo.org/records/10973241
workflows_json: []
content:
The application of supervised and unsupervised **Deep Learning (DL)** methods in bioimage analysis have been constantly increasing in biomedical research in the last decades ({% cite esteva2021deep %}). 
DL algorithms allow automatically classifying complex biological structures by learning complex patterns and features directly from large-scale imaging data, medical scans, or high-throughput biological datasets ({% cite franco2025biapy %}). Furthermore, trained models can be easily
shared on online repositories (e.g., [BioImage.IO](https://bioimage.io/#/models)) to be reused by other scientists and support open science. 

However, running DL models often requires high-level programming skills which can often be a barrier to general audience especially the 
one without a proper computational background. Additionally, many DL models require GPU acceleration, which is not always accessible to all researchers. 
Such obstacles might limit the practical and routine adoption of DL models in bioimaging. 

So, how to make DL models accessible to a larger audience? Well, [BiaPy](https://biapyx.github.io/) is an open source framework that streamlines the use of common deep-learning workflows for a large variety of bioimage analysis tasks, including 2D and 3D semantic segmentation, instance segmentation, object detection, image denoising, single image super-resolution, self-supervised learning (for model pretraining), image classification and image-to-image translation. 

In this training, you will learn how to execute a BiaPy workflow directly in Galaxy by running [inference](https://en.wikipedia.org/wiki/Deep_learning) on a set of images using two pre-trained models from BioImage.IO defined in a 
BiaPy YAML configuration file. In particular, we will execute the CartoCell pipeline a high-content pipeline for 3D image analysis, unveils cell morphology patterns in epithelia ({% cite andres2023cartocell %}).

![example-yaml.png](../../images/biapy/example-yaml.png "Example of a BiaPy YAML file where the model with ID in BioImage.IO 'merry-water-buffalo' is defined (red box). A BiaPy YAML configuration file includes information about the hardware to be used, such as the number of CPUs or GPUs, the specific image analysis task, the model to be used, optional hyperparameters, the optimizer, and the paths for loading and storing data.")

You will perform a comparative analysis of the segmentation performance of two models from BioImage.IO, namely
[venomous-swan](https://bioimage.io/#/artifacts/venomous-swan) and [merry-water-buffalo](https://bioimage.io/#/artifacts/merry-water-buffalo).
Both model can perform cyst segmentation for fluorescence microscopy images and have the same 3D [U-Net](https://en.wikipedia.org/wiki/U-Net) + Residual Blocks base architecture. 
However, venomous-swan enhances the 3D Residual U-Net with Squeeze-and-Excitation (SE) blocks.

Our goal is to check if whether the inclusion of SE blocks in the venomous-swan model leads to improved segmentation accuracy 
compared to the merry-water-buffalo model. This comparison will help us understand the impact of the SE blocks on the model's ability to segment 
cysts in fluorescence microscopy images.

Finally, we will assess the models using various metrics as well as a qualitative assessments of the segmentation masks will be conducted to 
visually inspect the differences in segmentation quality between the two models.

Let's start with BiaPy!

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Getting test data and the BiaPy YAML configuration file

The dataset required for this tutorial is available from [Zenodo]({{ page.zenodo_link }}). The CartoCell dataset contains whole epithelial cysts acquired at low resolution with minimal human intervention ([more information]({{ page.zenodo_link }})). The dataset is divided into *test*, *train* and *validation* data, each folder containing images and associated segmentation masks.

In order to simplify the upload, we already prepared the test images and YAML files in the **Data Library** that you can access on the left panel in Galaxy.

{% snippet faqs/galaxy/datasets_import_from_data_library.md %}

After importing the data from the Data Library, you should have the following files in your history: 
- `01_raw_image.tiff`
- `01_raw_mask.tiff`
- `02_raw_image.tiff`
- `02_raw_mask.tiff`
- `conf_cartocell_swam.yaml`
- `conf_cartocell_buffalo.yaml`

## Run inference using the BioImage.IO pre-trained model

Now we can set up the BiaPy tool with the ['venomous-swam' model](https://bioimage.io/#/artifacts/venomous-swan) which is defined in `conf_cartocell_swam.yaml`

> <hands-on-title>Configure the BiaPy Tool with 'venomous-swam'</hands-on-title>
>
> 1. {% tool [Build a workflow with BiaPy](toolshed.g2.bx.psu.edu/repos/iuc/biapy/biapy/3.6.5+galaxy0) %} with the following parameters to extract metadata from the image:
>
>- *Do you have a configuration file?* : `Yes, I have one and I want to run BiaPy directly`
>
>- *Select a configuration file*: `conf_cartocell_swam.yaml`
>
>- *Select the model checkpoint (if needed)* : Leave it blank. A checkpoint is a local file containing the trained model weights (e.g. .safetensors/.pth). In this tutorial we load a pre-trained model from the BioImage Model Zoo (BioImage.IO), so no local checkpoint file is required.
>
>- In the test data section, select the raw images to run predictions on and the ground truth/target images to evaluate those predictions. If no target data is provided, evaluation metrics will not be computed. **Make sure the files are in the same order so each raw image is paired with its corresponding target image**.
>
>     - *Specify the test raw images*: `01_raw_image.tiff` and `02_raw_image.tiff`
>
>     - *Specify the test ground truth/target images*: `01_raw_mask.tiff` and `02_raw_mask.tiff`
>
>- On *Select output* check the boxes:
>     - {% icon param-check %} `Test predictions (if exist)`
>     - {% icon param-check %} `Post-processed test predictions (if exist)` 
>     - {% icon param-check %} `Evaluation metrics (if exist, on test data)`
{: .hands_on}

Once the tool finishes running, you will have three different datasets in your history.

**1. Test predictions**: Full-size output images produced by the model on the test set. Because the model predicts small, overlapping patches, these patch outputs are merged back together to form one prediction per original image.

![test-prediction.png](../../images/biapy/test-prediction.png "Test predition on 22th z-stack of 02_raw_image.tiff"){: style="width:50%;" } 

**2. Post-Processed Test Prediction**: Test predictions after automatic “clean-up” steps defined in the configuration. These steps can refine the raw output (for example, removing small spurious regions or separating touching objects). 
In the YAML file definition, [Voronoi tessellation](https://en.wikipedia.org/wiki/Voronoi_diagram) is automatically applied to ensure that all instances touch each other.

![post-test-prediction.png](../../images/biapy/post-test-prediction.png "Post test predition on 22th z-stack of 02_raw_image.tiff"){: style="width:50%;" } 

**3. Test metrics:** Numerical scores that measure how well the predictions match the ground truth (if provided). 
In instance segmentation, the report typically includes:

- Intersection Over Union (IoU) per output channel (how well pixel regions overlap). This metric, also referred as the [Jaccard index](https://en.wikipedia.org/wiki/Jaccard_index), is essentially a method to quantify the percent of overlap between the target mask and the prediction output.

- Matching metrics (how well individual predicted objects match true objects), shown for raw predictions and post-processed predictions.

...but you can find more info on the test metrics in [BiaPy documentation](https://biapy.readthedocs.io/en/latest/)!

## Visualize the results

As first step, we can visualize one slice of the segmentation on the original image. We will work with `02_raw_image.tiff`

> <hands-on-title>Extract 2D results from the BiaPy output</hands-on-title>
>
> 1. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with the following parameters:
>- Input List: '"Build a workflow with BiaPy on dataset 2, 3, and others: Post-processed test predictions"'
>- How should a dataset be selected?:
>     - Select by Index
>         - Element index: `1`
>
> 2. Rename {% icon galaxy-pencil %} the dataset to `biapy_prediction_swam.tiff`
>
> 3. {% tool [Convert image format](toolshed.g2.bx.psu.edu/repos/imgteam/bfconvert/ip_convertimage/6.7.0+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Input Image"*: `biapy_prediction_swam.tiff`
>    - *"Extract series"*: `All series`
>    - *"Extract timepoint"*: `All timepoints`
>    - *"Extract channel"*: `All channels`
>    - *"Extract z-slice"*: `Extract z-slice`
>        - *"Z-slice id"* `22`
>    - *"Extract range"*: `All images`
>    - *"Extract crop"*: `Full image`
>    - *"Tile image"*: `No tiling`
>    - *"Pyramid image"*: `No Pyramid`
>
> 4. Rename {% icon galaxy-pencil %} the dataset to `stack_biapy_prediction_swam.tiff`
>
> 5. {% tool [Convert image format](toolshed.g2.bx.psu.edu/repos/imgteam/bfconvert/ip_convertimage/6.7.0+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Input Image"*: `01_raw_image.tiff`
>    - *"Extract series"*: `All series`
>    - *"Extract timepoint"*: `All timepoints`
>    - *"Extract channel"*: `All channels`
>    - *"Extract z-slice"*: `Extract z-slice`
>        - *"Z-slice id"* `22`
>    - *"Extract range"*: `All images`
>    - *"Extract crop"*: `Full image`
>    - *"Tile image"*: `No tiling`
>    - *"Pyramid image"*: `No Pyramid`
>
> 6. Rename {% icon galaxy-pencil %} the dataset to `stack_raw_image.tiff`
>
> 7. {% tool [Overlay images](toolshed.g2.bx.psu.edu/repos/imgteam/overlay_images/ip_overlay_images/0.0.4+galaxy4) %} with the following parameters to convert the image to PNG:
>    - *"Type of the overlay"*: `Segmentation contours over image`
>    - {% icon param-file %} *"Intensity image"*: `stack_raw_image.tiff` file
>    - {% icon param-file %} *"Label map"*: `stack_biapy_prediction_swam.tiff` file (output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %})
>    - *"Contour thickness"*: `1`
>    - *"Contour color"*: `green`
>    - *"Show labels"*: `no`
>8. Rename {% icon galaxy-pencil %} the dataset to `2D-overlay_biapy_swam.tiff`
{: .hands_on}

The segmentation results for the 22th z-stack are shown below:

![comparison_2D_swam.png](../../images/biapy/comparison_2D_swam.png "Segmentation results obtained with the 'venomous-swan' model. From left to right: the raw input image, predicted segmentation labels, and an overlay of the labels on the raw image."){: width="50%"}

We can also do better and visualize the full 3D segmentation using the [LibCarna](https://github.com/kostrykin/LibCarna) tool in Galaxy!

> <hands-on-title>Visual 3D</hands-on-title>
>
> 1. {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input image (3-D)"*: `01_raw_image.tiff` 
>    - *"Unit of the intensity values"*: `No unit`
>    - *"Coordinate system"*: `Point Y to the Top`
>    - *"Rendering mode"*: `Maximum Intensity Projection (MIP)`
>    - *"Color map"*: `gist_gray`
>    - *"Camera parameters"*:
>      - *"Distance"*: `100`
>    - *"Render mask overlay"*:
>      - {% icon param-file %} *"Mask overlay (3-D) "*: `biapy_prediction_swam.tiff`
>    - *"Video parameters"*:
>      - *"Frames"*: `400`
{: .hands_on}

<video loop="true" autoplay="autoplay" muted width="75%">
    <source src="../../images/biapy/carto_segm.mp4" type="video/mp4"/>
</video>

Pretty cool, eh? 

We can do the same for also for `02_raw_image.tiff`:

<video loop="true" autoplay="autoplay" muted width="75%">
    <source src="../../images/biapy/carto_segm_2.mp4" type="video/mp4"/>
</video>

## Compare different pre-trained models 

Let's now run the BiaPy tool again but this time with the ['merry-water-buffalo'](https://bioimage.io/#/artifacts/merry-water-buffalo) model:

> <hands-on-title>Configure the BiaPy Tool for 'merry-water-buffalo'</hands-on-title>
>
> 1. {% tool [Build a workflow with BiaPy](toolshed.g2.bx.psu.edu/repos/iuc/biapy/biapy/3.6.5+galaxy0) %} with the following parameters to extract metadata from the image:
>
>- *Do you have a configuration file?* : `Yes, I have one and I want to run BiaPy directly`
>
>- *Select a configuration file*: `conf_cartocell_buffalo.yaml`
>
>- *Select the model checkpoint (if needed)* : Leave it blank. We will load the pre-trained model directly from the BioImage.IO, so no checkpoint file is required.
>
>- In the test data section, select the raw images to run predictions on and the ground truth/target images to evaluate those predictions. If no target data is provided, evaluation metrics will not be computed. **Make sure the files are in the same order so each raw image is paired with its corresponding target image**.
>
>     - *Specify the test raw images*: `01_raw_image.tiff` and `02_raw_image.tiff`
>
>     - *Specify the test ground truth/target images*: `01_raw_mask.tiff` and `02_raw_mask.tiff`
>
>- On *Select output* check the boxes:
>     - {% icon param-check %} `Test predictions (if exist)`
>     - {% icon param-check %} `Post-processed test predictions (if exist)` 
>     - {% icon param-check %} `Evaluation metrics (if exist, on test data)`
{: .hands_on}

We can visualize again the results using the previous approach:

> <hands-on-title>Extract the results from the BiaPy output</hands-on-title>
>
> 1. {% tool [Extract Dataset](__EXTRACT_DATASET__) %} with the following parameters:
>- Input List: '"Build a workflow with BiaPy on dataset 2, 3, and others: Post-processed test predictions"'
>- How should a dataset be selected?:
>     - Select by Index
>         - Element index: `1`
>
> 2. Rename {% icon galaxy-pencil %} the dataset to `biapy_prediction_buffalo.tiff`
>
> 3. {% tool [Convert image format](toolshed.g2.bx.psu.edu/repos/imgteam/bfconvert/ip_convertimage/6.7.0+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Input Image"*: `biapy_prediction_buffalo.tiff`
>    - *"Extract series"*: `All series`
>    - *"Extract timepoint"*: `All timepoints`
>    - *"Extract channel"*: `All channels`
>    - *"Extract z-slice"*: `Extract z-slice`
>        - *"Z-slice id"* `22`
>    - *"Extract range"*: `All images`
>    - *"Extract crop"*: `Full image`
>    - *"Tile image"*: `No tiling`
>    - *"Pyramid image"*: `No Pyramid`
>
> 4. Rename {% icon galaxy-pencil %} the dataset to `stack_biapy_prediction_buffalo.tiff`
>
> 5. {% tool [Convert image format](toolshed.g2.bx.psu.edu/repos/imgteam/bfconvert/ip_convertimage/6.7.0+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Input Image"*: `01_raw_image.tiff`
>    - *"Extract series"*: `All series`
>    - *"Extract timepoint"*: `All timepoints`
>    - *"Extract channel"*: `All channels`
>    - *"Extract z-slice"*: `Extract z-slice`
>        - *"Z-slice id"* `22`
>    - *"Extract range"*: `All images`
>    - *"Extract crop"*: `Full image`
>    - *"Tile image"*: `No tiling`
>    - *"Pyramid image"*: `No Pyramid`
>
> 6. Rename {% icon galaxy-pencil %} the dataset to `stack_raw_image.tiff`
>
> 7. {% tool [Overlay images](toolshed.g2.bx.psu.edu/repos/imgteam/overlay_images/ip_overlay_images/0.0.4+galaxy4) %} with the following parameters to convert the image to PNG:
>    - *"Type of the overlay"*: `Segmentation contours over image`
>    - {% icon param-file %} *"Intensity image"*: `stack_raw_image.tiff` file
>    - {% icon param-file %} *"Label map"*: `stack_biapy_prediction_buffalo.tiff` file (output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %})
>    - *"Contour thickness"*: `1`
>    - *"Contour color"*: `green`
>    - *"Show labels"*: `no`
>8. Rename {% icon galaxy-pencil %} the dataset to `2D-overlay_biapy_buffalo.tiff`
{: .hands_on}

Results will look like this:

![comparison_2D_buffalo.png](../../images/biapy/comparison_2D_buffalo.png "Segmentation results with the 'merry-water-buffalo' model from BioImage.IO"){: width="50%"}

From a visual inspection of the results, the **'venomous-swam'** model appears to produce sharper contours and more clearly defined cells, whereas **'merry-water-buffalo'** seems better at capturing cells with fewer merges. However, its segmentation is slightly noisier.

It is hard to say that a prediction is better than other by just looking at a slice when working in 3D! 

The **Test metrics** will give us a better overview on which model is performing better!

**'venomous-swam'** struggles mainly with missing objects (low recall)

Let's evaluate segmentation at a IoU ≥ 0.5, a moderate and not-too-strict matching threshold:

    Precision: 0.328
    Recall: 0.171
    F1: 0.222
    PQ: 0.157

**water-buffalo** is clearly stronger at instance detection and segmentation

At a commonly used matching threshold (IoU ≥ 0.5), averaged across the 2 test images:

    Precision: 0.761
    Recall: 0.631
    F1: 0.689
    PQ: 0.473

So **Water-buffalo** is better both in:

- finding objects (higher recall),
- keeping predictions correct (higher precision).

## Conclusions

In this tutorial, you executed BiaPy inference pipelines directly in Galaxy using YAML configuration files, and compared two pre-trained BioImage.IO models on the same 3D dataset.

You learned how to:

- Run BiaPy in **“configuration-file driven”** mode, which makes analyses easier to reproduce and share
- Load **pre-trained BioImage.IO models** without providing a local checkpoint
- Inspect results both as a **2D slice overlay** and as a **3D rendering**
- Use the **evaluation metrics** to compare models objectively rather than relying on visual inspection alone

In our example, the two models showed different strengths: one produced cleaner contours in a slice view, while the other achieved stronger quantitative performance (higher object-level matching metrics at common IoU thresholds). This illustrates why combining **qualitative visualization** with **quantitative scoring** is important when selecting a model.

### Where to go next

- Edit the YAML to test different post-processing settings (e.g., instance separation parameters)
- Run inference on additional images or your own data (keeping image/mask pairing consistent)
- Try other BioImage.IO models and compare them using the same workflow and plots

# Optional: Extract complete training workflow from history
As an optional step, you can extract a complete workflow from your Galaxy history. This allows you to save and reuse the entire training process as a reproducible and shareable workflow.

{% snippet faqs/galaxy/workflows_extract_from_history.md %}

tags_json:
["Image segmentation", "Image annotation", "Deep learning", "Conversion", "Overlay", "3D image", "Volume rendering"]


topic: imaging
tutorial: process-image-bioimageio
title: Using BioImage.IO models for image analysis in Galaxy
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["Image segmentation", "Image annotation", "Deep learning"]
content_hash: 72d84d46c14924f92b1137819eced382
last_modified: 2026-02-26T21:12:16.815090
zenodo_link: https://zenodo.org/record/6647674
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/process-image-bioimageio/tutorial.html

content:
Deep learning models are increasingly used in bioimage analysis to perform processing steps such as segmentation, classification, and restoration tasks (e.g., {% cite Moen2019 %}). The [BioImage Model Zoo, (BioImage.IO)](https://bioimage.io/#/)({% cite Wei2021 %}) is a repository that provides access to pre-trained AI models, sharing a common metadata model that allows their reuse in different tools and platforms.

Each model in BioImage.IO is tailored for a specific biological task — for example, segmenting nuclei, detecting mitochondria, or identifying neuronal structures — and trained on specific imaging modalities such as electron or fluorescence microscopy (e.g., {% cite vonChamier2021 %}, {% cite GomezdeMariscal2021 %}).

This tutorial will guide you through the process of applying one of these BioImage.IO models to an input image using Galaxy ({% cite Batut2024 %}). You will learn how to upload and configure the model, set the correct input parameters, and interpret the output files.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Available BioImage.IO models in Galaxy

As of the version {% tool [Process image using a BioImage.IO model](toolshed.g2.bx.psu.edu/repos/bgruening/bioimage_inference/bioimage_inference/2.4.1+galaxy3) %}, only the PyTorch-based BioImage.IO models listed in the table below are compatible with the Galaxy tool:

| Model name | Task | Imaging modality | Sample / species | Link |
|------------|------|------------------|------------------|------|
| 🪴 PlatynereisEMnucleiSegmentationBoundaryModel | Nuclei segmentation | Electron microscopy | Platynereis | [View model](https://bioimage.io/#/?id=10.5281%2Fzenodo.6028097) |
| 🪴 PlatynereisEMcellsSegmentationBoundaryModel | Cell segmentation | Electron microscopy | Platynereis | [View model](https://bioimage.io/#/?id=10.5281%2Fzenodo.6028280) |
| 🦠 LiveCellSegmentationBoundaryModel | Live cell segmentation | Phase-contrast Microscopy | Various cell types | [View model](https://bioimage.io/#/?id=10.5281%2Fzenodo.5869899) |
| 🔬 HyLFM-Net-stat | Light field reconstruction | Light field and Fluorescence light microscopy | Zebrafish | [View model](https://bioimage.io/#/?tags=HyLFM-Net-stat&id=ambitious-sloth) |
| 🪴 3DUNetArabidopsisApicalStemCells | Stem cell segmentation | Confocal / light sheet | Arabidopsis root | [View model](https://bioimage.io/#/?id=emotional-cricket) |
| 🧬 CovidIFCellSegmentationBoundaryModel | Cell segmentation | Fluorescence light microscopy | Infected human cells | [View model](https://bioimage.io/#/?tags=Covid&id=10.5281%2Fzenodo.5847355) |
| 🧬 NucleiSegmentationBoundaryModel| Nucleus segmentation | Fluorescence light microscopy| Generic / various | [View model](https://bioimage.io/#/?id=10.5281%2Fzenodo.5764892) |
| 🧬 HPANucleusSegmentation | Nucleus segmentation | Immunofluorescence | Human Protein Atlas | [View model](https://bioimage.io/#/?tags=HPA&id=10.5281%2Fzenodo.6200999) |
| 🧠 NeuronSegmentationInEM (Membrane prediction) | Neuron segmentation | Electron microscopy | Brain tissue | [View model](https://bioimage.io/#/?id=10.5281%2Fzenodo.5874741) |
| 🧫 HPACellSegmentationModel | Cell segmentation | Immunofluorescence | Human Protein Atlas | [View model](https://bioimage.io/#/?tags=hpa&id=10.5281%2Fzenodo.6200635) |
| 🧪 MitochondriaEMSegmentationBoundaryModel | Mitochondria segmentation | Electron microscopy | Human | [View model](https://bioimage.io/#/?id=10.5281%2Fzenodo.5874841) |

# Example: Segmentation

Here we illustrate the type of information that is both useful for understanding the model's biological context and necessary for using the Galaxy tool — specifically, the input axes and input size parameters.

As an example, we consider the following model: **🧬 NucleiSegmentationBoundaryModel**

This model segments nuclei in fluorescence microscopy images. It predicts *boundary maps* and *foreground probabilities* for nucleus segmentation, primarily in images stained with DAPI. The outputs are designed to be post-processed with methods such as Multicut or Watershed to achieve instance-level segmentation (object-based segmentation).

- **Imaging modality**: Fluorescence microscopy
- **Task**: Nucleus segmentation (boundary-aware)
- **Model link**: [View on BioImage.IO](https://bioimage.io/#/r/ilastik/stardist_dsb_training_data)
- **Citation**: [10.5281/zenodo.5764893](https://doi.org/10.5281/zenodo.5764893)


> <tip-title> Where to find this information on BioImage.IO </tip-title>
>
> You can find similar details for other models directly on [BioImage.IO](https://bioimage.io) by viewing each model's card. Look under the “inputs” section of the RDF file to find the required `axes` and `input size` values. These parameters are essential for running the model correctly in Galaxy.
>
{: .tip}


## Get the data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial.
>
> 2. Download the following image and import it into your Galaxy history.
>    For the purpose of this tutorial, we will use one image to test only one of the 11 available models:
>
>    - [`test_image_nuclei.png`](../../images/process-image-bioimageio/input_nucleisegboundarymodel.png)
>
>    If you are importing the image via URL:
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    If you are importing the image from the shared data library:
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets appropriately if needed (e.g. `"BioImage.IO model"`, `"Test image"`)
>
> 4. Confirm the datatypes are correct (`pt` for the model, `tiff` or `png` for the image)
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Import the BioImage.IO model from the Galaxy file repository:
>
>    - Click on **Upload Data**
>    - Go to the **Choose from repository** tab
>    - Navigate to: `ML models` → `bioimaging-models`
>    - Select the desired model file (for this tutorial, choose `nucleisegmentationboundarymodel.pt`)
>    - Click **Import** to add it to your history
>
>    If you are importing the model from the shared data library:
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}


## Run the model on your image

> <hands-on-title> Run BioImage.IO model </hands-on-title>
>
> 1. {% tool [Process image using a BioImage.IO model](toolshed.g2.bx.psu.edu/repos/bgruening/bioimage_inference/bioimage_inference/2.4.1+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"BioImage.IO model"*: `nucleisegmentationboundarymodel.pt`
>    - {% icon param-file %} *"Input image"*: `test_image_nuclei.png`
>    - {% icon param-text %} *"Size of the input image"*: `256,256,1,1`
>    - {% icon param-select %} *"Axes of the input image"*: `Four axes (e.g., bcyx, byxc)`
>
>    > <comment-title>Axes and size</comment-title>
>    >
>    > The {% icon param-text %} *"Size of the input image"* and the {% icon param-select %} *"Axes of the input image"* are crucial to transform the input image into the format that the BioImage.IO model requires. The correct values are provided in the RDF file that comes with the chosen model on [BioImage.IO](https://bioimage.io).
>    {: .comment}
>
{: .hands_on}

The model will process the input image and generate two outputs:
- Two predicted images (written in one TIFF file)
- A predicted tensor matrix (`.npy`)

[Figure 1](#figure-1) below is a visualization of the two predicted images generated by the **🧬 NucleiSegmentationBoundaryModel**. Predicted Image 1 are the *foreground probabilities* and Predicted Image 2 are the *boundary map*.

![Example tiff output of the nuclei segmentation model](../../images/process-image-bioimageio/output-nucleus-seg-model.png "The output generated by the nuclei segmentation model for the example data. The intensity values in all three images are ranging from 0 (black) to 1 (white) with gray values in between.")

> <tip-title> Visualising the output images</tip-title>
>
> Galaxy provides a basic preview using its `.tiff` visualization tool. However, BioImage.IO models sometimes produce tiff files with several predicted images residing in the same tiff file.
>
> To properly explore the results, it is recommended to click on the **visualize** icon in the output file, this will give you the option to display the dataset using the **Avivator tool**.
> ![Inspect output with Avivator](../../images/process-image-bioimageio/display-avivator.png "Visualize your Tiff output with Avivator in Galaxy")
>
> An alternative is to download the file and open it locally using image analysis tools such as **Fiji/ImageJ**, **napari**, or **QuPath**.
>
{: .tip}

> <question-title> Check your understanding </question-title>
>
> 1. Why do the image axes matter when using a model?
> 2. What happens if the image size does not match the model input?
> 3. What are TIFF and NPY formats?
> 4. How can you interpret the output of the model, and what does it tell you about your input image?
>
> > <solution-title></solution-title>
> >
> > 1. Because deep learning models are trained on specific image shapes and dimensions; mismatches will cause errors or wrong results.
> > 2. The model will fail to run or produce invalid output.
> > 3. **TIFF (.tiff)** is a standard format for storing image data, commonly used in microscopy and bioimaging. It can be easily viewed and interpreted visually.
> >    **NPY (.npy)** is a binary format used by NumPy to store arrays. In this case, it contains the raw prediction tensor produced by the model, which can be useful for further analysis or visualization with Python tools.
> > 4. The model generates a **predicted image** that highlights or segments specific structures (e.g. nuclei, cells, mitochondria) based on what it learned during training. By comparing the output image to the input, users can see which regions were detected or classified, helping to extract biological meaning from the raw data.
> >
>{: .solution}
{: .question}


## Post-processing of the model output

There are two challenges when it comes to using the model output for subsequent analysis. First, the model produces a single output file with two images (boundary maps and foreground probabilities), so for subsequent analysis we need to extract the corresponding information from that file. Second, neither of the two images produced by the model directly corresponds to segmentation results. Albeit the extracted image (Predicted Image 1) looks like a binary image with intensity 0 for the image background and intensity 1 for the image foreground, it is not. For example, there are fine contours of intensity values subtly below 1 between closely clustered cell nuclei. Thus, to obtain segmentation results, we first need to threshold the foreground probabilities (values ranging between 0 and 1) to determine the image foreground (as a binary image without any values between 0 and 1).

However, directly thresholding the foreground probabilities is going to lose information when it comes to closely clustered cell nuclei, where the crucial information is stored in the boundary map. To cope with that, we will extract both images (the *foreground probabilities* and the *boundary map*) from the output file, combine their information into a single image, and then perform thresholding.

> <hands-on-title>Extract the segmentation results from the model output</hands-on-title>
>
> 1. {% tool [Split image along axes](toolshed.g2.bx.psu.edu/repos/imgteam/split_image/ip_split_image/2.2.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input Image"*: the output from "Run BioImage.IO model"
>    - {% icon param-select %} *"Axis to split along"*: `Q-axis (other or unknown axis)`
>    - {% icon param-check %} *"Squeeze result images"*
>
>    This produces a dataset collection with two items (the two images). Next, we need to extract the first dataset (Predicted Image 1) from this collection.
>
> 2. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: the output from the previous step
>    - {% icon param-select %} *"How should a dataset be selected"*: `Select by index`
>    - {% icon param-select %} *"Element index"*: `0`
>
>    This will yield the file `1.tiff` in your history (Predicted Image 1).
>
> 2. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: the output from the previous step
>    - {% icon param-select %} *"How should a dataset be selected"*: `Select by index`
>    - {% icon param-select %} *"Element index"*: `1`
>
>    This will yield the file `2.tiff` in your history (Predicted Image 2).
>
>    Next, we will combine the information from the two images into a single image.
>
> 3. {% tool [Process images using arithmetic expressions](toolshed.g2.bx.psu.edu/repos/imgteam/image_math/image_math/1.26.4+galaxy2) %}
>    - {% icon param-text %} *"Expression"*: `foreground - boundaries`
>    - {% icon param-repeat %} *"Input images"*:
>      - {% icon param-file %} *"Image"*: `1.tiff`
>      - {% icon param-text %} *"Variable for representation of the image within the expression"*: `foreground`
>    - {% icon param-repeat %} *"Input images"*:
>      - {% icon param-file %} *"Image"*: `2.tiff`
>      - {% icon param-text %} *"Variable for representation of the image within the expression"*: `boundaries`
>
>    > <question-title></question-title>
>    >
>    > What is the motivation for combining the information from the two images with this arithmetic expression?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > Each pixel of the `foreground` image (Predicted Image 1) corresponds to the probability of that pixel being part of a foreground object. We have also seen that the `boundaries` image (Predicted Image 2) uses white (intensity value 1) to encode pixels which likely correspond to boundaries of cell nuclei, and lower intensity values for others. Thus, we can interpret the `boundaries` image as *boundary probabilities*, in the sense that each pixel of that image corresponds to the probability of that pixel being part of an object boundary (i.e. the boundary of a nucleus). By considering the expression `foreground - boundaries`, we essentially consider the probability of each pixel being part of the image foreground, *plus* the probability of that point being *not* part of an object boundary.
>    > >
>    > > Thus, in the resulting image, the intensity of each pixel pixel can be interpreted as the probability of that pixel being part of the *interior* of a foreground object. This better preserves information about the individual cell nuclei in the image, which is especially crucial for closely clustered cell nuclei, as opposed to considering the *foregorund probabilities* solely.
>    > >
>    > > This interpretation also gives rise to the choice of `0.6` as the threshold value for in the next step (see below). Naturally, we would choose `0.5` to determine the image regions for which the probability is higher than 50% that the image pixels correspond to the interior of foreground objects (cell nuclei), but to improve the separation of closely clustered cell nuclei, it is a good practice to choose a threshold that is somewhat higher.
>    > >
>    > {: .solution }
>    {: .question}
>
> 4. {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.18.1+galaxy3) %}
>    - {% icon param-file %} *"Input image"*: the output of the {% tool [Process images using arithmetic expressions](toolshed.g2.bx.psu.edu/repos/imgteam/image_math/image_math/1.26.4+galaxy2) %} tool
>    - {% icon param-select %} *"Thresholding method"*: `Manual`
>    - {% icon param-text %} *"Threshold value"*: `0.6`
>
{: .hands_on}

Finally, you could follow the ["Hands On: Segment image" from the "Introduction to Image Analysis using Galaxy" tutorial]({% link topics/imaging/tutorials/imaging-introduction/tutorial.md#hands-on-segment-image %}) to create a segmentation overlay (e.g., see [Figure 3](#figure-3) below) or to perform cell counting.

![Segmentation overlay](../../images/process-image-bioimageio/segmentation.png "Overlay of the original input image and the contours of the segmentation results obtained using the BioImage.IO model and post-processing.")


# Conclusion

In this tutorial, you learned how to run a BioImage.IO model on a biological image using Galaxy. By uploading a compatible model and image, setting the appropriate size and axes, and running the tool, you obtained both a predicted image and a tensor matrix representing the model output.

This provides a fast, reproducible way to apply deep learning models in the context of bioimage analysis — all within Galaxy.


topic: imaging
tutorial: hela-screen-analysis
title: Analyse HeLa fluorescence siRNA screen
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
content_hash: ea08a73bb8a6c5b3667b10e07d64db9d
last_modified: 2026-02-26T21:12:16.738974
zenodo_link: https://zenodo.org/record/3362976
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/hela-screen-analysis/tutorial.html

content:
This tutorial shows how to segment and extract features from cell nuclei Galaxy for image analysis. As example use case, this tutorial shows you how to compare the phenotypes of PLK1 threated cells in comparison to a control. The data used in this tutorial is available at [Zenodo]({{ page.zenodo_link }}).

RNA interference (RNAi) is used in the example use case for silencing genes by way of mRNA degradation. Gene knockdown by this method is achieved by introducing small double-stranded interfering RNAs (siRNA) into the cytoplasm. Small interfering RNAs can originate from inside the cell or can be exogenously introduced into the cell. Once introduced into the cell, exogenous siRNAs are processed by the RNA-induced silencing complex (RISC).The siRNA is complementary to the target mRNA to be silenced, and the RISC uses the siRNA as a template for locating the target mRNA. After the RISC localizes to the target mRNA, the RNA is cleaved by a ribonuclease. RNAi is widely used as a laboratory technique for genetic functional analysis. RNAi in organisms such as C. elegans and Drosophila melanogaster provides a quick and inexpensive means of investigating gene function. Insights gained from experimental RNAi use may be useful in identifying potential therapeutic targets, drug development, or other applications. RNA interference is a very useful research tool, allowing investigators to carry out large genetic screens in an effort to identify targets for further research related to a particular pathway, drug, or phenotype.

The example used in this tutorial deals with PLK1 knocked down cells. PLK1 is an early trigger for G2/M transition. PLK1 supports the functional maturation of the centrosome in late G2/early prophase and establishment of the bipolar spindle. PLK1 is being studied as a target for cancer drugs. Many colon and lung cancers are caused by K-RAS mutations. These cancers are dependent on PLK1.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Getting data

The dataset required for this tutorial contains a screen of DAPI stained HeLa nuclei ([more information]({{ page.zenodo_link }})). We will use a sample image from this dataset for training basic image processing skills in Galaxy.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. If you are logged in, create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import {% icon galaxy-upload %} the following dataset from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor).
>    - **Important:** Choose the type of data as `zip`.
>
>    ```
>    {{ page.zenodo_link }}/files/B2.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"input_file"*: `B2.zip`
>    - *"Extract single file"*: `Single file`
>    - *"Filepath"*: `B2--W00026--P00001--Z00000--T00000--dapi.tif`
>
> 4. Rename {% icon galaxy-pencil %} the dataset to `testinput.tiff`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"input_file"*: `B2.zip`
>    - *"Extract single file"*: `All files`
>
> 6. Rename {% icon galaxy-pencil %} the resulting collection to `control`
>
>    {% snippet faqs/galaxy/collections_rename.md %}
>
> 7. Import {% icon galaxy-upload %} the following dataset from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor).
>    - **Important:** Choose the type of data as `zip`.
>    ```
>    {{ page.zenodo_link }}/files/B3.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 8. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} to extract the zipped screen:
>    - {% icon param-file %} *"input_file"*: `B3.zip`
>    - *"Extract single file"*: `All files`
>
> 9. Rename {% icon galaxy-pencil %} the collection to `PLK1`
> 9. Upload {% icon galaxy-upload %} the following segmentation filter rules as a new pasted file (format: `tabular`):
>    ```
>    	area	eccentricity
>    min	500	0.
>    max	100000	0.5
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" %}
>
> 9. Rename {% icon galaxy-pencil %} dataset to `rules`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
{: .hands_on}


# Create feature extraction workflow
First, we will create and test a workflow which extracts mean DAPI intensity, area, and major axis length of cell nuclei from an image.

> <hands-on-title>Create feature extraction workflow</hands-on-title>
>
> 1. {% tool [Filter 2-D image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_simple_filter/ip_filter_standard/1.12.0+galaxy1) %} with the following parameters to smooth the image:
>    - {% icon param-file %} *"Input image"*: `testinput.tiff` file
>    - *"Filter type"*: `Gaussian`
>    - *"Sigma"*: `3`
> 2. {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.18.1+galaxy3) %} with the following parameters to segment the image:
>    - {% icon param-file %} *"Input image"*: output of {% tool [Filter 2-D image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_simple_filter/ip_filter_standard/1.12.0+galaxy1) %}
>    - *"Thresholding method"*: `Globally adaptive / Otsu`
> 3. {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %} with the following parameters to split touching objects:
>    - {% icon param-file %} *"Binary image"*: output of {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.18.1+galaxy3) %}
>    - *"Mode":* `Watershed transform`
>    - *"Minimum distance between two objects"*: `20`
> 4. {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.18.1+galaxy0) %} with the following parameters to extract features from the segmented objects:
>    - {% icon param-file %} *"Label map"*: output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %}
>    - *"Use the intensity image to compute additional features"*: `No intensity image`
>    - *"Select features to compute"*: `Select features`
>    - *"Available features"*:
>        - {% icon param-check %} `Label from the label map`
>        - {% icon param-check %} `Area`
>        - {% icon param-check %} `Eccentricity`
>        - {% icon param-check %} `Major axis length`
> 5. {% tool [Filter label map by rules](toolshed.g2.bx.psu.edu/repos/imgteam/2d_filter_segmentation_by_features/ip_2d_filter_segmentation_by_features/0.0.1-4) %} with the following parameters to filter the label map from 3. with the extracted features and a set of rules:
>    - {% icon param-file %} *"Label map"*: output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %}
>    - {% icon param-file %} *"Features"*: output of {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.18.1+galaxy0) %}
>    - {% icon param-file %} *"Rules"*: `rules` file
> 6. {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.18.1+galaxy0) %} with the following parameters to extract features the final readout from the segmented objects:
>    - {% icon param-file %} *"Label map"*: output of {% tool [Filter label map by rules](toolshed.g2.bx.psu.edu/repos/imgteam/2d_filter_segmentation_by_features/ip_2d_filter_segmentation_by_features/0.0.1-4) %}
>    - *"Use the intensity image to compute additional features"*: `Use intensity image`
>    - {% icon param-file %} *"Intensity image"*: `testinput.tiff` file
>    - *"Select features to compute"*: `Select features`
>    - *"Available features"*:
>      - {% icon param-check %} `Mean Intensity (requires original image)`
>      - {% icon param-check %} `Area`
>      - {% icon param-check %} `Major axis length`
> 7. Now we can extract the workflow for batch processing:
>    - Name it "feature_extraction".
>    - Don't treat `B2.zip` and `B3.zip` as inputs (the workflow is supposed to be applied to the images directly).
>    - Exclude {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} by unchecking the tool (3 times).
>
>    {% snippet faqs/galaxy/workflows_extract_from_history.md %}
>
> 8. Edit the workflow you just created:
>    - Select "Input dataset" from the list of tools. The step {% icon param-file %} **8: Input Dataset** appears.
>    - Change the "Label" of {% icon param-file %} **8: Input Dataset** to `input image`.
>    - Change the "Label" of {% icon param-file %} **1: rules** to `filter rules`.
>    - Connect the output of {% icon param-file %} **8: input image** to the input of {% icon tool %} **2: Filter 2-D image**.
>    - Connect the output of {% icon param-file %} **8: input image** to the "Intensity image" input of {% icon tool %} **7: Extract image features**.
>    - Mark the results of {% icon tool %} **6: Filter label map by rules** and {% icon tool %} **7: Extract image features** as the primary outputs of the workflow (by clicking on the checkboxes of the outputs).
>
{: .hands_on}

The resulting workflow should look something like this:

![feature extraction workflow](../../images/hela-screen-analysis/feature_extraction_workflow.png "Feature extraction workflow.")

# Apply workflow to screen

Now we want to apply our extracted workflow to a series of images and merge the results. For this purpose, we create a workflow which uses the previously created workflow as a sub-workflow.

> <hands-on-title>Create screen analysis workflow</hands-on-title>
>
> 1. Create a new workflow in the workflow editor.
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
> 2. Select "Input dataset collection" from the list of tools. The step {% icon param-collection %} **1: Input Dataset Collection** appears in your workflow. Change the "Label" of this step to `input images`.
> 3. Add the input dataset {% icon param-file %} **2: rules** to your workflow (select "Input dataset" from the list of tools and set the "Label" of the newly created step to `rules`).
> 4. Add the {% icon workflow %} **feature_extraction** workflow as a sub-workflow:
>    - Expand the "Workflows" section in the list of tools and select "feature_extraction" to add it to the workflow.
>    - Connect the output of {% icon param-file %} **1: input images** to the "input image" input of {% icon workflow %} **3: feature_extraction**.
>    - Connect the output of {% icon param-file %} **2: rules** to the "filter rules" input of {% icon workflow %} **3: feature_extraction**.
> 5. Create the step {% icon tool %} **4: Collapse Collection** in the workflow (by choosing "Collapse Collection" from the list of tools).
>    - Connect the output "output (tabular)" of {% icon workflow %} **3: feature_extraction** to {% icon tool %} **4: Collapse Collection**.
>    - Set *"Keep one header line"* of {% icon tool %} **4: Collapse Collection**: `Yes`
>    - Set *"Prepend File name"* of {% icon tool %} **4: Collapse Collection**: `No`
>    - Mark the output of {% icon tool %} **4: Collapse Collection** as the primary workflow output.
> 6. Save your workflow and name it `analyze_screen`.
{: .hands_on}

The resulting workflow should look something like this:

![screen analysis workflow](../../images/hela-screen-analysis/analyze_screen_workflow.png "Full screen analysis workflow.")


> <hands-on-title>Run screen analysis workflow</hands-on-title>
>
> 1. Run the "analyze_screen" workflow on the `control` screen using the `rules` file.
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> 2. Run the "analyze_screen" workflow on the `PLK1` screen using the `rules` file.
>
{: .hands_on}

# Plot feature extraction results

Finally, we want to plot the results for better interpretation.

> <hands-on-title>Plot feature extraction results</hands-on-title>
>
> 1. Click on the **Visualize** {% icon galaxy-barchart %} icon of the {% icon tool %} **4: Collapse Collection** results.
> 2. Run **Box plot (jqPlot)** with the following parameters:
>    - *"Provide a title"*: `Screen features`
>    - *"X-Axis label"*:
>    - *"Y-Axis label"*:
>    - *"1: Data series"*:
>        - *"Provide a label"*: `Mean intensity`
>        - *"Observations"*: `Column 1`
>    - *"2: Data series"*:
>        - *"Provide a label"*: `Area`
>        - *"Observations"*: `Column 2`
>    - *"3: Data series"*:
>        - *"Provide a label"*: `Major axis length`
>        - *"Observations"*: `Column 3`
>
>    > <question-title></question-title>
>    >
>    > Plot the feature distribution of PLK1 and control. What differences do you observe between the screens?
>    >
>    > > <solution-title></solution-title>
>    > > The phenotype of PLK1 threated cells show a higher mean intensity and a shorter major axis in comparison to the control.
>    > {: .solution }
>    {: .question}
{: .hands_on}


One of the resulting plots should look something like this:

![feature extraction results box plot](../../images/hela-screen-analysis/result_boxplot.png){: width="100%"}

# Conclusion


In this exercise you imported images into Galaxy, segmented cell nuclei, filtered segmentations by morphological features, extracted features from segmentations, scaled your workflow to a whole screen, and plotted the feature extraction results using Galaxy.

tags_json:
["Image segmentation", "Object feature extraction", "Image thresholding", "High-throughput screening", "Overlay", "Fluorescence microscopy"]


topic: imaging
tutorial: imaging-introduction
title: Introduction to Image Analysis using Galaxy
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
content_hash: e74d09cd932c40d196b7a16bfd70885b
last_modified: 2026-02-26T21:12:16.760216
zenodo_link: https://zenodo.org/record/3362976
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/imaging-introduction/tutorial.html

content:
Image analysis is the extraction of meaningful information from images by means of digital image processing techniques. Imaging is an important component in a wide range of scientific fields of study, such as astronomy, medicine, physics, biology, geography, chemistry, robotics, and industrial manufacturing.

This tutorial shows how to use Galaxy to perform basic image analysis tasks such as format conversion, image enhancement, segmentation, and feature extraction.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Getting Data

The dataset required for this tutorial is available from [Zenodo]({{ page.zenodo_link }}) and
contains a screen of [DAPI](https://en.wikipedia.org/wiki/DAPI) stained [HeLa](https://en.wikipedia.org/wiki/HeLa) nuclei ([more information]({{ page.zenodo_link }})). We will use a sample image from this dataset for training basic image processing skills in Galaxy.

Our objective is to automatically count the number of cells contained in this image. In order to achieve this, we will enhance the quality of the image, automatically detect the nuclei and segment the nuclei and count them.


> <hands-on-title>Data upload</hands-on-title>
>
> 1. If you are logged in, create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following dataset from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor).
>    - **Important:** Choose the type of data as `zip`.
>
>    ```
>    {{ page.zenodo_link }}/files/B2.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"input_file"*: `B2.zip`
>    - *"Extract single file"*: `Single file`
>    - *"Filepath"*: `B2--W00026--P00001--Z00000--T00000--dapi.tif`
>
> 4. Rename {% icon galaxy-pencil %} the dataset to `input.tiff`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
{: .hands_on}


# Image Metadata Extraction

Now, we can extract metadata from an image.

> <hands-on-title>Extract Image Metadata</hands-on-title>
>
> 1. {% tool [Show image info](toolshed.g2.bx.psu.edu/repos/imgteam/image_info/ip_imageinfo/5.7.1+galaxy1) %} with the following parameters to extract metadata from the image:
>    - {% icon param-file %} *"Input Image"*: `input.tiff` file (output of the previous step)
> 2. Click on the {% icon galaxy-eye %} (eye) icon next to the file name, to look at the file content and search for image acquisition information
>
>    > <question-title></question-title>
>    >
>    > 1. What is the datatype?
>    > 2. What are the pixel dimentions?
>    > 3. How many bits per pixel are used?
>    >
>    > > <solution-title></solution-title>
>    > > 1. TIFF
>    > > 2. 1344x1024
>    > > 3. 16
>    > {: .solution }
>    {: .question}
{: .hands_on}

# Visual Inspection of TIFF Images

Not all tools can handle all image formats. Especially proprietary microscope image formats should be converted to TIFF ([supported formats](https://docs.openmicroscopy.org/bio-formats/5.7.1/supported-formats.html)). However, TIFF cannot be displayed in most web browsers directly. Therefore, for visual inspection of TIFF images, we use a TIFF visualization plugin in Galaxy.

> <hands-on-title>Visual Inspection of TIFF Images</hands-on-title>
>
> 1. Click on the title of your file to see the row of small icons for saving, linking, etc.:
> ![Screenshot of Galaxy icons. Seven small blue icons are shown on a green background. From left to right they are: floppy disk, link, information, redo, bar chart, flow chart and a question mark.](../../images/imaging-introduction/LittleJobIcons.png)
> 2. Click on the **visualise icon** {% icon galaxy-visualise %} and then select the **Tiff Viewer** visualization plugin.
>
{: .hands_on}

Your image should look something like this:

![raw input image](../../images/imaging-introduction/viz_input.png){: width="75%"}

> <question-title></question-title>
>
> You can observe that the image content is barely visible. Why?
>
> > <solution-title></solution-title>
> > The original image is 16-bit and the intensity values of the image (33069 to 36863) are spread over a very small fraction (only about 6%) of the intensity values that can be represented using 16 bits (0 to 65535, where 0 corresponds to black and 65535 corresponds to white). Therefore, for improved visibility the intensity histogram of the image should be normalized first.
> {: .solution }
{: .question}


Next we will normalize the histogram to improve the contrast. We do this using a [Contrast Limited Adaptive Histogram Equalization (CLAHE)](https://en.wikipedia.org/wiki/Adaptive_histogram_equalization) approach.

> <hands-on-title>Normalize Histogram</hands-on-title>
>
> 1. {% tool [Perform histogram equalization](toolshed.g2.bx.psu.edu/repos/imgteam/2d_histogram_equalization/ip_histogram_equalization/0.18.1+galaxy0) %} with the following parameters to normalize the histogram of the image:
>    - {% icon param-file %} *"Input image"*: `input.tiff` file
>    - *"Histogram equalization algorithm"*: `CLAHE`
> 2. Rename {% icon galaxy-pencil %} the generated file to `input_normalized`.
> 3. Click on the **visualise icon** {% icon galaxy-visualise %} of the file to visually inspect the image using the **Tiff Viewer** visualization plugin.
{: .hands_on}

Your image should now look something like this:

![viz_normalized output image](../../images/imaging-introduction/viz_normalized.png){: width="75%"}

We can now clearly make out the presence of the stained nuclei. Next we will automatically detect these features and segment the image.

# Image Filtering

Specific features of interest (e.g., edges, noise) can be enhanced or suppressed by using an image filter.

> <hands-on-title>Filter image</hands-on-title>
>
> 1. {% tool [Filter 2-D image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_simple_filter/ip_filter_standard/1.12.0+galaxy1) %} with the following parameters to smooth the image:
>    - *"Filter type"*: `Gaussian`
>    - *"Sigma"*: `3`
>    - {% icon param-file %} *"Source file"*: `input.tiff` file
> 2. Rename {% icon galaxy-pencil %} the generated file to `input_smoothed`.
> 3. {% tool [Perform histogram equalization](toolshed.g2.bx.psu.edu/repos/imgteam/2d_histogram_equalization/ip_histogram_equalization/0.18.1+galaxy0) %} with the following parameters to normalize the histogram of the image:
>    - {% icon param-file %} *"Input image"*: `input_smoothed` file (output of {% tool [Filter 2-D image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_simple_filter/ip_filter_standard/1.12.0+galaxy1) %})
>    - *"Histogram equalization algorithm"*: `CLAHE`
> 4. Rename {% icon galaxy-pencil %} the generated file to `input_smoothed_normalized`.
> 5. Click on the **visualise icon** {% icon galaxy-visualise %} of the file to visually inspect the image and compare the result with `input_normalized`. You can observe that `input_smoothed_normalized` has significantly reduced noise.
{: .hands_on}

Your image should now look something like this:

![viz_smoothed_normalized output image](../../images/imaging-introduction/viz_smoothed_normalized.png){: width="75%"}


# Segmentation

Objects of interest like nuclei can be segmented by using a smoothed image and thresholding. Moreover, the results can be overlayed with the original image.

> <hands-on-title>Segment image</hands-on-title>
>
> 1. {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.18.1+galaxy3) %} with the following parameters to segment the image:
>    - {% icon param-file %} *"Input image"*: `input_smoothed` file (output of {% tool [Filter 2-D image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_simple_filter/ip_filter_standard/1.12.0+galaxy1) %})
>    - *"Thresholding method"*: `Globally adaptive / Otsu`
> 2. Rename {% icon galaxy-pencil %} the generated file to `input_segmented`.
> 3. {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %} with the following parameters to segment the image:
>    - {% icon param-file %} *"Binary image"*: `input_segmented` file (output of {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.18.1+galaxy3) %})
> 4. Rename {% icon galaxy-pencil %} the generated file to `input_segmented_labeled`
>
>    > <question-title></question-title>
>    >
>    > 1. What does {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %} do? (Hint: check the tool help section)
>    > 2. View the `input_segmented_labeled` image from the last step, what do you see? Can you explain this result?
>    > 3. Exercise: Try to make the information in this image better visible (Hint: use {% tool [Perform histogram equalization](toolshed.g2.bx.psu.edu/repos/imgteam/2d_histogram_equalization/ip_histogram_equalization/0.18.1+galaxy0) %})
>    >
>    > > <solution-title></solution-title>
>    > > 1. The tool assigns each connected component (e.g., segmented cell) in the image a unique object ID called *label* and stores it as the intensity value.
>    > > 2. The image looks completely black.
>    > >    The object labels generated by {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %} are relatively low. Since the labels are stored as intensity values, these are too low to be visible in this case. Nevertheless, there is more information in this image than meets the eye.
>    > > 3. To make the labeled objects visible, the values have to be stretched to a larger range of visible intensity values. We
>    > >    can do that by equalizing the histogram again. To this end, use {% tool [Perform histogram equalization](toolshed.g2.bx.psu.edu/repos/imgteam/2d_histogram_equalization/ip_histogram_equalization/0.18.1+galaxy0) %} with the following parameters to normalize the intensity values:
>    > >    - {% icon param-file %} *"Input image"*: `input_segmented_labeled` file (output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %})
>    > >    - *"Histogram equalization algorithm"*: `CLAHE`
>    > >
>    > >    The information contained in the original image has now become visible to the human eye:
>    > >    ![normalized viz_segmented file](../../images/imaging-introduction/viz_segmented.png)
>    > >
>    > {: .solution }
>    {: .question}
>
>
> 7. {% tool [Overlay images](toolshed.g2.bx.psu.edu/repos/imgteam/overlay_images/ip_overlay_images/0.0.4+galaxy4) %} with the following parameters to convert the image to PNG:
>    - *"Type of the overlay"*: `Segmentation contours over image`
>    - {% icon param-file %} *"Intensity image"*: `input_normalized` file
>    - {% icon param-file %} *"Label map"*: `input_segmented_labeled` file (output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %})
>    - *"Contour thickness"*: `2`
>    - *"Contour color"*: `red`
>    - *"Show labels"*: `yes`
>    - *"Label color"*: `yellow`
> 8. Click on the {% icon galaxy-eye %} (eye) icon next to the file name, to look at the file content and assess the segmentation performance.
> 9. {% tool [Count objects in label map](toolshed.g2.bx.psu.edu/repos/imgteam/count_objects/ip_count_objects/0.0.5-2) %} with the following parameters to count the segmented objects in the image:
>    - {% icon param-file %} *"Source file"*: `input_segmented_labeled` file (output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %})
>
>    > <question-title></question-title>
>    >
>    > How many objects were segmented?
>    >
>    > > <solution-title></solution-title>
>    > >  The {% tool [Count objects in label map](toolshed.g2.bx.psu.edu/repos/imgteam/count_objects/ip_count_objects/0.0.5-2) %} tool counted 425 objects.
>    > {: .solution }
>    {: .question}
{: .hands_on}

The resulting image should look something like this:

![segmentation mask output image](../../images/imaging-introduction/viz_segmentation_mask.png){: width="75%"}

We see the segmentation mask overlayed; each detected object (nucleus) is labeled with its ID value.

We see that with the help of just a few simple steps, we were able to detect the locations of the stained nuclei, and count them.

# Conclusion


In this exercise you imported images into Galaxy, extracted meta information from an image, learned how to visualize microscopy images, filtered the image, and segmented cells using Galaxy.

tags_json:
["Image segmentation", "Image thresholding", "Conversion", "Object counting", "Overlay", "Fluorescence microscopy"]


topic: imaging
tutorial: detection-of-mitoflashes
title: Tracking of mitochondria and capturing mitoflashes
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["Isolated object tracking", "Temporal analysis", "Time-lapse data"]
content_hash: 19d17402786410d30a2a5e70aff76f88
last_modified: 2026-02-26T21:12:16.724331
zenodo_link: https://zenodo.org/records/14071552
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/detection-of-mitoflashes/tutorial.html

content:
Mitochondria act like fuel stations for the cell, supplying the energy needed to keep it functioning and healthy. During certain activities, they can produce bursts of **superoxide**, known as '**mitoflashes**.' These short, intense events occur in individual mitochondria and can be observed in cardiomyocytes, skeletal muscle, hippocampal neurons, chondrocytes, isolated mitochondria, and among other types of eukaryotic cells using confocal microscopy. Commonly detected with a mitochondria-targeted circularly permuted fluorescent protein (mt-cpYFP), mitoflashes provide real-time insights into mitochondrial respiration function _in situ_ and act as a biosensor for superoxide levels, reflecting the activity of the mitochondrial electron transport chain.

Mitoflashes are triggered by elevated **reactive oxygen species (ROS)** levels, and various physiological and stress-related signals including fluctuations in mitochondrial membrane potential, transient openings of the mitochondrial permeability transition pore (mPTP), calcium level changes, and bioenergetic stress (like nutrient deprivation or high metabolic demand, {% cite Wang2022 %}). Additionally, they can be induced during specific cellular processes, like differentiation and certain developmental stages, where mitochondrial function is tightly regulated. This burst event activates the mPTP, causing a depolarization of the mitochondrial membrane potential (ΔΨm) and a mild alkalization of the matrix ({% cite Feng2019 %}). While mitoflashes signal mitochondrial activity and ROS changes, they do not precisely quantify ROS levels or other cellular signatures. These events, however, are able to distinguish active from static mitochondria, making them valuable biomarkers for tracking mitochondrial health and function.

Importantly, the frequency and kinetics of mitoflashes hold physiological and pathophysiological implications. They correlate with key processes such as muscle contraction, cell differentiation, neuron development, wound healing, and lifespan prediction. The frequency and characteristics of mitoflashes vary by cell type; for example, adult cardiomyocytes display approximately 3.8 ± 0.5 mitoflashes, while primary cultured hippocampal neurons show around 31 ± 4 per cell ({% cite Li2020 %}).

In this tutorial, you will learn to track mitochondria in live-cell imaging data and detect mitoflashes using Galaxy tools for image analysis. You will work with time-lapse microscopy data, often stored as TIFF files with image stacks, to observe and analyze mitochondrial events across multiple time points. By identifying these events and quantifying their frequency through intensity measurements fitted to a curve, you'll gain insights into mitochondrial behavior and activity over time.

This tutorial is based on a workflow originally described in {% cite Wang2022 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. Understanding the concept and biological relevance of mitoflashes.
> 2. Processing live-cell imaging data for mitochondrial tracking.
> 3. Detecting and analyzing mitoflashes with Galaxy tools.
> 4. Analysis and Interpretation of mitoflash events.
>
{: .agenda}

# Preparing Your Data

First, we need to upload the data we'll work with. Ensure that you have the necessary files prepared from Zenodo or a shared data library within Galaxy.

## Data Upload

> <hands-on-title>Data Upload</hands-on-title>
>
> 1. Create a new history for this tutorial in Galaxy.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the mitoflash imaging data from [Zenodo]({{ page.zenodo_link }}) or from the shared data library:
>    - **Important:** Choose the correct data type if prompted (tiff).
>
>    ```
>    {{ page.zenodo_link }}/files/mitoflashes_8bit.tiff
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>    
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
> 
> 3. Confirm that the datatypes are correct for each file:
>    - `mitoflashes_8bit.tiff` should be an image in **tiff** file format.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
> 4. Tag each dataset with a label like "mitoflash_1" for easy identification later if you are working with multiple images.
>
{: .hands_on}

# Analyzing Mitochondrial Movement

In this section, we will focus on the detection of the mitochondria in a time-lapse image sequence based on local intensity maxima. This will provide insights into mitochondrial energy production and cellular health dynamics in the subsequent analysis.

## Step 1: Mitochondrion Detection

> <hands-on-title>Mitochondrion Detection</hands-on-title>
>
> 1. {% tool [Spot Detection](toolshed.g2.bx.psu.edu/repos/imgteam/spot_detection_2d/ip_spot_detection_2d/0.0.1) %} with the following recommended parameters:
>    - {% icon param-file %} *"Image input"*: `mitoflashes_8bit.tiff`
>    - **Starting time point**: `1` (first frame in the stack)
>    - **Ending time point**: `0` (use `0` to process until the last frame)
>    - **Intensity measurement method**: Set to `Smoothed` for a more averaged intensity or `Robust` for resilience to outliers.
>    - **Threshold**: Start at 10%; lower to 5-8% to catch weaker signals, or raise to 15-20% to focus on the strongest signals.
>    - **Gaussian filter sigma**: Use a value around `1` to suppress noise; increase to `1.5` if noise is high.
>    - **Boundary pixels**: Set to `10` to ignore spots within the image edges, adjusting as needed for image size.
>
>    This tool detects mitochondria based on intensity maxima. Adjusting the threshold parameter can improve the accuracy of spot detection, allowing to reduce the number of missed or spuriously detected mitochondria (as a rule of thumb, the latter is better tolerable for our analysis).
>
{: .hands_on}


## Step 2: Tracking Mitochondrial Movement

Mitochondria are dynamic organelles, constantly moving, and changing position within the cell. Without tracking, intensity measurements could be inconsistent or misleading, as they might reflect overlapping signals from different mitochondria rather than continuous changes in a single organelle. 

By linking each mitochondrion’s position across frames, we ensure that intensity fluctuations represent true changes within the same mitochondrial structure, enabling precise analysis of events like mitoflashes.

> <hands-on-title>Tracking Mitochondrial Movement</hands-on-title>
>
> 1. {% tool [Perform linking in time series (nearest neighbors)](toolshed.g2.bx.psu.edu/repos/imgteam/points_association_nn/ip_points_association_nn/0.0.3-2) %} with the following recommended parameters:
>    - {% icon param-file %} *"Coordinates (and intensities) of input points"*: Output from the **Spot Detection** tool.
>    - **Neighborhood size**: Set to `6` pixels as a starting value. Adjust within `1-10` pixels based on the density of points and proximity between spots across frames.
>    - **Intensity threshold**: A recommended starting value is `25%` of the global maximum intensity. Increase to `30-40%` for stricter filtering if spots have low intensity.
>    - **Minimum track length**: Use `50%` of the sequence length as a baseline. Adjust based on the expected duration of mitoflash events.
>    
>    This tool links detected spots across frames to track their movement. The **Neighborhood size** parameter controls the maximum allowed distance for associating points between frames (in pixels). Increase the **Intensity threshold** parameter to filter out more weak signals (e.g., to remove spurious tracks).
>
{: .hands_on}

# Detecting Mitoflashes

Mitoflashes are identified based on sudden changes in fluorescence intensity in mitochondria, signifying superoxide bursts. We will use a curve-fitting tool to capture these fluctuations and measure their characteristics.

## Step 3: Curve Fitting for Mitoflash Detection

> <hands-on-title>Mitoflash Detection</hands-on-title>
>
> 1. {% tool [Perform curve fitting](toolshed.g2.bx.psu.edu/repos/imgteam/curve_fitting/ip_curve_fitting/0.0.3-2) %} with the following parameters:
>    - {% icon param-file %} *"File name of input data points (xlsx)"*: Output from the **Perform linking in time series (nearest neighbors)** tool.
>    - **Degree of the polynomial function**: Set to `2nd degree`.
>    - **Penalty**: Choose *Least absolute deviations (LAD)* for robust fitting to intensity fluctuations.
>    - **Alpha**: Set a significance level, such as `0.01`, to generate assistive curves if needed.
>
>    This step uses polynomial curve fitting to model the basic underlying intensity patterns over time. The chosen **Penalty** ensures that the curve remains robust against brief, high-intensity deviations (e.g., mitoflashes or image noise), preserving the basic intensity trends. Mitoflash events are then identified as intensity values that significantly exceed the fitted curve by a specified threshold. To aid in this identification, an assistive curve is generated to represent the significance threshold, allowing mitoflash events to be easily recognized as those values exceeding the assistive curve.
>
{: .hands_on}

## Analysis and Interpretation of Results

Congratulations! You’ve reached the final step of your analysis and now have a fully prepared results table, ready for interpretation and further exploration! The output is an Excel file containing the columns **FRAME**, **POS_X**, **POS_Y**, **INTENSITY**, **CURVE**, and **CURVE_A**. This information is available for all detected spots in your image stack and can be used to calculate metrics for interpreting and plotting your mitoflash events results. 

Therefore, we can now analyze the _frequency_, _duration_, and _intensity_ of the mitoflashes. This data will allow us to understand the physiological significance of mitoflashes in processes like muscle contraction, neuron development, and wound healing. In the upcoming section, we will expand on how to interpret and plot these mitoflash events to give them biological meaning.

Here is an example of what the final output from the curve fitting tool in this tutorial looks like:

![Curve fitting results mitoflash](../../images/detection-of-mitoflashes/Curve_fitting_results_mitoflash.png "Mitoflash Curve Fitting Results: Data for plotting.")

> <question-title>Example Analysis</question-title>
>
> Consider the following example data table.
>
> | FRAME | POS_X | POS_Y | INTENSITY | CURVE   | CURVE_A |
> |-------|-------|-------|-----------|---------|---------|
> | 1     | 57    | 474   | 114.84    | 116.10  | 165.38  |
> | 2     | 57    | 474   | 115.91    | 116.11  | 165.39  |
> | 3     | 57    | 474   | 166.00    | 166.05  | 165.40  |
> | 4     | 57    | 474   | 116.24    | 116.13  | 165.42  |
> | 5     | 57    | 474   | 116.40    | 116.15  | 165.43  |
> | 6     | 57    | 474   | 164.50    | 165.00  | 165.44  |
> | 7     | 57    | 474   | 165.10    | 165.20  | 165.45  |
> | 8     | 57    | 474   | 115.69    | 116.33  | 165.62  |
>
>
> 1. Which frames show **INTENSITY** values that correspond to mitoflash events?
> 2. What does a case like Frame 7 (where both **INTENSITY** and **CURVE** are close to **CURVE_A**) indicate in terms of biological significance or computational interpretation?
>
> > <solution-title>Solutions</solution-title>
> >
> > 1. Frame 3 demonstrates a brief but significant increase of the intensity value (**INTENSITY > CURVE_A**), that corresponds to a mitoflash event. Frame 6 shows **CURVE > CURVE_A**, while **INTENSITY** remains below **CURVE_A**.
> > 2. Frame 7 represents a borderline case where both **INTENSITY** and **CURVE** are near **CURVE_A**. This could reflect computational noise, weak mitoflash activity, or background fluctuations that do not meet the threshold for a significant event.
> >
> {: .solution}
>
{: .question}

### Key Parameters for Analysis and Plotting

1. **FRAME**: Represents time points in the data, with each row corresponding to a frame in the time-lapse sequence. Use **FRAME** as the x-axis for time-based plotting.

2. **POS_X** and **POS_Y**: These columns provide the coordinates of each detected mitoflash, allowing visualization of mitochondrial movement and spatial distribution within the cell.

3. **INTENSITY**: Shows the raw intensity values for each mitochondrion at each time point. Plotting **INTENSITY** against **FRAME** reveals fluctuations over time, with peaks indicating mitoflash events.

4. **CURVE** and **CURVE_A**: Contain fitted curve data that smooths out noise to highlight true intensity peaks. **CURVE** is the main fit, while **CURVE_A** may show adjusted fits if available.
   
   - **Amplitude (F/F0)**: The peak value in **CURVE** or **CURVE_A**, representing the magnitude of each mitoflash. Higher amplitudes indicate stronger bursts, which can have biological significance.
   - **Tpk (Time to Peak)**: The frame where **CURVE** or **CURVE_A** reaches maximum intensity, indicating the speed of each mitoflash.
   - **T50 (Duration)**: The frames where the curve values are at least half of the peak value, representing the mitoflash duration. Longer durations may reflect sustained mitochondrial activity.

### Suggested Plots

- **Mitoflash Intensity Over Time**: Plot **FRAME** (x-axis) against **INTENSITY** or **CURVE** (y-axis) to observe intensity fluctuations over time, with peaks marking mitoflash events.
  
- **Amplitude and Duration Analysis**: Create a histogram or scatter plot of **Amplitude** (from **CURVE** peaks) and **T50** to explore the distribution of mitoflash magnitudes and durations.

- **Spatial Mapping of Mitoflashes**: Use **POS_X** and **POS_Y** for a 2D plot, with each point representing a mitoflash location. Color-code points by **Amplitude** to show intensity distribution across the cell.

This format seamlessly integrates the example data and a question for the user to reflect on and interact with the tutorial, helping them understand how to interpret differences between **INTENSITY**, **CURVE**, and **CURVE_A**.

These analyses and visualizations provide an understanding of mitochondrial activity and responses to cellular conditions, enabling insights into mitochondrial health and function in various states.

![Image of the workflow](../../images/detection-of-mitoflashes/Workflow_mitoflash.png "Overview of the workflow for detecting mitoflash using Mitochondrion detection (Step 1), Mitochondrion tracking (Step 2), and Curve fitting to measured intensities (Step 3).")

# Conclusion

In this tutorial, we have covered key techniques for tracking mitochondria and detecting mitoflashes. By processing and analyzing live-cell imaging data, you can gain valuable insights into mitochondrial behavior and health in cells. Tracking mitoflashes can contribute to studies in metabolism, aging, and diseases linked to mitochondrial dysfunction in verious cell types and eukaryotic organisms.


topic: imaging
tutorial: multiplex-tissue-imaging-TMA
title: End-to-End Tissue Microarray Image Analysis with Galaxy-ME
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["Image segmentation", "Tissue imaging"]
content_hash: 0226498b560e6dc393abc80f34c3f97a
last_modified: 2026-02-26T21:12:16.806799
zenodo_link: https://doi.org/10.5281/zenodo.7622545
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/multiplex-tissue-imaging-TMA/tutorial.html

content:
Multiplex tissue images are large, multi-channel images that contain intensity data for numerous biomarkers. The methods for generating multiplex tissue images are diverse, and each method can require specialized knowledge for downstream processing and analysis. The MCMICRO ({% cite Schapiro2021 %}) pipeline was developed to process multiplex images into single-cell data, and to have the range of tools to accommodate for different imaging methods. The tools used in the MCMICRO pipeline, in addition to tools for single-cell analysis, spatial analysis, and interactive visualization are available in Galaxy to facilitate comprehensive and accessible analyses of multiplex tissue images. The MCMICRO tools available in Galaxy are capable of processing Whole Slide Images (WSI) and Tissue Microarrays (TMA). WSIs are images in which a tissue section from a single sample occupies the entire microscope slide; whereas, TMAs multiplex smaller cores from multiple samples onto a single slide. This tutorial will demonstrate how to use the Galaxy multiplex imaging tools to process and analyze publicly available TMA test data provided by MCMICRO (Figure 1); however, **the majority of the steps in this tutorial are the same for both TMAs and WSIs and notes are made throughout the tutorial where processing of these two imaging types diverge.**

Find a full [example history here.](https://cancer.usegalaxy.org/u/watsocam/h/answer-key-history-gtnexemplar002tmaworkflowfeb2025)

![Aviator screenshot, described in figure caption](../../images/multiplex-tissue-imaging-TMA/ex2_combined_avivator.png "Fully registered image of the MCMICRO Exemplar-002 Tissue microarray. Exemplar-002 consists of four cores, each with a distinct tissue organization and expression of biomarkers. In the image, there are six biomarkers shown: DNA (white), CD163 (yellow), CD3D (blue), CD31 (red), VDAC1 (green), and Keratin (orange). This image is being viewed using Avivator, an interactive tool that allows the user to selectively view channels and adjust channel intensities.")


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Get data

Multiplex tissue images come in a variety of forms and file-types depending on the modality or platform used. For this tutorial, the Exemplar-002 data was imaged using Cyclic Immunofluorescence (CycIF) with a RareCyte slide scanner. Many of the steps in this workflow have platform-specific parameters, and the hands-on sections will show the best parameters for CycIF RareCyte images; however, notes will be made where critical differences may occur depending on the modality or platform throughout the tutorial.

> <hands-on-title> Data import to history </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{page.topic_name}}`
>     -> `{{page.title}}`):
>    ```
>    {{ page.zenodo_link }}/files/markers.csv
>    {{ page.zenodo_link }}/files/exemplar_002_phenotypes.csv
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-01.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-02.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-03.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-04.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-05.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-06.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-07.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-08.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-09.ome.tiff
>    {{ page.zenodo_link }}/files/exemplar-002-cycle-10.ome.tiff
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Group the datasets into [collections]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}). Make a collection of the OME-TIFF, ordering the files by cycle number.
>
{: .hands_on}

> <warning-title>Imaging platform differences</warning-title>
>
> The Exemplar-002 raw images are in *ome.tiff* format; however, commonly seen raw file-types are *ome.tiff*, *tiff*, *czi*, and *svs*. If your input images are not *ome.tiff* or *tiff*, you may have to edit the dataset attributes in Galaxy to allow tools to recognize them as viable inputs.
>
{: .warning}

Alternatively, the raw files for each round (10 in total) of the Exemplar-002 data are available on [cancer.usegalaxy.org](https://cancer.usegalaxy.org) under **Data Libraries** (Figure 2). Import the raw files into a new history as a **list collection**.


![Screenshot of the Galaxy data libraries on cancer.usegalaxy.org, highlighting the path to the dataset, Libraries, Exemplar 002, raw. The UI shows all datasets in that folder selected before using the Export to History button to import them as a Collection.](../../images/multiplex-tissue-imaging-TMA/ex2_getData.png "Finding the Exemplar-002 data on cancer.usegalaxy.org Data Libraries.")

# Tile illumination correction with **BaSiC Illumination**

Commonly, raw MTI data will consist of one image per round of imaging. These individual round images are frequently captured in tiles, and there can be slight variations in how each tile was illuminated across the course of imaging. Prior to tile stitching and image registration, the tiles have to undergo illumination correction with **BaSiC Illumination** ({% cite Peng2017 %}) to account for this. Unlike many of the other tools in this workflow, BaSiC has no extra parameters to think about: Just input the collection of raw images and press *go*!

Two new list collections will appear in the history upon completion:

  - BaSiC Illumination on Collection `X`: FFP (flat-field)
  - BaSiC Illumination on Collection `X`: DFP (deep-field)

> <hands-on-title> Illumination correction </hands-on-title>
>
> 1. {% tool [BaSiC Illumination](toolshed.g2.bx.psu.edu/repos/perssond/basic_illumination/basic_illumination/1.1.1+galaxy2) %} with the following parameters:
>
>    - {% icon param-collection %} *"Raw Cycle Images: "*: List collection of raw images
>
{: .hands_on}


# Stitching and registration with **ASHLAR**

After illumination is corrected across round tiles, the tiles must be stitched together, and subsequently, each round mosaic must be registered together into a single pyramidal OME-TIFF file. **ASHLAR** ({% cite Muhlich2022 %}) from MCMICRO provides both of these functions.

> <comment-title>Naming image channels using a marker metadata file</comment-title>
>
> **ASHLAR** optionally reads a marker metadata file to name the channels in the output OME-TIFF image. This marker file will also be used in later steps. Make sure that the marker file is comma-separated and has the `marker_names` as the third column (Figure 3).
>
> ![screenshot of the markers table](../../images/multiplex-tissue-imaging-TMA/ex2_markersFile.png "Markers file, used both in ASHLAR and downstream steps. Critically, the marker_names are in the third column.")
>
{: .comment}

> <hands-on-title>: Image stitching and registration </hands-on-title>
>
> 1. {% tool [ASHLAR](toolshed.g2.bx.psu.edu/repos/perssond/ashlar/ashlar/1.18.0+galaxy1) %} with the following parameters:
>
>    - {% icon param-collection %} *"Raw Images"*: List collection of raw images
>    - {% icon param-collection %} *"Deep Field Profile Images"*: List collection of DFP images produced by **BaSiC Illumination**
>    - {% icon param-collection %} *"Flat Field Profile Images"*: List collection of FFP images produced by **BaSiC Illumination**
>    - *"Rename channels in OME-XML metadata"*: `Yes`
>        - {% icon param-file %} *"Markers File"*: Comma-separated markers file with marker_names in third column
>
{: .hands_on}

> <warning-title>Imaging platform differences</warning-title>
>
> ASHLAR, among other tools in the MCMICRO and Galaxy-ME pre-processing tools have some parameters that are specific to the
> imaging patform used. By default, ASHLAR is oriented to work with images from RareCyte scanners. AxioScan scanners render images
> in a different orientation. Because of this, when using ASHLAR on AxioScan images, it is important to select the **Flip Y-Axis**
> parameter to *Yes*
>
> ASHLAR will work for most imaging modalities; however, certain modalities require different tools to be registered. For example,
> multiplex immunohistochemistry (mIHC) images must use an aligner that registers each moving image to a reference Hematoxylin image.
> For this, Galaxy-ME includes the alternative registration tool {% tool [palom](toolshed.g2.bx.psu.edu/repos/watsocam/palom/palom/2022.8.2+galaxy0) %}.
>
{: .warning}


# TMA dearray with **UNetCoreograph**

Many downstream processing and analysis steps require each individual core from the TMA to be in a separate image file. To accomplish this from our registered ome.tiff image, we can use **UNetCoreograph** to detect and crop each core into separate files.

UNetCoreograph will output images (used for downstream steps), masks, and a preview image (Figure 4.).

![Image of four cores (numbered 1-4) on a slide.](../../images/multiplex-tissue-imaging-TMA/ex2_dearray.png "Preview image from UNetCoreograph, outlines show detection of each individual core in the TMA.")

> <hands-on-title> TMA dearray </hands-on-title>
>
> 1. {% tool [UNetCoreograph](toolshed.g2.bx.psu.edu/repos/perssond/coreograph/unet_coreograph/2.2.8+galaxy1) %} with the following parameters:
>
>    - {% icon param-file %} *"Registered TIFF"*: The output of **ASHLAR** (registered, pyramidal OME-TIFF file)
>
>    > <comment-title>What about Whole Slide Images?</comment-title>
>    >
>    > Whole slide images do not need to be dearrayed, so in most cases, this step can be skipped; however, UNetCoreograph has the *"Tissue"* option, which when selected, can act to separate the whole tissue from the background in a whole slide image which can be useful. In this case, it is important to toggle the *"Downsample factor"* as this often needs to be higher when extracting whole tissues.
>    {: .comment}
>
{: .hands_on}


# Perform segmentation of multiplexed tissue data with **Mesmer**

Nuclear and/or cellular segmentation is the basis for all downstream single-cell analyses. Different segmentation tools work highly variably depending on the imaging modality or platform used. Because of this, Galaxy-ME has incorporated several segmentation tools so users may find the tool that works optimally for their data.

Available segmentation tools in Galaxy-ME:

  - Mesmer ({% cite Greenwald2021 %})
  - UnMicst and s3segmenter ({% cite Yapp2022 %})
  - Cellpose ({% cite Stringer2020 %})

In this tutorial, we use **Mesmer** because it tends to perform generally well on a diverse range of image types, and can be adapted to suite different modalities.

> <comment-title>Important detail: Running images in batches</comment-title>
>
> Now that each image has been split into individual core images, downstream tools must be run on the images separately. Luckily, Galaxy makes this easy by including the option to run each tool in batch across a collection of inputs. Next to the input for the tool, select {% icon param-collection %} (**Dataset collection**) as the input type, and pass the collection output by UNetCoreograph as input. If these tools are being used for a single WSI, this can be ignored and the tools can be run on single files individually. If users are processing multiple WSIs simultaneously, the steps would be identical to the TMA case, storing images in collections and running tools in batches.
>
{: .comment}

> <hands-on-title> Nuclear segmentation </hands-on-title>
>
> 1. {% tool [Mesmer](toolshed.g2.bx.psu.edu/repos/goeckslab/mesmer/mesmer/0.12.3+galaxy3) %} with the following parameters:
>    - {% icon param-collection %} *"Image containing all the marker(s) "*: Collection output of UNetCoreograph (images)
>    - *"The numerical indices of the channel(s) for the nuclear markers"*: `0`
>    - *"Resolution of the image in microns-per-pixel"*: `0.65`
>    - *"Compartment for segmentation prediction:"*: `Nuclear`
>
>    > <comment-title>np.squeeze</comment-title>
>    >
>    > Make sure the **np.squeeze** parameter is set to `Yes` to make the output compatible with next steps.
>    > This is the default behavior.
>    {: .comment}
>
{: .hands_on}

> <warning-title>Imaging platform differences: Image resolution</warning-title>
>
> A crucial parameter for Mesmer and other segmentation tools is the **Image resolution**. This is reported in microns/pixel, and can vary depending on the imaging platform used and the settings at image acquisition. Mesmer accepts the resolution in microns/pixel; however, if using UnMICST, the resolution must be reported as a ratio of the resolution of UnMICST's training images (0.65). For example, when using UnMICST, if your images were captured at a resolution of 0.65, then the UnMICST value would be 1, but if your images were captured at 0.325 microns/pixel, then the value you would enter for UnMICST would be 0.5.
>
{: .warning}


# Calculate single-cell features with **MCQUANT**

After generating a segmentation mask, the mask and the original registered image can be used to extract mean intensities for each marker in the panel, spatial coordinates, and morphological features for every cell. This step is performed by MCMICRO's quantification module (**MCQUANT**).

Once again, as this is a TMA, we will be running this in batch mode for every core image and its segmentation mask.

The quantification step will produce a CSV cell feature table for every image in the batch.

> <hands-on-title> Quantification </hands-on-title>
>
> 1. {% tool [MCQUANT](toolshed.g2.bx.psu.edu/repos/perssond/quantification/quantification/1.6.0+galaxy0) %} with the following parameters:
>
>    - {% icon param-collection %} *"Registered TIFF "*: Collection output of UNetCoreograph (images)
>    - {% icon param-collection %} *"Primary Cell Mask "*: Collection output of Mesmer (or other segmentation tool)
>    - {% icon param-collection %} *"Additional Cell Masks "*: `Nothing Selected` (Other tools may produce multiple mask types)
>    - {% icon param-file %} *"Marker channels"*: Comma-separated markers file with marker_names in third column
>
>    > <comment-title>Mask metrics and Intensity metrics</comment-title>
>    >
>    > Leaving the *"mask metrics"* and *"intensity metrics"* blank will by default run all available metrics
>    >
>    {: .comment}
>
{: .hands_on}


# Convert McMicro Output to Anndata

Anndata ({% cite Virshup2021 %}) is a Python package and file format schema for working with annotated data matrices that has gained popularity in the single-cell analysis community. Many downstream analysis tools, including Scimap ({%cite Nirmal2024 %}), Scanpy ({% cite Wolf2018 %}), and Squidpy ({% cite Palla2022 %}) are built around anndata format files (h5ad). This tool splits the marker intensity data into a separate dataframe (`X`), and places all observational data (spatial coordinates, morphological features, etc.) in the cell feature table into a separate dataframe (`obs`) that shares the same indices as `X`. In downstream analyses, new categorical variables, such as phenotype assignments for each cell, are stored in the `obs` dataframe.

Learn more about this file format at the [anndata documentation](https://anndata.readthedocs.io/en/latest/index.html).

> <hands-on-title> Conver to Anndata </hands-on-title>
>
> 1. {% tool [Convert McMicro Output to Anndata](toolshed.g2.bx.psu.edu/repos/goeckslab/scimap_mcmicro_to_anndata/scimap_mcmicro_to_anndata/2.1.0+galaxy2) %} with the following parameters:
>
>    - {% icon param-collection %} *"Select the input image or images"*: Collection output of Quantification (cellMaskQuant)
>    - In *"Advanced Options"*:
>        - *"Whether to remove the DNA channels from the final output"*: `No`
>        - *"Whether to use unique name for cells/rows"*: `No`
>
>    > <warning-title>Important parameter: Unique names for cells/rows</warning-title>
>    >
>    > Setting *"Whether to use unique name for cells/rows"* to `No` to ensures that downstream interactive visualizations will be able to map observational features to the mask CellIDs.
>    {: .warning}
>
{: .hands_on}

> <comment-title>Connection to Scanpy and other Galaxy tool suites</comment-title>
>
> Conversion to Anndata format allows users to feed their MTI datasets into a broad array of single-cell analysis tools that have been developed by the Galaxy community. For example, the Python package {% tool [Scanpy ComputeGraph](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.9.3+galaxy0) %} is available as a Galaxy tool suite and is extremely useful for filtering, feature scaling and normalization, dimensionality reduction, community detection, and many other downstream analysis tasks.
>
{: .comment}

# Scimap: Single Cell Phenotyping

There are several ways to classify cells available in Galaxy-ME. Unsupervised approaches, such as Leiden clustering, can be performed on all cells and phenotypes can be manually annotated based on marker expression patterns observed by the user. Alternatively, the R-based tool {% tool [CELESTA cell typing](toolshed.g2.bx.psu.edu/repos/goeckslab/celesta/celesta/0.0.0.9+galaxy3) %} can be used to perform a spatially-aware hierarchical cell phenotyping ({% cite Zhang2022 %}). These approaches can take a fair amount of trial-and-error, so here we will demonstrate automated phenotyping based on thresholds of specific lineage markers using the Scimap Python package ({%cite Nirmal2024 %}). Scimap phenotyping can either be provided a table of manual gate values for each marker of interest (which can be determined using the **GateFinder** tool in Galaxy-ME), or by default, Scimap will fit a Gaussian Mixture Model (GMM) to the intensity distribution for each marker to determine positive and negative populations for that marker. The marker intensity values are rescaled between (0,1) with 0.5 being the cut-off between negative and positive populations. Scimap uses a 'Phenotype workflow' to guide the classification of cells (Figure 5.). For more on how to construct a Scimap workflow, see the [Scimap documentation](https://scimap-doc.readthedocs.io/en/latest/tutorials/scimap-tutorial-cell-phenotyping/).


![Screenshot of the phenotypes table](../../images/multiplex-tissue-imaging-TMA/ex2_phenotypeWF.png "Example of a phenotype workflow compatible with Scimap. 'Pos' means that the marker must be positive to be classified as the respective phenotype. 'Anypos' means any, but not necessarily all, of the listed markers can be positive to call the respective phenotype.")

> <hands-on-title> Single Cell Phenotyping with Scimap </hands-on-title>
>
> 1. {% tool [Single Cell Phenotyping](toolshed.g2.bx.psu.edu/repos/goeckslab/scimap_phenotyping/scimap_phenotyping/2.1.0+galaxy2) %} with the following parameters:
>
>    - {% icon param-collection %} *"Select the input anndata"*: Output of **Convert MCMICRO output to Anndata**
>    - *"Whether to log the data prior to rescaling"*: `True`
>    - {% icon param-file %} *"Select the dataset containing manual gate information"*: (Optional) manually determined gates in CSV format. Gates will be determined automatically using a GMM for each marker if this file is not provided
>    - {% icon param-file %} *"Select the dataset containing gating workflow"*: `exemplar_002_phenotypes.csv`, CSV phenotype workflow (Figure 5.)
>
>    > <comment-title>Limitations of GMM automated phenotyping</comment-title>
>    >
>    > When manual gates are not provided, Scimap fits a 2-component Gaussian Mixture Model (GMM) to determine a threshold between positive and negative cells. This automated gating can work well when markers are highly abundant within the tissue, and the data shows a bimodal distribution; **however**, this is usually not the case, and it is recommended to provide manual gates which can be found using {% tool [GateFinder](toolshed.g2.bx.psu.edu/repos/goeckslab/gate_finder/gate_finder/3.5.1+galaxy0) %}
>    >
>    {: .comment}
>
{: .hands_on}


# Interactive visualization of multiplex tissue images

Visual analysis is an important part of multiplex tissue imaging workflows. Galaxy-ME has several tools that make interactive visualization easy, and can be used at various stages of analysis.

## Converting UNetCoreograph images to OME-TIFF using the **Convert image** tool

UNetCoreograph outputs each individual core image in `tiff` format. Interactive visualization tools, such as **Vitessce** and **Avivator** require the images to be in `OME-TIFF` format to be viewed. Galaxy-ME includes a conversion tool that can accomodate this, along with many other useful conversion functions.

> <hands-on-title> Convert image </hands-on-title>
>
> 1. {% tool [Convert image](toolshed.g2.bx.psu.edu/repos/imgteam/bfconvert/ip_convertimage/6.7.0+galaxy0) %} with the following parameters:
>    -  {% icon param-collection %} *"Input Image"*: `UNetCoreograph Images`
>    - *"Output data type"*: `OME TIFF`
>    - *"Tile image"*: `Tile image`
>    - *"Pyramid image"*: `Generate Pyramid`
>
{: .hands_on}


## Rename OME-TIFF Channels

Some tools can cause the channel names in an OME-TIFF image to be lost. To fix this, or to change the channel names to whatever the user prefers, the **Rename OME-TIFF Channels** tool can be invoked using a markers file similar to the one used in previous steps.

> <hands-on-title> Rename channels </hands-on-title>
>
> 1. {% tool [Rename OME-TIFF channels](toolshed.g2.bx.psu.edu/repos/goeckslab/rename_tiff_channels/rename_tiff_channels/0.0.2+galaxy1) %} with the following parameters:
>
>    - {% icon param-file %} *"Input image in OME-tiff format"*: `Convert image`
>    - *"Format of input image"*: `ome.tiff`
>    - {% icon param-file %} *"Channel metadata CSV"*: `markers.csv`, Comma-separated markers file with marker_names in third column
>
{: .hands_on}


## Initial visualization with **Avivator**

For any `OME-TIFF` image in a Galaxy-ME history, there will be an option to view the image using **Avivator**. This is a great way to perform an initial inspection of an image for QC purposes before continuing with downstream steps. The **Avivator** window can be launched by clicking the {% icon dataset-visualize %} **Dataset visualization** icon under an expanded dataset, which will open a page with visualization options. From here, click the **Display at Avivator link** (Figure 6).

![A screenshot showing the 'display at aviator' link which can be accessed through the dataset visualizations button.](../../images/multiplex-tissue-imaging-TMA/avivator_new_display.png "The 'display at Avivator' link which can be accessed through the dataset visualizations button.")

Clicking the link will open a new window with the Avivator interactive dashboard. Here, users can select channels to view, adjust the channel contrast and colors for viewing, and zoom in to inspect the image (Figure 7).

![A screenshot showing a TMA core visualized in the Avivator dashboard.](../../images/multiplex-tissue-imaging-TMA/ex2_core2_avivator.png "The Avivator dashboard allows for initial image exploration, channel selection, and zooming.")

> <tip-title>Accessing Avivator on older Galaxy instances</tip-title>
>
> Some Galaxy instances that are running older versions of Galaxy will have the **Display at Avivator** link directly in the expanded dataset view (Figure 8.).
>
> ![A screenshot showing the old 'display at aviator' link which is found on the expanded dataset view.](../../images/multiplex-tissue-imaging-TMA/avivator_old_display.png "Older Galaxy versions will have the 'display at Avivator' link directly on the expanded dataset view.")
>
>
{: .tip}

## Generating an interactive visualization dashboard with **Vitessce**

**Vitessce** is a powerful visualization tool that creates interactive dashboards (Figure 9.) to look at a multiplex `OME-TIFF` images in conjunction with data generated during analysis and stored in an anndata file. The segmentation mask can be overlaid onto the image to qualitatively assess segmentation performance, and visually explore cell-type annotations and marker intensities. The different visualization components are linked by cell index, so users can analyze the image, UMAP representations, heatmaps, and marker intensity violin plots for comprehensive data exploration.

![Screenshot of the vitessce dashboard.](../../images/multiplex-tissue-imaging-TMA/ex2_fullVitessce.png "A Full view of a vitesse dashboard for one core from Exemplar-002.")

> <hands-on-title> Vitessce visualization </hands-on-title>
>
> 1. {% tool [Vitessce](toolshed.g2.bx.psu.edu/repos/goeckslab/vitessce_spatial/vitessce_spatial/3.5.1+galaxy0) %} with the following parameters:
>
>    - {% icon param-collection %} *"Select the OME Tiff image"*: OME-TIFF image to be viewed (or collection of files to run in batch)
>    - {% icon param-collection %} *"Select masks for the OME Tiff image (Optional)"*: Output of Mesmer (or other segmentation tool)
>    - *"Whether to do phenotyping"*: `Yes`
>        - *"Select the anndata contaning phenotyping info"*: `Single Cell Phenotyping`, an anndata file that includes includes cell phenotype annotations.
>        - *"Select an embedding algorithm for scatterplot"*: `UMAP`
>        - *"Input phenotyping keys"*: `Multiple choices`
>            - *"Select the key(s)"*: `phenotype`
>
{: .hands_on}

> <tip-title>Customizing the Vitessce dashboard during use</tip-title>
>
> When running a Vitessce dashboard, the components may look squished and difficult to use. Individual components can be expanded using a handle at the bottom right corner of each component. Additionally, components can be removed to make room for others by clicking the 'X' in the upper right corner of each component. Once the dashboard is closed, it will return to its original state and all original components will be restored when opened again.
>
> ![screenshot of vitessce interface with some of the components removed or resized.](../../images/multiplex-tissue-imaging-TMA/ex2_vitessce_zoomed.png "Each window in the dashboard can be resized or removed to view the components in more detail. Closing and re-opening the dashboard will restore the original state.")
>
>
{: .tip}

# Next steps: Compositional and spatial analyses

Galaxy-ME includes additional tools from **Scimap** and tools from the **Squidpy** package ({% cite Palla2022 %}) that can be used to perform a variety of downstream analyses. For example, once phenotypes have been assigned to individual cells, **Squidpy** has several methods for understanding the spatial organization of the tissue. Using **Squidpy**, a spatial neighborhood graph is first generated, from which the organization of specific phenotype groups and their interactions can be quantified.

> <hands-on-title> Spatial analysis with Squidpy </hands-on-title>
>
> 1. {% tool [Analyze and visualize spatial multi-omics data with Squidpy](toolshed.g2.bx.psu.edu/repos/goeckslab/squidpy_spatial/squidpy_spatial/1.5.0+galaxy2) %} generate a spatial neighborhood graph with the following parameters:
>
>    - {% icon param-collection %} *"Select the input anndata"*: Anndata file containing phenotype information (or other variable of interest)
>    - *"Select an analysis"*: `Spatial neighbors -- Create a graph from spatial coordinates`
>
> 2. {% tool [Analyze and visualize spatial multi-omics data with Squidpy](toolshed.g2.bx.psu.edu/repos/goeckslab/squidpy_spatial/squidpy_spatial/1.5.0+galaxy2) %} compute and plot a neighborhood enrichment analysis with the following parameters:
>
>    - {% icon param-collection %} *"Select the input anndata"*: Output of step 1 (anndata file with spatial neighborhood graph)
>    - *"Select an analysis"*: `nhood_enrichment -- Compute neighborhood enrichment by permutation test`
>    - *"Key in anndata.AnnData.obs where clustering is stored"*: `phenotype`
>
>    > <comment-title>Neighborhood enrichment plot</comment-title>
>    >
>    > **Squidpy** was used to calculate neighborhood enrichments for each phenotype in core 2 of exemplar 2 (Figure 11). This shows which phenotypes co-locate most frequently within the tissue.
>    >
>    > ![Heatmap showing phenotype vs neighbourhood enrichment. Most of the heatmap is blue/green (low) but one cell under epithelial is bright yellow (high)](../../images/multiplex-tissue-imaging-TMA/ex2_squidpy_enrichment.png "The output of Squidpy's neighborhood enrichment on core 2 from Exemplar-002.")
>    >
>    {: .comment}
>
> 3. {% tool [Analyze and visualize spatial multi-omics data with Squidpy](toolshed.g2.bx.psu.edu/repos/goeckslab/squidpy_spatial/squidpy_spatial/1.5.0+galaxy2) %} calculate Ripley's L curves for each phenotype with the following parameters:
>
>    - {% icon param-collection %} *"Select the input anndata"*: Output of step 1 (anndata file with spatial neighborhood graph)
>    - *"Select an analysis"*: `nhood_enrichment -- Compute neighborhood enrichment by permutation test`
>    - *"Key in anndata.AnnData.obs where clustering is stored"*: `phenotype`
>    - In *"Advanced Graph Options"*:
>        - *"Which Ripley's statistic to compute"*: `L`
>    - In *"Plotting Options"*:
>        - *"Ripley's statistic to be plotted"*: `L`
>
>    > <comment-title>Ripley's L plot</comment-title>
>    >
>    > **Squidpy** was used to calculate Ripley's L curves for each phenotype in core 2 of exemplar 2 (Figure 12). This shows the overall organization of each phenotype in the tissue. If the curve for a given phenotype lies above the light grey null line (Example: Epithelial cells in Figure 11.), the phenotype is statistically significantly clustered. If the curve lies on the null line (Example: Myeloid lineage in Figure 11.), it's spatial distribution within the tissue is random. If the curve is underneath the null line (Example: T cells in Figure 11.), it's spatial distribution is statistically significantly dispersed.
>    >
>    > ![Graph of Ripley's L. Value is plotted against bins, all of which show cursves starting at 0 and increasing as bins increase. Epithelial is the highest curve.](../../images/multiplex-tissue-imaging-TMA/ex2_squidpy_ripleys.png "The output of Squidpy's Ripley's L curve on core 2 from Exemplar-002.")
>    >
>    {: .comment}
>
> 4. {% tool [Create spatial scatterplot with Squidpy](toolshed.g2.bx.psu.edu/repos/goeckslab/squidpy_spatial/squidpy_spatial/1.5.0+galaxy2) %} generate a spatial scatterplot of cell phenotypes with the following parameters:
>
>    - {% icon param-collection %} *"Select the input anndata"*: Output of step 1 (anndata file with spatial neighborhood graph)
>    - *"Key for annotations in adata.obs or variables/genes"*: `phenotype` (In this case we use the default; however this could be any observational variable (phenotype, leiden, louvain, etc.) or continuous variable name (CD45, CD3, Area, etc.))
>
>    > <comment-title>Spatial scatterplot</comment-title>
>    >
>    > **Squidpy** was used to generate a spatial scatterplot of core 2 of exemplar 2 (Figure 13). This is useful for visualizing the distribution of cell types, or any other categorical or continuous variable in the anndata file.
>    >
>    > ![Spatial scatterplot of TMA core showing cell types annotated by colors](../../images/multiplex-tissue-imaging-TMA/ex2_squidpy_scatter.png "The output of Squidpy's spatial scatterplot tool on core 2 from Exemplar-002.")
>    >
>    {: .comment}
>
{: .hands_on}


# Conclusion

In this tutorial, we demonstrated a complete multiplex tissue imaging analysis workflow performed entirely in a web browser using Galaxy-ME. Using an example tissue microarray imaged with cyclic immunofluorescence provided by MCMICRO, we...

  - Corrected illumination between imaging tiles
  - Stitched and registered input images to produce a single, pyramidal OME-TIFF image that is viewable in multiple built-in interactive viewing tools (Avivator, Vitessce)
  - Split the TMA into separate images for each core
  - Processed each core in parallel, beginning with nuclear segmentation
  - Quantified the mean marker intensities, morphological features, and spatial coordinates of each cell in each core
  - Converted the resulting tabular data to anndata format for convenient downstream anaylses and visualizations
  - Performed marker-based, automatically gated, phenotyping of cells
  - Prepared the dearrayed images and viewed them interactively in a dashboard combined with observational data

![Image of a galaxy workflow with all of the tools from this tutorial.](../../images/multiplex-tissue-imaging-TMA/ex2_galaxyWF.png "The entire workflow used in this tutorial.")


topic: imaging
tutorial: omero-suite
title: Overview of the Galaxy OMERO-suite - Upload images and metadata in OMERO using Galaxy
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["Data management", "Data handling", "Data sharing", "ROI selection"]
content_hash: 0539f2e042472ead520349b296e0aaee
last_modified: 2026-02-26T21:12:16.810010
zenodo_link: https://zenodo.org/records/14205500
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/omero-suite/tutorial.html

content:
The efficient and accurate treatment of microscopy metadata is of great importance, as it 
provides insights that are essential for effective image management, search, organisation, 
interpretation, and sharing. Considering this, it is vital to find ways to properly deal with the huge amount 
of complex and unstructured data for implementing **[Findable, Accessible, 
Interoperable and Reusable (FAIR)](https://www.nature.com/articles/sdata201618)** concepts in bio-imaging.

One of the most flexible and used open-source tools for image and metadata management can be
identified as **[OMERO
(Open Microscopy Environment Remote Objects)](https://www.nature.com/articles/nmeth.1896)**. **OMERO** is an open-source 
software platform designed to manage, visualise, and analyse large sets of biological image
data. Developed by the Open Microscopy Environment consortium, **OMERO** provides researchers 
with a centralised repository to store images and metadata, tools for collaborative 
sharing, and advanced functionalities for image processing and analysis.

In this tutorial, you will learn how to use the different tools of the **Galaxy OMERO-suite**. 
The  **Galaxy OMERO-suite** is based on the Python packages omero-py and ezomero, and it allows interactively building pipelines to upload and fetch image data in OMERO 
using a Galaxy workflow. 
Images can automatically be enriched with metadata (i.e. key-value pairs, tags, raw data, regions 
of interest) and uploaded to an OMERO server. The tools give the possibility 
to the user to intuitively fetch images from the local server and 
perform image analysis.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. Learn the different tools of the Galaxy OMERO-suite to manage image data and metadata
> 2. Get data from an OMERO instance using the Galaxy OMERO-suite
> 3. Build a pipeline for image and metadata upload into OMERO
{: .agenda}


## Before Starting - Set up the OMERO credentials
Before starting you need to set up your OMERO credential. 
This is necessary to connect to a target OMERO instance and needs to be performed just one time.

The OMERO credentials have to be saved in your preferences 
(*User → Preferences → Manage information*).

![omero_credential.png](../../images/omero-suite/omero_credential.png)

This feature is embedded in Galaxy and needs to be enabled by the admin 
of the server you are using. You can now use it in [UseGalaxy.eu](https://usegalaxy.eu/) or ask your 
admin to add it. For more info about the configuration on the admin side, 
please take a look at the [README file of the tool](https://github.com/galaxyproject/tools-iuc/blob/main/tools/idr_download/README.md).

**Please Notice**: The configuration file, which contains your OMERO password and username, will be stored in the job working directory.
    This directory only exists during the runtime of the job and should only be accessible by the system user that runs the job.
    However, please be aware that your username and password **may be exposed** to users with administrative rights.
    We are working on increasing the security of the OMERO suite

# Prepare your data

We need a dataset to upload into your OMERO instance. 
To this end, we have prepared an integrative dataset with images, metadata and regions of interest (ROIs) in Zenodo. 


## Data Upload - Image, metadata and ROI files

> <hands-on-title>Data Upload</hands-on-title>
>
> 1. Create a new history for this tutorial in Galaxy.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the nuclei imaging data from [Zenodo]({{ page.zenodo_link }}) or 
> from the shared data library:
>    - **Important:** Choose the correct data type if prompted.
>
>    ```
>    {{ page.zenodo_link }}
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>    
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>   
>    - Select the following files: 
>    
>       - `image85-H-01-00.tif`
>       - `image85-H-01-00.tif.tabular`
>       - `image86-H-02-00.tif`
>       - `image86-H-02-00.tif.tabular`
>       - `image87-H-03-00.tif`
>       - `image87-H-03-00.tif.tabular`
>       - `metadata_dataset.csv`
> 
> 3. Tag each dataset with a label like "image" (tif files), "rois" (tabular files) and "metadata" (CSV file)
> for easy identification
{: .hands_on}

# Upload images and metadata

In this section, we will focus on how uploading images and metadata into a user-defined OMERO instance.
This is done by using the **OMERO Image Import** and **OMERO Metadata Import** tools.

## Step 1: Image Upload

> <hands-on-title>Upload images into OMERO</hands-on-title>
>
> 1. {% tool [OMERO Image Import](toolshed.g2.bx.psu.edu/repos/ufz/omero_import/omero_import/5.18.0+galaxy3) %} with the following recommended parameters:
>    - {% icon param-file %} *Images to import into OMERO*: You can select here different files to import into OMERO.
>    Select the datasets `image85-H-01-00.tiff`, `image86-H-02-00.tiff` and `image87-H-03-00.tiff`
>    - **OMERO host URL**: Input the URL of your OMERO instance.
>    - **OMERO port**: Input the OMERO port (pre-defined value, *4064*)
>    - {% icon param-file %} **Target Dataset Name**: Type "My_OMERO_Test"
> 
>    A log file text file will be
>    created with the OMERO ID of the newly imported images
{: .hands_on}

After tool execution, you can check the images in your OMERO instance!
![img.png](../../images/omero-suite/omero_import.png)

Obviously, the Dataset and Image ID change according to your instance... In this case, we have created a new
dataset called "My_OMERO_Test" with ID 4005, while images have IDs 30781, 30782 and 30783.

## Step 2: Upload metadata

You can upload two different kinds of metadata files using the OMERO-suite.

- **Key Value Pairs:** Useful for displaying dataset and image metadata in OMERO.web. 
Key Value pairs are for unstructured data where every object can have different keys, and they can be easily edited ad-hoc by users.
- **Tables:** Useful for showing results for multiple images at the same time. Tables are suitable for e.g. analysis results or large structured data, columns have defined types (kv pairs are all strings) and can be queried like a database table.

We will learn now how to upload **Key-Value pairs** using the **OMERO Metadata Import** tool

> <hands-on-title>Upload Key-Value Pairs into OMERO</hands-on-title>
>
> 1. {% tool [OMERO Metadata Import](toolshed.g2.bx.psu.edu/repos/ufz/omero_metadata_import/omero_metadata_import/5.18.0+galaxy3) %} with the following recommended parameters:
>    - {% icon param-file %} *Annotation file*: `metadata_file.tsv`
>    - **OMERO host URL**: Input the URL of your OMERO instance.
>    - **OMERO port**: Input the OMERO port (pre-defined value, *4064*)
>    - {% icon param-file %} **Target Object Type**: Select *"Dataset"*
>    - **Selection**: Select *"Target an existing object"*
>    - **Object ID**: Input the ID of the previously created dataset. *4005* for this training
>    - **Annotation type**: Select *"KV"*
>    - **Annotation Name**: Type *"REMBI_Annotation"*
{: .hands_on}

Switch back to your OMERO instance. 
The Key Values are now in OMERO, associated with a target dataset! Well done!
![img_2.png](../../images/omero-suite/omero_metadata.png)

# Upload region of interest (ROIs)
OMERO support the ROI visualization in OMERO.viewer. 
In this section you will learn to associate ROIs coordinates to
an image stored in OMERO using the **OMERO ROI Import** tool.

Please note that this tool supports just polygon ROIs given in a specific format.
The tool [Analyze particles](https://toolshed.g2.bx.psu.edu/view/imgteam/imagej2_analyze_particles_binary/862af85a50ec)
can automatically create the ROIs in this specific format.

Check the `roi_file.tsv` to have an example. Everything is explained in the tools help section!


> <hands-on-title>Upload images into OMERO</hands-on-title>
>
> 1. {% tool [OMERO ROI Import](toolshed.g2.bx.psu.edu/repos/ufz/omero_roi_import/omero_roi_import/5.18.0+galaxy4) %} with the following recommended parameters:
>    - {% icon param-file %} **Tab File with ROIs**: `roi_file.tsv`
>    - **OMERO host URL**: Input the URL of your OMERO instance.
>    - **OMERO port**: Input the OMERO port (pre-defined value, *4064*)
>    - **Image ID where annotate the ROIs**: Select the **image ID** where to annotate the ROIs. We will go for *30782*
{: .hands_on}

Switch back to your OMERO instance and the image by double-clicking it.
This will open the OMERO.viewer. 
Go now to the "ROIs" sub-panel.
The ROIs are now annotated in OMERO and associated with the target image! **Awesome!**
![img_3.png](../../images/omero-suite/omero_rois.png)



# Get OMERO object IDs and Annotations
You can get information on projects, datasets and image IDs. 
Furthermore, you can fetch annotations and tables
associated with an OMERO object. This is done with the **OMERO get IDs** and 
**OMERO get Object** tools, respectively.

First of all, let's try to get all the image IDs present in all Datasets!

> <hands-on-title>Upload images into OMERO</hands-on-title>
>
> 1. {% tool [OMERO get IDs](toolshed.g2.bx.psu.edu/repos/ufz/omero_get_id/omero_get_id/5.18.0+galaxy0) %} with the following recommended parameters:
>    - **OMERO host URL**: Input the URL of your OMERO instance.
>    - **OMERO port**: Input the OMERO port (pre-defined value, *4064*)
>    - **Type of object to fetch ID:**: Select *"Dataset IDs"*
>    - **Which datasets?**: Select *"All datasets"*
>    - **ID of the project**: *0*
{: .hands_on}

Since we have just one dataset in your OMERO instance, the tool will produce a tabular file
with the IDs of the three images we just uploaded.

| 1     |
|-------|
| 30781 |
| 30782 |
| 30783 |

Now, let's try to get an annotation file...

> <hands-on-title>Get an annotation file from OMERO</hands-on-title>
>
> 1. {% tool [OMERO get Object](toolshed.g2.bx.psu.edu/repos/ufz/omero_get_value/omero_get_value/5.18.0+galaxy0) %} with the following recommended parameters:
>    - **OMERO host URL**: The target OMERO host URL
>    - **OMERO port**: The OMERO port, pre-defined to *4064*
>    - **Type of object to fetch:**: Select *"Annotation"*
>    - **How do you provide the ID(s) of the OMERO object?**: Select *"Comma separated values"*
>    - **ID(s) of the object(s) to fetch on OMERO separated by comma**: 4005
{: .hands_on}

Perfect, you fetched the Annotation file "REMBI_Annotation". 
**OMERO get Object** is particularly useful when you want to fetch data associated with your dataset or 
images!

# Filter OMERO object based on filename, tags and Key-Value Pairs
Finally, you filter OMERO objects based on their features... One option is to filter by filename,
a useful parameter to apply if you want to get certain images. 

This can be done by using the **OMERO IDs** Tool

> <hands-on-title>Upload images into OMERO</hands-on-title>
>
> 1. {% tool [OMERO IDs](toolshed.g2.bx.psu.edu/repos/ufz/omero_filter/omero_filter/5.18.0+galaxy0) %} with the following recommended parameters:
>    - **OMERO host URL**: The target OMERO host URL
>    - **OMERO port**: The OMERO port, pre-defined to *4064*
>    - **Filename to search among the image IDs**: Type `image85-H-01-00_tif.tiff`
>    - **List of images IDs**: Type *30781,30782,30783*
{: .hands_on}

As an output, you will have a tabular file with the following info:

| 1     |
|-------|
| 30782 |

Which is ID associated to the image with file name "image85-H-01-00_tif.tiff"! 

However, you can also use this tool to fetch images with specific file names or tags,
making data fetching an easy task...

# A full workflow for data management with the OMERO suite 

In this section, you will learn to integrate all the different tools to build a workflow for uploading
an images, metadata and ROIs  into OMERO.

## Step 1 - Define the OMERO inputs
To make the workflow work, you need to define six inputs. 

This will make this workflow reusable with different datasets.

> <hands-on-title>Create the inputs for the OMERO pipeline</hands-on-title>
>
> 1. Create a new workflow in the workflow editor.
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
> 2. Select {% icon tool %} **Input dataset collection** from the list of tools:
>    - {% icon param-collection %} **1: Input dataset collection** appears in your workflow. 
>    Change the "Label" of this input to *Input image Dataset*.
> 3. Add two {% icon tool %} **Input dataset**:
>    - {% icon param-file %} **2: Input Dataset** and {% icon param-file %} **3: Input Dataset** appears in your workflow. 
> Change the "Label" of these inputs to *Input Tabular ROIs* and *Input Metadata File*
> 4. Add three {% icon tool %}**Simple Input for Workflow Logic**:
>    - **4: Simple input for workflow logic**, **5: Simple input for workflow logic**, 
>    **6: Simple input for workflow logic**  appear in your workflow. 
>    Change the "Label" of these inputs to *Target Dataset Name*, *OMERO instance address* and
>    *Annotation type*
{: .hands_on}

## Step 2 - OMERO Import
We now add the step for the image import

> <hands-on-title>Add the image upload step to the workflow</hands-on-title>
>
>1. While in the workflow editor add {% tool [OMERO Image Import](toolshed.g2.bx.psu.edu/repos/ufz/omero_import/omero_import/5.18.0+galaxy3) %}  from the list of tools:
>   - Connect the output of {% icon param-file %} **1: Input image Datasets** to the "Images to Import in OMERO" 
>   input of {% icon tool %} **7: OMERO Image Import**.
>   - Connect the output of {% icon param-file %} **4: Target Dataset Name** to the "Target Dataset Name" 
>   input of {% icon tool %} **7: OMERO Image Import**.
>   - Connect the output of {% icon param-file %} **5: OMERO instance address** to the "OMERO host url"
>   input of {% icon tool %} **7: OMERO Image Import**.
{: .hands_on}

## Step 3 - Dynamically parse image IDs 
We need now a small series of steps to get the ID of the image we just upload into OMERO
This can be done with the following tool sequence:

> <hands-on-title>Parse Images IDs </hands-on-title>
>
>1. Add {% icon tool %} **Convert** from the list of tools:
>   - Connect the output of {% icon tool %} **7: OMERO Image Import** to the {% icon tool %} **8: Convert** input
>2. Add {% icon tool %} **Replace text**  from the list of tools:
>   - Connect the output of {% icon tool %} **8: Convert** to the {% icon tool %} **Replace text** input 
>3. Add {% icon tool %} **Split File**  from the list of tools:
>   - Connect the output of {% icon tool %} **9: Replace text** to the {% icon tool %} **10: Split File** 
>4. Add {% icon tool %} **Parse parameter value**  from the list of tools:
>   - Connect the output of {% icon tool %} **10: Split File**  to the {% icon tool %} **11: Parse parameter value**
{: .hands_on}

We got the OMERO image ID! We can now use it as an input for next sequence of tools...

## Step 4 - Metadata and ROIs input

The last section is to upload metadata and ROIs

> <hands-on-title> Add tools to upload metadata and ROIs </hands-on-title>
>
>1. Add {% tool [OMERO Metadata Import](toolshed.g2.bx.psu.edu/repos/ufz/omero_metadata_import/omero_metadata_import/5.18.0+galaxy3) %}  from the list of tools:
>   - Connect the output of {% icon param-file %} **5: OMERO instance address** to the "OMERO host url"
>   input of {% icon tool %} **12: OMERO Metadata Import**.
>   - Connect the output of {% icon param-file %} **11: Parse parameter value** to the "Object ID"
>   input of {% icon tool %} **12: OMERO Metadata Import**.
>   - Connect the output of {% icon param-file %} **6: Annotation type** to the "Annotation type"
>   input of {% icon tool %} **12: OMERO Metadata Import**.
>   - Connect the output of {% icon param-file %} **3: Input Metadata File** to the "Annotation file"
>   input of {% icon tool %} **12: OMERO Metadata Import**.
>2. Add {% tool [OMERO ROI Import](toolshed.g2.bx.psu.edu/repos/ufz/omero_roi_import/omero_roi_import/5.18.0+galaxy4) %}  from the list of tools:
>   - Connect the output of {% icon param-file %} **5: OMERO instance address** to the "OMERO host url"
>   input of {% icon tool %} **13: OMERO ROI Import**.
>   - Connect the output of {% icon param-file %} **2: Input Tabular ROIs** to the "Tab File with ROIs"
>   input of {% icon tool %} **13: OMERO ROI Import**.
>   - Connect the output of {% icon param-file %} **11: Parse parameter value** to the "Image ID where annotate the ROIs"
>   input of {% icon tool %} **13: OMERO ROI Import**.
>3. Save your workflow and name it `OMERO_basic_upload`.
{: .hands_on}


You are done! The final workflow should  look like this:
![img.png](../../images/omero-suite/workflow.png)

And you can easily add all parameters during the workflow invocation:
![img_1.png](../../images/omero-suite/workflow_invocation.png)

This workflow can be easily re-used or turned into a sub-workflow for an image processing
pipeline.

# Conclusion

In this exercise, you imported images into OMERO using Galaxy. You also learn how to
import metadata and ROIs, as well as get information such as image ID and annotation file
from a target OMERO instance.


# References

- Allan, C., Burel, JM., Moore, J. et al. OMERO: flexible, model-driven data management for experimental biology. 
Nat Methods 9, 245–253 (2012). https://doi.org/10.1038/nmeth.1896
- Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for Scientific Data Management and
stewardship. Sci Data 3, 160018 (2016). https://doi.org/10.1038/sdata.2016.18


topic: imaging
tutorial: dicom-anatomical-3d
title: Segmentation of Anatomical Structures in Medical 3-D Images
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
content_hash: 3cf20ff88d1cfc2d9d771a9ee463781e
last_modified: 2026-02-26T21:12:16.725174
zenodo_link: https://zenodo.org/records/18225140
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/dicom-anatomical-3d/tutorial.html

content:
Image data in medical imaging is often stored and exchanged in the DICOM file format. A DICOM dataset is a file that contains rich metadata (patient, study info) and the actual medical image data stored as pixels (for 2-D images) or voxel (for 3-D images). The image data can be single-channel or multi-channel, and it can also be organized in multiple frames (e.g., spatial tiles of a mosaic, z-slices of a 3-D image, or time steps).

Even though, technically, DICOM datasets can contain heterogeneous frames for different axes (e.g., z-slices and temporal frames), in practice, the frames of a DICOM dataset are usually designated for only one specific purpose and axis (e.g., single z-slices or temporal frames). For such cases, DICOM series are more widely adopted: A list of DICOM datasets can form a DICOM series to store multi-dimensional data (e.g., a list of 3-D images for consecutive time steps, or just the z-slices of a 3-D image, where each slice is a single DICOM dataset).

In this tutorial, we will show how DICOM series and DICOM datasets can be converted to TIFF, which is a general-purpose image file format that is well-supported by a majority of tools in Galaxy. We will use that to showcase segmentation and visualization of anatomical structures in 3-D image data from computed tomography (CT). The Galaxy Image Analysis tools and techniques utilized in this tutorial are domain-agnostic and can also be adapted to other imaging modalities (e.g., 3-D cell imaging).

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Getting the Image Data

In this tutorial, we will be using a publicly available CT dataset of a human torso, scanned between the thorax and the pelvis.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. If you are logged in, create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following dataset from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor).
>
>    ```
>    {{ page.zenodo_link }}/files/PCIR_98890234_20010101_7.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `PCIR_98890234_20010101_7.zip`
>    - *"What to extract"*: `All files`
>
> 4. Rename {% icon galaxy-pencil %} the dataset collection to `DICOM series`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Change the datatype of {% icon galaxy-pencil %} `DICOM series` to `Auto-detect`
>    
>    <!-- For some reason, there is no "Auto-detect" option when using the non-bulk version of the "Change data type" function of the collection -->
>    {% snippet faqs/galaxy/bulk_change_datatype.md datasets_description="the `DICOM series` dataset collection"  n="1" datatype="Auto-detect" %}
{: .hands_on}

# Pre-processing

In this DICOM series, each DICOM dataset contains the z-slice of the 3-D image. The metadata of each DICOM dataset in the series contains the z-position of the slice, so the order of the individual DICOM datasets in the collection does not matter (note that for other images, in general, this information might be missing in the metadata, in which case the order of the datasets might be crucial).

Our first step will be to convert the DICOM series into a single TIFF image, that represents the whole 3-D dataset.

> <hands-on-title>Convert DICOM series to TIFF image</hands-on-title>
>
> 1. {% tool [Convert DICOM to TIFF](toolshed.g2.bx.psu.edu/repos/imgteam/highdicom_dicom2tiff/highdicom_dicom2tiff/0.27.0+galaxy0) %} with the following parameters to convert the DICOM series (list of DICOM datasets) into a list of TIFF images:
>    - {% icon param-files %} *"DICOM dataset"*: `DICOM series` collection
> 2. {% tool [Concatenate images or channels](toolshed.g2.bx.psu.edu/repos/imgteam/concat_channels/ip_concat_channels/0.5) %} with the following parameters to concatenate the stack of TIFF images along the z-axis into a single 3-D image:
>    - {% icon param-files %} *"Images to concatenate"*: Output of {% tool [Convert DICOM to TIFF](toolshed.g2.bx.psu.edu/repos/imgteam/highdicom_dicom2tiff/highdicom_dicom2tiff/0.27.0+galaxy0) %}
>    - *"Concatenation axis"*: `Z-axis (concatenate images as slices of a 3-D image or image sequence)`
>    - *"Scaling of values"*: `Preserve range of values`
>    - *"Sort images before concatenating"*: `Sort images by their position along the Z-axis`
>
>    > <comment-title>Why do we convert DICOM to TIFF?</comment-title>
>    >
>    > Although DICOM is *the* standard file format in medical imaging, in research, it is convenient to use to TIFF for image analysis for several reasons:
>    > 
>    > **Advantages of TIFF for image analysis:**
>    > - **Tool compatibility:** Most image analysis and visualization tools in Galaxy and other scientific computing platforms are optimized for standard, general-purpose image formats like TIFF. In addition, TIFF is also well-supported by programming languages (e.g., Python, MATLAB), which can be handy for advanced analyses.
>    > - **Simpler structure:** TIFF has more straightforward data organization for computational processing. A single multi-dimensional TIFF file is easier to handle than hundreds or thousands of separate DICOM datasets.
>    > - **Metadata handling:** The extensive clinical metadata of DICOM (e.g., patient info, acquisition protocols) can be challenging to parse and process for general-purpose image analysis tools.
>    > 
>    > **Important considerations:**
>    > - **Metadata preservation is critical:** During conversion, spatial calibration data (voxel spacing, slice positions, orientation) must be preserved. Without these informations, it is not possible to translate spatial image analysis readouts (e.g., object sizes) into physical units (e.g., mm, cm), rendering quantitative analysis meaningless.
>    > - **DICOM advantages:** DICOM remains essential for clinical workflows, PACS integration, and regulatory compliance. For pure clinical use, stay in DICOM format.
>    {: .comment}
{: .hands_on}

Some of the tools that we will be using in our analysis require that the voxel size of the image is isotropic. To ensure that this is the case, we will re-sample the image data to an isotropic voxel size:

> <hands-on-title>Re-sample image data for isotropic pixel/voxel sizes</hands-on-title>
>
> 1. {% tool [Scale image](toolshed.g2.bx.psu.edu/repos/imgteam/scale_image/ip_scale_image/0.25.2+galaxy0) %} with the following parameters to re-sample the image data to an isotropic voxel size:
>    - {% icon param-file %} *"Input image"*: Output of {% tool [Concatenate images or channels](toolshed.g2.bx.psu.edu/repos/imgteam/concat_channels/ip_concat_channels/0.5) %}
>    - *"How to scale?"*: `Scale to spatially isotropic pixels/voxels`
>    - *"Method"*: `Down-sample (might lose information)`
{: .hands_on}

If the image data already would have been isotropic, this tool would yield the original data.

CT image data, such as the dataset used in this tutorial, typically has intensity values that correspond to the Hounsfield scale:

> The *Hounsfield* scale, named after Sir Godfrey Hounsfield, is a quantitative scale for describing radiodensity. It is frequently used in CT scans, where its value is also termed *CT number*.
{: .quote cite="https://en.wikipedia.org/w/index.php?title=Hounsfield_scale&oldid=1313893505" author="Wikipedia contributors"}

Using Hounsfield Units (HU) is advantageous, because it allows to directly identify specific materials or tissues solely based on the image intensities. Typical Hounsfield unit (HU) values are −1000 HU for air, approximately −950 to −650 HU for normally aerated lung tissue, −190 to −30 HU for subcutaneous fat, 0 HU for water, +30 to +60 HU for muscle tissue, and >+150 HU for bone tissue (often >+1000 HU for cortical bone; e.g., {% cite chougule2018clinical %}, {% cite LimFat2011 %}). In research, often larger ranges are used to account for pathological cases or contrast enhancement (e.g., –29 to +150 HU for muscle tissue).

The image data may also contain values outside of the range of –1000 to 3071 HU that correspond to, for example, parts of the CT imaging setup, or imaging artifacts. To effectively “erase” those parts from the image data, we will clip the image intensities to the meaningful range of –1000 to 3071 HU as a final step of pre-processing:

> <hands-on-title>Clipping the image intensities</hands-on-title>
>
> 1. {% tool [Clip image intensities](toolshed.g2.bx.psu.edu/repos/imgteam/clip_image/clip_image/0.7.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input image"*: Output of {% tool [Scale image](toolshed.g2.bx.psu.edu/repos/imgteam/scale_image/ip_scale_image/0.25.2+galaxy0) %}
>    - *"Lower bound"*: `-1000`
>    - *"Upper bound"*: `3071`
{: .hands_on}

# Exploring the Image Data

Galaxy's built-in TIFF viewer can be used to roughly explore the image data. To do so, click on the {% icon galaxy-eye %} (eye) icon next to the output of the {% tool [Clip image intensities](toolshed.g2.bx.psu.edu/repos/imgteam/clip_image/clip_image/0.7.3+galaxy0) %} tool. This will bring up a basic user interface that shows a z-slice of the 3-D image, along with the option to navigate to a different slice along the *z-axis* (slices are presented as "pages"). The viewer shows the *x-axis* as *left-to-right* and the *y-axis* as *top-to-bottom*.

![Galaxy's built-in TIFF viewer](../../images/dicom-anatomical-3d/tiff_viewer.png "Galaxy's built-in TIFF viewer.")

The buttons in the toolbar can be used to zoom and pan the view, or this can be accomplished by using the scroll wheel or dragging the image with a pressed mouse button. Even though this viewer is very simple, we can make two important observations that provide some rough orientation within the image data that we are dealing with:

1. The *y-axis* is the **anteroposterior axis** (or *anterior-posterior*, from the front to the back of the torso).
2. The *z-axis* is the **craniocaudal axis** (or *cranial-caudal*, from the bottom to the top of the torso).

![Anatomical axes](../../images/dicom-anatomical-3d/axes.jpg "Anatomical axes. <a href='https://commons.wikimedia.org/wiki/File:Anatomical_axes.jpg'>Edoarado and Mikael H&auml;ggstr&ouml;m</a>, <a href='https://creativecommons.org/licenses/by-sa/3.0'>CC BY-SA 3.0</a>, via Wikimedia Commons.")

We will now use this information to get a visually more informative representation of the image data by creating a 3-D volume rendering (geometrical projection of the 3-D image data onto the 2-D screen plane).

Inspecting 3-D data as a 2-D projection is intrinsically challenging because it requires transformations of the data that inherently reduce the information along the way—but we as humans, who are also subject to this limitation due to our two-dimensional vision, we have learned to deal with it naturally *by taking looks from different angles*. The tool that we are going to use for visualization mimics this intuition by generating *videos* instead of plain images, that use different angles for the projection:

> <hands-on-title>Visual inspection via pre-rendered video</hands-on-title>
>
> 1. {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input image (3-D)"*: Output of {% tool [Clip image intensities](toolshed.g2.bx.psu.edu/repos/imgteam/clip_image/clip_image/0.7.3+galaxy0) %}
>    - *"Unit of the intensity values"*: `Hounsfield`
>    - *"Rendering mode"*: `Maximum Intensity Projection (MIP)`
>    - *"Color map"*: `rainbow`
>    - *"Camera parameters"*:
>      - *"Distance"*: `350`
>    - *"Video parameters"*:
>      - *"Frames"*: `400`
>
>    > <comment-title>How do we establish proper orientation of the image data in 3-D?</comment-title>
>    >
>    > The parameter for the *"Coordinate system"* is set to `Point Z to the top` by default. This is the setting that we must use here, since we have identified the z-axis as the *craniocaudal* axis. For other image data it might be necessary to instead `Point Y to the top`.
>    {: .comment}
{: .hands_on}

Click on the {% icon galaxy-eye %} (eye) icon next to the output of the {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} tool to inspect the obtained visualization:

<!--
    Command to compress screen recordings to reasonable file size:
    ffmpeg -ss 0.5 -t 16 -i input.mov -vf "fps=15,scale=-1:512:flags=lanczos" -c:v libx264 -preset slow -crf 26 -movflags +faststart -pix_fmt yuv420p -an output.mp4

    An alternative is to use webp, but this yields larger files with worse quality:
    ffmpeg -ss 0.5 -t 16 -i input.mov -vf "fps=12,scale=-1:400:flags=lanczos" -loop 0 -c:v libwebp_anim -quality 40 -compression_level 5 -speed 2 -an output.webp
-->
<video loop="true" autoplay="autoplay" muted width="100%">
    <source src="../../images/dicom-anatomical-3d/mip_rainbow.mp4" type="video/mp4"/>
</video>

The *Maximum Intensity Projection (MIP)* is useful to get an idea of the spatial distribution of the image intensities within the image. At each time step of the video, the MIP is computed by casting a ray from each image pixel through the 3-D volume of image intensities, and taking the maximum intensity value along the line by sampling the image intensities at fixed steps (the rendering technique is thus called *ray marching*).

# Skeletal Segmentation

The imaged section of the torso in our dataset contains parts of prominent skeletal structures such as the pelvis, the spine, and the thorax. Segmentation of these structures is easy due to the distinct ranges of the Hounsfield scale:

> <hands-on-title>Skeletal segmentation in a CT image</hands-on-title>
>
> 1. {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.25.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input image"*: Output of {% tool [Clip image intensities](toolshed.g2.bx.psu.edu/repos/imgteam/clip_image/clip_image/0.7.3+galaxy0) %}
>    - *"Thresholding method"*: `Manual`
>    - *"Threshold value"*: `120`
>
>    > <question-title></question-title>
>    >
>    > Why do we use a threshold value of just 120, if the typical range of HU values for bone tissue starts at about 150 HU?
>    >
>    > > <solution-title></solution-title>
>    > > Some bones in the imaged data are quite thin (e.g., the ends of the ribs). This leads to *partial volume effects*: voxels at the edges of thin structures contain a mixture of bone and surrounding soft tissue (e.g., cartilage), resulting in intermediate intensity values lower than pure bone. When we re-sampled the image data to an isotropic voxel size, we chose to *down-sample* the image data. A consequence of this is that voxel intensity values are locally averaged, which can lead to reduced intensity values of thin structures.
>    > {: .solution }
>    {: .question}
{: .hands_on}

Lets inspect the segmentation result using a 3-D overlay of the input image and the segmentation mask:

> <hands-on-title>Visualization of 3-D segmentation results</hands-on-title>
>
> 1. {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input image (3-D)"*: Output of {% tool [Clip image intensities](toolshed.g2.bx.psu.edu/repos/imgteam/clip_image/clip_image/0.7.3+galaxy0) %}
>    - *"Unit of the intensity values"*: `Hounsfield`
>    - *"Rendering mode"*: `Direct Volume Rendering (DVR)`
>    - *"Color map"*: `BrBG`
>      - *"Ramp function"*: `Enabled`
>        - *"Ramp start"*:
>          - *"Type of the intensity value"*: `Absolute intensity value`
>          - *"Intensity value"*: `100`
>        - *"Ramp end"*:
>          - *"Type of the intensity value"*: `Absolute intensity value`
>          - *"Intensity value"*: `150`
>    - *"Camera parameters"*:
>      - *"Distance"*: `350`
>    - *"Render mask overlay"*: `Render mask overlay`
>      - *"Mask overlay (3-D)"*: Output of {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.25.2+galaxy0) %}
>    - *"Video parameters"*:
>      - *"Frames"*: `400`
{: .hands_on}

The *Direct Volume Rendering (DVR)* is rendering technique similar to MIP, but instead of taking the maximum intensity for each image pixel it simulates the absorption of light and thus produces realistically looking surface renderings. Click on the {% icon galaxy-eye %} (eye) icon next to the output of the {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} tool to inspect the obtained visualization:

<video loop="true" autoplay="autoplay" muted width="100%">
    <source src="../../images/dicom-anatomical-3d/segm_skeletal1.mp4" type="video/mp4"/>
</video>

It can be seen from the visualization that objects corresponding to parts of the imaging setup (behind the spine) also fall into the selected HU range, and is thus falsely included in the segmentation skeletal segmentation result (leakage). To improve the segmentation result, we will thus remove all objects from the segmentation that are behind the spine and pelvis. To identify the spine and pelvis in the segmentation result, we exploit that they correspond to the largest connected component in our segmentation.

The first step is to determine the sizes of the connected components:

> <hands-on-title>Feature extraction for the connected components of the segmentation result</hands-on-title>
>
> 1. {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.7.3+galaxy0) %} with the following parameters to assign a unique label to each connected component of the segmentation result:
>    - *"Mode"*: `Connected component analysis`
>    - {% icon param-file %} *"Binary image"*: Output of {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.25.2+galaxy0)  %}
>
> 2. {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.25.2+galaxy1) %} with the following parameters to measure the size and location of each connected component:
>    - {% icon param-file %} *"Label map"*: Output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.7.3+galaxy0)  %}
>    - *"Available features"*:
>      - `Label from the label map`
>      - `Area`
>      - `Centroid`
{: .hands_on}

Next, we inspect the tabular output yielded by the {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.25.2+galaxy1) %} tool:

> |label|area   |centroid_x       |centroid_y        |centroid_z       |
> |:----|:------|:----------------|:-----------------|:----------------|
> |1    |61710.0|88.83420839410144|103.37395883973424|36.61537838275806|
> |2    |10465.0|97.36292403248925|139.6089823220258 |52.41691352126135|
> |3    |1.0    |87.0             |137.0             |0.0              |
> |4    |2.0    |44.0             |159.0             |0.5              |
> |5    |7.0    |132.0            |159.0             |3.0              |
> |6    |6.0    |48.5             |160.0             |1.0              |
> |…    |…      |…                |…                 |…                |
{: .matrix}

In this table, each row corresponds to a connected component in the segmentation result (identified by its unique label). The centroid of the connected components tells us where the components are located (the y-axis is the *anteroposterior axis* and points from the front to the back of the torso). By inspecting this table, we can easily conclude that the centroid of the largest connected component is located at a y-coordinate of 103.37 (in voxels).

Since this coordinate corresponds to the centroid of the spine and pelvis, removing all objects with a y-coordinate of more than 103.37 voxels is likely to also remove some ribs—which we do not want to happen. Hence, we will add a tolerance margin for objects: Instead of strictly removing all objects that are behind the centroid of the spine and pelvis, we will only remove those that are 1.5 cm or further behind.

The size of the margin needs to be given in voxels. To determine that, we inspect the standard output of the {% tool [Scale image](toolshed.g2.bx.psu.edu/repos/imgteam/scale_image/ip_scale_image/0.25.2+galaxy0) %} tool, that contains many details:

> <hands-on-title>Inspect the standard output of the "Scale image" tool</hands-on-title>
>
> 1. Expand the history item for the output of the {% tool [Scale image](toolshed.g2.bx.psu.edu/repos/imgteam/scale_image/ip_scale_image/0.25.2+galaxy0) %} tool.
>
> 2. Click on the {% icon details %} icon.
>
> 3. Scroll down to the **Job Information** section to view the "Tool Standard Output" log:
>
>  > <code-out-title>Standard output of the "Scale image" tool</code-out-title>
>  > ```
>  > Input axes: ZYX
>  > Input resolution: (1.137778101526753, 1.137778101526753), unit: mm, z_position: -248.60999337383177, z_spacing: 2.4999998490566036
>  > scale: (1, 1, 1.0, 0.35156242119108805, 0.35156242119108805)
>  > order: 1
>  > preserve_range: True
>  > channel_axis: 5
>  > anti_aliasing: True
>  > anti_aliasing_sigma: (0, 0, 0, 0.9222225410383957, 0.9222225410383957, 0)
>  > Output shape: (107, 180, 180)
>  > Output axes: ZYX
>  > Output resolution: (0.4000000241509449, 0.4000000241509449), unit: mm, z_position: -248.60999337383177, z_spacing: 2.4999998490566036
>  > ```
>  {: .code-out}
{: .hands_on}

What we are interested in here is the line for the `Output resolution`. The first tuple `(0.4000000241509449, 0.4000000241509449)` corresponds to the number of voxels per millimeter along the x- and y-axes. From this, we can deduce, using basic algebra, that one voxel corresponds to 2.5 mm along the y-axis (in fact, we can also read off the value for the `z_spacing`, which is identical due to the isotropic re-sampling). Thus, a margin of 1.5 cm corresponds to 6 voxels.

With this information, we can now write a *rules* file for removing the leakage from the segmentation result:

> <hands-on-title>Write a rules file to remove spurious objects from the segmentation</hands-on-title>
>
> 1. Create a new file via the {% icon galaxy-upload %} **Upload Data** tool in Galaxy, use the name `rules_skeletal`, paste the following content, and set the format to tabular:
>    ```
>    feature   	min	max
>    centroid_y	0	109.37
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" name="rules_skeletal" %}
{: .hands_on}

The upper bound of 109.37 for the y-coordinate of the centroids of the objects that we will retain in the segmentation is obtained by adding the margin of 6 voxels to the previously determined y-coordinate of the centroid of the spine and pelvis.

Now we are all set to remove the leakage from the segmentation result:

> <hands-on-title>Remove spurious objects from the segmentation</hands-on-title>
>
> 1. {% tool [Filter label map by rules](toolshed.g2.bx.psu.edu/repos/imgteam/2d_filter_segmentation_by_features/ip_2d_filter_segmentation_by_features/0.7.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Label map"*: Output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.7.3+galaxy0) %}
>    - {% icon param-file %} *"Features"*: Output of {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.25.2+galaxy1) %}
>    - {% icon param-file %} *"Rules"*: `rules_skeletal`
>
> 2. {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input image (3-D)"*: Output of {% tool [Clip image intensities](toolshed.g2.bx.psu.edu/repos/imgteam/clip_image/clip_image/0.7.3+galaxy0) %}
>    - *"Unit of the intensity values"*: `Hounsfield`
>    - *"Rendering mode"*: `Direct Volume Rendering (DVR)`
>    - *"Color map"*: `BrBG`
>      - *"Ramp function"*: `Enabled`
>        - *"Ramp start"*:
>          - *"Type of the intensity value"*: `Absolute intensity value`
>          - *"Intensity value"*: `100`
>        - *"Ramp end"*:
>          - *"Type of the intensity value"*: `Absolute intensity value`
>          - *"Intensity value"*: `150`
>    - *"Camera parameters"*:
>      - *"Distance"*: `350`
>    - *"Render mask overlay"*: `Render mask overlay`
>      - *"Mask overlay (3-D)"*: Output of {% tool [Filter label map by rules](toolshed.g2.bx.psu.edu/repos/imgteam/2d_filter_segmentation_by_features/ip_2d_filter_segmentation_by_features/0.7.3+galaxy1) %}
>    - *"Video parameters"*:
>      - *"Frames"*: `400`
{: .hands_on}

Click on the {% icon galaxy-eye %} (eye) icon next to the output of the {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} tool to visually inspect the improved segmentation result:

<video loop="true" autoplay="autoplay" muted width="100%">
    <source src="../../images/dicom-anatomical-3d/segm_skeletal2.mp4" type="video/mp4"/>
</video>

> <question-title>Further improvements of the segmentation result</question-title>
>
> The visualization above shows that there are still small, spurious objects included in the segmentation. How to remove these?
>
> > <solution-title></solution-title>
> > Add another rule to the `rules_skeletal` file that you have created, then re-run the {% tool [Filter label map by rules](toolshed.g2.bx.psu.edu/repos/imgteam/2d_filter_segmentation_by_features/ip_2d_filter_segmentation_by_features/0.7.3+galaxy1) %} tool. That rule should be for the `area` feature and impose a reasonable `min` value for the minimum size of retained connected components. The value for `max` could be `100000`, for example.
> {: .solution }
{: .question}

# Vessel Segmentation

As another example of the segmentation of anatomical structures, we will now perform a segmentation of the aortic bifurcation, a key vascular landmark where the abdominal aorta divides into the left and right common iliac arteries. Vessel segmentation is very different from skeletal segmentation, because vessels do not exhibit a specific HU intensity value and generally have a very low contrast compared to the surrounding tissue. Moreover, they often are very thin, elongated structures, with and without branches, which poses additional challenges compared to segmentation of large connected image regions.

In CT imaging, vessels usually appear as thin lines of subtly higher intensity than the surrounding tissue (just like the ridges of mountains on a relief map). Image filters that enhance such structures are thus called *ridge filters*. The most prominent ridge filter for vessel enhancement in 3-D images is the Frangi filter ({% cite Frangi1998 %}).
> <comment-title>Contrast-enhanced vs. non-contrast vessel imaging</comment-title>
>
> Vessel segmentation feasibility is highly dependent on the imaging protocol:
>
> **Non-contrast CT:**
> - Blood vessels: ~30–50 HU (similar to muscle: 40–60 HU)
> - Can also be much less due to *partial volume effects*
> - Minimal contrast with surrounding tissue
> - Less suited for detailed vascular examination (e.g., segmentation of whole vessel trees)
> - Frangi filter unreliable for very thin vessels (e.g., coronary vessels)
>
> **Contrast-enhanced CT (CTA):**
> - Contrast-filled vessels: 250–400 HU (with iodinated contrast agent)
> - Excellent contrast against surrounding tissue (40–60 HU)
> - Frangi filter performs reliably for vessel detection
> - Clinical standard for vascular imaging and segmentation
>
> The image data used in this tutorial is *non-contrast* CT.
{: .comment}
Our workflow for the segmentation of the aortic bifurcation is as follows:

- First, we will apply a *Frangi filter* to perform image enhancement for vessels and other vessel-like structures. The responses of Frangi filters range between 0 and 1, which directly indicates the "vesselness" of an image point.
- Second, we will perform *hysteresis thresholding* to segment connected parts of the vascular tree with a particularly high filter response (i.e. a particularly high vesselness).
- Third, we will apply *feature extraction* and *label map filtering* to select the largest connected component of the segmented vascular tree (like we did for skeletal segmentation).

The individual workflow steps are as follows:

> <hands-on-title>Vessel segmentation and feature extraction</hands-on-title>
>
> 1. {% tool [Apply ridge filter](toolshed.g2.bx.psu.edu/repos/imgteam/ridge_filter/ridge_filter_skimage/0.22.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input image"*: Output of {% tool [Clip image intensities](toolshed.g2.bx.psu.edu/repos/imgteam/clip_image/clip_image/0.7.3+galaxy0) %}
>    - *"Filter"*: `Frangi vesselness filter`
>    - *"Mode of operation"*: `Enhance bright ridges (high image intensities)`
>    - *"Minimum sigma"*: `0.5`
>    - *"Maximum sigma"*: `1.5`
>    - *"Number of sigma steps for multi-scale analysis"*: `3`
>    - *"Alpha"*: `0.5`
>    - *"Beta"*: `0.5`
>    - *"Gamma"*: `5.0`
>
> 2. {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.25.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input image"*: Output of {% tool [Apply ridge filter](toolshed.g2.bx.psu.edu/repos/imgteam/ridge_filter/ridge_filter_skimage/0.22.0+galaxy2) %}
>    - *"Thresholding method"*: `Manual`
>    - *"Threshold value"*: `0.25`
>    - *"Second threshold value for hysteresis thresholding"*: `0.3`
>
> 3. {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.7.3+galaxy0) %} with the following parameters:
>    - *"Mode"*: `Connected component analysis`
>    - {% icon param-file %} *"Binary image"*: Output of {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.25.2+galaxy0)  %}
>
> 4. {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.25.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Label map"*: Output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.7.3+galaxy0)  %}
>    - *"Available features"*:
>      - `Label from the label map`
>      - `Area`
>
> 5. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: Output of {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.25.2+galaxy1) %}
>    - *"Number of header lines"*: `1`
>    - *"Column selections"*: `1: Column selections`
>      - *"on column"*: `Column 2`
>      - *"in"*: `Descending order`
>      - *"Flavor"*: `Fast numeric sort (-n)`
{: .hands_on}

We inspect the tabular output yielded by the {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} tool:

> |label|area   |
> |:----|:------|
> |26   |11167.0|
> |132  |8739.0 |
> |176  |2910.0 |
> |175  |2821.0 |
> |42   |2157.0 |
> |…    |…      |
{: .matrix}

From this table, we can deduce that the largest connected component is the one with label 26. We now create the corresponding *rules* file for removing all other components from the segmentation:

> <hands-on-title>Segmentation and visualization of the aortic bifurcation</hands-on-title>
>
> 1. Create a new file via the {% icon galaxy-upload %} **Upload Data** tool in Galaxy, use the name `rules_aorticbif`, paste the following content, and set the format to tabular:
>    ```
>    feature	min	max
>    label  	26	26
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" name="rules_aorticbif" %}
>
> 2. {% tool [Filter label map by rules](toolshed.g2.bx.psu.edu/repos/imgteam/2d_filter_segmentation_by_features/ip_2d_filter_segmentation_by_features/0.7.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Label map"*: Output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.7.3+galaxy0) %}
>    - {% icon param-file %} *"Features"*: Output of {% tool [Extract image features](toolshed.g2.bx.psu.edu/repos/imgteam/2d_feature_extraction/ip_2d_feature_extraction/0.25.2+galaxy1) %}
>    - {% icon param-file %} *"Rules"*: `rules_aorticbif`
>
> 3. Create a new file via the {% icon galaxy-upload %} **Upload Data** tool in Galaxy, use the name `colormap_white`, paste the following content, and set the format to tabular:
>    ```
>    color	intensity	type
>    #ffffff00	0	relative
>    #ffffffff	1 	relative
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" name="colormap_white" %}
>
> 4. {% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input image (3-D)"*: Output of {% tool [Filter label map by rules](toolshed.g2.bx.psu.edu/repos/imgteam/2d_filter_segmentation_by_features/ip_2d_filter_segmentation_by_features/0.7.3+galaxy1) %}
>    - *"Unit of the intensity values"*: `No unit`
>    - *"Rendering mode"*: `Direct Volume Rendering (DVR)`
>    - *"Color map"*: `Custom`
>    - {% icon param-file %} *"Custom color map"*: `colormap_white`
>    - *"Add a color bar"*: `No`
>    - *"Camera parameters"*:
>      - *"Distance"*: `250`
>    - *"Render mask overlay"*: `No overlay`
>    - *"Video parameters"*:
>      - *"Frames"*: `400`
{: .hands_on}

The obtained visualization shows the segmentation of the aortic bifurcation:

<video loop="true" autoplay="autoplay" muted width="100%">
    <source src="../../images/dicom-anatomical-3d/segm_aorticbif.mp4" type="video/mp4"/>
</video>

Vessel segmentation is generally challenging and still an active field of research. Segmentation of whole vascular trees may require more sophisticated methods (e.g., {% cite Glsn2008 %}, {% cite Biesdorf2015 %}).

# Fully automated workflow

This tutorial contained several interactive steps that required inspection of data from previous steps. Note that this steps can also be automated, as we show in the supplemental workflow:

{% snippet faqs/galaxy/workflows_run_trs.md path="topics/imaging/tutorials/dicom-anatomical-3d/workflows/dicom-anatomical-3d.ga" title="Segmentation of Anatomical Structures in Medical Images" %}

tags_json:
["Image segmentation", "Conversion", "Overlay", "Object feature extraction", "Medical imaging", "Computed tomography", "3D image", "Volume rendering"]


topic: imaging
tutorial: 2D-spot-detection
difficulty: introductory
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["IDR dataset", "Spot detection", "Rasterisation", "Overlay"]
content_hash: c98fbda4c3b5c0eef7fa5e1bd0f3d521
last_modified: 2026-02-26T21:12:16.702375
workflows_json: []
title:
Quantification of single-molecule RNA fluorescence in situ hybridization (smFISH) in yeast cell lines

url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/2D-spot-detection/tutorial.html

content:
The objective is to detect multiple bright spots in an image using a basic computer vision and image processing 
technique for identification and localization of regions 
of high intensity within an image. 
These spots often correspond to features of interest, such as fluorescent markers 
in biological imaging.
Such an approach can be beneficial in 
single-cell/single-molecule imaging experiments, such as RNA single-molecule
fluorescence in situ hybridization (smFISH), as these experiments can resolve the 
spatial and temporal distribution of individual RNA molecules with high resolution. 


In this tutorial, we will try to identify RNA molecules in yeast cell lines. However,
such an approach can be re-used for the identification of any fluorescence spots in biological 
images!

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data from IDR

The testing dataset for this tutorial can be obtained from the [Image Data Repository (IDR)](https://idr.openmicroscopy.org/). 
This dataset was published by [Li et al. (2019) in Scientific Data](https://www.nature.com/articles/s41597-019-0106-6).

In this publication, the authors used  smFISH to quantify the kinetic expression of STL1 and CTT1 mRNAs in single Saccharomyces cerevisiae
cells upon NaCl osmotic stress.

Let's fetch this **FAIR** dataset using a Galaxy Tool! We will download the channel 5-TMR 
and selected the 10th z-stack for analysis.

> <hands-on-title>Download images from the IDR</hands-on-title>
>
> 1. If you are logged in, create a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
> 2. {% tool [IDR Download](toolshed.g2.bx.psu.edu/repos/iuc/idr_download_by_ids/idr_download_by_ids/0.45) %} with the following parameters:
>    - *"How would you like to specify the IDs of images to download?"*: `As text (comma-separated list of IDs or a valid IDR link)`
>    -  *"Image IDs to download"*: `4496770`
>    -  *"Which Images do you want to download": `Exported TIFF (single channel, single stack)`
>    -  *"Name of the channel to download"*: `5-TMR`
>    -  *"z-plane of images to download"*: `10`
>    -  *"Image frame to download"*: `0`
>    -  *"Limit the download to a selected region of the image?"*: `No, download the entire image plane`
>    -  *"Skip failed retrievals?"*: `No`
>    -  *"Download images in a tarball?"*: `No`
> 3. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-collection %} *"Input List"*: `Download IDR/OMERO` (output of IDR Download tool)
>    - *"How should a dataset be selected?"*: `The first dataset`
> 4. Rename {% icon galaxy-pencil %} the generated file to `RNA_input.tiff`.
{: .hands_on}

# Improve image contrast 

When inspecting the image visually, we get the impression that the image is entirely black:

![2025_pre_clahe.png](../../images/2d_spot_detection/2025_pre_clahe.png){: width="75%"}

It's not the best quality for a first visual inspection!

As described in the [Galaxy Imaging Introduction Tutorial]({% link topics/imaging/tutorials/imaging-introduction/tutorial.md %}), the original image is 16-bit and the intensity values are spread
over a very small fraction of the range of intensity values that can be represented using 16 bits.

Therefore, for improved visibility the intensity histogram of the image can be normalized first.

We will normalize the histogram to improve the contrast. 
We do this using a [Contrast Limited Adaptive Histogram Equalization (CLAHE)](https://en.wikipedia.org/wiki/Adaptive_histogram_equalization) approach.

> <hands-on-title>Normalize Histogram</hands-on-title>
>
> 1. {% tool [Perform histogram equalization](toolshed.g2.bx.psu.edu/repos/imgteam/2d_histogram_equalization/ip_histogram_equalization/0.18.1+galaxy0) %} with the following parameters to normalize the histogram of the image:
>    - {% icon param-file %} *"Input image"*: `RNA_input.tiff` file
>    - *"Histogram equalization algorithm"*: `CLAHE`
> 2. Rename {% icon galaxy-pencil %} the generated file to `RNA_input_normalized.tiff`.
> 3. Click on the **visualise icon** {% icon galaxy-visualise %} of the file to visually inspect the image using the **Tiff Viewer** visualization plugin.
{: .hands_on}

Your image should now look a bit brighter:

![2025_post_clahe.png](../../images/2d_spot_detection/2025_post_clahe.png){: width="75%"}

We can now clearly make out the presence of many RNA spots!

# RNA spots detection

We can now try to detect blobs and measure their intensity... Let's run a specific tool for this!

> <hands-on-title>Perform 2-D spot detection</hands-on-title>
>
> 1. {% tool [Perform 2-D spot detection ](toolshed.g2.bx.psu.edu/repos/imgteam/spot_detection_2d/ip_spot_detection_2d/0.1+galaxy0) %} with the following parameters to detect the 2D spots:
>    - {% icon param-file %} *"Intensity image or a stack of images"*: `RNA_input_normalized.tiff` file
>    - *"Starting time point (1 for the first frame of the stack) "*: `1`
>    - *"Ending time point (0 for the last frame of the stack)"*: `0`
>    - *"Detection filter"*: `Laplacian of Gaussian`
>    - *"Minimum scale"*: `1`
>    - *"Maximum scale"*: `2`
>    - *"Minimum filter response (absolute)"*: `0.25`
>    - *"Minimum filter response (relative)"*: `0`
>    - *"Image boundary"*: `10`
> 2. Rename {% icon galaxy-pencil %} the generated file to `RNA_spot_detected`.
{: .hands_on}

The tool produces a TSV file containing all detections, with the following columns:

- **frame**: The frame of the image stack
- **pos_x**: The horizontal coordinate of the detection
- **pos_y**: The vertical coordinate of the detection
- **scale**: The scale at which the detection was found
- **radius**: The radius of the detected spot
- **intensity**: The mean intensity of the spot

The TSV output will have 1649 lines, meaning 1649 spots detected!

![TSV_output.png](../../images/2d_spot_detection/TSV_output.png)

# Convert spots coordinates to labels
Let's try now to visualize our results! The first step is to create a label map from the TSV file...
This can be nicely done by converting the point coordinates to a label map and we have a tool for it!

Please notice that the width and height of the image are necessary inputs to corretly overlay
the label map in the next step! 
You can get these info by running the "Show image info" tool on the original image!


> <hands-on-title>Create the label map from the points coordinates</hands-on-title>
>
> 1. {% tool [Convert point coordinates to label map](toolshed.g2.bx.psu.edu/repos/imgteam/points2labelimage/ip_points_to_label/0.4+galaxy0) %} with the following parameters to detect the 2D spots:
>    - {% icon param-file %} *"Tabular list of points"*: `RNA_spot_detected.tsv` file
>    - *"Width of output image"*: `2048`
>    - *"Height of output image "*: `2048`
>    - *"Tabular list of points has header"*: `Yes`
>    - *"Swap X and Y coordinates"*: `No`
>    - *"Produce binary image"*: `No`
> 2. Rename {% icon galaxy-pencil %} the generated file to `RNA_labels.tiff`.
> 3. Click on the **visualise icon** {% icon galaxy-visualise %} of the file to visually inspect the image using the **Tiff Viewer** visualization plugin.
{: .hands_on}

A label image will be created... If you want to visualize the labels, you can run again the normalize histogram
step:

![2025_visual_labels.png](../../images/2d_spot_detection/2025_visual_labels.png){: width="75%"}

but this is not necessary for the next and final step: overlay the label to the original image!

# Overlay labels to the original image

Results can be overlayed with the original image.

> <hands-on-title>Segment image</hands-on-title>
>
> 1. {% tool [Overlay images](toolshed.g2.bx.psu.edu/repos/imgteam/overlay_images/ip_overlay_images/0.0.4+galaxy4) %} with the following parameters to convert the image to PNG:
>    - *"Type of the overlay"*: `Segmentation contours over image`
>    - {% icon param-file %} *"Intensity image"*: `RNA_input_normalized.tiff` file
>    - {% icon param-file %} *"Label map"*: `RNA_labels.tiff` file (output of {% tool [Convert binary image to label map](toolshed.g2.bx.psu.edu/repos/imgteam/binary2labelimage/ip_binary_to_labelimage/0.5+galaxy0) %})
>    - *"Contour thickness"*: `1`
>    - *"Contour color"*: `red`
>    - *"Show labels"*: `No`
>    - *"Label color"*: `yellow`
{: .hands_on}

Final result should look like this:

![2025_overlay.png](../../images/2d_spot_detection/2025_overlay.png){: width="75%"}

# Create 2D spot detection workflow

The full workflow can now be created! 

Since you are probably going to
run it on several images fetched from IDR we suggest changing the 
setting of the IDR Download tool by downloading imaging in a tarball:

> <hands-on-title>Create the workflow for 2D spot detection</hands-on-title>
>
>    - Name it "2D_spot_detection".
>    - Change *"Download images in a tarball?"*: `Yes` in the **IDR Download** {% icon tool %} tool
>    - Include the {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %}
>    - Connect the output of {% icon param-file %} **4: Download IDR/OMERO** to the "input_file"
>   input of {% icon tool %} **5: Unzip**.
>    - Connect the output of {% icon param-file %} **4: Unzip** to the "Input Image"
>   input of {% icon tool %} **6: Perform histogram equalization**.
> 1. Edit the workflow you just created:
>    - Select "Input dataset" from the list of tools. The step {% icon param-file %} **1: Input Dataset** appears.
>    - Select "Text Input" from the list of tools. The step {% icon param-file %} **2: Text Input** appears.
>    - Select "Integer Input" from the list of tools. The step {% icon param-file %} **3: Integer Input** appears.
>    - Change the "Label" of {% icon param-file %} **1: Input Dataset** to `IDR IDs`.
>    - Change the "Label" of {% icon param-file %} **2: Text Input** to `Channel selection`.
>    - Change the "Label" of {% icon param-file %} **3: Integer Input** to `z-stack selection`.
>    - Connect the output of {% icon param-file %} **1: Input Dataset** to the "Image IDs to download" input of {% icon tool %} **4: Download IDR/OMERO**.
>    - Connect the output of {% icon param-file %} **2: Text Input** to the "Name of the channel to download" input of {% icon tool %} **4: Download IDR/OMERO**.
>    - Connect the output of {% icon param-file %} **3: Integer Input** to the "z-plane of images to download" input of {% icon tool %} **4: Download IDR/OMERO**.
>    - Mark the results of {% icon tool %} **7: Perform 2-D spot detection** and {% icon tool %} **9: Overlay images** as the primary outputs of the workflow (by clicking on the checkboxes of the outputs).
{: .hands_on}

The final workflow should look like this:

![2025_workflow.png](../../images/2d_spot_detection/2025_workflow.png)

# Conclusion

In this exercise, you imported images into Galaxy from IDR, 
detect RNA single molecules using the 2D spot detection tool 
and visualize them on the original intensity image.
Finally, you created a reusable workflow which can be used on a tarball of IDR images.


topic: imaging
tutorial: where-to-start-bioimaging-galaxy
title: Where to start with bioimage analysis in Galaxy
difficulty: introductory
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["Microscopy", "Bioimaging", "Histology"]
content_hash: 7785927af1170873a85cb7d061597645
last_modified: 2026-02-26T21:12:16.820402
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/imaging/tutorials/where-to-start-bioimaging-galaxy/tutorial.html

content:
Bioimage analysis is the process of extracting meaningful information, such as quantitative data, from images in the life sciences. In this field, images are typically acquired using microscopy. Whether examining stained tissue sections in histology or tracking fluorescently labeled proteins in live cells, the fundamental goal remains consistent: **turning pixels into numbers**.

In Galaxy, we provide an ecosystem to make this process reproducible and scalable. We focus on making image analysis **FAIR by design** _(Findable, Accesible, Interoperable, Reusable)_, ensuring that your image formats, metadata, tools, and workflows remain reusable, transparent, and traceable from the very first upload to the final figure ({% cite GalaxyCommunity2024 %}).

This tutorial serves as your **compass** for navigating the Galaxy imaging landscape, specifically for bioimage data preprocessing and analysis. You will learn how image data is structured, explore the tools available in Galaxy, and discover how to kickstart your image analysis journey. By the end, you'll understand how to transform raw, complex image data into structured tables of measurements ready for downstream statistical analysis—and know where to find the tools and resources to continue your work.

![Turning pixels into numbers!]({% link topics/imaging/images/where-to-start-bioimaging-galaxy/imaging_summary_image.png %} "Workflow showing the transformation from raw microscopy images to quantitative measurements.")

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# 1. Know your data (the "digital anatomy" of an image)

As Pete Bankhead points out in *[Introduction to Bioimage Analysis](https://bioimagebook.github.io/index.html)* ({% cite Bankhead2022 %}), _"an image is not just a picture, it is a collection of measurements."_ Therefore, before starting any analysis, it is important that you understand the structure of your data.

To the human eye, an image is a visual representation of a biological sample. However, to a computer, it is a **numerical array** ({% cite Sedgewick2010 %}). The dimensionality of this array depends on your data:

* A **2D grayscale image** is a **2D array** (matrix): rows × columns
* A **2D color image** is a **3D array** (tensor): rows × columns × channels (RGB)
* A **3D grayscale image (volume)** is a **3D array**: X × Y × Z
* A **3D color image (volume)** is a **4D array**: X × Y × Z × C
* A multi-dimensional hyperstack is a **tensor**: a multi-dimensional array (e.g., X × Y × Z × C × T)

Every point in that array—the **pixel** (2D) or **voxel** (3D)—is a data point representing the number of photons or the signal intensity detected at that specific coordinate. Understanding "digital anatomy" of an image means knowing exactly how those numbers were recorded, how they are organized across dimensions, how they are spaced and oriented in 3D space, and what the limitations of the employed imaging technique are (e.g., due to over/undersaturation that leads to clipped intensities).

![Example digital image and its pixel representation]({% link topics/imaging/images/where-to-start-bioimaging-galaxy/pete_bank_imaging.png %} "A digital image can be represented visually or numerically. <strong>(A–B)</strong> The image is visualized as a grid of small squares with different gray intensities, where each square corresponds to one pixel. This square pattern is only a display convention to help us interpret the image. <strong>(C)</strong> Internally, the image is stored as numerical values arranged in an array. Inspecting these raw numbers directly makes it difficult to recognize the image content. Figure from {% cite Bankhead2022 %} (CC BY 4.0).")


If you don't understand the numbers behind the colors, you risk performing "image processing" (simply making a pretty picture) rather than "image analysis" (extracting quantitative data and meaningful insights). Let's unpack the data behind an image step by step.

## Pixels and voxels

An image is a grid of **pixels** (2D images) or **voxels** (3D images). Think of an image as a vast mosaic where every tile is a "picture element." In 2D, these tiles are flat, but in 3D imaging, they have depth and are called **voxels** ("volumetric elements", e.g., {% cite Pawley2006 %}). Each one is like a small bucket that has captured a specific amount of signal (usually light), which the computer records as a single number.

Image intensities represent measurements from the imaging system. To make these numbers meaningful for scientific analysis, we must define two key properties: their **range** (the possible values intensities can take) and their **spatial extent** (the physical size represented by each pixel or voxel, which determines the image resolution).

## The 5 dimensions (5D)

In everyday photography, we usually deal with 2D color images. However, in the life sciences, we can capture **hyperstacks**: multi-dimensional data structures that represent a biological sample across space (`X`, `Y`, `Z` axes), spectrum (`C` axis), and time (`T` axis). Although many scientific images adopt the `XYZCT` ({% cite Goldberg2005 %}) or `TCZYX` axes order (e.g., ({% cite Moore2021%}), making wrong assumptions about that order is a frequent source of error. It is thus important to make sure that the axes order is properly annotated in the image metadata.

* **X, Y & Z (Spatial):** X and Y represent the width and height of your image (the 2D plane). Z represents multiple optical sections or "slices" taken at different focal planes to reconstruct a 3D volume. In confocal or light-sheet microscopy, Z captures different depths through the sample.
* **C (Channel):** Different wavelengths, fluorescent probes, or staining methods corresponding to specific biological structures or molecules. In fluorescence microscopy, examples include DAPI for nuclei, GFP for proteins, and Phalloidin for actin. In histology, different channels can represent various antibody stains (immunohistochemistry) or traditional stains like hematoxylin and eosin (H&E).
* **T (Time):** Separate frames captured over a duration (time-lapse), allowing you to track movement, growth, or dynamic changes in cellular processes.

However, it is important to note that while we describe them in `XYZCT` order, different microscope vendors, tools, and file formats store them in different orders (e.g., `TZCXY` or `XYCZT`). Ensuring that the dimensional order is properly annotated within the image metadata is essential so that Galaxy (and any image analysis software in general) doesn't accidentally interpret a Z-slice as a time-point or a channel as a Z-position (e.g., {% cite Linkert2010 %}).

> <tip-title> Respect the channels </tip-title> 
> While we often look at "merged" RGB images for presentations, you should **always perform quantification on the raw, individual channels.** Merging or converting to RGB often involves data compression, bit-depth reduction (typically down to 8-bit), or intensity scaling that distorts the underlying measurements and loses quantitative information ({% cite Cromey2010 %}). Each channel should be analyzed separately and then results can be compared or combined downstream.
> 
{: .tip}

> <question-title>Identify the dimensions</question-title>
>
> If you have a time-lapse experiment where you recorded 3 different fluorescent markers across 10 focal planes (Z-slices) every minute for an hour, how many total 2D images (planes) are in your file?
>
> > <solution-title></solution-title>
> > You would have: $$3 \text{ (Channels)} \times 10 \text{ (Z-slices)} \times 60 \text{ (Time points)} = 1,800 \text{ total 2D planes}$$
> >
> > Each individual plane is an $$X \times Y$$ image. Galaxy tools need to know the correct dimensional order (e.g., `XYZCT` vs `XYCZT` vs `TZCXY`) to ensure they correctly interpret which dimension is which—otherwise, you might accidentally measure intensity changes across Z-depth when you meant to measure changes over time!
> {: .solution}
{: .question}

## Bit depth (the range and precision limits)

The bit depth corresponds to an **upper limit of the precision** when storing image intensities by determining the range of possible intensity values and the smallest distinguishable difference between them. The **actual precision** may also be lower and depends on many factors in your imaging system, like sensor noise, photon statistics, optical aberrations, and sample preparation quality (e.g., {% cite Pawley2006 %}).

When it comes to the representation of the measurements of the imaging system as the image intensities, there generally are two main types of such representation:

### Integer representations

These are the natural format for raw camera data, as imaging sensors essentially count photons:

* **8-bit:** 2<sup>8</sup> = 256 possible values (range: 0 to 255 for unsigned, or -128 to 127 for signed). While this looks fine to our eyes, it is often too "coarse" for thorough quantitative analysis.

* **16-bit:** 2<sup>16</sup> = 65,536 possible values (range: 0 to 65,535 for unsigned, or -32,768 to 32,767 for signed). This is the **scientific gold standard for image acquisition and storage**, because it allows you to detect even subtle differences in image intensities that would be lost (e.g., due to rounding) when using an 8-bit representation {% cite Haase2022 %}.

* **32-bit:** 2<sup>32</sup> = 4,294,967,296 possible values. While less common in direct acquisition, 32-bit integer formats are sometimes used in intermediate processing steps or for label images where many distinct regions need to be encoded.

### Floating-point representations

After image processing operations (e.g., background subtraction, normalization, deconvolution), intensity values often become non-integer (i.e., decimal, fractional) or fall outside the original acquisition range. Floating-point formats accommodate this by representing both positive and negative values with decimal precision:

* **16-bit float (half precision):** Can represent values approximately in the range of ±65,504 with limited decimal precision. This format is increasingly used in machine learning and GPU-accelerated image processing because it saves memory while providing sufficient precision for many applications.

* **32-bit float (single precision):** Can represent values approximately in the range of ±10<sup>38</sup> with about 7 significant decimal digits of precision. This is the **most common format for image processing** as it balances precision, range, and computational efficiency.

* **64-bit float (double precision):** Can represent values approximately in the range of ±10<sup>308</sup> with about 15-16 significant decimal digits of precision, useful for iterative algorithms or when accumulating many operations where small errors could compound.

Unlike integer representations where the precision (smallest distinguishable difference) is constant across the entire range, floating-point precision varies: it is highest near zero and decreases as values approach the limits of the representable range.

Many analysis tools automatically convert to floating-point internally to preserve accuracy during calculations.

### Bit depth: why it matters for science

A common misconception is that if two images look identical on a desktop monitor, they contain the same data. However, our eyes are used to computer screens that are usually limited to 8-bit, while your microscope sensor is far more sensitive. 

When you perform certain pre-processing tasks, such as subtracting the image background or contrast enhancement, you are essentially **stretching the histogram** of the image data.

* In **8-bit**, with only 256 possible values, stretching creates more pronounced "gaps" in your histogram (quantization artifacts), potentially making your data distribution appear discontinuous.
* In **16-bit integer**, with 65,536 possible values, the same stretching operation creates smaller relative gaps. While quantization artifacts still occur, they are far less severe and less likely to impact downstream quantitative analysis ({% cite Cromey2010 %}).

The key difference is not whether artifacts appear, but their **magnitude in relation to your data range**. Think of it this way: spreading 256 values across a wider range creates larger "holes" between adjacent intensity levels than spreading 65,536 values across the same range.

![Bit depth: why it matters in science]({% link topics/imaging/images/where-to-start-bioimaging-galaxy/bit_depth.png %} "Comparison showing how different bit depths affect the ability to detect subtle differences in image intensities during quantitative analysis.")

> <tip-title> Precision </tip-title> 
> Always try to keep your data in **16-bit integer** or **floating-point formats** (16-bit, 32-bit, or 64-bit float) during analysis. Converting to 8-bit too early reduces your ability to detect subtle intensity differences and makes quantization artifacts more severe during processing. This lost precision can never be recovered ({% cite Haase2022 %}). 
>
{: .tip}

> <question-title> Why use floating-point instead of 16-bit integer? </question-title>
>
> We already covered different floating-point formats in the bit depth section above. But why would we need to convert from 16-bit integer to floating-point during analysis?
>
> > <solution-title></solution-title>
> >
> > 16-bit **integer** images can only store whole numbers (0–65,535). Many image processing operations produce non-integer results that would be lost through rounding:
> >
> > * Dividing pixel intensity 5 by 2 gives 2.5 (must be rounded to 2 or 3 in integer format)
> > * Background subtraction can produce negative values (impossible in unsigned integer format)
> > * Averaging two pixels with values 100 and 101 gives 100.5 (precision lost in integer)
> > * Normalizing intensities often produces fractional values
> >
> > Floating-point still suffer from rounding but are capable to preserve these exact values with a much higher precision, reducing cumulative rounding errors across multiple processing steps. Additionally, floating-point formats can represent values in a far larger range, which is useful when processing operations produce very small or very large values (e.g., summing image intensities across multiple images; e.g., {% cite Pawley2006 %}). Most image analysis software automatically converts images to 32-bit float internally for this reason.
> >
> {: .solution}
{: .question}

## Spatial calibration (the size)

Image pixels (or voxels) have no _intrinsic_ physical size; they are just units of storage and representation. **Spatial calibration** is the metadata that links these digital units to physical reality (e.g., $$1 \text{ pixel} = 0.25 \mu m$$). Without this "secret sauce," you can count objects, but you cannot accurately measure how big they are, how fast they move, or their concentration (e.g., {% cite Linkert2010 %}, {% cite Haase2022 %}). 

Metadata like the calibration information is usually stored in the image header. If you lose this metadata during a file conversion (e.g., saving as a standard .jpg), your analysis will only be able to provide quantitative results in "pixels," which may have no biological meaning.

> <tip-title> Avoid saturation </tip-title> 
> If you see an image intensity value of "0", the sensor may have detected nothing, or it could represent true absence of signal—context matters. If you see the maximum value (e.g., $$255$$ or $$65,535$$), your sensor was overwhelmed. These phenomena are called **saturation** (under- and oversaturation, respectively). Saturated pixels are often "clipped," meaning the true biological signal was lower or higher than what the camera could record (e.g., {% cite Pawley2006 %}). This data is lost forever and cannot be accurately quantified. 
>
{: .tip}

# 2. How to get your images into Galaxy

Galaxy is built to handle the complexity of biological data. However, microscopy images often come in "vendor-specific" or "proprietary" formats. Your entry point into Galaxy depends on how your data was saved:

* **Standard Formats (.tiff, .png):** Use the standard Galaxy **Upload** tool.
* **Open Formats:** OME-TIFF, which includes standardized metadata, and OME-Zarr, a cloud-friendly image format designed to efficiently store and access very large, multi-dimensional bioimaging datasets (such as spatial transcriptomics) along with their metadata.
* **Proprietary Formats (.czi, .nd2, .lif):** These formats "wrap" image data and metadata together. While you can often export TIFFs from your microscope software, using the **{% tool [Convert image format with Bioformats](toolshed.g2.bx.psu.edu/repos/imgteam/bfconvert/ip_convertimage/6.7.0+galaxy3) %}** tool allows Galaxy to "unlock" and standardize the metadata hidden inside these files ({% cite Moore2021 %}). For more information on supported formats, see the [Bio-Formats documentation](https://bio-formats.readthedocs.io/en/v8.4.0/).
* **OMERO Integration:** If your institution uses an **OMERO server**, you can import images directly via the **Remote Files** section in the upload panel. You can access OMERO using Galaxy. Additionally, you can fetch images directly using the {% tool [IDR Download by IDs](toolshed.g2.bx.psu.edu/repos/iuc/idr_download_by_ids/idr_download_by_ids/0.45) %} tool. See [Overview of the Galaxy OMERO-suite]({% link topics/imaging/tutorials/omero-suite/tutorial.md %}) for more details.

## Why use the Bio-Formats tool suite?

The **{% tool [Convert image format with Bio-formats](toolshed.g2.bx.psu.edu/repos/imgteam/bfconvert/ip_convertimage/6.7.0+galaxy3) %}** tool does more than just open a file; 

> <hands-on-title> Inspecting Image Metadata </hands-on-title>
>
> 1. {% tool [Show image info](toolshed.g2.bx.psu.edu/repos/imgteam/image_info/ip_imageinfo/5.7.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input image"*: `Select your uploaded microscopy file`
> 2. **Review the output:** Examine the resulting text file. Look for key metadata fields like "PhysicalSizeX" (pixel calibration) and "BitDepth". 
>
> Example output:
>
> ```
> Checking file format [Tagged Image File Format]
> Initializing reader
> TiffDelegateReader initializing /data/dnb10/galaxy_db/files/2/3/0/dataset_23048ef7-e850-41f4-9fc0-da24b2b4c36b.dat
> Reading IFDs
> Populating metadata
> Checking comment style
> Populating OME metadata
> Initialization took 1.994s
>
> Reading core metadata
> filename = /data/dnb10/galaxy_db/files/2/3/0/dataset_23048ef7-e850-41f4-9fc0-da24b2b4c36b.dat
> Series count = 1
> Series #0 :
> 	Image count = 1
> 	RGB = true (3) 
> 	Interleaved = false
> 	Indexed = false (false color)
> 	Width = 2752
> 	Height = 2208
> 	SizeZ = 1
> 	SizeT = 1
> 	SizeC = 3 (effectively 1)
> 	Thumbnail size = 128 x 102
> 	Endianness = intel (little)
> 	Dimension order = XYCZT (certain)
> 	Pixel type = uint8
> 	Valid bits per pixel = 8
> 	Metadata complete = true
> 	Thumbnail series = false
> 	-----
> 	Plane #0 <=> Z 0, C 0, T 0
>
>
> Reading global metadata
> BIG_TIFF: false
> BitsPerSample: 8
> Compression: LZW
> DATE_TIME_DIGITIZED: 2023:07:31 16:51:32
> DATE_TIME_ORIGINAL: 2023:07:31 16:51:32
> ImageLength: 2208
> ImageWidth: 2752
> LITTLE_ENDIAN: true
> MetaDataPhotometricInterpretation: RGB
> MetaMorph: no
> NewSubfileType: 0
> NumberOfChannels: 3
> PhotometricInterpretation: RGB
> PlanarConfiguration: Chunky
> Predictor: No prediction scheme
> ResolutionUnit: Inch
> SUB_SEC_TIME_DIGITIZED: 00
> SUB_SEC_TIME_ORIGINAL: 00
> SamplesPerPixel: 3
> Software: ZEN 3.1 (blue edition)
> XResolution: 300.0
> YResolution: 300.0
>
> Reading metadata
> ```
> > <comment-title>Why check this first?</comment-title>
> > Before starting a long analysis, this tool helps you verify if Galaxy correctly "unlocked" the metadata inside your proprietary file. If pixel sizes are missing here, your final results will be in pixels rather than biological units like micrometers ($$\mu m$$).
> {: .comment}
>
{: .hands_on}

For modern, large-scale, or cloud-based datasets, you can use **{% tool [Convert to OME-Zarr with Bio-formats](toolshed.g2.bx.psu.edu/repos/imgteam/bioformats2raw/bf2raw/0.7.0+galaxy3) %}**. This converts a wide range of formats into **OME-Zarr** following the OME-NGFF specification. OME-Zarr is specifically optimized for high-performance viewing and is a staple for massive datasets like those found in spatial transcriptomics.

> <tip-title> Which open format should I choose? </tip-title>
>
> While there are many options, two open community-endorsed formats are increasingly dominant in bioimaging:
> * **OME-TIFF:** Best for "classic" microscopy (2D/3D stacks) and maximum compatibility with classical software like ImageJ/Fiji or QuPath.
> * **OME-Zarr:** A cloud-optimized format for very large, multi-dimensional bioimaging datasets (including spatial transcriptomics), enabling scalable storage and efficient data access. {% cite Moore2021 %}
> 
> Both are superior to proprietary formats because they make it possible to preserve your metadata attached to your pixel data throughout the entire Galaxy workflow.
>
{: .tip}

> <question-title> Metadata testing </question-title>
>
> Why we should avoid bioimages to be "Save as JPEG" from your microscope or computer?
>
> > <solution-title></solution-title>
> >
> > 1. **Data Integrity:** JPEGs use "lossy" compression, which changes pixel values to save space. This approach literally modifies your scientific data.
> > 2. **Metadata Retention:** A standard JPEG or simple TIFF often "strips" the metadata. You would lose the information about the physical size a pixel represents, making it impossible to calculate the size of your objects later.
> >
> {: .solution}
{: .question}

# 3. Before you begin: diagnose your data and define your goals

The first step in any analysis is to characterize the data you have. Before clicking any tools, ask yourself these four questions:

 - **What was the imaging modality?** (Fluorescence microscopy, brightfield histology, high-content screening?)
 - **What is the biological subject?** (Individual cells, complex tissues, or subcellular structures?)
 - **What is the file format?** (Standard formats like .tif and .png, or proprietary vendor formats like .czi, .nd2, or .lif?)
 - **Where is the data stored?** (A local drive, an OMERO server, a public repository, or a remote URL?)

 Then, define your quantitative goals:

 - **What do you want to measure?** (Cell count? Nuclear size? Protein intensity? Colocalization? Movement over time?)
 - **What level of detail do you need?** (Measurements per object, summary statistics across many objects, or maps of where things are located?)
 - **What comparison will you make?** (Differences between conditions, changes across time, or descriptive characterization?)

Depending on your answers, your starting path in Galaxy will change.

# 4. The lifecycle of an analysis pipeline

A typical project in Galaxy is not a single click, but a sequence of logical steps. Think of it as a factory assembly line: you start with raw materials (pixels) and move through various stations until you have a finished product (a table of measurements). Let's examine these stages together.

## Stage A: Pre-processing (cleaning)
Raw images are rarely perfect. They often contain electronic noise from the camera, uneven illumination from the microscope, or staining artifacts. Pre-processing prepares your digital image for analysis by enhancing the structures you want to identify and suppressing unwanted features like noise or artifacts ({% cite Bankhead2022 %}).

* **Background subtraction:** Removes the "haze" or background fluorescence caused by out-of-focus light. This is crucial for accurate intensity measurements later on.
* **Denoising:** Applies linear or non-linear filters to reduce noise while preserving meaningful structures. A **Gaussian filter** is a linear method that performs spatially weighted averaging using a kernel whose coefficients decrease smoothly with distance from the center, following a Gaussian (bell-shaped) distribution. It is well suited for reducing noise that approximates [additive white Gaussian noise (AWGN)](https://en.wikipedia.org/wiki/Additive_white_Gaussian_noise). In contrast, a **Median filter** is a non-linear method that replaces each pixel value with the median of its local neighborhood. It is particularly effective for removing impulse noise such as [salt-and-pepper noise](https://en.wikipedia.org/wiki/Salt-and-pepper_noise) and can also help mitigate impulse-like components of [Poisson (shot) noise](https://en.wikipedia.org/wiki/Shot_noise). It is also appropriate for smoothing binary or label images because it preserves original intensity values. Compared to Gaussian smoothing, median filtering better maintains sharp edges such as cell boundaries (e.g., {% cite Haase2022 %}).

> <tip-title> Document your filters </tip-title> 
> Every filter you apply changes the pixel values. While this is necessary for segmentation, you must document these steps to ensure reproducibility. In Galaxy, this is done automatically by your history, which records every parameter used in your pre-processing steps.
{: .tip} 

In Galaxy, you can start the pre-processing stage with tools like **{% tool [Apply standard image filter](toolshed.g2.bx.psu.edu/repos/imgteam/2d_simple_filter/ip_filter_standard/1.16.3+galaxy1) %}**.

> <hands-on-title> Filtering Noise </hands-on-title>
>
> 1. {% tool [Apply standard image filter](toolshed.g2.bx.psu.edu/repos/imgteam/2d_simple_filter/ip_filter_standard/1.16.3+galaxy1) %} with the following parameters:
>    - {% icon param-select %} *"Type of image data to process"*: `2-D image data (or series thereof)`
>        - {% icon param-file %} *"Input image (2-D)"*: `your_uploaded_image.tif`
>        - {% icon param-select %} *"Filter type"*: `Median`
>            - {% icon param-text %} *"Size"*: `3`
>
> > <comment-title>Choosing the right filter</comment-title>
> > Try running this tool twice: once with a **Median** filter (using a small *Size* like 3) and once with a **Gaussian** filter (adjusting *Sigma*). Zoom in on the edges of your objects. You will notice that the Median filter preserves boundaries much better, while the Gaussian filter blurs them.
> {: .comment}
>
{: .hands_on}

## Stage B: Segmentation (defining objects)
This is the most critical step. Here, you tell the computer which pixels belong to an "object" (like a nucleus) and which belong to the "background." 

* **Thresholding:** A "cutoff" method where pixels above (or below) a certain intensity value are classified as object. This value can be set manually or determined automatically using algorithms like Otsu or Li. The result is typically a **Binary Mask**.
* **Inference (Deep Learning):** Advanced AI models like **Cellpose** use pre-trained neural networks to recognize complex shapes. These methods might be superior at specific segmentation tasks such as "untangling" cells that are touching or overlapping in high-density environments ({% cite stringer2025cellpose3 %}, {% cite Schmidt2018 %}, {% cite Pachitariu2025.04.28.651001 %}).

> <hands-on-title> Creating a Segmentation Mask </hands-on-title>
>
> 1. {% tool [Threshold image](toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.25.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input image"*: `your_filtered_image.tif`
>    - {% icon param-select %} *"Thresholding method"*: `Globally adaptive / Otsu`
>        - {% icon param-text %} *"Offset"*: `0`
>    - {% icon param-check %} *"Invert output labels"*: `No`
>
> 2. **Review the output:** You should now have a binary image where your biological objects are represented as white pixels (255) on a black background (0).
>
{: .hands_on}

> <comment-title>Masks vs. Label Images</comment-title>
> It is important to distinguish between these two types of images in Galaxy:
>
> * **Binary Mask:** This is what you created with the **Threshold** tool. Every pixel is either "Background" (0) or "Foreground" (255). It tells the computer *where* the objects are, but it doesn't know that one cell is different from the one next to it.
> * **Label Image:** Tools like **Label objects** (Connected Component Analysis) take a mask and assign a unique integer to every separate "blob." The first cell’s pixels all become `1`, the second cell `2`, and so on.
>
{: .comment}

### Stage B.1: The Region of Interest (ROI)

Once you have segmented your image, you have created **Regions of Interest (ROIs)**. This is a central concept in bioimaging. An ROI is a spatial selection that tells the software: "Ignore the background; only calculate values for these specific coordinates."

In Galaxy, ROIs can take three forms depending on the tool you use:
* **Bounding Boxes:** Simple rectangles used to crop an image or define a search area. In Galaxy, this is only possible to perform this step using interactive tools like QuPath.
* **Binary Masks:** A "stencil" where the ROI is white and everything else is black.
* **Label Images:** A "smart map" where every individual ROI has its own unique integer ID (e.g., {% cite Tosi2021 %}).

> <comment-title>ROIs in interactive vs. automated tools</comment-title>
>Interactive tools like QuPath allows users to inspect images and create ROIs either manually (e.g. drawing with the mouse) or through built-in detection algorithms that generate ROIs automatically. In automated Galaxy workflows, algorithms like Cellpose generate thousands of ROIs programmatically in a fully reproducible pipeline without user interaction ({% cite stringer2025cellpose3 %}, {% cite Pachitariu2025.04.28.651001 %}).
{: .comment}

## Stage C: Post-processing (Refining)
Computers are literal; they might detect tiny specs of dust as "cells." Post-processing is your "cleanup crew" for the mask you created in Stage B.
* **Size filtering:** Automatically removing any detected objects that are too small (noise) or too large (clumps of cells) to be biologically relevant.
* **Morphological operations:** Using operations like "Fill Holes" to fix gaps inside a detected cell or "Erode/Dilate" to slightly adjust the boundaries.

## Stage D: Quantification (Extracting numbers)
Now that you have a clean mask, Galaxy "overlays" it back onto your original raw image. Since the computer knows exactly which pixels belong to which cell, it can now do the math:
* **Intensity:** "How much protein (fluorescence) is inside this specific mask?"
* **Morphology:** "How large, round, or elongated is this nucleus?"
* **Spatial distribution:** "How far is the nucleus from the cell membrane?"

## Stage E: Validation (The sanity check)

**Never trust an algorithm blindly.** The final step is a visual inspection to ensure your numerical outputs represent biological reality rather than processing artifacts ({% cite Bankhead2022 %}). 

* **{% tool [Overlay outlines](toolshed.g2.bx.psu.edu/repos/bgruening/cp_overlay_outlines/cp_overlay_outlines/3.1.9+galaxy1) %}**: This tool takes your **original image** and draws the boundaries of your **detected objects** (from your Label Image) on top of it as colored lines. It is the gold standard for visual validation, allowing you to confirm that the computer's segmentation accurately followed the biological edges of your cells. For a detailed guide on visualizing results, see the "Visualizing Results" section in {% icon level %} [Object Tracking using CellProfiler]({% link topics/imaging/tutorials/object-tracking-using-cell-profiler/tutorial.md %}).

* **{% tool [Overlay images](toolshed.g2.bx.psu.edu/repos/imgteam/overlay_images/ip_overlay_images/0.0.5) %}**: A versatile tool for visualizing colocalization or blending two different channels to see where biological signals overlap.

* **{% tool [Render 3-D image data](toolshed.g2.bx.psu.edu/repos/imgteam/libcarna_render/libcarna_render/0.2.0+galaxy2) %}**: For 3-D images, this tool provides similar visualization functionality, allowing you to render and inspect volumetric data.

> <tip-title> Garbage In, Garbage Out </tip-title>
> If your segmentation (Stage B) is poor, your quantification (Stage D) will be meaningless. Always spend the most time optimizing your segmentation and validating it visually before trusting the final spreadsheet of numbers ({% cite Haase2022 %}).
{: .tip}

> <question-title> Mask vs. Image </question-title>
>
> If you calculate the **Mean Intensity** of a cell, should you use the pixels from the **Binary Mask** or the **Raw Image**?
>
> > <solution-title></solution-title>
> >
> > You must use the **Raw Image**. The mask is just a "stencil" or a map that tells the computer *where* to look. The actual biological information (the intensity values) is only found in the original, unprocessed image.
> >
> {: .solution}
{: .question}

# 5. Finding your workflow: modality and tools

Bioimage analysis is not "one size fits all." Galaxy provides a diverse suite of tools, from classical computer vision to state-of-the-art Deep Learning. Use the logical roadmap below to identify your specific path.

## The decision tree: your logical roadmap

![Microscopy imaging workflow decision tree]({% link topics/imaging/images/where-to-start-bioimaging-galaxy/Microscopy_Imaging_Workflow-2026.png %} "Decision tree to guide you through choosing the right imaging analysis approach.")

## The Galaxy imaging toolbox 

Once you have identified your path on the tree, use this guide to find the corresponding tools in Galaxy. We categorize these into **Standard Tools** (for automated batch processing) and **Interactive Tools** (for visual exploration).

### A. Standard tools (single images, high-performance & batch)

These tools are "wrapped" in Galaxy to build individual workflows and adapt your analysis to your biological question needs. 

These other set of "wrapped" tools allow you to run them on hundreds of images as part of a reproducible history.

| Tool Name | Primary Application | Key Benefit |
| --- | --- | --- |
| **{% tool [Cellpose](interactive_tool_cellpose) %}** | General cellular segmentation | Superior at "unsticking" crowded or overlapping cells using vector flow. |
| **{% tool [CellProfiler](interactive_tool_cellprofiler) %}** | High-content screening & automation | Allows you to build a complex multi-step "pipeline" and run it on thousands of images consistently. |
| **{% tool [Process image using a BioImage.IO model](toolshed.g2.bx.psu.edu/repos/bgruening/bioimage_inference/bioimage_inference/2.4.1+galaxy3) %}** | Advanced AI model inference | Gateway to the [BioImage Model Zoo](https://bioimage.io/), run pre-trained models (like 3D U-Nets) on your data. |

> <hands-on-title> Segmentation with Cellpose </hands-on-title>
>
> In Galaxy, you can choose between the standard Cellpose model or the newer Cellpose-SAM (Segment Anything Model) version. 
>
> Option 1: Standard Cellpose
> 1. {% tool [Run generalist cell and nucleus segmentation with Cellpose 3](toolshed.g2.bx.psu.edu/repos/bgruening/cellpose/cellpose/3.1.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Choose the image file for segmentation"*: `your_preprocessed_image.tif`
>    - {% icon param-select %} *"Choose the pre-trained model type"*: `cyto2` (standard cells) or `nuclei` (just nuclei)
>    - {% icon param-check %} *"Whether to show segmentation?"*: `Yes` (generates a PNG preview)
>    - In **Advanced Options**:
>        - {% icon param-text %} *"Cell or nuclei diameter in pixels"*: `Enter approximate diameter` (or leave blank for automated estimation)
>
> Option 2: Cellpose-SAM (Next Gen)
> 1. {% tool [Run generalist cell and nucleus segmentation with Cellpose-SAM](toolshed.g2.bx.psu.edu/repos/bgruening/cp_sam/cellposesam/4.0.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Choose the image file for segmentation"*: `your_preprocessed_image.tif`
>    - {% icon param-check %} *"Save masks as tiff?"*: `Yes` (Required for quantification later)
>    - {% icon param-check %} *"Save RGB outline images"*: `Yes` (Useful for Stage E: Validation)
>    - In **Advanced Options**:
>        - {% icon param-text %} *"Cell or nuclei diameter in pixels"*: `Enter approximate diameter`
>        - {% icon param-check %} *"Whether to use transformer backbone"*: `Yes` (to utilize the Cellpose3 transformer model)
>
> > <comment-title>Multi-channel Images</comment-title>
> > Cellpose tools in Galaxy typically expect single-channel 2D images. If you have a multi-channel image, you must first use the **Split image along axes** tool to extract the specific channel (e.g., the DAPI channel for nuclei) that you wish to segment.
> {: .comment}
{: .hands_on}

### B. Interactive tools (Visual exploration)

If you prefer a "hands-on" approach to see your results in real-time before scaling up, launch an **Interactive Tool (IT)** directly in your browser:

* **{% tool [QuPath IT](interactive_tool_qupath) %}:** The gold standard for digital pathology. Use this for large tissue sections and to access **StarDist**, a deep learning-based plugin for instance segmentation of star-convex shapes (e.g., nuclei), available in Galaxy through QuPath.
* **{% tool [Ilastik IT](interactive_tool_ilastik) %}:** Best for "training by example"—manually paint a few cells to teach the computer how to segment the rest based on texture.
* **{% tool [Cellpose IT](interactive_tool_cellpose) %}:** & **{% tool [Cellprofiler IT](interactive_tool_cellprofiler) %}:** Useful for building and fine-tuning your parameters visually before running a massive batch job.

## Identifying your modality 

To navigate the tree correctly, you must understand your image type:

* **Fluorescence:** Images where pixel intensity represents the amount of light emitted. While often visualized as multi-colored immunofluorescence, the computer treats each channel as an intensity map where higher values indicate more signal.

* **Brightfield/Histology (Color):** True-color images (RGB) where stains like H&E overlap. You must use **Color Deconvolution** to separate these into individual channels (e.g., separating Hematoxylin from DAB) before measurement. In Galaxy, you may find this tool as: {% tool [Perform color deconvolution or transformation](toolshed.g2.bx.psu.edu/repos/imgteam/color_deconvolution/ip_color_deconvolution/0.9+galaxy0) %}

* **High-density/Tissues:** For packed cells, classical thresholding fails. This is where **Inference** tools like **Cellpose** or **StarDist** shine, as they use pre-trained models to predict boundaries even in crowded environments.
[Image comparing simple thresholding versus AI-based instance segmentation in crowded tissues]

## Practice: applying the roadmap

> <question-title> Scenario 1: The High-Volume Screen </question-title>
>
> You have 500 images of crowded mitochondria. Which path do you choose?
> 
> > <solution-title></solution-title>
> >
> > You should look toward a **Deep Learning** approach (like **Cellpose**) within a **CellProfiler** workflow. Why?
> >
> > 1. **Crowding:** Cellpose handles overlapping structures better than simple thresholding.
> > 2. **Volume:** With 500 images, you need the batch-processing power of **CellProfiler** to stay efficient.
> > 
> {: .solution}
{: .question}

> <question-title> Scenario 2: The 3D Volume </question-title>
>
> You have a single 3D Z-stack of a zebrafish embryo with one fluorescent marker. What is your path?
>
> > <solution-title></solution-title>
> >
> > 1. **Microscopy?** Yes.
> > 2. **Color or B&W?** B&W (Single-channel fluorescence).
> > 3. **Dimensionality?** 3D volume.
> > **Result:** You are on the **Segmentation** path. Use **Cellpose** for automated results or **QuPath** interactively to test different StarDist models.
> > 
> {: .solution}
{: .question}
 

# 6. Common pitfalls to avoid

Even with the best tools, it is easy to accidentally "break" your data before you even start measuring. In bioimage analysis, these errors are often called **artifacts**.

## 1. The "JPG" trap
**Never use JPEG for science.** JPEGs use "lossy" compression, meaning the computer slightly changes pixel intensities to save space. This creates "blocky" artifacts that ruin your ability to measure protein concentration or fine textures.

* **The fix:** Always store quantitative microscopy data in **TIFF** or **OME-TIFF**. These formats preserve the original pixel values and metadata.

* **When is PNG OK?** PNG is lossless and suitable for small 2-D single-channel or RGB images where intensities are not used for quantitative measurements. For example: 2-D label images, segmentation masks, or visualization outputs. In these cases it can reduce file size by orders of magnitude compared to TIFF. However, it should not be used for raw or measurement-grade microscopy data.

## 2. The "merged image" mistake

Analyzing a "Merge" (RGB) image is risky because the intensities of different channels (like DAPI and GFP) are mathematically blended into a single color value.

* **The fix:** Always **Split Channels** in Galaxy. For this purpose you may use the {% tool [Split image along axes with NumPy](toolshed.g2.bx.psu.edu/repos/imgteam/split_image/ip_split_image/2.3.5+galaxy0) %} tool. Measure your DAPI (nuclei) and GFP (protein) separately to ensure scientific accuracy.

## 3. Ignoring saturation
If your image is too bright, you might hit the camera sensor's limit ($$255$$ for 8-bit or $$65,535$$ for 16-bit). This is called **Clipping**. 
* **The fix:** Check your histogram. If you see a giant "spike" at the very end of the graph, your data is saturated and you cannot accurately quantify the brightest parts of your sample.

> <question-title> The artifact detective </question-title>
>
> You notice that in your time-lapse, the image gets slightly dimmer with every frame. Is this a biological change or an artifact?
>
> > <solution-title> Solution </solution-title>
> >
> > This is likely **Photobleaching**, a common artifact where fluorescent molecules physically break down and lose their ability to emit light after repeated exposure to the excitation laser.
> >
> > 
> >
> > The standard approach to fix this is **Bleach Correction**. While a dedicated tool for this is not currently in this Galaxy workflow, you can use {% tool [Perform histogram equalization](toolshed.g2.bx.psu.edu/repos/imgteam/2d_histogram_equalization/ip_histogram_equalization/0.18.1+galaxy0) %} with the **CLAHE** algorithm as a workaround. 
> >
> > This does not "fix" the physical loss of signal, but it enhances the local contrast of the dim frames. This makes it possible for the segmentation tools to still detect and outline your objects in the later, darker stages of the experiment.
> >
> {: .solution}
{: .question}

# Conclusion

Congratulations! You have navigated the core principles of the Galaxy bioimaging landscape. By mastering these tools, you have moved beyond simply "looking" at images to performing rigorous, reproducible quantification.

In this tutorial, you have learned that:

* **Images are data:** Every pixel is a mathematical value. Whether you are adjusting bit depth or calibrating scales, you are ensuring that these numbers represent physical reality.
* **Context matters:** Using **Bio-Formats** and managing metadata ensures that your analysis respects the original biological dimensions, time-points, and channels.
* **Segmentation is a bridge:** Creating masks and label images is the critical step that allows a computer to recognize individual biological entities, such as cells or nuclei, for separate measurement.
* **Validation is essential:** Tools like **Overlay Outlines** and artifacts like **Photobleaching** remind us that automated analysis requires human oversight to ensure the results are biologically sound.

By building modular workflows in Galaxy, you have created a pipeline that is not only accurate but also shareable and reproducible, the **_gold standard_** of modern open science.

## Further Reading

For more inspiration and a complementary perspective on designing bioimage analysis pipelines, we recommend {% cite Fazeli2025 %}, which provides a decision tree approach to bioimage analysis that complements the workflows covered in this tutorial.

# Glossary of bioimage terms

* **Binary image:** A black-and-white image where pixels can only have two values: 0 (Background) and 1 (Object).
* **Bit depth:** The number of bits used to store each pixel value, determining the range and precision of intensity values an image can represent (e.g., 8-bit = 256 values, 16-bit = 65,536 values).
* **Color deconvolution:** The mathematical separation of overlapping color stains (like the purple Hematoxylin and pink Eosin in H&E histology) into individual intensity channels for measurement.
* **Hyperstack:** A multi-dimensional image data structure that extends beyond 2D or 3D, incorporating additional dimensions such as channels (C) and time (T), typically represented as X × Y × Z × C × T.
* **Label image:** An image where each individual detected object is assigned a unique integer value (e.g., cell 1 = 1, cell 2 = 2), allowing the computer to distinguish and measure each object separately. Unlike a binary mask, a label image encodes object identity.
* **Mask:** A binary "stencil" or digital cutout that tells the computer exactly which pixels belong to an object of interest and which should be ignored.
* **Metadata:** Information stored alongside image data that describes how, when, and where the image was acquired (e.g., pixel size, bit depth, dimensional order, microscope settings).
* **OME-TIFF:** An open-source, standardized file format designed for microscopy that packages raw pixel data together with its essential metadata.
* **Photobleaching:** An artifact in fluorescence microscopy where fluorescent molecules permanently lose their ability to emit light after repeated exposure to the excitation laser, causing images to appear progressively dimmer over time.
* **Pixel / Voxel:** The smallest unit of a digital image. A pixel (2D) or voxel (3D) represents a single data point recording the signal intensity detected at a specific location in the image.
* **ROI (Region of Interest):** A spatial selection that defines the area of an image to be measured or analyzed, telling the software to ignore the background and focus only on specific coordinates.
* **Saturation:** A digital clipping artifact that occurs when light intensity exceeds the camera sensor's maximum capacity, resulting in lost data and "flat" peaks where the brightest signals should be.
* **Segmentation:** The process of partitioning an image into meaningful regions by classifying pixels as belonging to either an object of interest or the background.
* **Spatial calibration:** Metadata that defines the physical size represented by each pixel or voxel (e.g., 1 pixel = 0.25 μm), linking digital units to real-world measurements and enabling biologically meaningful quantification.
* **Thresholding:** A segmentation method that classifies pixels as object or background based on whether their intensity value falls above or below a defined cutoff, producing a binary mask.

# Next Steps: choose your tutorial

Now that you have your "compass," it is time to choose a specific path. Pick the tutorial that matches your research goal:

* **The Basics:** {% icon level %} [Introduction to Image Analysis]({% link topics/imaging/tutorials/imaging-introduction/tutorial.md %}) – A deeper dive into the fundamental concepts of digital images.

* **Fluorescence & Screening:** {% icon level %} [HeLa Cell Screen Analysis]({% link topics/imaging/tutorials/hela-screen-analysis/tutorial.md %}) – Learn to process high-throughput screens using classical techniques.

* **Time-Lapse & Events:** {% icon level %} [Detection of MitoFlashes]({% link topics/imaging/tutorials/detection-of-mitoflashes/tutorial.md %}) – Track transient biological events over time.

* **Classical Segmentation:** {% icon level %} [Voronoi-based Segmentation]({% link topics/imaging/tutorials/voronoi-segmentation/tutorial.md %}) – A powerful approach for partitioning cells when they are touching but have clear centers.

* **Advanced AI:** {% icon level %} [Imaging using Bioimage Zoo Models]({% link topics/imaging/tutorials/process-image-bioimageio/tutorial.md %}) – Use pre-trained deep learning models for complex segmentation tasks.

* **Automated Pipelines:** {% icon level %} [CellProfiler in Galaxy]({% link topics/imaging/tutorials/tutorial-CP/tutorial.md %}) – Master the use of CellProfiler modules to build end-to-end automated workflows.

* **Tracking:** {% icon level %} [Object Tracking using CellProfiler]({% link topics/imaging/tutorials/object-tracking-using-cell-profiler/tutorial.md %}) – Move from static images to following individual objects through time and space.

* **Histology & Pathology:** *Coming Soon!* A dedicated tutorial on **Histology Staining and Color Deconvolution** is currently in development.


