# tutorials

topic: admin
tutorial: database-schema
title: Galaxy Database schema
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 07a0a0cb6053103cec415189dd2e5218
last_modified: 2026-05-27T09:01:47.636037
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/database-schema/tutorial.html

content:
For the hands-on examples you need access to a Galaxy server and access to its PostgreSQL database. You can set-up this yourself, or use the Galaxy Docker Image provided by Björn Grüning (https://github.com/bgruening/docker-galaxy-stable). During this tutorial, we will work with the Galaxy Docker Image.

Setting up Docker and using the Galaxy Docker Image:
(please do this before the tutorial, preferably when you are still at home using a fast internet connection)

Follow the instruction to install the Docker engine: https://docs.docker.com/engine/installation/   
Execute:  docker run -d -p 8080:80 bgruening/galaxy-stable
(this will download the Galaxy Docker Image, when executed the first time, and
start it)
Test the Docker image in your web browser: http://localhost:8080
(see first paragraph of Björn’s introduction, for special cases when using the
Docker Toolbox on Mac and Windows). Quit by: docker kill NAME (you get the name with:  docker ps )

# Introduction

## Database versus Object Model

The session description is database centric and we’ll be focusing on the relational
database that backs Galaxy servers.  But that’s only half the picture of the this data.
The other is the object model which is the object-oriented view of this same data.
The object model is used by the code to manipulate and access the database.
The translation between the two worlds is handled by an object-relational mapping implemented with SQLAlchemy (https://www.sqlalchemy.org).

Today we are covering the database and how to access it with SQL.  We aren’t going to cover the corresponding object model or object relational mapping.

## Database Platform

The default out-of-the-box Galaxy installation uses SQLite (https://www.sqlite.org).
SQLite is a lightweight database management system (DBMS) that can be packaged inside Galaxy and does not require any additional steps at initial setup time.

However, SQLite is not the recommended DBMS for running a Galaxy server. The recommended production DMBS for
Galaxy is PostgreSQL (https://www.postgresql.org). PostgreSQL offers a full set of DBMS features and robust support for multiple simultaneous users.

This workshop will be entirely based in PostgreSQL (also referred to as Postgres).

## What is in (and not in) the Galaxy database?

The Galaxy database contains management information about your server.  The database tracks users, groups, jobs, histories, datasets, workflows and so on.  

What’s not in the database is the data. Datasets are stored outside the database. The database does keep metadata – information about the datasets such as data type. The tools themselves are not stored in the database either.

## Understanding the Database Schema

### ER diagrams and SchemaSpy

Entity-relationship diagrams are a way to understand tables and the relationships between them
inside a relational database.  SchemaSpy (http://schemaspy.org) is a free (and remarkable tool) for
generating ER diagrams. Here's a [diagram of the Galaxy database](../../images/galaxy_schema.png)
(generated by SchemaSpy on June 6 2020; 6Mb).

The “Tables” tab is a good place to start learning the structure of the database.  Each table represents a different type of thing, and often that thing is itself a relationship. For example, each record in the dataset table has information about a specific dataset, while records in the history_dataset_association table have information about what histories that dataset is in.

Each SchemaSpy table’s page shows the attributes in that table, as well as any constraints on those attributes, and the relationships between that table and other tables.

Also see the “Run SchemaSpy in this container” section below for how to install and then run SchemaSpy yourself.

### Database conventions
The Galaxy database uses a number of naming and design conventions.  Understanding these can make navigating the database much easier.

### id attributes
Every table has an id column that uniquely identifies each row.  (The id column is the primary key in database terminology.) Beyond uniquely identifying a row in the table, ID values have no meaning.  ID values are unique within a table, but not across the database.

### Relationships between tables, and `_id` columns
Relationships between tables are implemented by exporting id columns from one table into another.  Imported ids are called foreign keys in database nomenclature, and are uniformly named
   table_the_id_came_from_id

There are a few notable exceptions to this rule.  If the ID is from a table that is prefixed with galaxy_, for example, galaxy_user or galaxy_session, the  galaxy_ will be dropped from the column name.  For example, galaxy_user.id becomes user_id in the over 50 tables it is imported into

### Relationship tables

As mentioned previously, some tables, such as history_dataset_association represent relationships between things, rather than things themselves.  In this case history_dataset_association describes relationships between datasets and histories.

Relationship table names typically contain the names of tables they are relating, suffixed with `_association`.

Why are nulls allowed in almost every column?
We have no idea.  In practice, they aren’t nulls in most of those columns.

Why aren’t there comments, on anything?
PostgreSQL supports comments to table definitions, but there are none shown in the SchemaSpy report. Why? The table definitions are actually generated by SQLAlchemy, the object-relational mapping software used by Galaxy, and SQLAlchemy does not support it.

There is nothing in the database that results from direct manipulation of the table definitions through DDL.  Everything comes in through SQLAlchemy.

## Start Docker and Galaxy

> <hands-on-title></hands-on-title>
>
>   1. Start the Galaxy Docker Image -  this time as an interactive session
>
>    ```sh
>       docker run -i -t -p 8080:80 bgruening/galaxy-stable /bin/bash
>    ```
>
>   2. Start Galaxy and its PostgreSQL server
>
>    ```sh
>       startup > log 2>&1 &
>    ```
>
>   3. Follow the startup process
>
>    ```sh
>       tail -f log
>    ```
{: .hands_on}

## Important tables

> <hands-on-title></hands-on-title>
>
>   1. Connect to the PostgreSQL database (change to user galaxy first)
>
>    ```sh 
>        su galaxy
>        psql -d galaxy -U galaxy
>    ```
>
>   2. List all tables
>
>    ```sql
>       \dt
>    ```
{: .hands_on}


Enter `q` to exit the view results page, and space to see the next results page.

### Table “galaxy_user”

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from galaxy_user;
>    ```
{: .hands_on}

As described in Björn’s introduction, an Admin user is already pre-set (email: ‘admin@galaxy.org’, password: ‘admin’). Now let’s add (i.e. register) a new user via the Galaxy website. And check the database:

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from galaxy_user;
>    ```
{: .hands_on}

### Table “job”

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from job;
>    ```
{: .hands_on}

Run a few jobs on the galaxy website (e.g _upload file_ a simple table and _add column_ with ‘Iterate’ no and yes) and check the database again:

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from job \x\g\x
>    ```
{: .hands_on}

### Table “job_parameter”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from job_parameter;
>   ```
{: .hands_on}


### Table “history”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from history;
>   ```
{: .hands_on}

Give your current history a name and check the database again.


### Table “dataset”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from dataset;
>   ```
{: .hands_on}

### Table “history_dataset_association”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from history_dataset_association;
>   ```
{: .hands_on}


## More (hands-on) Examples, not covered by the reports app

Have a look at the reports up (which is also provided in the Docker Image):

http://admin:admin@localhost:8080/reports/

Depending on your local needs, some queries are missing, like:

## Jobs per tool per year  /  jobs per tool since 2015

You can add the numbers per month from the reports, or:

> <hands-on-title></hands-on-title>
>   ```sql
>       select j.id, j.create_time from job j limit 5;
>   ```
>
>   ```sql
>        select j.id, j.create_time from job j
>            where j.create_time >= '2015-12-31'
>            and j.create_time < '2016-12-31';
>    ```
>
>   ```sql
>        select j.id,j.create_time from job j
>           where EXTRACT(year FROM j.create_time) = 2016
>           and j.tool_id='upload1';`
>   ```
>
>   ...and now include the user
>
>   ```sql
>        select count(j.id) from job j, galaxy_user u
>           where j.user_id = u.id
>           and u.email = 'hansrudolf.hotz@fmi.ch'
>           and EXTRACT(year FROM j.create_time) = 2016
>           and j.tool_id='upload1';
>   ```
>
>   ```sql
>        select u.email, count(*) from job j, galaxy_user u
>           where j.user_id = u.id
>           and EXTRACT(year FROM j.create_time) = 2016
>           and j.tool_id='upload1'
>           GROUP BY u.email;
>   ```
{: .hands_on}

### Jobs per tool of a certain version

Imagine the current version of a tool is working fine, however a previous version had a bug: now you wanna warn
all the users who have used the broken version, without alerting users who never used the broken one.

The following example is from the development server at the FMI

> <hands-on-title></hands-on-title>
>   ```sql
>       select distinct(j.tool_version) from job j
>           where j.tool_id = 'qAlign';
>   ```
>
>   ```sql
>        select j.user_id from job j
>           where j.tool_id = 'qAlign'
>           and j.tool_version = '1.0.4quasr';
>   ```
>
>   ```sql
>       select u.email, j.create_time from job j, galaxy_user u
>           where j.user_id = u.id
>           and j.tool_id = 'qAlign'
>           and j.tool_version = '1.0.4quasr';
>   ```
{: .hands_on}


## All users running a job using a certain parameter

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select jp.name, jp.value  from job_parameter jp
>           where name = 'iterate'`
>   ```
>
>   ```sql
>       select u.email, jp.name, jp.value
>           from job_parameter jp, job j, galaxy_user u
>           where jp.name = 'iterate'
>           and j.tool_id = 'addValue'
>           and jp.job_id = j.id
>           and j.user_id = u.id;
>   ```
>
{: .hands_on}

## Close PostgreSQL client and quit docker

> <hands-on-title></hands-on-title>
>   Close the PostgreSQL client
>
>   ```sql
>       \q
>   ```
>
>   Quit the interactive docker (change back to root first)
>
>   ```sh
>       exit
>       exit
>   ```
{: .hands_on}

## Other Topics

### How to move from MySQL to PostgreSQL

Slides: https://docs.google.com/presentation/d/1N3BDNQT3s7eQEO3BO89TQbTYwKp92fxHDRWQwC-T1kA

https://wiki.galaxyproject.org/Community/Log/2015/MySQL2PostgreSQL

### Is there ever a need to manually change the contents of a table

Slides:
https://docs.google.com/presentation/d/1l4DD0IaJjuvk1zAT1Sjv26bLyrSOg3VUm7rD-TQl_Zs

### Run SchemaSpy in this container

To run SchemaSpy in your container you’ll need to get it, and also install some required software packages.

> <hands-on-title>Schema Spy</hands-on-title>
>   ```sh
>   wget https://github.com/schemaspy/schemaspy/releases/download/v6.1.0/schemaspy-6.1.0.jar
>   apt-get update
>   apt-get install libpostgresql-jdbc-java
>   apt-get install graphviz
>   ```
>
>    To run SchemaSpy:
>   ```
>   java -jar schemaspy-6.1.0.jar -t pgsql -db galaxy -u galaxy -host localhost -s public -dp /usr/share/java/postgresql-jdbc4-9.2.jar -o SpyOut
>   ```
>
{: .hands_on}

The SpyOut directory will contain the generated reports and diagrams, anchored at index.html.



# Conclusion

There is a lot of information stored in the Galaxy database. Use this information for trouble shooting when necessary and use it as a source for extended user statistics.


topic: admin
tutorial: subdomain
title: Create a subdomain for your community on UseGalaxy.eu
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/subdomain/tutorial.html
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["subdomain", "community", "earth-system", "interactive-tools"]
content_hash: 7b83df8edab9fd7684d65f5f87b0e4d4
last_modified: 2026-05-15T22:28:10.046385
workflows_json: []
content:
This tutorial covers how to set up a subdomain on usegalaxy.eu. We will take here the example of the [earth system subdomain](https://earth-system.usegalaxy.eu/) and follow the step one by one. 

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Add your subdomain on a Galaxy server

> <hands-on-title>Subdomain name</hands-on-title>
>    - Clone the Github repository of the Galaxy server (of where you want your subdomain to be attached)
>    - For Galaxy europe clone the [Infrastructure repo](https://github.com/usegalaxy-eu/infrastructure)
>    - Create a branch on your fork
>    - Open the file **dns.tf** and edit it 
>    - You should add the name of your subdomain to the list my_new_and_shiny_subdomain.usegalaxy.eu or for the example here earth-system.usegalaxy.eu AND update **count** (number of subdomains) as shown below.
> 
> ![Image of the modification to make in the dns.tf file](../../images/create_subdomain/earth_system_subdo_infra.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}


# Enable the interactive tools 

> <hands-on-title>Galaxy Interactive Tool</hands-on-title>
>    - Clone the Github repository of the Galaxy server (of where you want your subdomain to be attached)
>    - For Galaxy europe clone the [Infrastructure-playbook repo](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master)
>    - Create a branch on your fork
>    - Open the file **sn06.yml** and edit it 
>    - You should add the name of your subdomain to the list "*.my_new_and_shiny_subdomain.usegalaxy.eu" or for the example here "*.earth-system.usegalaxy.eu" 
>    - AND add "*.interactivetoolentrypoint.interactivetool.earth-system.usegalaxy.eu" as shown below.
> 
> ![Image of the modification to make in the sn06.yml file](../../images/create_subdomain/earth_system_subdo_gxit.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

> <hands-on-title>Add the interactive tool section to the tool panel</hands-on-title>
>    - First, on your **Infrastructure-playbook** fork, create a new branch
>    - Go to **templates/galaxy/config/**
>    - Open the file **global_host_filters.py.j2** and edit it 
>    - In the function per_host_tool_labels add your subdomain as followed
> ```
> if "earth-system.usegalaxy.eu" in host:
>   return label.id in valid_labels"
> ```
>   - Then in the **DOMAIN_SECTIONS** add the the interactive tool section (this will allow to see on the subdomain interface the tool section "Interactive tools") as folowed:
> `'earth-system': ["interactivetools"],`
>
> ![Image of the modification to make in the global_host_filters.py.j2 file](../../images/create_subdomain/add_gxit.png)
>
>   - Second, on the same fork go to **files/traefik/rules/**
>   - Open and edit **template-subdomains.yml**, there you need to add the folowing line for your subdomain `{% raw %}{{template "subdomain" "my_new_and_shiny_subdomain"}}{% endraw %}`
>
>   - Finally, commit all your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}


# Make a nice tool panel (only for interactive tools)

> <hands-on-title>Customize the tool panel (only for the interactive tools)</hands-on-title>
>    - On your **Infrastructure-playbook** fork (don't forget to synchronize it)
>    - First, go to **templates/galaxy/config/**
>    - Open the file **tool_conf.xml.j2** and edit it 
>    - From there you can create your own tool sections where your different interactive tools can be organized. See the example of earth-system below.
> ![Image of the modification to make in the tool_conf.xml.j2 file](../../images/create_subdomain/customize_tool.png)
>
>   - Then, commit your changes
>   
>   - Secondly, still in **templates/galaxy/config/**
>   - Open the file **global_host_filters.py.j2** and edit it 
>   - Add the different new sections of tools created to your subdomain and prevent those new sections to appear in any other subdomain as shown below.
>  ![Image of the modification to make in the global_host_filters.py.j2 file to add your tool sections](../../images/create_subdomain/add_customize_tool.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

# Customize the front page 

> <hands-on-title>Update the homepage</hands-on-title>
>    - For Galaxy europe clone the [website repo](https://github.com/usegalaxy-eu/website/tree/master)
>    - Create a branch on your fork
>    - Create an index file in markdown for instance "index-earth-system.md"
>    - Once created let your imagnination flows to make the looks and feel of your homepage (if needed here is a [PR example](https://github.com/usegalaxy-eu/website/pull/1149))
>    - If you want to add images to your front page add them as folowed **assets/media/my_incredidle_image.png**
>
>   - Then commit your changes, write a nice message for the admin when you open your Pull Request.
> If in the future you want to change the front page just update the index file.
{: .hands_on}

> <tip-title>Add a custom welcome.html</tip-title>
> There is also the possibility to add a custom welcome.html, as well as other custom static files like the [singlecell subdomain did](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master/files/galaxy/subdomains/singlecell/static)
{: .tip}

> <hands-on-title>Add a theme</hands-on-title>
>    - On your **Infrastructure-playbook** fork (don't forget to synchronize it)
>    - Go to **group_vars/sn06/**
>    - Open the file **subdomains.yml** and edit it. There you have to add the name of your subdomain.
>    - In this same file you can also customize a theme for your subdomain. 
{: .hands_on}

# Let people know

> <hands-on-title>Make some communication on your new subdomain</hands-on-title>
>    - Clone the [galaxy-hub repo](https://github.com/galaxyproject/galaxy-hub/tree/master)
>    - Create a branch on your fork
>    - In **content/news** create a folder for your news example "2023-10-17-earth-system" and in it create an **index.md** file 
>    - In this index file write your blog post don't hesitate to add some nice photos
>
>   - Then commit your changes and open your Pull Request.
{: .hands_on}

# Conclusion
Your Subdomain is ready to be used !


topic: admin
tutorial: tiaas
title: Training Infrastructure as a Service (TIaaS)
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/tiaas/tutorial.html
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "training", "jobs", "git-gat"]
content_hash: c2f751efc7f688e88f49c42f39e0f0e5
last_modified: 2026-05-27T09:01:47.649129
workflows_json: []
content:
Galaxy is widely used for teaching. In order to facilitate instructors, the Galaxy Project has developed {TIaaS}.
Workshop instructors can apply for {TIaaS}, and on the day of their workshop, their participants will be placed in a special group and use dedicated
resources, thus reducing queue times on the day of the training.

![TIaaS concept](../../images/tiaas/tiaas_intro.png "With TIaaS, all of your users visit the same server. In the background, the scheduler recognises which users are training users, and directs their jobs to special resources. In the EU deployment of TIaaS jobs preferentially use private resources, but can spill over to the main queue if there is not enough space available."){: width="70%"}

This tutorial will go cover how to set up such a service on your own Galaxy server.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tiaas" %}


# Setting up TIaaS

> <hands-on-title>Setup TIaaS</hands-on-title>
>
> 1. In your `requirements.yml` add the {TIaaS} ansible role:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -50,3 +50,6 @@
>       version: 6f6fdf7f5ead491560783d52528b79e9e088bd5b
>     - src: cloudalchemy.grafana
>       version: 0.14.2
>    +# Training Infrastructure as a Service
>    +- src: galaxyproject.tiaas2
>    +  version: 2.1.5
>    {% endraw %}
>    ```
>    {: data-commit="Add tiaas2 requirement"}
>
>    And run the install step:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. In your `galaxyservers` group variables file, add the following:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -350,3 +350,8 @@ telegraf_plugins_extra:
>           - timeout = "10s"
>           - data_format = "influx"
>           - interval = "15s"
>    +
>    +# TIaaS setup
>    +tiaas_dir: /srv/tiaas
>    +tiaas_admin_user: admin
>    +tiaas_admin_pass: changeme
>    {% endraw %}
>    ```
>    {: data-commit="Configure tiaas"}
>
> 2. In the `galaxyservers` group variables file, we also need to set the database permissions correctly for TIaaS. It needs to be able to access some Galaxy tables, and we will carefully define only the ones we really need:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/dbservers.yml
>    +++ b/group_vars/dbservers.yml
>    @@ -3,6 +3,7 @@
>     postgresql_objects_users:
>       - name: "{{ galaxy_user_name }}"
>       - name: telegraf
>    +  - name: tiaas
>     postgresql_objects_databases:
>       - name: "{{ galaxy_db_name }}"
>         owner: "{{ galaxy_user_name }}"
>    @@ -11,7 +12,26 @@ postgresql_objects_privileges:
>         roles: telegraf
>         privs: SELECT
>         objs: ALL_IN_SCHEMA
>    -
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: galaxy_user,galaxy_session,job,history,workflow,workflow_invocation
>    +    type: table
>    +    privs: SELECT
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: user_group_association,galaxy_group,role,group_role_association
>    +    type: table
>    +    privs: SELECT,INSERT
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: group_role_association
>    +    type: table
>    +    privs: DELETE
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: role_id_seq,galaxy_group_id_seq,group_role_association_id_seq,user_group_association_id_seq
>    +    type: sequence
>    +    privs: USAGE,SELECT
>
>     # PostgreSQL Backups
>     postgresql_backup_dir: /data/backups
>    {% endraw %}
>    ```
>    {: data-commit="Add database privileges for TIaaS"}
>
>    > <tip-title>Why does TIaaS get `DELETE` privileges on Galaxy's Database?</tip-title>
>    > The `DELETE` privilege is limited in scope to one table: `group_role_association`. This allows TIaaS to
>    > disassociate training groups from roles in the Galaxy database after the training event date has passed, so that
>    > users who participated in a training return to using normal (non-training) resources after the training ends.
>    >
>    > The `galaxyproject.tiaas2` role will create a [cron](https://manpages.debian.org/stable/cron/cron.8.en.html) job
>    > to perform this process every night at midnight. You can control when this runs (or disable it) using
>    > [the tiaas_disassociate_training_roles variable](https://github.com/galaxyproject/ansible-tiaas2/blob/d5be2a064c49e010f67bfcea18e36812da23d7d8/defaults/main.yml#L20).
>    >
>    {: .tip}
>
>    > <tip-title>Running the playbook from scratch</tip-title>
>    > This is one of the few statements we've provided that presents difficulties when running the playbook completely from scratch on a blank machine. Setting postgresql roles is one of the first steps in our playbook, but the rules we've provided above depend on the Galaxy tables existing in that database. If those tables aren't there, it will fail. If you do someday run this from scratch, you'll find that you need to comment out those roles.
>    {: .tip}
>
> 3. We need to add the `galaxyproject.tiaas2` role before the `nginx` role, as TIaaS defines variables that Nginx needs.
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -47,6 +47,7 @@
>         - galaxyproject.nginx
>         - geerlingguy.docker
>         - usegalaxy_eu.rabbitmqserver
>    +    - galaxyproject.tiaas2
>         - galaxyproject.gxadmin
>         - galaxyproject.cvmfs
>         - dj-wasabi.telegraf
>    {% endraw %}
>    ```
>    {: data-commit="Add TIaaS role to the Galaxy playbook"}
>
> 4. Lastly we should add the routes for TIaaS to the NGINX template for Galaxy. TIaaS provides a set of default nginx routes that can be used.
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -114,4 +114,7 @@ server {
>     		proxy_pass http://127.0.0.1:3000/;
>     		proxy_set_header Host $http_host;
>     	}
>    +
>    +	{{ tiaas_nginx_routes }}
>    +
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Add nginx routes for TIaaS"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}


{TIaaS} should be available now! The following routes on your server are now configured (we will run through these in the next section)


| URL                                                                                       | Use                                                                         | Audience                 |
| :----                                                                                     | ----                                                                        | -----                    |
| [/tiaas/new/](https://my.gat.galaxy.training/?path=/tiaas/new/)                           | Request a new TIaaS training                                                | Instructors              |
| [/tiaas/admin/](https://my.gat.galaxy.training/?path=/tiaas/admin/)                       | Approve and Manage requests                                                 | Admin                    |
| [/tiaas/stats/](https://my.gat.galaxy.training/?path=/tiaas/stats/)                       | Overall TIaaS statistics ([EU Stats](https://usegalaxy.eu/tiaas/stats/))    | Admins, Funding Agencies |
| [/tiaas/calendar/](https://my.gat.galaxy.training/?path=/tiaas/calendar/)                 | Calendar of trainings ([EU Calendar](https://usegalaxy.eu/tiaas/calendar/)) | Admins, Funding Agencies |
| [/join-training/ID](https://my.gat.galaxy.training/?path=/join-training/ID)               | Join an TIaaS training                                                      | Participants             |
| [/join-training/ID/status](https://my.gat.galaxy.training/?path=/join-training/ID/status) | Dashboard with job states of trainees.                                      | Instructors              |


Let's see it in action!

> <hands-on-title>Using TIaaS</hands-on-title>
>
> 1. **Create a new TIaaS request**
>    - Go to [/tiaas/new](https://my.gat.galaxy.training/?path=/tiaas/new)
>    - Here you will find the request form users will fill in to request TIaaS:
>      ![TIaaS request form](../../images/tiaas/tiaas_request_form.png)
>    - For *"Training Identifier"*, fill in `gat`
>      - This is the `<training-id>` used in the URLs listed above used for:
>        1. Workshop participants to join the tiaas group
>        2. Workshop instructors to monitor the progress of their participants.
>    - Fill in the rest of the form as you like
>    - Submit the form and you should see a confirmation dialog:
>      ![TIaaS requested successfully](../../images/tiaas/tiaas_request_form_submitted.png)
>
> 2. **Approve TIaaS request**
>    - Next, the request will have to be approved by an admin
>    - Go to [/tiaas/admin](https://my.gat.galaxy.training/?path=/tiaas/admin)
>    - **Log in** using the values you configured `tiaas_admin_user` and `tiaas_admin_pass` in your group variables file
>      - Default values were `admin:changeme`
>    - You should now see the admin panel:
>      ![TIaaS admin console](../../images/tiaas/tiaas_admin_console.png)
>    - Click on **Trainings**, you should see the TIaaS request listed here:
>      ![TIaaS request list](../../images/tiaas/tiaas_request_list.png)
>    - **Approve the request**
>      - Click on the training
>      - Scroll down to the bottom
>      - Change *"Processed"* to `Approved` and **Save**
>        ![Approve TIaaS](../../images/tiaas/tiaas_request_approve.png)
>    - At this point, you would likely email the person who made the request to inform them of approval
>
> 3. **Join TIaaS Training**
>    - Make sure you are logged in to Galaxy
>    - On the day of the workshop, participants will visit a following URL to join the TIaaS group
>      - [/join-training/gat](https://my.gat.galaxy.training/?path=/join-training/gat)
>      - A confirmation dialog should appear if all went well:
>        ![Join TIaaS](../../images/tiaas/tiaas_join_training.png)
>
> 4. **Monitor TIaaS status**
>    - This is very useful for instructors to monitor the job state of their participants
>    - Go to [/join-training/gat](https://my.gat.galaxy.training/?path=/join-training/gat)
>    - In the Dasboard you should see that one user (you) has joined the training \
>    - Run some jobs to see the dashboard in action
>      ![TIaaS dashboard](../../images/tiaas/tiaas_dashboard.png)
>    - Scroll down to get some more information on a per-user level (anonymized)
>      - Every user designated by their own identifier and colour, but no personal information
>      ![TIaaS dashboard](../../images/tiaas/tiaas_dashboard2.png)
>
{: .hands_on}


> <comment-title>Note: GDPR assistance</comment-title>
>
> Since this setup tracks additional personal information (submitter name & email, users in the queue view), TIaaS includes some always-on features to assist with your GDPR compliance.
>
>  - Users in public status dashboard are only visible by an anonymized identifier and colour
>  - Email addressses in the TIaaS admin panel will be automatically expunged 60 days after a training event
>
>  Of course you need to review any GDPR compliance concerns with your group's legal representative(s), this only attempts to ensure some protections exist for the users of the system.
>
{: .comment}


# Job Configuration

While observability for teachers or trainers is already a huge benefit, one of the primary benefits of {TIaaS} is that your jobs get sent to dedicated compute resources, which won't be used by anyone else, during the period of the training. We will send all of the training jobs to pulsar if you have completed that tutorial, or one of the slurm destinations from the job configuration training.

In order to achieve this, we first need some way to sort the jobs of the training users into these private queues, while letting the other jobs continue on. So let's create a *traffic controller* to figure out where jobs belong.


> <hands-on-title>Writing a dynamic job destination</hands-on-title>
>
> 1. This destination will check that the `user_email` is in a training group (role starting with `training-`).
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -35,6 +35,15 @@ tools:
>           require:
>             - pulsar
>
>    +roles:
>    +  training.*:
>    +    max_cores: 2
>    +    max_mem: max_cores * 3.8  # TODO check multiplier
>    +    scheduling:
>    +      require:
>    +        - slurm
>    +        - training
>    +
>     destinations:
>       local_env:
>         runner: local_runner
>    @@ -64,6 +73,19 @@ destinations:
>         max_mem: 8
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)} --time={entity.params['walltime']}:00:00
>    +  slurm-training:
>    +    inherits: singularity
>    +    runner: slurm
>    +    max_accepted_cores: 12
>    +    max_accepted_mem: 120
>    +    max_cores: 2 # Limit the cores
>    +    max_mem: 8 # Limit the memory
>    +    params:
>    +      native_specification: --nodes=1 --ntasks=1 --mem={round(mem*1024)} --cpus-per-task={cores} --time=00:30:00
>    +    scheduling:
>    +      require:
>    +        - slurm
>    +        - training
>
>       pulsar:
>         runner: pulsar_runner
>    {% endraw %}
>    ```
>    {: data-commit="Add to list of deployed rules"}
>
> 6. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 7. Ensure your user is joined to a training
>
> 8. Run a job and observe the logs to see where it goes (`journalctl -u galaxy -f`)
>
{: .hands_on}

Congratulations! you have now set up {TIaaS} on your Galaxy server.

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=14 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tiaas" %}


topic: admin
tutorial: cvmfs-manual
title: Reference Data with CVMFS without Ansible
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 5a50ccc1a8ba6fb2ce7874b1eed2a3fb
last_modified: 2026-05-27T09:01:47.635654
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/cvmfs-manual/tutorial.html

content:
# Overview


The CernVM-FS is a distributed filesystem perfectly designed for sharing readonly data across the globe. We use it in the [Galaxy Project](https://galaxyproject.org) for sharing things that a lot of Galaxy servers need. Namely:
* **Reference Data**
    * Genome sequences for hundreds of useful species.
    * Indices for the genome sequences
    * Various bioinformatic tool indices for the available genomes
* **Tool containers**
    * [Singularity](https://www.sylabs.io/) containers of everything stored in [Biocontainers](https://biocontainers.pro/) (A bioinformatic tool container repository.) You get these for free every time you build a [Bioconda](https://bioconda.github.io/) recipe/package for a tool.
* Others too..

From the Cern website:

> The CernVM File System provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs."
>
{: .quote cite="https://cernvm.cern.ch/portal/filesystem"}

[A slideshow presentation on this subject]({% link topics/admin/tutorials/cvmfs/slides.html %}) is available. More [details are available on usegalaxy.org (Galaxy Main's) reference data setup](https://galaxyproject.org/admin/reference-data-repo/#usegalaxyorg-reference-data) and CVMFS system.

This exercise describes a manual process to install and configure CVMFS and Galaxy's access to CVMFS. For a tutorial that uses Ansible to perform these tasks, see [the Reference Data with CVMFS tutorial]({% link topics/admin/tutorials/cvmfs/tutorial.md %}).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}



# CVMFS and Galaxy without Ansible

> <comment-title>Manual version of Ansible Commands</comment-title>
> If you wish to perform the same thing that we've just done, but by building the ansible script manually, follow these instructions. Otherwise, you have already done everything below and do not need to re-do it.
{: .comment}

We are going to setup a CVMFS mount to the Galaxy reference data repository on our machines. To do this we have to install and configure the CVMFS client and then mount the appropriate CVMFS repository using the publicly available keys.

> <hands-on-title>Installing the CVMFS Client</hands-on-title>
>
> 1. On your remote machine, we need to first install the Cern software apt repo and then the CVMFS client and config utility:
>
>    ```bash
>    sudo apt install lsb-release
>    wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb
>    sudo dpkg -i cvmfs-release-latest_all.deb
>    rm -f cvmfs-release-latest_all.deb
>    sudo apt-get update
>
>    sudo apt install cvmfs cvmfs-config
>    ```
>
> 2. Now we need to run the CVMFS setup script.
>
>    ```bash
>    sudo cvmfs_config setup
>    ```
>
{: .hands_on}


## Configuring CVMFS

The configuration is not complex for CVMFS:

> <hands-on-title>Configuring CVMFS</hands-on-title>
>
> 1. Create a `/etc/cvmfs/default.local` file with the following contents:
>
>    ```
>    CVMFS_REPOSITORIES="data.galaxyproject.org"
>    CVMFS_HTTP_PROXY="DIRECT"
>    CVMFS_QUOTA_LIMIT="500"
>    CVMFS_CACHE_BASE="/srv/cvmfs/cache"
>    CVMFS_USE_GEOAPI=yes
>    ```
>
>    This tells CVMFS to mount the Galaxy reference data repository and use a specific location for the cache which is limited to 500MB in size and to use the instance's geo-location to choose the best CVMFS repo server to connect to. You can use the `cvmfs_quota_limit` role variable to control this setting.
>
> > <tip-title>What is a good size for CVMFS_QUOTA_LIMIT?</tip-title>
> > In production UseGalaxy.org.au uses 100GB, different sites have different needs and you can make your cache smaller depending on your usage. E.g. if your users only use one dataset from the reference data (e.g. just hg38) then perhaps you don't need such a large cache.
> {: .tip}
>
> 2. Create a `/etc/cvmfs/domain.d/galaxyproject.org.conf` file with the following contents:
>
>    ```
>    CVMFS_SERVER_URL="http://cvmfs1-tacc0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-iu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-psu0.galaxyproject.org/cvmfs/@fqrn@;http://galaxy.jrc.ec.europa.eu:8008/cvmfs/@fqrn@;http://cvmfs1-mel0.gvl.org.au/cvmfs/@fqrn@;http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/@fqrn@"
>    ```
>
>    This is a list of the available stratum 1 servers that have this repo.
>
> 3. Create a `/etc/cvmfs/keys/data.galaxyproject.org.pub` file with the following contents:
>
>    ```
>    -----BEGIN PUBLIC KEY-----
>    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5LHQuKWzcX5iBbCGsXGt
>    6CRi9+a9cKZG4UlX/lJukEJ+3dSxVDWJs88PSdLk+E25494oU56hB8YeVq+W8AQE
>    3LWx2K2ruRjEAI2o8sRgs/IbafjZ7cBuERzqj3Tn5qUIBFoKUMWMSIiWTQe2Sfnj
>    GzfDoswr5TTk7aH/FIXUjLnLGGCOzPtUC244IhHARzu86bWYxQJUw0/kZl5wVGcH
>    maSgr39h1xPst0Vx1keJ95AH0wqxPbCcyBGtF1L6HQlLidmoIDqcCQpLsGJJEoOs
>    NVNhhcb66OJHah5ppI1N3cZehdaKyr1XcF9eedwLFTvuiwTn6qMmttT/tHX7rcxT
>    owIDAQAB
>    -----END PUBLIC KEY-----
>    ```
>
> 4. Make a directory for the cache files
>
>    ```
>    sudo mkdir /srv/cvmfs
>    ```
{: .hands_on}


## Testing it out

Probe the connection.

> <hands-on-title>Testing it out</hands-on-title>
>
> 1. Run `sudo cvmfs_config probe data.galaxyproject.org`
>
>    > <question-title></question-title>
>    >
>    > What does it output?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > OK
>    > > ```
>    > >
>    > > If this doesn't return `OK` then you may need to restart autofs: `sudo systemctl restart autofs`
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
> 2. Change directory into `/cvmfs/` and list the files in that folder
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > You should see nothing, as CVMFS uses `autofs` in order to mount paths only upon request.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>
> 3. Change directory into `/cvmfs/data.galaxyproject.org/`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > cd /cvmfs/data.galaxyproject.org/
>    > ls
>    > ls byhand
>    > ls managed
>    > ```
>    {: .code-in}
>
>    > <question-title></question-title>
>    >
>    > What do you see now?
>    >
>    > > <solution-title></solution-title>
>    > >  You'll see `.loc` files, genomes and indices.
>    > > AutoFS only mounts the files when they're accessed, so it appears like there is no folder there.
>    > {: .solution }
>    >
>    {: .question}
>
>    And just like that we all have access to all the reference genomes and associated tool indices thanks to the Galaxy Project, IDC, and Nate's hard work!
>
>    > <tip-title>Contributing Reference Genomes</tip-title>
>    > If you are developing a new tool, and want to add a reference genome, we recommend you [talk to us on Gitter](https://gitter.im/galaxy-iuc/iuc). You can also look at one of the tools that uses reference data, and try and copy from that. If you’re developing the location files completely new, you need to write the data manager.
>    {: .tip}
>
{: .hands_on}

## Look at the repository

Now to configure Galaxy to use the CVMFS references we have just installed, see [the Ansible tutorial.]({% link topics/admin/tutorials/cvmfs/tutorial.md %}#configuring-galaxy-to-use-the-cvmfs-references)


topic: admin
tutorial: customization-manual
title: Customizing the look of Galaxy (Manual)
difficulty: intermediate
time_estimation: 45m
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 1e2e4e8bafa3707f0c116e4681b0b7bd
last_modified: 2026-05-27T09:01:47.634569
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization-manual/tutorial.html

content:
Customizing your Galaxy instance makes it more recognizable at a glance, and can help communicate its purpose to its users.
This tutorial will teach you three basic customizations you can make to Galaxy:

  - Setting a brand text
  - Adding a custom welcome page
  - Customizing the masthead using themes

For this tutorial we will be setting up a fictional instance called "Galaxy Mars".
Feel free to use the included material in the hands-on sections, or provide your own.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Custom Branding

The brand text in Galaxy refers to the text you can see in the masthead of some Galaxy instances.
This text will appear in the masthead, as well as the sites title.
It is an easy way to set your instance apart, and make it more identifiable.

![screenshot of the start page of Galaxy Europe](images/galaxy-europe.png "Galaxy Europe uses the brand text \"Europe\"")

> <hands-on-title>Customising the Branding</hands-on-title>
> 
> 1. Open your galaxy config file `config/galaxy.yml` and add the following option under `galaxy:`
> 
>    ```yaml
>    brand: Mars
>    logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>    ```
>
> 2. Start your galaxy server using `sh run.sh`
>
> 3. Navigate to `https://localhost:8080`
>
>    ![screenshot of fictional "Galaxy Mars" start page, with the brand text set to "Mars"](images/galaxy-mars-brand.png "Your Galaxy start page should now look like this")
{: .hands_on}

## Custom Welcome Page

The welcome page is an html document embedded in Galaxy's start page.
It is what users see when first navigating to your instance.
This page can be used to communicate what your instance is about, and share news and updates with your users.

> <hands-on-title>Creating a custom welcome page</hands-on-title>
>
> 0. (optional) Set the location of your welcome page in `galaxy.yml`
> 
>    Under `galaxy:` add the following option:
>
>    ```yaml
>    welcome_url: /static/your-welcome-page.html
>    ```
>
>    Not setting this will cause the url to default to `/static/welcome.html`
>
> 1. Open `/static/welcome.html`, or create a new html file at the location set in Step 0
>
>    It is advisable to keep a copy of the default `welcome.html`, as it can serve as a useful reference.
>
> 2. Add some Content
>
>    Let's add a title and some text to our welcome page:
>
>    ```html
>    <!DOCTYPE html>
>    <html lang="en">
>        <head>
>            <meta charset="utf-8">
>        </head>
>        <body>
>            <h1>Welcome to the Galaxy Mars instance!</h1>
>            <p>The only Galaxy instance on mars.</p>
>        </body>
>    </html>
>    ```
>
>    This works the same as any other html page. You can add styles in the head, or import some scripts.
>    
>    Make sure your welcome page has exactly one `h1` element, which describes the page.
>    This will act as the heading for your start-page, which can help assistive technologies.
{: .hands_on}

> <tip-title>Use an IFrame!</tip-title>
> A lot of the UseGalaxy.* instances choose to use their `welcome.html` to display an IFrame to a different site,
> e.g. the GalaxyProject.org site, or a sub-site thereof.
>
> This makes it easier for non-administrators to keep the homepage's contents
> up to date. By embedding (for example) a Wordpress blog in the center,
> non-technical contributors can still publish articles and announcements
> without having to re-deploy Galaxy every time.
{: .tip}

> <tip-title>Styling your Welcome</tip-title>
> As in our example welcome.html, we've loaded the default Galaxy CSS to keep
> it consistent with the surrounding material. If you wish to load a different
> CSS file, or set a different font for the main panel, you can easily do that
> by adding whatever CSS you like.
{: .tip}

## Custom Masthead Theme

Galaxy's theming system allows you to easily change the color of your Masthead, to give your instance an even more distinct look.
You can even offer several options, to allow users to switch to the default if they prefer it, or another look all together.

> <hands-on-title>Configuring Themes</hands-on-title>
>
> 0. (optional) Set the location of the themes configuration `galaxy.yml`
> 
>    Under `galaxy:` add the following option:
>
>    ```yaml
>    themes_config_file: your-themes-file.yml
>    ```
>
>    This location is relative to the config directory.
>    If unset, the default file `themes_conf.yml` will be used, if present.
>
> 1. Create your themes config file.
>
>    Open `themes_conf.yml.sample`, and copy its contents into a new file, either called `themes_conf.yml`, or the custom path you set in step 0.
>    Do not copy the conf file and remove the ending, as this file is a symbolic link.
>    Copying it copies a reference to the source file, not its contents.
>
> 2. Understanding Themes
>
>    The themes file can contain several themes.
>    Each theme is identified by its id, eg `blue`, and followed by a set of rules, which style the client.
>    The first theme in your theme file will be used as the default theme, which users see when visiting your instance,
>    and which users who haven't logged in will see.
>
>    At the time of writing, themes only support styling the masthead, but this is subject to change.
>    The first example theme should contain all available values you can change.
>
> 3. Creating our own Theme
>
>    Let's create a theme more fitting for our Mars instance.
>
>    Create a new theme, by adding `mars:` to the top of the themes config file.
>    Now add `masthead:` underneath, and then `color:`.
>    Set color to `"#e03e1d"`, a nice deep martian red.
>
>    Your theme file should now begin with:
>
>    ```yaml
>    mars:
>      masthead:
>        color: "#e03e1d"
>    ```
>
>    Restart your instance, and look at the masthead. It should be colored red.
>
> 4. You can also try logging in, and changing your theme under `Preferences`
{: .hands_on}


topic: admin
tutorial: stop-worrying-love-systemd
title: How I learned to stop worrying and love the systemd
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "systemd"]
content_hash: a6bb56fe92a01590e146f966051598e0
last_modified: 2026-05-15T22:28:10.045478
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/stop-worrying-love-systemd/tutorial.html

content:
Many linux sysadmins with years and years of experience bemoan systemd ("it's infecting everything! Now it wants to mess with time? And DNS???") and journalctl ("unix was supposed to be about files!") and while those are fair complaints and make systemd and friends wildly more opaque than traditional SysV and logging to files, there are some benefits that can be obtained, and may be interesting even to the wise old admins. There is a lot of convenience in systemd that can make the tradeoffs worth it.

This tutorial assumes a working knowledge of unix systems (files, directories, services, logging, `/var/log`, etc.).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# systemd

## Why systemd units, and not service shell scripts?

- CGroups
- Hard Memory Limits that include all children (also CPU, disk.)
- Dependencies (is your database up, before your service that needs it?)
- Fancy restarting (always/n times, delays, timeouts.)
- [Monitor it with Telegraf](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/systemd_units)

## Unit Files

```
[Unit]
Description=Galaxy
After=network.target
After=time-sync.target

[Service]
UMask=022
Type=simple
User=galaxy
Group=galaxy
WorkingDirectory=/srv/galaxy/server
TimeoutStartSec=60
ExecStart=/srv/galaxy/venv/bin/galaxyctl start --foreground --quiet

Environment=HOME=/srv/galaxy
Environment=VIRTUAL_ENV=/srv/galaxy/venv

MemoryAccounting=yes
MemoryLimit=16G
CPUAccounting=yes
IOAccounting=yes

[Install]
WantedBy=multi-user.target
```

Here's a simple unit file. It has a name, and some targets that must be up before it should be started, then the service definition featuring all of the common fields like:

- User/group and umask restriction
- Working directory
- Process to start (and optionally reload/stop commands.)
- Some environment variables
- Enabling of CPU and Mmeory accounting
- As well as limiting the memory to 16GB, at which time the child will be killed and restarted.

Unit files are very easy to create, be sure to see the [examples section](https://man.archlinux.org/man/systemd.service.5#EXAMPLES) of the documentation, and stick with `Type=simple` services which stay in the foreground for maximum convenience.


## Status

Status is generally your first step looking at any unit file.

```bash
$ systemctl galaxy
● galaxy.service - Galaxy
     Loaded: loaded (/etc/systemd/system/galaxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-06-28 15:17:04 CEST; 5 days ago
   Main PID: 217446
      Tasks: 73 (limit: 9521)
     Memory: 1.7G (limit: 16.0G)
        CPU: 18h 16min 20.685s
     CGroup: /system.slice/galaxy.service
             ├─217446 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/galaxyctl start --foreground --quiet
             ├─217467 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/supervisord -c /srv/galaxy/var/gravity/supervisor/supervisord.conf --nodaemon
             ├─217484 /bin/tail -f /srv/galaxy/var/gravity/supervisor/supervisord.log
             ├─217487 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/gunicorn galaxy.webapps.galaxy.fast_factory:factory() --timeout 300 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker -b unix:/srv/galaxy/var/config/gunicorn.sock --workers=2 --config python:galaxy.web_stack.gunicorn_config --preload --forwarded-allow-ips=*
             ├─245320 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/gunicorn galaxy.webapps.galaxy.fast_factory:factory() --timeout 300 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker -b unix:/srv/galaxy/var/config/gunicorn.sock --workers=2 --config python:galaxy.web_stack.gunicorn_config --preload --forwarded-allow-ips=*
             ├─245322 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/gunicorn galaxy.webapps.galaxy.fast_factory:factory() --timeout 300 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker -b unix:/srv/galaxy/var/config/gunicorn.sock --workers=2 --config python:galaxy.web_stack.gunicorn_config --preload --forwarded-allow-ips=*
             ├─245334 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/celery --app galaxy.celery worker --concurrency 2 --loglevel DEBUG --pool threads --queues celery,galaxy.internal,galaxy.external
             ├─245401 /srv/galaxy/venv/bin/python -c from multiprocessing.resource_tracker import main;main(7)
             ├─245582 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/celery --app galaxy.celery beat --loglevel DEBUG --schedule /srv/galaxy/var/gravity/celery-beat-schedule
             ├─245806 python ./lib/galaxy/main.py -c /srv/galaxy/config/galaxy.yml --server-name=handler_0 --attach-to-pool=job-handler --attach-to-pool=workflow-scheduler --pid-file=/srv/galaxy/var/gravity/supervisor/handler_0.pid
             ├─246274 python ./lib/galaxy/main.py -c /srv/galaxy/config/galaxy.yml --server-name=handler_1 --attach-to-pool=job-handler --attach-to-pool=workflow-scheduler --pid-file=/srv/galaxy/var/gravity/supervisor/handler_1.pid
             └─246737 python ./lib/galaxy/main.py -c /srv/galaxy/config/galaxy.yml --server-name=handler_2 --attach-to-pool=job-handler --attach-to-pool=workflow-scheduler --pid-file=/srv/galaxy/var/gravity/supervisor/handler_2.pid
```

Helpful things to note:

- the file defining the service is listed after `Loaded`, if you want to see what it does or how it's configured, or to edit that configuration
- You can see how long it's been running
- How many tasks (children) there are
- Memory and the limit, if memory limiting is configured.
- CPU time used
- And then many of the children.

## Restarting

Here's one of the nice new things you can do, once you've moved away from SysV, and to the joyous world of systemd, restart multiple processes at once! If you have multiple Galaxy processes, use a wildcard to restart them all.

```bash
systemctl restart nginx galaxy-*
```

## Failed Units

Easily check if any service has failed with:

```bash
systemctl --failed
```

## Enabling Units

In order to start at boot time, it must be "enabled", so you can run a service once knowing it won't come back when the machine reboots (or vice versa). But you should be sure that a unit is enabled if you want it available on restart. You can do this in separate commands

```bash
systemctl enable my.service
systemctl start my.service
```

Or simultaneously, [if you're on a systemd v220 or newer](https://unix.stackexchange.com/questions/374280/the-now-switch-of-systemctl):

```bash
systemctl enable --now my.service
```

## Editing Units or Overriding

Sometimes one of the system units provided will have some weird behaviour that you need to override (or your ansible role doesn't expose it), then you can use `systemctl edit unit` to override some settings.

Additional directives can be supplied, e.g. making your service start after another service is started, if you need to sequence their starts.

Note that if it was defined in the base unit file, then you need to set it to an empty value first, before overriding. E.g. `ExecStart=\nExecStart=command...`


## Masking Units

I never needed this until one day I did. Masking a unit makes it impossible to start, but if someone's gone and installed a GUI on your server, inclusive of power management systems that trigger hibernation when not in use, then this might be a useful trick. You can mask anything, service, or even targets like I had to in that case.

This command will prevent the device from reaching or activating any of those targets. Useful for servers when you're in a bind and don't know how to remove power management:

```console
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
```

## Unit Security Optimisation

Because systemd uses cgroups, it can also give us a nice overview of any security issues that might be worth looking into. Here we see the Galaxy unit has a lot of

```console
ubuntu@gat-1:~$ systemd-analyze security galaxy
  NAME                                       DESCRIPTION                                         EXPOSURE
✗ PrivateNetwork=                            Service has access to the host's network            0.5
✓ User=/DynamicUser=                         Service runs under a static non-root user identity
....
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT      Service may issue chroot()                          0.1
✗ ProtectHostname=                           Service may change system host/domainname           0.1
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND   Service may establish wake locks                    0.1
✗ CapabilityBoundingSet=~CAP_LEASE           Service may create file leases                      0.1
✗ CapabilityBoundingSet=~CAP_SYS_PACCT       Service may use acct()                              0.1
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG  Service may issue vhangup()                         0.1
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM      Service may program timers that wake up the system  0.1
✗ RestrictAddressFamilies=~AF_UNIX           Service may allocate local sockets                  0.1

→ Overall exposure level for galaxy.service: 9.2 UNSAFE 😨
```

Here we see a lot of low hanging fruit for restricting the abilities of Galaxy's processes. This can also be applied to e.g. the job runner, or your webserver for additional security.

## Testing with systemd-run

Would you like to test some of these limits manually? Or just to run a one-off process in a very sandboxed environment? Enter `systemd-run`, which allows just that. Here you can set a lot of the capabilities or restrictions:

```bash
systemd-run -p IOAccounting=yes -p CPUAccounting=true -p MemoryAccounting=true -p TasksAccounting=true -p ProtectSystem=strict -p PrivateDevices=yes -t -S
```

And `-S` puts you in an interactive terminal, allowing you to play around with this environment. You get a cleaned out environment, working inside of a cgroup (visible in `systemd-cgtop`, and `systemctl status`) and when you exit it tells how you many resources you've used. This is both an easy way to test processes, and an option to sandbox rarely run processes, by setting an extremely restrictive environment.

## Optimising Slow Boots

systemd features an analyze command to check why your booting is slow

```console
$ systemd-analyze
Startup finished in 3.841s (firmware) + 3.925s (loader) + 11.104s (kernel) + 1min 5.331s (userspace) = 1min 24.203s
graphical.target reached after 11.547s in userspace
```

You can even have it generate a fancy plot for you!

```bash
systemd-analyze plot > plot.svg
```

![waterfall graphic of systemd boot time, cloud-init has the most, largest red bars](images/plot.svg "Notice how many units wait for network to finish being ready, and are stuck until that is ready. This image was generated from a training VM. Can you spot Galaxy? It doesn't show the red bars, because it does not currently notify systemd when it's done starting up.")

## Pro/Cons of systemd

> > <code-out-title>Pros</code-out-title>
> > - CGroups, to reap children!
> > - CGroups, for hard memory/cpu limits
> > - CGroups, for security benefits, and dropping capabilities!
> > - Service dependencies and simultaneously booting multiple units.
> > - Easily override tasks
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > - It's new and requires time to learn
> >     - But all the distros have mostly switched so, might be time to learn.
> > - It is significantly more complicated
> > - Even if that complexity does have a lot of benefits
> {: .code-in}
{: .code-2col}

## Further Reading

- [Archi wiki page](https://wiki.archlinux.org/title/Systemd)
- [man 1 systemctl](https://man.archlinux.org/man/systemctl.1)
- [man 5 systemd.service](https://man.archlinux.org/man/systemd.service.5)

# timers

Do you use cron aggressively, like any other sysadmin? It's an amazing part of the system, no? Doing things periodically, and repeatably. But you've probably had to learn a lot of hard lessons along the way, right?

- Doing things idempotently
- Checking that you don't have a period cron job running, before starting this one, if it's a long one.
- What if there was a reboot?

Would you like to worry less about those issues? Try systemd timers today!

[The arch page](https://wiki.archlinux.org/title/Systemd/Timers) is an excellent reference on why they're interesting and useful. We'll reproduce their pro/con list below:

> > <code-out-title>Pros</code-out-title>
> > - Easier testing, you can trigger units any time
> > - Cgroups!
> > - dependencies (e.g. network!)
> > - automatic logging!!
> > - two flavours of timers: clock time based, and monotonic (e.g. time since last execution)
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > - Two files, instead of a single line. It *is* annoying, you're not wrong. Maybe write an ansible role that translates it.
> > - No `MALITO` functionality built in.
> {: .code-in}
{: .code-2col}

## Listing Timers

We have a lovely timer listing, which tells you when it'll next run, how much time is left, when it last ran, how much time since that passed.

```
$ systemctl list-timers
NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES
Mon 2022-07-04 12:32:45 CEST 6min left     Mon 2022-07-04 11:34:59 CEST 51min ago    anacron.timer                anacron.service
Mon 2022-07-04 14:07:08 CEST 1h 40min left Fri 2022-07-01 13:38:20 CEST 2 days ago   motd-news.timer              motd-news.service
Mon 2022-07-04 17:12:05 CEST 4h 45min left Mon 2022-07-04 10:20:49 CEST 2h 5min ago  ua-timer.timer               ua-timer.service
Mon 2022-07-04 17:14:33 CEST 4h 48min left Fri 2022-07-01 19:22:14 CEST 2 days ago   apt-daily.timer              apt-daily.service
Tue 2022-07-05 00:00:00 CEST 11h left      Mon 2022-07-04 09:32:16 CEST 2h 54min ago logrotate.timer              logrotate.service
Tue 2022-07-05 00:00:00 CEST 11h left      Mon 2022-07-04 09:32:16 CEST 2h 54min ago man-db.timer                 man-db.service
Tue 2022-07-05 01:44:11 CEST 13h left      Mon 2022-07-04 11:07:49 CEST 1h 18min ago fwupd-refresh.timer          fwupd-refresh.service
Tue 2022-07-05 06:28:44 CEST 18h left      Mon 2022-07-04 09:33:59 CEST 2h 52min ago apt-daily-upgrade.timer      apt-daily-upgrade.service
Tue 2022-07-05 10:12:36 CEST 21h left      Mon 2022-07-04 09:47:01 CEST 2h 39min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2022-07-10 03:10:47 CEST 5 days left   Mon 2022-07-04 09:32:52 CEST 2h 53min ago e2scrub_all.timer            e2scrub_all.service
Mon 2022-07-11 00:00:00 CEST 6 days left   Mon 2022-07-04 09:32:16 CEST 2h 54min ago fstrim.timer                 fstrim.service
```

In the last column we can see the timer units, and the services which they activate. In systemd everything is a unit, and hopefully by this point in the tutorial you've seen some of the benefits of that.

Let's look at the `motd-news` timer:

```console
$ systemctl cat motd-news.timer
[Unit]
Description=Message of the Day

[Timer]
OnCalendar=00,12:00:00
RandomizedDelaySec=12h
Persistent=true
OnStartupSec=1min

[Install]
WantedBy=timers.target
```

It has a description, an `OnCalendar` time since it's a clock based timer, potentially a random delay as well (despite being `Sec`, can be expressed in any unit). The associated `motd-news.service` does not need to be mentioned, it will be started by default based on the name of the timer.

What about logs? They're in journalctl now! Let's take advantage of some of the things we learned earlier, like, wildcards!

```console
journalctl -u motd-news.* --since '5 days ago'
```

You'll see the timer unit start mostly around reboots, usually it doesn't have much useful in it. The service unit has what we're more interested in, any logged output.

## Fixing those Cron Issues

Before we mentioned a few issues that admins dealt with with cron

- Doing things idempotently
- Checking that you don't have a period cron job running, before starting this one, if it's a long one.
- What if there was a reboot?

While we can't help with the first, the second is taken care of by having systemd services be state machines, if there's an already running copy of the `.service`, then the `.timer` won't do anything. No more `pgrep` to check for specific processes. And as for reboots, `Persistent=true` can cause a timer to trigger immediately, if it missed the last start time.

## Notifying you

As mentioned in the Arch wiki, you can setup a "notification" service. This is a service, typically a simple bash script, that, when run, sends a notification to a channel of your choice (Slack, XMPP, Email, etc.) This can be a really simple service:

```
/etc/systemd/system/failure-notification@.service

[Unit]
Description=Send a notification
After=network.target

[Service]
Type=simple
ExecStart=/path/to/failure-notification.sh %i
```

And then for any unit (timer's associated service, normal units, etc) where you'd like to be informed of the failure, simply update or override that to do the notification:

```
[Unit]
OnFailure=failure-notification@%n
```

Here it will be templated out with `%n` meaning the unit name, which will replace the `%i` in the notification unit, and be included in the notification to you.


## Further Reading

- [The arch page](https://wiki.archlinux.org/title/Systemd/Timers)
- [man 5 systemd.timer](https://man.archlinux.org/man/systemd.timer.5)
- [man 7 systemd.time](https://man.archlinux.org/man/systemd.time.7)

# journald

> Where did my files go?
{:.quote}

In a number of systems they're still there, to be honest, you've still got all of your old log files.

But now instead of search N different log files, or memorising which files have what contents, there's one firehose for you to watch instead: journalctl. So we need to make this manageable with filters!

## Per Unit Logs

> <code-out-title>The journalctl Way</code-out-title>
> ```bash
> journalctl -u galaxy
> ```
{: .code-out}

returns all of the logs for Galaxy.  The first line even tells us how far back our logs go:

```console
$ journalctl -u galaxy | head -n1
-- Logs begin at Tue 2022-06-28 08:31:06 UTC, end at Mon 2022-07-04 09:30:01 UTC. --
```

> > <code-out-title>Pros</code-out-title>
> > You did not need to know or remember:
> >
> > - Where the log files were (`/var/log`? `/srv/galaxy/log`? somewhere else?)
> > - If any of the old logs were compressed (or use `zless`/`zcat`/`zgrep`)
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > - Can't just `cat` it.
> {: .code-in}
{: .code-2col}

## Logs for a Group of Units

Sometimes you want to tail the logs of multiple proceses as once. Have you ever been debugging an issue, and wanted to see the logs of `nginx`, **all** `galaxy` processes, and `postgres`? Well you can't because `nginx` needs special configuration to write to journalctl (see, your files are still there, and it makes things extra confusing.)

Adding these two lines to your `nginx` configuration will place logs in journald

```nginx
error_log syslog:server=unix:/dev/log;
access_log syslog:server=unix:/dev/log;
```

And then you can simply

> <code-out-title>The journalctl Way</code-out-title>
> ```bash
> journalctl -f -u galaxy* -u nginx -u postgresql
> ```
{: .code-out}

If you have multiple similarly named units, the wildcard feature is incredibly helpful. UseGalaxy.eu had multiple named processes for: web handlers, workflow schedulers, job handlers. We could see logs simultaneously across all of these units.

> > <code-out-title>Pros</code-out-title>
> > - Tail multiple units at once
> > - Again don't have to know where the files are
> > - Or switch tools if you want to look at older logs
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > ?
> {: .code-in}
{: .code-2col}

## Dropping Logs

journald has the habit of throwing away logs if it gets too many messages simultaneously, to prevent {DOS} and denial-of-storage attacks. To disable that behaviour, you can tune

```
RateLimitBurst=0
```

(see `man 5 journald.conf`)

## Logs during a specific time period

This is my absolute favourite feature of journalctl, and I've regularly used it to narrow down logs from dozens of processes, especially when I don't know the root cause of an issue and need to watch all of the logs.

> > <code-in-title>The Old Way</code-in-title>
> > ```bash
> > zcat /var/log/* | egrep "(Jul\s*1 11:4.:..|2022-07-01 11:4.:..)"
> > ```
> > Hope everything logs in the same timezone! And no one uses UTC when there's a different system time configured.
> {: .code-in}
>
> > <code-out-title>The journalctl Way</code-out-title>
> > ```bash
> > journalctl --since "2022-07-01 11:40 CEST" --until "2022-07-01 11:50 CEST"
> > ```
> {: .code-out}
{: .code-2col}

This is very useful if you know when an error is, easily pull out entire system logs from around a specific timepoint. Everything is forced through journald's logging format, so you can be sure the timestamps are all correct, and `--since` and `--when` can be made timezone aware if you have reason to use that (e.g. a user reporting an issue in their current local time.) You can mix and match explicit time zone locales for even more fun:

```bash
journalctl --since "2022-06-21 14:24 Pacific/Auckland" --until "2022-06-21 14:30 Europe/Amsterdam"
```

The logs will be displayed in the last mentioned TZ.

## Logs during a specific time period, relatively

This is another flag that I use literally every day:

```bash
journalctl --since '1 hour ago'
```

If you know relatively when an issue started happening, it's a great way to filter on the logs that are interesting, without having to do clock math.

## Previous Boot Logs

Want to see what happened during a previous run of the system, during a previous boot? Use `--list-boots` to list all of the previous system restarts

```console
$ journalctl --list-boots
-14 143fdb7ac7a54370b41196c02affc3e8 Wed 2022-04-06 20:03:38 CEST—Mon 2022-04-11 08:26:00 CEST
-13 45801d31b0654fe3a4a7d5541e6ff71a Mon 2022-04-11 09:06:07 CEST—Wed 2022-04-20 13:37:31 CEST
-12 661e959635954aa0bbd6b800433e5097 Wed 2022-04-20 13:37:53 CEST—Thu 2022-04-28 17:26:32 CEST
-11 cedaef23e95849c39cfcdb1611b616a2 Thu 2022-04-28 17:40:33 CEST—Thu 2022-04-28 17:43:23 CEST
-10 6ac084d5256d4134bd6e395b490e99f2 Thu 2022-04-28 18:44:57 CEST—Mon 2022-05-09 11:21:33 CEST
 -9 680c9440c75d4699b6de48a56891d835 Mon 2022-05-09 11:21:54 CEST—Tue 2022-05-17 15:21:36 CEST
 -8 b8581fd6394144a9abc59b2832056037 Wed 2022-05-18 09:08:54 CEST—Fri 2022-05-20 17:33:36 CEST
 -7 8bb5c9d7d4ef4117ad8181d0314de7e1 Sun 2022-05-22 17:45:23 CEST—Thu 2022-06-02 16:35:45 CEST
 -6 5325407857054a828dbba6bb104efc80 Fri 2022-06-03 12:04:28 CEST—Thu 2022-06-09 10:15:40 CEST
 -5 0428cc6d11bc4aa9bd8e3999f9bd2c7f Thu 2022-06-09 10:20:05 CEST—Fri 2022-06-17 12:26:06 CEST
 -4 990910c70e79458394f5d9a97ae724ab Sat 2022-06-18 11:21:16 CEST—Tue 2022-06-21 12:14:47 CEST
 -3 d9a9088afacd4c6c9f85db49e9b321ce Tue 2022-06-21 14:29:20 CEST—Tue 2022-06-21 15:07:26 CEST
 -2 c3f0ec4fbc93410280f1e086d6533286 Tue 2022-06-21 15:07:45 CEST—Thu 2022-06-23 12:58:11 CEST
 -1 5d4d9c039daf4e5795a13211de4d6fd4 Thu 2022-06-23 12:58:31 CEST—Fri 2022-07-01 20:10:15 CEST
  0 490a1609fb5d422cad1e7135db88efe7 Mon 2022-07-04 09:32:12 CEST—Mon 2022-07-04 11:56:09 CEST
```

You can then see logs for those specific timeperiods with

```
$ journalctl -b -3 | head
-- Logs begin at Wed 2022-04-06 20:03:38 CEST, end at Mon 2022-07-04 11:59:13 CEST. --
jun 21 14:29:20 cosima kernel: microcode: microcode updated early to revision 0x88, date = 2021-03-31
jun 21 14:29:20 cosima kernel: Linux version 5.13.0-44-generic (buildd@lcy02-amd64-107) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #49~20.04.1-Ubuntu SMP Wed May 18 18:44:28 UTC 2022 (Ubuntu 5.13.0-44.49~20.04.1-generic 5.13.19)
```

All of the other filtering commands (`--since`, `--when`) natively search across boots so there's generally no need to supply this flag unless you want to restrict to a specific time period based on restarting.


## Advanced Log Filtering

There's a lot of additional information journald tracks with your logs, you can read them in the `verbose` format to see

```console
$ journalctl -o verbose
Wed 2022-04-06 20:03:38.281794 CEST [s=1b57c151fb2e49ad98a84ac45ed2e245;i=a310796;b=143fdb7ac7a54370b41196c02affc3e8;m=51917bf226;t=5dc002e4a3f42;x=4ae28fb4218f3691]
    _TRANSPORT=stdout
    _STREAM_ID=45b0adde513f46348564e6f9ce85e4b2
    PRIORITY=6
    SYSLOG_FACILITY=3
    SYSLOG_IDENTIFIER=influxd-systemd-start.sh
    MESSAGE=[httpd] 127.0.0.1 - - [06/Apr/2022:20:03:38 +0200] "POST /write?db=telegraf HTTP/1.1 " 204 0 "-" "Telegraf/1.21.4 Go/1.17.7" e265a273-b5d3-11ec-88d5-80fa5b94d68a>
    _PID=2491
    _UID=1001
    _GID=1001
    _COMM=influxd
    _EXE=/usr/bin/influxd
    _CMDLINE=/usr/bin/influxd -config /etc/influxdb/influxdb.conf
    _CAP_EFFECTIVE=0
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_CGROUP=/system.slice/influxdb.service
    _SYSTEMD_UNIT=influxdb.service
    _SYSTEMD_SLICE=system.slice
    _SYSTEMD_INVOCATION_ID=0ba7986947854acc938a93f020f96cee
    _BOOT_ID=143fdb7ac7a54370b41196c02affc3e8
    _MACHINE_ID=08854338f5f046d5a0d73f81ca5c2d79
    _HOSTNAME=cosima
```

All of these attributes can be used to filter your logs, just add them to your `journalctl` command like so:

```bash
journalctl _GID=1001
```

Or, for example, to get every log, by any process that logged, running under a specific user, like the Galaxy user:

```bash
journalctl _UID=$(getent passwd | grep galaxy | cut -f 3 -d : )
```

Or to see every log from a python program?

```bash
journalctl _COMM=python3
```

## Cleaning Logs

Journald comes with a nice tool for cleaning up logs, `journalctl --vacuum-...`. There are a couple different ways to invoke this. It's very useful if you left Galaxy poorly configured, reading bad XML files and crashing every minute, logging endlessly until you checked it again. That sort of behaviour generates an incredible number of logs.

First we can check how much disk space our logs are using:

```
$ journalctl --disk-usage
Archived and active journals take up 120.0M in the file system.
```

And then we can clean it!

Based on **size**:

```bash
journalctl --vacuum-size=4GB
```

Based on **age**

```bash
journalctl --vacuum-time="2 days"
```

Or based on number of **files**

```bash
journalctl --vacuum-files=3 # I've never used this one.
```

Note that this only applies to "archived" logs, not "active" ones, but you can use `journalctl --rotate` to force them to archived, before cleaning..


## Log Verification

Want to check if they've been tampered with (and someone forgot to recalculate checksums?). The `--verify` flag can be used to do that.

```
$ journalctl --verify
PASS: /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/user-1001@0005df9b9d17faa5-c4276c13359e16e1.journal~
File corruption detected at /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/system@0005df9b9c8ebfda-d41419e76ede7f2c.journal~:3f070f8 (of 67108864 bytes, 98%).
FAIL: /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/system@0005df9b9c8ebfda-d41419e76ede7f2c.journal~ (Bad message)
PASS: /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/system@ede9cabbb3874bd98bcb0b16ada49d8f-000000000a6ce684-0005de680b29a077.journal
7e5abf8: Unused data (entry_offset==0)
..  94%
```

## Further Reading

- [man 5 journald.conf](https://manpages.ubuntu.com/manpages/bionic/man5/journald.conf.5.html)
- [man 1 journalctl](https://manpages.ubuntu.com/manpages/bionic/man1/journalctl.1.html)
- [man 7 systemd.journal-fields](https://manpages.ubuntu.com/manpages/bionic/man7/systemd.journal-fields.7.html)


topic: admin
tutorial: upgrading
title: Upgrading Galaxy
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/upgrading/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible"]
content_hash: 1e877503edfc0503d87f3c024c78b2c1
last_modified: 2026-05-27T09:01:47.649483
workflows_json: []
content:
# Overview


One of the most important and recurring tasks for a Galaxy server admin is to keep the server up to date with security patches, minor fixes and major version upgrades.

Luckily for us, by using the best practice Ansible installation method that we have used throughout these tutorials, keeping our Galaxy server up to date is extremely easy.

We can get all the latest patches and fixes for our currently installed version of Galaxy by just re-running the Ansible playbook on a regular basis.

We can upgrade our Galaxy version by modifying the Ansible var files and re-running the playbook. It's that simple.

This tutorial will show you how and discuss some of the things you need to keep in mind whenever you are updating or upgrading your Galaxy server.

## Some notes on Galaxy versioning and releases.

* As at June 2021, Galaxy has three new version releases per year - in each of January, May and September. The versions are denominated by the year and month - `YY.MM`. i.e. January 2021's Galaxy release is version *21.01*.
* Official Galaxy versions are tagged on the [Galaxy Project's GitHub releases page](https://github.com/galaxyproject/galaxy/releases) as `vYY.MM`. e.g. *v21.01*
* Each version also has an official branch in the GitHub repo named `release_YY.MM`. e.g. *release_21.01*
* Galaxy versions will be supported with security fixes, bug patches and other improvements for [one year](https://github.com/galaxyproject/galaxy/blob/dev/SECURITY.md#supported-versions) from the time of release at which time they will become "End of Life" and will no longer be supported.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## This tutorial assumes that:

- You have a VM or machine where you you have Galaxy installed.
- You have completed at least the "Galaxy Installation with Ansible" tutorial.
- You have command line and sudo access to the VM/computer where it is installed.

> <tip-title>Galaxy server version installed by the tutorials</tip-title>
> At the time of writing, the version of Galaxy that was installed with the "Galaxy Installation with Ansible" tutorial is *20.09*. This tutorial is designed to upgrade the server to *21.01*. If you already have a version equal to or higher than *21.01* running on your server, never fear! The process outlined below can still be used to upgrade to the latest version of Galaxy - just swap out *21.01* used below for the current latest version.
{: .tip}

# Find your current Galaxy server version

The first thing we will do in this tutorial is find the current installed version of the Galaxy. There are a few methods for determining this - You can find it in the GUI if you are an admin user or you can get it via the server API.

Once we know what our current version is, we can compare it with the latest available version

> <hands-on-title>Get the current server version</hands-on-title>
>
> **Method 1: Via the Admin UI page**
> The currently installed version of Galaxy is shown on the Galaxy server's Admin UI page.
> 1. Click on the **Admin** menu item to bring up the Admin UI.
>
> 2. The version of Galaxy currently running is displayed in the top left corner of the screen.
>
> ![Screenshot of the Admin UI page showing the Galaxy server version](../../images/admin_page_version.png "Screenshot of the Admin UI page showing the Galaxy server version.")
>
> **Method 2: Via the API**
>
> 1. In your browser - go to the following URL: [`https://galaxy.example.org/api/version`](https://my.gat.galaxy.training/?path=/api/version) (replacing `galaxy.example.org` with the hostname of your Galaxy server). The version will be displayed as follows:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```shell
>    > curl https://galaxy.example.org/api/version
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>Browser window</code-out-title>
>    > ```JSON
>    > {
>    >   "version_major": "20.09",
>    >   "extra": {}
>    > }
>    > ```
>    {: .code-out}
>
> Both methods above show the version of Galaxy that has been installed. At the time of writing, on this server it's *20.09*. Your version may differ depending on which version you installed initially.
{: .hands_on}


# Finding the latest release version

So, we have found out which version we are running on our server. But what is the current latest version of Galaxy that is available? There are quite a few ways of finding out.

* Galaxy is released 3 times per year. If the time of year is around January/February, April/May or September/October there may be a new release about to be announced.
* You can check the Galaxy Project Newsletter and home page [https://galaxyproject.org](https://galaxyproject.org)

Probably the easiest way of finding the version number of the latest release is via the extensive Galaxy Documentation Website located at: [https://docs.galaxyproject.org](https://docs.galaxyproject.org).

This site shows all of the release notes of all the available Galaxy versions and always has the latest release at the top of the list.

![Screenshot of the Galaxy documentation webpage showing the release versions](../../images/galaxy_doc_releases.png "Screenshot of the Galaxy documentation webpage (<a href="https://docs.galaxyproject.org">https://docs.galaxyproject.org</a>) showing the release versions. The latest version is at the top of the 'Releases' list.")

We can see from this page that the latest release version of Galaxy is *21.01* (at the time of writing.)

> <question-title>What is the current latest version?</question-title>
> As previously mentioned, the current latest release of Galaxy may not be *21.01*. What version is the latest that you can see?
{: .question}

## The importance of the release notes

Each release of Galaxy is accompanied by extensive release notes in two different "flavours."

* "Full release notes" which have extensive detail about all changes made since the last release. This version of the release notes is targeted to Administrators who may need to know about minor fixes, configuration details, security issues, new features and other things found in the latest version.
* "User facing" release notes which provide an visual and interactive summary of the release highlights, additions and new features targeted towards end users, trainers and funders etc.

Before upgrading your Galaxy server to the latest version (or even a new version for that matter) - It is **HIGHLY recommended** that you read the release notes for that version. The notes are *very extensive* and may contain important information on any new features, links  and how to configure them for your server.

![Example section of release notes for release 21.01](../../images/release_notes_2101.png "An example selection from the release notes for Galaxy version 21.01 showing details on enhancements made with their associated Pull Requests in the GitHub repository.")

# Back up the Galaxy database

At the heart of every Galaxy server beats its database. The database is the most important part of the Galaxy server as it contains all of the metadata required for the server to operate and records details of all users, jobs, datasets etc since the server was started.

Sometimes new versions of Galaxy make changes to the database schema that require a database upgrade.

It is highly important that we back it up before making any substantial changes to the Galaxy server as things sometimes go wrong (inopportune power failures, disk crashes etc) and losing or corrupting the database is possible during an upgrade operation.

In a production environment it is really important to make regular backups of the database in any case. The **usegalaxy.\*** servers all perform at least daily backups of their respective databases and even have replicate database servers *just in case.*

> <hands-on-title>Back up the Galaxy database</hands-on-title>
>
> We will be using the PostgreSQL `pg_dump` command to dump the complete contents of the Galaxy database to a file. From the usage text of the command:
>
>   {% raw %}
>   ```
>   pg_dump --help
>   pg_dump dumps a database as a text file or to other formats.
>
>   Usage:
>     pg_dump [OPTION]... [DBNAME]
>   ```
>   {% endraw %}
>
> We will use `pg_dump` and pipe its output through `gzip` to an output file to create a compressed text (SQL) representation of the database. We will need to do this as the `galaxy` user. It's a good idea to create the backup file name with a sensible name and then copy it somewhere else like a cloud service.
>
> From your terminal -
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sudo su -l galaxy
>    > pg_dump galaxy | gzip > galaxy_database-2009-1.sql.gz
>    > ls -lah galaxy_database*
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    > ```bash
>    > -rw-rw-r-- 1 galaxy galaxy 94K Jun  8 04:12 galaxy_database_2009-1.sql.gz
>    > ```
>    {: .code-out}
>
> As you can see, the database on our server isn't currently huge though this will changes as your server gets more users, runs more jobs and stores more data. The current size for Galaxy Australia's compressed database backup is ~11GB and growing.
{: .hands_on}

Now that we have backed up our database we can move on to upgrading our Galaxy server to the latest version. For the sake of this tutorial, we will assume that the installed version is different to the latest version (at the time of writing - *20.09* vs *21.01*). If that is not the case for you, then you can instead use this tutorial as a reference in the future.

# Upgrading Galaxy to a new version

If you've been following best practices and using Ansible to install and configure your Galaxy server then you're in luck - this step is going to be very easy! We just need to adjust the Galaxy version number in our `group_vars/galaxyservers.yml` file and re-run the playbook. If you haven't been using Ansible then there are some notes on performing an upgrade located at: [https://galaxyproject.org/admin/maintenance/](https://galaxyproject.org/admin/maintenance/). The Ansible method is much preferred and simple!

There are however a few things to consider before doing the actual upgrade, especially in a production environment.

* Galaxy will be restarted.
    * Is this an appropriate time for that to happen?
    * Does it need to be scheduled with your users?
* The database schema may change
    * Have you backed up the database?

Once you've made sure of these things, it's time to do it!

> <hands-on-title>Back up the Galaxy database</hands-on-title>
>
> In this tutorial we will upgrade the server to version *21.01*. This may be different for you. Please use the latest version instead.
>
> 1. Edit the `group_vars/galaxyservers.yml` file and change the `galaxy_commit_id` to the latest version. Use the form `release_YY.MM` or in this case `release_21.01`.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -21,7 +21,7 @@ galaxy_manage_paths: true
>     galaxy_layout: root-dir
>     galaxy_root: /srv/galaxy
>     galaxy_user: {name: galaxy, shell: /bin/bash}
>    -galaxy_commit_id: release_20.09
>    +galaxy_commit_id: release_21.01
>     galaxy_force_checkout: true
>     miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
>     miniconda_version: 4.7.12
>    ```
>    {% endraw %}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Now run the playbook!
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

You should see in the Ansible output that Galaxy had to rebuild the client, update the database, rebuild its virtual environment and restart. If you weren't using Ansible, these steps would all be manual and important not to get wrong. Ansible makes the entire process super simple for us.

Congratulations, you're Galaxy server should now be updated to the latest version. You can check this by going to the Admin UI page or by using the API as described above.

# Ongoing maintenance

The Galaxy development community continually support release versions of Galaxy [for 1 year.](https://github.com/galaxyproject/galaxy/blob/dev/SECURITY.md#supported-versions) They regularly update releases with bug fixes, patches and security enhancements. Therefore it is important that we as administrators keep our Galaxy servers up to date regardless of which release we are running.

The easiest way to do this is to regularly re-run the Ansible playbook. It will check with GitHub and grab any new changes to Galaxy since the last time you ran it and automatically keep everything up to date. The large **usegalaxy.\*** servers all re-run the playbook on a regular schedule for this purpose.

Regular running of the playbook also encourages you and your team members to make changes to the Ansible scripts not the live system (as live changes will be overwritten). This leads to a much more robust and consistent production Galaxy server for your users.

# Conclusion

This tutorial has shown you how to keep your Galaxy server up to date with the latest version painlessly and simply following current best practice. We hope you enjoyed this tutorial and got something out of it.


topic: admin
tutorial: celery
title: Setting up Celery Workers for Galaxy
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/celery/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "git-gat"]
content_hash: 7aa29a7236e9c8a03df25e1bd9d77ec9
last_modified: 2026-05-27T09:01:47.633283
workflows_json: []
content:
Celery is a distributed task queue written in Python that can spawn multiple workers and enables asynchronous task processing on multiple nodes. It supports scheduling, but focuses more on real-time operations.

From the Celery website:

> "Task queues are used as a mechanism to distribute work across threads or machines.
>
>A task queue’s input is a unit of work called a task. Dedicated worker processes constantly monitor task queues for new work to perform.
>
>Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker.
>
>A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.
>
>Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there’s node-celery and node-celery-ts for Node.js, and a PHP client.
>
>Language interoperability can also be achieved exposing an HTTP endpoint and having a task that requests it (webhooks)."
>
{: .quote cite="https://docs.celeryq.dev/en/stable/getting-started/introduction.html#what-s-a-task-queue"}

[A slideshow presentation on this subject is available](slides.html).

If you are not interesting in managing Redis and Flower, you might be interested in the [lower-configuration deployment option]({% link topics/admin/tutorials/celeryless/tutorial.md %}).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="celery" %}

The agenda we're going to follow today is: We're going to enable and configure celery, install a Redis server, the Flower dashboard and start Celery workers.

# Installing and Configuring

To proceed from here it is expected that:

<!--  TODO: port assumptions are not correct for GAT. -->

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have set up a working Galaxy instance as described in the [ansible-galaxy](../ansible-galaxy/tutorial.html) tutorial.
>
> 2. You have a working RabbitMQ server installed and added the connection string to the galaxy configuration. (RabbitMQ is installed when doing the [Pulsar](../pulsar/tutorial.html) tutorial.)
>
> 3. Your VM has a public DNS name: this tutorial sets up SSL certificates from the start and as an integral part of the tutorial.
>
> 4. You have the following ports exposed:
>
>    - 22 for SSH, this can be a different port or via VPN or similar.
>    - 80 for HTTP, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 443 for HTTPs, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 5671 for AMQP for Pulsar, needed if you plan to setup Pulsar for remote job running.
>
{: .comment}
In order to run a production ready Celery setup, we need to discuss and install some other software that works together with Celery.  
We already learned about RabbitMQ in the Pulsar tutorial. The RabbitMQ server you already installed there will be our broker for Celery.
As a backend we are going to use Redis.  
Redis is a very popular key-value-store database. It is very fast and easy to set up backend for Celery.
If you want to learn more about Redis, visit their website: [https://redis.io/](https://redis.io/)

For monitoring and debugging Celery, we use the [Flower](https://github.com/mher/flower) dashboard.  
Flower is lightweight and has a clear but powerful UI and can be installed in Galaxy's venv using our role.

# Installing and Configuring

First we need to add our new Ansible Roles to the `requirements.yml`:

> <hands-on-title>Set up Redis, Flower, Systemd and Celery with Ansible</hands-on-title>
>
> 1. In your working directory, add the roles to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -37,3 +37,8 @@
>       version: 1.4.4
>     - src: galaxyproject.pulsar
>       version: 1.0.10
>    +# Celery, Redis, and Flower (dashboard)
>    +- name: geerlingguy.redis
>    +  version: 1.8.0
>    +- name: usegalaxy_eu.flower
>    +  version: 2.1.0
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement" data-ref="add-req"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the roles with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true" data-ref="req-install"}
>    {: .code-in}
>
> 3. Let's go now through all the Roles step-by-step:
>
>     1. Since we can stick to the basic default settings of Redis, we will look only at a few variables:
>
>        | Variable               | Type    | Description                                                                 |
>        | ----------             | ------- | -------------                                                               |
>        | `redis_port`           | integer | The port Redis should listen on. 6379 by default.                           |
>        | `redis_bind_interface` | string  | The interface Redis should listen to. 127.0.0.1 is default.                 |
>        | `redis_conf_path`      | string  | The path where your redis configuration will be stored. Default: /etc/redis |
>
>        Luckily we can leave them all on default and don't need to change anything for Redis in the vars.  
>     2. We only need to add Redis' Python package in the `group_vars/galaxyservers.yml`:
>        {% raw %}
>        ```diff
>        --- a/group_vars/galaxyservers.yml
>        +++ b/group_vars/galaxyservers.yml
>        @@ -283,3 +283,7 @@ rabbitmq_users:
>         # TUS
>         galaxy_tusd_port: 1080
>         galaxy_tus_upload_store: /data/tus
>        +
>        +#Redis
>        +galaxy_additional_venv_packages:
>        +  - redis
>        {% endraw %}
>        ```
>        {: data-commit="Configure rabbitmq users"}
>        
>     3. Let's add the role to our playbook then:
>        {% raw %}
>        ```diff
>        --- a/galaxy.yml
>        +++ b/galaxy.yml
>        @@ -42,6 +42,7 @@
>             - role: galaxyproject.miniconda
>               become: true
>               become_user: "{{ galaxy_user_name }}"
>        +    - geerlingguy.redis
>             - galaxyproject.nginx
>             - geerlingguy.docker
>             - usegalaxy_eu.rabbitmqserver
>        {% endraw %}
>        ```
>        {: data-commit="Add requirement" data-ref="add-req"}
>
>     4. Because Flower needs it's own RabbitMQ user, we should add that to the respective part of our vars
>        Edit your `group_vars/secret.yml` and define some random passwords:
>
>        ><code-in-title>Bash</code-in-title>
>        > ```
>        > ansible-vault edit group_vars/secret.yml
>        > ```
>        {: .code-in}
>
>        ```yaml
>        vault_rabbitmq_password_flower: "a-really-long-password-here"
>        vault_rabbitmq_password_galaxy: "a-different-really-long-password"
>        vault_flower_user_password: "another-different-really-long-password"
>        ```
>
>        <!-- Ignore this, just for the gat-automation. Vaults are ugly to work with :(
>
>        {% raw %}
>        ```diff
>        --- a/group_vars/secret.yml
>        +++ b/group_vars/secret.yml
>        @@ -1,13 +1,22 @@
>         $ANSIBLE_VAULT;1.1;AES256
>        -62346261323266656232393034396134316636376533376139666437363535393562663838613938
>        -6336666266633563346337623265353935646361326337610a393834333233313461346439376438
>        -63383338346530656561636631666134373238366364363164313166346461383736613162653237
>        -3461363334323431370a656132303965653262386130353332623937376261396530393761353834
>        -38336565666437666436643163363831633331333766653266356163613138393734656465323634
>        -39366362383433366437353534663134313330316337393335383962613961386665633261616237
>        -35366635373063313631323939396164336330356361393464326636353037336461323531336434
>        -35613933303333623031353936393265636130363335376533393335663266313863376135383338
>        -36613464373231623938373434306266373234633036343636633963353361356631363533353066
>        -39323064336237646432323530313065303331326636353334343862373330313133326363363063
>        -38383564636161396435666164643334656435393533643163393434623434656238633631633939
>        -33353232666432376661
>        +66653366633665383231303635643739396466653465626163633662623230643534346666303434
>        +3531356339646666623364383435306363373534373833370a333964306130653236373438373264
>        +34366433396135303064643932313135643064373034323865363939333565623238386161333465
>        +3932623162353034370a653663333363383938393936623663343639376430653366646662353830
>        +34383964326161383966306361633162653366353162633564623738353137333936313232363764
>        +38383962396637636136316366633936316365643038333030333932336365326335373737663834
>        +39363639303764343430336435313663313762353335366562656232646334356438343535303032
>        +35636131323163323036383163363964643237313137333131303737346662373233663562343031
>        +39396531306539623661376166313534303462623532323334393736316634616262313633626637
>        +37393066663239303564356232366466333334316565363631363230626665333039386133383933
>        +31656234363064646334623938383637393534313638643635396662643163346633346237353664
>        +34666161373764663166653737323736373261363565666633653831363833393264333666356633
>        +30373062333638366666316132623736386336383933326539663065653538386363323766313664
>        +63323737396264306439303366343834326164393763376663316366353766663131303966393039
>        +61343732313865613761623733356465336263633963376161663931653864383162393839643834
>        +34373239386461666462643162613536316166656135323332316163336635393035653164363138
>        +34336265383033663436306436663536383238366665653239313661666431666339323364646633
>        +64366361653034393962316562373835623631356339313062393933633634653735656636373031
>        +65376162633866623733623961303664663931636130373936326461623465363932373165356563
>        +36323838323865326464346337393164343765613333663830636234666663303535616532323936
>        +3230
>        {% endraw %}
>        ```
>        {: data-commit="Add rabbitmq passwords to the vault"}
>
>        -->
>
>        This is going in the vault as they are secrets we need to set. Flower needs it's own RabbitMQ user with admin access and we want a different vhost for galaxy and celery.
>
>        Replace both with long random (or not) string.  
>        Now add new users to the RabbitMQ configuration:
>        {% raw %}
>        ```diff
>        --- a/group_vars/galaxyservers.yml
>        +++ b/group_vars/galaxyservers.yml
>        @@ -270,6 +270,7 @@ rabbitmq_config:
>
>         rabbitmq_vhosts:
>           - /pulsar/pulsar_au
>        +  - galaxy_internal
>
>         rabbitmq_users:
>           - user: admin
>        @@ -279,6 +280,13 @@ rabbitmq_users:
>           - user: pulsar_au
>             password: "{{ vault_rabbitmq_password_vhost }}"
>             vhost: /pulsar/pulsar_au
>        +  - user: galaxy
>        +    password: "{{ vault_rabbitmq_password_galaxy }}"
>        +    vhost: galaxy_internal
>        +  - user: flower
>        +    password: "{{ vault_rabbitmq_password_flower }}"
>        +    tags: administrator
>        +    vhost: galaxy_internal
>
>         # TUS
>         galaxy_tusd_port: 1080
>        {% endraw %}
>        ```
>        {: data-commit="Configure rabbitmq users"}
>     5. Flower
>        Flower has a few variables, too, for example, we need to point it to our virtual environment:
>
>        | Variable             | Type          | Description                                                                                                                                                                    |
>        | ----------           | -------       | -------------                                                                                                                                                                  |
>        | `flower_python_version`         | string        | Python version to use when installing flower to a venv. Default: python39                               |
>        | `flower_port`         | integer        | The port Flower should listen on. 5555 by default.                               |
>        | `flower_bind_interface`         | string | The interface Flower should listen to. 0.0.0.0 is default.  |
>        | `flower_conf_dir`  | string | The path where your Flower configuration will be stored. Default: /etc/flower |
>        | `flower_venv_dir`  | string | The path to the venv where Flower should be installed. Default: `/home/{{ flower_user }}/.local` |
>        | `flower_user`  | string | User that owns the flower process. Default: galaxy |
>        | `flower_group`  | string | Group that owns the flower process. Default: galaxy |
>        | `flower_ui_users`  | list of dicts | Name and password of the UI users for basic auth. |
>        | `flower_app_dir`  | string | Root directory of your Python app to run with Celery. In our case `galaxy_root` |
>        | `flower_app_name`  | string | Python module to import. In our case 'galaxy.celery' |
>        | `flower_python_path`  | string | Should point to galaxy's `server/lib` directory (default) |
>        | `flower_broker_api`  | string | URL to broker's API with login credentials. |
>        | `flower_broker_url`  | string | Flower's RabbitMQ connection string. |
>        | `flower_db_file`  | string | When Flower is in persistent mode, use this path for the database. |
>
>        Let's add variables to our `group_vars/galaxyservers.yml`:
>
>        {% raw %}
>        ```diff
>        --- a/group_vars/galaxyservers.yml
>        +++ b/group_vars/galaxyservers.yml
>        @@ -295,3 +295,22 @@ galaxy_tus_upload_store: /data/tus
>         #Redis
>         galaxy_additional_venv_packages:
>           - redis
>        +
>        +# Flower
>        +flower_python_version: python3
>        +flower_app_dir: "{{ galaxy_root }}"
>        +flower_python_path: "{{ galaxy_root }}/server/lib"
>        +flower_venv_dir: "{{ galaxy_venv_dir }}"
>        +flower_app_name: galaxy.celery
>        +flower_db_file: "{{ galaxy_root }}/var/flower.db"
>        +flower_persistent: true
>        +flower_broker_api: "https://flower:{{ vault_rabbitmq_password_flower }}@localhost:5671/api/"
>        +flower_broker_url: "amqp://flower:{{ vault_rabbitmq_password_flower }}@localhost:5671/galaxy_internal?ssl=true"
>        +flower_proxy_prefix: /flower
>        +
>        +flower_ui_users:
>        +  - name: admin
>        +    password: "{{ vault_flower_user_password}}"
>        +
>        +flower_environment_variables:
>        +  GALAXY_CONFIG_FILE: "{{ galaxy_config_file }}"
>        {% endraw %}
>        ```
>        {: data-commit="Configure flower"}
>     6. It has a dashboard, so we need to expose that via nginx:
>
>        {% raw %}
>        ```diff
>        --- a/templates/nginx/galaxy.j2
>        +++ b/templates/nginx/galaxy.j2
>        @@ -94,4 +94,13 @@ server {
>         		proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
>         		proxy_set_header X-Forwarded-Proto $scheme;
>         	}
>        +
>        +	location /flower {
>        +		proxy_pass http://localhost:5555;
>        +		proxy_set_header Host $host;
>        +		proxy_redirect off;
>        +		proxy_http_version 1.1;
>        +		proxy_set_header Upgrade $http_upgrade;
>        +		proxy_set_header Connection "upgrade";
>        +	}
>         }
>        {% endraw %}
>        ```
>        {: data-commit="Add nginx routes"}
>
>     7. Now we can add the Flower Role to our Playbook:
>    
>        {% raw %}
>        ```diff
>        --- a/galaxy.yml
>        +++ b/galaxy.yml
>        @@ -43,6 +43,7 @@
>               become: true
>               become_user: "{{ galaxy_user_name }}"
>             - geerlingguy.redis
>        +    - usegalaxy_eu.flower
>             - galaxyproject.nginx
>             - geerlingguy.docker
>             - usegalaxy_eu.rabbitmqserver
>        {% endraw %}
>        ```
>        {: data-commit="Add flower role" data-ref="add-req"}
>
> 4. Now it is time to change the `group_vars/galaxyservers.yml` and enable celery in galaxy.gravity config.
>    Add the following lines to your file:
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -130,6 +130,11 @@ galaxy_config:
>           preload: true
>         celery:
>           concurrency: 2
>    +      enable_beat: true
>    +      enable: true
>    +      queues: celery,galaxy.internal,galaxy.external
>    +      pool: threads
>    +      memory_limit: 2
>           loglevel: DEBUG
>         tusd:
>           enable: true
>    {% endraw %}
>    ```
>    {: data-commit="Add celery" data-ref="add-req"}
>
>    Now add the second part, Galaxy's Celery configuration:
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -111,6 +111,11 @@ galaxy_config:
>         # Data Library Directories
>         library_import_dir: /libraries/admin
>         user_library_import_dir: /libraries/user
>    +    # Celery
>    +    amqp_internal_connection: "pyamqp://galaxy:{{ vault_rabbitmq_password_galaxy }}@localhost:5671/galaxy_internal?ssl=1"
>    +    celery_conf:
>    +      result_backend: "redis://localhost:6379/0"
>    +    enable_celery_tasks: true
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add celery-redis" data-ref="add-req"}
>
> 6. We are done with the changes and you can enter the command to run your playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>    This should also restart Galaxy and spawn the amount of Celery workers, that we defined in the Gravity configuration.
>
{: .hands_on}

# Test Celery

Now that everything is running, we want to test celery and watch it processing tasks.
We can simply do that by starting an upload to our Galaxy.

> <hands-on-title>Test Celery and monitor tasks with Flower</hands-on-title>
> 1. First, open a new tab and enter your machines hostname followed by `/flower/dashboard` then log in with `username: admin` and your password.
>    You should see an overview with active workers.  
>    Keep that tab open
> 2. In split view, open a second browser window and open your Galaxy page.
>    Click on {% icon galaxy-upload %} Upload Data, select a file from your computer and click `upload`.
> 3. The Workers should now receive a new tasks. Click on `Succeeded` and then on the UUID of the last upload task.  
>    You should see all its details here and the info that it was successful.
{: .hands_on}

{% snippet topics/admin/faqs/missed-something.md step=12 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="celery" %}


topic: admin
tutorial: k8s-deploying-galaxy
title: Galaxy Installation on Kubernetes
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["kubernetes"]
content_hash: 254d1e0691bf605492077060dd4e4ce3
last_modified: 2026-05-15T22:28:10.038110
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/k8s-deploying-galaxy/tutorial.html

content:
# Galaxy Helm Chart

## Overview

This tutorial describes how to use the Galaxy Helm Chart to deploy a production
grade instance of Galaxy on a Kubernetes cluster. The Helm Chart has been designed
to follow best practices adopted by the community, including the usegalaxy.* federation,
and will install a Galaxy with the following features by default:
- Zero-downtime configuration changes and upgrades
- Scalable web and job handlers
- Automatic failure recovery based on liveness and readiness probes
- A built-in nginx for efficiently serving large files
- TUSD for resumable uploads
- Celery for background jobs
- Access to CVMFS reference data
- A toolset matching the usegalaxy.* federation (also served off CVMFS)
- Interactive tools (wildcard DNS mapping required)
- Minimal privileges, with jobs running as non-root and only having access to datasets they need
- Automatic maintenance scripts to cleanup the galaxy database and tmp directories

Optionally, the chart can be configured with
- High-availability components - this includes trivial scaling of clustered Postgres, Rabbit MQ etc.
- Replacement components - You can replace the built-in operators with a managed or existing Postgres database (e.g. Amazon RDS), RabbitMQ cluster etc.
- Use S3 as an alternative to CVMFS
- Automatic scraping of metrics which can be sent to Influxdb

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Prerequisites
Some familarity with Kubernetes is assumed. This includes general administrative
familarity and how to install and configure Helm Charts.

A running Kubernetes cluster is also required (1.27 or higher), with Helm
(3.5 or higher) configured to access it.
For development and testing purposes this can be easily achieved by installing
[Docker Desktop locally and enabling Kubernetes][DfD]. Afterwards, install
[Helm](https://helm.sh).

For production deployments, we'll also need some storage resources for data
persistence. This can be done by either defining a [storage class][SC] or
creating a [Persistent Volume and a corresponding Persistent Volume Claim][PV].
Once created, just keep a note of the resources Persistent Volume Claim ID and
to use later.

## Deploying the Default Configuration
The default set of values for the Galaxy chart configures only a minimal set
of Galaxy options necessary. The configured options are required for suitable
operation of the system. Setting other options will depend on the environment
and it's best to refer to the general [Galaxy documentation][GxyDocs]; we'll
also take a look at how to make configuration changes in the context of the
chart later in this tutorial.

> <hands-on-title>Deploying the Galaxy Helm Chart</hands-on-title>
>
> 1. First, we need to add the helm repository for the chart. The chart is
> automatically packaged, versioned and uploaded to a helm repository on github
> with each accepted PR. Therefore, the latest version of the chart can be directly
> installed from that repository.
>
>    {% raw %}
>    ```bash
>    helm repo add galaxyproject https://raw.githubusercontent.com/galaxyproject/helm-charts/master/
>    helm repo update
>    ```
>    {% endraw %}
>
> 2. We can now deploy Galaxy via the Chart. Running this command will create a new
>    Helm release (i.e., chart installation) called `mygalaxy`.
>
>    {% raw %}
>    ```bash
>    helm install mygalaxy galaxyproject/galaxy
>    ```
>    {% endraw %}
>
> 3. It will take about a minute or two for the necessary containers to download,
>    the database to initialize, and Galaxy processes to start. Ultimately,
>    while this may depend on the Kubernetes cluster setup you are using,
>    Galaxy should be available at https://<root URI>/galaxy for the given machine. We can
>    always check the status of our release by typing `helm status galaxy`.
>
{: .hands_on}

## Setting the admin user and changing the brand
The chart is designed to follow standard Kubernetes and Helm idioms, and therefore,
it should be intuitively similar to the steps required to change configuration in
any other Helm chart. For example, ingress paths, resource allocations, container
images etc. can be changed following standard helm conventions. The list of
available configuration options are also documented in the Galaxy Helm
_[Chart repository](https://github.com/galaxyproject/galaxy-helm/tree/master?tab=readme-ov-file#configuration)_

To change Galaxy specific configuration, such as setting the admin user or change the brand in `galaxy.yml`,
we can follow the following steps. Once done, we will also rollback our change to demonstrate how Helm manages
configuration.

> <hands-on-title>Setting admin user and changing the brand</hands-on-title>
>
> 1. Modify the following entries in your `mygalaxy.yml`. Make sure to add these
>    keys under the `configs:` section of the file.
>
>    {% raw %}
>    ```
>    configs:
>      galaxy.yml:
>        galaxy:
>          brand: "Hello World"
>          admin_users: "admin@mydomain.com"
>    ```
>    {% endraw %}
>
> 2. Now, let’s upgrade the chart to apply the new configuration.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values -f mygalaxy.yml mygalaxy galaxyproject/galaxy
>    ```
>    {% endraw %}
>
> 3. Inspect the currently set Helm values by:
>
>    {% raw %}
>    ```bash
>    helm get values mygalaxy
>    ```
>    {% endraw %}
>
> 4. List the installed Helm charts again and note that the revision of the chart has changed as expected.
>
>    {% raw %}
>    ```bash
>    helm list
>    NAME  	  REVISION	UPDATED                 	STATUS  	CHART                 	APP VERSION	NAMESPACE
>    mygalaxy	2       	Wed Jun 26 14:51:17 2023	DEPLOYED	galaxy-5.14.2          	v24.0.2    	default
>    ```
>    {% endraw %}
>
> 5. Revisit the Galaxy Application in your browser to check whether the settings have changed. This will
>    take a short while (< 1 minute) for the new container to come up. You should experience no downtime.
>
> 6. Let’s now roll back to the previous revision.
>
>    {% raw %}
>    ```bash
>    helm rollback mygalaxy 1
>    ```
>    {% endraw %}
>
>    Use `helm get values` again to observe that the values have reverted to
>    the previous revision. After a short while, once the new container is up
>    and running, Kubernetes will automatically switch over to it and you can
>    see that the previous configuration has been restored.
>
{: .hands_on}

## Deleting a Deployed Helm Release
By default, the Helm chart is designed to install all required dependencies, so that it's easy
to get an instance up and running quickly for experimentation. However, in production, we
recommend installing the dependency charts separately, once per cluster, by installing
Galaxy with helm options
`--set postgresql.deploy=false --set s3csi.deploy=false --set cvmfs.deploy=false --set rabbitmq.deploy=false`.

This is particularly important during uninstallation, where orderly destruction of dependencies is often required
For example, if the rabbitmq operator is uninstalled before the rest of the Galaxy helm chart is deleted, there will be
no operator left to cleanup rabbitmq resources. Installing the aforementioned operators separately sidesteps this problem.

> <hands-on-title>Deleting a Deployed Helm Release</hands-on-title>
>
>    {% raw %}
>    ```bash
>    helm delete mygalaxy
>    helm delete mycvmfs # and any other operators
>    ```
>    {% endraw %}
>
{: .hands_on}

# Next Steps
This tutorial covers the basics of getting Galaxy deployed on Kubernetes using
Helm. There is a lot more to understanding all the configuration options for
the chart and the available deployment models. For more info on some of these
topics, take a look at the [Galaxy Helm chart] repository as well as
[other tutorials](../..) tagged with _kubernetes_. Also, feel free to reach out
on Gitter: [https://gitter.im/galaxyproject/FederatedGalaxy][fedG].


[Galaxy Helm chart]: https://github.com/galaxyproject/galaxy-helm
[DfD]: https://www.docker.com/products/kubernetes
[SC]: https://kubernetes.io/docs/concepts/storage/storage-classes/
[PV]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
[CSI]: https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/
[GxyDocs]: https://galaxyproject.org/admin/#configuration
[CVMFS]: https://training.galaxyproject.org/training-material/topics/admin/tutorials/cvmfs/tutorial.html
[BioContainers]: https://biocontainers.pro/#/
[Namespace]: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
[reclaim policy]: https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/
[fedG]: https://gitter.im/galaxyproject/FederatedGalaxy


topic: admin
tutorial: interactive-tools
title: Galaxy Interactive Tools
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "interactive-tools"]
content_hash: a5ab955eb3572baac1b92594dd56bf6c
last_modified: 2026-05-27T09:01:47.636704
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/interactive-tools/tutorial.html

content:
Galaxy Interactive Tools (GxITs) are a method to run containerized tools that are interactive in nature. Interactive Tools typically run a persistent service accessed on a specific port, until terminated by the user. One common example of such a tool is [Jupyter Notebook][jupyter]. Galaxy Interactive Tools are submitted through Galaxy's job management system and thus are scheduled the same as any other Galaxy tool - on a Slurm cluster, for instance. Galaxy Interactive Tools were introduced in Galaxy Release 19.09.

> <comment-title>Evolving Topic</comment-title>
> Galaxy Interactive Tools are a **relatively new and rapidly evolving feature** and there are some rough edges. Work to improve the experience of deploying and using them is ongoing. Please watch the [Galaxy Release Notes][galaxy-release-notes] for updates, changes, new documentation, and bug fixes.
>
> You may find extra information about Interactive Tools on the [Galaxy Documentation][galaxy-docs-interactivetools].
{: .comment}

[galaxy-release-notes]: https://docs.galaxyproject.org/en/master/releases/index.html
[galaxy-docs-interactivetools]: https://docs.galaxyproject.org/en/master/admin/special_topics/interactivetools.html

> <warning-title>Before You Continue - Wildcard DNS Certificates</warning-title>
> If you are *not* completing this tutorial as part of a [Galaxy Admin Training][gat] course, **you will need a wildcard DNS record for your Galaxy server and a method for obtaining a wildcard SSL certificate for your Galaxy server**.
>
> Galaxy Interactive Tools require a [wildcard SSL certificate][wildcard-cert]. Because the **Galaxy Installation with Ansible** tutorial fetches [Let's Encrypt][lets-encrypt] certificates, this tutorial fetches Let's Encrypt wildcard certificates. However, this process is only valid for Galaxy Admin Training courses, because Let's Encrypt wildcard certificates [can only be fetched using the DNS-01 challenge method][lets-encrypt-faq], which requires control of a [dynamic DNS][ddns] server (which we have preconfigured for use at training courses). Configuring your DNS service for dynamic updates is outside the scope of this tutorial, but it will show you how to request certificates using DNS-01, which can be adapted for your site.
>
> If you are using Let's Encrypt, [a list of available DNS plugins for Certbot][certbot-dns-plugins] can be found in the Certbot documentation. If you are not using Let's Encrypt, please consult your certificate vendor's documentation for information on how to obtain a wildcard certificate. You will need a certificate with (at least) the [subject alternative name][san]s `galaxy.example.org` and `*.ep.interactivetool.galaxy.example.org` (where `galaxy.example.org` is the hostname of your Galaxy server).
>
> You will also need a wildcard DNS `CNAME` record for `*.ep.interactivetool.galaxy.example.org`. You can verify that your Galaxy server has such a record using the `host` or `dig` command line tools like so:
>
>    ```console
>    $ host -t cname foo.ep.interactivetool.live.usegalaxy.eu
>    foo.ep.interactivetool.live.usegalaxy.eu is an alias for usegalaxy.eu.
>    $ host -t cname bar.ep.interactivetool.live.usegalaxy.eu
>    bar.ep.interactivetool.live.usegalaxy.eu is an alias for usegalaxy.eu.
>    ```
>
> Please consult your DNS server software or cloud provider's documentation for information on how to set up a wildcard record.
{: .warning}

[wildcard-cert]: https://en.wikipedia.org/wiki/Wildcard_certificate
[lets-encrypt]: https://letsencrypt.org/
[gat]: https://github.com/galaxyproject/dagobah-training
[lets-encrypt-faq]: https://letsencrypt.org/docs/faq/
[ddns]: https://en.wikipedia.org/wiki/Dynamic_DNS
[certbot-dns-plugins]: https://certbot.eff.org/docs/using.html#dns-plugins
[san]: https://en.wikipedia.org/wiki/Subject_Alternative_Name
[jupyter]: https://jupyter.org/
[gie-docs]: https://docs.galaxyproject.org/en/release_19.09/admin/special_topics/interactive_environments.html

> <warning-title>Uses Ansible!</warning-title>
> If the terms "Ansible," "role," and "playbook" mean nothing to you, please check out
> [the Ansible introduction slides]({% link topics/admin/tutorials/ansible/slides.html %}) and [the Ansible introduction tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %}).
{: .warning}

> <warning-title>Have you installed Galaxy?</warning-title>
> This tutorial builds upon the work in the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, please ensure that you have completed that tutorial first.
{: .warning}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Installing Docker

Galaxy Interactive Tools run in Docker containers. Thus, the first step is ensuring that the *nodes* that will run interactive tools have Docker installed. We will use the [Docker role][geerlingguy-docker] by the prolific Ansible Galaxy publisher, [Jeff Geerling (geerlingguy)][geerlingguy]. Have a look at the geerlingguy.docker [README][geerlingguy-docker-readme] and [defaults/main.yml][geerlingguy-docker-defaults] to get an understanding of what variables are used to control the role.

[geerlingguy-docker]: https://galaxy.ansible.com/geerlingguy/docker
[geerlingguy]: https://galaxy.ansible.com/geerlingguy
[geerlingguy-docker-readme]: https://github.com/geerlingguy/ansible-role-docker/blob/master/README.md
[geerlingguy-docker-defaults]: https://github.com/geerlingguy/ansible-role-docker/blob/master/defaults/main.yml

> <question-title></question-title>
>
> What variables might be relevant to using this role?
>
> > <solution-title></solution-title>
> >
> > The `docker_users` variable (a *list*) controls which users are able to interact with the Docker daemon, which our Galaxy user will need to do. Additionally, Docker Compose is configured by default, which we do not need, so it can be disabled with `docker_install_compose: false`.
> >
> {: .solution }
>
{: .question}

> <comment-title>Ansible Best Practices</comment-title>
> If you've set up your Galaxy server using the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, you will have created a `galaxyservers` group in your inventory file, `hosts`, and placed your variables in `group_vars/galaxyservers.yml`. Although for the purposes of this tutorial, the Galaxy server and cluster node are one and the same, in a real world deployment they are very likely to be different hosts. We will continue to use the `galaxyservers` group for simplicity, but in your own deployment you should consider creating an additional group for cluster nodes.
{: .comment}

> <hands-on-title>Installing Docker with Ansible</hands-on-title>
>
> 1. In your working directory, add the Docker role to your `requirements.yml`:
>
>    ```yaml
>    - src: geerlingguy.docker
>      version: 7.8.0
>    ```
>
> 2. Install the requirements with `ansible-galaxy`:
>
>    ```
>    ansible-galaxy role install -p roles -r requirements.yml
>    ```
>
> 3. Edit the group variables file, `group_vars/galaxyservers.yml`:
>
>    The relevant variables to set for this role are:
>
>    | Variable                 | Type            | Description                                     |
>    | ----------               | -------         | -------------                                   |
>    | `docker_users`           | list of strings | List of users to be added to the `docker` group |
>    | `docker_install_compose` | boolean         | Whether to install and configure Docker Compose |
>
>    Add the following lines to your `group_vars/galaxyservers.yml` file:
>
>    {% raw %}
>    ```yaml
>    # Interactive Tools
>    docker_install_compose: false
>    docker_users:
>      - "{{ galaxy_user.name }}"
>    ```
>    {% endraw %}
>
>    > <question-title></question-title>
>    >
>    > {% raw %}
>    > Why is `"{{ galaxy_user.name }}"` specified instead of just the user `galaxy`?
>    > {% endraw %}
>    >
>    > > <solution-title></solution-title>
>    > > Duplicating values is never a good idea. If we needed to change the Galaxy user down the line or wanted to reuse this playbook on another host where the Galaxy username was different, we would have to change the value in multiple locations.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
> 4. Add the new role to the list of roles under the `roles` key in your playbook, `galaxy.yml`:
>
>    ```yaml
>    ---
>    - hosts: galaxyservers
>      become: true
>      roles:
>        # ... existing roles ...
>        - geerlingguy.docker
>    ```
>
> 5. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
{: .hands_on}

Congratulations, you've set up Docker. Verify the installation using the `docker info` command (but keep in mind: what users did we authorize to interact with Docker?).

# Enabling the Interactive Tools Proxy

When an Interactive Tool's Docker container starts, it will be assigned a random port. In order to connect clients to the Interactive Tool, Galaxy needs to determine this port (and the node on which the tool is running) and configure a *proxy* from Galaxy to the GxIT's host and port. Consider the following example of running the Jupyter Notebook Interactive Tool, shown in Figure 1 below:

- nginx listens for requests from the client on **port 443** (https)
- Requests for Galaxy are delivered from nginx to Galaxy over a UNIX domain socket
- Requests for Interactive Tools are delivered from nginx to the Interactive Tools Proxy over (by default) **port 8000** (http)
  - GxIT http requests are forwarded by the proxy to Docker on the node on the container's (randomly assigned) **port 32768**
  - GxIT http requests are again forwarded by Docker to Jupyter on its in-container "published" **port 8888**

[//]: # The source for this figure can be found at: https://docs.google.com/presentation/d/1_4PtfM6A4mOxOlgGh6OGWvzFcxD1bdw4CydEWtm5n8k/

![Galaxy Interactive Tools Proxy Diagram](../../images/interactive-tools/gxit-proxy-diagram.png "Galaxy Interactive Tools Proxy Diagram")

As you can see, the client only ever speaks to nginx on the Galaxy server running on the standard https port (443), never directly to the interactive tool (which may be running on a node that does not even have a public IP address). By default, the mapping of GxIT invocation and its corresponding host/port is kept in a SQLite database known as the *Interactive Tools Session Map*, and the path to this database is important, since both Galaxy and the proxy need access to it.

The GxIT Proxy is written in [Node.js][nodejs]. A straightforward method to set it up is to use Galaxy's process manager, [Gravity](https://gravity.readthedocs.io/), to install and configure it.

[nodejs]: https://nodejs.org/

> <hands-on-title>Installing and configuring the Proxy using Gravity</hands-on-title>
>
> 1. Edit the group variables file, `group_vars/galaxyservers.yml`; the GxIT Proxy configuration lives under the `gravity['galaxy_config']['gx_it_proxy']` section. The available settings are:
>
>
>    | Setting         | Type    | Description                                                                                                                                                                                                                                         |
>    |-----------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
>    | `enable`        | bool    | Set to `true` to enable the GxIT Proxy. Defaults to `false`.                                                                                                                                                                                           |
>    | `version`       | str     | Version specifier for the GxIT Proxy Node.js package. Defaults to `>=0.0.6`.                                                                                                                                                                         |
>    | `ip`            | str     | Public-facing IP of the proxy. Defaults to `localhost`.                                                                                                                                                                                             |
>    | `port`          | int     | Public-facing port of the proxy. Defaults to `4002`.                                                                                                                                                                                                |
>    | `sessions`      | str     | Path to the Interactive Tools Session Map database. Defaults to `database/interactivetools_map.sqlite`.                                                                                                                                             |
>    | `verbose`       | bool    | Include verbose messages in the GxIT Proxy. Defaults to `true`.                                                                                                                                                                                      |
>    | `forward_ip`    | str     | Forward all requests to this IP. This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network.                                                                |
>    | `forward_port`  | int     | Forward all requests to this port. This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network.                                                              |
>    | `reverse_proxy` | str     | Rewrite location blocks with proxy port. This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network. Defaults to `false`.                                   |
>    | `umask`         | str     | Permissions mask under which service should be executed.                                                                                                                                                                                            |
>    | `start_timeout` | int     | Value of supervisor startsecs, systemd TimeoutStartSec. Defaults to `10`.                                                                                                                                                                           |
>    | `stop_timeout`  | int     | Value of supervisor stopwaitsecs, systemd TimeoutStopSec. Defaults to `10`.                                                                                                                                                                         |
>    | `memory_limit`  | float   | Memory limit (in GB). If the service exceeds the limit, it will be killed. Default is no limit or the value of the ``memory_limit`` setting at the top level of the Gravity configuration, if set. Ignored if ``process_manager`` is ``supervisor``. |
>    | `environment`   | mapping | Extra environment variables and their values to set when running the service. A dictionary where keys are the variable names.                                                                                                                       |
>
>    Configure the settings in the `group_vars/galaxyservers.yml` file:
>
>    {% raw %}
>    ```yaml
>    # ... other settings ... #
>    galaxy_config:
>      galaxy:
>        # ... other settings ... #
>        # Interactive Tools
>        interactivetools_enable: true
>        interactivetools_map: "{{ gxit_proxy_sessions_path }}"
>        # ... other settings ... #
>      gravity:
>        # ... other settings ... #
>        gx_it_proxy:
>          enable: true
>          port: 4002
>          sessions: "{{ gxit_proxy_sessions_path }}"
>        # ... other settings ... #
>      # ... other settings ... # 
>    # ... other settings ... #
>    gxit_proxy_sessions_path: "{{ galaxy_mutable_data_dir }}/interactivetools_map.sqlite"
>    ```
>    {% endraw %}
>
> 2. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
{: .hands_on}

[nodeenv]: https://github.com/ekalinin/nodeenv
[path-based_interactive_tools]: https://docs.galaxyproject.org/en/master/admin/special_topics/interactivetools.html#path-based-interactivetools

Because the proxy runs as a systemd service, you can inspect the log of the service using `journalctl`. The service name is `galaxy-gx-it-proxy`:

```console
$ sudo journalctl -eu galaxy-gx-it-proxy
Feb 14 17:38:49 gcc-4 systemd[1]: Started galaxy-gx-it-proxy.service - Galaxygx-it-proxy
...
Feb 14 17:38:49 gcc-4 node[3679]: Watching path /srv/galaxy/var/interactivetools_map.sqlite
Feb 14 17:38:49 gcc-4 node[3679]: Listening on localhost:4002
```

> <comment-title>Note</comment-title>
>
> You can ignore errors about failing to read the sessions map file for now - Galaxy will create it when it's needed.
>
{: .comment}

# Proxying the Proxy

As explained in the previous section, we will proxy the Interactive Tools Proxy with nginx so that it can serve requests on the standard HTTPS port, 443. Because we've configured nginx with Ansible, this is relatively simple.

> <hands-on-title>Configuring NGINX</hands-on-title>
>
> 1. Edit the group variables file, `group_vars/galaxyservers.yml` and add a new item to the **existing** `nginx_ssl_servers` so it matches:
>
>    {% raw %}
>    ```yaml
>    nginx_ssl_servers:
>      - galaxy
>      - galaxy-gxit-proxy
>    ```
>    {% endraw %}
>
>    The nginx configuration `galaxy-gxit-proxy` doesn't exist yet, but we'll create it in a moment.
>
> 2. Create `templates/nginx/galaxy-gxit-proxy.j2` with the following contents:
>
>    {% raw %}
>    ```nginx
>    server {
>        # Listen on port 443
>        listen       *:443 ssl;
>        # Match all requests for the interactive tools subdomain
>        server_name  *.interactivetool.{{ inventory_hostname }};
>
>        # Our log files will go here.
>        access_log  syslog:server=unix:/dev/log;
>        error_log   syslog:server=unix:/dev/log;
>
>        # Proxy all requests to the GxIT Proxy application
>        location / {
>            proxy_redirect off;
>            proxy_http_version 1.1;
>            proxy_set_header Host $host;
>            proxy_set_header X-Real-IP $remote_addr;
>            proxy_set_header Upgrade $http_upgrade;
>            proxy_set_header Connection "upgrade";
>            proxy_pass http://localhost:{{ galaxy_config.gravity.gx_it_proxy.port }};
>        }
>    }
>    ```
>    {% endraw %}
>
> 3. To enable [Path-based Interactive Tools][path-based_interactive_tools], open `templates/nginx/galaxy.j2` and add the following contents:
>
>    {% raw %}
>    ```nginx
>    server {
>        # ... existing settings ...
>
>        # Route all path-based interactive tool requests to the InteractiveTool proxy application
>        location ~* ^/(interactivetool/.+)$ {
>            proxy_redirect off;
>            proxy_http_version 1.1;
>            proxy_set_header Host $host;
>            proxy_set_header X-Real-IP $remote_addr;
>            proxy_set_header Upgrade $http_upgrade;
>            proxy_set_header Connection "upgrade";
>            proxy_pass http://localhost:{{ galaxy_config.gravity.gx_it_proxy.port }};
>        }
>
>        # ... other existing settings ...
>    }
>    ```
>    {% endraw %}
>
> 4. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
{: .hands_on}

# Getting a Wildcard SSL Certificate

During the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, we acquired an SSL certificate for our Galaxy server from [Let's Encrypt][lets-encrypt]. This certificate was issued for the hostname of your Galaxy server (e.g. `galaxy.example.org`). SSL certificates are valid *only for the name to which they were issued*. This presents a problem for us due to the way that Galaxy Interactive Tools work.

In order to ensure each Interactive Tool's cookies are unique, and to provide each tool with a unique entry point, they are served from a subdomain of your Galaxy server (e.g. `<unique-id>.ep.interactivetool.galaxy.example.org`). Your SSL cert is not valid for this subdomain. Further, in order to support the random `<unique-id>` in the hostname, we need a *wildcard certificate* for `*.ep.interactivetool.galaxy.example.org`.

This process is highly dependent on your site; specifically, your SSL certificate vendor, and your DNS server software or cloud provider.

Let's Encrypt, the SSL certificate vendor we use in our tutorials, [can only generate wildcard certificates using the DNS-01 challenge method][lets-encrypt-faq], which works by issuing a [dynamic DNS][ddns] update to set the requested domain's `TXT` record.

If you are completing this tutorial as part of a [Galaxy Admin Training][gat] course, we might have precreated a dynamic DNS server that you will use for this step. The *TSIG key* that allows you to perform dynamic DNS updates will be provided to you. Your instructor will also tell you which option to follow (1 or 2), depending on the DNS provider that was chosen for this course.

As we use Let's Encrypt in staging mode, the wildcard certificates generated with either option 1 or 2 will still be invalid, and you will still see a warning in your web browser when accessing an Interactive Tool. If this warning is not a problem for you, you can just skip this section of the tutorial, and move on to "Enabling Interactive Tools in Galaxy".

> <hands-on-title>Requesting a Wildcard Certificate with Certbot using Ansible - Option 1 (rfc2136)</hands-on-title>
>
> This method uses a DNS provider hosted by the Galaxy Project.
>
> 1. Edit the group variables file, `group_vars/galaxyservers.yml`:
>
>    The relevant variables to set for this role are:
>
>    | Variable                  | Type       | Description                                                                                                      |
>    | ----------                | -------    | -------------                                                                                                    |
>    | `certbot_domains`         | list       | List of domains to include as subject alternative names (the first will also be the certificate's *common name*) |
>    | `certbot_dns_provider`    | string     | Name of [Certbot DNS plugin][certbot-dns-plugins] to use                                                         |
>    | `certbot_dns_credentials` | dictionary | Plugin-specific credentials for performing dynamic DNS updates                                                   |
>    | `certbot_expand`          | boolean    | Whether to "expand" an existing certificate (add new domain names to it)                                         |
>
>    - Add a new item to the **existing** `certbot_domains` list so it matches:
>
>      {% raw %}
>      ```yaml
>      certbot_domains:
>        - "{{ inventory_hostname }}"
>        - "*.ep.interactivetool.{{ inventory_hostname }}"
>      ```
>      {% endraw %}
>
>    - Comment out the existing `certbot_auth_method` like so:
>
>      ```yaml
>      #certbot_auth_method: --webroot
>      ```
>
>        Although this is not explicitly required (setting `certbot_dns_provider` as we do overrides this setting), doing so is less confusing in the future, since it makes it clear that the "webroot" method for Let's Encrypt WEB-01 challenges is no longer in use for this server.
>
>    - Add the following lines to your `group_vars/galaxyservers.yml` file:
>
>      ```yaml
>      certbot_dns_provider: rfc2136
>      certbot_dns_credentials:
>        server: ns-training.galaxyproject.org
>        port: 53
>        name: certbot-training.
>        secret: <SECRET PROVIDED BY INSTRUCTOR>
>        algorithm: HMAC-SHA512
>      ```
>
> 2. Run the playbook **with `certbot_expand`**:
>
>    ```
>    ansible-playbook galaxy.yml -e certbot_expand=true
>    ```
>
>    > <question-title></question-title>
>    >
>    > What is the `-e` flag to `ansible-playbook` and why did we use it?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > As per `ansible-playbook --help`:
>    > >
>    > > ```
>    > >   -e EXTRA_VARS, --extra-vars EXTRA_VARS
>    > >                         set additional variables as key=value or YAML/JSON, if
>    > >                         filename prepend with @
>    > > ```
>    > >
>    > > We used this flag because `certbot_expand` only needs to be set *once*, when we are adding a new domain to the certificate. It should not be enabled on subsequent runs of the playbook, or else we would request a new certificate on each run! Thus, it does not make sense to add it to a vars file.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>    Be patient! The certificate request step can take time due to the time allowed for DNS propagation to occur.
>
{: .hands_on}


> <hands-on-title>Requesting a Wildcard Certificate with Certbot using Ansible - Option 2 (route53)</hands-on-title>
>
> This method uses route53, the Amazon Web Services DNS provider. To manage connection to AWS, we will first install a specific role.
>
> 1. In your working directory, add the aws_cli role to your `requirements.yml`:
>
>    ```yaml
>    - src: usegalaxy_eu.aws_cli
>      version: 0.0.1
>    ```
>
> 2. Install the requirements with `ansible-galaxy`:
>
>    ```
>    ansible-galaxy role install -p roles -r requirements.yml
>    ```
> 3. Open `galaxy.yml` with your text editor to add the role `usegalaxy_eu.aws_cli` just before the nginx role:
>
>    ```diff
>    diff --git a/galaxy.yml b/galaxy.yml
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -21,6 +21,7 @@
>           become: true
>           become_user: galaxy
>         - usegalaxy_eu.galaxy_systemd
>    +    - usegalaxy_eu.aws_cli
>         - galaxyproject.nginx
>         - geerlingguy.docker
>    ```
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 4. Edit the group variables file, `group_vars/galaxyservers.yml`:
>
>    The relevant variables to set for this role are:
>
>    | Variable                  | Type       | Description                                                                                                      |
>    | ----------                | -------    | -------------                                                                                                    |
>    | `certbot_domains`         | list       | List of domains to include as subject alternative names (the first will also be the certificate's *common name*) |
>    | `certbot_dns_provider`    | string     | Name of [Certbot DNS plugin][certbot-dns-plugins] to use                                                         |
>    | `certbot_dns_credentials` | dictionary | Plugin-specific credentials for performing dynamic DNS updates                                                   |
>    | `certbot_expand`          | boolean    | Whether to "expand" an existing certificate (add new domain names to it)                                         |
>
>    - Add a new item to the **existing** `certbot_domains` list so it matches:
>
>      {% raw %}
>      ```yaml
>      certbot_domains:
>        - "{{ inventory_hostname }}"
>        - "*.ep.interactivetool.{{ inventory_hostname }}"
>      ```
>      {% endraw %}
>
>    - Comment out the existing `certbot_auth_method` like so:
>
>      ```yaml
>      #certbot_auth_method: --webroot
>      ```
>
>        Although this is not explicitly required (setting `certbot_dns_provider` as we do overrides this setting), doing so is less confusing in the future, since it makes it clear that the "webroot" method for Let's Encrypt WEB-01 challenges is no longer in use for this server.
>
>    - Add the following lines to your `group_vars/galaxyservers.yml` file:
>
>      ```yaml
>      certbot_dns_provider: route53
>      aws_cli_credentials:
>        - access_key: "<SECRET PROVIDED BY INSTRUCTOR>"
>          secret_key: "<SECRET PROVIDED BY INSTRUCTOR>"
>          homedir: /root
>          owner: root
>          group: root
>      ```
>
> 5. Run the playbook **with `certbot_expand`**:
>
>    ```
>    ansible-playbook galaxy.yml -e certbot_expand=true
>    ```
>
>    > <question-title></question-title>
>    >
>    > What is the `-e` flag to `ansible-playbook` and why did we use it?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > As per `ansible-playbook --help`:
>    > >
>    > > ```
>    > >   -e EXTRA_VARS, --extra-vars EXTRA_VARS
>    > >                         set additional variables as key=value or YAML/JSON, if
>    > >                         filename prepend with @
>    > > ```
>    > >
>    > > We used this flag because `certbot_expand` only needs to be set *once*, when we are adding a new domain to the certificate. It should not be enabled on subsequent runs of the playbook, or else we would request a new certificate on each run! Thus, it does not make sense to add it to a vars file.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

You can verify that your certificate has been expanded using your browser's developer tools:

![Wildcard Certificate Dialog](../../images/interactive-tools/wildcard-cert.png "Wildcard Certificate Dialog")

# Enabling Interactive Tools in Galaxy

A few Interactive Tool wrappers are provided with Galaxy, but they are [commented out in Galaxy's default tool config file][tool-conf-gxits]. As a result, we need to instruct the [galaxyproject.galaxy role][galaxy-role] to install a tool panel configuration file containing at least one of these tools in order to try them out. For the purposes of this tutorial, a good choice is the [EtherCalc][ethercalc] GxIT, because it has a relatively small [Docker image][ethercalc-docker-image].

[tool-conf-gxits]: https://github.com/galaxyproject/galaxy/blob/9d788cb144a18570e7e5e948ac1f7bc04fec4e75/lib/galaxy/config/sample/tool_conf.xml.sample#L126
[galaxy-role]: https://github.com/galaxyproject/ansible-galaxy
[ethercalc]: https://ethercalc.net/
[ethercalc-docker-image]: https://hub.docker.com/r/shiltemann/ethercalc-galaxy-ie

> <hands-on-title>Enabling Interactive Tools in Galaxy</hands-on-title>
>
> 1. Rather than modifying the default tool configuration file, we'll add a new one that only references the Interactive Tools. This way, the default set of tools will still load without us having to incorporate the entire default tool config into our playbook.
>
>    If the folder does not exist, create `templates/galaxy/config` next to your `galaxy.yml` (`mkdir -p templates/galaxy/config/`)
>
>    Create `templates/galaxy/config/tool_conf_interactive.xml.j2` with the following contents:
>
>    ```xml
>    <toolbox monitor="true">
>        <section id="interactivetools" name="Interactive Tools">
>            <tool file="interactive/interactivetool_ethercalc.xml" />
>            <tool file="interactive/interactivetool_jupyter_notebook_1.0.1.xml" />
>        </section>
>    </toolbox>
>    ```
>
> 2. We need to instruct Galaxy on how to run Interactive Tools (and specifically, how to run them in Docker). Let's begin with a basic job configuration:
>
>    ```yaml
>    # ... other settings ... #
>    # Galaxy Job Configuration
>    galaxy_job_config:
>      runners:
>        local_runner:
>          load: galaxy.jobs.runners.local:LocalJobRunner
>          workers: 4
>      execution:
>        default: local_env
>        environments:
>          local_env:
>            runner: local_runner
>    # ... other settings ... #
>    ```
>
>    > <comment-title>Note</comment-title>
>    > Depending on the order in which you are completing this tutorial in relation to other tutorials, you may have already defined the `galaxy_job_config` mapping in your `group_vars/galaxyservers.yml` file, as well as `galaxy_config_templates`. If this is the case, be sure to **merge the changes in this section with your existing playbook**.
>    {: .comment}
>
> 3. Next, we need to configure the interactive tools destination. Add a destination for submitting jobs as Docker containers using the [advanced sample job configuration][job-conf-docker] as a guide. Finally, use the [EtherCalc GxIT's][ethercalc-tool-wrapper] tool ID to route executions of the EtherCalc GxIT to the newly created destination:
>
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>         execution:
>           default: local_env
>           environments:
>             local_env:
>               runner: local_runner
>    +        local_interactive_env:
>    +          runner: local_runner
>    +          params:
>    +            docker_enabled: true
>    +            docker_volumes: $defaults
>    +            docker_sudo: false
>    +            docker_net: bridge
>    +            docker_auto_rm: true
>    +            docker_set_user: ""
>    +            require_container: true
>    +    tools:
>    +      - id: interactive_tool_ethercalc
>    +        environment: local_interactive_env
>    +      - id: interactive_tool_jupyter_notebook
>    +        environment: local_interactive_env
>    ```
>
>    Of considerable note is the `docker_volumes` param: the variable expansions are explained in the [advanced sample job configuration][job-conf-docker].  We'll use this volume configuration for now but it has some considerable data security problems. We'll discuss a better solution at the end of this tutorial.
>
> 4. Inform `galaxyproject.galaxy` of what tool configuration files to load in your group variables (`group_vars/galaxyservers.yml`):
>
>    {% raw %}
>    ```yaml
>    galaxy_tool_config_files:
>      - "{{ galaxy_server_dir }}/config/tool_conf.xml.sample"
>      - "{{ galaxy_config_dir }}/tool_conf_interactive.xml"
>    ```
>    {% endraw %}
>
>    Next, deploy the new config template using the `galaxy_config_templates` var in your group vars:
>
>    {% raw %}
>    ```yaml
>    galaxy_config_templates:
>      # ... possible existing config file definitions
>      - src: templates/galaxy/config/tool_conf_interactive.xml.j2
>        dest: "{{ galaxy_config_dir }}/tool_conf_interactive.xml"
>    ```
>    {% endraw %}
>
> 5. If you've configured container resolvers, make sure the explit docker container resolver is enabled, e.g in templates/galaxy/config/container_resolvers_conf.yml.j2
>    {% raw %}
>    ```diff
>    --- a/templates/galaxy/config/container_resolvers_conf.yml.j2
>    +++ b/templates/galaxy/config/container_resolvers_conf.yml.j2
>    @@ -9,3 +9,4 @@
>     - type: build_mulled_singularity
>       auto_install: False
>       cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/built/"
>     +- type: explicit
>    ```
>    {% endraw %}
>
> 6. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
> 7. Follow the Galaxy logs with `galaxyctl follow`.
>
{: .hands_on}

[job-conf-docker]: https://github.com/galaxyproject/galaxy/blob/a2c847439528df0becf9e382c323f1dfe34d9633/lib/galaxy/config/sample/job_conf.xml.sample_advanced#L495
[ethercalc-tool-wrapper]: https://github.com/galaxyproject/galaxy/blob/6622ad1acb91866febb3d2f229de7cfb8af3a9f6/tools/interactive/interactivetool_ethercalc.xml

# Run an Interactive Tool

You should now be ready to run an Interactive Tool in Galaxy!

> <hands-on-title>Running an Interactive Tool</hands-on-title>
>
> 1. Ensure that you are logged in to your Galaxy server by checking the **User** menu in the masthead.
> 2. We'll need an input for our test GxIT (EtherCalc). Any tabular file can be used, such as Galaxy's [1.tabular][1-tabular] test data. Copy this file's URL:
>
>    ```
>    https://raw.githubusercontent.com/galaxyproject/galaxy/release_20.01/test-data/1.tabular
>    ```
>
> 3. Click {% icon galaxy-upload %} **Upload** at the top of the tool panel (on the left side of the Galaxy UI).
> 4. In the resulting modal dialog, click the **Paste/Fetch data** button.
> 5. Paste the URL in the **text field** that has just appeared.
> 6. Give the new dataset a name such as `tabular`, if you like.
> 7. Click **Start** and then **Close**.
> 8. From the tool menu, click the **Interactive Tools** section, then click **EtherCalc** {% icon tool %}.
> 9. Ensure that your newly uploaded tabular dataset is selected as the input **Some tabular dataset**, then click **Execute**.
> 10. Monitor the blue **info box** on the next page, which will inform you when the Interactive Tool is accessible and provide you with a link to access it.
>
>     <br/>
>
>     If you navigate away from this page, you can view your running Interactive Tools from the **Active InteractiveTools** menu item in the **User** menu.
> 11. Click the **click here to display** link.
>
{: .hands_on}

If everything has worked correctly, your browser will load EtherCalc with your tabular data preloaded. Once you're done working with the data, return to Galaxy and stop EtherCalc by deleting its output dataset from your history, or stopping it via the interface from the **Active InteractiveTools** menu item in the **User** menu.

[1-tabular]: https://raw.githubusercontent.com/galaxyproject/galaxy/release_20.01/test-data/1.tabular

# Securing Interactive Tools

Inspecting the Docker container of a running Interactive Tool shows the volume configuration expanded from `$galaxy_root` in the job destination's `docker_volumes` param:

```console
$ docker inspect $(docker ps -q) | jq '.[0].HostConfig.Binds'
[
  "/srv/galaxy/server:/srv/galaxy/server:ro",                                     # Galaxy server dir
  "/srv/galaxy/server/tools/interactive:/srv/galaxy/server/tools/interactive:ro", # EtherCalc tool wrapper parent dir
  "/srv/galaxy/jobs/000/1:/srv/galaxy/jobs/000/1:ro",                             # Per-job root dir
  "/srv/galaxy/jobs/000/1/outputs:/srv/galaxy/jobs/000/1/outputs:rw",             # Job outputs dir
  "/srv/galaxy/jobs/000/1/configs:/srv/galaxy/jobs/000/1/configs:rw",             # Job config files dir
  "/srv/galaxy/jobs/000/1/working:/srv/galaxy/jobs/000/1/working:rw",             # Job working (cwd) dir
  "/data:/data:rw",                                                               # GALAXY USER DATASETS DIR (RW!)
  "/srv/galaxy/server/tool-data:/srv/galaxy/server/tool-data:ro"                  # Galaxy reference data dir
]
```

As hinted earlier, there is a concerning state here: The directory containing **all** of the the user-generated data in Galaxy (not just the data for this job) has been mounted **read-write** in to the container. **This configuration grants users running interactive tools full access to all the data in Galaxy, which is a very bad idea.** Unlike standard Galaxy tools, where the tool's design prevents users from writing to arbitrary paths, Interactive Tools are fully user controllable. Although EtherCalc does not provide a mechanism for writing to this path, other Interactive Tools (such as Jupyter Notebook) do.

Two solutions are discussed in the [advanced sample job configuration][job-conf-docker]:

1. Use the `outputs_to_working_directory` job configuration option, which allows you to mount datasets read-only: this prevents manipulation, but still allows GxIT users to read any dataset in your Galaxy server.
2. Use [Pulsar][pulsar], Galaxy's remote job execution engine, to provide full job isolation: this avoids all access to Galaxy data, with the performance penalty of copying input dataset(s) to the job directory.

Because we want to maintain dataset privacy, Pulsar is the better choice here. And in fact, we don't even need to set up a Pulsar server: because we only need Pulsar's input staging and isolation features, we can use [Embedded Pulsar][job-conf-pulsar-embedded], which runs a Pulsar server within the Galaxy application to perform these tasks. Embedded Pulsar can even interface with your distributed resource manager (aka cluster scheduler) of choice, as long as your Galaxy server and cluster both have access to a common filesystem (otherwise, you will need to use Pulsar in standalone mode; see the [Running Jobs on Remote Resources with Pulsar]({% link topics/admin/tutorials/pulsar/tutorial.md %}) tutorial).

[pulsar]: https://github.com/galaxyproject/pulsar
[job-conf-pulsar-embedded]: https://github.com/galaxyproject/galaxy/blob/6622ad1acb91866febb3d2f229de7cfb8af3a9f6/lib/galaxy/config/sample/job_conf.xml.sample_advanced#L106

> <hands-on-title>Running Interactive Tools with Embedded Pulsar</hands-on-title>
>
> 1. Create a configuration file *template* for the Pulsar application at `templates/galaxy/config/pulsar_app.yml.j2`.
>
>    <br/>
>
>    If the folder does not exist, create `templates/galaxy/config` next to your `galaxy.yml` (`mkdir -p templates/galaxy/config/`).
>
>    <br/>
>
>    Add the following contents to the template:
>
>    {% raw %}
>    ```yaml
>    ---
>
>    # The path where per-job directories will be created
>    staging_directory: "{{ galaxy_job_working_directory }}/_interactive"
>
>    # Where Pulsar state information will be stored (e.g. currently active jobs)
>    persistence_directory: "{{ galaxy_mutable_data_dir }}/pulsar"
>
>    # Where to find Galaxy tool dependencies
>    tool_dependency_dir: "{{ galaxy_tool_dependency_dir }}"
>
>    # How to run jobs (see https://pulsar.readthedocs.io/en/latest/job_managers.html)
>    managers:
>      _default_:
>        type: queued_python
>        num_concurrent_jobs: 1
>    ```
>    {% endraw %}
>
> 2. Modify the job configuration in `group_vars/galaxyservers.yml` to configure Interactive Tools to use the embedded Pulsar runner.
>
>    <br/>
>
>    **Add** the embedded Pulsar runner to the `galaxy_job_config` section of the file:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>         runners:
>           local_runner:
>             load: galaxy.jobs.runners.local:LocalJobRunner
>             workers: 4
>    +      pulsar_embedded:
>    +        load: galaxy.jobs.runners.pulsar:PulsarEmbeddedJobRunner
>    +        pulsar_config: "{{ galaxy_config_dir }}/pulsar_app.yml"
>         execution:
>    ```
>    {% endraw %}
>
>    Next, **modify** the `interactive_local` destination to use the new runner and set the new parameter `container_monitor_result` to `callback` (explained in more detail in the next step):
>
>    > <warning-title>Untrusted SSL Certificates</warning-title>
>    > If you are completing this tutorial as part of a [Galaxy Admin Training][gat] course, you will also need the `env` setting shown below to prevent problems with the untrusted SSL certificates in use during the course. Galaxy servers with valid SSL certificates *do not need this option*.
>    {: .warning}
>
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>         execution:
>           default: local_env
>           environments:
>             local_env:
>               runner: local_runner
>             local_interactive_env:
>    -          runner: local_runner
>    +          runner: pulsar_embedded
>               params:
>                 docker_enabled: true
>                 docker_volumes: $defaults
>                 docker_sudo: false
>                 docker_net: bridge
>                 docker_auto_rm: true
>                 docker_set_user: ""
>                 require_container: true
>    +            container_monitor_result: callback
>    +          env:
>    +            - name: REQUESTS_CA_BUNDLE
>    +              value: /etc/ssl/certs/ca-certificates.crt
>         tools:
>    ```
>
> 3. Open your `galaxyservers` group variables file and instruct `galaxyproject.galaxy` to install the Pulsar configuration file:
>
>    > <comment-title>Note</comment-title>
>    > Depending on the order in which you are completing this tutorial in relation to other tutorials, you may have already defined `galaxy_config_templates`. If this is the case, be sure to **merge the changes in this step with your existing playbook**.
>    {: .comment}
>
>    {% raw %}
>    ```yaml
>    galaxy_config_templates:
>      - src: templates/galaxy/config/pulsar_app.yml.j2
>        dest: "{{ galaxy_config_dir }}/pulsar_app.yml"
>    ```
>    {% endraw %}
>
>    Additionally, you will need to set the `galaxy_infrastructure_url` config option:
>
>    {% raw %}
>    ```yaml
>    galaxy_config:
>      galaxy:
>        # ... existing configuration options in the `galaxy` section ...
>        galaxy_infrastructure_url: "https://{{ inventory_hostname }}/"
>    ```
>    {% endraw %}
>
>    > <details-title>Detail: Infrastructure URL/Callback</details-title>
>    > Galaxy must be made aware of the randomly selected port Docker has assigned after the GxIT begins operating, in order to update the proxy map. By default, this is done by writing a JSON file in the job directory. This method does not work with Pulsar since Pulsar uses a different job directory from the Galaxy job directory. As a result, Pulsar jobs use the `callback` method configured in the previous step to make a request to Galaxy's API, the URL for which is set in `galaxy_infrastructure_url`.
>    {: .details}
>
> 4. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
{: .hands_on}

Once the playbook run is complete and your Galaxy server has restarted, run the EtherCalc Interactive Tool again.

> <question-title></question-title>
>
> Once EtherCalc is running, check the mounts of its container. What do you observe?
>
> > <solution-title></solution-title>
> >
> > ```console
> > $ docker inspect $(docker ps -q) | jq '.[0].HostConfig.Binds'
> > [
> >   "/srv/galaxy/jobs/_interactive/2:/srv/galaxy/jobs/_interactive/2:ro",                       # Per-job root dir
> >   "/srv/galaxy/jobs/_interactive/2/tool_files:/srv/galaxy/jobs/_interactive/2/tool_files:ro", # EtherCalc tool wrapper parent dir
> >   "/srv/galaxy/jobs/_interactive/2/outputs:/srv/galaxy/jobs/_interactive/2/outputs:rw",       # Job outputs dir
> >   "/srv/galaxy/jobs/_interactive/2/working:/srv/galaxy/jobs/_interactive/2/working:rw",       # Job working (cwd) dir
> >   "/srv/galaxy/server/tool-data:/srv/galaxy/server/tool-data:ro"                              # Galaxy reference data dir
> > ]
> > ```
> >
> > Of note, the user data directory, `/data`, is no longer mounted in the container!
> >
> {: .solution }
>
{: .question }

# High availability setup with PostgresSQL (Optional)

> <comment-title></comment-title>
> This section is **only relevant if you are running a high-availability** setup, meaning that you have multiple copies of Galaxy running behind a load balancer.
>
> If you have installed Galaxy following the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, or are completing this tutorial as part of a [Galaxy Admin Training][gat] course, please skip this section, as you are then _not_ running a high-availability setup.
{: .comment}

In a _high availability_ setup, multiple redundant copies of Galaxy run simultaneously behind a load balancer to minimize downtime and service interruptions.

As explained in [one of the previous sections](#enabling-the-interactive-tools-proxy), the Galaxy Interactive Tools Proxy redirects requests to each Interactive Tool's host and port. By default, the mapping of GxIT invocations to their corresponding host/port is kept in a SQLite database known as the _Interactive Tools Session Map_.

By design, [SQLite is the wrong choice for high availability setups][sqlite_situations_where_a_client_server_rdbms_may_work_better], the showstopper being that the SQLite database file would have to be shared over a network filesystem, which are usually associated with too high latencies for RDBMS use. For this reason, Galaxy and the Interactive Tools Proxy can also store the **Session Map in a PostgreSQL database**. 

[sqlite_situations_where_a_client_server_rdbms_may_work_better]: https://www.sqlite.org/whentouse.html#situations_where_a_client_server_rdbms_may_work_better

> <hands-on-title>Preparing the database</hands-on-title>
> 
> First, you need to create a database for the Interactive Tools Proxy. 
>
> > <warning-title></warning-title>
> > Do **not** use the Galaxy database for this purpose. The main Galaxy database is reserved for Galaxy's core functionality, and Interactive Tools have not yet reached this stage. Since Galaxy does not expect to find the Interactive Tools Session Map in this database, storing it there can lead to errors.
> {: .warning }
>
> <br>
>
> 1. **On your database server**, access PostgresSQL and create a `gxitproxy` database to store the Interactive Tools Session Map. For simplicity, the same user that operates on the Galaxy main database, typically named `galaxy`, is also going to operate on this one and will own the new database.
> 
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > # one-liner that connects to Postgres, creates the database and assigns ownership
>    > sudo -u postgres createdb -O galaxy gxitproxy
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Connect to the `gxitproxy` database as `galaxy`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sudo -iu galaxy psql -d gxitproxy
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>SQL</code-out-title>
>    > ```
>    > psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
>    > Type "help" for help.
>    >
>    > gxitproxy=#
>    > ```
>    {: .code-out}
>
> 3. Create a `gxitproxy` table in the new database.
>
>    > <code-in-title>SQL</code-in-title>
>


topic: admin
tutorial: pulsar
title: Running Jobs on Remote Resources with Pulsar
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/pulsar/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["type: \"internal\"", "title: \"A server/VM on which to deploy Pulsar\""]
tags_json: ["ansible", "jobs", "git-gat"]
content_hash: 987c059cfcc6f80e947ac66db458671f
last_modified: 2026-05-27T09:01:47.638118
workflows_json: []
content:
Pulsar is the Galaxy Project's remote job running system. It was written by John Chilton ([@jmchilton](https://github.com/jmchilton)) of the Galaxy Project. It is a python server application that can accept jobs from a Galaxy server, submit them to a local resource and then send the results back to the originating Galaxy server.

More details on Pulsar can be found at:

- [Pulsar's Documentation](https://pulsar.readthedocs.io/en/latest/index.html)
- [Pulsar's Github Repository](https://github.com/galaxyproject/pulsar)
- [Pulsar Ansible Role](https://github.com/galaxyproject/ansible-pulsar)

Transport of data, tool information and other metadata can be configured as a web application via a RESTful interface or using a message passing system such as RabbitMQ.

At the Galaxy end, it is configured within the job configuraiton, and uses one of two special Galaxy job runners.
* `galaxy.jobs.runners.pulsar:PulsarRESTJobRunner` for the RESTful interface
* `galaxy.jobs.runners.pulsar:PulsarMQJobRunner` for the message passing interface.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="pulsar" %}

## This tutorial assumes that

- You have a VM or machine where you will install Pulsar, and a directory in which the installation will be done. This tutorial assumes it is `/mnt`
- You have completed the "Galaxy Installation with Ansible", "Connecting Galaxy to a Compute Cluster", and the "CVMFS" tutorials
- You have access to the VM/computer where it is installed.

> <tip-title>This is NOT intended as a standalone Pulsar guide</tip-title>
> This tutorial is not intended to be a standalone Pulsar setup guide. If you read carefully and understand Ansible, it is likely you can figure out which portions are required to just setup Pulsar.
{: .tip}

## Overview

We will be installing the RabbitMQ server daemon onto the Galaxy server to act as an intermediary message passing system between Galaxy and the remote Pulsar. The figure below shows a schematic representation of the system.

![Schematic diagram of Galaxy communicating with a Pulsar server via the RabbitMQ server](../../images/pulsar_amqp_schema.png "Schematic diagram of Galaxy communicating with a Pulsar server via the RabbitMQ server. Red arrows represent AMQP communications and blue represent file transfers (initiated by the Pulsar server.)")


### How it will work

1. Galaxy will send a message to the RabbitMQ server on the Pulsar server's particular queue saying that there is a job to be run and then will monitor the queue for job status updates.
2. The Pulsar server monitors this queue and when the job appears it will take control of it.
3. The Pulsar server will then download the required data etc. from the Galaxy server using `curl`.
4. The Pulsar server will install any required tools/tool dependencies using Conda.
5. The Pulsar server will start running the job using it's local mechanism and will send a message to the "queue" stating that the job has started.
6. Once the job has finished running, the Pulsar server will send a message to the queue stating that the job has finished.
7. Pulsar then sends the output data etc. back to the Galaxy server by `curl` again.
8. The Galaxy server acknowledges the job status and closes the job.

### Some notes

* RabbitMQ uses the Advanced Message Queueing Protocol (AMQP) to communicate with both the Galaxy server and the remote Pulsar VM.
* Transport of files, meta-data etc. occur via `curl` from the Pulsar end.
* RabbitMQ is written in erlang and does not add much overhead to the Galaxy VM, although in larger installations, RabbitMQ is commonly installed on a separate VM to Galaxy. e.g. Galaxy Europe, Galaxy Main and Galaxy Australia.

> <tip-title>Other file transport methods for Pulsar</tip-title>
>
>  Pulsar can use a variety of file transport methods including:
>  * Default: Galaxy initiates file transfer and stages files to Pulsar via http transfer.
>      * This requires that a http transfer port be open on the remote Pulsar.
>  * Remote transfer: Pulsar initiates file transfer. This can use a variety of lso available and can use a variety of methods:
>      * Curl
>      * Rsync
>      * Http
>
>  We use remote transfer using **Curl** here so we don't need an open port on the Pulsar server and tranfer robustness respectively.
>
{: .tip}


> <details-title>Why are we using Pulsar in MQ mode here and not the RESTful interface?</details-title>
> We are teaching you to install Pulsar and configure it in MQ mode in this tutorial. Configuring Pulsar in RESTful mode is also possible and is quite useful in certain situations. However, in the most common situation MQ mode is preferable for a number of reasons:
> * When running Pulsar in RESTful mode, all of the job control and data transfer is controlled by the Galaxy server usually using http transfers. This can place a limit on the size of files that can be transferred without constant configuring of the webserver.
> * When running in RESTful mode, Pulsar also needs to have an https server such as nginx, including securing it, configuring it, getting certificates and opening ports. This can be very difficult to do if you are attempting to submit jobs to an institutional HPC where the admins probably won't let you do any of these things.
> * In MQ mode, you only need to open a port for the RabbitMQ server on a machine you are more likely to control. The HPC side running Pulsar can just connect back to you.
>
> See the [Pulsar documentation](https://pulsar.readthedocs.io/en/latest/) for details.
{: .details}

## Install and configure a message queueing system

In this section we will install the RabbitMQ server on your Galaxy server VM.

RabbitMQ is an AMQP server that can queue messages between systems for all sorts of reasons. Here, we will be using the queue so that Galaxy and Pulsar can communicate jobs, job status and job metadata between them easily and robustly. More information on RabbitMQ can be found [on their website](https://www.rabbitmq.com/).

### Installing the roles

Firstly we will add and configure another *role* to our Galaxy playbook - a community role which runs RabbitMQ in docker and configures it. In order to have docker installed automatically, we use geelingguy's docker role, which luckly needs no configuration changes. Additionally we will use the Galaxy community role for deploying Pulsar

> <hands-on-title>Install the Ansible roles</hands-on-title>
>
> 1. From your ansible working directory, edit the `requirements.yml` file and add the following lines:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -30,3 +30,10 @@
>     # TPV Linting
>     - name: usegalaxy_eu.tpv_auto_lint
>       version: 0.4.3
>    +# RabbitMQ for Pulsar
>    +- src: geerlingguy.docker
>    +  version: 6.1.0
>    +- src: usegalaxy_eu.rabbitmqserver
>    +  version: 1.4.4
>    +- src: galaxyproject.pulsar
>    +  version: 1.0.10
>    {% endraw %}
>    ```
>    {: data-commit="Add requirements"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Now install it with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

### Configuring RabbitMQ

We need to configure RabbitMQ to be able to handle Pulsar messages. To do this we will need to create some queues, Rabbit users, some queue vhosts and set some passwords. We also need to configure rabbit to listen on various interfaces and ports.

#### Defining Virtual Hosts

Each set of queues in RabbitMQ are grouped and accessed via virtual hosts. We need to create one of these for the transactions between the Galaxy server and Pulsar server. They are set as an array under the `rabbitmq_vhosts` variable.

#### Defining users

Users need to be defined, given passwords and access to the various queues. We will need to create a user that can access this vhost. We will also create an admin user. The queue will need access to the Pulsar queue vhost. They are set as an array under the `rabbitmq_users` variable with the following structure:


```yaml
{% raw %}
rabbitmq_users:
  - user: username
    password: "{{ rabbitmq_password_username }}"
    vhost: /vhostname
{% endraw %}
```

Notice the variable we used instead of directly placing the password there. It will be read from vault instead.

> <tip-title>Tagging users</tip-title>
> Optional: You can add tags to each user if required. e.g. For an admin user it could be useful to add in a *administrator* tag. These tags allow you to grant permissions to every user with a specific tag.
{: .tip}

#### RabbitMQ server config

We also need to set some RabbitMQ server configuration variables. Such as where its security certificates are and which ports to listen on (both via localhost and network).

> <tip-title>Port accessibility is important!</tip-title>
> We will need to make sure that the RabbitMQ default port is open and accessible on the server we are installing RabbitMQ onto. (In our case this is the Galaxy server). Additionally, RabbitMQ provides a management interface for us, where we can see e.g. queues, connections and statistics.
>
> Port | Usage | Suggested Restrictions
> `5671` | SSL Listener | None
> `5672` | Plain text listener | Limit to just `172.0.0.0/8` to ensure only the host machine and other containers can access it, and we aren't sending plain text credentials over the internet.
> `15672` | Management interface | Limit to `172.0.0.0/8` to only permit accessing management interface locally.
{: .tip}

More information about the rabbitmq ansible role can be found [in the repository](https://github.com/usegalaxy-eu/ansible-rabbitmq).

### Add RabbitMQ configuration to Galaxy VM.

> <hands-on-title>Add RabbitMQ settings to Galaxy VM groupvars file.</hands-on-title>
>
> 1. Edit your `group_vars/secret.yml` and define some random passwords:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_rabbitmq_password_vhost: "a-really-long-password-here"
>    vault_rabbitmq_admin_password: "a-different-really-long-password"
>    ```
>
>    <!-- Ignore this, just for the gat-automation. Vaults are ugly to work with :(
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/secret.yml
>    +++ b/group_vars/secret.yml
>    @@ -1,7 +1,13 @@
>     $ANSIBLE_VAULT;1.1;AES256
>    -32653961383866636531396135663630386630346237333333653633313436663439643535323964
>    -6363626330336430363332643638646262316338313937320a666566306539373462386266383166
>    -30326165393863633463353234613561393939326164376432633732316264636464313061383161
>    -3532373937656138320a616361343664353264613332616236623231326137316635323465623562
>    -66656539346130353639623736633034653932373438663330646436656336666637313933666264
>    -3636313438626533633831323239373461373538646635613637
>    +62346261323266656232393034396134316636376533376139666437363535393562663838613938
>    +6336666266633563346337623265353935646361326337610a393834333233313461346439376438
>    +63383338346530656561636631666134373238366364363164313166346461383736613162653237
>    +3461363334323431370a656132303965653262386130353332623937376261396530393761353834
>    +38336565666437666436643163363831633331333766653266356163613138393734656465323634
>    +39366362383433366437353534663134313330316337393335383962613961386665633261616237
>    +35366635373063313631323939396164336330356361393464326636353037336461323531336434
>    +35613933303333623031353936393265636130363335376533393335663266313863376135383338
>    +36613464373231623938373434306266373234633036343636633963353361356631363533353066
>    +39323064336237646432323530313065303331326636353334343862373330313133326363363063
>    +38383564636161396435666164643334656435393533643163393434623434656238633631633939
>    +33353232666432376661
>    {% endraw %}
>    ```
>    {: data-commit="Add rabbitmq passwords to the vault"}
>
>    -->
>
>    This is going in the vault as they are secrets we need to set. Both of our services, Galaxy and Pulsar, need these variables, so we'll need to make sure they're in both playbooks. Both Galaxy in the job configuration, and Pulsar in its configuration.
>
>    Replace both with long random (or not) string.
>
> 2. From your ansible working directory, edit the `group_vars/galaxyservers.yml` file and add make the following changes.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -174,8 +174,11 @@ certbot_environment: staging
>     certbot_well_known_root: /srv/nginx/_well-known_root
>     certbot_share_key_users:
>       - www-data
>    +certbot_share_key_ids:
>    +  - "999:999"
>     certbot_post_renewal: |
>         systemctl restart nginx || true
>    +    docker restart rabbit_hole || true
>     certbot_domains:
>      - "{{ inventory_hostname }}"
>     certbot_agree_tos: --agree-tos
>    @@ -226,6 +229,47 @@ slurm_config:
>       SelectType: select/cons_tres
>       SelectTypeParameters: CR_CPU_Memory  # Allocate individual cores/memory instead of entire node
>
>    +#Install pip docker package for ansible
>    +pip_install_packages:
>    +  - name: docker
>    +# RabbitMQ
>    +rabbitmq_container:
>    +  name: rabbit_hole
>    +  image: rabbitmq:3.11-management
>    +  hostname: "{{ inventory_hostname }}"
>    +
>    +rabbitmq_plugins:
>    +  - rabbitmq_management
>    +
>    +rabbitmq_config:
>    +  listeners:
>    +    tcp: none
>    +  ssl_listeners:
>    +    default: 5671
>    +  ssl_options:
>    +    verify: verify_peer
>    +    cacertfile: /etc/ssl/certs/fullchain.pem
>    +    certfile: /etc/ssl/certs/cert.pem
>    +    keyfile: /etc/ssl/user/privkey-999:999.pem
>    +    fail_if_no_peer_cert: 'false'
>    +  management_agent:
>    +    disable_metrics_collector: "false"
>    +  management:
>    +    disable_stats: 'false'
>    +  consumer_timeout: 21600000 # 6 hours in milliseconds
>    +
>    +rabbitmq_vhosts:
>    +  - /pulsar/pulsar_au
>    +
>    +rabbitmq_users:
>    +  - user: admin
>    +    password: "{{ vault_rabbitmq_admin_password }}"
>    +    tags: administrator
>    +    vhost: /
>    +  - user: pulsar_au
>    +    password: "{{ vault_rabbitmq_password_vhost }}"
>    +    vhost: /pulsar/pulsar_au
>    +
>     # TUS
>     galaxy_tusd_port: 1080
>     galaxy_tus_upload_store: /data/tus
>    {% endraw %}
>    ```
>    {: data-commit="Configure RabbitMQ"}
>
>    > <tip-title>RabbitMQ installation errors?</tip-title>
>    > RabbitMQ depends on specific Erlang versions. If the Erlang version has been updated, you may need to change the value of `rabbitmq_version:` in the configuration above. [Information on the RabbitMQ Erlag version requirements.](https://www.rabbitmq.com/which-erlang.html)
>    {: .tip}
>
> 4. Update the Galaxy playbook to include the *usegalaxy_eu.rabbitmq* role.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -43,6 +43,8 @@
>           become: true
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>    +    - geerlingguy.docker
>    +    - usegalaxy_eu.rabbitmqserver
>         - galaxyproject.gxadmin
>         - galaxyproject.cvmfs
>       post_tasks:
>    {% endraw %}
>    ```
>    {: data-commit="Add role"}
>
>    > <tip-title>Why is this at the end?</tip-title>
>    > This is one of the constant problems with Ansible, how do you order everything correctly? Does an ordering exist such that a single run of the playbook will have everything up and working? We encounter one such instance of this problem now.
>    >
>    > Here are the dependencies between the roles:
>    >
>    > From             | To               | Purpose
>    > ---------------- | --               | -------
>    > nginx            | galaxy           | The nginx templates depend on variables only available after the Galaxy role is run
>    > SSL certificates | nginx            | A running nginx is required
>    > RabbitMQ         | SSL certificates | RabbitMQ will silently start with incorrect configuration if SSL certificates are not present at boot time.
>    > Galaxy           | RabbitMQ         | Galaxy needs the RabbitMQ available to submit jobs.
>    >
>    > And as you can see there is a circular dependency. Galaxy requires RabbitMQ, but RabbitMQ depends on a long chain of things that depends finally on Galaxy.
>    >
>    > There are some mitigating factors, some software will start with incomplete configuration. We can rely on Galaxy retrying access to RabbitMQ if it isn't already present. Additionally on first run, Galaxy is restarted by a handler which runs at the end. (Except that the nginx role triggers all pending handlers as part of the SSL certificate deployment.)
>    >
>    > We try to present the optimal version here but due to these interdependencies and Ansible specifics, sometimes it is not possible to determine a good ordering of roles, and multiple runs might be required.
>    >
>    {: .tip}
>
> 5. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 6. The rabbitmq server daemon will have been installed on your Galaxy VM. Check that it's running now:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > docker ps
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > ```ini
>    >CONTAINER ID   IMAGE             COMMAND                  CREATED        STATUS       PORTS                                                                                              NAMES
>    >d0d69d035768   rabbitmq:3.9.11   "docker-entrypoint.s…"   6 months ago   Up 5 hours   4369/tcp, 0.0.0.0:5671->5671/tcp, 5672/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp   rabbit_hole
>    > ```
>    {: .code-out.code-max-300}
>
> 7. But this doesn't tell the whole story, so run the diagnostics command to
>    check that the interfaces are setup and listening. RabbitMQ has a bad
>    habit of silently failing when processing the configuration, without any
>    logging information If RabbitMQ has any problem reading the configuration
>    file, it falls back to the default configuration (listens *without* ssl on
>    `tcp/5672`) so be sure to check that everything is OK before continuing.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > docker exec rabbit_hole rabbitmq-diagnostics status
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > ```ini
>    > ...
>    >
>    > Listeners
>    >
>    > Interface: [::], port: 15672, protocol: http, purpose: HTTP API
>    > Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
>    > Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
>    > Interface: 0.0.0.0, port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
>    > ```
>    >
>    {: .code-out.code-max-300}
>
> 8. Since we enabled metrics, let's check if the api works:
>
>    > <code-in-title>Bash</code-in-title>
>    > Make sure to replace \<password\> with the one from your vault.
>    > If you don't have jq installed, just leave that part with the pipe out, it just makes it prettier.
>    > ```bash
>    > curl --silent -s -u admin:<password> http://localhost:15672/api/whoami | jq
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > This should report the following response:
>    >
>    > ```console
>    > {
>    >   "name": "admin",
>    >   "tags": [
>    >     "administrator"
>    >   ]
>    > }
>    > ```
>    {: .code-out.code-max-300}
>
>    But wait! There are more ways it can go wrong. To be extra sure, run a quick `curl` command.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > curl http://localhost:5672
>    > ```
>    > (the non-encrypted port)
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > This should report the following response:
>    >
>    > ```console
>    > curl: (7) Failed to connect to localhost port 5672: Connection refused
>    > ```
>    >
>    > if it doesn't, consider the following debugging steps:
>    >
>    > 1. Restarting RabbitMQ
>    > 2. Check that the configuration looks correct (ssl private key path looks valid)
>    > 3. Check that the private key is shared correctly with the rabbitmq user
>    {: .code-out.code-max-300}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > curl -k --http0.9 https://localhost:5671 --output - && printf "\n"
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > This should report the following response:
>    >
>    > ```console
>    > AMQP
>    > ```
>    >
>    > if it doesn't, consider the following debugging steps:
>    >
>    > 1. Restarting RabbitMQ
>    > 2. Check that the configuration looks correct (ssl private key path looks valid)
>    > 3. Check that the private key is shared correctly with the rabbitmq user
>    {: .code-out.code-max-300}
>
{: .hands_on}

By this point you should have a functional RabbitMQ! Let's check out the dashboard:

> <hands-on-title>Accessing the RabbitMQ Dashboard</hands-on-title>
>
> 1. RabbitMQ has a fancy dashboard, so we should make that accessible with our NGINX:
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -84,4 +84,14 @@ server {
>     	location /training-material/ {
>     		proxy_pass https://training.galaxyproject.org/training-material/;
>     	}
>    +
>    +	location ~* /rabbitmq/(.*) {
>    +		rewrite ^/rabbitmq/(.*)$ /$1 break;
>    +		proxy_pass http://127.0.0.1:15672;
>    +		proxy_buffering                    off;
>    +		proxy_set_header Host              $http_host;
>    +		proxy_set_header X-Real-IP         $remote_addr;
>    +		proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
>    +		proxy_set_header X-Forwarded-Proto $scheme;
>    +	}
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Add proxy"}
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 1. Now we can take a look at the RabbitMQ dashboard.
>
>    1. Open a new tab in your browser and enter your server's hostname followed by `/rabbitmq/`
>    1. A login window should appear. Use the admin credentials you defined earlier in this tutorial.
>    1. Click on connections and you can see how connections will be established and appear there during the next parts of the tutorial.
>
{: .hands_on}

## Installing and configuring Pulsar on a remote machine

Now that we have a message queueing system running on our Galaxy VM, we need to install and configure Pulsar on our remote compute VM. To do this we need to create a new ansible playbook to install Pulsar.

### Configuring Pulsar

From the [`galaxyproject.pulsar` ansible role documentation](https://github.com/galaxyproject/ansible-pulsar#role-variables), we need to specify some variables.

There is one required variable:

`pulsar_server_dir` - The location in which to install pulsar

Then there are a lot of optional variables. They are listed here for information. We will set some for this tutorial but not all.

 Variable Name                 | Description                                                                                        | Default
---------------                | -------------                                                                                      | ---
`pulsar_yaml_config`           | a YAML dictionary whose contents will be used to create Pulsar's `app.yml`                         |
`pulsar_venv_dir`              | The role will create a virtualenv from which Pulsar will run                                       | `<pulsar_server_dir>/venv` if installing via pip, `<pulsar_server_dir>/.venv` if not.
`pulsar_config_dir`            | Directory that will be used for Pulsar configuration files.                                        | `<pulsar_server_dir>/config` if installing via pip, `<pulsar_server_dir>` if not
`pulsar_optional_dependencies` | List of optional dependency modules to install, depending on which features you are enabling.      | `None`
`pulsar_install_environments`  | Installing dependencies may require setting certain environment variables to compile successfully. |
`pulsar_create_user`           | Should a user be created for running pulsar?                                                       |
`pulsar_user`                  | Define the user details                                                                            |

Some of the other options we will be using are:

* We will set the tool dependencies to rely on **conda** for tool installs.

* You will need to know the FQDN or IP address of the Galaxy server VM that you installed RabbitMQ on.

> <hands-on-title>Configure pulsar group variables</hands-on-title>
>
>
> 2. Create a new file in `group_vars` called `pulsarservers.yml` and set some of the above variables as well as some others.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/pulsarservers.yml
>    @@ -0,0 +1,50 @@
>    +galaxy_server_hostname: "{{ groups['galaxyservers'][0] }}" # Important!!!
>    +# Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above.
>    +
>    +pulsar_root: /mnt/pulsar
>    +
>    +pulsar_pip_install: true
>    +pulsar_pycurl_ssl_library: openssl
>    +pulsar_systemd: true
>    +pulsar_systemd_runner: webless
>    +
>    +pulsar_create_user: true
>    +pulsar_user: {name: pulsar, shell: /bin/bash}
>    +
>    +pulsar_optional_dependencies:
>    +  - pyOpenSSL
>    +  # For remote transfers initiated on the Pulsar end rather than the Galaxy end
>    +  - pycurl
>    +  # drmaa required if connecting to an external DRM using it.
>    +  - drmaa
>    +  # kombu needed if using a message queue
>    +  - kombu
>    +  # psutil allows pulsar's local job runner to kill processes effictively.
>    +  - psutil
>    +  # temporary patch until setuptools is eliminated
>    +  - setuptools
>    +
>    +pulsar_yaml_config:
>    +  staging_directory: "{{ pulsar_staging_dir }}"
>    +  persistence_directory: "{{ pulsar_persistence_dir }}"
>    +  tool_dependency_dir: "{{ pulsar_dependencies_dir }}"
>    +  # The following are the settings for the pulsar server to contact the message queue with related timeouts etc.
>    +  message_queue_url: "pyamqp://pulsar_au:{{ vault_rabbitmq_password_vhost }}@{{ galaxy_server_hostname }}:5671//pulsar/pulsar_au?ssl=1"
>    +  min_polling_interval: 0.5
>    +  amqp_publish_retry: True
>    +  amqp_publish_retry_max_retries: 5
>    +  amqp_publish_retry_interval_start: 10
>    +  amqp_publish_retry_interval_step: 10
>    +  amqp_publish_retry_interval_max: 60
>    +  # We also need to create the dependency resolvers configuration so pulsar knows how to find and install dependencies
>    +  # for the tools we ask it to run. The simplest method which covers 99% of the use cases is to use conda auto installs
>    +  # similar to how Galaxy works.
>    +  dependency_resolution:
>    +    resolvers:
>    +      - type: conda
>    +        auto_init: true
>    +        auto_install: true
>    +
>    +# Pulsar should use the same job metrics plugins as Galaxy. This will automatically set `job_metrics_config_file` in
>    +# `pulsar_yaml_config` and create `{{ pulsar_config_dir }}/job_metrics_conf.yml`.
>    +pulsar_job_metrics_plugins: "{{ galaxy_job_metrics_plugins }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar group variables"}
>
>    > <details-title>Running non-conda tools</details-title>
>    > If the tool you want to run on Pulsar doesn't have a conda package, you will need to make alternative arrangements! This is complex and beyond our scope here. See the [Pulsar documentation](https://pulsar.readthedocs.io/en/latest/) for details.
>    {: .details}
>
> 3. Add the following lines to your `hosts` file:
>
>    {% raw %}
>    ```diff
>    --- a/hosts
>    +++ b/hosts
>    @@ -2,3 +2,5 @@
>     gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>     [dbservers:children]
>     galaxyservers
>    +[pulsarservers]
>    +gat-0.oz.galaxy.training ansible_user=ubuntu
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar host"}
>
{: .hands_on}

We will now write a new playbook for the pulsar installation as we are going to install it on a separate VM. We will also install the CVMFS client and the Galaxy CVMFS repos on this machine so Pulsar has the same access to reference data that Galaxy does.

We need to include a couple of pre-tasks to install virtualenv, git, etc.

> <hands-on-title>Creating the playbook</hands-on-title>
>
> 1. Create a `pulsar.yml` file with the following contents:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/pulsar.yml
>    @@ -0,0 +1,21 @@
>    +- hosts: pulsarservers
>    +  vars_files:
>    +    - group_vars/secret.yml
>    +  pre_tasks:
>    +    - name: Install some packages
>    +      package:
>    +        name:
>    +          - build-essential
>    +          - git
>    +          - python3-dev
>    +          - libcurl4-openssl-dev
>    +          - libssl-dev
>    +          - virtualenv
>    +          - python3-venv
>    +        state: present
>    +        update_cache: yes
>    +      become: yes
>    +  roles:
>    +    - role: galaxyproject.cvmfs
>    +      become: yes
>    +    - galaxyproject.pulsar
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar playbook"}
>
>    There are a couple of *pre-tasks* here. This is because we need to install some base packages on these very vanilla ubuntu instances as well as give ourselves ownership of the directory we are installing into.
>
{: .hands_on}

> <hands-on-title>Run the Playbook</hands-on-title>
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook pulsar.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    After the script has run, pulsar will be installed on the remote machines!
>
>    > <tip-title>Connection issues?</tip-title>
>    > If your remote pulsar machine uses a different key, you may need to supply the `ansible-playbook` command with the private key for the connection using the `--private-key key.pem` option.
>    {: .tip}
>
> 2. Log in to the machines and have a look in the `/mnt/pulsar` directory. You will see the venv and config directories. All the config files created by Ansible can be perused.
>
> 3. Run `journalctl -f -u pulsar`
>
>    A log will now start scrolling, showing the startup of pulsar. You'll notice that it will be initializing and installing conda. Once this is completed, Pulsar will be listening on the assigned port.
>
{: .hands_on}

## Configuring Galaxy to use Pulsar as a job destination

Now we have a Pulsar server up and running, we need to tell our Galaxy about it.

Galaxy talks to the Pulsar server via it's job configuration file. We need to let Galaxy know about Pulsar there and make sure Galaxy has loaded the requisite job runner, and has a destination set up.

There are three things we need to do here:

* Create a job runner which uses the  `galaxy.jobs.runners.pulsar:PulsarMQJobRunner` code.
* Create a job destination referencing the above job runner.
* Tell Galaxy which tools to send to this job destination.

For this tutorial, we will configure Galaxy to run the BWA and BWA-MEM tools on Pulsar.

> <hands-on-title>Configure Galaxy</hands-on-title>
>
> 1. In your `group_vars/galaxyservers.yml` file add the following job runner to the `<plugins>` section:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -24,6 +24,16 @@ galaxy_job_config:
>         slurm:
>           load: galaxy.jobs.runners.slurm:SlurmJobRunner
>           drmaa_library_path: /usr/lib/slurm-drmaa/lib/libdrmaa.so.1
>    +    pulsar_runner:
>    +      load: galaxy.jobs.runners.pulsar:PulsarMQJobRunner
>    +      amqp_url: "pyamqp://pulsar_au:{{ vault_rabbitmq_password_vhost }}@localhost:5671/{{ rabbitmq_vhosts[0] }}?ssl=1"
>    +      amqp_acknowledge: true
>    +      amqp_ack_republish_time: 1200
>    +      amqp_consumer_timeout: 2
>    +      amqp_publish_retry: true
>    +      amqp_publish_retry_max_retries: 60
>    +      galaxy_url: "https://{{ inventory_hostname }}"
>    +      manager: _default_
>       handling:
>         assign: ['db-skip-locked']
>       execution:
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar plugin"}
>
>    Add the following to the `destinations:` section of your `tpv_rules_local` file:
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -56,3 +56,18 @@ destinations:
>         max_mem: 8
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)} --time={entity.params['walltime']}:00:00
>    +
>    +  pulsar:
>    +    runner: pulsar_runner
>    +    params:
>    +      default_file_action: remote_transfer
>    +      dependency_resolution: remote
>    +      jobs_directory: /mnt/pulsar/files/staging
>    +      persistence_directory: /mnt/pulsar/files/persisted_data
>    +      remote_metadata: false
>    +      rewrite_parameters: true
>    +      transport: curl
>    +      outputs_to_working_directory: false
>    +    scheduling:
>    +      require:
>    +        - pulsar
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar destination"}
>
>    You'll notice we need to know a lot about the configuration of the remote end, this is an unfortunate requirement with pulsar. Changes to e.g. the staging directory need to be coordinated between Pulsar and Galaxy. That's fine if both are under your administration, but for a completely remote Pulsar it can be difficult.
>
>    Notably we also override `outputs_to_working_directory`, as this option is incompatible with running Pulsar, and, unnecessary. Pulsar already provides the same job isolation and safety that we request when we set that option by default in Galaxy's configuration.
>
> 2. Install the BWA and BWA-MEM tools, if needed.
>
>    {% snippet topics/admin/faqs/install_tool.md query="bwa" name="Map with BWA-MEM" section="Mapping" %}
>
> 3. We now need to tell Galaxy to send BWA and BWA-MEM jobs to the `pulsar` destination. We specify this in the `<tools>` section of your job conf.
>
>    Add the following to the end of the TPV rules file (inside the `tools:` section if it exists or create it if it doesn't.)
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -26,6 +26,14 @@ tools:
>             cores: int(job.get_param_values(app)['__job_resource']['cores'])
>             params:
>                walltime: "{int(job.get_param_values(app)['__job_resource']['time'])}"
>    +  .*/bwa/.*:
>    +    scheduling:
>    +      require:
>    +        - pulsar
>    +  .*/bwa_mem/.*:
>    +    scheduling:
>    +      require:
>    +        - pulsar
>
>     destinations:
>       local_env:
>    {% endraw %}
>    ```
>    {: data-commit="Send bwa and bwa-mem to pulsar"}
>
>    Note that here we are using the short tool IDs. If you want to run only a specific version of a tool in Pulsar, you have to use the full tool ID (e.g. `toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.4`) instead. The full tool ID can be found inside the `integrated_tool_panel.xml` file in the `mutable-config` directory.
>
> 4. Finally run the Galaxy playbook in order to deploy the updated job configuration, and to restart Galaxy.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1-pulsar.sh
> ```
> {: data-test="true"}
{: .hidden}


## Testing Pulsar

Now we will upload a small set of data to run bwa-mem with.

> <hands-on-title>Testing the Pulsar destination</hands-on-title>
>
> 1. Upload the following files from zenodo.
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    https://zenodo.org/record/582600/files/mutant_R2.fastq
>    ```
>
> 2. **Map with BWA-MEM** {% icon tool %} with the following parameters
>    - *"Will you select a reference genome from your history or use a built-in index"*: `Use a built-in genome index`
>    - *"Using reference genome"*: `Escherichia coli (str. K-12 substr MG1655): eschColi_K12`
>    - *"Single or Paired-end reads"*: `Paired end`
>    - {% icon param-file %} *"Select first set of reads"*: `mutant_R1.fastq`
>    - {% icon param-file %} *"Select second set of reads"*: `mutant_R2.fastq`
>
>    As soon as you press *execute* Galaxy will send the job to the pulsar server. You can watch the log in Galaxy using:
>
>    ```
>    journalctl -fu galaxy-*
>    ```
>
>    You can watch the log in Pulsar by ssh'ing to it and tailing the log file with:
>
>    ```
>    journalctl -fu pulsar
>    ```
>
{: .hands_on}

> ```bash
> 2-run-job.sh
> ```
> {: data-test="true"}
{: .hidden}

You'll notice that the Pulsar server has received the job (all the way in Australia!) and now should be installing bwa-mem via conda. Once this is complete (which may take a while - first time only) the job will run. When it starts running it will realise it needs the *E. coli* genome from CVMFS and fetch that, and then results will be returned to Galaxy!

How awesome is that? Pulsar in another continent with reference data automatically from CVMFS :)

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=11 %}

## Retries of the staging actions

When the staging actions are carried out by the Pulsar server itself (like in the case when driving Pulsar by message queue), there are some parameters that can be tweaked to ensure reliable communication between the Galaxy server and the remote Pulsar server.
The aim of these parameters is to control the retrying of staging actions in the event of a failure.

For each action (preprocess/input or postprocess/output), you can specify:
```text
 - *_action_max_retries    - the maximum number of retries before giving up
 - *_action_interval_start - how long start sleeping between retries (in seconds)
 - *_action_interval_step  - by how much the interval is increased for each retry (in seconds)
 - *_action_interval_max   - the maximum number of seconds to sleep between retries
```
substitute the * with `preprocess` or `postprocess`

In the following box, as an example, we have collected the values adopted in a Pulsar site with an unreliable network connection:

```yaml
preprocess_action_max_retries: 30
preprocess_action_interval_start: 2
preprocess_action_interval_step: 10
preprocess_action_interval_max: 300
postprocess_action_max_retries: 30
postprocess_action_interval_start: 2
postprocess_action_interval_step: 10
postprocess_action_interval_max: 300

```
In this case, for both actions, Pulsar will try to carry out the staging action 30 times, sleeping 2 secs after the first retry and adding 10 secs more to each next retries, until a maximum of 300 seconds between retries.

We hope you never have to experience a situation like this one, but if needed just adapt the numbers to your case and add the parameters in the `pulsar_yaml_config` section of your `pulsarservers.yml` file.

## Pulsar in Production

If you want to make use of Pulsar on a Supercomputer, you only need access to a submit node, and you will need to run Pulsar there. We recommend that if you need to run a setup with Pulsar, that you deploy an AMQP server (e.g. RabbitMQ) alongside your Galaxy. That way, you can run Pulsar on any submit nodes, and it can connect directly to the AMQP and Galaxy. Other Pulsar deployment options require exposing ports wherever Pulsar is running, and this requires significant more coordination effort.

For each new Pulsar server, you will need to add:
  1. In the RabbitMQ config:
      * A vhost
      * A user - configured with a password and the new vhost
  2. In the Galaxy job configuration
      * A new job runner with the new connection string
      * A new destination or multiple destinations for the new runner.

Pulsar servers can be the head node of a cluster. You can create a cluster and use your favourite job scheduler such as Slurm or PBS to schedule jobs. You can have many destinations in your Galaxy job conf file that change the number of cpus, amount of RAM etc. It can get quite complex and flexible if you like.

### Australia

You can also create multiple queues on your RabbitMQ server for multiple Pulsar servers. On Galaxy Australia, we run 5 different Pulsar servers spread out all around the country. They all communicate with Galaxy via the one RabbitMQ server.

![Map of australia with 6 pulsar nodes marked around the country.](../../images/pulsar_australia.png)

### Europe

Galaxy Europe has taken Pulsar and built [The Pulsar Network](https://pulsar-network.readthedocs.io/en/latest). This provides a framework for easily deploying Pulsar clusters in the cloud, something needed to support compute centers which might not have as much experience. This way they get an easy package they can deploy and the European Galaxy team can manage.

![Map of europe with pulsar nodes marked in many countries. An inset shows australia with a node there too.](https://pulsar-network.readthedocs.io/en/latest/_images/nodes.png)

The main purpose of this network is to support the workload of the UseGalaxy.eu instance by distributing it across several European data centers and clusters. If you're interested in setting up something similar, they [provide documentation](https://pulsar-network.readthedocs.io/en/latest/introduction.html) on how to install and configure a Pulsar network endpoint on a cloud infrastructure and how to connect it to your server.

## Conclusion

You're ready to ship your Galaxy jobs around the world! Now wherever you have compute space, you know how to setup a Pulsar node and connect it to Galaxy. Let us know if you come up with creative places to run your Galaxy jobs (coworker's laptops, your IoT fridge, the sky is the limit if it's x86 and has python)

{% snippet topics/admin/faqs/git-gat-path.md tutorial="pulsar" %}


topic: admin
tutorial: jenkins
title: Automation with Jenkins
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/jenkins/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "automation"]
content_hash: 52ce19eae8100c60d37fc0e8f203c9ae
last_modified: 2026-05-15T22:28:10.036444
workflows_json: []
content:
# Overview


Automation is a key component for making your life easier. There are dozens of regular, boring tasks involved in server administration, and many of these can be automated in order to make your life easier.
{CI} systems provide one way to accomplish this.
Many of these systems focus on testing and deploying code, and provide some built-in shortcuts to accomplish this.
Testing and deploying code boils down to "please run this command, when X happens", and they tend to function as "commands as a service".
You can often configure them to "test code when it is pushed to a branch on GitHub", or "test this code daily" or "deploy this code daily".
If you need to automatically run a command whenever some event happens, and then to report this back in some standardised way (GitHub PR statuses, automatically deploying, notifications in chat), then a CI system can do this for you.

There are many options for, some of these have different features which may be desirable in one or another scenario.
This tutorial will specifically cover automation with Jenkins as that is one of the tools `usegalaxy.*` use for a subset of their automation.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Comparison of CI Systems

There are dozens of {CI} systems available meeting different needs, which CI system you want to use depends largely on the task you are trying to automate. A couple of key factors you may wish to consider are:

- How long will jobs run for? Free services often have time and log-output limits.
- Are you concerned about secret storage? Self-hosted can be a better option here, but it depends on your security posture and threat matrix.
- Do you have infrastructure available? Self-hosted could be worth it for the extra freedoms.

In general, self-hosted options have the overhead of managing and securing and updating the service, but offer significantly more freedom if you need it. They generally provide poor per-job isolation and assume you trust all of the jobs and people with access to build scripts used in the build pipeline. The free services (Travis, Circle CI) often have much better user-experiences and are easier to get started with and provide better out-of-the-box isolation.

We will look at setting up and managing a Jenkins server today, how to setup tasks, and some of the things that `usegalaxy.*` automate with their Jenkins servers.

# Infrastructure

As per many of the tutorials, we will use Ansible to setup and manage our Jenkins server so we can easily leverage the work of a large community. Jenkins is one choice, there are other self-hosted systems you could use, but it is the one with which many of us are familiar.

## Jenkins

> <hands-on-title>Setting up Jenkins</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    ```yaml
>    - src: geerlingguy.java
>      version: 1.10.0
>    - src: geerlingguy.jenkins
>      version: 3.7.0
>    ```
>
> 2. Install the requirements
>
>    ```
>    ansible-galaxy install -p roles -r requirements.yml
>    ```
>
> 3. Create a new playbook, `build.yml` with the following:
>
>    ```yaml
>    ---
>    - hosts: build
>      become: true
>      pre_tasks:
>        - name: Allow jenkins user to execute things as root
>          copy:
>            content: |
>              jenkins ALL=(ALL:ALL) NOPASSWD:ALL
>            dest: /etc/sudoers.d/jenkins
>            validate: 'visudo -cf %s'
>            mode: 0440
>      roles:
>        - geerlingguy.java
>        - geerlingguy.jenkins
>    ```
>
>    During this tutorial we will install everything on the same host, but often one keeps the build infrastructure on a separate host. We permit the Jenkins user to run everything as root, because later we will run Ansible with Jenkins, and we have `become: true` in our `galaxy.yml` playbook, so the `jenkins` user will need to be able to become root.
>
> 4. Edit the inventory file (`hosts`) an add a group for monitoring like:
>
>    ```ini
>    [build]
>    localhost ansible_connection=local
>    ```
>
> 5. Create and edit the group variables for the `build` group, `group_vars/build.yml`, and include the following:
>
>    ```yml
>    jenkins_http_port: 4000
>    jenkins_admin_username: admin
>    # Please change at least the password to something more suitable
>    jenkins_admin_password: admin
>    jenkins_url_prefix: /jenkins
>    jenkins_plugins:
>    - matrix-auth
>    ```
>
> 6. Run the build server playbook:
>
>    ```
>    ansible-playbook -i hosts build.yml
>    ```
>
> 7. Update the nginx configuration in `templates/nginx/galaxy.j2` to proxy jenkins with the following. Add it *before the last curly brace* at the end of the file.
>
>    ```nginx
>        location /jenkins {
>            proxy_set_header        Host $host:$server_port;
>            proxy_set_header        X-Real-IP $remote_addr;
>            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
>            proxy_set_header        X-Forwarded-Proto $scheme;
>
>            # Fix the "It appears that your reverse proxy set up is broken" error.
>            proxy_pass          http://127.0.0.1:4000;
>            proxy_read_timeout  90;
>        }
>    ```
>
> 8. Run the galaxy playbook which contains Nginx:
>
>    ```
>    ansible-playbook -i hosts galaxy.yml
>    ```
>
{: .hands_on}

With this, we're done, Jenkins is set up and ready to use!

# Building things with Jenkins

We will look at a couple basic cases of how to build things with Jenkins, and wrap up with some advanced usage documentation that is not so practical for a tutorial environment.

Start by **visiting your Jenkins instance**, at `https://your-domain/jenkins`

![Welcome to Jenkins](../../images/jenkins-01-welcome.png "Welcome to Jenkins! This is how it looks once you've run the playbook and logged in. The menu on the left gives you some ability to configure Jenkins, below it is the job queue where running and queued jobs will appear. When you have configured some jobs, they will be visible in the central panel.")

## A simple job

> <hands-on-title>Create a simple job</hands-on-title>
>
> 1. Visit your Jenkins and login with the credentials you set up in the group variables file
>
> 2. Click **New Item** in the left hand menu
>
> 3. Give your job a name like `my-project`, and select that it is a `freestyle project`, and click "OK"
>
>    ![Create a job](../../images/jenkins-02-create-job.png "There are other types of jobs if you install plugins, but we will mostly use freestyle jobs as they are the most appropriate for our case.")
>
> 4. The job configuration interface is split into a few sections, visible in the tabs at the top of the page:
>
>    ![Job configuration interface](../../images/jenkins-03-job-config.png "The job configuration interface, split into 'General', 'Source Code Management', 'Build Triggers', 'Build', and 'Post-build Actions'")
>
> 5. In the **Build Triggers** section:
>
>    - Build Periodically
>        - *"Schedule"*: `H * * * * `
>
>    > <details-title>Cron syntax</details-title>
>    > Jenkins uses the [cron syntax](https://en.wikipedia.org/wiki/Cron#Overview), with the addition that `H` can be used to randomly choose a value for that place, rather than running at precisely that minute or hour. This allows you to spread the load of multiple jobs over a period of time, rather than having maybe 10 jobs all attempting to run simultaneously if they do not need to.
>    > For the above example it will choose one value for the `H`, a random minute in the hour, and then run every hour, every day, at that minute.
>    > You can enter different expressions to see when Jenkins would run it next.
>    {: .details}
>
> 6. Add a build step:
>
>    - Add build step
>        - **Click** "Execute shell"
>    - In the new step
>        - *"Command"*:
>          ```
>          echo Hello world!
>          date
>          env | sort
>          ```
>
>      ![Executing a shell](../../images/jenkins-05-job1.png)
>
>    > <warning-title>Danger!</warning-title>
>    > This field allows you to execute arbitrary bash scripts. This is a reason Jenkins **must** be secured with HTTPS and a good username/password.
>    {: .warning}
>
>
> 7. Click **Save** at the bottom
>
> 8. You will be brought back to the job information page. Click **Build Now**
>
> 9. Your job will appear in the *Build History* box at the left. **Click** the ball associated with your job, to see the console.
>
>    ![Seeing a Jenkins job's console](../../images/jenkins-06-job1-run.png "The build history box lets you see the status of previous, running, and queued builds. The 'sun' icon indicates that all builds have been successful, it will become more cloudy/overcast if you experience failures. The 'trend' button allows you to visualise how quickly builds have run historically.")
>
>    Clicking the number will bring you to a job information page, where you can click "Console Output" to see the jobs output
>
>    ![Jenkins job output](../../images/jenkins-07-job1-out.png "The output of your first Jenkins job. All jenkins jobs log both the commands that were run, and their outputs. Use this to see which step potentially failed.")
{: .hands_on}

Setting up Jenkins jobs is as simple as setting up a cron job, but the results are stored in a nice visual interface that may provide better visibility than a `cron` job.

> <comment-title>Author's commentary</comment-title>
> Cron jobs are great and often useful, but most people ignore the emails that cron sends them. I know I have mailboxes across N servers with dozens and dozens of unread emails from cron jobs. Jenkins is an improvement here because it shows "success" and "failure" messages that are clear, while storing the output in case you later decide that you want to look at it. For the most part, cron jobs write output that is never read, and we really only want to see the output if something has failed, but writing a cron job that behaves like this is unnecessarily complex.
{: .comment}

## Ansible in Jenkins

We will now setup Jenkins to run Ansible on cron. In the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial we emphasised that it is useful to often run the entire playbook to ensure that all changes are applied. UseGalaxy.eu likes to accomplish this by having Jenkins run the playbooks every day. We know that even if our coworkers made some changes to the servers, that by tomorrow it will be reverted to a known-good configuration.

> <hands-on-title>Jenkins running Ansible</hands-on-title>
>
> 1. Go back to the Jenkins homepage (click the logo in the top left)
>
> 2. Click **New Item** in the left hand menu.
>
> 3. Give your job a name like "ansible-galaxy", and select that it is a "freestyle project", and click "OK"
>
> 4. Configure your job
>
>    - Build Triggers
>      - Build Periodically
>        - *"Schedule"*: `H * * * * `
>    - Build
>      - Add build step
>        - **Click** "Execute shell"
>      - In the new step
>        - *"Command"*:
>          ```
>          cd /home/ubuntu/galaxy/ # Change this to wherever your galaxy.yml is, you can check with the command pwd
>          ansible-playbook -i hosts galaxy.yml --diff
>          ```
>    - Save
>
> 5. Click **Build Now** and check the console output of the job
>
>    What do you see? It should look like Jenkins is running the playbook:
>
>    ```
>    Started by user admin
>    Running as SYSTEM
>    Building in workspace /var/lib/jenkins/workspace/ansible-galaxy
>    [ansible-galaxy] $ /bin/sh -xe /tmp/jenkins6010384499214486550.sh
>    + cd /home/ubuntu/galaxy/
>    + ansible-playbook -i hosts galaxy.yml --diff
>
>    PLAY [galaxyservers] ***********************************************************
>
>    TASK [Gathering Facts] *********************************************************
>    ok: [gcc-3.training.galaxyproject.eu]
>
>    TASK [Install Dependencies] ****************************************************
>     [WARNING]: Could not find aptitude. Using apt-get instead
>    ok: [gcc-3.training.galaxyproject.eu]
>    ```
>
{: .hands_on}

With this we have Jenkins running our playbook for us! You can imaging several steps you could take to follow up from this:

1. Storing your playbooks in `git` like UseGalaxy.\*
	- [UseGalaxy.eu's playbooks](https://github.com/usegalaxy-eu/infrastructure-playbook) are public, anyone can contribute and if the PR is merged, it is automatically applied to our infrastructure within the next day. We store our job configuration in here as well which specifies how much memory and how many CPUs tools receive. Our teammates often can contribute fixes themselves when they notice a tool needs more memory.
	- [UseGalaxy.org's playbooks](https://github.com/galaxyproject/infrastructure-playbook) are also public, but run manually.
2. Running this job regularly
	- This can prevent mistakes, if a coworker edits some configuration on the server and forgets to commit it, this ensures everything is reverted back to the expected state.
	- The amount of time the Galaxy playbook takes can be quite annoying when done manually but no one is upset if it takes 30 minutes when it is run on cron.
3. Alerting you if something went wrong
	- There are numerous plugins that can send notifications of build success/failure to various places like Slack, etc.

These are all next steps that are great to look into, and will make your systems more reliable and trustworthy as you will know when things change, and you can be certain that the code is configured how you specified.

# Securing Jenkins

You have already secured Jenkins in that it is protected with a good password and HTTPS. This is one important thing. The other portion is the visibility of jobs and their outputs. In the above Ansible Galaxy deployment job we set the parameter `--diff`, so we could see what changed. This is an extremely helpful thing to do to be able to audit changes from a particular run. But! If you change the `id_secret` or other secret variables, then this diff will be visible in the job's console logs. This is useful to see for admins but needs to be restricted so that the whole world cannot see this.

> <hands-on-title>Securing Jenkins</hands-on-title>
>
> 1. Return to the Jenkins home page
>
> 2. Click on **Manage Jenkins**
>
> 3. Click on **Configure Global Security**
>
>    1. In the first block *Access Control* Change the *Authorization* strategy from "Logged-in users can do anything" to "Matrix-based security"
>
>    2. Click "Add user or group" and add the `admin` user
>
>    3. Configure the permissions like below. Clicking on the checked checkbox icon at the right will allow you to enable/disable all of the checkboxes for that row.
>
>       ![Configuring matrix based permissions](../../images/jenkins-09-permissions.png)
>
>       The matrix based security plugin is provided by a plugin, we installed this by setting the variable `jenkins_plugins: [matrix-auth]` in our group variables earlier.
>       The checkboxes give you the ability to grant or deny permissions to certain classes of users. Each column is an individual permission, and you can hover over each column to get more information about what that permission allows.
>
>       In the above example we have configured that:
>         - *Anonymous users* (not logged in) may `Read`, so they can access the Jenkins page, but by default will not see any jobs. The `Discover` permission is also granted so that if a user has bookmarked the ansible-galaxy job page, they will automatically be redirected through the login page, if they are not already logged in.
>         - Once they login and are *Authenticated Users*, they can see that `Manage` and `Run` Jobs and Views
>         - Only the *admin* user, however, can configure `Agents` to run those jobs.
>
>    3. Under **CSRF Protection**, check the box to prevent CSRF, and select the default crumb issuer, and then enable proxy compatibility.
>
> 4. Save
>
>    > <question-title></question-title>
>    >
>    > Try accessing your Jenkins from a private browsing session. How does it look?
>    >
>    > > <solution-title></solution-title>
>    > > It should appear as if there are no jobs, until you log in.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Advanced Topics

These topics are outside the scope of a training or do not lend themselves well to interesting exercises, but some discussion and configuration information will be provided below documenting common issues and questions that can be useful if you start using Jenkins "in anger".

## Updating Jenkins

Jenkins updates regularly due to security issues. We recommend that you either:

1. Regularly check the [Jenkins Changelog](https://jenkins.io/changelog/), and update the server whenever there are important security fixes
2. OR include some sort of yum or apt auto-updater on whichever system that Jenkins is running on

The downside of automatic updating is that Jenkins will restart and kill any jobs that had been running. Choose which works best for your environment. UseGalaxy.eu chose a completely automated setup because their jobs can be killed and few things are affected. UseGalaxy.org chose a more manual setup because their jobs are more critical, there if jobs fail, Galaxy pull request tests will fail unexpectedly and unhelpfully.

## Updating Plugins

Plugins also regularly receive updates due to security issues. You should either check in with your Jenkins server regularly and apply updates when they are available, or automate this as well. UseGalaxy.eu uses a script like:

```bash
#!/bin/bash
ARGS="-jar /opt/jenkins-cli.jar -s http://localhost:4000/ -auth user:password"
UPDATE_LIST=$(java $ARGS list-plugins | grep ')$' | awk '{print $1}')

if [ ! -z "${UPDATE_LIST}" ]; then
    echo Updating Jenkins Plugins: ${UPDATE_LIST};
    java $ARGS install-plugin ${UPDATE_LIST};
    java $ARGS safe-restart;
fi
```

And we run this on cron hourly. The `safe-restart` at the end causes Jenkins to stop processing new jobs, and wait until all jobs are complete before restarting. Unfortunately this is not an option for Jenkins server updates.

## Plugins

Jenkins provides an impressive number of plugins for a lot of different environments. Here are some of the ones we use and their purposes:

Plugin                           | Comments
------                           | --------
Git Plugin                       | Allow cloning git repositories to build
Build Timeout                    | Allows us to automatically kill jobs which run longer than we expect. We can use this to set per-build timeouts
Credentials Binding Plugin       | This can be used to inject secrets into builds, e.g. vault passwords. <!-- TODO: use this with vault in ansible-galaxy tuto -->
Embeddable Build Status Plugin   | Produces nice build statuses like Travis does, e.g. [UseGalaxy.eu's playbook statuses](https://github.com/usegalaxy-eu/infrastructure-playbook#build-statuses)
GitHub Authentication plugin     | The UseGalaxy.eu build server requires login with GitHub and ensures membership in the correct organisation to be able to access group secrets and build logs.
Job Configuration History plugin | This plugin tracks changes made in the configuration of jobs over time, so you can see what was changed in the Jenkins configuration which may have caused the job to start succeeding or failing.
ShiningPanda Plugin              | Allows easily running jobs within a per-job python virtualenv.
SSH Agent Plugin                 | Allows providing specific SSH credentials to a build

These plugins can be installed via the Ansible automation by expanding `jenkins_plugins` in your `group_vars/build.yml`, or installed with the web interface.

### Source Code Management

Jenkins supports most VCSs that are in use today. For the Git plugin, it allows you to clone a git repository into the job's directory, and then run scripts from there. It forms the foundation of most jobs.

You may need to specify credentials for cloning from private repositories, in this case you should beware that when you configure the SSH credentials you should specify the `user` as `git`. You will also need to ensure that the server's SSH public key is known to your Jenkins system which generally requires `su`ing as the Jenkins user and running `ssh git@...` once, and accepting the "Unknown host" key prompt. We recommend that you use repository URLs of the format `git@github.com:org/repo.git`, if you are using one of the GitHub PR builder plugins, then this URL has to match a value in the incoming webhook and it is not always obvious that this needs to be set like this.

The "Branches to build" section allows you to specify which branches. Whenever you click "Build now", Jenkins usually just picks the main branch to build, and doesn't run a build job per-branch. If you are automatically triggering Jenkins builds based on GitHub (or other) webhooks, then you can ensure that branches that should not be built are filtered out here. Real life example: UseGalaxy.eu has one Jenkins job which should only build the main branch of a repository, and another job that only builds the PRs for that repository. We know the first job will only ever build the main branch so we can hardcode that there. With the Git plugin, many "Additional Behaviours" are available. If you have submodules in your repository, you will need to enable one of these to ensure that the repository is cloned recursively. If you want to ensure a fresh start each time because you write out temporary files to the current working directory, then there is a behaviour to wipe out the repository before each build.

### Build Triggers

The GitHub Pull Request Builder plugin is not always easy to configure, here systems like Travis are generally significantly easier. Consider if that is an option. It is not a bad thing to use multiple {CI} systems, it provides some degree of redundancy where if one system is experiencing an outage it may not affect all CI jobs.

### Credential Binding

If you provide credentials to Jenkins builds, they will automatically be stripped from log outputs. So if your job will not "leak" any private information (e.g. in `--diff`) and you want to share the build logs with your users to prove that actions were taken or that things are working and automated, then the credential binding plugin ensures that any secrets will be stripped from output and replaced with `******`. They can be made available as environment variables, or in files that can be accessed.

We often upload a configuration file with secrets (e.g. `.parsec.yml` ), and then can just call our software `parsec -c $PARSEC_CONFIG_FILE_LOCATION ...` passing in that file, wherever Jenkins has stored it securely.

### SSH Agent

This allows you to automatically inject credentials into your build's SSH agent. If you want to `rsync` some build outputs to another server, or `ssh` to a server to run some one-off command, or to run `ansible`, then this is the feature you want. Beware that if you are pulling from and pushing to git repositories on GitHub with separate SSH credentials you may need to use a combination of the SSH Agent plugin, and injecting an SSH key through credential binding.

## Real Life Examples

Here are some examples of how Jenkins is used in the wild, maybe it can provide some reference for potential uses within your organisation or solutions to various problems. The examples are mostly pulled from UseGalaxy.eu's experiences, as the author is most familiar with those.

### Compute Nodes

We attach VMs as compute nodes for Jenkins to use as executors. We have an OpenStack cloud available so that is an easy option for us. We setup the VMs such that Jenkins just logs in as the default `centos` user and uses the home directory for building jobs. There is a Jenkins plugin to manage the VMs in the cloud automatically (scaling up/down as necessary, removing VMs with full disk) but our cloud was upgraded and we could not get this working. It is worth trying if that is an option! There are other cloud provider plugins available with similar features.

We also use a custom image that includes a lot of our requirements pre-installed. We include e.g. LaTeX, Ruby/RVM, Java, Node, anything we think we might need for our jobs. This also lets us reduce the privileges of the jobs, nothing *needs* `root` access to install packages globally. Python, Ruby, and Node all have something similar to virtualenvs.

### Job: Apollo Builder

This is a relatively straightforward job to build a WAR file and upload it to our depot server.

```console
# some env vars that are needed, it's kind of gross but it works :(
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/
export PERL5LIB=/home/centos/workspace/usegalaxy-eu/apollo-builder/extlib/lib/perl5:$PERL5LIB
# setting a variable from some build data
APOLLO_VERSION=$(cat application.properties | grep app.version | sed 's/.*=//g')

# Check if this version needs to be built
curl --silent https://usegalaxy.eu/static/vgcn/ | \
	grep -F "apollo-${APOLLO_VERSION}" && echo "Already built!" && exit 0;

# Build the thing
./apollo deploy

# Deploy it to our server
scp -i $PRIV_KEY \
	target/*.war \
    user@server:/static/apollo-${APOLLO_VERSION}.war
```

- We use the "Secret file" binding to inject a specific private key (available in the build as `$PRIV_KEY`) but the SSH agent plugin would work equally well.
- We could also use the "Archive Artifacts" plugin to have Jenkins retain all of the artefacts (build outputs that you care about) but we wanted to have an Ansible role pull those files and it was simpler to have them accessible at predictable URLs.

### Job: Chado Schema Builder

Similar to the Apollo builder but it uploads outputs as GitHub releases which is useful for public projects with complex build pipelines. The CSB takes on average 5-6 hours to run so we cannot easily run this on free infrastructure.

Here we store both the [build](https://github.com/hexylena/chado-schema-builder/blob/master/.ci/run.sh) and [deploy](https://github.com/hexylena/chado-schema-builder/blob/master/.ci/upload.sh) scripts in the GitHub repository. If you do this, you have to be careful about building PRs, as anyone can edit the build scripts and access your internal infrastructure!

### Job: Website

We build our own Jekyll website because we have some plugins GitHub will not compile for us. The configuration here is more complex:

- Throttle builds, since GitHub will only accept so many deploy requests in a time period
- One SSH key used to clone the repository using the Git SCM, with "Wipe out repository & force clone" set.
- Triggers on both:
  - Cron, daily
  - GitHub webhooks notifying us of pushes to master
- SSH Agent is configured with a deploy SSH key (because git really prefers you provide keys in the SSH agent, I don't know if it is possible to pass a keyfile on the CLI)
- The [.build.sh](https://github.com/usegalaxy-eu/website/blob/master/.build.sh) is run which activates the RVM pre-installed on the VMs
- If the master branch is checked out, [`.publish.sh`](https://github.com/usegalaxy-eu/website/blob/master/.publish.sh) is run
- Empty [`.nojekyll`](https://github.com/usegalaxy-eu/website/blob/master/.nojekyll) file to prevent GitHub from trying to build this.

### Job: Install Tools

For this job we use Jenkin's "Trigger other builds" feature:
- We have one job that fetches all of the latest updates for IUC tools and writes those into yaml files. This job runs on cron every week.
- This triggers a separate job to actually apply the changes to our server.

By splitting these up, we can run the "actually apply" portion without having to fetch updates to the tools first. Additionally we can have the second portion run only if the first portion succeeds, so we have some check that everything looks OK before trying to apply it.

We include a Galaxy API key as a credential and then use a `Makefile` contained in the project to do the steps. We don't automatically test PRs so we don't care if people change the Makefile because we visually validate it before merging their PR.

### Configuration: Themes

Many people find the "blue/red ball" icons a bit hard to understand. Using the "Simple Theme Plugin" you can easily add custom CSS files or directly enter custom CSS. UseGalaxy.\* use the [jenkins-material-theme](http://afonsof.com/jenkins-material-theme/) as it provides a slightly cleaner view of Jenkins with better iconography.

### Configuration: Folder Permissions

We have the aforementioned problem that our playbooks run with `--diff` and while we want to capture this output and be able to review it, it is not necessary that the public can see it. So permissions are configured not only on the global Jenkins level, but also within a single folder of jobs. The folder permissions involve setting:

- Properties
  - *"Inheritance Strategy"*: `Do not inherit permission grants from other ACLs`
  - Allowing `Discover` and nothing else for Anonymous + Authenticated
  - Explicitly granting all permissions for the admins


topic: admin
tutorial: monitoring
title: Galaxy Monitoring with Telegraf and Grafana
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "monitoring", "git-gat"]
content_hash: df132f1854cf51499e9a486a549aace2
last_modified: 2026-05-27T09:01:47.637092
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/monitoring/tutorial.html

content:
Monitoring is an incredibly important part of server monitoring and maintenance. Being able to observe trends and identify hot spots by collecting metrics gives you a significant ability to respond to any issues that arise in production. Monitoring is quite easy to get started with, it can be as simple as writing a quick shell script in order to start collecting metrics.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="monitoring" %}

This tutorial explicitly assumes you are starting with a setup like that created in the *Galaxy installation with Ansible* tutorial

# Data Flow

The monitoring setup we (UseGalaxy.\*) use involves compute nodes collecting metrics and forwarding the data to a central collector. On each node we have a data collector running (Telegraf) which collects data from numerous sources before aggregating it and forwarding it onwards. On another, sometimes public, node we accept and collect all of these metrics (InfluxDB) and provide an interface to visualise and query that data (Grafana).

For UseGalaxy.eu, we pre-install Telegraf on our compute node images. Whenever we launch new compute nodes, they automatically start sending data to our central collection point. Telegraf additionally provides a nice feature of aggregating metrics it collects, batching them up before sending them off to InfluxDB. Telegraf has built in data collectors, or can run command line scripts and tools which output specifically formatted data.

Telegraf communicates with InfluxDB using a JSON based HTTP API, making it easy to write custom tooling to send data there, if need be. Grafana provides the visualisation component, the most interesting component for most consumers of the metrics and monitoring.


# Infrastructure

Setting up the infrastructure is quite simple thanks to the automation provided by Ansible. We will first setup a playbook for a "monitoring" machine, which will collect and visualize our data using InfluxDB and Grafana. We will then expand the Galaxy playbook to include Telegraf on the machine to monitor both it and Galaxy itself. In this tutorial we will do everything on the same machine. In practice you can separate these services to different machines, if you wish. The only requirement is that Telegraf run on the machine from which you wish to collect data.

## InfluxDB

[InfluxDB](https://www.influxdata.com/) provides the data storage for monitoring. It is a {TSDB}, so it has been designed specifically for storing time-series data like monitoring and metrics. There are other TSBD options for storing data but we have had good experiences with this one. TSBDs commonly feature some form of automatic data expiration after a set period of time. In InfluxDB these are known as "retention policies". Outside of this feature, it is a [relatively normal database](https://docs.influxdata.com/influxdb/v1.7/concepts/crosswalk/).

The available Ansible roles for InfluxDB unfortunately do not support configuring databases or users or retention policies. Ansible itself contains [several modules](https://docs.ansible.com/ansible/2.9/modules/list_of_database_modules.html#influxdb) you can use to write your own roles, but nothing generic. UseGalaxy.eu wrote [their own role](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/roles/hxr.influxdb/tasks/main.yml) for setting up their InfluxDB database, but it is not reusable enough for it to be used here yet. If you plan to automate your entire setup, this tutorial can perhaps provide inspiration for writing your own Ansible role. However, in this case it is sufficient to manually create your users and retention policies as a one-off task.

> <hands-on-title>Setting up InfluxDB</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -42,3 +42,5 @@
>       version: 1.8.0
>     - name: usegalaxy_eu.flower
>       version: 2.1.0
>    +- src: usegalaxy_eu.influxdb
>    +  version: v6.0.7
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Create a new playbook, `monitoring.yml` with the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/monitoring.yml
>    @@ -0,0 +1,4 @@
>    +- hosts: monitoring
>    +  become: true
>    +  roles:
>    +    - usegalaxy_eu.influxdb
>    {% endraw %}
>    ```
>    {: data-commit="Setup the monitoring playbook"}
>
>    During this tutorial we will install everything on the same host, but often one keeps the monitoring infrastructure (Grafana, InfluxDB, Sentry) on a separate host.
>
> 4. Edit the inventory file (`hosts`) an add a group for monitoring like:
>
>    {% raw %}
>    ```diff
>    --- a/hosts
>    +++ b/hosts
>    @@ -4,3 +4,5 @@ gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>     galaxyservers
>     [pulsarservers]
>     gat-0.oz.galaxy.training ansible_user=ubuntu
>    +[monitoring]
>    +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    {% endraw %}
>    ```
>    {: data-commit="Add the monitoring host"}
>
>    **Ensure that the hostname is the full hostname of your machine.**
>
> 4. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook monitoring.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

This will setup an InfluxDB server listening on port `:8086`. The service is currently unauthenticated but it is only listening on `localhost` so it is less of a concern. The service can be authenticated and SSL configured quite easily but that is outside the scope of this tutorial.

You can access the InfluxDB service by running the command `influx`.

```sql
$ influx
Connected to http://localhost:8086 version 1.7.7
InfluxDB shell version: 1.7.7
> show databases
name: databases
name
----
_internal
> use _internal
Using database _internal
> show measurements
name: measurements
name
----
cq
database
httpd
queryExecutor
runtime
shard
subscriber
tsm1_cache
tsm1_engine
tsm1_filestore
tsm1_wal
write
```

The `influx` command provides command line access to InfluxDB in a similar fashion to `psql` for Postgresql. It provides commands like `show databases` and others, but we will not use this interface very often. Telegraf will automatically try to create any database needed, and no interaction is required to setup Grafana to talk to the database.

## Telegraf

We use [Telegraf](https://github.com/influxdata/telegraf) for monitoring as it is incredibly easy to get started with, and it natively integrates with InfluxDB.

### Data Input

Telegraf has extensive documentation on how to configure different types of monitoring, and it supports [a huge array of inputs](https://github.com/influxdata/telegraf#input-plugins). If Telegraf doesn't support a specific data source you wish to query, you can write a bash script to query this data, and have Telegraf [execute it regularly](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec). Telegraf supports several text formats here but the easiest to manage is the InfluxDB line protocol format. The InfluxDB format looks like:


```
weather,country=germany,city=freiburg temperature=25,wind=0 1453832006274169688
weather,country=usa,city=state-college temperature=33,wind=10 1453832006274169688
```

The first portion is the metric name, in both of these cases `weather`. This is just a unique key for a metric which describes the data being collected. Telegraf commonly collects metrics like `cpu`, or `disk`, or `mem`. After the metric name, up to the first space are tags. Tags are used to store categorical data usually, something that is [discrete](https://en.wikipedia.org/wiki/Continuous_or_discrete_variable) and [enumerable](https://en.wikipedia.org/wiki/Enumeration). If you're interested in the weather you would collect some information from a particular city, and then *tag* these values with information about where they're collected. After the space, are the various values or data points that are of interest. For weather this would be information like temperature, wind speed or direction, percentage cloud coverage, etc. After the final space is a timestamp, formatted as [Unix epoch](https://en.wikipedia.org/wiki/Unix_time) with nanosecond precision.

To apply this example to real world data, we'll look at the some example output of the [`disk` plugin](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/disk), which measures information about disks mounted to a server:


```
disk,device=devtmpfs,fstype=devtmpfs,host=stats.galaxyproject.eu,mode=rw,path=/dev free=952795136i,inodes_free=232283i,inodes_total=232616i,inodes_used=333i,total=952795136i,used=0i,used_percent=0 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/dev/shm free=963690496i,inodes_free=235275i,inodes_total=235276i,inodes_used=1i,total=963690496i,used=0i,used_percent=0 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/run free=862007296i,inodes_free=234817i,inodes_total=235276i,inodes_used=459i,total=963690496i,used=101683200i,used_percent=10.551437460684472 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=ro,path=/sys/fs/cgroup free=963690496i,inodes_free=235260i,inodes_total=235276i,inodes_used=16i,total=963690496i,used=0i,used_percent=0 1564485157000000000
disk,device=vda2,fstype=xfs,host=stats.galaxyproject.eu,mode=rw,path=/ free=6652604416i,inodes_free=6134444i,inodes_total=6290368i,inodes_used=155924i,total=12872298496i,used=6219694080i,used_percent=48.31844197780791 1564485157000000000
disk,device=vdb,fstype=xfs,host=stats.galaxyproject.eu,mode=rw,path=/vdb free=2013384704i,inodes_free=1044728i,inodes_total=1048576i,inodes_used=3848i,total=2136997888i,used=123613184i,used_percent=5.78443173454367 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/run/user/0 free=192741376i,inodes_free=235275i,inodes_total=235276i,inodes_used=1i,total=192741376i,used=0i,used_percent=0 1564485157000000000
disk,device=tmpfs,fstype=tmpfs,host=stats.galaxyproject.eu,mode=rw,path=/run/user/1000 free=192741376i,inodes_free=235275i,inodes_total=235276i,inodes_used=1i,total=192741376i,used=0i,used_percent=0 1564485157000000000

```

Reformatting one of the lines to be a bit easier to read:

```
disk,
	device=vdb,
	fstype=xfs,
	host=stats.galaxyproject.eu,
	mode=rw,
	path=/vdb

	free=2013384704i,
	inodes_free=1044728i,
	inodes_total=1048576i,
	inodes_used=3848i,
	total=2136997888i,
	used=123613184i,
	used_percent=5.78443173454367

	1564485157000000000
```

The plugin generates a line of output per disk. It is tagged with the `fstype`, the type of the filesystem (e.g. `ext3`/`ext4`/`xfs`/`autofs`/etc.), the mode by which the filesystem was mounted (`rw` or `ro`), and the path to which the filesystem was mounted. Then various numbers are collected, the `i` suffix meaning an [integer](https://en.wikipedia.org/wiki/Integer). Those without a suffix are [floats](https://en.wikipedia.org/wiki/Floating-point_arithmetic). In this example, you can see that it is an `xfs` formatted device, available as vdb, mounted read-write on the path /vdb. Telegraf automatically tags values with the name of the host from which the data was collected.

The values collected usually have quite self-explanatory names. Here there are `2013384704` bytes free on the disk, 1044728 [inodes](https://en.wikipedia.org/wiki/Inode), so many bytes total and used, and finally 5.784% disk usage.

### `gxadmin`

In the Galaxy world, the InfluxDB line protocol format and [exec](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec) plugin are commonly seen together in conjunction with [gxadmin](https://github.com/usegalaxy-eu/gxadmin) to run various database queries, and store the results into InfluxDB.

`gxadmin` provides various commands to inspect the database like `gxadmin query queue-detail`. It was written such that many of the queries can be automatically formatted for consumption by InfluxDB:

```
$ gxadmin iquery queue-overview --short-tool-id
queue-overview,tool_id=__SET_METADATA__,tool_version=1.0.1,state=new,handler=handler_main_9,destination_id=unknown,job_runner_name=unknown count=3
queue-overview,tool_id=iuc/mothur_pre_cluster/mothur_pre_cluster,tool_version=1.39.5.0,state=new,handler=handler_main_2,destination_id=unknown,job_runner_name=unknown count=1
```

If Telegraf runs this command in an exec block, the data will then be available for graphing. Here the only numeric value we're collecting is the *count* of how many jobs of that type are currently running.

```
queue-overview,
	tool_id=__SET_METADATA__,
	tool_version=1.0.1,
	state=new,
	handler=handler_main_9,
	destination_id=unknown,
	job_runner_name=unknown

	count=3
```

We capture information about what tool is running, the job state, and where it is running. This will allow us to produce nice graphs of the current queue status, how many jobs are new or queued or running.


### Configuring Telegraf

Setting up Telegraf is again very simple. We just add a single role to our playbook and set some variables.


> <hands-on-title>Dependencies</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -44,3 +44,7 @@
>       version: 2.1.0
>     - src: usegalaxy_eu.influxdb
>       version: v6.0.7
>    +# Monitoring
>    +- name: dj-wasabi.telegraf
>    +  src: https://github.com/dj-wasabi/ansible-telegraf
>    +  version: 6f6fdf7f5ead491560783d52528b79e9e088bd5b
>    {% endraw %}
>    ```
>    {: data-commit="Add Telegraf requirement"}
>
> 2. Install the requirements
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add an entry to the *end* of your `galaxy.yml` playbook under `roles:`
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -49,6 +49,7 @@
>         - usegalaxy_eu.rabbitmqserver
>         - galaxyproject.gxadmin
>         - galaxyproject.cvmfs
>    +    - dj-wasabi.telegraf
>       post_tasks:
>         - name: Setup gxadmin cleanup task
>           ansible.builtin.cron:
>    {% endraw %}
>    ```
>    {: data-commit="Add telegraf to the monitoring playbook"}
>
> 4. Create and edit `group_vars/all.yml` and add the following variables:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/all.yml
>    +++ b/group_vars/all.yml
>    @@ -20,3 +20,28 @@ galaxy_job_metrics_plugins:
>       - type: env
>       - type: cgroup
>       - type: hostname
>    +
>    +# Telegraf
>    +telegraf_agent_package_state: latest
>    +
>    +# Configure the output to point to an InfluxDB
>    +# running on localhost, and # place data in the
>    +# database "telegraf" which will be created if need be.
>    +telegraf_agent_output:
>    +  - type: influxdb
>    +    config:
>    +    - urls = ["http://127.0.0.1:8086"]
>    +    - database = "telegraf"
>    +
>    +# The default plugins, applied to any telegraf-configured host
>    +telegraf_plugins_default:
>    +  - plugin: cpu
>    +  - plugin: disk
>    +  - plugin: kernel
>    +  - plugin: processes
>    +  - plugin: diskio
>    +  - plugin: mem
>    +  - plugin: system
>    +  - plugin: swap
>    +  - plugin: net
>    +  - plugin: netstat
>    {% endraw %}
>    ```
>    {: data-commit="Setup telegraf's variables"}
>
>    This configures telegraf to output to the configured influxdb server in the `telegraf` database. A number of plugins are enabled as `defaults` like cpu or disk or memory, all of which is generically interesting to observe, across every host.
>
>    Any host that we setup Telegraf on, will have this base configuration. If you are setting up multiple hosts to be monitored, you will need to put the full hostname (or IP address) where InfluxDB will be available in the `telegraf_agent_output`, rather than `127.0.0.1`. That way every host with Telegraf will send data to the correct location.
>
> 5. Now with the generic configuration applied to all of our hosts, we will apply some specific configuration to the Galaxy server.
>
>    Open your `group_vars/galaxyservers.yml` file, and add the following variables at the end not under any category:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -331,3 +331,12 @@ flower_ui_users:
>
>     flower_environment_variables:
>       GALAXY_CONFIG_FILE: "{{ galaxy_config_file }}"
>    +
>    +# Telegraf
>    +telegraf_plugins_extra:
>    +  listen_galaxy_routes:
>    +    plugin: "statsd"
>    +    config:
>    +      - service_address = ":8125"
>    +      - metric_separator = "."
>    +      - allowed_pending_messages = 10000
>    {% endraw %}
>    ```
>    {: data-commit="Add extra monitoring for Galaxy"}
>
>    We have configured the `statsd` plugin for telegraf, as we will use it to receive Galaxy timing data. [StatsD](https://github.com/statsd/statsd) was an earlier time series database and had an associated line protocol with a different format. Telegraf supports data sent in this format, allowing us to reuse the long-present Galaxy support for this with our newer Telegraf/InfluxDB setup. Telegraf parses the data and converts it into a format that InfluxDB can understand.
>
> 5. Lastly, we need to enable Galaxy to send data to Telegraf:
>
>    In `group_vars/galaxyservers.yml`, edit the `galaxy_config` block, and add `statsd_host: localhost` and `statsd_influxdb: true` under the `galaxy` subsection. It should look like:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -116,6 +116,9 @@ galaxy_config:
>         celery_conf:
>           result_backend: "redis://localhost:6379/0"
>         enable_celery_tasks: true
>    +    # Monitoring
>    +    statsd_host: localhost
>    +    statsd_influxdb: true
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add extra monitoring for Galaxy"}
>
> 6. Run the `galaxy.yml` playbook
>
{: .hands_on}

> ```bash
> 3.sh
> ```
> {: data-test="true"}
{: .hidden}



# Monitoring with Grafana

## Grafana

[Grafana](https://grafana.com/) provides a visual interface to our metrics. It includes a nice query builder that provides a uniform experience across multiple backend databases, along with many attractive graphing and other visualization options. Each page in the Grafana webserver display is called a "dashboard." Dashboards can each have multiple visualizations and graphs, all responding to the data collected by InfluxDB. Another benefit of using Grafana is that many of the UseGalaxy.\* servers share their dashboards publicly, and you can easily copy these and use them on your own server.

There are some nice examples of dashboards available from the public Galaxies, we recommend that you peruse them to get an idea of the possibilities:

- [UseGalaxy.eu](https://stats.galaxyproject.eu/)
- [UseGalaxy.org.au](https://stats.genome.edu.au/)

> <hands-on-title>Setting up Grafana</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -48,3 +48,5 @@
>     - name: dj-wasabi.telegraf
>       src: https://github.com/dj-wasabi/ansible-telegraf
>       version: 6f6fdf7f5ead491560783d52528b79e9e088bd5b
>    +- src: cloudalchemy.grafana
>    +  version: 0.14.2
>    {% endraw %}
>    ```
>    {: data-commit="Add grafana requirement"}
>
> 2. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add `cloudalchemy.grafana` to your `monitoring.yml` playbook:
>
>    {% raw %}
>    ```diff
>    --- a/monitoring.yml
>    +++ b/monitoring.yml
>    @@ -2,3 +2,4 @@
>       become: true
>       roles:
>         - usegalaxy_eu.influxdb
>    +    - cloudalchemy.grafana
>    {% endraw %}
>    ```
>    {: data-commit="Add grafana to monitoring playbook"}
>
> 4. Edit the file `group_vars/monitoring.yml` and set the following variables:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/monitoring.yml
>    @@ -0,0 +1,17 @@
>    +grafana_url: "https://{{ inventory_hostname }}/grafana/"
>    +
>    +grafana_security:
>    +    # Please change at least the password to something more suitable
>    +    admin_user: admin
>    +    admin_password: password
>    +
>    +# These datasources will be automatically included into Grafana
>    +grafana_datasources:
>    + - name: Galaxy
>    +   type: influxdb
>    +   access: proxy
>    +   url: http://127.0.0.1:8086
>    +   isDefault: true
>    +   version: 1
>    +   editable: false
>    +   database: telegraf
>    {% endraw %}
>    ```
>    {: data-commit="Configure Grafana"}
>
> 5. Run the monitoring playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook monitoring.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 5. Update the nginx configuration in `templates/nginx/galaxy.j2` to include the following at the end, before the last curly brace
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -109,4 +109,9 @@ server {
>     		proxy_set_header X-Forwarded-Host $host;
>     		proxy_set_header X-Forwarded-Proto $scheme;
>     	}
>    +
>    +	location /grafana/ {
>    +		proxy_pass http://127.0.0.1:3000/;
>    +		proxy_set_header Host $http_host;
>    +	}
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Setup nginx location for grafana"}
>
>    Since we will setup everything on the same host, we will re-use the Nginx server we setup for Galaxy. If you had planned to run the Grafana and InfluxDB servers on a separate host, you would need to setup Nginx for this host separately.
>
> 5. Run the Galaxy playbook which includes Nginx:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

This has now deployed Grafana on your domain under `/grafana/`, with the username and password you set. The datasource, from which Grafana obtains data, is preconfigured. The Grafana web application will now be available, but currently there is no data available to it. We will return to Grafana shortly in the tutorial to configure dashboards once data is present.

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}

The stats have been collecting in InfluxDB for a few minutes, so now we will now configure Grafana with dashboards and alerting rules.

## Importing a dashboard

For any public Grafana dashboard, you can copy the dashboard for your own use. This is a nice feature of Grafana that has really helped it spread in the Galaxy community, any cool thing one of us builds, everyone else can copy and build upon.

> <hands-on-title>Import a dashboard</hands-on-title>
>
> 1. [Visit UseGalaxy.eu's Node Detail dashboard](https://stats.galaxyproject.eu/d/000000023/node-detail-infrastructure?orgId=1)
>
> 2. Look for the sharing icon at the top and click it
>
> 3. Under the "Export" tab, click "Save to file"
>
> 4. On your own Grafana server, on the home page, hover over the `+` icon and use "Import" from the menu.
>
> 5. Click "Upload .json file" and select the json dashboard you downloaded
>
> 6. Click "Import".
>
{: .hands_on}

With this, your first dashboard should be live! You should see some data from your Galaxy instance, like CPU/load/memory/etc. This can give you a nice `htop` like view into your systems, all collected in one easy dashboard. At the top you will see a box labelled "Host" with a dropdown. If you have more systems, you can click here to select between different machines.

## Setting up a Galaxy dashboard

Importing dashboards is a good start, but it's more interesting to create our own that's personalised to our needs.

> <hands-on-title>Create a dashboard</hands-on-title>
>
> 1. Again find the `+` icon in Grafana and create a dashboard. This will bring you to a new screen
>
> 2. Click *Add Query*, and you will be dropped into the Grafana query builder
>
>    ![Grafana query builder interface](../../images/grafana-query-builder.png)
>
>    This is the query builder interface. The interface somewhat resembles a SQL query, selecting data *from* a database, *where* it meets some condition, *select*ing some specific data, and *grouping by* time period. If this isn't immediately clear how it behaves, hopefully it will become more clear once you have built some queries.
>
> 3. Let's build a query:
>    - From:
>      - *"select measurement"*: `galaxy.`
>    - Select:
>      - *"field(value)"*: `field(mean)`
>    - Group by:
>      - *"fill(null)"*: `fill(none)`
>      - add new (+): `tag(path)`
>    - Alias by: `[[tag_path]]`
>
> 4. At the top of the page it probably says "Last 6 hours", click this to change to "Last 30 minutes"
>
> 5. Remember to save the dashboard (using the {% icon galaxy-save %} at the top), and give it a name like "Galaxy"
>
> 6. You can hit <kbd>Escape</kbd> to exit out of the graph editor, if need be.
>
{: .hands_on}

This will track how long it takes the interface to respond on various web routes and API routes. The collection of individual points is a bit hard to interpret the "feeling" of, so it's common to add a query like the [95th percentile](https://en.wikipedia.org/wiki/Percentile) of requests. This is a value that is calculated from all of the data points collected. The 95th percentile means that 95% of requests are responded to more quickly than this value.

> <hands-on-title>Add a second query to an existing graph</hands-on-title>
>
> 1. The top of the graph is probably labelled "Panel Title", unless you changed it. Click this to access a dropdown and click "Edit"
>
> 2. On the right side you will find a button **Add Query**, click it.
>
> 3. Let's build a query:
>    - From:
>      - *"select measurement"*: `galaxy.`
>    - Select:
>      - *"field(value)"*: `field(mean)`
>      - add new (+): Selectors → percentile
>    - Group by:
>      - *"fill(null)"*: `fill(none)`
>    - Alias by: `percentile`
>
> 4. Remember to save the dashboard
>
> 5. You can hit <kbd>Escape</kbd> to exit out of the graph editor
>
> 6. At any time you can drag the bottom right corner of the graph to resize it as needed.
>
{: .hands_on}

We should now have a graph that gives us not only individual data points, but also a more easily consumable overall representation. For many metrics there are so many data points individually collected that it can be overwhelming and so it is often useful to come up with an aggregate representation that summarises the points into a more easily consumed value. We will touch upon this again under the monitoring section.

## Styling

There is a significant amount of visual styling that one can do to the graphs to make data more or less prominent as you need.

![Grafana graph configuration sections](../../images/grafana-graph-sections.png "Graph configuration sections, from top to bottom: data, visualisation, configuration, and alerting")

We will update the panel we've added to highlight the important information and downplay less important facets, as well as configuring it to have a nicer title than "Panel Title"

> <hands-on-title>Styling the graph</hands-on-title>
>
> 1. Again edit the one graph we've added in our dashboard
>
> 2. On the left side, select the second icon, Visualisation
>
> 3. In the first section, we can edit some display attributes.
>
>    We are primarily interested in the 95th percentile line. Here we can change the display from points to lines or bars, change the tooltip that displays when we hover over the graph, among other facets.
>
>    - Click **Add series override**, click in the box that appears and type "percentile", to find the series we created early. Select it when it appears. We can now apply custom styling to this series.
>    - Click the "+" after the series override, and find **"Color → change"**, and choose a colour that you like.
>    - Click the "+" again, and find **"Line width → 5"** to make the line more visible. This is the most important facet of the graph for us, so we can emphasise it through colour and size.
>
> 3. In the second section, we can edit information about the Axes:
>
>    - Left Y
>      - *"Unit"*: Time → millisecond (ms)
>
>    This will cause the axis to display nicely at any scale. Before Grafana only knew it was a number, now it knows the type of number. If you have a request that takes 30 seconds, it will display as "30s" in Grafana, rather than "30000 ms".
>
>    This section is also useful for setting the scale if your data is better viewed with a logarithmic scale, or forcing specific axis bounds. Commonly on graphs that show things like "disk usage percentage", it can be useful to set the axis bounds to 0 and 100 to ensure that full context is available.
>
> 4. In the next section, "Legend" allows us to format the legend in more interesting ways.
>
>    - Options
>      - *"Show"*: yes
>      - *"As table"*: yes
>      - *"To right"*: yes
>    - Values
>      - *"Min"*: yes
>      - *"Max"*: yes
>      - *"Avg"*: yes
>
>    Clicking on the headers of the table that has appeared, you can force it to be sorted by one column or another.
>
> 5. On the left side, select the third icon, "General"
>
>    - *"Title"*: `Galaxy Request Times`
>
> 6. Save the dashboard
>
{: .hands_on}

Your graph should look something like the following:

![Final graph](../../images/grafana-graph-final.png "The final graph should look something like this, but probably different based on the requests to your server. For this image the author made repeated, automated requests to Galaxy to produce more data points than might normally be present.")

## Monitoring

Collecting all of this data is interesting to visualise but as an administrator you surely have more interesting things to do than to watch graphs all day. Many organisations like to display these dashboards on large monitors, but again this assumes someone is watching it. Everyone has better things to do with their time! So we will setup monitoring on the most important aspects of our system.

Doing monitoring effectively, without causing undue burden to the administrators (extraneous alerts that are not actionable), or the users (unexpected/unnoticed downtime), is a very complex topic. Recommended reading here includes the [monitoring chapter](https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/) of the Google SRE book {% cite Beyer:2016:SRE:3006357 %} which can provide some general guidance on the topic and what metrics may be interesting or annoying to alert upon.

> <comment-title>No generic advice</comment-title>
> We cannot easily provide generic and applicable recommendations, that work across every system and every scale. Some of these performance bounds or features you will need to discover yourself, either adding new metrics in support of this, or changing monitoring thresholds to match the values you need.
{: .comment }

We will add an example alert, to make you familiar with the process. This is not an alert that will probably be useful in production.

> <hands-on-title>Add an alert to your graph</hands-on-title>
>
> 1. Again edit the `Galaxy Request Times` graph
>
> 2. On the left side, select the last icon, Alerting
>
> 3. Click **Create Alert**
>
>    Alerts consist of a *Rule*, with some name, evaluated every N seconds, for a period of time. The *for* can be an important parameter, which you can read more about [in the Grafana documentation](https://grafana.com/docs/alerting/rules/).
>
>    Under some *conditions*, this alert will activate. We will change the conditions of the alert here:
>
>    - *"When"*: `avg()`
>    - *"OF"*: `query(B, 1m, now)`, here we select query B, the 95th percentile track, and the average over 1 minute
>    - *"IS ABOVE"*: `50`
>
>    We will not configure a notification channel in this tutorial. In practice, it is useful to do to ensure that the relevant people are notified automatically. If you and your team use [one of the many services](https://grafana.com/docs/alerting/notifications/), then you can configure this following their documentation.
>
> 4. Save the dashboard
>
{: .hands_on}

# Telegraf & `gxadmin`

Via this setup using `systemd` we collect metrics about Galaxy request times. To get statistics about other Galaxy-specific metrics such as the job queue status, we need to use `gxadmin` to query the Galaxy database and configure Telegraf to consume this data. In this section we will setup gxadmin, and to configure Telegraf to have permissions to run it.


## Installing gxadmin

It's simple to install gxadmin. Here's how you do it, if you haven't done it already:

> <hands-on-title>Installing gxadmin and configuring Telegraf</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    ```yml
>    - src: galaxyproject.gxadmin
>      version: 0.0.12
>    ```
>
> 2. Install the role with `ansible-galaxy install -p roles -r requirements.yml`
>
> 3. Add the role to your `galaxy.yml` playbook, it should come before the Telegraf role.
>
{: .hands_on}

You can run the playbook now, or wait until you have configured Telegraf below:

## Configuring Telegraf for gxadmin

> <hands-on-title>Configuring Telegraf</hands-on-title>
>
> 1. Edit the `group_vars/dbservers.yml`, we need to add some additional permissions to permit Telegraf to run `gxadmin`:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/dbservers.yml
>    +++ b/group_vars/dbservers.yml
>    @@ -2,9 +2,16 @@
>     # PostgreSQL
>     postgresql_objects_users:
>       - name: "{{ galaxy_user_name }}"
>    +  - name: telegraf
>     postgresql_objects_databases:
>       - name: "{{ galaxy_db_name }}"
>         owner: "{{ galaxy_user_name }}"
>    +postgresql_objects_privileges:
>    +  - database: galaxy
>    +    roles: telegraf
>    +    privs: SELECT
>    +    objs: ALL_IN_SCHEMA
>    +
>
>     # PostgreSQL Backups
>     postgresql_backup_dir: /data/backups
>    {% endraw %}
>    ```
>    {: data-commit="Configure database permissions for Telegraf"}
>
> 2. Edit the `group_vars/galaxyservers.yml`, we need to configure Telegraf to run `gxadmin`
>
>    Under `telegraf_plugins_extra`, where we already have set a Galaxy StatsD listener, add a stanza to monitor the Galaxy queue
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -343,3 +343,10 @@ telegraf_plugins_extra:
>           - service_address = ":8125"
>           - metric_separator = "."
>           - allowed_pending_messages = 10000
>    +  monitor_galaxy_queue:
>    +    plugin: "exec"
>    +    config:
>    +      - commands = ["/usr/bin/env PGDATABASE=galaxy /usr/local/bin/gxadmin iquery queue-overview --short-tool-id"]
>    +      - timeout = "10s"
>    +      - data_format = "influx"
>    +      - interval = "15s"
>    {% endraw %}
>    ```
>    {: data-commit="Add extra monitoring for Galaxy"}
>
>    This one is slightly more complex in the configuration. The command block does several things:
>
>    - it wraps the command with `env` which allows setting environment variables for a single command
>    - It sets the `PGDATABASE` to the Galaxy database, by default the `psql` will try and connect to a database with the same name of the user. So the `telegraf` user will attempt to connect to a (non-existent) `telegraf` database.
>    - Then it calls the gxadmin command `queue-overview`. By using `iquery` instead of `query`, the output is automatically converted to InfluxDB line protocol.
>    - The command is run every 15 seconds, and has a timeout of 10 seconds. If the command fails to finish in 10 seconds, it will be killed.
>
> 3. Run the Galaxy playbook
>
{: .hands_on}

With this, Telegraf will start monitoring the Galaxy queue by calling the query every few seconds to check the status every 15 seconds. This monitoring *will* miss jobs that complete within the 15 second interval, but for most servers this is not an issue. Most jobs are running for more than 15 seconds, and if not, it still gives an accurate point-in-time view.

We'll now create a graph for this, just like the one on [stats.galaxyproject.eu](https://stats.galaxyproject.eu)

> <hands-on-title>Building the queue graph</hands-on-title>
>
> 1. Click the **new graph** button at the top of Grafana's interface, and *Add a Query*
>
> 2. Let's build a query:
>    - From:
>      - *"select measurement"*: `queue-overview`
>    - Select:
>      - *"field(value)"*: `field(count)`
>      - add new (+): Aggregations → sum
>    - Group by:
>      - *"time(__interval)"*: `time(15s)`, because we set the interval to 15s
>      - add new (+): `tag(tool_id)`
>      - add new (+): `tag(tool_version)`
>    - Alias by: `[[tag_tool_id]]/[[tag_tool_version]]`
>
> 3. In the second tab on the left, Visualisation:
>
>    Under the first section:
>
>    - Draw Modes:
>      - *"Bars"*: `no`
>      - *"Lines"*: `yes`
>      - *"Points"*: `no`
>    - Mode Options:
>      - *"Staircase"*: `yes`
>    - Stacking & Null Value
>      - *"Stack"*: `yes`
>      - *"Null Value"*: `null as zero`
>
>    Below, under the *Legend* section,
>
>    - Options:
>      - *"Show"*: `yes`
>      - *"As table"*: `yes`
>      - *"To Right"*: `yes`
>    - Values:
>      - *"Max"*: `yes`
>      - *"Avg"*: `yes`
>      - *"Current"*: `yes`
>    - Hide Series:
>      - *"With only nulls"*: `yes`
>      - *"With only zeros"*: `yes`
>
> 4. In the third tab, General settings:
>
>    - *"Title"*: `Galaxy Queue Overview`
>
> 5. You can hit <kbd>Escape</kbd> to exit out of the graph editor, and remember to save your dashboard.
>
{: .hands_on}

Run some tools in Galaxy, try to generate a large number of jobs. It is relatively easy to upload a dataset, and then run the "Secure Hash / Message Digest" or another tool repeatedly, running it over every dataset in your history, repeating until you've generated a few dozen datasets. If you have a slower tool like `bwa` installed, this can be an option too.


![Final Graph](../../images/grafana-final-graph.png "Final graphs generated from this tutorial.")

You can also import a [copy of the dashboard]({{ site.baseurl }}{{ page.dir }}dashboard.json).

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=13 %}

# Galaxy Job Radar
Another possibility for data visualisation in Galaxy ecosystem is [Galaxy Job Radar](https://github.com/CESNET/gjr) (GJR). GJR is a project that visualizes traffic between [Galaxy server](https://galaxyproject.org) and its [Pulsars](https://github.com/galaxyproject/pulsar).

![image](../../images/gjr_logo.png "Logo of GJR project.")

Namely it shows jobs of a Galaxy instance, their distribution over Pulsar computational nodes, in which state they are and more. It supports both live view and history replay of traffic and past schedule evaluation.

[CESNET](https://www.cesnet.cz/en) manages a central instance of GJR at [https://gjr.metacentrum.cz](https://gjr.metacentrum.cz)

**Galaxy Job Radar admins invite any public Galaxy servers to join this effort and share the anonymous data about their jobs with the central instance.**

## How to join

### Summary

*Start with writing us an email at galaxy@cesnet.cz that you'd like to join. First we will celebrate and then happily walk you through what needs to be done.*

1. Your Galaxy needs to be sending data to your InfluxDB via [gxadmin scripts](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master/roles/usegalaxy-eu.job-radar-stats-influxdb)
2. You give us access to read the InfluxDB (we need: `influxdb_password_var_name;influxdb_host;influxdb_port;influxdb_username`) and some basic information about your server (we need: `name;lat;long;`)
3. You send us information about your Pulsar servers (we need: `galaxy;pulsar_id;lat;long;node_count;desc`)
4. Wait few days and your Galaxy and Pulsars show up at [https://gjr.metacentrum.cz](https://gjr.metacentrum.cz)

### Details

GJR periodically requests data from each connected Galaxy through their own `InfluxDB` instance. Schema of this setup:

![image](../../images/gjr_global_data_flow.png "Flow of data from all connected Galaxy servers to shared InfluxDB."){: width="583" height="449"}

For this to work we need the contributors to run an InfluxDB, fill it periodically with anonymous new data (with `gxadmin` scripts below) and give us read access to this data on InfluxDB. 

Note: If you do not run InfluxDB for monitoring yet there is a Galaxy [training section]({% link topics/admin/#st-monitoring %}) above available which will explain reasons and guide you through the setup. 

Simplified schema of the full setup:

![image](../../images/gjr_local_data_flow.png "How collected Galaxy statistics are being visualized."){: width="537" height="212"}

# Conclusion

Monitoring with Telegraf, InfluxDB, Grafana and Galaxy Job Radar can provide an easy solution to monitor your infrastructure. The UseGalaxy.\* servers use this stack and it has proven to be effective in production situations, with large Galaxy servers. The base monitoring done with Telegraf is easy to setup and extend on a per-site basis simply by adding scripts or commands to your servers which generate InfluxDB line protocol formatted output. Grafana provides an ideal visualisation solution as it encourages sharing, and allows you to import whatever dashboards have been developed by UseGalaxy.\*, and then to extend them to your own needs. Galaxy Job Radar is experimental solution still under development and ongoing testing. Still it is valuable if you join it so that it can bring admins and users overall view of Galaxy jobs traffic accros the world.

{% snippet topics/admin/faqs/git-gat-path.md tutorial="monitoring" %}


topic: admin
tutorial: sentry
title: Monitoring Galaxy and Pulsar with Sentry
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/sentry/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "git-gat"]
content_hash: 513d75a90f44b49181395fd5e54c8d41
last_modified: 2026-05-27T09:01:47.638477
workflows_json: []
content:
# Overview

Sentry is an error tracking software that helps admins and developers monitor and diagnose issues in their applications. It provides real-time alerts for errors and allows users to capture context information about each error, such as stack traces and user feedback. It is often possible to find and fix errors before users report them. Galaxy and Pulsar can log issues and failing tool runs to Sentry.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="sentry" %}

We're going to set up a local Sentry instance using docker-compose and connect Galaxy and Pulsar to that Sentry instance. Alternatively, you can use the hosted Sentry at https://sentry.io/.

# Installing and Configuring

To proceed from here it is expected that:

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have set up a working Galaxy instance as described in the [ansible-galaxy](../ansible-galaxy/tutorial.html) tutorial.
>
{: .comment}

# Installing and Configuring

First we need to add our new Ansible role to `requirements.yml`:

> <hands-on-title>Set up Sentry with Ansible</hands-on-title>
>
> 1. In your working directory, add the roles to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -53,3 +53,6 @@
>     # Training Infrastructure as a Service
>     - src: galaxyproject.tiaas2
>       version: 2.1.5
>    +# Sentry
>    +- name: mvdbeek.sentry_selfhosted
>    +  src: https://github.com/mvdbeek/ansible-role-sentry/archive/main.tar.gz
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement" data-ref="add-req"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the roles with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true" data-ref="req-install"}
>    {: .code-in}
>
> 3. Create a new playbook, `sentry.yml` with the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/sentry.yml
>    @@ -0,0 +1,7 @@
>    +- hosts: sentryservers
>    +  become: true
>    +  pre_tasks:
>    +    - pip:
>    +        name: docker-compose
>    +  roles:
>    +    - mvdbeek.sentry_selfhosted
>    {% endraw %}
>    ```
>    {: data-commit="Setup the sentry playbook"}
>
>    During this tutorial we will install everything on the same host, but often one keeps the monitoring infrastructure (Grafana, InfluxDB, Sentry) on a separate host.
>
> 4. Edit the inventory file (`hosts`) an add a group for Sentry like:
>
>    {% raw %}
>    ```diff
>    --- a/hosts
>    +++ b/hosts
>    @@ -6,3 +6,6 @@ galaxyservers
>     gat-0.oz.galaxy.training ansible_user=ubuntu
>     [monitoring]
>     gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    +
>    +[sentryservers]
>    +gat-0.eu.training.galaxyproject.eu ansible_connection=local ansible_user=ubuntu
>    {% endraw %}
>    ```
>    {: data-commit="Add the monitoring host"}
>
>    **Ensure that the hostname is the full hostname of your machine.**
>
>    Sentry requires its own (sub)domain. For the admin training we have set up the sentry.gat-N.eu.galaxy.training subdomain. If you run this tutorial outside of the training and you cannot obtain a domain or subdomain for sentry you can use the free [Duck DNS](https://www.duckdns.org/) service to map an IP address to a domain name.
>
> 5. Edit the file `group_vars/sentryservers.yml` and set the following variables:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/all.yml
>    +++ b/group_vars/all.yml
>    @@ -45,3 +45,6 @@ telegraf_plugins_default:
>       - plugin: swap
>       - plugin: net
>       - plugin: netstat
>    +
>    +# Used in galaxyservers, nginx and sentry
>    +sentry_domain: "sentry.{{ inventory_hostname }}"
>    diff --git a/group_vars/galaxyservers.yml b/group_vars/galaxyservers.yml
>    index 1bdac54..94b3fc3 100644
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -211,6 +211,7 @@ certbot_post_renewal: |
>         docker restart rabbit_hole || true
>     certbot_domains:
>      - "{{ inventory_hostname }}"
>    + - "{{ sentry_domain }}"
>     certbot_agree_tos: --agree-tos
>
>     # NGINX
>    diff --git a/group_vars/sentryservers.yml b/group_vars/sentryservers.yml
>    new file mode 100644
>    index 0000000..36298e3
>    --- /dev/null
>    +++ b/group_vars/sentryservers.yml
>    @@ -0,0 +1,6 @@
>    +sentry_version: 23.3.1
>    +sentry_url: "https://{{ sentry_domain }}"
>    +sentry_docker_compose_project_folder: /srv/sentry
>    +sentry_superusers:
>    +  - email:  admin@example.com
>    +    password: "{{ vault_sentry_password }}"
>    diff --git a/sentry.yml b/sentry.yml
>    index 285910b..ab32a34 100644
>    --- a/sentry.yml
>    +++ b/sentry.yml
>    @@ -1,7 +1,6 @@
>     - hosts: sentryservers
>    +  vars_files:
>    +    - group_vars/secret.yml
>       become: true
>    -  pre_tasks:
>    -    - pip:
>    -        name: docker-compose
>       roles:
>         - mvdbeek.sentry_selfhosted
>    {% endraw %}
>    ```
>    {: data-commit="Configure Sentry"}
>
> 9. We will add an associated admin password to the vault, do that now:
>
>    ><code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_sentry_password: 'some-super-secret-password'
>    ```
>
>    <!-- Ignore this, just for the gat-automation. Vaults are ugly to work with :(
>
>    +vault_sentry_password: password
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/secret.yml
>    +++ b/group_vars/secret.yml
>    @@ -1,22 +1,23 @@
>     $ANSIBLE_VAULT;1.1;AES256
>    -66653366633665383231303635643739396466653465626163633662623230643534346666303434
>    -3531356339646666623364383435306363373534373833370a333964306130653236373438373264
>    -34366433396135303064643932313135643064373034323865363939333565623238386161333465
>    -3932623162353034370a653663333363383938393936623663343639376430653366646662353830
>    -34383964326161383966306361633162653366353162633564623738353137333936313232363764
>    -38383962396637636136316366633936316365643038333030333932336365326335373737663834
>    -39363639303764343430336435313663313762353335366562656232646334356438343535303032
>    -35636131323163323036383163363964643237313137333131303737346662373233663562343031
>    -39396531306539623661376166313534303462623532323334393736316634616262313633626637
>    -37393066663239303564356232366466333334316565363631363230626665333039386133383933
>    -31656234363064646334623938383637393534313638643635396662643163346633346237353664
>    -34666161373764663166653737323736373261363565666633653831363833393264333666356633
>    -30373062333638366666316132623736386336383933326539663065653538386363323766313664
>    -63323737396264306439303366343834326164393763376663316366353766663131303966393039
>    -61343732313865613761623733356465336263633963376161663931653864383162393839643834
>    -34373239386461666462643162613536316166656135323332316163336635393035653164363138
>    -34336265383033663436306436663536383238366665653239313661666431666339323364646633
>    -64366361653034393962316562373835623631356339313062393933633634653735656636373031
>    -65376162633866623733623961303664663931636130373936326461623465363932373165356563
>    -36323838323865326464346337393164343765613333663830636234666663303535616532323936
>    -3230
>    +31383961303162656266343866376462343961326163356238636565386630363164353330643862
>    +3631663631383136356436363635373536323536353534650a393833343465396339373731663238
>    +38383763326230346561633061356634643566663863383838363765326162386133636330656136
>    +3935643333353365660a393932313965373166313838373739306234306634313139656334343839
>    +38393636396164373266646534326436316333313638366234323564323936643863633533366537
>    +66393535333263383837386538636530303137623761333161383765643938396232633863353838
>    +66363464613466353833663833613339666634333734343163326162393031363530643363396238
>    +36306562333763333435646432613565323537373032643335336639353762646662343865663266
>    +33646334623838333230343861353632626461363436343963636664393965613736306162396235
>    +64373239373739613164623165653832653139646433373531323562373534393563363962373634
>    +30393161303439376138653066646230303464353336656431663137366630323830316561356137
>    +65363663663165323666343335626238666334346664373938333839323161353131643336343738
>    +31646330386239363165643866316431386631643139613065663835323036353631646365333530
>    +37306161656562316665643666363132633766386330383864663937316233376261343264633133
>    +39356438623533376539393662326364666434386330626437646235386261323632396661656661
>    +31613232316237626562646330626564333566633266653762356338646236353137343863633634
>    +31623563393933656635323165326638376634383730393131373963353262366337383632666666
>    +64643264333463373334343433373564326632393637626237353336373837373438303231636633
>    +31666636316533313336613532643437646362636431306262623361326532613238303732396266
>    +62346131663135313935393439343433333439623335623131653066316561646631616361303632
>    +66633935356333643539323461333734376466316262656330396662346232333131313864343636
>    +6634613432653361653133383862383338353432666136303465
>    {% endraw %}
>    ```
>    {: data-commit="Add sentry passwords to the vault"}
>
>    -->
>
> 6. Add the nginx routes
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/nginx/sentry.j2
>    @@ -0,0 +1,20 @@
>    +server {
>    +	# Listen on port 443
>    +	listen        *:443 ssl;
>    +	# The virtualhost is our domain name
>    +	server_name   "{{ sentry_domain }}";
>    +
>    +	# Our log files will go here.
>    +	access_log  syslog:server=unix:/dev/log;
>    +	error_log   syslog:server=unix:/dev/log;
>    +
>    +	location / {
>    +		# This is the backend to send the requests to.
>    +		proxy_pass "http://localhost:9000";
>    +
>    +		proxy_set_header Host $http_host;
>    +		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>    +		proxy_set_header X-Forwarded-Proto $scheme;
>    +		proxy_set_header Upgrade $http_upgrade;
>    +	}
>    +}
>    {% endraw %}
>    ```
>    {: data-commit="Add nginx server"}
>
> 6. And make sure the sentry nginx configuration is deployed
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -220,6 +220,7 @@ nginx_servers:
>       - redirect-ssl
>     nginx_ssl_servers:
>       - galaxy
>    +  - sentry
>     nginx_enable_default_server: false
>     nginx_conf_http:
>       client_max_body_size: 1g
>    {% endraw %}
>    ```
>    {: data-commit="Deploy nginx server"}
>
> 7. Run the sentry playbook to deploy sentry and the galaxy playbook to update the nginx configuration.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook sentry.yml galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 8. Generate a project for Galaxy in Sentry
>  Go to the domain you configured for your Sentry instance. You need to log in with the username and admin you've set up in `group_vars/sentryservers.yml`. Click "continue" on the next page. Click "Projects", "Create Project", "Python", select "I'll create my own alerts later", and set "galaxy" as the Project Name. You'll see your project dsn that will look like `https://b0022427ee5345a8ad4cb072c73e62f4@sentry.gat-N.eu.galaxy.training/2`. We will need this string to let Galaxy know where to send data to. To avoid requesting an additional certificate for communication between Galaxy and Sentry we've set up communication via localhost:9000, so you can manually change the @ portion to localhost:9000.
>
> 9. We will add the galaxy project dsn to the vault. Edit your `group_vars/secret.yml` and add the sentry dsn.
>
>    ><code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_galaxy_sentry_dsn: 'http://b0022427ee5345a8ad4cb072c73e62f4@localhost:9000/2'
>    ```
>
> 9. Edit `group_vars/galaxyservers.yml` to reference the new vault secret:
>
>    This will let Galaxy know that captured logs should be sent to our Sentry instance.
>    We will also enable sending performance metrics to Sentry by setting the `sentry_traces_sample_rate` to `0.5`. This will send half of all transactions to Sentry. In a production environment you would reduce this to a smaller percentage of transactions.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -119,6 +119,8 @@ galaxy_config:
>         # Monitoring
>         statsd_host: localhost
>         statsd_influxdb: true
>    +    sentry_dsn: "{{ vault_galaxy_sentry_dsn }}"
>    +    sentry_traces_sample_rate: 0.5
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Configure Galaxy to report to Sentry"}
>
> 10. Run the galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on }

## Generate an error

Galaxy has a built in route that intentionally generates and error.
Just visit: [/error](https://my.gat.galaxy.training/?path=/error)

> <hands-on-title>Open the Galaxy Project in Sentry</hands-on-title>
> 1. Go to your Sentry instance and click on issues. You should see a couple of issues,
> one them should be the "Fake error" exception we generated by visiting [https://galaxy.example.org/error](https://my.gat.galaxy.training/?path=/error).
{: .hands_on }

## Sending tool error reports to Sentry

In addition to sending logging errors to Sentry you can also collect failing tool runs in Sentry. For this we will set up the error reporting configuration file and reference it in galaxy.yml. The `user_submission` parameter controls whether all reports will be collected in Sentry (when set to `false`) or only those that have been reported manually (when set to `true`). For testing purposes we'll also add a tool that will fail running so we can test that submitting tool errors to Sentry works as expected.

> <hands-on-title>Update Galaxy config to send tool error reports</hands-on-title>
>
> 1. Create the  `files/galaxy/config/error_reports.yml` file.
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/config/error_reports.yml
>    @@ -0,0 +1,2 @@
>    +- type: sentry
>    +  user_submission: false
>    {% endraw %}
>    ```
>    {: data-commit="Configure error reporting"}
>
> 2. Create a testing tool in `files/galaxy/tools/job_properties.xml`.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/tools/job_properties.xml
>    @@ -0,0 +1,65 @@
>    +<tool id="job_properties" name="Test Job Properties" version="1.0.0">
>    +    <stdio>
>    +        <exit_code range="127" level="fatal" description="Failing exit code." />
>    +    </stdio>
>    +    <version_command>echo 'v1.1'</version_command>
>    +    <command><![CDATA[
>    +#if $thebool
>    +    echo 'The bool is true' &&
>    +    echo 'The bool is really true' 1>&2 &&
>    +    echo 'This is a line of text.' > '$out_file1' &&
>    +    cp '$out_file1' '$one' &&
>    +    cp '$out_file1' '$two' &&
>    +    sleep $sleepsecs
>    +#else
>    +    echo 'The bool is not true' &&
>    +    echo 'The bool is very not true' 1>&2 &&
>    +    echo 'This is a different line of text.' > '$out_file1' &&
>    +    sleep $sleepsecs &&
>    +    sh -c 'exit 2'
>    +#end if
>    +#if $failbool
>    +    ## use ';' to concatenate commands so that the next one is run independently
>    +    ## of the exit code of the previous one
>    +    ; exit 127
>    +#end if
>    +    ]]></command>
>    +    <inputs>
>    +        <param name="sleepsecs" type="integer" value="0" label="Sleep this many seconds"/>
>    +        <param name="thebool" type="boolean" label="The boolean property" />
>    +        <param name="failbool" type="boolean" label="The failure property" checked="false" />
>    +    </inputs>
>    +    <outputs>
>    +        <data name="out_file1" format="txt" />
>    +        <collection name="list_output" type="list" label="A list output">
>    +            <data name="one" format="txt" />
>    +                <has_line line="The bool is true" />
>    +            </assert_stdout>
>    +            <assert_stderr>
>    +                <has_line line="The bool is really true" />
>    +            </assert_stderr>
>    +            <assert_command_version>
>    +                <has_text text="v1.1" />
>    +            </assert_command_version>
>    +        </test>
>    +        <test expect_exit_code="2">
>    +            <param name="thebool" value="false" />
>    +            <output name="out_file1" file="simple_line_alternative.txt" />
>    +            <assert_command>
>    +                <has_text text="very not" />
>    +            </assert_command>
>    +            <assert_stdout>
>    +                <has_line line="The bool is not true" />
>    +            </assert_stdout>
>    +            <assert_stderr>
>    +                <has_line line="The bool is very not true" />
>    +            </assert_stderr>
>    +        </test>
>    +        <test expect_exit_code="127" expect_failure="true">
>    +            <param name="thebool" value="true" />
>    +            <param name="failbool" value="true" />
>    +        </test>
>    +    </tests>
>    +    <help>
>    +    </help>
>    +</tool>
>    {% endraw %}
>    ```
>    {: data-commit="Configure a tool"}
>
> 3. Edit `group_vars/galaxyservers.yml` to reference the `error_reports.yml` file and the new testing tool.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -121,6 +121,7 @@ galaxy_config:
>         statsd_influxdb: true
>         sentry_dsn: "{{ vault_galaxy_sentry_dsn }}"
>         sentry_traces_sample_rate: 0.5
>    +    error_report_file: "{{ galaxy_config_dir }}/error_reports_file.yml"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    @@ -173,6 +174,8 @@ galaxy_config_files:
>         dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>       - src: files/galaxy/config/tpv_rules_local.yml
>         dest: "{{ tpv_mutable_dir }}/tpv_rules_local.yml"
>    +  - src: files/galaxy/config/error_reports.yml
>    +    dest: "{{ galaxy_config.galaxy.error_report_file }}"
>
>     galaxy_config_templates:
>       - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>    @@ -194,6 +197,7 @@ tpv_privsep: true
>
>     galaxy_local_tools:
>     - testing.xml
>    +- job_properties.xml
>
>     # Certbot
>     certbot_auto_renew_hour: "{{ 23 |random(seed=inventory_hostname)  }}"
>    {% endraw %}
>    ```
>    {: data-commit="Deploy files, error reporting"}
>
> 4. Run the galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on }

## Generating a tool error

To generate a tool error, run the job properties testing tool and set the `failbool` parameter to true.

> <hands-on-title>Open the Galaxy Project in Sentry</hands-on-title>
> 1. Go to your Sentry instance and click on issues. You should see an issue for the tool run error.
{: .hands_on }

## Reporting errors from the Pulsar server

It is also possible to report errors from the Pulsar server. You can either use the Galaxy project we created before in Sentry, or we can create a new project for Pulsar. We recommend creating a separate Pulsar project. Since the Pulsar server runs on a remote VM for this to work you need a valid certificate for the Sentry domain and you cannot use localhost.

> <hands-on-title>Add Sentry connection to Pulsar</hands-on-title>
> 1. Create a new dsn by creating a new pulsar project in Sentry.
> 2. We will add the project dsn to the vault. Edit your `group_vars/secret.yml` and add the sentry dsn.
>
>    ><code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_pulsar_sentry_dsn: 'https://f2a8a00d30224c2c9800a8f79194a32a@{{ groups['sentryservers'][0] }}/3'
>    ```
> 3. Add the sentry dsn to the pulsar group variables.
>    {% raw %}
>    ```diff
>    --- a/group_vars/pulsarservers.yml
>    +++ b/group_vars/pulsarservers.yml
>    @@ -44,6 +44,7 @@ pulsar_yaml_config:
>           - type: conda
>             auto_init: true
>             auto_install: true
>    +  sentry_dsn: "{{ vault_pulsar_sentry_dsn }}"
>
>     # Pulsar should use the same job metrics plugins as Galaxy. This will automatically set `job_metrics_config_file` in
>     # `pulsar_yaml_config` and create `{{ pulsar_config_dir }}/job_metrics_conf.yml`.
>    {% endraw %}
>    ```
>    {: data-commit="Configure pulsar for error reporting"}
>
> 4. Run the pulsar playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on }

Pulsar should now be set up to report errors to Sentry.

{% snippet topics/admin/faqs/missed-something.md step=12 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="sentry" %}


topic: admin
tutorial: job-destinations
title: Mapping Jobs to Destinations using TPV
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["jobs", "git-gat"]
content_hash: 3ecaaa53bd407a6dda014e2835b63380
last_modified: 2026-05-27T09:01:47.636950
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/job-destinations/tutorial.html

content:
This tutorial heavily builds on the [Connecting Galaxy to a compute cluster]({% link topics/admin/tutorials/connect-to-compute-cluster/tutorial.md %}) and it's expected you have completed this tutorial first.

Now that you have a working scheduler, we will start configuring which jobs are sent to which destinations.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="job-destinations" %}

# Mapping jobs to destinations

In order to run jobs in Galaxy, you need to assign them to a resource manager that can handle the task. This involves specifying the appropriate amount of memory and CPU cores. For production installations, the jobs
must be routed to a resource manager like SLURM, HTCondor, or Pulsar. Some tools may need specific resources such as GPUs or multi-core machines to work efficiently.

Sometimes, your available resources are spread out across multiple locations and resource managers. In such cases, you need a way to route your jobs to the appropriate location. Galaxy offers several methods for
routing jobs, ranging from simple static mappings to custom Python functions via [dynamic job destinations](https://docs.galaxyproject.org/en/latest/admin/jobs.html#dynamic-destination-mapping).

Recently, the Galaxy project has introduced a library named [{TPV}](https://total-perspective-vortex.readthedocs.io/) to simplify this process. {TPV} provides a admin-friendly YAML configuration that works for most scenarios.
For more complex cases, TPV also allows you to embed Python code into the configuration YAML file and implement fine-grained control over jobs. 

Lastly, TPV offers a shared global database of default resource requirements (more below). By leveraging this database, admins don't have to figure out the requirements for each tool
separately.

## Writing a testing tool

To demonstrate a real-life scenario and {TPV}'s role in it, let's plan on setting up a configuration where the VM designated for training jobs doesn't run real jobs and hence doesn't get overloaded. To start, we'll create a "testing" tool that we'll use in our configuration. This testing tool can run quickly, and without overloading our small machines.

> <hands-on-title>Deploying a Tool</hands-on-title>
>
> 1. Create the directory `files/galaxy/tools/` if it doesn't exist and edit a new file in `files/galaxy/tools/testing.xml` with the following contents:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/tools/testing.xml
>    @@ -0,0 +1,11 @@
>    +<tool id="testing" name="Testing Tool">
>    +    <command>
>    +        <![CDATA[echo "Running with '\${GALAXY_SLOTS:-1}' threads" > "$output1"]]>
>    +    </command>
>    +    <inputs>
>    +        <param name="input1" type="data" format="txt" label="Input Dataset"/>
>    +    </inputs>
>    +    <outputs>
>    +        <data name="output1" format="txt" />
>    +    </outputs>
>    +</tool>
>    {% endraw %}
>    ```
>    {: data-commit="Add testing tool"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Add the tool to the Galaxy group variables under the new item `galaxy_local_tools` :
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -162,6 +162,9 @@ galaxy_config_templates:
>     galaxy_extra_dirs:
>       - /data
>
>    +galaxy_local_tools:
>    +- testing.xml
>    +
>     # Certbot
>     certbot_auto_renew_hour: "{{ 23 |random(seed=inventory_hostname)  }}"
>     certbot_auto_renew_minute: "{{ 59 |random(seed=inventory_hostname)  }}"
>    {% endraw %}
>    ```
>    {: data-commit="Deploy testing tool"}
>
> 3. Run the Galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. Reload Galaxy in your browser and the new tool should now appear in the tool panel. If you have not already created a dataset in your history, upload a random text dataset. Once you have a dataset, click the tool's name in the tool panel, then click Execute.
>
>    > <question-title></question-title>
>    >
>    > What is the tool's output?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > Running with '1' threads
>    > > ```
>    > >
>    > {: .solution }
>    >
>    {: .question}
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

Of course, this tool doesn't actually *use* the allocated number of cores. In a real tool, you would call the tools's underlying command with whatever flag that tool provides to control the number of threads or processes it starts, such as `samtools sort -@ \${GALAXY_SLOTS:-1}`.

## Safeguard: TPV Linting

If we want to change something in production, it is always a good idea to have a safeguard in place. In our case, we would like to check the {TPV} configuration files for syntax errors, so nothing will break when we deploy broken yaml files or change them quickly on the server. TPV-lint-and-copy works with two separate locations:
- one where you can safely edit your files
- and the actual production config directory that galaxy reads.

Once you are done with your changes, you can run the script and it will automatically lint and copy over the files, if they are correct *and* mentioned in your job_conf.yml file, or in your `group_vars/galaxyservers.yml` inline `job_conf`.
And of course, Galaxy has an Ansible Role for that.

> <hands-on-title>Adding automated TPV-lind-and-copy-script</hands-on-title>
>
> 1. Add the role to your `requirements.yml`.
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -27,3 +27,6 @@
>       version: 0.0.3
>     - src: galaxyproject.slurm
>       version: 1.0.2
>    +# TPV Linting
>    +- name: usegalaxy_eu.tpv_auto_lint
>    +  version: 0.4.3
>    {% endraw %}
>    ```
>    {: data-commit="Add tpv-auto-lint to requirements"}
>
> 2. Install the missing role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Change your `group_vars/galaxyservers.yml`. We need to create a new directory where the TPV configs will be stored after linting, and add that directory name as variable for the role. The default name is 'TPV_DO_NOT_TOUCH' for extra safety 😉. If you want a different name, you need to change the `tpv_config_dir_name` variable, too. We also need to create a directory, `tpv_mutable_dir` (a role default variable), where TPV configs are copied before linting.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -145,6 +145,8 @@ galaxy_config:
>               - job-handlers
>               - workflow-schedulers
>
>    +galaxy_job_config_file: "{{ galaxy_config_dir }}/galaxy.yml"
>    +
>     galaxy_config_files_public:
>       - src: files/galaxy/welcome.html
>         dest: "{{ galaxy_mutable_config_dir }}/welcome.html"
>    @@ -161,6 +163,11 @@ galaxy_config_templates:
>
>     galaxy_extra_dirs:
>       - /data
>    +  - "{{ galaxy_config_dir }}/{{ tpv_config_dir_name }}"
>    +
>    +galaxy_extra_privsep_dirs:
>    +  - "{{ tpv_mutable_dir }}"
>    +tpv_privsep: true
>
>     galaxy_local_tools:
>     - testing.xml
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV config dir"}
> 4. Add the role to your `galaxy.yml` playbook.
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -38,6 +38,7 @@
>         - galaxyproject.slurm
>         - usegalaxy_eu.apptainer
>         - galaxyproject.galaxy
>    +    - usegalaxy_eu.tpv_auto_lint
>         - role: galaxyproject.miniconda
>           become: true
>           become_user: "{{ galaxy_user_name }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add tpv-auto-lint role"}
> 5. At this point we won't run the modified playbook just yet. Because TPV itself has not yet been installed,
>    the tpv_auto_lint role would fail at this point. So first, we'll have to install and configure TPV itself before the
>    linter can work.
>
{: .hands_on}
## Configuring TPV

We want our tool to run with more than one core. To do this, we need to instruct Slurm to allocate more cores for this job. First however, we need to configure Galaxy to use TPV.

> <hands-on-title>Adding TPV to your job configuration</hands-on-title>
>
> 1. Edit `group_vars/galaxyservers.yml` and add the following destination under your `job_config` section to route all jobs to TPV.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -27,38 +27,15 @@ galaxy_job_config:
>       handling:
>         assign: ['db-skip-locked']
>       execution:
>    -    default: slurm
>    +    default: tpv_dispatcher
>         environments:
>           local_env:
>             runner: local_runner
>             tmp_dir: true
>    -      slurm:
>    -        runner: slurm
>    -        singularity_enabled: true
>    -        # Enabling access to the reference data on CVMFS in the container
>    -        singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    -        env:
>    -        - name: LC_ALL
>    -          value: C
>    -        - name: APPTAINER_CACHEDIR
>    -          value: /tmp/singularity
>    -        - name: APPTAINER_TMPDIR
>    -          value: /tmp
>    -      singularity:
>    -        runner: local_runner
>    -        singularity_enabled: true
>    -        # Enabling access to the reference data on CVMFS in the container
>    -        singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    -        env:
>    -        # Ensuring a consistent collation environment is good for reproducibility.
>    -        - name: LC_ALL
>    -          value: C
>    -        # The cache directory holds the docker containers that get converted
>    -        - name: APPTAINER_CACHEDIR
>    -          value: /tmp/singularity
>    -        # Apptainer uses a temporary directory to build the squashfs filesystem
>    -        - name: APPTAINER_TMPDIR
>    -          value: /tmp
>    +      tpv_dispatcher:
>    +        runner: dynamic_tpv
>    +        tpv_config_files:
>    +          - "{{ tpv_config_dir }}/tpv_rules_local.yml"
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>           environment: local_env
>    @@ -154,6 +131,8 @@ galaxy_config_files_public:
>     galaxy_config_files:
>       - src: files/galaxy/themes.yml
>         dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>    +  - src: files/galaxy/config/tpv_rules_local.yml
>    +    dest: "{{ tpv_mutable_dir }}/tpv_rules_local.yml"
>
>     galaxy_config_templates:
>       - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV to job config"}
>
>  Note that we set the default execution environment to the tpv_dispatcher, added the tpv_dispatcher environment using the dynamic_tpv runner, and removed all other destinations.
>  Adding TPV as a runner will cause Galaxy to automatically install the `total-perspective-vortex` package on startup as a conditional dependency.
>  Finally, we added a new config file named `tpv_rules_local.yml`, which we will create next.
>
> 2. Create a new file named `tpv_rules_local.yml` in the `files/galaxy/config/` folder of your ansible playbook, so that it is copied to the config folder on the target.
>    The file should contain the following content:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -0,0 +1,32 @@
>    +tools:
>    +  .*testing.*:
>    +    cores: 2
>    +    mem: cores * 4
>    +
>    +destinations:
>    +  local_env:
>    +    runner: local_runner
>    +    max_accepted_cores: 1
>    +    params:
>    +      tmp_dir: true
>    +  singularity:
>    +    runner: local_runner
>    +    max_accepted_cores: 1
>    +    params:
>    +      singularity_enabled: true
>    +      # Enabling access to the reference data on CVMFS in the container
>    +      singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    +    env:
>    +      # Ensuring a consistent collation environment is good for reproducibility.
>    +      LC_ALL: C
>    +      # The cache directory holds the docker containers that get converted
>    +      APPTAINER_CACHEDIR: /tmp/singularity
>    +      # Singularity uses a temporary directory to build the squashfs filesystem
>    +      APPTAINER_TMPDIR: /tmp
>    +  slurm:
>    +    inherits: singularity
>    +    runner: slurm
>    +    max_accepted_cores: 16
>    +    params:
>    +      native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    +
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV local rules"}
>
>    In this TPV config, we have specified that the testing tool should use `2` cores. Memory has been defined as an expression and should be 4 times as much as cores, which, in this case, is 8GB.
>    Note that the tool id is matched via a regular expression against the full tool id. For example, a full tool id for hisat may look like: `toolshed.g2.bx.psu.edu/repos/iuc/hisat2/hisat2/2.1.0+galaxy7`
>    This enables complex matching, including matching against specific versions of tools.
>
>    Destinations must also be defined in TPV itself. Importantly, note that any destinations defined in the job conf are ignored by TPV. Therefore, we have moved all destinations from the job conf to TPV. In addition, we have removed some
>    redundancy by using the "inherits" clause in the `slurm` destination. This means that slurm will inherit all of the settings defined for singularity, but selectively override some settings. We have additionally
>    defined the `native_specification` param for SLURM, which is what SLURM uses to allocate resources per job. Note the use of the `{cores}` and `{mem}`
>    parameter within the native specification, which TPV will replace at runtime with the value of cores and memory assigned to the tool.
>
>    Finally, we have also defined a new property named `max_accepted_cores`, which is the maximum amount of cores this destination will accept. Since the testing tool requests 2 cores, but only the `slurm`
>    destination is able to accept jobs greater than 1 core, TPV will automatically route the job to the best matching destination, in this case, slurm.
>
> 3. Run the Galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. Click the rerun button on the last history item, or click **Testing Tool** in the tool panel, and then click the tool's Run Tool button.
>
>    > <question-title></question-title>
>    >
>    > What is the tool's output?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > Running with '2' threads
>    > > ```
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}


### Configuring defaults

Now that we've configured the resource requirements for a single tool, let's see how we can configure defaults for all tools, and reuse those defaults to reduce repetition.


> <hands-on-title>Configuring defaults and inheritance</hands-on-title>
>
> 1. Edit your `files/galaxy/config/tpv_rules_local.yml` and add the following settings.
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -1,4 +1,11 @@
>    +global:
>    +  default_inherits: default
>    +
>     tools:
>    +  default:
>    +    abstract: true
>    +    cores: 1
>    +    mem: cores * 4
>       .*testing.*:
>         cores: 2
>         mem: cores * 4
>    @@ -29,4 +36,3 @@ destinations:
>         max_accepted_cores: 16
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    -
>    {% endraw %}
>    ```
>    {: data-commit="Add TPV default inherits"}
>
>    We have defined a `global` section specifying that all tools and destinations should inherit from a specified `default`. We have then defined a tool named `default`, whose properties
>    are implicitly inherited by all tools at runtime. This means that our `testing` tool will also inherit from this default tool, but it explicitly overrides cores.
>    We can also explicitly specify an `inherits` clause if we wish to extend a specific tool or destination, as previously shown in the destinations section.
>
> 2. Run the Galaxy playbook. When the new `tpv_rules_local.yml` is copied, TPV will automatically pickup the changes without requiring a restart of Galaxy.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

### TPV reference documentation

Please see [TPV's dedicated documentation](https://total-perspective-vortex.readthedocs.io/en/latest/) for more information.


# Configuring the TPV shared database

The Galaxy Project maintains a [shared database of TPV rules](https://github.com/galaxyproject/tpv-shared-database) so that admins do not have to independently rediscover ideal resource allocations for specific tools. These rules are based
on settings that have worked well in the usegalaxy.* federation. The rule file can simply be imported directly, with local overrides applied on top.

> <hands-on-title>Adding the TPV shared database to job_conf</hands-on-title>
>
> 1. Edit `group_vars/galaxyservers.yml` and add the location of the TPV shared rule file to the `tpv_dispatcher` destination.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -35,6 +35,7 @@ galaxy_job_config:
>           tpv_dispatcher:
>             runner: dynamic_tpv
>             tpv_config_files:
>    +          - https://gxy.io/tpv/db.yml
>               - "{{ tpv_config_dir }}/tpv_rules_local.yml"
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>    {% endraw %}
>    ```
>    {: data-commit="Importing TPV shared database via job conf"}
>
>    Note how TPV allows the file to be imported directly via its http url. As many local and remote rule files as necessary can be combined, with rule files specified later overriding
>    any previously specified rule files. The TPV shared database does not define destinations, only cores and mem settings, as well as any required environment vars.
>    Take a look at the shared database of rules and note that some tools have very large recommended memory settings, which may or may not be available within your local cluster.
>    Nevertheless, you may still wish to execute these tools with memory adjusted to suit your cluster's capabilities. 
>
> 2. Edit your `files/galaxy/config/tpv_rules_local.yml` and make the following changes.
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -33,6 +33,9 @@ destinations:
>       slurm:
>         inherits: singularity
>         runner: slurm
>    -    max_accepted_cores: 16
>    +    max_accepted_cores: 24
>    +    max_accepted_mem: 256
>    +    max_cores: 2
>    +    max_mem: 8
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    {% endraw %}
>    ```
>    {: data-commit="TPV clamp max cores and mem"}
>
>    These changes indicate that the destination will accept jobs that are up to `max_accepted_cores: 24` and `max_accepted_mem: 256`. If the tool requests resources that exceed these limits, the tool will be rejected
>    by the destination. However, once accepted, the resources will be forcibly clamped down to 2 and 8 at most because of the `max_cores` and `max_mem` clauses. (E.g. a tool requesting 24 cores would only be submitted with 16 cores at maximum.) Therefore, a trick that can be used here to support
>    job resource requirements in the shared database that are much larger than your destination can actually support, is to combine `max_accepted_cores/mem/gpus` with `max_cores/mem/gpus` to accept the job and then
>    clamp it down to a supported range. This allows even the largest resource requirement in the shared database to be accomodated.
>
>    > <comment-title>Clamping in practice</comment-title>
>    > For the purposes of this tutorial, we've clamped down from 16 cores to 2 cores, and mem from 256 to 8, which is unlikely to work in practice. In production, you will probably need to manually test
>    > any tools that exceed your cluster's capabilities, and decide whether you want those tools to run in the first place.
>    {: .comment}
>
> 3. Run the Galaxy playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

# Basic access controls

You may wish to apply some basic restrictions on which users are allowed to run specific tools. {TPV} accomodates user and role specific rules. In addition, TPV supports tagging of tools, users, roles and destinations. These tags
can be matched up so that only desired combinations are compatible with each other. While these mechanisms are detailed in the TPV documentation, we will choose a different problem that highlights some other capabilities
- restricting a tool so that only an admin can execute that tool.

> <hands-on-title>Using conditionals to restrict a tool to admins only</hands-on-title>
>
> 1. Edit your `files/galaxy/config/tpv_rules_local.yml` and add the following rule.
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -9,6 +9,15 @@ tools:
>       .*testing.*:
>         cores: 2
>         mem: cores * 4
>    +    rules:
>    +      - id: admin_only_testing_tool
>    +        if: |
>    +          # Only allow the tool to be executed if the user is an admin
>    +          admin_users = app.config.admin_users
>    +          # last line in block must evaluate to a value - which determines whether the TPV if conditional matches or not
>    +          not user or user.email not in admin_users
>    +        fail: Unauthorized. Only admins can execute this tool.
>    +
>
>     destinations:
>       local_env:
>    {% endraw %}
>    ```
>    {: data-commit="TPV admin only tool"}
>
> Note the use of the `if` rule, which allows for conditional actions to be taken in TPV. An `if` block is evaluated as a multi-line python block, and can execute arbitrary code, but the last line of the block must evaluate to a value. That value
> determines whether the `if` condtional is matched or not. If the conditional is matched, the `fail` clause is executed in this case, and the message specified in the `fail` clause is displayed to the user.
> It is similarly possible to conditionally add job parameters, modify cores/mem/gpus and take other complex actions.
>
> 2. Run the Galaxy playbook to update the TPV rules.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Try running the tool as both an admin user and a non-admin user, non-admins should not be able to run it. You can start a private browsing session to test as a non-admin, anonymous user. Anonymous users were enabled in your Galaxy configuration.
>
{: .hands_on}

> ```bash
> 3.sh
> ```
> {: data-test="true"}
{: .hidden}

# Job Resource Selectors

Certain tools can benefit from allowing users to select appropriate job resource parameters, instead of having admins decide resource allocations beforehand. For example, a user might know that a particular set of parameters and inputs to a certain tool needs a larger memory allocation than the standard amount given to that tool.
Galaxy provides functionality to have extra form elements in the tool execution form to specify these additional job resource parameters.
This of course assumes that your users are well behaved enough not to choose the maximum whenever available, although such concerns can be mitigated somewhat by the use of concurrency limits on larger memory destinations.

Such form elements can be added to tools without modifying each tool's configuration file through the use of the **job resource parameters configuration file**

> <hands-on-title>Configuring a Resource Selector</hands-on-title>
>
> 1. Create and open `templates/galaxy/config/job_resource_params_conf.xml.j2`
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/galaxy/config/job_resource_params_conf.xml.j2
>    @@ -0,0 +1,7 @@
>    +<parameters>
>    +    <param label="Cores" name="cores" type="select" help="Number of cores to run job on.">
>    +        <option value="1">1 (default)</option>
>    +        <option value="2">2</option>
>    +    </param>
>    +  <param label="Time" name="time" type="integer" size="3" min="1" max="24" value="1" help="Maximum job time in hours, 'walltime' value (1-24). Leave blank to use default value." />
>    +</parameters>
>    {% endraw %}
>    ```
>    {: data-commit="Add job resource params configuration"}
>
>    This defines two resource fields, a select box where users can choose between 1 and 2 cores, and a text entry field where users can input an integer value from 1-24 to set the walltime for a job.
>
> 2. As usual, we need to instruct Galaxy of where to find this file:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -37,9 +37,17 @@ galaxy_job_config:
>             tpv_config_files:
>               - https://gxy.io/tpv/db.yml
>               - "{{ tpv_config_dir }}/tpv_rules_local.yml"
>    +  resources:
>    +    default: default
>    +    groups:
>    +      default: []
>    +      testing: [cores, time]
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>           environment: local_env
>    +    - id: testing
>    +      environment: tpv_dispatcher
>    +      resources: testing
>
>     galaxy_config:
>       galaxy:
>    @@ -59,6 +67,7 @@ galaxy_config:
>         object_store_store_by: uuid
>         id_secret: "{{ vault_id_secret }}"
>         job_config: "{{ galaxy_job_config }}" # Use the variable we defined above
>    +    job_resource_params_file: "{{ galaxy_config_dir }}/job_resource_params_conf.xml"
>         # SQL Performance
>         slow_query_log_threshold: 5
>         enable_per_request_sql_debugging: true
>    @@ -140,6 +149,8 @@ galaxy_config_templates:
>         dest: "{{ galaxy_config.galaxy.container_resolvers_config_file }}"
>       - src: templates/galaxy/config/dependency_resolvers_conf.xml
>         dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
>    +  - src: templates/galaxy/config/job_resource_params_conf.xml.j2
>    +    dest: "{{ galaxy_config.galaxy.job_resource_params_file }}"
>
>     galaxy_extra_dirs:
>       - /data
>    {% endraw %}
>    ```
>    {: data-commit="Configure resources in job conf"}
>
>    We have added a resources section. The group ID will be used to map a tool to job resource parameters, and the text value of the `<group>` tag is a comma-separated list of `name`s from `job_resource_params_conf.xml` to include on the form of any tool that is mapped to the defined `<group>`.
>
>    We have also listed the `testing` tool as a tool which uses resource parameters. When listed here, the resource parameter selection form will be displayed.
>
>    Finally, we have specified that the `job_resource_params_conf.xml.j2` should be copied across.
>
{: .hands_on}

This will set everything up to use the function. We have:

- A set of "job resources" defined which will let the user select the number of cores and walltime.
- A job configuration which says:
    - that our testing tool should allow selection of the cores and time parameters
    - directs it to TPV's `tpv_dispatcher` destination

This is a lot but we're still missing the last piece for it to work:

## Configuring TPV to process resource parameters

Lastly, we need to write a rule in TPV that will read the value of the job resource parameter form fields and decide how to submit the job.

> <hands-on-title>Processing job resource parameters in TPV</hands-on-title>
>
> 1. Create and edit `files/galaxy/config/tpv_rules_local.yml`. Create it with the following contents:
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -6,6 +6,8 @@ tools:
>         abstract: true
>         cores: 1
>         mem: cores * 4
>    +    params:
>    +      walltime: 8
>       .*testing.*:
>         cores: 2
>         mem: cores * 4
>    @@ -17,7 +19,13 @@ tools:
>               # last line in block must evaluate to a value - which determines whether the TPV if conditional matches or not
>               not user or user.email not in admin_users
>             fail: Unauthorized. Only admins can execute this tool.
>    -
>    +      - id: resource_params_defined
>    +        if: |
>    +          param_dict = job.get_param_values(app)
>    +          param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
>    +        cores: int(job.get_param_values(app)['__job_resource']['cores'])
>    +        params:
>    +           walltime: "{int(job.get_param_values(app)['__job_resource']['time'])}"
>
>     destinations:
>       local_env:
>    @@ -47,4 +55,4 @@ destinations:
>         max_cores: 2
>         max_mem: 8
>         params:
>    -      native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)}
>    +      native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)} --time={entity.params['walltime']}:00:00
>    {% endraw %}
>    ```
>    {: data-commit="process resource params in TPV"}
>
>    We define a conditional rule and check that the job_resource_params have in fact been defined by the user. If defined, we override the cores value with the one specified by the user. We also define a walltime param, setting it to 8 hours by default.
>    We also override walltime if the user has specified it.
>    It is important to note that **you are responsible for parameter validation, including the job resource selector**. This function only handles the job resource parameter fields, but it could do many other things - examine inputs, job queues, other tool parameters, etc.
>
>    Finally, we pass the walltime as part of the native specification.
>
>    > <comment-title>Rules for TPV code evaluation</comment-title>
>    > Notice how the `walltime` parameter is wrapped in braces, whereas the `cores` value isn't, yet they are both expressions. The reason for this difference is that when TPV evaluates an expression, all string fields (e.g. env, params) are evaluated as Python f-strings,
>    > while all non-string fields (integer fields like `cores` and `gpus`, float fields like `mem`, boolean fields like `if`) are evaluated as Python code-blocks. This rule enables the YAML file to be much more readable, but requires you to keep this simple rule in mind.
>    > Note also that Python code-blocks can be multi-line, and that the final line must evaluate to a value that can be assigned to the field.
>    {: .comment}
>
> 2. Run the Galaxy playbook to update the TPV rules.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Run the **Testing Tool** with various resource parameter selections
>
>    - Use default job resource parameters
>    - Specify job resource parameters:
>      - 1 core
>      - 2 cores
>      - Some value for walltime from 1-24
>
{: .hands_on}

The cores parameter can be verified from the output of the tool. The walltime can be verified with `scontrol`:

> <code-in-title>Bash</code-in-title>
> Your job number may be different.
> ```
> scontrol show job 24
> ```
{: .code-in}

> <code-out-title></code-out-title>
> Your output may look slightly different. Note that the `TimeLimit` for this job (which I gave a 12 hour time limit) was set to `12:00:00`.
> ```console
> JobId=24 JobName=g24_multi_anonymous_10_0_2_2
>    UserId=galaxy(999) GroupId=galaxy(999)
>    Priority=4294901747 Nice=0 Account=(null) QOS=(null)
>    JobState=COMPLETED Reason=None Dependency=(null)
>    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
>    RunTime=00:00:05 TimeLimit=12:00:00 TimeMin=N/A
>    SubmitTime=2016-11-05T22:01:09 EligibleTime=2016-11-05T22:01:09
>    StartTime=2016-11-05T22:01:09 EndTime=2016-11-05T22:01:14
>    PreemptTime=None SuspendTime=None SecsPreSuspend=0
>    Partition=debug AllocNode:Sid=gat2016:1860
>    ReqNodeList=(null) ExcNodeList=(null)
>    NodeList=localhost
>    BatchHost=localhost
>    NumNodes=1 NumCPUs=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
>    TRES=cpu=1,node=1
>    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
>    MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
>    Features=(null) Gres=(null) Reservation=(null)
>    Shared=OK Contiguous=0 Licenses=(null) Network=(null)
>    Command=(null)
>    WorkDir=/srv/galaxy/server/database/jobs/000/24
>    StdErr=/srv/galaxy/server/database/jobs/000/24/galaxy_24.e
>    StdIn=StdIn=/dev/null
>    StdOut=/srv/galaxy/server/database/jobs/000/24/galaxy_24.o
>    Power= SICP=0
> ```
{: .code-out}

{% snippet topics/admin/faqs/missed-something.md step=10 %}


# More on TPV

The goal of this tutorial is to provide a quick overview of some of the basic capabilities of TPV. However, there are numerous features that we have not convered such as:
a. Custom code blocks - execute arbitrary python code blocks, access additional context variables etc.
a. User and Role Handling - Add scheduling constraints based on the user's email or role
b. Metascheduling support - Perform advanced querying and filtering prior to choosing an appropriate destination
c. Job resubmissions - Resubmit a job if it fails for some reason
d. Linting, formatting and dry-run - Automatically format tpv rule files, catch potential syntax errors and perform a dry-run to check where a tool would get scheduled.

These features are covered in detail in the [TPV documentation](https://total-perspective-vortex.readthedocs.io/en/latest/).

## Further Reading

- The [sample dynamic tool destination config file](https://github.com/galaxyproject/galaxy/blob/dev/config/tool_destinations.yml.sample) fully describes the configuration language
- [Dynamic destination documentation](https://docs.galaxyproject.org/en/latest/admin/jobs.html)
- Job resource parameters are not as well documented as they could be, but the [sample configuration file](https://github.com/galaxyproject/usegalaxy-playbook/blob/master/env/test/files/galaxy/config/job_resource_params_conf.xml) shows some of the possibilities.
- [usegalaxy.org's job_conf.yml](https://github.com/galaxyproject/usegalaxy-playbook/blob/master/env/common/templates/galaxy/config/job_conf.yml.j2) is publicly available for reference.
- [usegalaxy.eu's job_conf.xml](https://github.com/usegalaxy-eu/infrastructure-playbook/search?l=YAML&q=galaxy_jobconf) is likewise (see the `group_vars/galaxy.yml` result)

{% snippet topics/admin/faqs/git-gat-path.md tutorial="job-destinations" %}


topic: admin
tutorial: surf-research-cloud-pulsar
title: Pulsar usage on SURF Research Cloud
difficulty: introductory
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"none\"", "type: \"none\"", "type: \"none\"", "type: \"none\""]
tags_json: []
content_hash: 768eeda092786f2f7c27ace700692541
last_modified: 2026-05-15T22:28:10.067013
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/surf-research-cloud-pulsar/tutorial.html

content:
Using Pulsar via the {SRC} allows researchers to start Pulsar instances on-demand to expand their computational resources and even access GPUs to help and analyze their data in a secure environment following the {GDPR}.

There are two main use cases we envision this role being useful for:

Saving costs on SRC
:  Maybe you're already running Galaxy in SRC, but you don't want to run a GPU node because it is very expensive. By using the SRC Pulsar Catalog Item, you can launch a node to do computations and then shut it down when you're done, saving money. Pulsar instances can be started and stopped on demand, depending on personal cases and requirements, giving you a lot of freedom!

Accessing a GPU from a local (in UMC/hospital Galaxy)
:  If you do not have a GPU easily available within your institute, it may be attractive to send jobs securely to SRC, by launching a Pulsar node in SRC and attaching it to your institute's Galaxy instance.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prerequisites

The instance provides secure authentication, where users must have a SURF Research account prior to this tutorial, have set the {SRAM} authentication method, and connect an SSH key to their accounts. In case you are not familiar with {SRC} and need help in setting up your accounts, please follow the instructions on the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/SURF+Research+Cloud)

Inside the SRC members should have access to all publicly available catalog items. If you are not able to create a catalog item, please [contact SURF servicedesk](mailto:servicedesk@surf.nl).

This tutorial assumes you are member of a {CO} in {SRAM} that has access to {SRC} and a wallet with budget in SRC with enough sources to create Galaxy and Pulsar catalog items. (For more information please refer to the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/Budgets%2C+wallets%2C+contracts).

You should have previous experience working with data inside Galaxy.

> <tip-title>SSH Access Required</tip-title>
> SSH access is required to reconfigure your Galaxy instance. Please make sure you set an SSH key in your SRAM account if you are planning to use Galaxy in SRC for this tutorial.
{: .tip}

> <hands-on-title>Log In to SRC</hands-on-title>
> 1. Log in to [SURF Research Cloud](https://portal.live.surfresearchcloud.nl/)
>
>    ![screenshot of SRC login portal with a bright yellow login button](images/surf1.png)
{: .hands_on}

# Launching Pulsar in SRC

> <hands-on-title>Launching Pulsar</hands-on-title>
> 1. In the **Workspaces Tab** on the bottom half of the screen, you'll find a **Plus Button** at right to add a new workspace
>
>    ![A small plus button is hovered over which says Add. Below galaxy-test is shown running with a yellow Access button](images/surf3.png)
>
> 2. Clicking that will let you choose any of the *Catalog Items* from SRC. They've got a wide selection but we're only interested in the two Pulsar Catalog Items
>
>    ![the two pulsar components in SRC: Galaxy Pulsar GPU Node (CUDA) and Galaxy Pulsar Node are shown. The second is expanded showing an author, Helena Rasche, and a description: let's you run galaxy jobs on another node](images/surf4.png)
>
>    > <warning-title>GPUs are Expensive</warning-title>
>    > The GPU nodes are *expensive*. In fact it
>    > was the motivating reason for building this catalog item: to enable you to
>    > launch a node, run some computations, and shut it down, saving you money.
>    {: .warning}
>
> 3. Creating a "workspace" from a catalog item (a template) is easy, most of the options are fixed for you, you just need to choose the size of the item. Pick an appropriate size for whatever computations you need to do.
>
>    ![workspace creation screen, choose the cloud provider is locked to SURF HPC, flavour is locked to 22.04, and the size options are available from 1GB/8CPU to 60C/750GB Hi-mem](images/surf5.png)
>
> 6. Pick a name, it can be anything, it does not matter. Check the expiration date to ensure it is just enough time for your computation and no more.
>
>    > <tip-title>Expiration Date</tip-title>
>    > The standard life-time of the VM is 5 days. If you need it for longer, this option can be changed once the machine is running.
>    > Note, that once the machine is expired and deleted it cannot be restored! Plan accordingly and migrate your data in time to prevent data loss!
>    >
>    > This is an incredibly useful feature as it saves you from forgetting to destroy a VM. Especially for GPU nodes it can help you ensure that they disappear after your computation is complete.
>    {:.tip}
>
>    ![almost there! some final details reads this page, asking for name and description. both have been filled out with 'pulsar'. a yellow submit button waits at the bottom](images/surf6.png)
>
> 7. Click submit when you are happy with the configuration.
>
{: .hands_on}

Once done, the workspace will be created for you. You'll need to wait ~5 minutes usually. Go for a beverage ☕️

![workspace list showing a workspace named pulsar being created.](images/surf7.png)

## Using Pulsar on SRC

> <hands-on-title>Access the Information Page</hands-on-title>
>
> 1. Once the workspace is up, you'll see an **Access** link:
>
>    ![A small plus button is hovered over which says Add. Below galaxy-test is shown running with a yellow Access button](images/surf3.png)
>
> 2. Click that will show you a Pulsar information page. This page is running on your pulsar node itself, and is restricted to ensure only authorised members can access the contents. It includes some configuration you will need to copy to your Galaxy node in order to make use of the Pulsar node.
>
>    ![pulsar configuration information page showing an about with admins and metadata like workspace fqdn. Configuration for galaxy is shown below including XML changes](images/surf8.png)
{: .hands_on}

This information page should have more than enough information to connect this Pulsar instance to your Galaxy server. You will need to reference information from this page in the following steps:

> <hands-on-title>Configuring Galaxy to use SRC Pulsar</hands-on-title>
> 1. Collect the requirements for accessing the Galaxy machine. You will need:
>
>    - your username from the first step
>    - your SSH key that is associated with your SRAM account
>
> 2. SSH into your *Galaxy* machine (not pulsar!).
>
>    ```
>    ssh -i path/to/your/sram-ssh-key username@galaxy.src-winter-school.src.surf-hosted.nl
>    ```
>
> 3. You will need to `sudo su` to do anything useful. Do that now.
> 4. `cd /srv/galaxy/` to move into the directory Galaxy configuration is stored.
> 5. The configuration is discussed fully in the Pulsar information, but it will be briefly covered here as well. Generally there are a few steps that must be followed:
>
>    - A runner must be registered
>    - A destination/environment must be added with the pulsar details
>    - Some tools should be redirected to this Pulsar
>
>    Here is an example of what those changes *might* look like in your Galaxy node.  In this example our pulsar node was called `p20` but that will be different for you.
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>
>    ```diff
>     runners:
>       local:
>         load: galaxy.jobs.runners.local:LocalJobRunner
>         workers: 4
>       condor:
>         load: galaxy.jobs.runners.condor:CondorJobRunner
>    +  pulsar:
>    +    load: galaxy.jobs.runners.pulsar:PulsarRESTJobRunner
>
>     execution:
>       default: docker_dispatch
>       environments:
>         local_destination:
>           runner: local
>         # ... probably some more environments here.
>
>    +    remote_p20:
>    +       runner: pulsar
>    +       url: https://p20.src-sensitive-i.src.surf-hosted.nl
>    +       private_token: ySgfM1rnGIsiVN8Xlfk
>    +       dependency_resolution: remote
>    +       manager: _default_
>    +       # Uncomment the following to enable interactive tools:
>    +       docker_enabled: true
>    +       docker_set_user: null
>    +       docker_memory: "8G"
>    +       singularity_enabled: false
>    +       tmp_dir: true
>    +       outputs_to_working_directory: false
>    +       container_resolvers:
>    +       - type: explicit
>    +       require_container: True
>    +       container_monitor_command: /mnt/pulsar/venv/bin/galaxy-container-monitor
>    +       container_monitor_result: callback
>    +       container_monitor_get_ip_method: command:echo p20.src-sensitive-i.src.surf-hosted.nl
>
>
>     tools:
>     - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>       environment: local_env
>     - id: Cut1
>       environment: condor_1x1
>    +- id: interactive_tool_jupyter_notebook
>    +  environment: remote_p20
>    +- id: interactive_tool_rstudio
>    +  environment: remote_p20
>    ```
{: .hands_on}

With the above, the minimal configuration is done, the Galaxy server will be aware of the Pulsar node, and two tools will be sent there: the Jupyter and RStudio Interactive Tools.

While you will simply copy-paste the runner and environment, you will need to identify yourself which tools should go to this Pulsar node.
You can find the tool ID from the dropdown at the top right, just to the left of "Execute" at the top:

![url bar and tool interface for the Cut1 tool](./images/id2.png)

> <tip-title>An Easy Configuration Option: Send Every Job to Pulsar</tip-title>
>
> If you are running jobs for a limited period of time, you might consider making this pulsar node the default destination. Remember to use the `remote_...` name of your pulsar node, based on what you copied. Not `remote_p20`.
>
> ```diff
>  execution:
> -  default: docker_dispatch
> +  default: remote_p20
>    environments:
>      local_destination:
>        runner: local
> ```
{: .tip}

With that, you're done, and for the length of time your node is running, your chosen tools (or everything) will be executed on that Pulsar node with more memory and CPU than the Galaxy host, and maybe a GPU as well!

> <hands-on-title>Launch a Job on Pulsar</hands-on-title>
> 1. Login to your Galaxy
> 2. Run one of the tools you have decided to send to Pulsar
> 3. On the pulsar machine, you can check that it runs by following the logs:
>
>    ```bash
>    sudo journalctl -fu pulsar
>    ```
>
{: .hands_on}

Congratulations on launching Pulsar in SRC! 🌌


topic: admin
tutorial: wireguard
title: Deploying Wireguard for private mesh networking
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/wireguard/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["wireguard", "networking"]
content_hash: 07b4430d9c7c033dfe08767cce2bc094
last_modified: 2026-05-15T22:28:10.075728
workflows_json: []
content:
In this tutorial we will briefly cover what [Wireguard](https://www.wireguard.com/) is and how you can leverage it for your needs. This will not make you an expert on Wireguard but will give you the tools you need in order to setup a local Wireguard network.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# What is Wireguard?

Wireguard is a VPN like OpenVPN or IPSec, but instead of the hub and spoke model of those where all traffic must go through a central node, Wireguard creates a mesh network where machines can all talk individually to each other. Wireguard also uses modern encryption only, ensuring your data stays safe.

## Is it right for me?

If you have machines that need to talk to each other privately, and you don't
have a better way to do it like a local network team, then yes, it's a great
solution to private, secure, fast networking.
It has [excellent performance](https://www.wireguard.com/performance/)
despite the encryption, and is built directly into the kernel.

By using wireguard, you can let services listen only on the wireguard interface, and thus only known and trusted machines can access those services.

# Setting up the infrastructure

> <hands-on-title>Configuration files</hands-on-title>
>
> 1. Create a `ansible.cfg` file (next to your playbook) to [configure settings](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html) like the inventory file (and save ourselves some typing!), or the Python interpreter to use:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/ansible.cfg
>    @@ -0,0 +1,6 @@
>    +[defaults]
>    +interpreter_python = /usr/bin/python3
>    +inventory = hosts
>    +retry_files_enabled = false
>    +[ssh_connection]
>    +pipelining = true
>    {% endraw %}
>    ```
>    {: data-commit="Add ansible.cfg"}
>
>    > <tip-title>CentOS7</tip-title>
>    > As mentioned in the "Ubuntu or Debian, CentOS or RHEL?" comment above, if you are using CentOS7 do not set `interpreter_python` in `ansible.cfg` .
>    {: .tip}
>
>    Pipelining will make [Ansible run faster](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html#ansible-pipelining) by significantly reducing the number of new SSH connections that must be opened.
>
> 2. Create the `hosts` inventory file if you have not done so yet, defining an `[wireguard]` group with every host you want to be part of the cluster. For each machine, also set a variable `wireguard_ip` with an address from `192.168.0.0/24`
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat hosts
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > Your hostname is probably different:
>    > >
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,5 @@
>    > > +[wireguard]
>    > > +1-wg.galaxy.training wireguard_ip=192.168.0.1
>    > > +2-wg.galaxy.training wireguard_ip=192.168.0.2
>    > > +3-wg.galaxy.training wireguard_ip=192.168.0.3
>    > > +4-wg.galaxy.training wireguard_ip=192.168.0.4
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add hosts"}
>    > {: .code-out}
>    {: .code-2col}
{: .hands_on}

Wireguard can use any of the [private network](https://en.wikipedia.org/wiki/Private_network) blocks, here we use `192.168.0.0/16` for familiarity, Tailscale uses `100.0.0.0/8`, which is the [Carrier Grade NAT](https://en.wikipedia.org/wiki/Carrier-grade_NAT) range, and behaves somewhat differently to classic Private Network prefixes.

## Writing the playbook

First lets set up the playbook and install Wireguard, without configuring it.

> <hands-on-title>Installing Wireguard</hands-on-title>
>
> 1. Create and open `wg.yml` which will be our playbook. Add the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/wg.yml
>    @@ -0,0 +1,16 @@
>    +---
>    +- hosts: all
>    +  become: yes
>    +  vars:
>    +    wireguard_mask_bits: 24
>    +    wireguard_port: 51871
>    +  tasks:
>    +    - name: update packages
>    +      apt:
>    +        update_cache: yes
>    +        cache_valid_time: 3600
>    +
>    +    - name: Install wireguard
>    +      apt:
>    +        name: wireguard
>    +        state: present
>    {% endraw %}
>    ```
>    {: data-commit="Add initial playbook, install wireguard"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook wg.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
{:.hands_on}

Now we can start configuring it. Wireguard relies on each node having a private/public keypair which is used for encrypting communications between the nodes. By default this uses modern elliptic cryptography which is quite simple and provably secure, wireguard only has ~4k LOC compared to 400k+ LOC for most VPN solutions. Let's generate those keys now:

> <hands-on-title>Generating a keypair</hands-on-title>
>
> 1. Edit `wg.yml` and add
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -14,3 +14,8 @@
>           apt:
>             name: wireguard
>             state: present
>    +
>    +    - name: Generate Wireguard keypair
>    +      shell: wg genkey | tee /etc/wireguard/privatekey | wg pubkey | tee /etc/wireguard/publickey
>    +      args:
>    +        creates: /etc/wireguard/privatekey
>    {% endraw %}
>    ```
>    {: data-commit="Generate the keys"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook wg.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Go check out what the keys look like!
{:.hands_on}

Ok, that's got wireguard setup, but it still isn't running. As of 2018 or so, systemd added built in support for wireguard and we'll use that to setup the network devices, and the network itself.

> <hands-on-title>Setting up the network</hands-on-title>
>
> 1. Edit `wg.yml` and add the following. We want to register the contents of these keys, as they'll be used to configure our networks later.
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -19,3 +19,13 @@
>           shell: wg genkey | tee /etc/wireguard/privatekey | wg pubkey | tee /etc/wireguard/publickey
>           args:
>             creates: /etc/wireguard/privatekey
>    +
>    +    - name: register private key
>    +      shell: cat /etc/wireguard/privatekey
>    +      register: wireguard_private_key
>    +      changed_when: false
>    +
>    +    - name: register public key
>    +      shell: cat /etc/wireguard/publickey
>    +      register: wireguard_public_key
>    +      changed_when: false
>    {% endraw %}
>    ```
>    {: data-commit="Register the keys"}
>
> 2. Again editing `wg.yml`, we'll setup the network device:
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -29,3 +29,28 @@
>           shell: cat /etc/wireguard/publickey
>           register: wireguard_public_key
>           changed_when: false
>    +
>    +    - name: Setup wg0 device
>    +      copy:
>    +        content: |
>    +          [NetDev]
>    +          Name=wg0
>    +          Kind=wireguard
>    +          Description=WireGuard tunnel wg0
>    +          [WireGuard]
>    +          ListenPort={{ wireguard_port }}
>    +          PrivateKey={{ wireguard_private_key.stdout }}
>    +          {% for peer in groups['all'] %}
>    +          {% if peer != inventory_hostname %}
>    +          [WireGuardPeer]
>    +          PublicKey={{ hostvars[peer].wireguard_public_key.stdout }}
>    +          AllowedIPs={{ hostvars[peer].wireguard_ip }}/32
>    +          Endpoint={{ hostvars[peer].inventory_hostname }}:{{ wireguard_port }}
>    +          PersistentKeepalive=25
>    +          {% endif %}
>    +          {% endfor %}
>    +        dest: /etc/systemd/network/99-wg0.netdev
>    +        owner: root
>    +        group: systemd-network
>    +        mode: 0640
>    +      notify: systemd network restart
>    {% endraw %}
>    ```
>    {: data-commit="Setup the netdev"}
>
>    Here we do a number of things:
>    - We configure a [NetDev](https://freedesktop.org/software/systemd/man/systemd.netdev.html), a virtual network device. Systemd supports many types but we'll use wireguard.
>    - Next we setup wireguard, specifying a `ListenPort` which is used for all wireguard communication, and specifies our PrivateKey which is used by the tunnel.
>    - For every peer (all other instances than ourselves), we setup a `WireGuardPeer` with that peer's public key, and which IPs are allowed to connect.
>
> 2. Again editing `wg.yml`, we'll setup the final bit, the network service.
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -54,3 +54,16 @@
>             group: systemd-network
>             mode: 0640
>           notify: systemd network restart
>    +
>    +    - name: Setup wg0 network
>    +      copy:
>    +        content: |
>    +          [Match]
>    +          Name=wg0
>    +          [Network]
>    +          Address={{ wireguard_ip }}/{{ wireguard_mask_bits }}
>    +        dest: /etc/systemd/network/99-wg0.network
>    +        owner: root
>    +        group: systemd-network
>    +        mode: 0640
>    +      notify: systemd network restart
>    {% endraw %}
>    ```
>    {: data-commit="Setup the network"}
>
>    Here we just setup the network, declare the interface name, and the address for it.
>
> 2. And last let's restart the appropriate services.
>
>    {% raw %}
>    ```diff
>    --- a/wg.yml
>    +++ b/wg.yml
>    @@ -67,3 +67,10 @@
>             group: systemd-network
>             mode: 0640
>           notify: systemd network restart
>    +
>    +  handlers:
>    +    - name: systemd network restart
>    +      service:
>    +        name: systemd-networkd
>    +        state: restarted
>    +        enabled: yes
>    {% endraw %}
>    ```
>    {: data-commit="Restart the network"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook wg.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Go check out the network! Try pinging each of the wireguard IPs to see if you can reach each of the machines.
{:.hands_on}


topic: admin
tutorial: reference-genomes
title: Reference Data with Data Managers
difficulty: intermediate
time_estimation: 1h
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 9da3c160bc82373f6a5838617fcc49db
last_modified: 2026-05-27T09:01:47.638245
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/reference-genomes/tutorial.html

content:
# Overview

## The problem

The Galaxy server administrator needed to know how to update each type of reference data, how to run the indexers, where to get the data from, and how to update Galaxy's *Tool Data Table* and *location* configuration files.

## Data managers to the rescue

Data Managers are a special class of Galaxy tool which allows for the download and/or creation of data that is stored within Tool Data Tables and their underlying flat location (e.g. .loc) files. These tools handle the creation of indices and the addition of entries/lines to the data table / .loc file via the Galaxy admin interface.

Data Managers can be defined locally in the data manager and tool data table configuration files or installed through the Tool Shed. When Data Managers are installed from the Tool Shed, their configuration is added to the shed versions of the data manager configuration and tool data table configuration files.

They are a flexible framework for adding reference data to Galaxy (not just genomic data). They are workflow compatible and can run via the Galaxy API. Data managers automatically update the appropriate location files when new data are installed.

For more background details on data managers, check out:
- [The data managers documentation on the hub](https://galaxyproject.org/admin/tools/data-managers/)
- [How to define a data manager for a tool](https://galaxyproject.org/admin/tools/data-managers/how-to/define/)
- There is also discussion of this topic in [the Reference Genomes in Galaxy slides]({% link topics/admin/tutorials/reference-genomes/slides.html %})

> <comment-title>Pre-built data are available</comment-title>
>
> The usegalaxy.\* servers and Galaxy Community have a large amount of reference data online and available for use by your Galaxy server. For instructions on how to access and use these data, see [the Reference Data with CVMFS tutorial]({% link topics/admin/tutorials/cvmfs/tutorial.md %}).
>
> If your data are not available as part of the CVMFS repository, Galaxy Data Managers can be used to locally install and build reference data.
>
{: .comment}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="reference-genomes" %}

# Galaxy Reference Data Components

As discussed in the overview, Galaxy Data Managers handle the population of numerous Galaxy components. From the highest level to the lowest:

- Addition of tool data table entries into the shed version of the tool data table configuration file, `shed_tool_data_table_conf.xml`. Multiple data tables with the same `name` attribute are automatically merged by Galaxy.
- Population of the corresponding location (`.loc`) file when new reference data are installed. The `.loc` file used corresponds to the Data Manager and its specific version that was executed to install the data. The path to the location file can be found in the Data Manager's entry in the tool data table configuration file (above).
- Generation of the reference data on disk, underneath the directory specified by `tool_data_path` in `galaxy.yml`.

# Installing and Using Data Managers with the Galaxy UI

## Install a the Fetch Genome Data Manager from the Tool Shed

> <comment-title>Galaxy Configuration</comment-title>
>
> No special configuration for Galaxy beyond creating an admin user is necessary in order to use Data Managers. However, you may want to be aware of the `enable_data_manager_user_view` option, which enables unprivileged user to browse data installed via Data Managers. This option is not required, and users will be able to *use* data installed via Data Managers regardless.
>
> Data are installed in the path specified by `tool_data_path` in `galaxy.yml`. If you prefer to keep hand-managed reference data separate from DM-managed reference data, you can set `galaxy_data_manager_data_path` instead.
>
{: .comment}

> <tip-title>Intall with Ephemeris</tip-title>
> This hands-on exercise installs data managers and reference data through the Galaxy UI, but you are encouraged to install tools in a deterministic, recordable way through the use of Ephemeris, which is described in [the Installing and Using Data Managers from the command line with Ephemeris section](#installing-and-using-data-managers-from-the-command-line-with-ephemeris) below.
{: .tip}

We will install a data manager that can fetch the various genome sequences from multiple sources.

> <hands-on-title>Install the Fetch Genome Data Manager</hands-on-title>
>
> 1. Access the **Admin** menu from the top bar (you need to be logged-in with an email specified in the `admin_users` setting)
> 2. Click **Install and Uninstall**, which can be found on the left, under **Tool Management**
> 3. Enter `fetch_genome` in the search interface
> 4. Click on the first hit, having `devteam` as owner
> 5. Click the **Install** button for the latest revision
{: .hands_on}

View in the file system where the various elements land. Have a look in the configuration files located in config directory.

> <question-title></question-title>
>
> What did this tool installation change?
>
> > <solution-title></solution-title>
> >
> > - The data manager and its data tables are added to the Galaxy-managed "shed" versions of the data manager config (`/srv/galaxy/var/config/shed_data_manager_conf.xml`) and data table config (`/srv/galaxy/var/config/shed_tool_data_table_conf.xml`)
> > - The data manager tool is installed along side other Galaxy tools in the shed tools directory
> >
> > > <code-in-title>Bash</code-in-title>
> > > Let's investigate the data manager config file.
> > > ```bash
> > > cat /srv/galaxy/var/config/shed_data_manager_conf.xml
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```xml
> > > <data_managers><data_manager id="fetch_genome_all_fasta_dbkeys" guid="toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager/fetch_genome_all_fasta_dbkeys/0.0.1" shed_conf_file="/srv/galaxy/var/config/shed_tool_conf.xml">
> > >         <tool file="toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/data_manager_fetch_genome_dbkeys_all_fasta/data_manager/data_manager_fetch_genome_all_fasta_dbkeys.xml" guid="toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4"><tool_shed>toolshed.g2.bx.psu.edu</tool_shed><repository_name>data_manager_fetch_genome_dbkeys_all_fasta</repository_name><repository_owner>devteam</repository_owner><installed_changeset_revision>4d3eff1bc421</installed_changeset_revision><id>toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4</id><version>0.0.4</version></tool><data_table name="all_fasta">
> > >             <output>
> > >                 <column name="value"/>
> > >                 <column name="dbkey"/>
> > >                 <column name="name"/>
> > >                 <column name="path" output_ref="out_file">
> > >                     <move type="file">
> > >                         <source>${path}</source>
> > >                         <target base="${GALAXY_DATA_MANAGER_DATA_PATH}">${dbkey}/seq/${path}</target>
> > >                     </move>
> > >                     <value_translation>${GALAXY_DATA_MANAGER_DATA_PATH}/${dbkey}/seq/${path}</value_translation>
> > >                     <value_translation type="function">abspath</value_translation>
> > >                 </column>
> > >             </output>
> > >         </data_table>
> > >         <data_table name="__dbkeys__">
> > >             <output>
> > >                 <column name="value"/>
> > >                 <column name="name"/>
> > >                 <column name="len_path" output_ref="out_file">
> > >                     <move type="file">
> > >                         <source>${len_path}</source>
> > >                         <target base="${GALAXY_DATA_MANAGER_DATA_PATH}">${value}/len/${len_path}</target>
> > >                     </move>
> > >                     <value_translation>${GALAXY_DATA_MANAGER_DATA_PATH}/${value}/len/${len_path}</value_translation>
> > >                     <value_translation type="function">abspath</value_translation>
> > >                 </column>
> > >             </output>
> > >         </data_table>
> > >     </data_manager>
> > > ```
> > {: .code-out.code-max-300}
> >
> > > <code-in-title>Bash</code-in-title>
> > > Let's also investigate the tool data table config file.
> > > ```bash
> > > cat /srv/galaxy/var/config/shed_tool_data_table_conf.xml
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```xml
> > > <?xml version="1.0" ?>
> > > <tables>
> > >     <table name="all_fasta" comment_char="#">
> > >         <columns>value, dbkey, name, path</columns>
> > >         <file path="/srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc"/>
> > >         <tool_shed_repository>
> > >             <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
> > >             <repository_name>data_manager_fetch_genome_dbkeys_all_fasta</repository_name>
> > >             <repository_owner>devteam</repository_owner>
> > >             <installed_changeset_revision>4d3eff1bc421</installed_changeset_revision>
> > >         </tool_shed_repository>
> > >     </table>
> > >     <table name="__dbkeys__" comment_char="#">
> > >         <columns>value, name, len_path</columns>
> > >         <file path="/srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/dbkeys.loc"/>
> > >         <tool_shed_repository>
> > >             <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
> > >             <repository_name>data_manager_fetch_genome_dbkeys_all_fasta</repository_name>
> > >             <repository_owner>devteam</repository_owner>
> > >             <installed_changeset_revision>4d3eff1bc421</installed_changeset_revision>
> > >         </tool_shed_repository>
> > >     </table>
> > > </tables>
> > > ```
> > {: .code-out.code-max-300}
> >
> {: .solution }
>
{: .question}

## Download and install a reference genome sequence

Next, we will install some reference data. Specifically, we will grab sacCer2 (version 2 of the *Saccharomyces cerevisiae* genome).

> <hands-on-title>Download and install sacCer2</hands-on-title>
>
> 1. Access the Admin menu from the top bar
> 2. [Click **Local Data**](https://my.gat.galaxy.training/?path=/admin/data_manager), which can be found on the left, under **Server**
>
>    You should see something like this:
>
>    ![nearly empty data manager tool list in Galaxy](../../images/dm-admin-local-data.png)
>
>    If you instead see the message "You do not currently have any Data Managers installed," you may need to restart your Galaxy server.
>
> 3. [Click **all_fasta**](https://my.gat.galaxy.training/?path=/admin/data_manager/table/all_fasta) under **View Tool Data Table Entries**
>
>    You should see the current contents of `tool-data/all_fasta.loc`, which will be empty.
>
> 4. Click **Local Data** to return to the previous page
>
> 5. Click {% tool [Create DBKey and Reference Genome - fetching](toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4) %} under **Installed Data Managers**.
>    - {% icon param-select %} *"Use existing dbkey or create a new one"*: `Existing`
>    - {% icon param-select %} *"DBKEY to assign to data"*: `sacCer2`
>    - {% icon param-text %} *"Name of sequence"*: `S. cerevisiae June 2008 (SGD/sacCer2)`
>    - {% icon param-select %} *"Choose the source for the reference genome"*: `UCSC`
>
>    > <details-title>What do these fields mean?</details-title>
>    >
>    > The data manager tool form fields correspond to the columns in the tool data table, as referenced in `shed_tool_data_table_conf.xml`.
>    >
>    > Tool Field                | Tool Data Table Entry         | Purpose
>    > ---                       | -----                         | ---
>    > *ID for sequence*         | `value` column in `all_fasta` | Identifier of specific build of the genome, if it differs from the DBKEY. Used for variant builds such as the `hg19female` build of `hg19`.
>    > *DBKEY to assign to data* | `dbkey` column in `all_fasta` | Identifier of the genome, the UCSC build ID for UCSC builds like `hg19`.
>    > *Name of sequence*        | `name` column in `all_fasta`  | Display name of the genome, displayed in Galaxy tool dropdowns.
>    {: .details}
>
> 5. Click **Execute**. In your history, you will see a new dataset for the data manager run. When the job has finished, go back to the Data Manager view on the Galaxy Admin page (Click **Local Data**).
> 6. Click **all_fasta** under *View Tool Data Table Entries*
>
>    You should see that *sacCer2* has been added to all_fasta.
>
>    ![populated all_fasta data table which now includes sacCer2 in the genome list](../../images/dm-all-fasta-populated.png)
>
{: .hands_on}

View in the file system where the changes to the location file and the reference genome fasta file.

> <question-title></question-title>
>
> What did this data manager execution change?
>
> > <solution-title></solution-title>
> >
> > - An entry for the *sacCer2* genome has been added to the `all_fasta.loc` file for the `data_manager_fetch_genome_all_fasta_dbkey` data manager. The path to the location file can be found in "shed" versions of the data manager config (`/srv/galaxy/var/config/shed_data_manager_conf.xml`) and data table config (`/srv/galaxy/var/config/shed_tool_data_table_conf.xml`)
> > - The data manager tool is installed along side other Galaxy tools in the shed tools directory
> >
> > > <code-in-title>Bash</code-in-title>
> > > Get the path to the `all_fasta.loc` file:
> > > ```bash
> > > grep 'data_manager_fetch_genome_dbkeys_all_fasta.*all_fasta.loc' /srv/galaxy/var/config/shed_tool_data_table_conf.xml
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```xml
> > >         <file path="/srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc"/>
> > > ```
> > {: .code-out}
> >
> > > <code-in-title>Bash</code-in-title>
> > > Let's investigate the contents of the location file identified above (the path to yours may be different if you installed a different version of the data manager):
> > > ```bash
> > > cat /srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```bash
> > > #This file lists the locations and dbkeys of all the fasta files
> > > #under the "genome" directory (a directory that contains a directory
> > > #for each build). The script extract_fasta.py will generate the file
> > > #all_fasta.loc. This file has the format (white space characters are
> > > #TAB characters):
> > > #
> > > #<unique_build_id>  <dbkey>     <display_name>  <file_path>
> > > #
> > > #So, all_fasta.loc could look something like this:
> > > #
> > > #apiMel3    apiMel3 Honeybee (Apis mellifera): apiMel3      /path/to/genome/apiMel3/apiMel3.fa
> > > #hg19canon  hg19        Human (Homo sapiens): hg19 Canonical        /path/to/genome/hg19/hg19canon.fa
> > > #hg19full   hg19        Human (Homo sapiens): hg19 Full         /path/to/genome/hg19/hg19full.fa
> > > #
> > > #Your all_fasta.loc file should contain an entry for each individual
> > > #fasta file. So there will be multiple fasta files for each build,
> > > #such as with hg19 above.
> > > #
> > > sacCer2	sacCer2	S. cerevisiae June 2008 (SGD/sacCer2)	/srv/galaxy/var/tool-data/sacCer2/seq/sacCer2.fa
> > > ```
> > {: .code-out}
> >
> > > <code-in-title>Bash</code-in-title>
> > > Finally, verify that we have some sequences in the genome fasta file:
> > > ```bash
> > > grep '^>' /srv/galaxy/var/tool-data/sacCer2/seq/sacCer2.fa
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```
> > > >2micron
> > > >chrI
> > > >chrII
> > > >chrIII
> > > >chrIV
> > > >chrIX
> > > >chrM
> > > >chrV
> > > >chrVI
> > > >chrVII
> > > >chrVIII
> > > >chrX
> > > >chrXI
> > > >chrXII
> > > >chrXIII
> > > >chrXIV
> > > >chrXV
> > > >chrXVI
> > > ```
> > {: .code-out}
> >
> {: .solution }
>
{: .question}

## Download and install the BWA data manager

Having the genome is a prerequisite for our ultimate goal, which is to use the sacCer2 genome as a *reference genome* for the BWA tool. BWA, like many tools, needs an *index* of the reference genome, and has its own format for that index. Thankfully, the BWA/BWA-MEM data manager will build that index for us.

In this part we will repeat the same process as when we installed the Fetch Genome data manager, except that we will install the BWA/BWA-MEM data manager this time.

> <hands-on-title>Install the BWA/BWA-MEM Data Manager </hands-on-title>
>
> 1. Access the **Admin** menu from the top bar
> 2. Click **Install and Uninstall**, which can be found on the left, under **Tool Management**
> 3. Enter `bwa_mem_index` in the search interface
> 4. Click on the first hit, having `devteam` as owner
> 5. Click the **Install** button for the latest revision
{: .hands_on}

## Build the BWA index for a reference genome

In this part we will actually build the BWA index for sacCer2. It will automatically be added to our list of available reference genomes in the BWA tool.

> <hands-on-title>Build the sacCer2 BWA index</hands-on-title>
>
> 1. Access the Admin menu from the top bar
> 2. Click **Local Data**, which can be found on the left, under **Server**
> 3. Click **{% tool [BWA-MEM index - builder](toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/bwa_mem_index_builder_data_manager/0.0.5) %}** under **Installed Data Managers**.
>    - {% icon param-select %} *"Source Fasta Sequence"*: `S. cerevisiae June 2008 (SGD/sacCer2)`
> 4. Click **Execute**.
> 5. Verify that the new BWA index for sacCer2 has been built and the .loc file has been filled in. From the **Local Data** page in the Admin section, click on **bwa_mem_indexes** under *View Tool Data Table Entries*
>
>    *S. cerevisiae* sacCer2 should now appear in the list!
{: .hands_on}

> <question-title></question-title>
>
> What changes were made by the **BWA-MEM index builder**?
>
> > <solution-title></solution-title>
> >
> > - An entry for the *sacCer2* BWA/BWA-MEM index has been added to the `bwa_mem_index.loc` file for the `data_manager_bwa_mem_index_builder` data manager.
> > - A symlink to the sacCer2 genome fasta file installed by the Fetch Genome data manager above was created in the sacCer2 BWA index directory, `/srv/galaxy/var/tool-data/sacCer2/bwa_mem_index/sacCer2/`.
> > - The BWA/BWA-MEM index for the sacCer2 genome has been built.
> >
> > > <code-in-title>Bash</code-in-title>
> > >
> > > ```bash
> > > ls -l /srv/galaxy/var/tool-data/sacCer2/bwa_mem_index/sacCer2
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Bash</code-out-title>
> > > ```console
> > > total 20800
> > > lrwxrwxrwx 1 galaxy galaxy       20 Dec  9 18:29 sacCer2.fa -> ../../seq/sacCer2.fa
> > > -rw-r--r-- 1 galaxy galaxy       14 Dec  9 18:29 sacCer2.fa.amb
> > > -rw-r--r-- 1 galaxy galaxy      591 Dec  9 18:29 sacCer2.fa.ann
> > > -rw-r--r-- 1 galaxy galaxy 12163076 Dec  9 18:29 sacCer2.fa.bwt
> > > -rw-r--r-- 1 galaxy galaxy  3040750 Dec  9 18:29 sacCer2.fa.pac
> > > -rw-r--r-- 1 galaxy galaxy  6081552 Dec  9 18:29 sacCer2.fa.sa
> > > ```
> > {: .code-out}
> >
> {: .solution }
>
{: .question}

# Installing and Using Data Managers from the command line with Ephemeris

The same process described in the previous section can also be performed from the command line, e.g. in a CI/CD
pipeline, using Ephemeris. For a more in-depth look at Ephemeris, especially in the tool installation context, please see [the Galaxy Tool Management with Ephemeris tutorial]({% link topics/admin/tutorials/tool-management/tutorial.md %}).

In order to accomplish this, you will need:

- The URL of your Galaxy server
- The API key for your account, which must be an admin

{% snippet faqs/galaxy/preferences_admin_api_key.md admin=true %}

## Install a the Fetch Genome Data Manager from the Tool Shed

> <hands-on-title>Install the Fetch Genome Data Manager with Ephemeris</hands-on-title>
>
> 1. Re-activate the virtualenv you created for the [ephemeris tool management tutorial]({% link topics/admin/tutorials/tool-management/tutorial.md %}).
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > . ~/ephemeris_venv/bin/activate
>    > ```
>    {: .code-in}
>
>    > <tip-title>missing?</tip-title>
>    >
>    > then you might need to re-run the steps:
>    >
>    > ```bash
>    > python3 -m venv ~/ephemeris_venv
>    > . ~/ephemeris_venv/bin/activate
>    > pip install ephemeris
>    > ```
>    {: .tip}
>
> 2. Install the `data_manager_fetch_genome_dbkeys_all_fasta` data manager tool owned by `devteam`.
>
>    > <code-in-title>Bash</code-in-title>
>    > Be sure to adjust the value of `-g` appropriately for your Galaxy server, and replace the value of `-a` with your API key.
>    > ```console
>    > shed-tools install -g https://galaxy.example.org -a <api-key> --name data_manager_fetch_genome_dbkeys_all_fasta --owner devteam
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_x9xeu8ro
>    > (1/1) Installing repository data_manager_fetch_genome_dbkeys_all_fasta from devteam to section "None" at revision 4d3eff1bc421 (TRT: 0:00:00.401143)
>    > 	repository data_manager_fetch_genome_dbkeys_all_fasta installed successfully (in 0:00:25.530604) at revision 4d3eff1bc421
>    > Installed repositories (1): [('data_manager_fetch_genome_dbkeys_all_fasta', '4d3eff1bc421')]
>    > Skipped repositories (0): []
>    > Errored repositories (0): []
>    > All repositories have been installed.
>    > Total run time: 0:00:25.932659
>    > ```
>    {: .code-out}
>
{: .hands-on}

## Download and install a reference genome sequence

> <hands-on-title>Download and install sacCer3 with Ephemeris</hands-on-title>
>
> 1. Create a config file for `run-data-managers` named `fetch-sacCer3.yml`:
>
>    {% raw %}
>    ```yaml
>    data_managers:
>      - id: toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4
>        params:
>          - 'dbkey_source|dbkey': '{{ item.dbkey }}'
>          - 'sequence_name': '{{ item.name }}'
>          - 'reference_source|reference_source_selector': 'ucsc'
>          - 'reference_source|requested_dbkey': '{{ item.dbkey }}'
>        items:
>          - dbkey: sacCer3
>            name: 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)'
>        data_table_reload:
>          - all_fasta
>          - __dbkeys__
>    ```
>    {% endraw %}
>
>    > <details-title>What do these config file compontents mean?</details-title>
>    >
>    > The `run-data-managers` config file options correspond to the options in the data manager tool XML file. To locate the tool XML file for the Fetch Genomes data manager, you can search for it in [the Tool Shed](https://toolshed.g2.bx.psu.edu) the same way you did when installing it via the UI. You can also open the data manager's tool form in the UI as if to run it, and, using the drop-down menu at the top right of the tool form, click "See in Tool Shed". From the Tool Shed, you can click the **Development repository** link and browse to the [tool XML file, `data_manager/data_manager_fetch_genome_all_fasta_dbkeys.xml`](https://github.com/galaxyproject/tools-iuc/blob/8487d2c73793be0afa5b34388b122e686ac8a094/data_managers/data_manager_fetch_genome_dbkeys_all_fasta/data_manager/data_manager_fetch_genome_all_fasta_dbkeys.xml).
>    >
>    > `run-data-managers` config file component          | Purpose
>    > ---                                                | ---
>    > `id`                                               | Data manager full (shed) tool ID, this can be found in `shed_data_manager_conf.xml`
>    > `params`                                           | Data manager tool params, these correspond to `<param>` tags in the tool XML file. Nested paramaters are specified using a pipe character (`|`).
>    > param `dbkey_source|dbkey`                         | Value of `<param name="dbkey" ...>` in `<conditional name="dbkey_source">`.
>    > param `sequence_name`                              | Value of `<param name="sequence_name" ...>`.
>    > param `reference_source|reference_source_selector` | Value of `<param name="reference_source_selector" ...>` in `<conditional name="reference_source">`.
>    > param `reference_source|requested_dbkey`           | Value of `<param name="requested_dbkey" ...>` in `<conditional name="reference_source">`.
>    > `items`                                            | A list of variables to template in to `params`, referenced in param fields with {% raw %}`{{ item }}`{% endraw %}. In the case of genomes, for example, you can run this DM with multiple genomes, or you could give multiple URLs.
>    > `data_table_reload`                                | Names of the data tables you want to reload after your DMs are finished running. This can be important for subsequent data managers.
>    >
>    {: .details}
>
>
> 2. Run the Genome Fetch DM with `run-data-managers`:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```console
>    > run-data-managers -g https://galaxy.example.org -a <api-key> --config fetch-sacCer3.yml
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_f6klyy7v
>    > Running data managers that populate the following source data tables: ['all_fasta']
>    > Dispatched job 1. Running DM: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4" with parameters: {'dbkey_source|dbkey': 'sacCer3', 'dbkey_source|dbkey_name': 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)', 'reference_source|reference_source_selector': 'ucsc', 'reference_source|requested_dbkey': 'sacCer3'}
>    > Job 1 finished with state ok.
>    > Running data managers that index sequences.
>    > Finished running data managers. Results:
>    > Successful jobs: 1
>    > Skipped jobs: 0
>    > Failed jobs: 0
>    > ```
>    {: .code-out}
>
> 3. In the Galaxy UI, access the Admin menu from the top bar
> 4. Click **Local Data**, which can be found on the left, under **Server**
> 5. Click **all_fasta** under *View Tool Data Table Entries*
>
>    You should see that *sacCer3* has been added to all_fasta.
>
>    ![populated all_fasta data table with sacCer3](../../images/dm-all-fasta-populated-2.png)
>
>    > <question-title></question-title>
>    >
>    > Can we view the changes from the command line? Where are they located?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The `all_fasta.loc` file contains the entry for our new genome. If you need a reminder on how to locate it, see the *What did this data manager execution change?* question in the UI section above.
>    > >
>    > > > <code-in-title>Bash</code-in-title>
>    > > > ```bash
>    > > > cat /srv/galaxy/var/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/4d3eff1bc421/all_fasta.loc
>    > > > ```
>    > > {: .code-in}
>    > >
>    > > > <code-out-title>Bash</code-out-title>
>    > > > ```bash
>    > > > #This file lists the locations and dbkeys of all the fasta files
>    > > > #under the "genome" directory (a directory that contains a directory
>    > > > #for each build). The script extract_fasta.py will generate the file
>    > > > #all_fasta.loc. This file has the format (white space characters are
>    > > > #TAB characters):
>    > > > #
>    > > > #<unique_build_id>  <dbkey>     <display_name>  <file_path>
>    > > > #
>    > > > #So, all_fasta.loc could look something like this:
>    > > > #
>    > > > #apiMel3    apiMel3 Honeybee (Apis mellifera): apiMel3      /path/to/genome/apiMel3/apiMel3.fa
>    > > > #hg19canon  hg19        Human (Homo sapiens): hg19 Canonical        /path/to/genome/hg19/hg19canon.fa
>    > > > #hg19full   hg19        Human (Homo sapiens): hg19 Full         /path/to/genome/hg19/hg19full.fa
>    > > > #
>    > > > #Your all_fasta.loc file should contain an entry for each individual
>    > > > #fasta file. So there will be multiple fasta files for each build,
>    > > > #such as with hg19 above.
>    > > > #
>    > > > sacCer2	sacCer2	S. cerevisiae June 2008 (SGD/sacCer2)	/srv/galaxy/var/tool-data/sacCer2/seq/sacCer2.fa
>    > > > sacCer3	sacCer3	S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)	/srv/galaxy/var/tool-data/sacCer3/seq/sacCer3.fa
>    > > > ```
>    > > {: .code-out}
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands-on}

> <warning-title>run-data-managers is not idempotent!</warning-title>
> Unlike `shed-tools install`, the Ephemeris `run-data-managers` utility is not idempotent. If run a second time on the same set of inputs, you will end up with two entries in your `all_fasta` data table, with the data from the second run overwriting the data from the first run.
>
> Please see [Galaxy issue #15188](https://github.com/galaxyproject/galaxy/issues/15188) for details.
>
{: .warning}

## Download and install the BWA data manager

> <hands-on-title>Install the BWA/BWA-MEM Data Manager </hands-on-title>
>
> 1. Install the `bwa_mem_index_builder_data_manager` data manager tool owned by `devteam`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```console
>    > shed-tools install -g https://galaxy.example.org -a <api-key> --name data_manager_bwa_mem_index_builder --owner devteam
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_2dyujjvi
>    > (1/1) Installing repository data_manager_bwa_mem_index_builder from devteam to section "None" at revision 63d5652be07a (TRT: 0:00:00.204350)
>    > 	repository data_manager_bwa_mem_index_builder installed successfully (in 0:00:05.599382) at revision 63d5652be07a
>    > Installed repositories (1): [('data_manager_bwa_mem_index_builder', '63d5652be07a')]
>    > Skipped repositories (0): []
>    > Errored repositories (0): []
>    > All repositories have been installed.
>    > Total run time: 0:00:05.804217
>    > ```
>    {: .code-out}
>
{: .hands-on}

## Build the BWA index for a reference genome

> <hands-on-title>Build the sacCer3 BWA index</hands-on-title>
>
> 1. Create a config file for `run-data-managers` named `build-sacCer3-bwa.yml`:
>
>    {% raw %}
>    ```yaml
>    data_managers:
>      - id: toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/bwa_mem_index_builder_data_manager/0.0.5
>        params:
>          - 'all_fasta_source': '{{ item.dbkey }}'
>          - 'sequence_name': '{{ item.name }}'
>        items:
>          - dbkey: sacCer3
>            name: 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)'
>        data_table_reload:
>          - bwa_mem_indexes
>    ```
>    {% endraw %}
>
> 2. Run the BWA-MEM index builder DM with `run-data-managers`:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```console
>    > run-data-managers -g https://galaxy.example.org -a <api-key> --config build-sacCer3-bwa.yml
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```console
>    > Storing log file in: /tmp/ephemeris_esecdef4
>    > Running data managers that populate the following source data tables: ['all_fasta']
>    > Running data managers that index sequences.
>    > Dispatched job 2. Running DM: "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/bwa_mem_index_builder_data_manager/0.0.5" with parameters: {'all_fasta_source': 'sacCer3', 'sequence_name': 'S. cerevisiae Apr. 2011 (SacCer_Apr2011/sacCer3)'}
>    > Job 2 finished with state ok.
>    > Finished running data managers. Results:
>    > Successful jobs: 1
>    > Skipped jobs: 0
>    > Failed jobs: 0
>    > ```
>    {: .code-out}
>
> 3. In the Galaxy UI, access the Admin menu from the top bar
> 4. Click **Local Data**, which can be found on the left, under **Server**
> 5. Click **bwa_mem_indexes** under *View Tool Data Table Entries*
>
>    You should see that *sacCer3* has been added to bwa_mem_indexes. You can also verify this via the command line using the methods you've learned above.
>
{: .hands-on}

# Verify that BWA can access the new reference data

Finally, we will verify that the BWA tool can see the new genome indexes.

> <hands-on-title>Configure Galaxy</hands-on-title>
>
> 1. Install the BWA and BWA-MEM tools, if needed. If installing with Ephemeris, the repo name for the `--name` parameter is `bwa` and the owner is `devteam`.
>
>    {% snippet topics/admin/faqs/install_tool.md query="bwa" name="Map with BWA" section="Mapping" %}
>
> 2. Click the "Home" icon at the top to return to the Galaxy analysis interface
>
> 3. Load the {% tool [Map with BWA](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.5) %} tool and verify that the sacCer2 genome (if you completed the "Galaxy UI" section) and/or sacCer3 genome (if you completed the "command line with Ephemeris" section)  appears in the {% icon param-select %} *"Using reference genome"* option.
{: .hands_on}

How cool is that? No editing `.loc` files, no making sure you've got TABs instead of spaces. Fully auto!

{% snippet topics/admin/faqs/git-gat-path.md tutorial="reference-genomes" %}

tools_json:
["Create DBKey and Reference Genome - fetching", "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_fetch_genome_dbkeys_all_fasta/data_manager_fetch_genome_all_fasta_dbkey/0.0.4", "BWA-MEM index - builder", "toolshed.g2.bx.psu.edu/repos/devteam/data_manager_bwa_mem_index_builder/bwa_mem_index_builder_data_manager/0.0.5", "Map with BWA", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.5"]


