# tutorials

topic: admin
tutorial: ansible
title: Ansible
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: []
tags_json: ["ansible"]
content_hash: a8483daffdd143707ade69d8c78f7c84
last_modified: 2026-02-26T21:12:14.132998
workflows_json: []
content:
# Overview

In this tutorial we will briefly cover what Ansible is and how to understand what it does. This guide is not meant to make you an expert on Ansible, but perhaps give you enough that you can debug broken roles and modify them to suit your needs. Maybe even [install Galaxy using Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) or contribute to the [Galaxy Project's Ansible roles](https://github.com/galaxyproject?q=ansible).

This will be a very practical training with emphasis on looking at examples from modules and becoming self sufficient.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/admin-testing.md %}

# What is Ansible?

Ansible runs commands on local or remote computers. It can move files around, create files from templates, and run command line tools. Primarily used for system administration tasks at scale. It has a push model rather than a pull model like Puppet. If you've used Puppet, Ansible doesn't evaluate what changes need to be made and make those, it just runs through all of commands every time.

Some terms that you should know first:

Inventory file
:    An Ansible-specific file that defines the systems ("hosts") and groups of hosts on which Ansible should operate.

Ansible module
:    A piece of Python code that converts some parameters into an invocation. An example would be the `command` module which converts parameters like `command: ls` into a command line that is executed. There are pre-built modules for just about everything.

Task
:    A call to an Ansible module that should be executed and the configuration for this module.

Role
:    A folder containing some tasks, templates, files, and default values for variables, with a predefined directory structure. People share roles on ["Ansible Galaxy"](https://galaxy.ansible.com/) (repository unrelated to the Galaxy Project).

Playbook
:    A YAML file listing a set of tasks and/or roles that should be applied to a group of hosts.

Vault
:    An encrypted YAML file. You put your secrets here and then you can use them in tasks, roles and playbooks.

Looking at each of these briefly:

## Inventory file

```ini
[webservers]
192.0.2.0
192.0.2.1

[databases]
192.0.2.3 ansible_user=root
```

Here we've defined two groups of computers, `webservers` and `databases`. `ansible_user` is used to
specify which user to connect with (you need to specify that if you SSH in with a username different
than your current local user account’s name).

As you can see, in this inventory we connect to multiple remote machines. This is common practice;
Ansible playbooks are stored on your laptop or a build server, and from there Ansible connects out to the
remote machines that are managed. The advantage of running remotely is that you can manage dozens of
machines simultaneously, rather than just the local machine. This scaling out to N machines is one of
the strengths of Ansible.

Additionally, playbooks are often stored in Git or other version control repositories to ensure that
if anything happens to the infrastructure you manage, you'll still have a copy of the information
required to rebuild everything with Ansible.

> <details-title>Ansible Inventory Documentation (Do you connect with a different user? Password? SSH key?)</details-title>
> For more advanced features of the inventory file, check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/intro_inventory.html).
{: .details}

## Roles

We can look at the [ansible-cvmfs role](https://github.com/galaxyproject/ansible-cvmfs) as an example for the layout of a role:

```
├── defaults
│   └── main.yml
├── files
│   ├── cvmfs_remount_sync.c
│   ├── cvmfs_remount_sync.centos_6
│   ├── cvmfs_remount_sync.centos_7
│   └── ...
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── tasks
│   ├── apache.yml
│   ├── ...
│   ├── main.yml
│   └── ...
├── templates
│   ├── ...
│   └── stratum1_squid.conf.j2
└── vars
    ├── ...
    ├── main.yml
    └── ...
```

These are the folders that are included in many complex roles. Simpler roles will often not need all of the folders.

Folder    | Usage
--------- | ------
defaults  | Default values for variables the user can set (e.g. "version of software to install").
files     | These are files which should be copied as-is over to the remote location.
handlers  | This is typically used for restarting processes.
meta      | Only needed to publish your role to Ansible Galaxy (repository unrelated to the Galaxy Project).
tasks     | **Always start reading here**. This is the most important folder and the best place to start when trying to understand what an unfamiliar role does. Anything that is loaded will be referenced here (e.g. variables to load, handlers, files, templates).
templates | Files that are templated out with variables before being copied.
vars      | Default values for variables the user should normally not change (e.g. name of a package in different Linux distributions).

> <details-title>Ansible Role Documentation</details-title>
>
> For more information check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/playbooks_reuse_roles.html).
{: .details}

## Modules and Tasks

A `tasks/main.yml` file calls multiple Ansible modules to accomplish its goal. A typical tasks file looks like:

```yaml
---
- name: Download cvmfs_preload utility when desired
  get_url:
    url: https://cvmrepo.web.cern.ch/cvmrepo/preload/cvmfs_preload
    dest: "{% raw %}{{ cvmfs_preload_path }}/cvmfs_preload{% endraw %}"
    owner: root
    group: root
    mode: 755
  when: cvmfs_preload_install | bool

- name: Ensure AutoFS is enabled + running
  service:
    name: autofs
    enabled: yes
    state: started
```

Here we have two tasks. Each has a `name` that will be shown to the person running the playbook.

The first example invokes the [`get_url`](https://docs.ansible.com/ansible/2.9/modules/get_url_module.html) module with 5 arguments (`url`, `dest`, `owner`, `group` and `mode`). This will download a file from the location indicated in `url` to the directory specified by `dest`, change user and group ownerships to `root`, and change file permissions to `755` (as assigned by the `mode` argument). The `dest` parameter uses a [Jinja2](http://jinja.pocoo.org/) template to evaluate the value of the variable `cvmfs_preload_path`. We can [grep through](https://github.com/galaxyproject/ansible-cvmfs/search?q=cvmfs_preload_path) the repository and see that `defaults/main.yml` sets that to `/usr/bin` by default. We can override this if we need, we'll come back to that later. The first task also has a `when` condition to ensure it only runs when the `cvmfs_preload_install` variable is set.

The second invokes the [`service`](https://docs.ansible.com/ansible/2.9/modules/service_module.html) module. The arguments to this one are quite legible and the functionality can be inferred from the names for the most part: The service `name: autofs` will be `enabled` and its `state` should be `started`.

[Many modules](https://docs.ansible.com/ansible/2.9/modules/modules_by_category.html) are available for you to use.

### Stylistic Choices

Ansible accepts two ways to format tasks:

YAML style:

```yaml
- package:
    name: "{% raw %}{{ package_name }}{% endraw %}"
    state: "{% raw %}{{ package_state }}{% endraw %}"
```

And an inline style.

```yaml
{% raw %}- package name={{ package_name }} state={{ package_state }}{% endraw %}
```

Some groups prefer one style or another. You can mix both of these but you probably shouldn't. In the YAML style the templated value needs to be quoted if the value after the colon starts with a `{` (see [this page](https://docs.ansible.com/ansible/9/playbook_guide/playbooks_variables.html#when-to-quote-variables-a-yaml-gotcha) for more details ). The inline style does not require quoting of templated values.

## Playbooks

```yaml
- name: CVMFS
  hosts: webservers
  vars:
    cvmfs_numfiles: 4096
  roles:
    - galaxyproject.cvmfs
```

This is a quite minimal playbook. It selects a `hosts` group named `webservers`, overrides the variable `cvmfs_numfiles`, and then says the following set of roles will be executed for this group of hosts. Ansible makes it easy to collect tasks that should apply to a group of hosts and run a playbook for all of those hosts. Some good uses of this are things like ensuring a certain set of users are installed on all of your managed machines, or using one of the package autoupdating roles to make sure your machines are up-to-date.

> <details-title>Ansible Playbook Documentation</details-title>
>
> For more information check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/playbooks_intro.html).
{: .details}

### Philosophies

Different groups use playbooks differently. Here is a non-exhaustive list of ways that the author has seen playbooks used:

Strategy                                | Use Cases
--------------------------------------- | ---
A single playbook that does everything  | Works well in cloud use cases where you would rather terminate a VM and just restart from nothing. If 100% of the tasks are idempotent (can be re-run without causing problems) then this can also work well for ensuring all machines that are managed are meeting your compliance standards.
Playbooks that execute one-off commands | Works well when you have commands that cannot safely be re-run (e.g. you are managing a repository on a remote machine and executing a 'commit' action.) Also works if you have multiple people who are responsible for managing a machine but you want to ensure exactly the same commands are run each time a task needs to be done, with no variation.

## Variables

There are a bunch of places variables can be set. [The list is ridiculous.](https://docs.ansible.com/ansible/2.9/user_guide/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable) Try and find a convention within your group and stick to that to help manage the chaos.

There are some special places you can put variables that will be loaded automatically:

- `group_vars/<group_name>.yml`: if you run a playbook which has `hosts: webservers`, and `group_vars/webservers.yml` exists, it will be loaded.
- `host_vars/<host_name>.yml`: if you run a playbook and it affects host (e.g.) `api01`, and `host_vars/api01.yml` exists, it will be loaded automatically.

For a real-life example, UseGalaxy.eu [generally attempts](https://github.com/usegalaxy-eu/infrastructure-playbook) to put variables in an appropriately named `group_vars/` file.

## Vaults

[`ansible-vault`](https://docs.ansible.com/ansible/2.9/user_guide/vault.html) is a super useful tool that lets you encrypt secrets for your group using a pre-shared key. This allows you to commit ALL of your Ansible playbooks and variables to source control, without the concern of leaking secrets. E.g. [UseGalaxy.eu](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/secret_group_vars/all.yml)'s vault file. The encrypted version is not very interesting to look at, but it is mostly to show that we confidently place an encrypted copy of our secrets online, under configuration management. This has made our life a lot easier.


# Your First Playbook and First Role

The above introduction was certainly not enough for you to feel confident in Ansible, so we will now build a small role and a playbook to run this role.

> <warning-title>Safety First</warning-title>
>
> Many of the things you can do with Ansible can be quite dangerous. As dangerous as normally being at the command line, but scaled across N machines. Be very careful with the changes you plan to make.
> Ansible provides some flags which can help you identify changes before they're made to production systems:
>
> **`--diff`**
> :    This will show the difference whenever a file is changed.
>
> **`--check`**
> :    Will do a dry-run of the playbook, attempting to show you which tasks will execute, which files will be updated. Not all actions can be predicted because commands are not run, if a downstream task depends on the result of a command execution task, Ansible has no way of knowing whether or not it will execute.
>
> In this tutorial we will write to files in `/tmp` as that is a *relatively* safe thing to do. The training material community does not have the resources to test this tutorial across all of the platforms you might want to run it on. Additionally we do not want to be responsible if you accidentally cause permanent damage by following this tutorial.
>
{: .warning}


> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have [Ansible installed](https://docs.ansible.com/ansible/2.9/installation_guide/intro_installation.html) on the machine you will run this tutorial from
>
>    > <comment-title>Running Ansible on remote machine</comment-title>
>    > It is possible to have Ansible installed on your laptop/local machine and run it against some remote hosts as well. We will **not** do that in this training.
>    {: .comment}
>
> 2. Your `ansible` version is `>=2.7`, you can check this by running `ansible --version`
>
{: .comment}


## A Basic Role

> <hands-on-title>Setting up our workspace</hands-on-title>
>
> 1. In this training we will run Ansible on the machine it will modify. This is not best practice, but it is convenient for trainings. You should probably run this in a VM either on the Cloud or in VirtualBox or similar
>
>    All of the steps are the same, no matter which machine Ansible will manage and where you run it. The only difference is the connection setup
>
> 3. **Create a directory named `intro` and `cd` into it.**
>
>    It's good practice to keep your deployments separated.
>
> 4. Create your inventory file (named `hosts`) in this folder
>
>    1. We will call our group `my_hosts`
>
>    2. Create [an inventory file](https://docs.ansible.com/ansible/2.9/user_guide/intro_inventory.html) with the group `my_hosts` and `localhost ansible_connection=local`, which tells Ansible to not use SSH, and just use the local connection. Additionally, you should explicitly set the `ansible_user` variable to the username to use when connecting to the server. Ansible has changed its behaviour over time regarding whether or not `ansible_user` is defined, and it is most effective to define it explicitly even when it can sometimes be inferred.
>
>       > <solution-title></solution-title>
>       > The file should look like:
>       >
>       > ```ini
>       > [my_hosts]
>       > localhost ansible_connection=local ansible_user=ubuntu
>       > ```
>       {: .solution }
>
>       > <details-title>For your own infrastructure, do you connect as a different user? Password? SSH key?</details-title>
>       > For more advanced features of the inventory file, check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/intro_inventory.html).
>       {: .details}
>
> 5. Create the roles directory, your role, and the tasks folder: `mkdir -p roles/my-role/tasks/`
>
> 6. Create a YAML file in that directory, `roles/my-role/tasks/main.yml` and open it for editing
>
> 7. Define a `copy` task like below:
>
>    ```yaml
>    ---
>    - name: Copy a file to the remote host
>      copy:
>        src: test.txt
>        dest: /tmp/test.txt
>    ```
>
>    You can read about all of the parameters available to the [`copy`](http://docs.ansible.com/ansible/2.9/modules/copy_module.html) module in Ansible's documentation.
>
>    > <details-title>Ansible Module Documentation</details-title>
>    > You can usually find a module for most commands you will run in a shell, e.g. by searching the internet for "ansible copy file to server" or "ansible restart service". If you cannot find a module that does it, there is the [`command`](http://docs.ansible.com/ansible/2.9/modules/command_module.html) module, but this should be avoided if possible. Expect to have a browser session with 10-30 different Ansible module documentation tabs if you work with Ansible regularly, no one remembers what arguments are available to every module.
>    >
>    {: .details }
>
> 8. Create a `roles/my-role/files` folder, and within it a file named `test.txt`, containing the content "Hello, Galaxy 🚀"
>
> 9. This is a complete role by itself and will copy the file `test.txt` from the `roles/my-role/files/` folder over to the target host and place it in `/tmp`.
>
> 10. Create and open `playbook.yml` file for editing in the `intro` directory you created. Place the following content in there:
>
>     ```yaml
>     ---
>     - hosts: my_hosts
>       roles:
>         - my-role
>     ```
>
>     > <question-title></question-title>
>     >
>     > How does your file tree look now? Use `find` or `tree`.
>     >
>     > > <solution-title></solution-title>
>     > >
>     > > ```
>     > > .
>     > > ├── hosts
>     > > ├── playbook.yml
>     > > └── roles
>     > >     └── my-role
>     > >         ├── files
>     > >         │   └── test.txt
>     > >         └── tasks
>     > >             └── main.yml
>     > > ```
>     > >
>     > {: .solution }
>     {: .question}
>
> 11. Run the playbook:
>
>     > > <code-in-title>Bash</code-in-title>
>     > > ```bash
>     > > ansible-playbook -i hosts playbook.yml
>     > > ```
>     > {: .code-in}
>     >
>     > > <code-out-title>Bash</code-out-title>
>     > > ```
>     > > PLAY [my_hosts] ****************************************************************
>     > > TASK [Gathering Facts] *********************************************************
>     > > ok: [localhost]
>     > > TASK [my-role : Copy a file to the remote host] ********************************
>     > > changed: [localhost]
>     > > PLAY RECAP *********************************************************************
>     > > localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
>     > > ```
>     > {: .code-out}
>     {: .code-2col}
>
> 12. Login to the appropriate host (unless your host was `localhost`) and `cat /tmp/test.txt` to see that the change was made.
>
{: .hands_on}

Now that you've done this, here are some starting points for exploration:

- Add more hosts, watch as Ansible executes over all of them in parallel.
- Identify a task you do regularly, e.g. restarting a service. Find the Ansible service module and add that to your playbook.

> <comment-title>Too Many Cows?</comment-title>
> If you've installed the `cowsay` tool, Ansible (for some reason) will take advantage of that to output a lot of the output with cowsay. To disable this you can `export ANSIBLE_NOCOWS=1` (Remember that exporting will only last as long as the current invocation of your terminal does, so consider adding this to your user profile if you wish to keep cowsay installed and still have legible output.)
>
> ```
>  ________________
> < PLAY [my_hosts] >
>  ----------------
>         \   ^__^
>          \  (oo)\_______
>             (__)\       )\/\
>                 ||----w |
>                 ||     ||
>  ________________________
> < TASK [Gathering Facts] >
>  ------------------------
>         \   ^__^
>          \  (oo)\_______
>             (__)\       )\/\
>                 ||----w |
>                 ||     ||
> ```
{: .comment}


## Facts

In the last step of the last hands-on, you ran your playbook. The first `TASK` that was executed was not one that you had written, it reads:

```
TASK [Gathering Facts] *********************************************************
ok: [localhost]
```

The [`setup`](https://docs.ansible.com/ansible/2.9/modules/setup_module.html) module runs by default for every host and gathers facts about the host.

> <hands-on-title>The Setup Module</hands-on-title>
>
> 1. Run the command `ansible -i hosts -m setup my_hosts | less`.
>
>    The `my_hosts` at the end of the command refers to the group we defined in our `hosts` inventory file.
>
> 2. Investigate the output. See what sort of information is made available to you.
>
>    > <question-title></question-title>
>    >
>    > 1. What variable stores the OS name?
>    > 2. Where are all of the places that you can find your machine's IP (v4) addresses?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The OS name is stored in `ansible_distribution`. We saw `ansible_os_family` used above in the `ansible-cvmfs` role. You can use these variables if you are writing a generic role but packages or commands are named different on different operating systems.
>    > >
>    > > 2. Ansible stores network information in quite a few places, sometimes one place is more convenient or more correct to use:
>    > >
>    > >    - `ansible_all_ipv4_addresses`
>    > >    - `ansible_default_ipv4.address`
>    > >    - `ansible_<interface_name>.ipv4`
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}


## Templates

Templates give you greater control over the files you are deploying to the target system. If you need to deploy a file to multiple hosts, but configure it differently on each host, you should use templates. For instance, deploying a service that should only listen on the correct IP address for that host would be a good use case for templates. All of the facts you discovered in the previous hands-on are available to you to use in templates, `when` statements (like the [ansible-cvmfs example we saw earlier](#modules-and-tasks)). Additionally all of the variables you've defined are available as well.

> <details-title>Template Syntax</details-title>
> Templates end with the `.j2` suffix and use Jinja2 syntax. If you are not familiar with it, you should [read about it](http://jinja.pocoo.org/docs/2.10/templates/) first, before moving on with the tutorial. Ansible fills the templates with variable values and copies the file to its remote destination without the `.j2` suffix.
{: .details}

> <hands-on-title>Variables and Templates</hands-on-title>
>
> 1. Create the directories: `roles/my-role/templates`, `roles/my-role/defaults`
>
> 2. Create and edit a file for your role's variables, `roles/my-role/defaults/main.yml`
>
> 3. Insert the following content:
>
>    ```yaml
>    ---
>    server_name: Cats!
>    ```
>
>    This will define a variable `server_name` that can then be used in templates.
>
> 4. Create and edit `roles/my-role/templates/test.ini.j2`
>
> 5. Insert the following content:
>
>    {% raw %}
>    ```ini
>    [example]
>    server_name = {{ server_name }}
>    listen = {{ ansible_default_ipv4.address }}
>    ```
>    {% endraw %}
>
> 6. Edit `roles/my-role/tasks/main.yml` and append a new task to the end to template this file:
>
>    ```yaml
>    - name: Template the configuration file
>      template:
>        src: test.ini.j2
>        dest: /tmp/test.ini
>    ```
>
> 7. Run the playbook again.
>
>
> 8. Check the contents of `/tmp/test.ini`
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat /tmp/test.ini
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > >
>    > > The file should look like:
>    > > ```
>    > > [example]
>    > > server_name = Cats!
>    > > listen = 192.168.0.2
>    > > ```
>    > > Where the last line has the machine's IP address.
>    > {: .code-out}
>    {: .code-2col}
>
>    Now that this has worked successfully, we will setup a `group_vars` folder
>    to show how a person using `my-role` would override the `server_name` variable.
>
> 9. Create the folder `group_vars/` (in the root of your directory)
>
> 10. Create and edit `group_vars/my_hosts.yml`
>
> 11. Insert the following:
>
>     ```yaml
>     ---
>     server_name: Dogs!
>     ```
>
>     {% snippet topics/admin/faqs/ansible-connection.md %}
>
> 12. Run the playbook again, but imagine you are worried about this change, and supply the `--check --diff` flag to see what changes are made before committing to make them.
>
>     > <code-in-title>Bash</code-in-title>
>     > ```bash
>     > ansible-playbook -i hosts playbook.yml --check --diff
>     > ```
>     {: .code-in}
>
>     {% snippet topics/admin/faqs/ansible_diff.md %}
>
>     > <code-out-title></code-out-title>
>     >
>     > ```
>     > PLAY [my_hosts] ****************************************************************
>     > TASK [Gathering Facts] *********************************************************
>     > ok: [localhost]
>     > TASK [my-role : Copy a file to the remote host] ********************************
>     > ok: [localhost]
>     > TASK [my-role : Template the configuration file] *******************************
>     > --- before: /tmp/test.ini
>     > +++ after: /home/hxr/.ansible/tmp/ansible-local-1906887dr2u6j8n/tmptx9pdelg/test.ini.j2
>     > @@ -1,3 +1,3 @@
>     >  [example]
>     > -server_name = Cats!
>     > +server_name = Dogs!
>     >  listen = 192.168.0.25
>     > changed: [localhost]
>     > PLAY RECAP **********************************************
>     > localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
>     > ```
>     > Here you can see that the `server_name` value will be changed. Despite Ansible reporting `changed=1`, no changes have actually been applied to the system.
>     {: .code-out}
>
> 13. Run the playbook again, without the `--check` flag to apply your changes.
{: .hands_on}

> <comment-title>Ansible Variable Templating</comment-title>
> In this hands-on we used some templated variables. We defined them in a template, but they are also commonly used in the group variables file. Our templated variable looked like: {% raw %}`listen = {{ ansible_default_ipv4.address }}`{% endraw %}.
>
> It is common to see things like this in Ansible roles:
>
> ```yaml
> root_dir = /opt/my-app
> config_dir = {% raw %}"{{ root_dir }}/config"{% endraw %}
> ```
>
> When Ansible runs:
>
> 1. It collects variables defined in group variables and other places
> 2. The first task for each machine is the [`setup` module](https://docs.ansible.com/ansible/2.9/modules/setup_module.html) which gathers facts about the host, which are added to the available variables
> 3. When multiple roles execute in a playbook:
>    1. Their defaults are added to the set of variables (the group variables having precedence over these variables)
>    2. They can also dynamically define more variables which may not be set until that role is run
> 4. Before use (in templates, commands, etc.), variables are resolved to their final value
>
>
{: .comment}

# Ansible Galaxy

Now that you've built a small role, you can imagine that building real roles that manage the full installation of a piece of software are not simple things. Ansible Galaxy (repository unrelated to the Galaxy Project) is the answer here. Many roles for common administration tasks, and software installation and setup are readily available on Ansible Galaxy.

**Warning**: This will install Memcached on the target machine.

> <hands-on-title>Installing a module using ansible-galaxy</hands-on-title>
>
> 1. Install the geerlingguy.memcached role with ansible-galaxy into your roles folder:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > ansible-galaxy install \
>    > >     -p roles/ geerlingguy.memcached
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > This will install the new role into your `roles` folder, alongside your own role.
>    > > ```
>    > > - downloading role 'memcached', owned by geerlingguy
>    > > - downloading role from https://github.com/geerlingguy/ansible-role-memcached/archive/1.1.0.tar.gz
>    > > - extracting geerlingguy.memcached to /home/ubuntu/ansible/intro/roles/geerlingguy.memcached
>    > > - geerlingguy.memcached (1.1.0) was installed successfully
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
> 2. Edit your playbook.yml and add the role `geerlingguy.memcached` at the bottom, after `my-role`
>
> 3. Run the playbook
>
>    > <question-title></question-title>
>    >
>    > Did something go wrong?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > Since you have been running the playbook as a non-root user (or at least you should have been!), the step to install a package fails.
>    > > The solution to this is to set `become: true`.
>    > >
>    > > 1. Edit your playbook.yml
>    > >   - add `become: true` just below `hosts: my_hosts`
>    > > 2. Run the playbook again
>    > >
>    > > `become` causes Ansible to attempt to become a different user (using sudo/su/whatever is appropriate), by default this is `root`. If you want to become a different user, just set `become_user`. Beware, the user should be able to privilege escalate without a password prompt. Otherwise when you execute the playbook you should set `--ask-become-pass`, using the privilege escalation password for that host.
>    > >
>    > > > <details-title>Ansible Become</details-title>
>    > > > See the [documentation](https://docs.ansible.com/ansible/2.9/user_guide/become.html) if you need to control this behaviour differently. `become` can be set either at the task level or the playbook level.
>    > > >
>    > > {: .details}
>    > >
>    > > > <tip-title>Should I use sudo or directly login as root?</tip-title>
>    > > > This is often site-specific policy. If your site doesn't have a dictated policy, then it's generally preferable to not allow direct login as root, and login as a separate user.
>    > > {: .tip}
>    > >
>    > >
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

## Choosing a Role

Picking the best role for a task from Ansible Galaxy is not always a trivial task. Sometimes there will only be a single role doing what you need. Other times you'll have to choose between 20 different roles that all look more or less the same. Here are some tips to guide you in identifying appropriate and well-written roles:

- The name should match the software you are using (I.e. ignore a role named `stackstorm` when you are trying to set up `rabbitmq`. Ansible Galaxy does not have perfect search.)
- `geerlingguy` wrote a huge number of roles for many pieces of standard software. If there is a role from this person, this is usually a safe choice. In the same way we recommend `galaxyproject` and `usegalaxy_eu` roles.
- Check the GitHub readme of each role you consider using. Look for:
  - Extensive documentation of all of the variables, their default values, and how they behave.
  - An example playbook using the role
- A large number of downloads is a good sign that other people found it useful

These are usually good proxies for quality, but do not treat them as strict rules. For an example of a role meeting many of these qualities, [`ansible-cvmfs`](https://github.com/galaxyproject/ansible-cvmfs) is good; the variables are well documented and there are example playbooks that you can (more or less) copy-and-paste and run.

Sometimes a role will accomplish 95% of what you need to do, but not everything. Once you have installed the role with `ansible-galaxy install`, you can edit it locally to make any changes. In an ideal world you would contribute this back, but this is not always a high priority. Many projects copy roles directly into their repositories, e.g. [galaxyproject](https://github.com/galaxyproject/infrastructure-playbook/tree/master/roles) and [usegalaxy.eu](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master/roles)

{% snippet topics/admin/faqs/ansible_role-properties.md %}

# Ansible Vault

Now that you have a small role built up, you might start thinking about deploying larger and more complex services and infrastructure. One last common task we want to cover here is the inclusion of secrets. Ansible Vault is really useful to include encrypted secrets in your playbook repository.

> <hands-on-title>Setting up secrets</hands-on-title>
>
> 0. Run `mkdir -p secret_group_vars`
>
> 1. Now we'll create a strong password: `openssl rand -base64 24 > vault-password.txt`
>
> 2. Run `ansible-vault create secret_group_vars/all.yml --vault-password-file=vault-password.txt`
>
>    This will open `secret_group_vars/all.yml` in your text editor.
>
> 3. In this file, enter the following contents and save it:
>
>    ```yaml
>    apikey: super-secret-api-key-wow!
>    ```
>
>    > <question-title></question-title>
>    >
>    > How does your file look? Is it readable? Run `cat secret_group_vars/all.yml`
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file will look like this, it is encrypted by Ansible Vault with AES256 encryption.
>    > > ```
>    > > $ cat secret_group_vars/all.yml
>    > > $ANSIBLE_VAULT;1.1;AES256
>    > > 64373665366130333437393639343534653134346538636239393363373062393830653333323966
>    > > 3134333366363130326139323162323131643763336236320a393262303938316262643764323862
>    > > 36393161666663353231366336613838633866323230303031313465646333613862363264323139
>    > > 3263383530626262370a666139666462663938343531656432353239346532316630366165376566
>    > > 34313765353766666330366632303836353863396430343264303032363739666139383830323565
>    > > 6133663637356331613062353834646561653366386665623930
>    > > ```
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. Use the new variable in our `.ini` file from earlier. Edit `roles/my-role/templates/test.ini.j2` and add the line `{% raw %}apikey = {{ apikey }}{% endraw %}`
>
> 5. Add encrypted variable file to `playbook.yml`
>
>    ```yaml
>    - hosts: my_hosts
>      ...
>      vars_files:
>       - secret_group_vars/all.yml
>    ```
>
> 6. Tell ansible where to find the decryption file. Create file `ansible.cfg` with content
>
>    ```yaml
>    [defaults]
>    vault_password_file=vault-password.txt
>    ```
> 7. Run the playbook
>
> 8. Check the contents of `/tmp/test.ini`
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat /tmp/test.ini
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > The file should look like:
>    > >
>    > > ```ini
>    > > [example]
>    > > server_name = Dogs!
>    > > listen = 192.168.0.2
>    > > apikey = super-secret-api-key-wow!
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
{: .hands_on}

In real life scenarios where you are sharing your playbooks publicly, be sure to encrypt all secrets from the start (or remove any secrets from the git history if you ever committed them in plain text). If you are storing your vault password in a file, remember to add it to your `.gitignore` (or VCS appropriate file.)

# Other Stuff

Ansible has a huge array of features and we can't cover them all. Some commonly used features are documented below:

## With Items

Duplicating tasks ten times to install ten packages is not efficient, so Ansible provides `loop` construct

```yaml
- name: Install stuff
  package:
    name: {% raw %}"{{ item }}"{% endraw %}
    state: installed
  loop:
    - htop
    - git
    - vim
```

This works for any task, not just package installation, if you have things you'd like to repeat.
Note that for this task one would normally list multiple packages in the `name` parameter of the package module.

```yaml
- name: Install stuff
  package:
    name:
      - htop
      - git
      - vim
    state: installed
```

## When Changed

Doing something only when a task is in the "changed" state is a common pattern. This is often used for reloading a service when some configuration files have changed.

```yaml
- name: Copy configuration file
  copy:
    src: main.conf
    dest: /etc/service/main.conf
    owner: root
    group: root
    mode: 0644
  register: service_conf

- name: Restart the service
  service:
    name: service
    enabled: yes
    state: restarted
  when: service_conf is changed
```

## Notifying Handlers

Often you want to restart a service whenever something has changed, like above. But if you need to restart the service whenever one of many different tasks has changed, there is a simpler way than writing `when: a.changed or b.changed or c.changed or ...`

First, move the restarting or reloading of the service into the `handlers/main.yml`

```yaml
- name: restart service
  service:
    name: service
    enabled: yes
    state: restarted

- name: reload httpd
  service:
    name: httpd
    enabled: yes
    state: reloaded
```

Now you can change your task definitions:

```yaml
- name: Copy configuration file
  copy:
    src: main.conf
    dest: "/etc/service/main.conf
    owner: "root"
    group: "root"
    mode: "0644"
  notify: 'restart service'
```

We no longer `register` the command, instead the `notify` attribute automatically marks that the appropriate handler should be called *at the end of the playbook run*. Additionally, tasks from outside of this role can use the same `notify`. So if you use a community built role for managing apache, and then have a custom role that builds the apache configuration, you can use changes there to reload the service using their definition of the reload/restart.


topic: admin
tutorial: file_sources
title: Adding file-sources to Galaxy
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: ["\"earth-system\"", "\"ocean\"", "\"data\""]
content_hash: 0af74b18328c70df83d496a37280851e
last_modified: 2026-02-26T21:12:14.144275
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/file_sources/tutorial.html

content:
This tutorial demonstrates how to implement an S3 bucket as a Galaxy file-source within Galaxy. We will add here the public Argo data Amazon S3 bucket.
Argo is an international program that observes the interior of the ocean with a fleet of profiling floats drifting in the deep ocean currents (https://argo.ucsd.edu). It started 20 years ago and is a dataset of 5 billion in situ ocean observations from 18.000 profiling floats (4.000 active). The Argo GDAC dataset is a collection of 18.000 NetCDF files. It is a major asset for ocean and climate science and a contributor to IOCCP reports.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Find the information you need
> <hands-on-title>Find an S3 bucket</hands-on-title>
> Go on [Amazon Sustainability Data Initiative](https://exchange.aboutamazon.com/data-initiative). 
>
> There you can visit the [catalog of data](https://registry.opendata.aws/collab/asdi/), and by searching for Argo you can directly get to the [Argo registry](https://registry.opendata.aws/argo-gdac-marinedata/).
>
> On this last page you'll find all the information you'll need to add the S3 bucket to Galaxy 
> ![Image of the Argo registry on amazon](../../images/file_sources/argo_registry.png)
{: .hands_on}


# Add the S3 bucket 

> <hands-on-title>Add on Galaxy</hands-on-title>
>    - If not already done clone the Galaxy Europe [Infrastructure-playbook repo](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master)
>    - Create a branch on your fork
>    - Go to the file **file_sources_conf.yml.j2** in **templates/galaxy/config/**
>
> There you can edit the file and add your S3 bucket by adding a Argo specific section, like in the following:
> ```
> - type: s3fs
>   label: Argo marine floats data and metadata from Global Data Assembly Centre (Argo GDAC)
>   id: argo-gdac-sandbox
>   doc: Argo is an international program to observe the interior of the ocean with a fleet of profiling floats drifting in the deep ocean currents (https://argo.ucsd.edu). 
>   bucket: argo-gdac-sandbox
>   anon: true
> ```
> ![Image of the file_sources_conf.yml.j2 file with th argo bucket added](../../images/file_sources/s3_in_galaxy.png)
>
> Finally, commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

# Conclusion 
Here you are all set and once your Pull Request is merged you'll soon be able to see your bucket in **upload data**, **Choose remote files**, and then search for your bucket label !


topic: admin
tutorial: ansible-galaxy
title: Galaxy Installation with Ansible
difficulty: intermediate
time_estimation: 2h30m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["ansible", "deploying", "git-gat"]
content_hash: 8c23b628c182c9df5ffef7c6fa9ea506
last_modified: 2026-02-26T21:12:14.132548
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible-galaxy/tutorial.html

content:
This tutorial assumes you have some familiarity with [Ansible](https://www.ansible.com/resources/get-started) and are comfortable with writing and running playbooks.  If not, please consider following our [Ansible Tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %}) first.

Here we'll see how to install a Galaxy server using an Ansible playbook. The Galaxy Project has decided on Ansible for all of its deployment recipes. For our project, Ansible is even more fitting due to its name:

> An ansible is a category of fictional device or technology capable of instantaneous or faster-than-light communication. It can send and receive messages to and from a corresponding device over any distance or obstacle whatsoever with no delay, even between star systems
{: .quote cite="https://en.wikipedia.org/wiki/Ansible"}

We want to give you a comprehensive understanding of how the Galaxy installation occurs, but we want to avoid you having to write a "custom" Galaxy installation playbook which you would eventually throw away, in order to use the official playbooks. Given these goals, we will go through the playbook in depth first, and then move to a hands-on portion later. If you are not interested in the inner workings, you can [skip to that section now](#installing-galaxy).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="ansible-galaxy" %}

{% snippet topics/admin/faqs/admin-testing.md %}

# Playbook Overview


## Configuration

We'll be using the [official Galaxy role](https://github.com/galaxyproject/ansible-galaxy) to install and manage Galaxy. This role is found in [Ansible Galaxy](https://galaxy.ansible.com/) (no relation - it is Ansible's system for sharing reusable Ansible roles) as [galaxyproject.galaxy](https://galaxy.ansible.com/galaxyproject/galaxy).

The official role is extremely configurable, everything that you want to change is exposed as a variable, and then tasks will change behaviour based on that. The [role documentation](https://github.com/galaxyproject/ansible-galaxy#role-variables) is the most up-to-date source of documentation for the variables. You should take a minute and read over the variables listed there.

The important variables for this tutorial are:

- `galaxy_root`
- `galaxy_commit_id`
- `galaxy_config`
- `galaxy_server_dir` (automatically set based on `galaxy_root`)

These are largely self explanatory: a directory for all of Galaxy's code and configuration, which commit should be installed, and the Galaxy configuration. We will not explain Galaxy configuration variables in detail as they are covered sufficiently in the `galaxy.yml` sample file or the [online documentation](https://docs.galaxyproject.org/en/master/admin/config.html#configuration-options).

The official recommendation is that you should have a variables file such as `group_vars/galaxyservers.yml` for storing all of the Galaxy configuration.

## Tasks

As with every role, the entry point for execution is the `tasks/main.yml` file. [In the case of the galaxyproject.galaxy role](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/main.yml), this includes a few groups of important tasks:

- [Clone (or Download) Galaxy](#cloning-galaxy)
- [Managing Configuration](#managing-configuration)
- [Fetching Dependencies](#dependencies)
- [Managing Mutable Setup](#mutable-setup)
- [Managing the Database](#managing-the-database)
- [Building the Client](#building-the-client)

The actions each set of tasks performs are described below.

### Cloning Galaxy

The [clone](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/clone.yml) task is the one which is primarily interesting to us, it downloads Galaxy, using git, at a specific commit (or more generally, any [git reference](https://git-scm.com/book/en/v2/Git-Internals-Git-References)).

1. Ansible tries to update Galaxy, cloning it if it is missing, or otherwise attempting to update to the correct commit (or latest commit of the given branch).
2. Any change is reported.
3. The virtualenv is set up:
    1. An empty virtualenv is created.
    2. Pip is updated within the virtualenv.
4. Any `.pyc` files are removed, as this can occasionally result in Python loading the cached code, even if the corresponding `.py` file is no longer present at the checked-out commit. For safety, all of these are removed.

With that Galaxy is cloned to disk and is ready to be configured by the next task.

### Managing Configuration

The [static configuration setup](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/static_setup.yml) is relatively straightforward:

1. The directories for Galaxy configuration data and for the shed tools are created
2. Any config files are copied over
3. Any templates are copied over
4. The `galaxy.yml` is deployed

The setup for deploying extra Galaxy configuration templates and files is a little bit non-standard by Ansible best practices. Here you are expected to provide your own templates and static config files, and then describe them as a list of files and where they should be deployed to.

Using the [UseGalaxy.eu](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/02ca578211bfee45044facf36635d28208e5dbb3/group_vars/galaxy.yml#L578) configuration as an example, we have something like:


{% raw %}
```yaml
galaxy_config_templates:
  - src: templates/galaxy/config/builds.txt
    dest: "{{ galaxy_config.galaxy.builds_file_path }}"
  - src: templates/galaxy/config/data_manager_conf.xml
    dest: "{{ galaxy_config.galaxy.data_manager_config_file }}"
  - src: templates/galaxy/config/datatypes_conf.xml
    dest: "{{ galaxy_config.galaxy.datatypes_config_file }}"
  - src: templates/galaxy/config/dependency_resolvers_conf.xml
    dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
  - src: templates/galaxy/config/disposable_email_blocklist.conf
    dest: "{{ galaxy_config.galaxy.blocklist_file }}"
```
{% endraw %}

The configuration here is a bit different, it references the `galaxy_config`, which is structured like:

{% raw %}
```yaml
galaxy_config:
  galaxy:
    builds_file_path: "{{ galaxy_config_dir  }}/builds.txt"
    datatypes_config_file: "{{ galaxy_config_dir  }}/datatypes_conf.xml"
```
{% endraw %}

So the references in `galaxy_config_templates` to `galaxy_config` are done to ensure that the setting for e.g. "location of the datatypes config file" is the same between where we have configured Galaxy to looking for it, and where the file has been deployed, without requiring us to make variables changes in numerous places.

{% snippet topics/admin/faqs/ansible_define-once.md %}

### Dependencies

Now that Galaxy is available on disk, Ansible is ready to start processing [dependencies](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/dependencies.yml) of Galaxy.

1. The virtualenv is updated with data from the `galaxy_requirements_file`, by default pointing to the requirements file in the codebase: {% raw %}`{{ galaxy_server_dir }}/lib/galaxy/dependencies/pinned-requirements.txt`.{% endraw %}
2. Any necessary conditional dependencies of Galaxy are [collected by processing the config file](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/dependencies/__init__.py)
3. and then installed to the virtualenv.

### Mutable Setup

[This task](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/mutable_setup.yml) creates a directory and initializes "mutable" (written/managed by Galaxy itself) configuration files. It also deploys any hand-managed mutable config files, but it is unlikely that you want to manage these directly, as Galaxy does a sufficient job. Any changes you make to Galaxy, for example installing some tools, would result in the tools being "forgotten about", if you re-ran the playbook and overwrote the `shed_tool_conf.xml` mutable config file with a hand-managed one.

### Managing the Database

The [database management tasks](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/database.yml) are extremely convenient; any time you run the playbook to update Galaxy, this will automatically run the database schema migration as needed.

1. The role first obtains the current DB version and the maximum possible DB version based on the codebase.
2. If needed, the database is created.
3. Both versions are reported for the runner of the playbook.
4. If the versions are different, then Ansible runs the command to upgrade the database to the latest version.

As an administrator who often forgot to run the upgrade, and would only notice it once Galaxy crashed during startup, having this process completely automated is extremely nice.

### Building the Client

Galaxy is a modern web application that includes both a server (written in Python) and a client (written in Javascript). After the server is installed and its database prepared, the next step is to build the client application. This means fetching its dependencies, bundling components, creating minified copies of static content, etc. This process ensures that the smallest possible amount of data is transferred to the user when accessing Galaxy over the web, which is important for the performance of the website.

This process can be lengthy and resource intensive. If you want to speed up your deployment, set `galaxy_client_use_prebuilt: true` and Ansible will download a prebuilt client using [yarn](https://classic.yarnpkg.com/en/docs/getting-started).

The client lives in the Galaxy code under the [client/](https://github.com/galaxyproject/galaxy/blob/dev/client/) directory, and the build process deploys it to the [static/](https://github.com/galaxyproject/galaxy/tree/dev/static/) directory, which we will configure a webserver to serve in this tutorial.

## Handlers

A number of the tasks that are executed will trigger a restart of Galaxy. The auto-magic implementation of this restart [handler](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html) should work for most cases, but it is possible to define your own custom handlers if necessary. If this is the case for you, the role provides a way to do this by notifying handlers that ["listen"](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html#naming-handlers) to certain notifications. See the [role handlers](https://github.com/galaxyproject/ansible-galaxy/tree/main/handlers) for details.

## Defaults

As with other roles, numerous [default values](https://github.com/galaxyproject/ansible-galaxy/blob/master/defaults/main.yml) are provided, but these are useful mostly as reference, and not to go through individually.

## Summary

Installation of Galaxy with the playbook follows generally the steps you would expect:

- Galaxy is cloned (or updated)
- A virtualenv is created if it doesn't exist
- Configuration files are installed
- Any missing dependencies are installed
- Any database updates are applied
- The client application is built and deployed (to save time set `galaxy_client_use_prebuilt: true`)

It would not be difficult to write a role that does this yourself, but by using
the `galaxyproject.galaxy` role, you know that you're getting all of the Galaxy
best practices and knowledge from previous admins codified for you.

# Installing Galaxy

With the necessary background in place, you are ready to install Galaxy with Ansible. The playbooks will start simple, and grow over time. We will start with the minimal Galaxy playbook which only requires setting the `galaxy_root` and expand from there. First, however, we need a database for Galaxy to connect to, so we will do that now.

To proceed from here it is expected that:

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have [Ansible installed](https://docs.ansible.com/ansible/2.9/installation_guide/intro_installation.html) on the machine where you will install Galaxy
>
>    > <comment-title>Running Ansible on remote machine</comment-title>
>    > It is possible to have Ansible installed on your laptop/local machine and run it against some remote hosts as well. We will **not** do that in this training.
>    {: .comment}
>
> 2. Your `ansible` version is `>=2.10`, you can check this by running `ansible --version`
> 3. You have an [inventory file](../ansible/tutorial.html#inventory-file) with the VM or host specified where you will deploy Galaxy and Galaxy's database. We will refer to these group of hosts as "galaxyservers" and "dbservers," respectively.
> 4. Your VM has a public DNS name: this tutorial sets up SSL certificates from the start and as an integral part of the tutorial.
> 5. Your VM has `python3` installed.
>
>    If you follow the official Ansible instructions to install Ansible on a blank machine, you will probably end up with py2 installed. You will need to install python3 in addition.
>
> 6. In your inventory file, you have written the full DNS hostname that has been provided, and **not** `localhost`, as we will be requesting SSL certificates.
>
> 7. You have the following ports exposed:
>
>    - 22 for SSH, this can be a different port or via VPN or similar.
>    - 80 for HTTP, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 443 for HTTPs, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 5671 for AMQP for Pulsar, needed if you plan to setup Pulsar for remote job running.
>
{: .comment}


> <comment-title>Ubuntu or Debian, CentOS or RHEL?</comment-title>
> The training tutorial is only tested on Ubuntu, as that is the training platform most often used. The training *should* work on RHEL/CentOS, and the roles we use are definitely compatible with both. If any of the variable values differ between Ubuntu and CentOS, we try to note it in the tutorial. Any places we don't note it are bugs.
>
> There are known issues with CentOS7 and python3, which is used in this tutorial. If you use this setup, you should have python2 and python3 coexisting, and use python2 for Ansible (i.e. do not set `interpreter_python` in your `ansible.cfg`) and python3 for Galaxy. This setup requires numerous changes that you will need to discover, which are not covered in this tutorial.
>
> On older versions, both `python-psycopg2` and `python3-psycopg2` may need to be installed.
>
> CentOS8 (and RHEL in general) have significantly different package names for some modules, beware!
{: .comment}


## Requirements

We have codified all of the dependencies you will need into a YAML file that `ansible-galaxy` can install.

> <hands-on-title>Installing roles</hands-on-title>
>
> 1. Create a new directory `galaxy` in your home folder, and `cd` into that directory
>
> 2. Create a new file in your working directory called `requirements.yml` and include the following contents:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/requirements.yml
>    @@ -0,0 +1,13 @@
>    +# Galaxy, Postgres, Nginx
>    +- src: galaxyproject.galaxy
>    +  version: 0.12.1
>    +- src: galaxyproject.nginx
>    +  version: 0.7.1
>    +- src: galaxyproject.postgresql
>    +  version: 1.1.2
>    +- src: galaxyproject.postgresql_objects
>    +  version: 1.2.0
>    +- src: galaxyproject.miniconda
>    +  version: 0.3.4
>    +- src: usegalaxy_eu.certbot
>    +  version: 0.1.13
>    {% endraw %}
>    ```
>    {: data-commit="Add requirements"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>    > <details-title>What do each of these roles do?</details-title>
>    > We'll cover it in more detail as we use each of the roles but briefly:
>    >
>    >  | Role | Purpose |
>    >  |------|---------|
>    >  |`galaxyproject.galaxy` | Installs and configures the Galaxy application|
>    >  |`galaxyproject.nginx` | Sets up a webserver|
>    >  |`galaxyproject.postgresql` | Installs our database, PostgreSQL|
>    >  |`galaxyproject.postgresql_objects` | Creates users and databases within PostgreSQL|
>    >  |`galaxyproject.miniconda` | Installs miniconda, which is used by Galaxy|
>    >  |`usegalaxy_eu.certbot` | Installs certbot and requests SSL certificates|
>    {: .details}
>
>    > <details-title>Role version vs Galaxy version?</details-title>
>    > Q: Is there a correspondence between galaxy role versions and galaxy versions?
>    > A: They are correlated (because generally new Galaxy versions could introduce e.g. new configuration options that the galaxy role would then need to adopt), but the Galaxy role can install older versions of Galaxy.
>    {: .details}
>
> 3. In the same directory, run:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    This will install all of the required modules for this training into the `roles/` folder. We choose to install to a folder to give you easy access to look through the different roles when you have questions on their behaviour.
>
> 4. Inspect the contents of the newly created `roles/` directory in your working directory.
>
> 5. It's good practice to put your playbooks and configuration files under version control (usually in a git repository), but there is no need to keep track of the content of the `roles/` directory, as all of that data can be perfectly recreated from the `requirements.yml` file.
>
>    If you plan to put this git repository online (e.g. on GitHub), any file containing clear-text passwords should *never* ever be committed to the repository, even if the repository is private. For this reason, we should tell git, ahead of time, to ignore one such file, `.vault-password.txt` , which will use later on in this lesson.
>
>    To configure git to ignore certain files and directories, they need to be listed in a file called `.gitignore` , which should be instead tracked as part of your git repository. You can now create the `.gitignore` file with the following content:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/.gitignore
>    @@ -0,0 +1,2 @@
>    +roles/
>    +.vault-password.txt
>    {% endraw %}
>    ```
>    {: data-commit="Add .gitignore"}
>
>    When you run `git status` you'll notice that the `roles/` folder is not listed among the other "Untracked files".
>
>    Now you can do `git add .` to add all of the files in the current directory to your repository, and not worry about committing these files and directories by mistake!
{: .hands_on}

> <hands-on-title>Configuration files</hands-on-title>
>
> 1. Create a `ansible.cfg` file (next to your playbook) to [configure settings](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html) like the inventory file (and save ourselves some typing!), or the Python interpreter to use:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/ansible.cfg
>    @@ -0,0 +1,11 @@
>    +[defaults]
>    +interpreter_python = /usr/bin/python3
>    +inventory = hosts
>    +# Use the YAML callback plugin.
>    +stdout_callback = yaml
>    +# Use the stdout_callback when running ad-hoc commands.
>    +bin_ansible_callbacks = True
>    +
>    +# Show diffs of changes
>    +[diff]
>    +always = True
>    {% endraw %}
>    ```
>    {: data-commit="Add ansible.cfg"}
>
>    > <tip-title>CentOS7</tip-title>
>    > As mentioned in the "Ubuntu or Debian, CentOS or RHEL?" comment above, if you are using CentOS7 do not set `interpreter_python` in `ansible.cfg` .
>    {: .tip}
>
>    There is an additional useful option that you might want to add to your `ansible.cfg` file if you are connecting over SSH:
>
>    ```ini
>    [ssh_connection]
>    pipelining = true
>    ```
>
>    Pipelining will make [Ansible run faster](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html#ansible-pipelining) by significantly reducing the number of new SSH connections that must be opened.
>
> 2. Create the `hosts` inventory file if you have not done so yet, defining a `[galaxyservers]` group with the address of the host where you want to install Galaxy. If you are running Ansible on the same machine where Galaxy will be installed to, you should set the `ansible_connection=local` variable. Lastly, you should explicitly set the `ansible_user` variable to the username to use when connecting to the server. Ansible has changed its behaviour over time regarding whether or not `ansible_user` is defined, and it is most effective to define it explicitly even when it can sometimes be inferred.
>
>    You should also define a `[dbservers]` group for the hosts that will run Galaxy's database. In many cases (and in the case of the Galaxy Admin Training), these will be the same host, but the tutorial is designed to support either scenario, and helps to strengthen understanding of the purpose of host groupings in Ansible for more advanced real-world deployments. Here however we suggest using `[dbservers:children]` and adding `galaxyservers` as the child of that.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat hosts
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > Your hostname is probably different:
>    > >
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,4 @@
>    > > +[galaxyservers]
>    > > +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    > > +[dbservers:children]
>    > > +galaxyservers
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add hosts"}
>    > {: .code-out}
>    {: .code-2col}
>
>    > <tip-title>Separate database server</tip-title>
>    > In the hosts file above, we have defined the `galaxyservers` group to be a "child" of the `dbservers` group. This means that the host defined in `[galaxyservers]` is also a member of `[dbservers]`. If we instead wanted to define a separate host as the Galaxy database server, we would replace the `[dbservers:children]` section with something like:
>    >
>    > ```ini
>    > [dbservers]
>    > galaxy-db.example.org
>    > ```
>    {: .tip}
>
> 3. Create the `group_vars` directory and create the file `group_vars/all.yml`. These variables are applied to *all* hosts, regardless of group membership. The variables we set will be used to ensure Python 3 is always used on older systems where Python 2 might also still be installed. We will also set a few variables that must be known by both the Galaxy server and the database server: Galaxy's system/database username, as well as the database name.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat group_vars/all.yml
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/group_vars/all.yml
>    > > @@ -0,0 +1,7 @@
>    > > +---
>    > > +# Python 3 support
>    > > +pip_virtualenv_command: /usr/bin/python3 -m venv  # usegalaxy_eu.certbot, usegalaxy_eu.tiaas2, galaxyproject.galaxy
>    > > +
>    > > +# Common variables needed by all hosts
>    > > +galaxy_user_name: galaxy
>    > > +galaxy_db_name: galaxy
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add initial all hosts variables file"}
>    > {: .code-out}
>    {: .code-2col}
>
{: .hands_on}

## PostgreSQL

Galaxy is capable of talking to multiple databases through SQLAlchemy drivers. SQLite is the development database, but PostgreSQL (/ˈpoʊstɡrɛs ˌkjuː ˈɛl/, POHST-gres kyoo el) is recommended in production. MySQL is a possibility, but does not receive the same testing or bugfixes from the main development team as PostgreSQL, so we will only show installation with PostgreSQL.

PostgreSQL maintains its own user database apart from the system user database. By default, PostgreSQL uses the "peer" authentication method which allows access for system users with matching PostgreSQL usernames (other authentication mechanisms are available, see the [PostgreSQL Client Authentication documentation](https://www.postgresql.org/docs/current/static/client-authentication.html).

For this tutorial, we will use the default "peer" authentication, so we need to create a PostgreSQL user matching the system user under which Galaxy will be running, i.e. `galaxy`. This is normally done with the PostgreSQL `createuser` command, and it must be run as the `postgres` user. In our case, we will use the `galaxyproject.postgresql_objects` role to handle this step.
> <hands-on-title>Installing PostgreSQL</hands-on-title>
>
> 1. Create and edit `group_vars/dbservers.yml` and add some variables to configure PostgreSQL:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/dbservers.yml
>    @@ -0,0 +1,7 @@
>    +---
>    +# PostgreSQL
>    +postgresql_objects_users:
>    +  - name: "{{ galaxy_user_name }}"
>    +postgresql_objects_databases:
>    +  - name: "{{ galaxy_db_name }}"
>    +    owner: "{{ galaxy_user_name }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add initial dbservers group variables file"}
>
>    Notice that we have used the variables we defined in `group_vars/all.yml` for the Galaxy user and database names in the previous section, implementing the principle of "define once, reference many times."
>
>    > <tip-title>Using postgres via the network</tip-title>
>    > To use postgres via another machine, or via the network, you can add lines like the following:
>    >
>    > ```yaml
>    > postgresql_pg_hba_conf:
>    > - host all all 127.0.0.1/32 md5
>    > ```
>    >
>    > Here you should either add multiple lines per IP address that needs access to the Postgres server (with netmask `/32`), or a less specific IP range defined also [via netmask.](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#IPv4_CIDR_blocks)
>    >
>    > Additionally you'll need to change the `postgresql_objects_users` statement to include a password (maybe stored in a vault, discussed later!)
>    >
>    > ```yaml
>    > postgresql_objects_users:
>    >   - name: galaxy
>    >     password: "{{ vault_postgres_galaxy_password }}"
>    > ```
>    {: .tip}
>
> 2. Create and open `galaxy.yml` which will be our playbook. Write the following:
>
>    - Create a play that will run on the `dbservers` group, as the root user (you will need `become`/`become_user`)
>    - Add a pre-task to install the necessary dependency at this stage: `acl`
>    - Use the role `galaxyproject.postgresql`. This will handle the installation of PostgreSQL.
>    - Use the role `galaxyproject.postgresql_objects`, run as the postgres user. This role allows for managing users and databases within PostgreSQL.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/galaxy.yml
>    @@ -0,0 +1,13 @@
>    +---
>    +- hosts: dbservers
>    +  become: true
>    +  become_user: root
>    +  pre_tasks:
>    +    - name: Install Dependencies
>    +      package:
>    +        name: 'acl'
>    +  roles:
>    +    - galaxyproject.postgresql
>    +    - role: galaxyproject.postgresql_objects
>    +      become: true
>    +      become_user: postgres
>    {% endraw %}
>    ```
>    {: data-commit="Add initial galaxy playbook"}
>
>    > <question-title></question-title>
>    >
>    > What should we expect to see in our folder?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - a playbook
>    > > - a hosts file
>    > > - a requirements file
>    > > - some group variables
>    > > - all of our roles
>    > >
>    > > > <code-in-title>Bash</code-in-title>
>    > > > Let's investigate our directory.
>    > > > ```bash
>    > > > tree -L 2
>    > > > ```
>    > > {: .code-in}
>    > >
>    > > > <code-out-title>Bash</code-out-title>
>    > > >
>    > > > Your directory should look approximately like this:
>    > > >
>    > > > ```ini
>    > > > .
>    > > > ├── ansible.cfg
>    > > > ├── galaxy.yml
>    > > > ├── group_vars
>    > > > │   ├── all.yml
>    > > > │   └── dbservers.yml
>    > > > ├── hosts
>    > > > ├── requirements.yml
>    > > > └── roles
>    > > >     ├── galaxyproject.galaxy
>    > > >     ├── galaxyproject.miniconda
>    > > >     ├── galaxyproject.nginx
>    > > >     ├── galaxyproject.postgresql
>    > > >     ├── galaxyproject.postgresql_objects
>    > > >     └── usegalaxy_eu.certbot
>    > > >
>    > > > 8 directories, 6 files
>    > > > ```
>    > > {: .code-out.code-max-300}
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>    {% snippet topics/admin/faqs/ansible_role-prefix.md %}
>
>    {% snippet topics/admin/faqs/ansible_yaml.md %}
>
> 3. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > Note: we have stripped out the "diff" output for compactness, your output will be more verbose.
>    >
>    > ```ini
>    > PLAY [dbservers] ***************************************************************
>    >
>    > TASK [Gathering Facts] *********************************************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [Install Dependencies] ****************************************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : include_tasks] ********************************
>    > included: /home/ubuntu/galaxy/roles/galaxyproject.postgresql/tasks/debian.yml for gat-0.eu.galaxy.training
>    >
>    > TASK [galaxyproject.postgresql : APT keyrings directory] ***********************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install pgdg package signing key (Debian/pgdg)] ***
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install pgdg repository (Debian/pgdg)] ********
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install PostgreSQL (Debian)] ******************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Get installed version] ************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set version fact] *****************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install psycopg2] *****************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set version fact] *****************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set OS-specific variables] ********************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set pgdata fact] ******************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set conf dir fact] ****************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : include_tasks] ********************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Create conf.d] ********************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Check for conf.d include in postgresql.conf] ***
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set conf.d include in postgresql.conf] ********
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Include 25ansible_postgresql.conf in postgresql.conf] ***
>    > [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created
>    > with a mode of 0700, this may cause issues when running as another user. To
>    > avoid this, create the remote_tmp dir with the correct permissions manually
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set config options] ***************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install pg_hba.conf] **************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : include_tasks] ********************************
>    > included: /home/ubuntu/galaxy/roles/galaxyproject.postgresql/tasks/backup.yml for gat-0.eu.galaxy.training
>    >
>    > TASK [galaxyproject.postgresql : Create backup directories] ********************
>    > changed: [gat-0.eu.galaxy.training] => (item=/var/lib/postgresql/backups)
>    > changed: [gat-0.eu.galaxy.training] => (item=/var/lib/postgresql/backups/bin)
>    >
>    > TASK [galaxyproject.postgresql : Create backup output directory] ***************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install backup script templates] **************
>    > changed: [gat-0.eu.galaxy.training] => (item=archive_wal.sh)
>    >
>    > TASK [galaxyproject.postgresql : Install backup script files] ******************
>    > changed: [gat-0.eu.galaxy.training] => (item=backup.py)
>    >
>    > TASK [galaxyproject.postgresql : Set WAL archive config options] ***************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Schedule backups] *****************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Remove PostgreSQL working WAL backup cron job] ***
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Ensure PostgreSQL is running] *****************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Warn if deprecated user priv param is set] ***
>    > skipping: [gat-0.eu.galaxy.training] => (item=galaxy)
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Revoke extra privileges] **************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Drop databases] ***********************
>    > skipping: [gat-0.eu.galaxy.training] => (item={'name': 'galaxy', 'owner': 'galaxy'})
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Create and drop users] ****************
>    > [WARNING]: Module remote_tmp /var/lib/postgresql/.ansible/tmp did not exist and
>    > was created with a mode of 0700, this may cause issues when running as another
>    > user. To avoid this, create the remote_tmp dir with the correct permissions
>    > manually
>    > changed: [gat-0.eu.galaxy.training] => (item=galaxy)
>    >
>    > TASK [galaxyproject.postgresql_objects : Create groups] ************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Add or remove users from groups] ******
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Drop groups] **************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Create databases] *********************
>    > changed: [gat-0.eu.galaxy.training] => (item=galaxy)
>    >
>    > TASK [galaxyproject.postgresql_objects : Grant extra privileges] ***************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > RUNNING HANDLER [galaxyproject.postgresql : Reload PostgreSQL] *****************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > PLAY RECAP *********************************************************************
>    > gat-0.eu.galaxy.training   : ok=27   changed=15   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
>    > ```
>    {: .code-out.code-max-300}
>
>    > <comment-title>Error: `postgresql_version is version_compare('9.3', '>=')' failed`</comment-title>
>    > You might see an error like this, if you're running with `--check` mode. Unfortunately here `--check` mode meets real life: not all modules support it because some rely on running command line tools to obtain version numbers, something that cannot be mocked with check mode. You can solve this by running it without `--check`.
>    >
>    > ```
>    > fatal: [gat-0.eu.galaxy.training]: FAILED! => {"msg": "The conditional check 'postgresql_version is version_compare('9.3', '>=')' failed. The error was: Input version value cannot be empty\n\nThe error appears to be in '/home/ubuntu/galaxy/roles/galaxyproject.postgresql/tasks/main.yml': line 42, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# etc.). So check for a match first and then add if there's no match.\n- name: Check for conf.d include in postgresql.conf\n ^ here\n"}
>    > ```
>    {: .comment}
>
>    > <comment-title>When running Ansible</comment-title>
>    > Always pay close attention to tasks reported as **changed** and ensure that the changes were expected!
>    {: .comment}
>
>    > <tip-title>Why didn't we use `-i` in our ansible command?</tip-title>
>    > In our `ansible.cfg` file we specified the inventory was stored in a file called `hosts`:
>    >
>    > ```ini
>    > [defaults]
>    > interpreter_python = /usr/bin/python3
>    > inventory = hosts
>    > retry_files_enabled = false
>    > ```
>    {: .tip}
>
>    {% snippet topics/admin/faqs/ansible_error-no-hosts-matched.md %}
>
> 4. Inspect the changes that have been made on your Galaxy server. Places to look include:
>
>    - `/etc/postgresql`
>    - Databases and users in PostgreSQL.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > List available databases.  You can now login and access the database, but only as the `postgres` user. You will need to `sudo -iu postgres` first, and then you can run `psql galaxy`.
>    > > ```bash
>    > > sudo -iu postgres psql -l
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > ```ini
>    > >                               List of databases
>    > >    Name    |  Owner   | Encoding | Collate |  Ctype  |   Access privileges
>    > > -----------+----------+----------+---------+---------+-----------------------
>    > >  galaxy    | galaxy   | UTF8     | C.UTF-8 | C.UTF-8 |
>    > >  postgres  | postgres | UTF8     | C.UTF-8 | C.UTF-8 |
>    > >  template0 | postgres | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
>    > >            |          |          |         |         | postgres=CTc/postgres
>    > >  template1 | postgres | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
>    > >            |          |          |         |         | postgres=CTc/postgres
>    > > (4 rows)
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
>    > <code-in-title>Bash</code-in-title>
>    > Access postgres and list some users
>    > ```bash
>    > sudo -iu postgres psql
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    > The database will currently be empty (check relations with \d and list of roles with \dg) as Galaxy has not yet connected to it. Once you install Galaxy in the next step, the database will be populated. (Quit database with \q)
>    > ```ini
>    > psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
>    > Type "help" for help.
>    >
>    > postgres=# \d
>    > Did not find any relations.
>    > postgres=# \dg
>    >                                    List of roles
>    >  Role name |                         Attributes                         | Member of
>    > -----------+------------------------------------------------------------+-----------
>    >  galaxy    |                                                            | {}
>    >  postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
>    >
>    > postgres=# \q
>    > ```
>    {: .code-out}
>
>    > > <code-in-title>Bash</code-in-title>
>    > > Inspect the postgres directory
>    > > ```bash
>    > > ls /etc/postgresql/*/main/
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > ```ini
>    > > conf.d
>    > > environment
>    > > pg_ctl.conf
>    > > pg_hba.conf
>    > > pg_hba.conf.3350.2020-07-08@11:51:47~
>    > > pg_ident.conf
>    > > postgresql.conf
>    > > start.conf
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
{: .hands_on}


## Galaxy

Next we will dive right in to deploying a copy of Galaxy onto our server.

For a normal Galaxy instance there are a few configuration changes you make very early during deployment:

- Changing the database connection
- Configuring the admin user list

Additionally we'll go ahead and set up the production-ready [Gunicorn + Webless](https://docs.galaxyproject.org/en/master/admin/scaling.html#deployment-options) deployment, which separates Galaxy's web and job handling into separate processes. This is done by configuring Galaxy's process manager, [Gravity](https://gravity.readthedocs.io).

Finally, best admin practices are to not run Galaxy as a user with `sudo` access, like your login user probably has. Additionally, it is best to install the Galaxy code and configs as a separate user, for security purposes. So we will instruct the `galaxyproject.galaxy` role to create a new user account specifically to run Galaxy under.

The configuration is quite simple thanks to the many sensible defaults that are provided in the Ansible roles.

> <hands-on-title>Minimal Galaxy Playbook</hands-on-title>
>
> 1. Open `galaxy.yml` with your text editor and do the following:
>
>    - Create a new play that will run on the `galaxyservers` group, as the root user (you will need `become`/`become_user`)
>    - Add a pre-task to install the necessary dependencies for the Galaxy server: `acl`, `bzip2`, `git`, `make`, `tar`, `python3-venv`, and `python3-setuptools`
>    - Use the roles `galaxyproject.galaxy` and `galaxyproject.miniconda` (in this order), with `galaxyproject.miniconda` run as the `galaxy` user.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -11,3 +11,16 @@
>         - role: galaxyproject.postgresql_objects
>           become: true
>           become_user: postgres
>    +
>    +- hosts: galaxyservers
>    +  become: true
>    +  become_user: root
>    +  pre_tasks:
>    +    - name: Install Dependencies
>    +      package:
>    +        name: ['acl', 'bzip2', 'git', 'make', 'tar', 'python3-venv', 'python3-setuptools']
>    +  roles:
>    +    - galaxyproject.galaxy
>    +    - role: galaxyproject.miniconda
>    +      become: true
>    +      become_user: "{{ galaxy_user_name }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add miniconda, galaxy to playbook"}
>
> 2. Create and edit `group_vars/galaxyservers.yml` and add some variables to configure Galaxy:
>
>    We need to set the following variables at the top level:
>
>    Variable                        | Value                                                                     | Purpose
>    ---                             | -----                                                                     | ---
>    `galaxy_create_user`            | `true`                                                                    | Instruct the role to create a Galaxy user
>    `galaxy_separate_privileges`    | `true`                                                                    | Enable separation mode to install the Galaxy code as `root` but run the Galaxy server as `galaxy`
>    `galaxy_manage_paths`           | `true`                                                                    | Instruct the role to create the needed directories.
>    `galaxy_layout`                 | `root-dir`                                                                | This enables the `galaxy_root` Galaxy deployment layout: all of the code, configuration, tools, and mutable-data (like caches, location files, etc.) folders will live by default beneath `galaxy_root`. User data is stored under `file_path`, a variable we will set later.
>    `galaxy_root`                   | `/srv/galaxy`                                                             | This is the root of the Galaxy deployment.
>    `galaxy_user`                   | {% raw %}`{name: "{{ galaxy_user_name }}", shell: /bin/bash}`{% endraw %} | The user that Galaxy will run as.
>    `galaxy_commit_id`              | `release_25.0`                                                            | The git reference to check out, which in this case is the branch for Galaxy Release 23.0
>    `galaxy_force_checkout`         | `true`                                                                    | If we make any modifications to the Galaxy codebase, they will be removed. This way we know we're getting an unmodified Galaxy and no one has made any unexpected changes to the codebase.
>    `galaxy_client_use_prebuilt`         | `true`                                                              | We are going to use a prebuilt client to save some time during this tutorial, if you want you can set it to false later and see how galaxy starts building the client itself.
>    `miniconda_prefix`              | {% raw %}`"{{ galaxy_tool_dependency_dir }}/_conda"`{% endraw %}          | We will manually install conda as well. Normally Galaxy will attempt to auto-install this, but since we will set up a production-ready instance with multiple handlers, there is the chance that they can become deadlocked.
>    `miniconda_version`             | `25.3`                                                                    | Install a specific miniconda version, the latest one at the time of writing that was tested and working.
>    `miniconda_distribution`             | `miniforge`                                                                    | Install a specific distribution of conda, see https://github.com/galaxyproject/ansible-miniconda for alternatives. This determines the default value for the `miniconda_executable` (`conda` for `miniforge`)
>    `miniconda_channels`          ` | `['conda-forge']`                                             | Use the free and community-maintained `conda-forge` channel.
>
>    > <tip-title>Different Galaxy Releases!</tip-title>
>    > In the time between this tutorial was last updated ({{ page.last_modified_at | date: "%Y-%m-%d" }}), and when you are now reading it, one or more new releases of Galaxy may have occured.
>    >
>    > If you wish to use a different version of Galaxy than the one listed, you can do so, but there is no guarantee this tutorial will work without modifications.
>    >
>    > Please ensure you consult the [releases documentation](https://docs.galaxyproject.org/en/master/releases/index.html)
>    {: .tip}
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/galaxyservers.yml
>    @@ -0,0 +1,15 @@
>    +# Galaxy
>    +galaxy_create_user: true # False by default, as e.g. you might have a 'galaxy' user provided by LDAP or AD.
>    +galaxy_separate_privileges: true # Best practices for security, configuration is owned by 'root' (or a different user) than the processes
>    +galaxy_manage_paths: true # False by default as your administrator might e.g. have root_squash enabled on NFS. Here we can create the directories so it's fine.
>    +galaxy_manage_systemd: true
>    +galaxy_layout: root-dir
>    +galaxy_root: /srv/galaxy
>    +galaxy_user: {name: "{{ galaxy_user_name }}", shell: /bin/bash}
>    +galaxy_commit_id: release_25.0
>    +galaxy_client_use_prebuilt: true
>    +galaxy_force_checkout: true
>    +miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
>    +miniconda_version: 25.3
>    +miniconda_distribution: miniforge
>    +miniconda_channels: ['conda-forge']
>    {% endraw %}
>    ```
>    {: data-commit="Configure miniforge and galaxy"}
>
>    > <tip-title>Following this training outside of a GAT course?</tip-title>
>    > Consider updating the Galaxy Commit ID to the latest version of Galaxy available, this will probably give better results (given that dependencies are always updating) than using a potentially outdated version.
>    {: .tip}
>
> 3. Again edit the galaxyservers group variables file and add a variable for `galaxy_config`. It will be a hash with one key, `galaxy` which will also be a hash. Inside here you can place all of your Galaxy configuration.
>
>    So the structure looks like:
>    ```yaml
>    galaxy_config:
>      galaxy:
>        key: value
>    ```
>
>    Now you should set:
>    1. `admin_users` to the email address you will use with this Galaxy.
>    3. `database_connection` to point to the database you setup earlier (`postgresql:///{{ galaxy_db_name }}?host=/var/run/postgresql`).
>    4. `file_path` to a place to store data, `/data` is fine for this lesson which sets up a single-node Galaxy. If you have separate compute machines, this will normally need to be storage shared between the Galaxy node and compute nodes.
>    5. `tool_data_path` to {% raw %}`{{ galaxy_mutable_data_dir }}/tool-data`{% endraw %}, so that when tools are installed, due to privilege separation, this will happen in a directory Galaxy can actually write into.
>    6. `object_store_store_by` to `uuid`, this is a better way of storing files that will ensure better filesystem balancing than the older system.
>    7. `id_secret` to {% raw %}`{{ vault_id_secret }}`{% endraw %}, we'll define this variable next but it will be used to encode the IDs used in Galaxy URLs and for securing session cookies.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -13,3 +13,17 @@ miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
>     miniconda_version: 25.3
>     miniconda_distribution: miniforge
>     miniconda_channels: ['conda-forge']
>    +
>    +galaxy_config:
>    +  galaxy:
>    +    # Main Configuration


topic: admin
tutorial: tool-management
title: Galaxy Tool Management with Ephemeris
difficulty: intermediate
time_estimation: 45m
tools_json: []
requirements_json: []
tags_json: ["tools", "git-gat"]
content_hash: 4d3b81e1ab8171b6d7e1dc1e6f762080
last_modified: 2026-02-26T21:12:14.166917
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/tool-management/tutorial.html

content:
This tutorial will introduce you to one of Galaxy's associated projects - [Ephemeris](https://ephemeris.readthedocs.io/). Ephemeris is a small Python library and set of scripts for managing the bootstrapping of Galaxy plugins - tools, index data, and workflows. It aims to help automate, and limit the quantity of manual actions admins have to do in order to maintain a Galaxy instance.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tool-management" %}

# Background

You are an administrator of a Galaxy server. A colleague has approached you with a request to run a specific [Galaxy workflow]({% link topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga %}) on their data. In order to enable this workflow for your users, you will have to:

- identify what tools are required for the workflow
- install these tools and their dependencies on your Galaxy instance.


# Requirements

To run this tutorial, you will need to [install Ephemeris](https://ephemeris.readthedocs.io/en/latest/installation.html). You would normally install it on your workstation, but during training courses we recommend to install it on the same virtual machine used for the Galaxy server.

> <hands-on-title>Installing Ephemeris in a Python virtual environment</hands-on-title>
>
> 1. Install the Python `venv` package if it is not already available. On Ubuntu this can be done with `sudo apt install python3-venv`
> 2. Create a virtual environment just for ephemeris, activate it and install ephemeris inside it:
>    ```console
>    python3 -m venv ~/ephemeris_venv
>    . ~/ephemeris_venv/bin/activate
>    pip install ephemeris
>    ```
{: .hands_on}

# Extracting Tools

A common request you will experience as an administrator is "I want to run this workflow". Since this is such a common workflow, Galaxy has a built in way to accomplish it. We can use Ephemeris to extract a list of tools from a Galaxy workflow document, and then use Ephemeris to install these tools and specific versions into your Galaxy.

However, Galaxy workflow files are complex JSON documents, and the process of mapping the tool IDs to a ToolShed repository and revision is not trivial. Workflow files contain tool IDs which look like:

```
toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.71
toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.4.3.1
toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.6
toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.4.2
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1
toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1
```

In order to actually install these tools, we need to convert each tool ID into a ToolShed repository name and revision. For example, FastQC version 0.71 corresponds to revision `ff9530579d1f` in the ToolShed.

```yaml
- name: fastqc
  owner: devteam
  revisions:
  - ff9530579d1f
  tool_panel_section_label: Tools from workflows
  tool_shed_url: https://toolshed.g2.bx.psu.edu
```

Ephemeris can take care of this process. Let's practice this on a real workflow.

> <hands-on-title>Extracting a list of tools from a workflow</hands-on-title>
>
> 1. Download the mapping workflow:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > wget {{ site.url }}{% link topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga %}
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Use the Ephemeris [`workflow-to-tools`](https://ephemeris.readthedocs.io/en/latest/commands/workflow-to-tools.html) command to extract the tool list from this workflow into a file named `workflow_tools.yml` in the folder `tools`.
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > > <code-in-title>Bash</code-in-title>
>    > > > ```bash
>    > > > workflow-to-tools -w mapping.ga -o tools/workflow_tools.yml -l Mapping
>    > > > ```
>    > > > {: data-cmd="true"}
>    > > >
>    > > > Or as a diff:
>    > > > {% raw %}
>    > > > ```diff
>    > > > --- /dev/null
>    > > > +++ b/tools/workflow_tools.yml
>    > > > @@ -0,0 +1,41 @@
>    > > > +install_tool_dependencies: True
>    > > > +install_repository_dependencies: True
>    > > > +install_resolver_dependencies: True
>    > > > +
>    > > > +tools:
>    > > > +- name: fastqc
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - e7b2202befea
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: trim_galore
>    > > > +  owner: bgruening
>    > > > +  revisions:
>    > > > +  - 949f01671246
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: multiqc
>    > > > +  owner: iuc
>    > > > +  revisions:
>    > > > +  - f7985e0479b9
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: bowtie2
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - 09b2cdb7ace5
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: samtools_stats
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - 24c5d43cb545
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: bamtools_filter
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - cb20f99fd45b
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > {% endraw %}
>    > > > ```
>    > > > {: data-commit="Add a tool list to install"}
>    > > {: .code-in}
>    > {: .solution }
>    {: .question}
>
> 3. Inspect the `workflow_tools.yml` file, which contains a tool list in YAML format.
>
{: .hands_on }

# Installing Tools

Now that you have extracted a list of tools, let's install these on your Galaxy instance. In order to accomplish this, you will need:

- The URL of your Galaxy server
- The API key for your account, which must be an admin

{% snippet faqs/galaxy/preferences_admin_api_key.md admin=true %}

There are two ways to install tools, depending on how you specify the tools to install:

> <hands-on-title>Installing a single tool</hands-on-title>
>
> 1. Use the Ephemeris [`shed-tools`](https://ephemeris.readthedocs.io/en/latest/commands/shed-tools.html) command to install the tool `bwa`, owned by `devteam` into a section named `Mapping`
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > Use your Galaxy URL and API key in the example command below:
>    > >
>    > > ```bash
>    > > shed-tools install -g https://galaxy.example.org -a <api-key> --name bwa --owner devteam --section_label Mapping
>    > > ```
>    > > {: data-cmd="true"}
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <tip-title>Certificate issues (GAT, uncommon)</tip-title>
>
> If your Galaxy instance is served via the HTTPS protocol (as it should be!), ephemeris will use the [requests](https://requests.readthedocs.io) Python library to encrypt the communication with Galaxy. Therefore, if your Galaxy uses a self-signed SSL certificate, `shed-tools` may fail with a `CERTIFICATE_VERIFY_FAILED` error.
>
> Under Ubuntu, you can allow the use of the unrecognized certificate as follows:
> 1. Get hold of the Certificate Authority (CA) certificate used to sign your Galaxy SSL certificate. For a [Galaxy Admin Training](https://github.com/galaxyproject/dagobah-training) course, this is usually the [Fake LE Root X1 certificate](https://letsencrypt.org/certs/fakelerootx1.pem).
> 2. Copy the CA certificate file into `/usr/local/share/ca-certificates/` with a `.crt` extension.
> 3. Run `update-ca-certificates` as root.
> 4. Execute `export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt`, as explained in [requests docs](https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification).
>
> Now you should be able to execute successfully the `shed-tools` commands.
{: .tip}


This provides an easy way to do a one-off installation of a tool, but is not very convenient if you want to install many tools.
For that, you can install from a YAML file:

> <hands-on-title>Installing tools from a tool list</hands-on-title>
>
> 1. (optional) Watch the installation proceed by running `journalctl -f` in a separate remote shell.
>
> 2. Use the Ephemeris [`shed-tools`](https://ephemeris.readthedocs.io/en/latest/commands/shed-tools.html) command to install all of the tools from the `workflow_tools.yml` file on your Galaxy.
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > Use your Galaxy URL and API key in the example command below:
>    > >
>    > > ```bash
>    > > shed-tools install -g https://galaxy.example.org -a <api-key> -t workflow_tools.yml
>    > > ```
>    > > {: data-cmd="true"}
>    > {: .solution}
>    {: .question}
>
> 4. Open your Galaxy's admin interface and check that the tools have been installed.
>
> 5. Using the UI import the workflow file that you used, [mapping.ga]({% link topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga %}).
>    1. Right-click or Ctrl-click on the link above and copy the link address
>    2. On your Galaxy instance click on `Workflows`, then `Import`.  Paste the URL into the `Import Archived URL` field.
{: .hands_on}

Occasionally the tool installation may fail due to network issues; if it does, just re-run the `shed-tools` installation process until it succeeds. This is a known issue the developers are working on.

{% snippet topics/admin/faqs/byobu-shortcuts.md %}

> <tip-title>Can I install tools without restarting?</tip-title>
> Yes. The default tool config (`config/tool_conf.xml.sample`, copy to `config/tool_conf.xml` to modify) has an option, `monitor="true"` set in the root `<toolbox>` tag. This instructs Galaxy to watch the tool files referenced in that config and load or reload them as necessary. It will also add any tools you have added.
{: .tip}

> <tip-title>Can I install tools without a ToolShed?</tip-title>
> Yes. The `galaxy_local_tools` option for the `galaxyproject.galaxy` Ansible role can be used to install local tools, or you can manage them in another way that fits your workflow better. UseGalaxy.eu, for example, maintains a repository of tools that are not installed from the ToolShed to aid their local developers. This is deployed to the server using the `git` module, rather than the Galaxy Ansible role.
{: .tip}


# Tool Testing

Having the tools installed is a good first step, but your users will expect that they actually work as well. You can use Ephemeris to automatically test all of the installed tools.

> <hands-on-title>Test the installed tools</hands-on-title>
>
> 1. Use the Ephemeris [`shed-tools`](https://ephemeris.readthedocs.io/en/latest/commands/shed-tools.html#test) command to test the `bamtools_filter` tool on your Galaxy.
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > Use your Galaxy URL and API key in the example command below:
>    > >
>    > > ```bash
>    > > shed-tools test -g https://galaxy.example.org -a <api-key> --name bamtools_filter --owner devteam
>    > > ```
>    > > {: data-cmd="true"}
>    > {: .solution}
>    {: .question}
> 2. Shed-tools test outputs a file with details of `tool_test_output.json` with details of jobs that have run.  Have a look at this file.
{: .hands_on}

This can give you some more confidence that things are working correctly. Oftentimes, users provide workflows for biological domains that we are not familiar with, so knowing how we can test these tools is impossible for us as admins. Leveraging the built-in tool test cases can give you reassurance that things are functional before you inform your users of the new tools.

> <tip-title>Tool test reports</tip-title>
> The ephemeris `shed-tools test` command produces an output file `tool_test_output.json` with information about the test jobs that have run. Galaxyproject's [Planemo](https://github.com/galaxyproject/planemo) can be used to generate [test reports](https://planemo.readthedocs.io/en/latest/commands/test_reports.html?highlight=test_reports#test-reports-command) from tool_test_output.json in HTML and other formats.
{: .tip}


# Obtaining a Tool List

Sometimes a user might ask you to install all the tools they were previously using on another Galaxy instance. Ephemeris can produce a `tool_list.yaml` file for all the tools installed on a server.

> <hands-on-title>Obtain UseGalaxy.eu's tool list</hands-on-title>
>
> 1. Use the Ephemeris [`get-tool-list`](https://ephemeris.readthedocs.io/en/latest/commands/get-tool-list.html) command to obtain the full set of tools installed on UseGalaxy.eu
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > This command does not require authentication and can be used to obtain the tool list from any public Galaxy server:
>    > >
>    > > > <code-in-title>Bash</code-in-title>
>    > > > ```bash
>    > > > get-tool-list -g "https://usegalaxy.eu" -o "eu_tool_list.yaml"
>    > > > ```
>    > > > {: data-cmd="true"}
>    > > {: .code-in}
>    > {: .solution}
>    {: .question}
> 2. Inpect the first few lines of tool list: Run `head -n 20 eu_tool_list.yaml`.
{: .hands_on}

We will not install all the tools from the EU Galaxy server as that server likely has more tools than any other Galaxy instance, but it is useful as an example of how you can use Ephemeris to facilitate the mirroring of another Galaxy instance.

> <tip-title>Non-shed tools</tip-title>
> The output of `get-tool-list` only includes ToolShed tools, not local non-TS tools.
{: .tip}

> <tip-title>But how many tools is it really?</tip-title>
> If you've seen the [European Galaxy tools view](https://usegalaxy.eu/tools/view) (this is available on any galaxy! Just access `/tools/view`)
> you'll notice they report somewhere over 2700 tools, however the `get-tool-list` output lists significantly fewer tools.
>
> This is for a couple reasons:
> - That's the number of repositories that are installed, and some repositories include multiple tools (e.g. [`circos`](https://toolshed.g2.bx.psu.edu/view/iuc/circos/df7356989ac1) has quite a few)
> - This only lists ToolShed tools, while EU also includes a number of non-TS tools
> - There are a number of tools built into Galaxy (e.g. the Collection Operation tools)
{: .tip}

# Production Best Practices

The servers which are part of the `usegalaxy.*` network use Ephemeris extensively to manage their large tool sets.

Interestingly, UseGalaxy.eu and UseGalaxy.org.au have different approaches:
- EU maintains [YAML files roughly per domain](https://github.com/usegalaxy-eu/usegalaxy-eu-tools), it is not as clear of an ordering. They maintain a YAML file where humans add the tools which should be installed in a given category, and lock files are automatically generated from these with the latest revision if it is missing. They follow a cycle of updating the lock files with the latest available revisions of tools, and then installing from these lock files any missing revisions. They use a [Jenkins server](https://build.galaxyproject.eu/job/usegalaxy-eu/job/install-tools/) to automatically run tool installation weekly.
- AU maintains [a separate YAML file per tool panel section](https://github.com/usegalaxy-au/usegalaxy-au-tools/tree/master/usegalaxy.org.au) as a record of all installed tools on the server.  They accept tool requests as pull requests and a Jenkins server is notified when a pull request is merged so that the tool can be automatically installed and tested. Like EU, they run automatic updates of installed tools once a week.
- Together, `usegalaxy.*` are working on a collaborative approach at [galaxyproject/usegalaxy-tools](https://github.com/galaxyproject/usegalaxy-tools) but this is not consumption ready yet.

If running ephemeris directly is not your preference, there is an Ansible [role](https://github.com/galaxyproject/ansible-galaxy-tools) and a sample [playbook](https://github.com/afgane/galaxy-tools-playbook) that can help automate some tasks.

> <tip-title>What if environments are not working</tip-title>
> It sometimes happens in Galaxy, that one environment isn't working anymore. It mostly happens from the start when it does happen. You can remove the environment on disk, or use the "Manage Dependencies" interface, select the environment, and delete it. Then re-install the dependency through the same Manage Dependencies interface.
{: .tip}

> <tip-title>Can you install multiple tools simultaneously?</tip-title>
> Previous experience with this is not good, there was a lot of unsafe code that would do things simultaneously that would destroy config files. With conda this is not recommended either, as this can corrupt conda environments. A solution for this is how [UseGalaxy.eu does it](https://github.com/usegalaxy-eu/usegalaxy-eu-tools), where they keep the full list of tools they want to install, and then a CI server (Jenkins) installs them. In this way they can enforce that only a single install process is running at any time.
{: .tip}

> <tip-title>Certificate Issues (GAT Only)</tip-title>
> If a student is running `shed-tools` on the VM, then it should work without certificate issues, because we installed the Fake LE X1 CA certificate, meaning that to your VM, the certificate chain is valid.
> We cannot (and would not recommend) setting that certificate on your local machine. That is the first way that comes to mind, that running an ephemeris command could generate that error.
{: .tip}

> <tip-title>Uninstalling tools</tip-title>
> While there is a function to accomplish this in [BioBlend](https://github.com/galaxyproject/bioblend), it has not been [included in Ephemeris yet.](https://github.com/galaxyproject/ephemeris/issues/83) If you're looking for a way to contribute to Galaxy, this would be great :)
{: .tip}

> <tip-title>"Recommended way" to install tools</tip-title>
> There is no one recommended way. Different people like to do different things. Some of us install things through the GUI still, some of us use ephemeris just for the automation.
>
> EU and others try and force all tool installation to go through ephemeris to make sure that rebuilding a server in case of disaster is easy. Anything not managed through their files will be lost.
>
> Others do a more mixed approach:
>
> > typically use Ephemeris, when I have to re-deploy a Galaxy instance (and thus install all tools) or during a Galaxy maintenance, when I want to update all installed toolshed tools in one go. Otherwise, when I need to install a specific version of one tool I use the GUI.
> {: .quote}
>
> There is no one right answer.
{: .tip}

> <tip-title>Tools not showing up? Refresh the toolbox!</tip-title>
> Sometimes the toolbox will fail to reload. You can correct for this by manually triggering the toolbox reload with a query:
>
> ```console
> curl -X PUT https://galaxy.example.org/api/configuration/toolbox -H "x-api-key: $GALAXY_API_KEY"
> ```
>
> This will request the toolbox to reload and you can check after if it's discovered your newly installed tools.
{: .tip}

<!--
{% snippet topics/admin/faqs/missed-something.md step=7 %}
-->

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tool-management" %}


topic: admin
tutorial: onedata-configuration
title: Configuring the Onedata connectors (remotes, Object Store, BYOS, BYOD)
difficulty: intermediate
time_estimation: 40m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["storage"]
content_hash: 13da590eec7f1e3cab25d0adc33d5f72
last_modified: 2026-02-26T21:12:14.150014
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/onedata-configuration/tutorial.html

content:
> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prerequisites

It's recommended that you have basic knowledge about Onedata and access 
to a Onedata ecosystem, so that you can test the configuration. If needed, follow 
[this tutorial]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html %})
first!


# Introduction

This tutorial will walk you through the configuration of four different types of Onedata 
connectors:

1. [Predefined Remote File Source](#predefined-remote-file-source) — provides all
   users with access to a predefined Onedata Space (e.g., for shared training
   datasets) without requiring individual configuration.

2. [BYOD (Remote File Source) templates](#byod-remote-file-source-templates) —
   similar to the Predefined Remote File Source, but uses templates and vault for secure
   credential storage, allowing users to configure multiple Onedata connections
   with their own credentials.

3. [Object Store — global Storage Location](#object-store---global-storage-location) — 
   configures Onedata as Galaxy's global storage backend (common for all users),
   where all datasets are stored in a specified Onedata Space.

4. [BYOS (Storage Location) templates](#byos-storage-location-templates) —
   enables users to configure their own Onedata Spaces as storage locations for
   their Galaxy datasets.


# Using Ansible?

The guides below pertain to files in the 
[Galaxy project codebase](https://github.com/galaxyproject/galaxy).
If you are using Ansible to manage your Galaxy deployment (which is the
recommended way for production instances), you will find tips at the end
of every subsection, showing how to achieve the same results with Ansible.

In case you are using jinja templates, make sure all of them are listed in
the `galaxy_config_templates` section for group_vars of each of the hosts.


# Common Configuration

Before configuring Onedata connectors, there are a few common elements that need
to be set up:

## Dependencies

For Remote File Sources (predefined and BYOD templates), install the
`fs.onedatarestfs` library (preferably adding it to the Python's requirements
file). Take a look at `lib/galaxy/dependencies/conditional-requirements.txt` for
the newest version:
```
fs.onedatarestfs>=21.2.5.2  # type: onedata, depends on onedatafilerestclient
```

For Object Store configuration, only `onedatafilerestclient` is required. However, 
if you're already installing `fs.onedatarestfs` for Remote File Sources, you don't 
need to add it separately as it's included as a dependency.

If you're only using Object Store functionality, you can add just:
```
onedatafilerestclient>=21.2.5.2
```

> <tip-title>Using Ansible</tip-title>
> Skip this step — conditional requirements should be collected automatically.
{: .tip}

## Vault Configuration

For BYOD and BYOS templates, you need to configure a vault to securely store user credentials:

1. Create `config/vault_conf.yml`:
```yaml
type: database
path_prefix: /galaxy
# Encryption keys must be valid fernet keys
# Generate them using Python:
# from cryptography.fernet import Fernet
# Fernet.generate_key().decode('utf-8')
encryption_keys:
  - KpVfbD7WfsNMxEnAEasql9aqSc1eCqR-S-tJCDyzcw8=
  - s1DDUEjZtFy2xknGIW2AiAE6EiZG5xSD0m_GzXi43G0=
  - qyb7f45l4XdkcuC5y06RlWtaENcPtUgHYekoQJBK1zM=
  # Do NOT use the keys above! Don't be lazy and generate your own.
```

The first key is used for encryption, while additional keys allow for key rotation. 

> <tip-title>Using Ansible</tip-title>
> Put the config in `templates/galaxy/config/vault_conf.yml.j2`, 
> like it has been done 
> [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/vault_conf.yml.j2).
>
> A `galaxy_vault_encryption_keys` var must be defined for the proper host in
> your playbook. For obvious reasons, it cannot be committed to your repository.
> For reference, take a look at [this PR](https://github.com/usegalaxy-eu/infrastructure-playbook/pull/1159).
{: .tip}

2. Enable it in `config/galaxy.yml`:
```yaml
galaxy:
  vault_config_file: vault_conf.yml
```

> <tip-title>Using Ansible</tip-title>
> Use the group vars file at `group_vars/gxconfig.yml`. 
> See [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
{: .tip}

For more details about vault configuration, see the [Galaxy Vault documentation](https://docs.galaxyproject.org/en/latest/admin/special_topics/vault.html).


# Predefined Remote File Source

While the [BYOD (Remote File Source) templates](#byod-remote-file-source-templates) 
allow users to configure their own Onedata access credentials, you can also set
up a predefined Onedata File Source that will be available to all users with
common configuration.

This approach is particularly useful when you want to provide access to shared
resources, such as public training datasets, without requiring user's access to
a Onedata ecosystem.

> <warning-title>Note the security of the credentials!</warning-title>
> This setup is perfect if you have a public Onedata access token that
> [can safely be shared](https://onedata.org/#/home/documentation/topic/stable/tokens-safe-publishing). If the token is not publicly shareable, use a vault to
> store it. Whatever the setup, always set the file source as `writable: false`,
> otherwise the users will be able to write (export) to the remote.
{: .warning}

Example configuration in `config/file_sources_conf.yml`:

```yaml
- type: onedata
  id: gtn_public_onedata
  label: GTN training data
  doc: Training data from the Galaxy Training Network (powered by Onedata)
  # The access token is public and can be shared
  onezone_domain: "datahub.egi.eu"
  writable: false
```

> <tip-title>Using Ansible</tip-title>
> Put the config in `templates/galaxy/config/file_sources_conf.yml.j2`, 
> like it has been done 
> [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/file_sources_conf.yml.j2).
{: .tip}

Note that you don't need to configure `user_preferences_extra_conf.yml` in this case, as all access parameters are specified directly in the File Sources configuration.


## Testing

Follow the Onedata 
[import]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html %})
guide.


# BYOD (Remote File Source) templates

BYOD (Bring Your Own Data) allows users to configure their own Onedata File
Sources using templates. This feature requires a vault for secure credential
storage, but users can create multiple File Sources with different
configurations, possibly connecting to different Onedata ecosystems.

1. Configure the vault as described in the [Vault Configuration section](#vault-configuration).

2. Enable the Onedata template by creating or modifying `config/file_source_templates.yml`:

   ```yaml
   - include: ./lib/galaxy/files/templates/examples/onedata.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Put the config in `templates/galaxy/config/file_source_templates.yml.j2`, 
   > like it has been done 
   > [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/file_source_templates.yml.j2).
   {: .tip}

3. Ensure there is the main Galaxy config file (`config/galaxy.yml`) and it includes the
   config file above:
   ```yaml
   galaxy:
     ...
     file_source_templates_config_file: file_source_templates.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Use the group vars file at `group_vars/gxconfig.yml`. 
   > See 
   > [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
   {: .tip}

This template defines a form that users will see when creating their own Onedata File Source. 
It collects the same configuration parameters that administrators specify when setting up 
a [Predefined Remote File Source](#predefined-remote-file-source) (like Onezone domain, access token, 
etc.), but allows users to provide their own values.

For required dependencies, see the [Dependencies section](#dependencies) in Common Configuration.


## Testing

Follow the Onedata 
[import]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html %})
and
[export]({% link topics/galaxy-interface/tutorials/onedata-remote-export/tutorial.html %})
guides.


# Object Store - global Storage Location

Galaxy supports various object store backends for storing datasets. While the [Object Store tutorial]({% link topics/admin/tutorials/object-store/tutorial.md %}) provides a comprehensive overview of the feature, this section focuses specifically on configuring Onedata as a global object store, available to all users.

To use Onedata as an object store:

1. Make sure there is a `config/object_store_conf.yml` configuration file with Onedata
   section. The relevant snippet can be found in config
   [samples](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/config/sample/object_store_conf.sample.yml).
   At the time of writing this tutorial, the config looks like the following:

   ```yaml
   type: onedata
   auth:
     access_token: MDAxY2xvY2F00aW9uIGRhdGFodWIuZWdpLmV1CjAw...
   connection:
     onezone_domain: datahub.egi.eu
     disable_tls_certificate_validation: false
   space:
     name: my-onedata-space
     galaxy_root_dir: ./galaxy-data
   cache:
     path: database/object_store_cache_onedata
     size: 1000
     cache_updated_data: true
   extra_dirs:
     - type: job_work
       path: database/job_working_directory_onedata
   ```
   
   Overwrite the following parameters with your values:
   - **access_token**: an access token suitable for data access (**write-enabled** and allowing calls to the Oneprovider REST API).
   - **onezone_domain**: the domain of the Onezone service (e.g., datahub.egi.eu).
   - **name**: the name of the Onedata Space where the Galaxy data will be stored.
   - **path**: the relative directory path in the Space under which the Galaxy data will be stored (optional, defaults to Space root).

   > <tip-title>Using Ansible</tip-title>
   > Put the config in `templates/galaxy/config/object_store_conf.yml.j2`.
   {: .tip}
   
2. Ensure there is the main Galaxy config file (`config/galaxy.yml`) and it includes the config:

   ```yaml
   galaxy:
     ...
     object_store_config_file: object_store_conf.yml
   ```
   
   > <tip-title>Using Ansible</tip-title>
   > Use the group vars file at `group_vars/gxconfig.yml`. 
   > See 
   > [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
   {: .tip}

> <warning-title>Switching object store types</warning-title>
> If you're switching from another object store type, you'll need to handle the migration of existing datasets. This process is not covered in this tutorial and should be carefully planned for production instances.
{: .warning}

For required dependencies, see the [Dependencies section](#dependencies) 
in Common Configuration.

For more details about object store configuration options and best practices, see the 
[Object Store tutorial]({% link topics/admin/tutorials/object-store/tutorial.md %}).


## Testing

Upload some new data to Galaxy, or run a workflow to produce results. Then
navigate to your Onedata account (e.g. [https://datahub.egi.eu](https://datahub.egi.eu))
and open the Space (and the path) that you have put down in the config. You
should see the Galaxy data:

![Data in Onedata](../../../galaxy-interface/images/onedata-byos/data-in-onedata.png)


# BYOS (Storage Location) templates

BYOS (Bring Your Own Storage) allows users to configure their own Onedata Storage Locations using templates. Similar to [BYOD](#byod-remote-file-source-templates), this feature requires a vault for secure credential storage.

1. Configure the vault as described in the [Vault Configuration section](#vault-configuration).

2. Enable the Onedata template by creating or modifying `config/object_store_templates.yml`:

   ```yaml
   - include: ./lib/galaxy/objectstore/templates/examples/onedata.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Put the config in `templates/galaxy/config/object_store_templates.yml.j2`, 
   > like it has been done 
   > [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/object_store_templates.yml.j2).
   {: .tip}

3. Ensure there is the main Galaxy config file (`config/galaxy.yml`) and it includes the
   config file above:
   ```yaml
   galaxy:
     ...
     object_store_templates_config_file: object_store_templates.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Use the group vars file at `group_vars/gxconfig.yml`. 
   > See 
   > [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
   {: .tip}

This template provides users with a form to configure their Onedata Storage Location, collecting information such as:
- Onezone domain and access token
- Space name and path for storing datasets
- Cache and performance settings

For required dependencies, see the [Dependencies section](#dependencies) 
in Common Configuration.


## Testing

Follow the Onedata 
[user-owned storage]({% link topics/galaxy-interface/tutorials/onedata-byos/tutorial.html %})
guide.


topic: admin
tutorial: cvmfs
title: Reference Data with CVMFS
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/cvmfs/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "git-gat"]
content_hash: 14341eb427eee377241f397fea2d0ea4
last_modified: 2026-02-26T21:12:14.142256
workflows_json: []
content:
> These words come from a transcript of Simon Gladman teaching this course. He
> is a bioinformatician at the University of Melbourne in Australia and also
> one of the administrators of Galaxy Australia.
>
> Hello everybody and welcome back to the Galaxy
> administrators course. in this session we're going to be talking about
> reference data in Galaxy using CVMFS.
>
> Hopefully by now you've all seen the video of
> the slides - if not I suggest you go and do
> look at those first as that explains how reference
> data works in Galaxy and also what CVMFS does to
> help us with that reference data.
>
> Some of the requirements from that we hope you've
> already completed: Galaxy server installation
> with Ansible and hopefully you
> understand what Ansible is and how
> it's used.
{: .spoken data-visual="gtn" data-target="#top-navbar" }

# Overview


The CernVM-FS {% cite Blomer_2011 %} is a distributed filesystem perfectly designed for sharing readonly data across the globe. We use it in the [Galaxy Project](https://galaxyproject.org) for sharing things that a lot of Galaxy servers need. Namely:
* **Reference Data**
    * Genome sequences for hundreds of useful species.
    * Indices for the genome sequences
    * Various bioinformatic tool indices for the available genomes
* **Tool containers**
    * [Singularity](https://www.sylabs.io/) containers of everything stored in [Biocontainers](https://biocontainers.pro/) (A bioinformatic tool container repository.) You get these for free every time you build a [Bioconda](https://bioconda.github.io/) recipe/package for a tool.
* Others too..

From the Cern website:

> The CernVM File System provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs."
>
{: .quote id="cvmfs-quote" cite="https://cernvm.cern.ch/portal/filesystem"}

> All right, just a quick recap. CVMFS or
> Cern-VMFS is a distributed file system perfectly
> designed for sharing read-only data across the
> globe and we use it extensively in the Galaxy
> project for sharing things that
> a lot of Galaxy servers need.
> Namely all the reference data; so the genome
> sequences for all the different genomes that
> we need to think about in in Galaxy and
> bioinformatics. Things like a human genome,
> mouse genome, etc., etc. And we need a lot of
> indices for the genome sequences and lots of
> tool indices for all those genomes. And we also
> have a repository that contains tool containers.
> So singularity containers for all of the
> bioinformatics tools that we might want to use.
> There's a tutorial idea already done this week
> or coming up soon this week that explains how to
> use those singularity containers within Galaxy as
> well. So we're just going to get on with things.
{: .spoken data-visual="gtn" data-target="#cvmfs-quote"}

[A slideshow presentation on this subject]({% link topics/admin/tutorials/cvmfs/slides.html %}) is available. More [details are available on usegalaxy.org (Galaxy Main's) reference data setup](https://galaxyproject.org/admin/reference-data-repo/#usegalaxyorg-reference-data) and CVMFS system.

This exercise uses Ansible to install and configure CVMFS and Galaxy's access to CVMFS. For a tutorial that does not use Ansible and gives a closer look at how reference data is configured in Galaxy, see [the Reference Data with CVMFS without Ansible tutorial]({% link topics/admin/tutorials/cvmfs-manual/tutorial.md %}).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="cvmfs" %}

> The agenda we're going to follow today is: We're going to install and
> configure Galaxy CVMFS reference data using ansible. We're going to explore
> the CVMFS installation and then we're going to configure Galaxy to use it.
{: .spoken data-visual="gtn" data-target="#agenda"}

# Ansible-CVMFS and Galaxy

> There are a few different repositories that um Galaxy project has created and
> shared with everybody. The first one is the reference data and indices and
> this is in data.galaxyproject.org. And then we have another one called
> singularity.galaxyproject.org.
>
> If you want to have a look at what's in there you can click on the "data
> cache", and this link here will show you all the things that are inside this
> data.galaxyproject.org
{: .spoken data-visual="gtn" data-target="#ansible-cvmfs-and-galaxy" }

The Galaxy project supports a few CVMFS repositories.


| Repository                 | Repository Address              | Contents                                                                         |
| ----------                 | ------------------              | --------                                                                         |
| Reference Data and Indices | `data.galaxyproject.org`        | Genome sequences and their tool indices, Galaxy `.loc` files for them as well    |
| Singularity Containers     | `singularity.galaxyproject.org` | Singularity containers for everything in Biocontainers for use in Galaxy systems |
| Galaxy Main Configuration  | `main.galaxyproject.org`        | The configuration files etc for Galaxy Main (usegalaxy.org)                      |

You can browse the contents of `data.galaxyproject.org` at the [datacache](http://datacache.galaxyproject.org/).

## Installing and Configuring

Luckily for us, the Galaxy Project has a lot of experience with using and configuring CVMFS and we are going to leverage off that. To get CVMFS working on our Galaxy server, we will use the Ansible role for CVMFS written by the Galaxy Project. Firstly, we need to install the role and then write a playbook for using it.

If the terms "Ansible", "role" and "playbook" mean nothing to you, please checkout [the Ansible introduction slides]({% link topics/admin/tutorials/ansible/slides.html %}) and [the Ansible introduction tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %})

{% snippet topics/admin/faqs/ansible_local.md %}

> Okay, so now we're going to move on to installing and configuring Galaxy's
> CVMFS reference data with Ansible. We are going to do some Ansible here and
> we're going to install CVMFS onto our Galaxy server. Hopefully you all have
> access to a Galaxy server.
{: .spoken data-visual="gtn" data-target="#installing-and-configuring"}

> Here is mine. And on it I am an admin user and I have access to the admin
> page. This was done as part of the installation of Galaxy um and hopefully
> you've installed a tool.
{: .spoken data-visual="galaxy" data-target="/" data-action="goto"}

> I have bwa and bwa-mem installed under Mapping.
{: .spoken data-visual="galaxy" data-target=".search-input input" data-action="fill" data-value="bwa-mem"}

> So if I click on that you can see
> here that I have bwa-mem but there are no options available for reference
> genomes, so we want to fix that. We want to connect to all of Galaxy's
> pre-built references and so we're going to use Galaxy's CVMFS system to let
> our own Galaxies connect and get access to all the pre-built caches and
> everything we already have.
{: .spoken data-visual="galaxy" data-target="/?tool_id=bwa_mem" data-action="goto"}

> Okay, so let's get started. If we go back to our
> tutorial here, it says that we need to install a CVMFS role into our
> requirements.yml and then add it to our Ansible.
{: .spoken data-visual="gtn" data-target="#hands-on-installing-cvmfs-with-ansible"}

> <hands-on-title>Installing CVMFS with Ansible</hands-on-title>
>
> 1. In your working directory, add the CVMFS role to your `requirements.yml`
>
>    > Well the first thing I need to do is log into my Galaxy machine in the
>    > terminal.
>    {: .spoken data-visual="terminal" data-cmd="whoami; hostname -f"}
>
>    > Have a look at he contents of this directory and I'll go into galaxy and here we have all of the Ansible scripts that hopefully everybody already has.
>    {: .spoken data-visual="terminal" data-cmd="ls -al"}
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -16,3 +16,6 @@
>       version: 0.0.14
>     - name: galaxyproject.tusd
>       version: 0.0.1
>    +# CVMFS Support
>    +- src: galaxyproject.cvmfs
>    +  version: 0.3.3
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement" data-ref="add-req"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>    > Okay, so the first thing I'm going to do is I'm going to add the CVMFS
>    > role to the requirements.yml.
>    > Edit requirements.yml and we need to add this to the bottom of that file. Copy. Paste. And save it.
>    {: .spoken data-visual="terminal" data-ref="add-req"}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true" data-ref="req-install"}
>    {: .code-in}
>
>    > And now install the role into our local Ansible scripts using the
>    > ansible-galaxy command. And as you can see, it's downloading the CVMFS
>    > role.
>    {: .spoken data-visual="terminal" data-ref="req-install"}
>
>    > And if we look into roles now you can see that we have galaxyproject.cmfs
>    {: .spoken data-visual="terminal" data-cmd="ls roles/"}
>
>    > Right, clear the screen.
>    {: .spoken data-visual="terminal" data-cmd="clear"}
>
> 3. The variables available in this role are:
>
>    > Okay, now what we need to do is we need to run this role and that will
>    > install the CVMFS client onto our Galaxy server. So the first thing we
>    > need to do is edit our group files galaxyservers file. There are a bunch
>    > of different variables that we can set. We can set the cvmfs role to be
>    > a client or stratum zero, stratum one server. URLs - so where we're
>    > getting all this data from, and if you remember in the slideshow, the
>    > data can come from Europe or America or Australia. Which repositories
>    > we want to have installed. And this is an important one - the quota
>    > limit. This is basically saying that CVMFS will cache some data on your
>    > local machine and the quota limit is the maximum size of that cache.
>    {: .spoken  data-visual="gtn" data-target="#variables-table"}
>
>    | Variable             | Type          | Description                                                                                                                                                                    |
>    | ----------           | -------       | -------------                                                                                                                                                                  |
>    | `cvmfs_role`         | string        | Type of CVMFS host: `client`, `stratum0`, `stratum1`, or `localproxy`. Controls what packages are installed and what configuration is performed.                               |
>    | `cvmfs_keys`         | list of dicts | Keys to install on hosts of all types.                                                                                                                                         |
>    | `cvmfs_server_urls`  | list of dicts | CVMFS server URLs, the value of `CVMFS_SERVER_URL` in `/etc/cvmfs/domain.d/<domain>.conf`.                                                                                     |
>    | `cvmfs_repositories` | list of dicts | CVMFS repository configurations, `CVMFS_REPOSITORIES` in `/etc/cvmfs/default.local` plus additional settings in `/etc/cvmfs/repositories.d/<repository>/{client,server}.conf`. |
>    | `cvmfs_quota_limit`  | integer in MB | Size of CVMFS client cache. Default is `4000`.                                                                                                                                 |
>    {: id="variables-table"}
>
>    But, luckily for us, the Galaxy Project CVMFS role has a lot of defaults for these variables which we can use by just setting `galaxy_cvmfs_repos_enabled` to `config-repo`. We will also set the `cvmfs_quota_limit` to something sensible (500MB) as we have relatively small disks on our instances. In a production setup, you should size this appropriately for the client.
>
>    > <tip-title>What is a good size for this?</tip-title>
>    > In production UseGalaxy.org.au uses 100GB, different sites have different needs and you can make your cache smaller depending on your usage. E.g. if your users only use one dataset from the reference data (e.g. just hg38) then perhaps you don't need such a large cache.
>    {: .tip}
>
>    > Okay but instead of just modifying galaxyservers.yml and adding in some
>    > of these variables - instead what we're going to do here is - we're
>    > going to create a new group file called all.yml.
>    >
>    > Because one of the things that we may want to do in the future - is we
>    > may want to create other machines. We might want to have worker nodes
>    > for our Galaxy cluster; or we may want to have other machines that we
>    > want to be able to create using these Ansible scripts that also have the
>    > CVMFS role installed. And instead of reproducing these variables in each
>    > of the group var files for those particular machines, we can create a
>    > special group vars file called all.yml. And whatever we put in there
>    > will be automatically available to all machines that we create with
>    > Ansible from this directory. Hopefully that makes a bit of sense and
>    > we're going to use that a bit later on if you come along to the pulsar
>    > tutorial where we will also be installing CVMFS on another machine - on
>    > a remote machine to run Pulsar.
>    {: .spoken data-visual="terminal" data-cmd="ls group_vars/"}
>
>    Add the following lines to your `group_vars/all.yml` file, creating it if it doesn't exist:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/all.yml
>    +++ b/group_vars/all.yml
>    @@ -5,3 +5,8 @@ pip_virtualenv_command: /usr/bin/python3 -m venv  # usegalaxy_eu.certbot, usegal
>     # Common variables needed by all hosts
>     galaxy_user_name: galaxy
>     galaxy_db_name: galaxy
>    +
>    +# CVMFS vars
>    +cvmfs_role: client
>    +galaxy_cvmfs_repos_enabled: config-repo
>    +cvmfs_quota_limit: 500
>    {% endraw %}
>    ```
>    {: data-commit="Configure CVMFS variables" data-ref="vars-all"}
>
>    > So what we're going to do, is we're going to create a new file called
>    > groupvars/all.yml and we're going to put some of these CVMFS variables
>    > inside it. So i'll just copy that. Okay, groupvars all dot yaml. And
>    > I'll paste this in. So basically the CVMFS role we want this machine to
>    > have is client. Which means that we just want it to be able to access
>    > all of our CVMFS reference data. And then we want we're going to set
>    > this one here to say that, "yes we want to set this up for Galaxy." And
>    > the config repo is the one that tells CVMFS how to set everything else
>    > up. And then this is the other important one - the CVMFS quota limit.
>    > We're setting to 500 megabytes and that's just so we don't fill the root
>    > disk of these machines. So I'll save that.
>    {: .spoken data-visual="terminal" data-ref="vars-all"}
>
>    > <tip-title>Why all.yml?</tip-title>
>    > We've integrated the cvmfs and pulsar tutorials better, such that CVMFS will be used for Pulsar as well, this configuration will be needed on all of our machines. This mirrors real life where you want CVMFS on every node that does computation.
>    {: .tip}
>
> 4. Add the new role to the list of roles under the `roles` key in your playbook, `galaxy.yml`:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -37,6 +37,7 @@
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>         - galaxyproject.gxadmin
>    +    - galaxyproject.cvmfs
>       post_tasks:
>         - name: Setup gxadmin cleanup task
>           ansible.builtin.cron:
>    {% endraw %}
>    ```
>    {: data-commit="Add role to playbook" data-ref="pb"}
>
>    > And now we need to add the role - we need to add the role to our Galaxy
>    > playbook. So we edit galaxy.yml which is our playbook and we just need to
>    > add the galaxyproject.cvmfs to the bottom of this. galaxyproject.cvmfs.
>    > And that's pretty much it.
>    {: .spoken data-visual="terminal" data-ref="pb"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true" data-ref="run-pb1"}
>    {: .code-in }
{: .hands_on}

> Okay now we just run the playbook. ansible-playbook. We want to run the
> Galaxy one. So. All right, we're about to get to the CVMFS repo here. We are
> now it's installing the um the apt package of CVMFS. It's going to get
> that out of this special um Cern apt repository. Okay, it's installing
> it. Hopefully it won't take too long. Okay, now setting up the
> repositories. And it's done.
{: .spoken data-visual="terminal" data-ref="run-pb1"}

Congratulations, you've set up CVMFS.

## Exploring the CVMFS Installation


> Okay, that was completed. We've now installed CVMFS client onto our machine
> and we've told it to go looking for certain repositories. Now to get access
> to them. We'll see what's in them. uh They'll be located at slash cvmfs. So
> under the cvmfs directory in your root directory. So, we can go to that. cd
> /
{: .spoken data-visual="terminal" data-cmd="cd /"}

> Do an ll. You can see here there's a directory here called cvmfs.
{: .spoken data-visual="terminal" data-cmd="ls -al"}

>  So we'll go in there and have a look and see what's in there.
{: .spoken data-visual="terminal" data-cmd="cd /cvmfs"}

> So let's have a look. Oh there's nothing in there!
> Well actually, what what's going to happen is; as soon as we go looking for
> something in this directory, so autofs will automatically mount the
> particular thing we're looking for. And so what we're going to do here is I'm
> going to go: cd data.galaxyproject.org, because I know that's one of the one
> of the repositories that should have been installed.
{: .spoken data-visual="terminal" data-cmd="ls -al"}

> And when I do that, autofs is automatically going to mount it for me on the
> fly. Like that.
{: .spoken data-visual="terminal" data-cmd="cd data.galaxyproject.org"}

> And now I've cd'd into it and if I do an ll, you can see here I've got some
> things in here now. I've got byhand and managed.
{: .spoken data-visual="terminal" data-cmd="ls -al"}

>  If I go into byhand you can see here that I have quite a lot of different
>  genomes and their tool indices.
{: .spoken data-visual="terminal" data-cmd="cd byhand; ls"}

> So if I'm going to, say, sacCer2, I can see in here there are bowtie index,
> the bwa index, the uh the picard index, the sam index; a whole bunch of
> other different things - including the original sequences etc. So yeah,
> quite a lot of data and we just have access to that on the fly. And then as
> soon as we try and look at any of these files, what will happen is CVMFS
> will automatically cache it to the local disk within within that 500
> megabyte cache that we uh we set up earlier. This is really cool. And we can
> tell Galaxy to look at all of this data and use it as its reference data.
{: .spoken data-visual="terminal" data-cmd="ls -al sacCer2/"}

> <hands-on-title>Exploring CVMFS</hands-on-title>
>
> 1. SSH into your machine
>
> 2. Change directory into `/cvmfs/` and list the files in that folder
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > You should see nothing, as CVMFS uses `autofs` in order to mount paths only upon request.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>
> 3. Change directory into `/cvmfs/data.galaxyproject.org/`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > cd /cvmfs/data.galaxyproject.org/
>    > ls
>    > ls byhand
>    > ls managed
>    > ```
>    {: .code-in}
>
>    > <question-title></question-title>
>    >
>    > What do you see now?
>    >
>    > > <solution-title></solution-title>
>    > >  You'll see `.loc` files, genomes and indices.
>    > > AutoFS only mounts the files when they're accessed, so it appears like there is no folder there.
>    > {: .solution }
>    >
>    {: .question}
>
>    And just like that we all have access to all the reference genomes and associated tool indices thanks to the Galaxy Project, IDC, and Nate's hard work!
>
{: .hands_on}

## Configuring Galaxy to use the CVMFS references.

Now that we have mounted the CVMFS repository we need to tell Galaxy how to find it and use it.

There are two primary directories in the reference data repository:

| Directory   | Contents                                                                                                                                                                  |
| ----------- | ----------                                                                                                                                                                |
| `/managed`  | Data generated with Galaxy Data Managers, organized by data table (index format), then by genome build.                                                                   |
| `/byhand`   | Data generated prior to the existence/use of Data Managers, manually curated. (For legacy reasons, this directory is shared as `/indexes` on the HTTP and rsync servers.) |

These directories have somewhat different structures:

* `/managed` is organized by index type, then by genome build (Galaxy dbkey)
* `/byhand` is organzied by genome build, then by index type

Both directories contain a location subdirectory, and each of these contain a `tool_data_table_conf.xml` file:

* `/managed/location/tool_data_table_conf.xml`
* `/byhand/location/tool_data_table_conf.xml`

Galaxy consumes these `tool_data_table_conf.xml` files and the `.loc` "location" files they reference. The paths contained in these files are valid if the data is mounted via CVMFS.

Examples of data include:

* twoBit (`.2bit`) and FASTA (`.fa`) sequence files
* Bowtie 2 and BWA indexes
* Multiple Alignment Format (`.maf`) files
* SAMTools FASTA indexes (`.fai`)

Now all we need to do is tell Galaxy how to find it! This tutorial assumes that you have run the tutorial in the requirements, [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}). The hands-on below will use the Galaxy Project Ansible role to configure everything.
{: id="spoken-7"}

> So that's what we're going to do now. Okay. So now we're going to try and
> configure Galaxy to use this CVMFS data. And and to have it so that we can run
> things like bwa and bwa mem and run them against the human genome or the bee
> genome or the mouse genome and take advantage of the fact that a lot of other
> people in the Galaxy community have done a lot of work for reference data for
> us already.
>
> So the way to do this is we're going to edit the groupvars galaxyservers
> file and we're going to add a variable called tool_data_table_config_path.
> And then we're going to point it to the two files that are in - there's one
> in byhand and one in managed.
{: .spoken  data-visual="gtn" data-target="#spoken-7"}


> If we go into byhand you can see here. And then we're going to location. As
> you can see in here, are all the lock files, but you'll also see there's an
> xml file here called tool_data_table_conf xml and we're going to point Galaxy
> at this file and there's another one in the same position in managed.
{: .spoken data-visual="terminal" data-cmd="cd /cvmfs/data.galaxyproject.org/byhand;  ls location;"}

> And you can see here there's another one in managed there. And so we're going to
> add both of these files to our Galaxy configuration and then Galaxy will be
> able to use all of the data contained within this repository.
{: .spoken data-visual="terminal" data-cmd="cd ../; ls managed/location"}

> Okay, so we'll go back to our Ansible directory.
{: .spoken data-visual="terminal" data-cmd="cd ~/galaxy/"}

> <hands-on-title>Configuring Galaxy to use CVMFS</hands-on-title>
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add a `tool_data_table_config_path` entry under the `galaxy` key of the `galaxy_config` section in the `group_vars/galaxyservers.yml` file. This new entry should be a list containing the paths to both `tool_data_table_conf.xml` files referenced above.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -76,6 +76,8 @@ galaxy_config:
>         # TUS
>         galaxy_infrastructure_url: "https://{{ inventory_hostname }}"
>         tus_upload_store: "{{ galaxy_tus_upload_store }}"
>    +    # CVMFS
>    +    tool_data_table_config_path: /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml,/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add tool_data_table_config_path to group variables" data-ref="gvconf"}
>
>    > This time I'm going to edit the groupvars galaxyservers.yml file. And in
>    > our Galaxy section. Which is here. Which is here. At the bottom of that
>    > I'm going to add a variable called tool_data_table_config_path. And I'm
>    > going to point it to the locations that we um I showed you before. um I
>    > can't remember what they are off the top of my head but luckily they're
>    > inside this solution box and so I will just copy them. And paste. And as
>    > you can see pointing to /cvmfs/data.galaxyproject.org/byhand/location
>    > and then that tool data table conf xml file. And then we have a list
>    > here and we separate it by commas and then we point it to the second
>    > one. Right, so we save this file.
>    {: .spoken data-visual="terminal" data-ref="gvconf"}
>
> 2. Re-run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true" data-ref="pb-run2"}
>    {: .code-in }
>
>    > So this time what we're going to do, all we're doing is making a minor change
>    > to the Galaxy yaml file in the Galaxy config to add that one line and then
>    > we're going to restart Galaxy. And yeah, Galaxy will suddenly automatically
>    > have access to all of that data. So you can see here we've changed the Galaxy
>    > configuration file. And then Galaxy is now restarting and it's done.
>    {: .spoken data-visual="terminal" data-ref="pb-run2"}
>
> 3. Install the BWA-MEM tool, if needed.
>
>    {% snippet topics/admin/faqs/install_tool.md query="bwa" name="Map with BWA-MEM" section="Mapping" %}
>
> 4. In your Galaxy server, open the **Map with BWA-MEM** {% icon tool %} tool. Now check that there are a lot more reference genomes available for use!
>
>    ![available_genomes.png](../../images/available_genomes.png)
>
> 5. Login to Galaxy as the admin user, and go to **Admin → Data Tables → bwa_mem indexes**
>
>    ![bwa_mem_indices.png](../../images/bwa_mem_indices.png)
>
{: .hands_on}

> Okay let's go and have a look at our Galaxy server and see if bwa can
> suddenly see all of those - that stuff. All right so we're back on our Galaxy
> server. I'll click on Analyze Data to just to reload the page.
{: .spoken data-visual="galaxy" data-target="#analysis a" data-action="click"}

> I'll go back to Mapping
{: .spoken data-visual="galaxy" data-target=".search-input input" data-action="click"}

>  and load bwa-mem.
{: .spoken data-visual="galaxy" data-target=".search-input input" data-action="fill" data-value="bwa-mem"}

> And then suddenly, instead of
> having no options available, you can see here we've got the Bee genome.
{: .spoken data-visual="galaxy" data-target="/?tool_id=bwa_mem" data-action="goto"}

>  Now click on that. Oh look at that, there are lots and lots and lots of
>  available genomes now including: lots of human, mouse, rat, yeast, all sorts
>  of things. And in fact if you want to see the list of all the different
>  available genomes now, that we have available to us.
>
>  If you go to admin. We go to data tables over here. You can see here that we
>  have um a couple of data tables for managed and for all fasta. So if we
>  click on that one, you can see that we have a lot of genomes available now
>  in the all fasta data table that Galaxy can get access to. If we go back to
>  the data tables again, and go down to bwa indexes or bwa mem indexes here.
>  You can see we have access to a lot of pre-built indexes for bwa for all of
>  these different genomes. That is pretty powerful.
>
>  So what did that take us? Maybe 30 minutes? And uh suddenly our Galaxy
>  server has access to all the uh the data the reference data and the tool
>  indices that the community have built over a number of years and it's super
>  simple.
{: .spoken data-visual="galaxy" data-target="s2id_field-uid-12_select a" data-action="click"}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=5 %}

# Common Production Questions

> <question-title>For the most used datasets (for ex. hg38) could we have a local copy, or would that be irrelevant?</question-title>
> This would be irrelevant, the most used datasets will stay in the cache. CVMFS uses a Least Recently Used (LRU) cache (see their [docs](https://cvmfs.readthedocs.io/en/latest/cpt-details.html#disk-cache)), so whenever it runs out of space, it will remove the least recently used file. If you have a file that is very commonly used, it will remain in the cache.
{: .question}

> <question-title>Could you explain how to calculate a good cache space?</question-title>
> Here are two approaches, there are others:
> 1. Allocate some cache, see how it is, make it larger if it is fully used + users complain of speed.
> 2. Enable reference data, and collect a week or two of data, analyse which reference datasets are being used, and allocate enough space for all of them.
>
> Essentially you just need data on how your users will behave and what reference data they want, combined with "when will they accept a wait period" to determine how much space you must allocate.
{: .question}

> <question-title>If I use a cluster, will I need to configure this FS in each node (given that the folder is at / directly)?</question-title>
> Yes. Often admins with a cluster keep a smaller cache local to each compute node, and then setup a Squid proxy to hold the most commonly accessed data on a machine with more storage. E.g. each compute node could have 10-50GB of CVMFS storage while you might setup a Squid proxy with 200-300 GB of storage that will store everything your site uses.
{: .question}

> <tip-title>Debugging failed mounting</tip-title>
>
> Are you having issues mounting your CVMFS mount? Is it giving strange errors like "Endpoint not connected"
> Try running this command as root:
>
> > > <code-in-title>Bash</code-in-title>
> > > ```console
> > > /usr/bin/cvmfs2 -d -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint singularity.galaxyproject.org /mnt
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Consolue</code-out-title>
> > > ```console
> > > Debug: using library /usr/lib/libcvmfs_fuse3_stub.so
> > > CernVM-FS: running in debug mode
> > > CernVM-FS: loading Fuse module... (cvmfs) Parsing config file /etc/cvmfs/default.conf    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) execve'd /bin/sh (PID: 280373)    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) Parsing config file /etc/cvmfs/default.d/50-cern-debian.conf    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) execve'd /bin/sh (PID: 280375)    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) Parsing config file /etc/cvmfs/default.d/80-ansible-galaxyproject-cvmfs.conf    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) execve'd /bin/sh (PID: 280378)    [07-21-2022 11:11:20 UTC]
> > > [...]
> > > (dns) empty hostname    [07-21-2022 11:11:20 UTC]
> > > (download) installed 1 proxies in 1 load-balance groups    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) DNS roaming is disabled for this repository.    [07-21-2022 11:11:20 UTC]
> > > (catalog) constructing client catalog manager    [07-21-2022 11:11:20 UTC]
> > > (catalog) Initialize catalog    [07-21-2022 11:11:20 UTC]
> > > (cache) unable to read local checksum    [07-21-2022 11:11:20 UTC]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfspublished to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfspublished    [07-21-2022 11:11:20 UTC]
> > > (download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 0)    [07-21-2022 11:11:20 UTC]
> > > (cache) miss ./e2/ab48b0984729d99951cb62c4312f501b3ddc6b (-2)    [07-21-2022 11:11:20 UTC]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX    [07-21-2022 11:11:20 UTC]
> > > (download) Verify downloaded url /data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX, proxy DIRECT (curl error 0)    [07-21-2022 11:11:20 UTC]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfswhitelist to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfswhitelist    [07-21-2022 11:11:20 UTC]
> > > [...]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/c7/f1555f421b1868b979291dc23f34a83132eadbC to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/c7/f1555f421b1868b979291dc23f34a83132eadbC    [07-21-2022 11:11:20 UTC]
> > > (download) Verify downloaded url /data/c7/f1555f421b1868b979291dc23f34a83132eadbC, proxy DIRECT (curl error 0)    [07-21-2022 11:11:25 UTC]
> > > (cache) finished downloading of /data/c7/f1555f421b1868b979291dc23f34a83132eadbC    [07-21-2022 11:11:25 UTC]
> > > (cache) commit ./c7/f1555f421b1868b979291dc23f34a83132eadb ./txn/fetchJWcwtt    [07-21-2022 11:11:25 UTC]
> > > (quota) pin into lru c7f1555f421b1868b979291dc23f34a83132eadb, path file catalog at singularity.galaxyproject.org:/ (c7f1555f421b1868b979291dc23f34a83132eadb)    [07-21-2022 11:11:25 UTC]
> > > (cache) commit failed: cannot pin c7f1555f421b1868b979291dc23f34a83132eadb    [07-21-2022 11:11:25 UTC]
> > > (catalog) failed to load catalog '' (2 - not enough space to load catalog)    [07-21-2022 11:11:25 UTC]
> > > (catalog) failed to initialize root catalog    [07-21-2022 11:11:25 UTC]
> > > Failed to initialize root file catalog (16 - file catalog failure)
> > > (cache) unpinning / unloading all catalogs    [07-21-2022 11:11:25 UTC]
> > > ```
> > {: .code-out}
> {: .code-2col}
{: .tip}

# Other Aspects

> Right, we'll go back to our tutorial. um Yeah. Just finally, just before we
> finish up. um If we are developing a new tool and you want to add a reference
> genome or a different index just give us - drop us a line on Gitter and we'll
> be able to add it into our - into the reference data for the community. um
> We're looking at automating the process of building all of this material
> using data managers and ephemeris. And we're working with a group of people
> called the IDC, which is the Intergalactic Data Commission, which is a funny
> name for everyone in Galaxy - in the Galaxy community who likes reference
> data. And we're looking at making a community controlled resource that will
> be semi-automatic. One of the other things that you can do is have automatic
> fallbacks. So, if say, you're in Australia and you're hooked up to the
> Australian mirror of the CVMFS repository and the Australian mirror dies, the
> CVMFS client is smart enough to automatically go to the next closest one and
> so you won't lose anything. If you're interested in looking at plant data
> there's a link here for that.
{: .spoken data-visual="gtn" data-target="#other-aspects" }

## Development

If you are developing a new tool, and want to add a reference genome, we recommend you [talk to us on Gitter](https://gitter.im/galaxy-iuc/iuc). You can also look at one of the tools that uses reference data, and try and copy from that. If you’re developing the location files completely new, you need to write the data manager.

## Automation

You can automate the process of installing and setting up data managers and data with ephemeris. We're working in the [IDC](https://github.com/galaxyproject/idc) to democratise this CVMFS repository, and make this a community-controlled resource. You can also look here for ideas on automating your data management.

## Access Control

It is not easily possible to filter access to reference data depending on the user's role or group.

You could set up a tool per user/group, [secure access to running this tool](https://galaxyproject.org/admin/config/access-control/), and then allow this private tool to access a private tool data table. But you will not get tool updates, you will have to copy and edit this tool every time it gets updated. Or write more advanced job control rules to reject specific jobs which use specific datasets.

## Proxying Recap

The client talks directly to the stratum 1 (or to a proxy), and manages the data, and exposes it to the user. The proxy stores an opaque cache, that can't really be used, except as a proxy to a stratum 1.

## Plant Data

If you are working with plants, you can find separate reference data here: [frederikcoppens/galaxy_data_management](https://github.com/frederikcoppens/galaxy_data_management)

> And finally if you could please click on this link here and give us some
> feedback on how you think the tutorial went, whether it was useful, if you
> enjoyed it or um if you have any criticisms, could you please put them in
> here as well. And if you end up using this to build a Galaxy server and you
> publish that Galaxy server um could you cite the tutorial for us please. That
> would be make a big difference to us. All right, thank you very much and I
> hope you enjoyed it and hopefully I'll get to meet some of you in person one
> day soon at a Galaxy conference. Thank you and goodbye.
{: .spoken data-visual="gtn" data-target="#feedback"}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="cvmfs" %}


topic: admin
tutorial: database-schema
title: Galaxy Database schema
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 07a0a0cb6053103cec415189dd2e5218
last_modified: 2026-02-26T21:12:14.143144
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/database-schema/tutorial.html

content:
For the hands-on examples you need access to a Galaxy server and access to its PostgreSQL database. You can set-up this yourself, or use the Galaxy Docker Image provided by Björn Grüning (https://github.com/bgruening/docker-galaxy-stable). During this tutorial, we will work with the Galaxy Docker Image.

Setting up Docker and using the Galaxy Docker Image:
(please do this before the tutorial, preferably when you are still at home using a fast internet connection)

Follow the instruction to install the Docker engine: https://docs.docker.com/engine/installation/   
Execute:  docker run -d -p 8080:80 bgruening/galaxy-stable
(this will download the Galaxy Docker Image, when executed the first time, and
start it)
Test the Docker image in your web browser: http://localhost:8080
(see first paragraph of Björn’s introduction, for special cases when using the
Docker Toolbox on Mac and Windows). Quit by: docker kill NAME (you get the name with:  docker ps )

# Introduction

## Database versus Object Model

The session description is database centric and we’ll be focusing on the relational
database that backs Galaxy servers.  But that’s only half the picture of the this data.
The other is the object model which is the object-oriented view of this same data.
The object model is used by the code to manipulate and access the database.
The translation between the two worlds is handled by an object-relational mapping implemented with SQLAlchemy (https://www.sqlalchemy.org).

Today we are covering the database and how to access it with SQL.  We aren’t going to cover the corresponding object model or object relational mapping.

## Database Platform

The default out-of-the-box Galaxy installation uses SQLite (https://www.sqlite.org).
SQLite is a lightweight database management system (DBMS) that can be packaged inside Galaxy and does not require any additional steps at initial setup time.

However, SQLite is not the recommended DBMS for running a Galaxy server. The recommended production DMBS for
Galaxy is PostgreSQL (https://www.postgresql.org). PostgreSQL offers a full set of DBMS features and robust support for multiple simultaneous users.

This workshop will be entirely based in PostgreSQL (also referred to as Postgres).

## What is in (and not in) the Galaxy database?

The Galaxy database contains management information about your server.  The database tracks users, groups, jobs, histories, datasets, workflows and so on.  

What’s not in the database is the data. Datasets are stored outside the database. The database does keep metadata – information about the datasets such as data type. The tools themselves are not stored in the database either.

## Understanding the Database Schema

### ER diagrams and SchemaSpy

Entity-relationship diagrams are a way to understand tables and the relationships between them
inside a relational database.  SchemaSpy (http://schemaspy.org) is a free (and remarkable tool) for
generating ER diagrams. Here's a [diagram of the Galaxy database](../../images/galaxy_schema.png)
(generated by SchemaSpy on June 6 2020; 6Mb).

The “Tables” tab is a good place to start learning the structure of the database.  Each table represents a different type of thing, and often that thing is itself a relationship. For example, each record in the dataset table has information about a specific dataset, while records in the history_dataset_association table have information about what histories that dataset is in.

Each SchemaSpy table’s page shows the attributes in that table, as well as any constraints on those attributes, and the relationships between that table and other tables.

Also see the “Run SchemaSpy in this container” section below for how to install and then run SchemaSpy yourself.

### Database conventions
The Galaxy database uses a number of naming and design conventions.  Understanding these can make navigating the database much easier.

### id attributes
Every table has an id column that uniquely identifies each row.  (The id column is the primary key in database terminology.) Beyond uniquely identifying a row in the table, ID values have no meaning.  ID values are unique within a table, but not across the database.

### Relationships between tables, and `_id` columns
Relationships between tables are implemented by exporting id columns from one table into another.  Imported ids are called foreign keys in database nomenclature, and are uniformly named
   table_the_id_came_from_id

There are a few notable exceptions to this rule.  If the ID is from a table that is prefixed with galaxy_, for example, galaxy_user or galaxy_session, the  galaxy_ will be dropped from the column name.  For example, galaxy_user.id becomes user_id in the over 50 tables it is imported into

### Relationship tables

As mentioned previously, some tables, such as history_dataset_association represent relationships between things, rather than things themselves.  In this case history_dataset_association describes relationships between datasets and histories.

Relationship table names typically contain the names of tables they are relating, suffixed with `_association`.

Why are nulls allowed in almost every column?
We have no idea.  In practice, they aren’t nulls in most of those columns.

Why aren’t there comments, on anything?
PostgreSQL supports comments to table definitions, but there are none shown in the SchemaSpy report. Why? The table definitions are actually generated by SQLAlchemy, the object-relational mapping software used by Galaxy, and SQLAlchemy does not support it.

There is nothing in the database that results from direct manipulation of the table definitions through DDL.  Everything comes in through SQLAlchemy.

## Start Docker and Galaxy

> <hands-on-title></hands-on-title>
>
>   1. Start the Galaxy Docker Image -  this time as an interactive session
>
>    ```sh
>       docker run -i -t -p 8080:80 bgruening/galaxy-stable /bin/bash
>    ```
>
>   2. Start Galaxy and its PostgreSQL server
>
>    ```sh
>       startup > log 2>&1 &
>    ```
>
>   3. Follow the startup process
>
>    ```sh
>       tail -f log
>    ```
{: .hands_on}

## Important tables

> <hands-on-title></hands-on-title>
>
>   1. Connect to the PostgreSQL database (change to user galaxy first)
>
>    ```sh 
>        su galaxy
>        psql -d galaxy -U galaxy
>    ```
>
>   2. List all tables
>
>    ```sql
>       \dt
>    ```
{: .hands_on}


Enter `q` to exit the view results page, and space to see the next results page.

### Table “galaxy_user”

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from galaxy_user;
>    ```
{: .hands_on}

As described in Björn’s introduction, an Admin user is already pre-set (email: ‘admin@galaxy.org’, password: ‘admin’). Now let’s add (i.e. register) a new user via the Galaxy website. And check the database:

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from galaxy_user;
>    ```
{: .hands_on}

### Table “job”

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from job;
>    ```
{: .hands_on}

Run a few jobs on the galaxy website (e.g _upload file_ a simple table and _add column_ with ‘Iterate’ no and yes) and check the database again:

> <hands-on-title></hands-on-title>
>
>    ```sql
>        select * from job \x\g\x
>    ```
{: .hands_on}

### Table “job_parameter”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from job_parameter;
>   ```
{: .hands_on}


### Table “history”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from history;
>   ```
{: .hands_on}

Give your current history a name and check the database again.


### Table “dataset”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from dataset;
>   ```
{: .hands_on}

### Table “history_dataset_association”

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select * from history_dataset_association;
>   ```
{: .hands_on}


## More (hands-on) Examples, not covered by the reports app

Have a look at the reports up (which is also provided in the Docker Image):

http://admin:admin@localhost:8080/reports/

Depending on your local needs, some queries are missing, like:

## Jobs per tool per year  /  jobs per tool since 2015

You can add the numbers per month from the reports, or:

> <hands-on-title></hands-on-title>
>   ```sql
>       select j.id, j.create_time from job j limit 5;
>   ```
>
>   ```sql
>        select j.id, j.create_time from job j
>            where j.create_time >= '2015-12-31'
>            and j.create_time < '2016-12-31';
>    ```
>
>   ```sql
>        select j.id,j.create_time from job j
>           where EXTRACT(year FROM j.create_time) = 2016
>           and j.tool_id='upload1';`
>   ```
>
>   ...and now include the user
>
>   ```sql
>        select count(j.id) from job j, galaxy_user u
>           where j.user_id = u.id
>           and u.email = 'hansrudolf.hotz@fmi.ch'
>           and EXTRACT(year FROM j.create_time) = 2016
>           and j.tool_id='upload1';
>   ```
>
>   ```sql
>        select u.email, count(*) from job j, galaxy_user u
>           where j.user_id = u.id
>           and EXTRACT(year FROM j.create_time) = 2016
>           and j.tool_id='upload1'
>           GROUP BY u.email;
>   ```
{: .hands_on}

### Jobs per tool of a certain version

Imagine the current version of a tool is working fine, however a previous version had a bug: now you wanna warn
all the users who have used the broken version, without alerting users who never used the broken one.

The following example is from the development server at the FMI

> <hands-on-title></hands-on-title>
>   ```sql
>       select distinct(j.tool_version) from job j
>           where j.tool_id = 'qAlign';
>   ```
>
>   ```sql
>        select j.user_id from job j
>           where j.tool_id = 'qAlign'
>           and j.tool_version = '1.0.4quasr';
>   ```
>
>   ```sql
>       select u.email, j.create_time from job j, galaxy_user u
>           where j.user_id = u.id
>           and j.tool_id = 'qAlign'
>           and j.tool_version = '1.0.4quasr';
>   ```
{: .hands_on}


## All users running a job using a certain parameter

> <hands-on-title></hands-on-title>
>
>   ```sql
>       select jp.name, jp.value  from job_parameter jp
>           where name = 'iterate'`
>   ```
>
>   ```sql
>       select u.email, jp.name, jp.value
>           from job_parameter jp, job j, galaxy_user u
>           where jp.name = 'iterate'
>           and j.tool_id = 'addValue'
>           and jp.job_id = j.id
>           and j.user_id = u.id;
>   ```
>
{: .hands_on}

## Close PostgreSQL client and quit docker

> <hands-on-title></hands-on-title>
>   Close the PostgreSQL client
>
>   ```sql
>       \q
>   ```
>
>   Quit the interactive docker (change back to root first)
>
>   ```sh
>       exit
>       exit
>   ```
{: .hands_on}

## Other Topics

### How to move from MySQL to PostgreSQL

Slides: https://docs.google.com/presentation/d/1N3BDNQT3s7eQEO3BO89TQbTYwKp92fxHDRWQwC-T1kA

https://wiki.galaxyproject.org/Community/Log/2015/MySQL2PostgreSQL

### Is there ever a need to manually change the contents of a table

Slides:
https://docs.google.com/presentation/d/1l4DD0IaJjuvk1zAT1Sjv26bLyrSOg3VUm7rD-TQl_Zs

### Run SchemaSpy in this container

To run SchemaSpy in your container you’ll need to get it, and also install some required software packages.

> <hands-on-title>Schema Spy</hands-on-title>
>   ```sh
>   wget https://github.com/schemaspy/schemaspy/releases/download/v6.1.0/schemaspy-6.1.0.jar
>   apt-get update
>   apt-get install libpostgresql-jdbc-java
>   apt-get install graphviz
>   ```
>
>    To run SchemaSpy:
>   ```
>   java -jar schemaspy-6.1.0.jar -t pgsql -db galaxy -u galaxy -host localhost -s public -dp /usr/share/java/postgresql-jdbc4-9.2.jar -o SpyOut
>   ```
>
{: .hands_on}

The SpyOut directory will contain the generated reports and diagrams, anchored at index.html.



# Conclusion

There is a lot of information stored in the Galaxy database. Use this information for trouble shooting when necessary and use it as a source for extended user statistics.


topic: admin
tutorial: subdomain
title: Create a subdomain for your community on UseGalaxy.eu
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/subdomain/tutorial.html
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["subdomain", "community", "earth-system", "interactive-tools"]
content_hash: 7b83df8edab9fd7684d65f5f87b0e4d4
last_modified: 2026-02-26T21:12:14.153462
workflows_json: []
content:
This tutorial covers how to set up a subdomain on usegalaxy.eu. We will take here the example of the [earth system subdomain](https://earth-system.usegalaxy.eu/) and follow the step one by one. 

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Add your subdomain on a Galaxy server

> <hands-on-title>Subdomain name</hands-on-title>
>    - Clone the Github repository of the Galaxy server (of where you want your subdomain to be attached)
>    - For Galaxy europe clone the [Infrastructure repo](https://github.com/usegalaxy-eu/infrastructure)
>    - Create a branch on your fork
>    - Open the file **dns.tf** and edit it 
>    - You should add the name of your subdomain to the list my_new_and_shiny_subdomain.usegalaxy.eu or for the example here earth-system.usegalaxy.eu AND update **count** (number of subdomains) as shown below.
> 
> ![Image of the modification to make in the dns.tf file](../../images/create_subdomain/earth_system_subdo_infra.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}


# Enable the interactive tools 

> <hands-on-title>Galaxy Interactive Tool</hands-on-title>
>    - Clone the Github repository of the Galaxy server (of where you want your subdomain to be attached)
>    - For Galaxy europe clone the [Infrastructure-playbook repo](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master)
>    - Create a branch on your fork
>    - Open the file **sn06.yml** and edit it 
>    - You should add the name of your subdomain to the list "*.my_new_and_shiny_subdomain.usegalaxy.eu" or for the example here "*.earth-system.usegalaxy.eu" 
>    - AND add "*.interactivetoolentrypoint.interactivetool.earth-system.usegalaxy.eu" as shown below.
> 
> ![Image of the modification to make in the sn06.yml file](../../images/create_subdomain/earth_system_subdo_gxit.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

> <hands-on-title>Add the interactive tool section to the tool panel</hands-on-title>
>    - First, on your **Infrastructure-playbook** fork, create a new branch
>    - Go to **templates/galaxy/config/**
>    - Open the file **global_host_filters.py.j2** and edit it 
>    - In the function per_host_tool_labels add your subdomain as followed
> ```
> if "earth-system.usegalaxy.eu" in host:
>   return label.id in valid_labels"
> ```
>   - Then in the **DOMAIN_SECTIONS** add the the interactive tool section (this will allow to see on the subdomain interface the tool section "Interactive tools") as folowed:
> `'earth-system': ["interactivetools"],`
>
> ![Image of the modification to make in the global_host_filters.py.j2 file](../../images/create_subdomain/add_gxit.png)
>
>   - Second, on the same fork go to **files/traefik/rules/**
>   - Open and edit **template-subdomains.yml**, there you need to add the folowing line for your subdomain `{% raw %}{{template "subdomain" "my_new_and_shiny_subdomain"}}{% endraw %}`
>
>   - Finally, commit all your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}


# Make a nice tool panel (only for interactive tools)

> <hands-on-title>Customize the tool panel (only for the interactive tools)</hands-on-title>
>    - On your **Infrastructure-playbook** fork (don't forget to synchronize it)
>    - First, go to **templates/galaxy/config/**
>    - Open the file **tool_conf.xml.j2** and edit it 
>    - From there you can create your own tool sections where your different interactive tools can be organized. See the example of earth-system below.
> ![Image of the modification to make in the tool_conf.xml.j2 file](../../images/create_subdomain/customize_tool.png)
>
>   - Then, commit your changes
>   
>   - Secondly, still in **templates/galaxy/config/**
>   - Open the file **global_host_filters.py.j2** and edit it 
>   - Add the different new sections of tools created to your subdomain and prevent those new sections to appear in any other subdomain as shown below.
>  ![Image of the modification to make in the global_host_filters.py.j2 file to add your tool sections](../../images/create_subdomain/add_customize_tool.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

# Customize the front page 

> <hands-on-title>Update the homepage</hands-on-title>
>    - For Galaxy europe clone the [website repo](https://github.com/usegalaxy-eu/website/tree/master)
>    - Create a branch on your fork
>    - Create an index file in markdown for instance "index-earth-system.md"
>    - Once created let your imagnination flows to make the looks and feel of your homepage (if needed here is a [PR example](https://github.com/usegalaxy-eu/website/pull/1149))
>    - If you want to add images to your front page add them as folowed **assets/media/my_incredidle_image.png**
>
>   - Then commit your changes, write a nice message for the admin when you open your Pull Request.
> If in the future you want to change the front page just update the index file.
{: .hands_on}

> <tip-title>Add a custom welcome.html</tip-title>
> There is also the possibility to add a custom welcome.html, as well as other custom static files like the [singlecell subdomain did](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master/files/galaxy/subdomains/singlecell/static)
{: .tip}

> <hands-on-title>Add a theme</hands-on-title>
>    - On your **Infrastructure-playbook** fork (don't forget to synchronize it)
>    - Go to **group_vars/sn06/**
>    - Open the file **subdomains.yml** and edit it. There you have to add the name of your subdomain.
>    - In this same file you can also customize a theme for your subdomain. 
{: .hands_on}

# Let people know

> <hands-on-title>Make some communication on your new subdomain</hands-on-title>
>    - Clone the [galaxy-hub repo](https://github.com/galaxyproject/galaxy-hub/tree/master)
>    - Create a branch on your fork
>    - In **content/news** create a folder for your news example "2023-10-17-earth-system" and in it create an **index.md** file 
>    - In this index file write your blog post don't hesitate to add some nice photos
>
>   - Then commit your changes and open your Pull Request.
{: .hands_on}

# Conclusion
Your Subdomain is ready to be used !


topic: admin
tutorial: tiaas
title: Training Infrastructure as a Service (TIaaS)
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/tiaas/tutorial.html
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "training", "jobs", "git-gat"]
content_hash: c2f751efc7f688e88f49c42f39e0f0e5
last_modified: 2026-02-26T21:12:14.166479
workflows_json: []
content:
Galaxy is widely used for teaching. In order to facilitate instructors, the Galaxy Project has developed {TIaaS}.
Workshop instructors can apply for {TIaaS}, and on the day of their workshop, their participants will be placed in a special group and use dedicated
resources, thus reducing queue times on the day of the training.

![TIaaS concept](../../images/tiaas/tiaas_intro.png "With TIaaS, all of your users visit the same server. In the background, the scheduler recognises which users are training users, and directs their jobs to special resources. In the EU deployment of TIaaS jobs preferentially use private resources, but can spill over to the main queue if there is not enough space available."){: width="70%"}

This tutorial will go cover how to set up such a service on your own Galaxy server.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tiaas" %}


# Setting up TIaaS

> <hands-on-title>Setup TIaaS</hands-on-title>
>
> 1. In your `requirements.yml` add the {TIaaS} ansible role:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -50,3 +50,6 @@
>       version: 6f6fdf7f5ead491560783d52528b79e9e088bd5b
>     - src: cloudalchemy.grafana
>       version: 0.14.2
>    +# Training Infrastructure as a Service
>    +- src: galaxyproject.tiaas2
>    +  version: 2.1.5
>    {% endraw %}
>    ```
>    {: data-commit="Add tiaas2 requirement"}
>
>    And run the install step:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. In your `galaxyservers` group variables file, add the following:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -350,3 +350,8 @@ telegraf_plugins_extra:
>           - timeout = "10s"
>           - data_format = "influx"
>           - interval = "15s"
>    +
>    +# TIaaS setup
>    +tiaas_dir: /srv/tiaas
>    +tiaas_admin_user: admin
>    +tiaas_admin_pass: changeme
>    {% endraw %}
>    ```
>    {: data-commit="Configure tiaas"}
>
> 2. In the `galaxyservers` group variables file, we also need to set the database permissions correctly for TIaaS. It needs to be able to access some Galaxy tables, and we will carefully define only the ones we really need:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/dbservers.yml
>    +++ b/group_vars/dbservers.yml
>    @@ -3,6 +3,7 @@
>     postgresql_objects_users:
>       - name: "{{ galaxy_user_name }}"
>       - name: telegraf
>    +  - name: tiaas
>     postgresql_objects_databases:
>       - name: "{{ galaxy_db_name }}"
>         owner: "{{ galaxy_user_name }}"
>    @@ -11,7 +12,26 @@ postgresql_objects_privileges:
>         roles: telegraf
>         privs: SELECT
>         objs: ALL_IN_SCHEMA
>    -
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: galaxy_user,galaxy_session,job,history,workflow,workflow_invocation
>    +    type: table
>    +    privs: SELECT
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: user_group_association,galaxy_group,role,group_role_association
>    +    type: table
>    +    privs: SELECT,INSERT
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: group_role_association
>    +    type: table
>    +    privs: DELETE
>    +  - database: galaxy
>    +    roles: tiaas
>    +    objs: role_id_seq,galaxy_group_id_seq,group_role_association_id_seq,user_group_association_id_seq
>    +    type: sequence
>    +    privs: USAGE,SELECT
>
>     # PostgreSQL Backups
>     postgresql_backup_dir: /data/backups
>    {% endraw %}
>    ```
>    {: data-commit="Add database privileges for TIaaS"}
>
>    > <tip-title>Why does TIaaS get `DELETE` privileges on Galaxy's Database?</tip-title>
>    > The `DELETE` privilege is limited in scope to one table: `group_role_association`. This allows TIaaS to
>    > disassociate training groups from roles in the Galaxy database after the training event date has passed, so that
>    > users who participated in a training return to using normal (non-training) resources after the training ends.
>    >
>    > The `galaxyproject.tiaas2` role will create a [cron](https://manpages.debian.org/stable/cron/cron.8.en.html) job
>    > to perform this process every night at midnight. You can control when this runs (or disable it) using
>    > [the tiaas_disassociate_training_roles variable](https://github.com/galaxyproject/ansible-tiaas2/blob/d5be2a064c49e010f67bfcea18e36812da23d7d8/defaults/main.yml#L20).
>    >
>    {: .tip}
>
>    > <tip-title>Running the playbook from scratch</tip-title>
>    > This is one of the few statements we've provided that presents difficulties when running the playbook completely from scratch on a blank machine. Setting postgresql roles is one of the first steps in our playbook, but the rules we've provided above depend on the Galaxy tables existing in that database. If those tables aren't there, it will fail. If you do someday run this from scratch, you'll find that you need to comment out those roles.
>    {: .tip}
>
> 3. We need to add the `galaxyproject.tiaas2` role before the `nginx` role, as TIaaS defines variables that Nginx needs.
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -47,6 +47,7 @@
>         - galaxyproject.nginx
>         - geerlingguy.docker
>         - usegalaxy_eu.rabbitmqserver
>    +    - galaxyproject.tiaas2
>         - galaxyproject.gxadmin
>         - galaxyproject.cvmfs
>         - dj-wasabi.telegraf
>    {% endraw %}
>    ```
>    {: data-commit="Add TIaaS role to the Galaxy playbook"}
>
> 4. Lastly we should add the routes for TIaaS to the NGINX template for Galaxy. TIaaS provides a set of default nginx routes that can be used.
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -114,4 +114,7 @@ server {
>     		proxy_pass http://127.0.0.1:3000/;
>     		proxy_set_header Host $http_host;
>     	}
>    +
>    +	{{ tiaas_nginx_routes }}
>    +
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Add nginx routes for TIaaS"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}


{TIaaS} should be available now! The following routes on your server are now configured (we will run through these in the next section)


| URL                                                                                       | Use                                                                         | Audience                 |
| :----                                                                                     | ----                                                                        | -----                    |
| [/tiaas/new/](https://my.gat.galaxy.training/?path=/tiaas/new/)                           | Request a new TIaaS training                                                | Instructors              |
| [/tiaas/admin/](https://my.gat.galaxy.training/?path=/tiaas/admin/)                       | Approve and Manage requests                                                 | Admin                    |
| [/tiaas/stats/](https://my.gat.galaxy.training/?path=/tiaas/stats/)                       | Overall TIaaS statistics ([EU Stats](https://usegalaxy.eu/tiaas/stats/))    | Admins, Funding Agencies |
| [/tiaas/calendar/](https://my.gat.galaxy.training/?path=/tiaas/calendar/)                 | Calendar of trainings ([EU Calendar](https://usegalaxy.eu/tiaas/calendar/)) | Admins, Funding Agencies |
| [/join-training/ID](https://my.gat.galaxy.training/?path=/join-training/ID)               | Join an TIaaS training                                                      | Participants             |
| [/join-training/ID/status](https://my.gat.galaxy.training/?path=/join-training/ID/status) | Dashboard with job states of trainees.                                      | Instructors              |


Let's see it in action!

> <hands-on-title>Using TIaaS</hands-on-title>
>
> 1. **Create a new TIaaS request**
>    - Go to [/tiaas/new](https://my.gat.galaxy.training/?path=/tiaas/new)
>    - Here you will find the request form users will fill in to request TIaaS:
>      ![TIaaS request form](../../images/tiaas/tiaas_request_form.png)
>    - For *"Training Identifier"*, fill in `gat`
>      - This is the `<training-id>` used in the URLs listed above used for:
>        1. Workshop participants to join the tiaas group
>        2. Workshop instructors to monitor the progress of their participants.
>    - Fill in the rest of the form as you like
>    - Submit the form and you should see a confirmation dialog:
>      ![TIaaS requested successfully](../../images/tiaas/tiaas_request_form_submitted.png)
>
> 2. **Approve TIaaS request**
>    - Next, the request will have to be approved by an admin
>    - Go to [/tiaas/admin](https://my.gat.galaxy.training/?path=/tiaas/admin)
>    - **Log in** using the values you configured `tiaas_admin_user` and `tiaas_admin_pass` in your group variables file
>      - Default values were `admin:changeme`
>    - You should now see the admin panel:
>      ![TIaaS admin console](../../images/tiaas/tiaas_admin_console.png)
>    - Click on **Trainings**, you should see the TIaaS request listed here:
>      ![TIaaS request list](../../images/tiaas/tiaas_request_list.png)
>    - **Approve the request**
>      - Click on the training
>      - Scroll down to the bottom
>      - Change *"Processed"* to `Approved` and **Save**
>        ![Approve TIaaS](../../images/tiaas/tiaas_request_approve.png)
>    - At this point, you would likely email the person who made the request to inform them of approval
>
> 3. **Join TIaaS Training**
>    - Make sure you are logged in to Galaxy
>    - On the day of the workshop, participants will visit a following URL to join the TIaaS group
>      - [/join-training/gat](https://my.gat.galaxy.training/?path=/join-training/gat)
>      - A confirmation dialog should appear if all went well:
>        ![Join TIaaS](../../images/tiaas/tiaas_join_training.png)
>
> 4. **Monitor TIaaS status**
>    - This is very useful for instructors to monitor the job state of their participants
>    - Go to [/join-training/gat](https://my.gat.galaxy.training/?path=/join-training/gat)
>    - In the Dasboard you should see that one user (you) has joined the training \
>    - Run some jobs to see the dashboard in action
>      ![TIaaS dashboard](../../images/tiaas/tiaas_dashboard.png)
>    - Scroll down to get some more information on a per-user level (anonymized)
>      - Every user designated by their own identifier and colour, but no personal information
>      ![TIaaS dashboard](../../images/tiaas/tiaas_dashboard2.png)
>
{: .hands_on}


> <comment-title>Note: GDPR assistance</comment-title>
>
> Since this setup tracks additional personal information (submitter name & email, users in the queue view), TIaaS includes some always-on features to assist with your GDPR compliance.
>
>  - Users in public status dashboard are only visible by an anonymized identifier and colour
>  - Email addressses in the TIaaS admin panel will be automatically expunged 60 days after a training event
>
>  Of course you need to review any GDPR compliance concerns with your group's legal representative(s), this only attempts to ensure some protections exist for the users of the system.
>
{: .comment}


# Job Configuration

While observability for teachers or trainers is already a huge benefit, one of the primary benefits of {TIaaS} is that your jobs get sent to dedicated compute resources, which won't be used by anyone else, during the period of the training. We will send all of the training jobs to pulsar if you have completed that tutorial, or one of the slurm destinations from the job configuration training.

In order to achieve this, we first need some way to sort the jobs of the training users into these private queues, while letting the other jobs continue on. So let's create a *traffic controller* to figure out where jobs belong.


> <hands-on-title>Writing a dynamic job destination</hands-on-title>
>
> 1. This destination will check that the `user_email` is in a training group (role starting with `training-`).
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -35,6 +35,15 @@ tools:
>           require:
>             - pulsar
>
>    +roles:
>    +  training.*:
>    +    max_cores: 2
>    +    max_mem: max_cores * 3.8  # TODO check multiplier
>    +    scheduling:
>    +      require:
>    +        - slurm
>    +        - training
>    +
>     destinations:
>       local_env:
>         runner: local_runner
>    @@ -64,6 +73,19 @@ destinations:
>         max_mem: 8
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)} --time={entity.params['walltime']}:00:00
>    +  slurm-training:
>    +    inherits: singularity
>    +    runner: slurm
>    +    max_accepted_cores: 12
>    +    max_accepted_mem: 120
>    +    max_cores: 2 # Limit the cores
>    +    max_mem: 8 # Limit the memory
>    +    params:
>    +      native_specification: --nodes=1 --ntasks=1 --mem={round(mem*1024)} --cpus-per-task={cores} --time=00:30:00
>    +    scheduling:
>    +      require:
>    +        - slurm
>    +        - training
>
>       pulsar:
>         runner: pulsar_runner
>    {% endraw %}
>    ```
>    {: data-commit="Add to list of deployed rules"}
>
> 6. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 7. Ensure your user is joined to a training
>
> 8. Run a job and observe the logs to see where it goes (`journalctl -u galaxy -f`)
>
{: .hands_on}

Congratulations! you have now set up {TIaaS} on your Galaxy server.

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=14 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tiaas" %}


topic: admin
tutorial: cvmfs-manual
title: Reference Data with CVMFS without Ansible
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 5a50ccc1a8ba6fb2ce7874b1eed2a3fb
last_modified: 2026-02-26T21:12:14.141609
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/cvmfs-manual/tutorial.html

content:
# Overview


The CernVM-FS is a distributed filesystem perfectly designed for sharing readonly data across the globe. We use it in the [Galaxy Project](https://galaxyproject.org) for sharing things that a lot of Galaxy servers need. Namely:
* **Reference Data**
    * Genome sequences for hundreds of useful species.
    * Indices for the genome sequences
    * Various bioinformatic tool indices for the available genomes
* **Tool containers**
    * [Singularity](https://www.sylabs.io/) containers of everything stored in [Biocontainers](https://biocontainers.pro/) (A bioinformatic tool container repository.) You get these for free every time you build a [Bioconda](https://bioconda.github.io/) recipe/package for a tool.
* Others too..

From the Cern website:

> The CernVM File System provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs."
>
{: .quote cite="https://cernvm.cern.ch/portal/filesystem"}

[A slideshow presentation on this subject]({% link topics/admin/tutorials/cvmfs/slides.html %}) is available. More [details are available on usegalaxy.org (Galaxy Main's) reference data setup](https://galaxyproject.org/admin/reference-data-repo/#usegalaxyorg-reference-data) and CVMFS system.

This exercise describes a manual process to install and configure CVMFS and Galaxy's access to CVMFS. For a tutorial that uses Ansible to perform these tasks, see [the Reference Data with CVMFS tutorial]({% link topics/admin/tutorials/cvmfs/tutorial.md %}).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}



# CVMFS and Galaxy without Ansible

> <comment-title>Manual version of Ansible Commands</comment-title>
> If you wish to perform the same thing that we've just done, but by building the ansible script manually, follow these instructions. Otherwise, you have already done everything below and do not need to re-do it.
{: .comment}

We are going to setup a CVMFS mount to the Galaxy reference data repository on our machines. To do this we have to install and configure the CVMFS client and then mount the appropriate CVMFS repository using the publicly available keys.

> <hands-on-title>Installing the CVMFS Client</hands-on-title>
>
> 1. On your remote machine, we need to first install the Cern software apt repo and then the CVMFS client and config utility:
>
>    ```bash
>    sudo apt install lsb-release
>    wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb
>    sudo dpkg -i cvmfs-release-latest_all.deb
>    rm -f cvmfs-release-latest_all.deb
>    sudo apt-get update
>
>    sudo apt install cvmfs cvmfs-config
>    ```
>
> 2. Now we need to run the CVMFS setup script.
>
>    ```bash
>    sudo cvmfs_config setup
>    ```
>
{: .hands_on}


## Configuring CVMFS

The configuration is not complex for CVMFS:

> <hands-on-title>Configuring CVMFS</hands-on-title>
>
> 1. Create a `/etc/cvmfs/default.local` file with the following contents:
>
>    ```
>    CVMFS_REPOSITORIES="data.galaxyproject.org"
>    CVMFS_HTTP_PROXY="DIRECT"
>    CVMFS_QUOTA_LIMIT="500"
>    CVMFS_CACHE_BASE="/srv/cvmfs/cache"
>    CVMFS_USE_GEOAPI=yes
>    ```
>
>    This tells CVMFS to mount the Galaxy reference data repository and use a specific location for the cache which is limited to 500MB in size and to use the instance's geo-location to choose the best CVMFS repo server to connect to. You can use the `cvmfs_quota_limit` role variable to control this setting.
>
> > <tip-title>What is a good size for CVMFS_QUOTA_LIMIT?</tip-title>
> > In production UseGalaxy.org.au uses 100GB, different sites have different needs and you can make your cache smaller depending on your usage. E.g. if your users only use one dataset from the reference data (e.g. just hg38) then perhaps you don't need such a large cache.
> {: .tip}
>
> 2. Create a `/etc/cvmfs/domain.d/galaxyproject.org.conf` file with the following contents:
>
>    ```
>    CVMFS_SERVER_URL="http://cvmfs1-tacc0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-iu0.galaxyproject.org/cvmfs/@fqrn@;http://cvmfs1-psu0.galaxyproject.org/cvmfs/@fqrn@;http://galaxy.jrc.ec.europa.eu:8008/cvmfs/@fqrn@;http://cvmfs1-mel0.gvl.org.au/cvmfs/@fqrn@;http://cvmfs1-ufr0.galaxyproject.eu/cvmfs/@fqrn@"
>    ```
>
>    This is a list of the available stratum 1 servers that have this repo.
>
> 3. Create a `/etc/cvmfs/keys/data.galaxyproject.org.pub` file with the following contents:
>
>    ```
>    -----BEGIN PUBLIC KEY-----
>    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5LHQuKWzcX5iBbCGsXGt
>    6CRi9+a9cKZG4UlX/lJukEJ+3dSxVDWJs88PSdLk+E25494oU56hB8YeVq+W8AQE
>    3LWx2K2ruRjEAI2o8sRgs/IbafjZ7cBuERzqj3Tn5qUIBFoKUMWMSIiWTQe2Sfnj
>    GzfDoswr5TTk7aH/FIXUjLnLGGCOzPtUC244IhHARzu86bWYxQJUw0/kZl5wVGcH
>    maSgr39h1xPst0Vx1keJ95AH0wqxPbCcyBGtF1L6HQlLidmoIDqcCQpLsGJJEoOs
>    NVNhhcb66OJHah5ppI1N3cZehdaKyr1XcF9eedwLFTvuiwTn6qMmttT/tHX7rcxT
>    owIDAQAB
>    -----END PUBLIC KEY-----
>    ```
>
> 4. Make a directory for the cache files
>
>    ```
>    sudo mkdir /srv/cvmfs
>    ```
{: .hands_on}


## Testing it out

Probe the connection.

> <hands-on-title>Testing it out</hands-on-title>
>
> 1. Run `sudo cvmfs_config probe data.galaxyproject.org`
>
>    > <question-title></question-title>
>    >
>    > What does it output?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > OK
>    > > ```
>    > >
>    > > If this doesn't return `OK` then you may need to restart autofs: `sudo systemctl restart autofs`
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
> 2. Change directory into `/cvmfs/` and list the files in that folder
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > You should see nothing, as CVMFS uses `autofs` in order to mount paths only upon request.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>
> 3. Change directory into `/cvmfs/data.galaxyproject.org/`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > cd /cvmfs/data.galaxyproject.org/
>    > ls
>    > ls byhand
>    > ls managed
>    > ```
>    {: .code-in}
>
>    > <question-title></question-title>
>    >
>    > What do you see now?
>    >
>    > > <solution-title></solution-title>
>    > >  You'll see `.loc` files, genomes and indices.
>    > > AutoFS only mounts the files when they're accessed, so it appears like there is no folder there.
>    > {: .solution }
>    >
>    {: .question}
>
>    And just like that we all have access to all the reference genomes and associated tool indices thanks to the Galaxy Project, IDC, and Nate's hard work!
>
>    > <tip-title>Contributing Reference Genomes</tip-title>
>    > If you are developing a new tool, and want to add a reference genome, we recommend you [talk to us on Gitter](https://gitter.im/galaxy-iuc/iuc). You can also look at one of the tools that uses reference data, and try and copy from that. If you’re developing the location files completely new, you need to write the data manager.
>    {: .tip}
>
{: .hands_on}

## Look at the repository

Now to configure Galaxy to use the CVMFS references we have just installed, see [the Ansible tutorial.]({% link topics/admin/tutorials/cvmfs/tutorial.md %}#configuring-galaxy-to-use-the-cvmfs-references)


topic: admin
tutorial: customization-manual
title: Customizing the look of Galaxy (Manual)
difficulty: intermediate
time_estimation: 45m
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 1e2e4e8bafa3707f0c116e4681b0b7bd
last_modified: 2026-02-26T21:12:14.140131
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization-manual/tutorial.html

content:
Customizing your Galaxy instance makes it more recognizable at a glance, and can help communicate its purpose to its users.
This tutorial will teach you three basic customizations you can make to Galaxy:

  - Setting a brand text
  - Adding a custom welcome page
  - Customizing the masthead using themes

For this tutorial we will be setting up a fictional instance called "Galaxy Mars".
Feel free to use the included material in the hands-on sections, or provide your own.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Custom Branding

The brand text in Galaxy refers to the text you can see in the masthead of some Galaxy instances.
This text will appear in the masthead, as well as the sites title.
It is an easy way to set your instance apart, and make it more identifiable.

![screenshot of the start page of Galaxy Europe](images/galaxy-europe.png "Galaxy Europe uses the brand text \"Europe\"")

> <hands-on-title>Customising the Branding</hands-on-title>
> 
> 1. Open your galaxy config file `config/galaxy.yml` and add the following option under `galaxy:`
> 
>    ```yaml
>    brand: Mars
>    logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>    ```
>
> 2. Start your galaxy server using `sh run.sh`
>
> 3. Navigate to `https://localhost:8080`
>
>    ![screenshot of fictional "Galaxy Mars" start page, with the brand text set to "Mars"](images/galaxy-mars-brand.png "Your Galaxy start page should now look like this")
{: .hands_on}

## Custom Welcome Page

The welcome page is an html document embedded in Galaxy's start page.
It is what users see when first navigating to your instance.
This page can be used to communicate what your instance is about, and share news and updates with your users.

> <hands-on-title>Creating a custom welcome page</hands-on-title>
>
> 0. (optional) Set the location of your welcome page in `galaxy.yml`
> 
>    Under `galaxy:` add the following option:
>
>    ```yaml
>    welcome_url: /static/your-welcome-page.html
>    ```
>
>    Not setting this will cause the url to default to `/static/welcome.html`
>
> 1. Open `/static/welcome.html`, or create a new html file at the location set in Step 0
>
>    It is advisable to keep a copy of the default `welcome.html`, as it can serve as a useful reference.
>
> 2. Add some Content
>
>    Let's add a title and some text to our welcome page:
>
>    ```html
>    <!DOCTYPE html>
>    <html lang="en">
>        <head>
>            <meta charset="utf-8">
>        </head>
>        <body>
>            <h1>Welcome to the Galaxy Mars instance!</h1>
>            <p>The only Galaxy instance on mars.</p>
>        </body>
>    </html>
>    ```
>
>    This works the same as any other html page. You can add styles in the head, or import some scripts.
>    
>    Make sure your welcome page has exactly one `h1` element, which describes the page.
>    This will act as the heading for your start-page, which can help assistive technologies.
{: .hands_on}

> <tip-title>Use an IFrame!</tip-title>
> A lot of the UseGalaxy.* instances choose to use their `welcome.html` to display an IFrame to a different site,
> e.g. the GalaxyProject.org site, or a sub-site thereof.
>
> This makes it easier for non-administrators to keep the homepage's contents
> up to date. By embedding (for example) a Wordpress blog in the center,
> non-technical contributors can still publish articles and announcements
> without having to re-deploy Galaxy every time.
{: .tip}

> <tip-title>Styling your Welcome</tip-title>
> As in our example welcome.html, we've loaded the default Galaxy CSS to keep
> it consistent with the surrounding material. If you wish to load a different
> CSS file, or set a different font for the main panel, you can easily do that
> by adding whatever CSS you like.
{: .tip}

## Custom Masthead Theme

Galaxy's theming system allows you to easily change the color of your Masthead, to give your instance an even more distinct look.
You can even offer several options, to allow users to switch to the default if they prefer it, or another look all together.

> <hands-on-title>Configuring Themes</hands-on-title>
>
> 0. (optional) Set the location of the themes configuration `galaxy.yml`
> 
>    Under `galaxy:` add the following option:
>
>    ```yaml
>    themes_config_file: your-themes-file.yml
>    ```
>
>    This location is relative to the config directory.
>    If unset, the default file `themes_conf.yml` will be used, if present.
>
> 1. Create your themes config file.
>
>    Open `themes_conf.yml.sample`, and copy its contents into a new file, either called `themes_conf.yml`, or the custom path you set in step 0.
>    Do not copy the conf file and remove the ending, as this file is a symbolic link.
>    Copying it copies a reference to the source file, not its contents.
>
> 2. Understanding Themes
>
>    The themes file can contain several themes.
>    Each theme is identified by its id, eg `blue`, and followed by a set of rules, which style the client.
>    The first theme in your theme file will be used as the default theme, which users see when visiting your instance,
>    and which users who haven't logged in will see.
>
>    At the time of writing, themes only support styling the masthead, but this is subject to change.
>    The first example theme should contain all available values you can change.
>
> 3. Creating our own Theme
>
>    Let's create a theme more fitting for our Mars instance.
>
>    Create a new theme, by adding `mars:` to the top of the themes config file.
>    Now add `masthead:` underneath, and then `color:`.
>    Set color to `"#e03e1d"`, a nice deep martian red.
>
>    Your theme file should now begin with:
>
>    ```yaml
>    mars:
>      masthead:
>        color: "#e03e1d"
>    ```
>
>    Restart your instance, and look at the masthead. It should be colored red.
>
> 4. You can also try logging in, and changing your theme under `Preferences`
{: .hands_on}


topic: admin
tutorial: stop-worrying-love-systemd
title: How I learned to stop worrying and love the systemd
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "systemd"]
content_hash: a6bb56fe92a01590e146f966051598e0
last_modified: 2026-02-26T21:12:14.152897
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/stop-worrying-love-systemd/tutorial.html

content:
Many linux sysadmins with years and years of experience bemoan systemd ("it's infecting everything! Now it wants to mess with time? And DNS???") and journalctl ("unix was supposed to be about files!") and while those are fair complaints and make systemd and friends wildly more opaque than traditional SysV and logging to files, there are some benefits that can be obtained, and may be interesting even to the wise old admins. There is a lot of convenience in systemd that can make the tradeoffs worth it.

This tutorial assumes a working knowledge of unix systems (files, directories, services, logging, `/var/log`, etc.).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# systemd

## Why systemd units, and not service shell scripts?

- CGroups
- Hard Memory Limits that include all children (also CPU, disk.)
- Dependencies (is your database up, before your service that needs it?)
- Fancy restarting (always/n times, delays, timeouts.)
- [Monitor it with Telegraf](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/systemd_units)

## Unit Files

```
[Unit]
Description=Galaxy
After=network.target
After=time-sync.target

[Service]
UMask=022
Type=simple
User=galaxy
Group=galaxy
WorkingDirectory=/srv/galaxy/server
TimeoutStartSec=60
ExecStart=/srv/galaxy/venv/bin/galaxyctl start --foreground --quiet

Environment=HOME=/srv/galaxy
Environment=VIRTUAL_ENV=/srv/galaxy/venv

MemoryAccounting=yes
MemoryLimit=16G
CPUAccounting=yes
IOAccounting=yes

[Install]
WantedBy=multi-user.target
```

Here's a simple unit file. It has a name, and some targets that must be up before it should be started, then the service definition featuring all of the common fields like:

- User/group and umask restriction
- Working directory
- Process to start (and optionally reload/stop commands.)
- Some environment variables
- Enabling of CPU and Mmeory accounting
- As well as limiting the memory to 16GB, at which time the child will be killed and restarted.

Unit files are very easy to create, be sure to see the [examples section](https://man.archlinux.org/man/systemd.service.5#EXAMPLES) of the documentation, and stick with `Type=simple` services which stay in the foreground for maximum convenience.


## Status

Status is generally your first step looking at any unit file.

```bash
$ systemctl galaxy
● galaxy.service - Galaxy
     Loaded: loaded (/etc/systemd/system/galaxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-06-28 15:17:04 CEST; 5 days ago
   Main PID: 217446
      Tasks: 73 (limit: 9521)
     Memory: 1.7G (limit: 16.0G)
        CPU: 18h 16min 20.685s
     CGroup: /system.slice/galaxy.service
             ├─217446 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/galaxyctl start --foreground --quiet
             ├─217467 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/supervisord -c /srv/galaxy/var/gravity/supervisor/supervisord.conf --nodaemon
             ├─217484 /bin/tail -f /srv/galaxy/var/gravity/supervisor/supervisord.log
             ├─217487 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/gunicorn galaxy.webapps.galaxy.fast_factory:factory() --timeout 300 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker -b unix:/srv/galaxy/var/config/gunicorn.sock --workers=2 --config python:galaxy.web_stack.gunicorn_config --preload --forwarded-allow-ips=*
             ├─245320 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/gunicorn galaxy.webapps.galaxy.fast_factory:factory() --timeout 300 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker -b unix:/srv/galaxy/var/config/gunicorn.sock --workers=2 --config python:galaxy.web_stack.gunicorn_config --preload --forwarded-allow-ips=*
             ├─245322 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/gunicorn galaxy.webapps.galaxy.fast_factory:factory() --timeout 300 --pythonpath lib -k galaxy.webapps.galaxy.workers.Worker -b unix:/srv/galaxy/var/config/gunicorn.sock --workers=2 --config python:galaxy.web_stack.gunicorn_config --preload --forwarded-allow-ips=*
             ├─245334 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/celery --app galaxy.celery worker --concurrency 2 --loglevel DEBUG --pool threads --queues celery,galaxy.internal,galaxy.external
             ├─245401 /srv/galaxy/venv/bin/python -c from multiprocessing.resource_tracker import main;main(7)
             ├─245582 /srv/galaxy/venv/bin/python /srv/galaxy/venv/bin/celery --app galaxy.celery beat --loglevel DEBUG --schedule /srv/galaxy/var/gravity/celery-beat-schedule
             ├─245806 python ./lib/galaxy/main.py -c /srv/galaxy/config/galaxy.yml --server-name=handler_0 --attach-to-pool=job-handler --attach-to-pool=workflow-scheduler --pid-file=/srv/galaxy/var/gravity/supervisor/handler_0.pid
             ├─246274 python ./lib/galaxy/main.py -c /srv/galaxy/config/galaxy.yml --server-name=handler_1 --attach-to-pool=job-handler --attach-to-pool=workflow-scheduler --pid-file=/srv/galaxy/var/gravity/supervisor/handler_1.pid
             └─246737 python ./lib/galaxy/main.py -c /srv/galaxy/config/galaxy.yml --server-name=handler_2 --attach-to-pool=job-handler --attach-to-pool=workflow-scheduler --pid-file=/srv/galaxy/var/gravity/supervisor/handler_2.pid
```

Helpful things to note:

- the file defining the service is listed after `Loaded`, if you want to see what it does or how it's configured, or to edit that configuration
- You can see how long it's been running
- How many tasks (children) there are
- Memory and the limit, if memory limiting is configured.
- CPU time used
- And then many of the children.

## Restarting

Here's one of the nice new things you can do, once you've moved away from SysV, and to the joyous world of systemd, restart multiple processes at once! If you have multiple Galaxy processes, use a wildcard to restart them all.

```bash
systemctl restart nginx galaxy-*
```

## Failed Units

Easily check if any service has failed with:

```bash
systemctl --failed
```

## Enabling Units

In order to start at boot time, it must be "enabled", so you can run a service once knowing it won't come back when the machine reboots (or vice versa). But you should be sure that a unit is enabled if you want it available on restart. You can do this in separate commands

```bash
systemctl enable my.service
systemctl start my.service
```

Or simultaneously, [if you're on a systemd v220 or newer](https://unix.stackexchange.com/questions/374280/the-now-switch-of-systemctl):

```bash
systemctl enable --now my.service
```

## Editing Units or Overriding

Sometimes one of the system units provided will have some weird behaviour that you need to override (or your ansible role doesn't expose it), then you can use `systemctl edit unit` to override some settings.

Additional directives can be supplied, e.g. making your service start after another service is started, if you need to sequence their starts.

Note that if it was defined in the base unit file, then you need to set it to an empty value first, before overriding. E.g. `ExecStart=\nExecStart=command...`


## Masking Units

I never needed this until one day I did. Masking a unit makes it impossible to start, but if someone's gone and installed a GUI on your server, inclusive of power management systems that trigger hibernation when not in use, then this might be a useful trick. You can mask anything, service, or even targets like I had to in that case.

This command will prevent the device from reaching or activating any of those targets. Useful for servers when you're in a bind and don't know how to remove power management:

```console
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
```

## Unit Security Optimisation

Because systemd uses cgroups, it can also give us a nice overview of any security issues that might be worth looking into. Here we see the Galaxy unit has a lot of

```console
ubuntu@gat-1:~$ systemd-analyze security galaxy
  NAME                                       DESCRIPTION                                         EXPOSURE
✗ PrivateNetwork=                            Service has access to the host's network            0.5
✓ User=/DynamicUser=                         Service runs under a static non-root user identity
....
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT      Service may issue chroot()                          0.1
✗ ProtectHostname=                           Service may change system host/domainname           0.1
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND   Service may establish wake locks                    0.1
✗ CapabilityBoundingSet=~CAP_LEASE           Service may create file leases                      0.1
✗ CapabilityBoundingSet=~CAP_SYS_PACCT       Service may use acct()                              0.1
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG  Service may issue vhangup()                         0.1
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM      Service may program timers that wake up the system  0.1
✗ RestrictAddressFamilies=~AF_UNIX           Service may allocate local sockets                  0.1

→ Overall exposure level for galaxy.service: 9.2 UNSAFE 😨
```

Here we see a lot of low hanging fruit for restricting the abilities of Galaxy's processes. This can also be applied to e.g. the job runner, or your webserver for additional security.

## Testing with systemd-run

Would you like to test some of these limits manually? Or just to run a one-off process in a very sandboxed environment? Enter `systemd-run`, which allows just that. Here you can set a lot of the capabilities or restrictions:

```bash
systemd-run -p IOAccounting=yes -p CPUAccounting=true -p MemoryAccounting=true -p TasksAccounting=true -p ProtectSystem=strict -p PrivateDevices=yes -t -S
```

And `-S` puts you in an interactive terminal, allowing you to play around with this environment. You get a cleaned out environment, working inside of a cgroup (visible in `systemd-cgtop`, and `systemctl status`) and when you exit it tells how you many resources you've used. This is both an easy way to test processes, and an option to sandbox rarely run processes, by setting an extremely restrictive environment.

## Optimising Slow Boots

systemd features an analyze command to check why your booting is slow

```console
$ systemd-analyze
Startup finished in 3.841s (firmware) + 3.925s (loader) + 11.104s (kernel) + 1min 5.331s (userspace) = 1min 24.203s
graphical.target reached after 11.547s in userspace
```

You can even have it generate a fancy plot for you!

```bash
systemd-analyze plot > plot.svg
```

![waterfall graphic of systemd boot time, cloud-init has the most, largest red bars](images/plot.svg "Notice how many units wait for network to finish being ready, and are stuck until that is ready. This image was generated from a training VM. Can you spot Galaxy? It doesn't show the red bars, because it does not currently notify systemd when it's done starting up.")

## Pro/Cons of systemd

> > <code-out-title>Pros</code-out-title>
> > - CGroups, to reap children!
> > - CGroups, for hard memory/cpu limits
> > - CGroups, for security benefits, and dropping capabilities!
> > - Service dependencies and simultaneously booting multiple units.
> > - Easily override tasks
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > - It's new and requires time to learn
> >     - But all the distros have mostly switched so, might be time to learn.
> > - It is significantly more complicated
> > - Even if that complexity does have a lot of benefits
> {: .code-in}
{: .code-2col}

## Further Reading

- [Archi wiki page](https://wiki.archlinux.org/title/Systemd)
- [man 1 systemctl](https://man.archlinux.org/man/systemctl.1)
- [man 5 systemd.service](https://man.archlinux.org/man/systemd.service.5)

# timers

Do you use cron aggressively, like any other sysadmin? It's an amazing part of the system, no? Doing things periodically, and repeatably. But you've probably had to learn a lot of hard lessons along the way, right?

- Doing things idempotently
- Checking that you don't have a period cron job running, before starting this one, if it's a long one.
- What if there was a reboot?

Would you like to worry less about those issues? Try systemd timers today!

[The arch page](https://wiki.archlinux.org/title/Systemd/Timers) is an excellent reference on why they're interesting and useful. We'll reproduce their pro/con list below:

> > <code-out-title>Pros</code-out-title>
> > - Easier testing, you can trigger units any time
> > - Cgroups!
> > - dependencies (e.g. network!)
> > - automatic logging!!
> > - two flavours of timers: clock time based, and monotonic (e.g. time since last execution)
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > - Two files, instead of a single line. It *is* annoying, you're not wrong. Maybe write an ansible role that translates it.
> > - No `MALITO` functionality built in.
> {: .code-in}
{: .code-2col}

## Listing Timers

We have a lovely timer listing, which tells you when it'll next run, how much time is left, when it last ran, how much time since that passed.

```
$ systemctl list-timers
NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES
Mon 2022-07-04 12:32:45 CEST 6min left     Mon 2022-07-04 11:34:59 CEST 51min ago    anacron.timer                anacron.service
Mon 2022-07-04 14:07:08 CEST 1h 40min left Fri 2022-07-01 13:38:20 CEST 2 days ago   motd-news.timer              motd-news.service
Mon 2022-07-04 17:12:05 CEST 4h 45min left Mon 2022-07-04 10:20:49 CEST 2h 5min ago  ua-timer.timer               ua-timer.service
Mon 2022-07-04 17:14:33 CEST 4h 48min left Fri 2022-07-01 19:22:14 CEST 2 days ago   apt-daily.timer              apt-daily.service
Tue 2022-07-05 00:00:00 CEST 11h left      Mon 2022-07-04 09:32:16 CEST 2h 54min ago logrotate.timer              logrotate.service
Tue 2022-07-05 00:00:00 CEST 11h left      Mon 2022-07-04 09:32:16 CEST 2h 54min ago man-db.timer                 man-db.service
Tue 2022-07-05 01:44:11 CEST 13h left      Mon 2022-07-04 11:07:49 CEST 1h 18min ago fwupd-refresh.timer          fwupd-refresh.service
Tue 2022-07-05 06:28:44 CEST 18h left      Mon 2022-07-04 09:33:59 CEST 2h 52min ago apt-daily-upgrade.timer      apt-daily-upgrade.service
Tue 2022-07-05 10:12:36 CEST 21h left      Mon 2022-07-04 09:47:01 CEST 2h 39min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2022-07-10 03:10:47 CEST 5 days left   Mon 2022-07-04 09:32:52 CEST 2h 53min ago e2scrub_all.timer            e2scrub_all.service
Mon 2022-07-11 00:00:00 CEST 6 days left   Mon 2022-07-04 09:32:16 CEST 2h 54min ago fstrim.timer                 fstrim.service
```

In the last column we can see the timer units, and the services which they activate. In systemd everything is a unit, and hopefully by this point in the tutorial you've seen some of the benefits of that.

Let's look at the `motd-news` timer:

```console
$ systemctl cat motd-news.timer
[Unit]
Description=Message of the Day

[Timer]
OnCalendar=00,12:00:00
RandomizedDelaySec=12h
Persistent=true
OnStartupSec=1min

[Install]
WantedBy=timers.target
```

It has a description, an `OnCalendar` time since it's a clock based timer, potentially a random delay as well (despite being `Sec`, can be expressed in any unit). The associated `motd-news.service` does not need to be mentioned, it will be started by default based on the name of the timer.

What about logs? They're in journalctl now! Let's take advantage of some of the things we learned earlier, like, wildcards!

```console
journalctl -u motd-news.* --since '5 days ago'
```

You'll see the timer unit start mostly around reboots, usually it doesn't have much useful in it. The service unit has what we're more interested in, any logged output.

## Fixing those Cron Issues

Before we mentioned a few issues that admins dealt with with cron

- Doing things idempotently
- Checking that you don't have a period cron job running, before starting this one, if it's a long one.
- What if there was a reboot?

While we can't help with the first, the second is taken care of by having systemd services be state machines, if there's an already running copy of the `.service`, then the `.timer` won't do anything. No more `pgrep` to check for specific processes. And as for reboots, `Persistent=true` can cause a timer to trigger immediately, if it missed the last start time.

## Notifying you

As mentioned in the Arch wiki, you can setup a "notification" service. This is a service, typically a simple bash script, that, when run, sends a notification to a channel of your choice (Slack, XMPP, Email, etc.) This can be a really simple service:

```
/etc/systemd/system/failure-notification@.service

[Unit]
Description=Send a notification
After=network.target

[Service]
Type=simple
ExecStart=/path/to/failure-notification.sh %i
```

And then for any unit (timer's associated service, normal units, etc) where you'd like to be informed of the failure, simply update or override that to do the notification:

```
[Unit]
OnFailure=failure-notification@%n
```

Here it will be templated out with `%n` meaning the unit name, which will replace the `%i` in the notification unit, and be included in the notification to you.


## Further Reading

- [The arch page](https://wiki.archlinux.org/title/Systemd/Timers)
- [man 5 systemd.timer](https://man.archlinux.org/man/systemd.timer.5)
- [man 7 systemd.time](https://man.archlinux.org/man/systemd.time.7)

# journald

> Where did my files go?
{:.quote}

In a number of systems they're still there, to be honest, you've still got all of your old log files.

But now instead of search N different log files, or memorising which files have what contents, there's one firehose for you to watch instead: journalctl. So we need to make this manageable with filters!

## Per Unit Logs

> <code-out-title>The journalctl Way</code-out-title>
> ```bash
> journalctl -u galaxy
> ```
{: .code-out}

returns all of the logs for Galaxy.  The first line even tells us how far back our logs go:

```console
$ journalctl -u galaxy | head -n1
-- Logs begin at Tue 2022-06-28 08:31:06 UTC, end at Mon 2022-07-04 09:30:01 UTC. --
```

> > <code-out-title>Pros</code-out-title>
> > You did not need to know or remember:
> >
> > - Where the log files were (`/var/log`? `/srv/galaxy/log`? somewhere else?)
> > - If any of the old logs were compressed (or use `zless`/`zcat`/`zgrep`)
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > - Can't just `cat` it.
> {: .code-in}
{: .code-2col}

## Logs for a Group of Units

Sometimes you want to tail the logs of multiple proceses as once. Have you ever been debugging an issue, and wanted to see the logs of `nginx`, **all** `galaxy` processes, and `postgres`? Well you can't because `nginx` needs special configuration to write to journalctl (see, your files are still there, and it makes things extra confusing.)

Adding these two lines to your `nginx` configuration will place logs in journald

```nginx
error_log syslog:server=unix:/dev/log;
access_log syslog:server=unix:/dev/log;
```

And then you can simply

> <code-out-title>The journalctl Way</code-out-title>
> ```bash
> journalctl -f -u galaxy* -u nginx -u postgresql
> ```
{: .code-out}

If you have multiple similarly named units, the wildcard feature is incredibly helpful. UseGalaxy.eu had multiple named processes for: web handlers, workflow schedulers, job handlers. We could see logs simultaneously across all of these units.

> > <code-out-title>Pros</code-out-title>
> > - Tail multiple units at once
> > - Again don't have to know where the files are
> > - Or switch tools if you want to look at older logs
> {: .code-out}
>
> > <code-in-title>Cons</code-in-title>
> > ?
> {: .code-in}
{: .code-2col}

## Dropping Logs

journald has the habit of throwing away logs if it gets too many messages simultaneously, to prevent {DOS} and denial-of-storage attacks. To disable that behaviour, you can tune

```
RateLimitBurst=0
```

(see `man 5 journald.conf`)

## Logs during a specific time period

This is my absolute favourite feature of journalctl, and I've regularly used it to narrow down logs from dozens of processes, especially when I don't know the root cause of an issue and need to watch all of the logs.

> > <code-in-title>The Old Way</code-in-title>
> > ```bash
> > zcat /var/log/* | egrep "(Jul\s*1 11:4.:..|2022-07-01 11:4.:..)"
> > ```
> > Hope everything logs in the same timezone! And no one uses UTC when there's a different system time configured.
> {: .code-in}
>
> > <code-out-title>The journalctl Way</code-out-title>
> > ```bash
> > journalctl --since "2022-07-01 11:40 CEST" --until "2022-07-01 11:50 CEST"
> > ```
> {: .code-out}
{: .code-2col}

This is very useful if you know when an error is, easily pull out entire system logs from around a specific timepoint. Everything is forced through journald's logging format, so you can be sure the timestamps are all correct, and `--since` and `--when` can be made timezone aware if you have reason to use that (e.g. a user reporting an issue in their current local time.) You can mix and match explicit time zone locales for even more fun:

```bash
journalctl --since "2022-06-21 14:24 Pacific/Auckland" --until "2022-06-21 14:30 Europe/Amsterdam"
```

The logs will be displayed in the last mentioned TZ.

## Logs during a specific time period, relatively

This is another flag that I use literally every day:

```bash
journalctl --since '1 hour ago'
```

If you know relatively when an issue started happening, it's a great way to filter on the logs that are interesting, without having to do clock math.

## Previous Boot Logs

Want to see what happened during a previous run of the system, during a previous boot? Use `--list-boots` to list all of the previous system restarts

```console
$ journalctl --list-boots
-14 143fdb7ac7a54370b41196c02affc3e8 Wed 2022-04-06 20:03:38 CEST—Mon 2022-04-11 08:26:00 CEST
-13 45801d31b0654fe3a4a7d5541e6ff71a Mon 2022-04-11 09:06:07 CEST—Wed 2022-04-20 13:37:31 CEST
-12 661e959635954aa0bbd6b800433e5097 Wed 2022-04-20 13:37:53 CEST—Thu 2022-04-28 17:26:32 CEST
-11 cedaef23e95849c39cfcdb1611b616a2 Thu 2022-04-28 17:40:33 CEST—Thu 2022-04-28 17:43:23 CEST
-10 6ac084d5256d4134bd6e395b490e99f2 Thu 2022-04-28 18:44:57 CEST—Mon 2022-05-09 11:21:33 CEST
 -9 680c9440c75d4699b6de48a56891d835 Mon 2022-05-09 11:21:54 CEST—Tue 2022-05-17 15:21:36 CEST
 -8 b8581fd6394144a9abc59b2832056037 Wed 2022-05-18 09:08:54 CEST—Fri 2022-05-20 17:33:36 CEST
 -7 8bb5c9d7d4ef4117ad8181d0314de7e1 Sun 2022-05-22 17:45:23 CEST—Thu 2022-06-02 16:35:45 CEST
 -6 5325407857054a828dbba6bb104efc80 Fri 2022-06-03 12:04:28 CEST—Thu 2022-06-09 10:15:40 CEST
 -5 0428cc6d11bc4aa9bd8e3999f9bd2c7f Thu 2022-06-09 10:20:05 CEST—Fri 2022-06-17 12:26:06 CEST
 -4 990910c70e79458394f5d9a97ae724ab Sat 2022-06-18 11:21:16 CEST—Tue 2022-06-21 12:14:47 CEST
 -3 d9a9088afacd4c6c9f85db49e9b321ce Tue 2022-06-21 14:29:20 CEST—Tue 2022-06-21 15:07:26 CEST
 -2 c3f0ec4fbc93410280f1e086d6533286 Tue 2022-06-21 15:07:45 CEST—Thu 2022-06-23 12:58:11 CEST
 -1 5d4d9c039daf4e5795a13211de4d6fd4 Thu 2022-06-23 12:58:31 CEST—Fri 2022-07-01 20:10:15 CEST
  0 490a1609fb5d422cad1e7135db88efe7 Mon 2022-07-04 09:32:12 CEST—Mon 2022-07-04 11:56:09 CEST
```

You can then see logs for those specific timeperiods with

```
$ journalctl -b -3 | head
-- Logs begin at Wed 2022-04-06 20:03:38 CEST, end at Mon 2022-07-04 11:59:13 CEST. --
jun 21 14:29:20 cosima kernel: microcode: microcode updated early to revision 0x88, date = 2021-03-31
jun 21 14:29:20 cosima kernel: Linux version 5.13.0-44-generic (buildd@lcy02-amd64-107) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #49~20.04.1-Ubuntu SMP Wed May 18 18:44:28 UTC 2022 (Ubuntu 5.13.0-44.49~20.04.1-generic 5.13.19)
```

All of the other filtering commands (`--since`, `--when`) natively search across boots so there's generally no need to supply this flag unless you want to restrict to a specific time period based on restarting.


## Advanced Log Filtering

There's a lot of additional information journald tracks with your logs, you can read them in the `verbose` format to see

```console
$ journalctl -o verbose
Wed 2022-04-06 20:03:38.281794 CEST [s=1b57c151fb2e49ad98a84ac45ed2e245;i=a310796;b=143fdb7ac7a54370b41196c02affc3e8;m=51917bf226;t=5dc002e4a3f42;x=4ae28fb4218f3691]
    _TRANSPORT=stdout
    _STREAM_ID=45b0adde513f46348564e6f9ce85e4b2
    PRIORITY=6
    SYSLOG_FACILITY=3
    SYSLOG_IDENTIFIER=influxd-systemd-start.sh
    MESSAGE=[httpd] 127.0.0.1 - - [06/Apr/2022:20:03:38 +0200] "POST /write?db=telegraf HTTP/1.1 " 204 0 "-" "Telegraf/1.21.4 Go/1.17.7" e265a273-b5d3-11ec-88d5-80fa5b94d68a>
    _PID=2491
    _UID=1001
    _GID=1001
    _COMM=influxd
    _EXE=/usr/bin/influxd
    _CMDLINE=/usr/bin/influxd -config /etc/influxdb/influxdb.conf
    _CAP_EFFECTIVE=0
    _SELINUX_CONTEXT=unconfined
    _SYSTEMD_CGROUP=/system.slice/influxdb.service
    _SYSTEMD_UNIT=influxdb.service
    _SYSTEMD_SLICE=system.slice
    _SYSTEMD_INVOCATION_ID=0ba7986947854acc938a93f020f96cee
    _BOOT_ID=143fdb7ac7a54370b41196c02affc3e8
    _MACHINE_ID=08854338f5f046d5a0d73f81ca5c2d79
    _HOSTNAME=cosima
```

All of these attributes can be used to filter your logs, just add them to your `journalctl` command like so:

```bash
journalctl _GID=1001
```

Or, for example, to get every log, by any process that logged, running under a specific user, like the Galaxy user:

```bash
journalctl _UID=$(getent passwd | grep galaxy | cut -f 3 -d : )
```

Or to see every log from a python program?

```bash
journalctl _COMM=python3
```

## Cleaning Logs

Journald comes with a nice tool for cleaning up logs, `journalctl --vacuum-...`. There are a couple different ways to invoke this. It's very useful if you left Galaxy poorly configured, reading bad XML files and crashing every minute, logging endlessly until you checked it again. That sort of behaviour generates an incredible number of logs.

First we can check how much disk space our logs are using:

```
$ journalctl --disk-usage
Archived and active journals take up 120.0M in the file system.
```

And then we can clean it!

Based on **size**:

```bash
journalctl --vacuum-size=4GB
```

Based on **age**

```bash
journalctl --vacuum-time="2 days"
```

Or based on number of **files**

```bash
journalctl --vacuum-files=3 # I've never used this one.
```

Note that this only applies to "archived" logs, not "active" ones, but you can use `journalctl --rotate` to force them to archived, before cleaning..


## Log Verification

Want to check if they've been tampered with (and someone forgot to recalculate checksums?). The `--verify` flag can be used to do that.

```
$ journalctl --verify
PASS: /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/user-1001@0005df9b9d17faa5-c4276c13359e16e1.journal~
File corruption detected at /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/system@0005df9b9c8ebfda-d41419e76ede7f2c.journal~:3f070f8 (of 67108864 bytes, 98%).
FAIL: /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/system@0005df9b9c8ebfda-d41419e76ede7f2c.journal~ (Bad message)
PASS: /var/log/journal/08854338f5f046d5a0d73f81ca5c2d79/system@ede9cabbb3874bd98bcb0b16ada49d8f-000000000a6ce684-0005de680b29a077.journal
7e5abf8: Unused data (entry_offset==0)
..  94%
```

## Further Reading

- [man 5 journald.conf](https://manpages.ubuntu.com/manpages/bionic/man5/journald.conf.5.html)
- [man 1 journalctl](https://manpages.ubuntu.com/manpages/bionic/man1/journalctl.1.html)
- [man 7 systemd.journal-fields](https://manpages.ubuntu.com/manpages/bionic/man7/systemd.journal-fields.7.html)


topic: admin
tutorial: upgrading
title: Upgrading Galaxy
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/upgrading/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible"]
content_hash: 1e877503edfc0503d87f3c024c78b2c1
last_modified: 2026-02-26T21:12:14.167836
workflows_json: []
content:
# Overview


One of the most important and recurring tasks for a Galaxy server admin is to keep the server up to date with security patches, minor fixes and major version upgrades.

Luckily for us, by using the best practice Ansible installation method that we have used throughout these tutorials, keeping our Galaxy server up to date is extremely easy.

We can get all the latest patches and fixes for our currently installed version of Galaxy by just re-running the Ansible playbook on a regular basis.

We can upgrade our Galaxy version by modifying the Ansible var files and re-running the playbook. It's that simple.

This tutorial will show you how and discuss some of the things you need to keep in mind whenever you are updating or upgrading your Galaxy server.

## Some notes on Galaxy versioning and releases.

* As at June 2021, Galaxy has three new version releases per year - in each of January, May and September. The versions are denominated by the year and month - `YY.MM`. i.e. January 2021's Galaxy release is version *21.01*.
* Official Galaxy versions are tagged on the [Galaxy Project's GitHub releases page](https://github.com/galaxyproject/galaxy/releases) as `vYY.MM`. e.g. *v21.01*
* Each version also has an official branch in the GitHub repo named `release_YY.MM`. e.g. *release_21.01*
* Galaxy versions will be supported with security fixes, bug patches and other improvements for [one year](https://github.com/galaxyproject/galaxy/blob/dev/SECURITY.md#supported-versions) from the time of release at which time they will become "End of Life" and will no longer be supported.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## This tutorial assumes that:

- You have a VM or machine where you you have Galaxy installed.
- You have completed at least the "Galaxy Installation with Ansible" tutorial.
- You have command line and sudo access to the VM/computer where it is installed.

> <tip-title>Galaxy server version installed by the tutorials</tip-title>
> At the time of writing, the version of Galaxy that was installed with the "Galaxy Installation with Ansible" tutorial is *20.09*. This tutorial is designed to upgrade the server to *21.01*. If you already have a version equal to or higher than *21.01* running on your server, never fear! The process outlined below can still be used to upgrade to the latest version of Galaxy - just swap out *21.01* used below for the current latest version.
{: .tip}

# Find your current Galaxy server version

The first thing we will do in this tutorial is find the current installed version of the Galaxy. There are a few methods for determining this - You can find it in the GUI if you are an admin user or you can get it via the server API.

Once we know what our current version is, we can compare it with the latest available version

> <hands-on-title>Get the current server version</hands-on-title>
>
> **Method 1: Via the Admin UI page**
> The currently installed version of Galaxy is shown on the Galaxy server's Admin UI page.
> 1. Click on the **Admin** menu item to bring up the Admin UI.
>
> 2. The version of Galaxy currently running is displayed in the top left corner of the screen.
>
> ![Screenshot of the Admin UI page showing the Galaxy server version](../../images/admin_page_version.png "Screenshot of the Admin UI page showing the Galaxy server version.")
>
> **Method 2: Via the API**
>
> 1. In your browser - go to the following URL: [`https://galaxy.example.org/api/version`](https://my.gat.galaxy.training/?path=/api/version) (replacing `galaxy.example.org` with the hostname of your Galaxy server). The version will be displayed as follows:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```shell
>    > curl https://galaxy.example.org/api/version
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>Browser window</code-out-title>
>    > ```JSON
>    > {
>    >   "version_major": "20.09",
>    >   "extra": {}
>    > }
>    > ```
>    {: .code-out}
>
> Both methods above show the version of Galaxy that has been installed. At the time of writing, on this server it's *20.09*. Your version may differ depending on which version you installed initially.
{: .hands_on}


# Finding the latest release version

So, we have found out which version we are running on our server. But what is the current latest version of Galaxy that is available? There are quite a few ways of finding out.

* Galaxy is released 3 times per year. If the time of year is around January/February, April/May or September/October there may be a new release about to be announced.
* You can check the Galaxy Project Newsletter and home page [https://galaxyproject.org](https://galaxyproject.org)

Probably the easiest way of finding the version number of the latest release is via the extensive Galaxy Documentation Website located at: [https://docs.galaxyproject.org](https://docs.galaxyproject.org).

This site shows all of the release notes of all the available Galaxy versions and always has the latest release at the top of the list.

![Screenshot of the Galaxy documentation webpage showing the release versions](../../images/galaxy_doc_releases.png "Screenshot of the Galaxy documentation webpage (<a href="https://docs.galaxyproject.org">https://docs.galaxyproject.org</a>) showing the release versions. The latest version is at the top of the 'Releases' list.")

We can see from this page that the latest release version of Galaxy is *21.01* (at the time of writing.)

> <question-title>What is the current latest version?</question-title>
> As previously mentioned, the current latest release of Galaxy may not be *21.01*. What version is the latest that you can see?
{: .question}

## The importance of the release notes

Each release of Galaxy is accompanied by extensive release notes in two different "flavours."

* "Full release notes" which have extensive detail about all changes made since the last release. This version of the release notes is targeted to Administrators who may need to know about minor fixes, configuration details, security issues, new features and other things found in the latest version.
* "User facing" release notes which provide an visual and interactive summary of the release highlights, additions and new features targeted towards end users, trainers and funders etc.

Before upgrading your Galaxy server to the latest version (or even a new version for that matter) - It is **HIGHLY recommended** that you read the release notes for that version. The notes are *very extensive* and may contain important information on any new features, links  and how to configure them for your server.

![Example section of release notes for release 21.01](../../images/release_notes_2101.png "An example selection from the release notes for Galaxy version 21.01 showing details on enhancements made with their associated Pull Requests in the GitHub repository.")

# Back up the Galaxy database

At the heart of every Galaxy server beats its database. The database is the most important part of the Galaxy server as it contains all of the metadata required for the server to operate and records details of all users, jobs, datasets etc since the server was started.

Sometimes new versions of Galaxy make changes to the database schema that require a database upgrade.

It is highly important that we back it up before making any substantial changes to the Galaxy server as things sometimes go wrong (inopportune power failures, disk crashes etc) and losing or corrupting the database is possible during an upgrade operation.

In a production environment it is really important to make regular backups of the database in any case. The **usegalaxy.\*** servers all perform at least daily backups of their respective databases and even have replicate database servers *just in case.*

> <hands-on-title>Back up the Galaxy database</hands-on-title>
>
> We will be using the PostgreSQL `pg_dump` command to dump the complete contents of the Galaxy database to a file. From the usage text of the command:
>
>   {% raw %}
>   ```
>   pg_dump --help
>   pg_dump dumps a database as a text file or to other formats.
>
>   Usage:
>     pg_dump [OPTION]... [DBNAME]
>   ```
>   {% endraw %}
>
> We will use `pg_dump` and pipe its output through `gzip` to an output file to create a compressed text (SQL) representation of the database. We will need to do this as the `galaxy` user. It's a good idea to create the backup file name with a sensible name and then copy it somewhere else like a cloud service.
>
> From your terminal -
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sudo su -l galaxy
>    > pg_dump galaxy | gzip > galaxy_database-2009-1.sql.gz
>    > ls -lah galaxy_database*
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    > ```bash
>    > -rw-rw-r-- 1 galaxy galaxy 94K Jun  8 04:12 galaxy_database_2009-1.sql.gz
>    > ```
>    {: .code-out}
>
> As you can see, the database on our server isn't currently huge though this will changes as your server gets more users, runs more jobs and stores more data. The current size for Galaxy Australia's compressed database backup is ~11GB and growing.
{: .hands_on}

Now that we have backed up our database we can move on to upgrading our Galaxy server to the latest version. For the sake of this tutorial, we will assume that the installed version is different to the latest version (at the time of writing - *20.09* vs *21.01*). If that is not the case for you, then you can instead use this tutorial as a reference in the future.

# Upgrading Galaxy to a new version

If you've been following best practices and using Ansible to install and configure your Galaxy server then you're in luck - this step is going to be very easy! We just need to adjust the Galaxy version number in our `group_vars/galaxyservers.yml` file and re-run the playbook. If you haven't been using Ansible then there are some notes on performing an upgrade located at: [https://galaxyproject.org/admin/maintenance/](https://galaxyproject.org/admin/maintenance/). The Ansible method is much preferred and simple!

There are however a few things to consider before doing the actual upgrade, especially in a production environment.

* Galaxy will be restarted.
    * Is this an appropriate time for that to happen?
    * Does it need to be scheduled with your users?
* The database schema may change
    * Have you backed up the database?

Once you've made sure of these things, it's time to do it!

> <hands-on-title>Back up the Galaxy database</hands-on-title>
>
> In this tutorial we will upgrade the server to version *21.01*. This may be different for you. Please use the latest version instead.
>
> 1. Edit the `group_vars/galaxyservers.yml` file and change the `galaxy_commit_id` to the latest version. Use the form `release_YY.MM` or in this case `release_21.01`.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -21,7 +21,7 @@ galaxy_manage_paths: true
>     galaxy_layout: root-dir
>     galaxy_root: /srv/galaxy
>     galaxy_user: {name: galaxy, shell: /bin/bash}
>    -galaxy_commit_id: release_20.09
>    +galaxy_commit_id: release_21.01
>     galaxy_force_checkout: true
>     miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
>     miniconda_version: 4.7.12
>    ```
>    {% endraw %}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Now run the playbook!
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

You should see in the Ansible output that Galaxy had to rebuild the client, update the database, rebuild its virtual environment and restart. If you weren't using Ansible, these steps would all be manual and important not to get wrong. Ansible makes the entire process super simple for us.

Congratulations, you're Galaxy server should now be updated to the latest version. You can check this by going to the Admin UI page or by using the API as described above.

# Ongoing maintenance

The Galaxy development community continually support release versions of Galaxy [for 1 year.](https://github.com/galaxyproject/galaxy/blob/dev/SECURITY.md#supported-versions) They regularly update releases with bug fixes, patches and security enhancements. Therefore it is important that we as administrators keep our Galaxy servers up to date regardless of which release we are running.

The easiest way to do this is to regularly re-run the Ansible playbook. It will check with GitHub and grab any new changes to Galaxy since the last time you ran it and automatically keep everything up to date. The large **usegalaxy.\*** servers all re-run the playbook on a regular schedule for this purpose.

Regular running of the playbook also encourages you and your team members to make changes to the Ansible scripts not the live system (as live changes will be overwritten). This leads to a much more robust and consistent production Galaxy server for your users.

# Conclusion

This tutorial has shown you how to keep your Galaxy server up to date with the latest version painlessly and simply following current best practice. We hope you enjoyed this tutorial and got something out of it.


topic: admin
tutorial: celery
title: Setting up Celery Workers for Galaxy
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/celery/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "git-gat"]
content_hash: 7aa29a7236e9c8a03df25e1bd9d77ec9
last_modified: 2026-02-26T21:12:14.137913
workflows_json: []
content:
Celery is a distributed task queue written in Python that can spawn multiple workers and enables asynchronous task processing on multiple nodes. It supports scheduling, but focuses more on real-time operations.

From the Celery website:

> "Task queues are used as a mechanism to distribute work across threads or machines.
>
>A task queue’s input is a unit of work called a task. Dedicated worker processes constantly monitor task queues for new work to perform.
>
>Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker.
>
>A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.
>
>Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there’s node-celery and node-celery-ts for Node.js, and a PHP client.
>
>Language interoperability can also be achieved exposing an HTTP endpoint and having a task that requests it (webhooks)."
>
{: .quote cite="https://docs.celeryq.dev/en/stable/getting-started/introduction.html#what-s-a-task-queue"}

[A slideshow presentation on this subject is available](slides.html).

If you are not interesting in managing Redis and Flower, you might be interested in the [lower-configuration deployment option]({% link topics/admin/tutorials/celeryless/tutorial.md %}).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="celery" %}

The agenda we're going to follow today is: We're going to enable and configure celery, install a Redis server, the Flower dashboard and start Celery workers.

# Installing and Configuring

To proceed from here it is expected that:

<!--  TODO: port assumptions are not correct for GAT. -->

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have set up a working Galaxy instance as described in the [ansible-galaxy](../ansible-galaxy/tutorial.html) tutorial.
>
> 2. You have a working RabbitMQ server installed and added the connection string to the galaxy configuration. (RabbitMQ is installed when doing the [Pulsar](../pulsar/tutorial.html) tutorial.)
>
> 3. Your VM has a public DNS name: this tutorial sets up SSL certificates from the start and as an integral part of the tutorial.
>
> 4. You have the following ports exposed:
>
>    - 22 for SSH, this can be a different port or via VPN or similar.
>    - 80 for HTTP, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 443 for HTTPs, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 5671 for AMQP for Pulsar, needed if you plan to setup Pulsar for remote job running.
>
{: .comment}
In order to run a production ready Celery setup, we need to discuss and install some other software that works together with Celery.  
We already learned about RabbitMQ in the Pulsar tutorial. The RabbitMQ server you already installed there will be our broker for Celery.
As a backend we are going to use Redis.  
Redis is a very popular key-value-store database. It is very fast and easy to set up backend for Celery.
If you want to learn more about Redis, visit their website: [https://redis.io/](https://redis.io/)

For monitoring and debugging Celery, we use the [Flower](https://github.com/mher/flower) dashboard.  
Flower is lightweight and has a clear but powerful UI and can be installed in Galaxy's venv using our role.

# Installing and Configuring

First we need to add our new Ansible Roles to the `requirements.yml`:

> <hands-on-title>Set up Redis, Flower, Systemd and Celery with Ansible</hands-on-title>
>
> 1. In your working directory, add the roles to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -37,3 +37,8 @@
>       version: 1.4.4
>     - src: galaxyproject.pulsar
>       version: 1.0.10
>    +# Celery, Redis, and Flower (dashboard)
>    +- name: geerlingguy.redis
>    +  version: 1.8.0
>    +- name: usegalaxy_eu.flower
>    +  version: 2.1.0
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement" data-ref="add-req"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the roles with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true" data-ref="req-install"}
>    {: .code-in}
>
> 3. Let's go now through all the Roles step-by-step:
>
>     1. Since we can stick to the basic default settings of Redis, we will look only at a few variables:
>
>        | Variable               | Type    | Description                                                                 |
>        | ----------             | ------- | -------------                                                               |
>        | `redis_port`           | integer | The port Redis should listen on. 6379 by default.                           |
>        | `redis_bind_interface` | string  | The interface Redis should listen to. 127.0.0.1 is default.                 |
>        | `redis_conf_path`      | string  | The path where your redis configuration will be stored. Default: /etc/redis |
>
>        Luckily we can leave them all on default and don't need to change anything for Redis in the vars.  
>     2. We only need to add Redis' Python package in the `group_vars/galaxyservers.yml`:
>        {% raw %}
>        ```diff
>        --- a/group_vars/galaxyservers.yml
>        +++ b/group_vars/galaxyservers.yml
>        @@ -283,3 +283,7 @@ rabbitmq_users:
>         # TUS
>         galaxy_tusd_port: 1080
>         galaxy_tus_upload_store: /data/tus
>        +
>        +#Redis
>        +galaxy_additional_venv_packages:
>        +  - redis
>        {% endraw %}
>        ```
>        {: data-commit="Configure rabbitmq users"}
>        
>     3. Let's add the role to our playbook then:
>        {% raw %}
>        ```diff
>        --- a/galaxy.yml
>        +++ b/galaxy.yml
>        @@ -42,6 +42,7 @@
>             - role: galaxyproject.miniconda
>               become: true
>               become_user: "{{ galaxy_user_name }}"
>        +    - geerlingguy.redis
>             - galaxyproject.nginx
>             - geerlingguy.docker
>             - usegalaxy_eu.rabbitmqserver
>        {% endraw %}
>        ```
>        {: data-commit="Add requirement" data-ref="add-req"}
>
>     4. Because Flower needs it's own RabbitMQ user, we should add that to the respective part of our vars
>        Edit your `group_vars/secret.yml` and define some random passwords:
>
>        ><code-in-title>Bash</code-in-title>
>        > ```
>        > ansible-vault edit group_vars/secret.yml
>        > ```
>        {: .code-in}
>
>        ```yaml
>        vault_rabbitmq_password_flower: "a-really-long-password-here"
>        vault_rabbitmq_password_galaxy: "a-different-really-long-password"
>        vault_flower_user_password: "another-different-really-long-password"
>        ```
>
>        <!-- Ignore this, just for the gat-automation. Vaults are ugly to work with :(
>
>        {% raw %}
>        ```diff
>        --- a/group_vars/secret.yml
>        +++ b/group_vars/secret.yml
>        @@ -1,13 +1,22 @@
>         $ANSIBLE_VAULT;1.1;AES256
>        -62346261323266656232393034396134316636376533376139666437363535393562663838613938
>        -6336666266633563346337623265353935646361326337610a393834333233313461346439376438
>        -63383338346530656561636631666134373238366364363164313166346461383736613162653237
>        -3461363334323431370a656132303965653262386130353332623937376261396530393761353834
>        -38336565666437666436643163363831633331333766653266356163613138393734656465323634
>        -39366362383433366437353534663134313330316337393335383962613961386665633261616237
>        -35366635373063313631323939396164336330356361393464326636353037336461323531336434
>        -35613933303333623031353936393265636130363335376533393335663266313863376135383338
>        -36613464373231623938373434306266373234633036343636633963353361356631363533353066
>        -39323064336237646432323530313065303331326636353334343862373330313133326363363063
>        -38383564636161396435666164643334656435393533643163393434623434656238633631633939
>        -33353232666432376661
>        +66653366633665383231303635643739396466653465626163633662623230643534346666303434
>        +3531356339646666623364383435306363373534373833370a333964306130653236373438373264
>        +34366433396135303064643932313135643064373034323865363939333565623238386161333465
>        +3932623162353034370a653663333363383938393936623663343639376430653366646662353830
>        +34383964326161383966306361633162653366353162633564623738353137333936313232363764
>        +38383962396637636136316366633936316365643038333030333932336365326335373737663834
>        +39363639303764343430336435313663313762353335366562656232646334356438343535303032
>        +35636131323163323036383163363964643237313137333131303737346662373233663562343031
>        +39396531306539623661376166313534303462623532323334393736316634616262313633626637
>        +37393066663239303564356232366466333334316565363631363230626665333039386133383933
>        +31656234363064646334623938383637393534313638643635396662643163346633346237353664
>        +34666161373764663166653737323736373261363565666633653831363833393264333666356633
>        +30373062333638366666316132623736386336383933326539663065653538386363323766313664
>        +63323737396264306439303366343834326164393763376663316366353766663131303966393039
>        +61343732313865613761623733356465336263633963376161663931653864383162393839643834
>        +34373239386461666462643162613536316166656135323332316163336635393035653164363138
>        +34336265383033663436306436663536383238366665653239313661666431666339323364646633
>        +64366361653034393962316562373835623631356339313062393933633634653735656636373031
>        +65376162633866623733623961303664663931636130373936326461623465363932373165356563
>        +36323838323865326464346337393164343765613333663830636234666663303535616532323936
>        +3230
>        {% endraw %}
>        ```
>        {: data-commit="Add rabbitmq passwords to the vault"}
>
>        -->
>
>        This is going in the vault as they are secrets we need to set. Flower needs it's own RabbitMQ user with admin access and we want a different vhost for galaxy and celery.
>
>        Replace both with long random (or not) string.  
>        Now add new users to the RabbitMQ configuration:
>        {% raw %}
>        ```diff
>        --- a/group_vars/galaxyservers.yml
>        +++ b/group_vars/galaxyservers.yml
>        @@ -270,6 +270,7 @@ rabbitmq_config:
>
>         rabbitmq_vhosts:
>           - /pulsar/pulsar_au
>        +  - galaxy_internal
>
>         rabbitmq_users:
>           - user: admin
>        @@ -279,6 +280,13 @@ rabbitmq_users:
>           - user: pulsar_au
>             password: "{{ vault_rabbitmq_password_vhost }}"
>             vhost: /pulsar/pulsar_au
>        +  - user: galaxy
>        +    password: "{{ vault_rabbitmq_password_galaxy }}"
>        +    vhost: galaxy_internal
>        +  - user: flower
>        +    password: "{{ vault_rabbitmq_password_flower }}"
>        +    tags: administrator
>        +    vhost: galaxy_internal
>
>         # TUS
>         galaxy_tusd_port: 1080
>        {% endraw %}
>        ```
>        {: data-commit="Configure rabbitmq users"}
>     5. Flower
>        Flower has a few variables, too, for example, we need to point it to our virtual environment:
>
>        | Variable             | Type          | Description                                                                                                                                                                    |
>        | ----------           | -------       | -------------                                                                                                                                                                  |
>        | `flower_python_version`         | string        | Python version to use when installing flower to a venv. Default: python39                               |
>        | `flower_port`         | integer        | The port Flower should listen on. 5555 by default.                               |
>        | `flower_bind_interface`         | string | The interface Flower should listen to. 0.0.0.0 is default.  |
>        | `flower_conf_dir`  | string | The path where your Flower configuration will be stored. Default: /etc/flower |
>        | `flower_venv_dir`  | string | The path to the venv where Flower should be installed. Default: `/home/{{ flower_user }}/.local` |
>        | `flower_user`  | string | User that owns the flower process. Default: galaxy |
>        | `flower_group`  | string | Group that owns the flower process. Default: galaxy |
>        | `flower_ui_users`  | list of dicts | Name and password of the UI users for basic auth. |
>        | `flower_app_dir`  | string | Root directory of your Python app to run with Celery. In our case `galaxy_root` |
>        | `flower_app_name`  | string | Python module to import. In our case 'galaxy.celery' |
>        | `flower_python_path`  | string | Should point to galaxy's `server/lib` directory (default) |
>        | `flower_broker_api`  | string | URL to broker's API with login credentials. |
>        | `flower_broker_url`  | string | Flower's RabbitMQ connection string. |
>        | `flower_db_file`  | string | When Flower is in persistent mode, use this path for the database. |
>
>        Let's add variables to our `group_vars/galaxyservers.yml`:
>
>        {% raw %}
>        ```diff
>        --- a/group_vars/galaxyservers.yml
>        +++ b/group_vars/galaxyservers.yml
>        @@ -295,3 +295,22 @@ galaxy_tus_upload_store: /data/tus
>         #Redis
>         galaxy_additional_venv_packages:
>           - redis
>        +
>        +# Flower
>        +flower_python_version: python3
>        +flower_app_dir: "{{ galaxy_root }}"
>        +flower_python_path: "{{ galaxy_root }}/server/lib"
>        +flower_venv_dir: "{{ galaxy_venv_dir }}"
>        +flower_app_name: galaxy.celery
>        +flower_db_file: "{{ galaxy_root }}/var/flower.db"
>        +flower_persistent: true
>        +flower_broker_api: "https://flower:{{ vault_rabbitmq_password_flower }}@localhost:5671/api/"
>        +flower_broker_url: "amqp://flower:{{ vault_rabbitmq_password_flower }}@localhost:5671/galaxy_internal?ssl=true"
>        +flower_proxy_prefix: /flower
>        +
>        +flower_ui_users:
>        +  - name: admin
>        +    password: "{{ vault_flower_user_password}}"
>        +
>        +flower_environment_variables:
>        +  GALAXY_CONFIG_FILE: "{{ galaxy_config_file }}"
>        {% endraw %}
>        ```
>        {: data-commit="Configure flower"}
>     6. It has a dashboard, so we need to expose that via nginx:
>
>        {% raw %}
>        ```diff
>        --- a/templates/nginx/galaxy.j2
>        +++ b/templates/nginx/galaxy.j2
>        @@ -94,4 +94,13 @@ server {
>         		proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
>         		proxy_set_header X-Forwarded-Proto $scheme;
>         	}
>        +
>        +	location /flower {
>        +		proxy_pass http://localhost:5555;
>        +		proxy_set_header Host $host;
>        +		proxy_redirect off;
>        +		proxy_http_version 1.1;
>        +		proxy_set_header Upgrade $http_upgrade;
>        +		proxy_set_header Connection "upgrade";
>        +	}
>         }
>        {% endraw %}
>        ```
>        {: data-commit="Add nginx routes"}
>
>     7. Now we can add the Flower Role to our Playbook:
>    
>        {% raw %}
>        ```diff
>        --- a/galaxy.yml
>        +++ b/galaxy.yml
>        @@ -43,6 +43,7 @@
>               become: true
>               become_user: "{{ galaxy_user_name }}"
>             - geerlingguy.redis
>        +    - usegalaxy_eu.flower
>             - galaxyproject.nginx
>             - geerlingguy.docker
>             - usegalaxy_eu.rabbitmqserver
>        {% endraw %}
>        ```
>        {: data-commit="Add flower role" data-ref="add-req"}
>
> 4. Now it is time to change the `group_vars/galaxyservers.yml` and enable celery in galaxy.gravity config.
>    Add the following lines to your file:
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -130,6 +130,11 @@ galaxy_config:
>           preload: true
>         celery:
>           concurrency: 2
>    +      enable_beat: true
>    +      enable: true
>    +      queues: celery,galaxy.internal,galaxy.external
>    +      pool: threads
>    +      memory_limit: 2
>           loglevel: DEBUG
>         tusd:
>           enable: true
>    {% endraw %}
>    ```
>    {: data-commit="Add celery" data-ref="add-req"}
>
>    Now add the second part, Galaxy's Celery configuration:
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -111,6 +111,11 @@ galaxy_config:
>         # Data Library Directories
>         library_import_dir: /libraries/admin
>         user_library_import_dir: /libraries/user
>    +    # Celery
>    +    amqp_internal_connection: "pyamqp://galaxy:{{ vault_rabbitmq_password_galaxy }}@localhost:5671/galaxy_internal?ssl=1"
>    +    celery_conf:
>    +      result_backend: "redis://localhost:6379/0"
>    +    enable_celery_tasks: true
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add celery-redis" data-ref="add-req"}
>
> 6. We are done with the changes and you can enter the command to run your playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>    This should also restart Galaxy and spawn the amount of Celery workers, that we defined in the Gravity configuration.
>
{: .hands_on}

# Test Celery

Now that everything is running, we want to test celery and watch it processing tasks.
We can simply do that by starting an upload to our Galaxy.

> <hands-on-title>Test Celery and monitor tasks with Flower</hands-on-title>
> 1. First, open a new tab and enter your machines hostname followed by `/flower/dashboard` then log in with `username: admin` and your password.
>    You should see an overview with active workers.  
>    Keep that tab open
> 2. In split view, open a second browser window and open your Galaxy page.
>    Click on {% icon galaxy-upload %} Upload Data, select a file from your computer and click `upload`.
> 3. The Workers should now receive a new tasks. Click on `Succeeded` and then on the UUID of the last upload task.  
>    You should see all its details here and the info that it was successful.
{: .hands_on}

{% snippet topics/admin/faqs/missed-something.md step=12 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="celery" %}


topic: admin
tutorial: k8s-deploying-galaxy
title: Galaxy Installation on Kubernetes
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["kubernetes"]
content_hash: 254d1e0691bf605492077060dd4e4ce3
last_modified: 2026-02-26T21:12:14.148127
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/k8s-deploying-galaxy/tutorial.html

content:
# Galaxy Helm Chart

## Overview

This tutorial describes how to use the Galaxy Helm Chart to deploy a production
grade instance of Galaxy on a Kubernetes cluster. The Helm Chart has been designed
to follow best practices adopted by the community, including the usegalaxy.* federation,
and will install a Galaxy with the following features by default:
- Zero-downtime configuration changes and upgrades
- Scalable web and job handlers
- Automatic failure recovery based on liveness and readiness probes
- A built-in nginx for efficiently serving large files
- TUSD for resumable uploads
- Celery for background jobs
- Access to CVMFS reference data
- A toolset matching the usegalaxy.* federation (also served off CVMFS)
- Interactive tools (wildcard DNS mapping required)
- Minimal privileges, with jobs running as non-root and only having access to datasets they need
- Automatic maintenance scripts to cleanup the galaxy database and tmp directories

Optionally, the chart can be configured with
- High-availability components - this includes trivial scaling of clustered Postgres, Rabbit MQ etc.
- Replacement components - You can replace the built-in operators with a managed or existing Postgres database (e.g. Amazon RDS), RabbitMQ cluster etc.
- Use S3 as an alternative to CVMFS
- Automatic scraping of metrics which can be sent to Influxdb

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Prerequisites
Some familarity with Kubernetes is assumed. This includes general administrative
familarity and how to install and configure Helm Charts.

A running Kubernetes cluster is also required (1.27 or higher), with Helm
(3.5 or higher) configured to access it.
For development and testing purposes this can be easily achieved by installing
[Docker Desktop locally and enabling Kubernetes][DfD]. Afterwards, install
[Helm](https://helm.sh).

For production deployments, we'll also need some storage resources for data
persistence. This can be done by either defining a [storage class][SC] or
creating a [Persistent Volume and a corresponding Persistent Volume Claim][PV].
Once created, just keep a note of the resources Persistent Volume Claim ID and
to use later.

## Deploying the Default Configuration
The default set of values for the Galaxy chart configures only a minimal set
of Galaxy options necessary. The configured options are required for suitable
operation of the system. Setting other options will depend on the environment
and it's best to refer to the general [Galaxy documentation][GxyDocs]; we'll
also take a look at how to make configuration changes in the context of the
chart later in this tutorial.

> <hands-on-title>Deploying the Galaxy Helm Chart</hands-on-title>
>
> 1. First, we need to add the helm repository for the chart. The chart is
> automatically packaged, versioned and uploaded to a helm repository on github
> with each accepted PR. Therefore, the latest version of the chart can be directly
> installed from that repository.
>
>    {% raw %}
>    ```bash
>    helm repo add galaxyproject https://raw.githubusercontent.com/galaxyproject/helm-charts/master/
>    helm repo update
>    ```
>    {% endraw %}
>
> 2. We can now deploy Galaxy via the Chart. Running this command will create a new
>    Helm release (i.e., chart installation) called `mygalaxy`.
>
>    {% raw %}
>    ```bash
>    helm install mygalaxy galaxyproject/galaxy
>    ```
>    {% endraw %}
>
> 3. It will take about a minute or two for the necessary containers to download,
>    the database to initialize, and Galaxy processes to start. Ultimately,
>    while this may depend on the Kubernetes cluster setup you are using,
>    Galaxy should be available at https://<root URI>/galaxy for the given machine. We can
>    always check the status of our release by typing `helm status galaxy`.
>
{: .hands_on}

## Setting the admin user and changing the brand
The chart is designed to follow standard Kubernetes and Helm idioms, and therefore,
it should be intuitively similar to the steps required to change configuration in
any other Helm chart. For example, ingress paths, resource allocations, container
images etc. can be changed following standard helm conventions. The list of
available configuration options are also documented in the Galaxy Helm
_[Chart repository](https://github.com/galaxyproject/galaxy-helm/tree/master?tab=readme-ov-file#configuration)_

To change Galaxy specific configuration, such as setting the admin user or change the brand in `galaxy.yml`,
we can follow the following steps. Once done, we will also rollback our change to demonstrate how Helm manages
configuration.

> <hands-on-title>Setting admin user and changing the brand</hands-on-title>
>
> 1. Modify the following entries in your `mygalaxy.yml`. Make sure to add these
>    keys under the `configs:` section of the file.
>
>    {% raw %}
>    ```
>    configs:
>      galaxy.yml:
>        galaxy:
>          brand: "Hello World"
>          admin_users: "admin@mydomain.com"
>    ```
>    {% endraw %}
>
> 2. Now, let’s upgrade the chart to apply the new configuration.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values -f mygalaxy.yml mygalaxy galaxyproject/galaxy
>    ```
>    {% endraw %}
>
> 3. Inspect the currently set Helm values by:
>
>    {% raw %}
>    ```bash
>    helm get values mygalaxy
>    ```
>    {% endraw %}
>
> 4. List the installed Helm charts again and note that the revision of the chart has changed as expected.
>
>    {% raw %}
>    ```bash
>    helm list
>    NAME  	  REVISION	UPDATED                 	STATUS  	CHART                 	APP VERSION	NAMESPACE
>    mygalaxy	2       	Wed Jun 26 14:51:17 2023	DEPLOYED	galaxy-5.14.2          	v24.0.2    	default
>    ```
>    {% endraw %}
>
> 5. Revisit the Galaxy Application in your browser to check whether the settings have changed. This will
>    take a short while (< 1 minute) for the new container to come up. You should experience no downtime.
>
> 6. Let’s now roll back to the previous revision.
>
>    {% raw %}
>    ```bash
>    helm rollback mygalaxy 1
>    ```
>    {% endraw %}
>
>    Use `helm get values` again to observe that the values have reverted to
>    the previous revision. After a short while, once the new container is up
>    and running, Kubernetes will automatically switch over to it and you can
>    see that the previous configuration has been restored.
>
{: .hands_on}

## Deleting a Deployed Helm Release
By default, the Helm chart is designed to install all required dependencies, so that it's easy
to get an instance up and running quickly for experimentation. However, in production, we
recommend installing the dependency charts separately, once per cluster, by installing
Galaxy with helm options
`--set postgresql.deploy=false --set s3csi.deploy=false --set cvmfs.deploy=false --set rabbitmq.deploy=false`.

This is particularly important during uninstallation, where orderly destruction of dependencies is often required
For example, if the rabbitmq operator is uninstalled before the rest of the Galaxy helm chart is deleted, there will be
no operator left to cleanup rabbitmq resources. Installing the aforementioned operators separately sidesteps this problem.

> <hands-on-title>Deleting a Deployed Helm Release</hands-on-title>
>
>    {% raw %}
>    ```bash
>    helm delete mygalaxy
>    helm delete mycvmfs # and any other operators
>    ```
>    {% endraw %}
>
{: .hands_on}

# Next Steps
This tutorial covers the basics of getting Galaxy deployed on Kubernetes using
Helm. There is a lot more to understanding all the configuration options for
the chart and the available deployment models. For more info on some of these
topics, take a look at the [Galaxy Helm chart] repository as well as
[other tutorials](../..) tagged with _kubernetes_. Also, feel free to reach out
on Gitter: [https://gitter.im/galaxyproject/FederatedGalaxy][fedG].


[Galaxy Helm chart]: https://github.com/galaxyproject/galaxy-helm
[DfD]: https://www.docker.com/products/kubernetes
[SC]: https://kubernetes.io/docs/concepts/storage/storage-classes/
[PV]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
[CSI]: https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/
[GxyDocs]: https://galaxyproject.org/admin/#configuration
[CVMFS]: https://training.galaxyproject.org/training-material/topics/admin/tutorials/cvmfs/tutorial.html
[BioContainers]: https://biocontainers.pro/#/
[Namespace]: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
[reclaim policy]: https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/
[fedG]: https://gitter.im/galaxyproject/FederatedGalaxy


topic: admin
tutorial: interactive-tools
title: Galaxy Interactive Tools
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "interactive-tools"]
content_hash: a5ab955eb3572baac1b92594dd56bf6c
last_modified: 2026-02-26T21:12:14.146560
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/interactive-tools/tutorial.html

content:
Galaxy Interactive Tools (GxITs) are a method to run containerized tools that are interactive in nature. Interactive Tools typically run a persistent service accessed on a specific port, until terminated by the user. One common example of such a tool is [Jupyter Notebook][jupyter]. Galaxy Interactive Tools are submitted through Galaxy's job management system and thus are scheduled the same as any other Galaxy tool - on a Slurm cluster, for instance. Galaxy Interactive Tools were introduced in Galaxy Release 19.09.

> <comment-title>Evolving Topic</comment-title>
> Galaxy Interactive Tools are a **relatively new and rapidly evolving feature** and there are some rough edges. Work to improve the experience of deploying and using them is ongoing. Please watch the [Galaxy Release Notes][galaxy-release-notes] for updates, changes, new documentation, and bug fixes.
>
> You may find extra information about Interactive Tools on the [Galaxy Documentation][galaxy-docs-interactivetools].
{: .comment}

[galaxy-release-notes]: https://docs.galaxyproject.org/en/master/releases/index.html
[galaxy-docs-interactivetools]: https://docs.galaxyproject.org/en/master/admin/special_topics/interactivetools.html

> <warning-title>Before You Continue - Wildcard DNS Certificates</warning-title>
> If you are *not* completing this tutorial as part of a [Galaxy Admin Training][gat] course, **you will need a wildcard DNS record for your Galaxy server and a method for obtaining a wildcard SSL certificate for your Galaxy server**.
>
> Galaxy Interactive Tools require a [wildcard SSL certificate][wildcard-cert]. Because the **Galaxy Installation with Ansible** tutorial fetches [Let's Encrypt][lets-encrypt] certificates, this tutorial fetches Let's Encrypt wildcard certificates. However, this process is only valid for Galaxy Admin Training courses, because Let's Encrypt wildcard certificates [can only be fetched using the DNS-01 challenge method][lets-encrypt-faq], which requires control of a [dynamic DNS][ddns] server (which we have preconfigured for use at training courses). Configuring your DNS service for dynamic updates is outside the scope of this tutorial, but it will show you how to request certificates using DNS-01, which can be adapted for your site.
>
> If you are using Let's Encrypt, [a list of available DNS plugins for Certbot][certbot-dns-plugins] can be found in the Certbot documentation. If you are not using Let's Encrypt, please consult your certificate vendor's documentation for information on how to obtain a wildcard certificate. You will need a certificate with (at least) the [subject alternative name][san]s `galaxy.example.org` and `*.ep.interactivetool.galaxy.example.org` (where `galaxy.example.org` is the hostname of your Galaxy server).
>
> You will also need a wildcard DNS `CNAME` record for `*.ep.interactivetool.galaxy.example.org`. You can verify that your Galaxy server has such a record using the `host` or `dig` command line tools like so:
>
>    ```console
>    $ host -t cname foo.ep.interactivetool.live.usegalaxy.eu
>    foo.ep.interactivetool.live.usegalaxy.eu is an alias for usegalaxy.eu.
>    $ host -t cname bar.ep.interactivetool.live.usegalaxy.eu
>    bar.ep.interactivetool.live.usegalaxy.eu is an alias for usegalaxy.eu.
>    ```
>
> Please consult your DNS server software or cloud provider's documentation for information on how to set up a wildcard record.
{: .warning}

[wildcard-cert]: https://en.wikipedia.org/wiki/Wildcard_certificate
[lets-encrypt]: https://letsencrypt.org/
[gat]: https://github.com/galaxyproject/dagobah-training
[lets-encrypt-faq]: https://letsencrypt.org/docs/faq/
[ddns]: https://en.wikipedia.org/wiki/Dynamic_DNS
[certbot-dns-plugins]: https://certbot.eff.org/docs/using.html#dns-plugins
[san]: https://en.wikipedia.org/wiki/Subject_Alternative_Name
[jupyter]: https://jupyter.org/
[gie-docs]: https://docs.galaxyproject.org/en/release_19.09/admin/special_topics/interactive_environments.html

> <warning-title>Uses Ansible!</warning-title>
> If the terms "Ansible," "role," and "playbook" mean nothing to you, please check out
> [the Ansible introduction slides]({% link topics/admin/tutorials/ansible/slides.html %}) and [the Ansible introduction tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %}).
{: .warning}

> <warning-title>Have you installed Galaxy?</warning-title>
> This tutorial builds upon the work in the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, please ensure that you have completed that tutorial first.
{: .warning}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Installing Docker

Galaxy Interactive Tools run in Docker containers. Thus, the first step is ensuring that the *nodes* that will run interactive tools have Docker installed. We will use the [Docker role][geerlingguy-docker] by the prolific Ansible Galaxy publisher, [Jeff Geerling (geerlingguy)][geerlingguy]. Have a look at the geerlingguy.docker [README][geerlingguy-docker-readme] and [defaults/main.yml][geerlingguy-docker-defaults] to get an understanding of what variables are used to control the role.

[geerlingguy-docker]: https://galaxy.ansible.com/geerlingguy/docker
[geerlingguy]: https://galaxy.ansible.com/geerlingguy
[geerlingguy-docker-readme]: https://github.com/geerlingguy/ansible-role-docker/blob/master/README.md
[geerlingguy-docker-defaults]: https://github.com/geerlingguy/ansible-role-docker/blob/master/defaults/main.yml

> <question-title></question-title>
>
> What variables might be relevant to using this role?
>
> > <solution-title></solution-title>
> >
> > The `docker_users` variable (a *list*) controls which users are able to interact with the Docker daemon, which our Galaxy user will need to do. Additionally, Docker Compose is configured by default, which we do not need, so it can be disabled with `docker_install_compose: false`.
> >
> {: .solution }
>
{: .question}

> <comment-title>Ansible Best Practices</comment-title>
> If you've set up your Galaxy server using the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, you will have created a `galaxyservers` group in your inventory file, `hosts`, and placed your variables in `group_vars/galaxyservers.yml`. Although for the purposes of this tutorial, the Galaxy server and cluster node are one and the same, in a real world deployment they are very likely to be different hosts. We will continue to use the `galaxyservers` group for simplicity, but in your own deployment you should consider creating an additional group for cluster nodes.
{: .comment}

> <hands-on-title>Installing Docker with Ansible</hands-on-title>
>
> 1. In your working directory, add the Docker role to your `requirements.yml`:
>
>    ```yaml
>    - src: geerlingguy.docker
>      version: 7.8.0
>    ```
>
> 2. Install the requirements with `ansible-galaxy`:
>
>    ```
>    ansible-galaxy role install -p roles -r requirements.yml
>    ```
>
> 3. Edit the group variables file, `group_vars/galaxyservers.yml`:
>
>    The relevant variables to set for this role are:
>
>    | Variable                 | Type            | Description                                     |
>    | ----------               | -------         | -------------                                   |
>    | `docker_users`           | list of strings | List of users to be added to the `docker` group |
>    | `docker_install_compose` | boolean         | Whether to install and configure Docker Compose |
>
>    Add the following lines to your `group_vars/galaxyservers.yml` file:
>
>    {% raw %}
>    ```yaml
>    # Interactive Tools
>    docker_install_compose: false
>    docker_users:
>      - "{{ galaxy_user.name }}"
>    ```
>    {% endraw %}
>
>    > <question-title></question-title>
>    >
>    > {% raw %}
>    > Why is `"{{ galaxy_user.name }}"` specified instead of just the user `galaxy`?
>    > {% endraw %}
>    >
>    > > <solution-title></solution-title>
>    > > Duplicating values is never a good idea. If we needed to change the Galaxy user down the line or wanted to reuse this playbook on another host where the Galaxy username was different, we would have to change the value in multiple locations.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
> 4. Add the new role to the list of roles under the `roles` key in your playbook, `galaxy.yml`:
>
>    ```yaml
>    ---
>    - hosts: galaxyservers
>      become: true
>      roles:
>        # ... existing roles ...
>        - geerlingguy.docker
>    ```
>
> 5. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
{: .hands_on}

Congratulations, you've set up Docker. Verify the installation using the `docker info` command (but keep in mind: what users did we authorize to interact with Docker?).

# Enabling the Interactive Tools Proxy

When an Interactive Tool's Docker container starts, it will be assigned a random port. In order to connect clients to the Interactive Tool, Galaxy needs to determine this port (and the node on which the tool is running) and configure a *proxy* from Galaxy to the GxIT's host and port. Consider the following example of running the Jupyter Notebook Interactive Tool, shown in Figure 1 below:

- nginx listens for requests from the client on **port 443** (https)
- Requests for Galaxy are delivered from nginx to Galaxy over a UNIX domain socket
- Requests for Interactive Tools are delivered from nginx to the Interactive Tools Proxy over (by default) **port 8000** (http)
  - GxIT http requests are forwarded by the proxy to Docker on the node on the container's (randomly assigned) **port 32768**
  - GxIT http requests are again forwarded by Docker to Jupyter on its in-container "published" **port 8888**

[//]: # The source for this figure can be found at: https://docs.google.com/presentation/d/1_4PtfM6A4mOxOlgGh6OGWvzFcxD1bdw4CydEWtm5n8k/

![Galaxy Interactive Tools Proxy Diagram](../../images/interactive-tools/gxit-proxy-diagram.png "Galaxy Interactive Tools Proxy Diagram")

As you can see, the client only ever speaks to nginx on the Galaxy server running on the standard https port (443), never directly to the interactive tool (which may be running on a node that does not even have a public IP address). By default, the mapping of GxIT invocation and its corresponding host/port is kept in a SQLite database known as the *Interactive Tools Session Map*, and the path to this database is important, since both Galaxy and the proxy need access to it.

The GxIT Proxy is written in [Node.js][nodejs]. A straightforward method to set it up is to use Galaxy's process manager, [Gravity](https://gravity.readthedocs.io/), to install and configure it.

[nodejs]: https://nodejs.org/

> <hands-on-title>Installing and configuring the Proxy using Gravity</hands-on-title>
>
> 1. Edit the group variables file, `group_vars/galaxyservers.yml`; the GxIT Proxy configuration lives under the `gravity['galaxy_config']['gx_it_proxy']` section. The available settings are:
>
>
>    | Setting         | Type    | Description                                                                                                                                                                                                                                         |
>    |-----------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
>    | `enable`        | bool    | Set to `true` to enable the GxIT Proxy. Defaults to `false`.                                                                                                                                                                                           |
>    | `version`       | str     | Version specifier for the GxIT Proxy Node.js package. Defaults to `>=0.0.6`.                                                                                                                                                                         |
>    | `ip`            | str     | Public-facing IP of the proxy. Defaults to `localhost`.                                                                                                                                                                                             |
>    | `port`          | int     | Public-facing port of the proxy. Defaults to `4002`.                                                                                                                                                                                                |
>    | `sessions`      | str     | Path to the Interactive Tools Session Map database. Defaults to `database/interactivetools_map.sqlite`.                                                                                                                                             |
>    | `verbose`       | bool    | Include verbose messages in the GxIT Proxy. Defaults to `true`.                                                                                                                                                                                      |
>    | `forward_ip`    | str     | Forward all requests to this IP. This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network.                                                                |
>    | `forward_port`  | int     | Forward all requests to this port. This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network.                                                              |
>    | `reverse_proxy` | str     | Rewrite location blocks with proxy port. This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network. Defaults to `false`.                                   |
>    | `umask`         | str     | Permissions mask under which service should be executed.                                                                                                                                                                                            |
>    | `start_timeout` | int     | Value of supervisor startsecs, systemd TimeoutStartSec. Defaults to `10`.                                                                                                                                                                           |
>    | `stop_timeout`  | int     | Value of supervisor stopwaitsecs, systemd TimeoutStopSec. Defaults to `10`.                                                                                                                                                                         |
>    | `memory_limit`  | float   | Memory limit (in GB). If the service exceeds the limit, it will be killed. Default is no limit or the value of the ``memory_limit`` setting at the top level of the Gravity configuration, if set. Ignored if ``process_manager`` is ``supervisor``. |
>    | `environment`   | mapping | Extra environment variables and their values to set when running the service. A dictionary where keys are the variable names.                                                                                                                       |
>
>    Configure the settings in the `group_vars/galaxyservers.yml` file:
>
>    {% raw %}
>    ```yaml
>    # ... other settings ... #
>    galaxy_config:
>      galaxy:
>        # ... other settings ... #
>        # Interactive Tools
>        interactivetools_enable: true
>        interactivetools_map: "{{ gxit_proxy_sessions_path }}"
>        # ... other settings ... #
>      gravity:
>        # ... other settings ... #
>        gx_it_proxy:
>          enable: true
>          port: 4002
>          sessions: "{{ gxit_proxy_sessions_path }}"
>        # ... other settings ... #
>      # ... other settings ... # 
>    # ... other settings ... #
>    gxit_proxy_sessions_path: "{{ galaxy_mutable_data_dir }}/interactivetools_map.sqlite"
>    ```
>    {% endraw %}
>
> 2. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
{: .hands_on}

[nodeenv]: https://github.com/ekalinin/nodeenv
[path-based_interactive_tools]: https://docs.galaxyproject.org/en/master/admin/special_topics/interactivetools.html#path-based-interactivetools

Because the proxy runs as a systemd service, you can inspect the log of the service using `journalctl`. The service name is `galaxy-gx-it-proxy`:

```console
$ sudo journalctl -eu galaxy-gx-it-proxy
Feb 14 17:38:49 gcc-4 systemd[1]: Started galaxy-gx-it-proxy.service - Galaxygx-it-proxy
...
Feb 14 17:38:49 gcc-4 node[3679]: Watching path /srv/galaxy/var/interactivetools_map.sqlite
Feb 14 17:38:49 gcc-4 node[3679]: Listening on localhost:4002
```

> <comment-title>Note</comment-title>
>
> You can ignore errors about failing to read the sessions map file for now - Galaxy will create it when it's needed.
>
{: .comment}

# Proxying the Proxy

As explained in the previous section, we will proxy the Interactive Tools Proxy with nginx so that it can serve requests on the standard HTTPS port, 443. Because we've configured nginx with Ansible, this is relatively simple.

> <hands-on-title>Configuring NGINX</hands-on-title>
>
> 1. Edit the group variables file, `group_vars/galaxyservers.yml` and add a new item to the **existing** `nginx_ssl_servers` so it matches:
>
>    {% raw %}
>    ```yaml
>    nginx_ssl_servers:
>      - galaxy
>      - galaxy-gxit-proxy
>    ```
>    {% endraw %}
>
>    The nginx configuration `galaxy-gxit-proxy` doesn't exist yet, but we'll create it in a moment.
>
> 2. Create `templates/nginx/galaxy-gxit-proxy.j2` with the following contents:
>
>    {% raw %}
>    ```nginx
>    server {
>        # Listen on port 443
>        listen       *:443 ssl;
>        # Match all requests for the interactive tools subdomain
>        server_name  *.interactivetool.{{ inventory_hostname }};
>
>        # Our log files will go here.
>        access_log  syslog:server=unix:/dev/log;
>        error_log   syslog:server=unix:/dev/log;
>
>        # Proxy all requests to the GxIT Proxy application
>        location / {
>            proxy_redirect off;
>            proxy_http_version 1.1;
>            proxy_set_header Host $host;
>            proxy_set_header X-Real-IP $remote_addr;
>            proxy_set_header Upgrade $http_upgrade;
>            proxy_set_header Connection "upgrade";
>            proxy_pass http://localhost:{{ galaxy_config.gravity.gx_it_proxy.port }};
>        }
>    }
>    ```
>    {% endraw %}
>
> 3. To enable [Path-based Interactive Tools][path-based_interactive_tools], open `templates/nginx/galaxy.j2` and add the following contents:
>
>    {% raw %}
>    ```nginx
>    server {
>        # ... existing settings ...
>
>        # Route all path-based interactive tool requests to the InteractiveTool proxy application
>        location ~* ^/(interactivetool/.+)$ {
>            proxy_redirect off;
>            proxy_http_version 1.1;
>            proxy_set_header Host $host;
>            proxy_set_header X-Real-IP $remote_addr;
>            proxy_set_header Upgrade $http_upgrade;
>            proxy_set_header Connection "upgrade";
>            proxy_pass http://localhost:{{ galaxy_config.gravity.gx_it_proxy.port }};
>        }
>
>        # ... other existing settings ...
>    }
>    ```
>    {% endraw %}
>
> 4. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
{: .hands_on}

# Getting a Wildcard SSL Certificate

During the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, we acquired an SSL certificate for our Galaxy server from [Let's Encrypt][lets-encrypt]. This certificate was issued for the hostname of your Galaxy server (e.g. `galaxy.example.org`). SSL certificates are valid *only for the name to which they were issued*. This presents a problem for us due to the way that Galaxy Interactive Tools work.

In order to ensure each Interactive Tool's cookies are unique, and to provide each tool with a unique entry point, they are served from a subdomain of your Galaxy server (e.g. `<unique-id>.ep.interactivetool.galaxy.example.org`). Your SSL cert is not valid for this subdomain. Further, in order to support the random `<unique-id>` in the hostname, we need a *wildcard certificate* for `*.ep.interactivetool.galaxy.example.org`.

This process is highly dependent on your site; specifically, your SSL certificate vendor, and your DNS server software or cloud provider.

Let's Encrypt, the SSL certificate vendor we use in our tutorials, [can only generate wildcard certificates using the DNS-01 challenge method][lets-encrypt-faq], which works by issuing a [dynamic DNS][ddns] update to set the requested domain's `TXT` record.

If you are completing this tutorial as part of a [Galaxy Admin Training][gat] course, we might have precreated a dynamic DNS server that you will use for this step. The *TSIG key* that allows you to perform dynamic DNS updates will be provided to you. Your instructor will also tell you which option to follow (1 or 2), depending on the DNS provider that was chosen for this course.

As we use Let's Encrypt in staging mode, the wildcard certificates generated with either option 1 or 2 will still be invalid, and you will still see a warning in your web browser when accessing an Interactive Tool. If this warning is not a problem for you, you can just skip this section of the tutorial, and move on to "Enabling Interactive Tools in Galaxy".

> <hands-on-title>Requesting a Wildcard Certificate with Certbot using Ansible - Option 1 (rfc2136)</hands-on-title>
>
> This method uses a DNS provider hosted by the Galaxy Project.
>
> 1. Edit the group variables file, `group_vars/galaxyservers.yml`:
>
>    The relevant variables to set for this role are:
>
>    | Variable                  | Type       | Description                                                                                                      |
>    | ----------                | -------    | -------------                                                                                                    |
>    | `certbot_domains`         | list       | List of domains to include as subject alternative names (the first will also be the certificate's *common name*) |
>    | `certbot_dns_provider`    | string     | Name of [Certbot DNS plugin][certbot-dns-plugins] to use                                                         |
>    | `certbot_dns_credentials` | dictionary | Plugin-specific credentials for performing dynamic DNS updates                                                   |
>    | `certbot_expand`          | boolean    | Whether to "expand" an existing certificate (add new domain names to it)                                         |
>
>    - Add a new item to the **existing** `certbot_domains` list so it matches:
>
>      {% raw %}
>      ```yaml
>      certbot_domains:
>        - "{{ inventory_hostname }}"
>        - "*.ep.interactivetool.{{ inventory_hostname }}"
>      ```
>      {% endraw %}
>
>    - Comment out the existing `certbot_auth_method` like so:
>
>      ```yaml
>      #certbot_auth_method: --webroot
>      ```
>
>        Although this is not explicitly required (setting `certbot_dns_provider` as we do overrides this setting), doing so is less confusing in the future, since it makes it clear that the "webroot" method for Let's Encrypt WEB-01 challenges is no longer in use for this server.
>
>    - Add the following lines to your `group_vars/galaxyservers.yml` file:
>
>      ```yaml
>      certbot_dns_provider: rfc2136
>      certbot_dns_credentials:
>        server: ns-training.galaxyproject.org
>        port: 53
>        name: certbot-training.
>        secret: <SECRET PROVIDED BY INSTRUCTOR>
>        algorithm: HMAC-SHA512
>      ```
>
> 2. Run the playbook **with `certbot_expand`**:
>
>    ```
>    ansible-playbook galaxy.yml -e certbot_expand=true
>    ```
>
>    > <question-title></question-title>
>    >
>    > What is the `-e` flag to `ansible-playbook` and why did we use it?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > As per `ansible-playbook --help`:
>    > >
>    > > ```
>    > >   -e EXTRA_VARS, --extra-vars EXTRA_VARS
>    > >                         set additional variables as key=value or YAML/JSON, if
>    > >                         filename prepend with @
>    > > ```
>    > >
>    > > We used this flag because `certbot_expand` only needs to be set *once*, when we are adding a new domain to the certificate. It should not be enabled on subsequent runs of the playbook, or else we would request a new certificate on each run! Thus, it does not make sense to add it to a vars file.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>    Be patient! The certificate request step can take time due to the time allowed for DNS propagation to occur.
>
{: .hands_on}


> <hands-on-title>Requesting a Wildcard Certificate with Certbot using Ansible - Option 2 (route53)</hands-on-title>
>
> This method uses route53, the Amazon Web Services DNS provider. To manage connection to AWS, we will first install a specific role.
>
> 1. In your working directory, add the aws_cli role to your `requirements.yml`:
>
>    ```yaml
>    - src: usegalaxy_eu.aws_cli
>      version: 0.0.1
>    ```
>
> 2. Install the requirements with `ansible-galaxy`:
>
>    ```
>    ansible-galaxy role install -p roles -r requirements.yml
>    ```
> 3. Open `galaxy.yml` with your text editor to add the role `usegalaxy_eu.aws_cli` just before the nginx role:
>
>    ```diff
>    diff --git a/galaxy.yml b/galaxy.yml
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -21,6 +21,7 @@
>           become: true
>           become_user: galaxy
>         - usegalaxy_eu.galaxy_systemd
>    +    - usegalaxy_eu.aws_cli
>         - galaxyproject.nginx
>         - geerlingguy.docker
>    ```
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 4. Edit the group variables file, `group_vars/galaxyservers.yml`:
>
>    The relevant variables to set for this role are:
>
>    | Variable                  | Type       | Description                                                                                                      |
>    | ----------                | -------    | -------------                                                                                                    |
>    | `certbot_domains`         | list       | List of domains to include as subject alternative names (the first will also be the certificate's *common name*) |
>    | `certbot_dns_provider`    | string     | Name of [Certbot DNS plugin][certbot-dns-plugins] to use                                                         |
>    | `certbot_dns_credentials` | dictionary | Plugin-specific credentials for performing dynamic DNS updates                                                   |
>    | `certbot_expand`          | boolean    | Whether to "expand" an existing certificate (add new domain names to it)                                         |
>
>    - Add a new item to the **existing** `certbot_domains` list so it matches:
>
>      {% raw %}
>      ```yaml
>      certbot_domains:
>        - "{{ inventory_hostname }}"
>        - "*.ep.interactivetool.{{ inventory_hostname }}"
>      ```
>      {% endraw %}
>
>    - Comment out the existing `certbot_auth_method` like so:
>
>      ```yaml
>      #certbot_auth_method: --webroot
>      ```
>
>        Although this is not explicitly required (setting `certbot_dns_provider` as we do overrides this setting), doing so is less confusing in the future, since it makes it clear that the "webroot" method for Let's Encrypt WEB-01 challenges is no longer in use for this server.
>
>    - Add the following lines to your `group_vars/galaxyservers.yml` file:
>
>      ```yaml
>      certbot_dns_provider: route53
>      aws_cli_credentials:
>        - access_key: "<SECRET PROVIDED BY INSTRUCTOR>"
>          secret_key: "<SECRET PROVIDED BY INSTRUCTOR>"
>          homedir: /root
>          owner: root
>          group: root
>      ```
>
> 5. Run the playbook **with `certbot_expand`**:
>
>    ```
>    ansible-playbook galaxy.yml -e certbot_expand=true
>    ```
>
>    > <question-title></question-title>
>    >
>    > What is the `-e` flag to `ansible-playbook` and why did we use it?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > As per `ansible-playbook --help`:
>    > >
>    > > ```
>    > >   -e EXTRA_VARS, --extra-vars EXTRA_VARS
>    > >                         set additional variables as key=value or YAML/JSON, if
>    > >                         filename prepend with @
>    > > ```
>    > >
>    > > We used this flag because `certbot_expand` only needs to be set *once*, when we are adding a new domain to the certificate. It should not be enabled on subsequent runs of the playbook, or else we would request a new certificate on each run! Thus, it does not make sense to add it to a vars file.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

You can verify that your certificate has been expanded using your browser's developer tools:

![Wildcard Certificate Dialog](../../images/interactive-tools/wildcard-cert.png "Wildcard Certificate Dialog")

# Enabling Interactive Tools in Galaxy

A few Interactive Tool wrappers are provided with Galaxy, but they are [commented out in Galaxy's default tool config file][tool-conf-gxits]. As a result, we need to instruct the [galaxyproject.galaxy role][galaxy-role] to install a tool panel configuration file containing at least one of these tools in order to try them out. For the purposes of this tutorial, a good choice is the [EtherCalc][ethercalc] GxIT, because it has a relatively small [Docker image][ethercalc-docker-image].

[tool-conf-gxits]: https://github.com/galaxyproject/galaxy/blob/9d788cb144a18570e7e5e948ac1f7bc04fec4e75/lib/galaxy/config/sample/tool_conf.xml.sample#L126
[galaxy-role]: https://github.com/galaxyproject/ansible-galaxy
[ethercalc]: https://ethercalc.net/
[ethercalc-docker-image]: https://hub.docker.com/r/shiltemann/ethercalc-galaxy-ie

> <hands-on-title>Enabling Interactive Tools in Galaxy</hands-on-title>
>
> 1. Rather than modifying the default tool configuration file, we'll add a new one that only references the Interactive Tools. This way, the default set of tools will still load without us having to incorporate the entire default tool config into our playbook.
>
>    If the folder does not exist, create `templates/galaxy/config` next to your `galaxy.yml` (`mkdir -p templates/galaxy/config/`)
>
>    Create `templates/galaxy/config/tool_conf_interactive.xml.j2` with the following contents:
>
>    ```xml
>    <toolbox monitor="true">
>        <section id="interactivetools" name="Interactive Tools">
>            <tool file="interactive/interactivetool_ethercalc.xml" />
>            <tool file="interactive/interactivetool_jupyter_notebook_1.0.1.xml" />
>        </section>
>    </toolbox>
>    ```
>
> 2. We need to instruct Galaxy on how to run Interactive Tools (and specifically, how to run them in Docker). Let's begin with a basic job configuration:
>
>    ```yaml
>    # ... other settings ... #
>    # Galaxy Job Configuration
>    galaxy_job_config:
>      runners:
>        local_runner:
>          load: galaxy.jobs.runners.local:LocalJobRunner
>          workers: 4
>      execution:
>        default: local_env
>        environments:
>          local_env:
>            runner: local_runner
>    # ... other settings ... #
>    ```
>
>    > <comment-title>Note</comment-title>
>    > Depending on the order in which you are completing this tutorial in relation to other tutorials, you may have already defined the `galaxy_job_config` mapping in your `group_vars/galaxyservers.yml` file, as well as `galaxy_config_templates`. If this is the case, be sure to **merge the changes in this section with your existing playbook**.
>    {: .comment}
>
> 3. Next, we need to configure the interactive tools destination. Add a destination for submitting jobs as Docker containers using the [advanced sample job configuration][job-conf-docker] as a guide. Finally, use the [EtherCalc GxIT's][ethercalc-tool-wrapper] tool ID to route executions of the EtherCalc GxIT to the newly created destination:
>
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>         execution:
>           default: local_env
>           environments:
>             local_env:
>               runner: local_runner
>    +        local_interactive_env:
>    +          runner: local_runner
>    +          params:
>    +            docker_enabled: true
>    +            docker_volumes: $defaults
>    +            docker_sudo: false
>    +            docker_net: bridge
>    +            docker_auto_rm: true
>    +            docker_set_user: ""
>    +            require_container: true
>    +    tools:
>    +      - id: interactive_tool_ethercalc
>    +        environment: local_interactive_env
>    +      - id: interactive_tool_jupyter_notebook
>    +        environment: local_interactive_env
>    ```
>
>    Of considerable note is the `docker_volumes` param: the variable expansions are explained in the [advanced sample job configuration][job-conf-docker].  We'll use this volume configuration for now but it has some considerable data security problems. We'll discuss a better solution at the end of this tutorial.
>
> 4. Inform `galaxyproject.galaxy` of what tool configuration files to load in your group variables (`group_vars/galaxyservers.yml`):
>
>    {% raw %}
>    ```yaml
>    galaxy_tool_config_files:
>      - "{{ galaxy_server_dir }}/config/tool_conf.xml.sample"
>      - "{{ galaxy_config_dir }}/tool_conf_interactive.xml"
>    ```
>    {% endraw %}
>
>    Next, deploy the new config template using the `galaxy_config_templates` var in your group vars:
>
>    {% raw %}
>    ```yaml
>    galaxy_config_templates:
>      # ... possible existing config file definitions
>      - src: templates/galaxy/config/tool_conf_interactive.xml.j2
>        dest: "{{ galaxy_config_dir }}/tool_conf_interactive.xml"
>    ```
>    {% endraw %}
>
> 5. If you've configured container resolvers, make sure the explit docker container resolver is enabled, e.g in templates/galaxy/config/container_resolvers_conf.yml.j2
>    {% raw %}
>    ```diff
>    --- a/templates/galaxy/config/container_resolvers_conf.yml.j2
>    +++ b/templates/galaxy/config/container_resolvers_conf.yml.j2
>    @@ -9,3 +9,4 @@
>     - type: build_mulled_singularity
>       auto_install: False
>       cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/built/"
>     +- type: explicit
>    ```
>    {% endraw %}
>
> 6. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
> 7. Follow the Galaxy logs with `galaxyctl follow`.
>
{: .hands_on}

[job-conf-docker]: https://github.com/galaxyproject/galaxy/blob/a2c847439528df0becf9e382c323f1dfe34d9633/lib/galaxy/config/sample/job_conf.xml.sample_advanced#L495
[ethercalc-tool-wrapper]: https://github.com/galaxyproject/galaxy/blob/6622ad1acb91866febb3d2f229de7cfb8af3a9f6/tools/interactive/interactivetool_ethercalc.xml

# Run an Interactive Tool

You should now be ready to run an Interactive Tool in Galaxy!

> <hands-on-title>Running an Interactive Tool</hands-on-title>
>
> 1. Ensure that you are logged in to your Galaxy server by checking the **User** menu in the masthead.
> 2. We'll need an input for our test GxIT (EtherCalc). Any tabular file can be used, such as Galaxy's [1.tabular][1-tabular] test data. Copy this file's URL:
>
>    ```
>    https://raw.githubusercontent.com/galaxyproject/galaxy/release_20.01/test-data/1.tabular
>    ```
>
> 3. Click {% icon galaxy-upload %} **Upload** at the top of the tool panel (on the left side of the Galaxy UI).
> 4. In the resulting modal dialog, click the **Paste/Fetch data** button.
> 5. Paste the URL in the **text field** that has just appeared.
> 6. Give the new dataset a name such as `tabular`, if you like.
> 7. Click **Start** and then **Close**.
> 8. From the tool menu, click the **Interactive Tools** section, then click **EtherCalc** {% icon tool %}.
> 9. Ensure that your newly uploaded tabular dataset is selected as the input **Some tabular dataset**, then click **Execute**.
> 10. Monitor the blue **info box** on the next page, which will inform you when the Interactive Tool is accessible and provide you with a link to access it.
>
>     <br/>
>
>     If you navigate away from this page, you can view your running Interactive Tools from the **Active InteractiveTools** menu item in the **User** menu.
> 11. Click the **click here to display** link.
>
{: .hands_on}

If everything has worked correctly, your browser will load EtherCalc with your tabular data preloaded. Once you're done working with the data, return to Galaxy and stop EtherCalc by deleting its output dataset from your history, or stopping it via the interface from the **Active InteractiveTools** menu item in the **User** menu.

[1-tabular]: https://raw.githubusercontent.com/galaxyproject/galaxy/release_20.01/test-data/1.tabular

# Securing Interactive Tools

Inspecting the Docker container of a running Interactive Tool shows the volume configuration expanded from `$galaxy_root` in the job destination's `docker_volumes` param:

```console
$ docker inspect $(docker ps -q) | jq '.[0].HostConfig.Binds'
[
  "/srv/galaxy/server:/srv/galaxy/server:ro",                                     # Galaxy server dir
  "/srv/galaxy/server/tools/interactive:/srv/galaxy/server/tools/interactive:ro", # EtherCalc tool wrapper parent dir
  "/srv/galaxy/jobs/000/1:/srv/galaxy/jobs/000/1:ro",                             # Per-job root dir
  "/srv/galaxy/jobs/000/1/outputs:/srv/galaxy/jobs/000/1/outputs:rw",             # Job outputs dir
  "/srv/galaxy/jobs/000/1/configs:/srv/galaxy/jobs/000/1/configs:rw",             # Job config files dir
  "/srv/galaxy/jobs/000/1/working:/srv/galaxy/jobs/000/1/working:rw",             # Job working (cwd) dir
  "/data:/data:rw",                                                               # GALAXY USER DATASETS DIR (RW!)
  "/srv/galaxy/server/tool-data:/srv/galaxy/server/tool-data:ro"                  # Galaxy reference data dir
]
```

As hinted earlier, there is a concerning state here: The directory containing **all** of the the user-generated data in Galaxy (not just the data for this job) has been mounted **read-write** in to the container. **This configuration grants users running interactive tools full access to all the data in Galaxy, which is a very bad idea.** Unlike standard Galaxy tools, where the tool's design prevents users from writing to arbitrary paths, Interactive Tools are fully user controllable. Although EtherCalc does not provide a mechanism for writing to this path, other Interactive Tools (such as Jupyter Notebook) do.

Two solutions are discussed in the [advanced sample job configuration][job-conf-docker]:

1. Use the `outputs_to_working_directory` job configuration option, which allows you to mount datasets read-only: this prevents manipulation, but still allows GxIT users to read any dataset in your Galaxy server.
2. Use [Pulsar][pulsar], Galaxy's remote job execution engine, to provide full job isolation: this avoids all access to Galaxy data, with the performance penalty of copying input dataset(s) to the job directory.

Because we want to maintain dataset privacy, Pulsar is the better choice here. And in fact, we don't even need to set up a Pulsar server: because we only need Pulsar's input staging and isolation features, we can use [Embedded Pulsar][job-conf-pulsar-embedded], which runs a Pulsar server within the Galaxy application to perform these tasks. Embedded Pulsar can even interface with your distributed resource manager (aka cluster scheduler) of choice, as long as your Galaxy server and cluster both have access to a common filesystem (otherwise, you will need to use Pulsar in standalone mode; see the [Running Jobs on Remote Resources with Pulsar]({% link topics/admin/tutorials/pulsar/tutorial.md %}) tutorial).

[pulsar]: https://github.com/galaxyproject/pulsar
[job-conf-pulsar-embedded]: https://github.com/galaxyproject/galaxy/blob/6622ad1acb91866febb3d2f229de7cfb8af3a9f6/lib/galaxy/config/sample/job_conf.xml.sample_advanced#L106

> <hands-on-title>Running Interactive Tools with Embedded Pulsar</hands-on-title>
>
> 1. Create a configuration file *template* for the Pulsar application at `templates/galaxy/config/pulsar_app.yml.j2`.
>
>    <br/>
>
>    If the folder does not exist, create `templates/galaxy/config` next to your `galaxy.yml` (`mkdir -p templates/galaxy/config/`).
>
>    <br/>
>
>    Add the following contents to the template:
>
>    {% raw %}
>    ```yaml
>    ---
>
>    # The path where per-job directories will be created
>    staging_directory: "{{ galaxy_job_working_directory }}/_interactive"
>
>    # Where Pulsar state information will be stored (e.g. currently active jobs)
>    persistence_directory: "{{ galaxy_mutable_data_dir }}/pulsar"
>
>    # Where to find Galaxy tool dependencies
>    tool_dependency_dir: "{{ galaxy_tool_dependency_dir }}"
>
>    # How to run jobs (see https://pulsar.readthedocs.io/en/latest/job_managers.html)
>    managers:
>      _default_:
>        type: queued_python
>        num_concurrent_jobs: 1
>    ```
>    {% endraw %}
>
> 2. Modify the job configuration in `group_vars/galaxyservers.yml` to configure Interactive Tools to use the embedded Pulsar runner.
>
>    <br/>
>
>    **Add** the embedded Pulsar runner to the `galaxy_job_config` section of the file:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>         runners:
>           local_runner:
>             load: galaxy.jobs.runners.local:LocalJobRunner
>             workers: 4
>    +      pulsar_embedded:
>    +        load: galaxy.jobs.runners.pulsar:PulsarEmbeddedJobRunner
>    +        pulsar_config: "{{ galaxy_config_dir }}/pulsar_app.yml"
>         execution:
>    ```
>    {% endraw %}
>
>    Next, **modify** the `interactive_local` destination to use the new runner and set the new parameter `container_monitor_result` to `callback` (explained in more detail in the next step):
>
>    > <warning-title>Untrusted SSL Certificates</warning-title>
>    > If you are completing this tutorial as part of a [Galaxy Admin Training][gat] course, you will also need the `env` setting shown below to prevent problems with the untrusted SSL certificates in use during the course. Galaxy servers with valid SSL certificates *do not need this option*.
>    {: .warning}
>
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>         execution:
>           default: local_env
>           environments:
>             local_env:
>               runner: local_runner
>             local_interactive_env:
>    -          runner: local_runner
>    +          runner: pulsar_embedded
>               params:
>                 docker_enabled: true
>                 docker_volumes: $defaults
>                 docker_sudo: false
>                 docker_net: bridge
>                 docker_auto_rm: true
>                 docker_set_user: ""
>                 require_container: true
>    +            container_monitor_result: callback
>    +          env:
>    +            - name: REQUESTS_CA_BUNDLE
>    +              value: /etc/ssl/certs/ca-certificates.crt
>         tools:
>    ```
>
> 3. Open your `galaxyservers` group variables file and instruct `galaxyproject.galaxy` to install the Pulsar configuration file:
>
>    > <comment-title>Note</comment-title>
>    > Depending on the order in which you are completing this tutorial in relation to other tutorials, you may have already defined `galaxy_config_templates`. If this is the case, be sure to **merge the changes in this step with your existing playbook**.
>    {: .comment}
>
>    {% raw %}
>    ```yaml
>    galaxy_config_templates:
>      - src: templates/galaxy/config/pulsar_app.yml.j2
>        dest: "{{ galaxy_config_dir }}/pulsar_app.yml"
>    ```
>    {% endraw %}
>
>    Additionally, you will need to set the `galaxy_infrastructure_url` config option:
>
>    {% raw %}
>    ```yaml
>    galaxy_config:
>      galaxy:
>        # ... existing configuration options in the `galaxy` section ...
>        galaxy_infrastructure_url: "https://{{ inventory_hostname }}/"
>    ```
>    {% endraw %}
>
>    > <details-title>Detail: Infrastructure URL/Callback</details-title>
>    > Galaxy must be made aware of the randomly selected port Docker has assigned after the GxIT begins operating, in order to update the proxy map. By default, this is done by writing a JSON file in the job directory. This method does not work with Pulsar since Pulsar uses a different job directory from the Galaxy job directory. As a result, Pulsar jobs use the `callback` method configured in the previous step to make a request to Galaxy's API, the URL for which is set in `galaxy_infrastructure_url`.
>    {: .details}
>
> 4. Run the playbook:
>
>    ```
>    ansible-playbook galaxy.yml
>    ```
>
{: .hands_on}

Once the playbook run is complete and your Galaxy server has restarted, run the EtherCalc Interactive Tool again.

> <question-title></question-title>
>
> Once EtherCalc is running, check the mounts of its container. What do you observe?
>
> > <solution-title></solution-title>
> >
> > ```console
> > $ docker inspect $(docker ps -q) | jq '.[0].HostConfig.Binds'
> > [
> >   "/srv/galaxy/jobs/_interactive/2:/srv/galaxy/jobs/_interactive/2:ro",                       # Per-job root dir
> >   "/srv/galaxy/jobs/_interactive/2/tool_files:/srv/galaxy/jobs/_interactive/2/tool_files:ro", # EtherCalc tool wrapper parent dir
> >   "/srv/galaxy/jobs/_interactive/2/outputs:/srv/galaxy/jobs/_interactive/2/outputs:rw",       # Job outputs dir
> >   "/srv/galaxy/jobs/_interactive/2/working:/srv/galaxy/jobs/_interactive/2/working:rw",       # Job working (cwd) dir
> >   "/srv/galaxy/server/tool-data:/srv/galaxy/server/tool-data:ro"                              # Galaxy reference data dir
> > ]
> > ```
> >
> > Of note, the user data directory, `/data`, is no longer mounted in the container!
> >
> {: .solution }
>
{: .question }

# High availability setup with PostgresSQL (Optional)

> <comment-title></comment-title>
> This section is **only relevant if you are running a high-availability** setup, meaning that you have multiple copies of Galaxy running behind a load balancer.
>
> If you have installed Galaxy following the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial, or are completing this tutorial as part of a [Galaxy Admin Training][gat] course, please skip this section, as you are then _not_ running a high-availability setup.
{: .comment}

In a _high availability_ setup, multiple redundant copies of Galaxy run simultaneously behind a load balancer to minimize downtime and service interruptions.

As explained in [one of the previous sections](#enabling-the-interactive-tools-proxy), the Galaxy Interactive Tools Proxy redirects requests to each Interactive Tool's host and port. By default, the mapping of GxIT invocations to their corresponding host/port is kept in a SQLite database known as the _Interactive Tools Session Map_.

By design, [SQLite is the wrong choice for high availability setups][sqlite_situations_where_a_client_server_rdbms_may_work_better], the showstopper being that the SQLite database file would have to be shared over a network filesystem, which are usually associated with too high latencies for RDBMS use. For this reason, Galaxy and the Interactive Tools Proxy can also store the **Session Map in a PostgreSQL database**. 

[sqlite_situations_where_a_client_server_rdbms_may_work_better]: https://www.sqlite.org/whentouse.html#situations_where_a_client_server_rdbms_may_work_better

> <hands-on-title>Preparing the database</hands-on-title>
> 
> First, you need to create a database for the Interactive Tools Proxy. 
>
> > <warning-title></warning-title>
> > Do **not** use the Galaxy database for this purpose. The main Galaxy database is reserved for Galaxy's core functionality, and Interactive Tools have not yet reached this stage. Since Galaxy does not expect to find the Interactive Tools Session Map in this database, storing it there can lead to errors.
> {: .warning }
>
> <br>
>
> 1. **On your database server**, access PostgresSQL and create a `gxitproxy` database to store the Interactive Tools Session Map. For simplicity, the same user that operates on the Galaxy main database, typically named `galaxy`, is also going to operate on this one and will own the new database.
> 
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > # one-liner that connects to Postgres, creates the database and assigns ownership
>    > sudo -u postgres createdb -O galaxy gxitproxy
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Connect to the `gxitproxy` database as `galaxy`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sudo -iu galaxy psql -d gxitproxy
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>SQL</code-out-title>
>    > ```
>    > psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
>    > Type "help" for help.
>    >
>    > gxitproxy=#
>    > ```
>    {: .code-out}
>
> 3. Create a `gxitproxy` table in the new database.
>
>    > <code-in-title>SQL</code-in-title>
>


topic: admin
tutorial: pulsar
title: Running Jobs on Remote Resources with Pulsar
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/pulsar/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["type: \"internal\"", "title: \"A server/VM on which to deploy Pulsar\""]
tags_json: ["ansible", "jobs", "git-gat"]
content_hash: 987c059cfcc6f80e947ac66db458671f
last_modified: 2026-02-26T21:12:14.151415
workflows_json: []
content:
Pulsar is the Galaxy Project's remote job running system. It was written by John Chilton ([@jmchilton](https://github.com/jmchilton)) of the Galaxy Project. It is a python server application that can accept jobs from a Galaxy server, submit them to a local resource and then send the results back to the originating Galaxy server.

More details on Pulsar can be found at:

- [Pulsar's Documentation](https://pulsar.readthedocs.io/en/latest/index.html)
- [Pulsar's Github Repository](https://github.com/galaxyproject/pulsar)
- [Pulsar Ansible Role](https://github.com/galaxyproject/ansible-pulsar)

Transport of data, tool information and other metadata can be configured as a web application via a RESTful interface or using a message passing system such as RabbitMQ.

At the Galaxy end, it is configured within the job configuraiton, and uses one of two special Galaxy job runners.
* `galaxy.jobs.runners.pulsar:PulsarRESTJobRunner` for the RESTful interface
* `galaxy.jobs.runners.pulsar:PulsarMQJobRunner` for the message passing interface.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="pulsar" %}

## This tutorial assumes that

- You have a VM or machine where you will install Pulsar, and a directory in which the installation will be done. This tutorial assumes it is `/mnt`
- You have completed the "Galaxy Installation with Ansible", "Connecting Galaxy to a Compute Cluster", and the "CVMFS" tutorials
- You have access to the VM/computer where it is installed.

> <tip-title>This is NOT intended as a standalone Pulsar guide</tip-title>
> This tutorial is not intended to be a standalone Pulsar setup guide. If you read carefully and understand Ansible, it is likely you can figure out which portions are required to just setup Pulsar.
{: .tip}

## Overview

We will be installing the RabbitMQ server daemon onto the Galaxy server to act as an intermediary message passing system between Galaxy and the remote Pulsar. The figure below shows a schematic representation of the system.

![Schematic diagram of Galaxy communicating with a Pulsar server via the RabbitMQ server](../../images/pulsar_amqp_schema.png "Schematic diagram of Galaxy communicating with a Pulsar server via the RabbitMQ server. Red arrows represent AMQP communications and blue represent file transfers (initiated by the Pulsar server.)")


### How it will work

1. Galaxy will send a message to the RabbitMQ server on the Pulsar server's particular queue saying that there is a job to be run and then will monitor the queue for job status updates.
2. The Pulsar server monitors this queue and when the job appears it will take control of it.
3. The Pulsar server will then download the required data etc. from the Galaxy server using `curl`.
4. The Pulsar server will install any required tools/tool dependencies using Conda.
5. The Pulsar server will start running the job using it's local mechanism and will send a message to the "queue" stating that the job has started.
6. Once the job has finished running, the Pulsar server will send a message to the queue stating that the job has finished.
7. Pulsar then sends the output data etc. back to the Galaxy server by `curl` again.
8. The Galaxy server acknowledges the job status and closes the job.

### Some notes

* RabbitMQ uses the Advanced Message Queueing Protocol (AMQP) to communicate with both the Galaxy server and the remote Pulsar VM.
* Transport of files, meta-data etc. occur via `curl` from the Pulsar end.
* RabbitMQ is written in erlang and does not add much overhead to the Galaxy VM, although in larger installations, RabbitMQ is commonly installed on a separate VM to Galaxy. e.g. Galaxy Europe, Galaxy Main and Galaxy Australia.

> <tip-title>Other file transport methods for Pulsar</tip-title>
>
>  Pulsar can use a variety of file transport methods including:
>  * Default: Galaxy initiates file transfer and stages files to Pulsar via http transfer.
>      * This requires that a http transfer port be open on the remote Pulsar.
>  * Remote transfer: Pulsar initiates file transfer. This can use a variety of lso available and can use a variety of methods:
>      * Curl
>      * Rsync
>      * Http
>
>  We use remote transfer using **Curl** here so we don't need an open port on the Pulsar server and tranfer robustness respectively.
>
{: .tip}


> <details-title>Why are we using Pulsar in MQ mode here and not the RESTful interface?</details-title>
> We are teaching you to install Pulsar and configure it in MQ mode in this tutorial. Configuring Pulsar in RESTful mode is also possible and is quite useful in certain situations. However, in the most common situation MQ mode is preferable for a number of reasons:
> * When running Pulsar in RESTful mode, all of the job control and data transfer is controlled by the Galaxy server usually using http transfers. This can place a limit on the size of files that can be transferred without constant configuring of the webserver.
> * When running in RESTful mode, Pulsar also needs to have an https server such as nginx, including securing it, configuring it, getting certificates and opening ports. This can be very difficult to do if you are attempting to submit jobs to an institutional HPC where the admins probably won't let you do any of these things.
> * In MQ mode, you only need to open a port for the RabbitMQ server on a machine you are more likely to control. The HPC side running Pulsar can just connect back to you.
>
> See the [Pulsar documentation](https://pulsar.readthedocs.io/en/latest/) for details.
{: .details}

## Install and configure a message queueing system

In this section we will install the RabbitMQ server on your Galaxy server VM.

RabbitMQ is an AMQP server that can queue messages between systems for all sorts of reasons. Here, we will be using the queue so that Galaxy and Pulsar can communicate jobs, job status and job metadata between them easily and robustly. More information on RabbitMQ can be found [on their website](https://www.rabbitmq.com/).

### Installing the roles

Firstly we will add and configure another *role* to our Galaxy playbook - a community role which runs RabbitMQ in docker and configures it. In order to have docker installed automatically, we use geelingguy's docker role, which luckly needs no configuration changes. Additionally we will use the Galaxy community role for deploying Pulsar

> <hands-on-title>Install the Ansible roles</hands-on-title>
>
> 1. From your ansible working directory, edit the `requirements.yml` file and add the following lines:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -30,3 +30,10 @@
>     # TPV Linting
>     - name: usegalaxy_eu.tpv_auto_lint
>       version: 0.4.3
>    +# RabbitMQ for Pulsar
>    +- src: geerlingguy.docker
>    +  version: 6.1.0
>    +- src: usegalaxy_eu.rabbitmqserver
>    +  version: 1.4.4
>    +- src: galaxyproject.pulsar
>    +  version: 1.0.10
>    {% endraw %}
>    ```
>    {: data-commit="Add requirements"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Now install it with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

### Configuring RabbitMQ

We need to configure RabbitMQ to be able to handle Pulsar messages. To do this we will need to create some queues, Rabbit users, some queue vhosts and set some passwords. We also need to configure rabbit to listen on various interfaces and ports.

#### Defining Virtual Hosts

Each set of queues in RabbitMQ are grouped and accessed via virtual hosts. We need to create one of these for the transactions between the Galaxy server and Pulsar server. They are set as an array under the `rabbitmq_vhosts` variable.

#### Defining users

Users need to be defined, given passwords and access to the various queues. We will need to create a user that can access this vhost. We will also create an admin user. The queue will need access to the Pulsar queue vhost. They are set as an array under the `rabbitmq_users` variable with the following structure:


```yaml
{% raw %}
rabbitmq_users:
  - user: username
    password: "{{ rabbitmq_password_username }}"
    vhost: /vhostname
{% endraw %}
```

Notice the variable we used instead of directly placing the password there. It will be read from vault instead.

> <tip-title>Tagging users</tip-title>
> Optional: You can add tags to each user if required. e.g. For an admin user it could be useful to add in a *administrator* tag. These tags allow you to grant permissions to every user with a specific tag.
{: .tip}

#### RabbitMQ server config

We also need to set some RabbitMQ server configuration variables. Such as where its security certificates are and which ports to listen on (both via localhost and network).

> <tip-title>Port accessibility is important!</tip-title>
> We will need to make sure that the RabbitMQ default port is open and accessible on the server we are installing RabbitMQ onto. (In our case this is the Galaxy server). Additionally, RabbitMQ provides a management interface for us, where we can see e.g. queues, connections and statistics.
>
> Port | Usage | Suggested Restrictions
> `5671` | SSL Listener | None
> `5672` | Plain text listener | Limit to just `172.0.0.0/8` to ensure only the host machine and other containers can access it, and we aren't sending plain text credentials over the internet.
> `15672` | Management interface | Limit to `172.0.0.0/8` to only permit accessing management interface locally.
{: .tip}

More information about the rabbitmq ansible role can be found [in the repository](https://github.com/usegalaxy-eu/ansible-rabbitmq).

### Add RabbitMQ configuration to Galaxy VM.

> <hands-on-title>Add RabbitMQ settings to Galaxy VM groupvars file.</hands-on-title>
>
> 1. Edit your `group_vars/secret.yml` and define some random passwords:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > ansible-vault edit group_vars/secret.yml
>    > ```
>    {: .code-in}
>
>    ```yaml
>    vault_rabbitmq_password_vhost: "a-really-long-password-here"
>    vault_rabbitmq_admin_password: "a-different-really-long-password"
>    ```
>
>    <!-- Ignore this, just for the gat-automation. Vaults are ugly to work with :(
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/secret.yml
>    +++ b/group_vars/secret.yml
>    @@ -1,7 +1,13 @@
>     $ANSIBLE_VAULT;1.1;AES256
>    -32653961383866636531396135663630386630346237333333653633313436663439643535323964
>    -6363626330336430363332643638646262316338313937320a666566306539373462386266383166
>    -30326165393863633463353234613561393939326164376432633732316264636464313061383161
>    -3532373937656138320a616361343664353264613332616236623231326137316635323465623562
>    -66656539346130353639623736633034653932373438663330646436656336666637313933666264
>    -3636313438626533633831323239373461373538646635613637
>    +62346261323266656232393034396134316636376533376139666437363535393562663838613938
>    +6336666266633563346337623265353935646361326337610a393834333233313461346439376438
>    +63383338346530656561636631666134373238366364363164313166346461383736613162653237
>    +3461363334323431370a656132303965653262386130353332623937376261396530393761353834
>    +38336565666437666436643163363831633331333766653266356163613138393734656465323634
>    +39366362383433366437353534663134313330316337393335383962613961386665633261616237
>    +35366635373063313631323939396164336330356361393464326636353037336461323531336434
>    +35613933303333623031353936393265636130363335376533393335663266313863376135383338
>    +36613464373231623938373434306266373234633036343636633963353361356631363533353066
>    +39323064336237646432323530313065303331326636353334343862373330313133326363363063
>    +38383564636161396435666164643334656435393533643163393434623434656238633631633939
>    +33353232666432376661
>    {% endraw %}
>    ```
>    {: data-commit="Add rabbitmq passwords to the vault"}
>
>    -->
>
>    This is going in the vault as they are secrets we need to set. Both of our services, Galaxy and Pulsar, need these variables, so we'll need to make sure they're in both playbooks. Both Galaxy in the job configuration, and Pulsar in its configuration.
>
>    Replace both with long random (or not) string.
>
> 2. From your ansible working directory, edit the `group_vars/galaxyservers.yml` file and add make the following changes.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -174,8 +174,11 @@ certbot_environment: staging
>     certbot_well_known_root: /srv/nginx/_well-known_root
>     certbot_share_key_users:
>       - www-data
>    +certbot_share_key_ids:
>    +  - "999:999"
>     certbot_post_renewal: |
>         systemctl restart nginx || true
>    +    docker restart rabbit_hole || true
>     certbot_domains:
>      - "{{ inventory_hostname }}"
>     certbot_agree_tos: --agree-tos
>    @@ -226,6 +229,47 @@ slurm_config:
>       SelectType: select/cons_tres
>       SelectTypeParameters: CR_CPU_Memory  # Allocate individual cores/memory instead of entire node
>
>    +#Install pip docker package for ansible
>    +pip_install_packages:
>    +  - name: docker
>    +# RabbitMQ
>    +rabbitmq_container:
>    +  name: rabbit_hole
>    +  image: rabbitmq:3.11-management
>    +  hostname: "{{ inventory_hostname }}"
>    +
>    +rabbitmq_plugins:
>    +  - rabbitmq_management
>    +
>    +rabbitmq_config:
>    +  listeners:
>    +    tcp: none
>    +  ssl_listeners:
>    +    default: 5671
>    +  ssl_options:
>    +    verify: verify_peer
>    +    cacertfile: /etc/ssl/certs/fullchain.pem
>    +    certfile: /etc/ssl/certs/cert.pem
>    +    keyfile: /etc/ssl/user/privkey-999:999.pem
>    +    fail_if_no_peer_cert: 'false'
>    +  management_agent:
>    +    disable_metrics_collector: "false"
>    +  management:
>    +    disable_stats: 'false'
>    +  consumer_timeout: 21600000 # 6 hours in milliseconds
>    +
>    +rabbitmq_vhosts:
>    +  - /pulsar/pulsar_au
>    +
>    +rabbitmq_users:
>    +  - user: admin
>    +    password: "{{ vault_rabbitmq_admin_password }}"
>    +    tags: administrator
>    +    vhost: /
>    +  - user: pulsar_au
>    +    password: "{{ vault_rabbitmq_password_vhost }}"
>    +    vhost: /pulsar/pulsar_au
>    +
>     # TUS
>     galaxy_tusd_port: 1080
>     galaxy_tus_upload_store: /data/tus
>    {% endraw %}
>    ```
>    {: data-commit="Configure RabbitMQ"}
>
>    > <tip-title>RabbitMQ installation errors?</tip-title>
>    > RabbitMQ depends on specific Erlang versions. If the Erlang version has been updated, you may need to change the value of `rabbitmq_version:` in the configuration above. [Information on the RabbitMQ Erlag version requirements.](https://www.rabbitmq.com/which-erlang.html)
>    {: .tip}
>
> 4. Update the Galaxy playbook to include the *usegalaxy_eu.rabbitmq* role.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -43,6 +43,8 @@
>           become: true
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>    +    - geerlingguy.docker
>    +    - usegalaxy_eu.rabbitmqserver
>         - galaxyproject.gxadmin
>         - galaxyproject.cvmfs
>       post_tasks:
>    {% endraw %}
>    ```
>    {: data-commit="Add role"}
>
>    > <tip-title>Why is this at the end?</tip-title>
>    > This is one of the constant problems with Ansible, how do you order everything correctly? Does an ordering exist such that a single run of the playbook will have everything up and working? We encounter one such instance of this problem now.
>    >
>    > Here are the dependencies between the roles:
>    >
>    > From             | To               | Purpose
>    > ---------------- | --               | -------
>    > nginx            | galaxy           | The nginx templates depend on variables only available after the Galaxy role is run
>    > SSL certificates | nginx            | A running nginx is required
>    > RabbitMQ         | SSL certificates | RabbitMQ will silently start with incorrect configuration if SSL certificates are not present at boot time.
>    > Galaxy           | RabbitMQ         | Galaxy needs the RabbitMQ available to submit jobs.
>    >
>    > And as you can see there is a circular dependency. Galaxy requires RabbitMQ, but RabbitMQ depends on a long chain of things that depends finally on Galaxy.
>    >
>    > There are some mitigating factors, some software will start with incomplete configuration. We can rely on Galaxy retrying access to RabbitMQ if it isn't already present. Additionally on first run, Galaxy is restarted by a handler which runs at the end. (Except that the nginx role triggers all pending handlers as part of the SSL certificate deployment.)
>    >
>    > We try to present the optimal version here but due to these interdependencies and Ansible specifics, sometimes it is not possible to determine a good ordering of roles, and multiple runs might be required.
>    >
>    {: .tip}
>
> 5. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 6. The rabbitmq server daemon will have been installed on your Galaxy VM. Check that it's running now:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > docker ps
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > ```ini
>    >CONTAINER ID   IMAGE             COMMAND                  CREATED        STATUS       PORTS                                                                                              NAMES
>    >d0d69d035768   rabbitmq:3.9.11   "docker-entrypoint.s…"   6 months ago   Up 5 hours   4369/tcp, 0.0.0.0:5671->5671/tcp, 5672/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp   rabbit_hole
>    > ```
>    {: .code-out.code-max-300}
>
> 7. But this doesn't tell the whole story, so run the diagnostics command to
>    check that the interfaces are setup and listening. RabbitMQ has a bad
>    habit of silently failing when processing the configuration, without any
>    logging information If RabbitMQ has any problem reading the configuration
>    file, it falls back to the default configuration (listens *without* ssl on
>    `tcp/5672`) so be sure to check that everything is OK before continuing.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > docker exec rabbit_hole rabbitmq-diagnostics status
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > ```ini
>    > ...
>    >
>    > Listeners
>    >
>    > Interface: [::], port: 15672, protocol: http, purpose: HTTP API
>    > Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
>    > Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
>    > Interface: 0.0.0.0, port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
>    > ```
>    >
>    {: .code-out.code-max-300}
>
> 8. Since we enabled metrics, let's check if the api works:
>
>    > <code-in-title>Bash</code-in-title>
>    > Make sure to replace \<password\> with the one from your vault.
>    > If you don't have jq installed, just leave that part with the pipe out, it just makes it prettier.
>    > ```bash
>    > curl --silent -s -u admin:<password> http://localhost:15672/api/whoami | jq
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > This should report the following response:
>    >
>    > ```console
>    > {
>    >   "name": "admin",
>    >   "tags": [
>    >     "administrator"
>    >   ]
>    > }
>    > ```
>    {: .code-out.code-max-300}
>
>    But wait! There are more ways it can go wrong. To be extra sure, run a quick `curl` command.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > curl http://localhost:5672
>    > ```
>    > (the non-encrypted port)
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > This should report the following response:
>    >
>    > ```console
>    > curl: (7) Failed to connect to localhost port 5672: Connection refused
>    > ```
>    >
>    > if it doesn't, consider the following debugging steps:
>    >
>    > 1. Restarting RabbitMQ
>    > 2. Check that the configuration looks correct (ssl private key path looks valid)
>    > 3. Check that the private key is shared correctly with the rabbitmq user
>    {: .code-out.code-max-300}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > curl -k --http0.9 https://localhost:5671 --output - && printf "\n"
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > This should report the following response:
>    >
>    > ```console
>    > AMQP
>    > ```
>    >
>    > if it doesn't, consider the following debugging steps:
>    >
>    > 1. Restarting RabbitMQ
>    > 2. Check that the configuration looks correct (ssl private key path looks valid)
>    > 3. Check that the private key is shared correctly with the rabbitmq user
>    {: .code-out.code-max-300}
>
{: .hands_on}

By this point you should have a functional RabbitMQ! Let's check out the dashboard:

> <hands-on-title>Accessing the RabbitMQ Dashboard</hands-on-title>
>
> 1. RabbitMQ has a fancy dashboard, so we should make that accessible with our NGINX:
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -84,4 +84,14 @@ server {
>     	location /training-material/ {
>     		proxy_pass https://training.galaxyproject.org/training-material/;
>     	}
>    +
>    +	location ~* /rabbitmq/(.*) {
>    +		rewrite ^/rabbitmq/(.*)$ /$1 break;
>    +		proxy_pass http://127.0.0.1:15672;
>    +		proxy_buffering                    off;
>    +		proxy_set_header Host              $http_host;
>    +		proxy_set_header X-Real-IP         $remote_addr;
>    +		proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
>    +		proxy_set_header X-Forwarded-Proto $scheme;
>    +	}
>     }
>    {% endraw %}
>    ```
>    {: data-commit="Add proxy"}
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 1. Now we can take a look at the RabbitMQ dashboard.
>
>    1. Open a new tab in your browser and enter your server's hostname followed by `/rabbitmq/`
>    1. A login window should appear. Use the admin credentials you defined earlier in this tutorial.
>    1. Click on connections and you can see how connections will be established and appear there during the next parts of the tutorial.
>
{: .hands_on}

## Installing and configuring Pulsar on a remote machine

Now that we have a message queueing system running on our Galaxy VM, we need to install and configure Pulsar on our remote compute VM. To do this we need to create a new ansible playbook to install Pulsar.

### Configuring Pulsar

From the [`galaxyproject.pulsar` ansible role documentation](https://github.com/galaxyproject/ansible-pulsar#role-variables), we need to specify some variables.

There is one required variable:

`pulsar_server_dir` - The location in which to install pulsar

Then there are a lot of optional variables. They are listed here for information. We will set some for this tutorial but not all.

 Variable Name                 | Description                                                                                        | Default
---------------                | -------------                                                                                      | ---
`pulsar_yaml_config`           | a YAML dictionary whose contents will be used to create Pulsar's `app.yml`                         |
`pulsar_venv_dir`              | The role will create a virtualenv from which Pulsar will run                                       | `<pulsar_server_dir>/venv` if installing via pip, `<pulsar_server_dir>/.venv` if not.
`pulsar_config_dir`            | Directory that will be used for Pulsar configuration files.                                        | `<pulsar_server_dir>/config` if installing via pip, `<pulsar_server_dir>` if not
`pulsar_optional_dependencies` | List of optional dependency modules to install, depending on which features you are enabling.      | `None`
`pulsar_install_environments`  | Installing dependencies may require setting certain environment variables to compile successfully. |
`pulsar_create_user`           | Should a user be created for running pulsar?                                                       |
`pulsar_user`                  | Define the user details                                                                            |

Some of the other options we will be using are:

* We will set the tool dependencies to rely on **conda** for tool installs.

* You will need to know the FQDN or IP address of the Galaxy server VM that you installed RabbitMQ on.

> <hands-on-title>Configure pulsar group variables</hands-on-title>
>
>
> 2. Create a new file in `group_vars` called `pulsarservers.yml` and set some of the above variables as well as some others.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/pulsarservers.yml
>    @@ -0,0 +1,50 @@
>    +galaxy_server_hostname: "{{ groups['galaxyservers'][0] }}" # Important!!!
>    +# Put your Galaxy server's fully qualified domain name (FQDN) (or the FQDN of the RabbitMQ server) above.
>    +
>    +pulsar_root: /mnt/pulsar
>    +
>    +pulsar_pip_install: true
>    +pulsar_pycurl_ssl_library: openssl
>    +pulsar_systemd: true
>    +pulsar_systemd_runner: webless
>    +
>    +pulsar_create_user: true
>    +pulsar_user: {name: pulsar, shell: /bin/bash}
>    +
>    +pulsar_optional_dependencies:
>    +  - pyOpenSSL
>    +  # For remote transfers initiated on the Pulsar end rather than the Galaxy end
>    +  - pycurl
>    +  # drmaa required if connecting to an external DRM using it.
>    +  - drmaa
>    +  # kombu needed if using a message queue
>    +  - kombu
>    +  # psutil allows pulsar's local job runner to kill processes effictively.
>    +  - psutil
>    +  # temporary patch until setuptools is eliminated
>    +  - setuptools
>    +
>    +pulsar_yaml_config:
>    +  staging_directory: "{{ pulsar_staging_dir }}"
>    +  persistence_directory: "{{ pulsar_persistence_dir }}"
>    +  tool_dependency_dir: "{{ pulsar_dependencies_dir }}"
>    +  # The following are the settings for the pulsar server to contact the message queue with related timeouts etc.
>    +  message_queue_url: "pyamqp://pulsar_au:{{ vault_rabbitmq_password_vhost }}@{{ galaxy_server_hostname }}:5671//pulsar/pulsar_au?ssl=1"
>    +  min_polling_interval: 0.5
>    +  amqp_publish_retry: True
>    +  amqp_publish_retry_max_retries: 5
>    +  amqp_publish_retry_interval_start: 10
>    +  amqp_publish_retry_interval_step: 10
>    +  amqp_publish_retry_interval_max: 60
>    +  # We also need to create the dependency resolvers configuration so pulsar knows how to find and install dependencies
>    +  # for the tools we ask it to run. The simplest method which covers 99% of the use cases is to use conda auto installs
>    +  # similar to how Galaxy works.
>    +  dependency_resolution:
>    +    resolvers:
>    +      - type: conda
>    +        auto_init: true
>    +        auto_install: true
>    +
>    +# Pulsar should use the same job metrics plugins as Galaxy. This will automatically set `job_metrics_config_file` in
>    +# `pulsar_yaml_config` and create `{{ pulsar_config_dir }}/job_metrics_conf.yml`.
>    +pulsar_job_metrics_plugins: "{{ galaxy_job_metrics_plugins }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar group variables"}
>
>    > <details-title>Running non-conda tools</details-title>
>    > If the tool you want to run on Pulsar doesn't have a conda package, you will need to make alternative arrangements! This is complex and beyond our scope here. See the [Pulsar documentation](https://pulsar.readthedocs.io/en/latest/) for details.
>    {: .details}
>
> 3. Add the following lines to your `hosts` file:
>
>    {% raw %}
>    ```diff
>    --- a/hosts
>    +++ b/hosts
>    @@ -2,3 +2,5 @@
>     gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>     [dbservers:children]
>     galaxyservers
>    +[pulsarservers]
>    +gat-0.oz.galaxy.training ansible_user=ubuntu
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar host"}
>
{: .hands_on}

We will now write a new playbook for the pulsar installation as we are going to install it on a separate VM. We will also install the CVMFS client and the Galaxy CVMFS repos on this machine so Pulsar has the same access to reference data that Galaxy does.

We need to include a couple of pre-tasks to install virtualenv, git, etc.

> <hands-on-title>Creating the playbook</hands-on-title>
>
> 1. Create a `pulsar.yml` file with the following contents:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/pulsar.yml
>    @@ -0,0 +1,21 @@
>    +- hosts: pulsarservers
>    +  vars_files:
>    +    - group_vars/secret.yml
>    +  pre_tasks:
>    +    - name: Install some packages
>    +      package:
>    +        name:
>    +          - build-essential
>    +          - git
>    +          - python3-dev
>    +          - libcurl4-openssl-dev
>    +          - libssl-dev
>    +          - virtualenv
>    +          - python3-venv
>    +        state: present
>    +        update_cache: yes
>    +      become: yes
>    +  roles:
>    +    - role: galaxyproject.cvmfs
>    +      become: yes
>    +    - galaxyproject.pulsar
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar playbook"}
>
>    There are a couple of *pre-tasks* here. This is because we need to install some base packages on these very vanilla ubuntu instances as well as give ourselves ownership of the directory we are installing into.
>
{: .hands_on}

> <hands-on-title>Run the Playbook</hands-on-title>
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook pulsar.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    After the script has run, pulsar will be installed on the remote machines!
>
>    > <tip-title>Connection issues?</tip-title>
>    > If your remote pulsar machine uses a different key, you may need to supply the `ansible-playbook` command with the private key for the connection using the `--private-key key.pem` option.
>    {: .tip}
>
> 2. Log in to the machines and have a look in the `/mnt/pulsar` directory. You will see the venv and config directories. All the config files created by Ansible can be perused.
>
> 3. Run `journalctl -f -u pulsar`
>
>    A log will now start scrolling, showing the startup of pulsar. You'll notice that it will be initializing and installing conda. Once this is completed, Pulsar will be listening on the assigned port.
>
{: .hands_on}

## Configuring Galaxy to use Pulsar as a job destination

Now we have a Pulsar server up and running, we need to tell our Galaxy about it.

Galaxy talks to the Pulsar server via it's job configuration file. We need to let Galaxy know about Pulsar there and make sure Galaxy has loaded the requisite job runner, and has a destination set up.

There are three things we need to do here:

* Create a job runner which uses the  `galaxy.jobs.runners.pulsar:PulsarMQJobRunner` code.
* Create a job destination referencing the above job runner.
* Tell Galaxy which tools to send to this job destination.

For this tutorial, we will configure Galaxy to run the BWA and BWA-MEM tools on Pulsar.

> <hands-on-title>Configure Galaxy</hands-on-title>
>
> 1. In your `group_vars/galaxyservers.yml` file add the following job runner to the `<plugins>` section:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -24,6 +24,16 @@ galaxy_job_config:
>         slurm:
>           load: galaxy.jobs.runners.slurm:SlurmJobRunner
>           drmaa_library_path: /usr/lib/slurm-drmaa/lib/libdrmaa.so.1
>    +    pulsar_runner:
>    +      load: galaxy.jobs.runners.pulsar:PulsarMQJobRunner
>    +      amqp_url: "pyamqp://pulsar_au:{{ vault_rabbitmq_password_vhost }}@localhost:5671/{{ rabbitmq_vhosts[0] }}?ssl=1"
>    +      amqp_acknowledge: true
>    +      amqp_ack_republish_time: 1200
>    +      amqp_consumer_timeout: 2
>    +      amqp_publish_retry: true
>    +      amqp_publish_retry_max_retries: 60
>    +      galaxy_url: "https://{{ inventory_hostname }}"
>    +      manager: _default_
>       handling:
>         assign: ['db-skip-locked']
>       execution:
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar plugin"}
>
>    Add the following to the `destinations:` section of your `tpv_rules_local` file:
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -56,3 +56,18 @@ destinations:
>         max_mem: 8
>         params:
>           native_specification: --nodes=1 --ntasks=1 --cpus-per-task={cores} --mem={round(mem*1024)} --time={entity.params['walltime']}:00:00
>    +
>    +  pulsar:
>    +    runner: pulsar_runner
>    +    params:
>    +      default_file_action: remote_transfer
>    +      dependency_resolution: remote
>    +      jobs_directory: /mnt/pulsar/files/staging
>    +      persistence_directory: /mnt/pulsar/files/persisted_data
>    +      remote_metadata: false
>    +      rewrite_parameters: true
>    +      transport: curl
>    +      outputs_to_working_directory: false
>    +    scheduling:
>    +      require:
>    +        - pulsar
>    {% endraw %}
>    ```
>    {: data-commit="Add pulsar destination"}
>
>    You'll notice we need to know a lot about the configuration of the remote end, this is an unfortunate requirement with pulsar. Changes to e.g. the staging directory need to be coordinated between Pulsar and Galaxy. That's fine if both are under your administration, but for a completely remote Pulsar it can be difficult.
>
>    Notably we also override `outputs_to_working_directory`, as this option is incompatible with running Pulsar, and, unnecessary. Pulsar already provides the same job isolation and safety that we request when we set that option by default in Galaxy's configuration.
>
> 2. Install the BWA and BWA-MEM tools, if needed.
>
>    {% snippet topics/admin/faqs/install_tool.md query="bwa" name="Map with BWA-MEM" section="Mapping" %}
>
> 3. We now need to tell Galaxy to send BWA and BWA-MEM jobs to the `pulsar` destination. We specify this in the `<tools>` section of your job conf.
>
>    Add the following to the end of the TPV rules file (inside the `tools:` section if it exists or create it if it doesn't.)
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/config/tpv_rules_local.yml
>    +++ b/files/galaxy/config/tpv_rules_local.yml
>    @@ -26,6 +26,14 @@ tools:
>             cores: int(job.get_param_values(app)['__job_resource']['cores'])
>             params:
>                walltime: "{int(job.get_param_values(app)['__job_resource']['time'])}"
>    +  .*/bwa/.*:
>    +    scheduling:
>    +      require:
>    +        - pulsar
>    +  .*/bwa_mem/.*:
>    +    scheduling:
>    +      require:
>    +        - pulsar
>
>     destinations:
>       local_env:
>    {% endraw %}
>    ```
>    {: data-commit="Send bwa and bwa-mem to pulsar"}
>
>    Note that here we are using the short tool IDs. If you want to run only a specific version of a tool in Pulsar, you have to use the full tool ID (e.g. `toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.17.4`) instead. The full tool ID can be found inside the `integrated_tool_panel.xml` file in the `mutable-config` directory.
>
> 4. Finally run the Galaxy playbook in order to deploy the updated job configuration, and to restart Galaxy.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1-pulsar.sh
> ```
> {: data-test="true"}
{: .hidden}


## Testing Pulsar

Now we will upload a small set of data to run bwa-mem with.

> <hands-on-title>Testing the Pulsar destination</hands-on-title>
>
> 1. Upload the following files from zenodo.
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    https://zenodo.org/record/582600/files/mutant_R2.fastq
>    ```
>
> 2. **Map with BWA-MEM** {% icon tool %} with the following parameters
>    - *"Will you select a reference genome from your history or use a built-in index"*: `Use a built-in genome index`
>    - *"Using reference genome"*: `Escherichia coli (str. K-12 substr MG1655): eschColi_K12`
>    - *"Single or Paired-end reads"*: `Paired end`
>    - {% icon param-file %} *"Select first set of reads"*: `mutant_R1.fastq`
>    - {% icon param-file %} *"Select second set of reads"*: `mutant_R2.fastq`
>
>    As soon as you press *execute* Galaxy will send the job to the pulsar server. You can watch the log in Galaxy using:
>
>    ```
>    journalctl -fu galaxy-*
>    ```
>
>    You can watch the log in Pulsar by ssh'ing to it and tailing the log file with:
>
>    ```
>    journalctl -fu pulsar
>    ```
>
{: .hands_on}

> ```bash
> 2-run-job.sh
> ```
> {: data-test="true"}
{: .hidden}

You'll notice that the Pulsar server has received the job (all the way in Australia!) and now should be installing bwa-mem via conda. Once this is complete (which may take a while - first time only) the job will run. When it starts running it will realise it needs the *E. coli* genome from CVMFS and fetch that, and then results will be returned to Galaxy!

How awesome is that? Pulsar in another continent with reference data automatically from CVMFS :)

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=11 %}

## Retries of the staging actions

When the staging actions are carried out by the Pulsar server itself (like in the case when driving Pulsar by message queue), there are some parameters that can be tweaked to ensure reliable communication between the Galaxy server and the remote Pulsar server.
The aim of these parameters is to control the retrying of staging actions in the event of a failure.

For each action (preprocess/input or postprocess/output), you can specify:
```text
 - *_action_max_retries    - the maximum number of retries before giving up
 - *_action_interval_start - how long start sleeping between retries (in seconds)
 - *_action_interval_step  - by how much the interval is increased for each retry (in seconds)
 - *_action_interval_max   - the maximum number of seconds to sleep between retries
```
substitute the * with `preprocess` or `postprocess`

In the following box, as an example, we have collected the values adopted in a Pulsar site with an unreliable network connection:

```yaml
preprocess_action_max_retries: 30
preprocess_action_interval_start: 2
preprocess_action_interval_step: 10
preprocess_action_interval_max: 300
postprocess_action_max_retries: 30
postprocess_action_interval_start: 2
postprocess_action_interval_step: 10
postprocess_action_interval_max: 300

```
In this case, for both actions, Pulsar will try to carry out the staging action 30 times, sleeping 2 secs after the first retry and adding 10 secs more to each next retries, until a maximum of 300 seconds between retries.

We hope you never have to experience a situation like this one, but if needed just adapt the numbers to your case and add the parameters in the `pulsar_yaml_config` section of your `pulsarservers.yml` file.

## Pulsar in Production

If you want to make use of Pulsar on a Supercomputer, you only need access to a submit node, and you will need to run Pulsar there. We recommend that if you need to run a setup with Pulsar, that you deploy an AMQP server (e.g. RabbitMQ) alongside your Galaxy. That way, you can run Pulsar on any submit nodes, and it can connect directly to the AMQP and Galaxy. Other Pulsar deployment options require exposing ports wherever Pulsar is running, and this requires significant more coordination effort.

For each new Pulsar server, you will need to add:
  1. In the RabbitMQ config:
      * A vhost
      * A user - configured with a password and the new vhost
  2. In the Galaxy job configuration
      * A new job runner with the new connection string
      * A new destination or multiple destinations for the new runner.

Pulsar servers can be the head node of a cluster. You can create a cluster and use your favourite job scheduler such as Slurm or PBS to schedule jobs. You can have many destinations in your Galaxy job conf file that change the number of cpus, amount of RAM etc. It can get quite complex and flexible if you like.

### Australia

You can also create multiple queues on your RabbitMQ server for multiple Pulsar servers. On Galaxy Australia, we run 5 different Pulsar servers spread out all around the country. They all communicate with Galaxy via the one RabbitMQ server.

![Map of australia with 6 pulsar nodes marked around the country.](../../images/pulsar_australia.png)

### Europe

Galaxy Europe has taken Pulsar and built [The Pulsar Network](https://pulsar-network.readthedocs.io/en/latest). This provides a framework for easily deploying Pulsar clusters in the cloud, something needed to support compute centers which might not have as much experience. This way they get an easy package they can deploy and the European Galaxy team can manage.

![Map of europe with pulsar nodes marked in many countries. An inset shows australia with a node there too.](https://pulsar-network.readthedocs.io/en/latest/_images/nodes.png)

The main purpose of this network is to support the workload of the UseGalaxy.eu instance by distributing it across several European data centers and clusters. If you're interested in setting up something similar, they [provide documentation](https://pulsar-network.readthedocs.io/en/latest/introduction.html) on how to install and configure a Pulsar network endpoint on a cloud infrastructure and how to connect it to your server.

## Conclusion

You're ready to ship your Galaxy jobs around the world! Now wherever you have compute space, you know how to setup a Pulsar node and connect it to Galaxy. Let us know if you come up with creative places to run your Galaxy jobs (coworker's laptops, your IoT fridge, the sky is the limit if it's x86 and has python)

{% snippet topics/admin/faqs/git-gat-path.md tutorial="pulsar" %}


topic: admin
tutorial: jenkins
title: Automation with Jenkins
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/jenkins/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "automation"]
content_hash: 52ce19eae8100c60d37fc0e8f203c9ae
last_modified: 2026-02-26T21:12:14.147091
workflows_json: []
content:
# Overview


Automation is a key component for making your life easier. There are dozens of regular, boring tasks involved in server administration, and many of these can be automated in order to make your life easier.
{CI} systems provide one way to accomplish this.
Many of these systems focus on testing and deploying code, and provide some built-in shortcuts to accomplish this.
Testing and deploying code boils down to "please run this command, when X happens", and they tend to function as "commands as a service".
You can often configure them to "test code when it is pushed to a branch on GitHub", or "test this code daily" or "deploy this code daily".
If you need to automatically run a command whenever some event happens, and then to report this back in some standardised way (GitHub PR statuses, automatically deploying, notifications in chat), then a CI system can do this for you.

There are many options for, some of these have different features which may be desirable in one or another scenario.
This tutorial will specifically cover automation with Jenkins as that is one of the tools `usegalaxy.*` use for a subset of their automation.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Comparison of CI Systems

There are dozens of {CI} systems available meeting different needs, which CI system you want to use depends largely on the task you are trying to automate. A couple of key factors you may wish to consider are:

- How long will jobs run for? Free services often have time and log-output limits.
- Are you concerned about secret storage? Self-hosted can be a better option here, but it depends on your security posture and threat matrix.
- Do you have infrastructure available? Self-hosted could be worth it for the extra freedoms.

In general, self-hosted options have the overhead of managing and securing and updating the service, but offer significantly more freedom if you need it. They generally provide poor per-job isolation and assume you trust all of the jobs and people with access to build scripts used in the build pipeline. The free services (Travis, Circle CI) often have much better user-experiences and are easier to get started with and provide better out-of-the-box isolation.

We will look at setting up and managing a Jenkins server today, how to setup tasks, and some of the things that `usegalaxy.*` automate with their Jenkins servers.

# Infrastructure

As per many of the tutorials, we will use Ansible to setup and manage our Jenkins server so we can easily leverage the work of a large community. Jenkins is one choice, there are other self-hosted systems you could use, but it is the one with which many of us are familiar.

## Jenkins

> <hands-on-title>Setting up Jenkins</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    ```yaml
>    - src: geerlingguy.java
>      version: 1.10.0
>    - src: geerlingguy.jenkins
>      version: 3.7.0
>    ```
>
> 2. Install the requirements
>
>    ```
>    ansible-galaxy install -p roles -r requirements.yml
>    ```
>
> 3. Create a new playbook, `build.yml` with the following:
>
>    ```yaml
>    ---
>    - hosts: build
>      become: true
>      pre_tasks:
>        - name: Allow jenkins user to execute things as root
>          copy:
>            content: |
>              jenkins ALL=(ALL:ALL) NOPASSWD:ALL
>            dest: /etc/sudoers.d/jenkins
>            validate: 'visudo -cf %s'
>            mode: 0440
>      roles:
>        - geerlingguy.java
>        - geerlingguy.jenkins
>    ```
>
>    During this tutorial we will install everything on the same host, but often one keeps the build infrastructure on a separate host. We permit the Jenkins user to run everything as root, because later we will run Ansible with Jenkins, and we have `become: true` in our `galaxy.yml` playbook, so the `jenkins` user will need to be able to become root.
>
> 4. Edit the inventory file (`hosts`) an add a group for monitoring like:
>
>    ```ini
>    [build]
>    localhost ansible_connection=local
>    ```
>
> 5. Create and edit the group variables for the `build` group, `group_vars/build.yml`, and include the following:
>
>    ```yml
>    jenkins_http_port: 4000
>    jenkins_admin_username: admin
>    # Please change at least the password to something more suitable
>    jenkins_admin_password: admin
>    jenkins_url_prefix: /jenkins
>    jenkins_plugins:
>    - matrix-auth
>    ```
>
> 6. Run the build server playbook:
>
>    ```
>    ansible-playbook -i hosts build.yml
>    ```
>
> 7. Update the nginx configuration in `templates/nginx/galaxy.j2` to proxy jenkins with the following. Add it *before the last curly brace* at the end of the file.
>
>    ```nginx
>        location /jenkins {
>            proxy_set_header        Host $host:$server_port;
>            proxy_set_header        X-Real-IP $remote_addr;
>            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
>            proxy_set_header        X-Forwarded-Proto $scheme;
>
>            # Fix the "It appears that your reverse proxy set up is broken" error.
>            proxy_pass          http://127.0.0.1:4000;
>            proxy_read_timeout  90;
>        }
>    ```
>
> 8. Run the galaxy playbook which contains Nginx:
>
>    ```
>    ansible-playbook -i hosts galaxy.yml
>    ```
>
{: .hands_on}

With this, we're done, Jenkins is set up and ready to use!

# Building things with Jenkins

We will look at a couple basic cases of how to build things with Jenkins, and wrap up with some advanced usage documentation that is not so practical for a tutorial environment.

Start by **visiting your Jenkins instance**, at `https://your-domain/jenkins`

![Welcome to Jenkins](../../images/jenkins-01-welcome.png "Welcome to Jenkins! This is how it looks once you've run the playbook and logged in. The menu on the left gives you some ability to configure Jenkins, below it is the job queue where running and queued jobs will appear. When you have configured some jobs, they will be visible in the central panel.")

## A simple job

> <hands-on-title>Create a simple job</hands-on-title>
>
> 1. Visit your Jenkins and login with the credentials you set up in the group variables file
>
> 2. Click **New Item** in the left hand menu
>
> 3. Give your job a name like `my-project`, and select that it is a `freestyle project`, and click "OK"
>
>    ![Create a job](../../images/jenkins-02-create-job.png "There are other types of jobs if you install plugins, but we will mostly use freestyle jobs as they are the most appropriate for our case.")
>
> 4. The job configuration interface is split into a few sections, visible in the tabs at the top of the page:
>
>    ![Job configuration interface](../../images/jenkins-03-job-config.png "The job configuration interface, split into 'General', 'Source Code Management', 'Build Triggers', 'Build', and 'Post-build Actions'")
>
> 5. In the **Build Triggers** section:
>
>    - Build Periodically
>        - *"Schedule"*: `H * * * * `
>
>    > <details-title>Cron syntax</details-title>
>    > Jenkins uses the [cron syntax](https://en.wikipedia.org/wiki/Cron#Overview), with the addition that `H` can be used to randomly choose a value for that place, rather than running at precisely that minute or hour. This allows you to spread the load of multiple jobs over a period of time, rather than having maybe 10 jobs all attempting to run simultaneously if they do not need to.
>    > For the above example it will choose one value for the `H`, a random minute in the hour, and then run every hour, every day, at that minute.
>    > You can enter different expressions to see when Jenkins would run it next.
>    {: .details}
>
> 6. Add a build step:
>
>    - Add build step
>        - **Click** "Execute shell"
>    - In the new step
>        - *"Command"*:
>          ```
>          echo Hello world!
>          date
>          env | sort
>          ```
>
>      ![Executing a shell](../../images/jenkins-05-job1.png)
>
>    > <warning-title>Danger!</warning-title>
>    > This field allows you to execute arbitrary bash scripts. This is a reason Jenkins **must** be secured with HTTPS and a good username/password.
>    {: .warning}
>
>
> 7. Click **Save** at the bottom
>
> 8. You will be brought back to the job information page. Click **Build Now**
>
> 9. Your job will appear in the *Build History* box at the left. **Click** the ball associated with your job, to see the console.
>
>    ![Seeing a Jenkins job's console](../../images/jenkins-06-job1-run.png "The build history box lets you see the status of previous, running, and queued builds. The 'sun' icon indicates that all builds have been successful, it will become more cloudy/overcast if you experience failures. The 'trend' button allows you to visualise how quickly builds have run historically.")
>
>    Clicking the number will bring you to a job information page, where you can click "Console Output" to see the jobs output
>
>    ![Jenkins job output](../../images/jenkins-07-job1-out.png "The output of your first Jenkins job. All jenkins jobs log both the commands that were run, and their outputs. Use this to see which step potentially failed.")
{: .hands_on}

Setting up Jenkins jobs is as simple as setting up a cron job, but the results are stored in a nice visual interface that may provide better visibility than a `cron` job.

> <comment-title>Author's commentary</comment-title>
> Cron jobs are great and often useful, but most people ignore the emails that cron sends them. I know I have mailboxes across N servers with dozens and dozens of unread emails from cron jobs. Jenkins is an improvement here because it shows "success" and "failure" messages that are clear, while storing the output in case you later decide that you want to look at it. For the most part, cron jobs write output that is never read, and we really only want to see the output if something has failed, but writing a cron job that behaves like this is unnecessarily complex.
{: .comment}

## Ansible in Jenkins

We will now setup Jenkins to run Ansible on cron. In the [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) tutorial we emphasised that it is useful to often run the entire playbook to ensure that all changes are applied. UseGalaxy.eu likes to accomplish this by having Jenkins run the playbooks every day. We know that even if our coworkers made some changes to the servers, that by tomorrow it will be reverted to a known-good configuration.

> <hands-on-title>Jenkins running Ansible</hands-on-title>
>
> 1. Go back to the Jenkins homepage (click the logo in the top left)
>
> 2. Click **New Item** in the left hand menu.
>
> 3. Give your job a name like "ansible-galaxy", and select that it is a "freestyle project", and click "OK"
>
> 4. Configure your job
>
>    - Build Triggers
>      - Build Periodically
>        - *"Schedule"*: `H * * * * `
>    - Build
>      - Add build step
>        - **Click** "Execute shell"
>      - In the new step
>        - *"Command"*:
>          ```
>          cd /home/ubuntu/galaxy/ # Change this to wherever your galaxy.yml is, you can check with the command pwd
>          ansible-playbook -i hosts galaxy.yml --diff
>          ```
>    - Save
>
> 5. Click **Build Now** and check the console output of the job
>
>    What do you see? It should look like Jenkins is running the playbook:
>
>    ```
>    Started by user admin
>    Running as SYSTEM
>    Building in workspace /var/lib/jenkins/workspace/ansible-galaxy
>    [ansible-galaxy] $ /bin/sh -xe /tmp/jenkins6010384499214486550.sh
>    + cd /home/ubuntu/galaxy/
>    + ansible-playbook -i hosts galaxy.yml --diff
>
>    PLAY [galaxyservers] ***********************************************************
>
>    TASK [Gathering Facts] *********************************************************
>    ok: [gcc-3.training.galaxyproject.eu]
>
>    TASK [Install Dependencies] ****************************************************
>     [WARNING]: Could not find aptitude. Using apt-get instead
>    ok: [gcc-3.training.galaxyproject.eu]
>    ```
>
{: .hands_on}

With this we have Jenkins running our playbook for us! You can imaging several steps you could take to follow up from this:

1. Storing your playbooks in `git` like UseGalaxy.\*
	- [UseGalaxy.eu's playbooks](https://github.com/usegalaxy-eu/infrastructure-playbook) are public, anyone can contribute and if the PR is merged, it is automatically applied to our infrastructure within the next day. We store our job configuration in here as well which specifies how much memory and how many CPUs tools receive. Our teammates often can contribute fixes themselves when they notice a tool needs more memory.
	- [UseGalaxy.org's playbooks](https://github.com/galaxyproject/infrastructure-playbook) are also public, but run manually.
2. Running this job regularly
	- This can prevent mistakes, if a coworker edits some configuration on the server and forgets to commit it, this ensures everything is reverted back to the expected state.
	- The amount of time the Galaxy playbook takes can be quite annoying when done manually but no one is upset if it takes 30 minutes when it is run on cron.
3. Alerting you if something went wrong
	- There are numerous plugins that can send notifications of build success/failure to various places like Slack, etc.

These are all next steps that are great to look into, and will make your systems more reliable and trustworthy as you will know when things change, and you can be certain that the code is configured how you specified.

# Securing Jenkins

You have already secured Jenkins in that it is protected with a good password and HTTPS. This is one important thing. The other portion is the visibility of jobs and their outputs. In the above Ansible Galaxy deployment job we set the parameter `--diff`, so we could see what changed. This is an extremely helpful thing to do to be able to audit changes from a particular run. But! If you change the `id_secret` or other secret variables, then this diff will be visible in the job's console logs. This is useful to see for admins but needs to be restricted so that the whole world cannot see this.

> <hands-on-title>Securing Jenkins</hands-on-title>
>
> 1. Return to the Jenkins home page
>
> 2. Click on **Manage Jenkins**
>
> 3. Click on **Configure Global Security**
>
>    1. In the first block *Access Control* Change the *Authorization* strategy from "Logged-in users can do anything" to "Matrix-based security"
>
>    2. Click "Add user or group" and add the `admin` user
>
>    3. Configure the permissions like below. Clicking on the checked checkbox icon at the right will allow you to enable/disable all of the checkboxes for that row.
>
>       ![Configuring matrix based permissions](../../images/jenkins-09-permissions.png)
>
>       The matrix based security plugin is provided by a plugin, we installed this by setting the variable `jenkins_plugins: [matrix-auth]` in our group variables earlier.
>       The checkboxes give you the ability to grant or deny permissions to certain classes of users. Each column is an individual permission, and you can hover over each column to get more information about what that permission allows.
>
>       In the above example we have configured that:
>         - *Anonymous users* (not logged in) may `Read`, so they can access the Jenkins page, but by default will not see any jobs. The `Discover` permission is also granted so that if a user has bookmarked the ansible-galaxy job page, they will automatically be redirected through the login page, if they are not already logged in.
>         - Once they login and are *Authenticated Users*, they can see that `Manage` and `Run` Jobs and Views
>         - Only the *admin* user, however, can configure `Agents` to run those jobs.
>
>    3. Under **CSRF Protection**, check the box to prevent CSRF, and select the default crumb issuer, and then enable proxy compatibility.
>
> 4. Save
>
>    > <question-title></question-title>
>    >
>    > Try accessing your Jenkins from a private browsing session. How does it look?
>    >
>    > > <solution-title></solution-title>
>    > > It should appear as if there are no jobs, until you log in.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Advanced Topics

These topics are outside the scope of a training or do not lend themselves well to interesting exercises, but some discussion and configuration information will be provided below documenting common issues and questions that can be useful if you start using Jenkins "in anger".

## Updating Jenkins

Jenkins updates regularly due to security issues. We recommend that you either:

1. Regularly check the [Jenkins Changelog](https://jenkins.io/changelog/), and update the server whenever there are important security fixes
2. OR include some sort of yum or apt auto-updater on whichever system that Jenkins is running on

The downside of automatic updating is that Jenkins will restart and kill any jobs that had been running. Choose which works best for your environment. UseGalaxy.eu chose a completely automated setup because their jobs can be killed and few things are affected. UseGalaxy.org chose a more manual setup because their jobs are more critical, there if jobs fail, Galaxy pull request tests will fail unexpectedly and unhelpfully.

## Updating Plugins

Plugins also regularly receive updates due to security issues. You should either check in with your Jenkins server regularly and apply updates when they are available, or automate this as well. UseGalaxy.eu uses a script like:

```bash
#!/bin/bash
ARGS="-jar /opt/jenkins-cli.jar -s http://localhost:4000/ -auth user:password"
UPDATE_LIST=$(java $ARGS list-plugins | grep ')$' | awk '{print $1}')

if [ ! -z "${UPDATE_LIST}" ]; then
    echo Updating Jenkins Plugins: ${UPDATE_LIST};
    java $ARGS install-plugin ${UPDATE_LIST};
    java $ARGS safe-restart;
fi
```

And we run this on cron hourly. The `safe-restart` at the end causes Jenkins to stop processing new jobs, and wait until all jobs are complete before restarting. Unfortunately this is not an option for Jenkins server updates.

## Plugins

Jenkins provides an impressive number of plugins for a lot of different environments. Here are some of the ones we use and their purposes:

Plugin                           | Comments
------                           | --------
Git Plugin                       | Allow cloning git repositories to build
Build Timeout                    | Allows us to automatically kill jobs which run longer than we expect. We can use this to set per-build timeouts
Credentials Binding Plugin       | This can be used to inject secrets into builds, e.g. vault passwords. <!-- TODO: use this with vault in ansible-galaxy tuto -->
Embeddable Build Status Plugin   | Produces nice build statuses like Travis does, e.g. [UseGalaxy.eu's playbook statuses](https://github.com/usegalaxy-eu/infrastructure-playbook#build-statuses)
GitHub Authentication plugin     | The UseGalaxy.eu build server requires login with GitHub and ensures membership in the correct organisation to be able to access group secrets and build logs.
Job Configuration History plugin | This plugin tracks changes made in the configuration of jobs over time, so you can see what was changed in the Jenkins configuration which may have caused the job to start succeeding or failing.
ShiningPanda Plugin              | Allows easily running jobs within a per-job python virtualenv.
SSH Agent Plugin                 | Allows providing specific SSH credentials to a build

These plugins can be installed via the Ansible automation by expanding `jenkins_plugins` in your `group_vars/build.yml`, or installed with the web interface.

### Source Code Management

Jenkins supports most VCSs that are in use today. For the Git plugin, it allows you to clone a git repository into the job's directory, and then run scripts from there. It forms the foundation of most jobs.

You may need to specify credentials for cloning from private repositories, in this case you should beware that when you configure the SSH credentials you should specify the `user` as `git`. You will also need to ensure that the server's SSH public key is known to your Jenkins system which generally requires `su`ing as the Jenkins user and running `ssh git@...` once, and accepting the "Unknown host" key prompt. We recommend that you use repository URLs of the format `git@github.com:org/repo.git`, if you are using one of the GitHub PR builder plugins, then this URL has to match a value in the incoming webhook and it is not always obvious that this needs to be set like this.

The "Branches to build" section allows you to specify which branches. Whenever you click "Build now", Jenkins usually just picks the main branch to build, and doesn't run a build job per-branch. If you are automatically triggering Jenkins builds based on GitHub (or other) webhooks, then you can ensure that branches that should not be built are filtered out here. Real life example: UseGalaxy.eu has one Jenkins job which should only build the main branch of a repository, and another job that only builds the PRs for that repository. We know the first job will only ever build the main branch so we can hardcode that there. With the Git plugin, many "Additional Behaviours" are available. If you have submodules in your repository, you will need to enable one of these to ensure that the repository is cloned recursively. If you want to ensure a fresh start each time because you write out temporary files to the current working directory, then there is a behaviour to wipe out the repository before each build.

### Build Triggers

The GitHub Pull Request Builder plugin is not always easy to configure, here systems like Travis are generally significantly easier. Consider if that is an option. It is not a bad thing to use multiple {CI} systems, it provides some degree of redundancy where if one system is experiencing an outage it may not affect all CI jobs.

### Credential Binding

If you provide credentials to Jenkins builds, they will automatically be stripped from log outputs. So if your job will not "leak" any private information (e.g. in `--diff`) and you want to share the build logs with your users to prove that actions were taken or that things are working and automated, then the credential binding plugin ensures that any secrets will be stripped from output and replaced with `******`. They can be made available as environment variables, or in files that can be accessed.

We often upload a configuration file with secrets (e.g. `.parsec.yml` ), and then can just call our software `parsec -c $PARSEC_CONFIG_FILE_LOCATION ...` passing in that file, wherever Jenkins has stored it securely.

### SSH Agent

This allows you to automatically inject credentials into your build's SSH agent. If you want to `rsync` some build outputs to another server, or `ssh` to a server to run some one-off command, or to run `ansible`, then this is the feature you want. Beware that if you are pulling from and pushing to git repositories on GitHub with separate SSH credentials you may need to use a combination of the SSH Agent plugin, and injecting an SSH key through credential binding.

## Real Life Examples

Here are some examples of how Jenkins is used in the wild, maybe it can provide some reference for potential uses within your organisation or solutions to various problems. The examples are mostly pulled from UseGalaxy.eu's experiences, as the author is most familiar with those.

### Compute Nodes

We attach VMs as compute nodes for Jenkins to use as executors. We have an OpenStack cloud available so that is an easy option for us. We setup the VMs such that Jenkins just logs in as the default `centos` user and uses the home directory for building jobs. There is a Jenkins plugin to manage the VMs in the cloud automatically (scaling up/down as necessary, removing VMs with full disk) but our cloud was upgraded and we could not get this working. It is worth trying if that is an option! There are other cloud provider plugins available with similar features.

We also use a custom image that includes a lot of our requirements pre-installed. We include e.g. LaTeX, Ruby/RVM, Java, Node, anything we think we might need for our jobs. This also lets us reduce the privileges of the jobs, nothing *needs* `root` access to install packages globally. Python, Ruby, and Node all have something similar to virtualenvs.

### Job: Apollo Builder

This is a relatively straightforward job to build a WAR file and upload it to our depot server.

```console
# some env vars that are needed, it's kind of gross but it works :(
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/
export PERL5LIB=/home/centos/workspace/usegalaxy-eu/apollo-builder/extlib/lib/perl5:$PERL5LIB
# setting a variable from some build data
APOLLO_VERSION=$(cat application.properties | grep app.version | sed 's/.*=//g')

# Check if this version needs to be built
curl --silent https://usegalaxy.eu/static/vgcn/ | \
	grep -F "apollo-${APOLLO_VERSION}" && echo "Already built!" && exit 0;

# Build the thing
./apollo deploy

# Deploy it to our server
scp -i $PRIV_KEY \
	target/*.war \
    user@server:/static/apollo-${APOLLO_VERSION}.war
```

- We use the "Secret file" binding to inject a specific private key (available in the build as `$PRIV_KEY`) but the SSH agent plugin would work equally well.
- We could also use the "Archive Artifacts" plugin to have Jenkins retain all of the artefacts (build outputs that you care about) but we wanted to have an Ansible role pull those files and it was simpler to have them accessible at predictable URLs.

### Job: Chado Schema Builder

Similar to the Apollo builder but it uploads outputs as GitHub releases which is useful for public projects with complex build pipelines. The CSB takes on average 5-6 hours to run so we cannot easily run this on free infrastructure.

Here we store both the [build](https://github.com/hexylena/chado-schema-builder/blob/master/.ci/run.sh) and [deploy](https://github.com/hexylena/chado-schema-builder/blob/master/.ci/upload.sh) scripts in the GitHub repository. If you do this, you have to be careful about building PRs, as anyone can edit the build scripts and access your internal infrastructure!

### Job: Website

We build our own Jekyll website because we have some plugins GitHub will not compile for us. The configuration here is more complex:

- Throttle builds, since GitHub will only accept so many deploy requests in a time period
- One SSH key used to clone the repository using the Git SCM, with "Wipe out repository & force clone" set.
- Triggers on both:
  - Cron, daily
  - GitHub webhooks notifying us of pushes to master
- SSH Agent is configured with a deploy SSH key (because git really prefers you provide keys in the SSH agent, I don't know if it is possible to pass a keyfile on the CLI)
- The [.build.sh](https://github.com/usegalaxy-eu/website/blob/master/.build.sh) is run which activates the RVM pre-installed on the VMs
- If the master branch is checked out, [`.publish.sh`](https://github.com/usegalaxy-eu/website/blob/master/.publish.sh) is run
- Empty [`.nojekyll`](https://github.com/usegalaxy-eu/website/blob/master/.nojekyll) file to prevent GitHub from trying to build this.

### Job: Install Tools

For this job we use Jenkin's "Trigger other builds" feature:
- We have one job that fetches all of the latest updates for IUC tools and writes those into yaml files. This job runs on cron every week.
- This triggers a separate job to actually apply the changes to our server.

By splitting these up, we can run the "actually apply" portion without having to fetch updates to the tools first. Additionally we can have the second portion run only if the first portion succeeds, so we have some check that everything looks OK before trying to apply it.

We include a Galaxy API key as a credential and then use a `Makefile` contained in the project to do the steps. We don't automatically test PRs so we don't care if people change the Makefile because we visually validate it before merging their PR.

### Configuration: Themes

Many people find the "blue/red ball" icons a bit hard to understand. Using the "Simple Theme Plugin" you can easily add custom CSS files or directly enter custom CSS. UseGalaxy.\* use the [jenkins-material-theme](http://afonsof.com/jenkins-material-theme/) as it provides a slightly cleaner view of Jenkins with better iconography.

### Configuration: Folder Permissions

We have the aforementioned problem that our playbooks run with `--diff` and while we want to capture this output and be able to review it, it is not necessary that the public can see it. So permissions are configured not only on the global Jenkins level, but also within a single folder of jobs. The folder permissions involve setting:

- Properties
  - *"Inheritance Strategy"*: `Do not inherit permission grants from other ACLs`
  - Allowing `Discover` and nothing else for Anonymous + Authenticated
  - Explicitly granting all permissions for the admins


