# tutorials

topic: assembly
tutorial: largegenome
title: Large genome assembly and polishing
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["assembly", "polishing", "nanopore", "plants", "biodiversity"]
content_hash: 3478f12d8a54651e5f820eb09d516fcf
last_modified: 2026-05-27T09:01:47.814966
zenodo_link: https://doi.org/10.5281/zenodo.7055935
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/largegenome/tutorial.html

content:
*Note: We recommend running this tutorial on either the Galaxy Europe or Galaxy Australia servers. Other servers (such as Galaxy main) have not yet been configured fully for all the tools in this analysis.*

## What is genome assembly?

A genome is a representation of the set of DNA in an organism, such as the set of chromosomes. When the DNA is extracted from the sample, it is broken up into fragments much smaller than the lengths of DNA in the chromosomes. After being sequenced, these fragments (and their copies) are called sequencing reads. To assemble the genome, we need to join the reads back into, ideally, chromosome-sized lengths.

!["Image showing the concept of a genome assembly: a sample (a tree), with an arrow going to sequencing reads, with an arrow going to a genome (a set of diploid chromosomes)"](images/sampletogenome.png){: width="800"}

### Assembly challenges

In reality, we rarely get chromosome-length assemblies, due to many challenges. Here are some examples of particular challenges in a diploid genome assembly:

!["Image showing potential assembly challenges: a set of diploid chromosomes with descriptions of different problems which are: a contig may be a consensus from both chromosomes; telomeres and centromeres are difficult to assemble; contigs may be assembled separately as a copy in both chromosomes but it is unclear which parental chromosome they are from; some contigs are short or missing; some contigs are not placed in the assembly."](images/challenges.png){: width="800"}


Even though most assemblies are not chromosome-length, the assembly in contigs is still valuable for many research questions. Lengths of assembled contigs are increasing as sequencing technology and assembly tools improve.

### Can I use my own data with these workflows?

This tutorial has been tested on real-sized data sets and should work with your own data. However, there will most likely be some modifications required to tools and settings. Furthermore, as most species have never had their genome sequenced, it is not possible to guarantee existing workflows are optimal for new data.

It is most likely that any new genome assembly will have its own set of required workflow and analysis customisations to account for things such as ploidy and repeats. Usually, an assembly workflow will need testing and customising, in concert with reading the biological domain literature.

Before using these workflows on real datasets, we recommend completing all the relevant GTN tutorials about Data QC and Asssembly to give extra familiarity with the concepts. Then, we suggest creating subsampled data sets from your full-sized data, for testing. For best results, it is recommended to test each tool separately, and consider the parameters and their relevance and suitability for your own research questions.

> <agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Analysis workflow

In this tutorial, we will follow these steps:

!["An image showing the steps in the tutorial in a stylized form: quality control and kmer counting, represented by bold lines as sequencing reads; trimming and filtering reads, represented by truncated reads; assembly, represented by joined reads into one long piece; polishing, represented by corrections (ticks) in the assembly; the finished genome assembly, represented by a set of diploid chromosomes; quality assessment, represented by ticks and crosses at different chromosome locations."](images/steps.png){: width="800"}

* Each of these steps is described in a section in this tutorial.
* For each step, we will run a workflow.
* We will stay in the same Galaxy history throughout.

## How to run a workflow in Galaxy

* All of these workflows have been uploaded to [workflowhub.eu](https://workflowhub.eu/) and have been tagged with "Large-genome-assembly". There, each workflow is accompanied by an image of the workflow canvas and the tool connections.
* The workflows are also linked to this tutorial - see above and import them into your own Galaxy Account.
* If you are using Galaxy Australia, the workflows are published with the tag "lg-wf".
* A note: as of September 2022, these workflows have been tested on Galaxy Australia and we are now in the process of testing them on Galaxy Europe and Galaxy Main.

Here is a list of the workflows:

|Tutorial section | Workflow name|
|-----------------|--------------|
|Quality Control | Data QC|
|Kmer counting | kmer counting - meryl|
|Trim and filter reads | Trim and filter reads - fastp|
|Assembly | Assembly with Flye |
|Polish assembly | Assembly polishing |
|Assess quality | Assess genome quality|

> <tip-title>Importing the workflows from WorkflowHub</tip-title>
>
> [WorkflowHub](https://workflowhub.eu/) is a workflow management system which allows workflows to be FAIR (Findable, Accessible, Interoperable, and Reusable), citable, have managed metadata profiles, and be openly available for review and analytics.
>
> {% snippet faqs/galaxy/workflows_import_search.md box_type="none" trs_server="workflowhub.eu" search_query='"large genome assembly"' %}
>
{: .tip}

After importing the workflows into your account:

* Look at your of Workflows. (Galaxy top panel: Workflow)
* For the workflow you want to run, go to the right hand side and see the arrow button (a triangle), click
* This brings up the workflow in the centre Galaxy panel
* Click "Expand to full workflow form"
* For "Send results to a new history", leave it as "No".
* Each time you run a workflow, you need to specify the input data set (or sets). Galaxy will try to guess which file this is, but change if required using the drop-down arrow.
* At the top right, click "Run Workflow".
* The result files will appear at the top of your current history

Each workflow will be discussed in a separate section.

# Upload data

Let's start with uploading the data.

## What sequence data are we using in the tutorial?

* The data sets for genome projects can be very large and tools can take some time to run. It is a good idea to test that your planned tools and workflows will work on smaller-sized test data sets, as it is much quicker to find out about any problems.
* In this tutorial we will use a subset of real sequencing data from a plant genome, the snow gum, *Eucalyptus pauciflora*, from a genome project described in this paper: {% cite Wang2020 %}. Data is hosted at NCBI BioProject number: PRJNA450887.

## How has this data subset been prepared?

* From NCBI, three read files were imported into Galaxy for this tutorial: nanopore reads (SRR7153076), and paired Illumina reads (SRR7153045).
* These were randomly subsampled to 10% of the original file size.
* Plant genomes may contain an excess of reads from the chloroplast genome (of which there are many copies per cell). To ensure our test data sets are not swamped from excessive chloroplast-genome reads, reads that mapped to a set of known chloroplast gene sequences were discarded.

We are also using a reference genome *Arabidopsis thaliana* for a later comparison step - file TAIR10_chr_all.fas downloaded from [The Arabidopsis Information Resource](https://www.arabidopsis.org/download/index-auto.jsp?dir=%2Fdownload_files%2FGenes%2FTAIR10_genome_release%2FTAIR10_chromosome_files).

## Import the data

> <hands-on-title>Import the data</hands-on-title>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import from [Zenodo](https://zenodo.org/record/7055935) or a data library (ask your instructor):
>
>    - 2 FASTQ files with illumina reads: these files have a R1 or R2 in the name
>    - 1 FASTQ file with nanopore reads: this file has nano in the name
>    - 1 FASTQ file for the genome Arabidopsis
>
>    ```
>    https://zenodo.org/record/7055935/files/Eucalyptus_subsample_ten_pc_SRR7153045_R1.fastq.gz
>    https://zenodo.org/record/7055935/files/Eucalyptus_subsample_ten_pc_SRR7153045_R2.fastq.gz
>    https://zenodo.org/record/7055935/files/Eucalyptus_subsample_ten_pc_nano_SRR7153076.fastq.gz
>    https://zenodo.org/record/7055935/files/Ref_genome_Arabidopsis.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check that the datatypes for the three files of sequencing reads are `fastq.gz`, not `fastqsanger.gz` and change datatype if needed.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

* This tutorial uses these input files and gives some examples from the results.
* It is likely that your results will differ slightly (e.g. number of bases in the genome assembly). This is common, because many tools start from different random seeds.
* Also, tool versions are being constantly updated.  Newer versions may be available since this tutorial was written and could give slightly different results.

# Check read quality

Let's look at how many reads we have and their quality scores using the Data QC workflow.

## Data QC workflow

What it does: Reports statistics from sequencing reads

Inputs:
* long reads (fastq.gz format)
* short reads (R1 and R2) (fastq.gz format)

Outputs:
* For long reads: a nanoplot report (the HTML report summarizes all the information).
* For short reads: a MultiQC report

Tools used:
* Nanoplot
* FastQC
* MultiQC

Workflow steps:
* Long reads are analysed by Nanoplot.
* Short reads (R1 and R2) are analysed by FastQC
* the resulting reports are processed by MultiQC

Options:
* See the tool settings options at runtime and change as required.
* Alternative tool option: fastp

> <hands-on-title>Run the Data QC workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Data_QC.ga" title="Galaxy Workflow Data QC" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Data QC** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: long reads"*: the nanopore fastq.gz file
>    - {% icon param-file %} *"2: Input file: Illumina reads R1"*: the illumina R1 fastq.gz file
>    - {% icon param-file %} *"3: Input file: Illumina reads R2"*: the illumina R2 fastq.gz file
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

## Data QC results

> <question-title></question-title>
>
> 1. What are the results from the two output files? Are the reads long enough and of high enough quality for our downstream analyses?  Will reads need any trimming or filtering?
>
> > <solution-title></solution-title>
> >
> > 1. Common things to check are average read length, average quality, and whether quality varies by position in the reads.
> > * Look at the plot for MultiQC for the Sequence Quality Histograms. This shows how the read quality of Illumina reads (y axis) varies according to base position (x axis).You may see for Illumina reads that there is some drop-off in quality towards the end of the reads, which may benefit from trimming.
> > * Look at the plot from Nanoplot for "Read lengths vs Average read quality". The nanopore reads have a mean read quality of 9.0. Depending on the size of our input read sets and the research question, we may filter out reads below a certain average quality. If we had a lot of reads, we may be able to set a higher threshold for filtering according to read quality.
> {: .solution}
{: .question}

> <comment-title>Additional information about the tools</comment-title>
> More about interpreting nanoplot plots:
> * [Nanoplot on Github](https://github.com/wdecoster/NanoPlot)
> * [Example nanoplots](https://gigabaseorgigabyte.wordpress.com/2017/06/01/example-gallery-of-nanoplot/)
>
> More about FastQC results:
> * [FastQC webpage](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
> * [FastQC tips](https://timkahlke.github.io/LongRead_tutorials/QC_F.html)
{: .comment}

# Determine genome characteristics

To prepare for genome assembly you might want to know things about your genome such as size, ploidy level (how many sets of chromosomes) and heterozygosity (how variable the sequence is between homologous chromosomes). A relatively fast way to estimate these things is to count small fragments of the sequencing reads (called kmers).

A read broken into kmers:

!["An image of the a read broken into kmers: the read is a bold bar with the letters ABCDEF; each kmer is a smaller bar with three letters in it: the kmers are ABC, BCD, CDE, and DEF."](images/kmers.png){: width="650"}

> <comment-title>What is kmer counting?</comment-title>
>
> Kmer counting is usually done with high-accuracy short reads, not long reads which may have high error rates.  After counting how many times each kmer is seen in the reads, we can see what sorts of counts are common. For example, lots of kmers may have been found 24 or 25 times. A graph shows the number of different kmers (y axis) found at different counts, or depths (x axis).
>
> Many different kmers will be found the same number of times; e.g. X25. If kmer length approaches read length, this means the average depth of your sequencing is also ~X25, and there would be a peak in the graph at this position (smaller kmers = higher kmer depth). There may be smaller peaks of kmer counts at higher depths, e.g. X50 or X100, indicating repeats in the genome. There may be other smaller peaks of kmers found at half the average depth, indicating a diploid genome with a certain amount of difference between the homologous chromosomes - this is known as heterozygosity. Thus, the plot of how many different kmers are found at all the depths will help inform estimates of sequencing depth, ploidy level, heterozygosity, and genome size.
{: .comment}

## Kmer counting workflow

What it does: Estimates genome size and heterozygosity based on counts of kmers

Inputs: One set of short reads: e.g. R1.fq.gz

Outputs: GenomeScope graphs

Tools used:
* Meryl
* GenomeScope

Workflow steps:
* The tool meryl counts kmers in the input reads (k=21), then converts this into a histogram.
* GenomeScope: runs a model on the histogram; reports estimates. k-mer size set to 21.

Options:
* Use a different kmer counting tool. e.g. khmer. If so, for the settings, advanced parameters:
* k-mer size: 21 (as per [this recommendation](https://github.com/schatzlab/genomescope/issues/32)).
* n_tables: 4. tablesize: set at 8 billion (as per [this recommendation](https://khmer.readthedocs.io/en/v1.0/choosing-table-sizes.html)).
* Will also need to run some formatting steps to convert khmer output to a two-column matrix, for the  Genomscope. See [this workflow](https://usegalaxy.org.au/u/anna/w/kmer-counting-khmer).
* Note: khmer: to use both R1 and R2 read sets, khmer needs these paired reads in interleaved format.


> <hands-on-title>Run the Kmer counting workflow</hands-on-title>
>
> 1. **Import the Kmer counting workflow** into Galaxy:
>    - Copy the URL (e.g. via right-click) of [this workflow]({{ site.baseurl }}{{ page.dir }}workflows/Galaxy-Workflow-kmer_counting.ga) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-kmer_counting.ga" title="Kmer counting workflow" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Kmer counting** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: Illumina reads R1"*: the illumina R1 fastq.gz file
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}


## Kmer counting results

GenomeScope transformed linear plot:

!["An image of the Genomescope plot, described fully in the text following this image."](images/genomescope.png){: width="800"}

Here we can see a central peak - showing that most of the different kmers were found at counts of ~ 120. These are kmers from single-copy homozygous alleles. To the left, a smaller peak at around half the coverage, showing kmers from heterozygous alleles (note that this peak gets higher than the main peak when heterozygosity is only ~ 1.2%). To the right, another smaller peak showing kmers at higher coverage, from repeat regions. Information from these three peaks provide a haploid genome length estimate of ~240,000 bp (note this is test data so smaller than whole plant genome size).

The output Summary file shows more detail:
* Genome unique length: from single copy homozygous and heterozygous alleles (under the main and left peak).
* Genome repeat length: from repeat copies (under the graph to the right of the main peak).
* Genome haploid length: unique length + repeat length

More about kmer counting: See [this tutorial](https://bioinformatics.uconn.edu/genome-size-estimation-tutorial/#)

For more about Meryl see {% cite Rhie2020 %}.

For more about Genomescope see {% cite Vurture2017 %}(Note: the supplementary information is very informative).

# Trim and filter reads

Using information from Data QC and kmer counting, we may want to trim and/or filter reads. The settings for trimming and filtering depend on many things, including:
* your aim (accuracy; contiguity)
* your data: type, error rate, read depth, lengths, quality (average, variation by position)
* the ploidy and heterozygosity of your sample
* choice of assembly tool (e.g. it may automatically deal with adapters, low qualities, etc.)

Because of all these factors, few specific recommendations are made here, but the workflow is provided and can be customised. If you are unsure how to start, use your test data to try different settings and see the effect on the resulting size and quality of the reads, and the downstream assembly contigs. Newer assemblers are often configured to work well with long-read data and in some cases, read trimming/filtering for long reads may be unnecessary.

Trimming and filtering reads:

!["An image of the concepts of trimming and filtering reads, with reads represented by bold bars; trimming represented by crosses over various locations in each read."](images/trimfilterreads.png){: width="800"}

## Trimming and filtering workflow


What it does: Trims and filters raw sequence reads according to specified settings.

Inputs:
* Long reads (format fastq)
* Short reads R1 and R2 (format fastq)

Outputs: Trimmed and filtered reads:
* fastp_filtered_long_reads.fastq.gz (But note: no trimming or filtering is on by default)
* fastp_filtered_R1.fastq.gz
* fastp_filtered_R2.fastq.gz

Tools used:  fastp (Note. The latest version (0.20.1) of fastp has an issue displaying plot results. Using version 0.19.5 here instead until this is rectified).

Input parameters: None required, but recommend removing the long reads from the workflow if not using any trimming/filtering settings.

Workflow parameter settings:
* Long reads: fastp settings:
  * These settings have been changed from the defaults (so that all filtering and trimming settings are now disabled).
  * Adapter trimming options: Disable adapter trimming: yes.
  * Filter options: Quality filtering options: Disable quality filtering: yes.
  * Filter options: Length filtering options: Disable length filtering: yes.
  * Read modification options: PolyG tail trimming: Disable.
  * Output options: output JSON report: yes
* Short reads: fastp settings:
  * adapter trimming (default setting: adapters are auto-detected)
  * quality filtering (default: phred quality 15), unqualified bases limit (default = 40%), number of Ns allowed in a read (default = 5)
  * length filtering (default length = min 15)
  * polyG tail trimming (default = on for NextSeq/NovaSeq data which is auto detected)
* Output options: output JSON report: yes

Options:
* Change any settings in fastp for any of the input reads.
* Adapter trimming: input the actual adapter sequences. (Alternative tool for long read adapter trimming: Porechop.)
* Trimming n bases from ends of reads if quality less than value x  (Alternative tool for trimming long reads: NanoFilt.)
* Discard post-trimmed reads if length is < x (e.g. for long reads, 1000 bp)
* Example filtering/trimming that you might do on long reads: remove adapters (can also be done with Porechop), trim bases from ends of the reads with low quality (can also be done with NanoFilt), after this can keep only reads of length x (e.g. 1000 bp)
* If not running any trimming/filtering on nanopore reads, could delete this step from the workflow entirely.


> <hands-on-title>Run the Trim and Filter Reads workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Trim_and_filter_reads.ga" title="Trim and Filter reads" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Trim and Filter Reads** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: long reads"*: the nanopore fastq.gz file
>    - {% icon param-file %} *"2: Input file: Illumina reads R1"*: the illumina R1 fastq.gz file
>    - {% icon param-file %} *"3: Input file: Illumina reads R2"*: the illumina R2 fastq.gz file
>    - *Note: the workflow parameter settings described above are already set, and don't need to be changed when running the workflow, unless you wish to.*
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

## Trim and filter reads: results

There are two fastp reports - one for the illumina reads and one for the nanopore reads. We have only processed illumina reads in this example. Look at the fastp illumina report. (Note: the title in the report refers to only one of the input read sets but the report is for both read sets. This is a known issue under investigation.)

Filtering results from fastp on short reads:

!["Image of a table of results from fastp, showing the percentage of reads that were low quality, had too many Ns, or too short."](images/fastp.png){: width="800"}

Here we can see that less than 0.5 % of the reads were discarded based on quality. If our read set had high enough coverage for downstream analyses, we might choose to apply a stricter quality filter.

## Summary of read data for genome assembly

> <question-title></question-title>
> 1. How many reads do we have now for our genome assembly? Is the read coverage high enough?
>
> > <solution-title></solution-title>
> > * Genome size: From kmer counting, the estimated genome size is ~ 240,000 bp  (this is only subsampled data; full data would likely suggest a size of 0.5 - 1 Gbp for a typical plant genome)
> > * Genome coverage (or depth):  total base pairs in the reads / base pairs in genome
> > * Short reads: From the fastp report after trimming and filtering short reads, there are 3.2 million reads, comprising 482 million base pairs
> > * Short read coverage:  = X2008
> > * Long reads: From the fastp report of the long reads (although no filtering and trimming performed) there are 85 thousand reads, comprising 761 million base pairs.  From the nanoplot tool we ran in the Data QC section , we know that the mean read length is almost 9,000 base pairs, and the longest read is > 140,000 base pairs.
> > * Long read coverage:  = X3170
> > These coverages are very high but are ok to use with tutorial data. With a typical full data set, coverage would be more in the order of X40 to X200.
> >
> {: .solution}
{: .question}

# Genome Assembly

Genome assembly means joining the reads up to make contiguous sections of the genome. A simplified way to imagine this is overlapping all the different sequencing reads to make a single length or contig, ideally one for each original chromosome. The output is a set of contigs and a graph showing how contigs are connected.

Extreme simplification of genome assembly:

!["Image of the simplified process of genome assembly, from a tree, to reads (represented by bold bars), to an assembly (represented by one long bold bar), to contigs (various long bold bars) and to an assembly graph."](images/assembly.png){: width="800"}

Genome assembly algorithms use different approaches to work with the complexities of large sequencing read data sets, large genomes, different sequencing error rates, and computational resources. Many use graph-based algorithms.  For more about genome assembly algorithms see [these tutorials by Ben Langmead](https://langmead-lab.org/teaching-materials/).

## Which assembly tool and approach to use?

Here, we will use the assembly tool called Flye to assemble the long reads. This is fast and deals well with the high error rate. Then, we will polish (correct) the assembly using information from the long reads (in their unassembled state), as well as the more accurate short Illumina reads.

There are many other approaches and combinations of using short and long reads, and the polishing steps. For example, the long reads can be polished before assembly (with themselves, or with short reads). This may increase accuracy of the assembly, but it may also introduce errors if similar sequences are "corrected" into an artificial consensus. Long reads are usually used in the assembly, but it is possible to assemble short reads and then scaffold these into longer contigs using information from long reads.

For more about the differences between current assembly and polishing tools see {% cite Chen2021%} and {% cite McCartney2021 %}.

## Assembly with Flye workflow


What it does: Assembles long reads with the tool Flye

Inputs: long reads (may be raw, or filtered, and/or corrected); fastq.gz format

Outputs:
* Flye assembly fasta.
* Fasta stats on assembly.fasta
* Assembly graph image from Bandage
* Bar chart of contig sizes
* Quast reports of genome assembly

Tools used:
* Flye
* Fasta statistics
* Bandage
* Bar chart
* Quast

Input parameters: None required, but recommend setting assembly mode to match input sequence type

Workflow steps:
* Long reads are assembled with Flye, using default tool settings. Note: the default setting for read type ("mode") is nanopore raw. Change this at runtime if required.
* Statistics are computed from the assembly.fasta file output, using Fasta Statistics and Quast (is genome large: Yes; distinguish contigs with more that 50% unaligned bases: no)
* The graphical fragment assembly file is visualized with the tool Bandage.
* Assembly information sent to bar chart to visualize contig sizes

Options
* See other Flye options.
* Use a different assembler (in a different workflow).
* Bandage image options - change size (max size is 32767), labels - add (e.g. node lengths). You can also install Bandage on your own computer and download the "graphical fragment assembly" file to view in greater detail.


> <hands-on-title>Run the Assembly with Flye workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Assembly_with_Flye.ga" title="Assembly with Flye" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Assembly with Flye** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: long reads"*: the fastp filtered long reads fastq.gz file
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}



## Assembly results

The assembled contigs are in the "Flye assembly on data X (consensus)" (X is a number that will vary depending on where it sits in your history).
Open the Quast tabular report to see the assembly statistics:

!["Image of the Quast tabular report, showing various assembly statistics."](images/quast.png){: width="800"}

There are 153 contigs, largest is ~246,000 bp, and total length almost 10 million bp. This is a fair bit longer than the estimated genome size from kmer counting (which was ~240,000 bp), but the difference is likely mainly due to idiosyncrasies of using a subsampled data set. The read coverage was likely <1, causing many kmers to have frequency of <1 and be classed as errors, rather than contributing to the genome size estimate.

Open the Quast HTML report, then click on "View in Icarus contig browser".  This is a way to visualize the contigs and their sizes:

!["Image of the Icarus contig browser, showing the set of contigs arranged from larges to smallest, represented by rectangles of different widths."](images/icarus.png){: width="800"}

View the Bandage image of the assembly graph:

!["Image of the bandage graph showing one multi-part set of joined contigs, some smaller one- or two- piece contigs, and many single contigs."](images/bandagegraphpartial.png){: width="800"}

As this is a subsampled data set, it is not surprising that most of the contigs are unjoined. The joined contigs at the top left are likely to be part of the mitochondrial genome as these reads were probably over-represented in our subsampled data set.

### What about centromeres and telomeres?

Some genomic areas such as centromeres, telomeres, and ribosomal DNA arrays, are much harder to assemble. These are long stretches of very similar repeats. With improved sequencing accuracy, length, and technologies (particularly long-range scaffolding), these may soon be much easier to assemble. The latest human genome assembly has a good demonstration of the techniques used for this.   See {% cite Nurk2021 %}, and in particular, Figure 2: Bandage graphs of the human genome chromosomes, with the grey shading showing centromeric regions.

### What about haplotigs?

Although our sample may be diploid, with pairs of chromosomes, the resulting assembly is often a haploid (or "collapsed") assembly. This is not the sequence of one of the chromosomes, but a mix of the two.

Some assemblers will produce extra contigs called haplotigs. These are parts of the assembly from heterozygous regions (that is, the sequence is relatively different between the chromosome pair). There are tools to remove haplotigs from the assembly if that is preferred.

For more on differences between collapsed, primary/alternate and partially-phased assemblies, with a great visual representation: see [Heng Li's Github page](http://lh3.github.io/2021/04/17/concepts-in-phased-assemblies)

For more on the phased assemblies, particularly for diploids or polyploids, see {% cite Garg2021 %}.

# Assembly polishing

We will polish the assembly using both the long reads and short reads. This process aligns the reads to the assembly contigs, and makes corrections to the contigs where warranted.  For more, see {% cite Aury2021 %}, particularly for a discussion about polishing diploid genomes.

Assembly polishing:

!["Image of the concept of assembly polishing, showing reads mapped to the assembly contigs."](images/polish.png){: width="800"}

## Polishing workflow

Workflow structure: The workflow includes two subworkflows:
* Racon polish with long reads, x 4
* Racon polish with illumina reads, x2

What it does: Polishes (corrects) an assembly, using long reads (with the tools Racon and Medaka) and short reads (with the tool Racon). (Note: medaka is only for nanopore reads, not PacBio reads).

Inputs:
* assembly to be polished:  assembly.fasta
* long reads - the same set used in the assembly (e.g. may be raw or filtered) fastq.gz format
* short reads, R1 only, in fastq.gz format

Outputs:
* Racon+Medaka+Racon polished_assembly. fasta
* Fasta statistics after each polishing tool

Tools used:
* Minimap2
* Racon
* Fasta statistics
* Medaka

Input parameters: None required, but recommended to set the Medaka model correctly (default = r941_min_high_g360). See drop down list for options.

Workflow steps for Part 1, Polish with long reads: using Racon
* Long reads and assembly contigs => Racon polishing (subworkflow):
* minimap2 : long reads are mapped to assembly => overlaps.paf.
* overaps, long reads, assembly => Racon => polished assembly 1
* using polished assembly 1 as input; repeat minimap2 + racon => polished assembly 2
* using polished assembly 2 as input, repeat minimap2 + racon => polished assembly 3
* using polished assembly 3 as input, repeat minimap2 + racon => polished assembly 4
* Racon long-read polished assembly => Fasta statistics
* Note: The Racon tool panel can be a bit confusing and is under review for improvement. Presently it requires sequences (= long reads), overlaps (= the paf file created by minimap2), and target sequences (= the contigs to be polished) as per "usage" described at this link [https://github.com/isovic/racon/blob/master/README.md](https://github.com/isovic/racon/blob/master/README.md)
* Note: Racon: the default setting for "output unpolished target sequences?" is No. This has been changed to Yes for all Racon steps in these polishing workflows.  This means that even if no polishes are made in some contigs, they will be part of the output fasta file.
* Note: the contigs output by Racon have new tags in their headers. For more on this see [this issue](https://github.com/isovic/racon/issues/85).

Workflow steps for Part 2, Polish with long reads: using Medaka
* Racon polished assembly + long reads => medaka polishing X1 => medaka polished assembly
* Medaka polished assembly => Fasta statistics

Workflow steps for Part 3, Polish with short reads: using Racon
* Short reads and Medaka polished assembly =>Racon polish (subworkflow):
* minimap2: short reads (R1 only) are mapped to the assembly => overlaps.paf. Minimap2 setting is for short reads.
* overlaps + short reads + assembly => Racon => polished assembly 1
* using polished assembly 1 as input; repeat minimap2 + racon => polished assembly 2
* Racon short-read polished assembly => Fasta statistics

Options:
* Change settings for Racon long read polishing if using PacBio reads:  The default profile setting for Racon long read polishing: minimap2 read mapping is "Oxford Nanopore read to reference mapping", which is specified as an input parameter to the whole Assembly polishing workflow, as text: map-ont. If you are not using nanopore reads and/or need a different setting, change this input. To see the other available settings, open the minimap2 tool, find "Select a profile of preset options", and click on the drop down menu. For each described option, there is a short text in brackets at the end (e.g. map-pb). This is the text to enter into the assembly polishing workflow at runtime instead of the default (map-ont).
* Other options: change the number of polishes (in Racon and/or Medaka). There are ways to assess how much improvement in assembly quality has occurred per polishing round (for example, the number of corrections made; the change in Busco score - see section "Genome quality assessment" for more on Busco).
* Option: change polishing settings for any of these tools. Note: for Racon - these will have to be changed within those subworkflows first. Then, in the main workflow, update the

> <hands-on-title>Run the Assembly polishing workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Assembly_polishing.ga" title="Assembly polishing workflow" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Assembly polishing** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Assembly to be polished"*: the assembly fasta file
>    - {% icon param-file %} *"2: long reads"*: the nanopore reads, the same set that was used for the assembly
>    - {% icon param-file %} *"3: Illumina reads R1"*: the illumina R1 fastq.gz file
>    - No other settings need to be changed, as they are set as described above, but recommended to set the Medaka model correctly (default = r941_min_high_g360). See drop down list for options.
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}


## Polishing results

The polished assembly is the final Racon file.

Look at the Fasta Statistics output files for the status of the assemblies after each polishing tool. From these, we can see that some polishes decrease or increase the size of the assembly. The final size is approximately 300,000 bp shorter than the original Flye assembly. (These numbers may vary slightly even if the same input data is used).

# Genome quality assessment

The polished genome is in 145 contigs with a total length of ~ 9.6 million base pairs.  We have some idea of how these contigs may be joined (or not) from the Bandage assembly graph.

> <question-title></question-title>
>
> 1. How good is the assembly?
>
> > <solution-title></solution-title>
> >
> > 1.  One measure is the N50, a number that indicates how large the contigs are (although, have the contigs been joined correctly)? Another measure is to see if expected gene sequences are found in the assembly, using a tool called BUSCO. For a discussion on these and other methods, see {% cite Wang2020%}.
> >
> {: .solution}
{: .question}

Here, we will use the BUSCO tool to annotate the genome and then assess whether expected genes are found. Note: this is a brief annotation only, not the full genome annotation that would typically be done following genome assembly.

For more about Busco see {% cite Simo2015 %}.

Genome assessment:

!["Image of genome assessment, showing contigs as bold bars, with one contig annotated (various sections labelled by coloured boxes), and then compared to an expected annotation."](images/assess.png){: width="800"}

We will also map the assembled contigs to a known reference genome using the tool Quast, to see how they align. For more about quast see [the manual](http://quast.sourceforge.net/docs/manual.html).

For more about the Icarus browser see {% cite Mikheenko2016 %}.

## Genome assessment workflow

What it does:
* Assesses the quality of the genome assembly: generate some statistics and determine if expected genes are present
* Align contigs to a reference genome.

Inputs:
* polished assembly
* reference_genome.fasta (e.g. of a closely-related species, if available).

Outputs
* Busco table of genes found
* Quast HTML report, and link to Icarus contigs browser, showing contigs aligned to a reference genome

Tools used:
* Busco
* Quast

Input parameters: None required

Workflow steps:
* Polished assembly => Busco:
* First: predict genes in the assembly: using Metaeuk
* Second: compare the set of predicted genes to the set of expected genes in a particular lineage. Default setting for lineage: Eukaryota
* Polished assembly and a reference genome => Quast:
* For the tutorial we will use the Arabidopsis genome. This is not closely related to our Eucalyptus species but will give an idea of how to use Quast.
* Contigs/scaffolds file: polished assembly
* Type of assembly: Genome
* Use a reference genome: Yes
* Reference genome: Arabidopsis genome
* Is the genome large (> 100Mbp)? Yes. (Our test data set won't be, but this will still give us some results).
* All other settings as defaults, except second last setting: Distinguish contigs with more than 50% unaligned bases as a separate group of contigs?: change to No

Options:
* **Gene prediction:**
* Change tool used by Busco to predict genes in the assembly: instead of Metaeuk, use Augustus (Metaeuk is meant to be faster; unsure which is better).
* select: Use Augustus; Use another predefined species model; then choose from the drop down list.
* select from a database of trained species models. Link to lists of species models: [https://github.com/Gaius-Augustus/Augustus/tree/master/config/species](https://github.com/Gaius-Augustus/Augustus/tree/master/config/species)
* Note: if using Augustus: it may fail if the input assembly is too small (e.g. a test-size data assembly). It can't do the training part properly.
* **Compare genes found to other lineage:**
* Busco has databases of lineages and their expected genes. Option to change lineage. Not all lineages are available - there is a mix of broader and narrower lineages.
* link to list of lineages [https://busco.ezlab.org/list_of_lineages.html](https://busco.ezlab.org/list_of_lineages.html)
* link to groups in taxonomic hierarchies [https://busco.ezlab.org/frames/euka.htm](https://busco.ezlab.org/frames/euka.htm)
* For example,  if you have a plant species from Fabales, you could set that as the lineage.
* The narrower the taxonomic group, the more total genes are expected.


> <hands-on-title>Run the Assess Genome Quality workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Assess_genome_quality.ga" title="Assess Genome Quality" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Assess Genome Quality** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: Polished assembly"*: the polished assembly fasta file from the previous polishing workflow
>    - {% icon param-file %} *"2: Input file: Reference genome"*: Arabidopsis.fasta
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}


## Assessment results

The output is a set of Quast and Busco reports.

Busco: As this is a test dataset, the assembly is small (~ 10 million base pairs, rather than ~ 1 billion base pairs), and likely missing most of the real genome (and genes). Thus, for this test case, we would expect most of the genes not to be found, but we can view the Busco results to see an example of how it works. Open the Busco short summary file: only 1 complete BUSCO has been found, out of 255 expected. You can re-run the workflow and change the lineage to "Embryophyta" to see that more BUSCOs are found (8 out of 1614).

Open the Quast HTML report, and at the top of this, click on "View in Icarus contig browser".  This shows how our assembly contigs have mapped to the reference genome. For this reference genome there are 5 chromosomes and two organelles. As this reference genome species is not closely related, not many contigs have mapped well. But we can see that some of them match the organelles (which is expected, as these reads are likely to be overrepresented in the test data). For the nuclear genome, there are some matches to parts of chromosome 2 and 3.

!["Image of quast results showing contig lengths."](images/quast2.png){: width="650"}

Click on the "mitochondria" to see how the assembly contigs align to the reference mitochondrial genome. Click the -5x button to zoom out to the full length.

!["Image of mitochondrial contigs."](images/mitocontigs.png){: width="650"}

# Combining workflows

We can combine these galaxy workflows into a single workflow. (See [this GTN tutorial]({% link topics/galaxy-interface/tutorials/workflow-editor/tutorial.md %}) for more information.)

## Combined assembly and polishing workflow

> <hands-on-title>Run the Combined Assembly and Polishing workflow</hands-on-title>
> * Find this workflow, enter the correct input files, and run.
> * What it does:  A workflow for genome assembly, containing subworkflows:
>   * Data QC
>   * Kmer counting
>   * Trim and filter reads
>   * Assembly with Flye
>   * Assembly polishing
>   * Assess genome quality
> * Inputs
>   * long reads and short reads in fastq format
>   * reference genome for Quast
> * Outputs
>   * Data information - QC, kmers
>   * Filtered, trimmed reads
>   * Genome assembly, assembly graph, stats
>   * Polished assembly, stats
>   * Quality metrics - Busco, Quast
> * Tools used: Sum of tools in each of the subworkflows
> * Input parameters: None required
> * Workflow steps: For detail see each subworkflow
> * Options:
>   * Omit some steps - e.g. Data QC and kmer counting
>   * Replace a module with one using a different tool - e.g. change assembly tool
{: .hands_on}

# Summary

In this tutorial we have assembled sequencing reads into contigs, using tools and workflows in Galaxy. Although we have used test data, these tools and workflows should work on real-sized eukaryotic data sets although please see our recommendations in the Introduction section.

A summary of the workflow steps and the main tools used:

!["A graphical summary of the steps in the assembly: quality control, kmer counting, trimming/filtering reads, assembly, polishing and assessment. A composite of all the earlier images in the tutorial."](images/summary.png){: width="650"}

We hope this has been useful for both learning about genome assembly concepts and as a customisable example for your own assembly data.


topic: admin
tutorial: terraform
title: Deploying a compute cluster in OpenStack via Terraform
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/terraform/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: []
tags_json: ["terraform", "deploying", "cloud"]
content_hash: bb6612e2474507861f0d1c2e2ffa4aa9
last_modified: 2026-05-15T22:28:10.068592
workflows_json: []
content:
# Overview


In this tutorial we will briefly cover what [Terraform](https://www.terraform.io) is and how you can leverage it for your needs. This will not make you an expert on Terraform but will give you the tools you need in order to maintain your cloud infrastructure as code.

This will be a very practical training with emphasis on looking at examples from modules and becoming self sufficient. This tutorial uses the OpenStack provider for Terraform. Other Cloud providers are available, but their invocation will be different from that which is described here.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Cattle vs. Pets

For those of you with more experience as System Administrators, you may have heard the term "cattle vs. pets."

Pets
:    These servers are managed by hand, with admins SSHing in to make changes directly on the machine, and no log of this. If a "pet" dies, everyone is sad.

Cattle
:    These servers are managed in bulk, you no longer own a beloved pet but hundreds of identical ones. You stop caring if an individual server lives or dies as you have many identical ones ready to take its place, and you can easily replace them.


# What is Terraform?

Terraform is a tool you can use to "sync" your infrastructure (VMs in Clouds, DNS records, etc.) with code you have written in configuration files. This is known as **infrastructure as code**.
Knowing that your infrastructure is exactly what you expect it to be can simplify your operations significantly. You can have confidence that if anything changes, any images crash or are accidentally deleted, that you can immediately re-build your infrastructure.

UseGalaxy.eu used Terraform to migrate easily between clouds, when our old cloud shut down and our new cloud was launched. We simply updated the credentials for connecting to the cloud, and ran Terraform. It recognised that none of our existing resources were there and recreated all of them. This made life incredibly easy for us, we knew that our infrastructure would be exactly correct relative to what it looked like in the previous cloud.

## What can be managed

Terraform supports [many providers](https://www.terraform.io/docs/providers/index.html), allowing you to easily manage resources no matter where they are located. Primarily this consists of resources like virtual machines and DNS records.

If you have VMs you are managing, whether for individual projects or for large scale infrastructure like UseGalaxy.eu, your work or research can be simpler and more reproducible if you are doing it with automation like Terraform.

## What should be managed

Support for certain databases and other software is available, but whether or not this is a good idea depends heavily on your workflow. UseGalaxy.eu found that managing databases and database users was not a good fit with our workflow. We launch VMs with terraform and then provision them with ansible. To then have a second step where we re-connect and provision the database with terraform was an awkward workflow, so we mostly let Ansible manage these resources.

Some groups use Ansible or Bash scripts in order to launch VMs. This can be a better workflow for certain cases. Launching VMs that manage themselves or are automatically scaled by some external process is not a good fit. Terraform expects the resource to be there the next time, with the same ID, with the same values, and will alert you if it isn't.

# Managing a Single VM

We will start small, by managing a single VM in our cloud account. Make sure you have your OpenStack credentials available.

> <tip-title>Obtaining credentials from OpenStack dashboard</tip-title>
>
> You can download the environment file with the credentials from the OpenStack dashboard.
>
> - Log in to the OpenStack dashboard, choose the project for which you want to download the OpenStack RC file, and click "Access & Security".
> - Click "Download OpenStack RC File" and save the file.
{: .tip}

<!-- TODO(hxr): add example DO config? -->

## Keypair

Terraform reads all files with the extension `.tf` in your current directory. Resources can be in a single file, or organised across several different files. We had the best experience by separating out logical groups of resources:

- DNS entries in a single file
- Security groups in separate `.tf` files
- Keypairs in a single file
- Servers/Instances in individual files.

> <hands-on-title>Setting up Terraform</hands-on-title>
>
> 1. [Install Terraform](https://www.terraform.io/downloads.html)
>
> 2. Create a new directory and go into it.
>
> 3. Create a `providers.tf` file with the following contents:
>
>    ```ini
>    provider "openstack" {}
>    ```
>
>    This specifies the configuration for the OpenStack plugin. You can either specify the configuration in the plugin, or it will automatically load the values from the normal OpenStack environment variable names.
>
> 4. Run `terraform init`
>
>    > <question-title></question-title>
>    >
>    > What did the output look like?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > If this is not the first time you've run it, you will see Terraform download any missing providers:
>    > >
>    > > ```
>    > > Initializing provider plugins...
>    > > - Checking for available provider plugins on https://releases.hashicorp.com...
>    > > - Downloading plugin for provider "openstack" (1.10.0)...
>    > >
>    > > The following providers do not have any version constraints in configuration,
>    > > so the latest version was installed.
>    > >
>    > > To prevent automatic upgrades to new major versions that may contain breaking
>    > > changes, it is recommended to add version = "..." constraints to the
>    > > corresponding provider blocks in configuration, with the constraint strings
>    > > suggested below.
>    > >
>    > > * provider.openstack: version = "~> 1.10"
>    > >
>    > > Terraform has been successfully initialized!
>    > >
>    > > You may now begin working with Terraform. Try running "terraform plan" to see
>    > > any changes that are required for your infrastructure. All Terraform commands
>    > > should now work.
>    > >
>    > > If you ever set or change modules or backend configuration for Terraform,
>    > > rerun this command to reinitialize your working directory. If you forget, other
>    > > commands will detect it and remind you to do so if necessary.
>    > > ```
>    > >
>    > {: .solution }
>    {: .question}
{:.hands_on}

This is the minimum to get started with Terraform, defining which providers we will use and any parameters for them. We have two options now:

1. We can load the OpenStack credentials as environment variables
2. We can code the OpenStack credentials in the `providers.tf` file

We recommend the first option, as often Terraform plans are [made publicly available](https://github.com/usegalaxy-eu/infrastructure) in order to collaborate on them, and this prevents accidentally committing your credentials to the git repository where.

We will start by managing your SSH keypair for the cloud as this is an easy thing to add.

> <comment-title>Your Operating System</comment-title>
>
> If you are on Windows and do not know your public key, please skip to the "Adding an Instance" section, as you probably do not have the tools installed to do this, and we cannot document how to do it. Instead you can simply reference the key by name later. We will point out this location.
>
{: .comment}

> <hands-on-title>Keypairs</hands-on-title>
>
> 1. Find the public key you will use for connecting to the your new VM. It is usually known as `id_rsa.pub`
>
>    > <comment-title>No public key</comment-title>
>    > If you can find the private key file (possibly a `cloud.pem` file you downloaded earlier from OpenStack), then you can find the public key by running the command:
>    >
>    > ```shell
>    > $ ssh-keygen -y -f /path/to/key.pem
>    > ```
>    {: .comment}
>
> 2. Create a new file, `main.tf` with the following structure. In `public_key`, write the complete value of your public key that you found in the first step.
>
>    ```ini
>    resource "openstack_compute_keypair_v2" "my-cloud-key" {
>      name       = "my-key"
>      public_key = "ssh-rsa AAAAB3Nz..."
>    }
>    ```
>
> 3. Run `terraform plan`
>
>    > <question-title></question-title>
>    >
>    > What does the output look like? Did it execute successfully?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > If you have not sourced your OpenStack credentials file, you will see something like the following:
>    > >
>    > > ```
>    > > Error: Error running plan: 1 error(s) occurred:
>    > >
>    > > * provider.openstack: One of 'auth_url' or 'cloud' must be specified
>    > > ```
>    > >
>    > > You should source your openstack credentials first. This is the file which has lines like:
>    > >
>    > > ```bash
>    > > export OS_AUTH_URL=https://...
>    > > export OS_PROJECT_ID=...
>    > > export OS_PROJECT_NAME="..."
>    > > export OS_USER_DOMAIN_NAME="Default"
>    > > ```
>    > >
>    > > You should run `source /path/to/file.sh` in your terminal, and then re-run `terraform plan`:
>    > >
>    > > ```
>    > > An execution plan has been generated and is shown below.
>    > > Resource actions are indicated with the following symbols:
>    > >   + create
>    > >
>    > > Terraform will perform the following actions:
>    > >
>    > >   + openstack_compute_keypair_v2.my-cloud-key
>    > >       id:          <computed>
>    > >       fingerprint: <computed>
>    > >       name:        "my-key"
>    > >       private_key: <computed>
>    > >       public_key:  "ssh-rsa AAAAB...."
>    > >       region:      <computed>
>    > >
>    > >
>    > > Plan: 1 to add, 0 to change, 0 to destroy.
>    > > ```
>    > >
>    > > If you see this, then everything ran successfully
>    > >
>    > {: .solution }
>    {: .question}
>
{: .hands_on }

Let's look at the output in detail:

```
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + openstack_compute_keypair_v2.my-cloud-key
      id:          <computed>
      fingerprint: <computed>
      name:        "my-key"
      private_key: <computed>
      public_key:  "ssh-rsa AAAAB..."
      region:      <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
```

Terraform informs us first about the different symbols used. Here it tells us that it will `+ create` a resource. Sometimes it will `- delete` or `~ update in-place`. Next it describes in detail what will be done and why. For creating a resource it does not give us much information, we will see more types of changes later.

Lastly it informs us that we did not save our plan. Terraform can maintain a concept of what the remote resource's state looks like between your `terraform plan` and `terraform apply` steps. This is a more advanced feature and will not be covered today.

> <hands-on-title>Applying our plan</hands-on-title>
>
> Now that you've reviewed the plan and everything looks good, we're ready to apply our changes.
>
> 1. Run `terraform apply`. Because we did not re-use our plan from the previous step, terraform will re-run the plan step first, and request your confirmation:
>
>    ```
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      + create
>
>    Terraform will perform the following actions:
>
>      + openstack_compute_keypair_v2.my-cloud-key
>          id:          <computed>
>          fingerprint: <computed>
>          name:        "my-key"
>          private_key: <computed>
>          public_key:  "" => "ssh-rsa AAAAB3..."
>          region:      <computed>
>
>
>    Plan: 1 to add, 0 to change, 0 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
> 2. Confirm that everything looks good and enter the value 'yes', and hit enter.
>
>    ```
>    openstack_compute_keypair_v2.my-cloud-key: Creating...
>      fingerprint: "" => "<computed>"
>      name:        "" => "my-key"
>      private_key: "" => "<computed>"
>      public_key:  "" => "ssh-rsa AAAAB3..."
>      region:      "" => "<computed>"
>    openstack_compute_keypair_v2.my-cloud-key: Creation complete after 3s (ID: my-key)
>
>    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
>    ```
>
>
>    > <comment-title>"Key pair 'my-key' already exists"</comment-title>
>    >
>    > If you see an error message that says:
>    >
>    > ```
>    > {"conflictingRequest": {"message": "Key pair 'my-key' already exists.", "code": 409}}
>    > ```
>    >
>    > Then you already have a keypair with this name. You should update the `name = "my-key"` to use a different name.
>    {: .comment}
>
{: .hands_on}

We should now have a keypair in our cloud!

## Adding an Instance

We now have:

- Terraform installed
- The OpenStack plugin initialized
- A keypair in OpenStack

> <comment-title>Doing this training outside of a training event</comment-title>
> If you are doing this training outside of an event, then you will likely need to do some additional steps specific to your Cloud:
>
> 1. Identify a small image flavor that you can use for your testing.
>
> 2. Upload this `raw` format image available from [UseGalaxy.eu](https://usegalaxy.eu/static/vgcn/denbi-centos7-j10-2e08aa4bfa33-master.raw) to your OpenStack
>
> You will need to do both of these things before you can proceed.
{: .comment}

You are now ready to launch an instance!

> <hands-on-title>Launching an Instance</hands-on-title>
>
> 1. Open your `main.tf` in your text editor and add the following.
>
>    > <warning-title>Correct image/flavor/network/security_group names</warning-title>
>    > The documentation below notes some specific values for the `image_name`, `flavor_name`, `security_groups`, and `network` properties. These *may not be correct* for your training, instead your instructor will provide these values to you.
>    {: .warning}
>
>    ```ini
>    resource "openstack_compute_instance_v2" "test" {
>      name            = "test-vm"
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>      flavor_name     = "m1.tiny"
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>      security_groups = ["default"]
>
>      network {
>        name = "public"
>      }
>    }
>    ```
>
>    There are some important things to note, we reproduce the above configuration but with comments:
>
>    "openstack_compute_instance_v2" is the 'type' of a resource, 'test' is the name of this specific resource.
>    ```
>    resource "openstack_compute_instance_v2" "test" {
>    ```
>
>    This will become the server name
>    ```
>      name            = "test-vm"
>    ```
>
>    The image which will be launched
>    ```
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    ```
>
>    Which flavor
>    ```
>      flavor_name     = "m1.tiny"
>    ```
>
>    This uses Terraform's knowledge of the
>    `openstack_compute_keypair_v2.my-cloud-key` resource which you
>    previously described. Then it accesses the `.name` attribute.
>    This allow your to update the name at any time, and still have your
>    resource definitions be correct.
>    ```
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    ```
>
>    The security group to apply. This is a comma separated list, but
>    `default` should work for most OpenStack clouds.
>    ```
>      security_groups = ["default"]
>    ```
>
>    A network that is accessible to you.
>    ```
>      network {
>        name = "public"
>      }
>    ```
>
> 2. Run `terraform apply`. Running the `plan` step is not necessary, it is just useful to see what changes will be applied without starting the apply process.
>
>    Note that first terraform "refreshes" the state of the remote resources that it already manages, before checking what changes need to be made.
>
>    ```
>    $ terraform apply
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      + create
>
>    Terraform will perform the following actions:
>
>      + openstack_compute_instance_v2.test
>          id:                         <computed>
>          access_ip_v4:               <computed>
>          access_ip_v6:               <computed>
>          all_metadata.%:             <computed>
>          availability_zone:          <computed>
>          flavor_id:                  <computed>
>          flavor_name:                "m1.tiny"
>          force_delete:               "false"
>          image_id:                   <computed>
>          image_name:                 "denbi-centos7-j10-2e08aa4bfa33-master"
>          key_pair:                   "my-key"
>          name:                       "test-vm"
>          network.#:                  "1"
>          network.0.access_network:   "false"
>          network.0.fixed_ip_v4:      <computed>
>          network.0.fixed_ip_v6:      <computed>
>          network.0.floating_ip:      <computed>
>          network.0.mac:              <computed>
>          network.0.name:             "public"
>          network.0.port:             <computed>
>          network.0.uuid:             <computed>
>          power_state:                "active"
>          region:                     <computed>
>          security_groups.#:          "1"
>          security_groups.3814588639: "default"
>          stop_before_destroy:        "false"
>
>
>    Plan: 1 to add, 0 to change, 0 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
> 3. If everything looks good, enter 'yes'.
>
>    ```
>    openstack_compute_instance_v2.test: Creating...
>      access_ip_v4:               "" => "<computed>"
>      access_ip_v6:               "" => "<computed>"
>      all_metadata.%:             "" => "<computed>"
>      availability_zone:          "" => "<computed>"
>      flavor_id:                  "" => "<computed>"
>      flavor_name:                "" => "m1.tiny"
>      force_delete:               "" => "false"
>      image_id:                   "" => "<computed>"
>      image_name:                 "" => "denbi-centos7-j10-2e08aa4bfa33-master"
>      key_pair:                   "" => "my-key"
>      name:                       "" => "test-vm"
>      network.#:                  "" => "1"
>      network.0.access_network:   "" => "false"
>      network.0.fixed_ip_v4:      "" => "<computed>"
>      network.0.fixed_ip_v6:      "" => "<computed>"
>      network.0.floating_ip:      "" => "<computed>"
>      network.0.mac:              "" => "<computed>"
>      network.0.name:             "" => "public"
>      network.0.port:             "" => "<computed>"
>      network.0.uuid:             "" => "<computed>"
>      power_state:                "" => "active"
>      region:                     "" => "<computed>"
>      security_groups.#:          "" => "1"
>      security_groups.3814588639: "" => "default"
>      stop_before_destroy:        "" => "false"
>    openstack_compute_instance_v2.test: Still creating... (10s elapsed)
>    openstack_compute_instance_v2.test: Still creating... (20s elapsed)
>    openstack_compute_instance_v2.test: Creation complete after 30s (ID: 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa)
>
>    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
>    ```
{: .hands_on}

You now have a running instance! We do not know the IP address so we cannot login yet. You can obtain that from the OpenStack dashboard, or via the `terraform show` command


> <hands-on-title>`terraform show` and logging in</hands-on-title>
>
> 1. Run `terraform show`
>
>    The values in the following output will not match yours.
>
>    ```
>    openstack_compute_instance_v2.test:
>      id = 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa
>      access_ip_v4 = 192.52.32.231
>      access_ip_v6 =
>      all_metadata.% = 0
>      availability_zone = nova
>      flavor_id = a9a58bcf-29a3-47f4-b4c3-e71e40127833
>      flavor_name = m1.tiny
>      force_delete = false
>      image_id = 922ae172-275b-4e6a-bb9d-c7ace52fc8d4
>      image_name = denbi-centos7-j10-2e08aa4bfa33-master
>      key_pair = my-key
>      name = test-vm
>      network.# = 1
>      network.0.access_network = false
>      network.0.fixed_ip_v4 = 192.52.32.231
>      network.0.fixed_ip_v6 =
>      network.0.floating_ip =
>      network.0.mac = fa:16:3e:e6:cd:6c
>      network.0.name = public
>      network.0.port =
>      network.0.uuid = 6a68b40c-356f-4d4c-95e9-e5c72855fb35
>      power_state = active
>      region = Freiburg
>      security_groups.# = 1
>      security_groups.3814588639 = default
>      stop_before_destroy = false
>    openstack_compute_keypair_v2.my-cloud-key:
>      id = my-key
>      fingerprint = 4a:72:2c:7e:d2:b7:f2:c4:5a:9e:bb:ca:c7:8f:d7:a3
>      name = my-key
>      private_key =
>      public_key = ssh-rsa AAAAB3...
>      region = Freiburg
>    ```
>
> 2. Here we can obtain the IP address. In the above output, it is `192.52.32.231`
>
> 3. Login with `ssh`: `ssh centos@<your-ip-address>`. You should see something like the following:
>
>    ```
>    The authenticity of host '192.52.32.231 (192.52.32.231)' can't be established.
>    ECDSA key fingerprint is SHA256:oEHgotonwT3a47K0kLdsNVd/QqctBjDK7F829J0VnwQ.
>    Are you sure you want to continue connecting (yes/no)? yes
>    Warning: Permanently added '192.52.32.231' (ECDSA) to the list of known hosts.
>          _        _   _ ____ _____
>         | |      | \ | |  _ \_   _| ██      ██
>       __| | ___  |  \| | |_) || |     ██████
>      / _' |/ _ \ | . ' |  _ < | |     ██████
>     | (_| |  __/_| |\  | |_) || |_  ██  ██  ██
>      \__,_|\___(_)_| \_|____/_____| ██  ██  ██
>    ====================================================
>    Hostname..........: test-vm.novalocal
>    Release...........: CentOS Linux release 7.5.1804 (Core)
>    Build.............: de.NBI Generic denbi-centos7-j10-2e08aa4bfa33-master
>    Build Date........: 2018-10-24T14:28:38Z
>    Current user......: centos
>    Load..............: 0.00 0.07 0.05 2/148 1644
>    Uptime............: up 5 minutes
>    ====================================================
>    [centos@test-vm ~]$
>    ```
>
{: .hands_on}

With this, you're done. You've launched a single VM with Terraform. The image we are using comes with:

- HTCondor
- Docker
- Singularity
- Conda
- CVMFS connected to Galaxyproject's biological databases and Singularity containers

This is very similar to the standard image that UseGalaxy.eu uses for all of its compute nodes.

# Managing a Cluster of VMs

When you are done playing around with the individual VM, we're ready to explore launching an entire cluster.

- We will launch a VM to act as our HTCondor central manager
- We will launch a VM to act as an NFS server for our "cluster"
- We will launch two execution nodes to process jobs

We use a very similar setup for our production cloud cluster that powers UseGalaxy.eu's compute and training infrastructure.

## Configuration

We will start by setting up the new nodes and launching them as a test:

> <hands-on-title>Adding configuration for other nodes</hands-on-title>
>
> 1. Update the name of the instance you already have, `test` to `name = "central-manager"`. This will be our scheduler central manager.
>
>    ```
>    resource "openstack_compute_instance_v2" "test" {
>      name            = "central-manager"
>      ...
>    }
>    ```
>
> 2. Add the NFS server. It is mostly the same configuration you've done before, just with a different `name` and resource name.
>
>    ```
>    resource "openstack_compute_instance_v2" "nfs" {
>      name            = "nfs-server"
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>      flavor_name     = "m1.tiny"
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>      security_groups = ["default"]
>
>      network {
>        name = "public"
>      }
>    }
>    ```
>
> 3. Lastly, we'll add the execution nodes. Here we will launch two servers at once by using the `count` parameter. We can add `count` in the main portion of a Terraform resource definition and it will create that many copies of that same resource. In the case where we wish to be able to distinguish between them, we can use `${count.index}` in the name or another field to distinguish them.
>
>    ```ini
>    resource "openstack_compute_instance_v2" "exec" {
>      name            = "exec-${count.index}"
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>      flavor_name     = "m1.tiny"
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>      security_groups = ["default"]
>      count           = 2
>
>      network {
>        name = "public"
>      }
>    }
>    ```
>
> 4. Run `terraform apply`
>
>    You will see something like the following:
>
>    ```
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>    openstack_compute_instance_v2.test: Refreshing state... (ID: 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      + create
>      ~ update in-place
>
>    Terraform will perform the following actions:
>
>      + openstack_compute_instance_v2.exec[0]
>          id:                         <computed>
>          access_ip_v4:               <computed>
>          access_ip_v6:               <computed>
>          all_metadata.%:             <computed>
>          availability_zone:          <computed>
>          ...
>
>      ~ openstack_compute_instance_v2.test
>          name:                       "test-vm" => "central-manager"
>
>
>    Plan: 3 to add, 1 to change, 0 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
>    This time, Terraform is able to modify the `openstack_compute_instance_v2.test` resource in place, it can just update the name of the VM. It will also create the other three requested resources.
>
> 5. Enter `yes` and terraform will make the requested changes
>
> 6. It should complete successfully
>
>    > <question-title></question-title>
>    >
>    > What did the output look like?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > openstack_compute_instance_v2.nfs: Creation complete after 2m55s (ID: 421f0899-f2c1-43c9-8bbe-9b0415c8f4f4)
>    > > openstack_compute_instance_v2.exec[0]: Creation complete after 2m57s (ID: 1ae02775-a0ff-47a5-907f-513190a8548e)
>    > > openstack_compute_instance_v2.exec[1]: Creation complete after 2m59s (ID: d339ebeb-4288-498c-9923-42fce32a3808)
>    > >
>    > > Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
>    > > ```
>    > {: .solution}
>    {: .question}
{: .hands_on}

## `cloud-init`

These VMs that we have launched currently are all absolutely identical, except for the host name. There is no specialisation. They all do nothing. We will fix that and make these machines have individual roles by booting them with metadata attached to each instance.

[`cloud-init`](https://cloud-init.io/) is used for this process, it allows for injecting files and executing commands as part of the boot process.


> <details-title>Other actions</details-title>
> cloud-init allows for many more actions to be executed as well, you can read about them in [the documentation](https://cloudinit.readthedocs.io/en/latest/).
{: .details}

The `cloud-init` configuration is a YAML file. Here we can see an example of how it is used, we have a YAML array at the top level. Inside, a hash of `content`, `owner`, `path`, and `permissions`.

```yaml
#cloud-config
write_files:
- content: |
    CONDOR_HOST = 127.0.0.1
    ALLOW_WRITE = *
    ALLOW_READ = $(ALLOW_WRITE)
    ALLOW_NEGOTIATOR = $(ALLOW_WRITE)
    DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD
    FILESYSTEM_DOMAIN = denbi
    UID_DOMAIN = denbi
    TRUST_UID_DOMAIN = True
    SOFT_UID_DOMAIN = True
  owner: root:root
  path: /etc/condor/condor_config.local
  permissions: '0644'
```

This will create a file with the value from `content`, owned by root/group root, in `/etc/condor/condor_config.local`. So let's re-structure our `main.tf` file to have some configuration:

> <hands-on-title>cloud-init</hands-on-title>
>
> 1. Edit your central-manager server and add a block like the following at the end.
>
>    ```
>    user_data = <<-EOF
>      #cloud-config
>      write_files:
>      - content: |
>          CONDOR_HOST = localhost
>          ALLOW_WRITE = *
>          ALLOW_READ = $(ALLOW_WRITE)
>          ALLOW_NEGOTIATOR = $(ALLOW_WRITE)
>          DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD
>          FILESYSTEM_DOMAIN = terraform-training
>          UID_DOMAIN = terraform-training
>          TRUST_UID_DOMAIN = True
>          SOFT_UID_DOMAIN = True
>        owner: root:root
>        path: /etc/condor/condor_config.local
>        permissions: '0644'
>      - content: |
>          /data           /etc/auto.data          nfsvers=3
>        owner: root:root
>        path: /etc/auto.master.d/data.autofs
>        permissions: '0644'
>      - content: |
>          share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>        owner: root:root
>        path: /etc/auto.data
>        permissions: '0644'
>    EOF
>    ```
>
>    Near the end we see an interesting thing, `${openstack_compute_instance_v2.nfs.access_ip_v4}`, this will template the `user_data` that is passed to `cloud-init`, with the IP address of the NFS server.
>
> 2. Edit your NFS server and add a user data block like:
>
>    ```
>    user_data = <<-EOF
>      #cloud-config
>      write_files:
>      - content: |
>          /data/share *(rw,sync)
>        owner: root:root
>        path: /etc/exports
>        permissions: '0644'
>      runcmd:
>       - [ mkdir, -p, /data/share ]
>       - [ chown, "centos:centos", -R, /data/share ]
>       - [ systemctl, enable, nfs-server ]
>       - [ systemctl, start, nfs-server ]
>       - [ exportfs, -avr ]
>    EOF
>    ```
>
>    This will:
>
>    1. Make the `/data/share` directory
>    2. Set permissions on it
>    3. Launch the NFS service
>    4. Export `/data/share` to the network
>
> 3. Edit the `exec` node and add a user data block like:
>
>    ```
>    user_data = <<-EOF
>      #cloud-config
>      write_files:
>      - content: |
>          CONDOR_HOST = ${openstack_compute_instance_v2.test.access_ip_v4}
>          ALLOW_WRITE = *
>          ALLOW_READ = $(ALLOW_WRITE)
>          ALLOW_ADMINISTRATOR = *
>          ALLOW_NEGOTIATOR = $(ALLOW_ADMINISTRATOR)
>          ALLOW_CONFIG = $(ALLOW_ADMINISTRATOR)
>          ALLOW_DAEMON = $(ALLOW_ADMINISTRATOR)
>          ALLOW_OWNER = $(ALLOW_ADMINISTRATOR)
>          ALLOW_CLIENT = *
>          DAEMON_LIST = MASTER, SCHEDD, STARTD
>          FILESYSTEM_DOMAIN = terraform-training
>          UID_DOMAIN = terraform-training
>          TRUST_UID_DOMAIN = True
>          SOFT_UID_DOMAIN = True
>          # run with partitionable slots
>          CLAIM_PARTITIONABLE_LEFTOVERS = True
>          NUM_SLOTS = 1
>          NUM_SLOTS_TYPE_1 = 1
>          SLOT_TYPE_1 = 100%
>          SLOT_TYPE_1_PARTITIONABLE = True
>          ALLOW_PSLOT_PREEMPTION = False
>          STARTD.PROPORTIONAL_SWAP_ASSIGNMENT = True
>        owner: root:root
>        path: /etc/condor/condor_config.local
>        permissions: '0644'
>      - content: |
>          /data           /etc/auto.data          nfsvers=3
>        owner: root:root
>        path: /etc/auto.master.d/data.autofs
>        permissions: '0644'
>      - content: |
>          share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>        owner: root:root
>        path: /etc/auto.data
>        permissions: '0644'
>    EOF
>    ```
> 4. Compare your final configuration against ours:
>
>    > <solution-title>Final Configuration</solution-title>
>    > ```
>    > resource "openstack_compute_keypair_v2" "my-cloud-key" {
>    >   name       = "my-key"
>    >   public_key = "ssh-rsa AAAAB3..."
>    > }
>    >
>    > resource "openstack_compute_instance_v2" "test" {
>    >   name            = "central-manager"
>    >   image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    >   flavor_name     = "m1.tiny"
>    >   key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    >   security_groups = ["default"]
>    >
>    >   network {
>    >     name = "public"
>    >   }
>    >
>    >   user_data = <<-EOF
>    >     #cloud-config
>    >     write_files:
>    >     - content: |
>    >         CONDOR_HOST = localhost
>    >         ALLOW_WRITE = *
>    >         ALLOW_READ = $(ALLOW_WRITE)
>    >         ALLOW_NEGOTIATOR = $(ALLOW_WRITE)
>    >         DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD
>    >         FILESYSTEM_DOMAIN = terraform-training
>    >         UID_DOMAIN = terraform-training
>    >         TRUST_UID_DOMAIN = True
>    >         SOFT_UID_DOMAIN = True
>    >       owner: root:root
>    >       path: /etc/condor/condor_config.local
>    >       permissions: '0644'
>    >     - content: |
>    >         /data           /etc/auto.data          nfsvers=3
>    >       owner: root:root
>    >       path: /etc/auto.master.d/data.autofs
>    >       permissions: '0644'
>    >     - content: |
>    >         share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>    >       owner: root:root
>    >       path: /etc/auto.data
>    >       permissions: '0644'
>    >   EOF
>    > }
>    >
>    > resource "openstack_compute_instance_v2" "nfs" {
>    >   name            = "nfs-server"
>    >   image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    >   flavor_name     = "m1.tiny"
>    >   key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    >   security_groups = ["default"]
>    >
>    >   network {
>    >     name = "public"
>    >   }
>    >
>    >   user_data = <<-EOF
>    >     #cloud-config
>    >     write_files:
>    >     - content: |
>    >         /data/share *(rw,sync)
>    >       owner: root:root
>    >       path: /etc/exports
>    >       permissions: '0644'
>    >     runcmd:
>    >      - [ mkdir, -p, /data/share ]
>    >      - [ chown, "centos:centos", -R, /data/share ]
>    >      - [ systemctl, enable, nfs-server ]
>    >      - [ systemctl, start, nfs-server ]
>    >      - [ exportfs, -avr ]
>    >   EOF
>    > }
>    >
>    > resource "openstack_compute_instance_v2" "exec" {
>    >   name            = "exec-${count.index}"
>    >   image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    >   flavor_name     = "m1.tiny"
>    >   key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    >   security_groups = ["default"]
>    >   count           = 2
>    >
>    >   network {
>    >     name = "public"
>    >   }
>    >
>    >   user_data = <<-EOF
>    >     #cloud-config
>    >     write_files:
>    >     - content: |
>    >         CONDOR_HOST = ${openstack_compute_instance_v2.test.access_ip_v4}
>    >         ALLOW_WRITE = *
>    >         ALLOW_READ = $(ALLOW_WRITE)
>    >         ALLOW_ADMINISTRATOR = *
>    >         ALLOW_NEGOTIATOR = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_CONFIG = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_DAEMON = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_OWNER = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_CLIENT = *
>    >         DAEMON_LIST = MASTER, SCHEDD, STARTD
>    >         FILESYSTEM_DOMAIN = terraform-training
>    >         UID_DOMAIN = terraform-training
>    >         TRUST_UID_DOMAIN = True
>    >         SOFT_UID_DOMAIN = True
>    >         # run with partitionable slots
>    >         CLAIM_PARTITIONABLE_LEFTOVERS = True
>    >         NUM_SLOTS = 1
>    >         NUM_SLOTS_TYPE_1 = 1
>    >         SLOT_TYPE_1 = 100%
>    >         SLOT_TYPE_1_PARTITIONABLE = True
>    >         ALLOW_PSLOT_PREEMPTION = False
>    >         STARTD.PROPORTIONAL_SWAP_ASSIGNMENT = True
>    >       owner: root:root
>    >       path: /etc/condor/condor_config.local
>    >       permissions: '0644'
>    >     - content: |
>    >         /data           /etc/auto.data          nfsvers=3
>    >       owner: root:root
>    >       path: /etc/auto.master.d/data.autofs
>    >       permissions: '0644'
>    >     - content: |
>    >         share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>    >       owner: root:root
>    >       path: /etc/auto.data
>    >       permissions: '0644'
>    >   EOF
>    > }
>    > ```
>    >
>    {: .solution}
>
> 5. Run `terraform apply`
>
>    It will look slightly different this time:
>
>    ```
>    $ terraform apply
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>    openstack_compute_instance_v2.nfs: Refreshing state... (ID: 421f0899-f2c1-43c9-8bbe-9b0415c8f4f4)
>    openstack_compute_instance_v2.test: Refreshing state... (ID: 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa)
>    openstack_compute_instance_v2.exec[0]: Refreshing state... (ID: 1ae02775-a0ff-47a5-907f-513190a8548e)
>    openstack_compute_instance_v2.exec[1]: Refreshing state... (ID: d339ebeb-4288-498c-9923-42fce32a3808)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>    -/+ destroy and then create replacement
>
>    Terraform will perform the following actions:
>
>    -/+ openstack_compute_instance_v2.exec[0] (new resource required)
>          id:                         "1ae02775-a0ff-47a5-907f-513190a8548e" => <computed> (forces new resource)
>          access_ip_v4:               "192.52.32.255" => <computed>
>          ...
>          user_data:                  "" => "07d27d50dbdf12a5647d9fd12a3510861d32203c" (forces new resource)
>
>    ...
>    Plan: 4 to add, 0 to change, 4 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
>    Here Terraform has detected that the userdata has changed. It cannot change this dynamically at runtime, only at boot time. So it decides that it must destroy and then replace that resource.
>
{:.hands_on}

## Login / HTCondor / A Test Job

We now have a running cluster! Let's log in

> <hands-on-title>Cluster Usage</hands-on-title>
>
> 1. Now that we have a cluster in the cloud, let's login. Look through the output of `terraform show` to find your `central-manager` server and its IP address.
>
> 2. `ssh centos@<your manager ip>`
>
> 3. Run `condor_status` which will show you the status of your cluster.
>
>    After your central manager VM booted, the executor nodes booted as well. Then, using the IP address of the central manager, the executors contacted that machine, and advertised their availability to run jobs.
>
>    > <question-title></question-title>
>    >
>    > What does the output look like? How many executor nodes do you see?
>    >
>    > > <solution-title></solution-title>
>    > > ```
>    > > [centos@central-manager share]$ condor_status
>    > > Name                   OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime
>    > >
>    > > slot1@exec-0.novalocal LINUX      X86_64 Unclaimed Idle      0.150  991  0+00:15:28
>    > > slot1@exec-1.novalocal LINUX      X86_64 Unclaimed Idle      0.000  991  0+00:15:32
>    > >
>    > >                      Machines Owner Claimed Unclaimed Matched Preempting  Drain
>    > >
>    > >         X86_64/LINUX        2     0       0         2       0          0      0
>    > >
>    > >                Total        2     0       0         2       0          0      0
>    > > ```
>    > {: .solution}
>    {: .question}
>
> 4. Change into the `/data/share/`
>
> 5. Create a file, `test.job` with the following contents:
>
>    ```
>    Universe = vanilla
>    Executable = /data/share/test.sh
>    Log = test.$(process).log
>    Output = test.$(process).out
>    Error = test.$(process).err
>    request_cpus = 1
>    Queue 10
>    ```
>
> 6. Create a file `test.sh` with the following:
>
>    ```
>    #!/bin/bash
>    echo "$(hostname)"
>    sleep 1
>    ```
> 7. Run `chmod +x test.sh`
>
> 8. Run `condor_submit test.job`
>
> 9. (Quickly) Run `condor_q` to see the queue status. You can invoke this repeatedly to watch the queue update.
>
> 10. Run `cat *.out` and you should see the hostnames where the jobs were run
>
{:.hands_on}

# Infrastructure Graph

Terraform has the ability to produce a graph showing the relationship of resources in your infrastructure. We will produce a graphic for our current terraform plan:

> <hands-on-title>Infrastructure Graph</hands-on-title>
>
> 1. Run:
>
>    ```
>    terraform graph | dot -Tpng > graph.png
>    ```
>
>    You may need the `graphviz` package installed in order to produce the graph
>
> 2. Open up `graph.png` in an image viewer
>
>    > <question-title></question-title>
>    >
>    > What did the output look like?
>    >
>    > > <solution-title></solution-title>
>    > > ![A simple graph](../../images/graph.png "A simple graph showing the structure of the infrastructure in this lesson")
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

This is a very simple resrouce graph:

![A simple graph](../../images/graph.png "A simple graph showing the structure of the infrastructure in this lesson")

The dependencies follow the arrows, the NFS depends on the keypair being setup. The `test` machine (central manager) depends on the NFS server being available (and having an IP), the exec nodes depend on the central manager server. This is a somewhat simplified view, there are more dependencies which are not shown for simplicity (e.g. the exec nodes depend on NFS)

If you have variables this can produce a more complex dependency graph:

![A more complex graph showing variables](../../images/graph2.png "A more complex graph showing variables")

Once you develop complex infrastructure, these graphics become less useful.

# Tearing Everything Down

> <hands-on-title>Cleanup</hands-on-title>
>
> 1. Simple delete your `main.tf` file (or rename it without the `.tf` extension)
>
> 2. Run `terraform apply`. Terraform will see that the resource is no longer part of your code and remove it.
>
>    ```
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>    openstack_compute_instance_v2.nfs: Refreshing state... (ID: 0b58d613-48f2-49c4-9d8a-3623d17f1c93)
>    openstack_compute_instance_v2.test: Refreshing state... (ID: e433f772-6f17-4609-aa61-053f7602533f)
>    openstack_compute_instance_v2.exec[0]: Refreshing state... (ID: 034a4206-b0ad-4d0e-9c75-27d3966f99ac)
>    openstack_compute_instance_v2.exec[1]: Refreshing state... (ID: fe576776-397e-4f1f-a8e7-4a9203764567)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      - destroy
>
>    Terraform will perform the following actions:
>
>      - openstack_compute_instance_v2.exec[0]
>
>      - openstack_compute_instance_v2.exec[1]
>
>      - openstack_compute_instance_v2.nfs
>
>      - openstack_compute_instance_v2.test
>
>      - openstack_compute_keypair_v2.my-cloud-key
>
>
>    Plan: 0 to add, 0 to change, 5 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
{: .hands_on}

## Destroy time provisioners

Sometimes, immediately terminating the VM is not ideal behaviour. Especially if you're managing a cluster like HTCondor which can gracefully terminate jobs, you might want to permit that. Terraform provides 'destroy-time provisioners', code snippets that are run before the VM is destroyed. If they return successfully, destruction continues. If they exit with an error code, then destruction is halted. We can write a simple one for HTCondor like so:

```bash
set -ex
condor_drain $(hostname) || true;
my_ip=$(/sbin/ifconfig eth0 | grep 'inet ' | awk '{print $2}')
# Find slots which have an address associated with us.
slots=$(condor_status -autoformat Name State MyAddress | grep "<${my_ip}?9618" | wc -l)
# If there are more than one slots, leave it.
if (( slots > 1 )); then
    exit 1;
else
    # Otherwise, poweroff.
    /usr/sbin/condor_off -graceful
    exit 0;
fi
```

Which can be provided to your VM like:

```
provisioner "remote-exec" {
  when = "destroy"

  scripts = [
    "prepare-restart.sh",
  ]

  connection {
    type        = "ssh"
    user        = "centos"
    private_key = "${file("~/.ssh/id_rsa")}"
  }
}
```

Terraform will SSH in with those credentials, copy over the script in `prepare-restart.sh`, and execute it.

# UseGalaxy.eu's Terraform Usage

All of [our virtual infrastructure](https://github.com/usegalaxy-eu/infrastructure/) is managed, publicly, with Terraform. We hope that this can inspire others and give people ideas of the sort of things they can accomplish with Terraform. If you have questions over the way we have done certain things, feel free to file an issue and ask us!


topic: admin
tutorial: external-auth
title: External Authentication
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["authentication"]
content_hash: 193ad1de042ab871692f4c091f64c22a
last_modified: 2026-05-15T22:28:10.029916
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/external-auth/tutorial.html

content:
# Overview


For this exercise we will use a basic password file method for authenticating - this is probably not a very useful method in production, but it demonstrates how the proxy server can be configured to provide the correct header to Galaxy, and how Galaxy integrates with upstream authentication providers. This same method can be used with NGINX and Apache modules for CAS or SAML authentication.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Configuring Authentication

> <hands-on-title>Configuring everything</hands-on-title>
>
> 1. Edit the galaxy.j2 template file in the templates/nginx directory and update the main location block defined for serving galaxy. Add the parameters:
>      - `auth_basic galaxy;`
>      - `auth_basic_user_file /etc/nginx/passwd;`
>      - `proxy_set_header HTTP_REMOTE_USER $remote_user;`
>      - `proxy_set_header HTTP_GX_SECRET SOME_SECRET_STRING;`
>
>    It should look like:
>
>    ```diff
>    @@ -14,6 +14,10 @@
>    +        auth_basic           galaxy;
>    +        auth_basic_user_file /etc/nginx/passwd;
>    +        proxy_set_header          HTTP_REMOTE_USER $remote_user;
>    +        proxy_set_header          HTTP_GX_SECRET SOME_SECRET_STRING;
>         }
>     
>         # Static files can be more efficiently served by Nginx. Why send the
>    ```
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>    > <tip-title>Running this tutorial <i>just</i> for Reports?</tip-title>
>    > Add the `auth_basic` and `auth_basic_user_file` lines to your `location /reports/`
>    {: .tip}
>
>
>    `auth_basic` enables validation of username and password using the "HTTP Basic Authentication" protocol. Its value `galaxy` is used as a realm name to be displayed to the user when prompting for credentials.
>    `auth_basic_user_file` specifies the file that keeps usernames and passwords.
>    `proxy_set_header` adds `HTTP_REMOTE_USER` to the special variables passed by nginx to Galaxy, with value `$remote_user`, which is a nginx embedded variable containing the username supplied with the Basic authentication.
>
>    `GX_SECRET` is added as a header for security purposes, to prevent any other users on the system impersonating nginx and sending requests to Galaxy. NGINX and other webservers like Apache will strip any user-sent `REMOTE_USER` headers, as that header defines the authenticated user. If you can talk directly to Galaxy (e.g. via curl) and provide the `REMOTE_USER` header, you can impersonate any other use. While having Galaxy listen on `127.0.0.1` prevents any requests from outside of the system reaching Galaxy, anyone on the system can still send requests to that port. Here you can choose to switch to a unix socket with permissions only permitting Galaxy and Nginx to connect. `GX_SECRET` adds additional security as it needs to match `remote_user_secret` in your galaxy configutation.
>
>    > <tip-title>Proxy bypass</tip-title>
>    > Users can bypass the authentication only if they can talk directly to the gunicorn processes (if you have socket/http: 0.0.0.0, or if it is directly responsible for serving galaxy, and there is no proxy.)
>    > This can happen mostly when some users have command line access to the Galaxy server, which is considered a bad practice.
>    {: .tip}
>
> 2. Add a pre_task using the [`pip`](https://docs.ansible.com/ansible/2.9/modules/pip_module.html) module which installs the library `passlib`, which is required for `htpasswd`.
>
>    Add a pre_task using the [`htpasswd`](https://docs.ansible.com/ansible/2.4/htpasswd_module.html) module which sets up a password file in `/etc/nginx/passwd`, with owner and group set to root, and a name and password, and a mode of 0640.
>    > <question-title></question-title>
>    >
>    > How does your final configuration look?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```diff
>    > > @@ -7,6 +7,15 @@
>    > >          name: galaxy
>    > >          state: restarted
>    > >    pre_tasks:
>    > > +    - pip:
>    > > +        name: passlib
>    > > +    - htpasswd:
>    > > +        path: /etc/nginx/passwd
>    > > +        name: user1        # Pick a username
>    > > +        password: changeme # and a password
>    > > +        owner: www-data    # nginx on centos
>    > > +        group: root
>    > > +        mode: 0640
>    > >      - name: Install Dependencies
>    > >        package:
>    > >          name: ['git', 'make', 'python3-psycopg2', 'virtualenv']
>    > > ```
>    > {: .solution }
>    >
>    {: .question}
>
> 3. Galaxy needs to be instructed to expect authentication to come from the upstream proxy. In order to do this, set the following two options in your Galaxy group variables:
>
>    ```yaml
>    ...
>    galaxy_config:
>      galaxy:
>        ...
>        use_remote_user: true
>        remote_user_maildomain: "{% raw %}{{ inventory_hostname }}{% endraw %}"
>        remote_user_secret: SOME_SECRET_STRING
>    ```
>
>    > <tip-title>Running this tutorial <i>just</i> for Reports?</tip-title>
>    > You don't need to make the above changes for Galaxy then.
>    {: .tip}
>
>    Set the `remote_user_maildomain` option to the appropriate domain name for your site.
>
> 4. Run the playbook
>
{: .hands_on}

> <comment-title>Access denied</comment-title>
>
> If you see this message, it is because nginx is not correctly sending the `REMOTE_USER` or the `GX_SECRET` values.
>
> ![access denied message](../../images/access_denied.png)
>
{: .comment}


# Testing

You should now be presented with a password dialog when attempting to load the Galaxy UI.

> <hands-on-title>Testing</hands-on-title>
>
> 1. Log in using the username and password you provided when creating the `passwd` file. If your username and the value of `remote_user_maildomain` match an existing user, you will be logged in to that account. If not, a new account will be created for that user.
>
> 2. Click on the "User" menu at the top, to see how the username appears.
>
> 3. Note that some user features are not available when remote user support is enabled.
>
>    Try logging out by selecting **User** -> **Logout**. You will discover that when returning to the user interface, you are still logged in. This is because Galaxy has no way of logging you out of the proxy's authentication system. Instead, you should set `remote_user_logout_href` in `galaxy.yml` to point to the URL of your authentication system's logout page.
>
{: .hands_on}

# API access

If you wish your Galaxy to be accessible to command line clients (e.g. bioblend, blend4j, parsec), you will need to add an exception for authentication on the API. Galaxy will still be secure and protected, but non-browser access will be permitted with an API key.

```
location /api/ {
    satisfy any;
    allow all;
}
```

> <tip-title>Notification of Registration</tip-title>
> There is no built-in way to be notified if users are registered, with external authentication or built-in. However, you could automate this easily. There is a [gxadmin](https://github.com/usegalaxy-eu/gxadmin) command we use called `gxadmin query latest-users` which Björn uses often. Other sites have other methods, e.g. Nicola's [cron script](https://gist.github.com/nsoranzo/f023e26aa60024ef6a7e3a3fe5fb2e4f) which runs daily on his server, to add new users to a group according to their email domain name.
{: .tip}

# Reverting

We don't want to leave Galaxy this way for the rest of our workshop.

> <hands-on-title>Reverting the changes</hands-on-title>
>
> 1. Edit your group variables file and comment out:
>
>    - the NGINX changes
>    - `use_remote_user: true`
>
> 2. Run the playbook
>
{: .hands_on}


topic: admin
tutorial: data-library
title: Data Libraries
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["ansible", "storage", "git-gat"]
content_hash: 7f5556452f0b68b7058eb112da9c3b49
last_modified: 2026-05-27T09:01:47.635922
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/data-library/tutorial.html

content:
Data libraries are a great way to provide structured repositories of data to
everyone on your server, or to a subset of your users. Datasets in libraries do
not count against user quotas, so they are commonly used to provide the results
of a sequencing run or similar project to a group of users on the servers.

![screenshot of data libraries]({{site.baseurl}}/topics/admin/images/data/data-library-screenshot.png)


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="data-library" %}

# Setup

Before we can import local data, we need to configure Galaxy to permit this. Additionally we will setup an example data library which we can use for demonstrative purposes.

> <hands-on-title>Setting up Data Libraries</hands-on-title>
>
> 1. We will add a pre-task to clone [a data repository](https://github.com/usegalaxy-eu/libraries-training-repo) into your machine. We will use this as the source for a library dataset.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -29,6 +29,9 @@
>           package:
>             name: ['tmpreaper']
>           when: ansible_os_family == 'Debian'
>    +    - git:
>    +        repo: 'https://github.com/usegalaxy-eu/libraries-training-repo'
>    +        dest: /libraries/
>       roles:
>         - galaxyproject.tusd
>         - usegalaxy_eu.apptainer
>    {% endraw %}
>    ```
>    {: data-commit="Add the git repository to the pre-tasks"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Take a minute to explore the [folders in our sample library.](https://github.com/usegalaxy-eu/libraries-training-repo). These will be important when we start loading data.
>
> 4. Edit the file `group_vars/galaxyservers.yml` and set the following variables:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -96,6 +96,9 @@ galaxy_config:
>         # Tool Dependencies
>         dependency_resolvers_config_file: "{{ galaxy_config_dir }}/dependency_resolvers_conf.xml"
>         container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>    +    # Data Library Directories
>    +    library_import_dir: /libraries/admin
>    +    user_library_import_dir: /libraries/user
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Configure the library import directories"}
>
>    Note that the `/libraries/admin` will refer to a folder within the libraries-training-repo that we cloned in the pre-task, and likewise for `/libraries/user`
>
> 5. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

# Importing Data

There are multiple options for importing data from your server, we'll go through all of your choices below. But first, let's take a quick look at the example library structure we've provided.

> > <code-in-title>Bash</code-in-title>
> > ```bash
> > tree /libraries
> > ```
> {: .code-in}
>
> > <code-out-title>Bash</code-out-title>
> > ```
> > /libraries/
> > ├── admin
> > │   └── admin-wildtype.fna
> > ├── example-library.yaml
> > ├── README.md
> > └── user
> >     ├── admin@example.com
> >     │   └── user-wildtype.fna
> >     └── admin@example.org
> >         └── user-wildtype.fna
> >
> > 4 directories, 5 files
> > ```
> {: .code-out}
{: .code-2col}

> <tip-title>admin@example.org</tip-title>
> Note that in the user directories, admin@example.com and example.org are used, if you've used a different email address for *your* admin user, you'll need to copy one of these directories.
{: .tip}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

We have a directory named `admin`, which will be available to all admin users (we set `library_import_dir: /libraries/admin` earlier.)

Additionally we have a `user` directory, below the user directory are more directories with the user's email as they directory key. Data can be placed in here, and it will become accessible to those users (we set `user_library_import_dir: /libraries/user` for this.)

![An add datasets dropdown menu in galaxy showing the options from history, from user directory, and under admins only, from import directory](../../images/data/import-menu.png)

## from History

This is by far the easiest and most convenient option for small datasets, or datasets that are just already in a history

![A select box is shown listing files in the user's history](../../images/data/import-history.png)

You can easily select multiple files and quickly import them.

## from User Directory

If user directories are configured, as we did at the beginning of this tutorial, then users will be able to import any files under their personal directory. This can be used for a wide variety of setups, e.g. providing the output of sequencing machines to users. This can point to the same directory structure that's used by the FTP service, if you want your users to be able to import files directly from FTP.

![Import popup with a list of files with one file, user-wildtype.fna, and buttons for configuring import behaviour.](../../images/data/import-user.png)

This will enable the option for everyone, any unprivileged user with a folder in the `user_library_import_dir` directory *and* permissions on a data library can import things from their import directory.

> <tip-title>Do you not see a directory here?</tip-title>
> We pre-created a directory for someone registered as `admin@example.org`, but if you are logged in with an account registered with a different email, you'll see nothing.
>
> You can fix this by going into /libraries/user and `cp -Rv admin@example.org whatever-email-you-used`
> Then it should appear in Galaxy.
{: .tip}

## from import Directory (Admins only)

Similarly to the user import directories, there is the option to have an admin only import area. If one admin is responsible for creating the data libraries and importing data this can be a good option.

![Same as the previous image, import popup listing options and one file, admin-wildtype.fna](../../images/data/import-admin.png)

An important feature of data libraries importing is the ability to "link files instead of copying". If you know your data will not be deleted (e.g. sequencing data sitting on an NFS), then you can use this option to further decrease the amount of storage your Galaxy server requires. The datasets can be used as if they were normal datasets, imported by users and analysed, but not imported into the Galaxy data storage area.

# Automatically Populating a Data Library

If your data is accessible via URL, you can write a yaml file to import and setup the data library automatically.
We've included this file in the example data libraries repository we cloned at the beginning:

> <code-in-title>Bash</code-in-title>
> ```bash
> cat /libraries/example-library.yaml
> ```
{: .code-in}

> <code-out-title>Bash</code-out-title>
> ```yaml
> ---
> destination:
>   type: library
>   name: Mouse sequencing project
>   description: some data
>   synopsis: samples collected from somewhere
> items:
> - url: https://zenodo.org/records/582600/files/wildtype.fna
>   src: url
>   ext: fasta
>   info: https://doi.org/10.5281/zenodo.582600
> - name: A directory
>   description: Exome sequencing means that all protein-coding genes in a genome are
>   items:
>   - url: https://zenodo.org/records/582600/files/mutant_R1.fastq
>     src: url
>     ext: fastqsanger
>     info: https://doi.org/10.5281/zenodo.582600
>   - url: https://zenodo.org/records/582600/files/mutant_R2.fastq
>     src: url
>     ext: fastqsanger
>     info: https://doi.org/10.5281/zenodo.582600
> ```
{: .code-out}

Let's try setting that up in our Galaxy!

{% snippet faqs/galaxy/preferences_admin_api_key.md admin=true %}

> <hands-on-title>importing data library yaml.</hands-on-title>
>
> 1. re-activate the virtualenv you created for the [ephemeris tool management tutorial]({% link topics/admin/tutorials/tool-management/tutorial.md %}).
>
>    > <code-in-title>input: bash</code-in-title>
>    > ```
>    > . ~/ephemeris_venv/bin/activate
>    > ```
>    {: .code-in}
>
>    > <tip-title>missing?</tip-title>
>    >
>    > then you might need to re-run the steps:
>    >
>    > ```bash
>    > python3 -m venv ~/ephemeris_venv
>    > . ~/ephemeris_venv/bin/activate
>    > pip install ephemeris
>    > ```
>    {: .tip}
>
> 2. we'll use the `setup-data-libraries` command to install the data in this yaml file into a library in our galaxy.
>
>    > <code-in-title>input: bash</code-in-title>
>    > ```bash
>    > setup-data-libraries -g https://galaxy.example.org -a <api-key> --training -i /libraries/example-library.yaml --legacy
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>output</code-out-title>
>    > ```
>    > library name: mouse sequencing project
>    > ```
>    {: .code-out}
>
> 3. this command is safe to re-run. for example if you update the yaml, it will simply report that the library exists before ensuring all files exist in their appropriate location:
>
>    ```bash
>    library name: mouse sequencing project
>    library already exists! id: f597429621d6eb2b
>    ```
>
{: .hands_on}

![Screenshot of data libraries, we're in a library folder named "Mouse sequencing project" and a directory and single file are shown. The file has an ugly URL as its name](../../images/data/imported.png)

That's it! You should be able to [see your newly created data library in your Galaxy.](https://my.gat.galaxy.training/?path=/libraries)

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/library-permissions.md %}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=8 %}

Note that we've used some special flags here, `--training` and `--legacy`. Training sets some defaults that make sense for the GTN (mostly around library descriptions / etc.)

## `--legacy`

This enables the use of legacy APIs even for newer Galaxies that should have a batch upload API enabled. Unfortunately the new batch upload API is not able to update existing data libraries in place and will always create new libraries with the same name. So currently `--legacy` is quite useful for maintaining a YAML file, and running `setup-data-libraries` regularly whenever that file has updates.

But this comes with the downside that the entire URL is used for the filename.

# GTN Data

This process scales quite well. Galaxy Europe, in their mission to support all of the GTN materials on their server setup [a shared-data repository](https://github.com/usegalaxy-eu/shared-data), a single giant YAML file with all of the data from the GTN tutorials. This was then expanded to other UseGalaxy.* servers to provide synced training libraries for all of the major servers.

Do you want the training data synced on your server? If so join the shared data repository! If you provide us with a non-admin API key and a data library to upload data into, we can sync this automatically.

# Using Data from Libraries

Users can now conveniently use datasets in libraries when they are running analyses, without having to import them first.

The tool form provides a button on the right of every dataset selector which allows users to "Browse datasets"

![tool form with the folder icon to the right of the dataset selector shown, the button with tooltip "Browse datasets" is highlighted](../../images/data/select-from-lib0.png)

Users can then choose from datasets in their history, or browse through the data libraries

![Popup with "Data Libraries", datasets from histories, and an upload button](../../images/data/select-from-lib1.png)

Here users can see every file accessible to them in the data library to begin analysing.

![Same popup as previous image, but the file listing now shows the contents of the data library we created during the hands-on step with Mouse data](../../images/data/select-from-lib2.png)

{% snippet topics/admin/faqs/git-gat-path.md tutorial="data-library" %}


topic: admin
tutorial: ftp
title: Enable upload via FTP
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/ftp/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["data", "git-gat"]
content_hash: 9969c40913585d5f10229af33df731f7
last_modified: 2026-05-27T09:01:47.636239
workflows_json: []
content:
This tutorial will guide you to setup an {FTP} server so galaxy users can use it to upload large files. Indeed, as written on the [galaxy community hub](https://galaxyproject.org/ftp-upload/), uploading data directly from the browser can be unreliable and cumbersome. FTP will allow users to monitor the upload status as well as resume interrupted transfers.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="ftp" %}

# FTP

{FTP} is a very old and reliable communication protocol that has been around since 1971 {% cite rfc114 %}. It requires a server (here our galaxy server) and a client (user's computer). The FTP server requires to have at least 2 ports accessible from outside (one for the commands and one for the transfer). Usually the port for the command is 21.

FTP supports two different modes: active, and passive. Active mode requires that the user's computer be reachable from the internet, which in the age of {NAT} and firewalls is usually unusable. So passive mode is the most commonly used. In passive mode, a client connects to the FTP server, and requests a channel for sending files. The server responds with an IP and port, from its range of "Passive Ports".

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> Your VM or wherever you are installing Galaxy needs to have the following ports available:
>
> - 21
> - Some high range of ports not used by another service, e.g. 56k-60k
>
> You need to know which ports are open so you can use them for the transfer (PassivePorts). In this training we assume that 56k to 60k are open.
>
> Which ports precisely is not important, and these numbers can differ between sites.
{: .comment}

# FTP and Galaxy

To allow your user to upload via FTP, you will need to:

- configure Galaxy to know where the files are uploaded.
- install a FTP server
- allow your FTP server to read Galaxy's database so users can use their credential and upload in the good directory.

For secure transmission we will use SSL/TLS (FTPS), not the SSH File Transfer Protocol (SFTP) as the Galaxy users don't correspond to users on the machine.

## Installing and Configuring

Luckily for us, there is an ansible role written by the Galaxy Project for this purpose. It will install proftpd. Firstly, we need to install the role and then update our playbook for using it.

If the terms "Ansible", "role" and "playbook" mean nothing to you, please checkout [the Ansible introduction slides]({% link topics/admin/tutorials/ansible/slides.html %}) and [the Ansible introduction tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %})

{% snippet topics/admin/faqs/ansible_local.md %}

> <hands-on-title>Setting up ftp upload with Ansible</hands-on-title>
>
> 1. In your playbook directory, add the `galaxyproject.proftpd` role to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -56,3 +56,6 @@
>     # Sentry
>     - name: mvdbeek.sentry_selfhosted
>       src: https://github.com/mvdbeek/ansible-role-sentry/archive/main.tar.gz
>    +# Our FTP Server
>    +- src: galaxyproject.proftpd
>    +  version: 0.3.1
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. As in this training we are using certbot, we will ask for a private key for proftpd. Add the following line to your `group_vars/galaxyserver.yml` file:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -210,11 +210,13 @@ certbot_environment: staging
>     certbot_well_known_root: /srv/nginx/_well-known_root
>     certbot_share_key_users:
>       - www-data
>    +  - proftpd
>     certbot_share_key_ids:
>       - "999:999"
>     certbot_post_renewal: |
>         systemctl restart nginx || true
>         docker restart rabbit_hole || true
>    +    systemctl restart proftpd || true
>     certbot_domains:
>      - "{{ inventory_hostname }}"
>      - "{{ sentry_domain }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add proftpd in certbot"}
>
>    This will make a copy of the current letsencrypt key available as `/etc/ssl/user/privkey-proftpd.pem`, and automatically restart proftpd every time the key is updated.
>
> 4. We will configure Galaxy to enable ftp file upload. Add the following line to your `group_vars/galaxyserver.yml` file in the galaxy_config/galaxy section:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -122,6 +122,9 @@ galaxy_config:
>         sentry_dsn: "{{ vault_galaxy_sentry_dsn }}"
>         sentry_traces_sample_rate: 0.5
>         error_report_file: "{{ galaxy_config_dir }}/error_reports_file.yml"
>    +    # FTP
>    +    ftp_upload_dir: /data/uploads
>    +    ftp_upload_site: "{{ inventory_hostname }}"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add ftp vars in galaxy"}
>
> To check the other options for setting up ftp in Galaxy, please check the [Galaxy configuration documentation](https://docs.galaxyproject.org/en/master/admin/galaxy_options.html?highlight=ftp_upload_site#ftp-upload-dir).
>
> 5. Then we will set the different variables for proftpd. Add the following lines to your `group_vars/galaxyserver.yml` file. Please replace the PassivePorts below with the range of ports that are appropriate for your machine!
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -368,3 +368,24 @@ telegraf_plugins_extra:
>     tiaas_dir: /srv/tiaas
>     tiaas_admin_user: admin
>     tiaas_admin_pass: changeme
>    +
>    +# Proftpd:
>    +proftpd_galaxy_auth: yes
>    +galaxy_ftp_upload_dir: "{{ galaxy_config.galaxy.ftp_upload_dir }}"
>    +proftpd_display_connect: |
>    +  {{ inventory_hostname }} FTP server
>    +
>    +  Unauthorized access is prohibited
>    +proftpd_create_ftp_upload_dir: yes
>    +proftpd_options:
>    +  - User: galaxy
>    +  - Group: galaxy
>    +  - Port: 21
>    +proftpd_sql_db: galaxy@/var/run/postgresql
>    +proftpd_sql_user: galaxy
>    +proftpd_conf_ssl_certificate: /etc/ssl/certs/cert.pem
>    +proftpd_conf_ssl_certificate_key: /etc/ssl/user/privkey-proftpd.pem
>    +proftpd_global_options:
>    +  - PassivePorts: 56000 60000
>    +proftpd_use_mod_tls_shmcache: false
>    +proftpd_tls_options: NoSessionReuseRequired
>    {% endraw %}
>    ```
>    {: data-commit="Add proftpd variables"}
>
>    Here is a description of the set variables:
>
>    | Variable                           | Description                                                                                                  |
>    | ----------                         | -------------                                                                                                |
>    | `proftpd_galaxy_auth`              | Attempt to authenticate users against a Galaxy database.                                                     |
>    | `galaxy_ftp_upload_dir`            | Path to the Galaxy FTP upload directory, should match `ftp_upload_dir` in your Galaxy config.                |
>    | `proftpd_display_connect`          | Message to display when users connect to the FTP server. This should be the message, not the path to a file. |
>    | `proftpd_create_ftp_upload_dir`    | Whether to allow the role to create this with owner `galaxy_user`.                                           |
>    | `proftpd_options`                  | Any option for proftpd, we will just set up the user and group of the `galaxy_user`.                         |
>    | `proftpd_sql_db`                   | Database name to connect to for authentication info.                                                         |
>    | `proftpd_sql_user`                 | (default: the value of galaxy_user): Value of the username parameter to SQLConnectInfo.                      |
>    | `proftpd_conf_ssl_certificate`     | Path on the remote host where the SSL certificate file is.                                                   |
>    | `proftpd_conf_ssl_certificate_key` | Path on the remote host where the SSL private key file is.                                                   |
>    | `proftpd_global_options`           | Set arbitrary options in the <Global> context. We set here the PassivePorts range.                           |
>    | `proftpd_use_mod_tls_shmcache`     | By default proftpd uses `mod_tls_shmcache` which is not installed on the server so we just disable it.       |
>    | `proftpd_tls_options`              | Additional options for tls. We will use `NoSessionReuseRequired`                                             |
>
>    > <tip-title>Why NoSessionReuseRequired?</tip-title>
>    > `mod_tls` only accepts SSL/TLS data connections that reuse the SSL session of the control connection, as a security measure. Unfortunately, there are some clients (e.g. curl/Filezilla) which do not reuse SSL sessions.
>    > To relax the requirement that the SSL session from the control connection be reused for data connections we set `NoSessionReuseRequired`.
>    {: .tip}
>
> 6. Add the new role to the list of roles under the `roles` key in your playbook, `galaxy.yml`:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -45,6 +45,7 @@
>         - geerlingguy.redis
>         - usegalaxy_eu.flower
>         - galaxyproject.nginx
>    +    - galaxyproject.proftpd
>         - geerlingguy.docker
>         - usegalaxy_eu.rabbitmqserver
>         - galaxyproject.tiaas2
>    {% endraw %}
>    ```
>    {: data-commit="Add role to playbook"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in }
>
{: .hands_on}

Congratulations, you've set up FTP for Galaxy.

## Check it works

> <hands-on-title>Checking proftpd from the server</hands-on-title>
>
> 1. SSH into your machine
>
> 2. Check the active status of proftpd by `systemctl status proftpd`.
>
> 3. Check the port has been correctly attributed by `sudo lsof -i -P -n`.
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > You should see all the ports used by the server. What interests us is the line with proftpd.
>    > > You should see TCP *:21 (LISTEN).
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
> 4. Check the directory `/data/uploads/` has been created and is empty.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > sudo tree /data/uploads/
>    > ```
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

> <hands-on-title>Checking galaxy detected the ftp possibility</hands-on-title>
>
> 1. Open your galaxy in a browser.
>
> 2. Log in with a user (FTP is only possible for authenticated sessions).
>
> 3. Click on the upload button. You should now see on the bottom "Choose FTP files"
>
> 4. Click on the Choose FTP files button. You should see a message "Your FTP directory does not contain any files."
>
{: .hands_on}

It's working!

> <hands-on-title>Upload your first file</hands-on-title>
>
> There are three options for uploading files, you can choose whichever is easiest for you.
>
> 1. **FileZilla**
>
>    1. Follow the [tutorial](https://galaxyproject.org/ftp-upload/) to upload a file.
>    2. You will have a message which ask you to approve the certificate, approve it.
>
> 2. **lftp**
>
>    You can use locally lftp to test the ftp.
>
>    1. Install lftp with `sudo apt-get install lftp`.
>    2. Add the public certificate to the list of known certificates (only for LetsEncrypt Staging Certificates!):
>       > <code-in-title>Bash</code-in-title>
>       > ```
>       > mkdir .lftp
>       > echo "set ssl:ca-file \"/etc/ssl/certs/cert.pem\"" > .lftp/rc
>       > ```
>       {: .code-in}
>
>    3. Connect to the server with for example the admin account:
>       > <code-in-title>Bash</code-in-title>
>       > ```
>       > lftp admin@example.org@$HOSTNAME
>       > ```
>       {: .code-in}
>
>    4. Enter the password of the admin@example.org galaxy user.
>    5. Put a random file:
>
>       `put /srv/galaxy/server/CITATION`
>
>    6. Check it is there with `ls`.
>    7. Leave lftp with `quit`.
>    {: .tip}
>
> 3. **Curl**
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > curl -T {"/srv/galaxy/server/CITATION"} ftp://localhost --user admin@example.org:password --ssl -k
>    > ```
>    > Here `-T` says to upload a file, `--ssl` ensures that the FTP connection is SSL/TLS encrypted, and `-k` ignores any certificate issues as the hostname `localhost` will not match the certificate we have.
>    {: .code-in}
>
{: .hands_on}

> <hands-on-title>Check where the file has been uploaded</hands-on-title>
>
> 1. SSH into your machine
>
> 4. Check the directory `/uploads/`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > sudo tree /uploads/
>    > ```
>    {: .code-in}
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > As I uploaded a file called `CITATION` with the admin@example.org user I see:
>    > > ```
>    > > /uploads/
>    > > └── admin@example.org
>    > >     └── CITATION
>    > > ```
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

> <hands-on-title>Use it in galaxy</hands-on-title>
>
> 1. Open your galaxy in a browser.
>
> 2. Log in with the user you used to upload the file.
>
> 3. Click on the upload button.
>
> 4. Click on the Choose FTP files button. You should see your file.
>
> 5. Click on it and click on Start to launch the upload. It should go to your history as a new dataset.
>
> 6. Click again on Choose FTP files button. Your file has disappeared. By default, the files are removed from the FTP at import.
>
>    > <tip-title>You want to change this behaviour?</tip-title>
>    > You just need to add `ftp_upload_purge: false` to the galaxy_config/galaxy variables (next to `ftp_upload_dir`).
>    {: .tip}
>
{: .hands_on}

Congratulations! Let your users know this is an option, many of them will prefer to start large uploads from an FTP client.

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=16 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="ftp" %}


topic: admin
tutorial: gxadmin
title: Galaxy Monitoring with gxadmin
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/gxadmin/tutorial.html
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["monitoring", "ansible", "git-gat"]
content_hash: 440e42fbd4c25e4227d77b2b7ce1f09d
last_modified: 2026-05-15T22:28:10.033937
workflows_json: []
content:
We will just briefly cover the features available in `gxadmin`, there are lots of queries that may or may not be useful for your Galaxy instance and you will have to read the documentation before using them.

It started life as a small shell script that Helena wrote because she couldn't remember what [Gravity](https://github.com/galaxyproject/gravity) was called or where it could be found. Some of the functions needed for things like swapping zerglings are still included in gxadmin but are highly specific to UseGalaxy.eu and not generally useful.

Since then it became the home for "all of the SQL queries we [galaxy admins] run regularly." @gtn:hexylena and @gtn:natefoo often shared SQL queries with each other in private chats, but this wasn't helpful to the admin community at large, so they decided to put them all in `gxadmin` and make it as easy to install as possible. We are continually trying to make this tool more generic and generally useful, if you notice something that's missing or broken, or have a new query you want to run, just [let us know](https://github.com/usegalaxy-eu/gxadmin/issues/new).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="gxadmin" %}

## Installing gxadmin

It's simple to install gxadmin. Here's how you do it, if you haven't done it already.

> <hands-on-title>Installing gxadmin with Ansible</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -28,3 +28,5 @@
>       version: 6.1.0
>     - name: usegalaxy_eu.rabbitmqserver
>       version: 1.4.1
>    +- src: galaxyproject.gxadmin
>    +  version: 0.0.12
>    {% endraw %}
>    ```
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add the role to your playbook:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -34,3 +34,4 @@
>         - galaxyproject.nginx
>         - galaxyproject.tusd
>         - galaxyproject.cvmfs
>    +    - galaxyproject.gxadmin
>    {% endraw %}
>    ```
>
> 4. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

With that, `gxadmin` should be installed! Now, test it out:

> <hands-on-title>Test out gxadmin</hands-on-title>
>
> 1. Run `gxadmin` as the galaxy user and list recently registered users:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sudo -u galaxy gxadmin query latest-users
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    >  id |          create_time          | disk_usage | username |       email        | groups | active
>    > ----+-------------------------------+------------+----------+--------------------+--------+--------
>    >   1 | 2021-06-09 12:25:59.299651+00 | 218 kB     | admin    | admin@example.org  |        | f
>    > (1 rows)
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

## Configuration

If `psql` runs without any additional arguments, and permits you to access your galaxy database then you do not need to do any more configuration for gxadmin.
Otherwise, you may need to set some of the [PostgreSQL environment variables](https://github.com/usegalaxy-eu/gxadmin#postgres)

## Overview

`gxadmin` has several categories of commands, each with different focuses. This is not a technically meaningful separation, it is just done to make the interface easier for end users.

Category      | Keyword             | Purpose
---           | --                  | --
Configuration | `config`            | Commands relating to galaxy's configuration files like XML validation.
Filters       | `filter`            | Transforming streams of text.
Galaxy Admin  | `galaxy`            | Miscellaneous galaxy related commands like a cleanup wrapper.
uWSGI         | `uwsgi`             | If you're using [systemd for Galaxy](https://github.com/usegalaxy-eu/ansible-galaxy-systemd/) and a handler/zergling setup, then this lets you manage your handlers and zerglings.
DB Queries    | `{csv,tsv,i,}query` | Queries against the database which return tabular output.
Report        | `report`            | Queries which return more complex and structured markdown reports.
Mutations     | `mutate`            | These are like queries, except they mutate the database. All other queries are read-only.
Meta          | `meta`              | More miscellaneous commands, and a built-in updating function.

## Admin Favourite Queries

**@gtn:slugger70's favourite**: `gxadmin query old-histories`. He contributed this function to find old histories, as their instance has a 90 day limit on histories, anything older than that might be automatically removed. This helps their group identify any histories that can be purged in order to save space. Running this on UseGalaxy.eu, we have some truly ancient histories, and maybe could benefit from a similar policy.

> <code-in-title></code-in-title>
> ```
> gxadmin query old-histories
> ```
{: .code-in}

> <code-out-title></code-out-title>
>
> id  |        update-time         | user-id | email |           name           | published | deleted | purged | hid-counter
> --- | -------------------------- | ------- | ----- | ------------------------ | --------- | ------- | ------ | ------------
> 361 | 2013-02-24 16:27:29.197572 |     xxx | xxxx  | Unnamed history          | f         | f       | f      | 6
> 362 | 2013-02-24 15:31:05.804747 |     xxx | xxxx  | Unnamed history          | f         | f       | f      | 1
> 347 | 2013-02-22 15:59:12.044169 |     xxx | xxxx  | Unnamed history          | f         | f       | f      | 19
> 324 | 2013-02-22 15:57:54.500637 |     xxx | xxxx  | Exercise 5               | f         | f       | f      | 64
> 315 | 2013-02-22 15:50:51.398894 |     xxx | xxxx  | day5 practical           | f         | f       | f      | 90
> 314 | 2013-02-22 15:45:47.75967  |     xxx | xxxx  | 5. Tag Galaxy-Kurs       | f         | f       | f      | 78
>
{: .code-out}

**@gtn:natefoo's favourite**: `gxadmin query job-inputs`. He contributed this function which helps him debug jobs which are not running and should be.

> <code-in-title></code-in-title>
> ```
> gxadmin query job-inputs 5 # Or another job ID
> ```
{: .code-in}

hda-id   | hda-state | hda-deleted | hda-purged |  d-id   | d-state | d-deleted | d-purged | object-store-id
-------- | --------- | ----------- | ---------- | ------- | ------- | --------- | -------- | ----------------
8638197  |           | f           | f          | 8246854 | running | f         | f        | files9
8638195  |           | f           | f          | 8246852 | running | f         | f        | files9
8638195  |           | f           | f          | 8246852 | running | f         | f        | files9

**@gtn:bgruening's favourite**: `gxadmin query latest-users` let's us see who has recently joined our server. We sometimes notice that people are running a training on our infrastructure and they haven't registered for [training infrastructure as a service](https://galaxyproject.eu/tiaas) which helps us coordinate infrastructure for them so they don't have bad experiences.

> <code-in-title></code-in-title>
> ```
> gxadmin query latest-users
> ```
{: .code-in}

id    |        create_time         | disk_usage | username | email | groups | active
----- | -------------------------- | ---------- | -------- | ----- | ------ | -------
3937  | 2019-01-27 14:11:12.636399 | 291 MB     | xxxx     | xxxx  |        | t
3936  | 2019-01-27 10:41:07.76126  | 1416 MB    | xxxx     | xxxx  |        | t
3935  | 2019-01-27 10:13:01.499094 | 2072 kB    | xxxx     | xxxx  |        | t
3934  | 2019-01-27 10:06:40.973938 | 0 bytes    | xxxx     | xxxx  |        | f
3933  | 2019-01-27 10:01:22.562782 |            | xxxx     | xxxx  |        | f

**@gtn:hexylena's favourite** `gxadmin report job-info`. This command gives more information than you probably need on the execution of a specific job, formatted as markdown for easy sharing with fellow administrators.

> <code-in-title></code-in-title>
> ```
> gxadmin report job-info 1
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```text
> # Galaxy Job 5132146
> 
> Property      | Value
> ------------- | -----
>          Tool | toolshed.g2.bx.psu.edu/repos/bgruening/canu/canu/1.7
>         State | running
>       Handler | handler_main_2
>       Created | 2019-04-20 11:04:40.854975+02 (3 days 05:49:30.451719 ago)
> Job Runner/ID | condor / 568537
>         Owner | e08d6c893f5
> 
> ## Destination Parameters
> 
> Key                     |   Value
> ---                     |   ---
> description             |   `canu`
> priority                |   `-128`
> request_cpus            |   `20`
> request_memory          |   `64G`
> requirements            |   `GalaxyGroup == "compute"`
> tmp_dir                 |   `True`
> 
> ## Dependencies
> 
> Name   |   Version   |   Dependency Type   |   Cacheable   |   Exact   |   Environment Path                          |   Model Class
> ---    |   ---       |   ---               |   ---         |   ---     |   ---                                       |   ---
> canu   |   1.7       |   conda             |   false       |   true    |   /usr/local/tools/_conda/envs/__canu@1.7   |   MergedCondaDependency
> 
> ## Tool Parameters
> 
> Name                 |   Settings
> ---------            |   ------------------------------------
> minOverlapLength     |   500
> chromInfo            |   /opt/galaxy/tool-data/shared/ucsc/chrom/?.len
> stage                |   all
> contigFilter         |   {lowCovDepth: 5, lowCovSpan: 0.5, minLength: 0, minReads: 2, singleReadSpan: 1.0}
> s                    |   null
> mode                 |   -nanopore-raw
> dbkey                |   ?
> genomeSize           |   300000
> corOutCoverage       |   40
> rawErrorRate         |
> minReadLength        |   1000
> correctedErrorRate   |
> 
> ## Inputs
> 
> Job ID    |   Name                |   Extension     |   hda-id    |   hda-state   |   hda-deleted   |   hda-purged   |   ds-id     |   ds-state   |   ds-deleted   |   ds-purged   |   Size
> ----      |   ----                |   ----          |   ----      |   ----        |   ----          |   ----         |   ----      |   ----       |   ----         |   ----        |   ----
> 4975404   |   Osur_record.fastq   |   fastqsanger   |   9517188   |               |   t             |   f            |   9015329   |   ok         |   f            |   f           |   3272 MB
> 4975404   |   Osur_record.fastq   |   fastqsanger   |   9517188   |               |   t             |   f            |   9015329   |   ok         |   f            |   f           |   3272 MB
> 
> ## Outputs
> 
> Name                                          |   Extension   |   hda-id    |   hda-state   |   hda-deleted   |   hda-purged   |   ds-id     |   ds-state   |   ds-deleted   |   ds-purged   |   Size
> ----                                          |   ----        |   ----      |   ----        |   ----          |   ----         |   ----      |   ----       |   ----         |   ----        |   ----
> Canu assembler on data 41 (trimmed reads)     |   fasta.gz    |   9520369   |               |   f             |   f            |   9018510   |   running    |   f            |   f           |
> Canu assembler on data 41 (corrected reads)   |   fasta.gz    |   9520368   |               |   f             |   f            |   9018509   |   running    |   f            |   f           |
> Canu assembler on data 41 (unitigs)           |   fasta       |   9520367   |               |   f             |   f            |   9018508   |   running    |   f            |   f           |
> Canu assembler on data 41 (unassembled)       |   fasta       |   9520366   |               |   f             |   f            |   9018507   |   running    |   f            |   f           |
> Canu assembler on data 41 (contigs)           |   fasta       |   9520365   |               |   f             |   f            |   9018506   |   running    |   f            |   f           |
> ```
{: .code-out}

**@gtn:cat-bro contributed** the 'jobs' query: `gxadmin query jobs` lets you list jobs that have been run on your Galaxy. It's a *lot* more flexible than `queue-overview` and we suggest using it instead, in most places. E.g. to find `circos` jobs that were recently run:


> <code-in-title></code-in-title>
> ```
> gxadmin query jobs --limit 2  --tool circos
> ```
{: .code-in}

job_id    |     create_time     |     update_time     | user_id | state |                            tool_id                            |    handler     | destination  | external_id 
--------- | ------------------- | ------------------- | ------- | ----- | ------------------------------------------------------------- | -------------- | ------------ | ------------
58483488  | 2023-04-04 18:42:40 | 2023-04-04 18:43:30 |         | error | toolshed.g2.bx.psu.edu/repos/iuc/circos/circos/0.69.8+galaxy8 | handler_sn06_5 | 1cores_10.0G | 42071736
58483208  | 2023-04-04 18:36:24 | 2023-04-04 18:40:43 |         | error | toolshed.g2.bx.psu.edu/repos/iuc/circos/circos/0.69.8+galaxy9 | handler_sn06_3 | 1cores_10.0G | 42071486

or to see recent jobs from a specific user (e.g. to help answer their email queries when they just send you a screenshot rather than a proper bug report)


> <code-in-title></code-in-title>
> ```
> gxadmin query jobs --limit 2 --user helena-rasche --terminal
> ```
{: .code-in}

job_id    |     create_time     |     update_time     | user_id | state |                                     tool_id                                      |    handler     | destination | external_id 
--------- | ------------------- | ------------------- | ------- | ----- | -------------------------------------------------------------------------------- | -------------- | ----------- | ------------
58277473  | 2023-03-31 09:53:36 | 2023-03-31 09:53:36 |     580 | ok    | __TAG_FROM_FILE__                                                                | _default_      |             | 
58277410  | 2023-03-31 09:47:16 | 2023-03-31 09:51:26 |     580 | ok    | toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/1.1.4 | handler_sn06_0 | 1cores_4.0G | 41859564

## `gxadmin` for Monitoring

`gxadmin` already supported `query`, `csvquery`, and `tsvquery` for requesting data from the Galaxy database in tables, CSV, or TSV formats, but we recently implemented `influx` queries which output data in a format that [Telegraf](https://github.com/influxdata/telegraf) can consume.

So running `gxadmin query queue-overview` normally shows something like:

> <code-in-title></code-in-title>
> ```
> gxadmin query queue-overview
> ```
{: .code-in}

tool_id                                                                                |  tool_version  |    destination_id    |     handler     |  state  | job_runner_name | count
-------------------------------------------------------------------------------------- | -------------- | -------------------- | --------------- | ------- | --------------- | ------
toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0                           | 0.4.6.0        | 12cores_180G_special | handler_main_4  | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0                           | 0.4.6.0        | 12cores_180G_special | handler_main_5  | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.1.0.46-0                    | 1.1.0.46-0     | 12cores_12G          | handler_main_3  | running | condor          |     2
toolshed.g2.bx.psu.edu/repos/iuc/qiime_extract_barcodes/qiime_extract_barcodes/1.9.1.0 | 1.9.1.0        | 4G_memory            | handler_main_1  | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/iuc/hisat2/hisat2/2.1.0+galaxy3                           | 2.1.0+galaxy3  | 8cores_20G           | handler_main_11 | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72                                | 0.72           | 20G_memory           | handler_main_11 | running | condor          |     4
ebi_sra_main                                                                           | 1.0.1          | 4G_memory            | handler_main_3  | running | condor          |     2
ebi_sra_main                                                                           | 1.0.1          | 4G_memory            | handler_main_4  | running | condor          |     3


The `gxadmin iquery queue-overview` is run by our Telegraf monitor on a regular basis, allowing us to consume the data:

> <code-in-title></code-in-title>
> ```
> gxadmin iquery queue-overview
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0,tool_version=0.4.6.0,state=running,handler=handler_main_4,destination_id=12cores_180G_special,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0,tool_version=0.4.6.0,state=running,handler=handler_main_5,destination_id=12cores_180G_special,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.1.0.46-0,tool_version=1.1.0.46-0,state=running,handler=handler_main_3,destination_id=12cores_12G,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/vcffilter/vcffilter2/1.0.0_rc1+galaxy1,tool_version=1.0.0_rc1+galaxy1,state=queued,handler=handler_main_11,destination_id=4G_memory,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/iuc/hisat2/hisat2/2.1.0+galaxy3,tool_version=2.1.0+galaxy3,state=running,handler=handler_main_11,destination_id=8cores_20G,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72,tool_version=0.72,state=running,handler=handler_main_11,destination_id=20G_memory,job_runner_name=condor count=4
> queue-overview,tool_id=ebi_sra_main,tool_version=1.0.1,state=running,handler=handler_main_3,destination_id=4G_memory,job_runner_name=condor count=2
> queue-overview,tool_id=ebi_sra_main,tool_version=1.0.1,state=running,handler=handler_main_4,destination_id=4G_memory,job_runner_name=condor count=3
> ```
{: .code-out}

And produce [some nice graphs](https://stats.galaxyproject.eu/) from it.

You can use an influx configuration like:

```toml
[[inputs.exec]]
    commands = ["/usr/bin/galaxy-queue-size"]
    timeout = "10s"
    data_format = "influx"
    interval = "1m"
```

This often requires a wrapper script, because you'll need to pass environment variables to the `gxadmin` invocation, e.g.:

```bash
#!/bin/bash
export PGUSER=galaxy
export PGHOST=dbhost
gxadmin iquery queue-overview --short-tool-id
gxadmin iquery workflow-invocation-status
```

> <tip-title>Which queries support iquery?</tip-title>
> This data is not currently exposed, so, just try the queries. But it's easy to add influx support when missing! [Here is an example](https://github.com/usegalaxy-eu/gxadmin/blob/e0ec0174ebbdce1acd8c40c7431308934981aa0c/parts/22-query.sh#L54), we set the variables in a function:
>
> ```
> fields="count=1"
> tags="tool_id=0"
> ```
>
> This means: column 0 is a tag named tool_id, and column 1 is a field (real value) named count.
> [Here is an example](https://github.com/usegalaxy-eu/gxadmin/blob/e0ec0174ebbdce1acd8c40c7431308934981aa0c/parts/22-query.sh#L1987) that has multiple fields that are stored.
>
{: .tip}

# Summary

There are a lot of queries, all tailored to specific use cases, some of these may be interesting for you, some may not. These are [all documented](https://github.com/usegalaxy-eu/gxadmin#commands) with example inputs and outputs in the gxadmin readme, and help is likewise available from the command line.

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=10 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="gxadmin" %}


topic: admin
tutorial: celeryless
title: Alternative Celery Deployment for Galaxy
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible"]
content_hash: bc4a2a81a80d9d6576ab2b0ff40a81a8
last_modified: 2026-05-15T22:28:10.012937
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/celeryless/tutorial.html

content:
Celery is a new component to the Galaxy world (ca 2023) and is a distributed task queue that *can* be used to run tasks asynchronously. It isn't mandatory, but you might find some features you expect to use to be missing without it.

If you are running a large production deployment you probably want to follow the [Celery+Redis+Flower Tutorial]({% link topics/admin/tutorials/celery/tutorial.md %}).

However, if you are running a smaller Galaxy you may not want to manage deploying Celery (past what Gravity does for you automatically), you may not want to add Redis to your stack, and you may not have need of Flower!

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Configuring Galaxy to use Postgres

AMQP is a message queue protocol which processes can use to pass messages between each other. While a real message queue like RabbitMQ is perhaps the most robust choice, there is an easier option: Postgres

Add the following to your Galaxy configuration to use Postgres:

```yaml
amqp_internal_connection: "sqlalchemy+postgresql:///galaxy?host=/var/run/postgresql"
```

# Configuring Celery to use Postgres

Celery would prefer you use Redis (a Key-Value store) as a backend to store results. But we have a database! So let's try using that instead:

```yaml
enable_celery_tasks: true
celery_conf:
  broker_url: null  # This should default to using amqp_internal_connection
  result_backend: "db+postgresql:///galaxy?host=/var/run/postgresql"
  task_routes:
    galaxy.fetch_data: galaxy.external
    galaxy.set_job_metadata: galaxy.external
```


With that we should now be able to [use useful features like](https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks):

- Changing the datatype of a collection.
- Exporting histories
- other things!

# Configuring with Ansible

If you're using Ansible, this could also look like:

```yaml
amqp_internal_connection: "sqlalchemy+{{ database_connection }}"
enable_celery_tasks: true
celery_conf:
  broker_url: null  # This should default to using amqp_internal_connection
  result_backend: "db+{{ database_connection }}"
  task_routes:
    galaxy.fetch_data: galaxy.external
    galaxy.set_job_metadata: galaxy.external
```


topic: admin
tutorial: customization
title: Customizing the look of Galaxy
difficulty: intermediate
time_estimation: 45m
tools_json: []
requirements_json: ["[]"]
tags_json: ["git-gat"]
content_hash: 140c7dc2b7caa1221de77bb5ac261675
last_modified: 2026-05-27T09:01:47.635515
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/tutorial.html

content:
Customizing your Galaxy instance makes it more recognizable at a glance, and can help communicate its purpose to its users.
This tutorial will teach you three basic customizations you can make to Galaxy:

  - Setting a brand text
  - Setting the contact information
  - Adding a custom welcome page
  - Customizing the masthead using themes

For this tutorial we will be setting up a fictional instance called "Galaxy Mars".
Feel free to use the included material in the hands-on sections, or provide your own.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Custom Branding

The brand text in Galaxy refers to the text you can see in the masthead of some Galaxy instances.
This text will appear in the masthead, as well as the sites title.
It is an easy way to set your instance apart, and make it more identifiable.

![screenshot of the start page of Galaxy Europe](images/galaxy-europe.png "Galaxy Europe uses the brand text \"Europe\"")

> <hands-on-title>Customising the Branding</hands-on-title>
> 
> 1. Open your `group_vars/galaxyserers.yml` and the following option under `galaxy_config.galaxy`:
> 
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -35,6 +35,9 @@ galaxy_job_config:
>
>     galaxy_config:
>       galaxy:
>    +    # Branding
>    +    brand: Mars 🚀
>    +    logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>         # Main Configuration
>         admin_users:
>         - admin@example.org
>    {% endraw %}
>    ```
>    {: data-commit="Add brand"}
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    ![screenshot of fictional "Galaxy Mars" start page, with the brand text set to "Mars"](images/galaxy-mars-brand.png "Your Galaxy start page should now look something like this")
{: .hands_on}

## Configuring Support

The Galaxy Help Site (https://help.galaxyproject.org) receives a lot of user support questions, sometimes for Galaxies that we do not manage. Including support information inside your Galaxy can help users find the right place to ask about issues with tools or quotas!

> <hands-on-title>Adding Support Information</hands-on-title>
>
> 1. Open your `group_vars/galaxyserers.yml` and the following option under `galaxy_config.galaxy`:
> 
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -38,6 +38,9 @@ galaxy_config:
>         # Branding
>         brand: Mars 🚀
>         logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>    +    # Support
>    +    support_url: "https://example.org/support"
>    +    terms_url: "https://example.org/terms-of-service"
>         # Main Configuration
>         admin_users:
>         - admin@example.org
>    {% endraw %}
>    ```
>    {: data-commit="Add support information"}
>
> Ideally this would point to a support URL and a terms of service that are appropriate to *your* deployment of Galaxy. You could, for instance, add a page on your group's blog with the appropriate contents of who should be contacted in case of issue, their email or a ticketing system. These options will help users from your site find you when they need help.
{: .hands_on}

## Custom Welcome Page

The welcome page is an html document embedded in Galaxy's start page.
It is what users see when first navigating to your instance.
This page can be used to communicate what your instance is about, and share news and updates with your users.

> <hands-on-title>Creating a custom welcome page</hands-on-title>
>
> 1. Set the location of your welcome page in `group_vars/galaxyservers.yml`
> 
>    Under `galaxy_config.galaxy` specify the location of your welcome page
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -99,6 +99,10 @@ galaxy_config:
>               - job-handlers
>               - workflow-schedulers
>
>    +galaxy_config_files_public:
>    +  - src: files/galaxy/welcome.html
>    +    dest: "{{ galaxy_mutable_config_dir }}/welcome.html"
>    +
>     galaxy_extra_dirs:
>       - /data
>
>    {% endraw %}
>    ```
>    {: data-commit="Add welcome url to the config"}
>
> 2. Let's then develop the associated template, add a title and some text to our welcome page:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/welcome.html
>    @@ -0,0 +1,11 @@
>    +<!DOCTYPE html>
>    +<html lang="en">
>    +    <head>
>    +        <meta charset="utf-8">
>    +        <link href="/static/dist/base.css" rel="stylesheet" type="text/css" />
>    +    </head>
>    +    <body>
>    +        <h1>Welcome to the Galaxy Mars instance!</h1>
>    +        <p>The only Galaxy instance on mars.</p>
>    +    </body>
>    +</html>
>    {% endraw %}
>    ```
>    {: data-commit="Add the template"}
>
>    This works the same as any other html page. You can add styles in the head, or import some scripts.
>
>    Make sure your welcome page has exactly one `h1` element, which describes the page.
>    This will act as the heading for your start-page, which can help assistive technologies.
>
> 1. Now that we have a proper welcome page, we'll also need to correct our nginx routes:
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -40,7 +40,7 @@ server {
>     	# automatically copied around. The welcome page is one of them. In
>     	# production, this step is skipped, so we will manually alias that.
>     	location /static/welcome.html {
>    -		alias {{ galaxy_server_dir }}/static/welcome.sample.html;
>    +		alias {{ galaxy_mutable_config_dir }}/welcome.html;
>     		expires 24h;
>     	}
>
>    {% endraw %}
>    ```
>    {: data-commit="Fix the nginx routes for the welcome page"}
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> <tip-title>Use an IFrame!</tip-title>
> A lot of the UseGalaxy.* instances choose to use their `welcome.html` to display an IFrame to a different site,
> e.g. the GalaxyProject.org site, or a sub-site thereof.
>
> This makes it easier for non-administrators to keep the homepage's contents
> up to date. By embedding (for example) a Wordpress blog in the center,
> non-technical contributors can still publish articles and announcements
> without having to re-deploy Galaxy every time.
{: .tip}

> <tip-title>Styling your Welcome</tip-title>
> As in our example welcome.html, we've loaded the default Galaxy CSS to keep
> it consistent with the surrounding material. If you wish to load a different
> CSS file, or set a different font for the main panel, you can easily do that
> by adding whatever CSS you like.
{: .tip}

## Custom Masthead Theme

Galaxy's theming system allows you to easily change the color of your Masthead, to give your instance an even more distinct look.
You can even offer several options, to allow users to switch to the default if they prefer it, or another look all together.

> <hands-on-title>Configuring Themes</hands-on-title>
>
> 0. Set the location of the themes configuration in your `group_vars/galaxyservers.yml`
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -38,6 +38,7 @@ galaxy_config:
>         # Branding
>         brand: Mars 🚀
>         logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>    +    themes_config_file: "{{ galaxy_config_dir }}/themes.yml"
>         # Support
>         support_url: "https://example.org/support"
>         terms_url: "https://example.org/terms-of-service"
>    @@ -103,6 +104,10 @@ galaxy_config_files_public:
>       - src: files/galaxy/welcome.html
>         dest: "{{ galaxy_mutable_config_dir }}/welcome.html"
>
>    +galaxy_config_files:
>    +  - src: files/galaxy/themes.yml
>    +    dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>    +
>     galaxy_extra_dirs:
>       - /data
>
>    {% endraw %}
>    ```
>    {: data-commit="Add brand"}
>
> 1. Create your themes config file in `files/galaxy/themes.yml`:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/themes.yml
>    @@ -0,0 +1,76 @@
>    +blue:
>    +  masthead:
>    +    color: "#2c3143"
>    +    text:
>    +      color: "#f8f9fa"
>    +      hover: gold
>    +      active: white
>    +    link:
>    +      color: transparent
>    +      hover: transparent
>    +      active: "#181a24"
>    +    logo:
>    +      img: "/static/favicon.svg"
>    +      img-secondary: null
>    +
>    +lightblue:
>    +  masthead:
>    +    color: "#384E77"
>    +    text:
>    +      color: white
>    +      hover: "#E6F9AF"
>    +      active: white
>    +    link:
>    +      color: transparent
>    +      hover: transparent
>    +      active: "#18314F"
>    +    logo:
>    +      img: "/static/favicon.svg"
>    +
>    +pride:
>    +  masthead:
>    +      color: >
>    +        linear-gradient(120deg,
>    +          #3c476d 0px 130px,
>    +          #fdda0f 131px 139px,
>    +          #fff 140px 148px,
>    +          #f4b0c9 149px 157px,
>    +          #7ccee6 158px 166px,
>    +          #93540c 167px 175px,
>    +          #000 176px 184px,
>    +          transparent 185px),
>    +        linear-gradient(270deg,
>    +          #3c476d 0px 110px,
>    +          #3c476d00 110px),
>    +        linear-gradient(180deg,
>    +          #FE0000 16.66%,
>    +          #FD8C00 16.66% 33.32%,
>    +          #FFE500 33.32% 49.98%,
>    +          #119F0B 49.98% 66.64%,
>    +          #0644B3 66.64% 83.3%,
>    +          #C22EDC 83.3%)
>    +      text:
>    +        color: white
>    +        hover: gold
>    +        active: white
>    +      link:
>    +        color: "#3c476d"
>    +        hover: "#323a53"
>    +        active: "#6170a6"
>    +      logo:
>    +        img: "/static/favicon.svg"
>    +        img-secondary: null
>    +
>    +smoky:
>    +  masthead:
>    +    color: "#0C0F0A"
>    +    text:
>    +      color: white
>    +      hover: "#FBFF12"
>    +      active: white
>    +    link:
>    +      color: transparent
>    +      hover: transparent
>    +      active: "#FF206E"
>    +    logo:
>    +      img: "/static/favicon.svg"
>    {% endraw %}
>    ```
>    {: data-commit="Add themes file"}
>
>    The themes file can contains themes for users to select in Galaxy.
>
>    Each theme is identified by its id, eg `blue`, and followed by a set of
>    rules, which style the client. The first theme in your theme file will be
>    used as the default theme, which users see when visiting your instance,
>    and which users who haven't logged in will see.
>
> 3. Creating our own Theme
>
>    Let's create a theme more fitting for our Mars instance.
>
>    Create a new theme, by adding `mars:` to the top of the themes config file.
>    Now add `masthead:` underneath, and then `color:`.
>    Set color to `"#e03e1d"`, a nice deep martian red.
>
>    Your theme file should now begin with:
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/themes.yml
>    +++ b/files/galaxy/themes.yml
>    @@ -1,3 +1,8 @@
>    +# Our Martian Theme
>    +mars:
>    +  masthead:
>    +    color: "#e03e1d"
>    +
>     blue:
>       masthead:
>         color: "#2c3143"
>    {% endraw %}
>    ```
>    {: data-commit="Add new default theme"}
>
> 3. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. You can also try logging in, and changing your theme under `Preferences`
{: .hands_on}

{% snippet topics/admin/faqs/missed-something.md step=3 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="customization" %}


topic: admin
tutorial: apptainer
title: Use Apptainer containers for running Galaxy jobs
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/apptainer/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["jobs", "ansible", "git-gat"]
content_hash: 9c0ca49bb184dbc562cd02b666e6bd43
last_modified: 2026-05-27T09:01:47.631294
workflows_json: []
content:
In this tutorial you will learn how to configure Galaxy to run jobs using [Apptainer](https://apptainer.org) containers provided by the [BioContainers](https://biocontainers.pro/) community.
Make sure to read the documentation on [container in Galaxy](https://docs.galaxyproject.org/en/master/admin/special_topics/mulled_containers.html) and in particular [container resolvers in Galaxy](https://docs.galaxyproject.org/en/master/admin/container_resolvers.html).


## Background

> BioContainers is a community-driven project that provides the infrastructure and basic guidelines to create, manage and distribute bioinformatics packages (e.g Conda) and containers (e.g Docker, Apptainer). BioContainers is based on the popular frameworks Conda, Docker and Apptainer.
>
{: .quote cite="https://biocontainers-edu.readthedocs.io/en/latest/what_is_biocontainers.html"}

Apptainer is an alternative to Docker that is much friendlier for HPCs

> <comment-title>Apptainer, Singularity, SingularityCE?</comment-title>
>
> Name                  | Singularity                                          | SingularityCE                                                  | Apptainer
> --------------------- | ---------------------------------------------------- | ---------------------                                          | ----------------
> Origin                | Original name, used by the project until 2021        | Name of Sylabs' Fork (CE for <b>C</b>ommunity <b>E</b>dition) | Name change when the project joined the Linux Foundation
> Status                | renamed in 2021                                      | currently active                                               | currently active
> RPM Package available | discontinued                                         | ❌                                                             | ✅
> CLI name              | `singularity`                                        | `singularity`                                                  | `apptainer` or `singularity`
>
> Many people still know Apptainer under its former name, Singularity.
> Singularity was forked in 2021, with the non-commercial fork being renamed to Apptainer and joining the Linux Foundation.
> Sylabs maintains their own free and open source fork, 'SingularityCE' (for <u>C</u>ommunity <u>E</u>dition), as well as a commercial version, 'SingularityPRO'.
>
> We will use the Name Apptainer for our training material, because most rpm packages are now named Apptainer.
{: .comment}

> Apptainer is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible.
{: .quote cite="https://sylabs.io/guides/3.7/user-guide/introduction.html"}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="apptainer" %}

# Installing Apptainer

First, we will install Apptainer using Ansible. Since there is a package available for major Linux distros now, we could simply install it in the pre tasks. However, we would have to enable additional repos, so we decided to create a role for this:

> <hands-on-title>Installing Apptainer with Ansible</hands-on-title>
>
> 1. In your working directory, add the Apptainer role to your `requirements.yml` file:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -19,3 +19,6 @@
>     # CVMFS Support
>     - src: galaxyproject.cvmfs
>       version: 0.3.3
>    +# Singularity/Apptainer
>    +- src: usegalaxy_eu.apptainer
>    +  version: 0.0.3
>    {% endraw %}
>    ```
>    {: data-commit="Add Apptainer ansible roles"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the requirements with `ansible-galaxy`:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. Add the new roles to your `galaxy.yml` playbook, before the Galaxy server itself. We'll do this because it's a dependency of Galaxy to run, so it needs to be there before Galaxy starts.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -31,6 +31,7 @@
>           when: ansible_os_family == 'Debian'
>       roles:
>         - galaxyproject.tusd
>    +    - usegalaxy_eu.apptainer
>         - galaxyproject.galaxy
>         - role: galaxyproject.miniconda
>           become: true
>    {% endraw %}
>    ```
>    {: data-commit="Add the roles to the playbook"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 6. Apptainer should now be installed on your Galaxy server. You can test this by connecting
>    to your server and run the following command:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > apptainer run docker://hello-world
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    > ```
>    > INFO:    Converting OCI blobs to SIF format
>    > INFO:    Starting build...
>    > Getting image source signatures
>    > Copying blob 0e03bdcc26d7 done
>    > Copying config b23a8f6569 done
>    > Writing manifest to image destination
>    > Storing signatures
>    > 2021/01/08 11:25:12  info unpack layer: sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689
>    > INFO:    Creating SIF file...
>    > WARNING: passwd file doesn't exist in container, not updating
>    > WARNING: group file doesn't exist in container, not updating
>    >
>    > Hello from Docker!
>    > This message shows that your installation appears to be working correctly.
>    > ...
>    > ```
>    {: .code-out}
{: .hands_on}

## Configure Galaxy to use Apptainer

Now, we will configure Galaxy to run tools using Apptainer containers, which will be automatically fetched from [the BioContainers repository](https://quay.io/organization/biocontainers).  

> <warning-title>Galaxy Calls It Singularity, not Apptainer</warning-title>
> Galaxy still uses `singularity` in most variables, they will be replaced successively.
{: .warning}

> <hands-on-title>Configure Galaxy to use Apptainer</hands-on-title>
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add a `dependency_resolvers_config_file` entry and a corresponding `galaxy_config_templates` entry:
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add a `container_resolvers_config_file` entry and a corresponding `galaxy_config_templates` entry:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -78,6 +78,9 @@ galaxy_config:
>         tus_upload_store: "{{ galaxy_tus_upload_store }}"
>         # CVMFS
>         tool_data_table_config_path: /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml,/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
>    +    # Tool Dependencies
>    +    dependency_resolvers_config_file: "{{ galaxy_config_dir }}/dependency_resolvers_conf.xml"
>    +    container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    @@ -117,6 +120,12 @@ galaxy_config_files:
>       - src: files/galaxy/themes.yml
>         dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>
>    +galaxy_config_templates:
>    +  - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>    +    dest: "{{ galaxy_config.galaxy.container_resolvers_config_file }}"
>    +  - src: templates/galaxy/config/dependency_resolvers_conf.xml
>    +    dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
>    +
>     galaxy_extra_dirs:
>       - /data
>
>    {% endraw %}
>    ```
>    {: data-commit="Configure the container and dependency resolvers"}
>
> 2. Create the `templates/galaxy/config` directory if it doesn't exist:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > mkdir -p templates/galaxy/config
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Create the new file `templates/galaxy/config/dependency_resolvers_conf.xml`. This will not enable any dependency resolvers like the legacy toolshed packages or Galaxy packages, and instead everything will be resolved through Apptainer.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/galaxy/config/dependency_resolvers_conf.xml
>    @@ -0,0 +1,2 @@
>    +<dependency_resolvers>
>    +</dependency_resolvers>
>    {% endraw %}
>    ```
>    {: data-commit="Configure the dependency resolvers"}
>
> 3. Create the new file `templates/galaxy/config/container_resolvers_conf.yml.j2`, this specifies the order in which to attempt container resolution.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/galaxy/config/container_resolvers_conf.yml.j2
>    @@ -0,0 +1,11 @@
>    +
>    +- type: cached_explicit_singularity
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/explicit/"
>    +- type: cached_mulled_singularity
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/mulled/"
>    +- type: mulled_singularity
>    +  auto_install: False
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/mulled/"
>    +- type: build_mulled_singularity
>    +  auto_install: False
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/built/"
>    {% endraw %}
>    ```
>    {: data-commit="Configure the container resolver"}
>
> 3. Now, we want to make Galaxy run jobs using Apptainer. Modify the file `group_vars/galaxyservers.yml`, by adding the `singularity_enabled` parameter:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -24,11 +24,26 @@ galaxy_job_config:
>       handling:
>         assign: ['db-skip-locked']
>       execution:
>    -    default: local_env
>    +    default: singularity
>         environments:
>           local_env:
>             runner: local_runner
>             tmp_dir: true
>    +      singularity:
>    +        runner: local_runner
>    +        singularity_enabled: true
>    +        # Enabling access to the reference data on CVMFS in the container
>    +        singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    +        env:
>    +        # Ensuring a consistent collation environment is good for reproducibility.
>    +        - name: LC_ALL
>    +          value: C
>    +        # The cache directory holds the docker containers that get converted
>    +        - name: APPTAINER_CACHEDIR
>    +          value: /tmp/singularity
>    +        # Apptainer uses a temporary directory to build the squashfs filesystem
>    +        - name: APPTAINER_TMPDIR
>    +          value: /tmp
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>           environment: local_env
>    {% endraw %}
>    ```
>    {: data-commit="Update the job_conf with singularity destination"}
>
> 4. Re-run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 5. In your Galaxy admin interface, install the minimap2 tool.
>
>    - Login to Galaxy as the admin user
>    - Click the "admin" menu at the top
>    - Under "Tool Management" on the left select "Install and Uninstall"
>    - search for `minimap2` and install the latest version with the Target Section "Mapping"
>
>    ![Screenshot of the install interface, minimap2 is entered in the search box and the latest revision shows it is currently cloning](../../images/install-minimap2.png)
>
> 6. Upload the following fasta file
>
>    ```
>    >testing
>    GATTACAGATHISISJUSTATESTGATTACA
>    ```
>
> 2. **Map with minimap2** {% icon tool %} with the following parameters
>    - *"Will you select a reference genome from your history or use a built-in index"*: `Use a genome from history and build index`
>    - *"Use the following dataset as the reference sequence"*: The fasta file you uploaded
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-file %} *"Select fastq dataset"*: The fasta file you uploaded
>
>    Your job should be executed using Apptainer with a BioContainer! You can watch the logs of Galaxy to see this happening.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > journalctl -f
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```
>    > gunicorn[1190010]: galaxy.tool_util.deps.containers INFO 2021-01-08 13:37:30,342 [p:1190010,w:0,m:2] [LocalRunner.work_thread-1] Checking with container resolver [MulledSingularityContainerResolver[namespace=biocontainers]] found description [ContainerDescription[identifier=docker://quay.io/biocontainers/mulled-v2-66534bcbb7031a148b13e2ad42583020b9cd25c4:e1ea28074233d7265a5dc2111d6e55130dff5653-0,type=singularity]]
>    > gunicorn[1190010]: galaxy.jobs.command_factory INFO 2021-01-08 13:37:30,418 [p:1190010,w:0,m:2] [LocalRunner.work_thread-1] Built script [/srv/galaxy/jobs/000/23/tool_script.sh] for tool command [minimap2 --version > /srv/galaxy/jobs/000/23/outputs/COMMAND_VERSION 2>&1; ln -f -s '/data/000/dataset_22.dat' reference.fa && minimap2           -t ${GALAXY_SLOTS:-4} reference.fa '/data/000/dataset_22.dat' -a | samtools sort -@${GALAXY_SLOTS:-2} -T "${TMPDIR:-.}" -O BAM -o '/data/000/dataset_23.dat' > '/data/000/dataset_23.dat']
>    > gunicorn[1190010]: galaxy.jobs.runners DEBUG 2021-01-08 13:37:30,441 [p:1190010,w:0,m:2] [LocalRunner.work_thread-1] (23) command is: mkdir -p working outputs configs
>    > gunicorn[1190010]: if [ -d _working ]; then
>    > gunicorn[1190010]:     rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
>    > gunicorn[1190010]: else
>    > gunicorn[1190010]:     cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
>    > gunicorn[1190010]: fi
>    > gunicorn[1190010]: cd working; SINGULARITYENV_GALAXY_SLOTS=$GALAXY_SLOTS SINGULARITYENV_HOME=$HOME SINGULARITYENV__GALAXY_JOB_HOME_DIR=$_GALAXY_JOB_HOME_DIR SINGULARITYENV__GALAXY_JOB_TMP_DIR=$_GALAXY_JOB_TMP_DIR SINGULARITYENV_TMPDIR=$TMPDIR SINGULARITYENV_TMP=$TMP SINGULARITYENV_TEMP=$TEMP singularity -s exec -B /srv/galaxy/server:/srv/galaxy/server:ro -B /srv/galaxy/var/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/minimap2/8c6cd2650d1f/minimap2:/srv/galaxy/var/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/minimap2/8c6cd2650d1f/minimap2:ro -B /srv/galaxy/jobs/000/23:/srv/galaxy/jobs/000/23 -B /srv/galaxy/jobs/000/23/outputs:/srv/galaxy/jobs/000/23/outputs -B /srv/galaxy/jobs/000/23/configs:/srv/galaxy/jobs/000/23/configs -B /srv/galaxy/jobs/000/23/working:/srv/galaxy/jobs/000/23/working -B /data:/data -B /srv/galaxy/var/tool-data:/srv/galaxy/var/tool-data:ro -B /srv/galaxy/var/tool-data:/srv/galaxy/var/tool-data:ro --home $HOME:$HOME docker://quay.io/biocontainers/mulled-v2-66534bcbb7031a148b13e2ad42583020b9cd25c4:e1ea28074233d7265a5dc2111d6e55130dff5653-0 /bin/bash /srv/galaxy/jobs/000/23/tool_script.sh > ../outputs/tool_stdout 2> ../outputs/tool_stderr; return_code=$?; cd '/srv/galaxy/jobs/000/23';
>    > ```
>    {: .code-out.code-max-300}
>
{: .hands_on}

> ```bash
> 1-run-minimap2.sh
> ```
> {: data-test="true"}
{: .hidden}

> <comment-title>Manage dependencies menu</comment-title>
> You can manually pull one or many containers for tools in the admin menu. Go to the admin menu, click Manage Dependencies and select the Containers tab. This will list all tools, their dependencies and whether containers are already pulled or can be pulled on demand.
>
> When a container has been resolved through Apptainer, you'll see something like this:
> ![Image of a table entry with minimap2 having requirements minimap2+singularity, a resolved column with a green checkmark next to via singularity, the resolver is mulled_singularity, and a container column with a path to /srv/galaxy/var/cache/singularity/mulled and some long hash.](../../images/singularity-resolved.png)
{: .comment}

> <tip-title>Apptainer, Conda, something else?</tip-title>
> We often hear
>
> > What would be the best practice, use Conda or Apptainer?
> {: .quote}
>
> Many of us are moving towards Apptainer. Conda environments can resolve differently if they were installed at different times, which isn't great for reproducibility. Apptainer images are never updated after generation which makes them fantastic. Also the isolation that's there by default is an incredible improvement for less-trustworthy binaries.
{: .tip}

> <tip-title>Does Apptainer fix issues with Conda dependencies resolution?</tip-title>
> Yes and no. Apptainer images are built from Conda environments. Only now you are no longer responsible for solving the conda environment, or ensuring that all of the dependencies are installed. The Galaxy project uses a system called "mulling" to bring together multiple Conda dependencies together in a single environment, and Apptainer images are produced for these dependencies as well. That said, complex or unresolvable Conda environments are not solved by Apptainer, because Apptainer is really just packaging Conda's environment into a single binary file.
{: .tip}


> <tip-title>Gateway Time-out (504) in Dependencies view</tip-title>
> When you open "Admin -> Tool Management -> Manage Dependencies -> Containers", it sometimes shows "Gateway Time-out (504)"
>
> Resolving all dependencies for all tools can take a bit, you can increase your timeout with the `proxy_read_timeout` setting in `templates/nginx/galaxy.j2`
{:.tip}

> <tip-title>Resolution is "unresolved"</tip-title>
> In "Admin -> Tool Management -> Manage Dependencies -> Dependencies", the Resolution for minimap2 @ 2.24 (as well as samtools @1.14) is "unresolved". How can I resolve this issue?
>
> Because our training uses containers for resolution it is expected that the non-container dependencies show as "unresolved". There is not currently a view which indicates if the containers have been resolved.
{: .tip}





<!--
## Use Apptainer containers from CVMFS

Galaxy can be configured to use pre-made Apptainer containers available from /cvmfs/singularity.galaxyproject.org/.
In order to do so, you will first need to set up CVMFS by doing the [CVMFS]({{ site.baseurl }}/topics/admin/tutorials/cvmfs/tutorial.html) tutorial.
After finishing the CVMFS tutorial, come back, and do this hands-on.

> <hands-on-title>Optional: Configure Galaxy to use Apptainer containers from CVMFS</hands-on-title>
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add `container_resolvers_config_file` and `galaxy_singularity_images_cvmfs_path`:
>    {% raw %}
>    ```yaml
>    galaxy_singularity_images_cvmfs_path: "/cvmfs/singularity.galaxyproject.org/all/"
>    galaxy_config:
>      galaxy:
>        ...
>        container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>    ```
>    {% endraw %}
>
> 2. Also in `group_vars/galaxyservers.yml`, add a `galaxy_config_templates` entry:
>    {% raw %}
>    ```yaml
>    galaxy_config_templates:
>      - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>        dest: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>    ```
>    {% endraw %}
>
> 3. Create the new file `templates/galaxy/config/container_resolvers_conf.yml.j2`:
>    {% raw %}
>    ```xml
>    <containers_resolvers>
>        <explicit_singularity />
>        <cached_mulled_singularity cache_directory="{{ galaxy_singularity_images_cvmfs_path }}"/>
>    </containers_resolvers>
>    ```
>    {% endraw %}
>
> 4. Re-run the playbook (`ansible-playbook galaxy.yml`)
>
> 5. Rerun your minimap2 job from the previous section. This time it should use a container served by CVMFS.
>
{: .hands_on}

> ```bash
> 1-run-minimap2.sh
> ```
> {: data-test="true"}
{: .hidden}

-->

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=6 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="apptainer" %}


topic: admin
tutorial: backup-cleanup
title: Server Maintenance: Cleanup, Backup, and Restoration
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["ansible", "deploying", "git-gat"]
content_hash: 1657abf9428ebc68ee867f328c90dbd5
last_modified: 2026-05-27T09:01:47.631409
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/backup-cleanup/tutorial.html

content:
Keeping your Galaxy cleaned up is an important way to retain space, especially since for many groups that is the
limiting factor in their deployment.

Additionally, backups are necessary to ensure that if you ever experience system level failures, you can safely recover
from these.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="backup-cleanup" %}

# Cleanups

There are two kinds of data that are produced when running a Galaxy: files users create and then delete or purge, and
then files Galaxy creates itself. Both of these can be cleaned to save space.

## User Created Files

You can use `gxadmin` to cleanup user created files. `gxadmin` is covered in more detail in [its own dedicated
tutorial]({% link topics/admin/tutorials/gxadmin/tutorial.md %}).

> <hands-on-title>Installing gxadmin with Ansible</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -11,3 +11,6 @@
>       version: 0.3.4
>     - src: usegalaxy_eu.certbot
>       version: 0.1.13
>    +# gxadmin (used in cleanup, and later monitoring.)
>    +- src: galaxyproject.gxadmin
>    +  version: 0.0.14
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add the role to your playbook:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -27,3 +27,4 @@
>           become: true
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>    +    - galaxyproject.gxadmin
>    {% endraw %}
>    ```
>    {: data-commit="Add the gxadmin role"}
>
> 3. Setup a cleanup task to run regularly:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -28,3 +28,11 @@
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>         - galaxyproject.gxadmin
>    +  post_tasks:
>    +    - name: Setup gxadmin cleanup task
>    +      ansible.builtin.cron:
>    +        name: "Cleanup Old User Data"
>    +        user: galaxy # Run as the Galaxy user
>    +        minute: "0"
>    +        hour: "0"
>    +        job: "SHELL=/bin/bash source {{ galaxy_venv_dir }}/bin/activate &&  GALAXY_LOG_DIR=/tmp/gxadmin/ GALAXY_ROOT={{ galaxy_root }}/server GALAXY_CONFIG_FILE={{ galaxy_config_file }} /usr/local/bin/gxadmin galaxy cleanup 60"
>    {% endraw %}
>    ```
>    {: data-commit="Configure gxadmin to cleanup data"}
>
>    This will cause datasets deleted for more than 60 days to be purged.
>
> 4. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

Whenever `gxadmin` runs, it will create logs you can read in `/tmp/gxadmin` which you can check later.

## Galaxy Created Files

Before we begin backing up our Galaxy data, let's set up automated cleanups to ensure we backup the minimal required set of data.

> <hands-on-title>Configuring PostgreSQL Backups</hands-on-title>
>
> 1. Edit `galaxy.yml` to install `tmpwatch` (if using RHEL/CentOS/Rocky) and `tmpreaper` if using Debian/Ubuntu
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -21,6 +21,14 @@
>         - name: Install Dependencies
>           package:
>             name: ['acl', 'bzip2', 'git', 'make', 'tar', 'python3-venv', 'python3-setuptools']
>    +    - name: Install RHEL/CentOS/Rocky specific dependencies
>    +      package:
>    +        name: ['tmpwatch']
>    +      when: ansible_os_family == 'RedHat'
>    +    - name: Install Debian/Ubuntu specific dependencies
>    +      package:
>    +        name: ['tmpreaper']
>    +      when: ansible_os_family == 'Debian'
>       roles:
>         - galaxyproject.galaxy
>         - role: galaxyproject.miniconda
>    {% endraw %}
>    ```
>    {: data-commit="Install tmpwatch/tmpreaper"}
>
> 1. Edit `group_vars/galaxyservers.yml` and add some variables to configure PostgreSQL:
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -3,6 +3,7 @@ galaxy_create_user: true # False by default, as e.g. you might have a 'galaxy' u
>     galaxy_separate_privileges: true # Best practices for security, configuration is owned by 'root' (or a different user) than the processes
>     galaxy_manage_paths: true # False by default as your administrator might e.g. have root_squash enabled on NFS. Here we can create the directories so it's fine.
>     galaxy_manage_systemd: true
>    +galaxy_manage_cleanup: true
>     galaxy_layout: root-dir
>     galaxy_root: /srv/galaxy
>     galaxy_user: {name: "{{ galaxy_user_name }}", shell: /bin/bash}
>    {% endraw %}
>    ```
>    {: data-commit="Configure automated cleanup"}
>
> 2. > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Check out the cleanup task which has been generated in: `/etc/cron.d/ansible_galaxy_tmpclean`
>
{: .hands_on}

This will setup `tmpwatch` to cleanup a few folders:

- the job working directory, important if you set `cleanup: onsuccess`, to cleanup old failed jobs once you're done debugging their failures.
- the new file upload path, to catch uploaded temporary files that are no longer necessary.

# Backups

There are a few important things to back up with your Ansible Galaxy:

- Galaxy
	- The Galaxy-managed config files
	- The playbooks
- The Database
- The Data

## Galaxy

By using Ansible, as long as you are storing your playbooks on another system, you are generally safe from failues of
the Galaxy node, and you'll be able to re-run your playbook at a later date.

However, playbooks often do not include:

- Which tools you've installed (have you ever installed a tool outside of ephemeris? This might be lost!)
- Conda environments, which will not always resolve identically over time. If strong guarantees of reproducibility are
  important, then consider backing these up as well.

## Database Backups

We're setting a couple of variables to control the automatic backups, they'll be placed in the `/data/backups` folder next to our user uploaded Galaxy data.

> <hands-on-title>Configuring PostgreSQL Backups</hands-on-title>
>
> 1. Edit `group_vars/dbservers.yml` and add some variables to configure PostgreSQL:
>    {% raw %}
>    ```diff
>    --- a/group_vars/dbservers.yml
>    +++ b/group_vars/dbservers.yml
>    @@ -5,3 +5,7 @@ postgresql_objects_users:
>     postgresql_objects_databases:
>       - name: "{{ galaxy_db_name }}"
>         owner: "{{ galaxy_user_name }}"
>    +
>    +# PostgreSQL Backups
>    +postgresql_backup_dir: /data/backups
>    +postgresql_backup_local_dir: "{{ '~postgres' | expanduser }}/backups"
>    {% endraw %}
>    ```
>    {: data-commit="Add backups"}
>
>    <!-- TODO: expanduser tip here -->
{: .hands_on}

This will setup our backups to run as a cron job. 

<!-- TODO: explore cron jobs -->

## Data Backup

With Galaxy it is *technically* only necessary to backup your inputs, as the downstream files *should*, *in theory* be re-createable due to the reproducibility of Galaxy.

In practice, some groups either choose to not backup, or to backup everything, often to extremely cheap and slow storage like Glacier or a tape library.

Most groups choose to implement this as a custom cron job, e.g.

```yaml
post_tasks:
  - name: Setup backup cron job
    ansible.builtin.cron:
      name: "Backup User Data"
      minute: "0"
      hour: "5,2"
      job: "rsync -avr /data/galaxy/ backup@backup.example.org:/backups/$(date -I)/"
```

> <tip-title>This isn't a backup strategy!</tip-title>
> People who, let's say, care strongly about backups will often insist that you need to version files. This is of course unnecessary in the Galaxy case as files are essentially {WORM}s, which is a really good file storage practice. Files can get removed so it isn't a true {WORM} strategy that you'd use for e.g. audit logs, but it is close.
> That said, since files never get changed, keeping multiple versions is unnecesary.
>
> Please consider communicating **very well** with your users what the data backup policy is.
{: .tip}

{% snippet topics/admin/faqs/missed-something.md step=2 %}

# Restoration

Sometimes failures happen! We're sorry you have to read this section.

## Restoring the Database

This procedure is more complicated, you can [read about the restoration procedure](https://github.com/galaxyproject/ansible-postgresql/pull/30#issuecomment-963600656) in the associated PR.

This step assumes you have pre-existing backups in place, you must check this first:

```console
ls /data/backups/
```

If you have backups, you're ready to restore:

```console
# Stop Galaxy, you do NOT want galaxy to connect mid-restoration in case it
# tries to modify the database.
sudo systemctl stop galaxy

# Stop the database
sudo systemctl stop postgresql
# Ensure that it is stopped
sudo systemctl status postgresql

# Begin the backup procedure by becoming postgres:
sudo su - postgres

# Move the current, live database to a backup location just in case:
mkdir /tmp/test/

# ====
# NOTE THAT THIS NUMBER MAY BE DIFFERENT FOR YOU!
# You will need to change 12 to whatever version of postgres you're running
# in every subsequent command
# ====
mv /var/lib/postgresql/12/main/* /tmp/test/

# Add backup
rsync -av /data/backups/YOUR_LATEST_BACKUP/ /var/lib/postgresql/12/main
# Add the restore_command, to your backup file:
# restore_command = 'cp "/tmp/backup/current/wal/%f" "%p"'
$EDITOR ./12/main/postgresql.auto.conf

# Touch a recovery file
touch /var/lib/postgresql/12/main/recovery.signal

# As $username (with sudo right)
sudo systemctl restart postgresql
sudo systemctl status postgresql
# Restart Galaxy
sudo systemctl start galaxy
```

If you encounter issues, we suggest reading [Lucille's log of her experiences restoring](https://github.com/lldelisle/galaxyduboule-infrastructure/blob/778e5b056f03970a1af5fc2f9dd133b4e4791cac/testUpgradeOnVM.sh#L53-L130) as you might encounter similar issues.

## Restoring Galaxy

Restoring Galaxy is easy via Ansible (maybe ensuring users cannot login by disabling the routes in nginx)

```console
ansible-playbook galaxy.yml
```

And if you are following best practices, you probably have your tools stored in a YAML file to use with [Ephemeris]({% link topics/admin/tutorials/tool-management/tutorial.md %}):

```console
shed-tools install -g https://galaxy.example.org -a <api-key> -t our_tools.yml
```

## Restoring User Data

This should simply be `rsync`ing your data from the backup location back into `/data/galaxy`.

{% snippet topics/admin/faqs/git-gat-path.md tutorial="backup-cleanup" %}


topic: admin
tutorial: wireguard-headscale
title: Deploying Tailscale/Headscale for private mesh networking
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["wireguard", "networking"]
content_hash: a5a136bad484eb94007d027130bd55cf
last_modified: 2026-05-15T22:28:10.075209
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/wireguard-headscale/tutorial.html

content:
[Tailscale](https://tailscale.com/) makes secure networking easy, it really is like magic. If you've used wireguard before, you know it takes a bit to setup and some configuration if you need to do anything fancy.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# What is Tailscale?

It's like Wireguard but easier and they built a **lot** of nice features on top of it. It's networking that "Just Works™" even more than Wireguard. If you prefer to use plain Wireguard without Headscale/Tailscale, or just want to get an understanding of the technology that Headscale/Tailscale build off of, [there is a tutorial for that]({% link topics/admin/tutorials/wireguard/tutorial.md %}) as well.

## Is it right for me?

if you have machines that need to talk to each other privately, and you don't
have a better way to do it like a local network team, then yes, it's a great
solution to private, secure, fast networking. if you need auditing, tailscale
will do that, rather than you having to build it out yourself.
it has [excellent performance](https://www.wireguard.com/performance/)
despite the encryption, and is built directly into the kernel.

By using wireguard, you can let services listen only on the wireguard
interface, and thus only known and trusted machines can access those services.

Tailscale makes wireguard setup even easier by removing the key management
step, which normally requires distributing keys to every machine. Instead that
step is handled centrally, and in the case of Tailscale enforceable with ACLs
and SSO and 2FA policies, however the networking remains meshed, and machines
connect directly to one another.

You can go one step further than trusting individual machines, with Tailscale,
as every device is tied to a single user, and you can ask Tailscale what is the
authenticated identity of the specific TCP connection, allowing automatically
logging in your users.

In the context of Galaxy, this can be useful for components like [Interactive 
Tools]({% link topics/admin/tutorials/interactive-tools/tutorial.md %}), which 
require a web proxy between Galaxy and the cluster node where the tool runs. If 
the cluster is not on the local network, Wireguard can be used to securely 
bridge the gap, and Headscale or Tailscale can greatly simplify that process.
# Setting up the infrastructure

{% include _includes/cyoa-choices.html option1="Tailscale" option2="Headscale" default="Tailscale" help="Here we provide a version of this tutorial that uses Headscale, if you want a more hands-on experience and feel like managing your own infrastructure." %}

> <hands-on-title>Configuration files</hands-on-title>
>
> 1. Create a `ansible.cfg` file (next to your playbook) to [configure settings](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html) like the inventory file (and save ourselves some typing!), or the Python interpreter to use:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/ansible.cfg
>    @@ -0,0 +1,6 @@
>    +[defaults]
>    +interpreter_python = /usr/bin/python3
>    +inventory = hosts
>    +retry_files_enabled = false
>    +[ssh_connection]
>    +pipelining = true
>    {% endraw %}
>    ```
>    {: data-commit="Add ansible.cfg"}
>
> 2. Create the `hosts` inventory file if you have not done so yet.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat hosts
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > Your hostname is probably different:
>    > > <div class="Headscale" markdown="1">
>    > > Pick one to be the `head`, and the rest to be the `tail`. The head will act as the coordination server, the `tail` will be the nodes that should talk to each other.
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,5 @@
>    > > +[head]
>    > > +1-wg.galaxy.training
>    > > +[tail]
>    > > +2-wg.galaxy.training
>    > > +3-wg.galaxy.training
>    > > +4-wg.galaxy.training
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add hosts"}
>    > > </div>
>    > >
>    > > <div class="Tailscale" markdown="1">
>    > > Place all of the nodes in a `tail` group
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,5 @@
>    > > +[tail]
>    > > +1-wg.galaxy.training
>    > > +2-wg.galaxy.training
>    > > +3-wg.galaxy.training
>    > > +4-wg.galaxy.training
>    > > ```
>    > > </div>
>    > {: .code-out}
>    {: .code-2col}
{: .hands_on}

<div class="Headscale" markdown="1">

> <details-title>Consider using Tailscale</details-title>
> Tailscale has implemented the coordination server and infrastructure with much better, more robust infrastructure that you won't have to be responsible for. If you can get your institution to pay for it, or grant money covering it, it's **probably worth it**. They add many new features often, and things like the mobile apps only work with their service. Tailscale is free for personal use (i.e. to test things out) and offers a free plan for open source projects that you may qualify for.
>
> For a training event obviously we want something free and quick to setup and destroy, so, we're using [Headscale](https://github.com/juanfont/headscale) since it's free and we're just going to destroy it immediately, and no one will accidentally get billed ;)
>
> Using Headscale will also teach you everything you need to know if you do choose to use Tailscale, which is simpler and has fewer components for you to manage yourself.
{: .details}

> <hands-on-title>Installing Headscale</hands-on-title>
>
> 1. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles ckstevenson.headscale
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Create and open `head.yml` which will be our playbook. Add the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/head.yml
>    @@ -0,0 +1,16 @@
>    +---
>    +- name: Headscale
>    +  hosts: head
>    +  become: true
>    +  vars:
>    +    headscale_user: 'headscale'
>    +    headscale_version: '0.15.0'
>    +    headscale_namespaces:
>    +    - galaxy
>    +  roles:
>    +    - ckstevenson.headscale
>    +  post_tasks:
>    +    - command: headscale --namespace galaxy preauthkeys create --reusable --expiration 1h
>    +      register: authkey
>    +    - debug:
>    +        msg: "{{ authkey.stdout.split('\n')[-1] }}"
>    {% endraw %}
>    ```
>    {: data-commit="Setup headscale"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook headscale.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. This will return a code in the debug output. Save this code, you'll need it shortly
{:.hands_on}
</div>

Now we can setup the nodes

> <hands-on-title>Configure the nodes</hands-on-title>
>
> 1. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles artis3n.tailscale
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 1. Edit `tail.yml` and add the following.
>
>    <div class="Headscale" markdown="1">
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/tail.yml
>    @@ -0,0 +1,17 @@
>    +---
>    +- name: Tailscale
>    +  hosts: tail
>    +  become: true
>    +  vars:
>    +    tailscale_args: "--advertise-exit-node --login-server http://{{ hostvars[groups['head'][0]].inventory_hostname }}:8080"
>    +  pre_tasks:
>    + - sysctl:
>    +     name: net.ipv4.ip_forward
>    +     value: '1'
>    +     state: present
>    + - sysctl:
>    +     name: net.ipv6.conf.all.forwarding
>    +     value: '1'
>    +     state: present
>    +  roles:
>    +    - artis3n.tailscale
>    {% endraw %}
>    ```
>    {: data-commit="Setup the clients"}
>    </div>
>
>    <div class="Tailscale" markdown="1">
>    ```diff
>    --- /dev/null
>    +++ b/tail.yml
>    @@ -0,0 +1,17 @@
>    +---
>    +- name: Tailscale
>    +  hosts: tail
>    +  become: true
>    +  vars:
>    +    tailscale_args: "--advertise-exit-node"
>    +  pre_tasks:
>    + - sysctl:
>    +     name: net.ipv4.ip_forward
>    +     value: '1'
>    +     state: present
>    + - sysctl:
>    +     name: net.ipv6.conf.all.forwarding
>    +     value: '1'
>    +     state: present
>    +  roles:
>    +    - artis3n.tailscale
>    ```
>    </div>
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook tail.yml -e tailscale_authkey=YOUR_CODE
>    > ```
>    > {: data-cmd="true"}
>    >
>    > <div class="Headscale" markdown="1">
>    > Remember, you can find this code from the output of the first playbook
>    > </div>
>    > <div class="Tailscale" markdown="1">
>    > You can generate an authentication key under your [Tailscale account page](https://login.tailscale.com/admin/authkeys)
>    > </div>
>    {: .code-in}
>
> 3. Go check out your tailnet! Play around with the tailscale command and pinging other nodes with the suffix `.galaxy.example.org`{:.Headscale} `<username>.org.github.beta.tailscale.net`{:.Tailscale}
>
>    > Note that you'll need to enable [MagicDNS](https://login.tailscale.com/admin/dns) in your acount settings.
>    {:.Tailscale}
{:.hands_on}

> <tip-title>Exit Nodes</tip-title>
> We've configured `--advertise-exit-node`, which means you can direct ALL of your traffic to use one of your tailscale endpoints as an exit node, just run `tailscale up --exit-node=...`
>
> Note that:
> - If you're using headscale you need to manually enable that route (check the node list via `headscale nodes list` and then enable the specific route via `headscale nodes routes enable -i ...`), this is automatic in Tailscale
> - If you enable it, on a remote machine, it will immediately become unresponsive. Only do this on your local machine, e.g. a laptop connected to your tailnet.
{: .tip}


topic: admin
tutorial: tus
title: Performant Uploads with TUS
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/tus/tutorial.html
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 1631939404deb78ca800779fa6c3ca0d
last_modified: 2026-05-27T09:01:47.649369
workflows_json: []
content:
Here you'll learn to setup [TUS](https://tus.io/) an open source resumable file upload server to process uploads for Galaxy. We use an external process here to offload the main Galaxy processes for more important work and not impact the entire system during periods of heavy uploading.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tus" %}

# TUS and Galaxy

To allow your user to upload via TUS, you will need to:

- configure Galaxy to know where the files are uploaded.
- install TUSd
- configure Nginx to proxy TUSd

## Installing and Configuring

> <hands-on-title>Setting up ftp upload with Ansible</hands-on-title>
>
> 1. In your playbook directory, add the `galaxyproject.tusd` role to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -14,3 +14,5 @@
>     # gxadmin (used in cleanup, and later monitoring.)
>     - src: galaxyproject.gxadmin
>       version: 0.0.14
>    +- name: galaxyproject.tusd
>    +  version: 0.0.1
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Configure it in your group variables
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -73,6 +73,9 @@ galaxy_config:
>         # Tool security
>         outputs_to_working_directory: true
>         new_user_dataset_access_role_default_private: true # Make datasets private by default
>    +    # TUS
>    +    galaxy_infrastructure_url: "https://{{ inventory_hostname }}"
>    +    tus_upload_store: "{{ galaxy_tus_upload_store }}"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    @@ -93,6 +96,10 @@ galaxy_config:
>         celery:
>           concurrency: 2
>           loglevel: DEBUG
>    +    tusd:
>    +      enable: true
>    +      tusd_path: /usr/local/sbin/tusd
>    +      upload_dir: "{{ galaxy_tus_upload_store }}"
>         handlers:
>           handler:
>             processes: 2
>    @@ -162,3 +169,7 @@ nginx_conf_http:
>     nginx_ssl_role: usegalaxy_eu.certbot
>     nginx_conf_ssl_certificate: /etc/ssl/certs/fullchain.pem
>     nginx_conf_ssl_certificate_key: /etc/ssl/user/privkey-www-data.pem
>    +
>    +# TUS
>    +galaxy_tusd_port: 1080
>    +galaxy_tus_upload_store: /data/tus
>    {% endraw %}
>    ```
>    {: data-commit="Configure TUS in your group variables"}
>
> 4. We proxy the service next to Galaxy. As it resides "under" the Galaxy path, clients will send cookies and authentication headers to TUS, which it can use to process the uploads before telling Galaxy when they're done.
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -28,6 +28,22 @@ server {
>     		proxy_set_header Upgrade $http_upgrade;
>     	}
>
>    +	location /api/upload/resumable_upload {
>    +		# Disable request and response buffering
>    +		proxy_request_buffering     off;
>    +		proxy_buffering             off;
>    +		proxy_http_version          1.1;
>    +
>    +		# Add X-Forwarded-* headers
>    +		proxy_set_header X-Forwarded-Host   $host;
>    +		proxy_set_header X-Forwarded-Proto  $scheme;
>    +
>    +		proxy_set_header Upgrade            $http_upgrade;
>    +		proxy_set_header Connection         "upgrade";
>    +		client_max_body_size        0;
>    +		proxy_pass http://localhost:{{ galaxy_tusd_port }}/files;
>    +	}
>    +
>     	# Static files can be more efficiently served by Nginx. Why send the
>     	# request to Gunicorn which should be spending its time doing more useful
>     	# things like serving Galaxy!
>    {% endraw %}
>    ```
>    {: data-commit="Proxy it via NGINX"}
>
> 5. Add to the end of your Galaxy playbook
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -30,6 +30,7 @@
>             name: ['tmpreaper']
>           when: ansible_os_family == 'Debian'
>       roles:
>    +    - galaxyproject.tusd
>         - galaxyproject.galaxy
>         - role: galaxyproject.miniconda
>           become: true
>    {% endraw %}
>    ```
>    {: data-commit="Add the role to the playbook"}
>
> 6. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in }
>
{: .hands_on}

Congratulations, you've set up TUS for Galaxy.

## Check it works

> <hands-on-title>Check that it works.</hands-on-title>
>
> 1. SSH into your machine
>
> 2. Check the active status of tusd by `systemctl status galaxy-tusd`.
>
> 3. Follow the tusd logs with `journalctl -fu galaxy-tusd`
>
> 4. Upload a small file! (Pasted text will not pass via TUS)
>
> 5. Make sure you see `event=UploadFinished` in the tusd logs
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=4 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tus" %}


topic: admin
tutorial: object-store
title: Distributed Object Storage
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["ansible", "storage"]
content_hash: d71254f5bcf0deb35f0c4f7638b90d4d
last_modified: 2026-05-27T09:01:47.637207
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/object-store/tutorial.html

content:
You may find that your Galaxy files directory has run out of space, but you don't want to move all of the files from one filesystem to another. One solution to this problem is to use Galaxy's hierarchical object store to add an additional file space for Galaxy.

Alternatively, you may wish to write new datasets to more than one filesystem. For this, you can use Galaxy's distributed object store.

# Expanding Storage

This tutorial assumes you have done the "Ansible for installing Galaxy" tutorial, it references the base configuration set up in that tutorial in numerous places.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <warning-title>switching object store types will cause issues</warning-title>
> Do not run this tutorial on your production instance! We will switch between object stores during this tutorial which has serious implications for production servers.
{: .warning}

# Hierarchical Object Store

First, note that your Galaxy datasets have been created thus far in the directory `/data`, due to `galaxy_config: galaxy: file_path`. In some cases, we run out of storage in a particular location. Galaxy allows us to add additional storage locations where it will create new datasets, while still looking in the old locations for old datasets. You will not have to migrate any of your datasets, and can just "plug and play" with new storage pools.


> <hands-on-title>Adding Hierarchical Storage</hands-on-title>
>
> 1. Open your group variables file and set the `object_store_config_file` variable:
>
>    ```yaml
>    galaxy_config:
>      galaxy:
>        object_store_config_file: {% raw %}"{{ galaxy_config_dir }}/object_store_conf.xml"{% endraw %}
>    ```
>
> 2. In your group variables file, add it to the `galaxy_config_templates` section:
>
>    ```yaml
>    galaxy_config_templates:
>      - src: templates/galaxy/config/object_store_conf.xml.j2
>        dest: {% raw %}"{{ galaxy_config.galaxy.object_store_config_file }}"{% endraw %}
>    ```
>
> 3. Create and edit `templates/galaxy/config/object_store_conf.xml.j2` with the following contents:
>
>    ```xml
>    <?xml version="1.0"?>
>    <object_store type="hierarchical">
>        <backends>
>            <backend id="newdata" type="disk" order="0">
>                <files_dir path="/data2" />
>                <extra_dir type="job_work" path="/data2/job_work_dir" />
>            </backend>
>            <backend id="olddata" type="disk" order="1">
>                <files_dir path="/data" />
>                <extra_dir type="job_work" path="/data/job_work_dir" />
>            </backend>
>        </backends>
>    </object_store>
>    ```
>
> 4. Add a `pre_task` in your playbook `galaxy.yml` file to create the `/data2` folder [using the file module](https://docs.ansible.com/ansible/2.9/modules/file_module.html).
>
>    ```
>        - name: Create the second storage directory
>          file:
>            owner: galaxy
>            group: galaxy
>            path: /data2
>            state: directory
>            mode: '0755'
>    ```
>
>    We've hardcoded the user/group because creating a storage directory is unusual. In normal practice someone provides you with an NFS mount and you will simply point your Galaxy there.
>
> 5. Run the playbook and restart Galaxy
>
> 6. Run a couple of jobs after Galaxy has restarted.
>
>    > <question-title></question-title>
>    >
>    > Where is the data now stored?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > You should see `/data2` in the `Full Path`, if not, something went wrong. Check that your "order" is correct
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

# Distributed Object Store

Rather than searching a hierarchy of object stores until the dataset is found, Galaxy can store the ID (in the database) of the object store in which a dataset is located when the dataset is created. This allows Galaxy to write to more than one object store for new datasets.


> <hands-on-title>Distributed Object Store</hands-on-title>
>
> 1. Edit your `templates/galaxy/config/object_store_conf.xml.j2` file and replace the contents with:
>
>    ```xml
>    <?xml version="1.0"?>
>    <object_store type="distributed">
>        <backends>
>            <backend id="newdata" type="disk" weight="1">
>                <files_dir path="/data2"/>
>                <extra_dir type="job_work" path="/data2/job_work_dir"/>
>            </backend>
>            <backend id="olddata" type="disk" weight="1">
>                <files_dir path="/data"/>
>                <extra_dir type="job_work" path="/data/job_work_dir"/>
>            </backend>
>        </backends>
>    </object_store>
>    ```
>
> 2. Run the playbook, restart Galaxy
>
> 3. Run 4 or so jobs, and check where the output appear. You should see that they are split relatively evenly between the two data directories.
>
{: .hands_on}

Sites like UseGalaxy.eu use the distributed object store in order to balance dataset storage across 10 different storage pools.

> <details-title>More documentation</details-title>
>
> More information can be found in the [sample file](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/config/sample/object_store_conf.xml.sample).
>
{: .details}

> <tip-title>Can I distribute objects based on the user?</tip-title>
>
> Yes! You must write your own dynamic job handler code to handle this.
> See [PR#6552](https://github.com/galaxyproject/galaxy/pull/6552) and [PR#10233](https://github.com/galaxyproject/galaxy/pull/10233)
>
> If you implement something like this, please [let the GTN
> know](https://github.com/galaxyproject/training-material/issues/new) with
> some example code, and we can include this as a training module for everyone.
>
{: .tip}

> <warning-title>switching object store types will cause issues</warning-title>
> We have switched between two different object stores here, but this is not supported. If you need to do this, you will need to update datasets in Galaxy's database. Any datasets that were created as hierarchical will lack the `object_store_id`, and you will need to supply the correct one. Do not just blindly copy these instructions, please understand what they do before running them and talk to us on [Gitter](http://gitter.im/galaxyproject/Lobby) for more help
>
> 1. Move the datasets to their new location: `sudo -u galaxy rsync -avr /hierarchical/000/ /distributed/000/`
>
> 2. Update the database: `sudo -Hu galaxy psql galaxy -c "UPDATE dataset SET object_store_id='data';" `
>
> 3. Restart your Galaxy
>
{: .warning}


# S3 Object Store

Many sites have access to an S3 service (either public AWS, or something private like Swift or Ceph), and you can take advantage of this for data storage.

we will set up a local S3-compatible object store, and then talk to the API of this service.

> <hands-on-title>Setting up an S3-compatible Object Store</hands-on-title>
>
> 1. Edit your `requirements.yml` file and add:
>
>    ```yml
>    - src: atosatto.minio
>      version: v1.1.0
>    ```
>
> 2. ```ansible-galaxy install -p roles -r requirements.yml```
>
> 3. Edit your group variables to configure the object store:
>
>    ```
>    minio_server_datadirs: ["/minio-test"]
>    minio_access_key: "my-access-key"
>    minio_secret_key: "my-super-extra-top-secret-key"
>    ```
>
> 4. Edit your playbook and add the minio role **before** `galaxyproject.galaxy`:
>
>    ```
>        - atosatto.minio
>    ```
>
>    Galaxy will need to use the bucket, and will want it to be there when it boots, so we need to setup the object store first.
>
> 5. Edit the `templates/galaxy/config/object_store_conf.xml.j2`, and configure the object store as one of the hierarchical backends. The object store does not play nicely with the distributed backend during training preparation. Additionally, reset the orders of the disk backends to be higher than the order of the swift backend.
>
>    {% raw %}
>    ```diff
>    @@ -1,13 +1,21 @@
>     <?xml version="1.0"?>
>    - <object_store type="distributed">
>    + <object_store type="hierarchical">
>         <backends>
>    -        <backend id="newdata" type="disk" weight="1">
>    +        <backend id="newdata" type="disk" order="1">
>                 <files_dir path="/data2"/>
>                 <extra_dir type="job_work" path="/data2/job_work_dir"/>
>             </backend>
>    -        <backend id="olddata" type="disk" weight="1">
>    +        <backend id="olddata" type="disk" order="2">
>                 <files_dir path="/data"/>
>                 <extra_dir type="job_work" path="/data/job_work_dir"/>
>             </backend>
>    +        <object_store id="swifty" type="swift" order="0">
>    +            <auth access_key="{{ minio_access_key }}" secret_key="{{ minio_secret_key }}" />
>    +            <bucket name="galaxy" use_reduced_redundancy="False" max_chunk_size="250"/>
>    +            <connection host="127.0.0.1" port="9091" is_secure="False" conn_path="" multipart="True"/>
>    +            <cache path="{{ galaxy_mutable_data_dir }}/database/object_store_cache" size="1000" />
>    +            <extra_dir type="job_work" path="{{ galaxy_mutable_data_dir }}/database/job_working_directory_swift"/>
>    +            <extra_dir type="temp" path="{{ galaxy_mutable_data_dir }}/database/tmp_swift"/>
>    +        </object_store>
>         </backends>
>     </object_store>
>    ```
>    {% endraw %}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 6. Run the playbook.
>
> 7. Galaxy should now be configure to use the object store!
>
> 8. When the playbook is done, upload a dataset to Galaxy, and check if it shows up in the bucket:
>
>    ```
>    $ sudo ls /minio-test/galaxy/000/
>    dataset_24.dat
>    ```
>
{: .hands_on}

# Dropbox

Dropbox is a well-known cloud storage service where you can store and share files with anyone.
As of [20.09](https://github.com/galaxyproject/galaxy/pull/9888), Galaxy has support for a couple of different file storage backends, including NextCloud (via webdavfs) and Dropbox.

This tutorial will help you setup the connection between Galaxy and Dropbox, allowing your users to add their account details and then access their Dropbox data within Galaxy

> <hands-on-title>Configure Galaxy to access the Dropbox service</hands-on-title>
>
> 1. If the folder does not exist, create `templates/galaxy/config` next to your `galaxy.yml` playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > mkdir -p templates/galaxy/config
>    > ```
>    {: .code-in}
>
> 2. Create `templates/galaxy/config/file_sources_conf.yml` with the following contents:
>
>    ```yaml
>    - type: dropbox
>      id: dropbox
>      label: Your Dropbox Files
>      doc: Your Dropbox files - configure an access token via the user preferences
>      accessToken: ${user.preferences.get('dropbox|access_token', '') if $user.preferences else ''}
>    ```
> 3. Create `templates/galaxy/config/user_preferences_extra_conf.yml` with the following contents:
>
>    ```yaml
>    preferences:
>        dropbox:
>            description: Your Dropbox account
>            inputs:
>                - name: access_token
>                  label: Dropbox access token
>                  type: password
>                  required: False
>    ```
> 4. Inform the `galaxyproject.galaxy` role of where you would like the `file_sources_conf.yml` and `user_preferences_extra_conf.yml` to reside, by setting it in your `group_vars/galaxyservers.yml`:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -35,6 +35,8 @@ galaxy_config:
>         tool_data_path: "{{ galaxy_mutable_data_dir }}/tool-data"
>         job_config_file: "{{ galaxy_config_dir }}/job_conf.yml"
>    +    file_sources_config_file: "{{ galaxy_config_dir }}/file_sources_conf.yml"
>    +    user_preferences_extra_conf_path: "{{ galaxy_config_dir }}/user_preferences_extra_conf.yml"
>    ```
>    {% endraw %}
>
> 5. Deploy the new config files using the `galaxy_config_files` var (also from the `galaxyproject.galaxy` role) in your group vars:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -65,6 +67,12 @@ galaxy_config_templates:
>       - src: templates/galaxy/config/job_conf.yml.j2
>         dest: "{{ galaxy_config.galaxy.job_config_file }}"
>     
>    +galaxy_config_files:
>    +  - src: files/galaxy/config/user_preferences_extra_conf.yml
>    +    dest: "{{ galaxy_config.galaxy.user_preferences_extra_conf_path }}"
>    +  - src: files/galaxy/config/file_sources_conf.yml
>    +    dest: "{{ galaxy_config.galaxy.file_sources_config_file }}"
>    +
>    ```
>    {% endraw %}
>
> 6. Run the playbook. At the very end, you should see output like the following indicating that Galaxy has been restarted:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```
>    > > ansible-playbook galaxy.yml
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Output</code-out-title>
>    > > ```
>    > > ...
>    > > RUNNING HANDLER [restart galaxy] ****************************************
>    > > changed: [gat-88.training.galaxyproject.eu]
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
{: .hands_on}

Now we are ready to configure a Galaxy's user account to upload dataset from Dropbox to the Galaxy server.

> <hands-on-title>Configure Galaxy to access the Dropbox service</hands-on-title>
>
> 1. Generate a Dropbox access token following the [Dropbox Oauth guide](https://www.dropbox.com/lp/developers/reference/oauth-guide)
>
> 2. Add the Dropbox access token in the Galaxy's user preferences
>    - Go to https://\<server\>/user/information
>    - Here you will find the form user will fill in with his own Dropbox access token:
>      ![user preferences form](../../images/storage_dropbox_form.png)
>
> 3. Click the upload icon toward the top left corner.
>    You will have a new "Choose remote files" button that will open the remote files windows with the link to reach your Dropbox files:
>     ![remote files window](../../images/storage_dropbox_remote_files_window.png)
>
{: .hands_on}


# Onedata

To configure Onedata as an Object Store or a Remote File Source, follow the
[dedicated tutorial]({% link topics/admin/tutorials/onedata-configuration/tutorial.html %}).


topic: admin
tutorial: surf-research-cloud-galaxy
title: Galaxy usage on SURF Research Cloud
difficulty: introductory
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"none\"", "type: \"none\"", "type: \"none\"", "type: \"none\""]
tags_json: []
content_hash: 8d3d8bb78147d198331e5fedf6080f97
last_modified: 2026-05-27T09:01:47.645669
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/surf-research-cloud-galaxy/tutorial.html

content:
Using Galaxy via the {SRC} allows researchers to start Galaxy instances on-demand and analyze their data in a secure environment following the {GDPR}. The instance provides secure authentication, where users must have a SURF Research account prior to this tutorial, have set the {SRAM} authentication method, and connect an SSH key to their accounts. In case you are not familiar with {SRC} and need help in setting up your accounts, please follow the instructions on the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/SURF+Research+Cloud)

In this training the aim is to focus on:

- Understanding how to start Galaxy on {SRC}
- Attaching external volumes for storage

Galaxy instances can be started and stopped on demand, depending on personal cases and requirements. Inside the SRC members should have access to all publicly available catalog items. If you are not able to create a catalog item, please [contact SURF servicedesk](mailto:servicedesk@surf.nl).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prerequisites

This tutorial assumes you are member of a {CO} in {SRAM} that has access to {SRC} and a wallet with budget in SRC with enough sources to create Galaxy and Pulsar catalog items. (For more information please refer to the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/Budgets%2C+wallets%2C+contracts).

You must have previous experience working with data inside Galaxy.

# Starting a Galaxy instance inside SRC step-by-step

> <hands-on-title>Access the SRC</hands-on-title>
> 1. The first and most important step is to have access to the SURF Research Cloud.
> 2. You will need to login to the [portal](https://portal.live.surfresearchcloud.nl).
>    ![login screen for SURF research cloud](./images/1.png)
>
> 3. Assuming you are a Dutch researcher, you should expect to see your educational institution listed on the authentication provider screen.
>    ![login with screen, showing an authentication provider being chosen.](./images/2.png)
> 4. Log in with your institute credentials and follow the SRAM authentication on your phone. Inside the environment, you can see if there are any active virtual machines, the options to create a workspace, new storage or request new wallet for your projects.
>    ![SRC dashboard, options like create new workspace, storage, or request a new wallet are available. one workspace is listed as running: galaxy test tim.](./images/3.png)
{: .hands_on}

With this, you're ready to get started launching resources within {SRC}.

## Create an external storage

The purpose of the storage is to attach it to your workspace, where you will save your data. This way, if the machine is deleted, your data will not be lost. We strongly advise anyone using the SRC to create storages and attach them to their machines!


> <hands-on-title>Create a Storage Volume</hands-on-title>
> 1. Locate the "Create new storage" option
>    ![top half of the SRC dashboard showing options like create new workspace, storage, or request a new wallet.](./images/4.png)
> 2. Click on "Create new".
> 3. Select the collaborative organisation you want the storage to be part of.
> 4. Select the desired cloud provider and the size of the volume you want to use.
>    > <tip-title>Unsure of the size required?</tip-title>
>    > If you are unsure about the size of the storage you need, contact your administrator. Consider also consulting [Galaxy from an Administrator's Point of View]({% link topics/admin/tutorials/introduction/slides.html %}#10)
>    {: .tip}
>
>    ![create your storage dashboard showing a selection of providers like surf and aws and azure, and then a selection of storage sizes below ranging from 5GB to 1.5TB](./images/5.png)
>
> 6. Name your storage however you like and press on submit.
>
>    ![form to provide name and preview what will be created.](./images/6.png)
>
> 7. You will be redirected to the main page, where you will be able to track the status of the storage creation.
>
>    ![bottom half of home page showing the Storage tab, and a resource named storage-galaxy in state creating.](./images/7.png)
{: .hands_on}

Once the storage is deployed, you can attach it to any of your machines! 🎉

Be aware, that in order to attach an external storage to an existing machine, you would need to pause the workspace first! (If you are not sure how to do that please refer to [External storage volumes](https://servicedesk.surf.nl/wiki/display/WIKI/External+storage+volumes))

# Create a new workspace

Depending on user’s rights, you can create a new workspace for your collaboration.

> <tip-title>A Workspace?</tip-title>
> Workspaces may also be known as *Virtual Machines/VMs* or *Instances* in other cloud providers.
{: .tip}

> <hands-on-title>Create a Workspace</hands-on-title>
> 1. On the right side under the "Workspaces" click on the "Add" button.
>
>    ![dashboard of workspaces showing a single running workspace titled galaxy test tim, and on the right a button to add a new workspace.](./images/8.png)
>
> 2. Then, you will be redirected to a new page, where you must first choose the collaborative organisation in which you want to create your workspace (in case you are a member of multiple organisations). Once chosen, you a new page will be loaded, where you will have access to all available catalog items.
>
>    ![Create your workspace screen showing 9 catalog items on the first page with pages more. Visible are Galaxy Pulsar node amongst some docker and ubuntu options.](./images/9.png)
>
> 3. Use the magnifying glass on the right side of the panel and search for Galaxy. Two catalog items will appear - the Galaxy instance designed for SURF and a Galaxy Pulsar node that can be connected to the instance. Select "Galaxy at SURF".
>
>    ![similar to previous image, but the items described above](./images/10.png)
>
> 4. SURF Research Cloud allows researchers to host their catalog items on different cloud providers. The Galaxy catalog item is currently supported only on the HPC Cloud and for Ubuntu 22.04. On this page, users can select the number of cores and RAM that they want on their machine. More sizes can be added in the future, and on request. Choose wisely and in case you are not certain contact your administrator!
>
>    ![workspace creation screen, choose the cloud provider is locked to SURF HPC, flavour is locked to 22.04, and the size options are available from 1GB/8CPU to 60C/750GB Hi-mem](./images/11.png)
>
> 5. Select the storage you have created earlier so it is attached to the new workspace.
>
>    ![create your workspace screen, storage only lists one option named storage-galaxy](./images/12.png)
>
> 6. Lastly, before the workspace is deployed, you need to choose for how long the machine will run.
>
>    > <tip-title>Expiration Date</tip-title>
>    > The standard life-time of the VM is 5 days. If you need it for longer, this option can be changed once the machine is running.
>    > Note, that once the machine is expired and deleted it cannot be restored! Plan accordingly and migrate your data in time to prevent data loss!
>    >
>    > This is an incredibly useful feature as it saves you from forgetting to destroy a VM. Especially for GPU nodes it can help you ensure that they disappear after your computation is complete.
>    {:.tip}
>
> 7. In this form, you can also select how to name your workspace, add a description (if you want) and specify the hostname. Then, scroll down, customise your Galaxy name, add your storage name and submit the workspace creation. 
>    ![almost there, some final details reads the heading. an expiration date is set and galaxy at surf is provided in the name. the hostname is set to galaxyatsurf and the description is blank](./images/17.png)
>    Change `my_storage` to the storage you attached earlier. In this case, `galaxy-storage`. Bonus: You can also rename the Galaxy to your liking! 
>    ![customise your Galaxy name and attach your storage. change the name of my_data to the storage you attached earlier, in this case, galaxy-storage](./images/18.png)
{: .hands_on}

With that, your workspace will start being created.

![A galaxy instance named galaxy demo is in status creating.](./images/14.png "Creating the workspace can take some time. You will see on the main page the status of your workspace.")

> <tip-title>Also launching Pulsar?</tip-title>
> While you are waiting, you can create your Galaxy Pulsar node simultaneously to save time.
{: .tip}

Once the workspace is running, press on the down arrow to check any information such as the ID, the owner of the workspace, the collaboration it is part of, when it is created and when it expires and the wallet it is using. Additionally, you can pause the workspace when you are not using it. Pressing on the storage option will display the storage that has been attached to the workspace.

![the details panel of the galaxy demo instance is shown including fields like an id, owner, date created and expiration date, url., etc](./images/15.png)

To access the Galaxy instance, press on the "Access" button. This will redirect you to a new webpage. The {SRAM} login will be displayed again, and after authenticating, you will be redirected to the Galaxy at SURF Research Cloud page!

![A galaxy instance homepage is shown, with a brand indicating that it was deployed on surf research cloud featuring a cute surfer emoji.](./images/16.png)

Congratulations! You have started a Galaxy instance inside the SURF Research Cloud! 🏄‍♀️🎉

If you have administrator rights, you can download tools that can be used by the members of the collaborative organisation. To obtain administrator rights you need to be part of the `src_co_admin` group inside the collaborative organisation. In case you are not sure how to download tools as a Galaxy administrator, please refer to [Installing Tools into Galaxy](https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/).

## Going Further with Pulsar?

Another important step when starting a local Galaxy instance on {SRC} is connecting it to a Galaxy Pulsar Node, which you can do [in this Pulsar SRC tutorial]({% link topics/admin/tutorials/surf-research-cloud-pulsar/tutorial.md %}). The Pulsar is an useful component that is used to execute your tasks on remote servers/clusters. In this case, the Pulsar is started as a new workspace, allowing users to also run interactive tools.

In SRC, this can also enable you to save costs. By choosing a smaller Galaxy instance, and attaching a large Pulsar node when you need to do computations, you can be sure to optimise your budget expenditures.


topic: admin
tutorial: k8s-managing-galaxy
title: Managing Galaxy on Kubernetes
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["kubernetes"]
content_hash: cfa2bbb2797d4802946df596b57cc581
last_modified: 2026-05-15T22:28:10.038916
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/k8s-managing-galaxy/tutorial.html

content:
# Managing Galaxy on Kubernetes

## Overview


A primary advantage of Galaxy on [Kubernetes](https://kubernetes.io/) is the ease with which common
administrative tasks can be performed reliably and without disruption of
service. In particular, because of containerization, Kubernetes provides a significant
advantage over managing individual virtual machines, where updates to system
libraries or components can cause unexpected breakage of dependent components.
With containerization, this becomes a simpler problem of swapping out a
container and replacing it with an updated version. It also reduces reliance
on the underlying operating system, allowing the OS to be upgraded and
have the latest security patches applied without having to worry about
how it will affect the applications running within. Kubernetes has built-in
functionality for draining a node of all containers and for transparently
moving those containers to a different node, allowing maintenance tasks to be
performed on the underlying node without disruption of service.

In this section, we will look at how to perform common management tasks on
a Galaxy deployment on Kubernetes, including:
- How to upgrade a deployment
- Change the configuration of a running Galaxy instance
- Map arbitrary files into Galaxy's config folder
- Rollback changes in the case of an error
- Scale the number of job and web handlers
- Delete a deployment

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Prerequisites
This tutorial builds on the material of the previous
[tutorial]({{ site.baseurl }}/topics/admin/tutorials/k8s-deploying-galaxy/tutorial.html)
and we recommend following it first to setup the required environment.
You must have some familiarity with Helm commands, know how to change values
in a Helm Chart and how to use the `kubectl` command.

# Changing the configuration of a Galaxy instance
We will start off by looking at how to change the configuration of a Galaxy
instance. We will first reduce the number of tools that are loaded for faster
startup, and then change some common settings in `galaxy.yml`.

## Changing tool configuration
We will change the Galaxy configuration to limit the initial list of tools
that Galaxy will use by pointing to our custom `tool_conf.xml`. This will make
your chart start up much faster for the remainder of this tutorial, as the
default configuration loads the full list of tools used by
[https://usegalaxy.org/](https://usegalaxy.org/).

> <hands-on-title>Creating a custom tool set</hands-on-title>
>
> 1. First, let's create a simpler list of tools by saving the following tool
>    config as a file called `custom_tool_conf.xml`.
>
>    {% raw %}
>    ```xml
>    <?xml version="1.0" ?>
>    <toolbox tool_path="/cvmfs/main.galaxyproject.org/shed_tools">
>        <section id="get_data" name="Get Data">
>            <tool file="data_source/upload.xml" />
>        </section>
>        <section id="chip_seq" name="ChIP-seq" version="">
>            <tool file="toolshed.g2.bx.psu.edu/repos/rnateam/chipseeker/1b9a9409831d/chipseeker/chipseeker.xml" guid="toolshed.g2.bx.psu.edu/repos/rnateam/chipseeker/chipseeker/1.18.0+galaxy1">
>                <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
>                <repository_name>chipseeker</repository_name>
>                <repository_owner>rnateam</repository_owner>
>                <installed_changeset_revision>1b9a9409831d</installed_changeset_revision>
>                <id>toolshed.g2.bx.psu.edu/repos/rnateam/chipseeker/chipseeker/1.18.0+galaxy1</id>
>                <version>1.18.0+galaxy1</version>
>            </tool>
>        </section>
>        <section id="fastq_quality_control" name="FASTQ Quality Control" version="">
>            <tool file="toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/51b771646466/trimmomatic/trimmomatic.xml" guid="toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.6">
>                <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
>                <repository_name>trimmomatic</repository_name>
>                <repository_owner>pjbriggs</repository_owner>
>                <installed_changeset_revision>51b771646466</installed_changeset_revision>
>                <id>toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.6</id>
>                <version>0.36.6</version>
>            </tool>
>        </section>
>        <section id="fastq_quality_control" name="FASTQ Quality Control" version="">
>            <tool file="toolshed.g2.bx.psu.edu/repos/devteam/fastqc/e7b2202befea/fastqc/rgFastQC.xml" guid="toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1">
>                <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
>                <repository_name>fastqc</repository_name>
>                <repository_owner>devteam</repository_owner>
>                <installed_changeset_revision>e7b2202befea</installed_changeset_revision>
>                <id>toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1</id>
>                <version>0.72+galaxy1</version>
>            </tool>
>        </section>
>    </toolbox>
>    ```
>    {% endraw %}
>
> 2. Next, let's create a new `galaxy.yml` file that uses this `custom_tool_conf.xml`.
>
>    Note that the content below is the same as the `configs` section of
>    `values-cvmfs.yaml` file from the Galaxy Helm chart with one exception:
>    `tool_config_file` entry is pointing to our custom tool list instead of the
>    full list from CVMFS.
>
>    {% raw %}
>    ```yaml
>    uwsgi:
>      virtualenv: /galaxy/server/.venv
>      processes: 1
>      http: 0.0.0.0:8080
>      static-map: /static/style=/galaxy/server/static/style/blue
>      static-map: /static=/galaxy/server/static
>      static-map: /favicon.ico=/galaxy/server/static/favicon.ico
>      pythonpath: /galaxy/server/lib
>      thunder-lock: true
>      manage-script-name: true
>      mount: {{.Values.ingress.path}}=galaxy.webapps.galaxy.buildapp:uwsgi_app()
>      buffer-size: 16384
>      offload-threads: 2
>      threads: 4
>      die-on-term: true
>      master: true
>      hook-master-start: unix_signal:2 gracefully_kill_them_all
>      enable-threads: true
>      py-call-osafterfork: true
>    galaxy:
>      database_connection: 'postgresql://{{.Values.postgresql.galaxyDatabaseUser}}:{{.Values.postgresql.galaxyDatabasePassword}}@{{ template "galaxy-postgresql.fullname" . }}/galaxy'
>      integrated_tool_panel_config: "/galaxy/server/config/mutable/integrated_tool_panel.xml"
>      sanitize_whitelist_file: "/galaxy/server/config/mutable/sanitize_whitelist.txt"
>      tool_config_file: "{{.Values.persistence.mountPath}}/config/editable_shed_tool_conf.xml,/galaxy/server/config/custom_tool_conf.xml"
>      tool_data_table_config_path: "{{ .Values.cvmfs.main.mountPath }}/config/shed_tool_data_table_conf.xml,{{.Values.cvmfs.data.mountPath}}/managed/location/tool_data_table_conf.xml,{{.Values.cvmfs.data.mountPath}}/byhand/location/tool_data_table_conf.xml"
>      tool_dependency_dir: "{{.Values.persistence.mountPath}}/deps"
>      builds_file_path: "{{.Values.cvmfs.data.mountPath}}/managed/location/builds.txt"
>      datatypes_config_file: "{{ .Values.cvmfs.main.mountPath }}/config/datatypes_conf.xml"
>      containers_resolvers_config_file: "/galaxy/server/config/container_resolvers_conf.xml"
>      workflow_schedulers_config_file: "/galaxy/server/config/workflow_schedulers_conf.xml"
>      build_sites_config_file: "/galaxy/server/config/build_sites.yml"
>    ```
>    {% endraw %}
>
> 3. Now, let's upgrade the chart to use `custom_tool_conf.xml` and
>    `galaxy.yml` by running the `helm upgrade` command.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values --set-file "configs.custom_tool_conf\.xml"=custom_tool_conf.xml --set-file "configs.galaxy\.yml"=configs/galaxy.yml galaxy galaxy/galaxy
>    ```
>    {% endraw %}
>
>    Note the `--reuse-values` flag, which instructs Helm to reuse any
>    previously set values, and apply the new ones on top.  The `--set-file`
>    option will set the value of the `configs.custom_tool_conf.xml`
>    key in your values file to the contents of the specified file, as a text
>    string. Each file under `configs` key in `values.yaml` is automatically
>    mapped into Galaxy's `config` directory within the running container.
>
> 4. Notice that while the chart is upgrading, the existing version continues
>    to function. The changeover will occur when the new container is online
>    and signals readiness to Kubernetes by responding to web requests on
>    the relevant port. Log into the Kubernetes dashboard and watch the logs
>    as the new pods come online.
>
> 5. List the installed helm charts again and note that the revision of the chart
>    has changed. These revisions are useful because it allows us to rollback
>    our changes if they are incorrect. This will be covered in a later section.
>
>    {% raw %}
>    ```bash
>    helm list
>    NAME  	REVISION	UPDATED                 	STATUS  	CHART                 	APP VERSION	NAMESPACE
>    cvmfs 	1       	Wed Jun 26 14:47:46 2019	DEPLOYED	galaxy-cvmfs-csi-1.0.1	1.0        	cvmfs
>    galaxy	2       	Wed Jun 26 14:51:17 2019	DEPLOYED	galaxy-3.0.0          	v19.05     	default
>    ```
>    {% endraw %}
>
> 6. Let's now exec into the running container and check where the files were
>    mapped in. First, let's get a list of running pods.
>
>    {% raw %}
>    ```bash
>    kubectl get pods
>    NAME                          READY   STATUS    RESTARTS   AGE
>    galaxy-galaxy-postgres-0      1/1     Running   0          2d6h
>    galaxy-job-69864b6797-zs5mn   1/1     Running   0          2d6h
>    galaxy-web-7568c58b94-jzkvm   1/1     Running   0          2d6h
>    ```
>    {% endraw %}
>
>    Exec into the web pod by running:
>    {% raw %}
>    ```bash
>    kubectl exec -it galaxy-web-7568c58b94-jzkvm /bin/bash
>    ```
>    {% endraw %}
>
>    Now run `ls /galaxy/server/config/` and note that the `galaxy.yml` contains
>    the content that you've provided and that `custom_tool_conf.xml` has also been
>    mapped into the config folder. In this same way, any of Galaxy's config
>    files can be overridden by simply mapping in the relevant file into the
>    config folder.
{: .hands_on}

## Setting the admin user and changing the brand
Next, we will set the admin user and change the brand in `galaxy.yml`. We will
rollback our change to understand how Helm manages configuration.

> <hands-on-title>Setting admin user and changing the brand</hands-on-title>
>
> 1. Modify the following entries in your `galaxy.yml`. Make sure to add these
>    keys under the `galaxy:` section of the file.
>
>    {% raw %}
>    ```
>    brand: "Hello World"
>    admin_users: "admin@mydomain.com"
>    ```
>    {% endraw %}
>
> 2. Now, let’s upgrade the chart to apply the new configuration.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values --set-file "configs.galaxy\.yml"=galaxy.yml galaxy galaxy/galaxy
>    ```
>    {% endraw %}
>
> 3. Inspect the currently set Helm values by:
>
>    {% raw %}
>    ```bash
>    helm get values galaxy
>    ```
>    {% endraw %}
>
> 4. List the installed Helm charts again and note that the revision of the chart has changed as expected.
>
>    {% raw %}
>    ```bash
>    helm list
>    NAME  	REVISION	UPDATED                 	STATUS  	CHART                 	APP VERSION	NAMESPACE
>    cvmfs 	1       	Wed Jun 26 14:47:46 2019	DEPLOYED	galaxy-cvmfs-csi-1.0.1	1.0        	cvmfs
>    galaxy	3       	Wed Jun 26 14:51:17 2019	DEPLOYED	galaxy-3.0.0          	v19.05     	default
>    ```
>    {% endraw %}
>
> 5. Let’s now roll back to the previous revision.
>
>    {% raw %}
>    ```bash
>    helm rollback galaxy 2
>    ```
>    {% endraw %}
>
>    Use `helm get values` again to observe that the values have reverted to
>    the previous revision. After a short while, once the new container is up
>    and running, Kubernetes will automatically switch over to it and you can
>    see that the previous configuration has been restored.
>
{: .hands_on}

# Scaling Galaxy
In Galaxy deployment on Kubernetes, there are two containers by default, one
web handler and one job handler. We will now look at how these can be scaled.

> <hands-on-title>Setting admin user and changing the brand</hands-on-title>
>
> 1. View the `values-cvmfs.yaml` file in the Galaxy Helm chart and note down the
>    number of web and job handlers.
>
>    {% raw %}
>    ```yaml
>    webHandlers:
>        replicaCount: 1
>    jobHandlers:
>        replicaCount: 1
>    ```
>    {% endraw %}
>
> 2. Let’s increase the number of web handlers by simply setting new values for the number of replicas.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values --set webHandlers.replicaCount=2 galaxy galaxy/galaxy
>    ```
>    {% endraw %}
>
> 3. Check whether the new replicas have been created.
>
>    {% raw %}
>    ```bash
>    kubectl get pods
>    NAME                          READY   STATUS    RESTARTS   AGE
>    galaxy-galaxy-postgres-0      1/1     Running   0          2d9h
>    galaxy-job-5cc75c6588-8dsbg   1/1     Running   0          7m13s
>    galaxy-web-7c9576cf89-49nlm   1/1     Running   0          7m13s
>    galaxy-web-7c9576cf89-r6rcj   0/1     Running   0          9s
>    ```
>    {% endraw %}
>
> 4. Follow the pod logs and check whether the new handler is receiving web requests
>    as expected.
>
>    {% raw %}
>    ```bash
>    kubectl logs -f galaxy-web-7c9576cf89-r6rcj
>    ```
>    {% endraw %}
>
>    You will notice that Kubernetes automatically load balances requests between the
>    available web handler replicas in a round-robin fashion.
>
{: .hands_on}

# Testing Kubernetes resilience
To observe how Kubernetes handles failures, let’s exec into a running container and
manually kill a process to simulate a possible process failure. Kubernetes
continuously monitors running containers, and attempts to bring the environment
back to the “desired” state. The moment it notices a failure, it will respawn
a new pod to replace the failed one. Typically, a Kubernetes container will also
have a [liveness probe][k8sliveness] defined. A liveness probe can be an http
request to a port, or even a manually executed shell script, which will test
whether the relevant container is healthy, and if not, Kubernetes will immediately
provision a new replacement.

> <hands-on-title>Handling failures</hands-on-title>
>
> 1. First list the available pods.
>
>    {% raw %}
>    ```bash
>    kubectl get pods
>    NAME                          READY   STATUS    RESTARTS   AGE
>    galaxy-galaxy-postgres-0      1/1     Running   0          2d9h
>    galaxy-job-5cc75c6588-8dsbg   1/1     Running   0          14m
>    galaxy-web-7c9576cf89-49nlm   1/1     Running   0          14m
>    galaxy-web-7c9576cf89-r6rcj   0/1     Running   1          7m36s
>    ```
>    {% endraw %}
>
>    Then exec into one:
>    {% raw %}
>    ```bash
>    kubectl exec -it galaxy-web-7c9576cf89-r6rcj /bin/bash
>    ```
>    {% endraw %}
>
> 2. Now kill the main container process.
>
>    {% raw %}
>    ```bash
>    kill 1
>    ```
>    {% endraw %}
>
> 3. If we run `kubectl get pods`, we can notice how Kubernetes immediately
>    starts a new pod to replace the failed one, bringing the environment back to
>    the desired state. Take a look at the liveness probe defined for the galaxy
>    web container in the helm chart source code
>    (`templates/deployment-web.yaml`).
>
{: .hands_on}

# Deleting Galaxy
Finally, let’s take a look at how we can uninstall Galaxy and remove all
related containers.

> <hands-on-title>Deleting Galaxy</hands-on-title>
>
> 1. To permanently delete the Galaxy release, run:
>
>    {% raw %}
>    ```bash
>    helm delete --purge galaxy
>    ```
>    {% endraw %}
>
>    The purge flag instructs helm to permanently remove galaxy from its history.
>
> 2. Use `kubectl get pods` to verify that the pods have been deleted.
>
{: .hands_on}

# Next Steps
This tutorial provided an overview of some common Galaxy administration tasks.
Advanced customizations would include running custom shell scripts on Galaxy
startup to perform additional tasks, running additional containers on
startup, administering and managing storage, building custom Galaxy containers
with desired modifications etc. For more info on some of these
topics, take a look at the [Galaxy Helm chart] repository as well as
[other tutorials](../..) tagged with _kubernetes_. Also, feel free to reach out
on Gitter: [https://gitter.im/galaxyproject/FederatedGalaxy][fedG].


[fedG]: https://gitter.im/galaxyproject/FederatedGalaxy
[k8sliveness]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
[Galaxy Helm chart]: https://github.com/galaxyproject/galaxy-helm


topic: admin
tutorial: ansible
title: Ansible
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: []
tags_json: ["ansible"]
content_hash: a8483daffdd143707ade69d8c78f7c84
last_modified: 2026-05-15T22:28:10.005210
workflows_json: []
content:
# Overview

In this tutorial we will briefly cover what Ansible is and how to understand what it does. This guide is not meant to make you an expert on Ansible, but perhaps give you enough that you can debug broken roles and modify them to suit your needs. Maybe even [install Galaxy using Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}) or contribute to the [Galaxy Project's Ansible roles](https://github.com/galaxyproject?q=ansible).

This will be a very practical training with emphasis on looking at examples from modules and becoming self sufficient.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/admin-testing.md %}

# What is Ansible?

Ansible runs commands on local or remote computers. It can move files around, create files from templates, and run command line tools. Primarily used for system administration tasks at scale. It has a push model rather than a pull model like Puppet. If you've used Puppet, Ansible doesn't evaluate what changes need to be made and make those, it just runs through all of commands every time.

Some terms that you should know first:

Inventory file
:    An Ansible-specific file that defines the systems ("hosts") and groups of hosts on which Ansible should operate.

Ansible module
:    A piece of Python code that converts some parameters into an invocation. An example would be the `command` module which converts parameters like `command: ls` into a command line that is executed. There are pre-built modules for just about everything.

Task
:    A call to an Ansible module that should be executed and the configuration for this module.

Role
:    A folder containing some tasks, templates, files, and default values for variables, with a predefined directory structure. People share roles on ["Ansible Galaxy"](https://galaxy.ansible.com/) (repository unrelated to the Galaxy Project).

Playbook
:    A YAML file listing a set of tasks and/or roles that should be applied to a group of hosts.

Vault
:    An encrypted YAML file. You put your secrets here and then you can use them in tasks, roles and playbooks.

Looking at each of these briefly:

## Inventory file

```ini
[webservers]
192.0.2.0
192.0.2.1

[databases]
192.0.2.3 ansible_user=root
```

Here we've defined two groups of computers, `webservers` and `databases`. `ansible_user` is used to
specify which user to connect with (you need to specify that if you SSH in with a username different
than your current local user account’s name).

As you can see, in this inventory we connect to multiple remote machines. This is common practice;
Ansible playbooks are stored on your laptop or a build server, and from there Ansible connects out to the
remote machines that are managed. The advantage of running remotely is that you can manage dozens of
machines simultaneously, rather than just the local machine. This scaling out to N machines is one of
the strengths of Ansible.

Additionally, playbooks are often stored in Git or other version control repositories to ensure that
if anything happens to the infrastructure you manage, you'll still have a copy of the information
required to rebuild everything with Ansible.

> <details-title>Ansible Inventory Documentation (Do you connect with a different user? Password? SSH key?)</details-title>
> For more advanced features of the inventory file, check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/intro_inventory.html).
{: .details}

## Roles

We can look at the [ansible-cvmfs role](https://github.com/galaxyproject/ansible-cvmfs) as an example for the layout of a role:

```
├── defaults
│   └── main.yml
├── files
│   ├── cvmfs_remount_sync.c
│   ├── cvmfs_remount_sync.centos_6
│   ├── cvmfs_remount_sync.centos_7
│   └── ...
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── tasks
│   ├── apache.yml
│   ├── ...
│   ├── main.yml
│   └── ...
├── templates
│   ├── ...
│   └── stratum1_squid.conf.j2
└── vars
    ├── ...
    ├── main.yml
    └── ...
```

These are the folders that are included in many complex roles. Simpler roles will often not need all of the folders.

Folder    | Usage
--------- | ------
defaults  | Default values for variables the user can set (e.g. "version of software to install").
files     | These are files which should be copied as-is over to the remote location.
handlers  | This is typically used for restarting processes.
meta      | Only needed to publish your role to Ansible Galaxy (repository unrelated to the Galaxy Project).
tasks     | **Always start reading here**. This is the most important folder and the best place to start when trying to understand what an unfamiliar role does. Anything that is loaded will be referenced here (e.g. variables to load, handlers, files, templates).
templates | Files that are templated out with variables before being copied.
vars      | Default values for variables the user should normally not change (e.g. name of a package in different Linux distributions).

> <details-title>Ansible Role Documentation</details-title>
>
> For more information check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/playbooks_reuse_roles.html).
{: .details}

## Modules and Tasks

A `tasks/main.yml` file calls multiple Ansible modules to accomplish its goal. A typical tasks file looks like:

```yaml
---
- name: Download cvmfs_preload utility when desired
  get_url:
    url: https://cvmrepo.web.cern.ch/cvmrepo/preload/cvmfs_preload
    dest: "{% raw %}{{ cvmfs_preload_path }}/cvmfs_preload{% endraw %}"
    owner: root
    group: root
    mode: 755
  when: cvmfs_preload_install | bool

- name: Ensure AutoFS is enabled + running
  service:
    name: autofs
    enabled: yes
    state: started
```

Here we have two tasks. Each has a `name` that will be shown to the person running the playbook.

The first example invokes the [`get_url`](https://docs.ansible.com/ansible/2.9/modules/get_url_module.html) module with 5 arguments (`url`, `dest`, `owner`, `group` and `mode`). This will download a file from the location indicated in `url` to the directory specified by `dest`, change user and group ownerships to `root`, and change file permissions to `755` (as assigned by the `mode` argument). The `dest` parameter uses a [Jinja2](http://jinja.pocoo.org/) template to evaluate the value of the variable `cvmfs_preload_path`. We can [grep through](https://github.com/galaxyproject/ansible-cvmfs/search?q=cvmfs_preload_path) the repository and see that `defaults/main.yml` sets that to `/usr/bin` by default. We can override this if we need, we'll come back to that later. The first task also has a `when` condition to ensure it only runs when the `cvmfs_preload_install` variable is set.

The second invokes the [`service`](https://docs.ansible.com/ansible/2.9/modules/service_module.html) module. The arguments to this one are quite legible and the functionality can be inferred from the names for the most part: The service `name: autofs` will be `enabled` and its `state` should be `started`.

[Many modules](https://docs.ansible.com/ansible/2.9/modules/modules_by_category.html) are available for you to use.

### Stylistic Choices

Ansible accepts two ways to format tasks:

YAML style:

```yaml
- package:
    name: "{% raw %}{{ package_name }}{% endraw %}"
    state: "{% raw %}{{ package_state }}{% endraw %}"
```

And an inline style.

```yaml
{% raw %}- package name={{ package_name }} state={{ package_state }}{% endraw %}
```

Some groups prefer one style or another. You can mix both of these but you probably shouldn't. In the YAML style the templated value needs to be quoted if the value after the colon starts with a `{` (see [this page](https://docs.ansible.com/ansible/9/playbook_guide/playbooks_variables.html#when-to-quote-variables-a-yaml-gotcha) for more details ). The inline style does not require quoting of templated values.

## Playbooks

```yaml
- name: CVMFS
  hosts: webservers
  vars:
    cvmfs_numfiles: 4096
  roles:
    - galaxyproject.cvmfs
```

This is a quite minimal playbook. It selects a `hosts` group named `webservers`, overrides the variable `cvmfs_numfiles`, and then says the following set of roles will be executed for this group of hosts. Ansible makes it easy to collect tasks that should apply to a group of hosts and run a playbook for all of those hosts. Some good uses of this are things like ensuring a certain set of users are installed on all of your managed machines, or using one of the package autoupdating roles to make sure your machines are up-to-date.

> <details-title>Ansible Playbook Documentation</details-title>
>
> For more information check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/playbooks_intro.html).
{: .details}

### Philosophies

Different groups use playbooks differently. Here is a non-exhaustive list of ways that the author has seen playbooks used:

Strategy                                | Use Cases
--------------------------------------- | ---
A single playbook that does everything  | Works well in cloud use cases where you would rather terminate a VM and just restart from nothing. If 100% of the tasks are idempotent (can be re-run without causing problems) then this can also work well for ensuring all machines that are managed are meeting your compliance standards.
Playbooks that execute one-off commands | Works well when you have commands that cannot safely be re-run (e.g. you are managing a repository on a remote machine and executing a 'commit' action.) Also works if you have multiple people who are responsible for managing a machine but you want to ensure exactly the same commands are run each time a task needs to be done, with no variation.

## Variables

There are a bunch of places variables can be set. [The list is ridiculous.](https://docs.ansible.com/ansible/2.9/user_guide/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable) Try and find a convention within your group and stick to that to help manage the chaos.

There are some special places you can put variables that will be loaded automatically:

- `group_vars/<group_name>.yml`: if you run a playbook which has `hosts: webservers`, and `group_vars/webservers.yml` exists, it will be loaded.
- `host_vars/<host_name>.yml`: if you run a playbook and it affects host (e.g.) `api01`, and `host_vars/api01.yml` exists, it will be loaded automatically.

For a real-life example, UseGalaxy.eu [generally attempts](https://github.com/usegalaxy-eu/infrastructure-playbook) to put variables in an appropriately named `group_vars/` file.

## Vaults

[`ansible-vault`](https://docs.ansible.com/ansible/2.9/user_guide/vault.html) is a super useful tool that lets you encrypt secrets for your group using a pre-shared key. This allows you to commit ALL of your Ansible playbooks and variables to source control, without the concern of leaking secrets. E.g. [UseGalaxy.eu](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/secret_group_vars/all.yml)'s vault file. The encrypted version is not very interesting to look at, but it is mostly to show that we confidently place an encrypted copy of our secrets online, under configuration management. This has made our life a lot easier.


# Your First Playbook and First Role

The above introduction was certainly not enough for you to feel confident in Ansible, so we will now build a small role and a playbook to run this role.

> <warning-title>Safety First</warning-title>
>
> Many of the things you can do with Ansible can be quite dangerous. As dangerous as normally being at the command line, but scaled across N machines. Be very careful with the changes you plan to make.
> Ansible provides some flags which can help you identify changes before they're made to production systems:
>
> **`--diff`**
> :    This will show the difference whenever a file is changed.
>
> **`--check`**
> :    Will do a dry-run of the playbook, attempting to show you which tasks will execute, which files will be updated. Not all actions can be predicted because commands are not run, if a downstream task depends on the result of a command execution task, Ansible has no way of knowing whether or not it will execute.
>
> In this tutorial we will write to files in `/tmp` as that is a *relatively* safe thing to do. The training material community does not have the resources to test this tutorial across all of the platforms you might want to run it on. Additionally we do not want to be responsible if you accidentally cause permanent damage by following this tutorial.
>
{: .warning}


> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have [Ansible installed](https://docs.ansible.com/ansible/2.9/installation_guide/intro_installation.html) on the machine you will run this tutorial from
>
>    > <comment-title>Running Ansible on remote machine</comment-title>
>    > It is possible to have Ansible installed on your laptop/local machine and run it against some remote hosts as well. We will **not** do that in this training.
>    {: .comment}
>
> 2. Your `ansible` version is `>=2.7`, you can check this by running `ansible --version`
>
{: .comment}


## A Basic Role

> <hands-on-title>Setting up our workspace</hands-on-title>
>
> 1. In this training we will run Ansible on the machine it will modify. This is not best practice, but it is convenient for trainings. You should probably run this in a VM either on the Cloud or in VirtualBox or similar
>
>    All of the steps are the same, no matter which machine Ansible will manage and where you run it. The only difference is the connection setup
>
> 3. **Create a directory named `intro` and `cd` into it.**
>
>    It's good practice to keep your deployments separated.
>
> 4. Create your inventory file (named `hosts`) in this folder
>
>    1. We will call our group `my_hosts`
>
>    2. Create [an inventory file](https://docs.ansible.com/ansible/2.9/user_guide/intro_inventory.html) with the group `my_hosts` and `localhost ansible_connection=local`, which tells Ansible to not use SSH, and just use the local connection. Additionally, you should explicitly set the `ansible_user` variable to the username to use when connecting to the server. Ansible has changed its behaviour over time regarding whether or not `ansible_user` is defined, and it is most effective to define it explicitly even when it can sometimes be inferred.
>
>       > <solution-title></solution-title>
>       > The file should look like:
>       >
>       > ```ini
>       > [my_hosts]
>       > localhost ansible_connection=local ansible_user=ubuntu
>       > ```
>       {: .solution }
>
>       > <details-title>For your own infrastructure, do you connect as a different user? Password? SSH key?</details-title>
>       > For more advanced features of the inventory file, check out [the official documentation on this topic](https://docs.ansible.com/ansible/2.9/user_guide/intro_inventory.html).
>       {: .details}
>
> 5. Create the roles directory, your role, and the tasks folder: `mkdir -p roles/my-role/tasks/`
>
> 6. Create a YAML file in that directory, `roles/my-role/tasks/main.yml` and open it for editing
>
> 7. Define a `copy` task like below:
>
>    ```yaml
>    ---
>    - name: Copy a file to the remote host
>      copy:
>        src: test.txt
>        dest: /tmp/test.txt
>    ```
>
>    You can read about all of the parameters available to the [`copy`](http://docs.ansible.com/ansible/2.9/modules/copy_module.html) module in Ansible's documentation.
>
>    > <details-title>Ansible Module Documentation</details-title>
>    > You can usually find a module for most commands you will run in a shell, e.g. by searching the internet for "ansible copy file to server" or "ansible restart service". If you cannot find a module that does it, there is the [`command`](http://docs.ansible.com/ansible/2.9/modules/command_module.html) module, but this should be avoided if possible. Expect to have a browser session with 10-30 different Ansible module documentation tabs if you work with Ansible regularly, no one remembers what arguments are available to every module.
>    >
>    {: .details }
>
> 8. Create a `roles/my-role/files` folder, and within it a file named `test.txt`, containing the content "Hello, Galaxy 🚀"
>
> 9. This is a complete role by itself and will copy the file `test.txt` from the `roles/my-role/files/` folder over to the target host and place it in `/tmp`.
>
> 10. Create and open `playbook.yml` file for editing in the `intro` directory you created. Place the following content in there:
>
>     ```yaml
>     ---
>     - hosts: my_hosts
>       roles:
>         - my-role
>     ```
>
>     > <question-title></question-title>
>     >
>     > How does your file tree look now? Use `find` or `tree`.
>     >
>     > > <solution-title></solution-title>
>     > >
>     > > ```
>     > > .
>     > > ├── hosts
>     > > ├── playbook.yml
>     > > └── roles
>     > >     └── my-role
>     > >         ├── files
>     > >         │   └── test.txt
>     > >         └── tasks
>     > >             └── main.yml
>     > > ```
>     > >
>     > {: .solution }
>     {: .question}
>
> 11. Run the playbook:
>
>     > > <code-in-title>Bash</code-in-title>
>     > > ```bash
>     > > ansible-playbook -i hosts playbook.yml
>     > > ```
>     > {: .code-in}
>     >
>     > > <code-out-title>Bash</code-out-title>
>     > > ```
>     > > PLAY [my_hosts] ****************************************************************
>     > > TASK [Gathering Facts] *********************************************************
>     > > ok: [localhost]
>     > > TASK [my-role : Copy a file to the remote host] ********************************
>     > > changed: [localhost]
>     > > PLAY RECAP *********************************************************************
>     > > localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
>     > > ```
>     > {: .code-out}
>     {: .code-2col}
>
> 12. Login to the appropriate host (unless your host was `localhost`) and `cat /tmp/test.txt` to see that the change was made.
>
{: .hands_on}

Now that you've done this, here are some starting points for exploration:

- Add more hosts, watch as Ansible executes over all of them in parallel.
- Identify a task you do regularly, e.g. restarting a service. Find the Ansible service module and add that to your playbook.

> <comment-title>Too Many Cows?</comment-title>
> If you've installed the `cowsay` tool, Ansible (for some reason) will take advantage of that to output a lot of the output with cowsay. To disable this you can `export ANSIBLE_NOCOWS=1` (Remember that exporting will only last as long as the current invocation of your terminal does, so consider adding this to your user profile if you wish to keep cowsay installed and still have legible output.)
>
> ```
>  ________________
> < PLAY [my_hosts] >
>  ----------------
>         \   ^__^
>          \  (oo)\_______
>             (__)\       )\/\
>                 ||----w |
>                 ||     ||
>  ________________________
> < TASK [Gathering Facts] >
>  ------------------------
>         \   ^__^
>          \  (oo)\_______
>             (__)\       )\/\
>                 ||----w |
>                 ||     ||
> ```
{: .comment}


## Facts

In the last step of the last hands-on, you ran your playbook. The first `TASK` that was executed was not one that you had written, it reads:

```
TASK [Gathering Facts] *********************************************************
ok: [localhost]
```

The [`setup`](https://docs.ansible.com/ansible/2.9/modules/setup_module.html) module runs by default for every host and gathers facts about the host.

> <hands-on-title>The Setup Module</hands-on-title>
>
> 1. Run the command `ansible -i hosts -m setup my_hosts | less`.
>
>    The `my_hosts` at the end of the command refers to the group we defined in our `hosts` inventory file.
>
> 2. Investigate the output. See what sort of information is made available to you.
>
>    > <question-title></question-title>
>    >
>    > 1. What variable stores the OS name?
>    > 2. Where are all of the places that you can find your machine's IP (v4) addresses?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The OS name is stored in `ansible_distribution`. We saw `ansible_os_family` used above in the `ansible-cvmfs` role. You can use these variables if you are writing a generic role but packages or commands are named different on different operating systems.
>    > >
>    > > 2. Ansible stores network information in quite a few places, sometimes one place is more convenient or more correct to use:
>    > >
>    > >    - `ansible_all_ipv4_addresses`
>    > >    - `ansible_default_ipv4.address`
>    > >    - `ansible_<interface_name>.ipv4`
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}


## Templates

Templates give you greater control over the files you are deploying to the target system. If you need to deploy a file to multiple hosts, but configure it differently on each host, you should use templates. For instance, deploying a service that should only listen on the correct IP address for that host would be a good use case for templates. All of the facts you discovered in the previous hands-on are available to you to use in templates, `when` statements (like the [ansible-cvmfs example we saw earlier](#modules-and-tasks)). Additionally all of the variables you've defined are available as well.

> <details-title>Template Syntax</details-title>
> Templates end with the `.j2` suffix and use Jinja2 syntax. If you are not familiar with it, you should [read about it](http://jinja.pocoo.org/docs/2.10/templates/) first, before moving on with the tutorial. Ansible fills the templates with variable values and copies the file to its remote destination without the `.j2` suffix.
{: .details}

> <hands-on-title>Variables and Templates</hands-on-title>
>
> 1. Create the directories: `roles/my-role/templates`, `roles/my-role/defaults`
>
> 2. Create and edit a file for your role's variables, `roles/my-role/defaults/main.yml`
>
> 3. Insert the following content:
>
>    ```yaml
>    ---
>    server_name: Cats!
>    ```
>
>    This will define a variable `server_name` that can then be used in templates.
>
> 4. Create and edit `roles/my-role/templates/test.ini.j2`
>
> 5. Insert the following content:
>
>    {% raw %}
>    ```ini
>    [example]
>    server_name = {{ server_name }}
>    listen = {{ ansible_default_ipv4.address }}
>    ```
>    {% endraw %}
>
> 6. Edit `roles/my-role/tasks/main.yml` and append a new task to the end to template this file:
>
>    ```yaml
>    - name: Template the configuration file
>      template:
>        src: test.ini.j2
>        dest: /tmp/test.ini
>    ```
>
> 7. Run the playbook again.
>
>
> 8. Check the contents of `/tmp/test.ini`
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat /tmp/test.ini
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > >
>    > > The file should look like:
>    > > ```
>    > > [example]
>    > > server_name = Cats!
>    > > listen = 192.168.0.2
>    > > ```
>    > > Where the last line has the machine's IP address.
>    > {: .code-out}
>    {: .code-2col}
>
>    Now that this has worked successfully, we will setup a `group_vars` folder
>    to show how a person using `my-role` would override the `server_name` variable.
>
> 9. Create the folder `group_vars/` (in the root of your directory)
>
> 10. Create and edit `group_vars/my_hosts.yml`
>
> 11. Insert the following:
>
>     ```yaml
>     ---
>     server_name: Dogs!
>     ```
>
>     {% snippet topics/admin/faqs/ansible-connection.md %}
>
> 12. Run the playbook again, but imagine you are worried about this change, and supply the `--check --diff` flag to see what changes are made before committing to make them.
>
>     > <code-in-title>Bash</code-in-title>
>     > ```bash
>     > ansible-playbook -i hosts playbook.yml --check --diff
>     > ```
>     {: .code-in}
>
>     {% snippet topics/admin/faqs/ansible_diff.md %}
>
>     > <code-out-title></code-out-title>
>     >
>     > ```
>     > PLAY [my_hosts] ****************************************************************
>     > TASK [Gathering Facts] *********************************************************
>     > ok: [localhost]
>     > TASK [my-role : Copy a file to the remote host] ********************************
>     > ok: [localhost]
>     > TASK [my-role : Template the configuration file] *******************************
>     > --- before: /tmp/test.ini
>     > +++ after: /home/hxr/.ansible/tmp/ansible-local-1906887dr2u6j8n/tmptx9pdelg/test.ini.j2
>     > @@ -1,3 +1,3 @@
>     >  [example]
>     > -server_name = Cats!
>     > +server_name = Dogs!
>     >  listen = 192.168.0.25
>     > changed: [localhost]
>     > PLAY RECAP **********************************************
>     > localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
>     > ```
>     > Here you can see that the `server_name` value will be changed. Despite Ansible reporting `changed=1`, no changes have actually been applied to the system.
>     {: .code-out}
>
> 13. Run the playbook again, without the `--check` flag to apply your changes.
{: .hands_on}

> <comment-title>Ansible Variable Templating</comment-title>
> In this hands-on we used some templated variables. We defined them in a template, but they are also commonly used in the group variables file. Our templated variable looked like: {% raw %}`listen = {{ ansible_default_ipv4.address }}`{% endraw %}.
>
> It is common to see things like this in Ansible roles:
>
> ```yaml
> root_dir = /opt/my-app
> config_dir = {% raw %}"{{ root_dir }}/config"{% endraw %}
> ```
>
> When Ansible runs:
>
> 1. It collects variables defined in group variables and other places
> 2. The first task for each machine is the [`setup` module](https://docs.ansible.com/ansible/2.9/modules/setup_module.html) which gathers facts about the host, which are added to the available variables
> 3. When multiple roles execute in a playbook:
>    1. Their defaults are added to the set of variables (the group variables having precedence over these variables)
>    2. They can also dynamically define more variables which may not be set until that role is run
> 4. Before use (in templates, commands, etc.), variables are resolved to their final value
>
>
{: .comment}

# Ansible Galaxy

Now that you've built a small role, you can imagine that building real roles that manage the full installation of a piece of software are not simple things. Ansible Galaxy (repository unrelated to the Galaxy Project) is the answer here. Many roles for common administration tasks, and software installation and setup are readily available on Ansible Galaxy.

**Warning**: This will install Memcached on the target machine.

> <hands-on-title>Installing a module using ansible-galaxy</hands-on-title>
>
> 1. Install the geerlingguy.memcached role with ansible-galaxy into your roles folder:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > ansible-galaxy install \
>    > >     -p roles/ geerlingguy.memcached
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > This will install the new role into your `roles` folder, alongside your own role.
>    > > ```
>    > > - downloading role 'memcached', owned by geerlingguy
>    > > - downloading role from https://github.com/geerlingguy/ansible-role-memcached/archive/1.1.0.tar.gz
>    > > - extracting geerlingguy.memcached to /home/ubuntu/ansible/intro/roles/geerlingguy.memcached
>    > > - geerlingguy.memcached (1.1.0) was installed successfully
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
> 2. Edit your playbook.yml and add the role `geerlingguy.memcached` at the bottom, after `my-role`
>
> 3. Run the playbook
>
>    > <question-title></question-title>
>    >
>    > Did something go wrong?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > Since you have been running the playbook as a non-root user (or at least you should have been!), the step to install a package fails.
>    > > The solution to this is to set `become: true`.
>    > >
>    > > 1. Edit your playbook.yml
>    > >   - add `become: true` just below `hosts: my_hosts`
>    > > 2. Run the playbook again
>    > >
>    > > `become` causes Ansible to attempt to become a different user (using sudo/su/whatever is appropriate), by default this is `root`. If you want to become a different user, just set `become_user`. Beware, the user should be able to privilege escalate without a password prompt. Otherwise when you execute the playbook you should set `--ask-become-pass`, using the privilege escalation password for that host.
>    > >
>    > > > <details-title>Ansible Become</details-title>
>    > > > See the [documentation](https://docs.ansible.com/ansible/2.9/user_guide/become.html) if you need to control this behaviour differently. `become` can be set either at the task level or the playbook level.
>    > > >
>    > > {: .details}
>    > >
>    > > > <tip-title>Should I use sudo or directly login as root?</tip-title>
>    > > > This is often site-specific policy. If your site doesn't have a dictated policy, then it's generally preferable to not allow direct login as root, and login as a separate user.
>    > > {: .tip}
>    > >
>    > >
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

## Choosing a Role

Picking the best role for a task from Ansible Galaxy is not always a trivial task. Sometimes there will only be a single role doing what you need. Other times you'll have to choose between 20 different roles that all look more or less the same. Here are some tips to guide you in identifying appropriate and well-written roles:

- The name should match the software you are using (I.e. ignore a role named `stackstorm` when you are trying to set up `rabbitmq`. Ansible Galaxy does not have perfect search.)
- `geerlingguy` wrote a huge number of roles for many pieces of standard software. If there is a role from this person, this is usually a safe choice. In the same way we recommend `galaxyproject` and `usegalaxy_eu` roles.
- Check the GitHub readme of each role you consider using. Look for:
  - Extensive documentation of all of the variables, their default values, and how they behave.
  - An example playbook using the role
- A large number of downloads is a good sign that other people found it useful

These are usually good proxies for quality, but do not treat them as strict rules. For an example of a role meeting many of these qualities, [`ansible-cvmfs`](https://github.com/galaxyproject/ansible-cvmfs) is good; the variables are well documented and there are example playbooks that you can (more or less) copy-and-paste and run.

Sometimes a role will accomplish 95% of what you need to do, but not everything. Once you have installed the role with `ansible-galaxy install`, you can edit it locally to make any changes. In an ideal world you would contribute this back, but this is not always a high priority. Many projects copy roles directly into their repositories, e.g. [galaxyproject](https://github.com/galaxyproject/infrastructure-playbook/tree/master/roles) and [usegalaxy.eu](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master/roles)

{% snippet topics/admin/faqs/ansible_role-properties.md %}

# Ansible Vault

Now that you have a small role built up, you might start thinking about deploying larger and more complex services and infrastructure. One last common task we want to cover here is the inclusion of secrets. Ansible Vault is really useful to include encrypted secrets in your playbook repository.

> <hands-on-title>Setting up secrets</hands-on-title>
>
> 0. Run `mkdir -p secret_group_vars`
>
> 1. Now we'll create a strong password: `openssl rand -base64 24 > vault-password.txt`
>
> 2. Run `ansible-vault create secret_group_vars/all.yml --vault-password-file=vault-password.txt`
>
>    This will open `secret_group_vars/all.yml` in your text editor.
>
> 3. In this file, enter the following contents and save it:
>
>    ```yaml
>    apikey: super-secret-api-key-wow!
>    ```
>
>    > <question-title></question-title>
>    >
>    > How does your file look? Is it readable? Run `cat secret_group_vars/all.yml`
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file will look like this, it is encrypted by Ansible Vault with AES256 encryption.
>    > > ```
>    > > $ cat secret_group_vars/all.yml
>    > > $ANSIBLE_VAULT;1.1;AES256
>    > > 64373665366130333437393639343534653134346538636239393363373062393830653333323966
>    > > 3134333366363130326139323162323131643763336236320a393262303938316262643764323862
>    > > 36393161666663353231366336613838633866323230303031313465646333613862363264323139
>    > > 3263383530626262370a666139666462663938343531656432353239346532316630366165376566
>    > > 34313765353766666330366632303836353863396430343264303032363739666139383830323565
>    > > 6133663637356331613062353834646561653366386665623930
>    > > ```
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. Use the new variable in our `.ini` file from earlier. Edit `roles/my-role/templates/test.ini.j2` and add the line `{% raw %}apikey = {{ apikey }}{% endraw %}`
>
> 5. Add encrypted variable file to `playbook.yml`
>
>    ```yaml
>    - hosts: my_hosts
>      ...
>      vars_files:
>       - secret_group_vars/all.yml
>    ```
>
> 6. Tell ansible where to find the decryption file. Create file `ansible.cfg` with content
>
>    ```yaml
>    [defaults]
>    vault_password_file=vault-password.txt
>    ```
> 7. Run the playbook
>
> 8. Check the contents of `/tmp/test.ini`
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat /tmp/test.ini
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > The file should look like:
>    > >
>    > > ```ini
>    > > [example]
>    > > server_name = Dogs!
>    > > listen = 192.168.0.2
>    > > apikey = super-secret-api-key-wow!
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
{: .hands_on}

In real life scenarios where you are sharing your playbooks publicly, be sure to encrypt all secrets from the start (or remove any secrets from the git history if you ever committed them in plain text). If you are storing your vault password in a file, remember to add it to your `.gitignore` (or VCS appropriate file.)

# Other Stuff

Ansible has a huge array of features and we can't cover them all. Some commonly used features are documented below:

## With Items

Duplicating tasks ten times to install ten packages is not efficient, so Ansible provides `loop` construct

```yaml
- name: Install stuff
  package:
    name: {% raw %}"{{ item }}"{% endraw %}
    state: installed
  loop:
    - htop
    - git
    - vim
```

This works for any task, not just package installation, if you have things you'd like to repeat.
Note that for this task one would normally list multiple packages in the `name` parameter of the package module.

```yaml
- name: Install stuff
  package:
    name:
      - htop
      - git
      - vim
    state: installed
```

## When Changed

Doing something only when a task is in the "changed" state is a common pattern. This is often used for reloading a service when some configuration files have changed.

```yaml
- name: Copy configuration file
  copy:
    src: main.conf
    dest: /etc/service/main.conf
    owner: root
    group: root
    mode: 0644
  register: service_conf

- name: Restart the service
  service:
    name: service
    enabled: yes
    state: restarted
  when: service_conf is changed
```

## Notifying Handlers

Often you want to restart a service whenever something has changed, like above. But if you need to restart the service whenever one of many different tasks has changed, there is a simpler way than writing `when: a.changed or b.changed or c.changed or ...`

First, move the restarting or reloading of the service into the `handlers/main.yml`

```yaml
- name: restart service
  service:
    name: service
    enabled: yes
    state: restarted

- name: reload httpd
  service:
    name: httpd
    enabled: yes
    state: reloaded
```

Now you can change your task definitions:

```yaml
- name: Copy configuration file
  copy:
    src: main.conf
    dest: "/etc/service/main.conf
    owner: "root"
    group: "root"
    mode: "0644"
  notify: 'restart service'
```

We no longer `register` the command, instead the `notify` attribute automatically marks that the appropriate handler should be called *at the end of the playbook run*. Additionally, tasks from outside of this role can use the same `notify`. So if you use a community built role for managing apache, and then have a custom role that builds the apache configuration, you can use changes there to reload the service using their definition of the reload/restart.


topic: admin
tutorial: file_sources
title: Adding file-sources to Galaxy
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: ["\"earth-system\"", "\"ocean\"", "\"data\""]
content_hash: 0af74b18328c70df83d496a37280851e
last_modified: 2026-05-15T22:28:10.030062
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/file_sources/tutorial.html

content:
This tutorial demonstrates how to implement an S3 bucket as a Galaxy file-source within Galaxy. We will add here the public Argo data Amazon S3 bucket.
Argo is an international program that observes the interior of the ocean with a fleet of profiling floats drifting in the deep ocean currents (https://argo.ucsd.edu). It started 20 years ago and is a dataset of 5 billion in situ ocean observations from 18.000 profiling floats (4.000 active). The Argo GDAC dataset is a collection of 18.000 NetCDF files. It is a major asset for ocean and climate science and a contributor to IOCCP reports.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Find the information you need
> <hands-on-title>Find an S3 bucket</hands-on-title>
> Go on [Amazon Sustainability Data Initiative](https://exchange.aboutamazon.com/data-initiative). 
>
> There you can visit the [catalog of data](https://registry.opendata.aws/collab/asdi/), and by searching for Argo you can directly get to the [Argo registry](https://registry.opendata.aws/argo-gdac-marinedata/).
>
> On this last page you'll find all the information you'll need to add the S3 bucket to Galaxy 
> ![Image of the Argo registry on amazon](../../images/file_sources/argo_registry.png)
{: .hands_on}


# Add the S3 bucket 

> <hands-on-title>Add on Galaxy</hands-on-title>
>    - If not already done clone the Galaxy Europe [Infrastructure-playbook repo](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master)
>    - Create a branch on your fork
>    - Go to the file **file_sources_conf.yml.j2** in **templates/galaxy/config/**
>
> There you can edit the file and add your S3 bucket by adding a Argo specific section, like in the following:
> ```
> - type: s3fs
>   label: Argo marine floats data and metadata from Global Data Assembly Centre (Argo GDAC)
>   id: argo-gdac-sandbox
>   doc: Argo is an international program to observe the interior of the ocean with a fleet of profiling floats drifting in the deep ocean currents (https://argo.ucsd.edu). 
>   bucket: argo-gdac-sandbox
>   anon: true
> ```
> ![Image of the file_sources_conf.yml.j2 file with th argo bucket added](../../images/file_sources/s3_in_galaxy.png)
>
> Finally, commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

# Conclusion 
Here you are all set and once your Pull Request is merged you'll soon be able to see your bucket in **upload data**, **Choose remote files**, and then search for your bucket label !


topic: admin
tutorial: ansible-galaxy
title: Galaxy Installation with Ansible
difficulty: intermediate
time_estimation: 2h30m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["ansible", "deploying", "git-gat"]
content_hash: 8c23b628c182c9df5ffef7c6fa9ea506
last_modified: 2026-05-27T09:01:47.630949
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible-galaxy/tutorial.html

content:
This tutorial assumes you have some familiarity with [Ansible](https://www.ansible.com/resources/get-started) and are comfortable with writing and running playbooks.  If not, please consider following our [Ansible Tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %}) first.

Here we'll see how to install a Galaxy server using an Ansible playbook. The Galaxy Project has decided on Ansible for all of its deployment recipes. For our project, Ansible is even more fitting due to its name:

> An ansible is a category of fictional device or technology capable of instantaneous or faster-than-light communication. It can send and receive messages to and from a corresponding device over any distance or obstacle whatsoever with no delay, even between star systems
{: .quote cite="https://en.wikipedia.org/wiki/Ansible"}

We want to give you a comprehensive understanding of how the Galaxy installation occurs, but we want to avoid you having to write a "custom" Galaxy installation playbook which you would eventually throw away, in order to use the official playbooks. Given these goals, we will go through the playbook in depth first, and then move to a hands-on portion later. If you are not interested in the inner workings, you can [skip to that section now](#installing-galaxy).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="ansible-galaxy" %}

{% snippet topics/admin/faqs/admin-testing.md %}

# Playbook Overview


## Configuration

We'll be using the [official Galaxy role](https://github.com/galaxyproject/ansible-galaxy) to install and manage Galaxy. This role is found in [Ansible Galaxy](https://galaxy.ansible.com/) (no relation - it is Ansible's system for sharing reusable Ansible roles) as [galaxyproject.galaxy](https://galaxy.ansible.com/galaxyproject/galaxy).

The official role is extremely configurable, everything that you want to change is exposed as a variable, and then tasks will change behaviour based on that. The [role documentation](https://github.com/galaxyproject/ansible-galaxy#role-variables) is the most up-to-date source of documentation for the variables. You should take a minute and read over the variables listed there.

The important variables for this tutorial are:

- `galaxy_root`
- `galaxy_commit_id`
- `galaxy_config`
- `galaxy_server_dir` (automatically set based on `galaxy_root`)

These are largely self explanatory: a directory for all of Galaxy's code and configuration, which commit should be installed, and the Galaxy configuration. We will not explain Galaxy configuration variables in detail as they are covered sufficiently in the `galaxy.yml` sample file or the [online documentation](https://docs.galaxyproject.org/en/master/admin/config.html#configuration-options).

The official recommendation is that you should have a variables file such as `group_vars/galaxyservers.yml` for storing all of the Galaxy configuration.

## Tasks

As with every role, the entry point for execution is the `tasks/main.yml` file. [In the case of the galaxyproject.galaxy role](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/main.yml), this includes a few groups of important tasks:

- [Clone (or Download) Galaxy](#cloning-galaxy)
- [Managing Configuration](#managing-configuration)
- [Fetching Dependencies](#dependencies)
- [Managing Mutable Setup](#mutable-setup)
- [Managing the Database](#managing-the-database)
- [Building the Client](#building-the-client)

The actions each set of tasks performs are described below.

### Cloning Galaxy

The [clone](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/clone.yml) task is the one which is primarily interesting to us, it downloads Galaxy, using git, at a specific commit (or more generally, any [git reference](https://git-scm.com/book/en/v2/Git-Internals-Git-References)).

1. Ansible tries to update Galaxy, cloning it if it is missing, or otherwise attempting to update to the correct commit (or latest commit of the given branch).
2. Any change is reported.
3. The virtualenv is set up:
    1. An empty virtualenv is created.
    2. Pip is updated within the virtualenv.
4. Any `.pyc` files are removed, as this can occasionally result in Python loading the cached code, even if the corresponding `.py` file is no longer present at the checked-out commit. For safety, all of these are removed.

With that Galaxy is cloned to disk and is ready to be configured by the next task.

### Managing Configuration

The [static configuration setup](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/static_setup.yml) is relatively straightforward:

1. The directories for Galaxy configuration data and for the shed tools are created
2. Any config files are copied over
3. Any templates are copied over
4. The `galaxy.yml` is deployed

The setup for deploying extra Galaxy configuration templates and files is a little bit non-standard by Ansible best practices. Here you are expected to provide your own templates and static config files, and then describe them as a list of files and where they should be deployed to.

Using the [UseGalaxy.eu](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/02ca578211bfee45044facf36635d28208e5dbb3/group_vars/galaxy.yml#L578) configuration as an example, we have something like:


{% raw %}
```yaml
galaxy_config_templates:
  - src: templates/galaxy/config/builds.txt
    dest: "{{ galaxy_config.galaxy.builds_file_path }}"
  - src: templates/galaxy/config/data_manager_conf.xml
    dest: "{{ galaxy_config.galaxy.data_manager_config_file }}"
  - src: templates/galaxy/config/datatypes_conf.xml
    dest: "{{ galaxy_config.galaxy.datatypes_config_file }}"
  - src: templates/galaxy/config/dependency_resolvers_conf.xml
    dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
  - src: templates/galaxy/config/disposable_email_blocklist.conf
    dest: "{{ galaxy_config.galaxy.blocklist_file }}"
```
{% endraw %}

The configuration here is a bit different, it references the `galaxy_config`, which is structured like:

{% raw %}
```yaml
galaxy_config:
  galaxy:
    builds_file_path: "{{ galaxy_config_dir  }}/builds.txt"
    datatypes_config_file: "{{ galaxy_config_dir  }}/datatypes_conf.xml"
```
{% endraw %}

So the references in `galaxy_config_templates` to `galaxy_config` are done to ensure that the setting for e.g. "location of the datatypes config file" is the same between where we have configured Galaxy to looking for it, and where the file has been deployed, without requiring us to make variables changes in numerous places.

{% snippet topics/admin/faqs/ansible_define-once.md %}

### Dependencies

Now that Galaxy is available on disk, Ansible is ready to start processing [dependencies](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/dependencies.yml) of Galaxy.

1. The virtualenv is updated with data from the `galaxy_requirements_file`, by default pointing to the requirements file in the codebase: {% raw %}`{{ galaxy_server_dir }}/lib/galaxy/dependencies/pinned-requirements.txt`.{% endraw %}
2. Any necessary conditional dependencies of Galaxy are [collected by processing the config file](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/dependencies/__init__.py)
3. and then installed to the virtualenv.

### Mutable Setup

[This task](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/mutable_setup.yml) creates a directory and initializes "mutable" (written/managed by Galaxy itself) configuration files. It also deploys any hand-managed mutable config files, but it is unlikely that you want to manage these directly, as Galaxy does a sufficient job. Any changes you make to Galaxy, for example installing some tools, would result in the tools being "forgotten about", if you re-ran the playbook and overwrote the `shed_tool_conf.xml` mutable config file with a hand-managed one.

### Managing the Database

The [database management tasks](https://github.com/galaxyproject/ansible-galaxy/blob/master/tasks/database.yml) are extremely convenient; any time you run the playbook to update Galaxy, this will automatically run the database schema migration as needed.

1. The role first obtains the current DB version and the maximum possible DB version based on the codebase.
2. If needed, the database is created.
3. Both versions are reported for the runner of the playbook.
4. If the versions are different, then Ansible runs the command to upgrade the database to the latest version.

As an administrator who often forgot to run the upgrade, and would only notice it once Galaxy crashed during startup, having this process completely automated is extremely nice.

### Building the Client

Galaxy is a modern web application that includes both a server (written in Python) and a client (written in Javascript). After the server is installed and its database prepared, the next step is to build the client application. This means fetching its dependencies, bundling components, creating minified copies of static content, etc. This process ensures that the smallest possible amount of data is transferred to the user when accessing Galaxy over the web, which is important for the performance of the website.

This process can be lengthy and resource intensive. If you want to speed up your deployment, set `galaxy_client_use_prebuilt: true` and Ansible will download a prebuilt client using [yarn](https://classic.yarnpkg.com/en/docs/getting-started).

The client lives in the Galaxy code under the [client/](https://github.com/galaxyproject/galaxy/blob/dev/client/) directory, and the build process deploys it to the [static/](https://github.com/galaxyproject/galaxy/tree/dev/static/) directory, which we will configure a webserver to serve in this tutorial.

## Handlers

A number of the tasks that are executed will trigger a restart of Galaxy. The auto-magic implementation of this restart [handler](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html) should work for most cases, but it is possible to define your own custom handlers if necessary. If this is the case for you, the role provides a way to do this by notifying handlers that ["listen"](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html#naming-handlers) to certain notifications. See the [role handlers](https://github.com/galaxyproject/ansible-galaxy/tree/main/handlers) for details.

## Defaults

As with other roles, numerous [default values](https://github.com/galaxyproject/ansible-galaxy/blob/master/defaults/main.yml) are provided, but these are useful mostly as reference, and not to go through individually.

## Summary

Installation of Galaxy with the playbook follows generally the steps you would expect:

- Galaxy is cloned (or updated)
- A virtualenv is created if it doesn't exist
- Configuration files are installed
- Any missing dependencies are installed
- Any database updates are applied
- The client application is built and deployed (to save time set `galaxy_client_use_prebuilt: true`)

It would not be difficult to write a role that does this yourself, but by using
the `galaxyproject.galaxy` role, you know that you're getting all of the Galaxy
best practices and knowledge from previous admins codified for you.

# Installing Galaxy

With the necessary background in place, you are ready to install Galaxy with Ansible. The playbooks will start simple, and grow over time. We will start with the minimal Galaxy playbook which only requires setting the `galaxy_root` and expand from there. First, however, we need a database for Galaxy to connect to, so we will do that now.

To proceed from here it is expected that:

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> 1. You have [Ansible installed](https://docs.ansible.com/ansible/2.9/installation_guide/intro_installation.html) on the machine where you will install Galaxy
>
>    > <comment-title>Running Ansible on remote machine</comment-title>
>    > It is possible to have Ansible installed on your laptop/local machine and run it against some remote hosts as well. We will **not** do that in this training.
>    {: .comment}
>
> 2. Your `ansible` version is `>=2.10`, you can check this by running `ansible --version`
> 3. You have an [inventory file](../ansible/tutorial.html#inventory-file) with the VM or host specified where you will deploy Galaxy and Galaxy's database. We will refer to these group of hosts as "galaxyservers" and "dbservers," respectively.
> 4. Your VM has a public DNS name: this tutorial sets up SSL certificates from the start and as an integral part of the tutorial.
> 5. Your VM has `python3` installed.
>
>    If you follow the official Ansible instructions to install Ansible on a blank machine, you will probably end up with py2 installed. You will need to install python3 in addition.
>
> 6. In your inventory file, you have written the full DNS hostname that has been provided, and **not** `localhost`, as we will be requesting SSL certificates.
>
> 7. You have the following ports exposed:
>
>    - 22 for SSH, this can be a different port or via VPN or similar.
>    - 80 for HTTP, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 443 for HTTPs, this needs to be available to the world if you want to follow the LetsEncrypt portion of the tutorial.
>    - 5671 for AMQP for Pulsar, needed if you plan to setup Pulsar for remote job running.
>
{: .comment}


> <comment-title>Ubuntu or Debian, CentOS or RHEL?</comment-title>
> The training tutorial is only tested on Ubuntu, as that is the training platform most often used. The training *should* work on RHEL/CentOS, and the roles we use are definitely compatible with both. If any of the variable values differ between Ubuntu and CentOS, we try to note it in the tutorial. Any places we don't note it are bugs.
>
> There are known issues with CentOS7 and python3, which is used in this tutorial. If you use this setup, you should have python2 and python3 coexisting, and use python2 for Ansible (i.e. do not set `interpreter_python` in your `ansible.cfg`) and python3 for Galaxy. This setup requires numerous changes that you will need to discover, which are not covered in this tutorial.
>
> On older versions, both `python-psycopg2` and `python3-psycopg2` may need to be installed.
>
> CentOS8 (and RHEL in general) have significantly different package names for some modules, beware!
{: .comment}


## Requirements

We have codified all of the dependencies you will need into a YAML file that `ansible-galaxy` can install.

> <hands-on-title>Installing roles</hands-on-title>
>
> 1. Create a new directory `galaxy` in your home folder, and `cd` into that directory
>
> 2. Create a new file in your working directory called `requirements.yml` and include the following contents:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/requirements.yml
>    @@ -0,0 +1,13 @@
>    +# Galaxy, Postgres, Nginx
>    +- src: galaxyproject.galaxy
>    +  version: 0.12.1
>    +- src: galaxyproject.nginx
>    +  version: 0.7.1
>    +- src: galaxyproject.postgresql
>    +  version: 1.1.2
>    +- src: galaxyproject.postgresql_objects
>    +  version: 1.2.0
>    +- src: galaxyproject.miniconda
>    +  version: 0.3.4
>    +- src: usegalaxy_eu.certbot
>    +  version: 0.1.13
>    {% endraw %}
>    ```
>    {: data-commit="Add requirements"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>    > <details-title>What do each of these roles do?</details-title>
>    > We'll cover it in more detail as we use each of the roles but briefly:
>    >
>    >  | Role | Purpose |
>    >  |------|---------|
>    >  |`galaxyproject.galaxy` | Installs and configures the Galaxy application|
>    >  |`galaxyproject.nginx` | Sets up a webserver|
>    >  |`galaxyproject.postgresql` | Installs our database, PostgreSQL|
>    >  |`galaxyproject.postgresql_objects` | Creates users and databases within PostgreSQL|
>    >  |`galaxyproject.miniconda` | Installs miniconda, which is used by Galaxy|
>    >  |`usegalaxy_eu.certbot` | Installs certbot and requests SSL certificates|
>    {: .details}
>
>    > <details-title>Role version vs Galaxy version?</details-title>
>    > Q: Is there a correspondence between galaxy role versions and galaxy versions?
>    > A: They are correlated (because generally new Galaxy versions could introduce e.g. new configuration options that the galaxy role would then need to adopt), but the Galaxy role can install older versions of Galaxy.
>    {: .details}
>
> 3. In the same directory, run:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    This will install all of the required modules for this training into the `roles/` folder. We choose to install to a folder to give you easy access to look through the different roles when you have questions on their behaviour.
>
> 4. Inspect the contents of the newly created `roles/` directory in your working directory.
>
> 5. It's good practice to put your playbooks and configuration files under version control (usually in a git repository), but there is no need to keep track of the content of the `roles/` directory, as all of that data can be perfectly recreated from the `requirements.yml` file.
>
>    If you plan to put this git repository online (e.g. on GitHub), any file containing clear-text passwords should *never* ever be committed to the repository, even if the repository is private. For this reason, we should tell git, ahead of time, to ignore one such file, `.vault-password.txt` , which will use later on in this lesson.
>
>    To configure git to ignore certain files and directories, they need to be listed in a file called `.gitignore` , which should be instead tracked as part of your git repository. You can now create the `.gitignore` file with the following content:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/.gitignore
>    @@ -0,0 +1,2 @@
>    +roles/
>    +.vault-password.txt
>    {% endraw %}
>    ```
>    {: data-commit="Add .gitignore"}
>
>    When you run `git status` you'll notice that the `roles/` folder is not listed among the other "Untracked files".
>
>    Now you can do `git add .` to add all of the files in the current directory to your repository, and not worry about committing these files and directories by mistake!
{: .hands_on}

> <hands-on-title>Configuration files</hands-on-title>
>
> 1. Create a `ansible.cfg` file (next to your playbook) to [configure settings](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html) like the inventory file (and save ourselves some typing!), or the Python interpreter to use:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/ansible.cfg
>    @@ -0,0 +1,11 @@
>    +[defaults]
>    +interpreter_python = /usr/bin/python3
>    +inventory = hosts
>    +# Use the YAML callback plugin.
>    +stdout_callback = yaml
>    +# Use the stdout_callback when running ad-hoc commands.
>    +bin_ansible_callbacks = True
>    +
>    +# Show diffs of changes
>    +[diff]
>    +always = True
>    {% endraw %}
>    ```
>    {: data-commit="Add ansible.cfg"}
>
>    > <tip-title>CentOS7</tip-title>
>    > As mentioned in the "Ubuntu or Debian, CentOS or RHEL?" comment above, if you are using CentOS7 do not set `interpreter_python` in `ansible.cfg` .
>    {: .tip}
>
>    There is an additional useful option that you might want to add to your `ansible.cfg` file if you are connecting over SSH:
>
>    ```ini
>    [ssh_connection]
>    pipelining = true
>    ```
>
>    Pipelining will make [Ansible run faster](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html#ansible-pipelining) by significantly reducing the number of new SSH connections that must be opened.
>
> 2. Create the `hosts` inventory file if you have not done so yet, defining a `[galaxyservers]` group with the address of the host where you want to install Galaxy. If you are running Ansible on the same machine where Galaxy will be installed to, you should set the `ansible_connection=local` variable. Lastly, you should explicitly set the `ansible_user` variable to the username to use when connecting to the server. Ansible has changed its behaviour over time regarding whether or not `ansible_user` is defined, and it is most effective to define it explicitly even when it can sometimes be inferred.
>
>    You should also define a `[dbservers]` group for the hosts that will run Galaxy's database. In many cases (and in the case of the Galaxy Admin Training), these will be the same host, but the tutorial is designed to support either scenario, and helps to strengthen understanding of the purpose of host groupings in Ansible for more advanced real-world deployments. Here however we suggest using `[dbservers:children]` and adding `galaxyservers` as the child of that.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat hosts
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > Your hostname is probably different:
>    > >
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,4 @@
>    > > +[galaxyservers]
>    > > +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
>    > > +[dbservers:children]
>    > > +galaxyservers
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add hosts"}
>    > {: .code-out}
>    {: .code-2col}
>
>    > <tip-title>Separate database server</tip-title>
>    > In the hosts file above, we have defined the `galaxyservers` group to be a "child" of the `dbservers` group. This means that the host defined in `[galaxyservers]` is also a member of `[dbservers]`. If we instead wanted to define a separate host as the Galaxy database server, we would replace the `[dbservers:children]` section with something like:
>    >
>    > ```ini
>    > [dbservers]
>    > galaxy-db.example.org
>    > ```
>    {: .tip}
>
> 3. Create the `group_vars` directory and create the file `group_vars/all.yml`. These variables are applied to *all* hosts, regardless of group membership. The variables we set will be used to ensure Python 3 is always used on older systems where Python 2 might also still be installed. We will also set a few variables that must be known by both the Galaxy server and the database server: Galaxy's system/database username, as well as the database name.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat group_vars/all.yml
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/group_vars/all.yml
>    > > @@ -0,0 +1,7 @@
>    > > +---
>    > > +# Python 3 support
>    > > +pip_virtualenv_command: /usr/bin/python3 -m venv  # usegalaxy_eu.certbot, usegalaxy_eu.tiaas2, galaxyproject.galaxy
>    > > +
>    > > +# Common variables needed by all hosts
>    > > +galaxy_user_name: galaxy
>    > > +galaxy_db_name: galaxy
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add initial all hosts variables file"}
>    > {: .code-out}
>    {: .code-2col}
>
{: .hands_on}

## PostgreSQL

Galaxy is capable of talking to multiple databases through SQLAlchemy drivers. SQLite is the development database, but PostgreSQL (/ˈpoʊstɡrɛs ˌkjuː ˈɛl/, POHST-gres kyoo el) is recommended in production. MySQL is a possibility, but does not receive the same testing or bugfixes from the main development team as PostgreSQL, so we will only show installation with PostgreSQL.

PostgreSQL maintains its own user database apart from the system user database. By default, PostgreSQL uses the "peer" authentication method which allows access for system users with matching PostgreSQL usernames (other authentication mechanisms are available, see the [PostgreSQL Client Authentication documentation](https://www.postgresql.org/docs/current/static/client-authentication.html).

For this tutorial, we will use the default "peer" authentication, so we need to create a PostgreSQL user matching the system user under which Galaxy will be running, i.e. `galaxy`. This is normally done with the PostgreSQL `createuser` command, and it must be run as the `postgres` user. In our case, we will use the `galaxyproject.postgresql_objects` role to handle this step.
> <hands-on-title>Installing PostgreSQL</hands-on-title>
>
> 1. Create and edit `group_vars/dbservers.yml` and add some variables to configure PostgreSQL:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/dbservers.yml
>    @@ -0,0 +1,7 @@
>    +---
>    +# PostgreSQL
>    +postgresql_objects_users:
>    +  - name: "{{ galaxy_user_name }}"
>    +postgresql_objects_databases:
>    +  - name: "{{ galaxy_db_name }}"
>    +    owner: "{{ galaxy_user_name }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add initial dbservers group variables file"}
>
>    Notice that we have used the variables we defined in `group_vars/all.yml` for the Galaxy user and database names in the previous section, implementing the principle of "define once, reference many times."
>
>    > <tip-title>Using postgres via the network</tip-title>
>    > To use postgres via another machine, or via the network, you can add lines like the following:
>    >
>    > ```yaml
>    > postgresql_pg_hba_conf:
>    > - host all all 127.0.0.1/32 md5
>    > ```
>    >
>    > Here you should either add multiple lines per IP address that needs access to the Postgres server (with netmask `/32`), or a less specific IP range defined also [via netmask.](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#IPv4_CIDR_blocks)
>    >
>    > Additionally you'll need to change the `postgresql_objects_users` statement to include a password (maybe stored in a vault, discussed later!)
>    >
>    > ```yaml
>    > postgresql_objects_users:
>    >   - name: galaxy
>    >     password: "{{ vault_postgres_galaxy_password }}"
>    > ```
>    {: .tip}
>
> 2. Create and open `galaxy.yml` which will be our playbook. Write the following:
>
>    - Create a play that will run on the `dbservers` group, as the root user (you will need `become`/`become_user`)
>    - Add a pre-task to install the necessary dependency at this stage: `acl`
>    - Use the role `galaxyproject.postgresql`. This will handle the installation of PostgreSQL.
>    - Use the role `galaxyproject.postgresql_objects`, run as the postgres user. This role allows for managing users and databases within PostgreSQL.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/galaxy.yml
>    @@ -0,0 +1,13 @@
>    +---
>    +- hosts: dbservers
>    +  become: true
>    +  become_user: root
>    +  pre_tasks:
>    +    - name: Install Dependencies
>    +      package:
>    +        name: 'acl'
>    +  roles:
>    +    - galaxyproject.postgresql
>    +    - role: galaxyproject.postgresql_objects
>    +      become: true
>    +      become_user: postgres
>    {% endraw %}
>    ```
>    {: data-commit="Add initial galaxy playbook"}
>
>    > <question-title></question-title>
>    >
>    > What should we expect to see in our folder?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - a playbook
>    > > - a hosts file
>    > > - a requirements file
>    > > - some group variables
>    > > - all of our roles
>    > >
>    > > > <code-in-title>Bash</code-in-title>
>    > > > Let's investigate our directory.
>    > > > ```bash
>    > > > tree -L 2
>    > > > ```
>    > > {: .code-in}
>    > >
>    > > > <code-out-title>Bash</code-out-title>
>    > > >
>    > > > Your directory should look approximately like this:
>    > > >
>    > > > ```ini
>    > > > .
>    > > > ├── ansible.cfg
>    > > > ├── galaxy.yml
>    > > > ├── group_vars
>    > > > │   ├── all.yml
>    > > > │   └── dbservers.yml
>    > > > ├── hosts
>    > > > ├── requirements.yml
>    > > > └── roles
>    > > >     ├── galaxyproject.galaxy
>    > > >     ├── galaxyproject.miniconda
>    > > >     ├── galaxyproject.nginx
>    > > >     ├── galaxyproject.postgresql
>    > > >     ├── galaxyproject.postgresql_objects
>    > > >     └── usegalaxy_eu.certbot
>    > > >
>    > > > 8 directories, 6 files
>    > > > ```
>    > > {: .code-out.code-max-300}
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>    {% snippet topics/admin/faqs/ansible_role-prefix.md %}
>
>    {% snippet topics/admin/faqs/ansible_yaml.md %}
>
> 3. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    >
>    > Note: we have stripped out the "diff" output for compactness, your output will be more verbose.
>    >
>    > ```ini
>    > PLAY [dbservers] ***************************************************************
>    >
>    > TASK [Gathering Facts] *********************************************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [Install Dependencies] ****************************************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : include_tasks] ********************************
>    > included: /home/ubuntu/galaxy/roles/galaxyproject.postgresql/tasks/debian.yml for gat-0.eu.galaxy.training
>    >
>    > TASK [galaxyproject.postgresql : APT keyrings directory] ***********************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install pgdg package signing key (Debian/pgdg)] ***
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install pgdg repository (Debian/pgdg)] ********
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install PostgreSQL (Debian)] ******************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Get installed version] ************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set version fact] *****************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install psycopg2] *****************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set version fact] *****************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set OS-specific variables] ********************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set pgdata fact] ******************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set conf dir fact] ****************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : include_tasks] ********************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Create conf.d] ********************************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Check for conf.d include in postgresql.conf] ***
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set conf.d include in postgresql.conf] ********
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Include 25ansible_postgresql.conf in postgresql.conf] ***
>    > [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created
>    > with a mode of 0700, this may cause issues when running as another user. To
>    > avoid this, create the remote_tmp dir with the correct permissions manually
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Set config options] ***************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install pg_hba.conf] **************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : include_tasks] ********************************
>    > included: /home/ubuntu/galaxy/roles/galaxyproject.postgresql/tasks/backup.yml for gat-0.eu.galaxy.training
>    >
>    > TASK [galaxyproject.postgresql : Create backup directories] ********************
>    > changed: [gat-0.eu.galaxy.training] => (item=/var/lib/postgresql/backups)
>    > changed: [gat-0.eu.galaxy.training] => (item=/var/lib/postgresql/backups/bin)
>    >
>    > TASK [galaxyproject.postgresql : Create backup output directory] ***************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Install backup script templates] **************
>    > changed: [gat-0.eu.galaxy.training] => (item=archive_wal.sh)
>    >
>    > TASK [galaxyproject.postgresql : Install backup script files] ******************
>    > changed: [gat-0.eu.galaxy.training] => (item=backup.py)
>    >
>    > TASK [galaxyproject.postgresql : Set WAL archive config options] ***************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Schedule backups] *****************************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Remove PostgreSQL working WAL backup cron job] ***
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql : Ensure PostgreSQL is running] *****************
>    > ok: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Warn if deprecated user priv param is set] ***
>    > skipping: [gat-0.eu.galaxy.training] => (item=galaxy)
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Revoke extra privileges] **************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Drop databases] ***********************
>    > skipping: [gat-0.eu.galaxy.training] => (item={'name': 'galaxy', 'owner': 'galaxy'})
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Create and drop users] ****************
>    > [WARNING]: Module remote_tmp /var/lib/postgresql/.ansible/tmp did not exist and
>    > was created with a mode of 0700, this may cause issues when running as another
>    > user. To avoid this, create the remote_tmp dir with the correct permissions
>    > manually
>    > changed: [gat-0.eu.galaxy.training] => (item=galaxy)
>    >
>    > TASK [galaxyproject.postgresql_objects : Create groups] ************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Add or remove users from groups] ******
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Drop groups] **************************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > TASK [galaxyproject.postgresql_objects : Create databases] *********************
>    > changed: [gat-0.eu.galaxy.training] => (item=galaxy)
>    >
>    > TASK [galaxyproject.postgresql_objects : Grant extra privileges] ***************
>    > skipping: [gat-0.eu.galaxy.training]
>    >
>    > RUNNING HANDLER [galaxyproject.postgresql : Reload PostgreSQL] *****************
>    > changed: [gat-0.eu.galaxy.training]
>    >
>    > PLAY RECAP *********************************************************************
>    > gat-0.eu.galaxy.training   : ok=27   changed=15   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0
>    > ```
>    {: .code-out.code-max-300}
>
>    > <comment-title>Error: `postgresql_version is version_compare('9.3', '>=')' failed`</comment-title>
>    > You might see an error like this, if you're running with `--check` mode. Unfortunately here `--check` mode meets real life: not all modules support it because some rely on running command line tools to obtain version numbers, something that cannot be mocked with check mode. You can solve this by running it without `--check`.
>    >
>    > ```
>    > fatal: [gat-0.eu.galaxy.training]: FAILED! => {"msg": "The conditional check 'postgresql_version is version_compare('9.3', '>=')' failed. The error was: Input version value cannot be empty\n\nThe error appears to be in '/home/ubuntu/galaxy/roles/galaxyproject.postgresql/tasks/main.yml': line 42, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# etc.). So check for a match first and then add if there's no match.\n- name: Check for conf.d include in postgresql.conf\n ^ here\n"}
>    > ```
>    {: .comment}
>
>    > <comment-title>When running Ansible</comment-title>
>    > Always pay close attention to tasks reported as **changed** and ensure that the changes were expected!
>    {: .comment}
>
>    > <tip-title>Why didn't we use `-i` in our ansible command?</tip-title>
>    > In our `ansible.cfg` file we specified the inventory was stored in a file called `hosts`:
>    >
>    > ```ini
>    > [defaults]
>    > interpreter_python = /usr/bin/python3
>    > inventory = hosts
>    > retry_files_enabled = false
>    > ```
>    {: .tip}
>
>    {% snippet topics/admin/faqs/ansible_error-no-hosts-matched.md %}
>
> 4. Inspect the changes that have been made on your Galaxy server. Places to look include:
>
>    - `/etc/postgresql`
>    - Databases and users in PostgreSQL.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > List available databases.  You can now login and access the database, but only as the `postgres` user. You will need to `sudo -iu postgres` first, and then you can run `psql galaxy`.
>    > > ```bash
>    > > sudo -iu postgres psql -l
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > ```ini
>    > >                               List of databases
>    > >    Name    |  Owner   | Encoding | Collate |  Ctype  |   Access privileges
>    > > -----------+----------+----------+---------+---------+-----------------------
>    > >  galaxy    | galaxy   | UTF8     | C.UTF-8 | C.UTF-8 |
>    > >  postgres  | postgres | UTF8     | C.UTF-8 | C.UTF-8 |
>    > >  template0 | postgres | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
>    > >            |          |          |         |         | postgres=CTc/postgres
>    > >  template1 | postgres | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
>    > >            |          |          |         |         | postgres=CTc/postgres
>    > > (4 rows)
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
>    > <code-in-title>Bash</code-in-title>
>    > Access postgres and list some users
>    > ```bash
>    > sudo -iu postgres psql
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    > The database will currently be empty (check relations with \d and list of roles with \dg) as Galaxy has not yet connected to it. Once you install Galaxy in the next step, the database will be populated. (Quit database with \q)
>    > ```ini
>    > psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
>    > Type "help" for help.
>    >
>    > postgres=# \d
>    > Did not find any relations.
>    > postgres=# \dg
>    >                                    List of roles
>    >  Role name |                         Attributes                         | Member of
>    > -----------+------------------------------------------------------------+-----------
>    >  galaxy    |                                                            | {}
>    >  postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
>    >
>    > postgres=# \q
>    > ```
>    {: .code-out}
>
>    > > <code-in-title>Bash</code-in-title>
>    > > Inspect the postgres directory
>    > > ```bash
>    > > ls /etc/postgresql/*/main/
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > ```ini
>    > > conf.d
>    > > environment
>    > > pg_ctl.conf
>    > > pg_hba.conf
>    > > pg_hba.conf.3350.2020-07-08@11:51:47~
>    > > pg_ident.conf
>    > > postgresql.conf
>    > > start.conf
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
{: .hands_on}


## Galaxy

Next we will dive right in to deploying a copy of Galaxy onto our server.

For a normal Galaxy instance there are a few configuration changes you make very early during deployment:

- Changing the database connection
- Configuring the admin user list

Additionally we'll go ahead and set up the production-ready [Gunicorn + Webless](https://docs.galaxyproject.org/en/master/admin/scaling.html#deployment-options) deployment, which separates Galaxy's web and job handling into separate processes. This is done by configuring Galaxy's process manager, [Gravity](https://gravity.readthedocs.io).

Finally, best admin practices are to not run Galaxy as a user with `sudo` access, like your login user probably has. Additionally, it is best to install the Galaxy code and configs as a separate user, for security purposes. So we will instruct the `galaxyproject.galaxy` role to create a new user account specifically to run Galaxy under.

The configuration is quite simple thanks to the many sensible defaults that are provided in the Ansible roles.

> <hands-on-title>Minimal Galaxy Playbook</hands-on-title>
>
> 1. Open `galaxy.yml` with your text editor and do the following:
>
>    - Create a new play that will run on the `galaxyservers` group, as the root user (you will need `become`/`become_user`)
>    - Add a pre-task to install the necessary dependencies for the Galaxy server: `acl`, `bzip2`, `git`, `make`, `tar`, `python3-venv`, and `python3-setuptools`
>    - Use the roles `galaxyproject.galaxy` and `galaxyproject.miniconda` (in this order), with `galaxyproject.miniconda` run as the `galaxy` user.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -11,3 +11,16 @@
>         - role: galaxyproject.postgresql_objects
>           become: true
>           become_user: postgres
>    +
>    +- hosts: galaxyservers
>    +  become: true
>    +  become_user: root
>    +  pre_tasks:
>    +    - name: Install Dependencies
>    +      package:
>    +        name: ['acl', 'bzip2', 'git', 'make', 'tar', 'python3-venv', 'python3-setuptools']
>    +  roles:
>    +    - galaxyproject.galaxy
>    +    - role: galaxyproject.miniconda
>    +      become: true
>    +      become_user: "{{ galaxy_user_name }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add miniconda, galaxy to playbook"}
>
> 2. Create and edit `group_vars/galaxyservers.yml` and add some variables to configure Galaxy:
>
>    We need to set the following variables at the top level:
>
>    Variable                        | Value                                                                     | Purpose
>    ---                             | -----                                                                     | ---
>    `galaxy_create_user`            | `true`                                                                    | Instruct the role to create a Galaxy user
>    `galaxy_separate_privileges`    | `true`                                                                    | Enable separation mode to install the Galaxy code as `root` but run the Galaxy server as `galaxy`
>    `galaxy_manage_paths`           | `true`                                                                    | Instruct the role to create the needed directories.
>    `galaxy_layout`                 | `root-dir`                                                                | This enables the `galaxy_root` Galaxy deployment layout: all of the code, configuration, tools, and mutable-data (like caches, location files, etc.) folders will live by default beneath `galaxy_root`. User data is stored under `file_path`, a variable we will set later.
>    `galaxy_root`                   | `/srv/galaxy`                                                             | This is the root of the Galaxy deployment.
>    `galaxy_user`                   | {% raw %}`{name: "{{ galaxy_user_name }}", shell: /bin/bash}`{% endraw %} | The user that Galaxy will run as.
>    `galaxy_commit_id`              | `release_25.0`                                                            | The git reference to check out, which in this case is the branch for Galaxy Release 23.0
>    `galaxy_force_checkout`         | `true`                                                                    | If we make any modifications to the Galaxy codebase, they will be removed. This way we know we're getting an unmodified Galaxy and no one has made any unexpected changes to the codebase.
>    `galaxy_client_use_prebuilt`         | `true`                                                              | We are going to use a prebuilt client to save some time during this tutorial, if you want you can set it to false later and see how galaxy starts building the client itself.
>    `miniconda_prefix`              | {% raw %}`"{{ galaxy_tool_dependency_dir }}/_conda"`{% endraw %}          | We will manually install conda as well. Normally Galaxy will attempt to auto-install this, but since we will set up a production-ready instance with multiple handlers, there is the chance that they can become deadlocked.
>    `miniconda_version`             | `25.3`                                                                    | Install a specific miniconda version, the latest one at the time of writing that was tested and working.
>    `miniconda_distribution`             | `miniforge`                                                                    | Install a specific distribution of conda, see https://github.com/galaxyproject/ansible-miniconda for alternatives. This determines the default value for the `miniconda_executable` (`conda` for `miniforge`)
>    `miniconda_channels`          ` | `['conda-forge']`                                             | Use the free and community-maintained `conda-forge` channel.
>
>    > <tip-title>Different Galaxy Releases!</tip-title>
>    > In the time between this tutorial was last updated ({{ page.last_modified_at | date: "%Y-%m-%d" }}), and when you are now reading it, one or more new releases of Galaxy may have occured.
>    >
>    > If you wish to use a different version of Galaxy than the one listed, you can do so, but there is no guarantee this tutorial will work without modifications.
>    >
>    > Please ensure you consult the [releases documentation](https://docs.galaxyproject.org/en/master/releases/index.html)
>    {: .tip}
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/group_vars/galaxyservers.yml
>    @@ -0,0 +1,15 @@
>    +# Galaxy
>    +galaxy_create_user: true # False by default, as e.g. you might have a 'galaxy' user provided by LDAP or AD.
>    +galaxy_separate_privileges: true # Best practices for security, configuration is owned by 'root' (or a different user) than the processes
>    +galaxy_manage_paths: true # False by default as your administrator might e.g. have root_squash enabled on NFS. Here we can create the directories so it's fine.
>    +galaxy_manage_systemd: true
>    +galaxy_layout: root-dir
>    +galaxy_root: /srv/galaxy
>    +galaxy_user: {name: "{{ galaxy_user_name }}", shell: /bin/bash}
>    +galaxy_commit_id: release_25.0
>    +galaxy_client_use_prebuilt: true
>    +galaxy_force_checkout: true
>    +miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
>    +miniconda_version: 25.3
>    +miniconda_distribution: miniforge
>    +miniconda_channels: ['conda-forge']
>    {% endraw %}
>    ```
>    {: data-commit="Configure miniforge and galaxy"}
>
>    > <tip-title>Following this training outside of a GAT course?</tip-title>
>    > Consider updating the Galaxy Commit ID to the latest version of Galaxy available, this will probably give better results (given that dependencies are always updating) than using a potentially outdated version.
>    {: .tip}
>
> 3. Again edit the galaxyservers group variables file and add a variable for `galaxy_config`. It will be a hash with one key, `galaxy` which will also be a hash. Inside here you can place all of your Galaxy configuration.
>
>    So the structure looks like:
>    ```yaml
>    galaxy_config:
>      galaxy:
>        key: value
>    ```
>
>    Now you should set:
>    1. `admin_users` to the email address you will use with this Galaxy.
>    3. `database_connection` to point to the database you setup earlier (`postgresql:///{{ galaxy_db_name }}?host=/var/run/postgresql`).
>    4. `file_path` to a place to store data, `/data` is fine for this lesson which sets up a single-node Galaxy. If you have separate compute machines, this will normally need to be storage shared between the Galaxy node and compute nodes.
>    5. `tool_data_path` to {% raw %}`{{ galaxy_mutable_data_dir }}/tool-data`{% endraw %}, so that when tools are installed, due to privilege separation, this will happen in a directory Galaxy can actually write into.
>    6. `object_store_store_by` to `uuid`, this is a better way of storing files that will ensure better filesystem balancing than the older system.
>    7. `id_secret` to {% raw %}`{{ vault_id_secret }}`{% endraw %}, we'll define this variable next but it will be used to encode the IDs used in Galaxy URLs and for securing session cookies.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -13,3 +13,17 @@ miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
>     miniconda_version: 25.3
>     miniconda_distribution: miniforge
>     miniconda_channels: ['conda-forge']
>    +
>    +galaxy_config:
>    +  galaxy:
>    +    # Main Configuration


topic: admin
tutorial: tool-management
title: Galaxy Tool Management with Ephemeris
difficulty: intermediate
time_estimation: 45m
tools_json: []
requirements_json: []
tags_json: ["tools", "git-gat"]
content_hash: 4d3b81e1ab8171b6d7e1dc1e6f762080
last_modified: 2026-05-27T09:01:47.649255
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/tool-management/tutorial.html

content:
This tutorial will introduce you to one of Galaxy's associated projects - [Ephemeris](https://ephemeris.readthedocs.io/). Ephemeris is a small Python library and set of scripts for managing the bootstrapping of Galaxy plugins - tools, index data, and workflows. It aims to help automate, and limit the quantity of manual actions admins have to do in order to maintain a Galaxy instance.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tool-management" %}

# Background

You are an administrator of a Galaxy server. A colleague has approached you with a request to run a specific [Galaxy workflow]({% link topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga %}) on their data. In order to enable this workflow for your users, you will have to:

- identify what tools are required for the workflow
- install these tools and their dependencies on your Galaxy instance.


# Requirements

To run this tutorial, you will need to [install Ephemeris](https://ephemeris.readthedocs.io/en/latest/installation.html). You would normally install it on your workstation, but during training courses we recommend to install it on the same virtual machine used for the Galaxy server.

> <hands-on-title>Installing Ephemeris in a Python virtual environment</hands-on-title>
>
> 1. Install the Python `venv` package if it is not already available. On Ubuntu this can be done with `sudo apt install python3-venv`
> 2. Create a virtual environment just for ephemeris, activate it and install ephemeris inside it:
>    ```console
>    python3 -m venv ~/ephemeris_venv
>    . ~/ephemeris_venv/bin/activate
>    pip install ephemeris
>    ```
{: .hands_on}

# Extracting Tools

A common request you will experience as an administrator is "I want to run this workflow". Since this is such a common workflow, Galaxy has a built in way to accomplish it. We can use Ephemeris to extract a list of tools from a Galaxy workflow document, and then use Ephemeris to install these tools and specific versions into your Galaxy.

However, Galaxy workflow files are complex JSON documents, and the process of mapping the tool IDs to a ToolShed repository and revision is not trivial. Workflow files contain tool IDs which look like:

```
toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.71
toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.4.3.1
toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.6
toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.4.2
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1
toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1
```

In order to actually install these tools, we need to convert each tool ID into a ToolShed repository name and revision. For example, FastQC version 0.71 corresponds to revision `ff9530579d1f` in the ToolShed.

```yaml
- name: fastqc
  owner: devteam
  revisions:
  - ff9530579d1f
  tool_panel_section_label: Tools from workflows
  tool_shed_url: https://toolshed.g2.bx.psu.edu
```

Ephemeris can take care of this process. Let's practice this on a real workflow.

> <hands-on-title>Extracting a list of tools from a workflow</hands-on-title>
>
> 1. Download the mapping workflow:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > wget {{ site.url }}{% link topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga %}
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Use the Ephemeris [`workflow-to-tools`](https://ephemeris.readthedocs.io/en/latest/commands/workflow-to-tools.html) command to extract the tool list from this workflow into a file named `workflow_tools.yml` in the folder `tools`.
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > > <code-in-title>Bash</code-in-title>
>    > > > ```bash
>    > > > workflow-to-tools -w mapping.ga -o tools/workflow_tools.yml -l Mapping
>    > > > ```
>    > > > {: data-cmd="true"}
>    > > >
>    > > > Or as a diff:
>    > > > {% raw %}
>    > > > ```diff
>    > > > --- /dev/null
>    > > > +++ b/tools/workflow_tools.yml
>    > > > @@ -0,0 +1,41 @@
>    > > > +install_tool_dependencies: True
>    > > > +install_repository_dependencies: True
>    > > > +install_resolver_dependencies: True
>    > > > +
>    > > > +tools:
>    > > > +- name: fastqc
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - e7b2202befea
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: trim_galore
>    > > > +  owner: bgruening
>    > > > +  revisions:
>    > > > +  - 949f01671246
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: multiqc
>    > > > +  owner: iuc
>    > > > +  revisions:
>    > > > +  - f7985e0479b9
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: bowtie2
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - 09b2cdb7ace5
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: samtools_stats
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - 24c5d43cb545
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > +- name: bamtools_filter
>    > > > +  owner: devteam
>    > > > +  revisions:
>    > > > +  - cb20f99fd45b
>    > > > +  tool_panel_section_label: Mapping
>    > > > +  tool_shed_url: https://toolshed.g2.bx.psu.edu/
>    > > > {% endraw %}
>    > > > ```
>    > > > {: data-commit="Add a tool list to install"}
>    > > {: .code-in}
>    > {: .solution }
>    {: .question}
>
> 3. Inspect the `workflow_tools.yml` file, which contains a tool list in YAML format.
>
{: .hands_on }

# Installing Tools

Now that you have extracted a list of tools, let's install these on your Galaxy instance. In order to accomplish this, you will need:

- The URL of your Galaxy server
- The API key for your account, which must be an admin

{% snippet faqs/galaxy/preferences_admin_api_key.md admin=true %}

There are two ways to install tools, depending on how you specify the tools to install:

> <hands-on-title>Installing a single tool</hands-on-title>
>
> 1. Use the Ephemeris [`shed-tools`](https://ephemeris.readthedocs.io/en/latest/commands/shed-tools.html) command to install the tool `bwa`, owned by `devteam` into a section named `Mapping`
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > Use your Galaxy URL and API key in the example command below:
>    > >
>    > > ```bash
>    > > shed-tools install -g https://galaxy.example.org -a <api-key> --name bwa --owner devteam --section_label Mapping
>    > > ```
>    > > {: data-cmd="true"}
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <tip-title>Certificate issues (GAT, uncommon)</tip-title>
>
> If your Galaxy instance is served via the HTTPS protocol (as it should be!), ephemeris will use the [requests](https://requests.readthedocs.io) Python library to encrypt the communication with Galaxy. Therefore, if your Galaxy uses a self-signed SSL certificate, `shed-tools` may fail with a `CERTIFICATE_VERIFY_FAILED` error.
>
> Under Ubuntu, you can allow the use of the unrecognized certificate as follows:
> 1. Get hold of the Certificate Authority (CA) certificate used to sign your Galaxy SSL certificate. For a [Galaxy Admin Training](https://github.com/galaxyproject/dagobah-training) course, this is usually the [Fake LE Root X1 certificate](https://letsencrypt.org/certs/fakelerootx1.pem).
> 2. Copy the CA certificate file into `/usr/local/share/ca-certificates/` with a `.crt` extension.
> 3. Run `update-ca-certificates` as root.
> 4. Execute `export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt`, as explained in [requests docs](https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification).
>
> Now you should be able to execute successfully the `shed-tools` commands.
{: .tip}


This provides an easy way to do a one-off installation of a tool, but is not very convenient if you want to install many tools.
For that, you can install from a YAML file:

> <hands-on-title>Installing tools from a tool list</hands-on-title>
>
> 1. (optional) Watch the installation proceed by running `journalctl -f` in a separate remote shell.
>
> 2. Use the Ephemeris [`shed-tools`](https://ephemeris.readthedocs.io/en/latest/commands/shed-tools.html) command to install all of the tools from the `workflow_tools.yml` file on your Galaxy.
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > Use your Galaxy URL and API key in the example command below:
>    > >
>    > > ```bash
>    > > shed-tools install -g https://galaxy.example.org -a <api-key> -t workflow_tools.yml
>    > > ```
>    > > {: data-cmd="true"}
>    > {: .solution}
>    {: .question}
>
> 4. Open your Galaxy's admin interface and check that the tools have been installed.
>
> 5. Using the UI import the workflow file that you used, [mapping.ga]({% link topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga %}).
>    1. Right-click or Ctrl-click on the link above and copy the link address
>    2. On your Galaxy instance click on `Workflows`, then `Import`.  Paste the URL into the `Import Archived URL` field.
{: .hands_on}

Occasionally the tool installation may fail due to network issues; if it does, just re-run the `shed-tools` installation process until it succeeds. This is a known issue the developers are working on.

{% snippet topics/admin/faqs/byobu-shortcuts.md %}

> <tip-title>Can I install tools without restarting?</tip-title>
> Yes. The default tool config (`config/tool_conf.xml.sample`, copy to `config/tool_conf.xml` to modify) has an option, `monitor="true"` set in the root `<toolbox>` tag. This instructs Galaxy to watch the tool files referenced in that config and load or reload them as necessary. It will also add any tools you have added.
{: .tip}

> <tip-title>Can I install tools without a ToolShed?</tip-title>
> Yes. The `galaxy_local_tools` option for the `galaxyproject.galaxy` Ansible role can be used to install local tools, or you can manage them in another way that fits your workflow better. UseGalaxy.eu, for example, maintains a repository of tools that are not installed from the ToolShed to aid their local developers. This is deployed to the server using the `git` module, rather than the Galaxy Ansible role.
{: .tip}


# Tool Testing

Having the tools installed is a good first step, but your users will expect that they actually work as well. You can use Ephemeris to automatically test all of the installed tools.

> <hands-on-title>Test the installed tools</hands-on-title>
>
> 1. Use the Ephemeris [`shed-tools`](https://ephemeris.readthedocs.io/en/latest/commands/shed-tools.html#test) command to test the `bamtools_filter` tool on your Galaxy.
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > Use your Galaxy URL and API key in the example command below:
>    > >
>    > > ```bash
>    > > shed-tools test -g https://galaxy.example.org -a <api-key> --name bamtools_filter --owner devteam
>    > > ```
>    > > {: data-cmd="true"}
>    > {: .solution}
>    {: .question}
> 2. Shed-tools test outputs a file with details of `tool_test_output.json` with details of jobs that have run.  Have a look at this file.
{: .hands_on}

This can give you some more confidence that things are working correctly. Oftentimes, users provide workflows for biological domains that we are not familiar with, so knowing how we can test these tools is impossible for us as admins. Leveraging the built-in tool test cases can give you reassurance that things are functional before you inform your users of the new tools.

> <tip-title>Tool test reports</tip-title>
> The ephemeris `shed-tools test` command produces an output file `tool_test_output.json` with information about the test jobs that have run. Galaxyproject's [Planemo](https://github.com/galaxyproject/planemo) can be used to generate [test reports](https://planemo.readthedocs.io/en/latest/commands/test_reports.html?highlight=test_reports#test-reports-command) from tool_test_output.json in HTML and other formats.
{: .tip}


# Obtaining a Tool List

Sometimes a user might ask you to install all the tools they were previously using on another Galaxy instance. Ephemeris can produce a `tool_list.yaml` file for all the tools installed on a server.

> <hands-on-title>Obtain UseGalaxy.eu's tool list</hands-on-title>
>
> 1. Use the Ephemeris [`get-tool-list`](https://ephemeris.readthedocs.io/en/latest/commands/get-tool-list.html) command to obtain the full set of tools installed on UseGalaxy.eu
>
>    > <question-title></question-title>
>    > What did your command look like?
>    >
>    > > <solution-title></solution-title>
>    > > This command does not require authentication and can be used to obtain the tool list from any public Galaxy server:
>    > >
>    > > > <code-in-title>Bash</code-in-title>
>    > > > ```bash
>    > > > get-tool-list -g "https://usegalaxy.eu" -o "eu_tool_list.yaml"
>    > > > ```
>    > > > {: data-cmd="true"}
>    > > {: .code-in}
>    > {: .solution}
>    {: .question}
> 2. Inpect the first few lines of tool list: Run `head -n 20 eu_tool_list.yaml`.
{: .hands_on}

We will not install all the tools from the EU Galaxy server as that server likely has more tools than any other Galaxy instance, but it is useful as an example of how you can use Ephemeris to facilitate the mirroring of another Galaxy instance.

> <tip-title>Non-shed tools</tip-title>
> The output of `get-tool-list` only includes ToolShed tools, not local non-TS tools.
{: .tip}

> <tip-title>But how many tools is it really?</tip-title>
> If you've seen the [European Galaxy tools view](https://usegalaxy.eu/tools/view) (this is available on any galaxy! Just access `/tools/view`)
> you'll notice they report somewhere over 2700 tools, however the `get-tool-list` output lists significantly fewer tools.
>
> This is for a couple reasons:
> - That's the number of repositories that are installed, and some repositories include multiple tools (e.g. [`circos`](https://toolshed.g2.bx.psu.edu/view/iuc/circos/df7356989ac1) has quite a few)
> - This only lists ToolShed tools, while EU also includes a number of non-TS tools
> - There are a number of tools built into Galaxy (e.g. the Collection Operation tools)
{: .tip}

# Production Best Practices

The servers which are part of the `usegalaxy.*` network use Ephemeris extensively to manage their large tool sets.

Interestingly, UseGalaxy.eu and UseGalaxy.org.au have different approaches:
- EU maintains [YAML files roughly per domain](https://github.com/usegalaxy-eu/usegalaxy-eu-tools), it is not as clear of an ordering. They maintain a YAML file where humans add the tools which should be installed in a given category, and lock files are automatically generated from these with the latest revision if it is missing. They follow a cycle of updating the lock files with the latest available revisions of tools, and then installing from these lock files any missing revisions. They use a [Jenkins server](https://build.galaxyproject.eu/job/usegalaxy-eu/job/install-tools/) to automatically run tool installation weekly.
- AU maintains [a separate YAML file per tool panel section](https://github.com/usegalaxy-au/usegalaxy-au-tools/tree/master/usegalaxy.org.au) as a record of all installed tools on the server.  They accept tool requests as pull requests and a Jenkins server is notified when a pull request is merged so that the tool can be automatically installed and tested. Like EU, they run automatic updates of installed tools once a week.
- Together, `usegalaxy.*` are working on a collaborative approach at [galaxyproject/usegalaxy-tools](https://github.com/galaxyproject/usegalaxy-tools) but this is not consumption ready yet.

If running ephemeris directly is not your preference, there is an Ansible [role](https://github.com/galaxyproject/ansible-galaxy-tools) and a sample [playbook](https://github.com/afgane/galaxy-tools-playbook) that can help automate some tasks.

> <tip-title>What if environments are not working</tip-title>
> It sometimes happens in Galaxy, that one environment isn't working anymore. It mostly happens from the start when it does happen. You can remove the environment on disk, or use the "Manage Dependencies" interface, select the environment, and delete it. Then re-install the dependency through the same Manage Dependencies interface.
{: .tip}

> <tip-title>Can you install multiple tools simultaneously?</tip-title>
> Previous experience with this is not good, there was a lot of unsafe code that would do things simultaneously that would destroy config files. With conda this is not recommended either, as this can corrupt conda environments. A solution for this is how [UseGalaxy.eu does it](https://github.com/usegalaxy-eu/usegalaxy-eu-tools), where they keep the full list of tools they want to install, and then a CI server (Jenkins) installs them. In this way they can enforce that only a single install process is running at any time.
{: .tip}

> <tip-title>Certificate Issues (GAT Only)</tip-title>
> If a student is running `shed-tools` on the VM, then it should work without certificate issues, because we installed the Fake LE X1 CA certificate, meaning that to your VM, the certificate chain is valid.
> We cannot (and would not recommend) setting that certificate on your local machine. That is the first way that comes to mind, that running an ephemeris command could generate that error.
{: .tip}

> <tip-title>Uninstalling tools</tip-title>
> While there is a function to accomplish this in [BioBlend](https://github.com/galaxyproject/bioblend), it has not been [included in Ephemeris yet.](https://github.com/galaxyproject/ephemeris/issues/83) If you're looking for a way to contribute to Galaxy, this would be great :)
{: .tip}

> <tip-title>"Recommended way" to install tools</tip-title>
> There is no one recommended way. Different people like to do different things. Some of us install things through the GUI still, some of us use ephemeris just for the automation.
>
> EU and others try and force all tool installation to go through ephemeris to make sure that rebuilding a server in case of disaster is easy. Anything not managed through their files will be lost.
>
> Others do a more mixed approach:
>
> > typically use Ephemeris, when I have to re-deploy a Galaxy instance (and thus install all tools) or during a Galaxy maintenance, when I want to update all installed toolshed tools in one go. Otherwise, when I need to install a specific version of one tool I use the GUI.
> {: .quote}
>
> There is no one right answer.
{: .tip}

> <tip-title>Tools not showing up? Refresh the toolbox!</tip-title>
> Sometimes the toolbox will fail to reload. You can correct for this by manually triggering the toolbox reload with a query:
>
> ```console
> curl -X PUT https://galaxy.example.org/api/configuration/toolbox -H "x-api-key: $GALAXY_API_KEY"
> ```
>
> This will request the toolbox to reload and you can check after if it's discovered your newly installed tools.
{: .tip}

<!--
{% snippet topics/admin/faqs/missed-something.md step=7 %}
-->

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tool-management" %}


topic: admin
tutorial: onedata-configuration
title: Configuring the Onedata connectors (remotes, Object Store, BYOS, BYOD)
difficulty: intermediate
time_estimation: 40m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["storage"]
content_hash: 13da590eec7f1e3cab25d0adc33d5f72
last_modified: 2026-05-27T09:01:47.637616
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/onedata-configuration/tutorial.html

content:
> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prerequisites

It's recommended that you have basic knowledge about Onedata and access 
to a Onedata ecosystem, so that you can test the configuration. If needed, follow 
[this tutorial]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html %})
first!


# Introduction

This tutorial will walk you through the configuration of four different types of Onedata 
connectors:

1. [Predefined Remote File Source](#predefined-remote-file-source) — provides all
   users with access to a predefined Onedata Space (e.g., for shared training
   datasets) without requiring individual configuration.

2. [BYOD (Remote File Source) templates](#byod-remote-file-source-templates) —
   similar to the Predefined Remote File Source, but uses templates and vault for secure
   credential storage, allowing users to configure multiple Onedata connections
   with their own credentials.

3. [Object Store — global Storage Location](#object-store---global-storage-location) — 
   configures Onedata as Galaxy's global storage backend (common for all users),
   where all datasets are stored in a specified Onedata Space.

4. [BYOS (Storage Location) templates](#byos-storage-location-templates) —
   enables users to configure their own Onedata Spaces as storage locations for
   their Galaxy datasets.


# Using Ansible?

The guides below pertain to files in the 
[Galaxy project codebase](https://github.com/galaxyproject/galaxy).
If you are using Ansible to manage your Galaxy deployment (which is the
recommended way for production instances), you will find tips at the end
of every subsection, showing how to achieve the same results with Ansible.

In case you are using jinja templates, make sure all of them are listed in
the `galaxy_config_templates` section for group_vars of each of the hosts.


# Common Configuration

Before configuring Onedata connectors, there are a few common elements that need
to be set up:

## Dependencies

For Remote File Sources (predefined and BYOD templates), install the
`fs.onedatarestfs` library (preferably adding it to the Python's requirements
file). Take a look at `lib/galaxy/dependencies/conditional-requirements.txt` for
the newest version:
```
fs.onedatarestfs>=21.2.5.2  # type: onedata, depends on onedatafilerestclient
```

For Object Store configuration, only `onedatafilerestclient` is required. However, 
if you're already installing `fs.onedatarestfs` for Remote File Sources, you don't 
need to add it separately as it's included as a dependency.

If you're only using Object Store functionality, you can add just:
```
onedatafilerestclient>=21.2.5.2
```

> <tip-title>Using Ansible</tip-title>
> Skip this step — conditional requirements should be collected automatically.
{: .tip}

## Vault Configuration

For BYOD and BYOS templates, you need to configure a vault to securely store user credentials:

1. Create `config/vault_conf.yml`:
```yaml
type: database
path_prefix: /galaxy
# Encryption keys must be valid fernet keys
# Generate them using Python:
# from cryptography.fernet import Fernet
# Fernet.generate_key().decode('utf-8')
encryption_keys:
  - KpVfbD7WfsNMxEnAEasql9aqSc1eCqR-S-tJCDyzcw8=
  - s1DDUEjZtFy2xknGIW2AiAE6EiZG5xSD0m_GzXi43G0=
  - qyb7f45l4XdkcuC5y06RlWtaENcPtUgHYekoQJBK1zM=
  # Do NOT use the keys above! Don't be lazy and generate your own.
```

The first key is used for encryption, while additional keys allow for key rotation. 

> <tip-title>Using Ansible</tip-title>
> Put the config in `templates/galaxy/config/vault_conf.yml.j2`, 
> like it has been done 
> [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/vault_conf.yml.j2).
>
> A `galaxy_vault_encryption_keys` var must be defined for the proper host in
> your playbook. For obvious reasons, it cannot be committed to your repository.
> For reference, take a look at [this PR](https://github.com/usegalaxy-eu/infrastructure-playbook/pull/1159).
{: .tip}

2. Enable it in `config/galaxy.yml`:
```yaml
galaxy:
  vault_config_file: vault_conf.yml
```

> <tip-title>Using Ansible</tip-title>
> Use the group vars file at `group_vars/gxconfig.yml`. 
> See [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
{: .tip}

For more details about vault configuration, see the [Galaxy Vault documentation](https://docs.galaxyproject.org/en/latest/admin/special_topics/vault.html).


# Predefined Remote File Source

While the [BYOD (Remote File Source) templates](#byod-remote-file-source-templates) 
allow users to configure their own Onedata access credentials, you can also set
up a predefined Onedata File Source that will be available to all users with
common configuration.

This approach is particularly useful when you want to provide access to shared
resources, such as public training datasets, without requiring user's access to
a Onedata ecosystem.

> <warning-title>Note the security of the credentials!</warning-title>
> This setup is perfect if you have a public Onedata access token that
> [can safely be shared](https://onedata.org/#/home/documentation/topic/stable/tokens-safe-publishing). If the token is not publicly shareable, use a vault to
> store it. Whatever the setup, always set the file source as `writable: false`,
> otherwise the users will be able to write (export) to the remote.
{: .warning}

Example configuration in `config/file_sources_conf.yml`:

```yaml
- type: onedata
  id: gtn_public_onedata
  label: GTN training data
  doc: Training data from the Galaxy Training Network (powered by Onedata)
  # The access token is public and can be shared
  onezone_domain: "datahub.egi.eu"
  writable: false
```

> <tip-title>Using Ansible</tip-title>
> Put the config in `templates/galaxy/config/file_sources_conf.yml.j2`, 
> like it has been done 
> [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/file_sources_conf.yml.j2).
{: .tip}

Note that you don't need to configure `user_preferences_extra_conf.yml` in this case, as all access parameters are specified directly in the File Sources configuration.


## Testing

Follow the Onedata 
[import]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html %})
guide.


# BYOD (Remote File Source) templates

BYOD (Bring Your Own Data) allows users to configure their own Onedata File
Sources using templates. This feature requires a vault for secure credential
storage, but users can create multiple File Sources with different
configurations, possibly connecting to different Onedata ecosystems.

1. Configure the vault as described in the [Vault Configuration section](#vault-configuration).

2. Enable the Onedata template by creating or modifying `config/file_source_templates.yml`:

   ```yaml
   - include: ./lib/galaxy/files/templates/examples/onedata.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Put the config in `templates/galaxy/config/file_source_templates.yml.j2`, 
   > like it has been done 
   > [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/file_source_templates.yml.j2).
   {: .tip}

3. Ensure there is the main Galaxy config file (`config/galaxy.yml`) and it includes the
   config file above:
   ```yaml
   galaxy:
     ...
     file_source_templates_config_file: file_source_templates.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Use the group vars file at `group_vars/gxconfig.yml`. 
   > See 
   > [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
   {: .tip}

This template defines a form that users will see when creating their own Onedata File Source. 
It collects the same configuration parameters that administrators specify when setting up 
a [Predefined Remote File Source](#predefined-remote-file-source) (like Onezone domain, access token, 
etc.), but allows users to provide their own values.

For required dependencies, see the [Dependencies section](#dependencies) in Common Configuration.


## Testing

Follow the Onedata 
[import]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html %})
and
[export]({% link topics/galaxy-interface/tutorials/onedata-remote-export/tutorial.html %})
guides.


# Object Store - global Storage Location

Galaxy supports various object store backends for storing datasets. While the [Object Store tutorial]({% link topics/admin/tutorials/object-store/tutorial.md %}) provides a comprehensive overview of the feature, this section focuses specifically on configuring Onedata as a global object store, available to all users.

To use Onedata as an object store:

1. Make sure there is a `config/object_store_conf.yml` configuration file with Onedata
   section. The relevant snippet can be found in config
   [samples](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/config/sample/object_store_conf.sample.yml).
   At the time of writing this tutorial, the config looks like the following:

   ```yaml
   type: onedata
   auth:
     access_token: MDAxY2xvY2F00aW9uIGRhdGFodWIuZWdpLmV1CjAw...
   connection:
     onezone_domain: datahub.egi.eu
     disable_tls_certificate_validation: false
   space:
     name: my-onedata-space
     galaxy_root_dir: ./galaxy-data
   cache:
     path: database/object_store_cache_onedata
     size: 1000
     cache_updated_data: true
   extra_dirs:
     - type: job_work
       path: database/job_working_directory_onedata
   ```
   
   Overwrite the following parameters with your values:
   - **access_token**: an access token suitable for data access (**write-enabled** and allowing calls to the Oneprovider REST API).
   - **onezone_domain**: the domain of the Onezone service (e.g., datahub.egi.eu).
   - **name**: the name of the Onedata Space where the Galaxy data will be stored.
   - **path**: the relative directory path in the Space under which the Galaxy data will be stored (optional, defaults to Space root).

   > <tip-title>Using Ansible</tip-title>
   > Put the config in `templates/galaxy/config/object_store_conf.yml.j2`.
   {: .tip}
   
2. Ensure there is the main Galaxy config file (`config/galaxy.yml`) and it includes the config:

   ```yaml
   galaxy:
     ...
     object_store_config_file: object_store_conf.yml
   ```
   
   > <tip-title>Using Ansible</tip-title>
   > Use the group vars file at `group_vars/gxconfig.yml`. 
   > See 
   > [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
   {: .tip}

> <warning-title>Switching object store types</warning-title>
> If you're switching from another object store type, you'll need to handle the migration of existing datasets. This process is not covered in this tutorial and should be carefully planned for production instances.
{: .warning}

For required dependencies, see the [Dependencies section](#dependencies) 
in Common Configuration.

For more details about object store configuration options and best practices, see the 
[Object Store tutorial]({% link topics/admin/tutorials/object-store/tutorial.md %}).


## Testing

Upload some new data to Galaxy, or run a workflow to produce results. Then
navigate to your Onedata account (e.g. [https://datahub.egi.eu](https://datahub.egi.eu))
and open the Space (and the path) that you have put down in the config. You
should see the Galaxy data:

![Data in Onedata](../../../galaxy-interface/images/onedata-byos/data-in-onedata.png)


# BYOS (Storage Location) templates

BYOS (Bring Your Own Storage) allows users to configure their own Onedata Storage Locations using templates. Similar to [BYOD](#byod-remote-file-source-templates), this feature requires a vault for secure credential storage.

1. Configure the vault as described in the [Vault Configuration section](#vault-configuration).

2. Enable the Onedata template by creating or modifying `config/object_store_templates.yml`:

   ```yaml
   - include: ./lib/galaxy/objectstore/templates/examples/onedata.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Put the config in `templates/galaxy/config/object_store_templates.yml.j2`, 
   > like it has been done 
   > [in this example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/object_store_templates.yml.j2).
   {: .tip}

3. Ensure there is the main Galaxy config file (`config/galaxy.yml`) and it includes the
   config file above:
   ```yaml
   galaxy:
     ...
     object_store_templates_config_file: object_store_templates.yml
   ```

   > <tip-title>Using Ansible</tip-title>
   > Use the group vars file at `group_vars/gxconfig.yml`. 
   > See 
   > [an example](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/group_vars/gxconfig.yml).
   {: .tip}

This template provides users with a form to configure their Onedata Storage Location, collecting information such as:
- Onezone domain and access token
- Space name and path for storing datasets
- Cache and performance settings

For required dependencies, see the [Dependencies section](#dependencies) 
in Common Configuration.


## Testing

Follow the Onedata 
[user-owned storage]({% link topics/galaxy-interface/tutorials/onedata-byos/tutorial.html %})
guide.


topic: admin
tutorial: cvmfs
title: Reference Data with CVMFS
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/cvmfs/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible", "git-gat"]
content_hash: 14341eb427eee377241f397fea2d0ea4
last_modified: 2026-05-27T09:01:47.635786
workflows_json: []
content:
> These words come from a transcript of Simon Gladman teaching this course. He
> is a bioinformatician at the University of Melbourne in Australia and also
> one of the administrators of Galaxy Australia.
>
> Hello everybody and welcome back to the Galaxy
> administrators course. in this session we're going to be talking about
> reference data in Galaxy using CVMFS.
>
> Hopefully by now you've all seen the video of
> the slides - if not I suggest you go and do
> look at those first as that explains how reference
> data works in Galaxy and also what CVMFS does to
> help us with that reference data.
>
> Some of the requirements from that we hope you've
> already completed: Galaxy server installation
> with Ansible and hopefully you
> understand what Ansible is and how
> it's used.
{: .spoken data-visual="gtn" data-target="#top-navbar" }

# Overview


The CernVM-FS {% cite Blomer_2011 %} is a distributed filesystem perfectly designed for sharing readonly data across the globe. We use it in the [Galaxy Project](https://galaxyproject.org) for sharing things that a lot of Galaxy servers need. Namely:
* **Reference Data**
    * Genome sequences for hundreds of useful species.
    * Indices for the genome sequences
    * Various bioinformatic tool indices for the available genomes
* **Tool containers**
    * [Singularity](https://www.sylabs.io/) containers of everything stored in [Biocontainers](https://biocontainers.pro/) (A bioinformatic tool container repository.) You get these for free every time you build a [Bioconda](https://bioconda.github.io/) recipe/package for a tool.
* Others too..

From the Cern website:

> The CernVM File System provides a scalable, reliable and low-maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications. CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs."
>
{: .quote id="cvmfs-quote" cite="https://cernvm.cern.ch/portal/filesystem"}

> All right, just a quick recap. CVMFS or
> Cern-VMFS is a distributed file system perfectly
> designed for sharing read-only data across the
> globe and we use it extensively in the Galaxy
> project for sharing things that
> a lot of Galaxy servers need.
> Namely all the reference data; so the genome
> sequences for all the different genomes that
> we need to think about in in Galaxy and
> bioinformatics. Things like a human genome,
> mouse genome, etc., etc. And we need a lot of
> indices for the genome sequences and lots of
> tool indices for all those genomes. And we also
> have a repository that contains tool containers.
> So singularity containers for all of the
> bioinformatics tools that we might want to use.
> There's a tutorial idea already done this week
> or coming up soon this week that explains how to
> use those singularity containers within Galaxy as
> well. So we're just going to get on with things.
{: .spoken data-visual="gtn" data-target="#cvmfs-quote"}

[A slideshow presentation on this subject]({% link topics/admin/tutorials/cvmfs/slides.html %}) is available. More [details are available on usegalaxy.org (Galaxy Main's) reference data setup](https://galaxyproject.org/admin/reference-data-repo/#usegalaxyorg-reference-data) and CVMFS system.

This exercise uses Ansible to install and configure CVMFS and Galaxy's access to CVMFS. For a tutorial that does not use Ansible and gives a closer look at how reference data is configured in Galaxy, see [the Reference Data with CVMFS without Ansible tutorial]({% link topics/admin/tutorials/cvmfs-manual/tutorial.md %}).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="cvmfs" %}

> The agenda we're going to follow today is: We're going to install and
> configure Galaxy CVMFS reference data using ansible. We're going to explore
> the CVMFS installation and then we're going to configure Galaxy to use it.
{: .spoken data-visual="gtn" data-target="#agenda"}

# Ansible-CVMFS and Galaxy

> There are a few different repositories that um Galaxy project has created and
> shared with everybody. The first one is the reference data and indices and
> this is in data.galaxyproject.org. And then we have another one called
> singularity.galaxyproject.org.
>
> If you want to have a look at what's in there you can click on the "data
> cache", and this link here will show you all the things that are inside this
> data.galaxyproject.org
{: .spoken data-visual="gtn" data-target="#ansible-cvmfs-and-galaxy" }

The Galaxy project supports a few CVMFS repositories.


| Repository                 | Repository Address              | Contents                                                                         |
| ----------                 | ------------------              | --------                                                                         |
| Reference Data and Indices | `data.galaxyproject.org`        | Genome sequences and their tool indices, Galaxy `.loc` files for them as well    |
| Singularity Containers     | `singularity.galaxyproject.org` | Singularity containers for everything in Biocontainers for use in Galaxy systems |
| Galaxy Main Configuration  | `main.galaxyproject.org`        | The configuration files etc for Galaxy Main (usegalaxy.org)                      |

You can browse the contents of `data.galaxyproject.org` at the [datacache](http://datacache.galaxyproject.org/).

## Installing and Configuring

Luckily for us, the Galaxy Project has a lot of experience with using and configuring CVMFS and we are going to leverage off that. To get CVMFS working on our Galaxy server, we will use the Ansible role for CVMFS written by the Galaxy Project. Firstly, we need to install the role and then write a playbook for using it.

If the terms "Ansible", "role" and "playbook" mean nothing to you, please checkout [the Ansible introduction slides]({% link topics/admin/tutorials/ansible/slides.html %}) and [the Ansible introduction tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %})

{% snippet topics/admin/faqs/ansible_local.md %}

> Okay, so now we're going to move on to installing and configuring Galaxy's
> CVMFS reference data with Ansible. We are going to do some Ansible here and
> we're going to install CVMFS onto our Galaxy server. Hopefully you all have
> access to a Galaxy server.
{: .spoken data-visual="gtn" data-target="#installing-and-configuring"}

> Here is mine. And on it I am an admin user and I have access to the admin
> page. This was done as part of the installation of Galaxy um and hopefully
> you've installed a tool.
{: .spoken data-visual="galaxy" data-target="/" data-action="goto"}

> I have bwa and bwa-mem installed under Mapping.
{: .spoken data-visual="galaxy" data-target=".search-input input" data-action="fill" data-value="bwa-mem"}

> So if I click on that you can see
> here that I have bwa-mem but there are no options available for reference
> genomes, so we want to fix that. We want to connect to all of Galaxy's
> pre-built references and so we're going to use Galaxy's CVMFS system to let
> our own Galaxies connect and get access to all the pre-built caches and
> everything we already have.
{: .spoken data-visual="galaxy" data-target="/?tool_id=bwa_mem" data-action="goto"}

> Okay, so let's get started. If we go back to our
> tutorial here, it says that we need to install a CVMFS role into our
> requirements.yml and then add it to our Ansible.
{: .spoken data-visual="gtn" data-target="#hands-on-installing-cvmfs-with-ansible"}

> <hands-on-title>Installing CVMFS with Ansible</hands-on-title>
>
> 1. In your working directory, add the CVMFS role to your `requirements.yml`
>
>    > Well the first thing I need to do is log into my Galaxy machine in the
>    > terminal.
>    {: .spoken data-visual="terminal" data-cmd="whoami; hostname -f"}
>
>    > Have a look at he contents of this directory and I'll go into galaxy and here we have all of the Ansible scripts that hopefully everybody already has.
>    {: .spoken data-visual="terminal" data-cmd="ls -al"}
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -16,3 +16,6 @@
>       version: 0.0.14
>     - name: galaxyproject.tusd
>       version: 0.0.1
>    +# CVMFS Support
>    +- src: galaxyproject.cvmfs
>    +  version: 0.3.3
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement" data-ref="add-req"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>    > Okay, so the first thing I'm going to do is I'm going to add the CVMFS
>    > role to the requirements.yml.
>    > Edit requirements.yml and we need to add this to the bottom of that file. Copy. Paste. And save it.
>    {: .spoken data-visual="terminal" data-ref="add-req"}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true" data-ref="req-install"}
>    {: .code-in}
>
>    > And now install the role into our local Ansible scripts using the
>    > ansible-galaxy command. And as you can see, it's downloading the CVMFS
>    > role.
>    {: .spoken data-visual="terminal" data-ref="req-install"}
>
>    > And if we look into roles now you can see that we have galaxyproject.cmfs
>    {: .spoken data-visual="terminal" data-cmd="ls roles/"}
>
>    > Right, clear the screen.
>    {: .spoken data-visual="terminal" data-cmd="clear"}
>
> 3. The variables available in this role are:
>
>    > Okay, now what we need to do is we need to run this role and that will
>    > install the CVMFS client onto our Galaxy server. So the first thing we
>    > need to do is edit our group files galaxyservers file. There are a bunch
>    > of different variables that we can set. We can set the cvmfs role to be
>    > a client or stratum zero, stratum one server. URLs - so where we're
>    > getting all this data from, and if you remember in the slideshow, the
>    > data can come from Europe or America or Australia. Which repositories
>    > we want to have installed. And this is an important one - the quota
>    > limit. This is basically saying that CVMFS will cache some data on your
>    > local machine and the quota limit is the maximum size of that cache.
>    {: .spoken  data-visual="gtn" data-target="#variables-table"}
>
>    | Variable             | Type          | Description                                                                                                                                                                    |
>    | ----------           | -------       | -------------                                                                                                                                                                  |
>    | `cvmfs_role`         | string        | Type of CVMFS host: `client`, `stratum0`, `stratum1`, or `localproxy`. Controls what packages are installed and what configuration is performed.                               |
>    | `cvmfs_keys`         | list of dicts | Keys to install on hosts of all types.                                                                                                                                         |
>    | `cvmfs_server_urls`  | list of dicts | CVMFS server URLs, the value of `CVMFS_SERVER_URL` in `/etc/cvmfs/domain.d/<domain>.conf`.                                                                                     |
>    | `cvmfs_repositories` | list of dicts | CVMFS repository configurations, `CVMFS_REPOSITORIES` in `/etc/cvmfs/default.local` plus additional settings in `/etc/cvmfs/repositories.d/<repository>/{client,server}.conf`. |
>    | `cvmfs_quota_limit`  | integer in MB | Size of CVMFS client cache. Default is `4000`.                                                                                                                                 |
>    {: id="variables-table"}
>
>    But, luckily for us, the Galaxy Project CVMFS role has a lot of defaults for these variables which we can use by just setting `galaxy_cvmfs_repos_enabled` to `config-repo`. We will also set the `cvmfs_quota_limit` to something sensible (500MB) as we have relatively small disks on our instances. In a production setup, you should size this appropriately for the client.
>
>    > <tip-title>What is a good size for this?</tip-title>
>    > In production UseGalaxy.org.au uses 100GB, different sites have different needs and you can make your cache smaller depending on your usage. E.g. if your users only use one dataset from the reference data (e.g. just hg38) then perhaps you don't need such a large cache.
>    {: .tip}
>
>    > Okay but instead of just modifying galaxyservers.yml and adding in some
>    > of these variables - instead what we're going to do here is - we're
>    > going to create a new group file called all.yml.
>    >
>    > Because one of the things that we may want to do in the future - is we
>    > may want to create other machines. We might want to have worker nodes
>    > for our Galaxy cluster; or we may want to have other machines that we
>    > want to be able to create using these Ansible scripts that also have the
>    > CVMFS role installed. And instead of reproducing these variables in each
>    > of the group var files for those particular machines, we can create a
>    > special group vars file called all.yml. And whatever we put in there
>    > will be automatically available to all machines that we create with
>    > Ansible from this directory. Hopefully that makes a bit of sense and
>    > we're going to use that a bit later on if you come along to the pulsar
>    > tutorial where we will also be installing CVMFS on another machine - on
>    > a remote machine to run Pulsar.
>    {: .spoken data-visual="terminal" data-cmd="ls group_vars/"}
>
>    Add the following lines to your `group_vars/all.yml` file, creating it if it doesn't exist:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/all.yml
>    +++ b/group_vars/all.yml
>    @@ -5,3 +5,8 @@ pip_virtualenv_command: /usr/bin/python3 -m venv  # usegalaxy_eu.certbot, usegal
>     # Common variables needed by all hosts
>     galaxy_user_name: galaxy
>     galaxy_db_name: galaxy
>    +
>    +# CVMFS vars
>    +cvmfs_role: client
>    +galaxy_cvmfs_repos_enabled: config-repo
>    +cvmfs_quota_limit: 500
>    {% endraw %}
>    ```
>    {: data-commit="Configure CVMFS variables" data-ref="vars-all"}
>
>    > So what we're going to do, is we're going to create a new file called
>    > groupvars/all.yml and we're going to put some of these CVMFS variables
>    > inside it. So i'll just copy that. Okay, groupvars all dot yaml. And
>    > I'll paste this in. So basically the CVMFS role we want this machine to
>    > have is client. Which means that we just want it to be able to access
>    > all of our CVMFS reference data. And then we want we're going to set
>    > this one here to say that, "yes we want to set this up for Galaxy." And
>    > the config repo is the one that tells CVMFS how to set everything else
>    > up. And then this is the other important one - the CVMFS quota limit.
>    > We're setting to 500 megabytes and that's just so we don't fill the root
>    > disk of these machines. So I'll save that.
>    {: .spoken data-visual="terminal" data-ref="vars-all"}
>
>    > <tip-title>Why all.yml?</tip-title>
>    > We've integrated the cvmfs and pulsar tutorials better, such that CVMFS will be used for Pulsar as well, this configuration will be needed on all of our machines. This mirrors real life where you want CVMFS on every node that does computation.
>    {: .tip}
>
> 4. Add the new role to the list of roles under the `roles` key in your playbook, `galaxy.yml`:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -37,6 +37,7 @@
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>         - galaxyproject.gxadmin
>    +    - galaxyproject.cvmfs
>       post_tasks:
>         - name: Setup gxadmin cleanup task
>           ansible.builtin.cron:
>    {% endraw %}
>    ```
>    {: data-commit="Add role to playbook" data-ref="pb"}
>
>    > And now we need to add the role - we need to add the role to our Galaxy
>    > playbook. So we edit galaxy.yml which is our playbook and we just need to
>    > add the galaxyproject.cvmfs to the bottom of this. galaxyproject.cvmfs.
>    > And that's pretty much it.
>    {: .spoken data-visual="terminal" data-ref="pb"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true" data-ref="run-pb1"}
>    {: .code-in }
{: .hands_on}

> Okay now we just run the playbook. ansible-playbook. We want to run the
> Galaxy one. So. All right, we're about to get to the CVMFS repo here. We are
> now it's installing the um the apt package of CVMFS. It's going to get
> that out of this special um Cern apt repository. Okay, it's installing
> it. Hopefully it won't take too long. Okay, now setting up the
> repositories. And it's done.
{: .spoken data-visual="terminal" data-ref="run-pb1"}

Congratulations, you've set up CVMFS.

## Exploring the CVMFS Installation


> Okay, that was completed. We've now installed CVMFS client onto our machine
> and we've told it to go looking for certain repositories. Now to get access
> to them. We'll see what's in them. uh They'll be located at slash cvmfs. So
> under the cvmfs directory in your root directory. So, we can go to that. cd
> /
{: .spoken data-visual="terminal" data-cmd="cd /"}

> Do an ll. You can see here there's a directory here called cvmfs.
{: .spoken data-visual="terminal" data-cmd="ls -al"}

>  So we'll go in there and have a look and see what's in there.
{: .spoken data-visual="terminal" data-cmd="cd /cvmfs"}

> So let's have a look. Oh there's nothing in there!
> Well actually, what what's going to happen is; as soon as we go looking for
> something in this directory, so autofs will automatically mount the
> particular thing we're looking for. And so what we're going to do here is I'm
> going to go: cd data.galaxyproject.org, because I know that's one of the one
> of the repositories that should have been installed.
{: .spoken data-visual="terminal" data-cmd="ls -al"}

> And when I do that, autofs is automatically going to mount it for me on the
> fly. Like that.
{: .spoken data-visual="terminal" data-cmd="cd data.galaxyproject.org"}

> And now I've cd'd into it and if I do an ll, you can see here I've got some
> things in here now. I've got byhand and managed.
{: .spoken data-visual="terminal" data-cmd="ls -al"}

>  If I go into byhand you can see here that I have quite a lot of different
>  genomes and their tool indices.
{: .spoken data-visual="terminal" data-cmd="cd byhand; ls"}

> So if I'm going to, say, sacCer2, I can see in here there are bowtie index,
> the bwa index, the uh the picard index, the sam index; a whole bunch of
> other different things - including the original sequences etc. So yeah,
> quite a lot of data and we just have access to that on the fly. And then as
> soon as we try and look at any of these files, what will happen is CVMFS
> will automatically cache it to the local disk within within that 500
> megabyte cache that we uh we set up earlier. This is really cool. And we can
> tell Galaxy to look at all of this data and use it as its reference data.
{: .spoken data-visual="terminal" data-cmd="ls -al sacCer2/"}

> <hands-on-title>Exploring CVMFS</hands-on-title>
>
> 1. SSH into your machine
>
> 2. Change directory into `/cvmfs/` and list the files in that folder
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > You should see nothing, as CVMFS uses `autofs` in order to mount paths only upon request.
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
>
> 3. Change directory into `/cvmfs/data.galaxyproject.org/`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > cd /cvmfs/data.galaxyproject.org/
>    > ls
>    > ls byhand
>    > ls managed
>    > ```
>    {: .code-in}
>
>    > <question-title></question-title>
>    >
>    > What do you see now?
>    >
>    > > <solution-title></solution-title>
>    > >  You'll see `.loc` files, genomes and indices.
>    > > AutoFS only mounts the files when they're accessed, so it appears like there is no folder there.
>    > {: .solution }
>    >
>    {: .question}
>
>    And just like that we all have access to all the reference genomes and associated tool indices thanks to the Galaxy Project, IDC, and Nate's hard work!
>
{: .hands_on}

## Configuring Galaxy to use the CVMFS references.

Now that we have mounted the CVMFS repository we need to tell Galaxy how to find it and use it.

There are two primary directories in the reference data repository:

| Directory   | Contents                                                                                                                                                                  |
| ----------- | ----------                                                                                                                                                                |
| `/managed`  | Data generated with Galaxy Data Managers, organized by data table (index format), then by genome build.                                                                   |
| `/byhand`   | Data generated prior to the existence/use of Data Managers, manually curated. (For legacy reasons, this directory is shared as `/indexes` on the HTTP and rsync servers.) |

These directories have somewhat different structures:

* `/managed` is organized by index type, then by genome build (Galaxy dbkey)
* `/byhand` is organzied by genome build, then by index type

Both directories contain a location subdirectory, and each of these contain a `tool_data_table_conf.xml` file:

* `/managed/location/tool_data_table_conf.xml`
* `/byhand/location/tool_data_table_conf.xml`

Galaxy consumes these `tool_data_table_conf.xml` files and the `.loc` "location" files they reference. The paths contained in these files are valid if the data is mounted via CVMFS.

Examples of data include:

* twoBit (`.2bit`) and FASTA (`.fa`) sequence files
* Bowtie 2 and BWA indexes
* Multiple Alignment Format (`.maf`) files
* SAMTools FASTA indexes (`.fai`)

Now all we need to do is tell Galaxy how to find it! This tutorial assumes that you have run the tutorial in the requirements, [Galaxy Installation with Ansible]({% link topics/admin/tutorials/ansible-galaxy/tutorial.md %}). The hands-on below will use the Galaxy Project Ansible role to configure everything.
{: id="spoken-7"}

> So that's what we're going to do now. Okay. So now we're going to try and
> configure Galaxy to use this CVMFS data. And and to have it so that we can run
> things like bwa and bwa mem and run them against the human genome or the bee
> genome or the mouse genome and take advantage of the fact that a lot of other
> people in the Galaxy community have done a lot of work for reference data for
> us already.
>
> So the way to do this is we're going to edit the groupvars galaxyservers
> file and we're going to add a variable called tool_data_table_config_path.
> And then we're going to point it to the two files that are in - there's one
> in byhand and one in managed.
{: .spoken  data-visual="gtn" data-target="#spoken-7"}


> If we go into byhand you can see here. And then we're going to location. As
> you can see in here, are all the lock files, but you'll also see there's an
> xml file here called tool_data_table_conf xml and we're going to point Galaxy
> at this file and there's another one in the same position in managed.
{: .spoken data-visual="terminal" data-cmd="cd /cvmfs/data.galaxyproject.org/byhand;  ls location;"}

> And you can see here there's another one in managed there. And so we're going to
> add both of these files to our Galaxy configuration and then Galaxy will be
> able to use all of the data contained within this repository.
{: .spoken data-visual="terminal" data-cmd="cd ../; ls managed/location"}

> Okay, so we'll go back to our Ansible directory.
{: .spoken data-visual="terminal" data-cmd="cd ~/galaxy/"}

> <hands-on-title>Configuring Galaxy to use CVMFS</hands-on-title>
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add a `tool_data_table_config_path` entry under the `galaxy` key of the `galaxy_config` section in the `group_vars/galaxyservers.yml` file. This new entry should be a list containing the paths to both `tool_data_table_conf.xml` files referenced above.
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -76,6 +76,8 @@ galaxy_config:
>         # TUS
>         galaxy_infrastructure_url: "https://{{ inventory_hostname }}"
>         tus_upload_store: "{{ galaxy_tus_upload_store }}"
>    +    # CVMFS
>    +    tool_data_table_config_path: /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml,/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add tool_data_table_config_path to group variables" data-ref="gvconf"}
>
>    > This time I'm going to edit the groupvars galaxyservers.yml file. And in
>    > our Galaxy section. Which is here. Which is here. At the bottom of that
>    > I'm going to add a variable called tool_data_table_config_path. And I'm
>    > going to point it to the locations that we um I showed you before. um I
>    > can't remember what they are off the top of my head but luckily they're
>    > inside this solution box and so I will just copy them. And paste. And as
>    > you can see pointing to /cvmfs/data.galaxyproject.org/byhand/location
>    > and then that tool data table conf xml file. And then we have a list
>    > here and we separate it by commas and then we point it to the second
>    > one. Right, so we save this file.
>    {: .spoken data-visual="terminal" data-ref="gvconf"}
>
> 2. Re-run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true" data-ref="pb-run2"}
>    {: .code-in }
>
>    > So this time what we're going to do, all we're doing is making a minor change
>    > to the Galaxy yaml file in the Galaxy config to add that one line and then
>    > we're going to restart Galaxy. And yeah, Galaxy will suddenly automatically
>    > have access to all of that data. So you can see here we've changed the Galaxy
>    > configuration file. And then Galaxy is now restarting and it's done.
>    {: .spoken data-visual="terminal" data-ref="pb-run2"}
>
> 3. Install the BWA-MEM tool, if needed.
>
>    {% snippet topics/admin/faqs/install_tool.md query="bwa" name="Map with BWA-MEM" section="Mapping" %}
>
> 4. In your Galaxy server, open the **Map with BWA-MEM** {% icon tool %} tool. Now check that there are a lot more reference genomes available for use!
>
>    ![available_genomes.png](../../images/available_genomes.png)
>
> 5. Login to Galaxy as the admin user, and go to **Admin → Data Tables → bwa_mem indexes**
>
>    ![bwa_mem_indices.png](../../images/bwa_mem_indices.png)
>
{: .hands_on}

> Okay let's go and have a look at our Galaxy server and see if bwa can
> suddenly see all of those - that stuff. All right so we're back on our Galaxy
> server. I'll click on Analyze Data to just to reload the page.
{: .spoken data-visual="galaxy" data-target="#analysis a" data-action="click"}

> I'll go back to Mapping
{: .spoken data-visual="galaxy" data-target=".search-input input" data-action="click"}

>  and load bwa-mem.
{: .spoken data-visual="galaxy" data-target=".search-input input" data-action="fill" data-value="bwa-mem"}

> And then suddenly, instead of
> having no options available, you can see here we've got the Bee genome.
{: .spoken data-visual="galaxy" data-target="/?tool_id=bwa_mem" data-action="goto"}

>  Now click on that. Oh look at that, there are lots and lots and lots of
>  available genomes now including: lots of human, mouse, rat, yeast, all sorts
>  of things. And in fact if you want to see the list of all the different
>  available genomes now, that we have available to us.
>
>  If you go to admin. We go to data tables over here. You can see here that we
>  have um a couple of data tables for managed and for all fasta. So if we
>  click on that one, you can see that we have a lot of genomes available now
>  in the all fasta data table that Galaxy can get access to. If we go back to
>  the data tables again, and go down to bwa indexes or bwa mem indexes here.
>  You can see we have access to a lot of pre-built indexes for bwa for all of
>  these different genomes. That is pretty powerful.
>
>  So what did that take us? Maybe 30 minutes? And uh suddenly our Galaxy
>  server has access to all the uh the data the reference data and the tool
>  indices that the community have built over a number of years and it's super
>  simple.
{: .spoken data-visual="galaxy" data-target="s2id_field-uid-12_select a" data-action="click"}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=5 %}

# Common Production Questions

> <question-title>For the most used datasets (for ex. hg38) could we have a local copy, or would that be irrelevant?</question-title>
> This would be irrelevant, the most used datasets will stay in the cache. CVMFS uses a Least Recently Used (LRU) cache (see their [docs](https://cvmfs.readthedocs.io/en/latest/cpt-details.html#disk-cache)), so whenever it runs out of space, it will remove the least recently used file. If you have a file that is very commonly used, it will remain in the cache.
{: .question}

> <question-title>Could you explain how to calculate a good cache space?</question-title>
> Here are two approaches, there are others:
> 1. Allocate some cache, see how it is, make it larger if it is fully used + users complain of speed.
> 2. Enable reference data, and collect a week or two of data, analyse which reference datasets are being used, and allocate enough space for all of them.
>
> Essentially you just need data on how your users will behave and what reference data they want, combined with "when will they accept a wait period" to determine how much space you must allocate.
{: .question}

> <question-title>If I use a cluster, will I need to configure this FS in each node (given that the folder is at / directly)?</question-title>
> Yes. Often admins with a cluster keep a smaller cache local to each compute node, and then setup a Squid proxy to hold the most commonly accessed data on a machine with more storage. E.g. each compute node could have 10-50GB of CVMFS storage while you might setup a Squid proxy with 200-300 GB of storage that will store everything your site uses.
{: .question}

> <tip-title>Debugging failed mounting</tip-title>
>
> Are you having issues mounting your CVMFS mount? Is it giving strange errors like "Endpoint not connected"
> Try running this command as root:
>
> > > <code-in-title>Bash</code-in-title>
> > > ```console
> > > /usr/bin/cvmfs2 -d -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint singularity.galaxyproject.org /mnt
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Consolue</code-out-title>
> > > ```console
> > > Debug: using library /usr/lib/libcvmfs_fuse3_stub.so
> > > CernVM-FS: running in debug mode
> > > CernVM-FS: loading Fuse module... (cvmfs) Parsing config file /etc/cvmfs/default.conf    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) execve'd /bin/sh (PID: 280373)    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) Parsing config file /etc/cvmfs/default.d/50-cern-debian.conf    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) execve'd /bin/sh (PID: 280375)    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) Parsing config file /etc/cvmfs/default.d/80-ansible-galaxyproject-cvmfs.conf    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) execve'd /bin/sh (PID: 280378)    [07-21-2022 11:11:20 UTC]
> > > [...]
> > > (dns) empty hostname    [07-21-2022 11:11:20 UTC]
> > > (download) installed 1 proxies in 1 load-balance groups    [07-21-2022 11:11:20 UTC]
> > > (cvmfs) DNS roaming is disabled for this repository.    [07-21-2022 11:11:20 UTC]
> > > (catalog) constructing client catalog manager    [07-21-2022 11:11:20 UTC]
> > > (catalog) Initialize catalog    [07-21-2022 11:11:20 UTC]
> > > (cache) unable to read local checksum    [07-21-2022 11:11:20 UTC]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfspublished to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfspublished    [07-21-2022 11:11:20 UTC]
> > > (download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 0)    [07-21-2022 11:11:20 UTC]
> > > (cache) miss ./e2/ab48b0984729d99951cb62c4312f501b3ddc6b (-2)    [07-21-2022 11:11:20 UTC]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX    [07-21-2022 11:11:20 UTC]
> > > (download) Verify downloaded url /data/e2/ab48b0984729d99951cb62c4312f501b3ddc6bX, proxy DIRECT (curl error 0)    [07-21-2022 11:11:20 UTC]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfswhitelist to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/.cvmfswhitelist    [07-21-2022 11:11:20 UTC]
> > > [...]
> > > (download) escaped http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/c7/f1555f421b1868b979291dc23f34a83132eadbC to http://cvmfs1-psu0.galaxyproject.org/cvmfs/singularity.galaxyproject.org/data/c7/f1555f421b1868b979291dc23f34a83132eadbC    [07-21-2022 11:11:20 UTC]
> > > (download) Verify downloaded url /data/c7/f1555f421b1868b979291dc23f34a83132eadbC, proxy DIRECT (curl error 0)    [07-21-2022 11:11:25 UTC]
> > > (cache) finished downloading of /data/c7/f1555f421b1868b979291dc23f34a83132eadbC    [07-21-2022 11:11:25 UTC]
> > > (cache) commit ./c7/f1555f421b1868b979291dc23f34a83132eadb ./txn/fetchJWcwtt    [07-21-2022 11:11:25 UTC]
> > > (quota) pin into lru c7f1555f421b1868b979291dc23f34a83132eadb, path file catalog at singularity.galaxyproject.org:/ (c7f1555f421b1868b979291dc23f34a83132eadb)    [07-21-2022 11:11:25 UTC]
> > > (cache) commit failed: cannot pin c7f1555f421b1868b979291dc23f34a83132eadb    [07-21-2022 11:11:25 UTC]
> > > (catalog) failed to load catalog '' (2 - not enough space to load catalog)    [07-21-2022 11:11:25 UTC]
> > > (catalog) failed to initialize root catalog    [07-21-2022 11:11:25 UTC]
> > > Failed to initialize root file catalog (16 - file catalog failure)
> > > (cache) unpinning / unloading all catalogs    [07-21-2022 11:11:25 UTC]
> > > ```
> > {: .code-out}
> {: .code-2col}
{: .tip}

# Other Aspects

> Right, we'll go back to our tutorial. um Yeah. Just finally, just before we
> finish up. um If we are developing a new tool and you want to add a reference
> genome or a different index just give us - drop us a line on Gitter and we'll
> be able to add it into our - into the reference data for the community. um
> We're looking at automating the process of building all of this material
> using data managers and ephemeris. And we're working with a group of people
> called the IDC, which is the Intergalactic Data Commission, which is a funny
> name for everyone in Galaxy - in the Galaxy community who likes reference
> data. And we're looking at making a community controlled resource that will
> be semi-automatic. One of the other things that you can do is have automatic
> fallbacks. So, if say, you're in Australia and you're hooked up to the
> Australian mirror of the CVMFS repository and the Australian mirror dies, the
> CVMFS client is smart enough to automatically go to the next closest one and
> so you won't lose anything. If you're interested in looking at plant data
> there's a link here for that.
{: .spoken data-visual="gtn" data-target="#other-aspects" }

## Development

If you are developing a new tool, and want to add a reference genome, we recommend you [talk to us on Gitter](https://gitter.im/galaxy-iuc/iuc). You can also look at one of the tools that uses reference data, and try and copy from that. If you’re developing the location files completely new, you need to write the data manager.

## Automation

You can automate the process of installing and setting up data managers and data with ephemeris. We're working in the [IDC](https://github.com/galaxyproject/idc) to democratise this CVMFS repository, and make this a community-controlled resource. You can also look here for ideas on automating your data management.

## Access Control

It is not easily possible to filter access to reference data depending on the user's role or group.

You could set up a tool per user/group, [secure access to running this tool](https://galaxyproject.org/admin/config/access-control/), and then allow this private tool to access a private tool data table. But you will not get tool updates, you will have to copy and edit this tool every time it gets updated. Or write more advanced job control rules to reject specific jobs which use specific datasets.

## Proxying Recap

The client talks directly to the stratum 1 (or to a proxy), and manages the data, and exposes it to the user. The proxy stores an opaque cache, that can't really be used, except as a proxy to a stratum 1.

## Plant Data

If you are working with plants, you can find separate reference data here: [frederikcoppens/galaxy_data_management](https://github.com/frederikcoppens/galaxy_data_management)

> And finally if you could please click on this link here and give us some
> feedback on how you think the tutorial went, whether it was useful, if you
> enjoyed it or um if you have any criticisms, could you please put them in
> here as well. And if you end up using this to build a Galaxy server and you
> publish that Galaxy server um could you cite the tutorial for us please. That
> would be make a big difference to us. All right, thank you very much and I
> hope you enjoyed it and hopefully I'll get to meet some of you in person one
> day soon at a Galaxy conference. Thank you and goodbye.
{: .spoken data-visual="gtn" data-target="#feedback"}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="cvmfs" %}


