# tutorials

topic: assembly
tutorial: assembly-with-preprocessing
title: Unicycler assembly of SARS-CoV-2 genome with preprocessing to remove human genome reads
difficulty: intermediate
time_estimation: "4h" # plus additional time for (optional) NCBI SRA downloads
tools_json: []
requirements_json: []
tags_json: ["covid19", "biodiversity"]
content_hash: ab2a461d6e6cd227b9ef23dc30cf479b
last_modified: 2026-02-26T21:12:14.400191
zenodo_link: https://zenodo.org/record/3732359
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/assembly-with-preprocessing/tutorial.html

content:
In some research or clinical contexts it is not possible, or very hard, to
purify DNA/RNA for sequencing from just the specimen of interest.
Instead you will isolate DNA that is contaminated, sometimes heavily, with
DNA/RNA of a different origin.
This is the case for example with microbiome samples, which typically display
considerable contamination with host DNA, or with samples of body fluids for
pathogen detection. Such contamination can pose an issue with certain types of
analyses, in particular with genome assembly.

This tutorial guides you through the preprocessing of sequencing data of
bronchoalveolar lavage fluid (BALF) samples obtained from early COVID-19
patients in China. Since such samples are expected to be contaminated
signficantly with human sequenced reads, the goal is to enrich the data for
SARS-CoV-2 reads by identifying and discarding reads of human origin before
trying to assemble the viral genome sequence.

> <comment-title>The usegalaxy.* COVID-19 analysis project</comment-title>
> This tutorial uses the same data as, and recapitulates to a large extent, the
> [Pre-processing](https://covid19.galaxyproject.org/genomics/1-PreProcessing/)
> and [Assembly](https://covid19.galaxyproject.org/genomics/2-Assembly/) steps
> of the [Genomics](https://covid19.galaxyproject.org/genomics/) section of
> [covid19.galaxyproject.org](https://covid19.galaxyproject.org/).
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

We are going to create an assembly of the SARS-CoV-2 genome based on mixed
short-reads (Illumina) and long-reads (Nanopore) data from a total of six
different samples, all of which are publicly accessible through the NCBI
and EBI short-reads archives (SRAs).

Since automated data downloads from SRAs can be unreliable at times and could
get broken by changes to the download interface on the NCBI/EBI side, this
tutorial offers two ways to access the sequenced reads input data:

1. Direct download from the NCBI SRA based on accession numbers and using the
   dedicated **Faster Download** tool

   Use this method if it works and is fast enough for you, and if you are
   interested in learning to obtain short-reads data directly from NCBI, in
   general.

2. Download of the same data deposited as a copy at [Zenodo](https://zenodo.org/record/3732359)
   This method uses Galaxy's generic data import functionality, and should be
   very reliable and faster than the download from NCBI.
   It also showcases **rule-based** uploads and demonstrates how they can be
   used to download several datasets and to arrange them into easy to handle
   data structures at the same time.

   > <details-title>Rule-based uploads</details-title>
   > In this tutorial you will only use the features of Galaxy's rule-based
   > uploader that are required to get the input data ready for our analysis,
   > and we will not explain those features in much detail.
   >
   > If, after this first taste, you are interested in a thorough introduction
   > we recommend the advanced tutorial
   > [Collections: Rule Based Uploader](../../../galaxy-interface/tutorials/upload-rules/tutorial.html).
   {: .details}

   Use this method if the direct download from the NCBI SRA does not work, or
   is too slow for your time frame, or if you are interested in advanced use
   of Galaxy's data import functionality.

The corresponding two step-by-step instructions below have been crafted to
produce identically arranged data structures in your history so all subsequent
steps are independent of the data source you choose.

## Get data from NCBI SRA

> <hands-on-title>Data upload to Galaxy from NCBI SRA</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Create a new dataset listing the SRA accession numbers of the Illumina paired-end input data for this tutorial:
>
>    ```
>    SRR10903401
>    SRR10903402
>    SRR10971381
>    ```
>
>    call it, *e.g.*, `Illumina accessions` and set its datatype to `tabular`.
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" %}
>
> 3. Create another new dataset listing the SRA accession numbers of the Nanopore input data for this tutorial:
>
>    ```
>    SRR10948550
>    SRR10948474
>    SRR10902284
>    ```
>
>    call it, *e.g.*, `Nanopore accessions` and set its datatype to `tabular`.
>
> 4. Add `#illumina`/`#nanopore` tags to the datasets
>
>    > <comment-title>Name tags in the analysis</comment-title>
>    > We are going to treat the Illumina- and the Nanopore-sequenced data
>    > separately in this tutorial up to the actual genome assembly step.
>    >
>    > To make it easier to keep track of the two branches of the analysis, we
>    > recommend the use of Galaxy's dataset **name tags**.
>    > A name tag will automatically propagate to any new dataset derived
>    > from the tagged dataset.
>    {: .comment}
>
>    You can create a name tag by attaching a tag starting with `#` to any
>    dataset.
>
>    Name tags are meant to help you identify the origin of datasets quickly.
>    Feel free to either use the suggested names above or choose ones you like.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 4. Retrieve the Illumina reads data from NCBI:
>
>    Run **Faster Download and Extract Reads in FASTQ** {% icon tool %} with
>    the following parameter settings:
>    - *"select input type"*: `List of SRA accession, one per line`
>      - {% icon param-file %} *"sra accession list"*:
>        the `Illumina accessions` dataset created above
>    - in *"Advanced Options"*
>      - *"Select how to split the spots"*: `--split-3`
>
>    The tool run should generate four new items in your history - three
>    collections and one *log* dataset with a summary of what happened.
>
>    Since all three datasets that we tried to retrieve contain only
>    paired-end reads, only the `Pair-end data` collection is expected to
>    contain downloaded data. Click on the other two collections to verify that
>    they are empty, then delete them from your history (since the collections
>    do not contain any datasets, it does not matter, which delete option -
>    "Collection Only", "Delete Datasets" or "Permanently Delete Datasets" you
>    are choosing when prompted).
>
> 5. Retrieve the Nanopore reads data from NCBI:
>
>    Run **Faster Download and Extract Reads in FASTQ** {% icon tool %} with
>    the following parameter settings:
>    - *"select input type"*: `List of SRA accession, one per line`
>      - {% icon param-file %} *"sra accession list"*:
>        the `Nanopore accessions` dataset created above
>    - in *"Advanced Options"*
>      - *"Select how to split the spots"*: `--split-3`
>
>    As in the previous step, the tool run should generate four new items in
>    your history.
>
>    Since all three datasets that we tried to retrieve in this run contain
>    only single-end reads, only the `Single-end data` collection is expected to
>    contain downloaded data this time. Click on the other two collections to
>    verify that they are empty, then delete them from your history.
>
{: .hands_on}


## Get data from Zenodo

> <hands-on-title>Data upload to Galaxy from Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import Illumina-sequenced reads data from [Zenodo](https://zenodo.org/record/3732359)
>
>    The Zenodo links for the data are these:
>    ```
>    https://zenodo.org/record/3732359/files/SRR10903401_r1.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10903401_r2.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10903402_r1.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10903402_r2.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10971381_r1.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10971381_r2.fq.gz
>    ```
>
>    To upload these data to your Galaxy history in structured form:
>
>      - Copy the above list of links
>      - Open the Galaxy Upload Manager
>        ({% icon galaxy-upload %} on the top right of the tool panel)
>      - In the pop-up window, switch to the **Rule-based** tab and select
>        - *"Upload data as:"*: `Collection(s)`
>        - *"Load tabular data from:"*: `Pasted Table`
>        - Paste the copied links into the text field
>        - Click **Build**
>      - In the next screen, select
>        - {% icon param-repeat %} *"Column"*: `Using a Regular expression`
>          - *"From Column"*: `A`
>          - {% icon param-check %} *"Create columns matching expression groups."*
>          - *"Regular Expression"*: `.+/(SRR\d+)_r(\d).fq.gz`
>          - *"Number of Groups"*: `2`
>          - Click **Apply**
>        - {% icon param-repeat %} *"Rules"*: `Add / Modify Column Definitions`
>          - {% icon param-repeat %} *"Add Definition"*: `URL`
>            - *"URL"*: `A`
>          - {% icon param-repeat %} *"Add Definition"*: `List Identifier(s)`
>            - *"List Identifier(s)"*: `B`
>          - {% icon param-repeat %} *"Add Definition"*: `Paired-end Indicator`
>            - *"Paired-end Indicator"*: `C`
>          - Click **Apply**
>        - *"Type"*: `fastqsanger.gz`
>        - *"Name"*: `Illumina PE data` (or similar)
>        - *"Add nametag for name:"* {% icon param-check %}
>
>          > <comment-title>Name tags in the analysis</comment-title>
>          > We are going to treat the Illumina- and the Nanopore-sequenced data
>          > separately in this tutorial up to the actual genome assembly step.
>          >
>          > To make it easier to keep track of the two branches of the analysis, we
>          > recommend the use of Galaxy's dataset **name tags**.
>          > A name tag will automatically propagate to any new dataset derived
>          > from the tagged dataset.
>          {: .comment}
>
>          Checking this option tells Galaxy to reuse the collection name above
>          as a name tag on the collection.
>
>        - Click **Upload**
>
> 3. Import Nanopore-sequenced reads data from [Zenodo](https://zenodo.org/record/3732359)
>
>    ```
>    https://zenodo.org/record/3732359/files/SRR10902284_ONT.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10948474_ONT.fq.gz
>    https://zenodo.org/record/3732359/files/SRR10948550_ONT.fq.gz
>    ```
>
>    Again, we want to upload this data to our history in structured form.
>    To do so:
>
>      - Copy the above list of links
>      - Open the Galaxy Upload Manager
>        ({% icon galaxy-upload %} on the top right of the tool panel)
>      - In the pop-up window, switch to the **Rule-based** tab and select
>        - *"Upload data as:"*: `Collection(s)`
>        - *"Load tabular data from:"*: `Pasted Table`
>        - Paste the copied links into the text field
>        - Click **Build**
>      - In the next screen, select
>        - {% icon param-repeat %} *"Column"*: `Using a Regular expression`
>          - *"From Column"*: `A`
>          - {% icon param-check %} *"Create columns matching expression groups."*
>          - *"Regular Expression"*: `.+/(SRR\d+)_ONT.fq.gz`
>          - *"Number of Groups"*: `1`
>          - Click **Apply**
>        - {% icon param-repeat %} *"Rules"*: `Add / Modify Column Definitions`
>          - {% icon param-repeat %} *"Add Definition"*: `URL`
>            - *"URL"*: `A`
>          - {% icon param-repeat %} *"Add Definition"*: `List Identifier(s)`
>            - *"List Identifier(s)"*: `B`
>          - Click **Apply**
>        - *"Type"*: `fastqsanger.gz`
>        - *"Name"*: `Nanopore data` (or similar)
>        - *"Add nametag for name:"* {% icon param-check %}
>        - Click **Upload**
>
{: .hands_on}


# Read trimming and quality control

In the end, we do not want to base our mapping on low-quality reads that may
cause misassembly of fragments or introduce sequencing artefacts into the final
assembled sequence. Since quality control, read filtering and read trimming are
all quite fast and computationally cheap operations compared to the read
mapping that we will use to identify and eliminate human reads, it is best to
perform these steps up front.

Due to their very different nature, however, Illumina- and Nanopore-sequenced
reads should be treated rather differently.

## Trimming and filtering of Illumina reads

Galaxy offers a panel of different NGS reads trimming/filtering tools. Here,
we use **fastp** {% icon tool %}, which is straightforward to configure,
and when combined with **MultiQC** {% icon tool %}, enables nice and
easy-to-interpret visualizations of the effects of preprocessing, in particular
for multiple samples.

In the following, we configure the tool to retain reads only if at most 20% of
their bases have a Phred-scaled quality >= 20 and if there length in bases after
trimming of adapter sequences (which the tool auto-detects for us) is at least
50.

The JSON-formatted report produced by the tool, can serve as input to
**MultiQC** {% icon tool %} for a direct visual comparison of key quality
metrics for all samples before and after preprocessing.

> <hands-on-title>Reads preprocessing and quality reporting</hands-on-title>
> 1. **fastp** {% icon tool %} with the following parameters
>    - *"Single-end or paired reads"*: `Paired Collection`
>      - *"Select paired collection(s)"*: the collection of Illumina-sequenced
>        reads as produced in the *Get Data* section
>    - in *"Filter Options"*
>      - in *"Quality filtering options"*
>        - *"Disable quality filtering"*: `No`
>        - *"Qualified quality phred"*: `20`
>        - *"Unqualified percent limit"*: `20`
>      - in *"Length filtering options"*
>        - *"Disable length filtering"*: `No`
>        - *"Length required"*: `50`
>    - in *"Output Options"*
>      - *"Output HTML report"*: `No`
>      - *"Output JSON report"*: `Yes`
>
>    The tool run produces two collections - one with the actual preprocessed
>    reads of all input samples, another one with a JSON-formatted report of
>    the processing for every sample.
>
> 2. **MultiQC** {% icon tool %} with the following parameters
>    - {% icon param-repeat %} *"Results"*
>      - *"Which tool was used generate logs?"*: `fastp`
>        - {% icon param-collection %} *"Output of fastp"*: the collection of
>          JSON-formatted reports, second collection produced by **fastp**
>          {% icon tool %}
>
>    This tool run generates a single webpage output with the combined quality reports
>    for all samples before and after processing with **fastp** {% icon tool %}.
>
{: .hands_on}

## Quality control of Nanopore reads

Nanopore-sequenced reads differ greatly in length from one another and are, on
average, of relatively low quality (in particular when compared to
Illumina-sequenced reads). These properties make it challenging to preprocess
them with standard tools. Quality assessment tools calibrated to work well with
Illumina-sequenced reads are less useful for Nanopore-sequenced reads, too, for
the same reasons. Here we restrict ourselves to a simple quality check with
**NanoPlot** {% icon tool %}, a dedicated QC tool for Nanopore-sequenced reads.

> <hands-on-title>Checking the quality of Nanopore reads with NanoPlot</hands-on-title>
> 1. **NanoPlot** {% icon tool %} with the following parameters
>    - *"Select multifile mode"*: `batch`
>      - *"Type of the file(s) to work on"*: `fastq`
>        - {% icon param-collection %} *"files"*: the collection of
>          Nanopore-sequenced reads as produced in the *Get Data* section
>    - in *"Options for filtering or transforming input prior to plotting"*
>      - *"Logarithmic scaling of lengths in plots."*: `Yes`
>
>    > <question-title></question-title>
>    >
>    > 1. Looking at the three generated quality reports, which of the three
>    >    samples seems to be of better quality overall than the other two,
>    >    and what are some criteria that support this conclusion?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Sample `SRR10948474` has the best overall quality.
>    > >
>    > >    It has both higher average read length and quality than the other
>    > >    two (see the *Summary statistics* table), has a distribution of
>    > >    read qualities that peaks around intermediate quality scores (not
>    > >    low ones as for the other samples), and contains some extra-long
>    > >    (though at least partly rather low-quality) reads. Those last two
>    > >    points become most obvious when looking at the *Read lengths vs
>    > >    Average read quality* plots.
>    >  {: .solution }
>    {: .question}
>
{: .hands_on}

# Subtraction of reads mapping to the human reference genome

## Mapping of Illumina reads

In this tutorial, we are using **Bowtie2** for mapping our short-reads data to
the human genome. *BWA-MEM* would be a good alternative for mapping the 150
nucleotides (see the QC report above) reads from our samples.

According to its authors, the *Minimap2* aligner, which we will be using for
mapping the Nanopore-sequenced data in the next step, is supposed to outcompete
*Bowtie2* and *BWA-MEM* in terms of speed even for Illumina-sequenced reads
of length > 100 nts, but we opt for the conservative approach of using a
widely-used, well-tested tool here.

> <hands-on-title>Mapping with Bowtie2</hands-on-title>
> 1. **Bowtie2** {% icon tool %} with the following parameters
>    - *"Is this single or paired library"*: `Paired-end Dataset Collection`
>       - *"FASTQ Paired Dataset"*: the collection of preprocessed
>         Illumina-sequenced reads, output of **fastp** {% icon tool %}
>       - *"Write unaligned reads (in fastq format) to separate file(s)"*: `No`
>
>         Activating this option may seem attractive since the unaligned reads
>         are what we are interested in, but filtering for those reads with
>         a dedicated tool in a separate step allows us to filter on the
>         properties of the read pairs instead of those of individual reads.
>
>       - *"Write aligned reads (in fastq format) to separate file(s)"*: `No`
>       - *"Do you want to set paired-end options?"*: `No`
>
>     - *"Will you select a reference genome from your history or use a built-in index?"*:
>       `Use a built-in genome index`
>       - *"Select reference genome"*: `Human (Homo sapiens): hg38 Full`
>     - *"Set read groups information?"*: `Do not set`
>     - *"Select analysis mode"*: `Default setting only`
>     - *"Do you want to tweak SAM/BAM Options?"*: `No`
>     - *"Save the bowtie2 mapping statistics to the history"*: `Yes`
>
>    The tool run should produce two collections of output datasets. One with
>    the actual mapped reads of the three samples and one with the corresponding
>    mapping statistics for each sample, which we want to have a brief look at
>    next.
>
> 2. Inspect the `mapping stats` of each sample by clicking on the corresponding collection, then on the {% icon galaxy-eye %} (eye) icon of each individual sample data
>
>    > <question-title></question-title>
>    >
>    > 1. What percentage of reads of each sample has been aligned to the `hg38` reference genome?
>    > 2. Which sample is the least contaminated with human reads?
>    > 3. Which sample contains the highest amount of SARS-CoV2 reads?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The samples have between 13% and 21% of reads aligned to `hg38`.
>    > >    The information can be found on the last line of output for each
>    > >    sample.
>    > > 2. Sample `SRR10971381` is the least contaminated with just above 13%
>    > >    of human reads
>    > > 3. You cannot answer this question with this data. While `SRR10971381`
>    > >    shows the lowest relative contamination with human reads, that
>    > >    does not mean that all other reads are from SARS-CoV-2. They could
>    > >    come from other species (*e.g.*, bacteria or other viruses)
>    > >    contained in this BALF sample.
>    >  {: .solution }
>    {: .question}
>
{: .hands_on}

## Mapping of Nanopore reads

For the mapping of the Nanopore-sequenced data we are using the **Minimap2**
aligner, which is particularly efficient for mapping long reads.

> <hands-on-title>Nanopore reads mapping</hands-on-title>
>
> 1. **Map with minimap2** {% icon tool %} with the following parameters
>    - *"Will you select a reference genome from your history or use a built-in index?"*:
>      `Use a built-in genome index`
>      - *"Using reference genome"*: `Human Dec. 2013 (GRCh38/hg38) (hg38)`
>
>    - *"Single or Paired-end reads"*: `Single`
>      - {% icon param-collection %} *"Select fastq dataset"*: the collection
>        of Nanopore-sequenced reads as set up in the *Get Data* section
>
>    - *"Select a profile of preset options"*: `Oxford Nanopore read to reference mapping. ...`
>
>    This tool run produces one collection with the actual mapped reads for
>    each Nanopore-sequenced sample. Unlike **Bowtie2** it does not have an
>    option to output mapping statistics directly. However, we can generate
>    that information through an extra step.
>
> 2. **Samtools stats** {% icon tool %} with the following parameters
>    - {% icon param-collection %} *"BAM file"*: the collection of mapped
>      Nanopore-sequenced reads, output of **Map with minimap2 {% icon tool %} (step 1)
>    - *"Output"*: `Separate datasets for each statistic`
>      - *"Desired output files"*
>        - {% icon param-check %} *"Summary numbers"*
>
>          These simple summary stats correspond approximately to the
>          statistics generated by **Bowtie2** and are enough for our purpose.
>
>    > <comment-title>Mapping stats for Nanopore-sequenced long reads</comment-title>
>    >
>    > Since, unlike Illumina-generated reads, Nanopore-sequenced reads can
>    > have very different lengths, it makes limited sense to calculate a
>    > ratio of mapped to overall reads for them.
>    >
>    > Instead, `bases mapped` / `total length` should give a more reliable
>    > estimate of which fraction of the data is due to human genome sequence.
>    >
>    > Try to calculate this ratio for the three samples on your own!
>    {: .comment}
>
{: .hands_on}

## Human reads subtraction

At this point you should have two collections of mapped reads - one holding the
mapping results obtained with **Bowtie2** of the three Illumina-sequenced
samples, the other one holding the **minimap2** output for the three
Nanopore-sequenced samples.

Next, we are going to filter the data from both collections to retain only
those reads that were *not* mapped to the human genome, *i.e* those of
potential viral origin.

> <hands-on-title>Mapped reads filtering</hands-on-title>
>
> 1. **Samtools view** {% icon tool %} to filter the Illumina-sequenced reads mapped with Bowtie2:
>    - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: the collection of
>      mapped Illumina-sequenced reads, output of **Bowtie2** {% icon tool %}
>    - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>      - in *"Configure filters"*
>        - *"Require that these flags are set"*: `Read is unmapped` and
>          `Mate is unmapped`
>
>      - *"What would you like to have reported?"*: `All reads retained after filtering and subsampling`
>        - *"Produce extra dataset with dropped reads?"*: `No`
>        - *"Output format"*: `BAM (-b)`
>
> 2. **Samtools view** {% icon tool %} to filter the Nanopore-sequenced reads mapped with minimap2:
>    - {% icon param-collection %} *"SAM/BAM/CRAM data set"*: the collection of
>      mapped Nanopore-sequenced reads, output of **minimap2** {% icon tool %}
>    - *"What would you like to look at?"*: `A filtered/subsampled selection of reads`
>      - In *"Configure filters"*
>        - *"Require that these flags are set"*: `Read is unmapped`
>
>      - *"What would you like to have reported?"*: `All reads retained after filtering and subsampling`
>        - *"Produce extra dataset with dropped reads?"*: `No`
>        - *"Output format"*: `BAM (-b)`
>
> 3. (Optional) Remove the database `hg38` attribute from the output files
>
>    > <details-title>Why do this?</details-title>
>    > When we ran the **Bowtie2** {% icon tool %} and **minimap2**
>    > {% icon tool %} mappers before, these tools set the *database* attribute
>    > on their outputs to `hg38` to indicate that the mapped reads in these
>    > outputs have been mapped against this version of the human reference
>    > genome - an important piece of information for further work with those
>    > mapped reads.
>    >
>    > In step 1 and 2 above, however, we have eliminated all mapped reads so
>    > the `database: hg38` info is misleading from this point onward in the
>    > analysis. While not directly harmful, it is best practice to remove this
>    > metadata now.
>    {: .details}
>
>    For the outputs of step 1 and step 2 above, reset the database/build
>    (dbkey) to `unspecified (?)`.
>
>    {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="unspecified (?)" %}
>
{: .hands_on}


# Format conversion of remaining reads

## Conversion to fastq format

Assembly tools, typically, expect their input data to be fastq-formatted, but
what we have after mapping and filtering is data in BAM format. Hence, we need
to convert the retained Illumina- and Nanopore-sequenced reads back into their
original format before proceeding to assembly.

> <hands-on-title>BAM to fastq format conversion</hands-on-title>
>
> 1. **Samtools fastx** {% icon tool %} to convert the filtered Illumina-sequenced reads to fastq format
>    - {% icon param-collection %} *"BAM or SAM file to convert"*: the collection of
>      filtered Illumina-sequenced reads, output of first **Samtools view** {% icon tool %} run
>    - *"Output format"*: `compressed FASTQ`
>    - *"outputs"*: `READ1` and `READ2`
>
> 2. **Samtools fastx** {% icon tool %} to convert the filtered Nanopore-sequenced reads to fastq format
>    - {% icon param-collection %} *"BAM or SAM file to convert"*: the collection of
>      filtered Nanopore-sequenced reads, output of second **Samtools view** {% icon tool %} run
>    - *"Output format"*: `compressed FASTQ`
>    - *"outputs"*: `unspecific`
>
{: .hands_on}

## Optional: Rearrange the filtered data into the original nested data structure

If you compare the outputs of the last step to the input data we started out
with, you will notice that the Illumina-sequenced data is arranged differently
now than initially. It is arranged now into separate collections of forward and
reverse reads, whereas we started with a single nested collection of the data.

We can easily cast the data back into its original structure with one of
Galaxy's collection manipulation tools, but note that we will not use the
resulting nested collection for this tutorial because the **Unicycler** tool
for assembling the reads would not be able to handle the nested data correctly.

Thus, the following just serves as an illustration and is entirely optional.

> <hands-on-title>Arrange two list collections into a list of pairs</hands-on-title>
>
> 1. **Zip Collection** {% icon tool %} with the following parameters
>    - {% icon param-collection %} *"Input Dataset (Forward)"*: the collection
>      of filtered Illumina-sequenced forward reads in fastq format,
>      first output of first **Samtools fastx** {% icon tool %} run
>    - {% icon param-collection %} *"Input Dataset (Reverse)"*: the collection
>      of filtered Illumina-sequenced reverse reads in fastq format,
>      second output of first **Samtools fastx** {% icon tool %} run
>
{: .hands_on}

## Merging of reads with collection operations

To merge reads from several samples into a combined final assembly, we need to
pass the data to **Unicycler** {% icon tool %} in partially merged form. The
forward and reverse reads of paired-end data should be kept separate, and so
should short and long reads. However, the tool has no option to combine data
from individual samples, so we need to merge the forward, reverse, and the long
reads data, respectively, across samples. Conveniently for us, the outputs of
the earlier **Samtools fastx** {% icon tool %} runs have already returned the
data structured into three corresponding collections for us.

> <hands-on-title>Collapsing each collection into a single dataset</hands-on-title>
>
> 1. **Collapse Collection** {% icon tool %} of Illumina-sequenced *forward* reads
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*:
>      the collection of filtered Illumina-sequenced forward reads in fastq format,
>      first output of first **Samtools fastx** {% icon tool %} run
>    - *"Keep one header line"*: `No`
>    - *"Prepend File name"*: `No`
>
> 2. **Collapse Collection** {% icon tool %} of Illumina-sequenced *reverse* reads
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*:
>      the collection of filtered Illumina-sequenced reverse reads in fastq format,
>      second output of first **Samtools fastx** {% icon tool %} run
>    - *"Keep one header line"*: `No`
>    - *"Prepend File name"*: `No`
>
> 3. **Collapse Collection** {% icon tool %} of Nanopore-sequenced reads
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*:
>      the collection of filtered Nanopore-sequenced reads in fastq format,
>      output of second **Samtools fastx** {% icon tool %} run
>    - *"Keep one header line"*: `No`
>    - *"Prepend File name"*: `No`
>
{: .hands_on}


# SARS-CoV-2 genome assembly

## Optional: Subsampling of reads

The actual assembly of the sequenced reads represents the real bottleneck in
this tutorial. Assembly of the full set of sequences can easily take 10 hours
and would best be conducted overnight.

If you do not have that much time, you should downsample the Illumina-sequenced
combined reads now. Which will reduce the time required to finish the
subsequent assembly step to approximately 1-2 hours.

> <comment-title>If you are in a hurry</comment-title>
> The downsampling parameters below have been chosen to have minimal impact on
> the assembly results. Further speed-ups are certainly possible, but will
> likely lead to poor assembly outcomes.
{: .comment}

> <hands-on-title>Subsampling of paired-end short-reads data</hands-on-title>
>
> 1. **seqtk_sample** {% icon tool %} with the following parameters
>    - {% icon param-files %} *"Input FASTA/Q file"*: The two datasets with the
>      combined Illumina-sequenced forward and reverse reads, outputs of the
>      first and the second run of **Collapse Collection**
>    - *"RNG seed"*: 4
>    - *"Subsample (decimal fraction or number)"*: 0.1
>
{: .hands_on}

## Create assembly

> <hands-on-title>Assembly of SARS-CoV2 genome</hands-on-title>
>
> 1. **Create assemblies with Unicycler** {% icon tool %} with the following parameters
>    - *"Paired or Single end data?"*: `Paired`
>      - {% icon param-file %} *"Select first set of reads"*: the combined
>        Illumina-sequenced forward reads from all samples, output of the
>        first **Collapse Collection** {% icon tool %} run (or first output of
>        **seqtk_sample** {% icon tool %} for a subsample-based assembly)
>      - {% icon param-file %} *"Select second set of reads"*: the combined
>        Illumina-sequenced reverse reads from all samples, output of the
>        second **Collapse Collection** {% icon tool %} run  (or second output
>        of **seqtk_sample** {% icon tool %} for a subsample-based assembly)
>
>    - {% icon param-file %} *"Select long reads. If there are no long reads, leave this empty"*:
>      Nanpore-sequenced reads from all samples, output of the third
>      **Collapse Collection** {% icon tool %} run
>    - *"Select Bridging mode"*: `Normal (moderate contig size and misassembly rate)`
>    - *"Exclude contigs from the FASTA file which are shorter than this length (bp)"*:
>      `100`
>    - *"The expected number of linear (i.e. non-circular) sequences in the assembly"*:
>      `1`
>
{: .hands_on}

## Explore assembly

The **Unicycler** {% icon tool %} run above should produce two output datasets:

- a final assembly in FASTA format
- an assembly graph

Of these, the assembly graph is more information-rich because it not only
contains the sequences of *all* assembled fragments (including the ones shorter
than the threshold length defined for inclusion of the fragments into the FASTA
output), but also indicates the relative average coverage of the fragments by
sequenced reads and how some of the fragments could potentially be bridged
after resolving ambiguities manually.

### Assembly inspection with Bandage

On the downside, the assembly graph format takes some getting used to before
you can make sense out of the information it provides.

This issue can be alleviated through the use of **Bandage**, a package for
exploring assembly graphs through summary reports and visualizations of their
contents.

> <hands-on-title>Assembly stats and visualization with Bandage</hands-on-title>
>
> 1. **Bandage Info** {% icon tool %} with the following parameters
>    - {% icon param-file %} *"Graphical Fragment Assembly"*: the assembly graph dataset produced by
>      **Unicycler**
>    - *"Output the information in a single tab-delimited line starting with the graph file"*:
>      `No`
>
> 2. **Bandage Image** {% icon tool %} with the following parameters
>    - {% icon param-file %} *"Graphical Fragment Assembly"*: the assembly graph dataset produced by
>      **Unicycler**
>    - *"Node name labels?"*: `Yes`
>    - *"Node length labels?"*: `Yes`
>
{: .hands_on}

Let us inspect the summary report produced by **Bandage Info** {% icon tool %}
first:

You may be rather disappointed by the large *percentage of of dead ends* in the
assembly graph (in general, lower is better here), and by the correspondingly
large *node count*. After all, should the viral sequence not be encoded on a
single small contig (a quick check at
[Wikipedia](https://en.wikipedia.org/wiki/Coronavirus#Genome) reveals that
coronaviruses have genomes in the size range of 30kb)?

On the other hand, there is the *Longest node* of 29768 bp of assembled
sequence, which is suspiciously close to the expected genome size, but a much
larger *Estimated sequence length*.

Next, take a look at the assembly graph visualization generated by **Bandage
Image** {% icon tool %} to see if that tells us more:

Indeed, this output shows that **Unicycler** {% icon tool %} managed to
assemble a good number of contigs of moderate size, then had trouble with a
number of really small fragments that it could only assemble with lots of
ambiguities (leading to that ugly clutter of nodes in the top row of the
image). Those small fragments will probably be hard to make sense of, but the
manageable list of moderate-size contigs (nodes 1-23, 25, 26) is encouraging.

Of these, node 1 is the longest node mentioned in the report with a size close
to our expectations.

Node 2 looks peculiar since **Unicycler** claims it is circular, while
Coronavirus genomes are known to be linear.

### Check origin of assembled sequences with BLAST

While we could view the actual contents of the assembly graph output of
**Unicycler** {% icon tool %} and extract node sequences of interest from it
(the longest node and that circular one could be a start), things are much
easier if we work with the FASTA output of **Unicycler** instead.

From the visualization with **Bandage Image** {% icon tool %} we know that the
separately assembled nodes are all longer than 1000 bp. We can extract those
sequences based on the length threshold in Galaxy, then BLAST all retained
sequences in one go.

> <hands-on-title>Filter FASTA sequences by their length</hands-on-title>
>
> 1. **Filter sequences by length** {% icon tool %} with the following parameters
>    - {% icon param-file %} *"Fasta file"*: the FASTA output produced by
>      **Unicycler**
>    - *"Minimal length"*: `1000`
>
>    This outputs a new FASTA datasets with only the sequences satisfying our
>    length threshold.
>
> > <tip-title>Apply length filters after instead of during assembly</tip-title>
> > You may have noted that in the **Unicycler** {% icon tool %} run we kept
> > the tool's *"Exclude contigs from the FASTA file which are shorter than
> > this length (bp)"* option at its default value of `100` instead of using
> > the 1,000 bp threshold there directly to save a step in the analysis.
> >
> > The reason we did this is that normally you will not know the exact length
> > threshold you want until *after* having explored the generated assembly.
> >
> > Length-filtering some FASTA sequences is a trivial process that takes very
> > little time, but you would not want to rerun an hours-long assembly job
> > just because you accidentally stripped some interesting assembled sequences
> > from the output.
> {: .tip}
>
{: .hands_on}

> <hands-on-title>NCBI BLAST of multiple contigs</hands-on-title>
>
> 1. View the output of **Filter sequences by length** {% icon tool %} by
>    clicking the {% icon galaxy-eye %} (eye) icon attached to that dataset.
> 2. Click into the middle panel, which should now display the content of the
>    dataset, select all sequences by pressing <kbd>Ctrl</kbd>+<kbd>A</kbd>,
>    then copy the selection to the clipboard with <kbd>Ctrl</kbd>+<kbd>C</kbd>
> 3. Head over to the
>    [NCBI BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE=MegaBlast&PROGRAM=blastn)
>    service, paste the copied content into the `Enter Query Sequence` text
>    box and click `BLAST` at the bottom of the form.
> 4. Wait for the *BLAST* run to finish.
> 5. On the results page, look for the drop-down menu next to `Results for`.
>    It lets you toggle the BLAST hits list further down to include only the
>    matches to individual sequences from your multi-sequence query.
>
{: .hands_on}

Now take a bit of time to explore the BLAST hits uncovered for some of your
assembled nodes. Pay attention, specifically, to the node with the longest
sequence (node #1) and the circular node #2, but also investigate the results
for a few others.

> <question-title></question-title>
>
> 1. Which genome is represented by node #1?
> 2. Which genome corresponds to node #2? Does this finding remind you of
>    something you have learnt before?
> 3. What do most other node sequences have in common? Do these additinal
>    findings make sense?
>
> > <solution-title></solution-title>
> > 1. The sequence of node #1 is the assembled SARS-CoV-2 sequence we are
> >    looking for. It is a perfect match to various SARS-CoV-2 genome
> >    sequences found in Genbank over the entire assembled length, and we have
> >    been able to assemble almost the entire genome even from the subsampled
> >    sequencing data.
> > 2. The circular sequence of node #2 corresponds to the 5,386 bp genome of
> >    bacteriophage phiX174. As explained in the more general
> >    [Unicycler assembly](../unicycler-assembly/tutorial.html) tutorial,
> >    this genome is often used as a spike-in in Illumina sequencing.
> >    Finding the complete sequence here is, thus,  another indication that
> >    our analysis worked and produced meaningful results.
> > 3. Almost all other assembled sequences appear to represent parts of
> >    bacterial genomes. The only exceptions are the node #10 sequence, for
> >    which no significant BLAST hits could be found, and the node #11
> >    sequence, which represents a small stretch of left-over human genomic
> >    DNA, which seems to have survived our subtraction approach.
> >
> >    What all the bacterial genomes have in common is that they represent
> >    genera of bacteria that are known to colonize the oral cavity and
> >    mucosa. Since all samples in this analysis are BALF samples the presence
> >    of DNA from such bacteria should not be surprising. In addition, some
> >    members of the identified genera are known as opportunistic pathogens.
> >    In particular, members of the genus *Prevotella* can infect the
> >    respiratory tract and contribute to inflammation under anaerobic
> >    conditions caused by primary infections. Hence, an alternative
> >    explanation for the presence of some of these sequences in the samples
> >    might be that the corresponding bacteria contributed to the clinical
> >    picture of some of the Covid-19 patients they were obtained from.
> {: .solution}
>
{: .question}

# Conclusion


The power of modern genome assembly tools is remarkable, and so is their
robustness in the face of data of metagenomic nature. Assembling reads derived
from a virus and a good handful of copurified bacteria back into separate
contigs is a challenging task, which Unicycler solved without major issues!

Nevertheless, good quality assemblies still rely on proper preprocessing and
filtering to reduce the number of misassembly events, ambiguous assemblies and
of incorporation of sequencing errors into the final assembly.


topic: assembly
tutorial: vgp_workflow_training
title: Using the VGP workflows to assemble a vertebrate genome with HiFi and Hi-C
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["pacbio", "eukaryote", "VGP", "biodiversity"]
content_hash: 2ad3bdb140c94f450154373905f09719
last_modified: 2026-02-26T21:12:14.497146
zenodo_link: https://zenodo.org/record/5887339
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/vgp_workflow_training/tutorial.html

content:
The {VGP}, a project of the {G10K} Consortium, aims to generate high-quality, near error-free, gap-free, chromosome-level, haplotype-phased, annotated reference genome assemblies for every vertebrate species ({% cite Rhie2021 %}). The VGP has developed a fully automated *de-novo* genome assembly pipeline, which uses a combination of three different technologies: Pacbio {HiFi}, {Hi-C} data, and (optionally) Bionano optical map data. The pipeline consists of nine distinct workflows. This tutorial provides a quick example of how to run these workflows for one particular scenario, which is, based on our experience, the most common: assembling genomes using {HiFi} Reads combined with {Hi-C} data (both generated from the same individual).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Getting started on Galaxy

This tutorial assumes you are comfortable getting data into Galaxy, running jobs, managing history, etc. If you are unfamiliar with Galaxy, we recommend you visit the [Galaxy Training Network](https://training.galaxyproject.org). Consider starting with the following trainings:
- [Introduction to Galaxy]({% link topics/introduction/tutorials/introduction/slides.html %})
- [Galaxy 101]({% link topics/introduction/tutorials/galaxy-intro-101/tutorial.md %})
- [Getting Data into Galaxy]({% link topics/galaxy-interface/tutorials/get-data/slides.html %})
- [Using Dataset Collections]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %})
- [Understanding the Galaxy History System]({% link topics/galaxy-interface/tutorials/history/tutorial.md %})
- [Introduction to Galaxy Analyses]({% link topics/introduction/index.md %})
- [Downloading and Deleting Data in Galaxy]({% link topics/galaxy-interface/tutorials/download-delete-data/tutorial.md %})

# The VGP-Galaxy pipeline

The {VGP} assembly pipeline has a modular organization, consisting in ten workflows (Fig. 1). It can used with the following types of input data:

|  Input data | Assembly quality  | Analysis trajectory <br>([Fig. 1)](#figure-1)|
|------|---------------|-----|
| HiFi | The minimum requirement | A |
| HiFi + HiC| Better continuity | B |
| HiFi + Bionano | Better continuity | C |
| HiFi + Hi-C + Bionano | Even better continuity | D |
| HiFi + parental data| Better haplotype resolution | E |
| HiFi + parental data + Hi-C| Better haplotype resolution and improved continuity | F |
| HiFi + parental + Bionano | Better haplotype resolution and improved continuity | G |
| HiFi + parental data + Hi-C + Bionano | Better haplotype resolution and ultimate continuity | H |

In this table, HiFi and Hi-C data are derived from the individual whose genome is being assembled. "Parental data" refers to high coverage whole genome resequencing data from the parents of the individual being assembled. Assemblies utilizing parental data for phasing are also called "*Trios*". Each combination of input datasets is supported by an *analysis trajectory*: a combination of workflows designed for generating assembly given a particular combination of inputs. These trajectories are listed in the table above and shown in the figure below. 

![The nine workflows of Galaxy assembly pipeline](../../images/vgp_assembly/VGP_workflow_modules.svg "Eight analysis trajectories are possible depending on the combination of input data. A decision on whether or not to invoke Workflow 6 is based on the analysis of QC output of workflows 3, 4, or 5. Thicker lines connecting Workflows 7, 8, and 9 represent the fact that these workflows are invoked separately for each phased assembly (once for maternal and once for paternal).")
<br>
The first stage of the pipeline is the generation of a *k*-mer profile of the raw reads to estimate genome size, heterozygosity, repetitiveness, and error rate necessary for parameterizing downstream workflows. The generation of *k*-mer counts can be done from HiFi data only (Workflow 1) or include data from parental reads for trio-based phasing (Workflow 2). The second stage is contig assembly. In addition to using only {HiFi} reads (Workflow 3), the contig-building (contiging) step can leverage {Hi-C} (Workflow 4) or parental read data (Workflow 5) to produce fully-phased haplotypes (hap1/hap2 or parental/maternal assigned haplotypes), using [`hifiasm`](https://github.com/chhylp123/hifiasm). The contiging workflows also produce a number of critical quality control (QC) metrics such as *k*-mer multiplicity profiles. Inspection of these profiles provides information to decide whether the third stage—purging of false duplication—is required. Purging (Workflow 6), using [`purge_dups`](https://github.com/dfguan/purge_dups) identifies and resolves haplotype-specific assembly segments incorrectly labeled as primary contigs, as well as heterozygous contig overlaps. This increases continuity and the quality of the final assembly. The purging stage is generally unnecessary for trio data, as thses assemblies achieve reliable haplotype resolution using *k*-mer set operations based on parental reads. Generally, HiC-phased contigs also do not need to undergo purging, but if so, then there exists Workflow 6b to purge a single haplotype at a time. The fourth stage, scaffolding, produces chromosome-level scaffolds using information provided by Bionano (Workflow 7), with [`Bionano Solve`](https://Bionano.com/software-downloads/) (optional) and Hi-C (Workflow 8) data and [`YaHS`](https://github.com/c-zhou/yahsscaffolding) algorithms. A final stage of decontamination (Workflow 9) removes exogenous sequences (*e.g.*, viral and bacterial sequences) from the scaffolded assembly. A separate workflow (WF0) is used to assemble and annotate a mitogenome.

> <comment-title>A note on data quality</comment-title>
> We suggest at least 30✕ PacBio HiFi coverage and 60✕ Hi-C coverage (both diploid coverage).
{: .comment}

# Getting the data

The following steps use PacBio {HiFi} and Illumina {Hi-C} data from baker's yeast ([*Saccharomyces cerevisiae*](https://en.wikipedia.org/wiki/Saccharomyces_cerevisiae)). The tutorial represents trajectory **B** from Fig. 1 above. For this tutorial, the first step is to get the datasets from Zenodo. Specifically, we will be uploading two datasets:

1. A set of PacBio {HiFi} reads in `fasta` format. Please note that your HiFi reads received from a sequencing center will usually be fastqsanger.gz format, but the dataset used in this tutorial has been converted to fasta for space..
2. A set of Illumina {Hi-C} reads in `fastqsanger.gz` format.

## Uploading `fasta` datasets from Zenodo

The following two steps demonstrate how to upload three PacBio {HiFi} datasets into your Galaxy history.


> <hands-on-title> Uploading <tt>FASTA</tt> datasets from Zenodo </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Copy the following URLs into clipboard.
>
>    (You can do this by clicking on {% icon copy %} button in the right upper corner of the box below. It will appear if you mouse over the box.)
>
>    ```
>    https://zenodo.org/record/6098306/files/HiFi_synthetic_50x_01.fasta
>    https://zenodo.org/record/6098306/files/HiFi_synthetic_50x_02.fasta
>    https://zenodo.org/record/6098306/files/HiFi_synthetic_50x_03.fasta
>    ```
>
> 3. Upload datasets into Galaxy.
>    - set the datatype to `fasta`
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>
>    {% snippet topics/assembly/tutorials/vgp_genome_assembly/faqs/dataset_upload_fasta_via_urls.md %}
>
{: .hands_on}


## Uploading `fastqsanger.gz` datasets from Zenodo

Illumina {Hi-C} data is uploaded in essentially the same way as shown in the following two steps.

> <warning-title> Make sure you the choose correct format!</warning-title>
> When selecting datatype in "**Type (set all)**" drop-down, make sure you select `fastqsanger` or `fastqsanger.gz` BUT NOT `fastqcssanger` or anything else!
{: .warning}

> <hands-on-title> Uploading <tt>fastqsanger.gz</tt> datasets from Zenodo </hands-on-title>
>
> 1. Copy the following URLs into clipboard.
>     - you can do this by clicking on {% icon copy %} button in the right upper corner of the box below. It will appear if you mouse over the box.
>
>    ```
>    https://zenodo.org/record/5550653/files/SRR7126301_1.fastq.gz
>    https://zenodo.org/record/5550653/files/SRR7126301_2.fastq.gz
>    ```
>
> 2. Upload datasets into Galaxy.
>    - set the datatype to `fastqsanger.gz`
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>
>    {% snippet topics/assembly/tutorials/vgp_genome_assembly/faqs/dataset_upload_fastqsanger_via_urls.md %}
>
> 3. Rename the datasets as follow: 
>    - Rename `SRR7126301_1.fastq.gz` as `Hi-C forward reads`
>    - Rename `SRR7126301_2.fastq.gz` as `Hi-C reverse reads`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}



> <warning-title>These datasets are large!</warning-title>
> Hi-C datasets are large. It will take some time (~15 min) for them to be fully uploaded. Please, be patient.
{: .warning}

## Organizing the data

If everything goes smoothly, your history will look like shown in Fig. 4 below. The three {HiFi} fasta files are better represented as a collection: {collection}. Also, importantly, the workflow we will be using for the analysis of our data takes a collection as an input (it does not access individual datasets). So let's create a collection using steps outlines in the Tip {% icon tip %} "Creating a dataset collection" that you can find below Fig. 4.

![AfterUpload](../../images/vgp_assembly/making_list.svg "History after uploading HiFi and HiC data (left). Creation of a list (collection) combines all HiFi datasets into a single history item called 'HiFi data' (right). See below for instruction on how to make this collection.")

{% snippet faqs/galaxy/collections_build_list.md %}

> <details-title>Other ways to upload the data</details-title>
> You can obviously upload your own datasets via URLs as illustrated above or from your own computer. In addition, you can upload data from a major repository called [GenomeArk](https://genomeark.org). GenomeArk is integrated directly into Galaxy Upload. To use GenomeArk following the steps in the Tip {% icon tip %} below:
>
> {% snippet faqs/galaxy/datasets_upload_from_genomeark.md %}
{: .details}


Once we have imported the datasets, the next step is to import the workflows necessary for the analysis of our data from [DockStore](https://dockstore.org).

# Importing workflows

All analyses described in this tutorial are performed using *workflows*--chains of tools--shown in [Fig. 1](#figure-1). Specifically, we will use four workflows corresponding to analysis trajectory **B**: 1, 4, 6, and 8. To use these four workflows you need to import them into your Galaxy account following the steps below. Note: these are not necessarily the latest versions of the actual workflows, but versions that have been tested for this tutorial. To see the latest versions, see the [Galaxy Project VGP workflows page](https://galaxyproject.org/projects/vgp/workflows/) and click on the Dockstore links to import workflows. **Alternatively, for each section of the tutorial, there will be a "Launch [workflow] (View on Dockstore)" link at the beginning, which you can use to launch the workflow.**

# Performing the assembly

Workflows listed in [Fig. 1](#figure-1) support a variety of "analysis trajectories". The majority of species that were sequenced by the {VGP} usually contain {HiFi} reads for the individual being sequenced supplemented with {Hi-C} data. As a result most assemblies performed by us follow the trajectory **B**. This is why this tutorial was designed to follow this trajectory as well.

## Genome profile analysis (WF1)

Before the assembly can be run, we need to collect metrics on the properties of the genome under consideration, such as the expected genome size according to our data. The present pipeline uses **Meryl** for generating the *k*-mer database and **Genomescope2** for determining genome characteristics based on a *k*-mer analysis.

### Launching the workflow

{% snippet faqs/galaxy/workflows_run_ds.md title="Genome profile analysis (WF1)" dockstore_id="github.com/iwc-workflows/kmer-profiling-hifi-VGP1/main" version="v0.1.7" %}

> <hands-on-title> Running <i>K</i>-mer profile analysis workflow </hands-on-title>
>
> 1. **Identify inputs**
>
>    The profiling workflow takes the following inputs:
>   
>    1. {HiFi} reads as a collection
>    2. *K*-mer length
>    3. Ploidy
>
> 2. **Launch *k*-mer profiling workflow**
>
>    1. Click on the **Workflow** menu, located in the top bar
>    2. Click on the {% icon workflow-run %} **Run workflow** buttom corresponding to `kmer-profiling-hifi-VGP1`
>    3. In the **Workflow: kmer-profiling-hifi-VGP1** menu:
>        - {% icon param-collection %} "*Collection of Pacbio Data*": `HiFi data` collection
>        - "*K-mer length*": `31`
>        - "*Ploidy*": `2`
>    4. Click on the {% icon workflow-run %} **Run workflow** button
>   
>    This should look like this:
>
>    ![Parameters of *k*-mer profiling workflow](../../images/vgp_assembly/wf1_launch_ui.png  "Workflow main menu. The workflow menu lists all the workflows that have been imported. It provides useful information for organizing the workflows, such as last update and the tags. The worklows can be run by clicking in the play icon, marked in red in the image.")
>
>    > <comment-title>K-mer length</comment-title>
>    > In this tutorial, we are using a *k*-mer length of 31. This can vary, but the VGP pipeline tends to use a *k*-mer length of 21, which tends to work well for most mammalian-size genomes. There is more discussion about *k*-mer length trade-offs in the extended VGP pipeline tutorial.
>    {: .comment}
>
>
> 3. **Refill your coffee**
>
>    Assembly is not exactly an instantaneous type of analysis - this workflow will take approximately 15 minutes to complete. The same is true for all analyses in tutorial.
>
{: .hands_on}

### Interpreting the results

Once the workflow has finished, we can evaluate the transformed linear plot generated by [**Genomescope**](https://github.com/schatzlab/Genomescope), which includes valuable information such as the observed *k*-mer profile, fitted models, and estimated parameters. This file corresponds to the dataset `18` in this [history](https://usegalaxy.org/u/delphinel/h/vertebrate-genome-assembly-training). 

![Genomescope plot described further in caption](../../images/vgp_assembly/genomescope_plot.png "GenomeScope2 <i>k</i>-mer profile. The first peak located at about 25&times; corresponds to the heterozygous peak. The second peak at 50&times; corresponds to the homozygous peak. The plot also includes information about the inferred total genome length (len), genome unique length percent (uniq), overall heterozygosity rate (ab), mean <i>k</i>-mer coverage for heterozygous bases (kcov), read error rate (err), average rate of read duplications (dup) and <i>k</i>-mer size (k).")

This distribution is the result of the Poisson process underlying the generation of sequencing reads. As we can see, the *k*-mer profile follows a bimodal distribution, indicative of a diploid genome. The distribution is consistent with the theoretical diploid model (model fit > 93%). Low frequency *k*-mers are the result of sequencing errors, and are indicated by the red line. Genomescope2 estimated a haploid genome size of around 11.7 Mbp, a value reasonably close to the *Saccharomyces* genome size. We are using the transformed linear plot because it shows the ploidy better by reducing low ccoverage peaks (like *k*-mers containing errors) and increasing higher coverage peaks ({% cite Ranallo_Benavidez_2020 %}). 

It is worth noting that the genome characteristics such as length, error percentage, etc., are based on the GenomeScope2 model, which is the black line in the plot. If the model (black line) does not fit your observed data (blue bars), then these estimated characteristics might be very off. In the case of this tutorial, the model is a good fit to our data, so we can trust the estimates. 

## Assembly (contiging) with `hifiasm` (WF4)

To generate {contigs} we will use the [**hifiasm**](https://github.com/chhylp123/hifiasm) assembler. It is a part of the "Assembly with HiC (WF4)" workflow . This workflow uses **hifiasm** (HiC mode) to generate HiC-phased haplotypes (hap1 and hap2). This is in contrast to its default mode, which generates primary and alternate pseudohaplotype assemblies. This workflow includes three tools for evaluating assembly quality: [**gfastats**](https://github.com/vgl-hub/gfastats), [**BUSCO**](https://busco.ezlab.org/) and [**Merqury**](https://github.com/marbl/merqury).

### Launching the workflow

{% snippet faqs/galaxy/workflows_run_ds.md title="Assembly HiFi-HiC phasing (WF4)" dockstore_id="github.com/iwc-workflows/Assembly-Hifi-HiC-phasing-VGP4/main" version="v0.2.1" %}

> <hands-on-title> Launching assembly (contiging) workflow </hands-on-title>
>
> 1. **Identify inputs**
>
>    The assembly workflow takes the following inputs:
>
>    1. {HiFi} reads as a collection
>    2. Forward Hi-C reads
>    3. Reverse Hi-C reads
>    4. `Genomescope` Model Parameters generated by previous (*k*-mer profiling) workflow
>    5. `Genomescope` Summary generated by previous (*k*-mer profiling) workflow
>    6. Meryl *k*-mer database generated by previous (*k*-mer profiling) workflow
>    7. Busco Database
>    8. Busco lineage
>
> 2. **Launch the workflow**
>
>    1. Click on the **Workflow** menu, located in the top bar
>    2. Click on the {% icon workflow-run %} **Run workflow** button corresponding to `Assembly-Hifi-HiC-phasing-VGP4`
>    3. In the **Workflow: Assembly-Hifi-HiC-phasing-VGP4** menu fill the following parameters:
>        - {% icon param-collection %} "*Pacbio Reads Collection*": `HiFi data` collection
>        - {% icon param-file %} "*HiC forward reads*": `Hi-C forward reads`
>        - {% icon param-file %} "*HiC reverse reads*": `Hi-C reverse reads`
>        - {% icon param-file %} "*GenomeScope Summary*": `GenomeScope on data X Summary` (contains tag "`GenomeScopeSummary`")
>        - {% icon param-file %} "*Meryl database*": `Meryl on data X: read-db.mertyldb`: the Meryl *k*-mer database (contains tag "`MerylDatabase`")
>        - "*Database for Busco Lineage*": `Busco v5 Lineage Datasets` (or the latest version available in the instance)
>        - "*Provide lineage for BUSCO (e.g., Vertebrata)*": `Ascomycota`
>        - {% icon param-file %} "*GenomeScope Model Parameters*": `GenomeScope on data X Model parameters` (contains tag "`GenomeScopeParameters`")
>    4. Click on the {% icon workflow-run %} **Run workflow** button
> 
{: .hands_on}

> <comment-title>A note about "Homozygous Read Coverage"</comment-title>
>
>  Hifiasm tries to estimate the homozygous read coverage, but sometimes it can mistake a different peak as the homozygous coverage peak. To get around this, the VGP workflows calculate the homozygous coverage from the haploid coverage estimate from the GenomeScope outputs generated in the *k*-mer profiling workflow. However, sometimes GenomeScope can also misidentify the haploid peak, leading to it feeding an incorrect homozygous read coverage value into hifiasm. *If this is the case, you can run the hifiasm workflows but specify the homozygous read coverage yourself. Otherwise, this parameter is fine to leave blank to let the workflows try to get it right on their own.*
>
{: .comment}



### Interpreting the results

> <warning-title>There will be two assemblies!</warning-title>
> Because we are running hifiasm in HiC-phasing mode, it will produce two assemblies: hap1 and hap2!
{: .warning}

Let's have a look at the stats generated by **gfastats**. This output summarizes some main assembly statistics, such as contig number, N50, assembly length, etc. The workflow provide a joined table to display the statistics for both haplotype assemblies side-by-side. Below we provide a partial output of this file called `Assembly statistics for Hap1 and Hap2` in your history:

>| Statistic | Hap 1 | Hap 2 |
>|-----------|----------:|------:|
>| # contigs | 16 | 17 |
>| Total contig length | 11,304,507 | 12,160,985 |
>| Average contig length | 706,531.69  | 715,352.06 |
>| Contig N50 |  922,430 | 923,452 |
>| Contig N50 | 922,430 | 923,452 |
>| Contig auN | 895,018.82  | 904,515.40 |
>| Contig L50 | 5 | 6 |
>| Contig L50 | 5 | 6 |
>| Contig NG50 | 813,311 | 923,452 |
>| Contig NG50 | 813,311 | 923,452 |
>| Contig auNG | 861,278.90 | 936,364.06 |
>| Contig LG50 | 6 | 6 |
>| Contig LG50 | 6 | 6 |
>| Largest contig | 1,532,843 | 1,531,728 |
>| Smallest contig | 185,154 | 85,850 |
{: .matrix}

According to the report, both assemblies are quite similar; the hap1 assembly includes 16 {contigs}, whose cumulative length is around 11.3 Mbp. The hap2 assembly includes 17 contigs, whose total length is 12.1Mbp. Both assemblies come close to the estimated genome size, which is as expected since we used hifiasm-HiC mode to generate phased assemblies, and this lowers the chance of false duplications that can inflate assembly size.

> <comment-title>Are you working with pri/alt assemblies?</comment-title>
> This tutorial uses the hifiasm-HiC workflow, which generates phased hap1 and hap2 assemblies. The phasing helps lower the chance of false duplications, since the phasing information helps the assembler know which genomic variation is heterozygosity at the same locus versus being two different loci entirely. If you are working with primary/alternate assemblies (especially if there is no internal purging in the initial assembly), you can expect higher false duplication rates than we observe here with the yeast HiC hap1/hap2.
{: .comment}

> <question-title></question-title>
>
> 1. What is the longest contig in the hap1 assembly? And in the hap2 one?
> 2. What is the N50 of the hap2 assembly?
>
> > <solution-title></solution-title>
> >
> > 1. The longest contig in the hap1 assembly is 1,532,843 bp, and 1,531,728 bp in the hap2 assembly.
> > 2. The N50 of the hap2 assembly is 923,452 bp.
> >
> {: .solution}
>
{: .question}

Next, we are going to evaluate the outputs generated by **BUSCO**. This tool provides qualitative assessment of the completeness of a genome assembly in terms of expected gene content. It relies on the analysis of genes that should be present only once in a complete assembly or gene set, while allowing for rare gene duplications or losses ({% cite Simo2015 %}).

<br>

![BUSCO assessment](../../images/vgp_assembly/busco_after_contiging.svg "A composite of BUSCO completeness summaries for hap1 (left) and hap2 (right)")

<br>

As we can see in the report, the results are simplified into four categories: *complete and single-copy*, *complete and duplicated*, *fragmented* and *missing*.

> <question-title></question-title>
>
> 1. How many complete BUSCO genes have been identified in the hap1 assembly?
> 2. How many BUSCOs genes are absent in the hap1 assembly?
>
> > <solution-title></solution-title>
> >
> > 1. According to the report, our assembly contains the 1,436 complete BUSCO genes -- this includes ones that are single-copy and ones that are duplicated.
> > 2. 208 BUSCO genes are missing.
> >
> {: .solution}
>
{: .question}

Despite **BUSCO** being robust for species that have been widely studied, it can be inaccurate when the newly assembled genome belongs to a taxonomic group that is not well represented in [OrthoDB](https://www.orthodb.org/). `Merqury` provides a complementary approach for assessing assembly quality in a reference-free manner via *k*-mer copy number analysis. Specifically, it takes our hap1 as the first genome assembly, hap2 as the second genome assembly, and the merylDB generated previously from our sequencing reads for *k*-mer counts.

By default, `Merqury` generates three collections as output: stats (completeness stats), plots and {QV} stats. Let's first have a look at the copy number (CN) spectrum plot, known as the *spectra-cn* plot. The spectra-cn plot looks at both of your assemblies (here, your two haplotypes) taken *together* (fig. 6a). 

![Figure 6: Merqury spectra-cn plot for initial yeast contigs](../../images/vgp_assembly/yeast_c_merqury_cn.svg "Merqury CN plot for yeast assemblies. The plot tracks the multiplicity of each <i>k</i>-mer found in the read set and colors it by the number of times it is found in a given assembly. Merqury connects the midpoint of each histogram bin with a line, giving the illusion of a smooth curve. <b>a)</b>. <i>K</i>-mer distribution of both haplotypes. <b>b)</b>. <i>K</i>-mer distribution of an individual haplotype (hap2)."){:width="100%"}

Our haplotypes look clean! In the spectra-cn plot for both haplotypes, the peaks are where we should expect them. There is a peak of *k*-mers present at 1-copy (so, only in either hap1 or hap2) with *k*-mer multiplicity of ~25, corresponding to heterozygous coverage. These are our heterozygous alleles being represented only once in our diplolid assemblies, which matches with their haploid coverage. We also have a peak of 2-copy *k*-mers present at diploid coverage, around ~50. This looks good so far, but we should also look at *k*-mer multiplicity for each individual haplotype separately, not just combined as they were in the previous plot. Figure 6b shows that most of the *k*-mers in our assembly are 1-copy -- we have one copy of all the homozygous regions (the ones with 50x coverage) and of about half the heterozygous regions (the ones with 25x coverage). At the 25x coverage point, there is a "read-only" peak, which is expected, as these are the alternative alleles for those heterozygous loci, and those *k*-mers are likely in the other assembly, since they were not missing from the overall spectra-cn plot.


## Hi-C scaffolding (WF8)

In this final stage, we will run the **Scaffolding HiC YAHS (WF8)** workflow, which uses the fact that the contact frequency between a pair of loci strongly correlates with the one-dimensional distance between them (*i.e.*, linear distance on a chromosome). This information allows [**YAHS**](https://github.com/c-zhou/yahs) -- the main tool in this workflow -- to generate scaffolds that are often chromosome-sized.

### Launching Hi-C scaffolding workflow

> <warning-title>The scaffolding workflow is run on <b>ONE</b> haplotype at a time.</warning-title>
> Contiging (VGP4) works with both (hap1/hap2, primary/alternate) assemblies simultaneously. This is not the case for contiging -- it has to be run independently for each haplotype assembly. In this example (below) we run contiging on the hap1 assembly only. You can run the same analysis on the second haplotype by replacing hap1 with hap2. 
{: .warning}

{% snippet faqs/galaxy/workflows_run_ds.md title="Scaffolding HiC (WF8)" dockstore_id="github.com/iwc-workflows/Scaffolding-HiC-VGP8/main" version="v0.2.7" %}

> <hands-on-title> Launching Hi-C scaffolding workflow </hands-on-title>
>
> 1. **Identify inputs**
>
>    The scaffolding workflow takes the following inputs:
>
>    1. An assembly graph
>    2. Forward Hi-C reads
>    3. Reverse Hi-C reads
>    4. Estimated genome size parsed from GenomeScope summary by the previous run of assembly workflow (VGP4).
>    5. Restriction enzymes used in Hi-C library preparation procedure
>    6. Busco Database
>    7. Busco lineage
>
> 2. **Launch scaffolding workflow (WF8)**
>
>    1. Click on the **Workflow** menu, located in the top bar
>    2. Click on the {% icon workflow-run %} **Run workflow** button corresponding to `Scaffolding-HiC-VGP8`
>    3. In the **Workflow: Scaffolding-HiC-VGP8** menu:
>        - {% icon param-file %} "*input GFA*": `usable hap1 gfa` Output of the contiging workflow (VGP4) with a tag `hic_hap1_gfa` for hap1 assembly.
>        - "*Database for Busco Lineage*": `Busco v5 Lineage Datasets` (or the latest version available in the instance)
>        - "*Provide lineage for BUSCO (e.g., Vertebrata)*": `Ascomycota`
>        - {% icon param-file %} "*HiC forward reads*": `Hi-C forward reads`
>        - {% icon param-file %} "*HiC reverse reads*": `Hi-C reverse reads`
>        - "*Restriction enzymes*": `Dovetail Omni-C: enzyme-free prep` For this tutorial, we'll use the Omni-C option as it is the equivalent of not specifying any restriction enzyme cutsites, but for your own data you would want to select the appropriate option. 
>        - {% icon param-file %} "*Estimated genome size - Parameter File*": `Estimated genome size`: An output of the contiging workflow (VGP4) with a tag `estimated_genome_size`.
>    4. Click on the {% icon workflow-run %} **Run workflow** button
{: .hands_on}

### Interpreting the results

In order to evaluate the Hi-C hybrid scaffolding, we are going to compare the contact maps before and after running the HiC hybrid scaffolding workflow (Fig. below). They will have the following tags:

- Before scaffolding: `pretext_s1`
- After scaffolding: `pretext_s2`

Below is the comparison of the two maps obtained from the yeast tutorial data versus real example from a zebra finch (*Taeniopygia guttata*) genome assembly:

![Pretext final contact map](../../images/vgp_assembly/hi-c_pretext_final.svg "Hi-C maps generated by Pretext before and after scaffolding with Hi-C data. The red circles indicate the differences between the contact maps generated before and after Hi-C hybrid scaffolding. The bottom two panels show results of scaffolding on zebra finch where scaffolding dramatically decreases the number of segments by merging multiple contigs into scaffolds.")

The regions marked with red circles highlight the most notable difference between the two contact maps, where inversion has been fixed.

# Conclusion

To sum up, it is worthwhile to compare the final assembly with the [_S. cerevisiae_ S288C reference genome](https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/146/045/GCF_000146045.2_R64/GCF_000146045.2_R64_assembly_stats.txt):

![Quast plot](../../images/vgp_assembly/quast_plot.png "Cumulative continuity plot comparing assembly generated here (red line) with existing yeast reference (black dotted line). Our assembly is slightly smaller (11,304,507 bp versus 12,071,326. Our assembly is lacking the mitochondrial genome (~86 kb) because the initial data does not include mitochondrial reads. This is partially responsible for this discrepancy. ")

With respect to the total sequence length, we can conclude that the size of our genome assembly is very similar to the reference genome. It is noteworthy that the reference genome consists of 17 sequences, while our assembly includes only 16 chromosomes. This is due to the fact that the reference genome also includes the sequence of the mitochondrial DNA, which consists of 85,779 bp. (The above comparison is performed using {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy1) %} using Primary assembly generated with scaffolding workflow (WF8) and yeast reference.)

![Comparison reference genome](../../images/vgp_assembly/hi-c_pretext_conclusion.svg "Comparison between contact maps generated using the final Primary assembly from this tutorial (left) and the reference genome (right).")

If we compare the contact map of our assembled genome with the reference assembly (Fig. above), we can see that the two are indistinguishable, suggesting that we have generated a chromosome level genome assembly.


topic: assembly
tutorial: mitochondrion-assembly
title: Assembly of the mitochondrial genome from PacBio HiFi reads
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["biodiversity"]
content_hash: 6d44cd40003a759ef23de4b3dd749b6a
last_modified: 2026-02-26T21:12:14.479929
zenodo_link: https://zenodo.org/records/13345315
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/mitochondrion-assembly/tutorial.html

content:
This tutorial will show you how to assemble a mitochondrial genome from PacBio HiFi data using MitoHiFi {% cite uliano2023 %}. 
Combined with the tutorials "Using the VGP workflows to assemble a vertebrate genome with HiFi and Hi-C data" and "Decontamination of a genome assembly", this allows you to produce a reference assembly for both the nuclear and the mitochondrial DNA of a vertebrate species. 

This tutorial uses data from the Zebra Finch (*Taeniopygia guttata*) generated by the [Vertebrate Genome Project](https://vertebrategenomesproject.org/). We downsampled the reads that didn't align with the mitochondrial genome so that the tutorial can run faster. 

> <comment-title>Run this analysis on "real" data</comment-title>
>
> If you want to run this analysis on a real sequencing library generated by the [Vertebrate Genome Project](https://vertebrategenomesproject.org/) you can find the PacBio HiFi data on [Genome Ark](https://www.genomeark.org/) as a remote repository and upload it to Galaxy (available on the three main Public Galaxy instances: .org, .eu, .org.au).
>
> {% snippet faqs/galaxy/datasets_import_from_remote_files.md location="Genome Ark -> species -> Taeniopygia_guttata -> bTaeGut2 -> genomic_data -> pacbio_hifi -> *_reads.fasta.gz" %}
>
>
{: .comment}

The assembly is using the wrapped workflow MitoHiFi. 
MitoHiFi: 
- Extracts mitochondrial reads (based on a BLAST against an existing reference mitogenome) and uses Hifiasm {% cite Cheng2021 %} to assemble them. 
- Removes nuclear mitochondrial DNA sequences (NUMTs) from the potential mitogenome contigs
- Generates a circularized and annotated genome for all potential mitogenome contigs
- Selects a representative for the final mitochondrial assembly


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/records/13345315/files/PacBio_reads.fastqsanger.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the datatype is `fastqsanger.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger.gz" %}
>
>
{: .hands_on}

# Download the mitogenome for a related species

To assemble the mitogenome from our PacBio Data,  MitoHiFi needs a reference mitogenome to fish out mitochondrial reads (the reads are blasted against the related reference). To download this reference, we use the tool MitoHiFi with the operation `Find a closely related species`.


> <hands-on-title>  Find a related mitogenome reference  </hands-on-title>
>
> 1. {% tool [MitoHiFi](toolshed.g2.bx.psu.edu/repos/bgruening/mitohifi/mitohifi/3+galaxy0) %} with the following parameters:
>    - *"Operation type selector"*: `Find a close-related mitochondrial reference genome`
>        - *"Species name"*: `Taeniopygia guttata` Enter the latin name of the species you are assembling
>        - *"Email"*: `your.email@service.com` Enter your email 
>        - *"Minimal appropriate length"*: `15000` As vertebrate mitochondrial genomes are typically at least 14kbp long, we are using a value in this range so that we get complete mitogenome results as our reference. 
>
>
{: .hands_on}


## Assemble the mitochondrial genome with **MitoHiFi**

> <hands-on-title>Assemble the mitochondrial genome</hands-on-title>
>
> 1. Create a collection with your PacBio HiFi Reads
>
>    {% snippet faqs/galaxy/collections_build_list.md name="PacBio Reads" datasets_description="the fastq.gz containing the HiFi reads" n="1" %}
> 
> 2. {% tool [MitoHiFi](toolshed.g2.bx.psu.edu/repos/bgruening/mitohifi/mitohifi/3+galaxy0) %} with the following parameters:
>    - *"Operation type selector"*: `Run MitoHiFi`
>        - *"Input mode"*: `Pacbio Hifi Reads`
>            - {% icon param-collection %} *"Pacbio Hifi reads"*: `PacBio Reads` (Input dataset collection)
>        - {% icon param-file %} *"Close-related mitogenome in fasta format"*: `MitoHiFi on : reference genome (FASTA)` (output of **MitoHiFi** {% icon tool %})
>        - {% icon param-file %} *"Close-related mitogenome in genbank format"*: `MitoHiFi on : reference genome (genbank)` (output of **MitoHiFi** {% icon tool %})
>        - *"Genetic code"*: `Vertebrate mitochondrial code`
>        - In *"Advanced options"*:
>            - *"Blast percentage identity"*: `70` This setting filters the potential mito contigs -- setting it to 70 means that we are retaining contigs with at least 70% of its length in the BLAST match. This parameter can be lowered if you are expecting more sequence divergence among mitogenomes of your taxa, or vice versa.
>
>
{: .hands_on}


Outputs of MitoHiFi:
- Final mitogenome (FASTA). The mitochondrial genome circularized and rotated to start at tRNA-Phe.
- Final mitogenome (genbank). The final mitogenome annotated in GenBank format.
- Final mitogenome annotation (png). The predicted genes in the final mitogenome.
- Final mitogenome coverage (png). The sequencing coverage along the final mitogenome.
- Contigs stats (TSV). Contains the statistics of your assembled mitogenomes such as the number of genes, size, whether it was circularized or not, if the sequence has frameshifts, and other metrics.
- Reads mapped to close-related mtDNA (FASTA). All and filtered by size.
- Hifiasm contigs (fasta). The results of running Hifiasm on the mitochondrial reads. 

![Image produced by MitoHiFi showing the genes annotated in our mitogenome](../../images/mito_annotation.png "Final mitogenome annotation")



# Conclusion

In this tutorial, we learned how to assemble the mitochondrial genome using PacBio HiFi reads and MitoHiFi. You can try this tutorial on your own data using the full HiFi read set you'd use for the nuclear genome assembly, since the filtering for mitochondrial reads happens within MitoHiFi.


topic: assembly
tutorial: assembly-quality-control
title: Genome Assembly Quality Control
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["assembly", "quality control", "biodiversity"]
content_hash: e28b72c4df2e7a742ddb5dd342e046a2
last_modified: 2026-02-26T21:12:14.399036
zenodo_link: https://zenodo.org/record/6947782
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/assembly-quality-control/tutorial.html

content:
In this tutorial, we will assess the assembly quality of 2 assemblies generated with Hifiasm and Flye using PacBio HiFi reads of a species of fungi, *Saccharomyces cerevisiae* INSC1019 and compare the results with the actual reference genome [*Saccharomyces cerevisiae* S288C](https://www.ncbi.nlm.nih.gov/genome/?term=Saccharomyces%20cerevisiae).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

We will use long quality reads sequencing data from PacBio HiFi sequencing of *Saccharomyces cerevisiae* INSC1019 genome. This data is a subset of data from ENA repository [SRR13577847](https://www.ebi.ac.uk/ena/browser/view/SRR13577847?show=reads). We will also use the reference genome assembly downloaded from the [NCBI website](https://www.ncbi.nlm.nih.gov/genome/?term=Saccharomyces%20cerevisiae) and we will use it as a comparison with our own assemblies.

## Get data from Zenodo

> <hands-on-title>Data upload from Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo](https://zenodo.org/record/5702408)
>
>    ```
>    https://zenodo.org/record/6947782/files/GCA_000146045.2_genomic.fna
>    https://zenodo.org/record/6947782/files/Scerevisiae-INSC1019.flye.30x.fa
>    https://zenodo.org/record/6947782/files/Scerevisiae-INSC1019.hifiasm.30x.fa
>    https://zenodo.org/record/6947782/files/SRR13577847_subreads.30x.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the datatype is `fastqsanger.gz` for `SRR13577847_subreads.30x.fastq.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

# Evaluation of assembly contiguity and correctness with **Quast**

A way to calculate metrics assembly is to use **QUAST = QUality ASsessment Tool**. Quast is a tool to evaluate genome assemblies by computing various metrics and to compare genome assembly with a reference genome. The manual of Quast is here: [Quast](http://quast.sourceforge.net/docs/manual.html#sec3)

> <comment-title>QUAST statistics</comment-title>
>
> QUAST will provide us with the following statistics:
>
> - Contigs: The total number of contigs in the assembly.
> - Largest contig: The length of the largest contig in the assembly.
> - Total length: The total number of bases in the assembly.
> - Nx: The largest contig length, *L*, such that contigs of length >= *L* account for at least *x*% of the bases of the assembly.
> - NGx: The contig length such that using equal or longer length contigs produces *x*% of the length of the reference genome, rather than *x*% of the assembly length.
> - GC content: the percentage of nitrogenous bases which are either guanine or cytosine.
>
> If a reference genome is provided, the report also indicates:
> - Genome fraction (%): total number of aligned bases in the reference genome, divided by the genome size
> - Duplication ratio: total number of aligned bases in the assembly, divided by the total number of aligned bases in the reference genome
> - NGA50: NG50 where the lengths of aligned blocks are counted instead of contig lengths. i.e., if a contig has a misassembly with respect to the reference, the contig is broken into smaller pieces. This metric is computed only if a reference genome is provided.
> LGA50: LG50 where aligned blocks are counted instead of contigs. i.e., if a contig has a misassembly with respect to the reference, the contig is broken into smaller pieces
>
> Many other metrics are available when clicking on `Extended report`!
{: .comment}

> <hands-on-title>assembly evaluation with Quast</hands-on-title>
>
> 1. {% tool [Quast](toolshed.g2.bx.psu.edu/repos/iuc/quast/quast/5.2.0+galaxy0) %} with the following parameters:
>    - *"Use customized names for the input files?"*: `No, use datasets names`
>    - {% icon param-files %} *"Contigs/scaffolds file"*: `Scerevisiae-INSC1019.flye.30x.fa` and `Scerevisiae-INSC1019.hifiasm.30x.fa`
>    - *"Reads options"*: `Pacbio SMRT reads`
>        - {% icon param-file %} *"FASTQ file"*: `SRR13577847_subreads.30x.fastq.gz`
>    - *"Type of assembly"*: `Genome`
>        - *"Use a reference genome?"*: `Yes`
>            - {% icon param-file %} *"Reference genome"*: `GCA_000146045.2_genomic.fna`
>    - *"Is genome large (>100Mpb)?"*: `No`
>
{: .hands_on}

> <question-title></question-title>
>
> With the informations generated by Quast, compare the two assemblies to the reference genome
>
> 1. Which assembly is closest to the reference genome?
> 2. What is the longest contig of the Flye and Hifiasm assemblies?
> 3. What is the N50 of the Flye assembly?
> 4. Which percentage of reads mapped to each assembly?
>
> > <solution-title></solution-title>
> >
> > 1. The Flye assembly, with a genome fraction of 99.57% while Hifiasm have a low genome fraction (75.15%) which can indicate a problematic assembly.
> > 2. The longest contig is 1,438,238 bp for Flye assembly while is 1,795,653 bp for Hifiasm
> > 3. The N50 of the Flye assembly is 929,061 bp.
> > 4. A total of 99.96% of reads mapped were mapped to the Flye assembly while only 91.02% against the Hifiasm assembly.
> >
> > Without a reference genome, it would have been harder to determine that assembly with Hifiasm is problematic. Metrics on the assembly itself are not always sufficient and therefore it is necessary to use other methodologies to ensure the quality of an assembly.
> {: .solution}
>
{: .question}

# Evaluation of assembly completness

## Core genes completness with **BUSCO**

**BUSCO (Benchmarking Universal Single-Copy Orthologs)** allows a measure for quantitative assessment of genome assembly based on evolutionarily informed expectations of gene content. Details for this tool are here: [Busco website](https://busco.ezlab.org/)

> <hands-on-title>assessing assembly completness with BUSCO</hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/5.3.2+galaxy0) %} with the following parameters:
>    - {% icon param-files %} *"Sequences to analyse"*:`Scerevisiae-INSC1019.flye.30x.fa`, `Scerevisiae-INSC1019.hifiasm.30x.fa` and `GCA_000146045.2_genomic.fna`
>    - *"Mode"*: `Genome assemblies (DNA)`
>        - *"Use Augustus instead of Metaeuk"*: `Use Metaeuk`
>    - *"Auto-detect or select lineage?"*: `Select lineage`
>       - *"Lineage"*: `Saccharomycetes`
>    - *"Which outputs should be generated"*: `short summary text` and `summary image`
>
{: .hands_on}

> <question-title></question-title>
>
> Compare the number of BUSCO genes identified in the Flye and Hifiasm assembly to the reference genome. What do you observe ?
>
> > <solution-title></solution-title>
> >
> > Short summary generated by BUSCO indicates that reference genome contains:
> > 1. 2129 Complete BUSCOs (of which 2128 are single-copy and 49 are duplicated),
> > 2. 2 fragmented BUSCOs,
> > 3. 6 missing BUSCOs.
> >
> > Short summary generated by BUSCO indicates that Flye assembly contains:
> > 1. 2127 complete BUSCOs (2127 single-copy and 49 duplicated),
> > 2. 2 fragmented BUSCOs
> > 3. 8 missing BUSCOs.
> >
> > Short summary generated by BUSCO indicates that Hifiasm assembly contains:
> > 1. 1663 complete BUSCOs (1662 single-copy and 27 duplicated),
> > 2. 5 fragmented BUSCOs
> > 3. 469 missing BUSCOs.
> >
> >
> > BUSCO analysis confirms that Flye assembly is better than Hifiasm assembly. Flye is also similar in completness to the reference genome with similar number of complete, fragmented and missing BUSCOs genes.
> >
> {: .solution}
>
{: .question}

Despite BUSCO being robust for species that have been widely studied, it can be inaccurate when the newly assembled genome belongs to a taxonomic group that is not well represented in [OrthoDB](https://www.orthodb.org/). Even in a well-represented taxonomic group, the bias on the selection of reference genomes selected to create OrthoDB can lead to an under-scoring of the newly assembled genome and is dependent on the evolution of the genomes. For example, in microsporidia, basal genomes have much lower scores due to the strong drive on gene loss and gain in these organisms.

## k-mer based assembly evaluation with **Merqury**

Meryl will allow us to generate the k-mer profile by decomposing the sequencing data into k-length substrings, counting the occurrence of each k-mer and determining its frequency. Meryl comprises three main modules: one for generating k-mer databases, one for filtering and combining databases, and one for searching databases.

> <hands-on-title>Generate k-mers count distribution</hands-on-title>
>
> 1. {% tool [Meryl](toolshed.g2.bx.psu.edu/repos/iuc/meryl/meryl/1.3+galaxy6) %} with the following parameters:
>    - *"Operation type selector"*: `Count operations`
>        - *"Count operations"*: `Count: count the ocurrences of canonical k-mers`
>        - {% icon param-file %} *"Input sequences"*: `SRR13577847_subreads.30x.fastq.gz`
>        - *"K-mer size selector"*: `Estimate the best k-mer size`
>            - "*Genome size*": `12000000`
>
{: .hands_on}

Merqury is used for reference-free assembly evaluation based on k-mer set operations. By comparing k-mers in a de novo assembly to those found in unassembled high-accuracy reads, Merqury estimates base-level accuracy and completeness. For trios, Merqury can also evaluate haplotype-specific accuracy, completeness, phase block continuity, and switch errors. Multiple visualizations and tables, such as k-mer spectrum plots, are generated for evaluation of the assembly quality.

> <hands-on-title>K-mer based evaluation with Merqury</hands-on-title>
>
> 1. {% tool [Merqury](toolshed.g2.bx.psu.edu/repos/iuc/merqury/merqury/1.3+galaxy2) %} with the following parameters:
>    - *"Evaluation mode"*: `Default mode`
>        - {% icon param-file %} *"K-mer counts database"*: `Meryldb` (output of **Meryl** {% icon tool %})
>        - *"Number of assemblies"*: `One assembly`
>            - {% icon param-files %} *"Genome assembly"*: `Scerevisiae-INSC1019.flye.30x.fa`, `Scerevisiae-INSC1019.hifiasm.30x.fa` and `GCA_000146045.2_genomic.fna`
>
{: .hands_on}

By default, Merqury generates three collections as output: stats, plots and QV stats. The "stats" and "plots" collection contains the completeness statistics, while the "QV stats" collection contains the quality value statistics.

> <question-title></question-title>
>
> Compare the k-mers plots for the Flye and Hifiasm assembly and the reference genome. What do you observe ?
>
> > <solution-title></solution-title>
> >
> > Merqury analysis confirms that Flye assembly is better than Hifiasm assembly. Hifiasm `assembly.spectra-cn.fl` shows that part of the data is specific to reads (black peak overlapping the red peak) indicating an absence of contigs in the assembly to represent these data.
> >
> > Flye it's similar in completness to the reference genome with a single read peak. We note however that part of the reference genome assembly is specific and absent from the reads (red barplot on the left). This is because the sequencing data is from a different strain (INSC1019) than the reference genome (S288C).
> >
> {: .solution}
>
{: .question}

# Evaluation against reference genome with **Chromeister**

Chromeister is used to generate dotplots to quickly compare sequence sets, even for large genomes. This is provide a synthetic similarity overview, highlighting repetitions, breaks and inversions.

> <hands-on-title>Synteny evaluation with Chromeister</hands-on-title>
>
> 1. {% tool [Chromeister](toolshed.g2.bx.psu.edu/repos/iuc/chromeister/chromeister/1.5.a+galaxy1) %} with the following parameters:
>    - {% icon param-files %} *"Query sequence"*: `Scerevisiae-INSC1019.flye.30x.fa` and `Scerevisiae-INSC1019.hifiasm.30x.fa`
>    - {% icon param-file %} *"Reference sequence"*: `GCA_000146045.2_genomic.fna`
>
{: .hands_on}

> <question-title></question-title>
>
> Compare the dotplots for the Hifiasm assembly and the reference genome. What do you observe ?
>
> > <solution-title></solution-title>
> >
> > ![Chromeister dotplot](../../images/genome-qc/chromeister-dotplot.png)
> >
> > With dotplots, we can quickly detect several inconsistencies between the assembly generated by Hifiasm and the reference genome:
> > - Orange arrows indicate contigs in the reference genome that are absent in the Hifiasm assembly
> > - Purple arrow indicates a contig specific to the Hifiasm assembly. This is not necessarily an inconsistency as it is 2 different strains of the same species.
> > - Brown arrows indicate contigs that are merged into the Hifiasm assembly. In the case of the top arrow, there is a fusion and a partial inversion.
> > - Green arrow indicates a contig partially assembled by Hifiasm.
> >
> {: .solution}
>
{: .question}

# Conclusion


This pipeline shows how to evaluate a genome assembly. Once you are satisfied with your genome sequence, you might want to purge it, make scaffolding and directly starting the annotation process!


topic: assembly
tutorial: vgp_genome_assembly
title: Vertebrate genome assembly using HiFi, Bionano and Hi-C data - Step by Step
difficulty: intermediate
time_estimation: 5h
tools_json: []
requirements_json: []
tags_json: []
content_hash: ab6ca5357a056aaa94f7eda00524c4c5
last_modified: 2026-02-26T21:12:14.495834
zenodo_link: https://zenodo.org/record/5887339
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/vgp_genome_assembly/tutorial.html

content:
Advances in sequencing technologies over the last few decades have revolutionized the field of genomics, allowing for a reduction in both the time and resources required to perform *de novo* genome assembly. Until recently, second-generation sequencing technologies (also known as {NGS}) produced highly accurate but short (up to 800bp) reads. These read lengths were not long enough to cope with the difficulties associated with repetitive regions. Today, so-called {TGS} technologies, also known as {SMRT} sequencing, have become dominant in *de novo* assembly of large genomes. TGS can use native DNA without amplification, reducing sequencing error and bias ({% cite Hon2020 %}, {% cite Giani2020 %}). In 2020, PacBio introduced {HiFi} sequencing, which produces reads 10-25 kbp in length with a minimum accuracy of 99% (Q20). In this tutorial, you will use HiFi reads in combination with data from additional sequencing technologies to generate a high-quality genome assembly.

Deciphering the structural organization of complex vertebrate genomes is currently one of the largest challenges in genomics ({% cite Frenkel2012 %}). Despite the significant progress made in recent years, a key question remains: what combination of data and tools can produce the highest quality assembly? In order to adequately answer this question, it is necessary to analyse two of the main factors that determine the difficulty of genome assembly processes: repetitive content and heterozygosity.

Repetitive elements can be grouped into two categories: interspersed repeats, such as {TE} that occur at multiple loci throughout the genome, and {TR} that occur at a single locus ({% cite Trresen2019 %}). Repetitive elements are an important component of eukaryotic genomes, constituting over a third of the genome in the case of mammals ({% cite SoteroCaio2017 %}, {% cite Chalopin2015 %}). In the case of tandem repeats, various estimates suggest that they are present in at least one third of human protein sequences ({% cite Marcotte1999 %}). {TE} content is among the main factors contributing to the lack of continuity in the reconstruction of genomes, especially in the case of large ones, as TE content is highly correlated with genome size ({% cite SoteroCaio2017 %}). On the other hand, {TR} usually lead to local genome assembly collapse, especially when their length is close to that of the reads ({% cite Trresen2019 %}).

Heterozygosity is also an important factor impacting genome assembly. Haplotype phasing, the identification of alleles that are co-located on the same chromosome, has become a fundamental problem in heterozygous and polyploid genome assemblies ({% cite Zhang2020 %}). When no reference sequence is available, the state-of-the-art strategy consists of constructing a string graph with vertices representing reads and edges representing consistent overlaps. In this kind of graph, after transitive reduction, heterozygous alleles in the string graph are represented by bubbles. When combined with {Hi-C} data, this approach allows complete diploid reconstruction ({% cite DominguezDelAngel2018 %}, {% cite Zhang2020 %}, {% cite Dida2021 %}).

The {G10K} launched the Vertebrate Genome Project ({VGP}), whose goal is generating high-quality, near-error-free, gap-free, chromosome-level, haplotype-phased, annotated reference genome assemblies for every vertebrate species ({% cite Rhie2021 %}). This tutorial will guide you step by step to assemble a high-quality genome using the VGP assembly pipeline, including multiple {QC} evaluations.

> <warning-title>Your results may differ!</warning-title>
>
> Some of your results may slightly differ from the results shown in this tutorial, depending on the versions of the tools used, since algorithms can change between versions.
>
{: .warning}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Important terms to know

Before getting into the thick of things, let's go over some terms you will often hear when learning about genome assembly. These concepts will be used often throughout this tutorial as well, so please refer to this section as necessary to help your understanding.

**Pseudohaplotype assembly**: A genome assembly that consists of long-phased haplotype blocks separated by regions where the haplotype cannot be distinguished (often homozygous regions). This can result in "switch errors", when the parental haplotypes alternate along the same sequence. These types of assemblies are usually represented by a _primary assembly_ and an _alternate assembly_. (This definition is largely taken from the [NCBI's Genome Assembly Model](https://www.ncbi.nlm.nih.gov/assembly/model/#asmb_def).)

**Primary assembly**: The primary assembly is traditionally the more complete representation of an individual's genome and consists of homozygous regions and one set of loci for heterozygous regions. Because the primary assembly contains both homo- and heterozygous regions, it is more complete than the _alternate assambly_ which often reports only the other allele for heterozygous regions. Thus, the primary assembly is usually what one would use for downstream analyses.

**Alternate assembly**: The alternate assembly consists of the alternate alleles not represented in the _primary assembly_ (heterozygous loci from the other haplotype). These types of sequences are often referred to as haplotigs. Traditionally, the alternate assembly is less complete compared to the primary assembly since homozygous regions are not represented.

**Phasing**: Phasing aims to partition the contigs for an individual according to the haplotype they are derived from. When parental data is available, this is done by identifying parental alleles using read data from the parents. Locally, this is achieved using linkage information in long read datasets. Recent approaches have managed to phase using long-range {Hi-C} linkage information from the same individual ({% cite Cheng2021 %}).

**Assembly graph**: A representation of the genome inferred from sequencing reads. Sequencing captures the genome as many fragmented pieces, instead of whole entire chromosomes at once (we eagerly await the day when this statement will be outdated!). The start of the assembly process pieces together these genome fragments to generate an assembly graph, which is a representation of the sequences and their overlaps. Visualizing assembly graphs can show where homozygous regions branch off into alternate paths on different haplotypes.

**Unitig**: Usually the smallest unit of an assembly graph, consistent with all the available sequencing data. A unitig is often constructed from an unambiguous path in the assembly graph where all the vertices have exactly one incoming and one outgoing edge, except the first vertex can have any number of incoming edges, while the last vertex can have any number of outgoing edges ({% cite Rahman2022 %}). In other words, the internal vertices in the unitig path can only be walked one way, so unitigs represent a path of confident sequence. In the assembly graph, unitig nodes can then have overlap edges with other unitigs.

**Contig**: A contiguous (*i.e.*, gapless) sequence in an assembly, usually inferred algorithmically from the unitig graph.

**False duplications**: Assembly errors that result in one region of the genome being represented twice in the same assembly as two separate regions. Not to be confused with optical or technical duplicates from PCR from short-read sequencing. False duplications can further be classified as either _haplotypic duplications_ or _overlaps_.

**Haplotypic duplication** can happen when a region that is heterozygous in the individual has the two haplotypes showing enough divergence that the assembler fails to interpret them as homologous. For example, say an individual is heterozygous in the region Chr1[1:100] and has Haplotype A from their mother and Haplotype B from their father; a false duplication can arise when the haplotypes are not recognized as being from the same region, and the assembler ends up placing both haplotypes in the same assembly, resulting in Chr1[1:100] being represented twice in one assembly. Ideally, a properly phased assembly would have Haplotype A in one assembly, *e.g.*, the primary, while Haplotype B is in the alternate.

False duplications via **overlaps** result from unresolved overlaps in the assembly graph. In this case, two different contigs end up representing the same sequence, effectively duplicating it. Overlaps happen when a branching point in an assembly graph is resolved such that the contig before the vertex and a contig after the vertex share the same overlapping sequence.

![Types of false duplication.](../../images/vgp_assembly/falseduplications.png "Schematic of types of false duplication. Image adapted from {% cite Rhie2021 %}.")

**Purging**: Purging aims to remove false duplications, collapsed repeats, and very low support/coverage regions from an assembly. When performed on a primary assembly, the haplotigs are removed from the primary and typically placed in the alternate assembly.

**Scaffold**: A scaffold refers to one or more contigs separated by gap (unknown) sequence. Scaffolds are usually generated with the aid of additional information, such as Bionano optical maps, linked reads, Hi-C chromatin information, etc. The regions between contigs are usually of unknown sequence, thus they are represented by sequences of _N_'s. Gap length in the sequence can be sized or arbitrary, depending on the technology used for scaffolding (*e.g.*, optical maps can introduce sized gaps, while scaffolding software using {Hi-C} information usually uses an arbitrary number of N's, such as 500 or 200).

For more about the specific scaffolding technologies used in the VGP pipeline (currently Bionano optical maps and Hi-C chromatin conformation data), please refer to those specific sections within this tutorial.

**HiFi reads**: PacBio {HiFi} reads are the focus of this tutorial. First described in 2019, they have revolutionized genome assembly by combining long (about 10-20 kbp) read lengths with high accuracy (>Q20) typically associated with short-read sequencing ({% cite Wenger2019 %}). These higher read lengths enable HiFi reads to traverse some repeat regions that are problematic to assemble with short reads.

**Ultra-long reads**: Ultra-long reads are typically defined as reads of over 100 kbp, and are usually generated using Oxford Nanopore Technology. Read quality is often lower than HiFi or Illumina (*i.e.*, have a higher error rate), but they are often significantly longer than any other current sequencing technology, and can help assembly algorithms walk complex repeat regions in the assembly graphs.

**Manual curation**: This term refers to manually evaluating and manipulating an assembly based on the raw supporting evidence (*e.g.*, using Hi-C contact map information). The user takes into account the original sequencing data to resolve potential _misassemblies_ and _missed joins_.

**Misassembly**: Misassemblies are a type of assembly error that usually refers to any structural error in the genome reconstruction, *e.g.*, sequences that are not adjacent in the genome being placed next to each other in the sequence. Misassemblies can be potentially identified and remedied by manual curation.

**Missed join**: A missed join happens when two sequences are adjacent to each other in the genome but are not represented contiguously in the final sequence. Missed joins can be identified and remedied in manual curation with Hi-C data.

**Telomere-to-telomere assembly**: Often abbreviated as "T2T", this term refers to an assembly where each chromosome is completely gapless from telomere to telomere. The term usually refers to the recently completed CHM13 human genome ({% cite Nurk2022 %}), though there is an increasing number of efforts to generate T2T genomes for other species.

# VGP assembly pipeline overview

The {VGP} assembly pipeline has a modular organization, consisting of ten workflows (Fig. 1). It can used with the following types of input data:

|  Input data | Assembly quality  | Analysis trajectory <br>([Fig. 2)](#figure-2)|
|------|---------------|-----|
| HiFi | The minimum requirement | A |
| HiFi + HiC | Better haplotype resolution (fewer switch errors) | B |
| HiFi + BioNano | Better contiguity | C |
| HiFi + Hi-C + BioNano | Even better contiguity | D |
| HiFi + parental data | Properly phased | E |
| HiFi + parental data + Hi-C | Better haplotype resolution | F |
| HiFi + parental + BioNano | Properly phased with improved contiguity | G |
| HiFi + parental data + Hi-C + BioNano | Properly phased with even more improved contiguity | H |

In this table, *HiFi* and *Hi-C* refer to HiFi and Hi-C data derived from the individual whose genome is being assembled. **This tutorial assumes you are assembling the genome of one individual; there are special considerations necessary for pooled data that are not covered in this tutorial.** (Note: you can use Hi-C data from another individual of the same species to scaffold, but you *cannot* use that data to phase the contigs in hifiasm.) *Parental data* is high-coverage whole genome resequencing data derived from the parents of the individual being assembled, and is the key component of trio-based genome assembly. Each combination of input datasets is demonstrated in Fig. 2 by an *analysis trajectory*: a combination of workflows designed for generating the best assembly given a particular combination of inputs. These trajectories are listed in the table above and shown in the figure below.

![The nine workflows of Galaxy assembly pipeline](../../images/vgp_assembly/VGP_workflow_modules.svg "Eight analysis trajectories are possible depending on the combination of input data. A decision on whether or not to invoke Workflow 6 is based on the analysis of QC output of workflows 3, 4, or 5. Thicker lines connecting Workflows 7, 8, and 9 represent the fact that these workflows are invoked separately for each phased assembly (once for maternal and once for paternal).")

The stages of genome assembly in the VGP-Galaxy pipeline are generally:

1. ***K*-mer profiling**: the generation of *k*-mer profiles of the raw reads to estimate genome size, heterozygosity, repetitiveness, and error rate. This is useful for getting an idea of the genome that lies within your reads, and is also useful for necessary for parameterizing downstream workflows. The generation of *k*-mer counts can be done from HiFi data only (Workflow 1), or include data from parental reads for trio-based phasing (Workflow 2), if one wants to generate *k*-mer spectra for the individual's parents, as well.

2. **Contig assembly**: In addition to using only {HiFi} reads (Workflow 3), the contig building (contiging) step can leverage {Hi-C} (Workflow 4) or parental read data (Workflow 5) to produce fully-phased haplotypes (hap1/hap2 or parental/maternal assigned haplotypes), using [`hifiasm`](https://github.com/chhylp123/hifiasm). The contiging workflows also produce a number of critical quality control (QC) metrics such as *k*-mer multiplicity profiles. Inspection of these profiles provides information to decide whether the third stage—purging of false duplication—is required.

3. (*Optional*) **Purging**: Purging duplicates (Workflow 6) using [`purge_dups`](https://github.com/dfguan/purge_dups) identifies and resolves haplotype-specific assembly segments incorrectly labeled as primary contigs, as well as heterozygous contig overlaps. This increases contiguity and the quality of the final assembly. The purging stage is generally unnecessary for trio data, as haplotype resolution is attained using set operations done on parental *k*-mers.

4. **Scaffolding**: Scaffolding produces chromosome-level scaffolds using information provided by Bionano (Workflow 7, with [`Bionano Solve`](https://bionano.com/software-downloads/) (optional)) and Hi-C (Workflow 8, with [`YaHS`](https://github.com/c-zhou/yahsscaffolding) algorithms).

5. **Decontamination**: A final step of decontamination (Workflow 9) removes non-target sequences (e.g., contamination as well as mitochondrial sequences) from the scaffolded assembly. A separate workflow (WF0) is used for mitochondrial assembly.

> <comment-title>A note on data quality</comment-title>
> For diploids, we suggest at least 30✕ PacBio HiFi coverage & around 60✕ Hi-C coverage, and up to 60✕ HiFi coverage to accurately assemble highly repetitive regions.
{: .comment}
This training has been organized into four main sections: genome profile analysis, assembly of {HiFi} reads with hifiasm, scaffolding with Bionano optical maps, and scaffolding with {Hi-C} data. Additionally, the **assembly with hifiasm** section has two possible paths in this tutorial: solo contiging or solo w/HiC contiging.

Throughout this tutorial, there will be **detail boxes** with additional background information on the science behind the sequencing technologies and software we use in the pipeline. These boxes are minimized by default, but please expand them to learn more about the data we utilize in this pipeline.

# Get data

In order to reduce computation time, we will assemble samples from the yeast _Saccharomyces cerevisiae_ S288C, a widely used laboratory strain isolated in the 1950s by Robert Mortimer. Using _S. cerevisae_, one of the most intensively studied eukaryotic model organisms, has the additional advantage of allowing us to evaluate the final result of our assembly with great precision. For this tutorial, we generated a set of synthetic HiFi reads corresponding to a theoretical diploid genome.

> <details-title>Generation of synthetic reads</details-title>
> The synthetic HiFi reads were generated by using the [S288C assembly](https://www.ncbi.nlm.nih.gov/genome/15?genome_assembly_id=22535) as reference genome. With this objective we used [HIsim](https://github.com/thegenemyers/HI.SIM), a HiFi shotgun simulator. The commands used are detailed below:
>
> ```bash
> ./FastK -T8 -Nsynthetic -k40 -t4 ./hifi_reads.fastq.gz
> ./FastK -T8 -Nsynthetic -k40 -p:synthetic.ktab .hifi_reads.fastq.gz
> ./Histex -G synthetic > synthetic.histogram
> ./GeneScopeFK.R -i synthetic.histogram -o ./synthetic -k 40 -p 1
> ./HImodel -v -osynthetic -g10:55 -e5 -T8 ./synthetic
> ./HIsim ./GCF_000146045.2_R64_genomic.fasta ./synthetic.model -p.2,.2 -c30 -r1 -U -f -h -e
> ```
> The selected mutation rate was 2%. Note that HIsim generates the synthetic reads in FASTA format. This is perfectly fine for illustrating the workflow, but you should be aware that usually you will work with HiFi reads in FASTQ format.
>
{: .details}

The first step is to get the datasets from Zenodo. Specifically, we will be uploading two datasets:

1. A set of PacBio {HiFi} reads in `fasta` format. Please note that your HiFi reads received from a sequencing center will usually be fastqsanger.gz format, but the dataset used in this tutorial has been converted to fasta for space.
2. A set of Illumina {Hi-C} reads in `fastqsanger.gz` format.

## Uploading `fasta` datasets from Zenodo

The following two steps demonstrate how to upload three PacBio {HiFi} datasets into your Galaxy history.

> <hands-on-title><b>Uploading <tt>FASTA</tt> datasets from Zenodo</b></hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Copy the following URLs into the clipboard.
>    - you can do this by clicking on {% icon copy %} button in the right upper corner of the box below. It will appear if you mouse over the box.
>
>    ```
>    https://zenodo.org/record/6098306/files/HiFi_synthetic_50x_01.fasta
>    https://zenodo.org/record/6098306/files/HiFi_synthetic_50x_02.fasta
>    https://zenodo.org/record/6098306/files/HiFi_synthetic_50x_03.fasta
>    ```
>
> 3. Upload datasets into Galaxy.
>    - set the datatype to `fasta`
>
> {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>
> {% snippet topics/assembly/tutorials/vgp_genome_assembly/faqs/dataset_upload_fasta_via_urls.md %}
>
{: .hands_on}

## Uploading `fastqsanger.gz` datasets from Zenodo

Illumina {Hi-C} data is uploaded in essentially the same way as shown in the following two steps.

> <warning-title>DANGER: Make sure you choose the correct format!</warning-title>
> When selecting datatype in "**Type (set all)**" drop-down, make sure you select `fastaqsanger` or `fastqsanger.gz` BUT NOT `fastqcssanger` or anything else!
{: .warning}

> <hands-on-title><b>Uploading <tt>fastqsanger.gz</tt> datasets from Zenodo</b></hands-on-title>
>
> 1. Copy the following URLs into the clipboard. You can do this by clicking on {% icon copy %} button in the right upper corner of the box below. It will appear if you mouse over the box.
>
>    ```
>    https://zenodo.org/record/5550653/files/SRR7126301_1.fastq.gz
>    https://zenodo.org/record/5550653/files/SRR7126301_2.fastq.gz
>    ```
>
> 2. Upload datasets into Galaxy and set the datatype to `fastqsanger.gz`
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fastqsanger.gz" %}
>
>    {% snippet topics/assembly/tutorials/vgp_genome_assembly/faqs/dataset_upload_fastqsanger_via_urls.md %}
>
> 3. Rename the datasets as follow: 
>     -  Rename `SRR7126301_1.fastq.gz` as `Hi-C_dataset_F`. It contains the forward reads.
>     -  Rename `SRR7126301_2.fastq.gz` as `Hi-C_dataset_R`. It contains the reverse reads.
>
{: .hands_on}

> <warning-title>These datasets are large!</warning-title>
> Hi-C datasets are large. It will take some time (~15 min) for them to be fully uploaded. Please, be patient.
{: .warning}

## Organizing the data

If everything goes smoothly your history will look like shown in the figure below. The three {HiFi} fasta files are better represented as a collection: {collection}. Also, importantly, the workflow we will be using for the analysis of our data takes a collection as input (it does not access individual datasets). So let's create a collection using steps outlined in the Tip {% icon tip %} "Creating a dataset collection":

{% snippet faqs/galaxy/collections_build_list.md %}

The view of your history should transition from what is shown in the left pane below to what looks like the right pane:

![AfterUpload](../../images/vgp_assembly/making_list.svg "History after uploading HiFi and HiC data (left). Creation of a list (collection) combines all HiFi datasets into a single history item called 'HiFi data' (right). See below for instructions on how to make this collection.")

> <details-title>Other ways to upload the data</details-title>
> You can obviously upload your own datasets via URLs as illustrated above or from your own computer. In addition, you can upload data from a major repository called [GenomeArk](https://genomeark.org). GenomeArk is integrated directly into Galaxy Upload. To use GenomeArk follow the steps in the Tip {% icon tip %} below:
>
> {% snippet faqs/galaxy/datasets_upload_from_genomeark.md %}
{: .details}

# HiFi reads preprocessing with **cutadapt**

Adapter trimming usually means trimming the adapter sequence off the ends of reads, which is where the adapter sequence is usually located in {NGS} reads. However, due to the nature of {SMRT} sequencing technology, adapters do not have a specific, predictable location in  {HiFi} reads. Additionally, the reads containing adapter sequences could be of generally lower quality compared to the rest of the reads. Thus, we will use **cutadapt** not to trim, but to remove the entire read if a read is found to have an adapter inside of it.

> <details-title>Background on PacBio HiFi reads</details-title>
>
> PacBio HiFi reads rely on {SMRT} sequencing technology. SMRT is based on real-time imaging of fluorescently tagged nucleotides as they are added to a newly synthesized DNA strand. HiFi further uses multiple subreads from the same circular template to produce one highly accurate consensus sequence (fig. 2).
>
> ![PacBio HiFi technology](../../images/vgp_assembly/pacbio_technology.png "HiFi reads are produced by calling consensus from subreads generated by multiple passes of the enzyme around a circularized template. This results in a HiFi read that is both long and accurate.")
>
> This technology allows to generate long-read sequencing data with read lengths in the range of 10-25 kb and minimum read consensus accuracy greater than 99% (Q20).
>
{: .details}

> <hands-on-title>Primer removal with Cutadapt</hands-on-title>
>
> 1. Run {% tool [Cutadapt](toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/4.4+galaxy0) %} with the following parameters:
>    - *"Single-end or Paired-end reads?"*: `Single-end`
>    - {% icon param-collection %} *"FASTQ/A file"*: `HiFi_collection`
>    - In *"Read 1 Options"*:
>        - In *"5' or 3' (Anywhere) Adapters"*:
>            - {% icon param-repeat %} *"Insert 5' or 3' (Anywhere) Adapters"*
>                - *"Source"*: `Enter custom sequence`
>                - *"Enter custom 5' or 3' adapter name"*: `First adapter`
>                - *"Enter custom 5' or 3' adapter sequence"*: `ATCTCTCTCAACAACAACAACGGAGGAGGAGGAAAAGAGAGAGAT`
>            - {% icon param-repeat %} *"Insert 5' or 3' (Anywhere) Adapters"*
>                - *"Source"*: `Enter custom sequence`
>                - *"Enter custom 5' or 3' adapter name"*: `Second adapter`
>                - *"Enter custom 5' or 3' adapter sequence"*: `ATCTCTCTCTTTTCCTCCTCCTCCGTTGTTGTTGTTGAGAGAGAT`
>    - In *"Adapter Options"*:
>        - *"Maximum error rate"*: `0.1`
>        - *"Minimum overlap length"*: `35`
>        - *"Look for adapters in the reverse complement"*: `Yes`
>    - In *"Filter Options"*:
>        - *"Discard Trimmed Reads"*: `Yes`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
> 2. Rename the output file as `HiFi_collection (trimmed)`.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Genome profile analysis

Before starting a *de novo* genome assembly project, it is useful to collect metrics on the properties of the genome under consideration, such as the expected genome size, so that you know what to expect from your assembly. Traditionally, DNA flow cytometry was considered the golden standard for estimating the genome size. Nowadays, experimental methods have been replaced by computational approaches ({% cite wang2020estimation %}). One widely used genome profiling methods is based on the analysis of *k*-mer frequencies. It allows one to provide information not only about the genomic complexity, such as the genome size and levels of heterozygosity and repeat content, but also about the data quality.

> <details-title><i>K</i>-mer size, sequencing coverage and genome size</details-title>
>
>*K*-mers are unique substrings of length *k* contained within a DNA sequence. For example, the DNA sequence *TCGATCACA* can be decomposed into five unique *k*-mers that have five bases long: *TCGAT*, *CGATC*, *GATCA*, *ATCAC* and *TCACA*. A sequence of length L will have  L-k+1 *k*-mers. On the other hand, the number of possible *k*-mers can be calculated as  n<sup>k</sup>, where n is the number of possible monomers and k is the *k*-mer size.
>
>
>---------| -------------|-----------------------
>  Bases  |  *k*-mer size  |  Total possible *k*-mers
>---------| -------------|-----------------------
>    4    |       1      |            4
>    4    |       2      |           16
>    4    |       3      |           64
>    4    |      ...     |          ...
>    4    |      10      |    1.048.576
>---------|--------------|-----------------------
>
> Thus, the *k*-mer size is a key parameter, which must be large enough to map uniquely to the genome, but not too large, since it can lead to wasting computational resources. In the case of the human genome, *k*-mers of 31 bases in length lead to 96.96% of unique *k*-mers.
>
> Each unique *k*-mer can be assigned a value for coverage based on the number of times it occurs in a sequence, whose distribution will approximate a Poisson distribution, with the peak corresponding to the average genome sequencing depth. From the genome coverage, the genome size can be easily computed.
{: .details}

## Generation of _k_-mer spectra with **Meryl**

Meryl will allow us to generate the *k*-mer profile by decomposing the sequencing data into *k*-length substrings, counting the occurrence of each *k*-mer and determining its frequency. The original version of Meryl was developed for the Celera Assembler. The current Meryl version comprises three main modules: one for generating *k*-mer databases, one for filtering and combining databases, and one for searching databases. *K*-mers are stored in lexicographical order in the database, similar to words in a dictionary ({% cite Rhie2020 %}).

> <comment-title><i>k</i>-mer size estimation</comment-title>
>
>  Given an estimated genome size (*G*) and a tolerable collision rate (*p*), an appropriate *k* can be computed as $$ k = \log_4\left(\frac{G(1-p)}{p}\right) $$ .
>
{: .comment}

In order to identify some key characteristics of the genome, we do genome profile analysis. To do this, we start by generating a histogram of the *k*-mer distribution in the raw reads (the *k*-mer spectrum). Then, GenomeScope creates a model fitting the spectrum that allows for estimation of genome characteristics. We work in parallel on each set of raw reads, creating a database of each file's *k*-mer counts, and then merge the databases of counts in order to build the histogram.

![Workflow of Kmer counting parallelization, described in the figure caption.](../../images/vgp_assembly/meryl_collections.png "K-mer counting is first done on the collection of FASTA files. Because these data are stored in a collection, a separate `count` job is launched for each FASTA file, thus parallelizing our work. After that, the collection of count datasets is merged into one dataset, which we can use to generate the histogram input needed for GenomeScope.")

> <hands-on-title>Generate <i>k</i>-mers count distribution</hands-on-title>
>
> 1. Run {% tool [Meryl](toolshed.g2.bx.psu.edu/repos/iuc/meryl/meryl/1.3+galaxy6) %} with the following parameters:
>    - *"Operation type selector"*: `Count operations`
>    - *"Count operations"*: `Count: count the occurrences of canonical k-mers`
>    - {% icon param-collection %} *"Input sequences"*: `HiFi_collection (trim)`
>    - *"k-mer size selector"*: `Set a k-mer size`
>    - "*k-mer size*": `31`
>
>    > <comment-title>Selection of <i>k</i>-mer size</comment-title>
>    >
>    > We used 31 as *k*-mer size, as this length has demonstrated to be sufficiently long that most *k*-mers are not repetitive and is short enough to be more robust to sequencing errors. For very large (haploid size > 10 Gb) and/or very repetitive genomes, larger *k*-mer length is recommended to increase the number of unique *k*-mers.
>    {: .comment}
>
> 2. Rename output as `meryldb`
>
> 3. Run {% tool [Meryl](toolshed.g2.bx.psu.edu/repos/iuc/meryl/meryl/1.3+galaxy6) %} again with the following parameters:
>    - *"Operation type selector"*: `Operations on sets of *k*-mers`
>    - *"Operations on sets of k-mers"*: `Union-sum: return k-mers that occur in any input, set the count to the sum of the counts`
>    - {% icon param-file %} *"Input meryldb"*: `Collection meryldb`
>
> 4. Rename it as `Merged meryldb`
>
> 5. Run {% tool [Meryl](toolshed.g2.bx.psu.edu/repos/iuc/meryl/meryl/1.3+galaxy6) %} for the third time with the following parameters:
>    - *"Operation type selector"*: `Generate histogram dataset`
>        - {% icon param-file %} *"Input meryldb"*: `Merged meryldb`
>
> 6. Finally, rename it as `meryldb histogram`.
>
{: .hands_on}

## Genome profiling with **GenomeScope2**

The next step is to infer the genome properties from the *k*-mer histogram generated by Meryl, for which we will use GenomeScope2. GenomeScope2 relies on a nonlinear least-squares optimization to fit a mixture of negative binomial distributions, generating estimated values for genome size, repetitiveness, and heterozygosity rates ({% cite RanalloBenavidez2020 %}).

> <hands-on-title>Estimate genome properties</hands-on-title>
>
> 1. Run {% tool [GenomeScope](toolshed.g2.bx.psu.edu/repos/iuc/genomescope/genomescope/2.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input histogram file"*: `meryldb histogram`
>    - *Ploidy for model to use*: `2`
>    - *"k-mer length used to calculate k-mer spectra"*: `31`
>    - In "*Output options*":
>        - Check `Summary of the analysis`
>    - In "*Advanced options*":
>        - *"Create testing.tsv file with model parameters"*: `Yes`
>
{: .hands_on}

Genomescope will generate six outputs:

- **Plots**:
    - Linear plot: *k*-mer spectra and fitted models: frequency (y-axis) versus coverage.
    - Log plot: logarithmic transformation of the previous plot.
    - Transformed linear plot: *k*-mer spectra and fitted models: frequency times coverage (y-axis) versus coverage (x-axis). This transformation increases the heights of higher-order peaks, overcoming the effect of high heterozygosity.
    - Transformed log plot: logarithmic transformation of the previous plot.
- **Model**: this file includes a detailed report about the model fitting.
- **Summary**: it includes the properties inferred from the model, such as genome haploid length and the percentage of heterozygosity. It is worth noting that the genome characteristics such as length, error percentage, etc., are based on the GenomeScope2 model, which is the black line in the plot. If the model (black line) does not fit your observed data (blue bars), then these estimated characteristics might be very off. In the case of this tutorial, the model is a good fit to our data, so we can trust the estimates.

Now, let's analyze the *k*-mer profiles, fitted models and estimated parameters shown below:

![Genomescope plot](../../images/vgp_assembly/genomescope_plot.png "GenomeScope2 31-mer profile. The first peak located at coverage 25✕ corresponds to the heterozygous peak. The second peak at coverage 50✕, corresponds to the homozygous peak. Estimate of the heterozygous portion is 0.576%. The plot also includes information about the inferred total genome length (len), genome unique length percent ('uniq'), overall heterozygosity rate ('ab'), mean *k*-mer coverage for heterozygous bases ('kcov'), read error rate ('err'), and average rate of read duplications ('dup'). It also reports the user-given parameters of *k*-mer size ('k') and ploidy ('p')."){:width="65%"}

This distribution is the result of the Poisson process underlying the generation of sequencing reads. As we can see, the *k*-mer profile follows a bimodal distribution, indicative of a diploid genome. The distribution is consistent with the theoretical diploid model (model fit > 93%). Low frequency *k*-mers are the result of sequencing errors. GenomeScope2 estimated a haploid genome size is around 11.7 Mb, a value reasonably close to *Saccharomyces* genome size. Additionally, it revealed that the variation across the genomic sequences is 0.576%. Some of these parameters can be used later on to parameterize running `purge_dups`. This is covered in the [**solo** contiging section](#solo_hic_switch) section of the tutorial.


# Assembly with **hifiasm**

Once we have finished the genome profiling stage, we can start the genome assembly with hifiasm,  a fast open-source *de novo* assembler specifically developed for PacBio HiFi reads. One of the key advantages of hifiasm is that it allows us to resolve near-identical, but not exactly identical, sequences, such as repeats and segmental duplications ({% cite Cheng2021 %}).

> <details-title>Hifiasm algorithm details</details-title>
>
> By default hifiasm performs three rounds of haplotype-aware error correction to correct sequence errors but keeping heterozygous alleles. A position on the target read to be corrected is considered informative if there are two different nucleotides at that position in the alignment, and each allele is supported by at least three reads.
>
> ![Hifiasm algorithm overview](../../images/vgp_assembly/hifiasm_algorithm.png "Hifiasm algorithm overview. Orange and blue bars represent the reads with heterozygous alleles carrying local phasing information, while green bars come from the homozygous regions without any heterozygous alleles.")
>
> Then, hifiasm builds a phased assembly string graph with local phasing information from the corrected reads. Only the reads coming from the same haplotype are connected in the phased assembly graph. After transitive reduction, a pair of heterozygous alleles is represented by a _bubble_ in the string graph. If there is no additional data, hifiasm arbitrarily selects one side of each bubble and outputs a primary assembly. In the case of a heterozygous genome, the primary assembly generated at this step may still retain haplotigs from the alternate allele.
>
{: .details}

The output of hifiasm will be [GFA](https://github.com/GFA-spec/GFA-spec) files. These differ from FASTA files in that they are a representation of the assembly graph instead of just linear sequences, so the {GFA} contains information about sequences, nodes, and edges (*i.e.*, overlaps). This output preserves the most information about how the reads assemble in graph space, and is useful to visualize in tools such as Bandage; however, our QV tools will expect FASTA files, so we will cover the GFA to FASTA conversion step later.

## `hifiasm` assembly modes

Hifiasm can be run in multiple modes depending on data availability

### **Solo** mode

**Solo**: generates a pseudohaplotype assembly, resulting in a primary & an alternate assembly.
- _Input: PacBio HiFi reads_
- _Output: scaffolded primary assembly, and alternate contigs_
![Diagram for hifiasm solo mode.](../../images/vgp_assembly/hifiasm_solo_schematic.png "The <b>solo</b> pipeline creates <b>primary</b> and <b>alternate</b> contigs, which then typically undergo purging with purge_dups to reconcile the haplotypes. During the purging process, haplotigs are removed from the primary assembly and added to the alternate assembly, which is then purged to generate the final alternate set of contigs. The purged primary contigs are then carried through scaffolding with Bionano and/or Hi-C data, resulting in one final draft primary assembly to be sent to manual curation.")

### **Hi-C** phased mode

**Hi-C-phased**: generates a hap1 assembly and a hap2 assembly, which are phased using the {Hi-C} reads from the same individual.
- _Input: PacBio HiFi & Illumina HiC reads_
- _Output: scaffolded hap1 assembly, and scaffolded hap2 assembly (assuming you run the scaffolding on **both** haplotypes)_
![Diagram for hifiasm hic mode.](../../images/vgp_assembly/hifiasm_hic_schematic.png "The <b>Hi-C-phased</b> mode produces <b>hap1</b> and <b>hap2</b> contigs, which have been phased using the HiC information as described in {% cite Cheng2021 %}. Typically, these assemblies do not need to undergo purging, but you should always look at your assemblies' QC to make sure. These contigs are then scaffolded <i>separately</i> using Bionano and/or Hi-C workflows, resulting in two scaffolded assemblies.")

### **Trio** mode

**Trio**: generates a maternal assembly and a paternal assembly, which are phased using reads from the parents.
- _Input: PacBio HiFi reads from child, Illumina reads from both parents._
- _Output: scaffolded maternal assembly, and scaffolded paternal assembly (assuming you run the scaffolding on **both** haplotypes)_
![Diagram for hifiasm trio mode.](../../images/vgp_assembly/hifiasm_trio_schematic.png "The <b>trio</b> mode produces <b>maternal</b> and <b>paternal</b> contigs, which have been phased using paternal short read data. Typically, these assemblies do not need to undergo purging, but you should always look at your assemblies' QC to make sure. These contigs are then scaffolded <i>separately</i> using Bionano and/or Hi-C workflows, resulting in two scaffolded assemblies.")

No matter which way you run hifiasm, you will have to evaluate the assemblies' {QC} to ensure your genome is in good shape. The VGP pipeline features several reference-free ways of evaluating assembly quality, all of which are automatically generated with our workflows; however, we will run them manually in this tutorial so we can familiarize ourselves with how each QC metric captures a different aspect of assembly quality. If you are interested in running the workflows with automatic QC generation, please see our [corresponding workflow tutorial]({% link topics/assembly/tutorials/vgp_workflow_training/tutorial.md %}) .

## Assembly evaluation

We use several tools for assessing various aspects of assembly quality:

- **gfastats**: manipulation & evaluation of assembly graphs and FASTA files, particularly used for summary statistics (*e.g.*, contig count, N50, NG50, etc.) ({% cite Formenti2022 %}).
![Schematic of N50 calculation.](../../images/vgp_assembly/n50schematic.jpg "<b>N50</b> is a commonly reported statistic used to represent genome contiguity. N50 is calculated by sorting contigs according to their lengths, and then taking the halfway point of the total genome length. The size of the contig at that halfway point is the N50 value. In the pictured example, the total genome length is 400 bp, so the N50 value is 60 because the contig at the halfway point is 60 bp long. N50 can be interpreted as the value where >50% of an assembly's contigs are at that value or higher. Image adapted from <a href='https://www.molecularecologist.com/2017/03/29/whats-n50/'>Elin Videvall at The Molecular Ecologist</a>.")
- **{BUSCO}**: assesses completeness of a genome from an evolutionarily informed functional point of view. BUSCO genes are genes that are expected to be present at single-copy in one haplotype for a certain clade, so their presence, absence, or duplication can inform scientists about if an assembly is likely missing important regions, or if it has multiple copies of them, which can indicate a need for purging ({% cite Simo2015 %}).
- **Merqury**: reference-free assessment of assembly completeness and phasing based on *k*-mers. Merqury compares *k*-mers in the reads to the *k*-mers found in the assemblies, as well as the {CN} of each *k*-mer in the assemblies ({% cite Rhie_merqury %}).

> <comment-title>How do I pick which assembly trajectory to use?</comment-title>
> The ideal scenario would be a trio assembly, where you can use parental data as a ground truth for phasing the haplotypes in the child. Unfortunately, attaining parental samples is difficult and often impossible when studying wild-caught organisms. When parental data is absent, the VGP recommends assembling using {Hi-C} phasing, if possible. This requires the Hi-C data to be derived from the same individual as the HiFi data. If this is not possible, then you cannot use the Hi-C data to phase the contigs, but you can still use it for scaffolding the primary assembly. Refer to the following decision tree for a visual representation of this logic. 
>
> ![Decision tree for picking workflow trajectory when one does not have parental data.](../../images/vgp_assembly/WF3vsWF4_decisiontree.png "If the HiFi and Hi-C data *do not* come from the same individual, then you can use WF3. After that, if you need to purge the primary, then run WF6 and then scaffolding. If the primary does not need purging, you can continue straight to scaffolding the primary. If the HiFi and Hi-C data *do* come from the same individual, then you can use WF4 to obtain hap1 and hap2 assemblies. If either of the assemblies need to be purged, then you can use WF6B to purge an individual haplotype. Then you can scaffold the assemblies separately. Accordingly, if neither need to be purged, then you can proceed to just scaffolding them separately.")
{: .comment}

<div id="solo_hic_switch">
<!-- For use as an anchor -->
</div>

{% include _includes/cyoa-choices.html option1="hic" option2="solo" default="hic"
       text="Use the following buttons to switch between contigging approaches. If you are assembling with only HiFi reads for an individual, then click <b><i>solo</i></b>. If you have HiC reads for the same indiviudal, then click <b><i>hic</i></b>. <b>NOTE: If you want to learn more about purging, then <u>please check out the <i>solo</i> tutorial for details on purging false duplications.</u></b>" %}

<div class = "hic" markdown="1">

<!---- BEGINNING OF HiC SECTION --->
<!---- BEGINNING OF HiC SECTION --->
<!---- BEGINNING OF HiC SECTION --->
<!---- BEGINNING OF HiC SECTION --->
<!---- BEGINNING OF HiC SECTION --->

## HiC-phased assembly with **hifiasm**

If you have the {Hi-C} data for the individual you are assembling with {HiFi} reads, then you can use that information to phase the contigs.

> <hands-on-title>Hi-C-phased assembly with <b>hifiasm</b></hands-on-title>
> 1. Run {% tool [Hifiasm](toolshed.g2.bx.psu.edu/repos/bgruening/hifiasm/hifiasm/0.19.8+galaxy0) %} with the following parameters:
>    - *"Assembly mode"*: `Standard`
>    - {% icon param-file %} *"Input reads"*: `HiFi_collection (trimmed)` (output of **Cutadapt** {% icon tool %})
>    - In *"Options for Hi-C-partition*" select `Specify`
>        - *"Hi-C R1 reads"*: `Hi-C_dataset_F`
>        - *"Hi-C R2 reads"*: `Hi-C_dataset_R`
>
> 2. After the tool has finished running, rename its outputs as follows:
>    - Rename the `Hi-C hap1 balanced contig graph` as `Hap1 contigs graph` and add a `#hap1` tag
>    - Rename the `Hi-C hap2 balanced contig graph` as `Hap2 contigs graph` and  add a `#hap2` tag
>
{: .hands_on}

We have obtained the fully phased contig graphs (as {GFA} files) of hap1 and hap2, but these must be converted to FASTA format for subsequent steps. We will use a tool developed from the VGP: [`gfastats`](https://github.com/vgl-hub/gfastats). `gfastats` is a tool suite that allows for manipulation and evaluation of FASTA and GFA files, but in this instance we will use it to convert our GFAs to FASTA files. Later on, we will use it to generate standard summary statistics for our assemblies.

> <hands-on-title>GFA to FASTA conversion for hifiasm Hi-C assembly</hands-on-title>
>
> 1. Run {% tool [gfastats](toolshed.g2.bx.psu.edu/repos/bgruening/gfastats/gfastats/1.3.6+galaxy0) %} with the following parameters:
>    - {% icon param-files %} *"Input GFA file"*: select `Hap1 contigs graph` and the `Hap2 contigs graph` datasets
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    - *"Tool mode"*: `Genome assembly manipulation`
>    - *"Output format"*: `FASTA`
>    - *"Generates the initial set of paths*": toggle to `yes`
>
> 2. Rename the outputs as `Hap1 contigs FASTA` and `Hap2 contigs FASTA`
>
{: .hands_on}

> <comment-title>Summary statistics</comment-title>
>
> gfastats will provide us with the following statistics:
>
> - **No. of contigs**: The total number of contigs in the assembly.
> - **Largest contig**: The length of the largest contig in the assembly.
> - **Total length**: The total number of bases in the assembly.
> - **Nx**: The largest contig length, *L*, such that contigs of length > *L* account for at least *x*% of the bases of the assembly.
> - **NGx**: The contig length such that using equal or longer length contigs produces *x*% of the length of the reference genome, rather than *x*% of the assembly length.
> - **GC content**: the percentage of nitrogenous bases which are either guanine or cytosine.
>
{: .comment}

Let's use gfastats to get a basic idea of what our assembly looks like. We'll run gfastats on the {GFA} files because gfastats can report graph-specific statistics as well. After generating the stats, we'll be doing some text manipulations to get the stats side-by-side in a pretty fashion.

> <hands-on-title>Assembly evaluation with gfastats</hands-on-title>
>
> 1. Run assembly statistics generation with {% tool [gfastats](toolshed.g2.bx.psu.edu/repos/bgruening/gfastats/gfastats/1.3.6+galaxy0) %} using the following parameters:
>
>    - {% icon param-files %} *"Input file"*: select `Hap1 contigs graph` and the `Hap2 contigs graph` datasets
>    - *"Tool mode": `Summary statistics generation`
>    - *"Expected genome size"*: `11747160` (remember we calculated this value [earlier using `GenomeScope2`](#genome-profiling-with-genomescope2). It is contained within `GenomeScope2` **Summary** output that should be in your history!)
>    - *"Thousands separator in output"*: Set to "No"
>
>
> 2. Rename outputs of `gfastats` step to as `Hap1 stats` and `Hap2 stats`
>
>    This would generate summary files that look like this (only the first six rows are shown):
>    
>    ```
>    Expected genome size    11747160
>    # scaffolds                    0
>    Total scaffold length          0
>    Average scaffold length      nan
>    Scaffold N50                   0
>    Scaffold auN                0.00
>    ```
>    
>    Because we ran `gfastats` on hap1 and hap2 outputs of `hifiasm` we need to join the two outputs together for easier interpretation:
>
> 3. Run {% tool [Column join](toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3) %} with the following parameters:
>    - {% icon param-files %} *"Input file"*: select `Hap1 stats` and the `Hap2 stats` datasets. Keep all other settings as they are.
>
> 4. Rename the output as `gfastats on hap1 and hap2 (full)`
>
>    This would generate a joined summary file that looks like this (only the first five rows are shown):
>    
>    ```
>    # gaps               0  0
>    # gaps in scaffolds  0  0
>    # paths              0  0
>    # segments          17 16
>    ```
>    
>    Now let's extract only relevant information by excluding all lines containing the word `scaffold` since there ar


topic: assembly
tutorial: largegenome
title: Large genome assembly and polishing
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["assembly", "polishing", "nanopore", "plants", "biodiversity"]
content_hash: 3478f12d8a54651e5f820eb09d516fcf
last_modified: 2026-02-26T21:12:14.477155
zenodo_link: https://doi.org/10.5281/zenodo.7055935
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/assembly/tutorials/largegenome/tutorial.html

content:
*Note: We recommend running this tutorial on either the Galaxy Europe or Galaxy Australia servers. Other servers (such as Galaxy main) have not yet been configured fully for all the tools in this analysis.*

## What is genome assembly?

A genome is a representation of the set of DNA in an organism, such as the set of chromosomes. When the DNA is extracted from the sample, it is broken up into fragments much smaller than the lengths of DNA in the chromosomes. After being sequenced, these fragments (and their copies) are called sequencing reads. To assemble the genome, we need to join the reads back into, ideally, chromosome-sized lengths.

!["Image showing the concept of a genome assembly: a sample (a tree), with an arrow going to sequencing reads, with an arrow going to a genome (a set of diploid chromosomes)"](images/sampletogenome.png){: width="800"}

### Assembly challenges

In reality, we rarely get chromosome-length assemblies, due to many challenges. Here are some examples of particular challenges in a diploid genome assembly:

!["Image showing potential assembly challenges: a set of diploid chromosomes with descriptions of different problems which are: a contig may be a consensus from both chromosomes; telomeres and centromeres are difficult to assemble; contigs may be assembled separately as a copy in both chromosomes but it is unclear which parental chromosome they are from; some contigs are short or missing; some contigs are not placed in the assembly."](images/challenges.png){: width="800"}


Even though most assemblies are not chromosome-length, the assembly in contigs is still valuable for many research questions. Lengths of assembled contigs are increasing as sequencing technology and assembly tools improve.

### Can I use my own data with these workflows?

This tutorial has been tested on real-sized data sets and should work with your own data. However, there will most likely be some modifications required to tools and settings. Furthermore, as most species have never had their genome sequenced, it is not possible to guarantee existing workflows are optimal for new data.

It is most likely that any new genome assembly will have its own set of required workflow and analysis customisations to account for things such as ploidy and repeats. Usually, an assembly workflow will need testing and customising, in concert with reading the biological domain literature.

Before using these workflows on real datasets, we recommend completing all the relevant GTN tutorials about Data QC and Asssembly to give extra familiarity with the concepts. Then, we suggest creating subsampled data sets from your full-sized data, for testing. For best results, it is recommended to test each tool separately, and consider the parameters and their relevance and suitability for your own research questions.

> <agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Analysis workflow

In this tutorial, we will follow these steps:

!["An image showing the steps in the tutorial in a stylized form: quality control and kmer counting, represented by bold lines as sequencing reads; trimming and filtering reads, represented by truncated reads; assembly, represented by joined reads into one long piece; polishing, represented by corrections (ticks) in the assembly; the finished genome assembly, represented by a set of diploid chromosomes; quality assessment, represented by ticks and crosses at different chromosome locations."](images/steps.png){: width="800"}

* Each of these steps is described in a section in this tutorial.
* For each step, we will run a workflow.
* We will stay in the same Galaxy history throughout.

## How to run a workflow in Galaxy

* All of these workflows have been uploaded to [workflowhub.eu](https://workflowhub.eu/) and have been tagged with "Large-genome-assembly". There, each workflow is accompanied by an image of the workflow canvas and the tool connections.
* The workflows are also linked to this tutorial - see above and import them into your own Galaxy Account.
* If you are using Galaxy Australia, the workflows are published with the tag "lg-wf".
* A note: as of September 2022, these workflows have been tested on Galaxy Australia and we are now in the process of testing them on Galaxy Europe and Galaxy Main.

Here is a list of the workflows:

|Tutorial section | Workflow name|
|-----------------|--------------|
|Quality Control | Data QC|
|Kmer counting | kmer counting - meryl|
|Trim and filter reads | Trim and filter reads - fastp|
|Assembly | Assembly with Flye |
|Polish assembly | Assembly polishing |
|Assess quality | Assess genome quality|

> <tip-title>Importing the workflows from WorkflowHub</tip-title>
>
> [WorkflowHub](https://workflowhub.eu/) is a workflow management system which allows workflows to be FAIR (Findable, Accessible, Interoperable, and Reusable), citable, have managed metadata profiles, and be openly available for review and analytics.
>
> {% snippet faqs/galaxy/workflows_import_search.md box_type="none" trs_server="workflowhub.eu" search_query='"large genome assembly"' %}
>
{: .tip}

After importing the workflows into your account:

* Look at your of Workflows. (Galaxy top panel: Workflow)
* For the workflow you want to run, go to the right hand side and see the arrow button (a triangle), click
* This brings up the workflow in the centre Galaxy panel
* Click "Expand to full workflow form"
* For "Send results to a new history", leave it as "No".
* Each time you run a workflow, you need to specify the input data set (or sets). Galaxy will try to guess which file this is, but change if required using the drop-down arrow.
* At the top right, click "Run Workflow".
* The result files will appear at the top of your current history

Each workflow will be discussed in a separate section.

# Upload data

Let's start with uploading the data.

## What sequence data are we using in the tutorial?

* The data sets for genome projects can be very large and tools can take some time to run. It is a good idea to test that your planned tools and workflows will work on smaller-sized test data sets, as it is much quicker to find out about any problems.
* In this tutorial we will use a subset of real sequencing data from a plant genome, the snow gum, *Eucalyptus pauciflora*, from a genome project described in this paper: {% cite Wang2020 %}. Data is hosted at NCBI BioProject number: PRJNA450887.

## How has this data subset been prepared?

* From NCBI, three read files were imported into Galaxy for this tutorial: nanopore reads (SRR7153076), and paired Illumina reads (SRR7153045).
* These were randomly subsampled to 10% of the original file size.
* Plant genomes may contain an excess of reads from the chloroplast genome (of which there are many copies per cell). To ensure our test data sets are not swamped from excessive chloroplast-genome reads, reads that mapped to a set of known chloroplast gene sequences were discarded.

We are also using a reference genome *Arabidopsis thaliana* for a later comparison step - file TAIR10_chr_all.fas downloaded from [The Arabidopsis Information Resource](https://www.arabidopsis.org/download/index-auto.jsp?dir=%2Fdownload_files%2FGenes%2FTAIR10_genome_release%2FTAIR10_chromosome_files).

## Import the data

> <hands-on-title>Import the data</hands-on-title>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import from [Zenodo](https://zenodo.org/record/7055935) or a data library (ask your instructor):
>
>    - 2 FASTQ files with illumina reads: these files have a R1 or R2 in the name
>    - 1 FASTQ file with nanopore reads: this file has nano in the name
>    - 1 FASTQ file for the genome Arabidopsis
>
>    ```
>    https://zenodo.org/record/7055935/files/Eucalyptus_subsample_ten_pc_SRR7153045_R1.fastq.gz
>    https://zenodo.org/record/7055935/files/Eucalyptus_subsample_ten_pc_SRR7153045_R2.fastq.gz
>    https://zenodo.org/record/7055935/files/Eucalyptus_subsample_ten_pc_nano_SRR7153076.fastq.gz
>    https://zenodo.org/record/7055935/files/Ref_genome_Arabidopsis.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check that the datatypes for the three files of sequencing reads are `fastq.gz`, not `fastqsanger.gz` and change datatype if needed.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

* This tutorial uses these input files and gives some examples from the results.
* It is likely that your results will differ slightly (e.g. number of bases in the genome assembly). This is common, because many tools start from different random seeds.
* Also, tool versions are being constantly updated.  Newer versions may be available since this tutorial was written and could give slightly different results.

# Check read quality

Let's look at how many reads we have and their quality scores using the Data QC workflow.

## Data QC workflow

What it does: Reports statistics from sequencing reads

Inputs:
* long reads (fastq.gz format)
* short reads (R1 and R2) (fastq.gz format)

Outputs:
* For long reads: a nanoplot report (the HTML report summarizes all the information).
* For short reads: a MultiQC report

Tools used:
* Nanoplot
* FastQC
* MultiQC

Workflow steps:
* Long reads are analysed by Nanoplot.
* Short reads (R1 and R2) are analysed by FastQC
* the resulting reports are processed by MultiQC

Options:
* See the tool settings options at runtime and change as required.
* Alternative tool option: fastp

> <hands-on-title>Run the Data QC workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Data_QC.ga" title="Galaxy Workflow Data QC" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Data QC** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: long reads"*: the nanopore fastq.gz file
>    - {% icon param-file %} *"2: Input file: Illumina reads R1"*: the illumina R1 fastq.gz file
>    - {% icon param-file %} *"3: Input file: Illumina reads R2"*: the illumina R2 fastq.gz file
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

## Data QC results

> <question-title></question-title>
>
> 1. What are the results from the two output files? Are the reads long enough and of high enough quality for our downstream analyses?  Will reads need any trimming or filtering?
>
> > <solution-title></solution-title>
> >
> > 1. Common things to check are average read length, average quality, and whether quality varies by position in the reads.
> > * Look at the plot for MultiQC for the Sequence Quality Histograms. This shows how the read quality of Illumina reads (y axis) varies according to base position (x axis).You may see for Illumina reads that there is some drop-off in quality towards the end of the reads, which may benefit from trimming.
> > * Look at the plot from Nanoplot for "Read lengths vs Average read quality". The nanopore reads have a mean read quality of 9.0. Depending on the size of our input read sets and the research question, we may filter out reads below a certain average quality. If we had a lot of reads, we may be able to set a higher threshold for filtering according to read quality.
> {: .solution}
{: .question}

> <comment-title>Additional information about the tools</comment-title>
> More about interpreting nanoplot plots:
> * [Nanoplot on Github](https://github.com/wdecoster/NanoPlot)
> * [Example nanoplots](https://gigabaseorgigabyte.wordpress.com/2017/06/01/example-gallery-of-nanoplot/)
>
> More about FastQC results:
> * [FastQC webpage](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
> * [FastQC tips](https://timkahlke.github.io/LongRead_tutorials/QC_F.html)
{: .comment}

# Determine genome characteristics

To prepare for genome assembly you might want to know things about your genome such as size, ploidy level (how many sets of chromosomes) and heterozygosity (how variable the sequence is between homologous chromosomes). A relatively fast way to estimate these things is to count small fragments of the sequencing reads (called kmers).

A read broken into kmers:

!["An image of the a read broken into kmers: the read is a bold bar with the letters ABCDEF; each kmer is a smaller bar with three letters in it: the kmers are ABC, BCD, CDE, and DEF."](images/kmers.png){: width="650"}

> <comment-title>What is kmer counting?</comment-title>
>
> Kmer counting is usually done with high-accuracy short reads, not long reads which may have high error rates.  After counting how many times each kmer is seen in the reads, we can see what sorts of counts are common. For example, lots of kmers may have been found 24 or 25 times. A graph shows the number of different kmers (y axis) found at different counts, or depths (x axis).
>
> Many different kmers will be found the same number of times; e.g. X25. If kmer length approaches read length, this means the average depth of your sequencing is also ~X25, and there would be a peak in the graph at this position (smaller kmers = higher kmer depth). There may be smaller peaks of kmer counts at higher depths, e.g. X50 or X100, indicating repeats in the genome. There may be other smaller peaks of kmers found at half the average depth, indicating a diploid genome with a certain amount of difference between the homologous chromosomes - this is known as heterozygosity. Thus, the plot of how many different kmers are found at all the depths will help inform estimates of sequencing depth, ploidy level, heterozygosity, and genome size.
{: .comment}

## Kmer counting workflow

What it does: Estimates genome size and heterozygosity based on counts of kmers

Inputs: One set of short reads: e.g. R1.fq.gz

Outputs: GenomeScope graphs

Tools used:
* Meryl
* GenomeScope

Workflow steps:
* The tool meryl counts kmers in the input reads (k=21), then converts this into a histogram.
* GenomeScope: runs a model on the histogram; reports estimates. k-mer size set to 21.

Options:
* Use a different kmer counting tool. e.g. khmer. If so, for the settings, advanced parameters:
* k-mer size: 21 (as per [this recommendation](https://github.com/schatzlab/genomescope/issues/32)).
* n_tables: 4. tablesize: set at 8 billion (as per [this recommendation](https://khmer.readthedocs.io/en/v1.0/choosing-table-sizes.html)).
* Will also need to run some formatting steps to convert khmer output to a two-column matrix, for the  Genomscope. See [this workflow](https://usegalaxy.org.au/u/anna/w/kmer-counting-khmer).
* Note: khmer: to use both R1 and R2 read sets, khmer needs these paired reads in interleaved format.


> <hands-on-title>Run the Kmer counting workflow</hands-on-title>
>
> 1. **Import the Kmer counting workflow** into Galaxy:
>    - Copy the URL (e.g. via right-click) of [this workflow]({{ site.baseurl }}{{ page.dir }}workflows/Galaxy-Workflow-kmer_counting.ga) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-kmer_counting.ga" title="Kmer counting workflow" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Kmer counting** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: Illumina reads R1"*: the illumina R1 fastq.gz file
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}


## Kmer counting results

GenomeScope transformed linear plot:

!["An image of the Genomescope plot, described fully in the text following this image."](images/genomescope.png){: width="800"}

Here we can see a central peak - showing that most of the different kmers were found at counts of ~ 120. These are kmers from single-copy homozygous alleles. To the left, a smaller peak at around half the coverage, showing kmers from heterozygous alleles (note that this peak gets higher than the main peak when heterozygosity is only ~ 1.2%). To the right, another smaller peak showing kmers at higher coverage, from repeat regions. Information from these three peaks provide a haploid genome length estimate of ~240,000 bp (note this is test data so smaller than whole plant genome size).

The output Summary file shows more detail:
* Genome unique length: from single copy homozygous and heterozygous alleles (under the main and left peak).
* Genome repeat length: from repeat copies (under the graph to the right of the main peak).
* Genome haploid length: unique length + repeat length

More about kmer counting: See [this tutorial](https://bioinformatics.uconn.edu/genome-size-estimation-tutorial/#)

For more about Meryl see {% cite Rhie2020 %}.

For more about Genomescope see {% cite Vurture2017 %}(Note: the supplementary information is very informative).

# Trim and filter reads

Using information from Data QC and kmer counting, we may want to trim and/or filter reads. The settings for trimming and filtering depend on many things, including:
* your aim (accuracy; contiguity)
* your data: type, error rate, read depth, lengths, quality (average, variation by position)
* the ploidy and heterozygosity of your sample
* choice of assembly tool (e.g. it may automatically deal with adapters, low qualities, etc.)

Because of all these factors, few specific recommendations are made here, but the workflow is provided and can be customised. If you are unsure how to start, use your test data to try different settings and see the effect on the resulting size and quality of the reads, and the downstream assembly contigs. Newer assemblers are often configured to work well with long-read data and in some cases, read trimming/filtering for long reads may be unnecessary.

Trimming and filtering reads:

!["An image of the concepts of trimming and filtering reads, with reads represented by bold bars; trimming represented by crosses over various locations in each read."](images/trimfilterreads.png){: width="800"}

## Trimming and filtering workflow


What it does: Trims and filters raw sequence reads according to specified settings.

Inputs:
* Long reads (format fastq)
* Short reads R1 and R2 (format fastq)

Outputs: Trimmed and filtered reads:
* fastp_filtered_long_reads.fastq.gz (But note: no trimming or filtering is on by default)
* fastp_filtered_R1.fastq.gz
* fastp_filtered_R2.fastq.gz

Tools used:  fastp (Note. The latest version (0.20.1) of fastp has an issue displaying plot results. Using version 0.19.5 here instead until this is rectified).

Input parameters: None required, but recommend removing the long reads from the workflow if not using any trimming/filtering settings.

Workflow parameter settings:
* Long reads: fastp settings:
  * These settings have been changed from the defaults (so that all filtering and trimming settings are now disabled).
  * Adapter trimming options: Disable adapter trimming: yes.
  * Filter options: Quality filtering options: Disable quality filtering: yes.
  * Filter options: Length filtering options: Disable length filtering: yes.
  * Read modification options: PolyG tail trimming: Disable.
  * Output options: output JSON report: yes
* Short reads: fastp settings:
  * adapter trimming (default setting: adapters are auto-detected)
  * quality filtering (default: phred quality 15), unqualified bases limit (default = 40%), number of Ns allowed in a read (default = 5)
  * length filtering (default length = min 15)
  * polyG tail trimming (default = on for NextSeq/NovaSeq data which is auto detected)
* Output options: output JSON report: yes

Options:
* Change any settings in fastp for any of the input reads.
* Adapter trimming: input the actual adapter sequences. (Alternative tool for long read adapter trimming: Porechop.)
* Trimming n bases from ends of reads if quality less than value x  (Alternative tool for trimming long reads: NanoFilt.)
* Discard post-trimmed reads if length is < x (e.g. for long reads, 1000 bp)
* Example filtering/trimming that you might do on long reads: remove adapters (can also be done with Porechop), trim bases from ends of the reads with low quality (can also be done with NanoFilt), after this can keep only reads of length x (e.g. 1000 bp)
* If not running any trimming/filtering on nanopore reads, could delete this step from the workflow entirely.


> <hands-on-title>Run the Trim and Filter Reads workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Trim_and_filter_reads.ga" title="Trim and Filter reads" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Trim and Filter Reads** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: long reads"*: the nanopore fastq.gz file
>    - {% icon param-file %} *"2: Input file: Illumina reads R1"*: the illumina R1 fastq.gz file
>    - {% icon param-file %} *"3: Input file: Illumina reads R2"*: the illumina R2 fastq.gz file
>    - *Note: the workflow parameter settings described above are already set, and don't need to be changed when running the workflow, unless you wish to.*
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

## Trim and filter reads: results

There are two fastp reports - one for the illumina reads and one for the nanopore reads. We have only processed illumina reads in this example. Look at the fastp illumina report. (Note: the title in the report refers to only one of the input read sets but the report is for both read sets. This is a known issue under investigation.)

Filtering results from fastp on short reads:

!["Image of a table of results from fastp, showing the percentage of reads that were low quality, had too many Ns, or too short."](images/fastp.png){: width="800"}

Here we can see that less than 0.5 % of the reads were discarded based on quality. If our read set had high enough coverage for downstream analyses, we might choose to apply a stricter quality filter.

## Summary of read data for genome assembly

> <question-title></question-title>
> 1. How many reads do we have now for our genome assembly? Is the read coverage high enough?
>
> > <solution-title></solution-title>
> > * Genome size: From kmer counting, the estimated genome size is ~ 240,000 bp  (this is only subsampled data; full data would likely suggest a size of 0.5 - 1 Gbp for a typical plant genome)
> > * Genome coverage (or depth):  total base pairs in the reads / base pairs in genome
> > * Short reads: From the fastp report after trimming and filtering short reads, there are 3.2 million reads, comprising 482 million base pairs
> > * Short read coverage:  = X2008
> > * Long reads: From the fastp report of the long reads (although no filtering and trimming performed) there are 85 thousand reads, comprising 761 million base pairs.  From the nanoplot tool we ran in the Data QC section , we know that the mean read length is almost 9,000 base pairs, and the longest read is > 140,000 base pairs.
> > * Long read coverage:  = X3170
> > These coverages are very high but are ok to use with tutorial data. With a typical full data set, coverage would be more in the order of X40 to X200.
> >
> {: .solution}
{: .question}

# Genome Assembly

Genome assembly means joining the reads up to make contiguous sections of the genome. A simplified way to imagine this is overlapping all the different sequencing reads to make a single length or contig, ideally one for each original chromosome. The output is a set of contigs and a graph showing how contigs are connected.

Extreme simplification of genome assembly:

!["Image of the simplified process of genome assembly, from a tree, to reads (represented by bold bars), to an assembly (represented by one long bold bar), to contigs (various long bold bars) and to an assembly graph."](images/assembly.png){: width="800"}

Genome assembly algorithms use different approaches to work with the complexities of large sequencing read data sets, large genomes, different sequencing error rates, and computational resources. Many use graph-based algorithms.  For more about genome assembly algorithms see [these tutorials by Ben Langmead](https://langmead-lab.org/teaching-materials/).

## Which assembly tool and approach to use?

Here, we will use the assembly tool called Flye to assemble the long reads. This is fast and deals well with the high error rate. Then, we will polish (correct) the assembly using information from the long reads (in their unassembled state), as well as the more accurate short Illumina reads.

There are many other approaches and combinations of using short and long reads, and the polishing steps. For example, the long reads can be polished before assembly (with themselves, or with short reads). This may increase accuracy of the assembly, but it may also introduce errors if similar sequences are "corrected" into an artificial consensus. Long reads are usually used in the assembly, but it is possible to assemble short reads and then scaffold these into longer contigs using information from long reads.

For more about the differences between current assembly and polishing tools see {% cite Chen2021%} and {% cite McCartney2021 %}.

## Assembly with Flye workflow


What it does: Assembles long reads with the tool Flye

Inputs: long reads (may be raw, or filtered, and/or corrected); fastq.gz format

Outputs:
* Flye assembly fasta.
* Fasta stats on assembly.fasta
* Assembly graph image from Bandage
* Bar chart of contig sizes
* Quast reports of genome assembly

Tools used:
* Flye
* Fasta statistics
* Bandage
* Bar chart
* Quast

Input parameters: None required, but recommend setting assembly mode to match input sequence type

Workflow steps:
* Long reads are assembled with Flye, using default tool settings. Note: the default setting for read type ("mode") is nanopore raw. Change this at runtime if required.
* Statistics are computed from the assembly.fasta file output, using Fasta Statistics and Quast (is genome large: Yes; distinguish contigs with more that 50% unaligned bases: no)
* The graphical fragment assembly file is visualized with the tool Bandage.
* Assembly information sent to bar chart to visualize contig sizes

Options
* See other Flye options.
* Use a different assembler (in a different workflow).
* Bandage image options - change size (max size is 32767), labels - add (e.g. node lengths). You can also install Bandage on your own computer and download the "graphical fragment assembly" file to view in greater detail.


> <hands-on-title>Run the Assembly with Flye workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Assembly_with_Flye.ga" title="Assembly with Flye" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Assembly with Flye** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: long reads"*: the fastp filtered long reads fastq.gz file
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}



## Assembly results

The assembled contigs are in the "Flye assembly on data X (consensus)" (X is a number that will vary depending on where it sits in your history).
Open the Quast tabular report to see the assembly statistics:

!["Image of the Quast tabular report, showing various assembly statistics."](images/quast.png){: width="800"}

There are 153 contigs, largest is ~246,000 bp, and total length almost 10 million bp. This is a fair bit longer than the estimated genome size from kmer counting (which was ~240,000 bp), but the difference is likely mainly due to idiosyncrasies of using a subsampled data set. The read coverage was likely <1, causing many kmers to have frequency of <1 and be classed as errors, rather than contributing to the genome size estimate.

Open the Quast HTML report, then click on "View in Icarus contig browser".  This is a way to visualize the contigs and their sizes:

!["Image of the Icarus contig browser, showing the set of contigs arranged from larges to smallest, represented by rectangles of different widths."](images/icarus.png){: width="800"}

View the Bandage image of the assembly graph:

!["Image of the bandage graph showing one multi-part set of joined contigs, some smaller one- or two- piece contigs, and many single contigs."](images/bandagegraphpartial.png){: width="800"}

As this is a subsampled data set, it is not surprising that most of the contigs are unjoined. The joined contigs at the top left are likely to be part of the mitochondrial genome as these reads were probably over-represented in our subsampled data set.

### What about centromeres and telomeres?

Some genomic areas such as centromeres, telomeres, and ribosomal DNA arrays, are much harder to assemble. These are long stretches of very similar repeats. With improved sequencing accuracy, length, and technologies (particularly long-range scaffolding), these may soon be much easier to assemble. The latest human genome assembly has a good demonstration of the techniques used for this.   See {% cite Nurk2021 %}, and in particular, Figure 2: Bandage graphs of the human genome chromosomes, with the grey shading showing centromeric regions.

### What about haplotigs?

Although our sample may be diploid, with pairs of chromosomes, the resulting assembly is often a haploid (or "collapsed") assembly. This is not the sequence of one of the chromosomes, but a mix of the two.

Some assemblers will produce extra contigs called haplotigs. These are parts of the assembly from heterozygous regions (that is, the sequence is relatively different between the chromosome pair). There are tools to remove haplotigs from the assembly if that is preferred.

For more on differences between collapsed, primary/alternate and partially-phased assemblies, with a great visual representation: see [Heng Li's Github page](http://lh3.github.io/2021/04/17/concepts-in-phased-assemblies)

For more on the phased assemblies, particularly for diploids or polyploids, see {% cite Garg2021 %}.

# Assembly polishing

We will polish the assembly using both the long reads and short reads. This process aligns the reads to the assembly contigs, and makes corrections to the contigs where warranted.  For more, see {% cite Aury2021 %}, particularly for a discussion about polishing diploid genomes.

Assembly polishing:

!["Image of the concept of assembly polishing, showing reads mapped to the assembly contigs."](images/polish.png){: width="800"}

## Polishing workflow

Workflow structure: The workflow includes two subworkflows:
* Racon polish with long reads, x 4
* Racon polish with illumina reads, x2

What it does: Polishes (corrects) an assembly, using long reads (with the tools Racon and Medaka) and short reads (with the tool Racon). (Note: medaka is only for nanopore reads, not PacBio reads).

Inputs:
* assembly to be polished:  assembly.fasta
* long reads - the same set used in the assembly (e.g. may be raw or filtered) fastq.gz format
* short reads, R1 only, in fastq.gz format

Outputs:
* Racon+Medaka+Racon polished_assembly. fasta
* Fasta statistics after each polishing tool

Tools used:
* Minimap2
* Racon
* Fasta statistics
* Medaka

Input parameters: None required, but recommended to set the Medaka model correctly (default = r941_min_high_g360). See drop down list for options.

Workflow steps for Part 1, Polish with long reads: using Racon
* Long reads and assembly contigs => Racon polishing (subworkflow):
* minimap2 : long reads are mapped to assembly => overlaps.paf.
* overaps, long reads, assembly => Racon => polished assembly 1
* using polished assembly 1 as input; repeat minimap2 + racon => polished assembly 2
* using polished assembly 2 as input, repeat minimap2 + racon => polished assembly 3
* using polished assembly 3 as input, repeat minimap2 + racon => polished assembly 4
* Racon long-read polished assembly => Fasta statistics
* Note: The Racon tool panel can be a bit confusing and is under review for improvement. Presently it requires sequences (= long reads), overlaps (= the paf file created by minimap2), and target sequences (= the contigs to be polished) as per "usage" described at this link [https://github.com/isovic/racon/blob/master/README.md](https://github.com/isovic/racon/blob/master/README.md)
* Note: Racon: the default setting for "output unpolished target sequences?" is No. This has been changed to Yes for all Racon steps in these polishing workflows.  This means that even if no polishes are made in some contigs, they will be part of the output fasta file.
* Note: the contigs output by Racon have new tags in their headers. For more on this see [this issue](https://github.com/isovic/racon/issues/85).

Workflow steps for Part 2, Polish with long reads: using Medaka
* Racon polished assembly + long reads => medaka polishing X1 => medaka polished assembly
* Medaka polished assembly => Fasta statistics

Workflow steps for Part 3, Polish with short reads: using Racon
* Short reads and Medaka polished assembly =>Racon polish (subworkflow):
* minimap2: short reads (R1 only) are mapped to the assembly => overlaps.paf. Minimap2 setting is for short reads.
* overlaps + short reads + assembly => Racon => polished assembly 1
* using polished assembly 1 as input; repeat minimap2 + racon => polished assembly 2
* Racon short-read polished assembly => Fasta statistics

Options:
* Change settings for Racon long read polishing if using PacBio reads:  The default profile setting for Racon long read polishing: minimap2 read mapping is "Oxford Nanopore read to reference mapping", which is specified as an input parameter to the whole Assembly polishing workflow, as text: map-ont. If you are not using nanopore reads and/or need a different setting, change this input. To see the other available settings, open the minimap2 tool, find "Select a profile of preset options", and click on the drop down menu. For each described option, there is a short text in brackets at the end (e.g. map-pb). This is the text to enter into the assembly polishing workflow at runtime instead of the default (map-ont).
* Other options: change the number of polishes (in Racon and/or Medaka). There are ways to assess how much improvement in assembly quality has occurred per polishing round (for example, the number of corrections made; the change in Busco score - see section "Genome quality assessment" for more on Busco).
* Option: change polishing settings for any of these tools. Note: for Racon - these will have to be changed within those subworkflows first. Then, in the main workflow, update the

> <hands-on-title>Run the Assembly polishing workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Assembly_polishing.ga" title="Assembly polishing workflow" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Assembly polishing** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Assembly to be polished"*: the assembly fasta file
>    - {% icon param-file %} *"2: long reads"*: the nanopore reads, the same set that was used for the assembly
>    - {% icon param-file %} *"3: Illumina reads R1"*: the illumina R1 fastq.gz file
>    - No other settings need to be changed, as they are set as described above, but recommended to set the Medaka model correctly (default = r941_min_high_g360). See drop down list for options.
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}


## Polishing results

The polished assembly is the final Racon file.

Look at the Fasta Statistics output files for the status of the assemblies after each polishing tool. From these, we can see that some polishes decrease or increase the size of the assembly. The final size is approximately 300,000 bp shorter than the original Flye assembly. (These numbers may vary slightly even if the same input data is used).

# Genome quality assessment

The polished genome is in 145 contigs with a total length of ~ 9.6 million base pairs.  We have some idea of how these contigs may be joined (or not) from the Bandage assembly graph.

> <question-title></question-title>
>
> 1. How good is the assembly?
>
> > <solution-title></solution-title>
> >
> > 1.  One measure is the N50, a number that indicates how large the contigs are (although, have the contigs been joined correctly)? Another measure is to see if expected gene sequences are found in the assembly, using a tool called BUSCO. For a discussion on these and other methods, see {% cite Wang2020%}.
> >
> {: .solution}
{: .question}

Here, we will use the BUSCO tool to annotate the genome and then assess whether expected genes are found. Note: this is a brief annotation only, not the full genome annotation that would typically be done following genome assembly.

For more about Busco see {% cite Simo2015 %}.

Genome assessment:

!["Image of genome assessment, showing contigs as bold bars, with one contig annotated (various sections labelled by coloured boxes), and then compared to an expected annotation."](images/assess.png){: width="800"}

We will also map the assembled contigs to a known reference genome using the tool Quast, to see how they align. For more about quast see [the manual](http://quast.sourceforge.net/docs/manual.html).

For more about the Icarus browser see {% cite Mikheenko2016 %}.

## Genome assessment workflow

What it does:
* Assesses the quality of the genome assembly: generate some statistics and determine if expected genes are present
* Align contigs to a reference genome.

Inputs:
* polished assembly
* reference_genome.fasta (e.g. of a closely-related species, if available).

Outputs
* Busco table of genes found
* Quast HTML report, and link to Icarus contigs browser, showing contigs aligned to a reference genome

Tools used:
* Busco
* Quast

Input parameters: None required

Workflow steps:
* Polished assembly => Busco:
* First: predict genes in the assembly: using Metaeuk
* Second: compare the set of predicted genes to the set of expected genes in a particular lineage. Default setting for lineage: Eukaryota
* Polished assembly and a reference genome => Quast:
* For the tutorial we will use the Arabidopsis genome. This is not closely related to our Eucalyptus species but will give an idea of how to use Quast.
* Contigs/scaffolds file: polished assembly
* Type of assembly: Genome
* Use a reference genome: Yes
* Reference genome: Arabidopsis genome
* Is the genome large (> 100Mbp)? Yes. (Our test data set won't be, but this will still give us some results).
* All other settings as defaults, except second last setting: Distinguish contigs with more than 50% unaligned bases as a separate group of contigs?: change to No

Options:
* **Gene prediction:**
* Change tool used by Busco to predict genes in the assembly: instead of Metaeuk, use Augustus (Metaeuk is meant to be faster; unsure which is better).
* select: Use Augustus; Use another predefined species model; then choose from the drop down list.
* select from a database of trained species models. Link to lists of species models: [https://github.com/Gaius-Augustus/Augustus/tree/master/config/species](https://github.com/Gaius-Augustus/Augustus/tree/master/config/species)
* Note: if using Augustus: it may fail if the input assembly is too small (e.g. a test-size data assembly). It can't do the training part properly.
* **Compare genes found to other lineage:**
* Busco has databases of lineages and their expected genes. Option to change lineage. Not all lineages are available - there is a mix of broader and narrower lineages.
* link to list of lineages [https://busco.ezlab.org/list_of_lineages.html](https://busco.ezlab.org/list_of_lineages.html)
* link to groups in taxonomic hierarchies [https://busco.ezlab.org/frames/euka.htm](https://busco.ezlab.org/frames/euka.htm)
* For example,  if you have a plant species from Fabales, you could set that as the lineage.
* The narrower the taxonomic group, the more total genes are expected.


> <hands-on-title>Run the Assess Genome Quality workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Assess_genome_quality.ga" title="Assess Genome Quality" %}
>
>    - Click "Expand to full workflow form"
>
> 2. Run **Assess Genome Quality** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: Input file: Polished assembly"*: the polished assembly fasta file from the previous polishing workflow
>    - {% icon param-file %} *"2: Input file: Reference genome"*: Arabidopsis.fasta
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}


## Assessment results

The output is a set of Quast and Busco reports.

Busco: As this is a test dataset, the assembly is small (~ 10 million base pairs, rather than ~ 1 billion base pairs), and likely missing most of the real genome (and genes). Thus, for this test case, we would expect most of the genes not to be found, but we can view the Busco results to see an example of how it works. Open the Busco short summary file: only 1 complete BUSCO has been found, out of 255 expected. You can re-run the workflow and change the lineage to "Embryophyta" to see that more BUSCOs are found (8 out of 1614).

Open the Quast HTML report, and at the top of this, click on "View in Icarus contig browser".  This shows how our assembly contigs have mapped to the reference genome. For this reference genome there are 5 chromosomes and two organelles. As this reference genome species is not closely related, not many contigs have mapped well. But we can see that some of them match the organelles (which is expected, as these reads are likely to be overrepresented in the test data). For the nuclear genome, there are some matches to parts of chromosome 2 and 3.

!["Image of quast results showing contig lengths."](images/quast2.png){: width="650"}

Click on the "mitochondria" to see how the assembly contigs align to the reference mitochondrial genome. Click the -5x button to zoom out to the full length.

!["Image of mitochondrial contigs."](images/mitocontigs.png){: width="650"}

# Combining workflows

We can combine these galaxy workflows into a single workflow. (See [this GTN tutorial]({% link topics/galaxy-interface/tutorials/workflow-editor/tutorial.md %}) for more information.)

## Combined assembly and polishing workflow

> <hands-on-title>Run the Combined Assembly and Polishing workflow</hands-on-title>
> * Find this workflow, enter the correct input files, and run.
> * What it does:  A workflow for genome assembly, containing subworkflows:
>   * Data QC
>   * Kmer counting
>   * Trim and filter reads
>   * Assembly with Flye
>   * Assembly polishing
>   * Assess genome quality
> * Inputs
>   * long reads and short reads in fastq format
>   * reference genome for Quast
> * Outputs
>   * Data information - QC, kmers
>   * Filtered, trimmed reads
>   * Genome assembly, assembly graph, stats
>   * Polished assembly, stats
>   * Quality metrics - Busco, Quast
> * Tools used: Sum of tools in each of the subworkflows
> * Input parameters: None required
> * Workflow steps: For detail see each subworkflow
> * Options:
>   * Omit some steps - e.g. Data QC and kmer counting
>   * Replace a module with one using a different tool - e.g. change assembly tool
{: .hands_on}

# Summary

In this tutorial we have assembled sequencing reads into contigs, using tools and workflows in Galaxy. Although we have used test data, these tools and workflows should work on real-sized eukaryotic data sets although please see our recommendations in the Introduction section.

A summary of the workflow steps and the main tools used:

!["A graphical summary of the steps in the assembly: quality control, kmer counting, trimming/filtering reads, assembly, polishing and assessment. A composite of all the earlier images in the tutorial."](images/summary.png){: width="650"}

We hope this has been useful for both learning about genome assembly concepts and as a customisable example for your own assembly data.


topic: admin
tutorial: terraform
title: Deploying a compute cluster in OpenStack via Terraform
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/terraform/tutorial.html
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: []
tags_json: ["terraform", "deploying", "cloud"]
content_hash: bb6612e2474507861f0d1c2e2ffa4aa9
last_modified: 2026-02-26T21:12:14.165960
workflows_json: []
content:
# Overview


In this tutorial we will briefly cover what [Terraform](https://www.terraform.io) is and how you can leverage it for your needs. This will not make you an expert on Terraform but will give you the tools you need in order to maintain your cloud infrastructure as code.

This will be a very practical training with emphasis on looking at examples from modules and becoming self sufficient. This tutorial uses the OpenStack provider for Terraform. Other Cloud providers are available, but their invocation will be different from that which is described here.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Cattle vs. Pets

For those of you with more experience as System Administrators, you may have heard the term "cattle vs. pets."

Pets
:    These servers are managed by hand, with admins SSHing in to make changes directly on the machine, and no log of this. If a "pet" dies, everyone is sad.

Cattle
:    These servers are managed in bulk, you no longer own a beloved pet but hundreds of identical ones. You stop caring if an individual server lives or dies as you have many identical ones ready to take its place, and you can easily replace them.


# What is Terraform?

Terraform is a tool you can use to "sync" your infrastructure (VMs in Clouds, DNS records, etc.) with code you have written in configuration files. This is known as **infrastructure as code**.
Knowing that your infrastructure is exactly what you expect it to be can simplify your operations significantly. You can have confidence that if anything changes, any images crash or are accidentally deleted, that you can immediately re-build your infrastructure.

UseGalaxy.eu used Terraform to migrate easily between clouds, when our old cloud shut down and our new cloud was launched. We simply updated the credentials for connecting to the cloud, and ran Terraform. It recognised that none of our existing resources were there and recreated all of them. This made life incredibly easy for us, we knew that our infrastructure would be exactly correct relative to what it looked like in the previous cloud.

## What can be managed

Terraform supports [many providers](https://www.terraform.io/docs/providers/index.html), allowing you to easily manage resources no matter where they are located. Primarily this consists of resources like virtual machines and DNS records.

If you have VMs you are managing, whether for individual projects or for large scale infrastructure like UseGalaxy.eu, your work or research can be simpler and more reproducible if you are doing it with automation like Terraform.

## What should be managed

Support for certain databases and other software is available, but whether or not this is a good idea depends heavily on your workflow. UseGalaxy.eu found that managing databases and database users was not a good fit with our workflow. We launch VMs with terraform and then provision them with ansible. To then have a second step where we re-connect and provision the database with terraform was an awkward workflow, so we mostly let Ansible manage these resources.

Some groups use Ansible or Bash scripts in order to launch VMs. This can be a better workflow for certain cases. Launching VMs that manage themselves or are automatically scaled by some external process is not a good fit. Terraform expects the resource to be there the next time, with the same ID, with the same values, and will alert you if it isn't.

# Managing a Single VM

We will start small, by managing a single VM in our cloud account. Make sure you have your OpenStack credentials available.

> <tip-title>Obtaining credentials from OpenStack dashboard</tip-title>
>
> You can download the environment file with the credentials from the OpenStack dashboard.
>
> - Log in to the OpenStack dashboard, choose the project for which you want to download the OpenStack RC file, and click "Access & Security".
> - Click "Download OpenStack RC File" and save the file.
{: .tip}

<!-- TODO(hxr): add example DO config? -->

## Keypair

Terraform reads all files with the extension `.tf` in your current directory. Resources can be in a single file, or organised across several different files. We had the best experience by separating out logical groups of resources:

- DNS entries in a single file
- Security groups in separate `.tf` files
- Keypairs in a single file
- Servers/Instances in individual files.

> <hands-on-title>Setting up Terraform</hands-on-title>
>
> 1. [Install Terraform](https://www.terraform.io/downloads.html)
>
> 2. Create a new directory and go into it.
>
> 3. Create a `providers.tf` file with the following contents:
>
>    ```ini
>    provider "openstack" {}
>    ```
>
>    This specifies the configuration for the OpenStack plugin. You can either specify the configuration in the plugin, or it will automatically load the values from the normal OpenStack environment variable names.
>
> 4. Run `terraform init`
>
>    > <question-title></question-title>
>    >
>    > What did the output look like?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > If this is not the first time you've run it, you will see Terraform download any missing providers:
>    > >
>    > > ```
>    > > Initializing provider plugins...
>    > > - Checking for available provider plugins on https://releases.hashicorp.com...
>    > > - Downloading plugin for provider "openstack" (1.10.0)...
>    > >
>    > > The following providers do not have any version constraints in configuration,
>    > > so the latest version was installed.
>    > >
>    > > To prevent automatic upgrades to new major versions that may contain breaking
>    > > changes, it is recommended to add version = "..." constraints to the
>    > > corresponding provider blocks in configuration, with the constraint strings
>    > > suggested below.
>    > >
>    > > * provider.openstack: version = "~> 1.10"
>    > >
>    > > Terraform has been successfully initialized!
>    > >
>    > > You may now begin working with Terraform. Try running "terraform plan" to see
>    > > any changes that are required for your infrastructure. All Terraform commands
>    > > should now work.
>    > >
>    > > If you ever set or change modules or backend configuration for Terraform,
>    > > rerun this command to reinitialize your working directory. If you forget, other
>    > > commands will detect it and remind you to do so if necessary.
>    > > ```
>    > >
>    > {: .solution }
>    {: .question}
{:.hands_on}

This is the minimum to get started with Terraform, defining which providers we will use and any parameters for them. We have two options now:

1. We can load the OpenStack credentials as environment variables
2. We can code the OpenStack credentials in the `providers.tf` file

We recommend the first option, as often Terraform plans are [made publicly available](https://github.com/usegalaxy-eu/infrastructure) in order to collaborate on them, and this prevents accidentally committing your credentials to the git repository where.

We will start by managing your SSH keypair for the cloud as this is an easy thing to add.

> <comment-title>Your Operating System</comment-title>
>
> If you are on Windows and do not know your public key, please skip to the "Adding an Instance" section, as you probably do not have the tools installed to do this, and we cannot document how to do it. Instead you can simply reference the key by name later. We will point out this location.
>
{: .comment}

> <hands-on-title>Keypairs</hands-on-title>
>
> 1. Find the public key you will use for connecting to the your new VM. It is usually known as `id_rsa.pub`
>
>    > <comment-title>No public key</comment-title>
>    > If you can find the private key file (possibly a `cloud.pem` file you downloaded earlier from OpenStack), then you can find the public key by running the command:
>    >
>    > ```shell
>    > $ ssh-keygen -y -f /path/to/key.pem
>    > ```
>    {: .comment}
>
> 2. Create a new file, `main.tf` with the following structure. In `public_key`, write the complete value of your public key that you found in the first step.
>
>    ```ini
>    resource "openstack_compute_keypair_v2" "my-cloud-key" {
>      name       = "my-key"
>      public_key = "ssh-rsa AAAAB3Nz..."
>    }
>    ```
>
> 3. Run `terraform plan`
>
>    > <question-title></question-title>
>    >
>    > What does the output look like? Did it execute successfully?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > If you have not sourced your OpenStack credentials file, you will see something like the following:
>    > >
>    > > ```
>    > > Error: Error running plan: 1 error(s) occurred:
>    > >
>    > > * provider.openstack: One of 'auth_url' or 'cloud' must be specified
>    > > ```
>    > >
>    > > You should source your openstack credentials first. This is the file which has lines like:
>    > >
>    > > ```bash
>    > > export OS_AUTH_URL=https://...
>    > > export OS_PROJECT_ID=...
>    > > export OS_PROJECT_NAME="..."
>    > > export OS_USER_DOMAIN_NAME="Default"
>    > > ```
>    > >
>    > > You should run `source /path/to/file.sh` in your terminal, and then re-run `terraform plan`:
>    > >
>    > > ```
>    > > An execution plan has been generated and is shown below.
>    > > Resource actions are indicated with the following symbols:
>    > >   + create
>    > >
>    > > Terraform will perform the following actions:
>    > >
>    > >   + openstack_compute_keypair_v2.my-cloud-key
>    > >       id:          <computed>
>    > >       fingerprint: <computed>
>    > >       name:        "my-key"
>    > >       private_key: <computed>
>    > >       public_key:  "ssh-rsa AAAAB...."
>    > >       region:      <computed>
>    > >
>    > >
>    > > Plan: 1 to add, 0 to change, 0 to destroy.
>    > > ```
>    > >
>    > > If you see this, then everything ran successfully
>    > >
>    > {: .solution }
>    {: .question}
>
{: .hands_on }

Let's look at the output in detail:

```
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + openstack_compute_keypair_v2.my-cloud-key
      id:          <computed>
      fingerprint: <computed>
      name:        "my-key"
      private_key: <computed>
      public_key:  "ssh-rsa AAAAB..."
      region:      <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
```

Terraform informs us first about the different symbols used. Here it tells us that it will `+ create` a resource. Sometimes it will `- delete` or `~ update in-place`. Next it describes in detail what will be done and why. For creating a resource it does not give us much information, we will see more types of changes later.

Lastly it informs us that we did not save our plan. Terraform can maintain a concept of what the remote resource's state looks like between your `terraform plan` and `terraform apply` steps. This is a more advanced feature and will not be covered today.

> <hands-on-title>Applying our plan</hands-on-title>
>
> Now that you've reviewed the plan and everything looks good, we're ready to apply our changes.
>
> 1. Run `terraform apply`. Because we did not re-use our plan from the previous step, terraform will re-run the plan step first, and request your confirmation:
>
>    ```
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      + create
>
>    Terraform will perform the following actions:
>
>      + openstack_compute_keypair_v2.my-cloud-key
>          id:          <computed>
>          fingerprint: <computed>
>          name:        "my-key"
>          private_key: <computed>
>          public_key:  "" => "ssh-rsa AAAAB3..."
>          region:      <computed>
>
>
>    Plan: 1 to add, 0 to change, 0 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
> 2. Confirm that everything looks good and enter the value 'yes', and hit enter.
>
>    ```
>    openstack_compute_keypair_v2.my-cloud-key: Creating...
>      fingerprint: "" => "<computed>"
>      name:        "" => "my-key"
>      private_key: "" => "<computed>"
>      public_key:  "" => "ssh-rsa AAAAB3..."
>      region:      "" => "<computed>"
>    openstack_compute_keypair_v2.my-cloud-key: Creation complete after 3s (ID: my-key)
>
>    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
>    ```
>
>
>    > <comment-title>"Key pair 'my-key' already exists"</comment-title>
>    >
>    > If you see an error message that says:
>    >
>    > ```
>    > {"conflictingRequest": {"message": "Key pair 'my-key' already exists.", "code": 409}}
>    > ```
>    >
>    > Then you already have a keypair with this name. You should update the `name = "my-key"` to use a different name.
>    {: .comment}
>
{: .hands_on}

We should now have a keypair in our cloud!

## Adding an Instance

We now have:

- Terraform installed
- The OpenStack plugin initialized
- A keypair in OpenStack

> <comment-title>Doing this training outside of a training event</comment-title>
> If you are doing this training outside of an event, then you will likely need to do some additional steps specific to your Cloud:
>
> 1. Identify a small image flavor that you can use for your testing.
>
> 2. Upload this `raw` format image available from [UseGalaxy.eu](https://usegalaxy.eu/static/vgcn/denbi-centos7-j10-2e08aa4bfa33-master.raw) to your OpenStack
>
> You will need to do both of these things before you can proceed.
{: .comment}

You are now ready to launch an instance!

> <hands-on-title>Launching an Instance</hands-on-title>
>
> 1. Open your `main.tf` in your text editor and add the following.
>
>    > <warning-title>Correct image/flavor/network/security_group names</warning-title>
>    > The documentation below notes some specific values for the `image_name`, `flavor_name`, `security_groups`, and `network` properties. These *may not be correct* for your training, instead your instructor will provide these values to you.
>    {: .warning}
>
>    ```ini
>    resource "openstack_compute_instance_v2" "test" {
>      name            = "test-vm"
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>      flavor_name     = "m1.tiny"
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>      security_groups = ["default"]
>
>      network {
>        name = "public"
>      }
>    }
>    ```
>
>    There are some important things to note, we reproduce the above configuration but with comments:
>
>    "openstack_compute_instance_v2" is the 'type' of a resource, 'test' is the name of this specific resource.
>    ```
>    resource "openstack_compute_instance_v2" "test" {
>    ```
>
>    This will become the server name
>    ```
>      name            = "test-vm"
>    ```
>
>    The image which will be launched
>    ```
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    ```
>
>    Which flavor
>    ```
>      flavor_name     = "m1.tiny"
>    ```
>
>    This uses Terraform's knowledge of the
>    `openstack_compute_keypair_v2.my-cloud-key` resource which you
>    previously described. Then it accesses the `.name` attribute.
>    This allow your to update the name at any time, and still have your
>    resource definitions be correct.
>    ```
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    ```
>
>    The security group to apply. This is a comma separated list, but
>    `default` should work for most OpenStack clouds.
>    ```
>      security_groups = ["default"]
>    ```
>
>    A network that is accessible to you.
>    ```
>      network {
>        name = "public"
>      }
>    ```
>
> 2. Run `terraform apply`. Running the `plan` step is not necessary, it is just useful to see what changes will be applied without starting the apply process.
>
>    Note that first terraform "refreshes" the state of the remote resources that it already manages, before checking what changes need to be made.
>
>    ```
>    $ terraform apply
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      + create
>
>    Terraform will perform the following actions:
>
>      + openstack_compute_instance_v2.test
>          id:                         <computed>
>          access_ip_v4:               <computed>
>          access_ip_v6:               <computed>
>          all_metadata.%:             <computed>
>          availability_zone:          <computed>
>          flavor_id:                  <computed>
>          flavor_name:                "m1.tiny"
>          force_delete:               "false"
>          image_id:                   <computed>
>          image_name:                 "denbi-centos7-j10-2e08aa4bfa33-master"
>          key_pair:                   "my-key"
>          name:                       "test-vm"
>          network.#:                  "1"
>          network.0.access_network:   "false"
>          network.0.fixed_ip_v4:      <computed>
>          network.0.fixed_ip_v6:      <computed>
>          network.0.floating_ip:      <computed>
>          network.0.mac:              <computed>
>          network.0.name:             "public"
>          network.0.port:             <computed>
>          network.0.uuid:             <computed>
>          power_state:                "active"
>          region:                     <computed>
>          security_groups.#:          "1"
>          security_groups.3814588639: "default"
>          stop_before_destroy:        "false"
>
>
>    Plan: 1 to add, 0 to change, 0 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
> 3. If everything looks good, enter 'yes'.
>
>    ```
>    openstack_compute_instance_v2.test: Creating...
>      access_ip_v4:               "" => "<computed>"
>      access_ip_v6:               "" => "<computed>"
>      all_metadata.%:             "" => "<computed>"
>      availability_zone:          "" => "<computed>"
>      flavor_id:                  "" => "<computed>"
>      flavor_name:                "" => "m1.tiny"
>      force_delete:               "" => "false"
>      image_id:                   "" => "<computed>"
>      image_name:                 "" => "denbi-centos7-j10-2e08aa4bfa33-master"
>      key_pair:                   "" => "my-key"
>      name:                       "" => "test-vm"
>      network.#:                  "" => "1"
>      network.0.access_network:   "" => "false"
>      network.0.fixed_ip_v4:      "" => "<computed>"
>      network.0.fixed_ip_v6:      "" => "<computed>"
>      network.0.floating_ip:      "" => "<computed>"
>      network.0.mac:              "" => "<computed>"
>      network.0.name:             "" => "public"
>      network.0.port:             "" => "<computed>"
>      network.0.uuid:             "" => "<computed>"
>      power_state:                "" => "active"
>      region:                     "" => "<computed>"
>      security_groups.#:          "" => "1"
>      security_groups.3814588639: "" => "default"
>      stop_before_destroy:        "" => "false"
>    openstack_compute_instance_v2.test: Still creating... (10s elapsed)
>    openstack_compute_instance_v2.test: Still creating... (20s elapsed)
>    openstack_compute_instance_v2.test: Creation complete after 30s (ID: 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa)
>
>    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
>    ```
{: .hands_on}

You now have a running instance! We do not know the IP address so we cannot login yet. You can obtain that from the OpenStack dashboard, or via the `terraform show` command


> <hands-on-title>`terraform show` and logging in</hands-on-title>
>
> 1. Run `terraform show`
>
>    The values in the following output will not match yours.
>
>    ```
>    openstack_compute_instance_v2.test:
>      id = 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa
>      access_ip_v4 = 192.52.32.231
>      access_ip_v6 =
>      all_metadata.% = 0
>      availability_zone = nova
>      flavor_id = a9a58bcf-29a3-47f4-b4c3-e71e40127833
>      flavor_name = m1.tiny
>      force_delete = false
>      image_id = 922ae172-275b-4e6a-bb9d-c7ace52fc8d4
>      image_name = denbi-centos7-j10-2e08aa4bfa33-master
>      key_pair = my-key
>      name = test-vm
>      network.# = 1
>      network.0.access_network = false
>      network.0.fixed_ip_v4 = 192.52.32.231
>      network.0.fixed_ip_v6 =
>      network.0.floating_ip =
>      network.0.mac = fa:16:3e:e6:cd:6c
>      network.0.name = public
>      network.0.port =
>      network.0.uuid = 6a68b40c-356f-4d4c-95e9-e5c72855fb35
>      power_state = active
>      region = Freiburg
>      security_groups.# = 1
>      security_groups.3814588639 = default
>      stop_before_destroy = false
>    openstack_compute_keypair_v2.my-cloud-key:
>      id = my-key
>      fingerprint = 4a:72:2c:7e:d2:b7:f2:c4:5a:9e:bb:ca:c7:8f:d7:a3
>      name = my-key
>      private_key =
>      public_key = ssh-rsa AAAAB3...
>      region = Freiburg
>    ```
>
> 2. Here we can obtain the IP address. In the above output, it is `192.52.32.231`
>
> 3. Login with `ssh`: `ssh centos@<your-ip-address>`. You should see something like the following:
>
>    ```
>    The authenticity of host '192.52.32.231 (192.52.32.231)' can't be established.
>    ECDSA key fingerprint is SHA256:oEHgotonwT3a47K0kLdsNVd/QqctBjDK7F829J0VnwQ.
>    Are you sure you want to continue connecting (yes/no)? yes
>    Warning: Permanently added '192.52.32.231' (ECDSA) to the list of known hosts.
>          _        _   _ ____ _____
>         | |      | \ | |  _ \_   _| ██      ██
>       __| | ___  |  \| | |_) || |     ██████
>      / _' |/ _ \ | . ' |  _ < | |     ██████
>     | (_| |  __/_| |\  | |_) || |_  ██  ██  ██
>      \__,_|\___(_)_| \_|____/_____| ██  ██  ██
>    ====================================================
>    Hostname..........: test-vm.novalocal
>    Release...........: CentOS Linux release 7.5.1804 (Core)
>    Build.............: de.NBI Generic denbi-centos7-j10-2e08aa4bfa33-master
>    Build Date........: 2018-10-24T14:28:38Z
>    Current user......: centos
>    Load..............: 0.00 0.07 0.05 2/148 1644
>    Uptime............: up 5 minutes
>    ====================================================
>    [centos@test-vm ~]$
>    ```
>
{: .hands_on}

With this, you're done. You've launched a single VM with Terraform. The image we are using comes with:

- HTCondor
- Docker
- Singularity
- Conda
- CVMFS connected to Galaxyproject's biological databases and Singularity containers

This is very similar to the standard image that UseGalaxy.eu uses for all of its compute nodes.

# Managing a Cluster of VMs

When you are done playing around with the individual VM, we're ready to explore launching an entire cluster.

- We will launch a VM to act as our HTCondor central manager
- We will launch a VM to act as an NFS server for our "cluster"
- We will launch two execution nodes to process jobs

We use a very similar setup for our production cloud cluster that powers UseGalaxy.eu's compute and training infrastructure.

## Configuration

We will start by setting up the new nodes and launching them as a test:

> <hands-on-title>Adding configuration for other nodes</hands-on-title>
>
> 1. Update the name of the instance you already have, `test` to `name = "central-manager"`. This will be our scheduler central manager.
>
>    ```
>    resource "openstack_compute_instance_v2" "test" {
>      name            = "central-manager"
>      ...
>    }
>    ```
>
> 2. Add the NFS server. It is mostly the same configuration you've done before, just with a different `name` and resource name.
>
>    ```
>    resource "openstack_compute_instance_v2" "nfs" {
>      name            = "nfs-server"
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>      flavor_name     = "m1.tiny"
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>      security_groups = ["default"]
>
>      network {
>        name = "public"
>      }
>    }
>    ```
>
> 3. Lastly, we'll add the execution nodes. Here we will launch two servers at once by using the `count` parameter. We can add `count` in the main portion of a Terraform resource definition and it will create that many copies of that same resource. In the case where we wish to be able to distinguish between them, we can use `${count.index}` in the name or another field to distinguish them.
>
>    ```ini
>    resource "openstack_compute_instance_v2" "exec" {
>      name            = "exec-${count.index}"
>      image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>      flavor_name     = "m1.tiny"
>      key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>      security_groups = ["default"]
>      count           = 2
>
>      network {
>        name = "public"
>      }
>    }
>    ```
>
> 4. Run `terraform apply`
>
>    You will see something like the following:
>
>    ```
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>    openstack_compute_instance_v2.test: Refreshing state... (ID: 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      + create
>      ~ update in-place
>
>    Terraform will perform the following actions:
>
>      + openstack_compute_instance_v2.exec[0]
>          id:                         <computed>
>          access_ip_v4:               <computed>
>          access_ip_v6:               <computed>
>          all_metadata.%:             <computed>
>          availability_zone:          <computed>
>          ...
>
>      ~ openstack_compute_instance_v2.test
>          name:                       "test-vm" => "central-manager"
>
>
>    Plan: 3 to add, 1 to change, 0 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
>    This time, Terraform is able to modify the `openstack_compute_instance_v2.test` resource in place, it can just update the name of the VM. It will also create the other three requested resources.
>
> 5. Enter `yes` and terraform will make the requested changes
>
> 6. It should complete successfully
>
>    > <question-title></question-title>
>    >
>    > What did the output look like?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```
>    > > openstack_compute_instance_v2.nfs: Creation complete after 2m55s (ID: 421f0899-f2c1-43c9-8bbe-9b0415c8f4f4)
>    > > openstack_compute_instance_v2.exec[0]: Creation complete after 2m57s (ID: 1ae02775-a0ff-47a5-907f-513190a8548e)
>    > > openstack_compute_instance_v2.exec[1]: Creation complete after 2m59s (ID: d339ebeb-4288-498c-9923-42fce32a3808)
>    > >
>    > > Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
>    > > ```
>    > {: .solution}
>    {: .question}
{: .hands_on}

## `cloud-init`

These VMs that we have launched currently are all absolutely identical, except for the host name. There is no specialisation. They all do nothing. We will fix that and make these machines have individual roles by booting them with metadata attached to each instance.

[`cloud-init`](https://cloud-init.io/) is used for this process, it allows for injecting files and executing commands as part of the boot process.


> <details-title>Other actions</details-title>
> cloud-init allows for many more actions to be executed as well, you can read about them in [the documentation](https://cloudinit.readthedocs.io/en/latest/).
{: .details}

The `cloud-init` configuration is a YAML file. Here we can see an example of how it is used, we have a YAML array at the top level. Inside, a hash of `content`, `owner`, `path`, and `permissions`.

```yaml
#cloud-config
write_files:
- content: |
    CONDOR_HOST = 127.0.0.1
    ALLOW_WRITE = *
    ALLOW_READ = $(ALLOW_WRITE)
    ALLOW_NEGOTIATOR = $(ALLOW_WRITE)
    DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD
    FILESYSTEM_DOMAIN = denbi
    UID_DOMAIN = denbi
    TRUST_UID_DOMAIN = True
    SOFT_UID_DOMAIN = True
  owner: root:root
  path: /etc/condor/condor_config.local
  permissions: '0644'
```

This will create a file with the value from `content`, owned by root/group root, in `/etc/condor/condor_config.local`. So let's re-structure our `main.tf` file to have some configuration:

> <hands-on-title>cloud-init</hands-on-title>
>
> 1. Edit your central-manager server and add a block like the following at the end.
>
>    ```
>    user_data = <<-EOF
>      #cloud-config
>      write_files:
>      - content: |
>          CONDOR_HOST = localhost
>          ALLOW_WRITE = *
>          ALLOW_READ = $(ALLOW_WRITE)
>          ALLOW_NEGOTIATOR = $(ALLOW_WRITE)
>          DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD
>          FILESYSTEM_DOMAIN = terraform-training
>          UID_DOMAIN = terraform-training
>          TRUST_UID_DOMAIN = True
>          SOFT_UID_DOMAIN = True
>        owner: root:root
>        path: /etc/condor/condor_config.local
>        permissions: '0644'
>      - content: |
>          /data           /etc/auto.data          nfsvers=3
>        owner: root:root
>        path: /etc/auto.master.d/data.autofs
>        permissions: '0644'
>      - content: |
>          share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>        owner: root:root
>        path: /etc/auto.data
>        permissions: '0644'
>    EOF
>    ```
>
>    Near the end we see an interesting thing, `${openstack_compute_instance_v2.nfs.access_ip_v4}`, this will template the `user_data` that is passed to `cloud-init`, with the IP address of the NFS server.
>
> 2. Edit your NFS server and add a user data block like:
>
>    ```
>    user_data = <<-EOF
>      #cloud-config
>      write_files:
>      - content: |
>          /data/share *(rw,sync)
>        owner: root:root
>        path: /etc/exports
>        permissions: '0644'
>      runcmd:
>       - [ mkdir, -p, /data/share ]
>       - [ chown, "centos:centos", -R, /data/share ]
>       - [ systemctl, enable, nfs-server ]
>       - [ systemctl, start, nfs-server ]
>       - [ exportfs, -avr ]
>    EOF
>    ```
>
>    This will:
>
>    1. Make the `/data/share` directory
>    2. Set permissions on it
>    3. Launch the NFS service
>    4. Export `/data/share` to the network
>
> 3. Edit the `exec` node and add a user data block like:
>
>    ```
>    user_data = <<-EOF
>      #cloud-config
>      write_files:
>      - content: |
>          CONDOR_HOST = ${openstack_compute_instance_v2.test.access_ip_v4}
>          ALLOW_WRITE = *
>          ALLOW_READ = $(ALLOW_WRITE)
>          ALLOW_ADMINISTRATOR = *
>          ALLOW_NEGOTIATOR = $(ALLOW_ADMINISTRATOR)
>          ALLOW_CONFIG = $(ALLOW_ADMINISTRATOR)
>          ALLOW_DAEMON = $(ALLOW_ADMINISTRATOR)
>          ALLOW_OWNER = $(ALLOW_ADMINISTRATOR)
>          ALLOW_CLIENT = *
>          DAEMON_LIST = MASTER, SCHEDD, STARTD
>          FILESYSTEM_DOMAIN = terraform-training
>          UID_DOMAIN = terraform-training
>          TRUST_UID_DOMAIN = True
>          SOFT_UID_DOMAIN = True
>          # run with partitionable slots
>          CLAIM_PARTITIONABLE_LEFTOVERS = True
>          NUM_SLOTS = 1
>          NUM_SLOTS_TYPE_1 = 1
>          SLOT_TYPE_1 = 100%
>          SLOT_TYPE_1_PARTITIONABLE = True
>          ALLOW_PSLOT_PREEMPTION = False
>          STARTD.PROPORTIONAL_SWAP_ASSIGNMENT = True
>        owner: root:root
>        path: /etc/condor/condor_config.local
>        permissions: '0644'
>      - content: |
>          /data           /etc/auto.data          nfsvers=3
>        owner: root:root
>        path: /etc/auto.master.d/data.autofs
>        permissions: '0644'
>      - content: |
>          share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>        owner: root:root
>        path: /etc/auto.data
>        permissions: '0644'
>    EOF
>    ```
> 4. Compare your final configuration against ours:
>
>    > <solution-title>Final Configuration</solution-title>
>    > ```
>    > resource "openstack_compute_keypair_v2" "my-cloud-key" {
>    >   name       = "my-key"
>    >   public_key = "ssh-rsa AAAAB3..."
>    > }
>    >
>    > resource "openstack_compute_instance_v2" "test" {
>    >   name            = "central-manager"
>    >   image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    >   flavor_name     = "m1.tiny"
>    >   key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    >   security_groups = ["default"]
>    >
>    >   network {
>    >     name = "public"
>    >   }
>    >
>    >   user_data = <<-EOF
>    >     #cloud-config
>    >     write_files:
>    >     - content: |
>    >         CONDOR_HOST = localhost
>    >         ALLOW_WRITE = *
>    >         ALLOW_READ = $(ALLOW_WRITE)
>    >         ALLOW_NEGOTIATOR = $(ALLOW_WRITE)
>    >         DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD
>    >         FILESYSTEM_DOMAIN = terraform-training
>    >         UID_DOMAIN = terraform-training
>    >         TRUST_UID_DOMAIN = True
>    >         SOFT_UID_DOMAIN = True
>    >       owner: root:root
>    >       path: /etc/condor/condor_config.local
>    >       permissions: '0644'
>    >     - content: |
>    >         /data           /etc/auto.data          nfsvers=3
>    >       owner: root:root
>    >       path: /etc/auto.master.d/data.autofs
>    >       permissions: '0644'
>    >     - content: |
>    >         share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>    >       owner: root:root
>    >       path: /etc/auto.data
>    >       permissions: '0644'
>    >   EOF
>    > }
>    >
>    > resource "openstack_compute_instance_v2" "nfs" {
>    >   name            = "nfs-server"
>    >   image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    >   flavor_name     = "m1.tiny"
>    >   key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    >   security_groups = ["default"]
>    >
>    >   network {
>    >     name = "public"
>    >   }
>    >
>    >   user_data = <<-EOF
>    >     #cloud-config
>    >     write_files:
>    >     - content: |
>    >         /data/share *(rw,sync)
>    >       owner: root:root
>    >       path: /etc/exports
>    >       permissions: '0644'
>    >     runcmd:
>    >      - [ mkdir, -p, /data/share ]
>    >      - [ chown, "centos:centos", -R, /data/share ]
>    >      - [ systemctl, enable, nfs-server ]
>    >      - [ systemctl, start, nfs-server ]
>    >      - [ exportfs, -avr ]
>    >   EOF
>    > }
>    >
>    > resource "openstack_compute_instance_v2" "exec" {
>    >   name            = "exec-${count.index}"
>    >   image_name      = "denbi-centos7-j10-2e08aa4bfa33-master"
>    >   flavor_name     = "m1.tiny"
>    >   key_pair        = "${openstack_compute_keypair_v2.my-cloud-key.name}"
>    >   security_groups = ["default"]
>    >   count           = 2
>    >
>    >   network {
>    >     name = "public"
>    >   }
>    >
>    >   user_data = <<-EOF
>    >     #cloud-config
>    >     write_files:
>    >     - content: |
>    >         CONDOR_HOST = ${openstack_compute_instance_v2.test.access_ip_v4}
>    >         ALLOW_WRITE = *
>    >         ALLOW_READ = $(ALLOW_WRITE)
>    >         ALLOW_ADMINISTRATOR = *
>    >         ALLOW_NEGOTIATOR = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_CONFIG = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_DAEMON = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_OWNER = $(ALLOW_ADMINISTRATOR)
>    >         ALLOW_CLIENT = *
>    >         DAEMON_LIST = MASTER, SCHEDD, STARTD
>    >         FILESYSTEM_DOMAIN = terraform-training
>    >         UID_DOMAIN = terraform-training
>    >         TRUST_UID_DOMAIN = True
>    >         SOFT_UID_DOMAIN = True
>    >         # run with partitionable slots
>    >         CLAIM_PARTITIONABLE_LEFTOVERS = True
>    >         NUM_SLOTS = 1
>    >         NUM_SLOTS_TYPE_1 = 1
>    >         SLOT_TYPE_1 = 100%
>    >         SLOT_TYPE_1_PARTITIONABLE = True
>    >         ALLOW_PSLOT_PREEMPTION = False
>    >         STARTD.PROPORTIONAL_SWAP_ASSIGNMENT = True
>    >       owner: root:root
>    >       path: /etc/condor/condor_config.local
>    >       permissions: '0644'
>    >     - content: |
>    >         /data           /etc/auto.data          nfsvers=3
>    >       owner: root:root
>    >       path: /etc/auto.master.d/data.autofs
>    >       permissions: '0644'
>    >     - content: |
>    >         share  -rw,hard,intr,nosuid,quota  ${openstack_compute_instance_v2.nfs.access_ip_v4}:/data/share
>    >       owner: root:root
>    >       path: /etc/auto.data
>    >       permissions: '0644'
>    >   EOF
>    > }
>    > ```
>    >
>    {: .solution}
>
> 5. Run `terraform apply`
>
>    It will look slightly different this time:
>
>    ```
>    $ terraform apply
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>    openstack_compute_instance_v2.nfs: Refreshing state... (ID: 421f0899-f2c1-43c9-8bbe-9b0415c8f4f4)
>    openstack_compute_instance_v2.test: Refreshing state... (ID: 7a2ed5ba-0801-49b5-bf1b-9bf9cec733fa)
>    openstack_compute_instance_v2.exec[0]: Refreshing state... (ID: 1ae02775-a0ff-47a5-907f-513190a8548e)
>    openstack_compute_instance_v2.exec[1]: Refreshing state... (ID: d339ebeb-4288-498c-9923-42fce32a3808)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>    -/+ destroy and then create replacement
>
>    Terraform will perform the following actions:
>
>    -/+ openstack_compute_instance_v2.exec[0] (new resource required)
>          id:                         "1ae02775-a0ff-47a5-907f-513190a8548e" => <computed> (forces new resource)
>          access_ip_v4:               "192.52.32.255" => <computed>
>          ...
>          user_data:                  "" => "07d27d50dbdf12a5647d9fd12a3510861d32203c" (forces new resource)
>
>    ...
>    Plan: 4 to add, 0 to change, 4 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
>
>    Here Terraform has detected that the userdata has changed. It cannot change this dynamically at runtime, only at boot time. So it decides that it must destroy and then replace that resource.
>
{:.hands_on}

## Login / HTCondor / A Test Job

We now have a running cluster! Let's log in

> <hands-on-title>Cluster Usage</hands-on-title>
>
> 1. Now that we have a cluster in the cloud, let's login. Look through the output of `terraform show` to find your `central-manager` server and its IP address.
>
> 2. `ssh centos@<your manager ip>`
>
> 3. Run `condor_status` which will show you the status of your cluster.
>
>    After your central manager VM booted, the executor nodes booted as well. Then, using the IP address of the central manager, the executors contacted that machine, and advertised their availability to run jobs.
>
>    > <question-title></question-title>
>    >
>    > What does the output look like? How many executor nodes do you see?
>    >
>    > > <solution-title></solution-title>
>    > > ```
>    > > [centos@central-manager share]$ condor_status
>    > > Name                   OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime
>    > >
>    > > slot1@exec-0.novalocal LINUX      X86_64 Unclaimed Idle      0.150  991  0+00:15:28
>    > > slot1@exec-1.novalocal LINUX      X86_64 Unclaimed Idle      0.000  991  0+00:15:32
>    > >
>    > >                      Machines Owner Claimed Unclaimed Matched Preempting  Drain
>    > >
>    > >         X86_64/LINUX        2     0       0         2       0          0      0
>    > >
>    > >                Total        2     0       0         2       0          0      0
>    > > ```
>    > {: .solution}
>    {: .question}
>
> 4. Change into the `/data/share/`
>
> 5. Create a file, `test.job` with the following contents:
>
>    ```
>    Universe = vanilla
>    Executable = /data/share/test.sh
>    Log = test.$(process).log
>    Output = test.$(process).out
>    Error = test.$(process).err
>    request_cpus = 1
>    Queue 10
>    ```
>
> 6. Create a file `test.sh` with the following:
>
>    ```
>    #!/bin/bash
>    echo "$(hostname)"
>    sleep 1
>    ```
> 7. Run `chmod +x test.sh`
>
> 8. Run `condor_submit test.job`
>
> 9. (Quickly) Run `condor_q` to see the queue status. You can invoke this repeatedly to watch the queue update.
>
> 10. Run `cat *.out` and you should see the hostnames where the jobs were run
>
{:.hands_on}

# Infrastructure Graph

Terraform has the ability to produce a graph showing the relationship of resources in your infrastructure. We will produce a graphic for our current terraform plan:

> <hands-on-title>Infrastructure Graph</hands-on-title>
>
> 1. Run:
>
>    ```
>    terraform graph | dot -Tpng > graph.png
>    ```
>
>    You may need the `graphviz` package installed in order to produce the graph
>
> 2. Open up `graph.png` in an image viewer
>
>    > <question-title></question-title>
>    >
>    > What did the output look like?
>    >
>    > > <solution-title></solution-title>
>    > > ![A simple graph](../../images/graph.png "A simple graph showing the structure of the infrastructure in this lesson")
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

This is a very simple resrouce graph:

![A simple graph](../../images/graph.png "A simple graph showing the structure of the infrastructure in this lesson")

The dependencies follow the arrows, the NFS depends on the keypair being setup. The `test` machine (central manager) depends on the NFS server being available (and having an IP), the exec nodes depend on the central manager server. This is a somewhat simplified view, there are more dependencies which are not shown for simplicity (e.g. the exec nodes depend on NFS)

If you have variables this can produce a more complex dependency graph:

![A more complex graph showing variables](../../images/graph2.png "A more complex graph showing variables")

Once you develop complex infrastructure, these graphics become less useful.

# Tearing Everything Down

> <hands-on-title>Cleanup</hands-on-title>
>
> 1. Simple delete your `main.tf` file (or rename it without the `.tf` extension)
>
> 2. Run `terraform apply`. Terraform will see that the resource is no longer part of your code and remove it.
>
>    ```
>    openstack_compute_keypair_v2.my-cloud-key: Refreshing state... (ID: my-key)
>    openstack_compute_instance_v2.nfs: Refreshing state... (ID: 0b58d613-48f2-49c4-9d8a-3623d17f1c93)
>    openstack_compute_instance_v2.test: Refreshing state... (ID: e433f772-6f17-4609-aa61-053f7602533f)
>    openstack_compute_instance_v2.exec[0]: Refreshing state... (ID: 034a4206-b0ad-4d0e-9c75-27d3966f99ac)
>    openstack_compute_instance_v2.exec[1]: Refreshing state... (ID: fe576776-397e-4f1f-a8e7-4a9203764567)
>
>    An execution plan has been generated and is shown below.
>    Resource actions are indicated with the following symbols:
>      - destroy
>
>    Terraform will perform the following actions:
>
>      - openstack_compute_instance_v2.exec[0]
>
>      - openstack_compute_instance_v2.exec[1]
>
>      - openstack_compute_instance_v2.nfs
>
>      - openstack_compute_instance_v2.test
>
>      - openstack_compute_keypair_v2.my-cloud-key
>
>
>    Plan: 0 to add, 0 to change, 5 to destroy.
>
>    Do you want to perform these actions?
>      Terraform will perform the actions described above.
>      Only 'yes' will be accepted to approve.
>
>      Enter a value:
>    ```
{: .hands_on}

## Destroy time provisioners

Sometimes, immediately terminating the VM is not ideal behaviour. Especially if you're managing a cluster like HTCondor which can gracefully terminate jobs, you might want to permit that. Terraform provides 'destroy-time provisioners', code snippets that are run before the VM is destroyed. If they return successfully, destruction continues. If they exit with an error code, then destruction is halted. We can write a simple one for HTCondor like so:

```bash
set -ex
condor_drain $(hostname) || true;
my_ip=$(/sbin/ifconfig eth0 | grep 'inet ' | awk '{print $2}')
# Find slots which have an address associated with us.
slots=$(condor_status -autoformat Name State MyAddress | grep "<${my_ip}?9618" | wc -l)
# If there are more than one slots, leave it.
if (( slots > 1 )); then
    exit 1;
else
    # Otherwise, poweroff.
    /usr/sbin/condor_off -graceful
    exit 0;
fi
```

Which can be provided to your VM like:

```
provisioner "remote-exec" {
  when = "destroy"

  scripts = [
    "prepare-restart.sh",
  ]

  connection {
    type        = "ssh"
    user        = "centos"
    private_key = "${file("~/.ssh/id_rsa")}"
  }
}
```

Terraform will SSH in with those credentials, copy over the script in `prepare-restart.sh`, and execute it.

# UseGalaxy.eu's Terraform Usage

All of [our virtual infrastructure](https://github.com/usegalaxy-eu/infrastructure/) is managed, publicly, with Terraform. We hope that this can inspire others and give people ideas of the sort of things they can accomplish with Terraform. If you have questions over the way we have done certain things, feel free to file an issue and ask us!


topic: admin
tutorial: external-auth
title: External Authentication
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["authentication"]
content_hash: 193ad1de042ab871692f4c091f64c22a
last_modified: 2026-02-26T21:12:14.144133
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/external-auth/tutorial.html

content:
# Overview


For this exercise we will use a basic password file method for authenticating - this is probably not a very useful method in production, but it demonstrates how the proxy server can be configured to provide the correct header to Galaxy, and how Galaxy integrates with upstream authentication providers. This same method can be used with NGINX and Apache modules for CAS or SAML authentication.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Configuring Authentication

> <hands-on-title>Configuring everything</hands-on-title>
>
> 1. Edit the galaxy.j2 template file in the templates/nginx directory and update the main location block defined for serving galaxy. Add the parameters:
>      - `auth_basic galaxy;`
>      - `auth_basic_user_file /etc/nginx/passwd;`
>      - `proxy_set_header HTTP_REMOTE_USER $remote_user;`
>      - `proxy_set_header HTTP_GX_SECRET SOME_SECRET_STRING;`
>
>    It should look like:
>
>    ```diff
>    @@ -14,6 +14,10 @@
>    +        auth_basic           galaxy;
>    +        auth_basic_user_file /etc/nginx/passwd;
>    +        proxy_set_header          HTTP_REMOTE_USER $remote_user;
>    +        proxy_set_header          HTTP_GX_SECRET SOME_SECRET_STRING;
>         }
>     
>         # Static files can be more efficiently served by Nginx. Why send the
>    ```
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
>    > <tip-title>Running this tutorial <i>just</i> for Reports?</tip-title>
>    > Add the `auth_basic` and `auth_basic_user_file` lines to your `location /reports/`
>    {: .tip}
>
>
>    `auth_basic` enables validation of username and password using the "HTTP Basic Authentication" protocol. Its value `galaxy` is used as a realm name to be displayed to the user when prompting for credentials.
>    `auth_basic_user_file` specifies the file that keeps usernames and passwords.
>    `proxy_set_header` adds `HTTP_REMOTE_USER` to the special variables passed by nginx to Galaxy, with value `$remote_user`, which is a nginx embedded variable containing the username supplied with the Basic authentication.
>
>    `GX_SECRET` is added as a header for security purposes, to prevent any other users on the system impersonating nginx and sending requests to Galaxy. NGINX and other webservers like Apache will strip any user-sent `REMOTE_USER` headers, as that header defines the authenticated user. If you can talk directly to Galaxy (e.g. via curl) and provide the `REMOTE_USER` header, you can impersonate any other use. While having Galaxy listen on `127.0.0.1` prevents any requests from outside of the system reaching Galaxy, anyone on the system can still send requests to that port. Here you can choose to switch to a unix socket with permissions only permitting Galaxy and Nginx to connect. `GX_SECRET` adds additional security as it needs to match `remote_user_secret` in your galaxy configutation.
>
>    > <tip-title>Proxy bypass</tip-title>
>    > Users can bypass the authentication only if they can talk directly to the gunicorn processes (if you have socket/http: 0.0.0.0, or if it is directly responsible for serving galaxy, and there is no proxy.)
>    > This can happen mostly when some users have command line access to the Galaxy server, which is considered a bad practice.
>    {: .tip}
>
> 2. Add a pre_task using the [`pip`](https://docs.ansible.com/ansible/2.9/modules/pip_module.html) module which installs the library `passlib`, which is required for `htpasswd`.
>
>    Add a pre_task using the [`htpasswd`](https://docs.ansible.com/ansible/2.4/htpasswd_module.html) module which sets up a password file in `/etc/nginx/passwd`, with owner and group set to root, and a name and password, and a mode of 0640.
>    > <question-title></question-title>
>    >
>    > How does your final configuration look?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```diff
>    > > @@ -7,6 +7,15 @@
>    > >          name: galaxy
>    > >          state: restarted
>    > >    pre_tasks:
>    > > +    - pip:
>    > > +        name: passlib
>    > > +    - htpasswd:
>    > > +        path: /etc/nginx/passwd
>    > > +        name: user1        # Pick a username
>    > > +        password: changeme # and a password
>    > > +        owner: www-data    # nginx on centos
>    > > +        group: root
>    > > +        mode: 0640
>    > >      - name: Install Dependencies
>    > >        package:
>    > >          name: ['git', 'make', 'python3-psycopg2', 'virtualenv']
>    > > ```
>    > {: .solution }
>    >
>    {: .question}
>
> 3. Galaxy needs to be instructed to expect authentication to come from the upstream proxy. In order to do this, set the following two options in your Galaxy group variables:
>
>    ```yaml
>    ...
>    galaxy_config:
>      galaxy:
>        ...
>        use_remote_user: true
>        remote_user_maildomain: "{% raw %}{{ inventory_hostname }}{% endraw %}"
>        remote_user_secret: SOME_SECRET_STRING
>    ```
>
>    > <tip-title>Running this tutorial <i>just</i> for Reports?</tip-title>
>    > You don't need to make the above changes for Galaxy then.
>    {: .tip}
>
>    Set the `remote_user_maildomain` option to the appropriate domain name for your site.
>
> 4. Run the playbook
>
{: .hands_on}

> <comment-title>Access denied</comment-title>
>
> If you see this message, it is because nginx is not correctly sending the `REMOTE_USER` or the `GX_SECRET` values.
>
> ![access denied message](../../images/access_denied.png)
>
{: .comment}


# Testing

You should now be presented with a password dialog when attempting to load the Galaxy UI.

> <hands-on-title>Testing</hands-on-title>
>
> 1. Log in using the username and password you provided when creating the `passwd` file. If your username and the value of `remote_user_maildomain` match an existing user, you will be logged in to that account. If not, a new account will be created for that user.
>
> 2. Click on the "User" menu at the top, to see how the username appears.
>
> 3. Note that some user features are not available when remote user support is enabled.
>
>    Try logging out by selecting **User** -> **Logout**. You will discover that when returning to the user interface, you are still logged in. This is because Galaxy has no way of logging you out of the proxy's authentication system. Instead, you should set `remote_user_logout_href` in `galaxy.yml` to point to the URL of your authentication system's logout page.
>
{: .hands_on}

# API access

If you wish your Galaxy to be accessible to command line clients (e.g. bioblend, blend4j, parsec), you will need to add an exception for authentication on the API. Galaxy will still be secure and protected, but non-browser access will be permitted with an API key.

```
location /api/ {
    satisfy any;
    allow all;
}
```

> <tip-title>Notification of Registration</tip-title>
> There is no built-in way to be notified if users are registered, with external authentication or built-in. However, you could automate this easily. There is a [gxadmin](https://github.com/usegalaxy-eu/gxadmin) command we use called `gxadmin query latest-users` which Björn uses often. Other sites have other methods, e.g. Nicola's [cron script](https://gist.github.com/nsoranzo/f023e26aa60024ef6a7e3a3fe5fb2e4f) which runs daily on his server, to add new users to a group according to their email domain name.
{: .tip}

# Reverting

We don't want to leave Galaxy this way for the rest of our workshop.

> <hands-on-title>Reverting the changes</hands-on-title>
>
> 1. Edit your group variables file and comment out:
>
>    - the NGINX changes
>    - `use_remote_user: true`
>
> 2. Run the playbook
>
{: .hands_on}


topic: admin
tutorial: data-library
title: Data Libraries
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["ansible", "storage", "git-gat"]
content_hash: 7f5556452f0b68b7058eb112da9c3b49
last_modified: 2026-02-26T21:12:14.142860
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/data-library/tutorial.html

content:
Data libraries are a great way to provide structured repositories of data to
everyone on your server, or to a subset of your users. Datasets in libraries do
not count against user quotas, so they are commonly used to provide the results
of a sequencing run or similar project to a group of users on the servers.

![screenshot of data libraries]({{site.baseurl}}/topics/admin/images/data/data-library-screenshot.png)


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="data-library" %}

# Setup

Before we can import local data, we need to configure Galaxy to permit this. Additionally we will setup an example data library which we can use for demonstrative purposes.

> <hands-on-title>Setting up Data Libraries</hands-on-title>
>
> 1. We will add a pre-task to clone [a data repository](https://github.com/usegalaxy-eu/libraries-training-repo) into your machine. We will use this as the source for a library dataset.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -29,6 +29,9 @@
>           package:
>             name: ['tmpreaper']
>           when: ansible_os_family == 'Debian'
>    +    - git:
>    +        repo: 'https://github.com/usegalaxy-eu/libraries-training-repo'
>    +        dest: /libraries/
>       roles:
>         - galaxyproject.tusd
>         - usegalaxy_eu.apptainer
>    {% endraw %}
>    ```
>    {: data-commit="Add the git repository to the pre-tasks"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Take a minute to explore the [folders in our sample library.](https://github.com/usegalaxy-eu/libraries-training-repo). These will be important when we start loading data.
>
> 4. Edit the file `group_vars/galaxyservers.yml` and set the following variables:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -96,6 +96,9 @@ galaxy_config:
>         # Tool Dependencies
>         dependency_resolvers_config_file: "{{ galaxy_config_dir }}/dependency_resolvers_conf.xml"
>         container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>    +    # Data Library Directories
>    +    library_import_dir: /libraries/admin
>    +    user_library_import_dir: /libraries/user
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Configure the library import directories"}
>
>    Note that the `/libraries/admin` will refer to a folder within the libraries-training-repo that we cloned in the pre-task, and likewise for `/libraries/user`
>
> 5. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

# Importing Data

There are multiple options for importing data from your server, we'll go through all of your choices below. But first, let's take a quick look at the example library structure we've provided.

> > <code-in-title>Bash</code-in-title>
> > ```bash
> > tree /libraries
> > ```
> {: .code-in}
>
> > <code-out-title>Bash</code-out-title>
> > ```
> > /libraries/
> > ├── admin
> > │   └── admin-wildtype.fna
> > ├── example-library.yaml
> > ├── README.md
> > └── user
> >     ├── admin@example.com
> >     │   └── user-wildtype.fna
> >     └── admin@example.org
> >         └── user-wildtype.fna
> >
> > 4 directories, 5 files
> > ```
> {: .code-out}
{: .code-2col}

> <tip-title>admin@example.org</tip-title>
> Note that in the user directories, admin@example.com and example.org are used, if you've used a different email address for *your* admin user, you'll need to copy one of these directories.
{: .tip}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

We have a directory named `admin`, which will be available to all admin users (we set `library_import_dir: /libraries/admin` earlier.)

Additionally we have a `user` directory, below the user directory are more directories with the user's email as they directory key. Data can be placed in here, and it will become accessible to those users (we set `user_library_import_dir: /libraries/user` for this.)

![An add datasets dropdown menu in galaxy showing the options from history, from user directory, and under admins only, from import directory](../../images/data/import-menu.png)

## from History

This is by far the easiest and most convenient option for small datasets, or datasets that are just already in a history

![A select box is shown listing files in the user's history](../../images/data/import-history.png)

You can easily select multiple files and quickly import them.

## from User Directory

If user directories are configured, as we did at the beginning of this tutorial, then users will be able to import any files under their personal directory. This can be used for a wide variety of setups, e.g. providing the output of sequencing machines to users. This can point to the same directory structure that's used by the FTP service, if you want your users to be able to import files directly from FTP.

![Import popup with a list of files with one file, user-wildtype.fna, and buttons for configuring import behaviour.](../../images/data/import-user.png)

This will enable the option for everyone, any unprivileged user with a folder in the `user_library_import_dir` directory *and* permissions on a data library can import things from their import directory.

> <tip-title>Do you not see a directory here?</tip-title>
> We pre-created a directory for someone registered as `admin@example.org`, but if you are logged in with an account registered with a different email, you'll see nothing.
>
> You can fix this by going into /libraries/user and `cp -Rv admin@example.org whatever-email-you-used`
> Then it should appear in Galaxy.
{: .tip}

## from import Directory (Admins only)

Similarly to the user import directories, there is the option to have an admin only import area. If one admin is responsible for creating the data libraries and importing data this can be a good option.

![Same as the previous image, import popup listing options and one file, admin-wildtype.fna](../../images/data/import-admin.png)

An important feature of data libraries importing is the ability to "link files instead of copying". If you know your data will not be deleted (e.g. sequencing data sitting on an NFS), then you can use this option to further decrease the amount of storage your Galaxy server requires. The datasets can be used as if they were normal datasets, imported by users and analysed, but not imported into the Galaxy data storage area.

# Automatically Populating a Data Library

If your data is accessible via URL, you can write a yaml file to import and setup the data library automatically.
We've included this file in the example data libraries repository we cloned at the beginning:

> <code-in-title>Bash</code-in-title>
> ```bash
> cat /libraries/example-library.yaml
> ```
{: .code-in}

> <code-out-title>Bash</code-out-title>
> ```yaml
> ---
> destination:
>   type: library
>   name: Mouse sequencing project
>   description: some data
>   synopsis: samples collected from somewhere
> items:
> - url: https://zenodo.org/records/582600/files/wildtype.fna
>   src: url
>   ext: fasta
>   info: https://doi.org/10.5281/zenodo.582600
> - name: A directory
>   description: Exome sequencing means that all protein-coding genes in a genome are
>   items:
>   - url: https://zenodo.org/records/582600/files/mutant_R1.fastq
>     src: url
>     ext: fastqsanger
>     info: https://doi.org/10.5281/zenodo.582600
>   - url: https://zenodo.org/records/582600/files/mutant_R2.fastq
>     src: url
>     ext: fastqsanger
>     info: https://doi.org/10.5281/zenodo.582600
> ```
{: .code-out}

Let's try setting that up in our Galaxy!

{% snippet faqs/galaxy/preferences_admin_api_key.md admin=true %}

> <hands-on-title>importing data library yaml.</hands-on-title>
>
> 1. re-activate the virtualenv you created for the [ephemeris tool management tutorial]({% link topics/admin/tutorials/tool-management/tutorial.md %}).
>
>    > <code-in-title>input: bash</code-in-title>
>    > ```
>    > . ~/ephemeris_venv/bin/activate
>    > ```
>    {: .code-in}
>
>    > <tip-title>missing?</tip-title>
>    >
>    > then you might need to re-run the steps:
>    >
>    > ```bash
>    > python3 -m venv ~/ephemeris_venv
>    > . ~/ephemeris_venv/bin/activate
>    > pip install ephemeris
>    > ```
>    {: .tip}
>
> 2. we'll use the `setup-data-libraries` command to install the data in this yaml file into a library in our galaxy.
>
>    > <code-in-title>input: bash</code-in-title>
>    > ```bash
>    > setup-data-libraries -g https://galaxy.example.org -a <api-key> --training -i /libraries/example-library.yaml --legacy
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title>output</code-out-title>
>    > ```
>    > library name: mouse sequencing project
>    > ```
>    {: .code-out}
>
> 3. this command is safe to re-run. for example if you update the yaml, it will simply report that the library exists before ensuring all files exist in their appropriate location:
>
>    ```bash
>    library name: mouse sequencing project
>    library already exists! id: f597429621d6eb2b
>    ```
>
{: .hands_on}

![Screenshot of data libraries, we're in a library folder named "Mouse sequencing project" and a directory and single file are shown. The file has an ugly URL as its name](../../images/data/imported.png)

That's it! You should be able to [see your newly created data library in your Galaxy.](https://my.gat.galaxy.training/?path=/libraries)

> ```bash
> 2.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/library-permissions.md %}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=8 %}

Note that we've used some special flags here, `--training` and `--legacy`. Training sets some defaults that make sense for the GTN (mostly around library descriptions / etc.)

## `--legacy`

This enables the use of legacy APIs even for newer Galaxies that should have a batch upload API enabled. Unfortunately the new batch upload API is not able to update existing data libraries in place and will always create new libraries with the same name. So currently `--legacy` is quite useful for maintaining a YAML file, and running `setup-data-libraries` regularly whenever that file has updates.

But this comes with the downside that the entire URL is used for the filename.

# GTN Data

This process scales quite well. Galaxy Europe, in their mission to support all of the GTN materials on their server setup [a shared-data repository](https://github.com/usegalaxy-eu/shared-data), a single giant YAML file with all of the data from the GTN tutorials. This was then expanded to other UseGalaxy.* servers to provide synced training libraries for all of the major servers.

Do you want the training data synced on your server? If so join the shared data repository! If you provide us with a non-admin API key and a data library to upload data into, we can sync this automatically.

# Using Data from Libraries

Users can now conveniently use datasets in libraries when they are running analyses, without having to import them first.

The tool form provides a button on the right of every dataset selector which allows users to "Browse datasets"

![tool form with the folder icon to the right of the dataset selector shown, the button with tooltip "Browse datasets" is highlighted](../../images/data/select-from-lib0.png)

Users can then choose from datasets in their history, or browse through the data libraries

![Popup with "Data Libraries", datasets from histories, and an upload button](../../images/data/select-from-lib1.png)

Here users can see every file accessible to them in the data library to begin analysing.

![Same popup as previous image, but the file listing now shows the contents of the data library we created during the hands-on step with Mouse data](../../images/data/select-from-lib2.png)

{% snippet topics/admin/faqs/git-gat-path.md tutorial="data-library" %}


topic: admin
tutorial: ftp
title: Enable upload via FTP
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/ftp/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["data", "git-gat"]
content_hash: 9969c40913585d5f10229af33df731f7
last_modified: 2026-02-26T21:12:14.144792
workflows_json: []
content:
This tutorial will guide you to setup an {FTP} server so galaxy users can use it to upload large files. Indeed, as written on the [galaxy community hub](https://galaxyproject.org/ftp-upload/), uploading data directly from the browser can be unreliable and cumbersome. FTP will allow users to monitor the upload status as well as resume interrupted transfers.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="ftp" %}

# FTP

{FTP} is a very old and reliable communication protocol that has been around since 1971 {% cite rfc114 %}. It requires a server (here our galaxy server) and a client (user's computer). The FTP server requires to have at least 2 ports accessible from outside (one for the commands and one for the transfer). Usually the port for the command is 21.

FTP supports two different modes: active, and passive. Active mode requires that the user's computer be reachable from the internet, which in the age of {NAT} and firewalls is usually unusable. So passive mode is the most commonly used. In passive mode, a client connects to the FTP server, and requests a channel for sending files. The server responds with an IP and port, from its range of "Passive Ports".

> <comment-title>Requirements for Running This Tutorial</comment-title>
>
> Your VM or wherever you are installing Galaxy needs to have the following ports available:
>
> - 21
> - Some high range of ports not used by another service, e.g. 56k-60k
>
> You need to know which ports are open so you can use them for the transfer (PassivePorts). In this training we assume that 56k to 60k are open.
>
> Which ports precisely is not important, and these numbers can differ between sites.
{: .comment}

# FTP and Galaxy

To allow your user to upload via FTP, you will need to:

- configure Galaxy to know where the files are uploaded.
- install a FTP server
- allow your FTP server to read Galaxy's database so users can use their credential and upload in the good directory.

For secure transmission we will use SSL/TLS (FTPS), not the SSH File Transfer Protocol (SFTP) as the Galaxy users don't correspond to users on the machine.

## Installing and Configuring

Luckily for us, there is an ansible role written by the Galaxy Project for this purpose. It will install proftpd. Firstly, we need to install the role and then update our playbook for using it.

If the terms "Ansible", "role" and "playbook" mean nothing to you, please checkout [the Ansible introduction slides]({% link topics/admin/tutorials/ansible/slides.html %}) and [the Ansible introduction tutorial]({% link topics/admin/tutorials/ansible/tutorial.md %})

{% snippet topics/admin/faqs/ansible_local.md %}

> <hands-on-title>Setting up ftp upload with Ansible</hands-on-title>
>
> 1. In your playbook directory, add the `galaxyproject.proftpd` role to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -56,3 +56,6 @@
>     # Sentry
>     - name: mvdbeek.sentry_selfhosted
>       src: https://github.com/mvdbeek/ansible-role-sentry/archive/main.tar.gz
>    +# Our FTP Server
>    +- src: galaxyproject.proftpd
>    +  version: 0.3.1
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. As in this training we are using certbot, we will ask for a private key for proftpd. Add the following line to your `group_vars/galaxyserver.yml` file:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -210,11 +210,13 @@ certbot_environment: staging
>     certbot_well_known_root: /srv/nginx/_well-known_root
>     certbot_share_key_users:
>       - www-data
>    +  - proftpd
>     certbot_share_key_ids:
>       - "999:999"
>     certbot_post_renewal: |
>         systemctl restart nginx || true
>         docker restart rabbit_hole || true
>    +    systemctl restart proftpd || true
>     certbot_domains:
>      - "{{ inventory_hostname }}"
>      - "{{ sentry_domain }}"
>    {% endraw %}
>    ```
>    {: data-commit="Add proftpd in certbot"}
>
>    This will make a copy of the current letsencrypt key available as `/etc/ssl/user/privkey-proftpd.pem`, and automatically restart proftpd every time the key is updated.
>
> 4. We will configure Galaxy to enable ftp file upload. Add the following line to your `group_vars/galaxyserver.yml` file in the galaxy_config/galaxy section:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -122,6 +122,9 @@ galaxy_config:
>         sentry_dsn: "{{ vault_galaxy_sentry_dsn }}"
>         sentry_traces_sample_rate: 0.5
>         error_report_file: "{{ galaxy_config_dir }}/error_reports_file.yml"
>    +    # FTP
>    +    ftp_upload_dir: /data/uploads
>    +    ftp_upload_site: "{{ inventory_hostname }}"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    {% endraw %}
>    ```
>    {: data-commit="Add ftp vars in galaxy"}
>
> To check the other options for setting up ftp in Galaxy, please check the [Galaxy configuration documentation](https://docs.galaxyproject.org/en/master/admin/galaxy_options.html?highlight=ftp_upload_site#ftp-upload-dir).
>
> 5. Then we will set the different variables for proftpd. Add the following lines to your `group_vars/galaxyserver.yml` file. Please replace the PassivePorts below with the range of ports that are appropriate for your machine!
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -368,3 +368,24 @@ telegraf_plugins_extra:
>     tiaas_dir: /srv/tiaas
>     tiaas_admin_user: admin
>     tiaas_admin_pass: changeme
>    +
>    +# Proftpd:
>    +proftpd_galaxy_auth: yes
>    +galaxy_ftp_upload_dir: "{{ galaxy_config.galaxy.ftp_upload_dir }}"
>    +proftpd_display_connect: |
>    +  {{ inventory_hostname }} FTP server
>    +
>    +  Unauthorized access is prohibited
>    +proftpd_create_ftp_upload_dir: yes
>    +proftpd_options:
>    +  - User: galaxy
>    +  - Group: galaxy
>    +  - Port: 21
>    +proftpd_sql_db: galaxy@/var/run/postgresql
>    +proftpd_sql_user: galaxy
>    +proftpd_conf_ssl_certificate: /etc/ssl/certs/cert.pem
>    +proftpd_conf_ssl_certificate_key: /etc/ssl/user/privkey-proftpd.pem
>    +proftpd_global_options:
>    +  - PassivePorts: 56000 60000
>    +proftpd_use_mod_tls_shmcache: false
>    +proftpd_tls_options: NoSessionReuseRequired
>    {% endraw %}
>    ```
>    {: data-commit="Add proftpd variables"}
>
>    Here is a description of the set variables:
>
>    | Variable                           | Description                                                                                                  |
>    | ----------                         | -------------                                                                                                |
>    | `proftpd_galaxy_auth`              | Attempt to authenticate users against a Galaxy database.                                                     |
>    | `galaxy_ftp_upload_dir`            | Path to the Galaxy FTP upload directory, should match `ftp_upload_dir` in your Galaxy config.                |
>    | `proftpd_display_connect`          | Message to display when users connect to the FTP server. This should be the message, not the path to a file. |
>    | `proftpd_create_ftp_upload_dir`    | Whether to allow the role to create this with owner `galaxy_user`.                                           |
>    | `proftpd_options`                  | Any option for proftpd, we will just set up the user and group of the `galaxy_user`.                         |
>    | `proftpd_sql_db`                   | Database name to connect to for authentication info.                                                         |
>    | `proftpd_sql_user`                 | (default: the value of galaxy_user): Value of the username parameter to SQLConnectInfo.                      |
>    | `proftpd_conf_ssl_certificate`     | Path on the remote host where the SSL certificate file is.                                                   |
>    | `proftpd_conf_ssl_certificate_key` | Path on the remote host where the SSL private key file is.                                                   |
>    | `proftpd_global_options`           | Set arbitrary options in the <Global> context. We set here the PassivePorts range.                           |
>    | `proftpd_use_mod_tls_shmcache`     | By default proftpd uses `mod_tls_shmcache` which is not installed on the server so we just disable it.       |
>    | `proftpd_tls_options`              | Additional options for tls. We will use `NoSessionReuseRequired`                                             |
>
>    > <tip-title>Why NoSessionReuseRequired?</tip-title>
>    > `mod_tls` only accepts SSL/TLS data connections that reuse the SSL session of the control connection, as a security measure. Unfortunately, there are some clients (e.g. curl/Filezilla) which do not reuse SSL sessions.
>    > To relax the requirement that the SSL session from the control connection be reused for data connections we set `NoSessionReuseRequired`.
>    {: .tip}
>
> 6. Add the new role to the list of roles under the `roles` key in your playbook, `galaxy.yml`:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -45,6 +45,7 @@
>         - geerlingguy.redis
>         - usegalaxy_eu.flower
>         - galaxyproject.nginx
>    +    - galaxyproject.proftpd
>         - geerlingguy.docker
>         - usegalaxy_eu.rabbitmqserver
>         - galaxyproject.tiaas2
>    {% endraw %}
>    ```
>    {: data-commit="Add role to playbook"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in }
>
{: .hands_on}

Congratulations, you've set up FTP for Galaxy.

## Check it works

> <hands-on-title>Checking proftpd from the server</hands-on-title>
>
> 1. SSH into your machine
>
> 2. Check the active status of proftpd by `systemctl status proftpd`.
>
> 3. Check the port has been correctly attributed by `sudo lsof -i -P -n`.
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > You should see all the ports used by the server. What interests us is the line with proftpd.
>    > > You should see TCP *:21 (LISTEN).
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
> 4. Check the directory `/data/uploads/` has been created and is empty.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > sudo tree /data/uploads/
>    > ```
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

> <hands-on-title>Checking galaxy detected the ftp possibility</hands-on-title>
>
> 1. Open your galaxy in a browser.
>
> 2. Log in with a user (FTP is only possible for authenticated sessions).
>
> 3. Click on the upload button. You should now see on the bottom "Choose FTP files"
>
> 4. Click on the Choose FTP files button. You should see a message "Your FTP directory does not contain any files."
>
{: .hands_on}

It's working!

> <hands-on-title>Upload your first file</hands-on-title>
>
> There are three options for uploading files, you can choose whichever is easiest for you.
>
> 1. **FileZilla**
>
>    1. Follow the [tutorial](https://galaxyproject.org/ftp-upload/) to upload a file.
>    2. You will have a message which ask you to approve the certificate, approve it.
>
> 2. **lftp**
>
>    You can use locally lftp to test the ftp.
>
>    1. Install lftp with `sudo apt-get install lftp`.
>    2. Add the public certificate to the list of known certificates (only for LetsEncrypt Staging Certificates!):
>       > <code-in-title>Bash</code-in-title>
>       > ```
>       > mkdir .lftp
>       > echo "set ssl:ca-file \"/etc/ssl/certs/cert.pem\"" > .lftp/rc
>       > ```
>       {: .code-in}
>
>    3. Connect to the server with for example the admin account:
>       > <code-in-title>Bash</code-in-title>
>       > ```
>       > lftp admin@example.org@$HOSTNAME
>       > ```
>       {: .code-in}
>
>    4. Enter the password of the admin@example.org galaxy user.
>    5. Put a random file:
>
>       `put /srv/galaxy/server/CITATION`
>
>    6. Check it is there with `ls`.
>    7. Leave lftp with `quit`.
>    {: .tip}
>
> 3. **Curl**
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > curl -T {"/srv/galaxy/server/CITATION"} ftp://localhost --user admin@example.org:password --ssl -k
>    > ```
>    > Here `-T` says to upload a file, `--ssl` ensures that the FTP connection is SSL/TLS encrypted, and `-k` ignores any certificate issues as the hostname `localhost` will not match the certificate we have.
>    {: .code-in}
>
{: .hands_on}

> <hands-on-title>Check where the file has been uploaded</hands-on-title>
>
> 1. SSH into your machine
>
> 4. Check the directory `/uploads/`.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > sudo tree /uploads/
>    > ```
>    {: .code-in}
>
>    > <question-title></question-title>
>    >
>    > What do you see?
>    >
>    > > <solution-title></solution-title>
>    > > As I uploaded a file called `CITATION` with the admin@example.org user I see:
>    > > ```
>    > > /uploads/
>    > > └── admin@example.org
>    > >     └── CITATION
>    > > ```
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

> <hands-on-title>Use it in galaxy</hands-on-title>
>
> 1. Open your galaxy in a browser.
>
> 2. Log in with the user you used to upload the file.
>
> 3. Click on the upload button.
>
> 4. Click on the Choose FTP files button. You should see your file.
>
> 5. Click on it and click on Start to launch the upload. It should go to your history as a new dataset.
>
> 6. Click again on Choose FTP files button. Your file has disappeared. By default, the files are removed from the FTP at import.
>
>    > <tip-title>You want to change this behaviour?</tip-title>
>    > You just need to add `ftp_upload_purge: false` to the galaxy_config/galaxy variables (next to `ftp_upload_dir`).
>    {: .tip}
>
{: .hands_on}

Congratulations! Let your users know this is an option, many of them will prefer to start large uploads from an FTP client.

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=16 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="ftp" %}


topic: admin
tutorial: gxadmin
title: Galaxy Monitoring with gxadmin
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/gxadmin/tutorial.html
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["monitoring", "ansible", "git-gat"]
content_hash: 440e42fbd4c25e4227d77b2b7ce1f09d
last_modified: 2026-02-26T21:12:14.146043
workflows_json: []
content:
We will just briefly cover the features available in `gxadmin`, there are lots of queries that may or may not be useful for your Galaxy instance and you will have to read the documentation before using them.

It started life as a small shell script that Helena wrote because she couldn't remember what [Gravity](https://github.com/galaxyproject/gravity) was called or where it could be found. Some of the functions needed for things like swapping zerglings are still included in gxadmin but are highly specific to UseGalaxy.eu and not generally useful.

Since then it became the home for "all of the SQL queries we [galaxy admins] run regularly." @gtn:hexylena and @gtn:natefoo often shared SQL queries with each other in private chats, but this wasn't helpful to the admin community at large, so they decided to put them all in `gxadmin` and make it as easy to install as possible. We are continually trying to make this tool more generic and generally useful, if you notice something that's missing or broken, or have a new query you want to run, just [let us know](https://github.com/usegalaxy-eu/gxadmin/issues/new).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="gxadmin" %}

## Installing gxadmin

It's simple to install gxadmin. Here's how you do it, if you haven't done it already.

> <hands-on-title>Installing gxadmin with Ansible</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -28,3 +28,5 @@
>       version: 6.1.0
>     - name: usegalaxy_eu.rabbitmqserver
>       version: 1.4.1
>    +- src: galaxyproject.gxadmin
>    +  version: 0.0.12
>    {% endraw %}
>    ```
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add the role to your playbook:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -34,3 +34,4 @@
>         - galaxyproject.nginx
>         - galaxyproject.tusd
>         - galaxyproject.cvmfs
>    +    - galaxyproject.gxadmin
>    {% endraw %}
>    ```
>
> 4. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

With that, `gxadmin` should be installed! Now, test it out:

> <hands-on-title>Test out gxadmin</hands-on-title>
>
> 1. Run `gxadmin` as the galaxy user and list recently registered users:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sudo -u galaxy gxadmin query latest-users
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    >  id |          create_time          | disk_usage | username |       email        | groups | active
>    > ----+-------------------------------+------------+----------+--------------------+--------+--------
>    >   1 | 2021-06-09 12:25:59.299651+00 | 218 kB     | admin    | admin@example.org  |        | f
>    > (1 rows)
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

## Configuration

If `psql` runs without any additional arguments, and permits you to access your galaxy database then you do not need to do any more configuration for gxadmin.
Otherwise, you may need to set some of the [PostgreSQL environment variables](https://github.com/usegalaxy-eu/gxadmin#postgres)

## Overview

`gxadmin` has several categories of commands, each with different focuses. This is not a technically meaningful separation, it is just done to make the interface easier for end users.

Category      | Keyword             | Purpose
---           | --                  | --
Configuration | `config`            | Commands relating to galaxy's configuration files like XML validation.
Filters       | `filter`            | Transforming streams of text.
Galaxy Admin  | `galaxy`            | Miscellaneous galaxy related commands like a cleanup wrapper.
uWSGI         | `uwsgi`             | If you're using [systemd for Galaxy](https://github.com/usegalaxy-eu/ansible-galaxy-systemd/) and a handler/zergling setup, then this lets you manage your handlers and zerglings.
DB Queries    | `{csv,tsv,i,}query` | Queries against the database which return tabular output.
Report        | `report`            | Queries which return more complex and structured markdown reports.
Mutations     | `mutate`            | These are like queries, except they mutate the database. All other queries are read-only.
Meta          | `meta`              | More miscellaneous commands, and a built-in updating function.

## Admin Favourite Queries

**@gtn:slugger70's favourite**: `gxadmin query old-histories`. He contributed this function to find old histories, as their instance has a 90 day limit on histories, anything older than that might be automatically removed. This helps their group identify any histories that can be purged in order to save space. Running this on UseGalaxy.eu, we have some truly ancient histories, and maybe could benefit from a similar policy.

> <code-in-title></code-in-title>
> ```
> gxadmin query old-histories
> ```
{: .code-in}

> <code-out-title></code-out-title>
>
> id  |        update-time         | user-id | email |           name           | published | deleted | purged | hid-counter
> --- | -------------------------- | ------- | ----- | ------------------------ | --------- | ------- | ------ | ------------
> 361 | 2013-02-24 16:27:29.197572 |     xxx | xxxx  | Unnamed history          | f         | f       | f      | 6
> 362 | 2013-02-24 15:31:05.804747 |     xxx | xxxx  | Unnamed history          | f         | f       | f      | 1
> 347 | 2013-02-22 15:59:12.044169 |     xxx | xxxx  | Unnamed history          | f         | f       | f      | 19
> 324 | 2013-02-22 15:57:54.500637 |     xxx | xxxx  | Exercise 5               | f         | f       | f      | 64
> 315 | 2013-02-22 15:50:51.398894 |     xxx | xxxx  | day5 practical           | f         | f       | f      | 90
> 314 | 2013-02-22 15:45:47.75967  |     xxx | xxxx  | 5. Tag Galaxy-Kurs       | f         | f       | f      | 78
>
{: .code-out}

**@gtn:natefoo's favourite**: `gxadmin query job-inputs`. He contributed this function which helps him debug jobs which are not running and should be.

> <code-in-title></code-in-title>
> ```
> gxadmin query job-inputs 5 # Or another job ID
> ```
{: .code-in}

hda-id   | hda-state | hda-deleted | hda-purged |  d-id   | d-state | d-deleted | d-purged | object-store-id
-------- | --------- | ----------- | ---------- | ------- | ------- | --------- | -------- | ----------------
8638197  |           | f           | f          | 8246854 | running | f         | f        | files9
8638195  |           | f           | f          | 8246852 | running | f         | f        | files9
8638195  |           | f           | f          | 8246852 | running | f         | f        | files9

**@gtn:bgruening's favourite**: `gxadmin query latest-users` let's us see who has recently joined our server. We sometimes notice that people are running a training on our infrastructure and they haven't registered for [training infrastructure as a service](https://galaxyproject.eu/tiaas) which helps us coordinate infrastructure for them so they don't have bad experiences.

> <code-in-title></code-in-title>
> ```
> gxadmin query latest-users
> ```
{: .code-in}

id    |        create_time         | disk_usage | username | email | groups | active
----- | -------------------------- | ---------- | -------- | ----- | ------ | -------
3937  | 2019-01-27 14:11:12.636399 | 291 MB     | xxxx     | xxxx  |        | t
3936  | 2019-01-27 10:41:07.76126  | 1416 MB    | xxxx     | xxxx  |        | t
3935  | 2019-01-27 10:13:01.499094 | 2072 kB    | xxxx     | xxxx  |        | t
3934  | 2019-01-27 10:06:40.973938 | 0 bytes    | xxxx     | xxxx  |        | f
3933  | 2019-01-27 10:01:22.562782 |            | xxxx     | xxxx  |        | f

**@gtn:hexylena's favourite** `gxadmin report job-info`. This command gives more information than you probably need on the execution of a specific job, formatted as markdown for easy sharing with fellow administrators.

> <code-in-title></code-in-title>
> ```
> gxadmin report job-info 1
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```text
> # Galaxy Job 5132146
> 
> Property      | Value
> ------------- | -----
>          Tool | toolshed.g2.bx.psu.edu/repos/bgruening/canu/canu/1.7
>         State | running
>       Handler | handler_main_2
>       Created | 2019-04-20 11:04:40.854975+02 (3 days 05:49:30.451719 ago)
> Job Runner/ID | condor / 568537
>         Owner | e08d6c893f5
> 
> ## Destination Parameters
> 
> Key                     |   Value
> ---                     |   ---
> description             |   `canu`
> priority                |   `-128`
> request_cpus            |   `20`
> request_memory          |   `64G`
> requirements            |   `GalaxyGroup == "compute"`
> tmp_dir                 |   `True`
> 
> ## Dependencies
> 
> Name   |   Version   |   Dependency Type   |   Cacheable   |   Exact   |   Environment Path                          |   Model Class
> ---    |   ---       |   ---               |   ---         |   ---     |   ---                                       |   ---
> canu   |   1.7       |   conda             |   false       |   true    |   /usr/local/tools/_conda/envs/__canu@1.7   |   MergedCondaDependency
> 
> ## Tool Parameters
> 
> Name                 |   Settings
> ---------            |   ------------------------------------
> minOverlapLength     |   500
> chromInfo            |   /opt/galaxy/tool-data/shared/ucsc/chrom/?.len
> stage                |   all
> contigFilter         |   {lowCovDepth: 5, lowCovSpan: 0.5, minLength: 0, minReads: 2, singleReadSpan: 1.0}
> s                    |   null
> mode                 |   -nanopore-raw
> dbkey                |   ?
> genomeSize           |   300000
> corOutCoverage       |   40
> rawErrorRate         |
> minReadLength        |   1000
> correctedErrorRate   |
> 
> ## Inputs
> 
> Job ID    |   Name                |   Extension     |   hda-id    |   hda-state   |   hda-deleted   |   hda-purged   |   ds-id     |   ds-state   |   ds-deleted   |   ds-purged   |   Size
> ----      |   ----                |   ----          |   ----      |   ----        |   ----          |   ----         |   ----      |   ----       |   ----         |   ----        |   ----
> 4975404   |   Osur_record.fastq   |   fastqsanger   |   9517188   |               |   t             |   f            |   9015329   |   ok         |   f            |   f           |   3272 MB
> 4975404   |   Osur_record.fastq   |   fastqsanger   |   9517188   |               |   t             |   f            |   9015329   |   ok         |   f            |   f           |   3272 MB
> 
> ## Outputs
> 
> Name                                          |   Extension   |   hda-id    |   hda-state   |   hda-deleted   |   hda-purged   |   ds-id     |   ds-state   |   ds-deleted   |   ds-purged   |   Size
> ----                                          |   ----        |   ----      |   ----        |   ----          |   ----         |   ----      |   ----       |   ----         |   ----        |   ----
> Canu assembler on data 41 (trimmed reads)     |   fasta.gz    |   9520369   |               |   f             |   f            |   9018510   |   running    |   f            |   f           |
> Canu assembler on data 41 (corrected reads)   |   fasta.gz    |   9520368   |               |   f             |   f            |   9018509   |   running    |   f            |   f           |
> Canu assembler on data 41 (unitigs)           |   fasta       |   9520367   |               |   f             |   f            |   9018508   |   running    |   f            |   f           |
> Canu assembler on data 41 (unassembled)       |   fasta       |   9520366   |               |   f             |   f            |   9018507   |   running    |   f            |   f           |
> Canu assembler on data 41 (contigs)           |   fasta       |   9520365   |               |   f             |   f            |   9018506   |   running    |   f            |   f           |
> ```
{: .code-out}

**@gtn:cat-bro contributed** the 'jobs' query: `gxadmin query jobs` lets you list jobs that have been run on your Galaxy. It's a *lot* more flexible than `queue-overview` and we suggest using it instead, in most places. E.g. to find `circos` jobs that were recently run:


> <code-in-title></code-in-title>
> ```
> gxadmin query jobs --limit 2  --tool circos
> ```
{: .code-in}

job_id    |     create_time     |     update_time     | user_id | state |                            tool_id                            |    handler     | destination  | external_id 
--------- | ------------------- | ------------------- | ------- | ----- | ------------------------------------------------------------- | -------------- | ------------ | ------------
58483488  | 2023-04-04 18:42:40 | 2023-04-04 18:43:30 |         | error | toolshed.g2.bx.psu.edu/repos/iuc/circos/circos/0.69.8+galaxy8 | handler_sn06_5 | 1cores_10.0G | 42071736
58483208  | 2023-04-04 18:36:24 | 2023-04-04 18:40:43 |         | error | toolshed.g2.bx.psu.edu/repos/iuc/circos/circos/0.69.8+galaxy9 | handler_sn06_3 | 1cores_10.0G | 42071486

or to see recent jobs from a specific user (e.g. to help answer their email queries when they just send you a screenshot rather than a proper bug report)


> <code-in-title></code-in-title>
> ```
> gxadmin query jobs --limit 2 --user helena-rasche --terminal
> ```
{: .code-in}

job_id    |     create_time     |     update_time     | user_id | state |                                     tool_id                                      |    handler     | destination | external_id 
--------- | ------------------- | ------------------- | ------- | ----- | -------------------------------------------------------------------------------- | -------------- | ----------- | ------------
58277473  | 2023-03-31 09:53:36 | 2023-03-31 09:53:36 |     580 | ok    | __TAG_FROM_FILE__                                                                | _default_      |             | 
58277410  | 2023-03-31 09:47:16 | 2023-03-31 09:51:26 |     580 | ok    | toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/1.1.4 | handler_sn06_0 | 1cores_4.0G | 41859564

## `gxadmin` for Monitoring

`gxadmin` already supported `query`, `csvquery`, and `tsvquery` for requesting data from the Galaxy database in tables, CSV, or TSV formats, but we recently implemented `influx` queries which output data in a format that [Telegraf](https://github.com/influxdata/telegraf) can consume.

So running `gxadmin query queue-overview` normally shows something like:

> <code-in-title></code-in-title>
> ```
> gxadmin query queue-overview
> ```
{: .code-in}

tool_id                                                                                |  tool_version  |    destination_id    |     handler     |  state  | job_runner_name | count
-------------------------------------------------------------------------------------- | -------------- | -------------------- | --------------- | ------- | --------------- | ------
toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0                           | 0.4.6.0        | 12cores_180G_special | handler_main_4  | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0                           | 0.4.6.0        | 12cores_180G_special | handler_main_5  | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.1.0.46-0                    | 1.1.0.46-0     | 12cores_12G          | handler_main_3  | running | condor          |     2
toolshed.g2.bx.psu.edu/repos/iuc/qiime_extract_barcodes/qiime_extract_barcodes/1.9.1.0 | 1.9.1.0        | 4G_memory            | handler_main_1  | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/iuc/hisat2/hisat2/2.1.0+galaxy3                           | 2.1.0+galaxy3  | 8cores_20G           | handler_main_11 | running | condor          |     1
toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72                                | 0.72           | 20G_memory           | handler_main_11 | running | condor          |     4
ebi_sra_main                                                                           | 1.0.1          | 4G_memory            | handler_main_3  | running | condor          |     2
ebi_sra_main                                                                           | 1.0.1          | 4G_memory            | handler_main_4  | running | condor          |     3


The `gxadmin iquery queue-overview` is run by our Telegraf monitor on a regular basis, allowing us to consume the data:

> <code-in-title></code-in-title>
> ```
> gxadmin iquery queue-overview
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0,tool_version=0.4.6.0,state=running,handler=handler_main_4,destination_id=12cores_180G_special,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0,tool_version=0.4.6.0,state=running,handler=handler_main_5,destination_id=12cores_180G_special,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/freebayes/freebayes/1.1.0.46-0,tool_version=1.1.0.46-0,state=running,handler=handler_main_3,destination_id=12cores_12G,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/vcffilter/vcffilter2/1.0.0_rc1+galaxy1,tool_version=1.0.0_rc1+galaxy1,state=queued,handler=handler_main_11,destination_id=4G_memory,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/iuc/hisat2/hisat2/2.1.0+galaxy3,tool_version=2.1.0+galaxy3,state=running,handler=handler_main_11,destination_id=8cores_20G,job_runner_name=condor count=1
> queue-overview,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72,tool_version=0.72,state=running,handler=handler_main_11,destination_id=20G_memory,job_runner_name=condor count=4
> queue-overview,tool_id=ebi_sra_main,tool_version=1.0.1,state=running,handler=handler_main_3,destination_id=4G_memory,job_runner_name=condor count=2
> queue-overview,tool_id=ebi_sra_main,tool_version=1.0.1,state=running,handler=handler_main_4,destination_id=4G_memory,job_runner_name=condor count=3
> ```
{: .code-out}

And produce [some nice graphs](https://stats.galaxyproject.eu/) from it.

You can use an influx configuration like:

```toml
[[inputs.exec]]
    commands = ["/usr/bin/galaxy-queue-size"]
    timeout = "10s"
    data_format = "influx"
    interval = "1m"
```

This often requires a wrapper script, because you'll need to pass environment variables to the `gxadmin` invocation, e.g.:

```bash
#!/bin/bash
export PGUSER=galaxy
export PGHOST=dbhost
gxadmin iquery queue-overview --short-tool-id
gxadmin iquery workflow-invocation-status
```

> <tip-title>Which queries support iquery?</tip-title>
> This data is not currently exposed, so, just try the queries. But it's easy to add influx support when missing! [Here is an example](https://github.com/usegalaxy-eu/gxadmin/blob/e0ec0174ebbdce1acd8c40c7431308934981aa0c/parts/22-query.sh#L54), we set the variables in a function:
>
> ```
> fields="count=1"
> tags="tool_id=0"
> ```
>
> This means: column 0 is a tag named tool_id, and column 1 is a field (real value) named count.
> [Here is an example](https://github.com/usegalaxy-eu/gxadmin/blob/e0ec0174ebbdce1acd8c40c7431308934981aa0c/parts/22-query.sh#L1987) that has multiple fields that are stored.
>
{: .tip}

# Summary

There are a lot of queries, all tailored to specific use cases, some of these may be interesting for you, some may not. These are [all documented](https://github.com/usegalaxy-eu/gxadmin#commands) with example inputs and outputs in the gxadmin readme, and help is likewise available from the command line.

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=10 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="gxadmin" %}


topic: admin
tutorial: celeryless
title: Alternative Celery Deployment for Galaxy
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ansible"]
content_hash: bc4a2a81a80d9d6576ab2b0ff40a81a8
last_modified: 2026-02-26T21:12:14.138072
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/celeryless/tutorial.html

content:
Celery is a new component to the Galaxy world (ca 2023) and is a distributed task queue that *can* be used to run tasks asynchronously. It isn't mandatory, but you might find some features you expect to use to be missing without it.

If you are running a large production deployment you probably want to follow the [Celery+Redis+Flower Tutorial]({% link topics/admin/tutorials/celery/tutorial.md %}).

However, if you are running a smaller Galaxy you may not want to manage deploying Celery (past what Gravity does for you automatically), you may not want to add Redis to your stack, and you may not have need of Flower!

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Configuring Galaxy to use Postgres

AMQP is a message queue protocol which processes can use to pass messages between each other. While a real message queue like RabbitMQ is perhaps the most robust choice, there is an easier option: Postgres

Add the following to your Galaxy configuration to use Postgres:

```yaml
amqp_internal_connection: "sqlalchemy+postgresql:///galaxy?host=/var/run/postgresql"
```

# Configuring Celery to use Postgres

Celery would prefer you use Redis (a Key-Value store) as a backend to store results. But we have a database! So let's try using that instead:

```yaml
enable_celery_tasks: true
celery_conf:
  broker_url: null  # This should default to using amqp_internal_connection
  result_backend: "db+postgresql:///galaxy?host=/var/run/postgresql"
  task_routes:
    galaxy.fetch_data: galaxy.external
    galaxy.set_job_metadata: galaxy.external
```


With that we should now be able to [use useful features like](https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks):

- Changing the datatype of a collection.
- Exporting histories
- other things!

# Configuring with Ansible

If you're using Ansible, this could also look like:

```yaml
amqp_internal_connection: "sqlalchemy+{{ database_connection }}"
enable_celery_tasks: true
celery_conf:
  broker_url: null  # This should default to using amqp_internal_connection
  result_backend: "db+{{ database_connection }}"
  task_routes:
    galaxy.fetch_data: galaxy.external
    galaxy.set_job_metadata: galaxy.external
```


topic: admin
tutorial: customization
title: Customizing the look of Galaxy
difficulty: intermediate
time_estimation: 45m
tools_json: []
requirements_json: ["[]"]
tags_json: ["git-gat"]
content_hash: 140c7dc2b7caa1221de77bb5ac261675
last_modified: 2026-02-26T21:12:14.141271
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/tutorial.html

content:
Customizing your Galaxy instance makes it more recognizable at a glance, and can help communicate its purpose to its users.
This tutorial will teach you three basic customizations you can make to Galaxy:

  - Setting a brand text
  - Setting the contact information
  - Adding a custom welcome page
  - Customizing the masthead using themes

For this tutorial we will be setting up a fictional instance called "Galaxy Mars".
Feel free to use the included material in the hands-on sections, or provide your own.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Custom Branding

The brand text in Galaxy refers to the text you can see in the masthead of some Galaxy instances.
This text will appear in the masthead, as well as the sites title.
It is an easy way to set your instance apart, and make it more identifiable.

![screenshot of the start page of Galaxy Europe](images/galaxy-europe.png "Galaxy Europe uses the brand text \"Europe\"")

> <hands-on-title>Customising the Branding</hands-on-title>
> 
> 1. Open your `group_vars/galaxyserers.yml` and the following option under `galaxy_config.galaxy`:
> 
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -35,6 +35,9 @@ galaxy_job_config:
>
>     galaxy_config:
>       galaxy:
>    +    # Branding
>    +    brand: Mars 🚀
>    +    logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>         # Main Configuration
>         admin_users:
>         - admin@example.org
>    {% endraw %}
>    ```
>    {: data-commit="Add brand"}
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    ![screenshot of fictional "Galaxy Mars" start page, with the brand text set to "Mars"](images/galaxy-mars-brand.png "Your Galaxy start page should now look something like this")
{: .hands_on}

## Configuring Support

The Galaxy Help Site (https://help.galaxyproject.org) receives a lot of user support questions, sometimes for Galaxies that we do not manage. Including support information inside your Galaxy can help users find the right place to ask about issues with tools or quotas!

> <hands-on-title>Adding Support Information</hands-on-title>
>
> 1. Open your `group_vars/galaxyserers.yml` and the following option under `galaxy_config.galaxy`:
> 
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -38,6 +38,9 @@ galaxy_config:
>         # Branding
>         brand: Mars 🚀
>         logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>    +    # Support
>    +    support_url: "https://example.org/support"
>    +    terms_url: "https://example.org/terms-of-service"
>         # Main Configuration
>         admin_users:
>         - admin@example.org
>    {% endraw %}
>    ```
>    {: data-commit="Add support information"}
>
> Ideally this would point to a support URL and a terms of service that are appropriate to *your* deployment of Galaxy. You could, for instance, add a page on your group's blog with the appropriate contents of who should be contacted in case of issue, their email or a ticketing system. These options will help users from your site find you when they need help.
{: .hands_on}

## Custom Welcome Page

The welcome page is an html document embedded in Galaxy's start page.
It is what users see when first navigating to your instance.
This page can be used to communicate what your instance is about, and share news and updates with your users.

> <hands-on-title>Creating a custom welcome page</hands-on-title>
>
> 1. Set the location of your welcome page in `group_vars/galaxyservers.yml`
> 
>    Under `galaxy_config.galaxy` specify the location of your welcome page
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -99,6 +99,10 @@ galaxy_config:
>               - job-handlers
>               - workflow-schedulers
>
>    +galaxy_config_files_public:
>    +  - src: files/galaxy/welcome.html
>    +    dest: "{{ galaxy_mutable_config_dir }}/welcome.html"
>    +
>     galaxy_extra_dirs:
>       - /data
>
>    {% endraw %}
>    ```
>    {: data-commit="Add welcome url to the config"}
>
> 2. Let's then develop the associated template, add a title and some text to our welcome page:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/welcome.html
>    @@ -0,0 +1,11 @@
>    +<!DOCTYPE html>
>    +<html lang="en">
>    +    <head>
>    +        <meta charset="utf-8">
>    +        <link href="/static/dist/base.css" rel="stylesheet" type="text/css" />
>    +    </head>
>    +    <body>
>    +        <h1>Welcome to the Galaxy Mars instance!</h1>
>    +        <p>The only Galaxy instance on mars.</p>
>    +    </body>
>    +</html>
>    {% endraw %}
>    ```
>    {: data-commit="Add the template"}
>
>    This works the same as any other html page. You can add styles in the head, or import some scripts.
>
>    Make sure your welcome page has exactly one `h1` element, which describes the page.
>    This will act as the heading for your start-page, which can help assistive technologies.
>
> 1. Now that we have a proper welcome page, we'll also need to correct our nginx routes:
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -40,7 +40,7 @@ server {
>     	# automatically copied around. The welcome page is one of them. In
>     	# production, this step is skipped, so we will manually alias that.
>     	location /static/welcome.html {
>    -		alias {{ galaxy_server_dir }}/static/welcome.sample.html;
>    +		alias {{ galaxy_mutable_config_dir }}/welcome.html;
>     		expires 24h;
>     	}
>
>    {% endraw %}
>    ```
>    {: data-commit="Fix the nginx routes for the welcome page"}
>
> 1. Run the playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

> <tip-title>Use an IFrame!</tip-title>
> A lot of the UseGalaxy.* instances choose to use their `welcome.html` to display an IFrame to a different site,
> e.g. the GalaxyProject.org site, or a sub-site thereof.
>
> This makes it easier for non-administrators to keep the homepage's contents
> up to date. By embedding (for example) a Wordpress blog in the center,
> non-technical contributors can still publish articles and announcements
> without having to re-deploy Galaxy every time.
{: .tip}

> <tip-title>Styling your Welcome</tip-title>
> As in our example welcome.html, we've loaded the default Galaxy CSS to keep
> it consistent with the surrounding material. If you wish to load a different
> CSS file, or set a different font for the main panel, you can easily do that
> by adding whatever CSS you like.
{: .tip}

## Custom Masthead Theme

Galaxy's theming system allows you to easily change the color of your Masthead, to give your instance an even more distinct look.
You can even offer several options, to allow users to switch to the default if they prefer it, or another look all together.

> <hands-on-title>Configuring Themes</hands-on-title>
>
> 0. Set the location of the themes configuration in your `group_vars/galaxyservers.yml`
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -38,6 +38,7 @@ galaxy_config:
>         # Branding
>         brand: Mars 🚀
>         logo_src: "https://training.galaxyproject.org/training-material/topics/admin/tutorials/customization/images/logo.png"
>    +    themes_config_file: "{{ galaxy_config_dir }}/themes.yml"
>         # Support
>         support_url: "https://example.org/support"
>         terms_url: "https://example.org/terms-of-service"
>    @@ -103,6 +104,10 @@ galaxy_config_files_public:
>       - src: files/galaxy/welcome.html
>         dest: "{{ galaxy_mutable_config_dir }}/welcome.html"
>
>    +galaxy_config_files:
>    +  - src: files/galaxy/themes.yml
>    +    dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>    +
>     galaxy_extra_dirs:
>       - /data
>
>    {% endraw %}
>    ```
>    {: data-commit="Add brand"}
>
> 1. Create your themes config file in `files/galaxy/themes.yml`:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/files/galaxy/themes.yml
>    @@ -0,0 +1,76 @@
>    +blue:
>    +  masthead:
>    +    color: "#2c3143"
>    +    text:
>    +      color: "#f8f9fa"
>    +      hover: gold
>    +      active: white
>    +    link:
>    +      color: transparent
>    +      hover: transparent
>    +      active: "#181a24"
>    +    logo:
>    +      img: "/static/favicon.svg"
>    +      img-secondary: null
>    +
>    +lightblue:
>    +  masthead:
>    +    color: "#384E77"
>    +    text:
>    +      color: white
>    +      hover: "#E6F9AF"
>    +      active: white
>    +    link:
>    +      color: transparent
>    +      hover: transparent
>    +      active: "#18314F"
>    +    logo:
>    +      img: "/static/favicon.svg"
>    +
>    +pride:
>    +  masthead:
>    +      color: >
>    +        linear-gradient(120deg,
>    +          #3c476d 0px 130px,
>    +          #fdda0f 131px 139px,
>    +          #fff 140px 148px,
>    +          #f4b0c9 149px 157px,
>    +          #7ccee6 158px 166px,
>    +          #93540c 167px 175px,
>    +          #000 176px 184px,
>    +          transparent 185px),
>    +        linear-gradient(270deg,
>    +          #3c476d 0px 110px,
>    +          #3c476d00 110px),
>    +        linear-gradient(180deg,
>    +          #FE0000 16.66%,
>    +          #FD8C00 16.66% 33.32%,
>    +          #FFE500 33.32% 49.98%,
>    +          #119F0B 49.98% 66.64%,
>    +          #0644B3 66.64% 83.3%,
>    +          #C22EDC 83.3%)
>    +      text:
>    +        color: white
>    +        hover: gold
>    +        active: white
>    +      link:
>    +        color: "#3c476d"
>    +        hover: "#323a53"
>    +        active: "#6170a6"
>    +      logo:
>    +        img: "/static/favicon.svg"
>    +        img-secondary: null
>    +
>    +smoky:
>    +  masthead:
>    +    color: "#0C0F0A"
>    +    text:
>    +      color: white
>    +      hover: "#FBFF12"
>    +      active: white
>    +    link:
>    +      color: transparent
>    +      hover: transparent
>    +      active: "#FF206E"
>    +    logo:
>    +      img: "/static/favicon.svg"
>    {% endraw %}
>    ```
>    {: data-commit="Add themes file"}
>
>    The themes file can contains themes for users to select in Galaxy.
>
>    Each theme is identified by its id, eg `blue`, and followed by a set of
>    rules, which style the client. The first theme in your theme file will be
>    used as the default theme, which users see when visiting your instance,
>    and which users who haven't logged in will see.
>
> 3. Creating our own Theme
>
>    Let's create a theme more fitting for our Mars instance.
>
>    Create a new theme, by adding `mars:` to the top of the themes config file.
>    Now add `masthead:` underneath, and then `color:`.
>    Set color to `"#e03e1d"`, a nice deep martian red.
>
>    Your theme file should now begin with:
>
>    {% raw %}
>    ```diff
>    --- a/files/galaxy/themes.yml
>    +++ b/files/galaxy/themes.yml
>    @@ -1,3 +1,8 @@
>    +# Our Martian Theme
>    +mars:
>    +  masthead:
>    +    color: "#e03e1d"
>    +
>     blue:
>       masthead:
>         color: "#2c3143"
>    {% endraw %}
>    ```
>    {: data-commit="Add new default theme"}
>
> 3. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. You can also try logging in, and changing your theme under `Preferences`
{: .hands_on}

{% snippet topics/admin/faqs/missed-something.md step=3 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="customization" %}


topic: admin
tutorial: apptainer
title: Use Apptainer containers for running Galaxy jobs
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/apptainer/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["jobs", "ansible", "git-gat"]
content_hash: 9c0ca49bb184dbc562cd02b666e6bd43
last_modified: 2026-02-26T21:12:14.133434
workflows_json: []
content:
In this tutorial you will learn how to configure Galaxy to run jobs using [Apptainer](https://apptainer.org) containers provided by the [BioContainers](https://biocontainers.pro/) community.
Make sure to read the documentation on [container in Galaxy](https://docs.galaxyproject.org/en/master/admin/special_topics/mulled_containers.html) and in particular [container resolvers in Galaxy](https://docs.galaxyproject.org/en/master/admin/container_resolvers.html).


## Background

> BioContainers is a community-driven project that provides the infrastructure and basic guidelines to create, manage and distribute bioinformatics packages (e.g Conda) and containers (e.g Docker, Apptainer). BioContainers is based on the popular frameworks Conda, Docker and Apptainer.
>
{: .quote cite="https://biocontainers-edu.readthedocs.io/en/latest/what_is_biocontainers.html"}

Apptainer is an alternative to Docker that is much friendlier for HPCs

> <comment-title>Apptainer, Singularity, SingularityCE?</comment-title>
>
> Name                  | Singularity                                          | SingularityCE                                                  | Apptainer
> --------------------- | ---------------------------------------------------- | ---------------------                                          | ----------------
> Origin                | Original name, used by the project until 2021        | Name of Sylabs' Fork (CE for <b>C</b>ommunity <b>E</b>dition) | Name change when the project joined the Linux Foundation
> Status                | renamed in 2021                                      | currently active                                               | currently active
> RPM Package available | discontinued                                         | ❌                                                             | ✅
> CLI name              | `singularity`                                        | `singularity`                                                  | `apptainer` or `singularity`
>
> Many people still know Apptainer under its former name, Singularity.
> Singularity was forked in 2021, with the non-commercial fork being renamed to Apptainer and joining the Linux Foundation.
> Sylabs maintains their own free and open source fork, 'SingularityCE' (for <u>C</u>ommunity <u>E</u>dition), as well as a commercial version, 'SingularityPRO'.
>
> We will use the Name Apptainer for our training material, because most rpm packages are now named Apptainer.
{: .comment}

> Apptainer is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible.
{: .quote cite="https://sylabs.io/guides/3.7/user-guide/introduction.html"}

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="apptainer" %}

# Installing Apptainer

First, we will install Apptainer using Ansible. Since there is a package available for major Linux distros now, we could simply install it in the pre tasks. However, we would have to enable additional repos, so we decided to create a role for this:

> <hands-on-title>Installing Apptainer with Ansible</hands-on-title>
>
> 1. In your working directory, add the Apptainer role to your `requirements.yml` file:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -19,3 +19,6 @@
>     # CVMFS Support
>     - src: galaxyproject.cvmfs
>       version: 0.3.3
>    +# Singularity/Apptainer
>    +- src: usegalaxy_eu.apptainer
>    +  version: 0.0.3
>    {% endraw %}
>    ```
>    {: data-commit="Add Apptainer ansible roles"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the requirements with `ansible-galaxy`:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 4. Add the new roles to your `galaxy.yml` playbook, before the Galaxy server itself. We'll do this because it's a dependency of Galaxy to run, so it needs to be there before Galaxy starts.
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -31,6 +31,7 @@
>           when: ansible_os_family == 'Debian'
>       roles:
>         - galaxyproject.tusd
>    +    - usegalaxy_eu.apptainer
>         - galaxyproject.galaxy
>         - role: galaxyproject.miniconda
>           become: true
>    {% endraw %}
>    ```
>    {: data-commit="Add the roles to the playbook"}
>
> 5. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 6. Apptainer should now be installed on your Galaxy server. You can test this by connecting
>    to your server and run the following command:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > apptainer run docker://hello-world
>    > ```
>    {: .code-in}
>
>    > <code-out-title>Bash</code-out-title>
>    > ```
>    > INFO:    Converting OCI blobs to SIF format
>    > INFO:    Starting build...
>    > Getting image source signatures
>    > Copying blob 0e03bdcc26d7 done
>    > Copying config b23a8f6569 done
>    > Writing manifest to image destination
>    > Storing signatures
>    > 2021/01/08 11:25:12  info unpack layer: sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689
>    > INFO:    Creating SIF file...
>    > WARNING: passwd file doesn't exist in container, not updating
>    > WARNING: group file doesn't exist in container, not updating
>    >
>    > Hello from Docker!
>    > This message shows that your installation appears to be working correctly.
>    > ...
>    > ```
>    {: .code-out}
{: .hands_on}

## Configure Galaxy to use Apptainer

Now, we will configure Galaxy to run tools using Apptainer containers, which will be automatically fetched from [the BioContainers repository](https://quay.io/organization/biocontainers).  

> <warning-title>Galaxy Calls It Singularity, not Apptainer</warning-title>
> Galaxy still uses `singularity` in most variables, they will be replaced successively.
{: .warning}

> <hands-on-title>Configure Galaxy to use Apptainer</hands-on-title>
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add a `dependency_resolvers_config_file` entry and a corresponding `galaxy_config_templates` entry:
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add a `container_resolvers_config_file` entry and a corresponding `galaxy_config_templates` entry:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -78,6 +78,9 @@ galaxy_config:
>         tus_upload_store: "{{ galaxy_tus_upload_store }}"
>         # CVMFS
>         tool_data_table_config_path: /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml,/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml
>    +    # Tool Dependencies
>    +    dependency_resolvers_config_file: "{{ galaxy_config_dir }}/dependency_resolvers_conf.xml"
>    +    container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    @@ -117,6 +120,12 @@ galaxy_config_files:
>       - src: files/galaxy/themes.yml
>         dest: "{{ galaxy_config.galaxy.themes_config_file }}"
>
>    +galaxy_config_templates:
>    +  - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>    +    dest: "{{ galaxy_config.galaxy.container_resolvers_config_file }}"
>    +  - src: templates/galaxy/config/dependency_resolvers_conf.xml
>    +    dest: "{{ galaxy_config.galaxy.dependency_resolvers_config_file }}"
>    +
>     galaxy_extra_dirs:
>       - /data
>
>    {% endraw %}
>    ```
>    {: data-commit="Configure the container and dependency resolvers"}
>
> 2. Create the `templates/galaxy/config` directory if it doesn't exist:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > mkdir -p templates/galaxy/config
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Create the new file `templates/galaxy/config/dependency_resolvers_conf.xml`. This will not enable any dependency resolvers like the legacy toolshed packages or Galaxy packages, and instead everything will be resolved through Apptainer.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/galaxy/config/dependency_resolvers_conf.xml
>    @@ -0,0 +1,2 @@
>    +<dependency_resolvers>
>    +</dependency_resolvers>
>    {% endraw %}
>    ```
>    {: data-commit="Configure the dependency resolvers"}
>
> 3. Create the new file `templates/galaxy/config/container_resolvers_conf.yml.j2`, this specifies the order in which to attempt container resolution.
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/templates/galaxy/config/container_resolvers_conf.yml.j2
>    @@ -0,0 +1,11 @@
>    +
>    +- type: cached_explicit_singularity
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/explicit/"
>    +- type: cached_mulled_singularity
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/mulled/"
>    +- type: mulled_singularity
>    +  auto_install: False
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/mulled/"
>    +- type: build_mulled_singularity
>    +  auto_install: False
>    +  cache_directory: "{{ galaxy_mutable_data_dir }}/cache/singularity/built/"
>    {% endraw %}
>    ```
>    {: data-commit="Configure the container resolver"}
>
> 3. Now, we want to make Galaxy run jobs using Apptainer. Modify the file `group_vars/galaxyservers.yml`, by adding the `singularity_enabled` parameter:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -24,11 +24,26 @@ galaxy_job_config:
>       handling:
>         assign: ['db-skip-locked']
>       execution:
>    -    default: local_env
>    +    default: singularity
>         environments:
>           local_env:
>             runner: local_runner
>             tmp_dir: true
>    +      singularity:
>    +        runner: local_runner
>    +        singularity_enabled: true
>    +        # Enabling access to the reference data on CVMFS in the container
>    +        singularity_volumes: $defaults,/cvmfs/data.galaxyproject.org:ro
>    +        env:
>    +        # Ensuring a consistent collation environment is good for reproducibility.
>    +        - name: LC_ALL
>    +          value: C
>    +        # The cache directory holds the docker containers that get converted
>    +        - name: APPTAINER_CACHEDIR
>    +          value: /tmp/singularity
>    +        # Apptainer uses a temporary directory to build the squashfs filesystem
>    +        - name: APPTAINER_TMPDIR
>    +          value: /tmp
>       tools:
>         - class: local # these special tools that aren't parameterized for remote execution - expression tools, upload, etc
>           environment: local_env
>    {% endraw %}
>    ```
>    {: data-commit="Update the job_conf with singularity destination"}
>
> 4. Re-run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 5. In your Galaxy admin interface, install the minimap2 tool.
>
>    - Login to Galaxy as the admin user
>    - Click the "admin" menu at the top
>    - Under "Tool Management" on the left select "Install and Uninstall"
>    - search for `minimap2` and install the latest version with the Target Section "Mapping"
>
>    ![Screenshot of the install interface, minimap2 is entered in the search box and the latest revision shows it is currently cloning](../../images/install-minimap2.png)
>
> 6. Upload the following fasta file
>
>    ```
>    >testing
>    GATTACAGATHISISJUSTATESTGATTACA
>    ```
>
> 2. **Map with minimap2** {% icon tool %} with the following parameters
>    - *"Will you select a reference genome from your history or use a built-in index"*: `Use a genome from history and build index`
>    - *"Use the following dataset as the reference sequence"*: The fasta file you uploaded
>    - *"Single or Paired-end reads"*: `Single`
>        - {% icon param-file %} *"Select fastq dataset"*: The fasta file you uploaded
>
>    Your job should be executed using Apptainer with a BioContainer! You can watch the logs of Galaxy to see this happening.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > journalctl -f
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```
>    > gunicorn[1190010]: galaxy.tool_util.deps.containers INFO 2021-01-08 13:37:30,342 [p:1190010,w:0,m:2] [LocalRunner.work_thread-1] Checking with container resolver [MulledSingularityContainerResolver[namespace=biocontainers]] found description [ContainerDescription[identifier=docker://quay.io/biocontainers/mulled-v2-66534bcbb7031a148b13e2ad42583020b9cd25c4:e1ea28074233d7265a5dc2111d6e55130dff5653-0,type=singularity]]
>    > gunicorn[1190010]: galaxy.jobs.command_factory INFO 2021-01-08 13:37:30,418 [p:1190010,w:0,m:2] [LocalRunner.work_thread-1] Built script [/srv/galaxy/jobs/000/23/tool_script.sh] for tool command [minimap2 --version > /srv/galaxy/jobs/000/23/outputs/COMMAND_VERSION 2>&1; ln -f -s '/data/000/dataset_22.dat' reference.fa && minimap2           -t ${GALAXY_SLOTS:-4} reference.fa '/data/000/dataset_22.dat' -a | samtools sort -@${GALAXY_SLOTS:-2} -T "${TMPDIR:-.}" -O BAM -o '/data/000/dataset_23.dat' > '/data/000/dataset_23.dat']
>    > gunicorn[1190010]: galaxy.jobs.runners DEBUG 2021-01-08 13:37:30,441 [p:1190010,w:0,m:2] [LocalRunner.work_thread-1] (23) command is: mkdir -p working outputs configs
>    > gunicorn[1190010]: if [ -d _working ]; then
>    > gunicorn[1190010]:     rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
>    > gunicorn[1190010]: else
>    > gunicorn[1190010]:     cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
>    > gunicorn[1190010]: fi
>    > gunicorn[1190010]: cd working; SINGULARITYENV_GALAXY_SLOTS=$GALAXY_SLOTS SINGULARITYENV_HOME=$HOME SINGULARITYENV__GALAXY_JOB_HOME_DIR=$_GALAXY_JOB_HOME_DIR SINGULARITYENV__GALAXY_JOB_TMP_DIR=$_GALAXY_JOB_TMP_DIR SINGULARITYENV_TMPDIR=$TMPDIR SINGULARITYENV_TMP=$TMP SINGULARITYENV_TEMP=$TEMP singularity -s exec -B /srv/galaxy/server:/srv/galaxy/server:ro -B /srv/galaxy/var/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/minimap2/8c6cd2650d1f/minimap2:/srv/galaxy/var/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/minimap2/8c6cd2650d1f/minimap2:ro -B /srv/galaxy/jobs/000/23:/srv/galaxy/jobs/000/23 -B /srv/galaxy/jobs/000/23/outputs:/srv/galaxy/jobs/000/23/outputs -B /srv/galaxy/jobs/000/23/configs:/srv/galaxy/jobs/000/23/configs -B /srv/galaxy/jobs/000/23/working:/srv/galaxy/jobs/000/23/working -B /data:/data -B /srv/galaxy/var/tool-data:/srv/galaxy/var/tool-data:ro -B /srv/galaxy/var/tool-data:/srv/galaxy/var/tool-data:ro --home $HOME:$HOME docker://quay.io/biocontainers/mulled-v2-66534bcbb7031a148b13e2ad42583020b9cd25c4:e1ea28074233d7265a5dc2111d6e55130dff5653-0 /bin/bash /srv/galaxy/jobs/000/23/tool_script.sh > ../outputs/tool_stdout 2> ../outputs/tool_stderr; return_code=$?; cd '/srv/galaxy/jobs/000/23';
>    > ```
>    {: .code-out.code-max-300}
>
{: .hands_on}

> ```bash
> 1-run-minimap2.sh
> ```
> {: data-test="true"}
{: .hidden}

> <comment-title>Manage dependencies menu</comment-title>
> You can manually pull one or many containers for tools in the admin menu. Go to the admin menu, click Manage Dependencies and select the Containers tab. This will list all tools, their dependencies and whether containers are already pulled or can be pulled on demand.
>
> When a container has been resolved through Apptainer, you'll see something like this:
> ![Image of a table entry with minimap2 having requirements minimap2+singularity, a resolved column with a green checkmark next to via singularity, the resolver is mulled_singularity, and a container column with a path to /srv/galaxy/var/cache/singularity/mulled and some long hash.](../../images/singularity-resolved.png)
{: .comment}

> <tip-title>Apptainer, Conda, something else?</tip-title>
> We often hear
>
> > What would be the best practice, use Conda or Apptainer?
> {: .quote}
>
> Many of us are moving towards Apptainer. Conda environments can resolve differently if they were installed at different times, which isn't great for reproducibility. Apptainer images are never updated after generation which makes them fantastic. Also the isolation that's there by default is an incredible improvement for less-trustworthy binaries.
{: .tip}

> <tip-title>Does Apptainer fix issues with Conda dependencies resolution?</tip-title>
> Yes and no. Apptainer images are built from Conda environments. Only now you are no longer responsible for solving the conda environment, or ensuring that all of the dependencies are installed. The Galaxy project uses a system called "mulling" to bring together multiple Conda dependencies together in a single environment, and Apptainer images are produced for these dependencies as well. That said, complex or unresolvable Conda environments are not solved by Apptainer, because Apptainer is really just packaging Conda's environment into a single binary file.
{: .tip}


> <tip-title>Gateway Time-out (504) in Dependencies view</tip-title>
> When you open "Admin -> Tool Management -> Manage Dependencies -> Containers", it sometimes shows "Gateway Time-out (504)"
>
> Resolving all dependencies for all tools can take a bit, you can increase your timeout with the `proxy_read_timeout` setting in `templates/nginx/galaxy.j2`
{:.tip}

> <tip-title>Resolution is "unresolved"</tip-title>
> In "Admin -> Tool Management -> Manage Dependencies -> Dependencies", the Resolution for minimap2 @ 2.24 (as well as samtools @1.14) is "unresolved". How can I resolve this issue?
>
> Because our training uses containers for resolution it is expected that the non-container dependencies show as "unresolved". There is not currently a view which indicates if the containers have been resolved.
{: .tip}





<!--
## Use Apptainer containers from CVMFS

Galaxy can be configured to use pre-made Apptainer containers available from /cvmfs/singularity.galaxyproject.org/.
In order to do so, you will first need to set up CVMFS by doing the [CVMFS]({{ site.baseurl }}/topics/admin/tutorials/cvmfs/tutorial.html) tutorial.
After finishing the CVMFS tutorial, come back, and do this hands-on.

> <hands-on-title>Optional: Configure Galaxy to use Apptainer containers from CVMFS</hands-on-title>
>
> 1. Edit the `group_vars/galaxyservers.yml` file and add `container_resolvers_config_file` and `galaxy_singularity_images_cvmfs_path`:
>    {% raw %}
>    ```yaml
>    galaxy_singularity_images_cvmfs_path: "/cvmfs/singularity.galaxyproject.org/all/"
>    galaxy_config:
>      galaxy:
>        ...
>        container_resolvers_config_file: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>    ```
>    {% endraw %}
>
> 2. Also in `group_vars/galaxyservers.yml`, add a `galaxy_config_templates` entry:
>    {% raw %}
>    ```yaml
>    galaxy_config_templates:
>      - src: templates/galaxy/config/container_resolvers_conf.yml.j2
>        dest: "{{ galaxy_config_dir }}/container_resolvers_conf.yml"
>    ```
>    {% endraw %}
>
> 3. Create the new file `templates/galaxy/config/container_resolvers_conf.yml.j2`:
>    {% raw %}
>    ```xml
>    <containers_resolvers>
>        <explicit_singularity />
>        <cached_mulled_singularity cache_directory="{{ galaxy_singularity_images_cvmfs_path }}"/>
>    </containers_resolvers>
>    ```
>    {% endraw %}
>
> 4. Re-run the playbook (`ansible-playbook galaxy.yml`)
>
> 5. Rerun your minimap2 job from the previous section. This time it should use a container served by CVMFS.
>
{: .hands_on}

> ```bash
> 1-run-minimap2.sh
> ```
> {: data-test="true"}
{: .hidden}

-->

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=6 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="apptainer" %}


topic: admin
tutorial: backup-cleanup
title: Server Maintenance: Cleanup, Backup, and Restoration
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["ansible", "deploying", "git-gat"]
content_hash: 1657abf9428ebc68ee867f328c90dbd5
last_modified: 2026-02-26T21:12:14.133897
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/backup-cleanup/tutorial.html

content:
Keeping your Galaxy cleaned up is an important way to retain space, especially since for many groups that is the
limiting factor in their deployment.

Additionally, backups are necessary to ensure that if you ever experience system level failures, you can safely recover
from these.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="backup-cleanup" %}

# Cleanups

There are two kinds of data that are produced when running a Galaxy: files users create and then delete or purge, and
then files Galaxy creates itself. Both of these can be cleaned to save space.

## User Created Files

You can use `gxadmin` to cleanup user created files. `gxadmin` is covered in more detail in [its own dedicated
tutorial]({% link topics/admin/tutorials/gxadmin/tutorial.md %}).

> <hands-on-title>Installing gxadmin with Ansible</hands-on-title>
>
> 1. Edit your `requirements.yml` and add the following:
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -11,3 +11,6 @@
>       version: 0.3.4
>     - src: usegalaxy_eu.certbot
>       version: 0.1.13
>    +# gxadmin (used in cleanup, and later monitoring.)
>    +- src: galaxyproject.gxadmin
>    +  version: 0.0.14
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Add the role to your playbook:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -27,3 +27,4 @@
>           become: true
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>    +    - galaxyproject.gxadmin
>    {% endraw %}
>    ```
>    {: data-commit="Add the gxadmin role"}
>
> 3. Setup a cleanup task to run regularly:
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -28,3 +28,11 @@
>           become_user: "{{ galaxy_user_name }}"
>         - galaxyproject.nginx
>         - galaxyproject.gxadmin
>    +  post_tasks:
>    +    - name: Setup gxadmin cleanup task
>    +      ansible.builtin.cron:
>    +        name: "Cleanup Old User Data"
>    +        user: galaxy # Run as the Galaxy user
>    +        minute: "0"
>    +        hour: "0"
>    +        job: "SHELL=/bin/bash source {{ galaxy_venv_dir }}/bin/activate &&  GALAXY_LOG_DIR=/tmp/gxadmin/ GALAXY_ROOT={{ galaxy_root }}/server GALAXY_CONFIG_FILE={{ galaxy_config_file }} /usr/local/bin/gxadmin galaxy cleanup 60"
>    {% endraw %}
>    ```
>    {: data-commit="Configure gxadmin to cleanup data"}
>
>    This will cause datasets deleted for more than 60 days to be purged.
>
> 4. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

Whenever `gxadmin` runs, it will create logs you can read in `/tmp/gxadmin` which you can check later.

## Galaxy Created Files

Before we begin backing up our Galaxy data, let's set up automated cleanups to ensure we backup the minimal required set of data.

> <hands-on-title>Configuring PostgreSQL Backups</hands-on-title>
>
> 1. Edit `galaxy.yml` to install `tmpwatch` (if using RHEL/CentOS/Rocky) and `tmpreaper` if using Debian/Ubuntu
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -21,6 +21,14 @@
>         - name: Install Dependencies
>           package:
>             name: ['acl', 'bzip2', 'git', 'make', 'tar', 'python3-venv', 'python3-setuptools']
>    +    - name: Install RHEL/CentOS/Rocky specific dependencies
>    +      package:
>    +        name: ['tmpwatch']
>    +      when: ansible_os_family == 'RedHat'
>    +    - name: Install Debian/Ubuntu specific dependencies
>    +      package:
>    +        name: ['tmpreaper']
>    +      when: ansible_os_family == 'Debian'
>       roles:
>         - galaxyproject.galaxy
>         - role: galaxyproject.miniconda
>    {% endraw %}
>    ```
>    {: data-commit="Install tmpwatch/tmpreaper"}
>
> 1. Edit `group_vars/galaxyservers.yml` and add some variables to configure PostgreSQL:
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -3,6 +3,7 @@ galaxy_create_user: true # False by default, as e.g. you might have a 'galaxy' u
>     galaxy_separate_privileges: true # Best practices for security, configuration is owned by 'root' (or a different user) than the processes
>     galaxy_manage_paths: true # False by default as your administrator might e.g. have root_squash enabled on NFS. Here we can create the directories so it's fine.
>     galaxy_manage_systemd: true
>    +galaxy_manage_cleanup: true
>     galaxy_layout: root-dir
>     galaxy_root: /srv/galaxy
>     galaxy_user: {name: "{{ galaxy_user_name }}", shell: /bin/bash}
>    {% endraw %}
>    ```
>    {: data-commit="Configure automated cleanup"}
>
> 2. > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Check out the cleanup task which has been generated in: `/etc/cron.d/ansible_galaxy_tmpclean`
>
{: .hands_on}

This will setup `tmpwatch` to cleanup a few folders:

- the job working directory, important if you set `cleanup: onsuccess`, to cleanup old failed jobs once you're done debugging their failures.
- the new file upload path, to catch uploaded temporary files that are no longer necessary.

# Backups

There are a few important things to back up with your Ansible Galaxy:

- Galaxy
	- The Galaxy-managed config files
	- The playbooks
- The Database
- The Data

## Galaxy

By using Ansible, as long as you are storing your playbooks on another system, you are generally safe from failues of
the Galaxy node, and you'll be able to re-run your playbook at a later date.

However, playbooks often do not include:

- Which tools you've installed (have you ever installed a tool outside of ephemeris? This might be lost!)
- Conda environments, which will not always resolve identically over time. If strong guarantees of reproducibility are
  important, then consider backing these up as well.

## Database Backups

We're setting a couple of variables to control the automatic backups, they'll be placed in the `/data/backups` folder next to our user uploaded Galaxy data.

> <hands-on-title>Configuring PostgreSQL Backups</hands-on-title>
>
> 1. Edit `group_vars/dbservers.yml` and add some variables to configure PostgreSQL:
>    {% raw %}
>    ```diff
>    --- a/group_vars/dbservers.yml
>    +++ b/group_vars/dbservers.yml
>    @@ -5,3 +5,7 @@ postgresql_objects_users:
>     postgresql_objects_databases:
>       - name: "{{ galaxy_db_name }}"
>         owner: "{{ galaxy_user_name }}"
>    +
>    +# PostgreSQL Backups
>    +postgresql_backup_dir: /data/backups
>    +postgresql_backup_local_dir: "{{ '~postgres' | expanduser }}/backups"
>    {% endraw %}
>    ```
>    {: data-commit="Add backups"}
>
>    <!-- TODO: expanduser tip here -->
{: .hands_on}

This will setup our backups to run as a cron job. 

<!-- TODO: explore cron jobs -->

## Data Backup

With Galaxy it is *technically* only necessary to backup your inputs, as the downstream files *should*, *in theory* be re-createable due to the reproducibility of Galaxy.

In practice, some groups either choose to not backup, or to backup everything, often to extremely cheap and slow storage like Glacier or a tape library.

Most groups choose to implement this as a custom cron job, e.g.

```yaml
post_tasks:
  - name: Setup backup cron job
    ansible.builtin.cron:
      name: "Backup User Data"
      minute: "0"
      hour: "5,2"
      job: "rsync -avr /data/galaxy/ backup@backup.example.org:/backups/$(date -I)/"
```

> <tip-title>This isn't a backup strategy!</tip-title>
> People who, let's say, care strongly about backups will often insist that you need to version files. This is of course unnecessary in the Galaxy case as files are essentially {WORM}s, which is a really good file storage practice. Files can get removed so it isn't a true {WORM} strategy that you'd use for e.g. audit logs, but it is close.
> That said, since files never get changed, keeping multiple versions is unnecesary.
>
> Please consider communicating **very well** with your users what the data backup policy is.
{: .tip}

{% snippet topics/admin/faqs/missed-something.md step=2 %}

# Restoration

Sometimes failures happen! We're sorry you have to read this section.

## Restoring the Database

This procedure is more complicated, you can [read about the restoration procedure](https://github.com/galaxyproject/ansible-postgresql/pull/30#issuecomment-963600656) in the associated PR.

This step assumes you have pre-existing backups in place, you must check this first:

```console
ls /data/backups/
```

If you have backups, you're ready to restore:

```console
# Stop Galaxy, you do NOT want galaxy to connect mid-restoration in case it
# tries to modify the database.
sudo systemctl stop galaxy

# Stop the database
sudo systemctl stop postgresql
# Ensure that it is stopped
sudo systemctl status postgresql

# Begin the backup procedure by becoming postgres:
sudo su - postgres

# Move the current, live database to a backup location just in case:
mkdir /tmp/test/

# ====
# NOTE THAT THIS NUMBER MAY BE DIFFERENT FOR YOU!
# You will need to change 12 to whatever version of postgres you're running
# in every subsequent command
# ====
mv /var/lib/postgresql/12/main/* /tmp/test/

# Add backup
rsync -av /data/backups/YOUR_LATEST_BACKUP/ /var/lib/postgresql/12/main
# Add the restore_command, to your backup file:
# restore_command = 'cp "/tmp/backup/current/wal/%f" "%p"'
$EDITOR ./12/main/postgresql.auto.conf

# Touch a recovery file
touch /var/lib/postgresql/12/main/recovery.signal

# As $username (with sudo right)
sudo systemctl restart postgresql
sudo systemctl status postgresql
# Restart Galaxy
sudo systemctl start galaxy
```

If you encounter issues, we suggest reading [Lucille's log of her experiences restoring](https://github.com/lldelisle/galaxyduboule-infrastructure/blob/778e5b056f03970a1af5fc2f9dd133b4e4791cac/testUpgradeOnVM.sh#L53-L130) as you might encounter similar issues.

## Restoring Galaxy

Restoring Galaxy is easy via Ansible (maybe ensuring users cannot login by disabling the routes in nginx)

```console
ansible-playbook galaxy.yml
```

And if you are following best practices, you probably have your tools stored in a YAML file to use with [Ephemeris]({% link topics/admin/tutorials/tool-management/tutorial.md %}):

```console
shed-tools install -g https://galaxy.example.org -a <api-key> -t our_tools.yml
```

## Restoring User Data

This should simply be `rsync`ing your data from the backup location back into `/data/galaxy`.

{% snippet topics/admin/faqs/git-gat-path.md tutorial="backup-cleanup" %}


topic: admin
tutorial: wireguard-headscale
title: Deploying Tailscale/Headscale for private mesh networking
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: ["wireguard", "networking"]
content_hash: a5a136bad484eb94007d027130bd55cf
last_modified: 2026-02-26T21:12:14.168642
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/wireguard-headscale/tutorial.html

content:
[Tailscale](https://tailscale.com/) makes secure networking easy, it really is like magic. If you've used wireguard before, you know it takes a bit to setup and some configuration if you need to do anything fancy.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# What is Tailscale?

It's like Wireguard but easier and they built a **lot** of nice features on top of it. It's networking that "Just Works™" even more than Wireguard. If you prefer to use plain Wireguard without Headscale/Tailscale, or just want to get an understanding of the technology that Headscale/Tailscale build off of, [there is a tutorial for that]({% link topics/admin/tutorials/wireguard/tutorial.md %}) as well.

## Is it right for me?

if you have machines that need to talk to each other privately, and you don't
have a better way to do it like a local network team, then yes, it's a great
solution to private, secure, fast networking. if you need auditing, tailscale
will do that, rather than you having to build it out yourself.
it has [excellent performance](https://www.wireguard.com/performance/)
despite the encryption, and is built directly into the kernel.

By using wireguard, you can let services listen only on the wireguard
interface, and thus only known and trusted machines can access those services.

Tailscale makes wireguard setup even easier by removing the key management
step, which normally requires distributing keys to every machine. Instead that
step is handled centrally, and in the case of Tailscale enforceable with ACLs
and SSO and 2FA policies, however the networking remains meshed, and machines
connect directly to one another.

You can go one step further than trusting individual machines, with Tailscale,
as every device is tied to a single user, and you can ask Tailscale what is the
authenticated identity of the specific TCP connection, allowing automatically
logging in your users.

In the context of Galaxy, this can be useful for components like [Interactive 
Tools]({% link topics/admin/tutorials/interactive-tools/tutorial.md %}), which 
require a web proxy between Galaxy and the cluster node where the tool runs. If 
the cluster is not on the local network, Wireguard can be used to securely 
bridge the gap, and Headscale or Tailscale can greatly simplify that process.
# Setting up the infrastructure

{% include _includes/cyoa-choices.html option1="Tailscale" option2="Headscale" default="Tailscale" help="Here we provide a version of this tutorial that uses Headscale, if you want a more hands-on experience and feel like managing your own infrastructure." %}

> <hands-on-title>Configuration files</hands-on-title>
>
> 1. Create a `ansible.cfg` file (next to your playbook) to [configure settings](https://docs.ansible.com/ansible/2.9/reference_appendices/config.html) like the inventory file (and save ourselves some typing!), or the Python interpreter to use:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/ansible.cfg
>    @@ -0,0 +1,6 @@
>    +[defaults]
>    +interpreter_python = /usr/bin/python3
>    +inventory = hosts
>    +retry_files_enabled = false
>    +[ssh_connection]
>    +pipelining = true
>    {% endraw %}
>    ```
>    {: data-commit="Add ansible.cfg"}
>
> 2. Create the `hosts` inventory file if you have not done so yet.
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > cat hosts
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Bash</code-out-title>
>    > >
>    > > Your hostname is probably different:
>    > > <div class="Headscale" markdown="1">
>    > > Pick one to be the `head`, and the rest to be the `tail`. The head will act as the coordination server, the `tail` will be the nodes that should talk to each other.
>    > > {% raw %}
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,5 @@
>    > > +[head]
>    > > +1-wg.galaxy.training
>    > > +[tail]
>    > > +2-wg.galaxy.training
>    > > +3-wg.galaxy.training
>    > > +4-wg.galaxy.training
>    > > {% endraw %}
>    > > ```
>    > > {: data-commit="Add hosts"}
>    > > </div>
>    > >
>    > > <div class="Tailscale" markdown="1">
>    > > Place all of the nodes in a `tail` group
>    > > ```diff
>    > > --- /dev/null
>    > > +++ b/hosts
>    > > @@ -0,0 +1,5 @@
>    > > +[tail]
>    > > +1-wg.galaxy.training
>    > > +2-wg.galaxy.training
>    > > +3-wg.galaxy.training
>    > > +4-wg.galaxy.training
>    > > ```
>    > > </div>
>    > {: .code-out}
>    {: .code-2col}
{: .hands_on}

<div class="Headscale" markdown="1">

> <details-title>Consider using Tailscale</details-title>
> Tailscale has implemented the coordination server and infrastructure with much better, more robust infrastructure that you won't have to be responsible for. If you can get your institution to pay for it, or grant money covering it, it's **probably worth it**. They add many new features often, and things like the mobile apps only work with their service. Tailscale is free for personal use (i.e. to test things out) and offers a free plan for open source projects that you may qualify for.
>
> For a training event obviously we want something free and quick to setup and destroy, so, we're using [Headscale](https://github.com/juanfont/headscale) since it's free and we're just going to destroy it immediately, and no one will accidentally get billed ;)
>
> Using Headscale will also teach you everything you need to know if you do choose to use Tailscale, which is simpler and has fewer components for you to manage yourself.
{: .details}

> <hands-on-title>Installing Headscale</hands-on-title>
>
> 1. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles ckstevenson.headscale
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Create and open `head.yml` which will be our playbook. Add the following:
>
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/head.yml
>    @@ -0,0 +1,16 @@
>    +---
>    +- name: Headscale
>    +  hosts: head
>    +  become: true
>    +  vars:
>    +    headscale_user: 'headscale'
>    +    headscale_version: '0.15.0'
>    +    headscale_namespaces:
>    +    - galaxy
>    +  roles:
>    +    - ckstevenson.headscale
>    +  post_tasks:
>    +    - command: headscale --namespace galaxy preauthkeys create --reusable --expiration 1h
>    +      register: authkey
>    +    - debug:
>    +        msg: "{{ authkey.stdout.split('\n')[-1] }}"
>    {% endraw %}
>    ```
>    {: data-commit="Setup headscale"}
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook headscale.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. This will return a code in the debug output. Save this code, you'll need it shortly
{:.hands_on}
</div>

Now we can setup the nodes

> <hands-on-title>Configure the nodes</hands-on-title>
>
> 1. Install the role
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles artis3n.tailscale
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 1. Edit `tail.yml` and add the following.
>
>    <div class="Headscale" markdown="1">
>    {% raw %}
>    ```diff
>    --- /dev/null
>    +++ b/tail.yml
>    @@ -0,0 +1,17 @@
>    +---
>    +- name: Tailscale
>    +  hosts: tail
>    +  become: true
>    +  vars:
>    +    tailscale_args: "--advertise-exit-node --login-server http://{{ hostvars[groups['head'][0]].inventory_hostname }}:8080"
>    +  pre_tasks:
>    + - sysctl:
>    +     name: net.ipv4.ip_forward
>    +     value: '1'
>    +     state: present
>    + - sysctl:
>    +     name: net.ipv6.conf.all.forwarding
>    +     value: '1'
>    +     state: present
>    +  roles:
>    +    - artis3n.tailscale
>    {% endraw %}
>    ```
>    {: data-commit="Setup the clients"}
>    </div>
>
>    <div class="Tailscale" markdown="1">
>    ```diff
>    --- /dev/null
>    +++ b/tail.yml
>    @@ -0,0 +1,17 @@
>    +---
>    +- name: Tailscale
>    +  hosts: tail
>    +  become: true
>    +  vars:
>    +    tailscale_args: "--advertise-exit-node"
>    +  pre_tasks:
>    + - sysctl:
>    +     name: net.ipv4.ip_forward
>    +     value: '1'
>    +     state: present
>    + - sysctl:
>    +     name: net.ipv6.conf.all.forwarding
>    +     value: '1'
>    +     state: present
>    +  roles:
>    +    - artis3n.tailscale
>    ```
>    </div>
>
> 2. Run the playbook:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook tail.yml -e tailscale_authkey=YOUR_CODE
>    > ```
>    > {: data-cmd="true"}
>    >
>    > <div class="Headscale" markdown="1">
>    > Remember, you can find this code from the output of the first playbook
>    > </div>
>    > <div class="Tailscale" markdown="1">
>    > You can generate an authentication key under your [Tailscale account page](https://login.tailscale.com/admin/authkeys)
>    > </div>
>    {: .code-in}
>
> 3. Go check out your tailnet! Play around with the tailscale command and pinging other nodes with the suffix `.galaxy.example.org`{:.Headscale} `<username>.org.github.beta.tailscale.net`{:.Tailscale}
>
>    > Note that you'll need to enable [MagicDNS](https://login.tailscale.com/admin/dns) in your acount settings.
>    {:.Tailscale}
{:.hands_on}

> <tip-title>Exit Nodes</tip-title>
> We've configured `--advertise-exit-node`, which means you can direct ALL of your traffic to use one of your tailscale endpoints as an exit node, just run `tailscale up --exit-node=...`
>
> Note that:
> - If you're using headscale you need to manually enable that route (check the node list via `headscale nodes list` and then enable the specific route via `headscale nodes routes enable -i ...`), this is automatic in Tailscale
> - If you enable it, on a remote machine, it will immediately become unresponsive. Only do this on your local machine, e.g. a laptop connected to your tailnet.
{: .tip}


topic: admin
tutorial: tus
title: Performant Uploads with TUS
url: https://training.galaxyproject.org/training-material/topics/admin/tutorials/tus/tutorial.html
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 1631939404deb78ca800779fa6c3ca0d
last_modified: 2026-02-26T21:12:14.167546
workflows_json: []
content:
Here you'll learn to setup [TUS](https://tus.io/) an open source resumable file upload server to process uploads for Galaxy. We use an external process here to offload the main Galaxy processes for more important work and not impact the entire system during periods of heavy uploading.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tus" %}

# TUS and Galaxy

To allow your user to upload via TUS, you will need to:

- configure Galaxy to know where the files are uploaded.
- install TUSd
- configure Nginx to proxy TUSd

## Installing and Configuring

> <hands-on-title>Setting up ftp upload with Ansible</hands-on-title>
>
> 1. In your playbook directory, add the `galaxyproject.tusd` role to your `requirements.yml`
>
>    {% raw %}
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -14,3 +14,5 @@
>     # gxadmin (used in cleanup, and later monitoring.)
>     - src: galaxyproject.gxadmin
>       version: 0.0.14
>    +- name: galaxyproject.tusd
>    +  version: 0.0.1
>    {% endraw %}
>    ```
>    {: data-commit="Add requirement"}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 2. Install the role with:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-galaxy install -p roles -r requirements.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Configure it in your group variables
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -73,6 +73,9 @@ galaxy_config:
>         # Tool security
>         outputs_to_working_directory: true
>         new_user_dataset_access_role_default_private: true # Make datasets private by default
>    +    # TUS
>    +    galaxy_infrastructure_url: "https://{{ inventory_hostname }}"
>    +    tus_upload_store: "{{ galaxy_tus_upload_store }}"
>       gravity:
>         process_manager: systemd
>         galaxy_root: "{{ galaxy_root }}/server"
>    @@ -93,6 +96,10 @@ galaxy_config:
>         celery:
>           concurrency: 2
>           loglevel: DEBUG
>    +    tusd:
>    +      enable: true
>    +      tusd_path: /usr/local/sbin/tusd
>    +      upload_dir: "{{ galaxy_tus_upload_store }}"
>         handlers:
>           handler:
>             processes: 2
>    @@ -162,3 +169,7 @@ nginx_conf_http:
>     nginx_ssl_role: usegalaxy_eu.certbot
>     nginx_conf_ssl_certificate: /etc/ssl/certs/fullchain.pem
>     nginx_conf_ssl_certificate_key: /etc/ssl/user/privkey-www-data.pem
>    +
>    +# TUS
>    +galaxy_tusd_port: 1080
>    +galaxy_tus_upload_store: /data/tus
>    {% endraw %}
>    ```
>    {: data-commit="Configure TUS in your group variables"}
>
> 4. We proxy the service next to Galaxy. As it resides "under" the Galaxy path, clients will send cookies and authentication headers to TUS, which it can use to process the uploads before telling Galaxy when they're done.
>
>    {% raw %}
>    ```diff
>    --- a/templates/nginx/galaxy.j2
>    +++ b/templates/nginx/galaxy.j2
>    @@ -28,6 +28,22 @@ server {
>     		proxy_set_header Upgrade $http_upgrade;
>     	}
>
>    +	location /api/upload/resumable_upload {
>    +		# Disable request and response buffering
>    +		proxy_request_buffering     off;
>    +		proxy_buffering             off;
>    +		proxy_http_version          1.1;
>    +
>    +		# Add X-Forwarded-* headers
>    +		proxy_set_header X-Forwarded-Host   $host;
>    +		proxy_set_header X-Forwarded-Proto  $scheme;
>    +
>    +		proxy_set_header Upgrade            $http_upgrade;
>    +		proxy_set_header Connection         "upgrade";
>    +		client_max_body_size        0;
>    +		proxy_pass http://localhost:{{ galaxy_tusd_port }}/files;
>    +	}
>    +
>     	# Static files can be more efficiently served by Nginx. Why send the
>     	# request to Gunicorn which should be spending its time doing more useful
>     	# things like serving Galaxy!
>    {% endraw %}
>    ```
>    {: data-commit="Proxy it via NGINX"}
>
> 5. Add to the end of your Galaxy playbook
>
>    {% raw %}
>    ```diff
>    --- a/galaxy.yml
>    +++ b/galaxy.yml
>    @@ -30,6 +30,7 @@
>             name: ['tmpreaper']
>           when: ansible_os_family == 'Debian'
>       roles:
>    +    - galaxyproject.tusd
>         - galaxyproject.galaxy
>         - role: galaxyproject.miniconda
>           become: true
>    {% endraw %}
>    ```
>    {: data-commit="Add the role to the playbook"}
>
> 6. Run the playbook
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ansible-playbook galaxy.yml
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in }
>
{: .hands_on}

Congratulations, you've set up TUS for Galaxy.

## Check it works

> <hands-on-title>Check that it works.</hands-on-title>
>
> 1. SSH into your machine
>
> 2. Check the active status of tusd by `systemctl status galaxy-tusd`.
>
> 3. Follow the tusd logs with `journalctl -fu galaxy-tusd`
>
> 4. Upload a small file! (Pasted text will not pass via TUS)
>
> 5. Make sure you see `event=UploadFinished` in the tusd logs
>
{: .hands_on}

> ```bash
> 1.sh
> ```
> {: data-test="true"}
{: .hidden}

{% snippet topics/admin/faqs/git-commit.md page=page %}

{% snippet topics/admin/faqs/missed-something.md step=4 %}

{% snippet topics/admin/faqs/git-gat-path.md tutorial="tus" %}


topic: admin
tutorial: object-store
title: Distributed Object Storage
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["ansible", "storage"]
content_hash: d71254f5bcf0deb35f0c4f7638b90d4d
last_modified: 2026-02-26T21:12:14.149731
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/object-store/tutorial.html

content:
You may find that your Galaxy files directory has run out of space, but you don't want to move all of the files from one filesystem to another. One solution to this problem is to use Galaxy's hierarchical object store to add an additional file space for Galaxy.

Alternatively, you may wish to write new datasets to more than one filesystem. For this, you can use Galaxy's distributed object store.

# Expanding Storage

This tutorial assumes you have done the "Ansible for installing Galaxy" tutorial, it references the base configuration set up in that tutorial in numerous places.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <warning-title>switching object store types will cause issues</warning-title>
> Do not run this tutorial on your production instance! We will switch between object stores during this tutorial which has serious implications for production servers.
{: .warning}

# Hierarchical Object Store

First, note that your Galaxy datasets have been created thus far in the directory `/data`, due to `galaxy_config: galaxy: file_path`. In some cases, we run out of storage in a particular location. Galaxy allows us to add additional storage locations where it will create new datasets, while still looking in the old locations for old datasets. You will not have to migrate any of your datasets, and can just "plug and play" with new storage pools.


> <hands-on-title>Adding Hierarchical Storage</hands-on-title>
>
> 1. Open your group variables file and set the `object_store_config_file` variable:
>
>    ```yaml
>    galaxy_config:
>      galaxy:
>        object_store_config_file: {% raw %}"{{ galaxy_config_dir }}/object_store_conf.xml"{% endraw %}
>    ```
>
> 2. In your group variables file, add it to the `galaxy_config_templates` section:
>
>    ```yaml
>    galaxy_config_templates:
>      - src: templates/galaxy/config/object_store_conf.xml.j2
>        dest: {% raw %}"{{ galaxy_config.galaxy.object_store_config_file }}"{% endraw %}
>    ```
>
> 3. Create and edit `templates/galaxy/config/object_store_conf.xml.j2` with the following contents:
>
>    ```xml
>    <?xml version="1.0"?>
>    <object_store type="hierarchical">
>        <backends>
>            <backend id="newdata" type="disk" order="0">
>                <files_dir path="/data2" />
>                <extra_dir type="job_work" path="/data2/job_work_dir" />
>            </backend>
>            <backend id="olddata" type="disk" order="1">
>                <files_dir path="/data" />
>                <extra_dir type="job_work" path="/data/job_work_dir" />
>            </backend>
>        </backends>
>    </object_store>
>    ```
>
> 4. Add a `pre_task` in your playbook `galaxy.yml` file to create the `/data2` folder [using the file module](https://docs.ansible.com/ansible/2.9/modules/file_module.html).
>
>    ```
>        - name: Create the second storage directory
>          file:
>            owner: galaxy
>            group: galaxy
>            path: /data2
>            state: directory
>            mode: '0755'
>    ```
>
>    We've hardcoded the user/group because creating a storage directory is unusual. In normal practice someone provides you with an NFS mount and you will simply point your Galaxy there.
>
> 5. Run the playbook and restart Galaxy
>
> 6. Run a couple of jobs after Galaxy has restarted.
>
>    > <question-title></question-title>
>    >
>    > Where is the data now stored?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > You should see `/data2` in the `Full Path`, if not, something went wrong. Check that your "order" is correct
>    > >
>    > {: .solution }
>    >
>    {: .question}
>
{: .hands_on}

# Distributed Object Store

Rather than searching a hierarchy of object stores until the dataset is found, Galaxy can store the ID (in the database) of the object store in which a dataset is located when the dataset is created. This allows Galaxy to write to more than one object store for new datasets.


> <hands-on-title>Distributed Object Store</hands-on-title>
>
> 1. Edit your `templates/galaxy/config/object_store_conf.xml.j2` file and replace the contents with:
>
>    ```xml
>    <?xml version="1.0"?>
>    <object_store type="distributed">
>        <backends>
>            <backend id="newdata" type="disk" weight="1">
>                <files_dir path="/data2"/>
>                <extra_dir type="job_work" path="/data2/job_work_dir"/>
>            </backend>
>            <backend id="olddata" type="disk" weight="1">
>                <files_dir path="/data"/>
>                <extra_dir type="job_work" path="/data/job_work_dir"/>
>            </backend>
>        </backends>
>    </object_store>
>    ```
>
> 2. Run the playbook, restart Galaxy
>
> 3. Run 4 or so jobs, and check where the output appear. You should see that they are split relatively evenly between the two data directories.
>
{: .hands_on}

Sites like UseGalaxy.eu use the distributed object store in order to balance dataset storage across 10 different storage pools.

> <details-title>More documentation</details-title>
>
> More information can be found in the [sample file](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/config/sample/object_store_conf.xml.sample).
>
{: .details}

> <tip-title>Can I distribute objects based on the user?</tip-title>
>
> Yes! You must write your own dynamic job handler code to handle this.
> See [PR#6552](https://github.com/galaxyproject/galaxy/pull/6552) and [PR#10233](https://github.com/galaxyproject/galaxy/pull/10233)
>
> If you implement something like this, please [let the GTN
> know](https://github.com/galaxyproject/training-material/issues/new) with
> some example code, and we can include this as a training module for everyone.
>
{: .tip}

> <warning-title>switching object store types will cause issues</warning-title>
> We have switched between two different object stores here, but this is not supported. If you need to do this, you will need to update datasets in Galaxy's database. Any datasets that were created as hierarchical will lack the `object_store_id`, and you will need to supply the correct one. Do not just blindly copy these instructions, please understand what they do before running them and talk to us on [Gitter](http://gitter.im/galaxyproject/Lobby) for more help
>
> 1. Move the datasets to their new location: `sudo -u galaxy rsync -avr /hierarchical/000/ /distributed/000/`
>
> 2. Update the database: `sudo -Hu galaxy psql galaxy -c "UPDATE dataset SET object_store_id='data';" `
>
> 3. Restart your Galaxy
>
{: .warning}


# S3 Object Store

Many sites have access to an S3 service (either public AWS, or something private like Swift or Ceph), and you can take advantage of this for data storage.

we will set up a local S3-compatible object store, and then talk to the API of this service.

> <hands-on-title>Setting up an S3-compatible Object Store</hands-on-title>
>
> 1. Edit your `requirements.yml` file and add:
>
>    ```yml
>    - src: atosatto.minio
>      version: v1.1.0
>    ```
>
> 2. ```ansible-galaxy install -p roles -r requirements.yml```
>
> 3. Edit your group variables to configure the object store:
>
>    ```
>    minio_server_datadirs: ["/minio-test"]
>    minio_access_key: "my-access-key"
>    minio_secret_key: "my-super-extra-top-secret-key"
>    ```
>
> 4. Edit your playbook and add the minio role **before** `galaxyproject.galaxy`:
>
>    ```
>        - atosatto.minio
>    ```
>
>    Galaxy will need to use the bucket, and will want it to be there when it boots, so we need to setup the object store first.
>
> 5. Edit the `templates/galaxy/config/object_store_conf.xml.j2`, and configure the object store as one of the hierarchical backends. The object store does not play nicely with the distributed backend during training preparation. Additionally, reset the orders of the disk backends to be higher than the order of the swift backend.
>
>    {% raw %}
>    ```diff
>    @@ -1,13 +1,21 @@
>     <?xml version="1.0"?>
>    - <object_store type="distributed">
>    + <object_store type="hierarchical">
>         <backends>
>    -        <backend id="newdata" type="disk" weight="1">
>    +        <backend id="newdata" type="disk" order="1">
>                 <files_dir path="/data2"/>
>                 <extra_dir type="job_work" path="/data2/job_work_dir"/>
>             </backend>
>    -        <backend id="olddata" type="disk" weight="1">
>    +        <backend id="olddata" type="disk" order="2">
>                 <files_dir path="/data"/>
>                 <extra_dir type="job_work" path="/data/job_work_dir"/>
>             </backend>
>    +        <object_store id="swifty" type="swift" order="0">
>    +            <auth access_key="{{ minio_access_key }}" secret_key="{{ minio_secret_key }}" />
>    +            <bucket name="galaxy" use_reduced_redundancy="False" max_chunk_size="250"/>
>    +            <connection host="127.0.0.1" port="9091" is_secure="False" conn_path="" multipart="True"/>
>    +            <cache path="{{ galaxy_mutable_data_dir }}/database/object_store_cache" size="1000" />
>    +            <extra_dir type="job_work" path="{{ galaxy_mutable_data_dir }}/database/job_working_directory_swift"/>
>    +            <extra_dir type="temp" path="{{ galaxy_mutable_data_dir }}/database/tmp_swift"/>
>    +        </object_store>
>         </backends>
>     </object_store>
>    ```
>    {% endraw %}
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
> 6. Run the playbook.
>
> 7. Galaxy should now be configure to use the object store!
>
> 8. When the playbook is done, upload a dataset to Galaxy, and check if it shows up in the bucket:
>
>    ```
>    $ sudo ls /minio-test/galaxy/000/
>    dataset_24.dat
>    ```
>
{: .hands_on}

# Dropbox

Dropbox is a well-known cloud storage service where you can store and share files with anyone.
As of [20.09](https://github.com/galaxyproject/galaxy/pull/9888), Galaxy has support for a couple of different file storage backends, including NextCloud (via webdavfs) and Dropbox.

This tutorial will help you setup the connection between Galaxy and Dropbox, allowing your users to add their account details and then access their Dropbox data within Galaxy

> <hands-on-title>Configure Galaxy to access the Dropbox service</hands-on-title>
>
> 1. If the folder does not exist, create `templates/galaxy/config` next to your `galaxy.yml` playbook.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```
>    > mkdir -p templates/galaxy/config
>    > ```
>    {: .code-in}
>
> 2. Create `templates/galaxy/config/file_sources_conf.yml` with the following contents:
>
>    ```yaml
>    - type: dropbox
>      id: dropbox
>      label: Your Dropbox Files
>      doc: Your Dropbox files - configure an access token via the user preferences
>      accessToken: ${user.preferences.get('dropbox|access_token', '') if $user.preferences else ''}
>    ```
> 3. Create `templates/galaxy/config/user_preferences_extra_conf.yml` with the following contents:
>
>    ```yaml
>    preferences:
>        dropbox:
>            description: Your Dropbox account
>            inputs:
>                - name: access_token
>                  label: Dropbox access token
>                  type: password
>                  required: False
>    ```
> 4. Inform the `galaxyproject.galaxy` role of where you would like the `file_sources_conf.yml` and `user_preferences_extra_conf.yml` to reside, by setting it in your `group_vars/galaxyservers.yml`:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -35,6 +35,8 @@ galaxy_config:
>         tool_data_path: "{{ galaxy_mutable_data_dir }}/tool-data"
>         job_config_file: "{{ galaxy_config_dir }}/job_conf.yml"
>    +    file_sources_config_file: "{{ galaxy_config_dir }}/file_sources_conf.yml"
>    +    user_preferences_extra_conf_path: "{{ galaxy_config_dir }}/user_preferences_extra_conf.yml"
>    ```
>    {% endraw %}
>
> 5. Deploy the new config files using the `galaxy_config_files` var (also from the `galaxyproject.galaxy` role) in your group vars:
>
>    {% raw %}
>    ```diff
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -65,6 +67,12 @@ galaxy_config_templates:
>       - src: templates/galaxy/config/job_conf.yml.j2
>         dest: "{{ galaxy_config.galaxy.job_config_file }}"
>     
>    +galaxy_config_files:
>    +  - src: files/galaxy/config/user_preferences_extra_conf.yml
>    +    dest: "{{ galaxy_config.galaxy.user_preferences_extra_conf_path }}"
>    +  - src: files/galaxy/config/file_sources_conf.yml
>    +    dest: "{{ galaxy_config.galaxy.file_sources_config_file }}"
>    +
>    ```
>    {% endraw %}
>
> 6. Run the playbook. At the very end, you should see output like the following indicating that Galaxy has been restarted:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```
>    > > ansible-playbook galaxy.yml
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>Output</code-out-title>
>    > > ```
>    > > ...
>    > > RUNNING HANDLER [restart galaxy] ****************************************
>    > > changed: [gat-88.training.galaxyproject.eu]
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
{: .hands_on}

Now we are ready to configure a Galaxy's user account to upload dataset from Dropbox to the Galaxy server.

> <hands-on-title>Configure Galaxy to access the Dropbox service</hands-on-title>
>
> 1. Generate a Dropbox access token following the [Dropbox Oauth guide](https://www.dropbox.com/lp/developers/reference/oauth-guide)
>
> 2. Add the Dropbox access token in the Galaxy's user preferences
>    - Go to https://\<server\>/user/information
>    - Here you will find the form user will fill in with his own Dropbox access token:
>      ![user preferences form](../../images/storage_dropbox_form.png)
>
> 3. Click the upload icon toward the top left corner.
>    You will have a new "Choose remote files" button that will open the remote files windows with the link to reach your Dropbox files:
>     ![remote files window](../../images/storage_dropbox_remote_files_window.png)
>
{: .hands_on}


# Onedata

To configure Onedata as an Object Store or a Remote File Source, follow the
[dedicated tutorial]({% link topics/admin/tutorials/onedata-configuration/tutorial.html %}).


topic: admin
tutorial: surf-research-cloud-galaxy
title: Galaxy usage on SURF Research Cloud
difficulty: introductory
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"none\"", "type: \"none\"", "type: \"none\"", "type: \"none\""]
tags_json: []
content_hash: 8d3d8bb78147d198331e5fedf6080f97
last_modified: 2026-02-26T21:12:14.162618
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/surf-research-cloud-galaxy/tutorial.html

content:
Using Galaxy via the {SRC} allows researchers to start Galaxy instances on-demand and analyze their data in a secure environment following the {GDPR}. The instance provides secure authentication, where users must have a SURF Research account prior to this tutorial, have set the {SRAM} authentication method, and connect an SSH key to their accounts. In case you are not familiar with {SRC} and need help in setting up your accounts, please follow the instructions on the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/SURF+Research+Cloud)

In this training the aim is to focus on:

- Understanding how to start Galaxy on {SRC}
- Attaching external volumes for storage

Galaxy instances can be started and stopped on demand, depending on personal cases and requirements. Inside the SRC members should have access to all publicly available catalog items. If you are not able to create a catalog item, please [contact SURF servicedesk](mailto:servicedesk@surf.nl).

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prerequisites

This tutorial assumes you are member of a {CO} in {SRAM} that has access to {SRC} and a wallet with budget in SRC with enough sources to create Galaxy and Pulsar catalog items. (For more information please refer to the [SURF Knowledge Base](https://servicedesk.surf.nl/wiki/display/WIKI/Budgets%2C+wallets%2C+contracts).

You must have previous experience working with data inside Galaxy.

# Starting a Galaxy instance inside SRC step-by-step

> <hands-on-title>Access the SRC</hands-on-title>
> 1. The first and most important step is to have access to the SURF Research Cloud.
> 2. You will need to login to the [portal](https://portal.live.surfresearchcloud.nl).
>    ![login screen for SURF research cloud](./images/1.png)
>
> 3. Assuming you are a Dutch researcher, you should expect to see your educational institution listed on the authentication provider screen.
>    ![login with screen, showing an authentication provider being chosen.](./images/2.png)
> 4. Log in with your institute credentials and follow the SRAM authentication on your phone. Inside the environment, you can see if there are any active virtual machines, the options to create a workspace, new storage or request new wallet for your projects.
>    ![SRC dashboard, options like create new workspace, storage, or request a new wallet are available. one workspace is listed as running: galaxy test tim.](./images/3.png)
{: .hands_on}

With this, you're ready to get started launching resources within {SRC}.

## Create an external storage

The purpose of the storage is to attach it to your workspace, where you will save your data. This way, if the machine is deleted, your data will not be lost. We strongly advise anyone using the SRC to create storages and attach them to their machines!


> <hands-on-title>Create a Storage Volume</hands-on-title>
> 1. Locate the "Create new storage" option
>    ![top half of the SRC dashboard showing options like create new workspace, storage, or request a new wallet.](./images/4.png)
> 2. Click on "Create new".
> 3. Select the collaborative organisation you want the storage to be part of.
> 4. Select the desired cloud provider and the size of the volume you want to use.
>    > <tip-title>Unsure of the size required?</tip-title>
>    > If you are unsure about the size of the storage you need, contact your administrator. Consider also consulting [Galaxy from an Administrator's Point of View]({% link topics/admin/tutorials/introduction/slides.html %}#10)
>    {: .tip}
>
>    ![create your storage dashboard showing a selection of providers like surf and aws and azure, and then a selection of storage sizes below ranging from 5GB to 1.5TB](./images/5.png)
>
> 6. Name your storage however you like and press on submit.
>
>    ![form to provide name and preview what will be created.](./images/6.png)
>
> 7. You will be redirected to the main page, where you will be able to track the status of the storage creation.
>
>    ![bottom half of home page showing the Storage tab, and a resource named storage-galaxy in state creating.](./images/7.png)
{: .hands_on}

Once the storage is deployed, you can attach it to any of your machines! 🎉

Be aware, that in order to attach an external storage to an existing machine, you would need to pause the workspace first! (If you are not sure how to do that please refer to [External storage volumes](https://servicedesk.surf.nl/wiki/display/WIKI/External+storage+volumes))

# Create a new workspace

Depending on user’s rights, you can create a new workspace for your collaboration.

> <tip-title>A Workspace?</tip-title>
> Workspaces may also be known as *Virtual Machines/VMs* or *Instances* in other cloud providers.
{: .tip}

> <hands-on-title>Create a Workspace</hands-on-title>
> 1. On the right side under the "Workspaces" click on the "Add" button.
>
>    ![dashboard of workspaces showing a single running workspace titled galaxy test tim, and on the right a button to add a new workspace.](./images/8.png)
>
> 2. Then, you will be redirected to a new page, where you must first choose the collaborative organisation in which you want to create your workspace (in case you are a member of multiple organisations). Once chosen, you a new page will be loaded, where you will have access to all available catalog items.
>
>    ![Create your workspace screen showing 9 catalog items on the first page with pages more. Visible are Galaxy Pulsar node amongst some docker and ubuntu options.](./images/9.png)
>
> 3. Use the magnifying glass on the right side of the panel and search for Galaxy. Two catalog items will appear - the Galaxy instance designed for SURF and a Galaxy Pulsar node that can be connected to the instance. Select "Galaxy at SURF".
>
>    ![similar to previous image, but the items described above](./images/10.png)
>
> 4. SURF Research Cloud allows researchers to host their catalog items on different cloud providers. The Galaxy catalog item is currently supported only on the HPC Cloud and for Ubuntu 22.04. On this page, users can select the number of cores and RAM that they want on their machine. More sizes can be added in the future, and on request. Choose wisely and in case you are not certain contact your administrator!
>
>    ![workspace creation screen, choose the cloud provider is locked to SURF HPC, flavour is locked to 22.04, and the size options are available from 1GB/8CPU to 60C/750GB Hi-mem](./images/11.png)
>
> 5. Select the storage you have created earlier so it is attached to the new workspace.
>
>    ![create your workspace screen, storage only lists one option named storage-galaxy](./images/12.png)
>
> 6. Lastly, before the workspace is deployed, you need to choose for how long the machine will run.
>
>    > <tip-title>Expiration Date</tip-title>
>    > The standard life-time of the VM is 5 days. If you need it for longer, this option can be changed once the machine is running.
>    > Note, that once the machine is expired and deleted it cannot be restored! Plan accordingly and migrate your data in time to prevent data loss!
>    >
>    > This is an incredibly useful feature as it saves you from forgetting to destroy a VM. Especially for GPU nodes it can help you ensure that they disappear after your computation is complete.
>    {:.tip}
>
> 7. In this form, you can also select how to name your workspace, add a description (if you want) and specify the hostname. Then, scroll down, customise your Galaxy name, add your storage name and submit the workspace creation. 
>    ![almost there, some final details reads the heading. an expiration date is set and galaxy at surf is provided in the name. the hostname is set to galaxyatsurf and the description is blank](./images/17.png)
>    Change `my_storage` to the storage you attached earlier. In this case, `galaxy-storage`. Bonus: You can also rename the Galaxy to your liking! 
>    ![customise your Galaxy name and attach your storage. change the name of my_data to the storage you attached earlier, in this case, galaxy-storage](./images/18.png)
{: .hands_on}

With that, your workspace will start being created.

![A galaxy instance named galaxy demo is in status creating.](./images/14.png "Creating the workspace can take some time. You will see on the main page the status of your workspace.")

> <tip-title>Also launching Pulsar?</tip-title>
> While you are waiting, you can create your Galaxy Pulsar node simultaneously to save time.
{: .tip}

Once the workspace is running, press on the down arrow to check any information such as the ID, the owner of the workspace, the collaboration it is part of, when it is created and when it expires and the wallet it is using. Additionally, you can pause the workspace when you are not using it. Pressing on the storage option will display the storage that has been attached to the workspace.

![the details panel of the galaxy demo instance is shown including fields like an id, owner, date created and expiration date, url., etc](./images/15.png)

To access the Galaxy instance, press on the "Access" button. This will redirect you to a new webpage. The {SRAM} login will be displayed again, and after authenticating, you will be redirected to the Galaxy at SURF Research Cloud page!

![A galaxy instance homepage is shown, with a brand indicating that it was deployed on surf research cloud featuring a cute surfer emoji.](./images/16.png)

Congratulations! You have started a Galaxy instance inside the SURF Research Cloud! 🏄‍♀️🎉

If you have administrator rights, you can download tools that can be used by the members of the collaborative organisation. To obtain administrator rights you need to be part of the `src_co_admin` group inside the collaborative organisation. In case you are not sure how to download tools as a Galaxy administrator, please refer to [Installing Tools into Galaxy](https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/).

## Going Further with Pulsar?

Another important step when starting a local Galaxy instance on {SRC} is connecting it to a Galaxy Pulsar Node, which you can do [in this Pulsar SRC tutorial]({% link topics/admin/tutorials/surf-research-cloud-pulsar/tutorial.md %}). The Pulsar is an useful component that is used to execute your tasks on remote servers/clusters. In this case, the Pulsar is started as a new workspace, allowing users to also run interactive tools.

In SRC, this can also enable you to save costs. By choosing a smaller Galaxy instance, and attaching a large Pulsar node when you need to do computations, you can be sure to optimise your budget expenditures.


topic: admin
tutorial: k8s-managing-galaxy
title: Managing Galaxy on Kubernetes
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["kubernetes"]
content_hash: cfa2bbb2797d4802946df596b57cc581
last_modified: 2026-02-26T21:12:14.148416
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/admin/tutorials/k8s-managing-galaxy/tutorial.html

content:
# Managing Galaxy on Kubernetes

## Overview


A primary advantage of Galaxy on [Kubernetes](https://kubernetes.io/) is the ease with which common
administrative tasks can be performed reliably and without disruption of
service. In particular, because of containerization, Kubernetes provides a significant
advantage over managing individual virtual machines, where updates to system
libraries or components can cause unexpected breakage of dependent components.
With containerization, this becomes a simpler problem of swapping out a
container and replacing it with an updated version. It also reduces reliance
on the underlying operating system, allowing the OS to be upgraded and
have the latest security patches applied without having to worry about
how it will affect the applications running within. Kubernetes has built-in
functionality for draining a node of all containers and for transparently
moving those containers to a different node, allowing maintenance tasks to be
performed on the underlying node without disruption of service.

In this section, we will look at how to perform common management tasks on
a Galaxy deployment on Kubernetes, including:
- How to upgrade a deployment
- Change the configuration of a running Galaxy instance
- Map arbitrary files into Galaxy's config folder
- Rollback changes in the case of an error
- Scale the number of job and web handlers
- Delete a deployment

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Prerequisites
This tutorial builds on the material of the previous
[tutorial]({{ site.baseurl }}/topics/admin/tutorials/k8s-deploying-galaxy/tutorial.html)
and we recommend following it first to setup the required environment.
You must have some familiarity with Helm commands, know how to change values
in a Helm Chart and how to use the `kubectl` command.

# Changing the configuration of a Galaxy instance
We will start off by looking at how to change the configuration of a Galaxy
instance. We will first reduce the number of tools that are loaded for faster
startup, and then change some common settings in `galaxy.yml`.

## Changing tool configuration
We will change the Galaxy configuration to limit the initial list of tools
that Galaxy will use by pointing to our custom `tool_conf.xml`. This will make
your chart start up much faster for the remainder of this tutorial, as the
default configuration loads the full list of tools used by
[https://usegalaxy.org/](https://usegalaxy.org/).

> <hands-on-title>Creating a custom tool set</hands-on-title>
>
> 1. First, let's create a simpler list of tools by saving the following tool
>    config as a file called `custom_tool_conf.xml`.
>
>    {% raw %}
>    ```xml
>    <?xml version="1.0" ?>
>    <toolbox tool_path="/cvmfs/main.galaxyproject.org/shed_tools">
>        <section id="get_data" name="Get Data">
>            <tool file="data_source/upload.xml" />
>        </section>
>        <section id="chip_seq" name="ChIP-seq" version="">
>            <tool file="toolshed.g2.bx.psu.edu/repos/rnateam/chipseeker/1b9a9409831d/chipseeker/chipseeker.xml" guid="toolshed.g2.bx.psu.edu/repos/rnateam/chipseeker/chipseeker/1.18.0+galaxy1">
>                <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
>                <repository_name>chipseeker</repository_name>
>                <repository_owner>rnateam</repository_owner>
>                <installed_changeset_revision>1b9a9409831d</installed_changeset_revision>
>                <id>toolshed.g2.bx.psu.edu/repos/rnateam/chipseeker/chipseeker/1.18.0+galaxy1</id>
>                <version>1.18.0+galaxy1</version>
>            </tool>
>        </section>
>        <section id="fastq_quality_control" name="FASTQ Quality Control" version="">
>            <tool file="toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/51b771646466/trimmomatic/trimmomatic.xml" guid="toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.6">
>                <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
>                <repository_name>trimmomatic</repository_name>
>                <repository_owner>pjbriggs</repository_owner>
>                <installed_changeset_revision>51b771646466</installed_changeset_revision>
>                <id>toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/trimmomatic/0.36.6</id>
>                <version>0.36.6</version>
>            </tool>
>        </section>
>        <section id="fastq_quality_control" name="FASTQ Quality Control" version="">
>            <tool file="toolshed.g2.bx.psu.edu/repos/devteam/fastqc/e7b2202befea/fastqc/rgFastQC.xml" guid="toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1">
>                <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
>                <repository_name>fastqc</repository_name>
>                <repository_owner>devteam</repository_owner>
>                <installed_changeset_revision>e7b2202befea</installed_changeset_revision>
>                <id>toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.72+galaxy1</id>
>                <version>0.72+galaxy1</version>
>            </tool>
>        </section>
>    </toolbox>
>    ```
>    {% endraw %}
>
> 2. Next, let's create a new `galaxy.yml` file that uses this `custom_tool_conf.xml`.
>
>    Note that the content below is the same as the `configs` section of
>    `values-cvmfs.yaml` file from the Galaxy Helm chart with one exception:
>    `tool_config_file` entry is pointing to our custom tool list instead of the
>    full list from CVMFS.
>
>    {% raw %}
>    ```yaml
>    uwsgi:
>      virtualenv: /galaxy/server/.venv
>      processes: 1
>      http: 0.0.0.0:8080
>      static-map: /static/style=/galaxy/server/static/style/blue
>      static-map: /static=/galaxy/server/static
>      static-map: /favicon.ico=/galaxy/server/static/favicon.ico
>      pythonpath: /galaxy/server/lib
>      thunder-lock: true
>      manage-script-name: true
>      mount: {{.Values.ingress.path}}=galaxy.webapps.galaxy.buildapp:uwsgi_app()
>      buffer-size: 16384
>      offload-threads: 2
>      threads: 4
>      die-on-term: true
>      master: true
>      hook-master-start: unix_signal:2 gracefully_kill_them_all
>      enable-threads: true
>      py-call-osafterfork: true
>    galaxy:
>      database_connection: 'postgresql://{{.Values.postgresql.galaxyDatabaseUser}}:{{.Values.postgresql.galaxyDatabasePassword}}@{{ template "galaxy-postgresql.fullname" . }}/galaxy'
>      integrated_tool_panel_config: "/galaxy/server/config/mutable/integrated_tool_panel.xml"
>      sanitize_whitelist_file: "/galaxy/server/config/mutable/sanitize_whitelist.txt"
>      tool_config_file: "{{.Values.persistence.mountPath}}/config/editable_shed_tool_conf.xml,/galaxy/server/config/custom_tool_conf.xml"
>      tool_data_table_config_path: "{{ .Values.cvmfs.main.mountPath }}/config/shed_tool_data_table_conf.xml,{{.Values.cvmfs.data.mountPath}}/managed/location/tool_data_table_conf.xml,{{.Values.cvmfs.data.mountPath}}/byhand/location/tool_data_table_conf.xml"
>      tool_dependency_dir: "{{.Values.persistence.mountPath}}/deps"
>      builds_file_path: "{{.Values.cvmfs.data.mountPath}}/managed/location/builds.txt"
>      datatypes_config_file: "{{ .Values.cvmfs.main.mountPath }}/config/datatypes_conf.xml"
>      containers_resolvers_config_file: "/galaxy/server/config/container_resolvers_conf.xml"
>      workflow_schedulers_config_file: "/galaxy/server/config/workflow_schedulers_conf.xml"
>      build_sites_config_file: "/galaxy/server/config/build_sites.yml"
>    ```
>    {% endraw %}
>
> 3. Now, let's upgrade the chart to use `custom_tool_conf.xml` and
>    `galaxy.yml` by running the `helm upgrade` command.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values --set-file "configs.custom_tool_conf\.xml"=custom_tool_conf.xml --set-file "configs.galaxy\.yml"=configs/galaxy.yml galaxy galaxy/galaxy
>    ```
>    {% endraw %}
>
>    Note the `--reuse-values` flag, which instructs Helm to reuse any
>    previously set values, and apply the new ones on top.  The `--set-file`
>    option will set the value of the `configs.custom_tool_conf.xml`
>    key in your values file to the contents of the specified file, as a text
>    string. Each file under `configs` key in `values.yaml` is automatically
>    mapped into Galaxy's `config` directory within the running container.
>
> 4. Notice that while the chart is upgrading, the existing version continues
>    to function. The changeover will occur when the new container is online
>    and signals readiness to Kubernetes by responding to web requests on
>    the relevant port. Log into the Kubernetes dashboard and watch the logs
>    as the new pods come online.
>
> 5. List the installed helm charts again and note that the revision of the chart
>    has changed. These revisions are useful because it allows us to rollback
>    our changes if they are incorrect. This will be covered in a later section.
>
>    {% raw %}
>    ```bash
>    helm list
>    NAME  	REVISION	UPDATED                 	STATUS  	CHART                 	APP VERSION	NAMESPACE
>    cvmfs 	1       	Wed Jun 26 14:47:46 2019	DEPLOYED	galaxy-cvmfs-csi-1.0.1	1.0        	cvmfs
>    galaxy	2       	Wed Jun 26 14:51:17 2019	DEPLOYED	galaxy-3.0.0          	v19.05     	default
>    ```
>    {% endraw %}
>
> 6. Let's now exec into the running container and check where the files were
>    mapped in. First, let's get a list of running pods.
>
>    {% raw %}
>    ```bash
>    kubectl get pods
>    NAME                          READY   STATUS    RESTARTS   AGE
>    galaxy-galaxy-postgres-0      1/1     Running   0          2d6h
>    galaxy-job-69864b6797-zs5mn   1/1     Running   0          2d6h
>    galaxy-web-7568c58b94-jzkvm   1/1     Running   0          2d6h
>    ```
>    {% endraw %}
>
>    Exec into the web pod by running:
>    {% raw %}
>    ```bash
>    kubectl exec -it galaxy-web-7568c58b94-jzkvm /bin/bash
>    ```
>    {% endraw %}
>
>    Now run `ls /galaxy/server/config/` and note that the `galaxy.yml` contains
>    the content that you've provided and that `custom_tool_conf.xml` has also been
>    mapped into the config folder. In this same way, any of Galaxy's config
>    files can be overridden by simply mapping in the relevant file into the
>    config folder.
{: .hands_on}

## Setting the admin user and changing the brand
Next, we will set the admin user and change the brand in `galaxy.yml`. We will
rollback our change to understand how Helm manages configuration.

> <hands-on-title>Setting admin user and changing the brand</hands-on-title>
>
> 1. Modify the following entries in your `galaxy.yml`. Make sure to add these
>    keys under the `galaxy:` section of the file.
>
>    {% raw %}
>    ```
>    brand: "Hello World"
>    admin_users: "admin@mydomain.com"
>    ```
>    {% endraw %}
>
> 2. Now, let’s upgrade the chart to apply the new configuration.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values --set-file "configs.galaxy\.yml"=galaxy.yml galaxy galaxy/galaxy
>    ```
>    {% endraw %}
>
> 3. Inspect the currently set Helm values by:
>
>    {% raw %}
>    ```bash
>    helm get values galaxy
>    ```
>    {% endraw %}
>
> 4. List the installed Helm charts again and note that the revision of the chart has changed as expected.
>
>    {% raw %}
>    ```bash
>    helm list
>    NAME  	REVISION	UPDATED                 	STATUS  	CHART                 	APP VERSION	NAMESPACE
>    cvmfs 	1       	Wed Jun 26 14:47:46 2019	DEPLOYED	galaxy-cvmfs-csi-1.0.1	1.0        	cvmfs
>    galaxy	3       	Wed Jun 26 14:51:17 2019	DEPLOYED	galaxy-3.0.0          	v19.05     	default
>    ```
>    {% endraw %}
>
> 5. Let’s now roll back to the previous revision.
>
>    {% raw %}
>    ```bash
>    helm rollback galaxy 2
>    ```
>    {% endraw %}
>
>    Use `helm get values` again to observe that the values have reverted to
>    the previous revision. After a short while, once the new container is up
>    and running, Kubernetes will automatically switch over to it and you can
>    see that the previous configuration has been restored.
>
{: .hands_on}

# Scaling Galaxy
In Galaxy deployment on Kubernetes, there are two containers by default, one
web handler and one job handler. We will now look at how these can be scaled.

> <hands-on-title>Setting admin user and changing the brand</hands-on-title>
>
> 1. View the `values-cvmfs.yaml` file in the Galaxy Helm chart and note down the
>    number of web and job handlers.
>
>    {% raw %}
>    ```yaml
>    webHandlers:
>        replicaCount: 1
>    jobHandlers:
>        replicaCount: 1
>    ```
>    {% endraw %}
>
> 2. Let’s increase the number of web handlers by simply setting new values for the number of replicas.
>
>    {% raw %}
>    ```bash
>    helm upgrade --reuse-values --set webHandlers.replicaCount=2 galaxy galaxy/galaxy
>    ```
>    {% endraw %}
>
> 3. Check whether the new replicas have been created.
>
>    {% raw %}
>    ```bash
>    kubectl get pods
>    NAME                          READY   STATUS    RESTARTS   AGE
>    galaxy-galaxy-postgres-0      1/1     Running   0          2d9h
>    galaxy-job-5cc75c6588-8dsbg   1/1     Running   0          7m13s
>    galaxy-web-7c9576cf89-49nlm   1/1     Running   0          7m13s
>    galaxy-web-7c9576cf89-r6rcj   0/1     Running   0          9s
>    ```
>    {% endraw %}
>
> 4. Follow the pod logs and check whether the new handler is receiving web requests
>    as expected.
>
>    {% raw %}
>    ```bash
>    kubectl logs -f galaxy-web-7c9576cf89-r6rcj
>    ```
>    {% endraw %}
>
>    You will notice that Kubernetes automatically load balances requests between the
>    available web handler replicas in a round-robin fashion.
>
{: .hands_on}

# Testing Kubernetes resilience
To observe how Kubernetes handles failures, let’s exec into a running container and
manually kill a process to simulate a possible process failure. Kubernetes
continuously monitors running containers, and attempts to bring the environment
back to the “desired” state. The moment it notices a failure, it will respawn
a new pod to replace the failed one. Typically, a Kubernetes container will also
have a [liveness probe][k8sliveness] defined. A liveness probe can be an http
request to a port, or even a manually executed shell script, which will test
whether the relevant container is healthy, and if not, Kubernetes will immediately
provision a new replacement.

> <hands-on-title>Handling failures</hands-on-title>
>
> 1. First list the available pods.
>
>    {% raw %}
>    ```bash
>    kubectl get pods
>    NAME                          READY   STATUS    RESTARTS   AGE
>    galaxy-galaxy-postgres-0      1/1     Running   0          2d9h
>    galaxy-job-5cc75c6588-8dsbg   1/1     Running   0          14m
>    galaxy-web-7c9576cf89-49nlm   1/1     Running   0          14m
>    galaxy-web-7c9576cf89-r6rcj   0/1     Running   1          7m36s
>    ```
>    {% endraw %}
>
>    Then exec into one:
>    {% raw %}
>    ```bash
>    kubectl exec -it galaxy-web-7c9576cf89-r6rcj /bin/bash
>    ```
>    {% endraw %}
>
> 2. Now kill the main container process.
>
>    {% raw %}
>    ```bash
>    kill 1
>    ```
>    {% endraw %}
>
> 3. If we run `kubectl get pods`, we can notice how Kubernetes immediately
>    starts a new pod to replace the failed one, bringing the environment back to
>    the desired state. Take a look at the liveness probe defined for the galaxy
>    web container in the helm chart source code
>    (`templates/deployment-web.yaml`).
>
{: .hands_on}

# Deleting Galaxy
Finally, let’s take a look at how we can uninstall Galaxy and remove all
related containers.

> <hands-on-title>Deleting Galaxy</hands-on-title>
>
> 1. To permanently delete the Galaxy release, run:
>
>    {% raw %}
>    ```bash
>    helm delete --purge galaxy
>    ```
>    {% endraw %}
>
>    The purge flag instructs helm to permanently remove galaxy from its history.
>
> 2. Use `kubectl get pods` to verify that the pods have been deleted.
>
{: .hands_on}

# Next Steps
This tutorial provided an overview of some common Galaxy administration tasks.
Advanced customizations would include running custom shell scripts on Galaxy
startup to perform additional tasks, running additional containers on
startup, administering and managing storage, building custom Galaxy containers
with desired modifications etc. For more info on some of these
topics, take a look at the [Galaxy Helm chart] repository as well as
[other tutorials](../..) tagged with _kubernetes_. Also, feel free to reach out
on Gitter: [https://gitter.im/galaxyproject/FederatedGalaxy][fedG].


[fedG]: https://gitter.im/galaxyproject/FederatedGalaxy
[k8sliveness]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
[Galaxy Helm chart]: https://github.com/galaxyproject/galaxy-helm


