# tutorials

topic: visualisation
tutorial: jbrowse
title: Genomic Data Visualisation with JBrowse
difficulty: intermediate
time_estimation: 1h
tools_json: ["JBrowse", "toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0"]
requirements_json: []
tags_json: ["gmod", "jbrowse1"]
content_hash: bf82026b60c419a88cdc5d72c0c7571f
last_modified: 2026-05-15T22:28:18.636819
zenodo_link: https://doi.org/10.5281/zenodo.3591856
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/visualisation/tutorials/jbrowse/tutorial.html

content:
> JBrowse ({% cite Buels_2016 %}) is a fast, embeddable genome browser built completely with JavaScript
> and HTML5, with optional run-once data formatting tools written in Perl.
>
{: .quote cite="https://jbrowse.org/jbrowse1.html"}

The Galaxy tool accepts data in many formats:

- Intervals/Annotation/Feature Tracks (GFF/GFF3, BED, GenBank)
- BAM Pileups
- Blast XML results
- Wig/BigWig
- VCF files

and executes the "run-once data formatting tools" mentioned in its description. The JBrowse tool has an incredibly extensive number of options, more than anyone needs most of the time. We'll go through them in detail but feel free to skip the sections that don't apply to the data types you use. Not everyone has Blast results to visualise. This tutorial also is very hands off, we will give some guidance but it is up to you to explore JBrowse and consider ways in which it can apply to your own data. Unlike other tutorials there is no biological story here, just some ways you can visualise common datatypes.

This tutorial covers version 1.16.5+ of the JBrowse tool, earlier versions will have different behaviour and tool layout.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Upload

> <hands-on-title>Getting the data</hands-on-title>
>
> 1. Create and name a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the datasets we will visualize:
>
>    ```
>    https://zenodo.org/record/3591856/files/blastp%20genes.gff3
>    https://zenodo.org/record/3591856/files/blastp%20vs%20swissprot.xml
>    https://zenodo.org/record/3591856/files/dna%20sequencing.bam
>    https://zenodo.org/record/3591856/files/dna%20sequencing%20coverage.bw
>    https://zenodo.org/record/3591856/files/genes%20(de%20novo).gff3
>    https://zenodo.org/record/3591856/files/genes%20(NCBI).gff3
>    https://zenodo.org/record/3591856/files/genome.fa
>    https://zenodo.org/record/3591856/files/RNA-Seq%20coverage%201.bw
>    https://zenodo.org/record/3591856/files/RNA-Seq%20coverage%202.bw
>    https://zenodo.org/record/3591856/files/variants.vcf
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

The data for today is a subset of real datasets from *E. coli MG1655 strain K-12*

# Simple Gene Tracks

We will start by adding a couple of gene call tracks. In our case these are genes and gene predictions, but they don't have to be. In the general case this can be any interesting region from an analysis, where a tool has pointed out some region for further inspection for some reason, then this data can be visualised with the "gene" track type.

> <hands-on-title>Build the JBrowse</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome.fa`
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Genes`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `genes (de novo).gff3`
> 4. Execute
>
> 5. View the contents of the file
>
>    ![Screenshot of JBrowse](../../images/jbrowse-gff-track.png "Screenshot of JBrowse")
>
>    > <tip-title>Error reading from name store</tip-title>
>    > ![error message](../../images/jbrowse/error.png)
>    >
>    > If you see this error message, click OK, this is a known bug.
>    {: .tip}
>
{: .hands_on}

If you are not familiar with the operation of JBrowse there are some important points:

- "Tracks" are shown on the left. Clicking the checkboxes will make the tracks visible or invisible
- You can use your mouse scrollwheel to move around the genome view area, or you can click and drag to move.
- Double clicking will zoom in on the genome, or you can use the magnifying glass icons to zoom in our out

> <tip-title>Naming Tracks</tip-title>
>
> * The JBrowse tool takes track names directly from file names
> * If you want to rename tracks: **Click** on the **pencil** icon, edit the **Name** and click **Save**.
> * You can now re-run the JBrowse tool and it will produce a new JBrowse instance with corrected names.
{: .tip}

# Complex Gene Tracks

All of the track types in the JBrowse tool support a wide array of features. We've only looked at a simple track with default options, however there are more tools available to us to help create user-friendly JBrowse instances that can embed rich data.

> <hands-on-title>Build a JBrowse for viewing Genes</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome.fa`
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Genes`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `genes (de novo).gff3`
>                        - *"JBrowse Track Type [Advanced]"*: `Canvas Features`
>                        - In *"JBrowse Feature Score Scaling & Coloring Options [Advanced]"*:
>                            - *"Color Score Algorithm"*: `Based on score`
>                                - *"How should minimum and maximum values be determined for the scores of the features"*: `Manually specify minimum and maximum expected scores for the feature track`
>                                    - *"Minimum expected score"*: `0`
>                                    - *"Maximum expected score"*: `1`
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `GFF/GFF3/BED Features`
>                        - {% icon param-file %} *"GFF/GFF3/BED Track Data"*: `genes (NCBI).gff3`
>                        - *"JBrowse Track Type [Advanced]"*: `Canvas Features`
>                        - In *"JBrowse Contextual Menu options [Advanced]"*:
>
>                          > <tip-title>Contextual Menus</tip-title>
>                          >
>                          > We're going to add a "Contextual Menu" which deserves a little explanation first. In JBrowse
>                          > terminology the right click menu of some features is called the contextual menus. You can customize
>                          > this menu to add new links and options that will be useful to the user. These links can be
>                          > templated with variables based on metadata of the feature that the user clicked upon.
>                          >
>                          > There are more valid values for templating than mentioned in the help. When
>                          > you click on a feature in the JBrowse instance, it will present all of the
>                          > properties of the feature. Any of the top level properties can be used
>                          > directly in your templating
>                          {: .tip}
>
>                            - In *"Track Menu"*:
>                                - {% icon param-repeat %} *"Insert Track Menu"*
>                                    - *"Menu label"*: `See protein at NCBI`
>                                    - *"Menu url"*: `https://www.ncbi.nlm.nih.gov/gene?term={locus_tag}%5BGene%20Name%5D`
>                                    - *"Menu icon"*: `Database`
>
> 2. View the output
>
>    ![genes in JBrowse](../../images/jbrowse/genes.png "Genes visualised in JBrowse. The lighter coloured genes on the denovo track are coloured based on the confidence score given to the prediction by AUGUSTUS. NCBI genes do not have scores, these are more like an official gene set.")
>
> 3. Turn on both tracks of data.
>
> 4. Navigate to `21,200`, either manually, or by copying and pasting the location block: `NC_000913.3:18351..24780`
>
> 5. Right click on the `yaaY` gene, and click the "See protein at NCBI" menu option.
>
>    This menu option is dynamic, try it with a few other features from the `genes (NCBI).gff3` track. These features have a `locus_tag` and the menu button we added will open a URL to an NCBI search page for the value of this `locus_tag` attribute.
>
>    ![ncbi view](../../images/jbrowse/ncbi.png "The NCBI iframe within JBrowse, you can see how to link to external databases from JBrowse and help the viewer obtain as much information or context as they might need... but you have to build it yourself.")
{: .hands_on}

Contextual menus can be used to link to more than just NCBI.

- The links can go anywhere such as web search services (e.g. Google) or genomics web services (e.g. EBI)
- Some sites use the IFrame action to link genes to local services where users are expected to submit annotation notes or data.

# Sequencing, Coverage, and Variation

This is the next major category of data that people wish to visualize: sequencing, coverage, and variation. The sequencing data can be of any type, and does not necessarily need to be sequencing data, as long as the results are formatted as BAM files. Here we will add several coverage tracks (essentially a line plot along the genome, associating a position with a value) with various visualisation options like scaling and display type. Each of these visualisation options can be useful in different situations, but it largely is a matter of preference or what you are used to seeing.

Next we will add a sequencing dataset, a BAM file which maps some sequencing reads against various locations along the genome. JBrowse helpfully highlights which reads have mapping issues, and any changes in bases between the reads and the genome. We "Autogenerate a SNP track", which produces an extra track we can enable in JBrowse. This track reads the same BAM file used for visualising reads, and then produces a SNP and coverage visualisation. **NB**: This only works for small BAM files, if your files are large (>200 Mb), then you should consider generating these coverage and SNP tracks by other means (e.g. `bamCoverage` and `FreeBayes` or similar tools) as it will be significantly faster. You can learn more about generating these files in [the Mapping tutorial]({% link topics/sequence-analysis/tutorials/mapping/tutorial.md %}).

> <hands-on-title></hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome.fa`
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Coverage`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BigWig XY`
>                        - {% icon param-file %} *"BigWig Track Data"*: `dna sequencing coverage.bw`
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BigWig XY`
>                        - {% icon param-file %} *"BigWig Track Data"*: `rna-seq coverage/1.bw`
>                        - *"Use XYPlot"*: `Yes`
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BigWig XY`
>                        - {% icon param-file %} *"BigWig Track Data"*: `rna-seq coverage/2.bw`
>                        - *"Use XYPlot"*: `Yes`
>                        - *"Show variance band"*: `Yes`
>                        - *"Track Scaling"*: `Autoscale (global)`
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - *"Track Category"*: `Sequencing & Variation`
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `BAM Pileups`
>                        - {% icon param-file %} *"BAM Track Data"*: `sequencing.bam`
>                        - *"Autogenerate SNP Track"*: `Yes`
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `VCF SNPs`
>                        - {% icon param-file %} *"SNP Track Data"*: `variants.vcf`
>
> 2. Execute and then explore the resulting data.
>
>    ![sequencing tracks in JBrowse](../../images/jbrowse/sequencing.png "All of the sequencing tracks in JBrowse. Try exploring! There are many menu and configuration options which can help you filter and sort your data.")
{: .hands_on}

Try:

- Clicking on individual variations in the dedicated `variants.vcf` track
- Hovering over the autogenerated SNPs/Coverage track
- Clicking on individual reads of the `sequencing.bam` track
- Changing the visualisation options of the BAM track

# Blast Results

The Blast visualisation module requires that you have a gff3 formatted set of features which you then exported as DNA or protein, and blasted. The reason is easy to understand: when you extract DNA/protein sequences for Blasting, this process looses information about where these sequences were along the genome. The results from Blast retains the identifiers from the DNA/protein sequences, so we need to "map" these identifiers, to proper features with locations.

The best way to accomplish this is through the `gffread` tool which can cleanup a gff3 file, and export various features, optionally translating them. With these outputs, the cleaned features and `fasta` formatted sequences, you can Blast the sequences, and then supply the resulting Blast XML outputs in addition to the cleaned features, allowing a script to re-associate these Blast results to their original locations along the genome.

> <hands-on-title>Building a JBrowse for Blast results</hands-on-title>
>
> 1. {% tool [JBrowse](toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/jbrowse/1.16.9+galaxy0) %} with the following parameters:
>    - *"Reference genome to display"*: `Use a genome from history`
>        - {% icon param-file %} *"Select the reference genome"*: `genome.fa`
>    - *"Genetic Code"*: `11. The Bacterial, Archaeal and Plant Plastid Code`
>    - In *"Track Group"*:
>        - {% icon param-repeat %} *"Insert Track Group"*
>            - In *"Annotation Track"*:
>                - {% icon param-repeat %} *"Insert Annotation Track"*
>                    - *"Track Type"*: `Blast XML`
>                        - {% icon param-file %} *"BlastXML Track Data"*: `blastp vs swissprot.xml`
>                        - {% icon param-file %} *"Features used in Blast Search"*: `blastp genes.gff3`
>                        - *"Minimum Gap Size"*: `5`
>                        - *"Is this a protein blast search?"*: `Yes`
>                        - In *"JBrowse Feature Score Scaling & Coloring Options [Advanced]"*:
>                            - *"Color Score Algorithm"*: `Based on score`
>                                - *"JBrowse style.color function's score scaling"*: `Blast scaling`
>
> 2. Execute and then explore the resulting data.
>
>    ![blast results in JBrowse](../../images/jbrowse/blast.png "Blast results, coloured according to their e-value. This sort of track is commonly used to help genome annotators have additional genomic context when they are annotating.")
{: .hands_on}

# Conclusion

This does not exhaustively cover JBrowse, and the tool is more extensible than can be easily documented, but hopefully these examples are illustrative and can give you some ideas about your next steps.


topic: dev
tutorial: tool-generators
title: ToolFactory: Generating Tools From Simple Scripts
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: []
content_hash: 33ce78b92c8baba1ad9a343e390ff0f6
last_modified: 2026-05-15T22:28:11.282528
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/tool-generators/tutorial.html

content:
The Toolfactory and these tutorials are for developers and researchers learning about Galaxy, who routinely develop their own analysis scripts using
bash, Python, Perl, Rscript or other common scientific scripting languages. The tutorials show a convenient way to bridge the gap between a
working command line script and a new tool that "wraps" that script so users can use it like any other tool in Galaxy.

The ToolFactory is a Galaxy tool. Tools are constructed through the normal Galaxy interface when it is run. It is distributed as a self-installing
configuration on a freshly cloned copy of the Galaxy source code. Generated tools are *immediately installed* and ready to run so you can see
what the end user will see. Jobs can be re-run to edit and update generated tools, so Galaxy becomes an integrated development environment for Galaxy tools.

A [`Hello Galaxy!` demonstration](https://youtu.be//DK1eKz5TRs4) using the ToolFactory is available if you'd like to see a walk-through of some of
the hands-on material in this tutorial. You can see whether it looks useful for your work and decide whether to read the material below.

This first tutorial is a slow introduction. For some developers, it may be too slow and the second tutorial may be a better place to start. This one steps
in some detail through the process of using the ToolFactory
to generate `Hello World!` style simple demonstration Galaxy tools.

The reader will soon learn if it might be adapted to their work. If so, an [Advanced ToolFactory tutorial]({% link topics/dev/tutorials/tool-generators-advanced/tutorial.md %}) is
available if the material here is relevant to your needs and you would like to learn more details about the different kinds of tools and features the ToolFactory offers.

Experienced galaxy tool developers already have specialised tools and training so may not gain much from this material.
Users new to Galaxy from other scientific disciplines not yet familiar with the manual tool development process,
may find the ToolFactory appliance useful for familiarising themselves with tool development in Galaxy.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Tools, tool wrappers and the ToolFactory in Galaxy.

Tools are the fundamental building blocks for analyses in Galaxy. Thousands are available in the [project tool library](https://toolshed.g2.bx.psu.edu) from
many different kinds of science. Galaxy
tools can be created from almost any Linux command line software packages or interpreter scripts. What happens
at tool execution is decoupled from the framework itself, so the framework is indifferent to scientific discipline or coding language.
This is one reason for rapid uptake in new data intensive scientific areas. In many cases, growing sets of discipline specific tools are attracting
whole new communities of scientists into the Galaxy community.

Tool execution is tightly constrained and secured. The only things that can be changed
before execution are tool form exposed user-controlled settings, and data input selections. Everything else is fixed.
This is ideal for non-programmer Galaxy users who rely on prepared tools for their work. Tools are
utterly predictable, to the point of being about as reproducible as any useful complex computing component is likely to be.
This is one of the strengths of the Galaxy framework for users requiring reproducible and shareable scientific analyses.

## Galaxy Interactive Tools and Interactive Environments (GxIT/GIE)

In contrast to tools, GIE allow unconstrained scripting in a Galaxy environment. They offer complete freedom from the constraints of existing tools for
appropriately skilled researchers and developers, because they allow code to run inside Galaxy that is not available in any existing tool.
These can be shared and reused and GIE can now be run in workflows.

If a shareable tool that performs exactly the same functions as a GIE script is needed, the code can be extracted and turned into a parameterised command line script.
Any functional script can be turned into a typical Galaxy tool.

## Pathways from scripts to tools

Any command line script that runs correctly with sample test data input files and parameter settings can be turned into a Galaxy tool.
These may be derived from notebook scripts that have been consolidated and rewritten to take positional or argparse style parameters and tested
in a shell using small data input files. Alternatively, skilled users can develop scripts and test them using small input data files on the command line
without using Galaxy.

The Galaxy developers support extensive software infrastructure for manually creating new tools including Planemo and the new Galaxy language server.
These are complex and powerful with substantial
learning curves but can be used to turn almost any software package into a tool.

For those new to Galaxy, in many simple cases, it may be possible to generate a new tool "wrapping" a script in a few minutes, using an XML code generator for tool wrappers
in a specialised Galaxy tool for developers.

---

# The ToolFactory development server

The ToolFactory implements an automated, form driven XML document generator, and an installer for newly generated tools so you can try them straight away
in Galaxy. The ToolFactory can be "popped up" as a docker container conveniently but all work must be exported and saved before shutting down because no changes are persisted in the docker image. It can be installed locally by cloning and running the setup script from the git repository - ready to run in about 20 minutes as a fully functional, local, throw-away Galaxy development server.

> <tip-title>The ToolFactory installs and configures a fresh development server clone.</tip-title>
> - Private local desktop Galaxy server or docker image
>     - ideal for tinkering and experimentation
>     - learning how the Galaxy server works and
>     - developing new tools for new kinds of scientists.
> - Any Galaxy tool from the toolshed can be installed and used.
> - Simple scripts can have tool wrapper XML generated, and installed for testing.
> - They appear as a shareable Toolshed ready archive in the history.
> - Newly generated tools appear in the tool menu after a refresh, and can be viewed and used as the user will see them.
> - Tool generation jobs can be rerun using the {% icon galaxy-refresh %} button on the history item after editing the form to make changes to the tool the user will see in Galaxy.
> - The development server is a self-installing clone of the Galaxy code, configured with the ToolFactory and sample tools to explore.
>    - It can be backed up and persisted for as long as required, or it can be treated as a throw-away instance and deleted when no longer needed.
>    - There is almost zero technical friction. Only time is required to initialise the server.
>    - On a modern workstation or well-endowed laptop, it will perform well out of the box.
>    - It is suitable only for development in a private deployment.
>    - Please, never expose as a public server.
{: .tip }

The server was developed for programmers who need scripts they write turned in to new Galaxy tools for their own use and if sufficiently useful,
for others to share. Any user comfortable with scientific or general scripting languages on a Linux command
line may find it useful if they ever need a Galaxy tool that wraps a working script. Linux command line utilities and scripting language interpreters supported by Conda can be used.
Some Conda packages can also be used without a script, but the focus is on scripts.

Generated tools pass Planemo lint and are functionally indistinguishable from equivalent
manually written tools. The tested toolshed archives contain a test based on the test data provided at tool generation.

Working demonstration scripts are provided that use bash, Python, Rscript, Lisp, Prolog, Perl, sed, BWA and samtools, as described below.
Many demonstrate ToolFactory features and all can be updated and changed easily, supporting learning by experimenting.
More useful tools can be developed using more complex scripts and as many inputs, outputs
and user supplied parameters as needed by that script. Note that many tool complexities are not easily automated, so the
XML generator provides only limited features. Tools using those limited features may still be useful in many situations but a specialised
tool developer will be needed for many requirements.

If you are a scientist/programmer or software developer new to Galaxy and new to the dark arts of Galaxy tool building, this tutorial may be of help.
It introduces an automated way to convert any useful script into a toolshed ready tool, quickly *inside* Galaxy.


> <tip-title>Alternative ways to generate and see tools:</tip-title>
> - The [Galaxy Language Server](https://github.com/galaxyproject/galaxy-language-server)
>   - Undergoing rapid development.
>   - Specialised semi-automated tool building environment with VCS bindings.
> - Planemo can manually [generate and serve tool XML](https://planemo.readthedocs.io/en/latest/writing_standalone.html) with an optional test.
>    - Recommended for developers who will focus on building Galaxy tools on the command line.
>    - Outstanding documentation.
>    - Widely used. Requires relatively little time to figure out - Galaxy tool syntax takes longer.
>    - No GUI, although can serve tools on a web port.
>    - Command line only. Can create archives with additional steps.
>    - Need to pass all i/o and parameter details at once on the command line.
>    - Takes longer to learn to use and less accessible to many users than a form driven GUI might be.
>    - Manual XML editing required for selects and collections.
>    - See the recommended next steps at the end of this tutorial for Planemo related training.
> - Many Unix utilities (sed, awk...) are already available as IUC tools.
>    - They are `generic` in the sense that a script is supplied at **run time** by the tool user.
>    - This is possible with the ToolFactory, but for reproducible workflows, a specific script can be permanently built-in to make a reproducible tool.
>        - The user supplies only those I/O and parameter values specified on the ToolFactory form.
>        - Nothing else can be changed - just like with most Galaxy tools.
> - Choose whichever one fits best for the task at hand.
{: .tip }

---

# `Hello World!` with the ToolFactory.

A `Hello World!` Galaxy tool is a good place to start, just like any other new programming environment.
It requires planning and preparation. The ToolFactory can automate the generation of a wrapper, but the developer must supply a working script and
configure the inputs, outputs, user supplied parameters and metadata for the tool to be useful.

## Planning the new tool

A very simple bash script can be used to say "hello" but we make it a little more like a real
Galaxy tool by adding a text box so the user may designate whatever they want to add after that
such as "Hello, Galaxy Training Network".

Save the following sample as `hello.sh`:

 > <code-in-title>Starting bash script: Hello World</code-in-title>
 > ```bash
 > #!/bin/bash
 > echo "Hello $1!"
 > ```
 {: .code-in}

Test it on the command line by running:

`bash hello.sh ToolFactory`

In this case, `ToolFactory` is the first command line parameter. `Hello ToolFactory!` should appear as the output.

Once the script works and produces the expected outputs, the next step is to plan how the generated tool form should look to the user when run as a Galaxy tool.
In this case, a single text string is needed from the user.

Tool definition involves configuring the major sections of the ToolFactory form for the new tool.

Five categories needed to generate a script:

- Conda dependency requirements
- History data inputs
- History outputs
- User controlled parameters
- Developer supplied code/scripts to embed

For the `hello` tool case:

Conda dependency requirements:

- There are no dependencies usually because bash is available and version is not important.
- For completeness, it could be included as a Conda package. It's your tool.

History data inputs:

- This tool requires no history input files.

History outputs - data and collections:
- It produces one text output file.

User controlled parameters:
- The tool form should show a single input text field for the user to supply.

Developer supplied code/scripts to embed:
- These are optional - many Conda packages will not need them.
- Most simple use-cases will involve developer supplied, known working code.
- Executing the tool is expected to write the decorated string to a new history item.
- This must be known to work with suitable inputs on the command line. 
  - Broken code == broken tool.
- Such as `echo "Hello $1!"` as a Bash script
- There are other advanced features, such as command over-rides where code can be embedded

At this point, the plan for this new tool is:

- The Galaxy user should see a helpfully labelled text input field on the tool form, and the usual tool `execute` button.
- When the tool executes, that text should be passed to the script running under bash, as the first positional parameter.
- The script output should appear as a new output file in the history.
- It should contain the expected decorated input text.
- Galaxy tools need a test.
   - A simple test would be to supply a default value for the text string, run the tool and check that the output is correct.

## Putting the plan into action using the ToolFactory

The form collects all the information needed for a new Galaxy tool. It is long and complex as a result, particularly with many repeated form elements for more complex tools.
Much of what is collected is used to construct a command line for the script when the generated tool runs.
Other information such as the name and dependencies are needed to construct the relevant
sections of the generated XML file in the toolshed archive. The ToolFactory form configured to generate the `Hello` example can be viewed below.

> <details-title>Detail to explore: Annotated ToolFactory form for the `Hello World` example</details-title>
> ![First part of the form](../../images/ToolFactory_hello1form.png "The first part of the form collects the new tool name and dependencies to be installed. In this case, no Conda dependency is used. bash can be specified as a conda dependency, but it is not very version dependent and usually available. Reproducibility is not an issue for this trivial example. When it is, specify the dependencies and their versions here and the generated tool will always use them. If run in a shell, the bash script <code>echo "Hello $1"</code> in the text box will emit a string that includes the first command line parameter - such as "Hello Galaxy Training Network" This will be collected from STDOUT (configured below) into a new history output file (named and configured below). Positional parameters are chosen so the first parameter on the command line will be emitted when the script runs.")
>
> ![Second part of the form](../../images/ToolFactory_hello2form.png "The second section shows the new generated history output. It uses the special name <code>STDOUT</code> so the tool will take whatever the bash script writes and create a new text file called <code>Hello_output</code>. When the test is generated, the pass criterion is that the default value <code>Galaxy Training Network</code> should appear as the message in <code>hello_output</code>. no difference. Other criteria including <code>sim_size</code> are available for the test applied to each output file. There is no limit (other than your patience) to the number of new generated history outputs. Note that this example has no history input files. Again, any number of these can be specified on the form using the repeat.")
>
> ![Third part of the form](../../images/ToolFactory_hello3form.png "The third section shows the form settings for the user supplied parameter to be passed in to the bash script on the command line. It will be the first positional parameter because the ordinal position is 1. Argparse parameters are shown in other samples. The help and label text for each input file and user defined parameter will appear on the generated tool form for the user so make them informative. This is where you can change the default from "World" to "Galaxy Training Network" on the sample provided and regenerate it to make a new tool later in the tutorial.")
>
{: .details }


The generated tool XML appears in the history after the ToolFactory is executed and the tool itself is installed in the `Local Tools` submenu.
Text on the form is specified in the XML and it all comes from the ToolFactory form.

> <details-title>Detail to explore: Generated XML and tool form</details-title>
>
> [Galaxy XML documentation is here](https://docs.galaxyproject.org/en/latest/dev/schema.html)
>
> Note how text from the form appears in the generated tool XML
>
> ```xml
> <tool name="hello_toolshed" id="hello_toolshed" version="0.01">
>   <!--Source in git at: https://github.com/fubar2/galaxy_tf_overlay-->
>   <!--Created by planemo@galaxyproject.org at 22/01/2021 13:48:27 using the Galaxy Tool Factory.-->
>   <description>Says hello</description>
>   <stdio>
>     <exit_code range="1:" level="fatal"/>
>   </stdio>
>  <version_command><![CDATA[echo "0.01"]]></version_command>
>  <command><![CDATA[bash
>  $runme
>  "$sayhelloto" > $Hello_output]]>
>  </command>
>   <configfiles>
>     <configfile name="runme"><![CDATA[
>  echo "Hello $1"
>  ]]></configfile>
>   </configfiles>
>   <inputs>
>     <param label="Say hello to" help="" value="Galaxy Training Network!!" type="text" name="sayhelloto" argument="sayhelloto"/>
>   </inputs>
>   <outputs>
>     <data name="Hello_output" format="txt" label="Hello_output" hidden="false"/>
>   </outputs>
>   <tests>
>     <test>
>       <output name="Hello_output" value="Hello_output_sample" compare="diff" lines_diff="0"/>
>       <param name="sayhelloto" value="Galaxy Training Network!!"/>
>     </test>
>   </tests>
>   <help><![CDATA[
>
> **What it Does**
>
> ToolFactory demonstration - hello world in Galaxy
>
>
>
> ------
>
>
> Script::
>
>     echo "Hello $1"
>
> ]]></help>
>   <citations>
>     <citation type="doi">10.1093/bioinformatics/bts573</citation>
>   </citations>
> </tool>
> ```
>
> Which, when seen loaded into Galaxy looks like an ordinary tool:
> ![Generated form seen by the new tool user](../../images/toolfactory_hello_demo_form.png "The form displayed when the generated Hello tool is executed is below. The user sees a text box to enter any string. When executed, it will be echoed to a new history file called <code>Hello_output</code>")
> When a user runs the tool and enters some text in the text box, the decorated output will appear in a new history text dataset.
> This may not seem very exciting but it provides a useful pattern that can easily be adapted and extended.
> The script could do something far more interesting and could take unlimited input datasets, user configurable parameters and can produce as many outputs as needed in the history.
>
{: .details}


> <tip-title>If this is confusing</tip-title>
>
> If you are not yet familiar with the basics of Galaxy tools covered in the
> [tool integration training material]({% link topics/dev/tutorials/tool-integration/slides.html %}), the example form and XML
> will be confusing. You may gain more by reviewing the introductory part of that material, and then coming back here?
> It's a lot to learn and it is complicated. While a form driven code generator can hide much of the complexity of generating the code,
> the user must supply valid inputs for the code to be useful.
>
{: .tip}

## Extending this trivial example

This demonstrates a script based model, that can be extended to do more useful things with more complex scripts. More useful tools will ingest one or more user supplied input files, emit more complex outputs such as collections, and allow more user controlled parameters. 
These ToolFactory features are illustrated in the examples, and discussed in the advanced Tutorial.

Bash was used for this demonstration. Literally any scripting language, or any other useful package available in Conda, can be made available for tool execution by adding the appropriate dependency to the ToolFactory form.

> <comment-title>ToolFactory limitations and scope</comment-title>
>  
> The ToolFactory includes a very simple, automated, form driven XML code generator. Code automation is limited to the most common and easily implemented features. For example, conditionals are not available, so many complicated tools cannot be automatically generated with this tool. Send code.
> - It works best wrapping simple R/Bash/Python and other interpreted scripts, with few user supplied parameters and a few input and output files.
> - Scripts are easier than some Conda packages
>   - Scripts can often be modified to suit any ToolFactory limitations.
>   - Advanced tool components such as conditional logic and related tricks, requiring manual coding, can sometimes be worked around.
>         - where those features are needed, a skilled developer will be required.
>   - Many Conda dependencies will require XML conditionals or other tool XML constructs that are not easy to generate automatically.
>   - While some simple requirements may be manageable, complex ones will not be suitable for the ToolFactory.
> - Compared to the more usual shell and a text editor, The ToolFactory in Galaxy is a slow and clumsy way to debug your scripts.
> - **Starting a new ToolFactory tool with a know good command line and data** is strongly recommended.
>      - You will know exactly what to expect from the tool test for a first sanity check.
> - Corrolary: Unless there is a working script that needs to be wrapped into a toolshed-ready Galaxy tool, the ToolFactory is of little use.
{: .comment}

---

# Installation Options


> <warning-title>Security advisory!</warning-title>
>- *Please do not install the ToolFactory on any public server*
>- Configured as a default server, it has none of the usual additional security layers required for a safe public Galaxy server.
>- Although it will only run for administrative users, it allows unlimited scripting and exposes unwanted risk.
>- Install it locally and do not expose to the public internet.
>- For this reason, the training materials can't make use of existing public Galaxy infrastructure like most of the GTN material.
{: .warning}


> <hands-on-title>Installing and managing a Galaxy ToolFactory development server</hands-on-title>
> 
> # Logging in as an administrator to a new ToolFactory server
> 
> Once you have a working installation running, as described below, the server should be ready after 20-30 seconds, at [http://localhost:8080](http://localhost:8080).
> *The ToolFactory will only execute for administrative users as a minimal security barrier.* 
> When the webserver starts, immediately login using the administrative user email *toolfactory@galaxy.org* with the password *ChangeMe!* which of course you should change.
>   
> # Docker
> Note: *Nothing is persistent in the image*. Useful work must be manually exported and saved.
> For following the GTN tutorial and for test driving the ToolFactory for the first time, the docker version is recommended.
> Non-persistent means it does not remember anything after you stop the container. The next time it starts it will be a fresh installation.
> You can save your work by exporting the histories and tool tarballs you want to keep. You need the history to rerun the job that generated a tarball, so the history is the most important thing to preserve if you make a useful tool.
> The image must be pulled first, then run with port 8080 open for the Galaxy server.
> 
> ```
> docker pull quay.io/fubar2/galaxy_toolfactory:latest
> docker run -d -p 8080:8080 quay.io/fubar2/galaxy_toolfactory:latest
> ```
> Check the docker logs until gunicorn is ready to serve or wait 
> about 20-30 seconds, then browse to [http://localhost:8080](http://localhost:8080)
> If a Galaxy server appears, proceed with the login instructions above and you should see a history containing all the example tools. There is also a workflow that can reproduce that history if you set all the inputs to the right input datasets in that history.
>  
> # Local workstation development Galaxy server installation
> 
> A persistent desktop development ToolFactory server can be built by cloning the [bootstrap github repository](https://github.com/fubar2/galaxy_tf_overlay), and using the included *localtf.sh* script, to build a complete, new development server with the ToolFactory installed and ready to run.
>
> From a convenient directory, download the overlay configuration code repository, then
> run the *localtf.sh* setup script from that cloned repository root directory:
>
> ```
> git clone https://github.com/fubar2/galaxy_tf_overlay.git
> cd galaxy_tf_overlay
> sh ./localtf.sh
> ```
> Running *localtf.sh* will create a new directory, *galaxytf*, in the parent directory of the *galaxy_tf_overlay* clone.
> The script will configure a fresh Galaxy 23.0 server, with the ToolFactory installed, in that new directory.
> This takes twenty minutes or more to complete since the client must be built once. Visualisation plugins are not built to save some time.
>
> The resulting development server directory will occupy ~9GB of disk space, so be sure your machine has plenty of room.
> It will be based in a single directory, *galaxytf* in the same level as the cloned galaxy_tf_overlay repository directory, where the script should be run as shown above.
>
> Rerunning the *localtf.sh* script will *destroy the entire galaxytf directory* - all ~9GB, and create a clean new installation.
> It should only need to be run once in the life of the development server.
>
> Remove the *galaxytf* directory to remove the entire development server when it is no longer needed. 
> Before doing that, save any useful histories, because the jobs in a ToolFactory history can be used to update a tool, because that history can be imported into a fresh development instance when needed and the job rerun with the necessary adjustments to the form.
>
>
> Once installation is complete:
>  * start the server from the *galaxytf* directory with *sh run.sh*. The logs will be displayed.
>  * ^c (control+c) will stop it from the console.
>  * In 23.0, *.venv/bin/galaxyctl start* and *.venv/bin/galaxyctl stop* should work.
>
>
{: .hands_on}


## Exploring the ToolFactory

- The best way to understand what can be done, is to look at the sample tools in the default administrator initial history.
- As you explore the forms for each sample tool, you can see how the various options have been configured and what kinds of scripts or Conda packages this could be used for in your work.
- The example script can be swapped out for another one known to work and additional new parameters added to suit, to extend the toy examples and create tools of use to your users.
- Change the tool name when you do this on the newly edited form, then press `execute`
  - The new wrapper XML will appear
  - The new tool will be installed in the `Local Tools` submenu.
- If the tool name is not changed before re-generating a tool, the original installed tool will be updated with the new configuration. The old job can still be rerun from the history if necessary. Galaxy can be a clumsy but not entirely useless integrated development environment.

---

# Hello World!

> <hands-on-title>Building the Hello World example</hands-on-title>
>
> 1. Run {% tool ToolFactory %} with the following parameters:
>    - "Dependencies, optional script and script interpreter"
>      - *"Interpreter for the script"*: `bash`
>      - *"Script for executable above to interpret"*: `echo "Hello $1"`
>    - "Data file input, output and settings forming the executable or script command line"
>      - *"Command line parameter passing method to use"*: `positional`
>      - "Input and output files"
>        - {% icon param-repeat %} *"Insert Outputs"*
>          - *"Name for this output to appear in new history"*: `Hello_output`
>          - *"Select the datatype for this output"*: `txt`
>          - *"Position"*: `STDOUT`
>      - "Arguments"
>        - {% icon param-repeat %} *"Insert Command Line Parameters"*
>          - *"Choose the name for this parameter - MUST not be blank!"*: `say_hello_to`
>          - *"Enter this parameter's label for the form"*: `Say hello to`
>          - *"Positional ordinal \| argparse argument name"*: `1`
>
> 2. Execute
>
> 3. Explore the outputs - do they match what you expected?
>
> 4. Refresh the page - click the home icon (or the "Analysis" tab) - to see the new tool in the `Local Tools` section of the tools menu.
>
> 5. Run the tool that has been added - Select the new tool and examine the form. Check that all the changes are as they should be.
>
{: .hands_on}

## The Development Cycle

1. Test on the command line and confirm it produces correct output with defaults and test data.
1. In the development server, start a new history
1. Upload all input samples used on the command line if any, for use in the tool test.
1. Open the ToolFactory tool form.
    1. Define the tool metadata, dependencies, interpreter and optionally, paste the script.
    1. Add the required history inputs using the small samples as examples.
    1. Specify all the output files to be created in the user's history.
    1. Add any user adjustable command line parameters such as text fields.
    1. Look at the samples to see how the ToolFactory form can be used.
1. Execute the tool when the form is completed.
1. When the job is complete, refresh the page (Home icon or Analysis tab). The new tool will be found in the `Local Tools` section, ready to run.
1. Run the new tool and check that it does what you expect, or re-generate after adjusting the form settings as needed.
1. If it needs any changes, open the XML history item created when the tool was generated and use the {% icon galaxy-refresh %} rerun button to
recreate the ToolFactory form as it was when you last ran it. Adjust as needed and use the tool form`Execute` button to run the ToolFactory again with updated settings.
    1. Rinse, repeat.
1. Warning: generating a tool with an existing tool id such as `mytool` will overwrite the installed version of any previously generated tool with id "mytool".
    1. Persisted jobs in user histories allow previous versions to be recreated to restore older versions.

Galaxy can be used as an Integrated Development Environment for tools - clunky but oddly satisfying.
Note this is distinct from debugging the script - that is not at all satisfying in Galaxy unless you like waiting for jobs to finish.

A shell is much better for that.

![Galaxy as an IDE for tools with the ToolFactory](../../images/ToolFactory_big_picture.png "Galaxy can be used as a tool development environment for users who can write their own scripts as shown in this process overview slide.")

## Hello World: Continued

> <hands-on-title>Modifying the Hello World example</hands-on-title>
>
> 1. Rerun the output of your previous job, and make the following changes
>
>    - "Dependencies, optional script and script interpreter"
>      - *"Script for executable above to interpret"*: `echo "Hello $1"; echo "Goodbye $2";`
>    - "Data file input, output and settings forming the executable or script command line"
>      - "Arguments"
>        - Add a second {% icon param-repeat %} *"Insert Command Line Parameters"*
>          - *"Choose the name for this parameter - MUST not be blank!"*: `say_bye_to`
>          - *"Enter this parameter's label for the form"*: `Say bye to`
>          - *"Positional ordinal \| argparse argument name"*: `2`
>
{: .hands_on}

## Hello Collections!

> <hands-on-title>Building a File Splitter</hands-on-title>
>
> 1. Run {% tool ToolFactory %} with the following parameters:
>    - *"New tool ID and title for outputs"*: `file_splitter`
>    - "Dependencies, optional script and script interpreter"
>      - *"Interpreter for the script"*: `bash`
>      - *"Script for executable above to interpret"*:
>
>        ```bash
>        mkdir -p outputs/;
>        split --lines=$2 --additional-suffix=.txt $1 outputs/
>        ```
>
>    - "Data file input, output and settings forming the executable or script command line"
>      - *"Command line parameter passing method to use"*: `positional`
>      - "Input and output files"
>        - {% icon param-repeat %} *"Insert Inputs"*
>          - *"Select an input file from your history"*: Choose any XML file from your history, we'll use this as an example
>          - *"Select the datatype for this output"*: `txt`
>          - *"This will become the user prompt for the form so please make it informative"*: `File to split`
>          - *"Positional: ordinal integer. Argparse: argument name. STDIN if the executable/script expects it"*: `1`
>        - {% icon param-repeat %} *"Insert Output Collections"*
>          - *"Select the kind of collection for this output"*: `List`
>          - *"Label for this collection"*: `File Parts`
>      - "Arguments"
>        - {% icon param-repeat %} *"Insert Command Line Parameters"*
>          - *"Choose the name for this parameter - MUST not be blank!"*: `lines`
>          - *"Select the type for this parameter"*: `Integer`
>            - *"Enter this parameter's default integer value"*: `4`
>          - *"Enter this parameter's label for the form"*: `Number of lines in each split file`
>          - *"Positional ordinal \| argparse argument name"*: `2`
>
> 2. Execute
>
> 3. Refresh the Galaxy Page and locate your `collections_test` tool
>
> 4. Run the tool, selecting any text file in your history, e.g. the XML output from the ToolFactory that created this tool.
>
> 5. You should see a collection filled with files named `aa` to `ao` (or so), each with 4 lines from your file.
{: .hands_on}

This is presented as a motivating example. It is up to you to imagine what else you might be able to accomplish with collection outputs!
You could extend this example to split a `.fastq` file which could speed up your processing, or you could use collections to store extra images or plots produced by your tool.
They are also a convenient way to keep outputs together that are unlikely to be used downstream in a workflow, such as reports and images
that a user might want to be able to easily view if they want without cluttering up the history.

> <warning-title>Collection Testing</warning-title>
> The default generated test for output collections always passes because it doesn't test anything.
> Supplying a test over-ride is recommended for collections.
> For a real test, one or more expected <element.../> tags must be provided so the test really does test something.
{: .warning}

## Done!

> <hands-on-title>To safely shut the server down</hands-on-title>
>
> 1. Type <kbd>Ctrl-C</kbd> in the terminal where you ran `sh run.sh`.
>
{: .hands_on}

### Limits, workarounds and value proposition

- The ToolFactory Appliance is a slightly clumsy but useable way to create, test and maintain Galaxy tools in a web GUI.
- The ToolFactory tool is an automated code generator installed in the appliance tool menu
- No generator can replace manual editing by a skilled developer other than in constrained simple cases.
- These are common enough in the daily work of most data intensive scientific fields to make a tool generator potentially worth keeping handy.
- For simple scripts and appropriate Conda packages, a professional Galaxy tool developer can probably do it quickly by hand, but those skills take time to acquire
and are not widely available, particularly in scientific fields coming to Galaxy.
- Tools can have command-override and test-override pasted in as in one of the BWA samples.
   - This can solve some of the limitations but if it is needed, it might be better to prepare the wrapper manually if a skilled developer is available.
   - Any logic in the `<command>` section can probably always be replaced by equivalent code in a script at the cost of time and effort compared to templating.
   - Other aspects of tool logic such as output filters based on other parameter values can only be implemented in the wrapper document, not in a tool script.
- The ToolFactory can help new scientists and developers to quickly get some simple tools working for their colleagues while awaiting help with the complex ones.

# Next Steps

Expand your knowledge further with the [Advanced ToolFactory tutorial]({% link topics/dev/tutorials/tool-generators-advanced/tutorial.md %})

# Acknowledgements

This tutorial is based on the work of thousands of individual contributers to the Galaxy project over the last 15 years or so.
Thanks all! It has been a lot of fun.

Special thanks to:

- @gtn:hexylena for
    - review and contribution to the tutorial and associated code.
    - the vision of instant installation of generated tools for developer feedback.
    - elegantly generated lint-free XML provided by [galaxyml code](https://github.com/hexylena/galaxyxml)
- @gtn:mvdbeek for thoughtful comments on the role of the ToolFactory that helped motivate the tutorial.


topic: dev
tutorial: debugging
title: Debugging Galaxy
url: https://training.galaxyproject.org/training-material/topics/dev/tutorials/debugging/tutorial.html
difficulty: intermediate
time_estimation: 4h
tools_json: []
requirements_json: ["type: none", "type: none", "type: none"]
tags_json: []
content_hash: b0ee2326deeeb5de5e183d044ea33ac9
last_modified: 2026-05-27T09:01:48.026287
workflows_json: []
content:
In this tutorial we will demonstrate how to find and fix common types of bugs you may encounter as a contributor to Galaxy. We will step you through the process of finding and fixing a bug - from locating specific errors in the logs of Galaxy's GitHub Actions, to identifying their cause, developing a solution and committing your edits

With the skills from this tutorial, it is our hope that you will feel more prepared to develop solutions for Galaxy and more confidently navigate any obstacles along the way.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Local development environment setup

For almost every type of error you encounter, there will be a relatively similar process:

1. Locate failing test or error on GitHub Actions
2. Run failing test locally
3. Identify issue causing failure
4. Develop a solution
5. Ensure test passes locally
6. Commit and push fixed branch to local fork

The process for dealing with runtime errors will be somewhat different, since there will be no failed test to review initially. However, in either case, the first step is to setup your local development environment.

## Contributing to Galaxy

{% snippet topics/dev/faqs/contributing.md %}

In this tutorial, instead of cloning Galaxy's default branch, we will clone a branch that has been created for this tutorial and contains several bugs that you'll need to identify and fix. Also, we won't be opening a pull request in the upstream Galaxy repository.

## Getting started: Pulling the buggy branch

Good news! All of the failing tests are on the same Galaxy branch! That means you only need to download and work off of one branch. The downside is that this is one buggy branch with issues all over the place. We have no doubts that you can solve them though!

> <hands-on-title>Setup your local Galaxy instance</hands-on-title>
>
> 1. Use GitHub UI to fork Galaxy's repository at `galaxyproject/galaxy`.
> 2. Clone your forked repository to a local path, further referred to as `GALAXY_ROOT` and `cd` into `GALAXY_ROOT`. Note that we specify the tutorial branch with the `-b` option:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git clone -b training2022 https://github.com/<your-username>/galaxy GALAXY_ROOT
>    > cd GALAXY_ROOT
>    > ```
>    {: .code-in}
>
>    Note: In the future, when you contribute to Galaxy, you'll need to clone the `dev` branch, which is the default, so you don't need to specify the `-b` option in the `git clone` command).
>
> 3. Before we can use Galaxy, we need to create a virtual environment and install the required dependencies. This is generally done with the `common_startup.sh` script:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > bash scripts/common_startup.sh --dev-wheels
>    > ```
>    {: .code-in}
>
>    {% icon warning %} Make sure you are using a [supported Python version](https://galaxyproject.org/admin/python/) (you can check your Python version with `python --version`). If your system uses an older version, you may specify an alternative Python interpreter using the `GALAXY_PYTHON` environment variable (`GALAXY_PYTHON=/path/to/alt/python bash scripts/common_startup.sh --dev-wheels`).
>
> 4. Activate your new virtual environment:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > . .venv/bin/activate
>    > ```
>    {: .code-in}
>
>    Once activated, you'll see the name of the virtual environment prepended to your shell prompt: `(.venv)$`.
>
> 5. Finally, let's create a new branch for your edits:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git checkout -b my-training
>    > ```
>    {: .code-in}
>
>    Now when you run `git branch` you'll see that your new branch is activated:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > git branch
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > ```bash
>    > >   training2022
>    > > * my-training
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
>    Note: `my-training` is just an example; you can call your new branch anything you like.
{: .hands_on}

# Unit test failure

Most of Galaxy unit tests are designed to test a separate component or function, are very fast, and do not require a running Galaxy instance. Finding the issue causing a unit test to fail is relatively straightforward.

> <hands-on-title>Fixing a failing unit test</hands-on-title>
>
> 1. **Finding the failing test on GitHub**
>
>    Head over to the [Galaxy repository on GitHub](https://github.com/galaxyproject/galaxy) and in the branch drop-down select the "training" branch. Click the red "x" next to the last commit, you'll get a drop-down listing of workflow results; find "Unit tests / Test (3..." and click "Details". This will display a detailed report that includes information about the failed tests. Try to find the failed tests.
>
>    > <solution-title></solution-title>
>    > The log lists 2 failed tests:
>    > ```
>    > - FAILED lib/galaxy/model/tags.py::galaxy.model.tags.TagHandler.parse_tags
>    > - FAILED test/unit/util/test_utils.py::test_strip_control_characters
>    > ```
>    > The first is a doctest. While it will provide us with hints on what may have caused the error, the unit test is generally more helpful.
>    {: .solution }
>
> 2. **Running the test locally**
>
>    An essential requirement for productive development is a fast feedback loop: we want to make a change, run a test, and get immediate feedback. If we push our edits to the remote fork (or submit a pull request to the upstream repository) and wait for the the tests to run remotely, it could take hours before we get any feedback, which is very inefficient (and detrimental to sustaining the joy of programming).
>
>    Instead, we will run the tests locally. Furthermore, we will run a specific test that we know is failing: this will give us the instant feedback we need.
>
>    > <code-in-title>Bash</code-in-title>
>    > Make sure you are in `GALAXY_ROOT` and your virtual environment is activated. If not, activate it:
>    > ```bash
>    > . .venv/bin/activate
>    > ```
>    {: .code-in}
>
>    The simplest way to run the failing test locally is using pytest directly:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > pytest test/unit/util/test_utils.py
>    > ```
>    {: .code-in}
>
>    Do you see the error message?
>
>    > <solution-title></solution-title>
>    > The test gives us a very specific error message:
>    > ```
>    > AssertionError: assert 'b l a' == 'bla'
>    > - bla
>    > + b l a
>    > ?  + +
>    > ```
>    > This means that instead of the expected `bla` we got `b l a` (with spaces).
>    {: .solution }
>
> 3. **Finding and fixing the issue**
>
>    Our next step is to open the test file and see what part of the codebase (i.e., what module, class + method or function) is failing to provide the expected result.
>
>    The failing test is calling the `strip_control_characters` function in the `util` module. That function, apparently, is not producing the expected output. Let's head over to `lib/galaxy/util/__init__.py`, where we'll find the function definition:
>
>    ```python
>    def strip_control_characters(s):
>        """Strip unicode control characters from a string."""
>        return " ".join(c for c in unicodify(s) if unicodedata.category(c) != "Cc")
>    ```
>
>    The function takes a string as input, and, in theory, returns a copy of that string with certain characters stripped away (it doesn't really matter what characters are stripped away). However, from the test we see that the string is modified in a different way: spaces have been added between the characters.
>
>    Can you see what's causing this behavior?
>
>    > <solution-title></solution-title>
>    > The genexp is evaluated and subsequently joined using a space as a delimiter. The space is, obviously, a typo: we don't want to _add_ anything to the string, so the delimiter should be the empty string instead:
>    > ```python
>    > return "".join(c for c in unicodify(s) if unicodedata.category(c) != "Cc")
>    > ```
>    {: .solution }
>
>    Make the change, save the file, run the same test. Now it should pass.
>
> 4. **Finishing up**
>
>    Now you are ready to commit your edits and push the branch to your fork. So, `git add [your modified files]` and `git commit -m "[insert your commit message here]"`. Please consult [Galaxy's contributor's guide](https://github.com/galaxyproject/galaxy/blob/dev/CONTRIBUTING.md#how-to-contribute) for details on how to write a useful and properly formatted commit message, and why it matters.
>
>    Congratulations - you found and fixed a unit test failure!
{: .hands_on}

# API test failure

API tests test various aspects of the Galaxy API, as well as general backend aspects of Galaxy using the API. These tests require a Galaxy instance.

> <hands-on-title>Fixing a failing API test</hands-on-title>
>
> 1. **Finding the failing test on GitHub**
>
>    We'll follow the same process we used in the unit test section. On GitHub, click the same red "x" and select the "API tests / Test (3..." workflow and click "Details". Try to find the failed tests.
>
>    > <solution-title></solution-title>
>    > The log lists multiple failed tests. Some of them were caused by bugs you'll be fixing in later sections of this tutorial. For this exercise, we care about these two failed tests:
>    > ```
>    > FAILED lib/galaxy_test/api/test_licenses.py::LicensesApiTestCase::test_404_on_unknown_license
>    > FAILED lib/galaxy_test/api/test_licenses.py::LicensesApiTestCase::test_get_license
>    > ```
>    > Note: occasionally, there could be more failing tests; however, for the purposes of this exercise we are concerned with the licenses tests.
>    {: .solution }
>
> 2. **Running the test locally**
>
>    Again, we will run the individual test locally. While it is possible to run the test directly with pytest, in this case we will use Galaxy's script `run_tests.sh`. The script optimizes test runs by sharing the same instance of Galaxy across multiple tests; without the script a new Galaxy will be started for each TestCase class. There are other reasons to run the script; for more information check the documentation block at the top of the script.
>
>    Make sure you are in `GALAXY_ROOT` (if your virtual environment is not activated, the script will do it for you).
>
>    Let's run the failed test:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ./run_tests.sh -api lib/galaxy_test/api/test_licenses.py
>    > ```
>    {: .code-in}
>
>    Two tests are failing. Do you see the error messages?
>
>    > <solution-title></solution-title>
>    > We see the following error messages:
>    > ```
>    > FAILED lib/galaxy_test/api/test_licenses.py::LicensesApiTestCase::test_404_on_unknown_license - AssertionError: Request status code (500) was not expected value 404. Body was INVALID JSON RESPONSE <Internal Server Error>
>    > FAILED lib/galaxy_test/api/test_licenses.py::LicensesApiTestCase::test_get_license - AssertionError: Request status code (404) was not expected value 200. Body was {'err_msg': "License 'Apache-2.0' not found", 'err_code': 404001}
>    > ```
>    > This means that instead of HTTP status code 200 (OK/success) we got a 500 (server error) and 404 (resource not found). Sometimes multiple test failures may be caused by the same error; this is the case here.
>    {: .solution }
>
> 3. **Finding and fixing the issue**
>
>    We can use either one of the failing tests to start debugging. Let's use `test_get_license`. As with the unit test, first let's take a look at the test function definition (located in `lib/galaxy_test/api/test_licenses.py`):
>
>    ```python
>    def test_get_license(self):
>        licenseId = "Apache-2.0"
>        response = self._get(f"licenses/{licenseId}")
>        self._assert_status_code_is(response, 200)
>        self._assert_matches_license_id(licenseId, response.json())
>    ```
>
>    The important line here is line 3: it tells us what API endpoint is called: `licenses/{licenseId}`.
>
>    Our next step is to examine the controller for this endpoint. For that, we head over to `lib/galaxy/webapps/galaxy/api/licenses.py`.
>
>    If you look at the FastAPI controller, you'll see the `get` method that is responsible for fetching the data in response to a `GET` HTTP request received at the `licenses/{licenseId}` endpoint:
>
>    ```python
>        @router.get('/api/licenses/{id}',
>            summary="Gets the SPDX license metadata associated with the short identifier",
>            response_description="SPDX license metadata")
>        async def get(self, id=LicenseIdPath) -> LicenseMetadataModel:
>            """Returns the license metadata associated with the given
>            [SPDX license short ID](https://spdx.github.io/spdx-spec/appendix-I-SPDX-license-list/)."""
>            return self.licenses_manager.get_license_by_id(id)
>    ```
>
>    The only thing in this method that matters to us is the last line:
>
>    ```python
>    return self.licenses_manager.get_license_by_id(id)
>    ```
>
>    Why? Because despite all this syntax, the method simply redirects the request to `self.licenses_manager`. If we read the code, we'll quickly see that this variable holds an instance of `LicencesManager`, which (look at the import statements at the top of the file) is located in `lib/galaxy/managers/licenses.py`. Which is where we should look next.
>
>    If we carefully trace the code in `LicensesManager`, we'll find the error. Do you see it?
>
>    > <solution-title></solution-title>
>    > The `get_license_by_id()` method raises an `ObjectNotFound` error (which results in the 404 status code we see in the test result). This error is a direct result of the LicensesManager `get` method called on the previous line (`license = self.get(id)`) - take a look at that method:
>    > ```python
>    >     def get(self, uri):
>    >         if uri not in self._by_index:
>    >             return self._by_index[uri]
>    >             ...
>    > ```
>    > Let's rewrite the first clause in pseudo code to expose the problem:
>    > ```
>    > if key not in dictionary:
>    >     return dictionary[key]
>    > ```
>    > Well, that makes no sense whatsoever! It should be the opposite: if key **is** in dictionary, then get the item with that key. The `not` is clearly a typo (or, more accurately, a wicked edit made for the purpose of setting up a learning experience). Remove this negation and rerun the test.
>    {: .solution }
>
> 4. **Finishing up**
>
>    Well, that was fun. Congratulations! Now commit your edit (don't forget to supply a useful commit message), and move on to client errors.
{: .hands_on}

# Client linting error

Client Linting is important to reduce errors, improve quality of the codebase, and enforce common rules. These rules are defined by the eslinter defined with Vue best practices, primarily. On Github, the command `eslint` is run to start the process for linting code on the client side.

> <comment-title>Python Linting</comment-title>
> For backend consistency, we also have a Python linter. While we won't be running you through that exercise today, you may encounter a Python linting issue in the future. The way to diagnose and treat these is very similar to the way to do it for the front end.
{: .comment}

> <hands-on-title>Fixing client linting error</hands-on-title>
>
> 1. **Finding the failing test on GitHub**
>
>    With this error, one of the failing test workflows on GitHub would normally be “get_code_and_test”. However, since we are working on a branch that contains multiple bugs by design, some of the test failures overlap, as a result of which the “get_code_and_test” workflow has been skipped in the last commit.
>
>    Therefore, in this section, for the remote results, we'll have to describe what would have been displayed.
>
>    One of the failing tests on GitHub says "get_code_and_test". Clicking on the word "Details" assocated with this failure will open a brief view of the CircleCI Checks, with a list of tests that are run as part of that command. You'll see that the one that is failing is js_lint. And if you click that, you'll be navigated to the CicleCI app, where you can see much more information about the failures.
>
>    Here you can see the exact command that ran to lint our codebase:
>
>    ```
>    cd client && yarn run eslint
>    ```
>
>    We can also see that it has found 1 error, where that error is located, and a description of the problem.
>
> 2. **Running the linter locally**
>
>    Alright, so we've seen the failure, but we want to verify it locally now. For this, you'll need to open up a terminal.
>
>    > <code-in-title>Bash</code-in-title>
>    > Make sure you are in `GALAXY_ROOT` and your virtual environment is activated. If not, activate it:
>    > ```bash
>    > . .venv/bin/activate
>    > ```
>    {: .code-in}
>
>    Then you've got two options. You can run the same exact command that CircleCI uses, `cd client && yarn run eslint` or `make client-lint`, which does `cd client && yarn run eslint && yarn run prettier-check`. This will also make sure your code conforms to formatting standards.
>
>    After running that, you should see the same linting error from CircleCI; you should know where it is and how to fix it too!
>
> 3. **Finding and fixing the issue**
>
>    > <solution-title></solution-title>
>    >
>    > You are currently in `GALAXY_ROOT/client`.
>    > In the file `src/components/RuleBuilder/RuleComponent.vue`, there is an unused import. Simply delete the line that says `import Vue from "vue";`
>    >
>    > Alternatively, you can run `yarn eslint --fix` or `yarn run prettier`.
>    {: .solution }
>
>    If you think you've got it, try running the test locally again to be sure.
>
> 4. **Finishing up**
>
>    Whoo-hoo! If you're here, you've identified the error, ran the appropriate command locally, found the problem, fixed it, confirmed that the command runs without errors, and are ready to commit your change. Write a useful commit message like `Fix a client linting issue in RuleComponent` or something similar, making sure it is clear what you've done.
>    (See [Galaxy's contributor's guide](https://github.com/galaxyproject/galaxy/blob/dev/CONTRIBUTING.md#how-to-contribute) for details on how to write a useful and properly formatted commit message.)
>
>    Awesome work - you now understand how to solve client linting issues!
{: .hands_on}

# Client unit test failure

Client tests are tests written by developers to test front-end code. In the case of the test we're going to walk through right now, we're looking at a Vue test failure, which means the component that it's testing is definitely a Vue component.

> <hands-on-title>Fixing a failing client unit test</hands-on-title>
>
> 1. **Finding the failing test on GitHub**
>    In this section we, again, can't demonstrate the remote output, so we descibe it instead.
>    One of the failing tests on GitHub says "Client Unit Testing / jest". Clicking on Details beside that failure, will open up a the terminal output from that test. Here you should be able to see what test is failing.
>
>    > <solution-title></solution-title>
>    >
>    > The failing test file is `src/components/RuleBuilder/SavedRulesSelector.test.js`, and the failing test in that file is `SavedRulesSelector › disables history icon if there is no history`.
>    {: .solution }
>
> 2. **Running the test locally**
>
>    > <code-in-title>Bash</code-in-title>
>    > Make sure you are in `GALAXY_ROOT` and your virtual environment is activated. If not, activate it:
>    > ```bash
>    > . .venv/bin/activate
>    > ```
>    {: .code-in}
>
>    To run the client tests locally, you must be in the client directory; if you're just in the galaxy directory, `cd client` should take you to the right place.
>
>    Next, type `yarn run test` or `yarn jest` to run all the client tests.
>
>    The output should match what you found on Github.
>
> 3. **Finding and fixing the issue**
>
>    Next up, finding the error. Here are some tips:
>
>    - Take a look at the failing test and the imports to find out which Vue component the test correlates to.
>    - Look at the test to see what it does. Is there an error in the test?
>    - Look at the Vue component where the failure seems to be taking place. Is the error located there?
>
>    The solution is hidden below, but try your hand at fixing it first:
>
>    > <solution-title></solution-title>
>    > The failure is only happening on the disabled test case, so take a look in the template of `SavedRulesSelector` where the conditions for adding disabled are defined.
>    >
>    > The way it's worded in the test (and the way one could think about it logically), is that if there is no history, the button should be disabled, but the code says
>    >
>    > ```javascript
>    > :class="{ disabled: numOfSavedRules == 1 }"
>    > ```
>    >
>    > -- whoops! A very silly off by one error. On that very same line, change the 1 to a 0, and run the test again.
>    {: .solution }
>
>    If you think you've got it, try running the test locally again to be sure.
>
>    > <comment-title>Running fewer client tests</comment-title>
>    > If you don't want to run the whole suite of client tests, you can add keywords that match the test path/name. For example both `yarn run jest rule` and `yarn jest selector` will work, but the former will run other tests with rule in the path/module name.
>    {: .comment}
>
> 4. **Finishing up**
>
>    Nice Job! If you're here, you've found the problem, ran it locally, fixed the problem, ran it locally again without errors, and are ready to commit your change. Write a useful commit message like `Fix logic error in SavedRuleSelector` or something similar, making sure it is clear what you've done.
>
>    (See [Galaxy's contributor's guide](https://github.com/galaxyproject/galaxy/blob/dev/CONTRIBUTING.md#how-to-contribute) for details on how to write a useful and properly formatted commit message.)
>
>    Awesome work - you now understand how to solve client linting issues!- you now have a handle on client tests!
{: .hands_on}

# Selenium test failure

Selenium is the end-to-end or integration testing framework that we use; so these tests often are designed to test both frontend and backend code. This makes fixing them a little difficult, but we are sure you can do it!

> <hands-on-title>Fixing a failing Selenium test</hands-on-title>
>
> 1. **Finding the failing test on GitHub**
>
>    There may be actually a few failing Selenium tests, but the good news is, they are the result of only one little error. The failing tests on GitHub are in the "Selenium Test 3.7,0" and "Selenium Test 3.7,1" workflows. When you click on the details of this one, it's not always very clear which test is causing the failure, but if you download the artifacts, it's very clear which tests have failed.
>
>    Try to identify which tests are failing based on artifacts you downloaded.
>
>    > <solution-title></solution-title>
>    >
>    > The failing tests are:
>    >
>    > - `test_build_list_and_show_items`
>    > - `test_upload_pair_specify_extension`
>    {: .solution }
>
> 2. **Running the tests locally**
>
>    Make sure you are in `GALAXY_ROOT` (if your virtual environment is not activated, the script will do it for you).
>
>    We do not want to run the full suite of Selenium tests here - that would take a lot of time and we already know that most of them passed.
>
>    Instead, we can build our commands to only run a subset of the tests. Here's an example: `./run_tests.sh -selenium lib/galaxy_test/selenium/test_sign_out.py`
>
>    A command like that will still run all of the tests defined in that python file, so we need to find out where the tests that failed are defined. Try seaching the baseline for our three failing Selenium tests; what files are they in?
>
>    > <solution-title></solution-title>
>    >
>    > The failing tests are defined in these files:
>    >
>    > - `lib/galaxy_test/selenium/test_collection_builders.py`
>    > - `lib/galaxy_test/selenium/test_uploads.py`
>    {: .solution }
>
>    Now that we have the files, we can run the selenium tests. The command to do this should be very similar to `./run_tests.sh -selenium lib/galaxy_test/selenium/test_sign_out.py`. The only difference should be the python file where the tests are defined. Try it yourself, before checking out the solution.
>
>    > <solution-title></solution-title>
>    >
>    > Either of these two commands will work:
>    >
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > ./run_tests.sh -selenium lib/galaxy_test/selenium/test_collection_builders.py
>    > > ```
>    > > or
>    > > ```bash
>    > > ./run_tests.sh -selenium lib/galaxy_test/selenium/test_uploads.py
>    > > ```
>    > {: .code-in}
>    {: .solution }
>
>    Running any of these tests will show us that the tests are still failing on our local machine. So, you should see a failing test that matches one of the three we identified from the downloaded artifacts. Onward to fixing them!
>
> 3. **Finding and fixing the issue**
>
>    Good news - there's an easy way to figure out what's going wrong with a selenium test!
>
>    In your terminal type `export GALAXY_TEST_SELENIUM_HEADLESS=false`. This will allow you to see the test running in real time.
>
>    Re-run the selenium test to watch the headless browser step through the test.
>
>    Looking at the test run in the headless browser, alongside the definition for the test should help you understand what might be going wrong. Try and figure out where the problem is on your own, before looking at the solution.
>
>    > <solution-title></solution-title>
>    >
>    > The problem is that the tests are expecting a different HID (history identifier) than what is being made available to them. Since all the failing tests have to do with Lists, you might want to check out the ListCollectionCreator.
>    >
>    > For comparisons sake, open one of the other CollectionCreators, as well and see if you can spot anything that is amiss.
>    >
>    > Pay careful attention in the template to the <collection-creator> props.
>    >
>    > Here you should notice that in the PairCollectionCreator or in the PairedListCollectionCreator, `:hide-source-items="hideSourceItems"` instead of `:hide-source-items="!hideSourceItems"`. It's a little change, just flipping the boolean, but it makes a big difference!
>    {: .solution }
>
>    Awesome! If you're here, you've found and fixed the problem, at least, in theory.
>
>    If you think you've got it, try running the tests locally again to be sure. In this simulated case, you can get by with just running one of the tests, but in reality, you should probably run all three Selenium tests to be sure that it's totally fixed.
>
> 4. **Finishing up**
>
>    Excellent! If you're here, you've found the problem, ran it locally, fixed the problem, ran it locally again without errors, and are ready to commit your change. Write a useful commit message like `Fix logic error in ListCollectionCreator` or something similar, making sure it is clear what you've done.
>    (See [Galaxy's contributor's guide](https://github.com/galaxyproject/galaxy/blob/dev/CONTRIBUTING.md#how-to-contribute) for details on how to write a useful and properly formatted commit message.)
>
>    Selenium tests are in the bag!
{: .hands_on}

# Runtime error

Our last error happens at runtime, which means we don't have a failing test; instead we have a bug report describing the error. Our goal is to fix the error. Our steps are as follows:

1. Reproduce the error
2. Locate the error
3. Identify the problem
4. Write a test exposing the bug. The test must fail.
5. Fix the error.
6. Ensure the error no longer occurs.

> <hands-on-title>Fixing a runtime error</hands-on-title>
>
> 1. **Reproduce the error**
>
>    Here's the bug report:
>
>    > In the User menu, clicking the Datasets option causes an error message to be displayed on the page: "Uncaught exception in exposed API method".
>    {: .quote}
>
>    Make sure you are in `GALAXY_ROOT`. Then start your local Galaxy using the `uvicorn` command*:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > GALAXY_CONFIG_FILE=config/galaxy.yml uvicorn --app-dir lib --factory galaxy.webapps.galaxy.fast_factory:factory
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > The first startup may take a few minutes. Eventually, you'll see the following output (the PID will be different):
>    > >
>    > > ```
>    > > Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
>    *NOTE: We are not using `run.sh` because that would prevent us from using the Python interactive debugger, which we will be utilizing in this section.
>
>    Now you can access your Galaxy instance from the browser at [`http://127.0.0.1:8000`](http://127.0.0.1:8000).
>
>    To reproduce this error, we need to access the User menu. For this, we need to be logged in to Galaxy:
>
>    1. Click on "Login or Register:
>    2. At the bottom of the form, click "Register here"
>    3. Fill out the form. You may use any data, as long as the email you provide is a valid email format and the password is at least 6 characters long. For this exercise, it doesn't matter what email, name or password you use.
>    4. Click "Create"
>
>    Now the User menu is available to us. Click "User", then select "Datasets" from the dropdown menu. You should be able to see the error message displayed on the page: "Uncaught exception in exposed API method".
>
>    Being able to reproduce a bug is good: this means you're on your way to fixing it, and you know what to look for! In a real scenario, it is not uncommon that a bug is not easily reproduceable, which makes fixing it a much more complicated task.
>
> 2. **Locate the problem**
>
>    To figure out what's happening, our best bet is to look at the Galaxy log (look at the terminal window from which you launched Galaxy). Can you see the error message?
>
>    > <solution-title></solution-title>
>    >
>    > You should see something like this:
>    >
>    > ```
>    > galaxy.web.framework.decorators ERROR 2021-06-08 17:52:08,350 [pN:main.web.1,p:1583948,w:1,m:0,tN:uWSGIWorker1Core1] Uncaught exception in exposed API method:
>    > Traceback (most recent call last):
>    >   File "lib/galaxy/web/framework/decorators.py", line 312, in decorator
>    >     rval = func(self, trans, *args, **kwargs)
>    >   File "lib/galaxy/webapps/galaxy/api/datasets.py", line 100, in index
>    >     raise Exception('This should not happen!')
>    > Exception: This should not happen!
>    > ```
>    >
>    > From this error log, we can easily tell that the error happens in the `lib/galaxy/webapps/galaxy/api/datasets.py` file on line 100. The specific error message doesn't tell us much - which is often what happens in the wild.
>    {: .solution }
>
>    Let's head over to the `datasets'py` file, line 100:
>
>    ```python
>     97         if str_as_bool(trans.app.config.get("show_datasets", "True")):
>     98             return self.service.index(trans, history_id, serialization_params, filter_query_params)
>     99         else:
>    100             raise Exception("This should not happen!")
>    ```
>
>    The good news is that the error happens inside an `if/else` block, which narrows down our search to line 97: that line evaluates to `False`, causing the `else` clause to execute. The bad news is that we have no idea what causes that line to evaluate to `False`.
>
> 3. **Identify the problem**
>
>    We will investigate using pdb - our trusted Python debugger!
>
>    pdb offers a wide range of functionality and is exceptionally useful for debugging at runtime. We encourage you to read its [documentation](https://docs.python.org/3/library/pdb.html); however for this exercise we will only use the built-in [breakpoint()](https://docs.python.org/3/library/functions.html#breakpoint) function, that drops us into pdb.
>
>    Add a `breakpoint()` statement to your code, right above the `if/else` block.
>
>    > <solution-title></solution-title>
>    >
>    > ```python
>    > 126         breakpoint()
>    >  97         if str_as_bool(trans.app.config.get("show_datasets", "True")):
>    >  98             return self.service.index(trans, history_id, serialization_params, filter_query_params)
>    >  99         else:
>    > 100             raise Exception("This should not happen!")
>    > ```
>    >
>    {: .solution }
>
>    Now we need to restart Galaxy. However, to use pdb, we need to enable Galaxy's debug configuration option. In general, it may be a good idea to enable this in your `galaxy.yml` configuration file (just copy or rename `config/galaxy.yml.sample` and uncomment the option you want to set). However, for the purposes of this exercise, it's enough to set the `GALAXY_CONFIG_DEBUG` environment variable when running Galaxy:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > GALAXY_CONFIG_DEBUG=1 GALAXY_CONFIG_FILE=config/galaxy.yml uvicorn --app-dir lib --factory galaxy.webapps.galaxy.fast_factory:factory
>    > ```
>    {: .code-in}
>
>    > <comment-title>Why we set the debug option</comment-title>
>    >
>    > The reason for setting the `debug` option for releases prior to 22.05 was quite esoteric: among other things, it prevents uWSGI (no longer used as of 22.05) from remapping `stdin` to `dev/null` which would prevent tools like pdb from running.
>    {: .comment}
>
>    Now your Galaxy is running in debug mode. Repeat your steps that reproduced the error. When you click "Datasets", head over to your terminal. At the bottom of the log you'll see something like this:
>
>    ```
>    > /home/sergey/2sandbox/galaxy/dev_training/lib/galaxy/webapps/galaxy/api/datasets.py(97)index()
>    -> if str_as_bool(trans.app.config.get('show_datasets', 'True')):
>    (Pdb)
>    ```
>
>    You're at the pdb prompt. From here you can execute Python code interactively. The logs printed to `stdout` can be distracting, so try to ignore them. Let's explore what's happening in the next line:
>
>    > <code-in-title>Pdb</code-in-title>
>    > ```
>    > (Pdb) trans
>    > <galaxy.webapps.base.webapp.GalaxyWebTransaction object at 0x7f3664eee2b0>
>    > (Pdb) trans.app
>    > <galaxy.app.UniverseApplication object at 0x7f36ccc18790>
>    > (Pdb) trans.app.config
>    > <galaxy.config.GalaxyAppConfiguration object at 0x7f369addcac0>
>    > (Pdb) trans.app.config.get
>    > <bound method CommonConfigurationMixin.get of <galaxy.config.GalaxyAppConfiguration object at 0x7f369addcac0>>
>    > (Pdb)
>    > ```
>    {: .code-in}
>
>    We could have just typed `trans.app.config.get`, of course; the steps above give us some additional context.
>
>    Now let's take a look at the definition of the `get` method. We need to find `CommonConfigurationMixin` which is used by `galaxy.config.GalaxyAppConfiguration`.
>
>    Head over to `lib/galaxy/config/__init__.py` and look for the class `CommonConfigurationMixin`. Once you find it, look for the definition of its `get` method. When you find it, look at the method signature. What is the meaning of the argument `'True'` that is passed to this method in `datasets.py` on line 97?
>
>    > <solution-title></solution-title>
>    >
>    > ```python
>    > 584     def get(self, key, default=None):
>    > 585         # Warning: the value of self.config_dict['foo'] may be different from self.foo
>    > 586         return self.config_dict.get(key, default)
>    > ```
>    >
>    > The argument `True` corresponds to the parameter `default`. In the method's body, we see that this value will be passed to `self.config_dict.get()`; `self.config_dict` is a Python dictionary, so the second argument to `get()` is the default value that will be retuned if `key` is not in `self.config_dict`.
>    {: .solution }
>
>    Back to `datasets.py`, line 97. Now we know that `True` is the default value that will be returned if the key `show_datasets` does not exist in the `config_dict` dictionary. Moving on:
>
>    > <code-in-title>Pdb</code-in-title>
>    > ```
>    > (Pdb) trans.app.config.get('show_datasets')
>    > (Pdb) trans.app.config.get('show_datasets', 'True')
>    > 'True'
>    > ```
>    {: .code-in}
>
>    So, we see that `show_datasets` does not exist and we get the default value, which is the string `'True'`.
>
>    Next step: `str_to_bool`. First, let's pass it the value `'True'`:
>
>    > <code-in-title>Pdb</code-in-title>
>    > ```
>    > (Pdb) str_as_bool('True')
>    > False
>    > ```
>    {: .code-in}
>
>    That doesn't look right! Exit the debugger (type in `q`, then `Enter`, then `CTRL-C ` to exit Galaxy.
>    Head over to the function's definition (you can tell from the import statement at the top of the file that it's in `lib/galaxy/util/__init__.py`) and try to figure out if there is an error.
>
>    > <solution-title></solution-title>
>    >
>    > ```python
>    > 117 def str_as_bool(string):
>    > 118     """This is for training only."""
>    > 119     if str(string) in ("true", "yes", "on", "1"):
>    > 120         return True
>    > 121     else:
>    > 122         return False
>    > ```
>    >
>    > At first glance, everything seems right. But think of what value we are passing: `'True'` - it is _not_ in the `('true', 'yes', 'on', '1')` tuple! Aparently, the developer did not account for upper vs. lower case!
>    >
>    > Do NOT fix the error just yet.
>    {: .solution }
>
> 4. **Write a test exposing the bug**
>
>    Before you fix code that was not covered by a test, write the test to expose the bug! In this case we have a simple, completely isolated function - a simple unit test should do. Your test should go into this module: `test/unit/util/test_utils.py`.
>
>    Start with a blank test function. Then call the funtion under test (`str_as_bool`) and assert that when you pass it the value `'True'`, it returns `True`. Can you think of more cases to test?
>
>    > <solution-title></solution-title>
>    >
>    > Your first step could be something like this:
>    >
>    > ```python
>    > def test_str_as_bool():
>    >     assert util.str_as_bool('True')
>    > ```
>    >
>    > However, keep in mind that there may be other cases of mixed or uppercase values. You should also test for lowercase values - to make sure that while fixing the bug you don't break the current functionality. And remember to test for the `False` cases as well.
>    >
>    > There are many ways to implement this, feel free to write your own. Here's one option:
>    >
>    > ```python
>    > def test_str_as_bool():
>    >     for value in ('true', 'yes', 'on', '1', 'True', 'Yes', 'On','TRUE', 'YES', 'ON'):
>    >         assert util.str_as_bool(value)
>    >     for value in ('false', '0', 'something else'):
>    >         assert not util.str_as_bool(value)
>    > ```
>    >
>    {: .solution }
>
>    Run your test. Does it fail? Good! Now it's time to fix the error.
>
> 5. **Fix the error**
>
>    Can you think of a simple way to fix the function so that our test passes?
>
>    > <solution-title></solution-title>
>    >
>    > One way to do it is to simply convert the input value to lowercase:
>    >
>    > ```python
>    > 117 def str_as_bool(string):
>    > 118     """This is for training only."""
>    > 119     if str(string).lower() in ("true", "yes", "on", "1"):
>    > 120         return True
>    > 121     else:
>    > 122         return False
>    > ```
>    >
>    > In fact, that's exactly what the real function does! Check line 1017 - you'll see almost the exact same function, named `string_as_bool`: we didn't modify it because that might have broken other parts of Galaxy which would have been a distraction from the core aspects of the exercise.
>    {: .solution }
>
>    Run the test - now it should pass!
>
> 6. **Ensure the error no longer occurs**
>
>    Now that we have the `str_as_bool` function covered by test, as a bonus, we can safely do some minor refactoring: the test will prevent us from breaking things. Can you simplify the function's code?
>
>    > <solution-title></solution-title>
>    >
>    > ```python
>    > 117 def str_as_bool(string):
>    > 118     """ This is for training only."""
>    > 119     return str(string).lower() in ('true', 'yes', 'on', '1')
>    > ```
>    >
>    > Much better!
>    {: .solution }
>
>    Finally, head back to your local Galaxy and verify that the runtime error no longer occurs:
>
>    1. Remove the `breakpoint()` statement you added to `lib/galaxy/webapps/galaxy/api/datasets.py`.
>    2. Start Galaxy and repeat the steps from the bug report (select "User" > "Datasets"). You should see the Datasets page now - no more error message!
>
>    Congratulations - you've completed the last and, possibly, most challenging section of this tutorial!
{: .hands_on}

# Conclusion

Finally, we're back in the Green - all tests pass!

First of all, thank you for completing this tutorial. We hope you feel more confident debugging Galaxy code and understanding how to navigate the testing frameworks we utilize. Of course, we are always available to help answer questions and support you!


topic: dev
tutorial: visualization-generic
title: Generic plugins
difficulty: intermediate
time_estimation: 90m
tools_json: []
requirements_json: []
tags_json: []
content_hash: ba65e2fc6300f0e6e10cd7551d2c5180
last_modified: 2026-05-15T22:28:11.289292
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/visualization-generic/tutorial.html

content:
Visualizations may be very helpful in understanding data better. There is a whole
range of visualizations, from rather simple scatter and barplots up to projections
of high dimensional data or even entire genomes. Many of these visualizations often
require a lot of tweaking and changes in settings like zooming in and assigning colors, etc.
Therefore, visualizations are ideally interactive, and changing settings is often
an initial step in exploring data. For this reason it may be inconvenient to make use
of static galaxy tools because it lacks these interactive features. For these situations Galaxy
offers the option to create *visualizations plugins*, file format specific javascripts
that integrate with the history menu, without making redundant copies of data.

In this tutorial we shall go through how this system works and create a simple visualization
plugin. The tool will create a visualization of the number of aligned reads per
chromosome of a BAM file, and we will discuss possible optimizations and advantages
and disadvantages of the proposed implementation.

If you want to make visualizations ready for production, it is essential to have a good
understanding of HTML5 and JavaScript as these are the basic languages in which they are written.
However, for this tutorial we will keep it basic.

Additional documentation about Galaxy visualizations can be found here:

- [DataProviders](https://galaxyproject.org/data-providers)
- [DataProviders/Cookbook](https://galaxyproject.org/data-providers/cookbook)

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Part 1

The visualization we are going to create in this tutorial, is a tool that shows the number of aligned
reads per chromosome of a BAM file. The first thing we need to do is to come up with a name.
Let's call it *alignment_rname_boxplot*. Note that the reference sequences (usually chromosomes)
to which we align are named `RNAME` in the BAM/SAM specification.

The development of a Galaxy visualization takes place within the Galaxy codebase.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Clone an instance of Galaxy in a path, further referred to as `$GALAXY_ROOT`
> 2. Explore the plugin directory as follows:
>
>    ```bash
>    $ cd $GALAXY_ROOT/config/plugins/visualizations
>    ```
>
> 3. Create a new directory for our new plugin project
>
>    ```bash
>    $ mkdir alignment_rname_boxplot
>    $ cd alignment_rname_boxplot
>    ```
>
> 4. Make three (sub-)directories to complete the structure of the project:
>
>    ```bash
>    $ mkdir config
>    $ mkdir static
>    $ mkdir templates
>    ```
{: .hands_on}

## Linking the plugin with Galaxy

To create a bridge between our not-yet-written plugin and Galaxy, we need to write a
configuration in XML format.

> <hands-on-title>Data upload</hands-on-title>
>
> Create the file  `config/alignment_rname_boxplot.xml` with the following contents:
>
> ```xml
> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE visualization SYSTEM "../../visualization.dtd">
> <visualization name="alignment_rname_boxplot">
>     <data_sources>
>         <data_source>
>             <model_class>HistoryDatasetAssociation</model_class>
>
>             <test type="isinstance" test_attr="datatype" result_type="datatype">binary.Bam</test>
>             <test type="isinstance" test_attr="datatype" result_type="datatype">tabular.Sam</test>
>
>             <to_param param_attr="id">dataset_id</to_param>
>         </data_source>
>     </data_sources>
>     <params>
>         <param type="dataset" var_name_in_template="hda" required="true">dataset_id</param>
>     </params>
>     <template>alignment_rname_boxplot.mako</template>
> </visualization>
> ```
>
{: .hands_on}

This configures the plugin's name, which shall appear on pressing the visualization button in
the history menu. It also links the plugin to two file formats: BAM and SAM, which means that
for any history item of these file formats the plugin will automatically become available.

It also includes a reference to a mako template file (HTML + Python syntax), to be found in the
`templates` directory (we will create this file in the next section). The `var_name_in_template`
parameter is set to the value `hda`, which will be the name of the variable in the mako template
corresponding to the dataset to be visualized.

## Creating the visualization

We have linked our visualization to a mako file (which we have not yet created). This file is a
blueprint for the visualization. This means that for every invocation of the visualization, the
mako file will be compiled to render an HTML file.

Beause we would like the visualization to load quickly, computationally intensive tasks should
not be done prior to loading. A bit of server-side rendering in itself is not a problem, but the
visualizations (written in HTML and/or JS) should do most of the actual calculations and
conversions on the client side (in the browser). Therefore, unlike regular Galaxy tools, parsing
files does not take place on the server, but instead data will be downloaded by the client via an
exposed Galaxy URL prior to client-side rendering.

The most basic part of the mako file are the variables used for further web development, given
below.

```html
<!DOCTYPE HTML>
<%
    import os

    ## Generates hash (hdadict['id']) of history item
    hdadict = trans.security.encode_dict_ids( hda.to_dict() )

    ## Finds the parent directory of galaxy (/, /galaxy, etc.)
    root     = h.url_for( '/' )

    ## Determines the exposed URL of the ./static directory
    app_root = root + 'plugins/visualizations/'+visualization_name+'/static/'

    ## Actual file URL:
    file_url = os.path.join(root, 'datasets', hdadict['id'], "display?to_ext="+hda.ext)
%>
```

- `hdadict` is a variable that contains a file identifier that has been encoded to it's
   exposed uid.
- `root` indicates location of Galaxy on the webserver (e.g. `/`, `/galaxy/`, `/galaxy-pub/`, etc).
- `app_root` contains the exposed url of the static files for this visualization.
- `file_url` contains the exposed url of the dataset selected (by user) for visualization.

We could obtain the BAM file client-side by downloading the BAM/SAM file in its entirety via
*file_url*. However, BAM files can become quite large and it is usually not desired to transfer
such datasets over the network. In our case it is also rather inconvenient to parse the BAM file
with Javascript, just to count the number of reads.

Fortunately, BAM files have indices. These indices are brief summaries describing the
number of entries per chromosome, in order to be able to access the data contained in them more
quickly. In the mako template we can access a BAM index as *hda.metadata.bam_index*.
(Note that this is a file path on the server, not an exposed URL).

Samtools has a command named `idxstats` which is able to leverage this BAM index you. However,
since visualizations do not have dependency management, it is very tricky to let the mako template
do a system call to samtools. Fortunately, the Galaxy ecosystem ships with a built-in `pysam`
dependency, a library that can do any native samtools command within python.

The `*.metadata.bam_index` is a special kind of file in the Galaxy ecosystem. It is actually an
invisible file in Galaxy, linked to another history item, but does have a unique filename.
So, for the BAM file `./database/files/000/dataset_001.dat`, our BAI file (index) is **not**
`./database/files/000/dataset_001.dat.bai` but could be `./database/files/000/dataset_002.dat` or
`./database/files/000/dataset_003.dat`. We can create a symlink to this index file to ensure the
bam file and its index share the same prefix, as expected by samtools and pysam.

We can create this symlink as follows in our mako template:

```python
## Ensure BAI index is symlinked
bai_target = hda.file_name+'.bai'
import os

if not os.path.isfile(bai_target):
   os.symlink(hda.metadata.bam_index.file_name, bai_target)
```

Now the BAM file has a `.bai` file with the same prefix, and we can run the `idxstats`
as follows:

```python
## Extract idxstats
import pysam
bam_idxstats_data = pysam.idxstats(hda.file_name)
```

With the lines of python code above, the idxstats data is parsed into the RAM of python
during compilation on the server, but is not yet exported into the HTML page nor parsed by JS.
To do that, we add an HTML section to the end of the mako file.

```html
<html>
    <head>
        <title>${hda.name | h} | ${visualization_name | h}</title>
    </head>
    <body>
        ${bam_idxstats_data | h}
    </body>
</html>
```

Here you see `${bam_idxstats_data | h}`, which prints the python variable into
the HTML page and also does HTML escaping by providing the ` | h`-flag (for security reasons).

Let's put this all together.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create the mako file `templates/alignment_rname_boxplot.mako`
> 2. Fill it with the following code:
>
>    ```html
>    <!DOCTYPE HTML>
>    <%
>        import os
>
>        ## Generates hash (hdadict['id']) of history item
>        hdadict = trans.security.encode_dict_ids( hda.to_dict() )
>
>        ## Finds the parent directory of galaxy (/, /galaxy, etc.)
>        root     = h.url_for( '/' )
>
>        ## Determines the exposed URL of the ./static directory
>        app_root = root + 'plugins/visualizations/'+visualization_name+'/static/'
>
>        ## Actual file URL:
>        file_url = os.path.join(root, 'datasets', hdadict['id'], "display?to_ext="+hda.ext)
>
>        ## Ensure BAI index is symlinked
>        bai_target = hda.file_name+'.bai'
>
>        if not os.path.isfile(bai_target):
>            os.symlink(hda.metadata.bam_index.file_name, bai_target)
>
>        ## Extract idxstats
>        import pysam
>        bam_idxstats_data = pysam.idxstats(hda.file_name)
>    %>
>    <html>
>        <head>
>            <title>${hda.name | h} | ${visualization_name | h}</title>
>        </head>
>        <body>
>            ${bam_idxstats_data | h}
>        </body>
>    </html>
>    ```
>
>    We are now ready to test this very basic visualization, we just need a (small) BAM file for it.
>
> 3. Download [the example BAM file](https://zenodo.org/record/248730/files/tutorial.bam)
> 4. Go the galaxy root directory and start Galaxy:
>
>    ```bash
>    $ cd $GALAXY_ROOT
>    $ ./run.sh
>    ```
>
> 5. Upload the example BAM file to your history
>
>    If everything went well, our plugin has appeared as a visualization option for the dataset
>
>    > <comment-title></comment-title>
>    > You must be logged in to be able to use visualizations
>    {: .comment}
>
{: .hands_on}

All the visualization does at the moment, is show the contents of idxstats, compiled to HTML:

```
['chrA\t5386\t2\t0\n', 'chrB\t5386\t4\t0\n', 'chrC\t5386\t1\t0\n', 'chrD\t5386\t6\t1\n',
'chrE\t5386\t3\t0\n', 'chrF\t5386\t2\t0\n', 'chrG\t5386\t1\t0\n', '*\t0\t0\t0\n']
```

It contains eight entries, one for each of our (made-up) chromosomes and one to `*`, which represents
the unmapped reads. Entries are tab delimited (`\t`) and for the `chrA` entry it indicates that the
length of the RNAME (chromosome) is 5386 bases and 2 reads are aligned to it.

To make the data a bit more usable for Javascript , we convert it into a simple dictionary
of the following syntax:

```python
{'chrA':2, 'chrB':4, 'chrC':1, 'chrD':6, 'chrE':3, 'chrF':2, 'chrG':1, '*': 0}
```

Although it is possible to do this in python we recommend doing this in JS.
The var dump provided by python/pysam is actually a valid syntax for Javascript too,
so getting the raw data into Javascript is rather easy:

```html
<script>
    bam_idxstats_data = ${bam_idxstats_data};
</script>
```

Converting the data is not the scope of the tutorial, so here we provide such a function:

```html
<script>
  bam_idxstats_data = ${bam_idxstats_data};
  function parse_data(bam_idxstats_data) {
    var output = {};

    for(var i = 0; i < data.length ; i++) {
      var line = data[i];
      var chunks = line.split("\t");

      if(chunks[0].split("_").length == 1) { // only if it does not contain underscore
        output[chunks[0]] = parseInt(chunks[2]);
      }
    }

   return output;
  }
  </script>
```

The great thing about the mako system is that it does not require to restart galaxy in order to make
functional changes to the mako files.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Change the mako file to the following:
>
>    ```html
>    <!DOCTYPE HTML>
>    <%
>        import os
>
>        ## Generates hash (hdadict['id']) of history item
>        hdadict = trans.security.encode_dict_ids( hda.to_dict() )
>
>        ## Finds the parent directory of galaxy (/, /galaxy, etc.)
>        root     = h.url_for( '/' )
>
>        ## Determines the exposed URL of the ./static directory
>        app_root = root + 'plugins/visualizations/'+visualization_name+'/static/'
>
>        ## Actual file URL:
>        file_url = os.path.join(root, 'datasets', hdadict['id'], "display?to_ext="+hda.ext)
>
>        ## Ensure BAI index is symlinked
>        bai_target = hda.file_name+'.bai'
>
>        if not os.path.isfile(bai_target):
>            os.symlink(hda.metadata.bam_index.file_name, bai_target)
>
>        ## Extract idxstats
>        import pysam
>        bam_idxstats_data = pysam.idxstats(hda.file_name)
>
>    %>
>    <html>
>        <head>
>           <title>${hda.name | h} | ${visualization_name | h}</title>
>            <script>
>                bam_idxstats_data = ${bam_idxstats_data};
>                function parse_data(data) {
>                    var output = {};
>                    for(var i = 0; i < data.length ; i++) {
>                        var line = data[i];
>                        var chunks = line.split("\t");
>
>                        if(chunks[0].split("_").length == 1) { // only if it does not contain underscore
>                            output[chunks[0]] = parseInt(chunks[2]);
>                        }
>                    }
>                    return output;
>                }
>            </script>
>        </head>
>        <body onload="bam_idxstats = parse_data(bam_idxstats_data);">
>            ${bam_idxstats_data | h}
>        </body>
>    </html>
>    ```
>
> 2. Retrigger the visualization and open the developers console of your browser: In the console, type: `bam_idxstats_data` and press <kbd>Enter</kbd>
>   This should give the parsed contents as a dictionary, which can directly be used in Javascript.
>
{: .hands_on}

From this point forward you are encouraged to continue on your own to see if you are able to create
a simple visualization from this dictionary. Think of tables, DIVs or even more complicated
solutions :).

Below is an example visualization, which creates a bar plot showing the number of reads per
chromosome.

![Example visualization](../../images/vis_plugins_example.png)

The full contents of this plugin are provided in the [GitHub repository related to this material in `tree/master/topics/dev/files/hands_on-visualizations/alignment_rname_boxplot`]({{ site.github_repository }}/tree/{{ site.github_repository_branch }}/topics/dev/files/hands_on-visualizations/alignment_rname_boxplot).
To try out this example, simply copy this folder to the `$GALAXY_ROOT/config/plugins/visualizations/` folder
on your (local) Galaxy and restart Galaxy.

The contents of the mako file for this example are given below.

```html
<!DOCTYPE HTML>
<%
    import os

    ## Generates hash (hdadict['id']) of history item
    hdadict = trans.security.encode_dict_ids( hda.to_dict() )

    ## Finds the parent directory of galaxy (/, /galaxy, etc.)
    root     = h.url_for( '/' )

    ## Determines the exposed URL of the ./static directory
    app_root = root + 'plugins/visualizations/'+visualization_name+'/static/'

    ## Actual file URL:
    file_url = os.path.join(root, 'datasets', hdadict['id'], "display?to_ext="+hda.ext)

    ## Ensure BAI index is symlinked
    bai_target = hda.file_name+'.bai'

    if not os.path.isfile(bai_target):
        os.symlink(hda.metadata.bam_index.file_name, bai_target)

    ## Extract idxstats
    import pysam
    bam_idxstats_data = pysam.idxstats(hda.file_name)
%>
<html>
     <head>
        <title>${hda.name | h} | ${visualization_name | h}</title>

        <style>
             .chart div {
               font: 10px sans-serif;
               background-color: steelblue;
               text-align: right;
               padding: 3px;
               margin: 1px;
               color: white;
             }
        </style>
        <script>
            bam_idxstats_data = ${bam_idxstats_data};
            function parse_data(data) {
                 /*
                  Data comes in as tuple of unsplit lines:
                  ["chr1\t1000\t0\t0", "chr2\t2500\t0\t0"]

                  We need to split it up, and ideally only keep reference names without an underscore
                 */
                 var output = {};

                 for(var i = 0; i < data.length ; i++) {
                     var line = data[i];
                     var chunks = line.split("\t");

                     if(chunks[0].split("_").length == 1) { // only if it does not contain underscore
                         output[chunks[0]] = parseInt(chunks[2]);
                     }
                 }

                 return output;
            }

            function calc_stats(parsed) {
                max = 0;
                sum = 0;
                for (var key in parsed) {
                    if (parsed[key] > max){
                        max = parsed[key];
                    }
                    sum += parsed[key]
                }
                return [max, sum];
            }

            function plot_data(parsed) {
                var max = calc_stats(parsed)[0];
                var sum = calc_stats(parsed)[1];

                for (var key in parsed) {
                     var value = parsed[key];
                     var ratio = 100.0 * value / sum;
                     var ratio2 = 100.0 * value / max;

                     var div = document.createElement("div");
                     div.innerHTML = '<nobr>'+key+'</nobr>';
                     document.getElementById("chart_names").appendChild(div);

                     var div = document.createElement("div");
                     div.innerHTML = '<nobr>'+value+" ("+Math.round(ratio*100)/100+"%)</nobr>";
                     div.title = key+': '+value+" ("+Math.round(ratio*100)/100+"%)";
                     div.style.width =  ratio2+'%';
                     document.getElementById("chart").appendChild(div);
                }
            }
        </script>
     </head>
     <body onload="plot_data(parse_data(bam_idxstats_data));">
         <center>
             <h1>Bam contents of ${hda.name | h}</h1>

             <table border="0" borderpadding="0" borderpanning=")" style="width: 500px;">
                 <tr>
                     <td style="width:50px;">
                         <div id="chart_names" class="chart" style="width: 100%; border: 1px dashed gray;text-align: left;" />
                     </td>
                     <td style="width:450px;">
                         <div id="chart" class="chart" style="width:100%; border: 1px dashed gray;text-align: left;" />
                     </td>
                 </tr>
             </table>
         </center>
     </body>
</html>

```

In the given example, the `RNAME` queries containing an underscore were removed.
This is because there are many alternative chromosomes, making the list very large
for certain reference genomes. However, for certain studies it might be desired to
just look at those.

The current plot is a box plot, but one can imagine a pie-chart may be convenient too.
All of those additional settings can be implemented for interactive behaviour,
contributing to quicker understanding of the data which is generally not so convenient
using static Galaxy tools.

> <comment-title>Static files</comment-title>
>
> In the example we included Javascript and CSS into the HTML website.
> Remember that for every new invocation of the visualization the entire CSS en JS are copied
> and transferred as well. This is a waste of (redundant) bandwidth as we could save the
> files in the static directory and refer to them within the HTML. The browser shall check
> it's cache for the presence of libs and style sheets and only update them if they have changed.
{: .comment}

### Improvements

Another thing you may realize is that we still do the calculation (pysam.idxstats) server side.
Although this is a marginal calculation, it is causing delay and the bigger the files, the worse
this delay becomes. The underlying problem is that we did not obtain and parse the BAI file via
Javascript. This would be a more elaborate solution, but requires more development time as we
need to develop a function able to parse the binary file.

Another thing we could do is create a static `samtools idxstats` tool, that creates a file of
datatype `tabular.Idxstats` and include that datatype into Galaxy. We then make the visualization
specific for that datatype, just plotting the results of the `idxstats`.

A fundamental and more complicated problem is that BAM files are simply too big to transfer for
these kind of applications. It would be ideal to have web server integration that allows querying
of specific locations or metadata within or from a BAM file where indexing operations are taken
care of at the server side. This is what has been done in Trackster.


### More examples

For more examples of visualization plugins, you can browse this
[GitHub repo](https://github.com/bgruening/galaxytools/tree/master/visualisations)

# Conclusion


We have just created a visualization plugin in Galaxy to visualize the number of alignments
per `RNAME` (chromosome) in a BAM file.


topic: dev
tutorial: bioblend-dev
title: Contributing to BioBlend as a developer
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: []
content_hash: f33cbf9c03c9f1d0b45023dc8f8cf67f
last_modified: 2026-05-15T22:28:11.245488
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/bioblend-dev/tutorial.html

content:
BioBlend ({% cite Sloggett2013 %}) is a Python library to enable simple interaction with Galaxy ({% cite Afgan2018 %}) via the command line or scripts.

Galaxy is a data analysis platform for accessible, reproducible and transparent computational research. It includes a web interface through which users can design and perform tasks in a visual and interactive manner. The Galaxy server also exposes this functionality through its REST-based {API}.

Computer programs can communicate with the Galaxy server through this {API} and perform similar tasks as can be achieved manually via the web interface. The program sends a network request to a URL of an {API} endpoint. The server then computes the result for this request and sends back a response to the program.

BioBlend provides classes and methods that handle the specific details of this communication for Python programs. Similar libraries also exist for interacting with Galaxy via other programming languages:

* blend4j (Java): [https://github.com/galaxyproject/blend4j](https://github.com/galaxyproject/blend4j)
* blend4php (PHP): [https://github.com/galaxyproject/blend4php](https://github.com/galaxyproject/blend4php)
* clj-blend (Clojure): [https://github.com/chapmanb/clj-blend](https://github.com/chapmanb/clj-blend)

> <agenda-title></agenda-title>
>
> In this tutorial, we will learn the basics of BioBlend development:
>
> 1. TOC
> {:toc}
>
{:.agenda}

# Development on GitHub

Galaxy and BioBlend are developed as open source projects on GitHub and contributions are welcome!

The GitHub repositories can be found at:
* Galaxy: [https://github.com/galaxyproject/galaxy](https://github.com/galaxyproject/galaxy)
* BioBlend: [https://github.com/galaxyproject/bioblend](https://github.com/galaxyproject/bioblend)

##  Contributing on GitHub

To contribute to BioBlend, a GitHub account is required.

Changes are proposed via a {PR}. This allows the project maintainers to review the changes and suggest improvements.

The general steps are as follows:

1. Fork the BioBlend repository
2. Make changes in a new branch
3. Open a pull request for this branch in the upstream BioBlend repository

It is generally a good idea to enable the “Allow edits and access to secrets by maintainers” option for the {PR}. Enabling this option gives maintainers more freedom to help out.

# Downloading Galaxy and BioBlend

Now we are ready to set up our development environment! Since BioBlend communicates with the Galaxy {API}, we must also set up a local Galaxy server in order to test BioBlend functionality.
We use the [git](https://git-scm.com) versioning tool to download the repositories.

For this we require a public SSH key associated with our GitHub account. It makes sense to set this up now, since pushing changes to GitHub without a public key will prompt for credentials every time. See the [GitHub Docs](https://docs.github.com/en/github-ae@latest/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) for information on setting up an SSH key.

> <hands-on-title>Downloading repositories</hands-on-title>
> Download Galaxy and set it up as a local repository:
>
> ```shell
> git clone git@github.com:galaxyproject/galaxy.git
> ```
>
> Likewise, download BioBlend:
>
> ```shell
> git clone git@github.com:galaxyproject/bioblend.git
> ```
>
> If you have already forked these repositories, you probably want to replace the links in the commands above to point to the forks.
{:.hands_on}

# Interacting with the API

## Structure of the Galaxy API

The source code for the {API} endpoints is contained in various files under [lib/galaxy/webapps/galaxy/api/](https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/webapps/galaxy/api) in the Galaxy source code. Each of these files contains a controller which exposes functionality for a specific entity. For example, the dataset-related functionality is contained in the `DatasetsController` class in the [datasets.py](https://github.com/galaxyproject/galaxy/blob/v21.01/lib/galaxy/webapps/galaxy/api/datasets.py#L33) file.

Additionally, the [buildapp.py](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/buildapp.py) file contains a complete listing of all the endpoints.

> <comment-title>Note</comment-title>
> Different versions of Galaxy differ in the functionality that is implemented. The development version is named `dev` and includes the most recent changes. Release versions are named according to their release date. These Galaxy releases can be accessed via git branches, e.g. `release_20.01`. At the time of writing BioBlend supports Galaxy releases 17.09 and later, although some functionality is not supported by all of these versions.
{:.comment}


> <tip-title>Going deeper into the Galaxy back-end code</tip-title>
> The various Galaxy {API} methods that are exposed to the outside are contained in controller classes. These contain the endpoint methods corresponding to the URLs of the Galaxy {API}. These endpoint methods handle the incoming requests from BioBlend.
>
> A focus for development of the Galaxy {API} is to make these controllers as “terse” as possible. This basically means that any logic not strictly required by the {API} endpoint method is moved to a corresponding manager class. This approach separates the {API} more cleanly from internal functionality. The manager classes should contain as much of the functionality as possible. An example is the [`DatasetManager`](https://github.com/galaxyproject/galaxy/blob/v21.01/lib/galaxy/managers/datasets.py#L26), referenced to regularly by the `DatasetsController` which is exposed directly to the {API}.
>
> At the time of writing this transition is ongoing. Therefore, source code of methods for one controller might look and function differently than those of another controller. This might be encountered when debugging a failing request in BioBlend.
>
{: .tip}



## Core concepts of the Galaxy interface

This section provides brief descriptions of the core concepts of the Galaxy interface, from the perspective of a BioBlend/Galaxy developer, rather than the typical Galaxy user. This list is by no means complete, but it should help those not already familiar with Galaxy.

BioBlend provides separate clients for interacting with each of these entities. For example, the BioBlend client for jobs is the `JobsClient` class. The source code for each `Client` can be found under [bioblend/galaxy/](https://github.com/galaxyproject/bioblend/tree/main/bioblend/galaxy).

<!-- This is the syntax for a definition list in markdown, it's semantically the most correct option here -->

Tool
: A Galaxy tool packages a particular piece of scientific software, keeping track of its required dependencies. It also contains additional metadata such as the BibTex references to the associated research paper(s). Tools generally take datasets and/or dataset collections as inputs and produce new datasets and/or collections as outputs.

Dataset
: A dataset represents digital data that can serve as input for a job. It can be associated with (contained in) a history or a library. A dataset can be shared between users. Each time a dataset is copied between histories or libraries, a new {HDA} or {LDA} is created. This naming scheme is used internally by the Galaxy relational database, but the terms might be encountered during BioBlend development as well.

Dataset Collection
: A dataset collection represents a group of related datasets. Like a dataset, it can be associated with a history or a library. Each time it is copied or shared, a new {HDCA} or a {LDCA} is created.<br><br>There are two kinds of dataset collections, `list` and `paired`. The former is a simple list of elements. The latter has a particular application in bioinformatics; it contains only two elements, corresponding to the `forward` and `reverse` strands of a piece of DNA.<br><br> Collection elements can be other collections; in other words, collections can be nested. These are designated (for example) as `list:list` or `list:paired`.

Job
: Jobs are associated with the execution of a tool on the Galaxy server. Each dataset is created by running a tool and therefore has an associated job. The computation of each step in a workflow has an associated job as well.<br><br>Jobs run as background processes on the Galaxy server. This is relevant because reading results before the appropriate jobs have finished can lead to missing data in the output. BioBlend methods must handle this by waiting for relevant jobs to finish before returning a response to the user.<br><br>Job creation can be different when dealing with dataset collections. If a tool input receives a collection instead of a dataset, it will create a separate job for each element, resulting in an output collection with the same structure as the input. This is referred to as 'mapping over' a collection.

History
: A history represents a container that keeps track of actions performed on its contents. Datasets are associated with a particular history; tools and workflows also need to be run in a particular history, which keeps track of the outputs produced by each of the resulting jobs or invocations. While the history is a very important concept to users of the graphical user interface, as it provides a constantly visible workspace to organize individual analyses or projects, it is slightly less relevant for BioBlend users.

Workflow
: A workflow represents a computation pipeline, made up of multiple tools chained together. It is created and contained in a history. A workflow is comprised of steps, which can be either inputs or tools. Its structure can be imagined as a directed graph of nodes that process inputs and pass on their outputs to the next nodes. The initial input to a workflow is commonly one or multiple datasets or dataset collections, as well as input parameters.

Invocation
: Just like a job represents a tool executed on certain inputs, an invocation represents the execution of a workflow on specified inputs and with certain specified parameters. Every invocation corresponds to only one workflow, and every step of the workflow has an associated invocation step. Each invocation step has jobs associated with it.

Library
: A library represents a container for datasets and dataset collections. Datasets that are contained in a library can be shared between multiple Galaxy users.

## Structure of the BioBlend library

BioBlend methods for the Galaxy API are associated with the various `Clients` already mentioned, stored under [bioblend/galaxy/](https://github.com/galaxyproject/bioblend/tree/main/bioblend/galaxy) in the BioBlend GitHub repository. The functionality for each Galaxy entity is in a separate folder. For example, the methods for interacting with workflows are stored under [bioblend/galaxy/workflows](https://github.com/galaxyproject/bioblend/tree/main/bioblend/galaxy/workflows) in the `WorkflowsClient` class.

The tests for these methods are stored under [tests/](https://github.com/galaxyproject/bioblend/tree/main/bioblend/_tests). For example, the tests related to workflows are in the file [tests/TestGalaxyWorkflows.py](https://github.com/galaxyproject/bioblend/blob/main/bioblend/_tests/TestGalaxyWorkflows.py).

The classic approach for accessing the Galaxy API is using the various clients and their methods. Each of these methods corresponds to one of the API endpoints. They send a request to the Galaxy server and return a Python object representing the parsed JSON response. Alternatively, there is also BioBlend.objects ({% cite Leo2014 %}), which provides an object-oriented API. Interaction with the API occurs via wrapper classes. These wrappers represent entity instances and provide methods to interact with and manipulate them. For example, a wrapper could represent a single workflow.


> <question-title>What is the difference between BioBlend and BioBlend.objects?</question-title>
>
> Look at the following code:
> ```python
> # classic BioBlend
> from bioblend.galaxy import GalaxyInstance
> gi = GalaxyInstance(url=galaxy_url, key=galaxy_key)
> workflows = gi.workflows.get_workflows()
> for wf_dict in workflows:
>     print(f"{wf_dict['name']} : {wf_dict['id']}"))
>
> # BioBlend.objects
> from bioblend.galaxy.objects import GalaxyInstance
> gi = GalaxyInstance(galaxy_url, galaxy_key)
> workflows = gi.workflows.get_previews()
> for wf in workflows:
>     print(f"{wf.name} : {wf.id}")
> ```
> It does the same thing (prints all available workflows) using both BioBlend and BioBlend.objects. What differences can you see?
> > <solution-title></solution-title>
> >
> > The output of the classic BioBlend code is provided as JSON, which the user can continue to manipulate themselves. The BioBlend.objects code provides the same information, but in the format of a Python object. This has various properties which the user can access, using for example `name` or `id`.
> >
> > You can view [numerous examples of both here](https://github.com/galaxyproject/bioblend/tree/main/docs/examples).
> {: .solution }
{: .question}


> <comment-title>Note</comment-title>
> BioBlend also provides methods for interacting with the Galaxy [ToolShed](https://galaxyproject.org/toolshed/) and [Cloudman](https://galaxyproject.org/cloudman/). However, this tutorial focuses on the part of BioBlend which provides access to the Galaxy API.
{:.comment}

# Communicating with the Galaxy API

We will make some manual requests to get a better feel for the Galaxy API. But first we need a valid API key in order to communicate with the Galaxy server through the API.

{% snippet faqs/galaxy/preferences_admin_api_key.md dev=true %}

In the following examples we make use of a simple networking tool named [curl](https://curl.se). Each command is listed together with the corresponding BioBlend and BioBlend.objects code.

We assume basic familiarity with HTTP request methods. For an overview, see the Mozilla Developer Network [documentation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods).

Now, open a terminal window and let's make a few manual requests! In each case, the corresponding BioBlend and BioBlend.objects code is also provided for comparison.

> <hands-on-title>Making requests to the API</hands-on-title>
> 1. __GET request__
>
>    This is a simplest type of request. We request data from a specific URL, but we do not provide any further information apart from our API key. In this example we request a listing of all our invocations.
>
>    > <code-in-title>curl</code-in-title>
>    > ```shell
>    > curl -X GET -H 'x-api-key: <API_KEY>' 'localhost:8080/api/invocations'
>    > ```
>    >
>    > The `-X` flag specifies the HTTP method of our request. In this case we are making a GET request. GET is the default method, so we could also simply omit it here.
>    >
>    > The `-H` flag is used to specify HTTP headers. We authenticate our requests by specifying our API key in the `x-api-key` header.
>    {:.code-in}
>
>    > <code-out-title>Example JSON response</code-out-title>
>    > ```javascript
>    > [{
>    >     'history_id': '2f94e8ae9edff68a',
>    >     'id': 'df7a1f0c02a5b08e',
>    >     'model_class': 'WorkflowInvocation',
>    >     'state': 'new',
>    >     'update_time': '2015-10-31T22:00:22',
>    >     'uuid': 'c8aa2b1c-801a-11e5-a9e5-8ca98228593c',
>    >     'workflow_id': '03501d7626bd192f',
>    > }]
>    > ```
>    >
>    > The above response would mean that our user has only invoked a workflow once. The workflow's encoded ID is listed (`workflow_id`), as well as the encoded ID of the history containing this workflow (`history_id`), and the ID of the invocation itself (`id`). The state `new` indicates that the request to invoke this workflow is still pending and that the invocation has not been scheduled yet.
>    {:.code-out}
>
>    BioBlend code:
>
>    ```python
>    from bioblend.galaxy import GalaxyInstance
>    gi = GalaxyInstance('http://localhost:8080', key=<API_KEY>)
>    invs = gi.invocations.get_invocations()
>    print(invs)
>    ```
>
>    BioBlend.objects code:
>
>    ```python
>    from bioblend.galaxy.objects import GalaxyInstance
>    obj_gi = GalaxyInstance('http://localhost:8080', key=<API_KEY>)
>    previews = obj_gi.invocations.get_previews()
>    for preview in previews:
>        print(preview.id)
>    ```
>
> 2. __GET request with query parameters__
>
>    > <code-in-title>curl</code-in-title>
>    > In this example we include two query parameters in our request, `limit` and `include_terminal`.
>    >
>    > ```shell
>    > curl -X GET -H 'x-api-key: <API_KEY>' \
>    >      'localhost:8080/api/invocations?limit=5&include_terminal=False'
>    > ```
>    >
>    > With `limit=5` we limit the result to at most 5 invocations. With `include_terminal=False` we indicate that invocations in a terminal state should be excluded from the results.
>    {:.code-in}
>
>    > <code-out-title>Example JSON response</code-out-title>
>    > ```javascript
>    > []
>    > ```
>    >
>    > The above response would mean that there are no workflows in a non-terminal state - any workflows which have been invoked have been successfully scheduled.
>    {:.code-out}
>
>    BioBlend code:
>    ```python
>    invs = gi.invocations.get_invocations(limit=5, include_terminal=False)
>    print(invs)
>    ```
>
>    BioBlend.objects code:
>    ```python
>    invs = obj_gi.invocations.list(limit=5, include_terminal=False)
>    for invocation in invs:
>        print(invocation.id)
>    ```
>
> 3. __POST request with a payload__
>
>    > <code-in-title>curl</code-in-title>
>    > ```shell
>    > curl -X POST \
>    >     -H 'x-api-key: <API_KEY>' \
>    >     -H 'Content-Type: application/json' \
>    >     -d '{"name":"MyNewHistory"}' \
>    >     'localhost:8080/api/histories'
>    > ```
>    >
>    > This creates a new history with the name ‘MyNewHistory’.
>    {:.code-in}
>
>    > <code-out-title>Example JSON response</code-out-title>
>    > ```javascript
>    > {'update_time': '2021-05-24T13:04:05.393590',
>    > 'user_id': 'e9376175b7f708de',
>    > 'name': 'MyNewHistory',
>    > 'genome_build': None,
>    > 'username_and_slug': None,
>    > 'deleted': False,
>    > 'published': False,
>    > 'slug': None,
>    > 'model_class': 'History',
>    > 'state': 'new',
>    > 'id': '464411b81613694d',
>    > 'empty': True,
>    > 'annotation': None,
>    > 'importable': False,
>    > 'state_ids': {'new': [],
>    >  'upload': [],
>    >  'queued': [],
>    >  'running': [],
>    >  'ok': [],
>    >  'empty': [],
>    >  'error': [],
>    >  'discarded': [],
>    >  'paused': [],
>    >  'setting_metadata': [],
>    >  'failed_metadata': []},
>    > 'create_time': '2021-05-24T13:04:05.393579',
>    > 'purged': False,
>    > 'tags': [],
>    > 'contents_url': '/api/histories/464411b81613694d/contents',
>    > 'size': 0,
>    > 'state_details': {'new': 0,
>    >  'upload': 0,
>    >  'queued': 0,
>    >  'running': 0,
>    >  'ok': 0,
>    >  'empty': 0,
>    >  'error': 0,
>    >  'discarded': 0,
>    >  'paused': 0,
>    >  'setting_metadata': 0,
>    >  'failed_metadata': 0},
>    > 'url': '/api/histories/464411b81613694d'}
>    > ```
>    >
>    > The response contains some basic information about the newly created history, including the `history_id`, which we would need for further API calls.
>    {:.code-out}
>
>    BioBlend code:
>    ```python
>    history = gi.histories.create_history('MyNewHistory')
>    print(history)
>    ```
>
>    BioBlend.objects code:
>
>    ```python
>    history = obj_gi.histories.create('MyNewHistory')
>    print(history.name)
>    ```
{:.hands_on}

> <tip-title>Encoded and non-encoded IDs in Galaxy</tip-title>
> While making the API calls now, we encountered various IDs - for example, we created a history with ID `464411b81613694d`. How do Galaxy IDs work?
>
> The Galaxy server assigns a numeric (integer) ID to every entity, which serves as its unique identifier. (The exception are tools, which have a predefined string ID specified in the [XML wrapper](https://docs.galaxyproject.org/en/latest/dev/schema.html#attributes) defining the tool.) For example, each workflow, invocation, history and dataset is assigned a unique ID. If you access the Galaxy database directly, using a tool such as [gxadmin](https://github.com/usegalaxy-eu/gxadmin), these integers are the IDs that you will see.
>
> However, these IDs are not (or should not be!) exposed through the API. The Galaxy server encodes the numeric IDs into hexadecimal strings before sending them to external API clients. These are referred to as encoded IDs. BioBlend only deals with these encoded IDs.
>
{: .tip}

> <tip-title>Params versus payload: what's the difference?</tip-title>
>
> In BioBlend, the `Client._get()` helper method is used for GET requests. It takes a `params` argument. The `Client._post()` and `Client._put()` helper methods make POST and PUT requests respectively. They expect a `payload` argument. In this section we briefly explain the difference.
>
> An HTTP request is comprised of (1) a request line, (2) headers with metadata, (3) an empty line and (4) an optional message body. The HTTP GET method uses query parameters to specify which resource should be retrieved from the server. These parameters are included in the request line as part of the URL. The HTTP POST and PUT methods create or update resources. This data is specified in the payload, which is included in the message body of the request.
>
> We can see the difference using [curl](https://curl.se) and [ncat](https://nmap.org/ncat/).
>
> Open a terminal and run the `ncat` command in listen mode on port 8080. The loop will print out any incoming requests; each request is separated by a line containing `---`.
>
> ```shell
> while ncat --listen 8080; do printf "\n---\n"; done
> ```
>
> > <comment-title>Note</comment-title>
> > The `ncat` command is a low-level networking tool that does not respond to any incoming connections. Since HTTP is done over [TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol), the `curl` command will hang expecting a response. Simply press <kbd>CTRL</kbd> + <kbd>C</kbd> to terminate the `curl` command.
> {:.comment}
>
> #### Parameters
>
> Open a second terminal and send a GET request with a query parameter:
>
> > <code-in-title>curl</code-in-title>
> > ```shell
> > curl -X GET localhost:8080/?message=Hello
> > ```
> {:.code-in}
>
> > <code-out-title>ncat</code-out-title>
> > ```
> > GET /?message=Hello HTTP/1.1
> > Host: localhost:8080
> > User-Agent: curl/7.76.1
> > Accept: */*
> >
> >
> > ```
> >
> > The two blank lines indicate that the payload is empty.
> {:.code-out}
>
> #### Payload
>
> Send a POST request with the payload ‘My name is curl’.
>
> > <code-in-title>curl</code-in-title>
> > ```shell
> > curl -X POST -d 'My name is curl' localhost:8080
> > ```
> {:.code-in}
>
> > <code-out-title>ncat</code-out-title>
> > ```
> > POST / HTTP/1.1
> > Host: localhost:8080
> > User-Agent: curl/7.76.1
> > Accept: */*
> > Content-Length: 15
> > Content-Type: application/x-www-form-urlencoded
> >
> > My name is curl
> > ```
> >
> > The message body now contains our payload.
> {:.code-out}
{: .tip}

# Creating a simple BioBlend method

As an introduction to BioBlend development, we will extend the BioBlend `ToolsClient` class by adding a new method named `uninstall_dependencies()`. This method will send a request to the Galaxy server to uninstall any dependencies for the specified tool.

> <comment-title>Note</comment-title>
> This example is based on the pull request “[Improving Tools API coverage](https://github.com/galaxyproject/bioblend/pull/390)” in the BioBlend repository.
{:.comment}

## Identifying the relevant Galaxy endpoint

We should check the Galaxy controller to verify whether this functionality is already implemented in the Galaxy API. Let's check the `dev` version of Galaxy. Our method interacts with the tools API, so let's check the [api/tools.py](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/tools.py) file.

As it turns out, a corresponding [`uninstall_dependencies()`](https://github.com/galaxyproject/galaxy/blob/v21.01/lib/galaxy/webapps/galaxy/api/tools.py#L293) endpoint method is already available in the Galaxy `dev` branch. Lucky us! If this was not the case, then we would have first had to implement the endpoint in Galaxy, before writing a BioBlend method to interact with it.

> <hands-on-title>Creating the method and a simple test</hands-on-title>
> 1. __Constructing the correct method signature__
>
>    Let's look at the signature of the endpoint method:
>
>    __File [api/tools.py](https://github.com/galaxyproject/galaxy/blob/v21.01/lib/galaxy/webapps/galaxy/api/tools.py#L293), line 291__:
>    ```python
>    def uninstall_dependencies(self, trans: GalaxyWebTransaction, id, **kwds):
>    ```
>    The `GalaxyWebTransaction` is an object which represents our API request. It is internal to the Galaxy server. We do not specify it as a parameter in BioBlend.
>
>    We can see that it also expects an `id` parameter, which is the tool ID. The `kwds` argument indicates that the endpoint might accept additional keyword arguments as well.
>
>    Let's check the method documentation:
>    ```python
>    """
>    DELETE /api/tools/{tool_id}/dependencies
>    Attempts to uninstall requirements via the dependency resolver
>
>    parameters:
>
>        index:
>            index of dependency resolver to use when installing dependency.
>            Defaults to using the highest ranking resolver
>
>        resolver_type:
>            Use the dependency resolver of this resolver_type to install
>            dependency
>    """
>    ```
>    The endpoint expects a DELETE request at URL `/api/tools/{tool_id}/dependencies`. The `id` parameter in the method signature corresponds to the variable `{tool_id}` present in the endpoint URL.
>
>    There are two additional parameters: `index` and `resolver_type`. These parameters specify how the server should resolve the tool dependencies. For an average API user these options are probably too specific as they require knowledge of the resolvers used by the Galaxy server back end. We will skip these parameters for this example.
>
>    Lastly, we need to find out what the endpoint returns. Unfortunately the return type is not listed in the method documentation. It is a reality that various parts of the Galaxy API lack complete documentation, although this is being worked on.
>
>    Let's look at the return statement of the endpoint:
>    ```python
>    return tool.tool_requirements_status
>    ```
>    Here the tool API controller calls the `tool_requirements_status` property of the `Tool` class instance, which represents the tool corresponding to the `id` parameter. Let's also check the definition of this property.
>
>    __File [tools/\_\_init\_\_.py](https://github.com/galaxyproject/galaxy/blob/v21.01/lib/galaxy/tools/__init__.py#L1833), line 1833__:
>    ```python
>    def tool_requirements_status(self):
>        """
>        Return a list of dictionaries for all tool dependencies with their
>        associated status
>        """
>        return self._view.get_requirements_status(
>            {self.id: self.tool_requirements},
>            self.installed_tool_dependencies
>        )
>    ```
>    The documentation of this method mentions that it returns a list of dictionaries for all tool dependencies with their associated status. Now we know the return type of the endpoint method!
>
>    Our BioBlend method will receive this list of dictionaries encoded as JSON in the server response. It will parse the JSON and return the list. Thus, the signature of our new `uninstall_dependencies()` method will be as follows:
>
>    ```python
>    def uninstall_dependencies(self, tool_id: str) -> List[dict]:
>    ```
>
>    The `self` parameter corresponds to the `ToolsClient` instance. The parameter `tool_id` corresponds to the encoded ID for a given tool. This ID will be substituted in place of `{tool_id}` in the request URL.
>
>
> 2. __Adding the method body__
>
>    Our method is part of the `ToolsClient` class. This means that our method has access to all its helper methods.
>    Let's recall that the endpoint expects a DELETE request at URL `/api/tools/{tool_id}/dependencies`.
>
>    To translate this into BioBlend source code, we do the following:
>
>    - We use the `self._make_url()` helper method to construct a valid URL with the supplied `tool_id` parameter. Finally, we need to append `'/dependencies'` for our endpoint.
>
>        ```python
>        url = self._make_url(tool_id) + '/dependencies'
>        ```
>
>    - BioBlend clients also include helper methods for making requests of various types. In this case we want to make a DELETE request, so we use the `self._delete()` helper method with our constructed URL as argument. These helper methods are inherited from the `Client` class located at [bioblend/galaxy/client.py](https://github.com/galaxyproject/bioblend/blob/v0.16.0/bioblend/galaxy/client.py#L19).
>
>
>    The `_delete()` method also expects a `payload` argument. Our payload will be empty (`{}`), since we use `tool_id` only to construct the URL and there are no additional parameters to pass. This helper method automatically parses the JSON response into a Python object, so we can directly return its result.
>
>    ```python
>    return self._delete(payload={}, url=url)
>    ```
>
> 3. __Adding the docstring__
>
>    We should document the method and its parameter for the user.
>    ```python
>    """
>    Uninstall dependencies for a given tool via a resolver.
>
>    :type tool_id: str
>    :param tool_id: id of the requested tool
>
>    :rtype: list of dicts
>    :return: Tool requirement statuses
>    """
>    ```
>
>    This is the docstring format used in BioBlend. At the top we provide a short general description of the method. Then we document each parameter with its type and a short description. The return value of the method is documented last. After the PR is merged, the docstring will be rendered as part of the [BioBlend documentation](https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html).
>
> 4. __Checking Galaxy version compatibility__
>
>    In case certain versions of Galaxy do not support our method's functionality, we should append a note to the method's docstring.
>
>    __Example__:
>
>    ```
>    .. note::
>        This method is only supported by Galaxy 19.09 or later.
>    ```
>
>    Previously we made sure that the `dev` version of Galaxy supported our method. Let's now check if older versions of Galaxy also support it. At the time of writing, Galaxy 17.09 is the earliest version supported by BioBlend, so it makes sense to check it first.
>
>    It turns out that Galaxy 17.09 also [supports](https://github.com/galaxyproject/galaxy/blob/v17.09/lib/galaxy/webapps/galaxy/api/tools.py#L159) our method! It's safe to assume that all later versions do as well, and thus we do not need to add a note.
>
>     > <tip-title>Determining the Galaxy version from within BioBlend</tip-title>
>     > In certain situations it might be useful to determine the version of the Galaxy server from BioBlend. We can determine the version using the `self.gi.config.get_version()` method.
>     >
>     > __Example__:
>     >
>     > ```python
>     > if self.gi.config.get_version()['version_major'] >= '21.01':
>     >     print('Newer version of Galaxy!')
>     > else:
>     >     print('Older version of Galaxy!')
>     > ```
>     > An example where this is useful is the [`download_dataset_collection()`](https://github.com/galaxyproject/bioblend/blob/v0.16.0/bioblend/galaxy/dataset_collections/__init__.py#L125) method. It downloads a dataset collection as an archive file. Early versions of Galaxy return a ‘tar.gz’ archive. Later versions return a ‘zip’ archive. The BioBlend method uses the Galaxy version to determine the archive type.
>     {: .tip}
>
> 5. __Putting it all together__
>
>    In the end, our new method should look something like this:
>
>    ```python
>    def uninstall_dependencies(self, tool_id: str) -> List[dict]:
>        """
>        Uninstall dependencies for a given tool via a resolver.
>
>        :type tool_id: str
>        :param tool_id: id of the requested tool
>
>        :rtype: list of dicts
>        :return: Tool requirement statuses
>        """
>        url = self._make_url(tool_id) + '/dependencies'
>        return self._delete(payload={}, url=url)
>    ```
>
> 6. __Writing a test for our new method__
>
>    It is best to keep BioBlend tests simple. We need to check that we get an expected response, but testing minute details is unnecessary, since those are the responsibility of the Galaxy server itself.<br>
>
>    Therefore, let's check that, when run on a given tool ID, the returned status indicates that the dependencies are null. We can reuse a tool ID from an existent test method. Let's take `CONVERTER_fasta_to_bowtie_color_index`, which has only one dependency.
>
>    ```python
>    def test_tool_dependency_uninstall(self):
>        statuses = self.gi.tools.uninstall_dependencies(
>             'CONVERTER_fasta_to_bowtie_color_index'
>        )
>        self.assertEqual(statuses[0]['model_class'], 'NullDependency')
>    ```
>
>     > <tip-title>Skip tests with the `@test_util.skip_unless_galaxy` decorator</tip-title>
>     > On our local machine we normally test our changes against a single version of Galaxy, for example `dev`. However, GitHub Actions continuous integration runs the BioBlend tests against all supported versions of Galaxy.
>     >
>     > At the time of writing BioBlend supports Galaxy 17.09 and later. However, certain functionality might not be available in all these versions. For example, `download_dataset_collection` is only supported by Galaxy 18.01 and later. We don't want to run tests specific to this functionality on earlier versions of Galaxy. They would simply fail. We can skip a test for specific versions of Galaxy by adding the `@test_util.skip_unless_galaxy` decorator above the test method in question.
>     >
>     > __Example__:
>     > ```python
>     > @test_util.skip_unless_galaxy('release_19.09')
>     > def test_some_functionality(self):
>     > ```
>     > This test will only be run against Galaxy 19.09 and later.
>     {: .tip}
{:.hands_on}


> <tip-title>Galaxy API parameter formats</tip-title>
>
> Implementing a BioBlend method often requires knowledge about the parameters which are accepted by the corresponding Galaxy API endpoint. At the time of writing there are a few different formats in which endpoints might accept their input parameters.
>
> #### Direct parameters
> A parameter can be specified explicitly in the signature of the endpoint method.
>
> __Example__:
>
> In the histories API [`delete()`](https://github.com/galaxyproject/galaxy/blob/v21.01/lib/galaxy/webapps/galaxy/api/histories.py#L384) endpoint method, the `id` parameter is specified explicitly.
>
> ```python
> def delete(self, trans, id, **kwd):
> ```
>
> #### Keyword parameters
>
> Other parameters might get passed to the endpoint as an element of `kwd`. If the documentation of the endpoint is lacking, these keyword parameters can be difficult to identify.
>
> #### The `q` and `qv` format
>
> Filtering parameters might also be expected as pairs of `q` and `qv` values. These are then parsed into filters by the Galaxy server.
>
> The general format is `q={filter}-{operation}` and `qv={value}`. The accepted values for these parameters must be identified in order to construct valid `q` and `qv` pairs.
>
> __Example__:
>
> The Datasets API [`index()` method](https://github.com/galaxyproject/galaxy/blob/v21.01/lib/galaxy/webapps/galaxy/api/datasets.py#L54) uses this format. The `q` and `qv` parameters are passed to the endpoint as part of `kwd`. They are not used in the endpoint itself, but rather implicitly passed into the `self.parse_filter_params()` helper method as part of `kwd`. See [the corresponding method](https://github.com/galaxyproject/bioblend/blob/v0.16.0/bioblend/galaxy/datasets/__init__.py#L146) of the `DatasetClient` to get a feeling how this can be handled in BioBlend.
>
{: .tip}

# Running the BioBlend tests

Finally, we should run our test to make sure that it passes. This would indicate that our method works correctly. After testing we will be ready to push our changes to GitHub and open a pull request!

This section outlines two approaches for running BioBlend tests: A simple approach that should suffice for basic testing, and a more involved approach that speeds up testing.

It makes sense to test our method against the most recent changes of Galaxy, so let's use Galaxy `dev`.

> <comment-title>Note</comment-title>
> On our local machine we generally only test against Galaxy `dev`, so it might happen that tests on GitHub still fail because there tests are run against all supported Galaxy versions. When this happens we simply need to update our changes to fix those errors as well, and we might need to check out older Galaxy versions for debugging locally.
{:.comment}

> <hands-on-title>Using the `run_bioblend_tests.sh` script</hands-on-title>
>
> This script is provided in the repository. It will lint the Python code and run the BioBlend tests. It works well for running all BioBlend tests at once, or for small tests that require little to no debugging or experimentation.
>
> Since BioBlend requires a Galaxy server, we must specify the path to the Galaxy server directory:
>
> -  `-g <galaxy_path>`
>
> Two useful optional parameters:
>
> -  `-e <python_version>`
> -  `-t <path_to_test>`
>
>     We can specify a subset of tests to run by supplying this argument. This is specified in [pytest](https://pytest.org/) format. See the [documentation](https://docs.pytest.org/en/latest/how-to/usage.html#specifying-which-tests-to-run) for more information.
>
> __Examples__:
>
> Let's assume that our galaxy directory is located at `../galaxy`.
>
> Then this command runs all BioBlend tests:
> ```shell
> ./run_bioblend_tests -g ../galaxy -e py39
> ```
> And this command runs only the test named `test_get_jobs`:
> ```shell
> ./run_bioblend_tests.sh \
>     -g ../galaxy \
>     -e py39 \
>     -t tests/TestGalaxyJobs.py::TestGalaxyJobs::test_get_jobs
> ```
> __Downside of the `run_bioblend_tests.sh` script__:
>
> The script starts and stops a new instance of the Galaxy server every time it is run. This makes it robust. However, it also means we have to wait for the server to start up every time. This makes it painful to debug more complicated tests.
{:.hands_on}

## Running the tests directly

We can speed up our test environment by controlling the Galaxy server and the BioBlend tests directly. The initial setup will be more involved, but this way we can keep the server running in the background while we do our testing.

> <warning-title>Caveat</warning-title>
>
> When the Galaxy server is not reset between test runs, tests that implicitly depend on a clean server state might start failing after the first run.
>
> For example, let's assume there is only a single test that creates a new history and checks that the total number of histories is equal to 1. This test will pass for a new Galaxy server because no histories exist yet. However, the next run there would be two histories and thus the test would fail.
>
> Although this is generally not a big issue, it is something to watch out for. There are a few BioBlend tests that will fail in this manner. Most of these tests measure an array length property that increases with every run and therefore fail after the first run.
>
> It is preferable to write tests that are robust in this regard!
{:.warning}

> <hands-on-title>Starting the Galaxy server with a custom configuration</hands-on-title>
>
> Open a terminal in the Galaxy base directory and execute the following commands:
>
> 1. __Declare an API key of our choosing to use for testing__
>
>    We will include this key in the Galaxy server configuration. The server will then accept incoming requests that use this key.
>
>    ```shell
>    API_KEY=LetMeIn
>    ```
>
> 2. __Create a temporary directory__
>
>    ```shell
>    TEMP_DIR=$(mktemp -d)
>    echo "Galaxy directory: $TEMP_DIR"
>    ```
>
>    The temporary directory will contain useful information for debugging tests, such as the ‘main.log' file and the ‘universe.sqlite' database.
>
> 3. __Export environment variables required by the Galaxy server__
>
>    ```shell
>    export SKIP_GALAXY_CLIENT_BUILD=1
>    export GALAXY_LOG_FILE=$TEMP_DIR/main.log
>    export GALAXY_CONFIG_FILE=$TEMP_DIR/test_galaxy.ini
>    ```
>
> 4. __Copy a sample tool configuration__
>
>    ```shell
>    cp config/tool_conf.xml.sample $TEMP_DIR/tool_conf.xml.sample
>    ```
>
> 5. __Declare the desired logging level__
>
>    ```shell
>    LOG_LEVEL=DEBUG
>    ```
>
>    The available logging levels can be found in the [documentation](https://docs.python.org/3/library/logging.html#logging-levels) of the Python [logging](https://docs.python.org/3/library/logging.html) library.
>
> 6. __Create the Galaxy configuration file__
>
>    ```shell
>    cat << EOF > $GALAXY_CONFIG_FILE
>    [server:main]
>
>    use = egg:Paste#http
>    port = 8080
>
>    [app:main]
>
>    log_level = $LOG_LEVEL
>    paste.app_factory = galaxy.web.buildapp:app_factory
>    database_connection = sqlite:///$TEMP_DIR/universe.sqlite?isolation_level=IMMEDIATE
>    file_path = $TEMP_DIR/files
>    new_file_path = $TEMP_DIR/tmp
>    tool_config_file = $TEMP_DIR/tool_conf.xml.sample
>    conda_auto_init = True
>    job_working_directory = $TEMP_DIR/jobs_directory
>    allow_library_path_paste = True
>    admin_users = test@test.test
>    allow_user_deletion = True
>    allow_user_dataset_purge = True
>    enable_beta_workflow_modules = True
>    master_api_key = $API_KEY
>    enable_quotas = True
>    cleanup_job = onsuccess
>    EOF
>    ```
>
> 7. __Start the Galaxy server__
>
>    ```shell
>    ./run.sh
>    ```
{:.hands_on}

Now we have a Galaxy server running with our specified configuration. We can run tests against this server in a separate terminal.

> <hands-on-title>Running the tests</hands-on-title>
>
> Navigate to the BioBlend base directory and execute the following commands:
>
> 1. __Declare the same API key we used for the Galaxy server__
>
>    ```shell
>    API_KEY=LetMeIn
>    ```
>
> 2. __Export environment variables required by BioBlend__
>
>    ```shell
>    export BIOBLEND_GALAXY_API_KEY=$API_KEY
>    export BIOBLEND_GALAXY_URL=http://localhost:8080/
>    ```
>
> 3. __Activate the virtual environment__
>
>    ```shell
>    source <GALAXY_DIRECTORY>/.venv/bin/activate
>    ```
>
> 4. __Declare the desired test logging level__
>
>    ```shell
>    LOG_LEVEL=WARNING
>    ```
>
> 5. __Run the desired BioBlend test(s)__
>
>    ```shell
>    # selection format: tests/<file>::<module>::<test>
>    pytest --override-ini log_cli_level=$LOG_LEVEL \
>        tests/TestGalaxyWorkflows.py::TestGalaxyWorkflows::test_show_versions
>    ```
>
> 6. __Lint our code to detect bad practices and potential errors__
>
>    ```shell
>    tox -e lint
>    ```
>
> 7. __Leave the virtual environment when we are done testing__
>
>    ```shell
>    deactivate
>    ```
{:.hands_on}

You may find your tests do not pass straight away. In this case, you need to identify the issue and modify the code to fix it. This might take several iterations.

Once tests are passing, the final step would be to [open a PR](https://github.com/galaxyproject/bioblend/compare) at the BioBlend GitHub repository! This will trigger automatic tests and after some time your new code will be reviewed by the BioBlend maintainers. After any concerns are addressed and all tests are passing, your PR will be merged into the `main` branch and incorporated into the next BioBlend release.

# Conclusion

We covered the basics of BioBlend development. We also touched briefly on some key concepts relating to the Galaxy API and the structure of the Galaxy back-end code.

Compared to Galaxy, BioBlend is a smaller project with limited complexity. Changes are generally not very hard to implement, which makes it a good candidate for contributions from those not yet familiar with Galaxy in its entirety.

Good luck!


topic: dev
tutorial: visualization-charts
title: JavaScript plugins
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: ebea1bae414785745db47e536cfbf036
last_modified: 2026-05-27T09:01:48.028031
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/visualization-charts/tutorial.html

content:
In this tutorial we are going to demonstrate how to add a third party
JavaScript-based visualization to Galaxy, and we'll talk about what the benefits
are. The plugin we've selected for this exercise is the [*PV-Javascript Protein
Viewer*](https://biasmv.github.io/pv/). It is an open source protein structure
viewer for `PDB`-files. There are many other popular protein structure viewers
available for the visualization of `PDB`-files such as e.g.
[NGL](https://arose.github.io/ngl/) (also available in Galaxy) and
[JSMol](https://chemapps.stolaf.edu/jmol/jsmol/jsmol.htm).

> <details-title>Background: What is the PDB (Protein Data Bank) file format?</details-title>
>
> The `PDB`-file format contains atomic coordinates of biomolecules derived through a range of experimental and computational methods. Most commonly the file contains a spatial crystallographic snapshot of a protein. There are 100s of thousands of protein structures publicly available at the Protein Data Bank ([https://www.rcsb.org](https://www.rcsb.org)). Proteins are usually labeled by a four-letter code.
> Here is an example of a `PDB`-file for a hydrolase bond to its inhibitor (PDB: [1ACB](https://www.rcsb.org/pdb/explore/explore.do?structureId=1acb)):
>
> ```
> HEADER    HYDROLASE/HYDROLASE INHIBITOR           08-NOV-91   1ACB
> TITLE     CRYSTAL AND MOLECULAR STRUCTURE OF THE BOVINE ALPHA-CHYMOTRYPSIN-EGLIN
> TITLE    2 C COMPLEX AT 2.0 ANGSTROMS RESOLUTION
> ...
> KEYWDS    SERINE PROTEASE, HYDROLASE-HYDROLASE INHIBITOR COMPLEX
> AUTHOR    R.Z.KRAMER,L.VITAGLIANO,J.BELLA,R.BERISIO,L.MAZZARELLA,
> AUTHOR   2 B.BRODSKY,A.ZAGARI,H.M.BERMAN
> ...
> REMARK   1 REFERENCE 1
> REMARK   1  AUTH   M.BOLOGNESI,L.PUGLIESE,G.GATTI,F.FRIGERIO,A.CODA,L.ANTOLINI,
> REMARK   1  AUTH 2 H.P.SCHNEBLI,E.MENEGATTI,G.AMICONI,P.ASCENZI
> REMARK   1  TITL   X-RAY CRYSTAL STRUCTURE OF THE BOVINE
> REMARK   1  TITL 2 ALPHA-CHYMOTRYPSIN(SLASH)EGLIN C COMPLEX AT 2.6 ANGSTROMS
> ...
> SEQRES   1 E  245  CYS GLY VAL PRO ALA ILE GLN PRO VAL LEU SER GLY LEU
> SEQRES   2 E  245  SER ARG ILE VAL ASN GLY GLU GLU ALA VAL PRO GLY SER
> SEQRES   3 E  245  TRP PRO TRP GLN VAL SER LEU GLN ASP LYS THR GLY PHE
> ...
> ATOM      1  N   CYS E   1       2.323 -16.405  18.812  1.00 43.48           N
> ATOM      2  CA  CYS E   1       3.017 -15.136  18.786  1.00 35.11           C
> ATOM      3  C   CYS E   1       4.134 -15.068  19.799  1.00 32.90           C
> ATOM      4  O   CYS E   1       4.173 -15.810  20.772  1.00 41.38           O
> ATOM      5  CB  CYS E   1       2.052 -13.969  19.139  1.00 31.14           C
> ATOM      6  SG  CYS E   1       1.246 -14.085  20.788  1.00 34.72           S
> ATOM      7  N   GLY E   2       4.993 -14.081  19.607  1.00 21.94           N
> ...
> HETATM 2292  O   HOH E 406      12.343   1.842  12.901  0.86 18.70           O
> HETATM 2293  O   HOH E 407      -4.767  17.237  10.630  1.00 59.78           O
> HETATM 2294  O   HOH E 408      11.489  -6.278  18.740  0.96 20.00           O
> ...
> ```
>
> More resources on this file format:
>
>   - [https://en.wikipedia.org/wiki/Protein_Data_Bank_(file_format) ](https://en.wikipedia.org/wiki/Protein_Data_Bank_(file_format))
>   - [https://www.wwpdb.org/documentation/file-format ](http://www.wwpdb.org/documentation/file-format)
{: .details}

As mentioned above we will be focusing on the *PV-Javascript Protein Viewer* in
this tutorial. Now that we have learned about the underlying file format, let us
continue by visiting the protein viewer developer site at
[https://biasmv.github.io/pv/](https://biasmv.github.io/pv/) to get familiar
with the plugin.

> <hands-on-title></hands-on-title>
>
> 1. View the plugin in action, rotate the molecule and change its style.
>
> 2. Under which license is this plugin distributed?
>
> 3. Can you find the minified code file of this plugin?
>
{: .hands_on}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Section 1 - Basic plugin setup

### Directory and plugin preparations

In this section we will download the viewer and add it to a local *Galaxy*
instance. All development takes place within the *Galaxy* codebase. The first
thing we are going to do is to clone a *Galaxy* instance and prepare the
directory structure for the new visualization plugin.

> <hands-on-title></hands-on-title>
>
> 1. Clone an instance of *Galaxy* in a path, further referred to as `$GALAXY_ROOT`:
>    ```bash
>    $ git clone https://github.com/galaxyproject/galaxy
>    ```
>
> 2. Navigate to the visualization plugin directory:
>    ```bash
>    $ cd $GALAXY_ROOT/config/plugins/visualizations
>    ```
>
> 3. Copy the existing example into a new directory:
>    ```bash
>    $ cp -r example myviz
>    ```
>
{: .hands_on}

This example visualization provides a great place to start with most of the
basics already in place.  Now that the directory structure is in place, let us
review the example visualization. Each visualization contains a set of <b>3</b>
files:

- Logo (`static/logo.png`) which will appear in *Chart*'s plugin selection interface.
- Configuration (`config/example.xml`) describing input parameters and options.
- Wrapper (`src/script.js`) which serves as a bridge between *Galaxy* and our 3rd-party plugin.

In the following sections we are going to discuss these files in more detail and
modify them to incorporate a new visualization. Let's start with the logo for
our visualization.

### Your visualization needs a logo

Each visualization is represented by a logo in the Galaxy interface. This makes
it easier for users to find and configure their visualization. The logo should
be in the `png`-file format. It will appear with a width of 120 pixels.

Here's an example [logo](../../files/charts-plugins/pdb/logo.png):

![Logo](../../files/charts-plugins/pdb/logo.png)

> <hands-on-title></hands-on-title>
>
> 1. Find an arbitrary image in `PNG`-file format. Possibly using *Google*'s [image search](https://images.google.com).
>
> 2. Copy it to the `myviz/static` directory and name it `logo.png`.
{: .hands_on}

### Configure the visualization

Each visualization has a configuration file. In this case it is named
`example.xml`. This file has conceptual similarities with a Tool's XML-file. It
allows developers to specify a variety of attributes and input parameters for
their visualization. Throughout this tutorial we are going to gradually augment
this file but for now we keep it simple.

> <hands-on-title></hands-on-title>
>
> 1. Rename the file to `config/myviz.xml`
>
> 2. Edit the file named `config/myviz.xml` and change the name and description.
>
> 3. Go to the Galaxy root directory to install dependencies, activate the virtual environment, and invoke the Galaxy client build.
>    ```bash
>    $ cd $GALAXY_ROOT
>    $ make setup-venv
>    $ source .venv/bin/activate
>    $ make client
>    ```
>
> 4. Run Galaxy
>    ```bash
>    $ ./run.sh
>    ```
>
{: .hands_on}

Your visualization should now be loaded.  You can verify that now by clicking
on `Visualize > Create Visualization` in the top menu bar of Galaxy and finding
your plugin with its new logo in the list there.

### Assign a new datatype to your visualization

> <hands-on-title></hands-on-title>
>
> 1. Open the file named `config/myviz.xml` and find the `<data_source>` section.
>
> 2. Replace the existing two `<test>` sections with:
>
>    `<test type="isinstance" test_attr="datatype" result_type="datatype">molecules.PDB</test>`
>
> 3. Remove the `<settings>` and `<groups>` sections.
>
{: .hands_on}

This links the plugins to the `PDB`-file format, which means that for any
history item of this file type the plugin will automatically be available.

### Modifying the wrapper

Now let's take a look at the wrapper which connects our visualization with
Galaxy. The wrapper consists of a module written in *JavaScript* and is
available at `src/script.js`:

The wrapper receives an `options` dictionary with the following <b>four</b> components:
 - *charts*: The model of the current visualization with attributes, settings etc.
 - *process*: A [jQuery.Deferred()](https://api.jquery.com/jquery.deferred/) object to allow asynchronous data requests within the wrapper
 - *dataset*: Details on the selected datasets such as url, ids etc. which can be used to access the dataset
 - *targets*: The DOM ids of the container elements to draw into

In this tutorial we will implement the *PV-Viewer* plugin. In order to execute a
3rd-party plugin we need to figure out how it works. This can be done by finding
a working example or documentation. Fortunately the *PV-Viewer* comes with both.
Let's take a look at the [documentation](https://pv.readthedocs.io/).

> <hands-on-title>Exploring the PV-Viewer</hands-on-title>
>
> 1. Identify the parameter which is needed to initialize the plugin when calling [*pv.Viewer()*](https://pv.readthedocs.io/en/v1.8.1/viewer.html#pv.Viewer).
>
> 2. Which of the wrapper option components represents this parameter?
>
> 3. Can you identify which `mode` settings are valid to render the structure with [*pv.Viewer.renderAs()*](https://pv.readthedocs.io/en/v1.8.1/viewer.html#pv.Viewer.renderAs)?
>
{: .hands_on}

Now that we have learned the basics on how the viewer plugin works, we can edit it and adjust  `script.js`.

> <hands-on-title></hands-on-title>
>
> 1. Access your visualization's `myviz/src` directory.
>    ```bash
>    $ cd $GALAXY_ROOT/config/plugins/visualizations/myviz
>    ```
>
> 2. Install the package for the *PV-Viewer*:
>    ```bash
>    yarn add bio-pv
>    ```
>
> 3. Modify your plugin's entrypoint script `src/script.js` to import the new `bio-pv` library we just added, add this at the top of the file:
>    ```js
>    import * as pv from 'bio-pv';
>    ```
>
> 4. Lastly, replace the `load` function contents in the same entrypoint file with the following code that will set up the protein viewer:
>     ```js
>     var viewer = pv.Viewer( document.getElementById( options.targets[ 0 ] ), {
>         width       : 'auto',
>         height      : 'auto',
>         antialias   : true,
>         outline     : true
>     });
>     $.ajax( {
>         url     : options.dataset.download_url,
>         success : function( response ) {
>             var structure = pv.io.pdb( response );
>             viewer.clear();
>             viewer.renderAs( 'protein', structure, 'cartoon', {} );
>             viewer.centerOn( structure );
>             viewer.autoZoom();
>             options.chart.state('ok', 'Chart drawn.');
>             options.process.resolve();
>         }
>     });
>     ```
{: .hands_on}

### Build the package

Now that we have completed the basic plugin definition, it is time to build the
scripts and libraries into a single bundle that Galaxy can use.  Galaxy
visualizations typically use [*Parcel*](https://parceljs.org) for this due to
its simplicity, and this is how the example project is already configured.  If
you're interested in more details, take a look at the `package.json` file in
your `myviz` plugin directory.

After it has been built and staged the plugin will be accessible through
*Galaxy*'s user interface. This process not require restarting your *Galaxy*
instance, just make sure to properly refresh your browser.

> <hands-on-title></hands-on-title>
>
> 1. Navigate to your visualization's root directory:
>    ```bash
>    $ cd $GALAXY_ROOT/config/plugins/visualizations/myviz
>    ```
>
> 2. Install any necessary JavaScript dependencies:
>    ```bash
>    $ yarn install
>    ```
>
> 3. Run the plugin build process using yarn:
>    ```bash
>    $ yarn run build
>    ```
>
> 4. Stage the scripts and run Galaxy:
>    ```bash
>    $ cd $GALAXY_ROOT
>    $ make client
>    $ ./run.sh
>    ```
>
{: .hands_on}

Lets test this.


> <comment-title>Clearing the Cache</comment-title>
>
> "For Developers" or anyone who needs to build multiple times, you may need to clear the cached files in multiple places using any of the following tasks:
>
> 1. Within the `client` directory so that the staged files are appropriately cleared out, to build more than once use: `yarn run gulp plugins`
>
> 2. If rebuild continues to fail: (1) delete the hidden *cache* directories in the `plugins/your-plugin/` directory, (2) delete the `plugin_build_hash.txt` file in order to force a rebuild, and (3) delete the transpiled `static/script.js` file in the `your-plugin/` directory.
>
{: .comment}


### Test the visualization

In this section we will select a `PDB`-file from the Protein Data Bank and visualize it with our new plugin.

> <hands-on-title></hands-on-title>
>
> 1. Visit [https://www.rcsb.org ](http://www.rcsb.org) and select a protein structure e.g. [1ACB](http://www.rcsb.org/pdb/explore/explore.do?structureId=1acb)
>
> 2. Copy the link to the raw `PDB`-file e.g.
>
>    ```bash
>    https://files.rcsb.org/view/1ACB.pdb
>    ```
>
> 3. Start your Galaxy instance
>
>    ```bash
>    $ cd $GALAXY_ROOT
>    $ ./run.sh
>    ```
>
> 4. Open the upload dialog, paste the above link and click on *Start*.
>
> 5. Close the upload dialog, and select the file from the history panel on the right.
>
> 6. Click on the *diagram* icon. (You must be logged in)
>
> 7. Find your visualization and click on its logo.
>
{: .hands_on}

![First view](../../images/pv_1.png)

## Section 2 - Allow different rendering modes

In this section we are going to augment the visualization such that users may
select different rendering modes. Similar to a Tool's XML file, developers may
specify input parameters which then will be presented to the user. The
definition of Tool and Visualization input parameters are similar, however the
latter is provided in *JavaScript* and not as XML.

More information on parameters can be found in the [wiki](https://docs.galaxyproject.org/en/latest/dev/schema.html).

> <hands-on-title></hands-on-title>
>
> 1. Add the following block into the `myviz.xml` file:
>    ```xml
>    <settings>
>       <input>
>           <name>mode</name>
>           <label>Display mode</label>
>           <type>select</type>
>           <display>radio</display>
>           <value>cartoon</value>
>           <help>Select the rendering mode.</help>
>           <data>
>               <data>
>                   <value>cartoon</value>
>                   <label>Cartoon</label>
>               </data>
>               <data>
>                   <value>lines</value>
>                   <label>Lines</label>
>               </data>
>               <data>
>                    <value>points</value>
>                    <label>Points</label>
>               </data>
>           </data>
>       </input>
>    </settings>
>    ```
>
> 2. Change the following line in `script.js`:
>    ```js
>    viewer.renderAs( 'protein', structure, 'cartoon', {} );
>    ```
>    to
>
>    ```js
>    var settings = options.chart.settings;
>    viewer.renderAs( 'protein', structure, settings.get( 'mode' ), {} );
>    ```
>
> 3. Rebuild the plugin
>
>    ```bash
>    $ yarn run build
>    $ cd $GALAXY_ROOT
>    $ make client
>    ```
>
> 4. Refresh your browser.
>
> 5. Load your visualization and test different rendering modes in the *Customization* tab of your visualization.
>
{: .hands_on}

## Conclusion


First of all, thank you for completing this tutorial. We have learned how to add
JavaScript visualizations to Galaxy utilizing the Charts framework.


topic: dev
tutorial: onedata-dev-instance
title: Setting up a dev Onedata instance
difficulty: introductory
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["storage"]
content_hash: 3052c74b0a79620c1e7524a5adc684d9
last_modified: 2026-05-27T09:01:48.026458
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/onedata-dev-instance/tutorial.html

content:
# Prerequisites

It's recommended that you have basic knowledge about Onedata. If needed, follow 
[this tutorial]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html %})
first!


# Introduction

If you are familiar with **docker**, you can easily set up a local Onedata
sandbox deployment. It may come in handy if you wish to test Galaxy & Onedata
integration or develop features related to data management in Galaxy.


# Demo mode

Use the official Onedata tutorial for running the system in
[demo mode](https://onedata.org/#/home/documentation/topic/stable/demo-mode).

If you are new to this, run a Onezone and Oneprovider instance in foreground.
When the setup is finished, log in to the Onezone interface, using the IP
address and credentials displayed in Onezone docker logs (as explained in 
[the guide](https://onedata.org/#/home/documentation/topic/stable/demo-mode)).

Use the Onezone **docker IP** as the **Onezone domain** in all Onedata related
configs in Galaxy. Use the predefined Space called **demo-space** for Storage
Location configs (global or BYOS).

> <warning-title>The demo mode runs on self-signed certificates</warning-title>
> You will need to accept the untrusted connection in your browser to enter the UI. 
> 
> Remember to disable TLS certificate validation in the Galaxy configs!
{: .warning}


Follow the 
[getting started tutorial]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html#access-tokens %})
to acquire an access token, or use the convenience scripts of the demo mode
(as explained in 
[the guide](https://onedata.org/#/home/documentation/topic/stable/demo-mode)).


# Related materials

The demo mode is used for setting up Onedata in Galaxy's Object Store
integration tests. 
[Take a look at the code](https://github.com/galaxyproject/galaxy/blob/dev/test/integration/objectstore/_base.py)
(search for "onedata").


topic: dev
tutorial: bioblend-api
title: Scripting Galaxy using the API and BioBlend
difficulty: introductory
time_estimation: 2h
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 089834f7e236dc49c3e8803d8d34f15c
last_modified: 2026-05-15T22:28:11.244284
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/bioblend-api/tutorial.html

content:
BioBlend ({% cite Sloggett2013 %}) is a Python library to enable simple interaction with Galaxy ({% cite Afgan2018 %}) via the command line or scripts.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Interacting with histories in Galaxy API

We are going to use the [requests](https://requests.readthedocs.io/) Python library to communicate via HTTP with the Galaxy server. To start, let's define the connection parameters.

**You need to insert the API key for your Galaxy server in the cell below**:
1. Open the Galaxy server in another browser tab
2. Click on "User" on the top menu, then "Preferences"
3. Click on "Manage API key"
4. Generate an API key if needed, then copy the alphanumeric string and paste it as the value of the `api_key` variable below.

```python
import json
from pprint import pprint
from urllib.parse import urljoin

import requests

server = 'https://usegalaxy.eu/'
api_key = ''
base_url = urljoin(server, 'api')
base_url
```

We now make a GET request to retrieve all histories owned by a user:

```python
headers = {"Content-Type": "application/json", "x-api-key": api_key}
r = requests.get(base_url + "/histories", headers=headers)
print(r.text)
hists = r.json()
pprint(hists)
```

As you can see, GET requests in Galaxy API return JSON strings, which need to be **deserialized** into Python data structures. In particular, GETting a resource collection returns a list of dictionaries.

Each dictionary returned when GETting a resource collection gives basic info about a resource, e.g. for a history you have:
- `id`: the unique **identifier** of the history, needed for all specific requests about this resource
- `name`: the name of this history as given by the user
- `deleted`: whether the history has been deleted.

There is no readily-available filtering capability, but it's not difficult to filter histories **by name**:

```python
pprint([_ for _ in hists if _['name'] == 'Unnamed history'])
```

If you are interested in more **details** about a given resource, you just need to append its `id` to the previous collection request, e.g. to the get more info for a history:

```python
hist0_id = hists[0]['id']
print(hist0_id)
r = requests.get(base_url + "/histories/" + hist0_id, headers=headers)
pprint(r.json())
```

As you can see, there are much more entries in the returned dictionary, e.g.:
- `create_time`
- `size`: total disk space used by the history
- `state_ids`: ids of history datasets for each possible state.

To get the list of **datasets contained** in a history, simply append `/contents` to the previous resource request.

```python
r = requests.get(base_url + "/histories/" + hist0_id + "/contents", headers=headers)
hdas = r.json()
pprint(hdas)
```

The dictionaries returned when GETting the history content give basic info about each dataset, e.g.: `id`, `name`, `deleted`, `state`, `url`...

To get the details about a specific dataset, you can use the `datasets` controller:

```python
hda0_id = hdas[0]['id']
print(hda0_id)
r = requests.get(base_url + "/datasets/" + hda0_id, headers=headers)
pprint(r.json())
```

Some of the interesting additional dictionary entries are:
- `create_time`
- `creating job`: id of the job which created this dataset
- `download_url`: URL to download the dataset
- `file_ext`: the Galaxy data type of this dataset
- `file_size`
- `genome_build`: the genome build (dbkey) associated to this dataset.

**New resources** are created with POST requests. The uploaded **data needs to be serialized** in a JSON string. For example, to create a new history:

```python
data = {'name': 'New history'}
r = requests.post(base_url + "/histories", data=json.dumps(data), headers=headers)
new_hist = r.json()
pprint(new_hist)
```

The return value of a POST request is a dictionary with detailed info about the created resource.

To **update** a resource, make a PUT request, e.g. to change the history name:

```python
data = {'name': 'Updated history'}
r = requests.put(base_url + "/histories/" + new_hist["id"], json.dumps(data), headers=headers)
print(r.status_code)
pprint(r.json())
```

The return value of a PUT request is usually a dictionary with detailed info about the updated resource.

Finally to **delete** a resource, make a DELETE request, e.g.:

```python
r = requests.delete(base_url + "/histories/" + new_hist["id"], headers=headers)
print(r.status_code)
```


## Exercise: Galaxy API

**Goal**: Upload a file to a new history, import a workflow and run it on the uploaded dataset.

> <question-title>Initialise</question-title>
> First, define the connection parameters. What variables do you need?
> > <solution-title></solution-title>
> > ```python
> > import json
> > from pprint import pprint
> > from urllib.parse import urljoin
> >
> > import requests
> >
> > server = 'https://usegalaxy.eu/'
> > api_key = ''
> > base_url = urljoin(server, 'api')
> > ```
> >
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>New History</question-title>
> Next, create a new Galaxy history via POST to the correct API.
> > <solution-title></solution-title>
> > ```python
> > headers = {"Content-Type": "application/json", "x-api-key": api_key}
> > data = {"name": "New history"}
> > r = requests.post(base_url + "/histories", data=json.dumps(data), headers=headers)
> > new_hist = r.json()
> > pprint(new_hist)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>Upload a dataset</question-title>
> **Upload** the local file `1.txt` to the new history. You need to run the special `upload1` tool by making a `POST` request to `/api/tools`. You don't need to pass any inputs to it apart from attaching the file as `files_0|file_data`. Also, note that when attaching a file the payload should **not** be serialized to a JSON string and you need to drop `Content-Type` from the request headers.
>
> You can obtain the `1.txt` file from the following URL, you'll need to download it first.
>
> ```
> https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/1.txt
> ```
>
> > <solution-title></solution-title>
> > ```python
> > data = {
> >     "history_id": new_hist["id"],
> >     "tool_id": "upload1"
> > }
> > with open("1.txt", "rb") as f:
> >     files = {"files_0|file_data": f}
> >     r = requests.post(base_url + "/tools", data=data, files=files, headers={"x-api-key": api_key})
> > ret = r.json()
> > pprint(ret)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>Find the dataset in your history</question-title>
> Find the new uploaded dataset, either from the dict returned by the POST request above or from the history contents.
> > <solution-title></solution-title>
> > ```python
> > hda = ret['outputs'][0]
> > pprint(hda)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>Import a workflow</question-title>
> **Import a workflow** from the local file `convert_to_tab.ga` by making a `POST` request to `/api/workflows`. The only needed data is `workflow`, which must be a deserialized JSON representation of the workflow `.ga` file.
>
> You can obtain the `convert_to_tab.ga` file from the following URL, you'll need to download it first.
>
> ```
> https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/convert_to_tab.ga
> ```
> > <solution-title></solution-title>
> >
> > ```python
> > with open("convert_to_tab.ga", "r") as f:
> >     workflow_dict = json.load(f)
> > data = {"workflow": workflow_dict}
> > r = requests.post(base_url + "/workflows", data=json.dumps(data), headers=headers)
> > wf = r.json()
> > pprint(wf)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>View the workflow details</question-title>
> View the details of the imported workflow by making a GET request to `/api/workflows`.
> > <solution-title></solution-title>
> > ```python
> > r = requests.get(base_url + "/workflows/" + wf["id"], headers=headers)
> > wf = r.json()
> > pprint(wf)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>Invoke the workflow</question-title>
> **Run** the imported workflow on the uploaded dataset **inside the same history** by making a `POST` request to `/api/workflows/WORKFLOW_ID/invocations`. The only needed data are `history` and `inputs`.
> > <solution-title></solution-title>
> >
> > ```python
> > inputs = {0: {'id': hda['id'], 'src': 'hda'}}
> > data = {
> >     'history': 'hist_id=' + new_hist['id'],
> >     'inputs': inputs}
> > r = requests.post(base_url + "/workflows/" + wf["id"] + "/invocations", data=json.dumps(data), headers=headers)
> > pprint(r.json())
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>View the results</question-title>
> View the results on the Galaxy server with your web browser. Were you successful? Did it run?
{: .question}

# Interacting with histories in BioBlend

If you need to install BioBlend into your Jupyter environment, you can execute:

``python
import sys
!{sys.executable} -m pip install bioblend
``

**You need to insert the API key for your Galaxy server in the cell below**:
1. Open the Galaxy server in another browser tab
2. Click on "User" on the top menu, then "Preferences"
3. Click on "Manage API key"
4. Generate an API key if needed, then copy the alphanumeric string and paste it as the value of the `api_key` variable below.

The user interacts with a Galaxy server through a `GalaxyInstance` object:

```python
from pprint import pprint

import bioblend.galaxy

server = 'https://usegalaxy.eu/'
api_key = ''
gi = bioblend.galaxy.GalaxyInstance(url=server, key=api_key)
```

The `GalaxyInstance` object gives you access to the various controllers, i.e. the resources you are dealing with, like `histories`, `tools` and `workflows`.
Therefore, method calls will have the format `gi.controller.method()`. For example, the call to retrieve all histories owned by the current user is:

```python
pprint(gi.histories.get_histories())
```

As you can see, methods in BioBlend do not return JSON strings, but **deserialize** them into Python data structures. In particular, `get_` methods return a list of dictionaries.

Each dictionary gives basic info about a resource, e.g. for a history you have:
- `id`: the unique **identifier** of the history, needed for all specific requests about this resource
- `name`: the name of this history as given by the user
- `deleted`: whether the history has been deleted.

**New resources** are created with `create_` methods, e.g. the call to create a new history is:

```python
new_hist = gi.histories.create_history(name='BioBlend test')
pprint(new_hist)
```

As you can see, to make POST requests in BioBlend it is **not necessary to serialize data**, you just pass them explicitly as parameters. The return value is a dictionary with detailed info about the created resource.

`get_` methods usually have **filtering** capabilities, e.g. it is possible to filter histories **by name**:

```python
pprint(gi.histories.get_histories(name='BioBlend test'))
```

To **upload** the local file `1.txt` to the new history, you can run the special upload tool by calling the `upload_file` method of the `tools` controller.

You can obtain the `1.txt` file from the following URL, you'll need to download it first.

```
https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/1.txt
```

```python
hist_id = new_hist["id"]
pprint(gi.tools.upload_file("1.txt", hist_id))
```

If you are interested in more **details** about a given resource for which you know the id, you can use the corresponding `show_` method. For example, to the get more info for the history we have just populated:

```python
pprint(gi.histories.show_history(history_id=hist_id))
```

As you can see, there are much more entries in the returned dictionary, e.g.:
- `create_time`
- `size`: total disk space used by the history
- `state_ids`: ids of history datasets for each possible state.

To get the list of **datasets contained** in a history, simply add `contents=True` to the previous call.

```python
hdas = gi.histories.show_history(history_id=hist_id, contents=True)
pprint(hdas)
```

The dictionaries returned when showing the history content give basic info about each dataset, e.g.: `id`, `name`, `deleted`, `state`, `url`...

To get the details about a specific dataset, you can use the `datasets` controller:

```python
hda0_id = hdas[0]['id']
print(hda0_id)
pprint(gi.datasets.show_dataset(hda0_id))
```

Some of the interesting additional dictionary entries are:
- `create_time`
- `creating job`: id of the job which created this dataset
- `download_url`: URL to download the dataset
- `file_ext`: the Galaxy data type of this dataset
- `file_size`
- `genome_build`: the genome build (dbkey) associated to this dataset.

To **update** a resource, use the `update_` method, e.g. to change the name of the new history:

```python
pprint(gi.histories.update_history(new_hist['id'], name='Updated history'))
```

The return value of `update_` methods is usually a dictionary with detailed info about the updated resource.

Finally to **delete** a resource, use the `delete_` method, e.g.:

```python
pprint(gi.histories.delete_history(new_hist['id']))
```

## Exercise: BioBlend

**Goal**: Upload a file to a new history, import a workflow and run it on the uploaded dataset.


> <question-title>Initialise</question-title>
> Create a `GalaxyInstance` object.
> > <solution-title></solution-title>
> > ```python
> > from pprint import pprint
> >
> > import bioblend.galaxy
> >
> > server = 'https://usegalaxy.eu/'
> > api_key = ''
> > gi = bioblend.galaxy.GalaxyInstance(url=server, key=api_key)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>New History</question-title>
> Create a new Galaxy history.
> > <solution-title></solution-title>
> > ```python
> > new_hist = gi.histories.create_history(name='New history')
> > pprint(new_hist)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>Upload a dataset</question-title>
> **Upload** the local file `1.txt` to the new history using `tools.upload_file()` .
>
> You can obtain the `1.txt` file from the following URL, you'll need to download it first.
>
> ```
> https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/1.txt
> ```
> > <solution-title></solution-title>
> > ```python
> > ret = gi.tools.upload_file("1.txt", new_hist["id"])
> > pprint(ret)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>Find the dataset in your history</question-title>
> Find the new uploaded dataset, either from the dict returned by `tools.upload_file()` or from the history contents.
> > <solution-title></solution-title>
> > ```python
> > hda = ret['outputs'][0]
> > pprint(hda)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```


> <question-title>Import a workflow</question-title>
> **Import a workflow** from the local file `convert_to_tab.ga` using `workflows.import_workflow_from_local_path()` .
>
> You can obtain the `convert_to_tab.ga` file from the following URL, you'll need to download it first.
>
> ```
> https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/convert_to_tab.ga
> ```
> > <solution-title></solution-title>
> > ```python
> > wf = gi.workflows.import_workflow_from_local_path("convert_to_tab.ga")
> > pprint(wf)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>View the workflow details</question-title>
> View the details of the imported workflow using `workflows.show_workflow()`
> > <solution-title></solution-title>
> > ```python
> > wf = gi.workflows.show_workflow(wf['id'])
> > pprint(wf)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>Invoke the workflow</question-title>
> **Run** the imported workflow on the uploaded dataset **inside the same history** using `workflows.invoke_workflow()` .
> > <solution-title></solution-title>
> > ```python
> > inputs = {0: {'id': hda['id'], 'src': 'hda'}}
> > ret = gi.workflows.invoke_workflow(wf['id'], inputs=inputs, history_id=new_hist['id'])
> > pprint(ret)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>View the results</question-title>
> View the results on the Galaxy server with your web browser. Were you successful? Did it run?
{: .question}

# Interacting with histories in BioBlend.objects

**You need to insert the API key for your Galaxy server in the cell below**:
1. Open the Galaxy server in another browser tab
2. Click on "User" on the top menu, then "Preferences"
3. Click on "Manage API key"
4. Generate an API key if needed, then copy the alphanumeric string and paste it as the value of the `api_key` variable below.

The user interacts with a Galaxy server through a `GalaxyInstance` object:

```python
from pprint import pprint

import bioblend.galaxy.objects

server = 'https://usegalaxy.eu/'
api_key = ''
gi = bioblend.galaxy.objects.GalaxyInstance(url=server, api_key=api_key)
```

All `GalaxyInstance` method calls have the `client.method()` format, where `client` is the name of the resources you dealing with. There are 2 methods to get the list of resources:

- `get_previews()`: lightweight (one GET request), retrieves basic resources' info, returns a list of **preview** objects
- `list()`: one GET request for each resource, retrieves full resources' info, returns a list of **full** objects.

For example, the call to retrieve previews of all histories owned by the current user is:

```python
pprint(gi.histories.get_previews())
```

**New resources** are created with `create()` methods, e.g. to create a new history:

```python
new_hist = gi.histories.create(name='BioBlend test')
pprint(new_hist)
```

As you can see, the `create()` methods in BioBlend.objects returns an object, not a dictionary.

Both `get_previews()` and `list()` methods usually have **filtering** capabilities, e.g. it is possible to filter histories **by name**:

```python
pprint(gi.histories.list(name='BioBlend test'))
```

To **upload** the local file `1.txt` to the new history, you can run the special upload tool by calling the `upload_file` method of the `History` object.

You can obtain the `1.txt` file from the following URL, you'll need to download it first.

```
https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/1.txt
```

```python
hda = new_hist.upload_file("1.txt")
pprint(hda)
```

Please note that with BioBlend.objects there is no need to find the upload dataset, since `upload_file()` already returns a `HistoryDatasetAssociation` object.

Both `HistoryPreview` and `History` objects have many of their properties available as **attributes**, e.g. the id.

If you need to specify the unique **id** of the resource to retrieve, you can use the `get()` method, e.g. to get back the history we created before:

```python
gi.histories.get(new_hist.id)
```

To get the list of **datasets contained** in a history, simply look at the `content_infos` attribute of the `History` object.

```python
pprint(new_hist.content_infos)
```

To get the details about one dataset, you can use the `get_dataset()` method of the `History` object:

```python
new_hist.get_dataset(hda.id)
```

You can also filter history datasets by name using the `get_datasets()` method of `History` objects.


To **update** a resource, use the `update()` method of its object, e.g. to change the history name:

```python
new_hist.update(name='Updated history')
```

The return value of `update()` methods is the updated object.

Finally to **delete** a resource, you can use the `delete()` method of the object, e.g.:

```python
new_hist.delete()
```


## Exercise: BioBlend.objects

**Goal**: Upload a file to a new history, import a workflow and run it on the uploaded dataset.

> <question-title>Initialise</question-title>
> Create a `GalaxyInstance` object.
> > <solution-title></solution-title>
> > ```python
> > from pprint import pprint
> >
> > import bioblend.galaxy
> >
> > server = 'https://usegalaxy.eu/'
> > api_key = ''
> > gi = bioblend.galaxy.objects.GalaxyInstance(url=server, api_key=api_key)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>New History</question-title>
> Create a new Galaxy history.
> > <solution-title></solution-title>
> > ```python
> > new_hist = gi.histories.create(name='New history')
> > pprint(new_hist)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>Upload a dataset</question-title>
> **Upload** the local file `1.txt` to the new history using the `upload_file()` method of `History` objects.
>
> You can obtain the `1.txt` file from the following URL, you'll need to download it first.
>
> ```
> https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/1.txt
> ```
> > <solution-title></solution-title>
> > ```python
> > hda = new_hist.upload_file("1.txt")
> > pprint(hda)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>Import a workflow</question-title>
> **Import a workflow** from the local file `convert_to_tab.ga` using `workflows.import_new()`
>
> You can obtain the `convert_to_tab.ga` file from the following URL, you'll need to download it first.
>
> ```
> https://raw.githubusercontent.com/nsoranzo/bioblend-tutorial/main/test-data/convert_to_tab.ga
> ```
> > <solution-title></solution-title>
> > ```python
> > with open("convert_to_tab.ga", "r") as f:
> >     wf_string = f.read()
> > wf = gi.workflows.import_new(wf_string)
> > pprint(wf)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>View the workflow inputs</question-title>
> > <solution-title></solution-title>
> > ```python
> > pprint(wf.inputs)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>Invoke the workflow</question-title>
> **Run** the imported workflow on the uploaded dataset **inside the same history** using the `invoke()` method of `Workflow` objects.
> > <solution-title></solution-title>
> > ```python
> > inputs = {'0': hda}
> > wf.invoke(inputs=inputs, history=new_hist)
> > ```
> {: .solution}
{: .question}

```python
# Try it out here!

```

> <question-title>View the results</question-title>
> View the results on the Galaxy server with your web browser. Were you successful? Did it run?
{: .question}


# Optional Extra Exercises

If you have completed the exercise, you can try to perform these extra tasks with the help of the online documentation:

1. Download the workflow result to your computer
2. Publish your history

{% snippet topics/admin/faqs/git-gat-path.md tutorial="bioblend-api" %}


topic: dev
tutorial: writing_tests
title: Writing Automated Tests for Galaxy
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: ["type: none", "type: none", "type: none"]
tags_json: []
content_hash: 75db62b36be17583a7165f6b444c7da1
last_modified: 2026-05-27T09:01:48.028253
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/writing_tests/tutorial.html

content:
The Galaxy code base contains thousands of tests that include tests of different types (unit vs. functional vs. end-to-end; client vs. backend, etc.) that are supported by a variety of testing frameworks and libraries. In this tutorial, we will offer a small, yet representative sample of the types of tests you might write, as well as the concepts and issues you may need to be familiar with when writing tests for Galaxy code, whether as part of a new feature you are implementing, or as a standalone contribution to Galaxy's testing code.

A good way to start learning about Galaxy's testing infrastructure and how to use it is to read the documentation article on the different types of tests that are present in the code base as well as how to determine which type is most appropriate for a given scenario (see [Writing Tests for Galaxy](https://docs.galaxyproject.org/en/master/dev/writing_tests.html)).

In addition to reading the documentation, it is essential to have a reasonable understanding of Galaxy's code organization. The best documentation resource for that is the [Galaxy Code Architecture slides]({% link topics/dev/ %}#st-architecture). In addition, we recommend the [Contributing a New Feature to Galaxy Core]({% link topics/dev/tutorials/core-contributing/tutorial.md %}) tutorial, which, among other things, combines some of the concepts from the architecture slides with the material covered in this tutorial.

The most detailed and up-to-date documentation on running Galaxy tests is located at the top of the `run_tests.sh` script in Galaxy's root directory.

To run Galaxy tests, you may also use the `pytest` command directly (except for client tests, which provide their own infrastructure and scripts described in ``client/README.md``. Using pytest directly is most convenient for running individual tests (although you might have to manually set the required environment variables for all but unit tests, as per documentation in `run_tests.sh`). However, keep in mind that the `run_scripts.sh` script optimizes the tests by reusing the same Galaxy instance and database for API tests, so running API tests in batch with pytest would be very inefficient.

Another useful resource is the [Debugging Galaxy]({% link topics/dev/tutorials/debugging/tutorial.md %}) tutorial which contains a lot of useful information on how to debug test failures that occur both locally and remotely.

Finally, nothing can substitute studying Galaxy's test code - we encourage you to always look for examples of similar tests and testing scenarios before you write your own.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Local development environment setup

{% snippet topics/dev/faqs/contributing.md %}

> <hands-on-title>Setup your local Galaxy instance</hands-on-title>
>
> 1. Use GitHub UI to fork Galaxy's repository at `galaxyproject/galaxy`.
> 2. Clone your forked repository to a local path, further referred to as `GALAXY_ROOT` and `cd` into `GALAXY_ROOT`. Note that we specify the tutorial branch with the `-b` option:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git clone https://github.com/<your-username>/galaxy GALAXY_ROOT
>    > cd GALAXY_ROOT
>    > ```
>    {: .code-in}
>
> 3. Before we can use Galaxy, we need to create a virtual environment and install the required dependencies. This is generally done with the `common_startup.sh` script:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > bash scripts/common_startup.sh --dev-wheels
>    > ```
>    {: .code-in}
>
>    {% icon warning %} Make sure you are using a [supported Python version](https://galaxyproject.org/admin/python/) (you can check your Python version with `python --version`). If your system uses an older version, you may specify an alternative Python interpreter using the `GALAXY_PYTHON` environment variable (`GALAXY_PYTHON=/path/to/alt/python bash scripts/common_startup.sh --dev-wheels`).
>
> 4. Activate your new virtual environment:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > . .venv/bin/activate
>    > ```
>    {: .code-in}
>
>    Once activated, you'll see the name of the virtual environment prepended to your shell prompt: `(.venv)$`.
>
> 5. Finally, let's create a new branch for your edits:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git checkout -b my-training
>    > ```
>    {: .code-in}
>
>    Now when you run `git branch` you'll see that your new branch is activated:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > git branch
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > ```bash
>    > >   training
>    > > * my-training
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
>    Note: `my-training` is just an example; you can call your new branch anything you like.
{: .hands_on}

# API tests

We turn to API tests when we need to test some feature that requires a running Galaxy instance and, in many cases, the Galaxy database. These tests use the Galaxy API to drive the test.

In a way, these tests may be the simplest to write. While the required setup of an API test is, certainly, more involved than that of a basic unit test, Galaxy's testing infrastructure takes care of all the heavy lifting, such as creating a test database, configuring and starting up a Galaxy instance, and tearing down the setup upon test completion. The testing infrastructure also provides a wealth of convenient abstractions that simplify pre-populating the database with the necessary state for each test, interacting with the API, as well as expressing expectations about the outcomes of a test. Thus, writing an API test boils down to calling the appropriate API endpoint and verifying the result.

We are not developing any new features in this tutorial (check out the [Contributing a New Feature to Galaxy Core]({% link topics/dev/tutorials/core-contributing/tutorial.md %}) tutorial). However, we need to exercise the API, so we will be using existing Galaxy functionality as our testing context. Much of that functionality is already covered by tests. Existing tests may include advanced concepts or details on Galaxy's internals that are beyond the scope of a tutorial on testing, so they are not optimal as examples for training. Therefore, the approach we will use is as follows:

1. Pick a controller `foo` in the `lib/galaxy/webapps/galaxy/api` directory.
2. Pick one of the existing endpoints from the selected controller. If the endpoint represents a `GET` request (e.g. `@router.get("/api/foos`), you may be able to view the results on a running instance at `[host]/api/foos`
3. Write a new test to verify the specified functionality.

Ideally, we'd like to follow the process of test-driven development: write a test, run it and watch it fail, implement the functionality under test, rerun the test to verify it passes. Although we are not developing any new functionality here, we still would like to verify that the test we have written is a legitimate test of the correctness of the underlying code. In other words, we want it to be deterministic, and fail if the input is invalid. For this, you may want to intentionally break the test on its first run to verify it doesn't pass under any condition. Then fix the test and rerun. Although this doesn't guarantee correctness, it helps prevent obvious errors in the testing code (such as forgetting to edit the endpoint after cutting and pasting code from a similar test function.

## Basic API test

Let's start by writing a basic test for a very simple API endpoint: `api/version`. The controller for this endpoint is located at `lib/galaxy/webapps/galaxy/api/configuration.py`. You can check the format of the data at `https://usegalaxy.org/api/version`.

First, you need to create a new file at `lib/galaxy_test/api/test_mytutorial.py`. For simplicity, we'll place all new tests in this module. Next, add a class definition for `TestMyTutorialApiTestCase` that should be a subclass of `ApiTestCase`. Then add a test method, `test_version_is_current`, where you will (1) call the API via the ``_get`` method that returns a response object; and (2) verify that the response contains the "24.1" version number (assuming you have cloned the "dev" branch; otherwise your version may be different).

If your test fails, one way to debug it is to insert a `breakpoint()` statement into the body of the test (right after the call to ``_get`` would be a logical spot), and then use [pdb](https://docs.python.org/3/library/pdb.html), Python's interactive debugger, to explore the response at runtime with the test paused (see [Debugging Galaxy]({% link topics/dev/tutorials/debugging/tutorial.md %}) for more details on using pdb to debug Galaxy).

{% include topics/dev/tutorials/writing_tests/api1.md %}

You may notice that in the provided solution we also call ``response.raise_for_status()``: this is a requests library function that verifies that the request was successful (of course, we can also explicitly check the status code). It is helpful to include this check, because otherwise, if an error is raised, the error message in the error log will be not as helpful. Try using a nonextistant endpoint and running the test with and without the `raise_for_status` call. Compare these error messages:

With the status check:
- `requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://127.0.0.1:9584/api/versionx`

Without the status check:
- `requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

## Test creating a simple object

Now let's write an API test for adding a simple object. By "simple", we mean that this object does not depend on any other objects, so, for example, we don't need to provide a reference to a History or User object to create it. We will be creating a role, using the `api/roles` `POST` endpoint located at `lib/galaxy/webapps/galaxy/api/roles.py`.

To create a role, we need to call the ``_post`` method, passing it 4 arguments:
- the string value of the endpoint
- the payload containing the data for the new Role object
- a boolean argument setting the format of the payload data: `json=True`
- a boolean argument setting the test user's admin status: `admin=True`

How do we know what data to include in the payload? Look at the method signature in the API:
```
self, trans: ProvidesUserContext = DependsOnTrans, role_definition_model: RoleDefinitionModel = Body(...)
```
`RoleDefinitionModel` is the Pydantic model that describes the structure of the data passed with this argument. Looking at its definition (check the import statement at the top of the file to find its location), we see the following:

```python
class RoleDefinitionModel(BaseModel):
    name: str = RoleNameField
    description: str = RoleDescriptionField
    user_ids: Optional[List[EncodedDatabaseIdField]] = Field(title="User IDs", default=[])
    group_ids: Optional[List[EncodedDatabaseIdField]] = Field(title="Group IDs", default=[])
```

Thus, we need to provide a name and a description, both of which are string values (the other members are optional), formatted as JSON. (For more details on how the API works, see [FastAPI's excellent documentation](https://fastapi.tiangolo.com/tutorial/).)

The response will return the Role object, so testing it is straightforward.

{% include topics/dev/tutorials/writing_tests/api2.md %}

## A better test for creating a simple object

Can we do better? We can verify that the new object has been added by using another endpoint to retrieve that object from the database. Here's a first take:

```python
def test_create_role_WRONG(self):
    ROLE_COUNT = 1
    # verify no roles except one built-in role for test user
    response = self._get("roles")
    response.raise_for_status()
    data = response.json()
    assert len(data) == ROLE_COUNT

    # create role
    name = 'my cool name'
    description = 'description of this cool role'
    payload = {
        "name": name,
        "description": description,
    }
    response = self._post("roles", payload, admin=True, json=True)
    response.raise_for_status()
    role = response.json()
    assert role["name"] == name
    assert role["description"] == description

    # verify role has been added
    response = self._get("roles")
    response.raise_for_status()
    data = response.json()
    assert len(data) == ROLE_COUNT + 1
```

Everything is straightforward: we verify that there is only n=1 role currently present (a built-in for the test user), we add the role, then retrieve all roles and verify that the new total is n + 1. Unfortunately, ***this is the wrong approach***. Try running this together with the previous version of the test - you'll get an assertion error: there's an extra role in the database!

```
FAILED lib/galaxy_test/api/test_mytutorial.py::TestMyTutorialApiTestCase::test_create_role_WRONG - AssertionError: assert 2 == 1
```

The reason for that is that we are using the same database for both tests, so whatever artifacts are created in one test will affect the following test if we make any kind of assumptions about the state of the database.

How do we rewrite this test without making assumptions about database state? We can't retrieve the roles and use their initial quantity as our baseline: that's a perfect recipe for a race condition (if some other test creates a role after we have counted the existing roles but before we have created our new role, our test will fail). Instead, we can use the role's name (which is unique across all roles) to verify that our object has been added.

We could generate our own unique name, but we can also simply use an existing method used across Galaxy's tests: ``get_random_name()`` (we need to add an import and override the ``setUp`` method for that: see the solution code). Now we have a test we can safely run together with our old test (or with any number of other tests that create role objects).

{% include topics/dev/tutorials/writing_tests/api3.md %}

Can't we destroy and create or re-populate the database for each test? We can, and there is infrastructure for that in `test/unit/data/model/db`. However, setting up and initializing the database is an expensive operation, which will be noticeable for a single test; Galaxy has hundreds of tests that rely on the database, so providing a fresh copy of the database for each test function is infeasible.

## Use Galaxy's populators for setting up database state

As our last example, we'll look at how to take advantage of some of the many abstractions provided by Galaxy's testing infrastructure. Supposing you'd like to test the "datasets-filter-by-history" functionality exposed via the ``api/datasets`` endpoint (see `lib/galaxy/webapps/galaxy/api/datasets.py`). The test design is straightforward:
- Create 2 history objects h1 and h2.
- Create n and m datasets associated with h1 and h2 respectively.
- Retrieve datasets filtering by h1. Verify there are n results.
- Retrieve datasets filtering by h2. Verify there are m results.

The question is, how do we create the datasets and the histories (and all their respective dependencies?). One way would be to use the API for each such object. However, that would result in a barely readable test function which would mostly consist of setup code. Instead, we want our tests to be simple and understandable at a glance, and contain as little infrastructure code as possible. Enter dataset populators! This is one of many abstractions available to you that make it much easier to write tests that otherwise would have required nontrivial setup.

{% include topics/dev/tutorials/writing_tests/api4.md %}

Populators are used extensively throughout API tests as well as integration and Selenium tests to both setup database state, as well as access information from the Galaxy server. For more details, see the populators module at ``lib/galaxy_test/base/populators.py``.

And we are done! Here's the final version of the code we added to this testing module.

{% include topics/dev/tutorials/writing_tests/api5.md %}

# Unit tests

Unit tests are best suited for testing well-defined, isolated functionality and, with rare exceptions, should not require a running Galaxy instance, a database, or any other external infrastructure. Unit tests are located in the `test/unit` directory.

There are numerous unit tests in the Galaxy code base. However, existing unit tests do not cover the entire code base and do not verify all the relevant logic (so there is plenty of space for improvement, and new contributions are always welcome!) In this tutorial, we will be writing unit tests for such code.

## Testing simple functions

We'll start by looking at the module `lib/galaxy/util/bytesize.py`. This module is not covered by unit tests. There are 2 obvious targets for unit tests: the `parse bytesize` function and the `to_unit` method of the `ByteSize` class. Our goal is to come up with a reasonable selection of unit tests that verify essential aspects of this functionality.

We'll start with the `parse_bytesize` function. Look at the code of the function and identify the logic that, you think, needs testing. Essentially, you want to ensure that any combination of valid input produces the expected output. Next, add a few tests. (see [https://docs.pytest.org/en/8.2.x/how-to/assert.html](https://docs.pytest.org/en/8.2.x/how-to/assert.html) for examples)

{% include topics/dev/tutorials/writing_tests/unit1.md %}

Run the tests to verify they pass. You can use the `pytest` command:

> <code-in-title>Bash</code-in-title>
> ```bash
> pytest test/unit/util/test_bytesize.py
> ```
{: .code-in}

## Verifying that an error is raised

Now let's add a test that verifies that invalid input raises an error, as expected.

{% include topics/dev/tutorials/writing_tests/unit2.md %}

Run the tests to verify they pass.

## Using fixtures to reduce code duplication

Let's move on to the `to_unit` method of the `ByteSize`class. Just like you did for the previous function, add a few tests that verify that valid input produces expected output, and invalid input raises an error.

{% include topics/dev/tutorials/writing_tests/unit3.md %}

Run the tests to verify they pass.

Our tests are looking good for the most part, although code duplication is starting to creep in. It's time to refactor!

Let's start by factoring out the ByteSize object creation into a fixture (see [https://docs.pytest.org/en/8.2.x/how-to/fixtures.html](https://docs.pytest.org/en/8.2.x/how-to/fixtures.html)). In this particular case moving this code into a fixture might be unnecessary, however, it provides an example of an approach that is very useful in more complex scenarios and is heavily used in Galaxy's testing code.

{% include topics/dev/tutorials/writing_tests/unit4.md %}

Run the tests to verify they pass.

## Parametrization of test functions

Finally, our `test_bytesize_to_unit` test has a lot of assert statements of the same form. We can do better! Let's use pytest's test parametrization feature (see [https://docs.pytest.org/en/8.2.x/how-to/parametrize.html](https://docs.pytest.org/en/8.2.x/how-to/parametrize.html)) to eliminate this redundancy. Again, as with the fixture example, this particular case would be fine without this feature. However, sometime you want to run the same test function on hundreds of different input combinations, in which case this feature is invaluable (e.g. Galaxy's tool tests, or integration tests for configuration settings).

{% include topics/dev/tutorials/writing_tests/unit5.md %}

Run the tests to verify they pass.

## Dealing with less trivial code

So far we've been dealing with simple clean functions that had no external dependencies. However, quite often you will encounter lengthy blocks of code implementing nontrivial logic, that depend on objects that would be hard to instantiate in the context of a test. There are many excellent resources on this topic (the book [Working Effectively with Legacy Code](https://www.oreilly.com/library/view/working-effectively-with/0131177052/) by Michael Feathers is but one example). In this tutorial, we'll demonstrate a few basic approaches to handling such cases.

Your target module for the following exercises is `lib/galaxy/security/validate_user_input.py`. The unit tests for this module are located at `test/unit/data/security/test_validate_user_input.py`. You will be adding your code to this testing module.

If you look at the tests for this module, you'll note that most of its functionality is directly or indirectly covered by tests. However, the `validate_email` function does not have tests - and that's a shame for, despite being relatively small, it contains logic that is not immediately obvious (consider how the function behaves under different combinations of the allowlist and blocklist). That's the function we'll be testing.

The `validate_email` function is not fun to test. It depends on a Transaction and User objects - one does not simply ~~walk into Mordor~~ instantiate those in the context of a unit test. It has multiple paths of execution, to determine which it accesses the GalaxyApplication object (which is a reference to a running instance of Galaxy) to access configuration values, the User object to check the value of its email attribute, and in addition to all that, of course, it calls the database. None of these dependencies require testing, at least not in the context of a unit test. However, we do want to verify the function's core logic: the cases when no validation is required, when the provided email already exists, and the different combinations of the allowlist and blocklist.

## Mocking a dependency

First, let's test the case when the function returns early - when the email argument is the empty string or when its value is the same as the value of the `email` attribute of the provided User object. The first case is trivial, but the next one requires a User object. However, if we think about it, we only care about that user having a given string as its `email` attribute. So let's use a "test double" - a stand-in empty object, and give it an `email` attribute. We'll call it `MockUser` (strictly speaking, it's a stub, and ["mocks aren't stubs"](https://martinfowler.com/articles/mocksArentStubs.html), but we'll follow the naming convention often used in Galaxy). We can pass `None` as the transaction object in both cases.

{% include topics/dev/tutorials/writing_tests/unit6.md %}

Now run the tests for this module:

> <code-in-title>Bash</code-in-title>
> ```bash
> pytest test/unit/data/security/test_validate_user_input.py
> ```
{: .code-in}

> <warning-title></warning-title>
> As a word of caution, mocking (or any kind of patching to accommodate testing) may lead to brittle tests: by relying too much on *how* the logic is implemented, we'll be forced to adjust the test each time that implementation changes. It's a tradeoff between having narrowly-scoped tests that will point to the exact location of the problem but may lock the code under test into a given state, and higher-level integration-type tests that test the end result without "looking under the hood", but may be less helpful when pinpointing the exact cause of a failed test. So, use sparingly and keep it simple.
{: .warning}

## Refactoring for testability

Next, we'd like to test the case when the provided email already exists, or doesn't exist. To determine the existence of an email the function calls the database - we don't want to do that. Instead, we can simulate both conditions, like we simulated the user email in the previous exercise.

For this, we need to factor out the code that calls the database into its own function - then we can override that function for our test. So let's modify `lib/galaxy/security/validate_user_input.py`.

{% include topics/dev/tutorials/writing_tests/unit7.md %}

## "Monkeypatching" the module under test

Now we can use pytest's "monkeypatch" built-in fixture to replace that function with a stub method and then use it to return `True` or `False` to test both cases. (see [https://docs.pytest.org/en/8.2.x/how-to/monkeypatch.html?highlight=monkeypatch](https://docs.pytest.org/en/8.2.x/how-to/monkeypatch.html?highlight=monkeypatch) for more details).

{% include topics/dev/tutorials/writing_tests/unit8.md %}

## More refactoring

Run the tests to verify they pass after our latest modifications. But one test fails! The problem is that our validation function is not yet ready for testing: it still relies on the transaction object to access the running instance of Galaxy to retrieve 2 configuration settings: `email_domain_allowlist_content` and `email_domain_blocklist_content`.

How would you fix this problem?

{% include topics/dev/tutorials/writing_tests/unit9.md %}

## More "monkeypatching"

Run the tests. They still fail. That's because we haven't replaced the new functions with a mock for our test. Try to do that - it's the same approach as we used for monkeypatching the `check_for_existing_email` function. Note that both lists should be `None` to be ignored.

{% include topics/dev/tutorials/writing_tests/unit10.md %}

Run the tests - they should pass now.

## Utilizing the mocks to test the logic

Now that our tests pass, it's time to test the last bit of logic: the allowlist and the blocklist. Our first step is to add a simple test to verify that if the allow-list is not empty, only emails with allowed domain names will be validated.

{% include topics/dev/tutorials/writing_tests/unit11.md %}

Run the tests to verify we haven't broken anything.

However, again, we are getting a lot of duplication from our monkeypatching code, and there are more tests to come. Time to move some of it into fixtures; **leave only the code that is specific to the test**.

{% include topics/dev/tutorials/writing_tests/unit12.md %}

Again, run the tests to verify we haven't broken anything.

## Reformatting for improved readability

Before we add more tests, let's reformat our code to group all the tests that target the `validate_email` function in one class. Keep in mind that each test gets its own instance of the class - so you cannot share instance state across tests. However, grouping related tests makes the testing module easier to navigate (there are more benefits; see [https://docs.pytest.org/en/8.2.x/getting-started.html#group-multiple-tests-in-a-class](https://docs.pytest.org/en/8.2.x/getting-started.html#group-multiple-tests-in-a-class)).

{% include topics/dev/tutorials/writing_tests/unit13.md %}

Again, run the tests to verify we haven't broken anything.

## Adding the remaining tests

Now we have all the infrastructure in place. We need to add 2 more tests. First, you may notice that if the allowlist is not empty, the blocklist is ignored. Let's write a test to verify this.

{% include topics/dev/tutorials/writing_tests/unit14.md %}

Run the tests - they should pass.

Finally, let's test the blocklist.

{% include topics/dev/tutorials/writing_tests/unit15.md %}

Run the tests one last time - they should pass.

And we are done! Here's the final version of the code we've added to this testing module.

{% include topics/dev/tutorials/writing_tests/unit16.md %}


topic: dev
tutorial: tool-annotation
title: Adding and updating best practice metadata for Galaxy tools using the bio.tools registry
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 63c69436a360c6332a4f53ab041c2bc6
last_modified: 2026-05-15T22:28:11.274581
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/tool-annotation/tutorial.html

content:
Galaxy offers thousands of tools. Many of these tools either have incomplete metadata or are not yet linked to sources of high-quality metadata such as [bio.tools](https://bio.tools/).

This prevents filtering for all tools in a specific research community or domain, and makes it all but impossible to employ advanced filtering with ontology terms like the ones from EDAM or to group tools based on an ontology to improve the Galaxy tool panel.

[EDAM](https://edamontology.org/page) ({% cite black2021edam %}) is a comprehensive ontology of well-established, familiar concepts that are prevalent within bioscientific data analysis and data management. It includes 4 main sections of concepts (sub-ontologies):

- **Topic**:  A category denoting a rather broad domain or field of interest, of study, application, work, data, or technology. Topics have no clearly defined borders between each other.
- **Operation**: A function that processes a set of inputs and results in a set of outputs, or associates arguments (inputs) with values (outputs).
- **Data**: Information, represented in an information artefact (data record) that is "understandable" by dedicated computational tools that can use the data as input or produce it as output.
- **Format**: A defined way or layout of representing and structuring data in a computer file, blob, string, message, or elsewhere.

![Simplified data flow diagram in EDAM architecture: boxes for concepts, lines for relations. Streamlined data management.](./images/EDAMrelations.png "EDAM architecture is simple. Boxes indicate top-level concepts (sections, sub-ontologies), and lines indicate types of relations. Source: <a href="https://edamontology.org/page">EDAM website</a>")

The ontology can be navigated using [EDAM Browser](https://edamontology.github.io/edam-browser/):

<iframe id="edam" src="https://edamontology.github.io/edam-browser/#operation_0291" frameBorder="1" width="80%" height="600px"> ![EDAM ontology browser](./images/edam_browser.png) </iframe>

A tool or software can then be characterized by different EDAM terms:
- A topic term, *e.g.*  [`Proteomics`](https://edamontology.github.io/edam-browser/#topic_0121),
- An operation (a specific scientific thing that a tool does) term, *e.g.* [`Peptide identification`](https://edamontology.github.io/edam-browser/#operation_3631),
- A data term for the type of biological data, *e.g.* [`Mass spectrum`](https://edamontology.github.io/edam-browser/#data_0943),
- A format term, *e.g.* [`Thermo RAW`](https://edamontology.github.io/edam-browser/#format_3712).

The annotation of tools can be done on [bio.tools](https://bio.tools/). bio.tools ({% cite ison2016tools %}) is a global portal for bioinformatics resources that helps researchers to find, understand, compare, and select resources suitable for their work. It relies on the EDAM ontology for standardizing the annotations.

In Galaxy, tools can be annotated with EDAM concepts, either by adding them directly to the `XML` wrapper or extracting them from their corresponding bio.tools entry by linking to it in the wrapper. The advantage of the second approach is that there is one source of truth (i.e. the bio.tools entry), which centralises the location for storage and update of metadata, including EDAM concepts, as well as preventing replication of metadata across multiple platforms.

The aim of this tutorial is to improve the annotation of a given Galaxy tool by either:

- Linking it to an existing bio.tools identifier,
- Creating a new bio.tools identifier first and then linking the Galaxy tool, or
- Updating an existing bio.tools entry with the proper EDAM concepts

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Choose a tool without a bio.tools identifier

To start, we need to select a tool without a bio.tools identifier.

> <hands-on-title>Choose a tool without a bio.tools identifier</hands-on-title>
>
> 1. Open [the list of Galaxy tools](https://galaxyproject.github.io/galaxy_codex/)
> 2. Click on **Add Condition**
> 3. Select *bio.tool id* in **Data** drop-down
> 4. Select *Empty* in **Condition** drop-down
> 5. Select a tool in the list
>
{: .hands_on}

# Determine if a bio.tools entry exists for your chosen tool

Now let's search for our selected tool on bio.tools.

> <hands-on-title>Search for a tool in bio.tools</hands-on-title>
>
> 1. Open [bio.tools](https://bio.tools/)
> 2. Type the name of your tool in the "Search bio.tools" bar on the top
> 3. Try also other naming possibilities of your tool, e.g. sometimes the galaxy tool name is not identical to the name of the tool in the publication.
>
{: .hands_on}

{% include _includes/cyoa-choices.html option1="No existing entry" option2="Existing entry" default="No bio.tool entry" text="Have you found the tool in bio.tools?" disambiguation="biotool"%}

<div class="No-existing-entry" markdown="1">

# Create a bio.tools entry for a tool

If the tool is not on bio.tools, we need to create a new entry and populate it with metadata.

> <hands-on-title> Create a bio.tools entry with minimum metadata </hands-on-title>
>
> 1. Sign up to [bio.tools](https://bio.tools/)
> 2. Select **Add a tool** from the drop-down **Menu**
> 3. Fill in general information in the first tab of the wizard. Required information includes:
>    1. **Tool name**
>    2. **Description**
>    3. **Homepage URL**
> 4. Add EDAM operation concepts, as well as EDAM data concepts for both inputs and outputs
>    
>    {% include topics/dev/tutorials/tool-annotation/add_edam_function.md %}
>
> 5. Add EDAM topic concepts
>    
>    {% include topics/dev/tutorials/tool-annotation/add_edam_topic.md %}
>
> 6. Add the tool type, license, language, and other metadata as needed
> 7. In the Permissions tab, select **Make this resource editable by anyone**
> 8. Click on **Validate** on the top
> 9. Click on **Save** to create the bio.tools entry
> 10. Copy the bio.tools id
{: .hands_on}

</div>

<div class="Existing-entry" markdown="1">

# Review and update the EDAM terms for an existing bio.tools entry

Before linking a Galaxy tool with its corresponding bio.tools entry, we need to check if the tool is correctly annotated with EDAM concepts.

> <hands-on-title>Check EDAM terms in a bio.tools entry</hands-on-title>
>
> 1. Open the bio.tools entry for the tool
> 2. Check the EDAM **Topic** terms by looking at the green boxes (if they exist) below the tool name, URL, and available versions
> 3. Check the EDAM **Operation** terms by looking at the blue boxes (if they exist) below the tool description
{: .hands_on}

{% include _includes/cyoa-choices.html option1="Terms to be modified" option2="Correct terms" default="Terms to be modified" text="What do you think about the EDAM terms in the bio.tools entry?" disambiguation="edamUpdate"%}

<div class="Terms-to-be-modified" markdown="1">

To modify EDAM terms in a bio.tools entry, we need to request editing rights and then modify this entry.

> <hands-on-title>Modify the EDAM concepts in a bio.tools entry</hands-on-title>
>
> 1. Sign up for [bio.tools](https://bio.tools/)
> 2. Click on **Request editing rights** at the bottom of the bio.tools entry page
> 3. Wait for the request to be approved
> 4. Click on **Update this record**
> 5. Update, or add, EDAM **Operation** term(s) and EDAM **Data** term(s) for both inputs and outputs
>    
>    {% include topics/dev/tutorials/tool-annotation/add_edam_function.md %}
>
> 6. Update, or add, EDAM **Topic** term(s)
>    
>    {% include topics/dev/tutorials/tool-annotation/add_edam_topic.md %}
> 7. Update, or add, the metadata for tool type, license, language, and other fields as needed
> 8. Click on **Validate** on the top
> 8. Click on **Save** to create the bio.tools entry
> 9. Copy the bio.tools ID
{: .hands_on}

</div>

</div>

# Linking a Galaxy tool to a bio.tools entry

To link a Galaxy tool to its corresponding bio.tools entry, we need to first find the source of the wrapper.

> <hands-on-title>Find the Galaxy wrapper</hands-on-title>
>
> 1. Go to the tool on any Galaxy server
> 2. Click on the drop-down menu next to the **Run tool** button
> 3. Select **See in Tool Shed**
> 4. Once in the Tool Shed, click on the link to the development repository
> 5. Fork the repository
>
> If the link to the development repository is not correct the column `Galaxy wrapper parsed folder` from the **Galaxy Codex** will also show you the location of the wrapper.
> 1. Open [Galaxy Codex](https://galaxyproject.github.io/galaxy_codex)
> 2. Search your tool
> 3. Expand the row
> 4. Open the link shown in the `Galaxy wrapper parsed folder` column
>
{: .hands_on}

Now we have the wrapper, and can add the bio.tools entry.

{% include _includes/cyoa-choices.html option1="Add entry to toolsuite" option2="Add entry to single tool" default="Ideal: Add bio.tools ID to toolsuite" text="How do you want to add the bio.tools entry?" disambiguation="biotool"%}

<div class="Add-entry-to-toolsuite" markdown="1">

> <hands-on-title>Check for an existing xrefs section</hands-on-title>
>
> 1. Open the Galaxy tool `macros.xml` file (*If this does not exist*: You have to add the entry to a single tool instead)
> 2. Check if a `xml name="xrefs"` section exists in this file.
>
{: .hands_on}

{% include _includes/cyoa-choices.html option1="Yes, an xrefs section exists" option2="No, an xrefs section does not exist" default="No, an xrefs section does not exist" text="Does an xref section already exist?" disambiguation="biotool"%}

<div class="yes-exists" markdown="1">

> <hands-on-title>Add bio.tools entry to the Galaxy toolsuite</hands-on-title>
>
> 1. In the `macros.xml` file, add the following lines into the existing `xrefs` section:
>    ```
>      <xrefs>
>          <xref type="bio.tools">biotool-id</xref>
>      </xrefs>
>    ```
> 2. Replace `biotool-id` in the example snippet above with the bio.tools ID for your tool
>
>    This section should now look a bit like this, but with other xref text inside:
>    <xml name="xrefs">
>      <xrefs>
>          <xref type="other cool stuff">supercool</xref>
>          <xref type="bio.tools">your-biotool-id</xref>
>      </xrefs>
>    </xml>
>
> 6. Commit the change on a new branch
> 7. Make a pull request (PR) against the original repository
> 8. Wait patiently for the PR to be merged, at which point the new bio.tools reference will be added to the Galaxy tool wrapper
> 9. Make sure to respond to any feedback from the owner of the wrapper
>
{: .hands_on}

</div>

<div class="no-exist" markdown="1">

> <hands-on-title>Add bio.tools entry to the Galaxy toolsuite</hands-on-title>
>
> 1. In the `macros.xml` file, add the following lines above any `requirements` sections, such that it does not break an existing block.
>    ```
>    <xml name="xrefs">
>      <xrefs>
>          <xref type="bio.tools">biotool-id</xref>
>      </xrefs>
>    </xml>
>    ```
> 2. Replace `biotool-id` in the example snippet above with the bio.tools ID for your tool
> 3. Commit the change on a new branch
> 4. Make a pull request (PR) against the original repository
> 5. Wait patiently for the PR to be merged, at which point the new bio.tools reference will be added to the Galaxy tool wrapper
> 6. Make sure to respond to any feedback from the owner of the wrapper
>
{: .hands_on}

</div>

Now, we need to make sure that each tool pulls the bio.tools entry from the macro.

> <hands-on-title>Add bio.tools line to each tool</hands-on-title>
>
> 1. In **each** tool `.xml` file, add the following line after the `macros` section. Usually, there are other similar lines that also `expand macro` to follow.
>    ```
>      <expand macro="xrefs"/>
>    ```
> 3. Commit the change on a new branch
> 4. Make a pull request (PR) against the original repository
> 5. Wait patiently for the PR to be merged, at which point the new bio.tools reference will be added to the Galaxy tool wrapper
> 6. Make sure to respond to any feedback from the owner of the wrapper
>
{: .hands_on}


<div class="Add-entry-to-single-tool" markdown="1">

> <hands-on-title>Add bio.tools entry to the Galaxy wrapper</hands-on-title>
>
> 1. Open the Galaxy tool XML file
> 2. Add the xref snippet indicated below:
>
>    ```
>    <xrefs>
>        <xref type="bio.tools">biotool-id</xref>
>    </xrefs>
>    ```
>
>    It should appear below the `macros` section and before the `requirements` section. If the tools already defines `xrefs` in the `macro.xml` file. The snippet should be placed there.
>
> 3. Replace `biotool-id` in the example snippet above with the bio.tools ID for your tool
> 4. Commit the change on a new branch
> 5. Make a pull request (PR) against the original repository
> 6. Wait patiently for the PR to be merged, at which point the new bio.tools reference will be added to the Galaxy tool wrapper
> 7. Make sure to respond to any feedback from the owner of the wrapper
>
{: .hands_on}

</div>

</div>

# Conclusion


topic: dev
tutorial: tool-from-notebook
title: nb2workflow: Generating Galaxy Tools From Jupyter Notebooks
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: []
content_hash: c126830b93ccc7dda5f5ec69f2416cda
last_modified: 2026-05-27T09:01:48.027775
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/tool-from-notebook/tutorial.html

content:
With the advance of python and Jupyter Notebooks, many astronomers have started using them for their data analysis pipelines. In response, members of the astronomy and data science communities have developed `nb2galaxy` -- a module of the `nb2workflow` library -- that allows for a quick and easy conversion of Jupyter Notebooks into Galaxy tools. While originally motivated by astronomical use cases, the tool is broadly applicable. This tutorial is therefore aimed at developers and researchers across disciplines who are interested in Galaxy.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Convert an example Jupyter notebook into a Galaxy tool

This tutorial starts with the setup of the required enviornment and a very basic Jupyter Notebook. Afterwards, each section introduces one or more concepts related to the conversion. The tutorial ends with a more complex example of inputs and outputs.
It is important to mention that the end result of this tutorial is a folder with a `.xml` file that describes the tool and other required files for the tool to function. In order for the tool to be on the UseGalaxy platform, one needs to publish the resulting folder on [toolshed](https://toolshed.g2.bx.psu.edu/) and UseGalaxy.


## Environment Setup

This tutorial uses the JupyterLab interface to edit notebooks, but any Jupyter-compatible editor (such as VS Code) will work just as well. 
The example notebook relies only on the `numpy` and `matplotlib` libraries. Two additional packages are required: `nb2workflow`  that provides the `nb2galaxy` conversion module and `planemo`  which is used to test and preview the generated Galaxy tool locally.
To create a virtual environment with all required packages:

```bash
mkdir nb2galaxy-example
cd nb2galaxy-example
python -m venv .venv
source .venv/bin/activate
pip install jupyterlab numpy matplotlib nb2workflow[galaxy] planemo
```

The initial notebook, that is used to plot a function $y=x^p$ in a given range, can be downloaded from:

```bash
wget https://raw.githubusercontent.com/esg-epfl-apc/nb2galaxy-example-repo/refs/tags/step-0/example_nb2workflow.ipynb
```

Open Jupyter lab to preview the notebook

```bash
jupyter lab
```

## Tagging Inputs and Outputs

Since `nb2galaxy` recognizes cell tags following the **Papermill** convention, one needs to create dedicated cells for the **input** and **output** of the notebook and then tag them accordingly. In the current example, the second cell is tagged as "parameters" and contains the  **input** values:

![parameters cell tag](../../images/nb2workflow-annotating-nb-inputs.png)

In order to obtain the **output**, one must save the figure to a file and provide its path in a new cell tagged as "outputs":

![outputs cell tag](../../images/nb2workflow-annotating-nb-outputs.png)


## Defining Dependencies

The automatic conversion of the notebook to a Galaxy tool requires an explicit definition of the dependencies. As long as Galaxy uses `conda` to install tool dependencies, the preferred way is to create an `environment.yml` file, even though `nb2galaxy` supports both `environment.yml` and `requirements.txt`. Here's an example `environment.yml`:

```yaml
name: nb2galaxy-example
channels:
  - conda-forge
dependencies:
  - numpy
  - matplotlib
```

When both `environment.yml` and `requirements.txt` are present, `nb2galaxy` attempts to reconcile them using Conda. First, each package from `requirements.txt` is searched by `conda`. If the package with the same name exists in the configured conda channels (only `conda-forge` by default), it is included in the final list of packages for reconciliation. Otherwise, the package is ignored, and a comment is added to the generated tool represented as an XML file.

In the end, all dependencies are resolved together to obtain fixed versions of the required packages that are written in the tool's XML file.

## Notebook to Galaxy tool conversion

At this stage, one can create a very simple Galaxy tool using `nb2galaxy` CLI:

```bash
nb2galaxy --environment_yml environment.yml example_nb2workflow.ipynb ./tooldir
```

This creates the `./tooldir` folder that contains a tool description `.xml` file and a python script file:

```
tooldir/
├── example_nb2workflow.py
└── example.xml

1 directory, 2 files
```

The minimal tool created from the notebook is ready. One can preview it locally with `planemo`:

```bash
planemo serve ./tooldir
```

## Improving the tool

In order to publish a tool on the UseGalaxy platform, the tool needs to pass the `planemo` linting tests. Currently, the example tool does not pass the tests: 

```bash
$ planemo lint ./tooldir/
Linting tool /home/dsavchenko/Projects/ESG/nb2galaxy-example-repo/tooldir/example.xml
.. CHECK (TestsNoValid): 1 test(s) found.
.. INFO (OutputsNumber): 1 outputs found.
.. INFO (InputsNum): Found 3 input parameters.
.. WARNING (HelpMissing): No help section found, consider adding a help section to your tool.
.. CHECK (ToolIDValid): Tool defines an id [example].
.. CHECK (ToolNameValid): Tool defines a name [example].
.. CHECK (ToolProfileValid): Tool specifies profile version [24.0].
.. CHECK (ToolVersionValid): Tool defines a version [0.1.0+galaxy0].
.. INFO (CommandInfo): Tool contains a command.
.. WARNING (CitationsMissing): No citations found, consider adding citations to your tool.
Failed linting
```

The tool requires a "help section" and citations. `nb2galaxy` can extract the "help section" from either `.rst` or `.md` files and citations from a BibTex `.bib` file.

Let us create a markdown file `galaxy_help.md` with the following help text:

```markdown
This tool is a simple example, automatically created from a Jupyter notebook using `nb2galaxy`.
```

and a `CITATION.bib` file:

```bibtex
@software{nb2galaxy-example,
  author = {Variu, Andrei and Savchenko, Denys},
  title = "Example of nb2galaxy",
  url = {https://github.com/esg-epfl-apc/nb2galaxy-example-repo},
  year = {2025}
}
```

Finally, by regenerating the tool

```bash
nb2galaxy --environment_yml environment.yml --citations_bibfile CITATION.bib --help_file galaxy_help.md example_nb2workflow.ipynb tooldir
```

the `planemo lint` tests should pass.

## Annotating input parameters

By default, `nb2galaxy` assumes all input parameters are of type `Integer`. Even though one can change the default configuration, one can explicitly provide parameter types using semantic annotations or python type annotations.

In this tutorial, we focus on semantic annotations because they allow for additional options. Out of the box, the conversion module uses the astronomy-specific ontology, described at <https://odahub.io/ontology/>, although a different ontology can be specified via a CLI option. 

Semantic annotations are added as comments following the parameter assignment. The syntax follows the truncated Turtle format, where the input parameter is implicitly considered the subject and the `a` predicate is optional.
In the current example:

```python
min_ = -10 # http://odahub.io/ontology#Float
max_ = 10 # http://odahub.io/ontology#Float
power = 3 # http://odahub.io/ontology#Integer
```

To improve readability, one can use the `oda:` prefix as a shorthand for <http://odahub.io/ontology#>, similar to how `rdfs:` shortens <http://www.w3.org/2000/01/rdf-schema#>. With this abbreviation, the annotations become:

```python
min_ = -10 # oda:Float
max_ = 10 # oda:Float
power = 3 # oda:Integer
```

Automatically, the variable names are used as labels in the tool interface. However, one can provide custom labels using semantic annotations, allowing for more user-friendly descriptions to be displayed:

```python
min_ = -10 # oda:Float; oda:label "Left bound of x-axis"
max_ = 10 # oda:Float; oda:label "Right bound of x-axis"
power = 3 # oda:Integer
```

Let us modify the notebook to add optional *y*-axis bounds:

![annotations](../../images/nb2workflow-annotating-nb-parameters-opt.png)

The resulting Jupyter notebook is available at <https://github.com/esg-epfl-apc/nb2galaxy-example-repo/blob/step-2/example_nb2workflow.ipynb>.

By recreating the tool 

```bash
nb2galaxy --environment_yml environment.yml --citations_bibfile CITATION.bib --help_file galaxy_help.md example_nb2workflow.ipynb tooldir
```

the `planemo lint` checks should pass and one can locally test the tool using `planemo serve` as before.

## Working with Input Datasets

Jupyter Notebooks often use datasets as inputs. In this section, we explain how to modify the previous notebook to accept a file as input. Specifically, the updated notebook should read polynomial coefficients from a `.csv` file, compute the roots and generate a plot of the corresponding polynomial function.

Firstly, create an input file, `dataset.csv`, which allows to run the notebook. This file also serves as test data for the generated tool.

```csv
-1.2,3.2,6,-2.1
```

Furthermore, replace the `power` input parameter from the notebook with:

```python
coeff_f = 'dataset.csv' # oda:POSIXPath
```

where every parameter annotated with `http://odahub.io/oontology#POSIXPath` or its subclasses is treated as an input dataset.

Read the polynomial coefficients from the input `.csv` file and use the `numpy.polynomial.Polynomial` function:

```python
coeff = np.genfromtxt(coeff_f, delimiter=',')
p = Polynomial(coeff)
```

Update the `plot` method invocation in the next cell

```python
x = np.linspace(min_, max_, 1000)
plt.figure(figsize=(5, 4))
im = plt.plot(x, p(x), label=f"{p}")
plt.xlabel("x")
plt.legend()
plt.ylim(ymin, ymax)

plt.savefig('.plot.png')
```

and add an additional output for the polynomial roots

```python
function_plot = '.plot.png' 
roots = list(p.roots())
```

You can consult the resulting notebook at <https://github.com/esg-epfl-apc/nb2galaxy-example-repo/blob/step-3/example_nb2workflow.ipynb>.

Afterwards, cleanup the tool directory `rm -rf tooldir` and regenerate the tool using the new name `polynome`:

```bash
nb2galaxy --name polynome --environment_yml environment.yml --citations_bibfile CITATION.bib --help_file galaxy_help.md example_nb2workflow.ipynb tooldir
```

Finally, lint the tool with `planemo lint tooldir` and preview it with `planemo serve tooldir`. The tool should accept a dataset as input and generate two outputs: a plot of the polynomial function and a JSON expression with the list of roots of the corresponding polynomial.

![Galaxy screenshot](../../images/nb2galaxy-final-example-screen.png)

## Acknowledgements

This tutorial is part of [EuroScienceGateway project](#funding).
Parts of `nb2workflow` library is developed and tested with support from [OSCARS](#funding), [ACME](#funding),
[AstroORDAS Explore](https://open-research-data-portal.ch/projects/development-of-open-research-data-analysis-services-supplementing-astronomical-open-research-data/) and
[AstroORDAS Establish](https://www.swissuniversities.ch/fileadmin/swissuniversities/Dokumente/Hochschulpolitik/ORD/Calls/Approved_Projects_Track_B.pdf)
projects.


topic: dev
tutorial: core-contributing
title: Contributing a New Feature to Galaxy Core
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: cce8c855a473da926335c3e82a9c2a3b
last_modified: 2026-05-27T09:01:48.025785
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/core-contributing/tutorial.html

content:
This tutorial walks you through developing an extension to Galaxy, and how to contribute back to the core project.

To setup the proposed extension imagine you're running a specialized Galaxy server and each of your users only use a few of Galaxy datatypes. You'd like to tailor the UI experience by allowing users of Galaxy to select their favorite extensions for additional filtering in downstream applications, UI extensions, etc..

Like many extensions to Galaxy, the proposed change requires persistent state. Galaxy stores most persistent state in a relational database. The Python layer that defines Galaxy's data model is setup by defining SQLAlchemy models.

The proposed extension could be implemented in several different ways on Galaxy's backend. We will choose one for this example for its simplicity, not for its correctness or cleverness, because our purpose here is to demonstrate modifying and extending various layers of Galaxy.

With simplicity in mind, we will implement our proposed extension to Galaxy by adding a single new table to Galaxy's data model called ``user_favorite_extension``. The concept of a favorite extension will be represented by a one-to-many relationship from the table that stores Galaxy's user records to this new table. The extension itself that will be favorited will be stored as a ``Text`` field in this new table. This table will also need to include an integer primary key named ``id`` to follow the example set by the rest of the Galaxy data model.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Forking Galaxy

{% snippet topics/dev/faqs/contributing.md %}

> <hands-on-title>Setup your local Galaxy instance</hands-on-title>
>
> 1. Use GitHub UI to fork Galaxy's repository at `galaxyproject/galaxy`.
> 2. Clone your forked repository to a local path, further referred to as `GALAXY_ROOT` and `cd` into `GALAXY_ROOT`. Note that we specify the tutorial branch with the `-b` option:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git clone https://github.com/<your-username>/galaxy GALAXY_ROOT
>    > cd GALAXY_ROOT
>    > ```
>    {: .code-in}
>
>
> 3. Before we can use Galaxy, we need to create a virtual environment and install the required dependencies. This is generally done with the `common_startup.sh` script:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > bash scripts/common_startup.sh --dev-wheels
>    > ```
>    {: .code-in}
>
>    {% icon warning %} Make sure you are using a [supported Python version](https://galaxyproject.org/admin/python/) (you can check your Python version with `python --version`). If your system uses an older version, you may specify an alternative Python interpreter using the `GALAXY_PYTHON` environment variable (`GALAXY_PYTHON=/path/to/alt/python bash scripts/common_startup.sh --dev-wheels`).
>
> 4. Activate your new virtual environment:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > . .venv/bin/activate
>    > ```
>    {: .code-in}
>
>    Once activated, you'll see the name of the virtual environment prepended to your shell prompt: `(.venv)$`.
>
> 5. Finally, let's create a new branch for your edits:
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git checkout -b my-feature
>    > ```
>    {: .code-in}
>
>    Now when you run `git branch` you'll see that your new branch is activated:
>
>    > > <code-in-title>Bash</code-in-title>
>    > > ```bash
>    > > git branch
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title></code-out-title>
>    > > ```bash
>    > >   dev
>    > > * my-feature
>    > > ```
>    > {: .code-out}
>    {: .code-2col}
>
>    Note: `my-feature` is just an example; you can call your new branch anything you like.
>
> 6. As one last step, you need to initialize your database. This only applies if you are working on
>    a clean clone and have not started Galaxy (starting Galaxy will initialize the database).
>    Initializing the database is necessary because you will be making changes to the database
>    schema, which cannot be applied to a database that has not been initialized.
>
>    To initialize the database, you can start Galaxy by executing the following script (might take some time when executing
>    for the first time):
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sh run.sh
>    > ```
>    {: .code-in}
>
{: .hands_on}

# Models

Galaxy uses a relational database to persist objects and object relationships. Galaxy's data model represents the object view of this data. To map objects and their relationships onto tables and rows in the database, Galaxy relies on SQLAlchemy, which is a SQL toolkit and object-relational mapper.

The mapping between objects and the database is defined in `lib/galaxy/model/__init__.py` via "declarative mapping", which means that models are defined as Python classes together with the database metadata that describes the database table corresponding to each class. For example, the definition of the JobParameter class includes the database table name:

```python
__tablename__ = "job_parameter"
```

and four `Column` attributes that correspond to table columns with the same names:

```python
id = Column(Integer, primary_key=True)
job_id = Column(Integer, ForeignKey("job.id"), index=True)
name = Column(String(255))
value = Column(TEXT)
```

Associations between objects are usually defined with the `relationship` construct. For example, the `UserAddress` model has an association with the `User` model and is defined with the `relationship` construct as the `user` attribute.

> <question-title>about Mapping</question-title>
>
> 1. What should be the SQLAlchemy model named corresponding to the table ``user_favorite_extension`` based on [other examples](https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/model/__init__.py)?
> 2. What table stores Galaxy's user records?
> 3. What is another simple table with a relationship with the Galaxy's user table?
>
> > <solution-title></solution-title>
> > 1. ``UserFavoriteExtension``
> > 2. ``galaxy_user``
> > 3. An example table might be the ``user_preference`` table.
> {: .solution }
{: .question }

To implement the required changes to add the new model, you need to create a new class in `lib/galaxy/model/__init__.py` with appropriate database metadata:

1. Add a class definition for your new class
2. Your class should be a subclass of `Base`
3. Add a `__tablename__` attribute
4. Add a `Column` attribute for the primary key (should be named `id`)
5. Add a `Column` attribute to store the extension
6. Add a `Column` attribute that will serve as a foreign key to the `User` model
7. Use the `relationship` function to define an association between your new model and the `User` model.

To define a regular `Column` attribute you must include the datatype:

```python
foo = Column(Integer)
```

To define an attribute that is the primary key, you need to include the `primary_key` argument:
(a primary key is one or more columns that uniquely identify a row in a database table)

```python
id = Column(Integer, primary_key=True)
```

To define an attribute that is a foreign key, you need to reference the associated table + its primary key column using the `ForeignKey` construct (the datatype will be derived from that column, so you don't have to include it):

```python
bar_id = Column(ForeignKey("bar.id"))
```

To define a relationship between tables, you need to set it on *both* tables:

```python
class Order(Base):
    …
    items = relationship("Item", back_populates="order")

class Item(Base):
    …
    order_id = Column(ForeignKey("order.id")
    awesome_order = relationship("Order", back_populates="items")
    # relationship not named "order" to avoid confusion: this is NOT the table name for Order
```

Now modify `lib/galaxy/model/__init__.py` to add a model class called `UserFavoriteExtension` as described above.

{% include topics/dev/tutorials/core-contributing/__init__.py_diff.md %}

# Migrations

There is one last database issue to consider before moving on to considering the API. Each successive release of Galaxy requires recipes for how to migrate old database schemas to updated ones. These recipes are called versions, or revisions, and are implemented using Alembic.

Galaxy's data model is split into the galaxy model and the install model. These models are persisted in one combined database or two separate databases and are represented by two migration branches: "gxy" (the galaxy branch) and "tsi" (the tool shed install branch). Schema changes for these branches are defined in these revision modules:
- `lib/galaxy/model/migrations/alembic/versions_gxy` (galaxy model)
- `lib/galaxy/model/migrations/alembic/versions_tsi` (install model)

We encourage you to read [Galaxy's documentation on migrations](https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/model/migrations), as well as relevant [Alembic documentation](https://alembic.sqlalchemy.org/en/latest/tutorial.html#create-a-migration-script).

For this tutorial, you'll need to do the following:
1. Create a revision template.
2. Edit the revision template, filling in the body of the upgrade and downgrade functions.
3. Run the migration.

> <question-title> about generating a revision template</question-title>
>
> What command should you run to generate a revision template?
>
> > <solution-title></solution-title>
> > ```bash
> > sh run_alembic.sh revision --head=gxy@head -m "Add user_favorite_extentions table"
> > ```
> > The title of the revision is an example only.
> {: .solution }
{: .question }

To fill in the revision template, you need to populate the body of the `upgrade` and `downgrade`
functions. The `upgrade` function is executed during a schema upgrade, so it should create your table.
The `downgrade` function is executed during a schema downgrade, so it should drop your table.

Note that although the table creation command looks similar to the one we used to define the model,
it is not the same. Here, the `Column` definitions are arguments to the `create_table` function.
Also, while you didn't have to specify the datatype of the `user_id` column in the model, you must
do that here.

{% include topics/dev/tutorials/core-contributing/revisionid_add_user_favorite_extensions.py_diff.md %}

> <question-title>about running the migration</question-title>
>
> What command should you run to upgrade your database to include the new table?
>
> > <solution-title></solution-title>
> > ```bash
> > sh manage_db.sh upgrade
> > ```
> {: .solution }
{: .question }


To verify that the table has been added to your database, you may use the SQLite CLI tool. First,
you login to your database; then you display the schema of the new table; and, finally, you verify
that the database version has been updated (the first record stored in the `alembic_version` table
is the revision identifier that corresponds to the revision identifier in the revision file you
added in a previous step.

{% include topics/dev/tutorials/core-contributing/sqlite_cli.md %}

# Test Driven Development

With the database model in place, we need to start adding the rest of
the Python plumbing required to implement this feature. We will do
this with a test-driven approach and start by implementing an API test
that exercises operations we would like to have available for favorite
extensions.

We will stick a test case for user extensions in
``lib/galaxy_test/api/test_users.py`` which is a relatively
straightforward file that contains tests for other user API
endpoints.

Various user-centered operations have endpoints under
``api/user/<user_id>`` and ``api/user/current`` is sometimes
substituable as the current user.

We will keep things very simple and only implement this functionality
for the current user.

We will implement three simple API endpoints.

Method   | Route                                                                  | Definition
------   | -----                                                                  | ---
`GET`    | `<galaxy_root_url>/api/users/current/favorites/extensions`             | This should return a list of favorited extensions for the current user.
`POST`   | `<galaxy_root_url>/api/users/current/favorites/extensions/<extension>` | This should mark an extension as a favorite for the current user.
`DELETE` | `<galaxy_root_url>/api/users/current/favorites/extensions/<extension>` | This should unmark an extension as a favorite for the current user.

Please review ``test_users.py`` and attempt to write a test case that:

- Verifies the test user's initially favorited extensions is an empty list.
- Verifies that a ``POST`` to ``<galaxy_root_url>/api/users/current/favorites/extensions/fasta`` returns a 200 status code indicating success.
- Verifies that after this ``POST`` the list of user favorited extensions contains ``fasta`` and is of size 1.
- Verifies that a ``DELETE`` to ``<galaxy_root_url>/api/users/current/favorites/extensions/fasta`` succeeds.
- Verifies that after this ``DELETE`` the favorited extensions list is again empty.

{% include topics/dev/tutorials/core-contributing/test_users.py_diff.md %}

# Run the Tests

Verify this test fails when running stand-alone.

> <code-in-title>Bash</code-in-title>
> ```bash
> ./run_tests.sh -api lib/galaxy_test/api/test_users.py::UsersApiTestCase::test_favorite_extensions
> ```
{: .code-in}

# Implementing the API

Add a new API implementation file to
``lib/galaxy/webapps/galaxy/api/`` called ``user_favorites.py`` with
an API implementation of the endpoints we just outlined.

To implement the API itself, add three methods to the user manager in
``lib/galaxy/managers/users.py``.

- ``get_favorite_extensions(user)``
- ``add_favorite_extension(user, extension)``
- ``delete_favorite_extension(user, extension)``

{% include topics/dev/tutorials/core-contributing/user_favorites.py_diff.md %}

{% include topics/dev/tutorials/core-contributing/users.py_diff.md %}

This part is relatively challenging and takes time to really become an
expert at - it requires greping around the backend to find similar examples,
lots of trial and error, debugging the test case and the implementation in unison,
etc..

Ideally, you'd start at the top of the test case - make sure it fails on the first API request,
implement ``get_favorite_extensions`` on the manager and the API code to wire it up, and continue
with ``add_favorite_extension`` before finishing with ``delete_favorite_extension``.

> <code-in-title>Bash</code-in-title>
> ```bash
> ./run_tests.sh -api lib/galaxy_test/api/test_users.py::UsersApiTestCase::test_favorite_extensions
> ```
{: .code-in}

# Building the UI

Once the API test is done, it is time to build a user interface for this addition to Galaxy. Let's get
some of the plumbing out of the way right away. We'd like to have a URL for viewing the current user's
favorite extensions in the UI. This URL needs to be registered as a client route in ``lib/galaxy/webapps/galaxy/buildapp.py``.

Add ``/user/favorite/extensions`` as a route for the client in ``buildapp.py``.

{% include topics/dev/tutorials/core-contributing/buildapp.py_diff.md %}

Let's add the ability to navigate to this URL and future component to
the "User" menu in the Galaxy masthead. The file ``client/src/layout/menu.js``
contains the "model" data describing the masthead. Add a link to the route
we described above to ``menu.js`` with an entry titled "Favorite Extensions".

{% include topics/dev/tutorials/core-contributing/menu.js_diff.md %}

The next piece of this plumbing is to respond to this route in the
analysis router. The analysis router maps URLs to UI components to
render.

Assume a VueJS component will be available called
``FavoriteExtensions`` in the file
``components/User/FavoriteExtensions/index.js``.  In
``client/src/entry/analysis/router.js`` respond to the route
added above in ``buildapp.py`` and render the fictitious VueJS component
``FavoriteExtensions``.

{% include topics/dev/tutorials/core-contributing/router.js_diff.md %}

There are many ways to perform the next steps, but like the API entry-point lets
start with a test case describing the UI component we want to write. Below is a
Jest unit test for a VueJS component that mocks out some API calls to ``/api/datatypes``
and the API entry points we implemented earlier and renders an editable list of
extensions based on it.

{% include topics/dev/tutorials/core-contributing/List.test.js_diff.md %}

Sketching out this unit test would take a lot of practice, this is a step
that might be best done just by copying the file over. Make sure the
individual components make sense before continuing though.

Next implement a VueJS component in that same directory called
``List.vue`` that fullfills the contract described by the unit test.

{% include topics/dev/tutorials/core-contributing/List.vue_diff.md %}

Finally, we added a level of indirection when we utilized this
component from the analysis router above by importing it from
``index.js``. Let's setup that file and import the component from
``List.vue`` and export as a component called
``FavoriteExtensions``.

{% include topics/dev/tutorials/core-contributing/index.js_diff.md %}


topic: dev
tutorial: interactive-tools
title: Galaxy Interactive Tools
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"none\""]
tags_json: []
content_hash: f1c2bda4fa4a8e2d95b1763731b73bf6
last_modified: 2026-05-15T22:28:11.256876
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/interactive-tools/tutorial.html

content:
<!--

Notes:

Recommendations for version numbering?

There are 2 Docker repositories:
https://github.com/abretaud/geoc_gxit_ansible/
hub.docker.com and http://quay.io/

Container dependancies on Galaxy:
https://docs.galaxyproject.org/en/latest/admin/special_topics/mulled_containers.html

The default port of dockerized RShiny app is 3838

The environment_variables would be needed to retrieve data from Galaxy History. In the example, this is therefore not necessary.

-->

This tutorial demonstrates how to build and deploy a Galaxy Interactive Tool (GxIT). GxITs are accessible through the Galaxy tool panel, like any installed Galaxy tool. Our example application is a simple R Shiny app that we call `Tabulator`.

There are three elements to a GxIT - an application script, a Docker container and a Galaxy tool XML file. This tutorial will take you through creating those components, and installing them as a new Interactive Tool into a local Galaxy instance and an existing Galaxy instance.


> <comment-title>If you plan to use an existing Galaxy instance</comment-title>
> The Galaxy server requires specific configuration in order to run Interactive Tools! Please refer to [this admin tutorial]({% link topics/admin/tutorials/interactive-tools/tutorial.md %}) for setting up a compatible Galaxy instance for development and testing of your GxIT.
> As well as updating the Galaxy server configuration, you will also have to configure the server's DNS provider to allow wildcard DNS records. This allows Galaxy to create unique host names (subdomains) for GxITs to be served over, separating them from the main Galaxy application.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## How do Interactive Tools work?

Interactive tools are a special breed of Galaxy tools, which is relatively
new to the Galaxy ecosystem - they are a work in progress!
GxITs enable the user to run an entire web application through Galaxy, which
opens as a new tab in the browser. This can enable users to explore and manipulate
data in a rich interface, such as Jupyter notebooks and RStudio. To see some
examples of GxITs in action, take a look at
[Galaxy EU "Live"](https://live.usegalaxy.eu/).


Interactive tool development builds on the canonical tool-wrapping process.
Instead of running a command, the tool feeds user input to a Docker container
running the application. Once it's up and running, the GxIT application can
then be accessed through a unique URL generated by the Galaxy server.
The user can then open the application, interact with their Galaxy data and
then terminate the tool. On termination, the Docker container is stopped and
removed, and the job is considered "complete".


## When is an Interactive Tool appropriate?

In a regular Galaxy tool the user passes data to the tool and waits for it to
run. They then get some output file(s) when the tool run is complete. In an
Interactive Tool, however, the users are provided with a graphical web interface
allowing them to interact with their data in real time. This is great for
visualising data, but if it is possible to provide the same
functionality with a regular tool (e.g. by rendering an HTML file as an output),
then an Interactive Tool might not be necessary.

If you are sure that a static output is not sufficient, then it's time to start building your first Interactive Tool!

> <comment-title>Interactive tool infrastructure</comment-title>
> Interactive tools require some rather complex infrastructure in order to work! However, most of the infrastructure requirements are taken care of by Galaxy core. As such, wrapping a new GxIT requires only three components:
> - Application script(s)
> - Docker container image
> - Galaxy tool XML
>
> However, as we will see in the next section, testing and deploying a GxIT is not so simple.
{: .comment}


# The development process

Since the infrastructure for building GxITs is not as well developed as regular
tool wrapping, the development process is unfortunately not so streamlined.
Where Planemo is typically used for tool linting and testing, the complex
architecture of GxITs requires a local instance or a development server to be built to manually
test and run the tool.
In addition, they are currently not supported by the Galaxy ToolShed and have to be installed
manually. This means that distributed GxITs can be found in [the Galaxy core codebase](https://github.com/galaxyproject/galaxy/tree/dev/tools/interactive),
and they can be manually enabled by the Galaxy server administrator.

However, the build process itself is not too complex!
We can break it down into just a few steps:

1. Find or create the application you wish to install on Galaxy
2. Find or create a Docker image containing this application
3. Write a Galaxy tool XML to pass IT details to Galaxy and pass user input to the Docker container
4. Add the tool XML to your Galaxy server (local or distant)
5. Try out the tool in the Galaxy interface. Error messages might appear in the Galaxy history.
6. If errors occur, revise the container or tool XML, and try again until the application is working.

The last step is likely where the most time is spent - the process requires
iterative development of the Docker image and tool XML until they work together.
As such, reducing the iteration time is the key to quick development! Throughout
the tutorial, we'll sprinkle in some tips on how to speed up the development
cycle.

> <comment-title>A note on architecture</comment-title>
> When building a GxIT, it is best to keep as much logic as possible in the tool XML, while keeping the Docker image as generic as possible. Why? Updating the tool XML is simple for Galaxy admins and developers in the future. They can view the tool XML directly on the server and understand how the tool works. The Docker image, meanwhile, is relatively opaque to other developers and administrators. To understand the container they must locate the original Dockerfile, which is not always available. Updating the container is more complex, as we will see later. Additionally, keeping the Docker container generic makes it testable outside of Galaxy.
>
> In short: updating the Docker container is hard, but updating the tool XML is easy!
{: .comment}


{% include _includes/cyoa-choices.html option1="Application" option2="JupyterLab" default="Application" text="Do you want to build a desktop application or a JupyterLab tool ?" %}

<div class="Application" markdown="1">

# The application

The application that we will wrap in this tutorial is a simple web tool which
allows the user to upload `csv` and `tsv` files, manipulate them and download
them. Our application is based on an R Shiny App hosted with a Shiny server.

Note that there is no link between this Interactive Tool and the Galaxy history.
More complex applications might be able to read and write outputs to the user's
history to create a more integrated experience - see the
[Additional components section](#galaxy-history-interaction)
to see an example of how this can be done.

Our example application can already be found [online](https://github.com/Lain-inrae/geoc-gxit).
In the following sections, we will study how it can be built into a GxIT.


> <hands-on-title></hands-on-title>
>
> First, let's clone the repository to take a quick look at it.
>
> ```console
> $ git clone https://github.com/Lain-inrae/geoc-gxit
> $ cd geoc-gxit
>
> $ tree .
> ├── Dockerfile
> ├── interactivetool_tabulator.xml
> ├── gxit
> │   ├── app.R
> │   └── install.R
> ├── Makefile
> └── README.md
> ```
> You'll find a Galaxy tool XML, a Dockerfile and two R scripts that will be injected into the container image.
>
{: .hands_on}

## The R scripts

- `app.R` defines the R Shiny application.
- `install.R` will be used by the docker container to install the R packages needed to run `app.R`.

These are specific to your container; these are required for an R-Shiny container, but won't be for other containers like a Jupyter notebook container.

## The Dockerfile

If you need some help to start your Dockerfile you can always get some inspiration from the previous interactive tools built in Galaxy. Go check some of the Dockerfiles, for instance the one for the [QGIS](https://github.com/usegalaxy-eu/docker-qgis) application or for [ODV](https://github.com/bgruening/docker-odv/tree/main).

> <tip-title>A brief primer on Docker</tip-title>
> Docker allows an entire application context to be containerized. A typical web application consists of an operating system, installed dependancies, web server configuration, database configuration and, of course, the codebase of the software itself. A Docker container can encapsulate all of these components in a single "image", which can be run on any machine with Docker installed.
>
> **Essentials of Docker:**
>
> 1. Write an image recipe as a Dockerfile. This single file selects an OS, installs software, pulls code repositories and copies files from the host machine (your computer).
> 2. Build the image from your recipe:
>
>    `docker build -t <image_name> .`
> 3. View existing images with
>
>    `docker image list`
> 4. Run a container with a specified command:
>
>    `docker run <image_name> <command>`
> 5. View running containers:
>
>    `docker ps`
> 6. Stop a running container:
>
>    `docker stop <container_name>`
> 7. Remove a stopped container:
>
>    `docker container rm <container_name>`
> 8. Remove an image:
>
>    `docker image rm <container_name>`
{: .tip}


Let's check out
[the Dockerfile](https://github.com/Lain-inrae/geoc-gxit/blob/master/Dockerfile)
that we'll use to containerize our application.

This container recipe can be used to build a Docker image which can be pushed to a
container registry in the cloud, ready for consumption by our Galaxy instance:

```dockerfile
# Set image to build upon
FROM rocker/shiny

# set author
MAINTAINER Lain Pavot <lain.pavot@inra.fr>

## we copy the installer and run it before copying the entire project to prevent
## reinstalling everything each time the project has changed

COPY ./gxit/install.R /tmp/

RUN \
        apt-get update                                \
    &&  apt-get install -y --no-install-recommends    \
        fonts-texgyre                                 \
    &&  Rscript /tmp/install.R                        \
    &&  apt-get clean autoclean                       \
    &&  apt-get autoremove --yes                      \
    &&  rm -rf /var/lib/{apt,dpkg,cache,log}/         \
    &&  rm -rf /tmp/*                                 ;


# ------------------------------------------------------------------------------

# These default values can be overridden when we run the container:
#     docker run -p 8080:8080 -e PORT=8080 -e LOG_PATH=/tmp/shiny/gxit.log <container_name>

# We can also bind the container $LOG_PATH to a local directory in order to
# follow the log file from the host machine as the container runs. This command
# will create the log/ directory in our current working directory at runtime -
# inside we will find our Shiny app log file:
#     docker run -p 8888:8888 -e LOG_PATH=/tmp/shiny/gxit.log -v $PWD/log:/tmp/shiny <container_name>

ARG PORT=8765
ARG LOG_PATH=/tmp/gxit.log

ENV LOG_PATH=$LOG_PATH
ENV PORT=$PORT

# ------------------------------------------------------------------------------

# Edit shiny-server config to use our port
RUN cat /etc/shiny-server/shiny-server.conf \
    | sed "s/3838/${PORT}/" > /etc/shiny-server/shiny-server.conf.1
RUN mv /etc/shiny-server/shiny-server.conf.1 /etc/shiny-server/shiny-server.conf

# ------------------------------------------------------------------------------

RUN mkdir -p $(dirname "${LOG_PATH}")
EXPOSE $PORT
COPY ./gxit/app.R /srv/shiny-server/

CMD ["/bin/sh", "-c", "shiny-server > ${LOG_PATH} 2>&1"]
```

> <tip-title>Shiny-based interactive tools</tip-title>
> In a previous version of this tutorial, we ran the Shiny App with `R -e "shiny:runApp()"`
> rather than using `shiny-server`. The latter is better practice, because it ensures that
> ports are mapped correctly for websocket functionality. With `shiny::runApp()` you will
> probably notice a websocket timeout in the app when run as a GxIT - the UI often greys
> out and becomes unresponsive after 20-30 seconds.
>
{: .tip}

This image is already hosted on [Docker Hub](https://hub.docker.com/r/ancelete/first-gxit)
, but anyone can use this Dockerfile to rebuild the image if necessary.
If so, don't forget to create a `gxit` folder containing `app.R` and `install.R`
next to your Dockerfile.

> <hands-on-title></hands-on-title>
>
> Let's start working on this Docker container.
>
> 1. Install Docker as described on the [docker website](https://docs.docker.com/engine/install/). Click on your distribution name to get specific information.
>
> 2. Now let's use the recipe to build our Docker image.
>
>    ```sh
>    # Build a container image from our Dockerfile
>    IMAGE_TAG="myimage"
>    LOG_PATH=`pwd`  # Create log output in current directory
>    PORT=8765
>    docker build -t $IMAGE_TAG --build-arg LOG_PATH=$LOG_PATH --build-arg PORT=$PORT .
>    ```
>
>    > <tip-title>Automating the build</tip-title>
>    > While developing the Docker container you may find yourself tweaking and rebuilding the container image many times.
>    > In the GitHub repository linked above, you'll notice that the author has used a `Makefile` to accelerate the build and deploy process.
>    > This allows the developer to simply run `make docker` and `make push_hub` to build and push the container, or `make` to rebuild the container after making changes during development. Check out the `Makefile` to see what commands can be run using `make` in this repository.
>    >
>    {: .tip}
{: .hands_on}

If you are lucky, you might find an available Docker image for the application you are trying to wrap. Some configuration changes can be needed such as:

1. Expose the correct port. The application, Docker and tool XML ports must be aligned!
2. Log output to an external file - useful for debugging.
3. Make the application callable from tool `<command>` - this sometimes requires a wrapper script to interface the application inside the container (we'll take a look at this later).



## Test the image

Before we go pushing our container to the cloud, we should give it a local test run to ensure that it's working correctly on our development machine. Have a play and see how our little web app works!

> <hands-on-title></hands-on-title>
> ```sh
> # Run our application in the container
> docker run -it -p 127.0.0.1:8765:$PORT $IMAGE_TAG
>
> # Or to save time, take advantage of the Makefile
> make it
>
> # Give it a few moments to start up, and the application should be available
> # in your browser at http://127.0.0.1:8765
> ```
{: .hands_on}

## Push the image

If you are happy with the image, we are ready to push it to a container registry
to make it accessible to our Galaxy server.

During development, we suggest making an account on
[Docker Hub](https://hub.docker.com/)
if you don't have one already. This can be used for hosting container images
during development.
[Docker Hub](https://hub.docker.com/)
has great documentation on creating repositories, authenticating with tokens
and pushing images.

> <hands-on-title></hands-on-title>
> ```sh
> # Set remote tag for your container. This should include your username and
> # repository name for Docker Hub.
> REMOTE=<DOCKERHUB_USERNAME>/my-first-gxit
>
> # Tag your image
> docker tag $IMAGE_TAG:latest $REMOTE:latest
>
> # Authenticate your DockerHub account
> docker login  # >>> Enter username and token for your account
>
> # Push the image
> docker push $REMOTE:latest
> ```
>
> > <tip-title>Production container hosting</tip-title>
> > For production deployment, the
> > [Galaxy standard](https://docs.galaxyproject.org/en/latest/admin/special_topics/mulled_containers.html)
> > for container image hosting is
> > [Biocontainers](https://biocontainers.pro).
> > This requires you to
> > [make a pull request](https://biocontainers-edu.readthedocs.io/en/latest/contributing.html)
> > against the Biocontainers GitHub repository, so this should only be done when an
> > image is considered production-ready. You can also push your image to a
> > repository on
> > [hub.docker.com](https://hub.docker.com) or
> > [quay.io](https://quay.io)
> > but please ensure that it links to a public code repository
> > (e.g. GitHub) to enable maintenance of the image by the Galaxy community!
> {: .tip}
{: .hands_on}

You should now have a container in the cloud, ready for action.
Check out your repo on Docker Hub and you should find the container image there.
Awesome!

Now we just need to write a tool XML that will enable Galaxy to pull and run
our new Docker container as a Galaxy tool.

## The tool XML

> <hands-on-title></hands-on-title>
>
> Create a Galaxy tool XML file named `interactivetool_tabulator.xml`. The file is similar to a regular tool XML, but calls on our remote Docker image as a dependency. The tags that we are most concerned with are:
> - A `<container>` (under the `<requirements>` tag)
> - A `<port>` which matches our container
> - An `<input>` file
> - The `<command>` section
>
> > <comment-title>Writing the tool command</comment-title>
> >
> > This step can cause a lot of confusion. Here are a few pointer that you will find critical to understanding the process:
> > - The `<command>` will be templated by Galaxy
> > - The templated command will run *inside* the Docker container
> >
> {: .comment}
>
> > <tip-title>Writing the GxIT tool XML</tip-title>
> >
> > * Refer to the [Galaxy tool XML docs](https://docs.galaxyproject.org/en/latest/dev/schema.html).
> > * You can take inspiration from [Askomics](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_askomics.xml), [QGIS](https://github.com/usegalaxy-eu/galaxy/blob/release_24.1_europe/tools/interactive/interactivetool_qgis3_34.xml), [ODV](https://github.com/usegalaxy-eu/galaxy/blob/release_24.1_europe/tools/interactive/interactivetool_odv.xml), and other [existing Interactive Tools](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive).
> > * Check XML syntax with [xmlvalidation.com](https://www.xmlvalidation.com/) or [w3schools XML validator](https://www.w3schools.com/xml/xml_validator.asp), or use a linter in your code editor.
> > * [planemo lint](https://planemo.readthedocs.io/en/latest/commands/lint.html) can also be used for XML linting. But be aware that `planemo test` won't work.
> > * When it comes to testing and debugging your tool XML, it can be easier to update the XML file directly on your Galaxy server between tests.
> {: .tip}
>
> > <solution-title></solution-title>
> >
> > {% raw  %}
> >
> > ```xml
> > <tool id="interactive_tool_tabulator" tool_type="interactive" name="Tabulator" version="0.1">
> >     <description>Tuto tool for Gxit</description>
> >
> >     <requirements>
> >         <container type="docker">ancelete/geoc-gxit:latest</container>
> >     </requirements>
> >
> >     <entry_points>
> >         <entry_point name="first gxit" requires_domain="True">
> >             <port>8765</port>
> >
> >             <!--
> >                  Some apps have a non-root entrypoint.
> >                  We can provide the URL with a <url> tag like this:
> >                  <url>/my/entrypoint</url>
> >              -->
> >             <url>/</url>
> >
> >         </entry_point>
> >     </entry_points>
> >
> >     <environment_variables>
> >         <!-- These will be accessible as environment variables inside the Docker container -->
> >     </environment_variables>
> >
> >     <command><![CDATA[
> >
> >         ## The command will be templated by Cheetah within Galaxy, and
> >         ## then run inside the Docker container!
> >
> >         ## This only works because Galaxy's user data directory is mapped
> >         ## onto the Docker container at runtime - enabling access to
> >         ## '$infile' and '$outfile' from inside the container.
> >
> >         shiny-server 2>&1 > /var/log/tuto-gxit-01.log
> >         ## The log file can be found inside the container, for debugging purposes
> >
> >     ]]>
> >     </command>
> >
> >     <inputs>
> >     </inputs>
> >
> >     <outputs>
> >         <!--
> >             Even if our IT doesn't export to Galaxy history,
> >             adding an output ensures to keep track of the IT
> >             execution in the history
> >         -->
> >
> >         <data name="file_output" format="txt"/>
> >     </outputs>
> >
> >     <tests>
> >         <!-- Tests are difficult with GxITs! -->
> >     </tests>
> >
> >     <help> <![CDATA[
> >
> >         Some help is always of interest ;)
> >
> >     ]]></help>
> >     <citations>
> >        <citation type="bibtex">
> >        @misc{
> >             author       = {Lain Pavot - lain.pavot@inrae.fr},
> >             title        = {first-gxit -  A tool to visualise tsv/csv files},
> >             publisher    = {INRAE},
> >             url          = {}
> >         }
> >         </citation>
> >     </citations>
> > </tool>
> > ```
> > {% endraw  %}
> {: .solution}
>
> Don't forget to change the image path (see the `$REMOTE` variable above) and the citation to fit your project settings.
{: .hands_on}

# Additional components

The GxIT that we wrapped in this tutorial was a simple example, and you should now understand what is required to create an Interactive Tool for Galaxy. However, there are a few additional components that can enhance the reliability and user experience of the tool. In addition, more complex applications may require some additional components or workarounds the create the desired experience for the user.

## Run script

In the case of our `Tabulator` application, the run script is simply the R script that renders our Shiny App. It is quite straightforward to call this from our Galaxy tool XML.
However, some web apps might require more elaborate commands to be run. In this situation, there are several solutions demonstrated in the `<command>` section of [existing GxITs](https://github.com/galaxyproject/galaxy/tree/dev/tools/interactive):
- [Guacamole Desktop](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_guacamole_desktop.xml): application startup with `startup.sh`
- [HiCBrowser](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_hicbrowser.xml): application startup with `supervisord`
- [AskOmics](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_askomics.xml): configuration with Python and Bash scripts, followed by `start_all.sh` to run the application.

## Templated config files

Using the `<configfiles>` section in the tool XML, we can enable complex user configuration for the application by templating a run script or configuration file to be read by the application. In this application, for example, we could use a `<configfiles>` section to template user input into the `app.R` script that runs the application within the Docker container. This could enable the user to customize the layout of the app before launch.

## Reserved environment variables

There are a few environment variables
that are accessible in the command section of the tool XML - these can be handy when writing your tool script.
[Check the docs](https://docs.galaxyproject.org/en/latest/dev/schema.html#reserved-variables) for a full reference on the tool XML.

```sh
$__tool_directory__
$__root_dir__
$__user_id__
$__user_email__
```

It can also be useful to create and inject environment variables into the tool context. This can be achieved using the `<environment variables>` tag in the tool XML. The [RStudio GxIT](https://github.com/galaxyproject/galaxy/blob/b180b7909dc3fe2750fbc8b90214e201eb276794/tools/interactive/interactivetool_rstudio.xml#L12) again provides an example of this:

```xml
<environment_variables>
    <environment_variable name="HISTORY_ID" strip="True">${__app__.security.encode_id($jupyter_notebook.history_id)}</environment_variable>
    <environment_variable name="REMOTE_HOST">${__app__.config.galaxy_infrastructure_url}</environment_variable>
    <environment_variable name="GALAXY_WEB_PORT">8080</environment_variable>
    <environment_variable name="GALAXY_URL">$__galaxy_url__</environment_variable>
    <environment_variable name="DEBUG">true</environment_variable>
    <environment_variable name="DISABLE_AUTH">true</environment_variable>
    <environment_variable name="API_KEY" inject="api_key" />
</environment_variables>
```

## Galaxy history interaction

We have demonstrated how to pass an input file to the Docker container. But what if the application needs to interact with the user's Galaxy history? For example, if the user creates a file within the application. That's where the environment variables created in the tool XML become useful.

> <tip-title>Access histories in R</tip-title>
> From the [R-Studio GxIT](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive/interactivetool_rstudio.xml) we can see that there is [an R library](https://github.com/hexylena/rGalaxyConnector) that allows us to interact with Galaxy histories.
>
> "The convenience functions `gx_put()` and `gx_get()` are available to you to interact with your current Galaxy history. You can save your workspace with `gx_save()`"
>
> Under the hood, this library uses [galaxy_ie_helpers](https://github.com/bgruening/galaxy_ie_helpers) - a Python interface to Galaxy histories written with [BioBlend](https://github.com/galaxyproject/bioblend). You could also use BioBlend directly (or even the Galaxy REST API) if your GxIT requires a more flexible interface than these wrappers provide.
>
{: .tip}


</div>

<div class="JupyterLab" markdown="1">

# The JupyterLab

The JupyterLab environment that we will wrap in this tutorial is a simple JupyterLab tool which
allows the user to upload data in a Jupyter environment, manipulate notebooks, and download
outputs. 

Our example JupyterLab can already be found [online](https://github.com/usegalaxy-eu/docker-copernicus-notebooks) .
But you can also check another implementation of [JupyterLabs](https://github.com/bgruening/docker-jupyter-notebook/blob/master/Dockerfile).
In the following sections, we will study how it can be built into a GxIT.


> <hands-on-title></hands-on-title>
>
> First, let's clone the repository to take a quick look at a basic implementation of a JupyterLab.
>
> ```console
> $ git clone https://github.com/bgruening/docker-jupyter-notebook.git
> $ cd docker-jupyter-notebook
>
> $ tree .
> ├── Dockerfile
> ├── LICENSE
> ├── README.md
> ├── .
> ├── .
> ├── .
> └── startup.sh
> ```
> You'll find a Dockerfile, startup.sh, and a README that will describe how the Dockerfile here for a GxIT works. We encourage you to read this README.
>
{: .hands_on}

## The Dockerfile

If you need some help to start your Dockerfile you can always get some inspiration from the previous interactive tools built in Galaxy. Go check some of the Dockerfiles, for instance, the one for the [Copernicus data space ecosystem JupyterLab](https://github.com/usegalaxy-eu/docker-copernicus-notebooks/blob/main/Dockerfile) or for a basic [JupyterLab here](https://github.com/bgruening/docker-jupyter-notebook/blob/master/Dockerfile).

> <tip-title>A brief primer on Docker</tip-title>
> Docker allows an entire application context to be containerized. A typical web application consists of an operating system, installed dependancies, web server configuration, database configuration and, of course, the codebase of the software itself. A Docker container can encapsulate all of these components in a single "image", which can be run on any machine with Docker installed.
>
> **Essentials of Docker:**
>
> 1. Write an image recipe as a Dockerfile. This single file selects an OS, installs software, pulls code repositories and copies files from the host machine (your computer).
> 2. Build the image from your recipe:
>
>    `docker build -t <image_name> .`
> 3. View existing images with
>
>    `docker image list`
> 4. Run a container with a specified command:
>
>    `docker run <image_name> <command>`
> 5. View running containers:
>
>    `docker ps`
> 6. Stop a running container:
>
>    `docker stop <container_name>`
> 7. Remove a stopped container:
>
>    `docker container rm <container_name>`
> 8. Remove an image:
>
>    `docker image rm <container_name>`
{: .tip}


Let's check out [the Dockerfile](https://github.com/usegalaxy-eu/docker-copernicus-notebooks/blob/main/Dockerfile) that we'll use to containerize our JupyterLab.

This container recipe can be used to build a Docker image which can be pushed to a
container registry in the cloud, ready for consumption by our Galaxy instance:

```dockerfile
# Jupyter container used for Galaxy copernicus notebooks (+other kernels) Integration

# from 5th March 2021
FROM jupyter/datascience-notebook:python-3.10

MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com

ENV DEBIAN_FRONTEND noninteractive
USER root

RUN apt-get -qq update && \
    apt-get install -y wget unzip net-tools procps && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Set channels to (defaults) > bioconda > conda-forge
RUN conda config --add channels conda-forge && \
    conda config --add channels bioconda
    #conda config --add channels defaults
RUN pip install --upgrade pip
RUN pip install --no-cache-dir bioblend galaxy-ie-helpers

ENV JUPYTER /opt/conda/bin/jupyter
ENV PYTHON /opt/conda/bin/python
ENV LD_LIBRARY_PATH /opt/conda/lib/

# Python packages
RUN conda config --add channels conda-forge && \
    conda config --add channels bioconda && \
    conda install --yes --quiet \
    bash_kernel \
    ansible-kernel \
    bioblend galaxy-ie-helpers \
```
With those packages installed, you have the bases to have a functional environment in your JupyterLab AND be able to link your JupyterLab to the Galaxy history with the package `bioblend galaxy-ie-helpers`.

Then you can add the package specific to the environment you want to have in your JupyterLab for instance 

```
    # specific sentinel, openeo packages
    sentinelhub \
    openeo \
    # other packages for notebooks
    geopandas \
    rasterio \
    ipyleaflet \
    netcdf4 \
    h5netcdf \
    # Jupyter widgets
    jupytext && \
    conda clean -yt && \
    pip install jupyterlab_hdf \
    fusets

```

Then, you can add some configurations to have some configuration and a welcome notebook by default (this is generic to all JupyterLab GxITs).

```
ADD ./startup.sh /startup.sh
ADD ./get_notebook.py /get_notebook.py

# We can get away with just creating this single file and Jupyter will create the rest of the
# profile for us.
RUN mkdir -p /home/$NB_USER/.ipython/profile_default/startup/ && \
    mkdir -p /home/$NB_USER/.jupyter/custom/

COPY ./ipython-profile.py /home/$NB_USER/.ipython/profile_default/startup/00-load.py
COPY jupyter_notebook_config.py /home/$NB_USER/.jupyter/
COPY jupyter_lab_config.py /home/$NB_USER/.jupyter/

ADD ./custom.js /home/$NB_USER/.jupyter/custom/custom.js
ADD ./custom.css /home/$NB_USER/.jupyter/custom/custom.css
ADD ./default_notebook.ipynb /home/$NB_USER/notebook.ipynb

```

You can also add your own set of notebooks to guide the user like that:

```
# Download notebooks
RUN cd /home/$NB_USER/ &&  \
    wget -O notebook-samples.zip https://github.com/eu-cdse/notebook-samples/archive/refs/heads/main.zip && \
    unzip notebook-samples.zip && \
    rm /home/$NB_USER/notebook-samples.zip && \
    mv /home/$NB_USER/notebook-samples-main/geo /home/$NB_USER && \
    mv /home/$NB_USER/notebook-samples-main/sentinelhub /home/$NB_USER && \
    mv /home/$NB_USER/notebook-samples-main/openeo /home/$NB_USER && \
    rm -r /home/$NB_USER/notebook-samples-main
```

Finally, some general variables of environment:

```
# ENV variables to replace conf file
ENV DEBUG=false \
    GALAXY_WEB_PORT=10000 \
    NOTEBOOK_PASSWORD=none \
    CORS_ORIGIN=none \
    DOCKER_PORT=none \
    API_KEY=none \
    HISTORY_ID=none \
    REMOTE_HOST=none \
    GALAXY_URL=none

# @jupyterlab/google-drive  not yet supported

USER root
WORKDIR /import

# Start Jupyter Notebook
CMD /startup.sh

```

And with all this your Dockerfile is ready.

You now need to publish this image on a public repository.
Anyone can use this Dockerfile to rebuild the image if necessary.
In any case don't forget to have all the files shown when we cloned this [repository](https://github.com/bgruening/docker-jupyter-notebook/tree/master) next to your Dockerfile.

> <hands-on-title></hands-on-title>
>
> Let's start working on this Docker container.
>
> 1. Install Docker as described on the [docker website](https://docs.docker.com/engine/install/). Click on your distribution name to get specific information.
>
> 2. Now let's use the recipe to build our Docker image.
>
>    ```sh
>    # Build a container image from our Dockerfile
>    IMAGE_TAG="myimage"
>    LOG_PATH=`pwd`  # Create log output in current directory
>    PORT=8765
>    docker build -t $IMAGE_TAG --build-arg LOG_PATH=$LOG_PATH --build-arg PORT=$PORT .
>    ```
>
>    > <tip-title>Automating the build</tip-title>
>    > While developing the Docker container you may find yourself tweaking and rebuilding the container image many times.
>    > In the GitHub repository linked above, you'll notice that the author has used a `Makefile` to accelerate the build and deploy process.
>    > This allows the developer to simply run `make docker` and `make push_hub` to build and push the container, or `make` to rebuild the container after making changes during development. Check out the `Makefile` to see what commands can be run using `make` in this repository.
>    >
>    {: .tip}
{: .hands_on}

If you are lucky, you might find an available Docker image for the application you are trying to wrap. Some configuration changes can be needed such as:


1. Expose the correct port. The application, Docker and tool XML ports must be aligned!
2. Log output to an external file - useful for debugging.
3. Make the application callable from tool `<command>` - this sometimes requires a wrapper script to interface the application inside the container (we'll take a look at this later).



## Test the image

Before we push our container to the cloud, we should give it a local test run to ensure that it's working correctly on our development machine. Have a play and see how our little web app `works!`

> <hands-on-title></hands-on-title>
> ```sh
> # Run our application in the container
> docker run -it -p 127.0.0.1:8765:$PORT $IMAGE_TAG
>
> # Or to save time, take advantage of the Makefile
> make it
>
> # Give it a few moments to start up, and the application should be available
> # in your browser at http://127.0.0.1:8765
> ```
{: .hands_on}

## Push the image

If you are happy with the image, we are ready to push it to a container registry
to make it accessible to our Galaxy server.

During development, we suggest making an account on
[Docker Hub](https://hub.docker.com/)
if you don't have one already. This can be used for hosting container images
during development.
[Docker Hub](https://hub.docker.com/)
has great documentation on creating repositories, authenticating with tokens
and pushing images.

> <hands-on-title></hands-on-title>
> ```sh
> # Set remote tag for your container. This should include your username and
> # repository name for Docker Hub.
> REMOTE=<DOCKERHUB_USERNAME>/my-first-gxit
>
> # Tag your image
> docker tag $IMAGE_TAG:latest $REMOTE:latest
>
> # Authenticate your DockerHub account
> docker login  # >>> Enter username and token for your account
>
> # Push the image
> docker push $REMOTE:latest
> ```
>
> > <tip-title>Production container hosting</tip-title>
> > For production deployment, the
> > [Galaxy standard](https://docs.galaxyproject.org/en/latest/admin/special_topics/mulled_containers.html)
> > for container image hosting is
> > [Biocontainers](https://biocontainers.pro).
> > This requires you to
> > [make a pull request](https://biocontainers-edu.readthedocs.io/en/latest/contributing.html)
> > against the Biocontainers GitHub repository, so this should only be done when an
> > image is considered production-ready. You can also push your image to a
> > repository on
> > [hub.docker.com](https://hub.docker.com) or
> > [quay.io](https://quay.io)
> > but please ensure that it links to a public code repository
> > (e.g. GitHub) to enable maintenance of the image by the Galaxy community!
> {: .tip}
{: .hands_on}

You should now have a container in the cloud, ready for action.
Check out your repo on Docker Hub and you should find the container image there.
Awesome!

Now we just need to write a tool XML that will enable Galaxy to pull and run
our new Docker container as a Galaxy tool.

## The tool XML

> <hands-on-title></hands-on-title>
>
> Create a Galaxy tool XML file named `interactivetool_copernicus.xml`. The file is similar to a regular tool XML, but calls on our remote Docker image as a dependency. The tags that we are most concerned with are:
> - A `<container>` (under the `<requirements>` tag)
> - A `<port>` which matches our container
> - An `<input>` file
> - The `<command>` section
>
> > <comment-title>Writing the tool command</comment-title>
> >
> > This step can cause a lot of confusion. Here are a few pointer that you will find critical to understanding the process:
> > - The `<command>` will be templated by Galaxy
> > - The templated command will run *inside* the Docker container
> >
> {: .comment}
>
> > <tip-title>Writing the GxIT tool XML</tip-title>
> >
> > * Refer to the [Galaxy tool XML docs](https://docs.galaxyproject.org/en/latest/dev/schema.html).
> > * You can take inspiration from [Copernicus](https://github.com/usegalaxy-eu/galaxy/blob/release_24.1_europe/tools/interactive/interactivetool_copernicus.xml) or for a more advanced [jupyterlab](https://github.com/usegalaxy-eu/galaxy/blob/release_24.1_europe/tools/interactive/interactivetool_jupyter_notebook_1.0.1.xml) [existing Interactive Tools](https://github.com/galaxyproject/galaxy/blob/dev/tools/interactive).
> > * Check XML syntax with [xmlvalidation.com](https://www.xmlvalidation.com/) or [w3schools XML validator](https://www.w3schools.com/xml/xml_validator.asp), or use a linter in your code editor.
> > * [planemo lint](https://planemo.readthedocs.io/en/latest/commands/lint.html) can also be used for XML linting. But be aware that `planemo test` won't work.
> > * When it comes to testing and debugging your tool XML, it can be easier to update the XML file directly on your Galaxy server between tests.
> {: .tip}
>
> > <solution-title></solution-title>
> >
> > {% raw  %}
> >
> > ```xml
> > <tool id="interactive_tool_copernicus_notebook" tool_type="interactive" name="Copernicus Data Space Ecosystem" version="@VERSION@">
> >     <description>sample notebooks to access and discover data</description>
> >     <macros>
> >         <token name="@VERSION@">0.0.1</token>
> >     </macros>
> >     <requirements>
> >         <container type="docker">quay.io/galaxy/copernicus-jupyterlab:@VERSION@</container>
> >     </requirements>
> >     <entry_points>
> >         <entry_point name="Jupyter Interactive Tool" requires_domain="True">
> >             <port>8888</port>
> >             <url>ipython/lab</url>
> >         </entry_point>
> >     </entry_points>
> >     <environment_variables>
> >         <environment_variable name="HISTORY_ID">$__history_id__</environment_variable>
> >         <environment_variable name="REMOTE_HOST">$__galaxy_url__</environment_variable>
> >         <environment_variable name="GALAXY_WEB_PORT">8080</environment_variable>
> >         <environment_variable name="GALAXY_URL">$__galaxy_url__</environment_variable>
> >         <  environment_variable name="API_KEY" inject="api_key" />
> >     </environment_variables>
> >     <command detect_errors="aggressive"><![CDATA[
> >     #import re
> >     export GALAXY_WORKING_DIR=`pwd` &&
> >     mkdir -p ./jupyter/outputs/ &&
> >     mkdir -p ./jupyter/data &&
> >     mkdir -p ./jupyter/notebooks &&
> >     mkdir -p ./jupyter/notebooks/geo &&
> >     mkdir -p ./jupyter/notebooks/openeo &&
> >     mkdir -p ./jupyter/notebooks/sentinelhub &&
> >
> >     #for $count, $file in enumerate($input):
> >         #set $cleaned_name = str($count + 1) + '_' + re.sub('[^\w\-\.\s]', '_', str($file.element_identifier))
> >         ln -sf '$file' './jupyter/data/${cleaned_name}' &&
> >     #end for
> >
> >     ## change into the directory where the notebooks are located
> >     cd ./jupyter/ &&
> >     export HOME=/home/jovyan/ &&
> >     export PATH=/home/jovyan/.local/bin:\$PATH &&
> >
> >     #if $mode.mode_select == 'scratch'
> >         ## copy all notebooks, workflows and data
> >         cp '$__tool_directory__/default_notebook.ipynb' ./ipython_galaxy_notebook.ipynb &&
> >         jupyter trust ./ipython_galaxy_notebook.ipynb &&
> >         cp -r /home/\$NB_USER/geo/* ./notebooks/geo/ &&
> >         cp -r /home/\$NB_USER/openeo/* ./notebooks/openeo/ &&
> >         cp -r /home/\$NB_USER/sentinelhub/* ./notebooks/sentinelhub/ &&
> >
> >
> >         ## provide all rights to copied files
> >         jupyter lab --allow-root --no-browser &&
> >         cp ./*.ipynb '$jupyter_notebook' &&
> >         
> >         cd outputs/ &&
> >         sleep 2 &&
> >         for file in *; do mv "\$file" "\$file.\${file\#\#*.}"; done
> >     #else
> >         #set $noteboook_name = re.sub('[^\w\-\.\s]', '_', str($mode.ipynb.element_identifier))
> >         cp '$mode.ipynb' './${noteboook_name}.ipynb' &&
> >         jupyter trust './${noteboook_name}.ipynb' &&
> >         #if $mode.run_it
> >             jupyter nbconvert --to notebook --execute --output ./ipython_galaxy_notebook.ipynb --allow-errors  ./*.ipynb &&
> >             #set $noteboook_name = 'ipython_galaxy_notebook'
> >         #else
> >             jupyter lab --allow-root --no-browser --NotebookApp.shutdown_button=True &&
> >         #end if
> >         cp './${noteboook_name}.ipynb' '$jupyter_notebook' &&
> >         
> >         cd outputs/ &&
> >         sleep 2 &&
> >         for file in *; do mv "\$file" "\$file.\${file\#\#*.}"; done
> >     #end if
> > ]]>
> >     </command>
> >     <inputs>
> >
> >         <conditional name="mode">
> >             <param name="mode_select" type="select" label="Do you already have a notebook?" help="If not, no problem we will provide you with a default one.">
> >                 <option value="scratch">Start with a fresh notebook</option>
> >                 <option value="previous">Load a previous notebook</option>
> >             </param>
> >             <when value="scratch"/>
> >             <when value="previous">
> >                 <param name="ipynb" type="data" format="ipynb" label="Copernicus Notebook"/>
> >                 <param name="run_it" type="boolean" truevalue="true" falsevalue="false" label="Copernicus notebook and return a new one."
> >                 help="This option is useful in workflows when you just want to execute a notebook and not dive into the webfrontend."/>
> >             </when>
> >         </conditional>
> >         <param name="input" multiple="true" type="data" optional="true" label="Include data into the environment"/>
> >     </inputs>
> >     <outputs>
> >         <data name="jupyter_notebook" format="ipynb" label="Executed Copernicus notebook"></data>
> >         <collection name="output_collection" type="list" label="Copernicus outputs collection">
> >             <discover_datasets pattern="__designation_and_ext__" directory="jupyter/outputs" visible="true"/>
> >         </collection>
> >     </outputs>
> >     <tests>
> >         <test expect_num_outputs="1">
> >             <param name="mode" value="previous" />
> >             <param name="ipynb" value="test.ipynb" />
> >             <param name="run_it" value="true" />
> >             <output name="jupyter_notebook" file="test.ipynb" ftype="ipynb"/>
> >         </test>
> >     </tests>
> >     <help>
> > This tool contains sample Jupyter notebooks for the Copernicus Data Space Ecosystem. Notebooks are grouped per kernel: sentinelhub, openeo and geo. To know more https:/github.com/eu-cdse/notebook-samples
> > 
> > To have more example notebooks produced by th OpenEO community that you can use in this jupyterlab go there https://github.com/Open-EO/openeo-community-examples/tree/main
> > 
> >     </help>
> >     <citations>
> >        <citation type="bibtex">
> >         @Manual{,
> >         title = {Copernicus Data Space Ecosystem },
> >         author = {The eu-cdse community},
> >         year = {2023},
> >         note = {https://github.com/eu-cdse}
> >         </citation>
> > 	</citations>    
> > </tool>
> > ```
> > {% endraw  %}
> {: .solution}
>
> In order to get your data from your galaxy history into your jupyterlab don't forget to have this kind of line in the executable
> ```
> #for $count, $file in enumerate($input):
>     #set $cleaned_name = str($count + 1) + '_' + re.sub('[^\w\-\.\s]', '_', str($file.element_identifier))
>     ln -sf '$file' './jupyter/data/${cleaned_name}' &&
> #end for
> ```
> AND to get your output in your history data 
> 
> ```
> <collection name="output_collection" type="list" label="Copernicus outputs collection">
>     <discover_datasets pattern="__designation_and_ext__" directory="jupyter/outputs" visible="true"/>
> </collection>
> ```
> Don't forget to change the image path and the citation to fit your project settings.
{: .hands_on}

</div>

# Testing locally

You would like to check your GxIT integration in Galaxy but don't have a development server or don't want to disturb your sysadmin at this point?
Let's check this integration on your machine. You can use a VM if you prefer not to modify your machine environment.

> <comment-title>A note on the OS</comment-title>
> This part of the tutorial has been tested on Ubuntu and Debian and there is no guaranteed success for other operating systems.
> If you have another OS on your machine (i.e. Windows or MacOS), you may need to use an Ubuntu virtual machine or perhaps try [Windows subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install).
{: .comment}

## Docker installation

> <hands-on-title>Install Docker</hands-on-title>
> Install Docker as described on the [docker website](https://docs.docker.com/engine/install/). Click on your distribution name to get specific information.
{: .hands_on}

## Galaxy installation

> <hands-on-title>Install Galaxy</hands-on-title>
>
> For Ubuntu:
> ```sh
> # Install git to get Galaxy project
> sudo apt-get install git
> # Create a working directory and move to it
> mkdir ~/GxIT && cd ~/GxIT
> # Get the galaxy project. A new directory named "galaxy" will be created.
> # This directory contains the whole project
> git clone https://github.com/galaxyproject/galaxy
> # Checkout the last stable version (v23.1 as the time of writing)
> cd galaxy && git checkout release_23.1
> ```
{: .hands_on}


## Galaxy configuration

> <hands-on-title></hands-on-title>
>
> ```sh
> cd ~/GxIT/galaxy/config
> # Create custom config files
> cat galaxy.yml.interactivetools > galaxy.yml
> cat tool_conf.xml.sample > tool_conf.xml
> ```
> In `galaxy.yml`, ensure that the `galaxy_infrastructure_url` parameter is present under the galaxy section:
> ```yaml
> galaxy:
>   galaxy_infrastructure_url: http://localhost:8080
> ```
> This will make galaxy to provide your GxIT using links like <http://your_gxit_identifier.localhost:8080>.
>
> Configure the tool panel by adding a section in `~/GxIT/galaxy/config/tool_conf.xml`:
> ```xml
>   <section id="interactivetools" name="Interactive tools">
>     <tool file="interactive/interactivetool_tabulator.xml" />
>     <tool file="interactive/interactivetool_copernicus.xml" />
>   </section>
> ```
> With these lines, Galaxy will create a new section named "Interactive tools" in the tool panel
> with our interactive tabulator inside.
> Choose whatever name and id you want as long as the id is unique.
> And of course, you have no obligation to put your GxITs in this section.
> You can put them in any section.
>
> Next, create a simple `job_conf.xml` with the following contents, which basically specifies how Galaxy should execute a job. It is also possible to execute the tools directly from the local environment, but here we would like the tool to make use of the container we have just made.
> ```xml
> <?xml version="1.0"?>
> <job_conf>
>     <plugins>
>         <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
>     </plugins>
>     <destinations default="docker_dispatch">
>         <destination id="local" runner="local"/>
>         <destination id="docker_local" runner="local">
>             <param id="docker_enabled">true</param>
>             <param id="docker_volumes">$defaults</param>
>             <param id="docker_sudo">false</param>
>             <param id="docker_net">bridge</param>
>             <param id="docker_auto_rm">true</param>
>             <param id="require_container">true</param>
>             <param id="container_monitor">true</param>
>             <param id="docker_set_user"></param>
>             <param id="docker_run_extra_arguments">--add-host localhost:host-gateway</param>
>         </destination>
>         <destination id="docker_dispatch" runner="dynamic">
>             <param id="type">docker_dispatch</param>
>             <param id="docker_destination_id">docker_local</param>
>             <param id="default_destination_id">local</param>
>         </destination>
>     </destinations>
> </job_conf>
> ```
> Finally, copy your GxIT wrapper to the Interactive Tool directory (dependin


topic: dev
tutorial: tool-from-scratch
title: Creating Galaxy tools from Conda Through Deployment
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 898d5a9cc09f15fe18bf5038cad3e33f
last_modified: 2026-05-15T22:28:11.276574
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/tool-from-scratch/tutorial.html

content:
Tools wrappers allow any command line runnable code or programs to be run inside a galaxy environment.
Although Galaxy has thousands of tools readily available, new software and packages will always be useful.
This tutorial is designed to allow anyone to create, run, and deploy new tools in a Galaxy environment.

This tutorial is intended to instruct the reader in the full process of integrating a tool into Galaxy.
In three parts, it will cover the creation of a bioconda recipe for a new tool, writing a Galaxy tool
wrapper, and finally the testing and deployment of this tool into both a local and public Galaxy environment.
Although this will include specific tools, the training is generalizable.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy Tools and Conda

Galaxy tools run within a conda environment, and are able to run any tool capable of running inside
that type of environment. The wrapper includes a set of conda packages available from either bioconda
or conda forge, and retrieves them on tool installation. Sometimes this may be a single package, as
with many command line tools, other times, there may be several. The first part of this tutorial discusses
the creation and deployment of these packages, making them available for Galaxy to retrieve.

## Writing a Bioconda Recipe

Bioconda is a repository of conda packages for software and tools relevant to the life sciences. Using conda
packages ensures better reproducibility, since each conda package is usually immutable, an exception being
if it is later discovered that the software in the package had always been non-functional and should be replaced.
Recipes should always define the following 6 sections in the `meta.yaml` file:

- package
- source
- build
- requirements
- test
- about

Let's write a Bioconda recipe for the tool we want to package: [bellerophon](https://github.com/davebx/bellerophon)

> <warning-title>Naming collision</warning-title>
> As this tool is already packaged in Bioconda, to prevent any naming collision, we will slightly modify its name from `bellerophon` to `bellerophon_bis` in the recipe.
> **This should never be done in real life, we only do it for this exercise!**
{: .warning}

> <hands-on-title>Writing a Bioconda Recipe</hands-on-title>
>
> 1. The first thing we should do is prepare our workspace. [Create a fork](https://help.github.com/articles/fork-a-repo/) of the [Bioconda repository](https://github.com/bioconda/bioconda-recipes/) on GitHub
>
> 2. Clone your fork of this repository to create a local copy on your computer
>
> 3. We create a branch in git, a conda environment with pip and
> conda-build preinstalled, and a folder for the meta.yaml file.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git checkout -b bellerophon_bioconda
>    > conda create -y --name bellerophon_bioconda pip python conda-build
>    > conda activate bellerophon_bioconda
>    > mkdir recipes/bellerophon_bis
>    > ```
>    {: .code-in}
>
> 4. Next we determine the SHA-256 checksum of the source tarball.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > wget -O bellerophon.tar.gz https://github.com/davebx/bellerophon/archive/1.0.tar.gz
> > sha256sum bellerophon.tar.gz # Copy the 64-character hexadecimal number that this outputs.
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > 036c5e23f53ed5b612525d3a32095acca073a9c8d2bf73883deb852c89f40dcf  bellerophon.tar.gz
>    > ```
>    {: .code-out}
>
> 5. Using the above information, we create the meta.yaml file where we'll then define the
> parameters that tell conda-build how to build this package, starting with variables for the name, version, and checksum.
> With these definitions, Bioconda's automatic version updater should recognize when a new version has been released and
> create a pull request to update the Bioconda package.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > vim recipes/bellerophon_bis/meta.yaml # vim can of course be replaced with any other editor.
>    > ```
>    {: .code-in}
>
>    and set the following content:
>
>    {% raw %}
>    ```yaml
>    {% set name = "bellerophon_bis" %}
>    {% set version = "1.0" %}
>    {% set sha256 = "036c5e23f53ed5b612525d3a32095acca073a9c8d2bf73883deb852c89f40dcf" %}
>    ```
>    {% endraw %}
>
> 6. Now we define the conda package metadata. This will be shown as bellerophon_bis-1.0 in anaconda and `conda search`. We
> plug in the relevant variables from the top of the file, lowering the name since conda package names should always be
> lowercase.
>
>    {% raw %}
>    ```yaml
>    package:
>      name: {{ name|lower }}
>      version: {{ version }}
>    ```
>    {% endraw %}
>
> 7. Of course conda-build needs to know where to get the source code for bellerophon. Since the recipe we are creating is on
> github, updates can be automated with the variables we defined in the second step, while the SHA-256 checksum ensures
> that conda-build is getting the right source code every time.
>
>    {% raw %}
>
>    ```yaml
>    source:
>      url: https://github.com/davebx/bellerophon/archive/{{ version }}.tar.gz
>      sha256: {{ sha256 }}
>    ```
>
>    {% endraw %}
>
> 8. Next, we move on to the build metadata. Since this is the first version of the conda recipe, the build number is 0.
> We use the externally defined `{% raw %}{{ PYTHON }}{% endraw %}` variable, which defines which python conda-build is using, to install it
> to the build prefix. The --no-deps and --ignore-installed flags are needed to ensure that conda-build only packages
> bellerophon itself. In this section, if necessary, we can also define patches that should be applied to the source code,
> with the `patches:` token under `build:`, and specify that the package should not be built for a given architecture with
> the `skip:` directive.
>
>    ```yaml
>    build:
>      noarch: python
>      number: 0
>      script: {% raw %}{{ PYTHON }}{% endraw %} -m pip install . --no-deps --ignore-installed -vv
>      #patches: /dev/null # Not used in this tutorial
>      #skip: True [osx] # Not used in this tutorial
>    ```
>
> 9. After the build metadata has been defined, we need to specify dependencies for at least building and running, with
> build-time dependencies specified in the `host:` section, and runtime dependencies in the `run:` section. With
> bellerophon, we know that it's a python package that uses pysam to operate on SAM/BAM files, so the only runtime
> dependencies we need are python and pysam.
>
>    ```yaml
>    requirements:
>      host:
>        - python
>        - pip
>      run:
>        - python
>        - pysam
>    ```
>
> 9. No recipe is complete without tests, and this recipe is no exception. Normally, it's sufficient to confirm that the
> program actually runs, e.g. with a `--version` command. Some software also has a self-test flag or parameter, though
> bellerophon is not among them, and we could even define a test script that uses test data either from the recipe or from
> the source archive.
>
>    ```yaml
>    test:
>      commands:
>        - bellerophon --version
>    ```
>
> 9. Finally, we add information about the software, such as code's is license type, the program's
> homepage, and optionally the github username of the person responsible for maintaining the recipe.
>
>    ```yaml
>    about:
>      home: https://github.com/davebx/bellerophon/
>      license: MIT
>      license_file: LICENSE
>      summary: "Filter reads that span a mapping junction, retaining the 5'-side."
>      maintainer: davebx # Optional
>    ```
>
> Putting all these parts together, we end up with a complete conda recipe for version 1.0 of bellerophon.
>
> > <question-title></question-title>
> >
> > What does your final file look like?
> >
> > > <solution-title></solution-title>
> > >
> > > {% raw %}
> > >
> > > ```yaml
> > > {% set name = "bellerophon_bis" %}
> > > {% set version = "1.0" %}
> > > {% set sha256 = "036c5e23f53ed5b612525d3a32095acca073a9c8d2bf73883deb852c89f40dcf" %}
> > >
> > > package:
> > >   name: {{ name|lower }}
> > >   version: {{ version }}
> > >
> > > source:
> > >   url: https://github.com/davebx/bellerophon/archive/{{ version }}.tar.gz
> > >   sha256: {{ sha256 }}
> > >
> > > build:
> > >   noarch: python
> > >   number: 0
> > >   script: {{ PYTHON }} -m pip install . --no-deps --ignore-installed -vv
> > >
> > > requirements:
> > >   host:
> > >     - python
> > >     - pip
> > >   run:
> > >     - python
> > >     - pysam
> > >
> > > test:
> > >   commands:
> > >     - bellerophon --version
> > >
> > > about:
> > >   home: https://github.com/davebx/bellerophon/
> > >   license: MIT
> > >   license_file: LICENSE
> > >   summary: "Filter reads that span a mapping junction, retaining the 5'-side."
> > >
> > > ```
> > >
> > > {% endraw %}
> > {: .solution}
> {: .question}
{: .hands_on}

> <comment-title>Using build.sh</comment-title>
>
> In this recipe, the command to execute for installing the package is very short, that's why we write it directly in the `meta.yaml` file. When you need to run more complex commands (like compilation steps), it is preferable to remove the `build > script` entry in the `meta.yaml` file, and write all the commands in a script named `build.sh` in the same directory.
>
{: .comment}

## Building the recipe locally

Now that your recipe is written, you can try to build it locally.

> <hands-on-title>Building the recipe locally</hands-on-title>
>
> 1. Run conda-build
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > conda build recipes/bellerophon_bis
>    > ```
>    {: .code-in}
>
> 2. Conda build will try to build the recipe locally and will run the the test to check if the package was successfully built.
> At the end, you should see something like this:
>
>    > <code-out-title></code-out-title>
>    >
>    > ```
>    > TEST END: /home/abretaud/miniconda3/conda-bld/noarch/bellerophon-1.0-py_0.tar.bz2
>    > Renaming work directory [...]
>    > # Automatic uploading is disabled
>    > # If you want to upload package(s) to anaconda.org later, type:
>    >
>    > # To have conda build upload to anaconda.org automatically, use
>    > # conda config --set anaconda_upload yes
>    > anaconda upload \
>    >    /home/xxxxx/miniconda3/conda-bld/noarch/bellerophon_bis-1.0-py_0.tar.bz2
>    > anaconda_upload is not set.  Not uploading wheels: []
>    > ```
>    {: .code-out}
>
> 3. In this example, the build was successful, and the resulting package is placed in `/home/xxxxx/miniconda3/conda-bld/noarch/bellerophon_bis-1.0-py_0.tar.bz2`. This is the file that gets uploaded to the bioconda channel when you create a Pull Request on the Bioconda GitHub repository.
{: .hands_on}

> <comment-title>Note on building locally</comment-title>
>
> While building locally is a quick way to check if a recipe is working, it's not the most reliable way.
> That's because the build is occurring directly on your system, which means you might get interference between the conda requirements in the recipe and your system-wide installed libraries and system configuration.
> The automatic testing of your recipe on GitHub when you create a Pull Request is more reliable as it is performed in a controlled and isolated environment.
>
{: .comment}

## Creating a Pull Request

After the recipe is complete, and when we have checked that it builds locally, we can commit and push to our fork, so that the recipe can eventually be integrated into Bioconda.

> <hands-on-title>Creating the PR</hands-on-title>
>
> 1. We start by making sure we're on a branch, and that `recipes/bellerophon/meta.yaml` is the only modified file.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git status
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    >
>    > ```
>    > On branch bellerophon_bioconda
>    > Untracked files:
>    >   (use "git add <file>..." to include in what will be committed)
>    >
>    > 	recipes/bellerophon_bis/
>    >
>    > no changes added to commit (use "git add" and/or "git commit -a")
>    > ```
>    {: .code-out}
>
> 2. Add your recipes folder, commit, and push it.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git add recipes/bellerophon_bis
>    > git commit -m 'Add recipe for bellerophon 1.0'
>    > git push origin -u bellerophon_bioconda
>    > ```
>    {: .code-in}
>
>    Git will give you a url for creating the pull request, which can be clicked to be taken to github.
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > $ git push origin -u bellerophon_bioconda
>    > Total 0 (delta 0), reused 0 (delta 0)
>    > remote:
>    > remote: Create a pull request for 'bellerophon_bioconda' on GitHub by visiting:
>    > remote:      https://github.com/davebx/bioconda-recipes/pull/new/bellerophon_bioconda
>    > remote:
>    > To ssh://github.com/davebx/bioconda-recipes.git
>    >  * [new branch]      bellerophon_bioconda -> bellerophon_bioconda
>    > Branch bellerophon_bioconda set up to track remote branch bellerophon_bioconda from origin.
>    > ```
>    {: .code-out}
>
> > <warning-title>Don't create the Pull request for real!</warning-title>
> >
> > You can click on the link to see the diff of what would be included in the Pull Request, and how the form looks like. But please don't click on the `Create pull request` button! We don't want to get duplicate versions of this recipe integrated in bioconda everytime someone follows this tutorial!
> >
> {: .warning}
>
> Once on the github pull request page, you see a preview of the changes we propose, and you can write a short description of the recipe we want to merge. Have a look at the guidelines in the text box to understand how to properly fill the form, and get your Pull Request merged in the end. In real life, when creating a real new recipe, you would then click the `Create pull request` button.
>
> ![Github PR](../../images/create_pr.png "An open pull request for the bellerophon conda recipe")
{: .hands_on}

## Installing a Conda Package

Once our new recipe has been merged, installing the package should be fairly painless. Galaxy, if configured to do so,
will automatically install any conda dependencies a tool asks for, whereas manual installation can be as easy as using
the `conda create` command to install python, bellerophon, and pysam into a conda environment named `bellerophon`. The
`-y` flag tells conda not to ask for confirmation before installing the software and its dependencies into the named
environment.


```bash
conda create -y --name bellerophon bellerophon=1.0 -c conda-forge -c bioconda
```

# Galaxy Tool Wrappers

Galaxy tool wrappers convert the available command line inputs and parameters of a given tool into a grapical
user interface, then translate the user's selections back into the command to be run. This section will focus
on writing a wrapper. Although this section will cover the full process and many of the options available to
anyone wrapping a tool, a more complete list is available in [the Galaxy tool docs](https://docs.galaxyproject.org/en/latest/dev/schema.html).

## Installing Planemo

Planemo is an important tool within the Galaxy development workflow. Although it will be covered in greater detail in the
next section of this tutorial, one of its functions is necessary for the tool wrapper development. `planemo` is available
via the python package manager `pip`. To that end, before beginning this section, please install planemo locally by running

> <hands-on-title>Installing `planemo` via pip</hands-on-title>
>
> It is advised to install `planemo` in a virtual environment (step 1-2).
>
> 1. Create a virtual environment: `virtualenv -p python3 ~/.venv/`. Here `~/.venv/` is the path where the virtual environment should be crated and you may adapt the path to your needs. With `-p python3` we make sure that a `python` intepreter version 3 is used.
> 2. Activate the virtual environment: `. ~/.venv/bin/activate`
> 3. Install `planemo` `pip install planemo`
{: .hands_on}

Note that for using `planemo`from a new shell you will need to activate the python virtual environment again.

> <hands-on-title>Testing `planemo` and getting help</hands-on-title>
>
> 1. In order to test if the installation worked execute
>
>    > <code-in-title>Input: Bash</code-in-title>
>    > ```bash
>    > planemo --version
>    > ```
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > This should output the version of `planemo`, e.g.
>    > ```bash
>    > planemo, version 0.74.3
>    > ```
>    {: .code-in}
>
> 2. `planemo --help` will show the available commands with a short desctiption (lint, test, and serve will be part of this tutorial)
> 3. `planemo SUBCOMMAND --help` will show the usage information for the corresponding subcommand. Try to obtain the information for the `lint` subcommand.
{: .hands_on}

## Initializing a Tool Wrapper

Initializing a tool wrapper to be run in Galaxy is simple

> <hands-on-title>Creating a Tool Wrapper Skeleton</hands-on-title>
> 1. Go to the "tools" directory in your Galaxy instance.
> 2. Make a new directory for tools you will be wrapping.
> 3. Use `planemo` to initialize a new tool wrapper with basic formatting.
>     - Here we will be using bellerophon, but the parameters can be set for any tool.
>     - The tool ID is the internal reference id for the tool, and may contain no spaces or capitalized letters
>     - The tool name is the text that will be visible to the user.
>
>     ```bash
>     cd galaxy/tools
>     mkdir -p myTools/bellerophon
>     cd myTools/bellerophon
>     planemo tool_init --id bellerophon --name bellerophon
>     ```
>
> 4. You can now open the new xml file that has been generated to begin the wrapping. It should look something like this:
>
>    ```xml
>    <tool id="bellerophon" name="bellerophon" version="0.1.0" python_template_version="3.5">
>        <requirements>
>        </requirements>
>        <command detect_errors="exit_code"><![CDATA[
>            TODO: Fill in command template.
>        ]]></command>
>        <inputs>
>        </inputs>
>        <outputs>
>        </outputs>
>        <help><![CDATA[
>            TODO: Fill in help.
>        ]]></help>
>    </tool>
>    ```
>
{: .hands_on}

## Galaxy Tool Wrappers

Galaxy tool wrapper xml files are made up of several sections:

- Tool
- Description
- Requirements
- Macros
- Command
- Inputs
- Outputs
- Tests
- Citations
- Help

While many of those are present in the wrapper by default, others need to be added.

### Tool section

The tool section at the head of the file defines several key pieces of information for both the user and Galaxy.
The tool ID and name are defined here as well as which minimum version of Galaxy is required to run it.

> <hands-on-title>Defining the tool section:</hands-on-title>
> For this section, the following is necessary:
>
> - Tool ID
> - Tool name
> - Version (Version of the Galaxy wrapper)
>   - This is set as `'<Conda wrapper version>'+'galaxy<galaxy wrapper version>'`
> - Profile
>   - This is the minimum version of Galaxy required to run the tool
> - Unset python_template version
>
> when complete, the line should appear as follows:
>
> ```xml
> <tool id="bellerophon" name="bellerophon" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="20.01">
> ```
{: .hands_on}

The @TOOL_VERSION@ and @VERSION_SUFFIX@ are what are referred to as "tokens", and will be furter discussed in the "Macros" section.

### Description section

This section adds help text to easily identify the tool. It is not intended as in-depth help, which should be in the `help` section.
The description is simply presented as plaintext between the tags. Bellerophon's would look like this:

> <hands-on-title>Adding a description</hands-on-title>
> Add the following description to your tool
> ```xml
> <description>chimeric reads from Arima Genomics</description>
> ```
{: .hands_on}

Which would present the tool's in the toolbar and at the top of the tool form as

```txt
bellerophon: chimeric reads from Arima Genomics
```

### Macros section

The macros section serves two functions -- to create meta-variables for inside both the xml text and the command text, called
tokens, and xml sections that can be slotted in repeatedly. In this section we will only use tokens, but xml macros are very
useful for tools which have multiple options/modes that reuse parameters. We will use a simple xml macro in the following section.

The @TOOL_VERSION@ and @VERSION_SUFFIX@ used in the Tool line are defined here. Each need their own 'token' line, resulting in

> <hands-on-title>Macros</hands-on-title>
> Add the following macros section to your tool:
> ```xml
> <macros>
>     <token name="@TOOL_VERSION@">1.0</token>
>     <token name="@VERSION_SUFFIX@">0</token>
> </macros>
> ```
{: .hands_on}

This will cause the version string at the beginning to read "1.0+galaxy0", and set the conda package that is pulled to be bellerophon version 1.0.
As versions are incremented, this will change. Any updates to the galaxy wrapper should iterate `@VERSION_SUFFIX@`, while any changes to the version
of the requirement would change the value for `@TOOL_VERSION@` (and reset the `@VERSION_SUFFIX@` to 0).

A basic example of the xml macros would look like this

```xml
<macros>
    <xml name="example_parameter">
        <param name="ex_1" type="integer" label="Example help text"/>
        <param name="ex_2" type="text" label="Example help text"/>
    </xml>
</macros>
```

One would then use that macro at any point in the xml it is relevant using the `<expand/>` tag like so:

```xml
<expand macro="example_parameter" />
```

This would add both of the `<param/>` tags at that location. Though inputs are used as an example macro here, any tag or group of tags can be made into a macro.


### Bio.tools ID

We will now add a useful macro entry. With the Galaxy ecosystem is becoming more reliant on metadata, a bio.tools ID can help Galaxy to pull standardized metadata for your tool. This enables users to find and use the tool more easily from across the web, so we highly recommend setting a bio.tools identifier. If an entry already exists on [https://bio.tools](https://bio.tools), it's a simple task.

> <hands-on-title>Setting a bio.tools ID:</hands-on-title>
>
> If we go to [https://bio.tools](https://bio.tools) and search "Bellerophon", we find that it already has a bio.tools entry:
>
> ![Bellerophon bio.tools webpage](../../images/biotools-profile.png)
>
> We just have to add the biotools ID as a `<macro>` tag.
>
> Let's create a macro xml file next to our tool xml: `bellerophon_macros.xml`
>
> ```xml
> <macros>
>   <xml name="bio_tools">
>     <xrefs>
>         <xref type="bio.tools">Bellerophon</xref>
>     </xrefs>
>   </xml>
> </macros>
> ```
>
> Now we can expand the `bio_tools` macro in our tool XML, beneath the tokens:
>
> ```xml
> <macros>
>     <token name="@TOOL_VERSION@">1.0</token>
>     <token name="@VERSION_SUFFIX@">0</token>
> </macros>
>
> <import>bellerophon_macros.xml</import>
> <expand macro="bio_tools"/>
>
> ```
>
> If your tool does not yet have an entry in bio.tools, we highly encourage you to create one!
> It takes just 5 minutes to register with bio.tools to start contributing towards this global registry of computational resources.
>
{: .hands_on}

### Requirements section

The requirement section is where conda packages and docker/singularity containers are set. As this is a very basic tutorial,
the container requirements will not be discussed here.

For each conda package added to the environment in which the tool will run, a `<requirement/>` tag needs to be included in the `<requirements/>` section.
This specifies the

- Bioconda or Conda Forge package ID
  - This is set as plain text betweeh the `<requirement/>` tags
- Specific Bioconda/Conda Forge package version
  - This does not have to be the most recent version

> <hands-on-title>Adding requirements</hands-on-title>
> In the case of bellerophon, which requires two dependencies, bellerophon and samtools, the requirements section appears like so. Add them to your tool XML.
>
> ```xml
>   <requirements>
>         <requirement type="package" version="@TOOL_VERSION@">bellerophon</requirement>
>         <requirement type="package" version="1.12">samtools</requirement>
>   </requirements>
> ```
{: .hands_on}

This uses the macro token for @TOOL_VERSION@ for the Bioconda package version to retrieve the version number for bellerophon, while samtools' version is set manually.

> <question-title></question-title>
>
> If you wanted to add a requirement for the Seurat package of version 3.2, what would that look like?
>
> > <solution-title></solution-title>
> >
> > ```xml
> >   <requirements>
> >         <requirement type="package" version="3.2">seurat</requirement>
> >   </requirements>
> > ```
> >
> {: .solution}
{: .question}

### Input section

Although the next section in the tool script is actually be the command section, the input section will be discussed first, as it sets variables that are used in the command.
The input section has three possible xml tags at the highest level -- `<conditional/>`, ` <section/>`, and `<param/>`. Param will set any variable, conditional allows different available
parameters based on previous selections, and section simply breaks up the GUI into multiple parts that can be viewed or hidden at the user's discression.

#### *Param*

The param tag sets the value of variables, and has several attributes associated with it:

- Name
  - The name of the variable to be used in the command. Cannot include spaces.
- Argument
  - The actual flag being set in the tool. If the name and the flag is the same as what the name attribute would be, only Argument needs to be set.
- Label
  - The name of the parameter as shown to the user. Can include spaces.
- Type
  - Sets the variable type. Available options are:
    - Text
      - A string input with a built-in sanitizer.
    - Integer
      - Also allows for a minimum and maximum value to be set.
    - Float
      - Like integer, also allows for a minimum and maximum value to be set.
    - Select
      - Allows the user to choose from a pre-set list of options.
    - Boolean
      - Allows the user to select between two options, not simply True or False.
    - Data
      - Input file, with required file format specified.
- Help
  - Help text shown below the input for the parameter (optional).
- Optional
  - Boolean flag that allows the user to leave the parameter blank to either use the default value or to not use the parameter at all (optional).

#### *Conditional*

Conditionals are used alongside param tags to change available options based on what has been selected and is used with select-type params. Writing
conditionals, then, is a three-tag or four-tag process -- the `<conditional/>` tag, the `<param/>` of select type, and the `<when/>` tag,
used to specify the new paramaters available only on the selection of certain options. The param is of type "select", a tag `<option/>` is used to specify
each possible value. A very generic select-type conditional might look like this:

```xml
<conditional name="example_conditional">
    <param name="example_select" label="Example Select" type="select">
      <option value="ex1"/>
      <option value="ex2"/>
    </param>
    <when value="ex1">
        <param name="example_input" label="Example input" type="integer" min="0" help="Example help text"/>
    </when>
    <when value="ex2">
        <param name="example_input2" label="Example input 2" type="boolean" truevalue="--flag" falsevalue="--otherflag" help="Example help text"/>
    </when>
</conditional>
```

#### *Section*

Section tags are useful for visually separating the parameters. They can make a long tool with many parameters more simple to navigate by breaking it into
groups of parameters with similar functions. Alternatively, it can hide "advanced" or optional parameters, so that the user doesn't have to deal with
otherwise-irrelevant parameters if they don't need to. There are only 4 attributes:

- Name
  - Sets the internal reference name for the section. Cannot contain spaces.
- Title
  - Sets the visible name of the section for the user. Can contain spaces.
- Expanded
  - A boolean value that sets the default starting position of the section. If false, the section will begin collapsed, and only the title will be visible by default.
- Help
  - Help text visible to the user that explains the purpose of the section.

#### *Inputs Section in Practice*

The bellerophon tool's help section reads

```txt
optional arguments:
  -h, --help            show this help message and exit
  --forward FORWARD, -f FORWARD
                        SAM/BAM/CRAM file with the first set of reads.
  --reverse REVERSE, -r REVERSE
                        SAM/BAM/CRAM file with the second set of reads.
  --output OUTPUT, -o OUTPUT
                        Output BAM file for filtered and paired reads.
  --quality QUALITY, -q QUALITY
                        Minimum mapping quality.
  --threads THREADS, -t THREADS
                        Threads.
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}, -l {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Log level.
  --version             show program's version number and exit
```

This means that there are 6 total functional flags, --forward, --reverse, --output, --quality, --threads and --log-level.
Among these flags, two are file inputs, or 'data' type params, one is an integer type param, and three are parameters not
necessary for the user to have direct access to. As Galaxy uses specific file names or patterns to find output files, the
output file names need to be specific to the wrapper, not the actual running step. Since the user doesn't know the capability
of the resources on which the tool will be running, they should not be given access to the threads flag. And since the log level
goes to the stderr instead of an output file, this can also be skipped. All three of these flags can be coded in the command,
but not exposed to the user.

> <hands-on-title>Adding parameters to your tool</hands-on-title>
> 1. Using information in the above sections, add parameters for the forward, reverse, and quality flags to your tool xml.
>
> > <question-title></question-title>
> >
> > What does your `<inputs/>` section look like?
> >
> > > <solution-title></solution-title>
> > > To include all of the necessary parameters, then, the inputs section would appear like this
> > >
> > > ```xml
> > > <inputs>
> > >     <param argument="--forward" type="data" format="qname_sorted.bam,sam" label="First set of reads" help="This is usually the forward reads in your experiment." />
> > >     <param argument="--reverse" type="data" format="qname_sorted.bam,sam" label="Second set of reads" help="This is usually the reverse reads in your experiment." />
> > >     <param argument="--quality" type="integer" value="20" min="0" label="Minimum mapping quality"/>
> > > </inputs>
> > > ```
> > {: .solution}
> {: .question}
{: .hands_on}

This makes the variables "forward", "reverse" and "quality" available in the command section and lets the users set them in the Galaxy interface. The "min" attribute prevents
a user from inputting a value below a specified number. Here, the quality flag cannot be set below 0.

### Outputs section

The outputs section defines the files that Galaxy makes available in the history after running a tool using the `<data/>` and
`<collection/>` tags. In general, it tells Galaxy to look for a specific file after the job completes, and return it to the user.
It also defines the format of that file and the name shown to the user in the history.

> <hands-on-title>Adding your outputs</hands-on-title>
> As Bellerophon has a single output file, add the following outputs section to your tool:
>
> ```xml
> <outputs>
>     <data name="outfile" label="${tool.name} on ${on_string}" format="bam" />
> </outputs>
> ```
{: .hands_on}

This generates a history item called "bellerophon on `<input file name>`", as the ${tool.name} and ${on_string} are reserved values
for the tool's label and the input file labels.

As the name is hard-set using the output flag, and is output into the working directory, the name attribute can be used for the file
search. Alternatively, the attribute "from_work_dir" allows a lower directory to be specified.

#### *Filtering the output*

In the case where a certain output only appears when certain parameters are set, the `<filter/>` flag can help. Though not used in this
tool, a user might have a boolean or select that generate new output files. In this case, the data tag would appear like this

```xml
      <data format="bam" label="example" name="output_bam">
          <filter>options['selection_mode'] == 'advanced' and options['bam_output']</filter>
      </data>
```

Meaning that file only appears in the history if the user specified the "advanced" mode of the tool, and requested a bam output file.

#### *Output collections*

Collections are also useful for groups of output files, or simply data meant to be kept together like paired-end files.
In this case, the data to be grouped in the history like this.

```xml
<collection name="paired_output" type="paired" label="Split Pair">
    <data name="forward" format="txt" />
    <data name="reverse" format_source="txt" from_work_dir="reverse.txt" />
</collection>
```

Though paired-end data is used here, the type can also be 'list' or 'paired list', specifying a larger group of data, or a group of
paired-end data respectively.

#### *Discover datasets*

Datasets can also be found automatically, in the case where the number of output files is unknown.

```xml
<data format="tsv" name="sample">
    <discover_datasets pattern="__name_and_ext__" directory="outputs"/>>
</data>
```

This, for example finds a tabuar dataset in a directory named "outputs" that follows the naming convention of "`<filename>.<extension>`",
where the extension is .tsv. This method is also able to be used in a collection, where the same function would find all datasets that
fit the pattern in the specified directory.

### Command section

Returning to before the inputs section, the command section is written in bash with a built-in python interpreter called Cheetah. Because of that, the command section is, for the most part,
a list of bash commands connected by '&&', with the conda packages listed in the requirement available to run.

#### *Cheetah commands*

Cheetah allows loops, logic gates, and assignments of variables in a command section. Creation of a logic gate is as simple as this

```bash
#if ${parameter} == "value":
  <command>
#end if
```

For an "else":

```bash
#if ${parameter} == "value":
    <command>
#else:
    <command_2>
#end if
```

For an "else if":

```bash
#if ${parameter} == "value":
    <command>
#else if ${parameter} == "other value":
    <command_2>
#else:
    <command_3>
#end if
```

Similarly, loops would appear as

```bash
#for value in ${param}
    <Loop commands>
#end for
```

For all open loops and gates, an "#end" must be used to close.

Finally, to define a variable within the command section, use

```bash
#set $variable = value
```

#### *Using other scripts*

In the case where languages other than bash are necessary, such as R or python, by listing the languages in the requirements,
they can be called in this section. Any scripts present in the tools folder can also be called.
For example

```bash
python helloworld.py &&
echo ${variable}
```

runs the helloworld.py script present in the same folder as the tool xml, then echoes the value of the param named "variable" from the input section.

#### *In practice*

> <hands-on-title>Writing the command block</hands-on-title>
>
> The bellerophon command section, based on the variables set previously, would be as follows. Please add it to your tool XML.
>
> ```xml
>     <command detect_errors="exit_code"><![CDATA[
>         #if $forward.is_of_type("sam"):
>             #set $forward_input = 'forward_input.sam'
>             ln -s '${forward}' '$forward_input' &&
>         #else:
>             #set $forward_input = 'forward_input.bam'
>             ln -s '${forward}' '$forward_input' &&
>         #end if
>         #if $reverse.is_of_type("sam"):
>             #set $reverse_input = 'reverse_input.sam'
>             ln -s '${reverse}' '$reverse_input' &&
>         #else:
>             #set $reverse_input = 'reverse_input.bam'
>             ln -s '${reverse}' '$reverse_input' &&
>         #end if
>         bellerophon
>         --forward $forward_input
>         --reverse $reverse_input
>         --quality $quality
>         --output 'merged_out.bam'
>         && samtools sort --no-PG -O BAM -o '$outfile' -@ \${GALAXY_SLOTS:-1} merged_out.bam
>         ]]>
>     </command>
> ```
{: .hands_on}

The variables set in the inputs and output sections generate a full command to be run inside the Galaxy environment. When inside a conditional or
section, the hierarchy is preserved using a period. For example

```txt
'$example.forward'
```

would look for the variable 'forward' within the section or conditional named 'example'.

This command will run, assuming two sam input files, as:

```bash
ln -s `<forward input sam file>` '$forward_input' &&
ln -s `<reverse input sam file>` '$reverse_input' &&
bellerophon --forward $forward_input --reverse $reverse_input --quality $quality --output 'merged_out.bam'
&& samtools sort --no-PG -O BAM -o `<outfile name generated by Galaxy>` -@ `<number of threads allocated>` merged_out.bam
```

#### Symlinks

The symlinks in the first two parts of this command serve to standardize the naming of files passed to the bellerophon command.
For tools which require specific naming conventions, such as a standard prefix across all input files, or which require specified
file extensions, this is necessary. When using non-symlinked files, Galaxy uses metadata for filetypes, but, by default, refers to
files using a `.dat` etension. For example, had the symlinks not been used here, the command actually given would look like this:

```bash
bellerophon --forward <Galaxy_internal_reference_string>.dat --reverse <Galaxy_internal_reference_string>.dat ...
```

By adding the symlinks, the files are specified and will use proper names in the command. Although not strictly necessary for
bellerophon, it is best practice for command legibility.

### Test section

The test section is an easy way to ensure your tool wrapper functions as intended. It runs the tool and compares the output Galaxy returns against a
specified output. The output can be a specific file, a check against the stdout/err, or simply a search to see if the output contains specific
text. There can be any number of tests for a given tool, and it is preferred that all parameters are tested at some point in these tests.

When writing a test, the `<param>` tag functions slightly differently. It finds the variable with the same name, and then provides a value to be input
for the test. These params can also be placed inside a conditional, simulating the same conditional present in the inputs section.

As all parameters in bellerophon are accessible at once, and do not contradict one another, only one test is necessary.

> <hands-on-title>Adding a test</hands-on-title>
> Add the following test case to your tool:
> ```xml
>     <tests>
>         <test expect_num_outputs="1">
>             <param name="forward" value="forward.bam" />
>             <param name="reverse" value="reverse.bam" />
>             <output name="outfile" file="merged-out.bam" ftype="bam" />
>         </test>
>     </tests>
> ```
{: .hands_on}

This runs the tool with the files forward.bam and reverse.bam from a folder named test-data in the tool's directory. Since no quality parameter was specified, it uses the default of 20. The parameter can also be manually set with the same param tag and value attribute, just inputting an applicable value such as "21". It then
compares the defined output called "outfile" from the outputs section against a file called "merged-out.bam" in the same test-data directory. It also
double checks that Galaxy only finds one output to put in the history with the "expect_num_outputs" attribute. This is very useful for tests on tools
that have filtered outputs or that use 'discover_datasets'.

#### *Collections and sections in tests*

Though not used in this example, the sections and conditionals from the input section are used in tests as well. For example

```xml
<conditional name="example_conditional">
    <param name="select_example" value="first">
    <param name="when_first" value="1"/>
</conditional>
```

simulates the selection of value 1 inside of a conditional when the initial select parameter is set to "first".

### Help section

This is help text to be shown at the bottom of the tool form. It uses standard markdown format.

### Citations section

The citation section gives proper credit to the tool's writers. Here, using either DOI format or a bibtex citation, the tool form will automatically
link out to the relevant papers/repositories. The format in this section is simple, and as bellerophon was written by a Galaxy team member as a translation
of an Arima Genomics tool in association with the Vertebrate Genomes project, that is the paper that will be cited.

There are two ways to write this section

DOI:

```xml
<citations>
    <citation type="doi">10.1038/s41586-021-03451-0</citation>
</citations>
```

and Bibtex:

{% raw %}
```xml
<citations>
    <citation type="bibtex">@article{Butler2018,
    abstract = {A new computational approach enables integrative analysis of disparate single-cell RNA–sequencing data sets by identifying shared patterns of variation between cell subpopulations.},
    author = {Butler, Andrew and Hoffman, Paul and Smibert, Peter and Papalexi, Efthymia and Satija, Rahul},
    doi = {10.1038/nbt.4096},
    issn = {1546-1696},
    journal = {Nature Biotechnology},
    number = {5},
    pages = {411--420},
    title = {{Integrating single-cell transcriptomic data across different conditions, technologies, and species}},
    url = {https://doi.org/10.1038/nbt.4096},
    volume = {36},
    year = {2018}
    }</citation>
</citations>
```
{% endraw %}

Multiple citations can be added by using additional citation tags.

> <hands-on-title>Adding a citation</hands-on-title>
> Add the citation above using either method (but not both!)
{: .hands_on}

## Final wrapper

With all sections complete, the final wrapper for bellerophon can be found [in the IUC GitHub repository](https://github.com/galaxyproject/tools-iuc/blob/master/tools/bellerophon/bellerophon.xml).

## Toolshed file

The .shed.yml file is placed in the same directory as the tool's xml file and saves metadata for the tool. It enables
toolshed organization and search by using tags and descriptions.

It is made up of the following sections:

Field               | YAML key                | Description
-----               | --------                | -----------
Name                | `name`                  | the tool's name
Owner               | `owner`                 | username of the person who wrapped the tool to be represented in the Galaxy Toolshed. If submitted through the Intergalactic Utilities Comission (IUC), can be set to `iuc`.
Description         | `description`           | short help text for the tool/package.
Homepage URL        | `homepage_url`          | URL of the tool's website. Can also just be the Github repository for the tool.
Long Description    | `long_description`      | longer, more in-depth help section.
Remote Reposity URL | `remote_repository_url` | link to the repository for the tool's files.
Type                | `type`                  | kind of files in the directory. Can be set as unrestricted (tool), tool_dependency_definition, or repository_suite_definition.
Categories          | `categories`            | toolshed categories that are relevant to the tool or package.

For Bellerophon, the .shed.yml file would be:

```yaml
name: bellerophon
owner: iuc
description: Filter mapped reads where the mapping spans a junction, retaining the 5-prime read.
homepage_url: https://github.com/davebx/bellerophon
long_description: |
  Filter mapped reads where the mapping spans a junction, retaining the 5-prime read.
remote_repository_url: https://github.com/davebx/bellerophon
categories:
- Sequence Analysis
```

In the case where the directory represents a group of tools or a 'suite', there are additional overarching sections into which the above tags fall:

- Auto Tool Repositories
- Suite

For more information on how to write automatic tool suites, visit the [Galaxy docs](https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/shed_yml.html).

# Testing Galaxy tool with `planemo`

`planemo` is a command line utility that helps developing Galaxy tools.
Among many other tasks it can:

1. Lint the tool source errors and conformance to best practices.
2. Test tools, i.e. executing the tool with the tests described in the tool's test section.
3. Serve tools, i.e. crate an ad-hoc Galaxy server running the tool.

For more information on `planemo` see its extensive [documentation](https://planemo.readthedocs.io/).

In this part of the tutorial we will be looking at `planemo` and
see it in action linting, testing, and serving the tool.

## Using `planemo` to lint tools

When linting a Galaxy tool `planemo` checks the sources for common errors and violations of best practice rules. Examples are:

- Presence to "TODO" in the help section
- Absence of tests or citations
- Missing when clauses in conditionals or the use of a boolean parameter as the selector in a conditional

If `planemo` finds a problem it outputs warnings or errors depending on the severity of the problem.
By default `planemo` will fail if any warning or error is found, i.e. return a non-zero exit code.

> <hands-on-title>Lint a tool with `planemo`</hands-on-title>
>
> 1. Change into the directory containing the tool
> 2. Run `planemo lint`
> 3. Inspect the output for any warnings or errors
> 4. Fix the problem(s) and rerun from step 2 until all problems are solved
{: .hands_on}

By default `planemo` will lint all tools contained in the current working directory (including subdirectories).
You can also specify a directory or a tool by giving its path as additional argument.

## Using `planemo` to test tools

Using `planemo` to execute the tests defined in a tool is probably the most frequently used feature.
In order to do so `planemo` will:

- Start a local Galaxy instance
- Stage all input data defined in the tests
- Run the tool using the datasets and parameters defined in the tests (this also involves the installation of all requirements which is done with conda by default)
- Check if the test assumptions are met, e.g. non-zero exit code, equivalence of the output files, number of outputs, ...

> <hands-on-title>Test a tool with `planemo`</hands-on-title>
>
> 1. Change into the directory containing the tool
> 2. Run `planemo test` (as with `planemo lint` you can also specify a path to a tool explicitly as extra argument)
>
>    The most important output of `planemo` is found at the end and looks like:
>
>    ```
>    Testing complete. HTML report is in "PATH/tool_test_output.html".
>    All X test(s) executed passed.
>    TOOL_NAME (Test #1): passed
>    TOOL_NAME (Test #2): passed
>    ...
>    ```
>
>    The last part lists for each test if it failed or passed. In case of a failure you can open the file `tool_test_output.html` (the path to this file is in the output) with a browser to get more information.
>
> 3. Open the HTML report `firefox PATH/tool_test_output.html` (replace the PATH as in `planemo`'s output).
>
>    The HTML report contains a table on the top giving the number of executed, successful, and failed tests:
>
>    ![Planemo HTML report](../../images/planemo_html.png "Planemo HTML report")
>
>    If there are passed or failed tests you will find the text `Passed tests`, `Failed tests`, resp., below the table.
>    Clicking on this text will unfold the list of passed or failed tests, resp. Clicking again on one of the tests
>    Will show additional information like stdout, stderr and the executed command line. In most cases this is sufficient
>    to determine the source of the test failure.
{: .hands_on}


> <tip-title>Useful parameters for `planemo test`</tip-title>
>
> * `--failed` Will make `planemo` rerun only the tests that failed in the previous execution.
> * `--update_test_data` If there are differences to output files defined in the tests these will be updated (and the tests will run again).
> * `--no_cleanup` `Planemo` will not cleanup the created Galaxy clone or the job working dir (see the details box below).
{: .tip}


> <details-title>Understanding `planemo`'s output in more detail</details-title>
>
> Sometimes it can be helpful to understand the `planemo` output in more detail.
>
> - The first lines show some `git` commands that create a local clone of the Galaxy sources (by default the `master` branch) and the command to start Galaxy. The end of the part is marked by the line `Activated a virtualenv for Galaxy`.
> - Then the python virtual environment used to run Galaxy is created and the requirements are installed
> - Then the worker threads and the Galaxy database are initialized, look for lines containing `Initializing main Galaxy Queue Worker on` and `Install database targetting Galaxy's database configuration`. The completion of the startup of Galaxy is indicated by the line `INFO:     Application startup complete.`
>
> The first two steps can take a while on the first run of `planemo test` (in particular on slow interne


topic: dev
tutorial: tool-generators-advanced
title: ToolFactory: Generating Tools From More Complex Scripts
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: []
content_hash: 908c0dd6ecdaead1f301711762c5468a
last_modified: 2026-05-15T22:28:11.277508
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/tool-generators-advanced/tutorial.html

content:
Galaxy users who write and share scripts useful for scientific analyses are likely to be reading this material, perhaps after seeing the "Hello Galaxy"
demonstration. It was written to help you find out about the capabilities and limits of the ToolFactory by experimenting with it yourself.
It is hoped that this advanced tutorial will introduce some features that potentially make the ToolFactory useful in your work.

> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Background and a user's guide to this training material

This training material is unlike most other GTN tutorials. There is no specific tool building curriculum on offer because it is hard to know how
that might be orgranised. The introductory `Hello Galaxy` demonstrations and
hands-on are the only formal attempt at explaining any tool's structure and construction.

The reason is that tools are mostly scientific domain specific, while the ToolFactory is applicable to any domain where Galaxy might be useful.
Developers and the tools they might be interested in creating are so diverse
that it was decided to provide the infrastructure and environment to encourage "learning by doing".
The result of editing and regenerating a tool is immediately available in the ToolFactory for evaluation to encourage self-guided exploration.
Developers are free to explore their own specific requirements.

Simple demonstration tools are provided. The are both the functional documentation, and useful models to adjust and explore.
They each address a specific feature (such as the `plotter` illustrating R parameter passing and collectionscollections),
and they illustrate some of the styles of tool building supported by the ToolFactory. The `sed` example does not use a script. All the others do.
The ToolFactory is likely to be more useful for scripts than directly driving Conda packages. Many Conda package interfaces have requirements that the ToolFactory cannot
currently address. Those will probably always require a skilled tool document developer. Conda dependencies requiring minimal conditional parameter
logic for the construction of a command line or for adjusting outputs, such as the `sed` example, may be more tractable.

This training material will introduce some ToolFactory features and will show how some of the limitations can be worked around.
The introductory tutorial walked through some simple tools. This advanced tutorial shows some examples but the focus is on self-guided
exploration of the sample tools to see how they might be used in your work.


## The ToolFactory is for scientists and developers who write command line scripts in their work.

The ToolFactory is distributed as a docker image or as a self-configuring clone of the Galaxy server code. This is intended to make it easy to create new tools, analyses and workflows can be developed to help speed uptake in areas of data-intensive science where
few Galaxy tools are currently available.

The ToolFactory automates much of the work needed to prepare a new Galaxy tool using information provided by the script writer
on a Galaxy tool form. It can generate XML to wrap any simple script that runs correctly on the linux command line with some small test input samples. This is potentially
handy for developers new to Galaxy, and for Galaxy users who are capable of correctly scripting on the command line for themselves, because those working scripts can
be wrapped and tested to make toolshed ready, shareable Galaxy tools, or if too trivial to be worth sharing, used on the host desktop Galaxy for real analyses.

Generated tools are immediately installed and ready to run. This provides instant feedback for the developer in an integrated tool
development environment for simple scripts. Jobs that generate tools can always be rerun using {% icon galaxy-refresh %}. The form reappears as it was
when the tool was generated. Data and user configurable parameters can be added or removed. Text that appears when the tool is run, such as user parameter or data input
labels and help, can be updated by editing the form. When the updated tool is generated, installed and run as a tool, the tool form will include all the changes, ready to run.


The development server is a self-configuring, disposable platform for a data intensive scientist to learn how Galaxy tools work, in preparation for the more flexible and efficient
Galaxy project supported manual tool building infrastructure. At the same time, they can wrap and refine new tools from existing scripts
on their own workstations, preparing them for sharing and deployment to production servers. It also may be useful as a private sandbox for learning about tools and
Galaxy administration or experimenting with code development.


> <tip-title>Under the hood:</tip-title>
>  - It uses [galaxyml](https://github.com/hexylena/galaxyxml) to generate the tool XML from ToolFactory form settings.
{: .tip}

---

## Limits and scope

- The ToolFactory can generate, install and run new tools from scripts on your desktop.
    - It is a fully functional development Galaxy instance, so it can import any existing tool from a toolshed and can handle as much data as your desktop disks will fit.
    - It takes 30 minutes to install locally or a few minutes as a docker image, and can be completely removed when no longer useful.
    - The user is entirely responsible for securing and backing up all their work in the docker version. The desktop version is a normal persistent Galaxy development server.
    - It is not recommended for production and is *not supported by the Galaxy team*.
    - A well endowed modern workstation or laptop with plenty of cores, disk and RAM is needed. Older commodity hardware may struggle.
- The ToolFactory works best wrapping simple R/Bash/Python and other interpreted scripts, with a few user supplied parameters and a few I/O history files.
- Conditional parameters, XML macros, output filters and many other advanced features are not currently supported. Manual coding will be needed.
- Scripts are easier than some Conda packages
    - This is because the tool builder can modify the code to respond to default empty parameters as if they had not been passed.
    - This may not sound like much, but as a result, advanced tool building elements such as conditionals and related tricks requiring manual coding, can sometimes be avoided.
    - In contrast, some Conda dependencies or combinations will require output filters
or other complex tool XML constructs outside the `<command>` section, that are not easy to generate automatically and can only be coded in the document.
    - While some simple requirements may be manageable, complex ones will not be suitable for the ToolFactory.
- Compared to the more usual linux shell and text editor, the ToolFactory appliance is a rather clumsy way to debug scripts.
    - Starting a new ToolFactory tool with a know good command line and data is strongly recommended.
      - You will know exactly what to expect from the tool test for a first sanity check.
    - Corrolary: Unless there is a working script that needs to be wrapped into a toolshed-ready Galaxy tool, the ToolFactory is of little use
      - other than for learning about Galaxy tools.
- The ToolFactory Appliance is for developers and code-writing scientists not yet familiar with the more flexible and complex manual tools, and who need to wrap scripts that are simple
enough for the ToolFactory.
    - Compared to the more flexible manual Galaxy tool development software, there is far less to learn to get up to speed with a form driven,
automated code generator in a tailored, readily deployed appliance.
    - The cost of this convenience is that ToolFactory is limited to a subset of simple script and package wrappers.
    - Much can be learned in the process of trying things out in the ToolFactory that will help acquire skills for manually building tools.

---

# Installation (Copied from the introductory tutorial)

> <warning-title>Security advisory!</warning-title>
>- *Please do not install the ToolFactory on any public server*
>- Configured as a default server, it has none of the usual additional security layers required for a safe public Galaxy server.
>- Although it will only run for administrative users, it allows unlimited scripting and exposes unacceptable risks.
>- Please install it locally.
>- For this reason, the training materials can't make use of existing public Galaxy infrastructure like most of the GTN material.
{: .warning}


> <hands-on-title>Installing and managing a Galaxy ToolFactory development server</hands-on-title>
> # Logging in as an administrator
> 
> Once you have a working installation running as described below, the server should be ready in 20-30 seconds or so, at [http://localhost:8080](http://localhost:8080).
> *The ToolFactory will only execute for administrative users as a minimal security barrier.* 
> When the webserver starts, immediately login using the administrative user email *toolfactory@galaxy.org* with the password *ChangeMe!* which of course you should change.
>   
> # Docker
> Note: *Nothing is persistent in the image*. Useful work must be manually exported and saved.
> For following the GTN tutorial and for test driving the ToolFactory for the first time, the docker version is recommended.
> Non-persistent means it does not remember anything after you stop the container. The next time it starts it will be a fresh installation.
> You can save your work by exporting the histories and tool tarballs you want to keep. You need the history to rerun the job that generated a tarball, so the history is the most important thing to preserve if you make a useful tool.
> The image must be pulled first, then run with port 8080 open for the Galaxy server.
> 
> ```
> docker pull quay.io/fubar2/galaxy_toolfactory:latest
> docker run -d -p 8080:8080 quay.io/fubar2/galaxy_toolfactory:latest
> ```
> Check the docker logs until gunicorn is ready to serve or wait 
> about 20-30 seconds, then browse to [http://localhost:8080](http://localhost:8080)
> If a Galaxy server appears, proceed with the login instructions above and you should see a history containing all the example tools. There is also a workflow that can reproduce that history if you set all the inputs to the right input datasets in that history.
>  
> # Local workstation development Galaxy server installation
> 
> A persistent desktop development ToolFactory server can be built by cloning the [bootstrap github repository](https://github.com/fubar2/galaxy_tf_overlay), and using the included *localtf.sh* script, to build a complete, new development server with the ToolFactory installed and ready to run.
>
> From a convenient directory, download the overlay configuration code repository, then
> run the *localtf.sh* setup script from that cloned repository root directory:
>
> ```
> git clone https://github.com/fubar2/galaxy_tf_overlay.git
> cd galaxy_tf_overlay
> sh ./localtf.sh
> ```
> Running *localtf.sh* will create a new directory, *galaxytf*, in the parent directory of the *galaxy_tf_overlay* clone.
> The script will configure a fresh Galaxy 23.0 server, with the ToolFactory installed, in that new directory.
> This takes 20 minutes or more to complete since the client must be built once.
>
> The resulting development server directory will occupy ~9GB of disk space, so be sure your machine has plenty of room.
> It will be based in a single directory, *galaxytf* in the same directory as the galaxy_tf_overlay repository was cloned into.
> That is where the script should be run as shown above.
>
> Rerunning the *localtf.sh* script will *destroy the entire galaxytf directory* - all ~9GB, and create a clean new installation.
> It should only need to be run once in the life of the development server.
>
> Remove the *galaxytf* directory to remove the entire development server when it is no longer needed. Save all your tools and histories,
> because the jobs in a history can be used to update a tool easily, and a history can be imported into a fresh development instance
> when needed.
>
>
> Once installation is complete:
>  * start the server from the *galaxytf* directory with *sh run.sh*. The logs will be displayed.
>  * ^c (control+c) will stop it from the console.
>  * In 23.0 that is equivalent to *.venv/bin/galaxyctl start* and *.venv/bin/galaxyctl stop*.
>
>
{: .hands_on}


The trivial `Hello World!` tool example is readily extended to suit many situations where a tool is needed quickly for a workflow. Try adding another parameter.
For example, the planemo `lint` tool example (described below) can be derived by adding a history toolshed archive as input, plus a few more lines of bash script.
In practice, it's a flexible basis for generating many simple tools.


# ToolFactory tips and tricks illustrated by some of the examples.

## Before you begin a new tool

- Make sure it runs correctly on a command line with your sample inputs and default parameter settings. That is how the test will be generated.
- You cannot specify any inputs on the form without providing samples, and those samples must run correctly, with the supplied defaults.
- Easiest to make sure those samples are in the history before you begin.
- Be well prepared. The ToolFactory cannot do that preparation for  you.

## STDIN and STDOUT

- Demonstration tools often capture output from a bash script using the special STDOUT designation for output files
- This can save sending the output path as a parameter to the script or executable
- STDIN is also available as a special designation for history inputs if the script takes input from STDIN when it runs.
- The option `Tool reads selected input file from STDIN and writes STDOUT with no parameters` in the parameter passing model
selection list will generate a simple filter tool, executing a script or Conda dependency with input on STDIN and output on STDOUT
for those rare situations where that's all you need. No i/o or other parameters for the user to set. Used in the tacrev demonstration tool.

## Repeats in the ToolFactory form theoretically permit *any* number of parameters.

- Inputs, outputs and additional user parameters are all presented within repeat elements on the form.
- For any given tool, repeats allow an unlimited number of each of those tool elements - from a technical perspective.
- The practical limit is reached when your patience runs out as you add them in the current Galaxy forms UI.
- The form gets very long, very quickly.
- A handful is easily manageable.

## Repeated parameters in generated tools allow the end user to supply multiple values

- Repeats *on the generated tool form* are supported for input and user edited parameters
- The generated tool form has the usual "Add..." button associated with the parameter, so the tool user can add any number of them.
- The script must be able to parse and deal correctly with multiple instances of the same parameter name.
- A Python sample script using argparse with `action="append"` parameters to deal with potentially any number of multiples of each parameter command
line will echo all the repeated parameters is shown in the example shown in the example below.
- Repeats do not make sense for `positional parameters` because their number is unpredictable. The ToolFactory will ignore repeats in positional mode. A warning is issued in the log.
- Repeats on `output parameters` are not supported - use a `collection` described below when an unpredictable number of output files are required.
- The sample shown below has one input data parameter and one text parameter. The `repeat` option is selected for both on the ToolFactory form.
- In the generated tool XML wrapper, these are embedded within `<repeat>` tags in the `<inputs>` and also in the `<tests>` sections automatically.
- Note that repeats are limited to *single parameters* - groups of parameters cannot currently be repeated. They will require manual tool writing.
- The end user sees the form with repeatable fields as shown at the end of the detail below.
- The demonstration returns whatever the user chose to repeat.


> <details-title>Repeats demonstration generated tool XML</details-title>
>
>
> ```xml
> <tool name="repeats_demo" id="repeats_demo" version="2.00">
>   <!--Source in git at: https://github.com/fubar2/galaxy_tf_overlay-->
>   <!--Created by admin@galaxy.org at 29/05/2021 07:46:08 using the Galaxy Tool Factory.-->
>   <description>Repeated parameter demonstration</description>
>   <requirements/>
>   <stdio>
>     <exit_code range="1:" level="fatal"/>
>   </stdio>
>   <version_command><![CDATA[echo "2.00"]]></version_command>
>   <command><![CDATA[python
> $runme
> #for $rep in $R_mi:
> --mi "$rep.mi"
> #end for
> #for $rep in $R_mp:
> --mp "$rep.mp"
> #end for
> >
> $repeats_out]]></command>
>   <configfiles>
>     <configfile name="runme"><![CDATA[#raw
>
> import argparse
> parser = argparse.ArgumentParser()
> a = parser.add_argument
> a("--mi", action="append")
> a("--mp", action="append")
> args = parser.parse_args()
> if args.mi:
>    print(" file and ".join(args.mi))
> if args.mp:
>    print(" string and ".join(args.mp))
> if not (args.mi or args.mp):
>    print('Nothing was selected')
>
> #end raw]]></configfile>
>   </configfiles>
>   <inputs>
>     <repeat name="R_mi" title="Add as many Multiple input files from your history - as many as you like as needed">
>       <param name="mi" type="data" optional="false" label="Multiple input files from your history - as many as you like" help="" format="html,txt,xml" multiple="false"/>
>     </repeat>
>     <repeat name="R_mp" title="Add as many Multiple user supplied text strings - as many different ones as you like as needed">
>       <param name="mp" type="text" value="Multiple user supplied text strings - as many different ones as you like" label="Multiple user supplied text strings - as many different ones as you like" help=""/>
>     </repeat>
>   </inputs>
>   <outputs>
>     <data name="repeats_out" format="txt" label="repeats_out" hidden="false"/>
>   </outputs>
>   <tests>
>     <test>
>       <output name="repeats_out" value="repeats_out_sample" compare="diff" lines_diff="6"/>
>       <repeat name="R_mi">
>         <param name="mi" value="mi_sample"/>
>       </repeat>
>       <repeat name="R_mp">
>         <param name="mp" value="Multiple user supplied text strings - as many different ones as you like"/>
>       </repeat>
>     </test>
>   </tests>
>   <help><![CDATA[
>
> **What it Does**
>
> Simple python Argparse sample to echo repeated user selections - how to use repeated inputs and user parameters.
>
> Unpredictable or messy "repeated" outputs can use a collection if they are not useful downstream but otherwise require manual wrapping - see the GTN advanced tutorial.
>
>
>
> ------
>
>
> Script::
>
>     import argparse
>     parser = argparse.ArgumentParser()
>     a = parser.add_argument
>     a("--mi", action="append")
>     a("--mp", action="append")
>     args = parser.parse_args()
>     if args.mi:
>        print(" file and ".join(args.mi))
>     if args.mp:
>        print(" string and ".join(args.mp))
>     if not (args.mi or args.mp):
>        print('Nothing was selected')
>
> ]]></help>
>   <citations>
>     <citation type="doi">10.1093/bioinformatics/bts573</citation>
>   </citations>
> </tool>
>
>
> ```
>
>  - The user sees the following form after adding 3 repeats for each of the two available items
>
>  ![User can add as many repeats as they want for repeated input file and parameter values](../../images/toolfactory_repeats_sample_form.png)
{: .details}

## ToolFactory `collection` outputs are handy for hiding dozens of miscellaneous tool outputs in a single history item

- If the script writes a large number of different output file types (images, reports..) that are not in themselves useful for downstream analyses.
- A Collection can be used to hide them in an organised list as a single new history item.
- Collections are special kinds of Galaxy history items that can present a directory full of files arranged inside one single new history item.
- When viewed, that history item presents them all as a list, each a typical, viewable Galaxy history item.
- They don't clutter the user's history because they are all hidden in the collection.
- The plotter example uses an Rscript.
- It generates as many pairs of random plots as the parameter supplied requires.
- The script sends them into the the collection that appears in the history after the job runs.
- The user's history shows only one new item after it runs - it must be viewed to see all the individual contents.

> <warning-title>The default generated test for output collections always passes because it doesn't test anything.</warning-title>
>
>    - Supplying a test over-ride is recommended for collections.
>    - Example code is shown on the sample tool's form and in the original example code below - removed from the current sample.
>    - For a real test, one or more expected <element.../> tags must be provided so the test really does test something.
>    - Add your own file names to the sample to make a real test for your own collections.
>    - Otherwise, the generated test will be empty, because the code generator knows nothing about what's going to appear in the collection at the time the `<test>` code is generated.
>    - Introspecting arbitrary scripts to reliably populate the test with actual file names?. Not likely any time soon.
>
{: .warning}

> <details-title>`plotter` collection output demonstration tool form, generated XML and outputs</details-title>
>
>  - The ToolFactory form for the plotter example tool
>  is configured as shown below, from "rerunning" the plotter job from the sample history.
>
> ![ToolFactory form configuration of the output collection in the plotter example](../../images/toolfactory_plotter_demo_form.png)
>
>  The Rscript is contained in a configfile so`#` is escaped - this is automatic.
>
> ```xml
> <tool name="plotter" id="plotter" version="0.01">
>   <!--Source in git at: https://github.com/fubar2/tgalaxy_tf_overlay-->
>   <!--Created by admin@galaxy.org at 24/01/2021 05:02:33 using the Galaxy Tool Factory.-->
>   <description>ToolFactory collection demonstration - random plots</description>
>   <requirements>
>     <requirement version="" type="package">r-base</requirement>
>   </requirements>
>   <stdio>
>     <exit_code range="1:" level="fatal"/>
>   </stdio>
>   <version_command><![CDATA[echo "0.01"]]></version_command>
>   <command><![CDATA[Rscript
> $runme
> "$nplot"]]></command>
>   <configfiles>
>     <configfile name="runme"><![CDATA[
> \# demo
> args = commandArgs(trailingOnly=TRUE)
> if (length(args)==0) {
>    n_plots = 3
> } else {
>    n_plots = as.integer(args[1]) }
> dir.create('plots')
> for (i in 1:n_plots) {
>     foo = runif(100)
>     bar = rnorm(100)
>     bar = foo + 0.05*bar
>     pdf(paste('plots/yet',i,"anotherplot.pdf",sep='_'))
>     plot(foo,bar,main=paste("Foo by Bar plot \#",i),col="maroon", pch=3,cex=0.6)
>     dev.off()
>     foo = data.frame(a=runif(100),b=runif(100),c=runif(100),d=runif(100),e=runif(100),f=runif(100))
>     bar = as.matrix(foo)
>     pdf(paste('plots/yet',i,"anotherheatmap.pdf",sep='_'))
>     heatmap(bar,main='Random Heatmap')
>     dev.off()
> }
>
> ]]></configfile>
>   </configfiles>
>   <inputs>
>     <param label="Number of random plots pairs to draw" help="" value="3" type="text" name="nplot" argument="nplot"/>
>   </inputs>
>   <outputs>
>     <collection name="plots" type="list" label="Plots">
>       <discover_datasets pattern="__name_and_ext__" directory="plots" visible="false"/>
>     </collection>
>   </outputs>
>
>
>   <tests>
>     <test>
>       <param name="nplot" value="3" />
>       <output_collection name="plots" type="list">
>      <element file="yet_1_anotherplot_sample" name="yet_1_anotherplot" ftype="pdf" compare="sim_size" delta_frac="0.05"/>
>     </output_collection>
>  </test>
>   </tests>
>
>
>
>   <help><![CDATA[
>
> **What it Does**
>
> Draws as many random plot pairs as you need
>
>
>
> ------
>
>
> Script::
>
>     # demo
>     args = commandArgs(trailingOnly=TRUE)
>     if (length(args)==0) {
>        n_plots = 3
>     } else {
>        n_plots = as.integer(args[1]) }
>     dir.create('plots')
>     for (i in 1:n_plots) {
>         foo = runif(100)
>         bar = rnorm(100)
>         bar = foo + 0.05*bar
>         pdf(paste('plots/yet',i,"anotherplot.pdf",sep='_'))
>         plot(foo,bar,main=paste("Foo by Bar plot #",i),col="maroon", pch=3,cex=0.6)
>         dev.off()
>         foo = data.frame(a=runif(100),b=runif(100),c=runif(100),d=runif(100),e=runif(100),f=runif(100))
>         bar = as.matrix(foo)
>         pdf(paste('plots/yet',i,"anotherheatmap.pdf",sep='_'))
>         heatmap(bar,main='Random Heatmap')
>         dev.off()
>     }
>
> ]]></help>
>   <citations>
>     <citation type="doi">10.1093/bioinformatics/bts573</citation>
>   </citations>
> </tool>
> ```
>  After requesting 25 pairs of plots from the sample tool, a collection appears in the history and is shown below.
>  One of them is displayed by clicking the "eye" icon.
>  Collections are ideal for messy analysis reporting outputs such as images, pdfs and other material that is not useful as an input to a downstream tool.
>  It is material that the user will want kept together, so a single history item is ideal to avoid unnecessary clutter.
>  As shown above, the script only has to write the files to a directory.
>  Note that the test is over-ridden in the ToolFactory form to generate this tool.
>  Without an over-ride to suit your script, an empty test will be generated that will not do any testing - see warning above.
>
>
> ![Plotter collection opens to show all the plots as separate items to be viewed or downloaded](../../images/toolfactory_plotter_sample_output.png)
{: .details}

## Selects as user supplied parameters

- Additional parameter types are selected from a drop down list. It includes text, numeric and select parameters.
- Selects offer a list of prespecified options to the user.
- There is a repeat on the ToolFactory form to collect options as pairs of names/values.
- It is clumsy and suitable only for a limited number of options.
- Galaxyxml generates appropriate select parameters on the generated tool as shown in the select demonstration tool.

> <details-title>`select_test` select field demonstration tool generated XML</details-title>
>
> The ToolFactory form section for user configurable command line settings is
>  configured as shown here for the select demonstration
>
> ![Form to configure an additional parameter as a select](../../images/toolfactory_select_demo_form.png)
>
> The generated XML is shown below.
>
> ```xml
> <tool name="select_test" id="select_test" version="0.01">
>   <!--Source in git at: https://github.com/fubar2/galaxy_tf_overlay-->
>   <!--Created by admin@galaxy.org at 24/01/2021 05:03:21 using the Galaxy Tool Factory.-->
>   <description>ToolFactory select demonstration</description>
>   <stdio>
>     <exit_code range="1:" level="fatal"/>
>   </stdio>
>   <version_command><![CDATA[echo "0.01"]]></version_command>
>   <command><![CDATA[bash
> $runme
> "$choose"
> >
> $select_out]]></command>
>   <configfiles>
>     <configfile name="runme"><![CDATA[
> echo "You chose \$1"
> ]]></configfile>
>   </configfiles>
>   <inputs>
>     <param label="Choose" help="" type="select" name="choose" argument="choose">
>       <option value="won">one</option>
>       <option value="too">two</option>
>       <option value="free">three</option>
>     </param>
>   </inputs>
>   <outputs>
>     <data name="select_out" format="txt" label="select_out" hidden="false"/>
>   </outputs>
>   <tests>
>     <test>
>       <output name="select_out" value="select_out_sample" compare="diff" lines_diff="0"/>
>       <param name="choose" value="won"/>
>     </test>
>   </tests>
>   <help><![CDATA[
>
> **What it Does**
>
> Echoes your selection
>
>
>
> ------
>
>
> Script::
>
>     echo "You chose $1"
>
> ]]></help>
>   <citations>
>     <citation type="doi">10.1093/bioinformatics/bts573</citation>
>   </citations>
> </tool>
> ```
>  The generated tool form from the select demonstration shows the three options and returns the one selected.
>
> ![Generated form seen by users of the select demonstration tool](../../images/toolfactory_select_test_tool.png)
{: .details}



## The ToolFactory can wrap some Conda packages correctly using a simple wrapper script to work around limitations.

- Many complex tools require manual coding to make them useable.
  - Often complex tools hide data being moved between multiple dependencies such as samtools in the bwa tool.
  - These are often not suitable for the simple automated tool generator in the ToolFactory.
     - Sometimes a suitable script may be able to perform the complications needed so a tool can be created.
     - A lot depends on the ingenuity and preferences of the developer.
     - For specialist use and complex tools, the project developer supported tool building infrastructure is recommended.

> <details-title>`planemo lint` demonstration tool generated XML</details-title>
> The labels for outputs have been edited to include the tool name in this sample - this is not possible at present in the ToolFactory.
>
> ```xml
> <tool name="planemo_lint" id="planemo_lint" version="0.01">
>   <!--Source in git at: https://github.com/fubar2/galaxy_tf_overlay->
>   <!--Created by admin@galaxy.org at 18/05/2021 01:16:17 using the Galaxy Tool Factory.-->
>   <description>Runs Planemo lint on any ToolFactory xml history file</description>
>   <requirements>
>     <requirement version="0.74.3" type="package">planemo</requirement>
>     <requirement version="3.7" type="package">python</requirement>
>     <requirement type="package" version="4.6.3">lxml</requirement>
>   </requirements>
>   <stdio>
>     <exit_code range="1:" level="fatal"/>
>   </stdio>
>   <version_command><![CDATA[echo "0.01"]]></version_command>
>   <command><![CDATA[python
> $runme
> $ToolFactory_XML_to_be_linted
> >
> $lint_output]]></command>
>   <configfiles>
>     <configfile name="runme"><![CDATA[#raw
>
> import lxml.etree as ET
> import os
> import subprocess
> import sys
>
> def main():
>     assert len(sys.argv) >= 2, 'Must have input xml on command line'
>     xmlin = sys.argv[1]
>     tree = ET.parse(xmlin)
>     root = tree.getroot()
>     toolname = root.get('id')
>     toolxml = os.path.join('/export/galaxy/tools/TFtools', toolname, '%s.xml' % toolname)
>     cl = ['planemo', 'lint', toolxml]
>     print('Running', cl)
>     p = subprocess.run(cl, shell=False)
>     if p.returncode > 0:
>          print('Planemo lint call returned error %d')
>     else:
>          print('Lint report ends')
> main()
>
>
> #end raw]]></configfile>
>   </configfiles>
>   <inputs>
>     <param name="ToolFactory_XML_to_be_linted" type="data" optional="false" label="ToolFactory XML to be linted" help="" format="xml" multiple="false"/>
>   </inputs>
>   <outputs>
>     <data name="lint_output" format="txt" label="${ToolFactory_XML_to_be_linted.name}_lint_output" hidden="false"/>
>   </outputs>
>   <tests>
>     <test>
>       <output name="lint_output" value="lint_output_sample" compare="diff" lines_diff="5"/>
>       <param name="ToolFactory_XML_to_be_linted" value="ToolFactory_XML_to_be_linted_sample"/>
>     </test>
>   </tests>
>   <help><![CDATA[
>
> **What it Does**
>
> ToolFactory demonstration script using bash to run planemo lint from a history XML representing a tool.
>
>
>
> ------
>
>
> Script::
>
>     import lxml.etree as ET
>     import os
>     import subprocess
>     import sys
>     def main():
>         assert len(sys.argv) >= 2, 'Must have input xml on command line'
>         xmlin = sys.argv[1]
>         tree = ET.parse(xmlin)
>         root = tree.getroot()
>         toolname = root.get('id')
>         toolxml = os.path.join('/export/galaxy/tools/TFtools', toolname, '%s.xml' % toolname)
>         cl = ['planemo', 'lint', toolxml]
>         print('Running', cl)
>         p = subprocess.run(cl, shell=False)
>         if p.returncode > 0:
>              print('Planemo lint call returned error %d')
>         else:
>              print('Lint report ends')
>     main()
> #end raw
> ]]></help>
>   <citations>
>     <citation type="doi">10.1093/bioinformatics/bts573</citation>
>   </citations>
> </tool>
>
>
>
> ```
{: .details}


## Other languages such as Lisp and Prolog scripts in Galaxy.

- Any interpreted language in Conda can be used, as long as it will take a command line script path or read from STDIN.
- Trivial samples using Lisp and Prolog are included in the demonstration history.
- They may be old, but they show how it is easy to use any Conda scripting language package in Galaxy for long running applications.
- The Prolog tool has an inbuilt script. Substitute the sample script for real code and add inputs, user-configurable parameters and outputs to produce a tool wrapping a Prolog script if you ever need one in Galaxy.
- The Lisp tool will try to execute the selected input text file, hoping it contains a Lisp program like the `hello world` example supplied. This is a terrible idea for a public server and is shown only as a skeletal example of what is possible, not what is sensible.

> <details-title>`Prolog` and `Lisp` demonstration tools</details-title>
> ```xml
> <tool name="prolog_demo" id="prolog_demo" version="0.01">
>   <!--Source in git at: https://github.com/fubar2/galaxy_tf_overlay-->
>   <!--Created by admin@galaxy.org at 04/04/2021 17:21:32 using the Galaxy Tool Factory.-->
>   <description>Runs a prolog script</description>
>   <requirements>
>     <requirement version="" type="package">swi-prolog</requirement>
>   </requirements>
>   <stdio>
>     <exit_code range="1:" level="fatal"/>
>   </stdio>
>   <version_command><![CDATA[echo "0.01"]]></version_command>
>   <command><![CDATA[swipl
> -q
> -g
> main
> -s
>
> $runme
> >
> $prolog_out]]></command>
>   <configfiles>
>     <configfile name="runme"><![CDATA[
> parent(pam,bob).
> parent(tom,bob).
> parent(tom,liz).
> parent(bob,ann).
> parent(bob,pat).
> parent(pat,jim).
>
> main :-
>     parent(X,jim),
>     format('~a is the parent of jim~n', [X]),
>     halt.
> ]]></configfile>
>   </configfiles>
>   <inputs/>
>   <outputs>
>     <data name="prolog_out" format="txt" label="prolog_out" hidden="false"/>
>   </outputs>
>   <tests>
>     <test>
>       <output name="prolog_out" value="prolog_out_sample" compare="diff" lines_diff="0"/>
>     </test>
>   </tests>
>   <help><![CDATA[
>
> **What it Does**
>
>
>
> Prolog demonstration in the ToolFactory
>
>
>
> ------
>
>
> Script::
>
>     parent(pam,bob).
>     parent(tom,bob).
>     parent(tom,liz).
>     parent(bob,ann).
>     parent(bob,pat).
>     parent(pat,jim).
>     main :-
>         parent(X,jim),
>         format('~a is the parent of jim~n', [X]),
>         halt.
>
> ]]></help>
>   <citations>
>     <citation type="doi">10.1093/bioinformatics/bts573</citation>
>   </citations>
> </tool>
> ```
>
>  Lisp too!
>  In this case, the tool takes a lisp script as input
>
>  A lisp expression was typed into the upload tool text box and saved as hello_lisp.txt ```(write-line "Hello, ToolFactory does Lisp!")```.
>  It or any other lisp script can be used as input to the SBCL Conda dependency.
>
> ```xml
> <tool name="lisp_demo" id="lisp_demo" version="0.01">
>   <!--Source in git at: https://github.com/fubar2/galaxy_tf_overlay-->
>   <!--Created by admin@galaxy.org at 04/04/2021 16:38:47 using the Galaxy Tool Factory.-->
>   <description>Runs SBCL hello world demonstration</description>
>   <requirements>
>     <requirement version="" type="package">sbcl</requirement>
>   </requirements>
>   <stdio>
>     <exit_code range="1:" level="fatal"/>
>   </stdio>
>   <version_command><![CDATA[echo "0.01"]]></version_command>
>   <command><![CDATA[sbcl
> --script
> $script
> >
> $lisp_out]]></command>
>   <inputs>
>     <param optional="false" label="SBCL lisp script to execute" help="" format="txt" multiple="false" type="data" name="script" argument="--script"/>
>   </inputs>
>   <outputs>
>     <data name="lisp_out" format="txt" label="lisp_out" hidden="false"/>
>   </outputs>
>   <tests>
>     <test>
>       <output name="lisp_out" value="lisp_out_sample" compare="diff" lines_diff="0"/>
>       <param name="script" value="script_sample"/>
>     </test>
>   </tests>
>   <help><![CDATA[
>
> **What it Does**
>
> Lisp in the ToolFactory
>
>  ]]></help>
>   <citations>
>     <citation type="doi">10.1093/bioinformatics/bts573</citation>
>   </citations>
> </tool>
> ```
{: .details}


## Command over-ride and test over-ride

- If necessary, the automatically generated `<command>` section and/or the `<test>` section can be over-written by a developer supplied text.
- This is a hybrid solution.
  - Where the generated command needs to be embellished, logic can be placed directly in the tool document this way
  - Only useful for developers comfortable with writing Galaxy XML namespace template code.
- There are two simple BWA wrappers based on a Planemo documentation advanced example
- One uses a command over-ride pasted into the appropriate text box on the ToolFactory form
- This was based on the one shown in the example it is copied from.
- It allows templating - `${reference_fasta_filename}` is replaced by the current value of the `reference_fasta_filename` parameter on the tool form.
- The pasted over-ride completely replaces the galaxyxml generated ones.

> <details-title>`bwa_test_command_override` sample - the command override</details-title>
>  ToolFactory command over-ride section adapted from the Planemo BWA example.
>
> ```
>  ## Build reference
> #set $reference_fasta_filename = "localref.fa"
> ln -s "${ref_file}" "${reference_fasta_filename}" ;
> bwa index -a is "${reference_fasta_filename}" ;
> bwa mem -t "2" -v 1 "${reference_fasta_filename}" "${fastq_input1}"  | samtools view -Sb - > temporary_bam_file.bam ;
> samtools sort -o "${bwa_test_commover_bam_output}" temporary_bam_file.bam
> ```
{: .details}

- There is another sample bwa_test tool that achieves the same results using a bash script.
- It does not need a command over-ride but is more typing because three positional parameters are named for readability.
- Bash is probably more familiar to many ToolFactory users than mako/cheetah templating.
- The effects of templating the command line can usually be achieved in a script, at the expense of needing to script the handling of parameters. It's just logic.
- The `<test>` section allows writing more useful tests than the automatically generated one, or for a real test when using collections as discussed above.
- At present, other embedded logic such as filters in output tags cannot be over-ridden.
  - These and other extensions are welcomed as pull requests if there is a need for them.
  - But if a tool needs them, it probably needs an experienced tool developer.

> <details-title>`bwa_test_toolfactory_positional_bash` sample alternative.</details-title>
> ToolFactory form bash script to replace above command over-ride section:
>
> ```
> REFFILE=$1
> FASTQ=$2
> BAMOUT=$3
> ln -s "$REFFILE" "refalias"
> bwa index -a is "refalias"
> bwa mem -t "2" -v 1 "refalias" "$FASTQ"  | samtools view -Sb - > temporary_bam_file.bam
> samtools sort -o "$BAMOUT" temporary_bam_file.bam
> ```
{: .details}


- Most users may never need to use the command over-ride option to access templating for wrapping their own scripts where they control the command line interpretation.
- Where the generated test is not sufficient, a hand written one can be substituted. Not needed for the simple BWA example.
- Test or command over-rides are likely to be edge cases more suited to the alternate, more powerful tools.

---

# Limits and workarounds

- The ToolFactory is an automated, form based code generator.
- A generator can replace manual editing by a skilled developer only in relatively constrained, simple cases.
- These are common enough in the daily work of most data intensive scientific fields to make a tool generator worth keeping handy.
- For simple scripts and appropriate Conda packages, it's potentially very useful.
- It is not hard to imagine using a Python wrapper to finesse some aspects of more complex tools just as bash was used in the `planemo lint` example.
- Logic in the `<command>` section can probably always be replaced by equivalent code in a script at the cost of extra work compared to templating.
- Other aspects of tool logic such as output filters based on other parameter values can only be implemented in the wrapper document, not in a tool script.
   - The ToolFactory relies on galaxyxml, so those kinds of extensions to galaxyxml will permit extensions to the ToolFactory. Pull requests are welcomed.
- The ToolFactory appliance is a convenient and efficient way to create and maintain simple Galaxy tools from simple working scripts.
- Tools can have command-override and test-override pasted in as in one of the BWA samples.
  - This can work around some of the current limitations on automation.
  - If the package requires that kind of complexity, it might be better to prepare the wrapper manually.

---

# Notes on common issues

## My Rscript generates a strange R error on STDOUT about an invalid operation on a closure called 'args' ?

- Did your code declare the `args vector` with something like `args = commandArgs(trailingOnly=TRUE)` before it tried to access args[1] ?
- See the plotter tool for a sample

## I want to use a collection for outputs but it always passes the test even when the script fails. What gives?

- Collections are tricky for generating tests.
  - The contents appear only after the tool has been run and even then may vary with settings.
- A manual test override is currently the only way to test collections properly.
- Automation is hard. If you can help, pull requests are welcomed.
- Until it's automated, please take a look at the plotter sample.
- It is recommended that you modify the test over-ride that appears in that sample form. Substitute one
or more of the file names you expect to see after the collection is filled by your new tool for the `<element.../>` used in the plotter sample's tool test.

## Only one ToolFactory job runs at a time. Why doesn't the server allow more than one at once?

- When a new dependency is being installed in the Conda repository, there is no locking to prevent a second process from overwriting or otherwise
interfering with it's own independent repository update. The result is not pretty.
- Allowing two tests to run at once has proven to be unstable so the job queue is currently limited to one.

## The tool I just generated is not in the tool menu?

- Did you refresh your browser's Galaxy page? Click on the "Analyse data" or "home" or the left side of the masthead to do that, then open the "ToolFactory Generated Tools" submenu.
It should be there if the XML for the new tool appeared without complaint. Please export and send me a copy of the history so I can see what's going on?


# Appendices - material of potential interest for those interested in details

## ToolFactory functional test workflows

- There are two workflows supplied in the Appliance.
- One will make and install all the test tools and runs pretty fast.
- The other will do the same but also test each one.
  - It is the Appliance functional test and a good stress test for your installation if you need one.
  - It will take a long time to run because there is a deliberate bottleneck as only one planemo process runs at a time.
  - This is necessary to avoid damage to the Planemo Conda installation.
    - When two or more processes try to install new dependencies, Bad Things Happen.
    - Conda does a pretty good job of recovering if it can but it is not pretty. Best avoided.
  - Planemo takes a while because it is doing a lot. Planemo test builds a fresh Galaxy first, and it is run twice - first to generate the test outputs then to do the real test.


## Why is the form so complex?

> <details-title>Summary: details needed and how they are used to generate a new tool</details-title>
>
> #### What information is needed to generate a tool ?
>
> - The code generator requires enough detail to be able to create the appropriate command line
> template to call the script or executable and pass the required file paths and other settings correctly.
> - Small input samples and default settings are used to construct a test for the newly generated tool. These should be known to work with the script, having been used to debug
> the script on the command line.
> - Upload the samples to the current history before
> starting a new tool in the ToolFactory. No tool will be generated without sample inputs. This test becomes part of the XML and of the toolshed archive.
> - The outputs from running the script during the first planemo run become sample outputs to be compared with test outputs in the archive.
>
> - In addition to an ID and name, a tool may have any combination of:
>
>     - Multiple dependencies. Conda is currently supported.
>     - Interpreters such as python, r-base and perl are typically used for ToolFactory tools.
>     - System utilities such as bash and sed can be used.
>     - Argparse (named) or positional (ordered) style parameter passing at tool execution time depending on the script requirements. Positional works well for bash scripts with only a handful of parameters. Argparse is preferred for clarity.
>     - Unlimited individual input data files to be selected from the user's history.
>     - Unlimited individual output files to be written to the user's history, paths determined at tool execution.
>     - Unlimited additional command line parameters that the user can control on the new tool form.
>     - an (optional) script to execute. Running a script to call an executable using parameters passed from the user can be useful to overcome some limitations of the ToolFactory for more complex tools.
>
> - Many of these generate parameter input boxes and history data selects on the new tool form.
> - Metadata about command line formatting together with text strings for the form seen by the user are needed.
>
> - Many of these are components of the generated command line template.
> - This can be seen in the new tool XML. Galaxy file paths for the script are only determined at generated tool execution. The generated template ensures that these are correct.
>
> #### The Galaxy UI imposes additional limits
>
> - The ToolFactory has limited flexibility and works best for simple tools.
> - Even then, the form becomes complicated as more parameters are added.
> - Tools can have unlimited numbers of some items, including input files, output files, citations and user parameters.
> - Each one has half a dozen metadata or text details. Galaxy form repeats are used for those.
> - As more repeats are added, the Galaxy UI becomes increasingly unwieldy.
> - In theory, the Toolfactory can potentially generate very complicated tools with large numbers if inputs, outputs and user modifiable parameters.
> - Great patience would be required.
> - That is why manual methods are likely more productive for complicated requirements.
>
>
{: .details}


## Motivation

- Uptake of Galaxy in new quantitative scientific fields requiring complex computing for analyses is arguably rate-limited early on by the availability of domain specific tools.
   - many scientists routinely write their own analysis code in quantitative disciplines - probably more commonly than in biological domains where Galaxy started.
   - lowering the barriers to those scientists generating their own new tools may speed up adoption of Galaxy in their domains.
- Galaxy tool wrapping has a well established and growing range of project supported infrastructure.
- Much complex Galaxy tool logic is embedded in the tool document namespace.
  - Some document logic cannot be easily automated. This includes output filters and other constructs that are hard to generate automatically.
  - Being in the namespace saves substantial effort compared to the alternative of writing the command  section logic in a script, as provided by the ToolFactory.
  - That extra effort is needed to pass those parameters on the command line and then to parse them in the script so the logic can be implemented.
- Manual templating is far more efficient of developer time and effort, particularly as conditional parameter and tool complexities grow.
  - It is widely preferred by dedicated developers.
  - It is the only way to satisfy some complex tool requirements.
    - An example is an output filter making an output datatype conditional on another parameter - that requires manual code in the output parameter.
  - The syntax requires some getting used to for newcomers.
  - Errors are not always pleasant or convenient to debug in the template.
- The ToolFactory may make it easier for some developers new to Galaxy to begin creating the tools needed for scientists from their new domain.
  - some may prefer a GUI tool form.
  - some may prefer the persistent IDE aspect.
  - some may feel more at home in their favourite scripting language.
  - any scripting language should be capable of implementing equivalent logic.
- Negatives include limited complexity and lower efficiency for complex tools.
  - Specialised manual development tools can make the process far more efficient.
  - ToolFactory form becomes increasingly unwieldy as complexity grows with many parameters and files.
    - sections help some but it requires determination.
- May be a convenient way to learn by building simple tools before diving into the project supported infrastructure.
- It extends the range of options available for creating


