# tutorials

topic: dev
tutorial: webhooks
title: Galaxy Webhooks
url: https://training.galaxyproject.org/training-material/topics/dev/tutorials/webhooks/tutorial.html
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 79487ed9881d7ac89cd55d25f1eb6097
last_modified: 2026-05-27T09:01:48.028138
workflows_json: []
content:
In this tutorial we are going to demonstrate how to add a webhook to the tool-execution endpoint. This is the web-page that appears
after you have executed a tool. As a more useful example we are going to ask [phdcomics](http://phdcomics.com) for a random comic that we can
display to entertain our users.


At first let's create a config file that defines the name and the type of your webhook. The `name` is `phdcomics` and with the type we define the entry-points
at which we modify the Galaxy user-interface. In our case we want to display an image at the `tool` and `workflow` entry-point.
The key `activate` gives you control per-webhook to activate or deactivate it.


> <hands-on-title></hands-on-title>
>
> 1. Create a file named `config/phdcomics.yaml` with the following content:
>
>    ```yaml
>       name: phdcomics
>       type:
>         - tool
>         - workflow
>       activate: true
>    ```
{: .hands_on}


The next step is to define HTML/JS part which will control the part of the Galaxy UI. We create a new PHDComicsAppView view extended from Backbone.
Inside this view we define a simple `div`-container with a button and a placeholder for our image called `phdcomics-img`. You can add additional
functionality to your view, for example getting a new image on button click. The essential functionality however is stored getRandomComic.
The big problem with phdcomics is that there is no nice API to retrive the comics, so we need to fallback to parse the HTML pages. We decided to do this
in Python to demonstrate webhooks abilitity to call self-defined python functions. Please note the `url = galaxyRoot + 'api/webhooks/phdcomics/get_data';`, which
calls an REST endpoint defined by use in the next step. The return value of this endpoint is retrived in JS and can be displayed or modified.

> <hands-on-title></hands-on-title>
>
> 1. Create a file named `static/script.js` with the following content:
>
>    ```js
>        $(document).ready(function() {
>
>            var galaxyRoot = typeof Galaxy != 'undefined' ? Galaxy.root : '/';
>
>            var PHDComicsAppView = Backbone.View.extend({
>                el: '#phdcomics',
>
>                appTemplate: _.template(
>                    '<div id="phdcomics-header">' +
>                       '<div id="phdcomics-name">PHD Comics</div>' +
>                        '<button id="phdcomics-random">Random</button>' +
>                    '</div>' +
>                    '<div id="phdcomics-img"></div>'
>                ),
>
>                imgTemplate: _.template('<img src="<%= src %>"">'),
>
>                events: {
>                    'click #phdcomics-random': 'getRandomComic'
>                },
>
>                initialize: function() {
>                    this.render();
>                },
>
>                render: function() {
>                    this.$el.html(this.appTemplate());
>                    this.$comicImg = this.$('#phdcomics-img');
>                    this.getRandomComic();
>                    return this;
>                },
>
>                getRandomComic: function() {
>                    var me = this,
>                        url = galaxyRoot + 'api/webhooks/phdcomics/get_data';
>
>                    this.$comicImg.html($('<div/>', {
>                        id: 'phdcomics-loader'
>                    }));
>
>                    $.getJSON(url, function(data) {
>                        if (data.success) {
>                            me.renderImg(data.src);
>                        } else {
>                            console.error('[ERROR] "' + url + '":\n' + data.error);
>                        }
>                    });
>                },
>
>                renderImg: function(src) {
>                    this.$comicImg.html(this.imgTemplate({src: src}));
>                }
>            });
>
>            new PHDComicsAppView();
>        });
>    ```
{: .hands_on}


The following hands-on will define an API endpoint that is called from the JS code of your webhook.
Make sure you name the python function `main` and that all third-party requirements are installed in your Galaxy virtual environment.
Please note that the `main()` can consume `params` from your client but also the Galaxy `trans` object, which will give you access to the
entire user-object, including histories and datasets.


> <hands-on-title></hands-on-title>
>
> 1. Create a file named `helper/__init__.py` with the following content:
>
>    ```python
>    import urllib
>    import re
>    import random
>    import logging
>
>    log = logging.getLogger(__name__)
>
>
>    def main(trans, webhook, params):
>        error = ''
>        comic_src = ''
>
>        try:
>            # Third-party dependencies
>            try:
>                from bs4 import BeautifulSoup
>            except ImportError as e:
>                log.exception(e)
>                return {'success': False, 'error': str(e)}
>
>            # Get latest id
>            if 'latest_id' not in webhook.config.keys():
>                url = 'https://phdcomics.com/gradfeed.php'
>                content = urllib.urlopen(url).read()
>                soap = BeautifulSoup(content, 'html.parser')
>                pattern = '(?:https://www\.phdcomics\.com/comics\.php\?f=)(\d+)'
>                webhook.config['latest_id'] = max([
>                    int(re.search(pattern, link.text).group(1))
>                    for link in soap.find_all('link', text=re.compile(pattern))
>                ])
>
>            random_id = random.randint(1, webhook.config['latest_id'])
>            url = 'https://www.phdcomics.com/comics/archive.php?comicid=%d' % \
>                random_id
>            content = urllib.urlopen(url).read()
>            soup = BeautifulSoup(content, 'html.parser')
>            comic_img = soup.find_all('img', id='comic2')
>
>            try:
>                comic_src = comic_img[0].attrs.get('src')
>            except IndexError:
>                pattern = '<img id=comic2 name=comic2 src=([\w:\/\.]+)'
>                comic_src = re.search(pattern, content).group(1)
>
>        except Exception as e:
>            error = str(e)
>
>        return {'success': not error, 'error': error, 'src': comic_src}
>   ```
{: .hands_on}


To make your webhook appealing you can also add custom CSS which you can use in your HTML/JS code.


> <hands-on-title></hands-on-title>
>
> 1. Create a file named `static/styles.css` with the following content:
>
>    ```css
>        #phdcomics {
>            border: 1px solid #52697d;
>            text-align: center;
>            border-radius: 3px;
>            overflow: hidden;
>        }
>
>        #phdcomics-header {
>            background: #52697d;
>            border-bottom: 1px solid #52697d;
>            padding: 15px 0;
>        }
>
>        #phdcomics-name {
>            color: #fff;
>            padding-bottom: 10px;
>        }
>
>        #phdcomics-header button {
>            color: #fff;
>            font-size: 14px;
>            background-color: #768fa5;
>            border: none;
>            border-radius: 7px;
>            box-shadow: 0 5px #5c768c;
>            padding: 5px 10px;
>        }
>
>        #phdcomics-header button:focus {
>            outline: 0;
>        }
>
>        #phdcomics-header button:hover {
>            background-color: #67839b;
>        }
>
>        #phdcomics-header button:active {
>            background-color: #67839b;
>            box-shadow: 0 0 #5c768c;
>            transform: translateY(5px);
>        }
>
>        #phdcomics-img {
>            background: #fff;
>        }
>
>        #phdcomics-img img {
>            padding: 10px;
>            max-width: 100%;
>            margin-bottom: -4px;
>        }
>
>        #phdcomics-loader {
>            border: 5px solid #f3f3f3;
>            border-top: 5px solid #52697d;
>            border-radius: 50%;
>            width: 25px;
>            height: 25px;
>            animation: spin 1.5s linear infinite;
>            margin: 15px auto;
>        }
>
>        @keyframes spin {
>            0% { transform: rotate(0deg); }
>            100% { transform: rotate(360deg); }
>        }
>    ```
{: .hands_on}


Please make sure you have activated webhooks in your `config/galaxy.yml` file by setting the `webhooks_dir` to the path in which your `phdcomics` folder is located.

> <hands-on-title></hands-on-title>
>
> 1. Submit one tool and see if your webhook is working on the tool-submit page.
>
{: .hands_on}


If successful it should look like this:

![First view](../../images/phdcomics.png)


## Conclusion

First of all, thank you for completing this tutorial. We have learned how to add webhooks to your Galaxy.


topic: dev
tutorial: data-source-integration
title: Data source integration
difficulty: intermediate
time_estimation: 10M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 8f509a1fe68725d0794b0a21f7edc380
last_modified: 2026-05-15T22:28:11.254119
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/dev/tutorials/data-source-integration/tutorial.html

content:
# Data Source Integration

An important goal of Galaxy is scalability. A major bottleneck when it comes to analysis of big data sets is the time and space it takes of copying these data sets.

![data_source_integration](../../images/data_source_integration.png)


Galaxy provides an interface such that it can communicate with other servers to get data directly into the Galaxy environment of a user without the need
of "downloading" the data. In this hands on, we will use the resource from [DoRiNA Server](http://dorina.mdc-berlin.de/) ({% cite Blin2014 %}).
but the main point about this short section is:
if you have a data source which you think is very important for your research with Galaxy let us know!

> <hands-on-title>Hands on!</hands-on-title>
>
> 1. Create a new history called "doRiNA"
> 2. Go to Get Data::doRiNA search
> 3. Choose hg19 from the drop-down list -> Search Database
> 4. Leave everything as is and choose from the Regulators (set A) drop-down list "hsa-let-7astar-CLASH" -> Search doRiNA
> 5. Use the "Send to Galaxy" button
> 6. Notice the new History Item
{: .hands_on}

You can see the [tutorial section ](http://dorina.mdc-berlin.de/tutorials) of the DoRiNA website for more detailed examples. That was very easy for all of you! If you want your database of choice to be accessible as easy as this **let us know**!


topic: climate
tutorial: fates
title: Functionally Assembled Terrestrial Ecosystem Simulator (FATES)
url: https://training.galaxyproject.org/training-material/topics/climate/tutorials/fates/tutorial.html
difficulty: intermediate
time_estimation: 4H
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 095c56cab675900a7bda7477fe8d1e05
last_modified: 2026-05-27T09:01:47.875158
zenodo_link: https://doi.org/10.5281/zenodo.4108341
workflows_json: []
content:
Terrestrial ecosystem models have been widely used to study the impact of climate changes on vegetation and terrestrial biogeochemical cycles in climate modelling community. They are also more and more applied in ecological studies to help ecologists to better understand the processes. But the technical challenges are still too high for most of the ecologists to use them. This practical aims at familiarizing you (especially ecologists) with running a terrestrial ecosystem model (i.e., CLM-FATES) at site-level in Galaxy and analyzing the model results.
It will also teach you on how to create Galaxy workflow for your site-level CLM-FATES simulations to make your research fully reproducible. We hope this tutorial will promote the use of CLM-FATES and other terrestrial ecosystem models by a broader community.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
> FATES is the “Functionally Assembled Terrestrial Ecosystem Simulator”, which is a vegetation demographic model ({% cite Fisher2017 %}).
> FATES needs what we call a "Host Land Model" (HLM) to run and in this tutorial
> we will be using the [Community Land Model](http://www.cesm.ucar.edu/models/clm/)
> of the [Community Terrestrial Systems Model](https://github.com/ESCOMP/CTSM) (CLM-CTSM).
> FATES was derived from the CLM Ecosystem Demography model (CLM(ED)), which was documented in
> {% cite Fisher2015 %} and {% cite Koven2020 %}.
> And this technical note was first published as an appendix to [that paper](https://pdfs.semanticscholar.org/396c/b9f172cb681421ed78325a2237bfb428eece.pdf).
> The [FATES documentation](https://fates-docs.readthedocs.io/en/latest/index.html) will provide some more insight on FATES too.
>
{:  .comment}

# Get CLM-FATES input data

Preparing CLM-FATES input data is out of scope for this tutorial. We assume the input data tarball contains the following folders:

```
atm   cpl   lnd   share
```

Each sub-folder will then contain all the necessary inputs for running your CLM-FATES case. For instance, 'atm' contains all the meteorological forcing data for running CLM-FATES. 'lnd' contains the data required to describe surface conditions (e.g., soil depth) for the model. More details about the model inputdata can be found in [CLM and FATES documentation](https://escomp.github.io/ctsm-docs/versions/master/html/tech_note/Ecosystem/CLM50_Tech_Note_Ecosystem.html#model-input-requirements).
For the purpose of this tutorial, input data for a single point location (ALP1) on the Norwegian alpine tundra ecosystem (Latitude: 61.0243N, Longitude: 8.12343E, Elevation: 1208 m) has been prepared and is ready to use. This is a site included in the modelling platform developed under [EMERALD project] (https://www.mn.uio.no/geo/english/research/projects/emerald/). More details about the sites can be found in {% cite Klanderud2015 %} and  {% cite Vandvik2020 %}

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *fates*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the [input data](https://doi.org/10.5281/zenodo.4108341) and the [restart dataset from Zenodo](https://doi.org/10.5281/zenodo.4126404)
>    or from the shared data library. Restart dataset will be used if you want to initialize the model from exisiting experiments rather than running the model from a cold start to shorten spin-up time needed for the model.
>
>    ```
>    https://zenodo.org/record/4108341/files/inputdata_version2.0.0_ALP1.tar
>    https://zenodo.org/record/4126404/files/CTSM_FATES-EMERALD_version2.0.0_ALP1_restart_2300-01-01.tar
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check the datatype (for both files) is **tar**
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Rename {% icon galaxy-pencil %} datasets
>      - Dataset names are the full URL, but this is not very nice to work with, and can even give errors for some tools
>      - It is good practice to change the dataset names to something more meaningful and without any special characters
>        -  E.g. by stripping off the beginning of the URL
>      - Example: rename `https://zenodo.org/record/4108341/files/inputdata_version2.0.0_ALP1.tar` to `inputdata_version2.0.0_ALP1.tar`
>      - Do the same for the other dataset
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Setting up a CLM-FATES simulation

We will be using the CTSM/FATES-EMERALD Galaxy tool.This tool is based on the version of [CLM-FATES](https://github.com/NordicESMhub/ctsm/tree/fates_emerald_api) that have been adapted to run at the sites included in the EMERALD project. More details about this model version can be found in [README_fates_emerald_api](https://github.com/NordicESMhub/ctsm/blob/fates_emerald_api/README_fates_emerald_api)

> <comment-title>Tip: Finding your tool</comment-title>
>
> Different Galaxy servers may have tools available under different sections, therefore it is often useful to use the **search bar** at the top of the tool panel to find your tool.
>
> Additionally different servers may have multiple, similarly named tools which accomplish similar functions. When following tutorials, you should use precisely the tools that they describe. For real analyses, however, you will need to search among the various options to find the one that works for you.
>
{: .comment}

> <comment-title>Tip: Pre-selected tool parameters</comment-title>
>
> When selecting a tool, Galaxy will pre-fill the tool parameters, selecting the first dataset with the corresponding type in your history.
> Be aware that very often, the default pre-selection is incorrect and do not correspond to the required dataset.
> So always check and update accordingly the tool parameters!
>
{: .comment}

> <hands-on-title>Creating a new CTSM/FATES-EMERALD case</hands-on-title>
>
> 1. {% tool [CTSM/FATES-EMERALD](toolshed.g2.bx.psu.edu/repos/climate/ctsm_fates/ctsm_fates/2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"inputdata for running FATES EMERALD"*: `inputdata_version2.0.0_ALP1.tar` file from your history
>    - *"Name of your case"*: ALP1_exp
>    - In section *"Customize the model run period"*:
>        - {% icon param-select %} *"Determines the model run initialization type:*: `hybrid`
>          - *"Reference case for hybrid or branch runs"*: `ALP1_refcase`
>          - *"Reference date for hybrid or branch runs (yyyy-mm-dd)"*: `2300-01-01`
>          - *"Run start date (yyyy-mm-dd). Only used for startup or hybrid runs"*: `0001-01-01`
>          - {% icon param-file %} *"Restart for running FATES EMERALD"*: `CTSM_FATES-EMERALD_version2.0.0_ALP1_restart_2300-01-01.tar`
>        - *"Provides a numerical count for STOP_OPTION"*: `5`
>        - *"Sets the run length along with STOP_N and STOP_DATE"*: `nyears`
>
>    > <comment-title>Startup versus Hybrid</comment-title>
>    >
>    >  When using **startup**, the FATES model will start from some arbitrary baseline state that is not linked to any previous run.
>    > Startup runs are typically initialized using a start date of 0001-01-01 except if you change it (start date option).
>    > For any scientific study, starting from an arbitraty baseline state implies you would need to run the model for a long period (between 100 and 200 years)
>    > before being able to use the model outputs. For this reason, we usually make a first simulation (spin-up) in **startup** mode and reuse this case as a baseline
>    > for our scientific study. We then use **hybrid** type and give additional inputs (restart files) to our simulation case. It is then important to specify the dates
>    > of your restart files. This is what we do in this tutorial.
>    >
>    {: .comment}
>
> 2. Check that the datatype {% icon galaxy-pencil %} of your outputs (history file) is **[netcdf](https://en.wikipedia.org/wiki/NetCDF)**
>      - If this is not the case, please change the datatype now
>
>    > <comment-title>About CLM-FATES history files</comment-title>
>    > All the CLM-FATES history files are organized in a collection.
>    >
>    {: .comment}
>
>    > <comment-title>About datatypes</comment-title>
>    >
>    > All the history files contain gridded data values written at specified times during the model run.
>    > Depending on the length of your simulation, you may have one or more history files that you can recognize from their names:
>    >`ALP1_exp.clm2.h0.yyyy-mm-dd-sssss.nc` (for non-monthly history files).
>    > Datatypes are, by default, automatically guessed. Here, as the prefix is `.nc`, the format is not always recognized as `netcdf` files.
>    > To cope with that, one can change the datatype manually, as shown below.
>    {: .comment}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 3. **Rename** {% icon galaxy-pencil %} the output dataset (history file) to `ALP1_exp.nc`
>
>    Our FATES model has run for 5 years only, so we get a single output file. As previously, we recommend
>    to rename all netCDF files so that they do not contain any special characters or dots (except for the file extension) or slashes. Some tools, in
>    particular Panoply, won't be able to recognize your file if not named properly.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. {% tool [NetCDF xarray Metadata Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1) %}  to get metadata information for CLM-FATES netCDF outputs:
>      - {% icon param-file %} *"Netcdf file"*: `ALP1_exp.nc`
>
> 5. **Inspect** {% icon galaxy-eye %} the generated output files
>      - Identify which variables would provide you some insights about canopy transpiration.
>
>    > <question-title></question-title>
>    >
>    > 1. What are the short names of the relevant variables? Which one will you pick if you want a result in **mm/s**?
>    > 2. What are the dimensions of these variables?
>    >
>    > > <solution-title></solution-title>
>    > > 1. **FCTR** is the canopy transpiration in W/m^2 and **QVEGT** is in mm/s. Therefore, we would select the latter.
>    > > 2. These variables are stored as a function of time and lndgrid and since we have only one grid cell, lngrid=1, hence the time series.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Quick visualization with Panoply

## Opening up Panoply

> <hands-on-title>Launch Panoply</hands-on-title>
>
>  [Panoply](https://www.giss.nasa.gov/tools/panoply/) plots geo-referenced and other arrays from netCDF and is available as a Galaxy interactive environment and may not be available on all Galaxy servers.
>
> > <tip-title>Only on UseGalaxy.eu</tip-title>
> > Currently Panoply in Galaxy is available on useGalaxy.eu instance, on the "Interactive tools" tool panel section or,
> > as all interactive tools, from the dedicated useGalaxy.eu subdomain: [live.useGalaxy.eu](https://live.usegalaxy.eu).
> > You may have to login again to [live.usegalaxy.eu](https://live.usegalaxy.eu) (use the same username and password than on other useGalaxy.eu subdomains)
> > and switch to the correct history.
> >
> > You can access the tool by clicking [here to launch it on EU](https://live.usegalaxy.eu/?tool_id=interactive_tool_panoply)
> {: .tip}
>
> 1. Open the {% tool [Panoply](interactive_tool_panoply) %}
> 2. Check **ALP1_exp.nc** dataset selected in the netcdf input field
> 3. Click Run Tool
> 4. The tool will start running and will stay running permanently
> 5. Click on the "User" menu at the top and go to "Active Interactive Tools" and locate the Panoply instance you started.
> 6. Click on your Panoply instance
>    ![Panoply dataset selection](../../images/select_dataset.png "Select dataset")
> 7. Click on **ALP1_exp.nc** dataset
{: .hands_on}

## Inspect metadata

> <hands-on-title>Inspect dataset</hands-on-title>
>
> 1. Inspect dataset content
>
>    Here you can look at the dataset (`ALP1_exp.nc`) and related variables (FSDS, FSA, AREA_TREE, BIOMASS_CANOPY, etc.)
>
>    > <question-title></question-title>
>    >
>    > 1. What is the long name of **MORTALITY**?
>    > 2. What is its physical unit?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Rate of total mortality per PFT (Plat functional types)
>    > > 2. indiv/ha/yr
>    > {: .solution}
>    {: .question}
>
>
> 2. Plot the total carbon in live plant leaves (**LEAFC**)
>
>    Cutomize your plot and save it as **png** file in the output folder. Remember that
>    if you do not save in the output folder, your plot will get lost.
>    > <question-title></question-title>
>    > 1. Can you observe any pattern? Does it make any sense?
>    >
>    > > <solution-title></solution-title>
>    > > 1. We can clearly see a seasonal cycle.
>    > > ![Panoply LEAFC timeseries](../../images/panoply_LEAFC_ALP1_exp.png "LEAFC")
>    > {: .solution}
>    {: .question}
>
>
> 3. Plot the rate of total mortality per PFT (MORTALITY)
>
>    Select a 2D plot with time as x-axis and colored by the rate of total mortality per PFT (Plant functional type).
>    Make sure to adjust the y-axis and save your plots in the output folder (as png file).
>    > <question-title></question-title>
>    > 1. Can you observe any pattern? Does it make any sense?
>    >
>    > > <solution-title></solution-title>
>    > > 1. We can clearly see a seasonal cycle of PFT2.
>    > > ![Panoply MORTALITY per PFT](../../images/panoply_MORTALITY_ALP1_exp.png "total mortality per PFT")
>    > {: .solution}
>    {: .question}
>
>    > <comment-title>Quit Panoply properly to save your plots!</comment-title>
>    >
>    > To make sure all your plots stored in **outputs** folder  get exported to Galaxy, you need to quit panoply: **File** --> **Quit Panoply**.
>    {: .comment}
{: .hands_on}

# Using Galaxy tools for analysing your CLM-FATES simulation

Panoply is a great tool for exploring the results of your simulations but what we would like is to automate the generation of the plots
so that we can reuse it for any simulations.

> <hands-on-title>Select and plot <b>LEAFC</b></hands-on-title>
>
> 1. {% tool [NetCDF xarray Selection](toolshed.g2.bx.psu.edu/repos/ecology/xarray_select/xarray_select/0.15.1) %} to select the total carbon in live plant leaves (**LEAFC**)
>      - {% icon param-file %} *"Input netcdf file"*: ALP1_exp.nc
>      - {% icon param-file %} *"Tabular of variables"*: Metadata info from `ALP1_exp.nc` (output of **NetCDF xarray Metadata Info** {% icon tool %})
>      - {% icon param-select %} *"Choose the variable to extract"*: `LEAFC`
>
> 2. **Rename** {% icon galaxy-pencil %} dataset to `NetCDF xarray Selection on ALP1_exp.nc`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. {% tool [Replace parts of text ](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/1.1.3) %} to clean date column for plotting:
>      - {% icon param-file %} *"File to process"*: NetCDF xarray Selection on ALP1_exp.nc
>      - {% icon param-text %} *"Find pattern"*: `00:00:00`
>      - *"Find-Pattern is a regular expression"*: `No`
>      - *"Replace all occurences of the pattern"*: `Yes`
>      - *"Case-Insensitive search"*: `No`
>      - *"Find whole-words"*: `Yes`
>      - *"Ignore first line"*: `Yes`
>      - {% icon param-select %} *"Find and Replace text in"*: `entire line`
>
> 4. **Rename** {% icon galaxy-pencil %} dataset to `LEAFC_clean.tabular`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. **Scatterplot w ggplot2**  {% icon tool %} to plot the total carbon in live plant leaves (**LEAFC**):
>      - {% icon param-file %} *"Input in tabular format"*: `LEAFC_clean.tabular`
>      - *"Column to plot on x-axis"*: 1
>      - *"Column to plot on y-axis"*: 4
>      - *"Plot title"*: `Total carbon in live plant leaves`
>      - *"Label for x axis"*: `Time`
>      - *"Label for y axis"*: `LEAFC (kgC ha-1)`
>      - In `Advanced Options`
>          - {% icon param-select %} *"Type of plot"*: `Points and Lines`
>      - In `Output options`
>          - *"width of output"*:`19.0`
>          - *"height of output"*: `5.0`
>
>
> 6. **View** {% icon galaxy-eye%} the resulting plot:
>
>    ![Snapshot of LEAFC resulting plot](../../images/LEAFC_ALP1_exp_ggplot.png)
>
{: .hands_on}

# Convert your analysis history into a Galaxy workflow

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. Go to the **History Options menu**  {% icon galaxy-gear %} menu
>    - Select the **Extract Workflow** option.
>    - Remove any unwanted steps, in particular all steps with Panoply as we do not want to have interactive tools in our automated workflow..
>
> 2. **Rename** the workflow to something descriptive
>    - For example: `CLM-FATES_ ALP1 simulation (5 years)`.
>    - If there are any steps that shouldn't be included in the workflow, you can **uncheck** them.
>
> 3. Click "Create Workflow"
>    - Click on "edit" and check your workflow
>    - Check all the steps
>
{: .hands_on}

# Change your CLM-FATES case and rerun your workflow

We would like to run a CLM-FATES case where the atmospheric Carbon Dioxyde Concentration (CO2) is increased by a factor of 4.

> <hands-on-title>Compare the two simulations</hands-on-title>
>
>  Using the results from your two CLM-FATES simulations and the generated plots, assess the impact
>  of an increase in the atmosperhic CO2 on the outputs of the model.
>
> 1. Open the **workflow editor**
>
>    {% snippet faqs/galaxy/workflows_edit.md %}
>
> 2. Edit your workflow and customize it to run your new CO2 experiment. For this you would need to:
>      - In "Advanced customization", change "Atmospheric CO2 molar ratio (by volume) only used when co2_type==constant (umol/mol)" from 367.0 to 1468.0.
>      - Add an extra step to extract the first history file from the history collection: {% tool [Extract Dataset](__EXTRACT_DATASET__) %}
>        and make sure to select "netcdf" in the change datatype field.
>      - Generate the corresponding plot.
>    The final workflow would be similar to the one shown below:
>
>    ![Snapshot of FATES workflow](../../images/fates_workflow.png "FATES workflow")
>
>    > <question-title></question-title>
>    > 1. Is the model response to this significant increase of atmospheric CO2 what you expected?
>    >   Justify your answer.
>    > 2. Is the current workflow (in particular the variables selected for the plots) the best choice?
>    >   What changes/additions would you recommend?
>    >
>    > > <solution-title></solution-title>
>    > > 1. Running 5 years is already sufficient to highlight significant changes.
>    > > ![LEAFC 4xCO2](../../images/LEAFC_ALP1_4CO2_ggplot.png)
>    > > 2. Many suggestions can be given here. One simple addition can be the generation of plots where
>    > > both simulations are represented on the same plot.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

{% snippet faqs/galaxy/histories_sharing.md %}

> <hands-on-title>Share history</hands-on-title>
>
> 1. Share your history with your neighbour (ask for his/her galaxy username).
> 2. Find the history shared by your neighbour. Histories shared with specific users can be accessed by those users under their top masthead "User" menu under `Histories shared with me`.
{: .hands_on}


> <comment-title>Publish your history to https://workflowhub.eu/</comment-title>
> One step further is to share your workflow on [https://workflowhub.eu](https://workflowhub.eu) where it
> will be stored in a Galaxy workflow format as well as in [Common Workflow Language](https://www.commonwl.org/).
> It provides standardised workflow identifiers and descriptions needed for workflow discovery, reuse, preservation, interoperability and monitoring and metadata harvesting using standard protocols.
> Please note that [https://workflowhub.eu](https://workflowhub.eu) is still under active development.
{:  .comment}

# Conclusion



We have learnt to run single-point simulations with FATES-CLM and generate workflows for multi-site scenarios.

tools_json:
["CTSM/FATES-EMERALD", "toolshed.g2.bx.psu.edu/repos/climate/ctsm_fates/ctsm_fates/2.0.1", "NetCDF xarray Metadata Info", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1", "Panoply", "interactive_tool_panoply", "NetCDF xarray Selection", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_select/xarray_select/0.15.1", "Replace parts of text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/1.1.3", "Extract Dataset", "__EXTRACT_DATASET__"]


topic: climate
tutorial: jupytergis_collaboration
title: Collaboration with JupyterGIS
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 51dd7c6791d3af63dea828eb15edc59e
last_modified: 2026-05-27T09:01:47.875373
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/jupytergis_collaboration/tutorial.html

content:
Welcome to the JupyterGIS collaborative features tutorial. JupyterGIS enables the seamless sharing of notebooks and GIS files, allowing teams—including GIS specialists, data analysts,
and other experts—to collaborate on spatial projects in a shared environment. This guide will provide the tools and steps needed to collaborate effectively,
using features like real-time editing, cursor tracking, follow mode, and annotations.

> <comment-title>Motivation</comment-title>
> Collaboration is at the heart of effective GIS projects. Teams often include members with diverse backgrounds, skills, and areas of expertise. 
> Without robust collaborative tools,
> it can become challenging to share insights, make real-time decisions, and maintain consistency across project workflows. JupyterGIS simplifies 
> collaboration by providing real-time editing, annotations, and interactive features that allow teams to seamlessly integrate their work.
> 
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will learn about:
>
>
> 1. TOC
> {:toc}
{: .agenda}


> <tip-title>Simulating Multi-User Environments</tip-title>
>
> The JupyterGIS collaborative training material is designed to help users explore geospatial workflows in a collaborative Jupyter environment. This training is intended to be used in
> pairs or small teams, even when following a self-paced approach. To simulate real-time collaboration, we recommend using an additional browser window or an anonymous session to represent
> another user. This setup allows participants to better understand multi-user interactions, shared geospatial analysis, and version control within Jupyter-based environments.
{: .tip}

# Launching Your Collaborative Session

In this section, we will learn how to launch a collaborative session in JupyterGIS and invite collaborators to join. This section of the tutorial is required to be completed by only one user (the lead user) in the session.

> <hands-on-title>Starting a Collaborative Session</hands-on-title>
>
> 1. Open your browser and navigate to [https://climate.usegalaxy.eu/](https://climate.usegalaxy.eu/), then log in or register for an account if you haven't already, to access the JupyterGIS platform.
> 2. On the left sidebar, click **Tools**.
> 3. In the search box, type **JupyterGIS**.
> 4. Click **Interactive JupyterGIS Notebook** and then click the **Run Tool** button.
>     ![Run Tool](../../images/jupytergis_collaboration/run_tool.png)
> 5. Click on **Interactive Tools** on the left sidebar.
> 6. Wait a few minutes until the job status changes to “running.”
> 7. Click on the **Jupyter Interactive GIS Tool** link that appears to open JupyterLab.
>     ![Start JGIS](../../images/jupytergis_collaboration/start_jgis.png)
> 8. Scroll down to the **Other** section and click **GIS File** to open a blank canvas for your project
>     ![New GIS File](../../images/jupytergis_collaboration/new_gis_file.png)
> 9. Notice that you are given an anonymous username, which you can see in the upper right corner. Every user in the project will be assigned an anonymous username.
>     ![Username](../../images/jupytergis_collaboration/username.png)
> 10. Click on the **Share** button in your interface in the upper right corner, then click on the **Copy Link** button to invite collaborators to your session.
>     ![Share](../../images/jupytergis_collaboration/share.png)
> 11. When your colleagues join using the link, their usernames appear in the top right corner. This lets you know exactly who is working on the document. There are two more collaborators in the session in the example below.
>     ![Shared Users](../../images/jupytergis_collaboration/shared_users.png)
{: .hands_on}


# Real-Time Collaboration on a GIS File

In this section, we will explore how to collaborate on a GIS file in real time. The remaining sections of the tutorial are required to be completed by both the lead and the collaborating users in the session.

When you make changes to your GIS file such as adding a new layer, or updating the layer appearance, the changes are reflected for all collaborators in your session in real time.

> <hands-on-title>Adding and Editing Layers</hands-on-title>
>
> 1. As the lead user, open the layer browser and select **OpenStreetMap.Mapnik**. Notice that all users in the session can see the new layer in real time.
> 2. Ask one of the collaborators to add the Libraries in France shapefile layer using the add shapefile layer menu 
> by clicking **+** -> **New Vector Layer** → **New Shapefile Layer** -> pasting the provided URL.
> 
> ```
> https://public.opendatasoft.com/api/explore/v2.1/catalog/datasets/osm-france-library/exports/shp
> ```
> 
> 
>
{: .hands_on}


{% include _includes/youtube.html id="3FbvYg3G9Gk" title="JupyterGIS tutorial show how to add layers" %}


> <question-title>Adding a New Layer</question-title>
> 
> 1. As the lead user, add the World Administrative Boundaries layer using the add shapefile layer menu.
> ```
>    https://public.opendatasoft.com/api/explore/v2.1/catalog/datasets/world-administrative-boundaries/exports/shp
> ```
> 3. Change the fill color to yellow.
> 4. Ask one of the collaborators to set the opacity to 0.2.
>
> > <solution-title></solution-title>
> >
> > - You can add the layer by clicking **+** → **New Vector Layer** → **New Shapefile Layer** → pasting the provided URL.
> > - To change the fill color, click on the layer in the left sidebar, then select **Edit Symbology** and choose yellow for the fill color. You can check the [Getting Started with JupyterGIS](https://jupytergis.readthedocs.io/en/latest/user_guide/tutorials/01-intro/index.html) tutorial for more details on how to customize the layer appearance.
> > - To set the opacity to 0.2, the collaborating user can click on the layer in the left sidebar, then adjust the opacity slider to 0.2 on the right sidebar, then click **Ok**.
> {: .solution}
{: .question}

# Tracking Collaborators with Cursors

Each user's cursor appears on the document in the same color as their icon. This feature makes it easy to see what your teammates are focused on.

> <hands-on-title>Tracking Collaborators with Cursors</hands-on-title>
>
> 1. Ask one of the collaborators to move their cursor around the document.
> 2. Spot their cursor to find out where on the map they are working.
>
> ![Cursor](../../images/jupytergis_collaboration/cursor.png)
>
{: .hands_on}


# Using Follow Mode
Follow mode allows you to track another user’s activity in the document in real time. When enabled, you’ll see their actions as they navigate and edit. This feature is ideal for live demonstrations, interactive sessions, and collaborative meetings, as it lets you quickly align your view with a teammate’s actions and provide immediate feedback.

> <hands-on-title>Activating Follow Mode</hands-on-title>
>
> 1. Click on a collaborator's user icon in the upper right corner to activate the follow mode. Observe that the document will then have a frame in their assigned color.
> 2. Ask the followed user to pan and zoom the map. Notice that your screen will follow their movements in real time.
> 3. Click on the user icon again to exit follow mode.
>   
{: .hands_on}

 {% include _includes/youtube.html id="-WzV1rcPlEw" title="JupyterGIS Collaboration using the follow mode" %}

> <question-title>Using Follow Mode</question-title>
>
> - When you activate follow mode, what color appears around the document frame?
> - Ask one of the collaborators to activate follow mode on you.
> - What color do they see?
> 
> > <solution-title></solution-title>
> > 
> > - The color of the frame is the same as the color of the user icon. This indicates the user whose activity is being followed.
> {: .solution}
{: .question}

# Adding Annotations and Comments
Annotations and comments let you add notes directly on your GIS file, which makes it easier for your team to track important details, provide context, ask questions, or share insights.

> <hands-on-title>Adding Annotations and Comments</hands-on-title>
>
> 1. Right-click anywhere on your GIS file to open the context menu, then select **Add Annotation** from the menu to create an annotation. All collaborators can see the new annotation in real time.
> 2. Click on the created annotation and type a comment.
> 3. Ask one of the collaborators to open the right sidebar to view all annotations and comments in the document, and click on the middle button to locate the annotation.
> 4. Click on the user icon again to exit follow mode.
>  
{: .hands_on}

 {% include _includes/youtube.html id="QqscEokpWIA" title="JupyterGIS annotations" %}

# Collaborating on Notebooks (Optional)
Real-time collaboration in notebooks is a powerful tool for teams working on code together. It enables multiple users to write, edit, and run code simultaneously. This feature is ideal for live coding sessions, debugging, and data analysis projects.

> <tip-title>Prerequisite</tip-title>
>
> This section requires basic Python programming knowledge. If you are not familiar with Python, we recommend completing the [Introduction to Python]({% link topics/data-science/tutorials/python-basics/tutorial.md %}) tutorial first.
{: .tip}

> <hands-on-title>Real-Time Code Collaboration</hands-on-title>
>
> 1. Click on the **+** icon to open the Launcher, then select the Python kernel under **Notebook** to create a notebook. Once a notebook is created, it is automatically accessible to all collaborators—no additional sharing is needed.
> ![Notebook](../../images/jupytergis_collaboration/create_notebook.png)
> 2. Ask one of the collaborators to open the newly created notebook by clicking on the explorer button in the left sidebar, then locating and clicking on the notebook. Anyone in the session can open, edit, and run the notebooks.
> ![Notebook](../../images/jupytergis_collaboration/open_notebook.png)
> 3. As you write or execute code, every change is visible to your team instantly. Create a new code cell and write the following code. Notice that all collaborators can see the code in real time.
> ```python
>    from jupytergis import GISDocument
>    doc = GISDocument('untitled.jGIS')
>    doc
> ```
> 4. Rename the "Custom Shapefile Layer" to "libraries" in the GIS file. 
> 5. Find the layer ID of the libraries layer.
> ```python
>    layers = doc.layers
>    libraries_layer_id = None
>    for layer_id, data in layers.items():
>        if data.get("name", "").lower() == "libraries":
>            libraries_layer_id = layer_id
>            break
> ```
> 6. Ask one of the collaborators to filter all libraries that have wheelchair access. They can do this by running the following code:
> ```python
>    doc.add_filter(libraries_layer_id, 'all', 'wheelchair', '==', 'yes')
> ```
>  
{: .hands_on}

 {% include _includes/youtube.html id="H5Pkg5zOSwA" title="JupyterGIS collaboration from the notebook" %}

> <question-title></question-title>
>
> - Create a new notebook and load your GIS document.
> - In the GIS file, add a new layer with the following URL:
>    ```https://public.opendatasoft.com/api/explore/v2.1/catalog/datasets/openaq/exports/geojson```
> - Ask one of the collaborators to open the notebook, write the code to remove the air quality layer of the GIS file and run the code cell.
> 
> > <solution-title></solution-title>
> >
> > - Create a new notebook from the JupyterLab launcher (select Python kernel).
> > - Load your GIS document using the following Python code:
> >   ```python
> >   from jupytergis import GISDocument
> >   doc = GISDocument("untitled.jGIS")
> >   ```
> > - Add the air quality layer by clicking **+** → **New Vector Layer** → **Add GeoJSON Layer** → pasting the provided URL.
> > - The collaborator can add and execute the following code to list all layers:
> >   ```python
> >   doc.layers
> >   ```
> > - Then they can find the air quality layer ID (the layer with the name Custom GeoJSON Layer) and remove it using:
> >   ```python
> >   air_quality_layer_id = "your_layer_id"
> >   doc.remove_layer(air_quality_layer_id)
> >   ````
> > 
> {: .solution}
{: .question}


Congratulations! You have completed the Collaboration Features of JupyerGIS tutorial. You now have the knowledge and tools to collaborate effectively with your team on GIS files and notebooks.

If you'd like to explore more of JupyterGIS' functionality, please view our
[JupyterGIS announcement blog post](https://blog.jupyter.org/real-time-collaboration-and-collaborative-editing-for-gis-workflows-with-jupyter-and-qgis-d25dbe2832a6)
for video demos of more features.


topic: climate
tutorial: pangeo-notebook
title: Pangeo Notebook in Galaxy - Introduction to Xarray
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["pangeo", "interactive-tools"]
content_hash: e5d0d00864a486120253bd202ab19326
last_modified: 2026-05-27T09:01:47.876803
zenodo_link: https://doi.org/10.5281/zenodo.5805953
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/pangeo-notebook/tutorial.html

content:
In this tutorial, we will learn about [Xarray](https://xarray.pydata.org/), one of the most used Python library from the [Pangeo](https://pangeo.io/) ecosystem.

We will be using data from [Copernicus Atmosphere Monitoring Service](https://ads.atmosphere.copernicus.eu/)
and more precisely PM2.5 ([Particle Matter < 2.5 μm](https://en.wikipedia.org/wiki/Particulates#Size,_shape_and_solubility_matter)) 4 days forecast from December, 22 2021. Parallel data analysis with Pangeo is not covered in this tutorial.

> <comment-title>Remark</comment-title>
>
> This tutorial uses data on a regular latitude-longitude grid. More complex and irregular grids are not discussed in this tutorial. In addition,
> this tutorial is not meant to cover all the different possibilities offered by Xarrays but shows functionalities we find useful for day to day
> analysis.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Analysis

## Import Python packages

Some packages may need to be installed first. For example `cmcrameri` is missing, so we need to install it by entering the following command in a new cell of your Jupyter Notebook:

```python
pip install cmcrameri
```

Then we need to import all the necessary packages in our Jupyter Notebook.

```python
import numpy as np
import xarray as xr
import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import cmcrameri.cm as cmc
import pandas as pd
```

## Open and read metadata

The netCDF dataset can now be opened with Xarray:

```python
dset = xr.open_dataset("data/CAMS-PM2_5-20211222.netcdf")
```

Once opened, we can get metadata using `print` statement.

```python
print(dset)
```

Below is what you should get if everything goes fine.

> <code-out-title></code-out-title>
> ```
>  <xarray.Dataset>
>  Dimensions:     (longitude: 700, latitude: 400, level: 1, time: 97)
>  Coordinates:
>    * longitude   (longitude) float32 -24.95 -24.85 -24.75 ... 44.75 44.85 44.95
>    * latitude    (latitude) float32 69.95 69.85 69.75 69.65 ... 30.25 30.15 30.05
>    * level       (level) float32 0.0
>    * time        (time) timedelta64[ns] 00:00:00 01:00:00 ... 4 days 00:00:00
> Data variables:
>   pm2p5_conc  (time, level, latitude, longitude) float32 0.4202 ... 7.501
> Attributes:
>   title:        PM25 Air Pollutant FORECAST at the Surface
>   institution:  Data produced by Meteo France
>   source:       Data from ENSEMBLE model
>   history:      Model ENSEMBLE FORECAST
>   FORECAST:     Europe, 20211222+[0H_96H]
>   summary:      ENSEMBLE model hourly FORECAST of PM25 concentration at the...
>   project:      MACC-RAQ (http://macc-raq.gmes-atmosphere.eu)
> ```
{: .code-out}

> <tip-title>Command not found</tip-title>
> If you get an error with the previous command, first check the location of the input file `CAMS-PM2_5-20211222.netcdf`:
> it needs to be in the same directory as your Jupyter Notebook.
{: .tip }

We can identify 4 different sections:
1. **Dimensions**: name of dimensions and corresponding number of elements;
2. **Coordinates**: contains coordinate arrays (longitude, latitude, level and time) with their values.
3. **Data variables**: contains all the variables available in the dataset. Here, we only have one variable. For each variable, we get information on its shape and values.
4. **Attributes**: at this level, we get all the attributes of the dataset.

We can also get metadata information for each coordinate and data variables using "." followed by the coordinate or data variable name.

```python
print(dset.time)
```

> <question-title>CAM PM2.5 Dataset</question-title>
>
> What is the name of the variable for Particle matter < 2.5 μm and its physical units?
>
> > <solution-title></solution-title>
> > 1. To get metadata information from `pm2p5_conc` Data variable, we use its variable name and print it. Printing it will only print metadata, not the values.
> >      - Variable name: `mass_concentration_of_pm2p5_ambient_aerosol_in_air`
> >      - Units: `µg/m3`
> >
> > > <code-in-title>Python</code-in-title>
> > > ```python
> > > print(dset.pm2p5_conc)
> > > ```
> > {: .code-in}
> >
> > > <code-out-title></code-out-title>
> > > ```bash
> > >    <xarray.DataArray 'pm2p5_conc' (time: 97, level: 1, latitude: 400, longitude: 700)>
> > >    [27160000 values with dtype=float32]
> > >    Coordinates:
> > >      * longitude  (longitude) float32 335.0 335.1 335.2 335.4 ... 44.75 44.85 44.95
> > >      * latitude   (latitude) float32 69.95 69.85 69.75 69.65 ... 30.25 30.15 30.05
> > >      * level      (level) float32 0.0
> > >      * time       (time) timedelta64[ns] 00:00:00 01:00:00 ... 4 days 00:00:00
> > >    Attributes:
> > >        species:        PM2.5 Aerosol
> > >        units:          µg/m3
> > >        value:          hourly values
> > >        standard_name:  mass_concentration_of_pm2p5_ambient_aerosol_in_air
> > > ```
> > {: .code-out}
> {: .solution }
{: .question }

> <comment-title>Different ways to access Data variables</comment-title>
>
> To access a variable or coordinate, we can use "." or specify its name as a string between squared brackets "[" "]". For example:
>
> ```python
> print(dset['pm2p5_conc'])
> # or alternatively
> print(dset.pm2p5_conc)
> ```
>
> When we print a variable or coordinate, we do not get all the individual values but a `DataArray` that contains a lot of very useful metadata such as coordinates (if they have some), all the attributes such as the name, the physical units, etc.
>
{: .comment}
## Select / Subset from coordinates

We often want to select elements from the coordinates for instance to subset a geographical area or select specific times or a specific time range.

There are two different ways to make a selection:
- by index
- by value

### Select elements from coordinate by index

```python
print(dset.isel(time=0))
```

You should see that the coordinate `time` "disappeared" from the `Dimensions` and now the variable `pm2p5_conc` is a 3D field with longitude, latitude and level.

### Select elements from coordinates by value

When selecting elements by the value of the coordinate, we need to use the same datatype. For instance, to select an element from
`time`, we need to use `timedelta64`. The code below will give the same result as `isel(time=0)`.

```python
print(dset.sel(time=np.timedelta64(0)))
```

The output will be very similar to what we did previously when selecting from coordinates by index.

> <question-title>Select a single time for PM2.5</question-title>
>
> How to select the forecast for December, 24th 2021 at 12:00 UTC?
>
> > <solution-title></solution-title>
> > Data starts on December, 22nd 2021 at 00:00 UTC so we need to add 2 days and 12 hours to select the correct time index.
> >
> > > <code-in-title>Python</code-in-title>
> > > ```python
> > > print(dset.sel(time=(np.timedelta64(2,'D')+ np.timedelta64(12,'h'))))
> > > ```
> > {: .code-in}
> >
> > > <code-out-title></code-out-title>
> > > ```bash
> > > <xarray.Dataset>
> > > Dimensions:     (longitude: 700, latitude: 400, level: 1)
> > > Coordinates:
> > >   * longitude   (longitude) float32 -24.95 -24.85 -24.75 ... 44.75 44.85 44.95
> > >   * latitude    (latitude) float32 69.95 69.85 69.75 69.65 ... 30.25 30.15 30.05
> > >   * level       (level) float32 0.0
> > >     time        timedelta64[ns] 2 days 12:00:00
> > > Data variables:
> > >     pm2p5_conc  (level, latitude, longitude) float32 0.4499 0.4421 ... 10.71
> > > Attributes:
> > >     title:        PM25 Air Pollutant FORECAST at the Surface
> > >     institution:  Data produced by Meteo France
> > >     source:       Data from ENSEMBLE model
> > >     history:      Model ENSEMBLE FORECAST
> > >     FORECAST:     Europe, 20211222+[0H_96H]
> > >     summary:      ENSEMBLE model hourly FORECAST of PM25 concentration at the...
> > >     project:      MACC-RAQ (http://macc-raq.gmes-atmosphere.eu)
> > > ```
> > {: .code-out}
> {: .solution }
{: .question }


## Plotting

To plot a map, you need to select a variable with data on geographical coordinates (latitude, longitude). In addition, coordinates need to be sorted, and preferably in increasing order. This is not the case for the coordinate "longitude" which is given between 360 and 0.

Let's shift the longitudes by 180 degrees so that they come in the range of -180 to 180.

### Shift longitudes

We print the longitudes before and after shifting them so we can see what is happening.

```python
print(dset.longitude)
```

The longitude values are between `335.05` and `44.95` degrees.


Let's now shift the longitudes to get values between `-180`, `180` degrees.

```python
dset.coords['longitude'] = (dset['longitude'] + 180) % 360 - 180
print(dset.longitude)
```

Indeed, the longitudes have been shifted and now the values are between `-24.95` and `44.95`.

### Visualize on a map PM2.5 for December, 24th 2021 at 12:00 UTC

```python
dset.sel(time=(np.timedelta64(2,'D')+ np.timedelta64(12,'h'))).pm2p5_conc.plot()
```
We will get a figure like the one below:

![CAMS PM2.5 December, 24th 2021 at 12:00 UTC](../../images/PM2_5_default.png)


> <comment-title>What about `level`</comment-title>
> Note that in the previous plot, we did not need to select `level` because there is one value only. However, if we had more than one level, we would need to add a selection on the level before plotting
>
{: .comment}

### Customize your plot

There are many ways to customize your plots and we will only detail what we think is important for creating publication ready figures:
- Define the size of the figure
- Choose to project data on a different projection.
- Add coastline
- Set the min and max values for plotting
- Add a title, change colorbar title
- Save figure into png

```python
fig = plt.figure(1, figsize=[15,10])

# We're using cartopy to project our data.
# (see documentation on cartopy)
ax = plt.subplot(1, 1, 1, projection=ccrs.Mercator())
ax.coastlines(resolution='10m')

# We need to project our data to the new projection and for this we use `transform`.
# we set the original data projection in transform (here PlateCarree)
dset.sel(time=(np.timedelta64(2,'D') + np.timedelta64(12,'h')))['pm2p5_conc'].plot(ax=ax,
                                                                                    transform=ccrs.PlateCarree(),
                                                                                    vmin = 0, vmax = 35,
                                                                                    cmap=cmc.roma_r)
# One way to customize your title
plt.title("Copernicus Atmosphere Monitoring Service PM2.5, 2 day forecasts\n 24th December 2021 at 12:00 UTC", fontsize=18)
plt.savefig("CAMS-PM2_5-fc-20211224.png")
```

And you should get the following plot:

![Customized plot for CAMS PM2.5 December, 24th 2021 at 12:00 UTC](../../images/CAMS-PM2_5-fc-20211224.png)

### Multi-plots

Now, we will plot several times on the same figure in different sub-plots; we will not plot all the times (too many) but the first 24 forecasted values.

Firstly, we need to create a list of times and convert it to `pandas datetime` in order to make it easier to format times when plotting:

```python
list_times = np.datetime64('2021-12-22') + dset.time.sel(time=slice(np.timedelta64(0),np.timedelta64(1,'D')))
print(pd.to_datetime(list_times).strftime("%d %b %H:%S UTC"))
```

Secondly, we need to use the same plotting method as earlier, but we pass additional parameters:

- `vmin = 0`and `vmax = 35` to set the minimum and maximum values when plotting (this is useful to highlight features in your plot)
- `subplot_kws={"projection": proj_plot}` to project data on a non-default projection. See [cartopy projection](https://scitools.org.uk/cartopy/docs/v0.15/crs/projections.html) for more information about projections.
- `col='time'` because we will plot several `time`;
-  `col_wrap=4` to have a maximum of 4 plots per row. If we have more times to plot, then the next figures will be on another row.
- `robust=True` and `aspect=dset.dims["longitude"] / dset.dims["latitude"]` are additional parameters to make each subplot with a "sensible" figsize.
- `cmap=cmc.roma_r` to select a non-default and color-blind friendly colormap (see [scientific colormaps](https://www.fabiocrameri.ch/colourmaps/)).


```python
fig = plt.figure(1, figsize=[10,10])

# We're using cartopy to project our data.
# (see documentation on cartopy)
proj_plot = ccrs.Mercator()

# We need to project our data to the new projection and for this we use `transform`.
# we set the original data projection in transform (here PlateCarree)
p = dset.sel(time=slice(np.timedelta64(1,'h'),np.timedelta64(1,'D')))['pm2p5_conc'].plot(transform=ccrs.PlateCarree(),
                                                                                       vmin = 0, vmax = 35,
                                                                                       subplot_kws={"projection": proj_plot},
                                                                                       col='time', col_wrap=4,
                                                                                       robust=True,
                                                                                      aspect=dset.dims["longitude"] / dset.dims["latitude"],  # for a sensible figsize
                                                                                       cmap=cmc.roma_r)
# We have to set the map's options on our axes
for ax,i in zip(p.axes.flat,  (np.datetime64('2021-12-22') + dset.time.sel(time=slice(np.timedelta64(0),np.timedelta64(1,'D')))).values):
      ax.coastlines('10m')
      ax.set_title("CAMS PM2.5 " + pd.to_datetime(i).strftime("%d %b %H:%S UTC"), fontsize=12)
# Save your figure
plt.savefig("CAMS-PM2_5-fc-multi.png")
```
In the second part of our plot, we are going to customize each subplot (this is why we loop for each of them and get their axes) by adding:

-  `coastlines`: we pass a parameter `10m` to get coastlines with a high resolution (non-default);
- `set_title` to set a title for each subplot.

![Customized multi-plot](../../images/CAMS-PM2_5-fc-multi.png)


> <question-title>PM2.5 over Italy</question-title>
>
> Using a Multi-plot between Rome and Naples, can you tell us if the forecasted PM2.5 will increase or decrease during the first 24 hours?
>
> > <solution-title></solution-title>
> > We will select a sub-area: 11. East to 15.0 East and 40. N to 43. N. PM2.5 will increase and reach values close to 35 μm.m-3.
> > We will use `slice` to select the area and we slice latitudes with `latitude=slice(47.3, 36.5)` and not `latitude=slice(36.5, 47.3)`.
> > The reason is that when using slice, you need to specify values using the same order as in the coordinates. Latitudes are specified in
> > decreasing order for CAMS.
> >
> > > <code-in-title>Python</code-in-title>
> > > ```python
> > > fig = plt.figure(1, figsize=[10,10])
> > >
> > > # We're using cartopy to project our data.
> > > # (see documentation on cartopy)
> > > proj_plot = ccrs.Mercator()
> > >
> > > # We need to project our data to the new projection and for this we use `transform`.
> > > # we set the original data projection in transform (here PlateCarree)
> > > p = dset.sel(time=slice(np.timedelta64(1,'h'),np.timedelta64(1,'D'))).sel(latitude=slice(43., 40.),
> > >                                                                           longitude=slice(11.,15.))['pm2p5_conc'].plot(transform=ccrs.PlateCarree(),
> > >                                                                                      vmin = 0, vmax = 35,
> > >                                                                                      subplot_kws={"projection": proj_plot},
> > >                                                                                      col='time', col_wrap=4,
> > >                                                                                      robust=True,
> > >                                                                                      aspect=dset.dims["longitude"] / dset.dims["latitude"],  # for a sensible figsize
> > >                                                                                      cmap=cmc.roma_r)
> > > # We have to set the map's options on all axes
> > > for ax,i in zip(p.axes.flat,  (np.datetime64('2021-12-22') + dset.time.sel(time=slice(np.timedelta64(0),np.timedelta64(1,'D')))).values):
> > >     ax.coastlines('10m')
> > >     ax.set_title("CAMS PM2.5 " + pd.to_datetime(i).strftime("%d %b %H:%S UTC"), fontsize=12)
> > > # Save your figure
> > > plt.savefig("CAMS-PM2_5-fc-multi-Italy.png")
> > > ```
> > {: .code-in}
> >
> >  ![PM2.5 over Italy](../../images/CAMS-PM2_5-fc-multi-Italy.png)
> >
> {: .solution }
{: .question }


## How to use the **where** method

Sometimes we may want to make more complex selections with criteria on the values of a given variable and not only on its coordinates. For this purpose,  we use the `where` method. For instance, we may want to only keep PM2.5 if values are greater than 25 μm.m-3 (or any threshold you would like to choose).

### Mask values that do not meet a criteria with `Where`

```python
print(dset.where(dset['pm2p5_conc'] > 25))
```

> <comment-title>What happened?</comment-title>
> Each element of the dataset where the criteria within the `where` statement is not met, e.g. when PM2.5 <= 25, will be set to `nan`.
> You may not see any changes when printing the dataset but if you look carefuly at `pm2p5_conc` values, you will see many `nan`.
>
{: .comment}

Let's plot one time to better see what happened:

```python
######################
# Plotting with mask #
######################

fig = plt.figure(1, figsize=[15,10])

# We're using cartopy to project our data.
# (see documentation on cartopy)
ax = plt.subplot(1, 1, 1, projection=ccrs.Mercator())
ax.coastlines(resolution='10m')

# We need to project our data to the new projection and for this we use `transform`.
# we set the original data projection in transform (here PlateCarree)
dset.where(dset['pm2p5_conc'] > 25).isel(time=0)['pm2p5_conc'].plot(ax=ax,
                                                                     transform=ccrs.PlateCarree(),
                                                                     vmin = 0, vmax = 35,
                                                                     cmap=cmc.roma_r)
# One way to customize your title
plt.title("Copernicus Atmosphere Monitoring Service PM2.5, 2 day forecasts\n 24th December 2021 at 12:00 UTC\n only values > 25", fontsize=18)
plt.savefig("CAMS-PM2_5-fc-20211224-25.png")
```
![PM2.5 over Italy with threshold at 25](../../images/CAMS-PM2_5-fc-20211224-25.png)


We can then make the same multi-plot as earlier (over Italy) but with a `where` statement to mask values lower than 25 μm.m-3:

### Multi-plot over Italy using a mask

```python
fig = plt.figure(1, figsize=[10,10])

# We're using cartopy to project our data.
# (see documentation on cartopy)
proj_plot = ccrs.Mercator()

# We need to project our data to the new projection and for this we use `transform`.
# we set the original data projection in transform (here PlateCarree)
p = dset.where(dset['pm2p5_conc'] > 25).sel(time=slice(np.timedelta64(1,'h'),np.timedelta64(1,'D'))).sel(latitude=slice(43., 40.),
                                                                           longitude=slice(11.,15.))['pm2p5_conc'].plot(transform=ccrs.PlateCarree(),
                                                                                      vmin = 0, vmax = 35,
                                                                                     subplot_kws={"projection": proj_plot},
                                                                                     col='time', col_wrap=4,
                                                                                     robust=True,
                                                                                     aspect=dset.dims["longitude"] / dset.dims["latitude"],  # for a sensible figsize
                                                                                     cmap=cmc.roma_r)
# We have to set the map's options on all four axes
for ax,i in zip(p.axes.flat,  (np.datetime64('2021-12-22') + dset.time.sel(time=slice(np.timedelta64(0),np.timedelta64(1,'D')))).values):
     ax.coastlines('10m')
     ax.set_title("PM2.5 > 25 μm.m-3" + pd.to_datetime(i).strftime("%d %b %H:%S UTC"), fontsize=12)
# Save your figure
plt.savefig("CAMS-PM2_5-fc-multi-Italy-25.png")
```

![Multi-plot of PM2.5 over Italy with threshold at 25](../../images/CAMS-PM2_5-fc-multi-Italy-25.png)


## Reduction operations
We often want to compute the mean of all our datasets, or along a dimension (for instance time). If you do not pass any argument to the operation then it is done over all dimensions.

### Details on the **mean** method
When we do not specify any parameters, we get a single value.

```python
print(dset.sel(latitude=slice(43., 40.), longitude=slice(11.,15.)).mean())
```

> <code-out-title></code-out-title>
> ```bash
> <xarray.Dataset>
> Dimensions:     ()
>  Data variables:
>     pm2p5_conc  float32 9.118
>```
{: .code-out}


> <question-title>Maximum PM2.5 over Italy</question-title>
>
> What is the maximum forecasted PM2.5 value over the Rome-Naples region?
>
> > <solution-title></solution-title>
> >  We  select the same sub-area: 11. East to 15.0 East and 40. N to 43. N and compute the maximum with `max`. The maximum PM2.5 value is **59.13694382** μm.m-3 (that is rounded to **59.14**).
> >
> > > <code-in-title>Python</code-in-title>
> > > ```python
> > > dset.sel(latitude=slice(43., 40.), longitude=slice(11.,15.)).max()
> > > ```
> > {: .code-in}
> >
> > > <code-out-title></code-out-title>
> > > ```bash
> > > xarray.Dataset
> > > Dimensions:
> > > Coordinates: (0)
> > > Data variables:
> > > pm2p5_conc
> > > ()
> > > float64
> > > 59.14
> > > array(59.13694382)
> > > Attributes: (0)
> > > ```
> > {: .code-out}
> {: .solution }
{: .question }

> <question-title>Find when the maximum PM2.5 is forecasted</question-title>
>
> When is the maximum PM2.5 value forecasted?
>
> > <solution-title></solution-title>
> > We will select a sub-area: 11. East to 15.0 East and 40. N to 43. N and average over the entire selected area and search where the maximum PM2.5 value of 59.13694382 μm.m-3 is found. The maximum PM2.5 value occurs on 2021-12-22 at 20:00 UTC.
> >
> > > <code-in-title>Python</code-in-title>
> > > ```python
> > > dset_tmean = dset.sel(latitude=slice(43., 40.), longitude=slice(11.,15.)).max(dim=('latitude', 'longitude'))
> > > dset_tmean_max = dset_tmean.where(dset_tmean['pm2p5_conc'] == 59.13694382, drop=True)
> > > print(dset_tmean_max)
> > > ```
> > {: .code-in}
> >
> > > <code-out-title></code-out-title>
> > > ```bash
> > > <xarray.Dataset>
> > > Dimensions:     (time: 1, level: 1)
> > > Coordinates:
> > >   * level       (level) float32 0.0
> > >   * time        (time) timedelta64[ns] 20:00:00
> > > Data variables:
> > >     pm2p5_conc  (time, level) float32 59.14
> > > ```
> > {: .code-out}
> {: .solution }
{: .question }

> <comment-title>Pixel size when averaging</comment-title>
> We average over a relatively small area so we do not make a weighted average. Use weighted averages when averaging over the entire globe or over a large area where the pixel sizes may vary (depending on the latitude).
>
{: .comment}


## Details on the **resample** method

### 1 day Resampling

 The resampling frequency is lower than our original data, so we would need to apply a global operation on the data we group together such as mean, min, max:

```python
print(dset.resample(time='1D').mean())
```

> <code-out-title></code-out-title>
> ```bash
> <xarray.Dataset>
> Dimensions:     (time: 5, longitude: 700, latitude: 400, level: 1)
> Coordinates:
>   * time        (time) timedelta64[ns] 0 days 1 days 2 days 3 days 4 days
>   * longitude   (longitude) float32 -24.95 -24.85 -24.75 ... 44.75 44.85 44.95
>   * latitude    (latitude) float32 69.95 69.85 69.75 69.65 ... 30.25 30.15 30.05
>   * level       (level) float32 0.0
> Data variables:
>     pm2p5_conc  (time, level, latitude, longitude) float32 0.4298 ... 7.501
> ```
{: .code-out}


### 30 minute resampling

When the resampling frequency is higher than the original data, we need to indicate how to fill the gaps, for instance, interpolate and indicate which interpolation method to apply or select nearest values, etc.:

```python
print(dset.resample(time='30min').interpolate('linear'))
```

> <comment-title>Be careful when sub-sampling!</comment-title>
> Increasing the frequency of your data e.g. artificially creating data may not be scientifically relevant. Please use it carefully! Interpolating is not always scientifically relevant and sometimes you may prefer to choose a different method, like taking the nearest value for instance:
>
>    > <code-in-title>Python</code-in-title>
>    >  ```python
>    >  dset.resample(time='30min').nearest()
>    >  ```
>    {: .code-in}
>
{: .comment}



> <question-title>PM2.5 over Italy in the next 4 days</question-title>
>
> Using a Multi-plot between Rome and Naples, and making averages per day, can you tell us if forecasted PM2.5 will increase or decrease?
>
> > <solution-title></solution-title>
> > PM2.5 over Italy is overall decreasing over the next 4 forecasted days.
> >
> > > <code-in-title>Python</code-in-title>
> > > ```python
> > > fig = plt.figure(1, figsize=[10,10])
> > >
> > > # We're using cartopy to project our data.
> > > # (see documentation on cartopy)
> > > proj_plot = ccrs.Mercator()
> > >
> > > sub_dset = dset.sel(latitude=slice(43., 40.), longitude=slice(11.,15.)).resample(time='1D').mean()
> > > # We need to project our data to the new projection and for this we use `transform`.
> > > # we set the original data projection in transform (here PlateCarree)
> > > p = sub_dset['pm2p5_conc'].plot(transform=ccrs.PlateCarree(),
> > >                                 vmin = 0, vmax = 35,
> > >                                 subplot_kws={"projection": proj_plot},
> > >                                 col='time', col_wrap=5,
> > >                                 robust=True,
> > >                                 aspect=dset.dims["longitude"] / dset.dims["latitude"],  # for a sensible figsize
> > >                                 cmap=cmc.roma_r)
> > > # We have to set the map's options on all axes
> > > for ax,i in zip(p.axes.flat,  (np.datetime64('2021-12-22') + dset.time.sel(time=slice(np.timedelta64(0),np.timedelta64(1,'D')))).values):
> > >     ax.coastlines('10m')
> > >     ax.set_title("CAMS PM2.5 " + pd.to_datetime(i).strftime("%d %b %H:%S UTC"), fontsize=12)
> > > # Save your figure
> > > plt.savefig("CAMS-PM2_5-fc-multi-Italy-mean-per-day.png")
> > > ```
> > {: .code-in}
> >
> > ![Daily mean for PM2.5 over Italy](../../images/CAMS-PM2_5-fc-multi-Italy-mean-per-day.png)
> {: .solution }
{: .question }




> <comment-title>`Grouby` versus `resample`</comment-title>
>
> Use `groupby` instead of `resample` when you wish to group over a dimension that is not `time`. `groupby` is very similar to resample but can be applied to any coordinates and not only to time.
>
{: .comment}


# Conclusion


Well done! [Pangeo](https://pangeo.io/) is a fantastic community with many more resources for learning and/or contributing! Please, if you use any Python packages from the Pangeo ecosystem, do not forget to cite Pangeo {% cite Abernathey2017 %}, {% cite Abernathey2021 %}, {% cite Gentemann2021 %} and {% cite Sambasivan2021 %}!

Have a look at the [Pangeo Tutorial Gallery](https://gallery.pangeo.io/repos/pangeo-data/pangeo-tutorial-gallery/) to pick up your next Pangeo training material!


topic: climate
tutorial: climate-101
title: Getting your hands-on climate data
difficulty: intermediate
time_estimation: 1H
tools_json: ["Panoply", "interactive_tool_panoply"]
requirements_json: []
tags_json: []
content_hash: b058e603cee7538189393b99ef74b241
last_modified: 2026-05-27T09:01:47.874801
zenodo_link: https://zenodo.org/record/3776500
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/climate-101/tutorial.html

content:
The practical aims at familiarzing you with Climate Science and the terminology used by climate scientists. The target audience is not a climate scientist but
anyone interested in learning about climate.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [Getting your hands-on Climate data](https://nordicesmhub.github.io/climate-data-tutorial/).
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
> [European Copernicus Climate Change Service (C3S)](https://climate.copernicus.eu/) provide authoritative information about the past, present
> and future climate. C3S is one of the many services provided by Copernicus, the European Union's Earth Observation Programme, looking
> at our planet and its environment for the ultimate benefit of all European citizens.
> The C3S [Climate Data Store (CDS)](https://cds.climate.copernicus.eu/#!/home) provides a single point of access to a wide range of
> quality-assured climate datasets distributed in the cloud.
> Access to data from the CDS is open and unrestricted. While registration is required, it is free of charge.
> We will be using freely available datasets from the CDS, including
> observations, historical climate data records, estimates of Essential Climate Variables (ECVs) derived from Earth observations,
> global and regional climate reanalyses of past observations, seasonal forecasts and climate projections. To make access easier and faster,
> the datasets have been prepared and deposited on Zenodo. At the end of the tutorial, feel free to visit the CDS to download alternative
> datasets.
{:  .comment}

For the purpose of this tutorial, sample datasets have been created from data downloaded from [C3S](https://climate.copernicus.eu/) through
[Copernicus Climate Data Store](https://cds.climate.copernicus.eu/#!/home):
- [E-OBS daily gridded meteorological data for Europe from 1950 to present derived from in-situ observations](https://cds.climate.copernicus.eu/cdsapp#!/dataset/insitu-gridded-observations-europe?tab=overview)
- [Essential climate variables for assessment of climate variability from 1979 to present](https://cds.climate.copernicus.eu/cdsapp#!/dataset/ecv-for-climate-change?tab=overview)

To reduce the volume of data, the data resolution (in space and/or time) has been significantly reduced and/or data has been selected on sample locations (Paris, Oslo and
Freiburg). The data format may also have been changed (for instance to tabular) to ease processing.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *climate101*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://doi.org/10.5281/zenodo.3776500) or from the shared data library
>
>    ```
>    https://zenodo.org/record/3776500/files/tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv
>    https://zenodo.org/record/3776500/files/ts_cities.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check that the datatype is **tabular**
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
>    If it is not `tabular` make sure to convert it using the Galaxy built-in format converters.
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert CSV to Tabular" %}
>
> 4. Rename Datasets
>
>    As "`https://zenodo.org/record/3776500/files/tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv`" is not a beautiful name and can give errors for some tools, it is a good practice to change the dataset name by something more meaningful.
>    For example by removing `https://zenodo.org/record/3776500/files/` to obtain `tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv` and `ts_cities.csv`, respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Add a tag to the dataset corresponding to `copernicus`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# What is climate?

According to [wikipedia](https://en.wikipedia.org/wiki/Climate),
Climate is defined as the average state of everyday's weather condition **over a period of 30 years**. It is measured by assessing
the patterns of variation in temperature, humidity, atmospheric pressure, wind, precipitation, atmospheric particle count and
other meteorological variables in a given region over a long period of time (usually 20 or 30 years).
Climate differs from weather, in that weather only describes the short-term conditions of these variables in a given region.


## Climate versus Weather

Quantities that climate scientists are interested in are similar to those used to assess the weather (temperature, precipitation, etc.).
But there is a big difference between climate and weather: **weather** varies from hour to hour and from day to day whereas **climate**
is defined as the average of weather over several decades or longer.

The figure below shows a woman walking her dog and we can use it to make an analogy to illustrate the difference between weather and climate.
if you focus your attention on the dog, you can see that it is all over the place, sometimes upwards, sometimes downwards: this can represent the weather and its
variability. The dog (weather) is not following a fully random pattern and varies around a main direction (trend) that is given by the woman: the woman is representing
the climate and gives us an indication of where both the woman and dog are likely to be in the future.

 ![Illustration of the difference between weather and climate](../../images/weather_versus_climate.png "Weather versus Climate")

*Source: [Animated short introduction to statistics in climate research](https://youtu.be/e0vj-0imOLw) from Norwegian infotainment program Siffer. Produced by TeddyTV for NRK. Animation by Ole Christoffer Haga*


You can also watch an animated illustration of the difference between climate and weather:

{% include _includes/youtube.html id="e0vj-0imOLw" title="Difference between climate and weather" %}

# What is the weather like in Paris?

In order to answer this question, we are going to inspect and visualize the dataset `tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv` using simple galaxy tools.

> <hands-on-title>Daily temperature time series</hands-on-title>
>
>    > <comment-title>Tip: search for the tool</comment-title>
>    >
>    > Many different tools can be used to answer to the questions. Here we give you some guidelines to help you to choose.
>    > Use the **tools search box** at the top of the tool panel to find **Select lines that match an expression** {% icon tool %} and **Datamash** {% icon tool %}.
>    {: .comment}
>
>    > <question-title></question-title>
>    >
>    > 1. What was the average temperature in Paris on the 14th of July 2003?
>    > 2. What is the minimum and maximum temperatures in Paris?
>    > 3. On which date did the minimum temperature occured?
>    > 4. On which date did the maximum temperature occured?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The average temperature in Paris on the 14th of July 2003 was 26.73 degrees Celcius. It can be found by using **Select lines that match an expression** {% icon tool %} with parameter **"the pattern"** set to 2003-07-14.
>    > > 2. The minimum temperature in Paris is -11.6799995 degrees celcius and the maximum temperature in Paris is 33.579998 degrees celcius. To find out, you can use **Datamash** {% icon tool %} with the following parameters:
>    > >      - {% icon param-file %} *"Input tabular dataset"*: `tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv`
>    > >      - *"Input file has a header line"*: `Yes`
>    > >      - *"Print header line"*: `Yes`
>    > >      - "Print all fields from input file": `No`
>    > >      - In *"Operation to perform on each group"*:
>    > >          - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>    > >              - *"Type"*: `minimum`
>    > >              - *"On column"*: `c2`
>    > >          - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>    > >              - *"Type"*: `maximum`
>    > >              - *"On column"*: `c2`
>    > >
>    > > 3. The minimum temperature (-11.6799995 degrees celcius) was observed on January 16 1985.
>    > >     You can use different Galaxy tools to find out the solution and here we show you how to use **Datamash**  {% icon tool %} with the following parameters:
>    > >      - {% icon param-file %} *"Input tabular dataset"*: `tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv`
>    > >      - *"Input file has a header line"*: `Yes`
>    > >      - *"Print header line"*: `Yes`
>    > >      - "Print all fields from input file": `Yes`
>    > >      - In *"Operation to perform on each group"*:
>    > >          - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>    > >              - *"Type"*: `minimum`
>    > >              - *"On column"*: `c2`
>    > >
>    > > 4. The maximum temperature (33.579998 degrees celcius) was observed on July 25 2019. For the **maximum**, repeat **Datamash**  {% icon tool %} with the following parameters:
>    > >
>    > >      - {% icon param-file %} *"Input tabular dataset"*: `tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv`
>    > >      - *"Input file has a header line"*: `Yes`
>    > >      - *"Print header line"*: `Yes`
>    > >      - "Print all fields from input file": `Yes`
>    > >      - In *"Operation to perform on each group"*:
>    > >          - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>    > >              - *"Type"*: `maximum`
>    > >              - *"On column"*: `c2`
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

# What is the climate in Paris?

To get some information about the (past and current) climate in Paris, we will first look at monthly averages.

## Seasonality

> <hands-on-title>What is the monthly climatological temperature in Paris?</hands-on-title>
>
>   To answer to this question, we will compute the global average temperatures over the entire period 1950 and 2019 for each month (January, February, etc.). Indeed,
>   this period of time is sufficiently long for computing monthly climatological temperature (more than 30 years).
>    > <question-title></question-title>
>    >
>    > 1. What is the warmest summer month e.g. between June, July and August (JJA) in Paris?
>    > 2. What is the coolest winter month e.g. between December, January and February (DJF) in Paris?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The warmest summer month in Paris is July (19.921018171429 degrees celcius). However, it is interesting to remark that on our dataset we see very little difference in the mean temperature between July and August.
>    > > 2. The coolest winter month in Paris is January (4.4669169722484 degrees celcius).
>    > >
>    > >     Below, we show you how we found these results.
>    > >     We will first split all the dates (first column) from YYYY-MM-DD (where YYYY is the year, MM the month and DD the day) to three column to get 3 columns: one for the year, one for the month and one for the day. Use **Text reformatting with awk** {% icon tool %} with parameters:
>    > >      - **File to process**: `tg_ens_mean_0.1deg_reg_v20.0e_Paris_daily.csv`
>    > >      - **AWK Program**: `gsub(/-/,"\t",$1){$1=$1} {print}`
>    > >
>    > >     Rename the resulting file to `split_dates_Paris.csv`.
>    > >
>    > >     Then use **Datamash**  {% icon tool %} with the following parameters:
>    > >      - {% icon param-file %} *"Input tabular dataset"*: `split_dates_Paris.csv`
>    > >      - *"Group by fields"*: 2
>    > >      - *"Input file has a header line"*: `Yes`
>    > >      - *"Print header line"*: `No`
>    > >      - "Sort input": `Yes`
>    > >      - "Print all fields from input file": `No`
>    > >      - In *"Operation to perform on each group"*:
>    > >          - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>    > >              - *"Type"*: `Mean`
>    > >              - *"On column"*: `c4`
>    > >
>    > >     Rename the resulting file to `climatology_Paris.csv`.
>    > >     Then use again **Datamash** to get the month where the minimum and maximum temperatures are found:
>    > >      - {% icon param-file %} *"Input tabular dataset"*: `climatology_Paris.csv`
>    > >      - *"Group by fields"*:
>    > >      - *"Input file has a header line"*: `No`
>    > >      - *"Print header line"*: `No`
>    > >      - "Print all fields from input file": `Yes`
>    > >      - In *"Operation to perform on each group"*:
>    > >             - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>    > >                 - *"Type"*: `minimum`
>    > >                 - *"On column"*: `c2`
>    > >
>    > >     Look at the resulting file and the first field will give you the month (07 e.g. July) where the maximum temperature is found.
>    > >
>    > >     For the **maximum**, repeat **Datamash**  {% icon tool %} with the following parameters:
>    > >      - {% icon param-file %} *"Input tabular dataset"*: `climatology_Paris.csv`
>    > >      - *"Group by fields"*: ``
>    > >      - *"Input file has a header line"*: `Yes`
>    > >      - *"Print header line"*: `No`
>    > >      - "Print all fields from input file": `Yes`
>    > >      - In *"Operation to perform on each group"*:
>    > >          - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>    > >              - *"Type"*: `maximum`
>    > >              - *"On column"*: `c2`
>    > >
>    > >    The result is in the first column of the resulting file which indicates `01` e.g. January.
>    > >
>    > >    Please note that you may use other Galaxy tools to reach the same results.
>    > >    Results can be slightly different when using different source of climate information. However, you will always observe the same pattern e.g. cool month in winter and warm month on summer. We can also clearly see that Paris has a mild climate with on average no extreme temperatures.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <tip-title>Using existing climatologies</tip-title>
>
> In this tutorial, we compute manually the monthly climatological temperatures to explain you the algorithm used behing.
> However, many data providers have pre-computed climatologies and can be directly downloaded. For instance, on the [CDS](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset), climatologies are provided for [Essential climate variables for assessment of climate variability from 1979 to present](https://cds.climate.copernicus.eu/cdsapp#!/dataset/ecv-for-climate-change?tab=overview).
{: .tip}


## Yearly average

> <hands-on-title>What is the trend (cooling/warming) in the climate for Paris between 1950 and 2019?</hands-on-title>
>
>  To answer to this question, we will compute yearly mean of the temperature in Paris and visualize it.
>
>    1. Use **Datamash**  {% icon tool %} with the following parameters:
>        - {% icon param-file %} *"Input tabular dataset"*: `split_dates_Paris.csv`
>        - *"Group by fields"*: 1
>        - *"Input file has a header line"*: `Yes`
>        - *"Print header line"*: `No`
>        - *"Print all fields from input file"*: `No`
>        - *"Sort input"*: `Yes`
>        - In *"Operation to perform on each group"*:
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>              - *"Type"*: `Mean`
>              - *"On column"*: `c4`
>
>    2. Rename the resulting file to `yearly_mean_Paris.csv`.
>
>    3. To make a plot, you can use **Scatterplot w ggplot2**  {% icon tool %} with the following parameters:
>        - *"Input in tabular format"*: `yearly_mean_Paris.csv`
>        - *"Column to plot on x-axis"*: 1
>        - *"Column to plot on y-axis"*: 2
>        - *"Plot title"*: Yearly mean temperature in Paris from 1950 to 2019
>        - *"Label for x axis"*: Year (YYYY)
>        - *"Label for y axis"*: Temperature (degrees celcius)
>        - And finally in `Advanced Options` change `Type of plot` to **Points and Lines**.
>
>    4. **View** {% icon galaxy-eye%} the resulting plot:
>
>    ![Resulting plot showing Yearly mean temperature in Paris](../../images/yearly_mean_temperature_Paris.png)
>
>    > <question-title></question-title>
>    >
>    > Can we easily observe a trend?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The plot clearly shows a slight increase in the yearly mean temperature between 1950 and 2019. Even though it looks no more than a few degrees celcius, it is
>    > > quite significant.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

## Anomalies

In climate change studies, temperature **anomalies** are more important than **absolute** temperature. A temperature anomaly is the difference from an average, or baseline,
temperature. The baseline temperature is typically computed by averaging 30 or more years of temperature data. A *positive anomaly* indicates the observed temperature was *warmer* than the baseline, while a *negative anomaly* indicates the observed temperature was *cooler* than the baseline.

> <hands-on-title>Climate stripes for Paris</hands-on-title>
>    Computing temperature anomalies is out of scope of this tutorial and we will therefore use pre-computed temperature anomalies `ts_cities.csv`.
>    A simple way to visualize anomalies and highlight cooling/warming over the years,  is to use **climate stripes from timeseries** {% icon tool %} with the following parameters:
>    - *"timeseries to plot"*: `ts_cities.csv`
>    - *"column name to use for plotting"*: `tg_anomalies_paris`
>    - *"plot title"*: `Climate stripes for Paris (1950-2019)`
>
>    **View** {% icon galaxy-eye%} the resulting plot:
>
>    ![Resulting plot showing Climate stripes in Paris](../../images/climate_stripes_temperature_Paris.png)
>
>    > <question-title>do you observe a warming or cooling between 1950 and 2019?</question-title>
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The climate stripes clearly show a warming between 1950 and 2019.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <tip-title>Copernicus Climate Bulletin</tip-title>
>
> [Copernicus Climate Bulletins](https://climate.copernicus.eu/climate-bulletins) presents the current condition of the climate using key climate change indicators.
> They also provide data, analysis of the maps and guidance on how they are produced. Datasets for temperature anomalies can be found and are
> regularly updated (with recent dates). For instance, in March 2020, [the corresponding dataset](https://climate.copernicus.eu/sites/default/files/2020-04/ts_1month_anomaly_Global_ea_2t_202003_v01.csv) can be found on the Copernicus site.
>
{: .tip}

# Climate variables

*Temperature* is often the first variable that comes to mind when we talk about climate. However, it is insufficient to fully characterize the climate, and scientists have agreed on a number of variables to systematically observe Earth`s changing climate.

That is what we call *Essential Climate Variables*.

## Essential Climate Variables

The [Global Climate Observing System](https://gcos.wmo.int/) (GCOS) and its GCOS expert panels maintain definitions of [Essential Climate Variables](https://gcos.wmo.int/en/essential-climate-variables) (ECVs).

GCOS is co-sponsored by the [World Meteorological Organization](https://public.wmo.int/en) (WMO), the [Intergovernmental Oceanographic Commission of the United Nations Educational, Scientific and Cultural Organization](http://www.ioc-unesco.org/) (IOC-UNESCO), the [United Nations Environment Programme](https://www.unenvironment.org/) (UN Environment), and the [International Science Council](https://council.science/) (ISC). It regularly assesses the status of global climate observations of the atmosphere, land and ocean and produces guidance for its improvement.

At the moment, there are [55 ECVs](https://gcos.wmo.int/en/essential-climate-variables).

*Source: [https://gcos.wmo.int/en/essential-climate-variables](https://gcos.wmo.int/en/essential-climate-variables)*

> <hands-on-title>Essential Climate Variables</hands-on-title>
>
> We will look at the [Water Vapor Essential Climate Variable ](https://gcos.wmo.int/en/essential-climate-variables/surface-vapour/):
> *The humidity of air near the surface of the Earth affects the comfort and health of humans, livestock and wildlife, the swarming behaviour of insects and the occurrence of plant disease. The humidity of air near the surface affects evaporation and the strength of the hydrological and energy cycles. Evaporation from the surface of the earth is the source of water in the atmosphere and so is responsible for important feedbacks in the climate system due to clouds and radiation.*
>
> 1. Import the file from [Zenodo](https://zenodo.org/records/15349982/files/rh_mean_july_1980_2018.nc?download=1) or from the shared data library
>
>    ```
>    https://zenodo.org/records/15349982/files/rh_mean_july_1980_2018.nc?download=1
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Ensure the data type is set to **netcdf**; if not, change it accordingly.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
>    If it is not `netcdf` make sure to convert it using the Galaxy built-in format converters.
>
> 3. Start Panoply interactive tool to generate (lat/lon) geographical maps
>    - *"Tool Parameters netCDF format"*: `rh_mean_july_1980_2018.nc`
>
>    > <tip-title>Launch Panoply in Galaxy</tip-title>
>    > Currently Panoply in Galaxy is available on useGalaxy.eu instance, on the "Interactive tools" tool panel section or, as all interactive tools, from the dedicated usGalaxy.eu subdomain: [Live.useGalaxy.eu](https://live.usegalaxy.eu)
>    >
>    > 1. Open the {% tool [Panoply](interactive_tool_panoply) %} [on UseGalaxy.eu](https://live.usegalaxy.eu/?tool_id=interactive_tool_panoply)
>    > 2. Check **ecv_1979.nc** dataset selected in the netcdf input field
>    > 3. Click Run Tool
>    > 4. The tool will start running and will stay running permanently
>    > 5. Click on the "User" menu at the top and go to "Active Interactive Tools" and locate the Panoply instance you started.
>    > 6. Click on your Panoply instance
>    >    ![Panoply dataset selection](../../images/select_dataset.png "Select dataset")
>    > 7. Click on **ecv_1979.nc** dataset
>    {: .tip}
>
> 4. **Create** a georeferenced Longitude-Latitude plot:
>    - Change the Color Table (Scale Tab) to "tokyo.cpt", and
>    - Change the Projection (Map Tab) to "Equirectangular (Regional) and the center latitude to "45" North to zoom over France.
>    - Switch between 1980 and 2018 in the 'Array(s)' tab to observe the changes between these two years.
>
>    ![Resulting plot showing Relative humidity in July 1980 and July 2018](../../images/rh_ecv_july_1980_2018.png)
>
>
>    > <question-title>Relative humidity</question-title>
>    >
>    > 1. Do you observe any significant changes relative humidity in France from 1979 to 2018?
>    > 2. Do we have sufficient information to make any conclusions on the change in climate?
>    >
>    > > <solution-title></solution-title>
>    > > 1. We can see significant changes on the plot over France. The relative humidity of air near the surface of the Earth is lower in July 2018 than in July 1980.
>    > > 2. We do not have sufficient information to draw any conclusions about the change in climate. In our analysis, we only used two different months (July 1980 and July 2018) and can only discuss the average changes in weather during these two periods (July 1980 and July 2018). We learnt that to draw any conclusions on the climate, we would need to make statistics over a long period of time e.g. we would need to download about 30 years of data and for instance compute anomalies in relative humidity to check if there is any trend. These aspects will be discussed further in other Galaxy tutorials.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Past, present and future climate?


When we talk about climate data, the type of data can vary significantly. We have very little actual observations at the scale of climate and usually not covering a large area. In addition to observations, we can make use of:
- Re-analyses where observations and numerical modelling are combined together.
- Climate models.

Observations and re-analyses provide information about the past and current climate while climate models can provide past, current and future climate information.
When it comes to future climate, we usually need to make some assumptions (such as how much CO2 emissions, etc.) and simulate different scenarios e.g. we run climate models using different assumptions and look at future trends under each of these scenarios: this is what we call **climate projections**. Climate projections will be discussed in a separate Galaxy tutorial.


# Conclusion



We have learnt to differentiate climate from weather and got an overview of the terminology used by climate scientists to identify the
various source of climate data.


topic: climate
tutorial: fates-jupyterlab
title: Functionally Assembled Terrestrial Ecosystem Simulator (FATES) with Galaxy Climate JupyterLab
difficulty: intermediate
time_estimation: 6H
tools_json: ["JupyterLab", "interactive_tool_jupyter_notebook"]
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 7b40d585ff771aa0e7e92caf5dfe368d
last_modified: 2026-05-27T09:01:47.875038
zenodo_link: https://doi.org/10.5281/zenodo.4108341
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/fates-jupyterlab/tutorial.html

content:
The practical aims at familiarizing you with running CLM-FATES within Galaxy Climate JupyterLab.

> ## Agenda
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
> FATES is the “Functionally Assembled Terrestrial Ecosystem Simulator”.
> FATES needs what we call a "Host Land Model" (HLM) to run and in this tutorial
> we will be using the [Community Land Model](http://www.cesm.ucar.edu/models/clm/)
> of the [Community Terrestrial Systems Model](https://github.com/ESCOMP/CTSM) (CLM-CTSM).
> FATES was derived from the CLM Ecosystem Demography model (CLM(ED)), which was documented in
> {% cite Fisher2015 %}.
> and this technical note was first published as an appendix to [that paper](https://pdfs.semanticscholar.org/396c/b9f172cb681421ed78325a2237bfb428eece.pdf).
> The [FATES documentation](https://fates-docs.readthedocs.io/en/latest/index.html) will provide some more insight on FATES too.
>
{:  .comment}

## Motivation

In this tutorial, we will be using a Galaxy interactive tool called Galaxy Climate JupyterLab for running CLM-FATES.
This interactive tool is only available from [LiveGalaxy.eu](https://live.usegalaxy.eu/).

This platform is meant to be used for:
- preparing new input datasets;
- testing new versions of fates, including code changes;
- developing new notebooks for analyzing and showing the model results;
- creating interactive publications (notebooks);
- teaching purposes.

The main advantage over the Galaxy CLM-FATES tool is that you can run any versions of FATES, including developments
that are not released yet. However, it is not recommended to run more than a few decades of simulation time.
In that particular case, we would suggest to use the Galaxy CLM-FATES Galaxy tool.

> <comment-title>CML-FATES in JupyterLab versus CLM-FATES Galaxy tool</comment-title>
> Do not use the interactive Galaxy Climate JupyterLab for running long and "operational" simulations and
> do not forget that you need to save back your results to your Galaxy history or local machine before
> stopping your JupyterLab.
>
{: .comment}

# Step-1: Get data

> <hands-on-title> Data upload </hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *fates-jupyterlab*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://doi.org/10.5281/zenodo.4108341) or from the shared data library
>
>    ```
>    https://zenodo.org/record/4108341/files/inputdata_version2.0.0_ALP1.tar
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check the datatype is **tar**
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Rename Datasets
>
>    As "https://zenodo.org/record/4108341/files/inputdata_version2.0.0_ALP1.tar" is not a beautiful name and can give errors for some tools,
>    it is a good practice to change the dataset name by something more meaningfull. For example by removing `https://zenodo.org/record/4108341/files/` to obtain `inputdata_version2.0.0_ALP1.tar`, respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Add a tag to the dataset corresponding to `fates-jupyterlab`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Step-2: Opening up Climate JupyterLab

> <hands-on-title> Launch JupyterLab for Ocean / Atmosphere / Land / Climate Python ecosystem in Galaxy </hands-on-title>
>
> Currently JupyterLab for Ocean / Atmosphere / Land / Climate Python ecosystem in Galaxy is available on [Live.useGalaxy.eu](https://live.usegalaxy.eu) only. JupyterLab for Ocean / Atmosphere / Land / Climate Python ecosystem and not the default JupyterLab in Galaxy contains all the python packages and additional software we need for running Earth System Model, including Functionally Assembled Terrestrial Ecosystem Simulator (FATES). The default JupyterLab in Galaxy would not be sufficient for executing all the tasks in th
> is tutorial.
>
> 1. Open the {% tool [JupyterLab](interactive_tool_jupyter_notebook) %} or opening directly on [live.usegalaxy](https://live.usegalaxy.eu/?tool_id=interactive_tool_climate_notebook)
> 2. Click Run Tool
> 3. The tool will start running and will stay running permanently
> 4. Click on the "User" menu at the top and go to "Active Interactive Tools" and locate the JupyterLab instance you started.
> 5. Click on your JupyterLab instance (please not that it may take a few minutes before you can click on the link to your jupyterLab instance).
>
{: .hands_on}


You should now be looking at a page with the JupyterLab interface:

![Jupyterlab climate session interface](../../images/jupyterlab_climate_session.png)


# Step-3: Create a new session CLM-FATES in JupyterLab

## Import input data to JupyterLab

In this part of the tutorial, we will be using the existing Jupyter Notebook called **ipython_galaxy_notebook.ipynb**

> <hands-on-title>Open a JupyterLab Terminal</hands-on-title>
> To open **ipython_galaxy_notebook.ipynb**, double click on it.
> More information on the JupyterLab interface can be found on the [JupyterLab documentation](https://jupyterlab.readthedocs.io/en/stable/user/interface.html).
>
> Import the FATES input dataset from your history:
> - In a new code cell:
>
> ```
> %%bash
>
> get -i inputdata_version2.0.0_ALP1.tar -t name
>```
>
> By default code cells execute Python 3 code (default kernel) so to execute the Shell command lines
> we will use `%%bash`.  In that case the cell runs with bash in a subprocess.
>
> Then untar this file:
>
> ```
> %%bash
> mkdir $HOME/inputdata
> tar xf /import/inputdata_version2.0.0_ALP1.tar --directory $HOME/inputdata
> ```
{: .hands_on}

> <comment-title>Direct download in JupyterLab from Zenodo</comment-title>
>
> You may also download the input dataset directly from Zenodo.
> - Open a JupyterLab Terminal and enter the following command:
> ```
> %%bash
> cd /import
> wget https://zenodo.org/record/4108341/files/inputdata_version2.0.0_ALP1.tar
> ```
{:  .comment}

> <comment-title>Using JupyterLab Terminal</comment-title>
>
> Most of the tutorial (except visualization) can be executed from a JupyterLab Terminal. In that case,
> you should not add `%%bash` to your commands. More on JupyterLab Terminal can be found on [Read the Docs](https://jupyterlab.readthedocs.io/en/stable/user/terminal.html).
{:  .comment}

## Get CLM-FATES EMERALD release

> <hands-on-title>Clone CLM-FATES for Nordic sites</hands-on-title>
>
> ```
> %%bash
> conda create --name fates -y fates-emerald=2.0.1
> ```
> The command above is required once only. It creates a new conda environment called **fates** and
> install **fates-emerald** version 2.0.1 conda package. It is important to always specify the version of
> CLM-FATES you would like to use as it needs to match your input dataset.
> Now a new **fates** conda environment has been
> created in your current JupyterLab session and can be use every time you activate it.
>
> Then to activate this new conda environment:
>
> ```
> %%bash
> source activate fates
> ```
> Please note that you would need to activate **fates** environment in every new code cell (because it starts a new Shell subprocess).
>
{: .hands_on}


## Create CLM-FATES new case

> <hands-on-title>Create CLM-FATES new case for ALP1 site</hands-on-title>
>
> ```
> %%bash
> source activate fates
>
> create_newcase --case $HOME/ctsm_cases/fates_alp1 --compset 2000_DATM%1PTGSWP3_CLM50%FATES_SICE_SOCN_MOSART_SGLC_SWAV --res 1x1_ALP1 --machine espresso --run-unsupported
> ```
>
{: .hands_on}

> <warning-title>Command not found!</warning-title>
> If you get an error when invoking `create_newcase` make sure you have switch to fates conda environment:
> ```
> %%bash
> source acticate fates
> create_newcase --help
> ```
{: .warning}

The 4 main arguments of create_newcase are explained on the figure below: ![Snapshot of create_newcase main arguments showing: what is the case name? which resolution? which model configuration? which set of components? which machine are you running on?](../../images/newcase_fates.png).

- **case**: specifies the name and location of the case being created. It creates a new case in `$HOME/ctsm_cases/` and its name is `fates_alp1`. make sure to give a meaningful name to your FATES experiments.
- **res**: specifies the model resolution (resolution of the grid). Here **1x1_ALP1** corresponds to a single point resolution.
- **compset**: specifies the component set, i.e., component models, forcing scenarios and physics options for those models.
  - The long name of the compset we have chosen is `2000_DATM%1PTGSWP3_CLM50%FATES_SICE_SOCN_MOSART_SGLC_SWAV`
  - The notation for the compset longname is: `TIME_ATM[%phys]_LND[%phys]_ICE[%phys]_OCN[%phys]_ROF[%phys]_GLC[%phys]_WAV[%phys][_BGC%phys]`
  - The compset longname has the specified order: **atm, lnd, ice, ocn, river, glc wave cesm-options** where:
    - **Initialization Time**:2000
    - **Atmosphere**: Data atmosphere DATM%1PTGSWP3
    - **Land**: CLM50%FATES
    - **Sea-Ice**: SICE Stub ICE
    - **Ocean**: SOCN Stub ocean
    - **River runoff**:MOSART: MOdel for Scale Adaptive River Transport
    - **Land Ice**: SGLC Stub Glacier (land ice) component
    - **Wave**-   SWAV Stub wave component
   See also the [list of available component sets](http://www.cesm.ucar.edu/models/cesm2/config/compsets.html).
- **mach**: specifies the machine where CLM-FATES will be compiled and run. We use `espresso` which is the local setup (see `$HOME/.cime/` folder).

## Setup, build and submit your first simulation

> <hands-on-title>Setup, build and submit</hands-on-title>
>
> Check the content of the directory and browse the sub-directories:
> - CaseDocs: namelists or similar
> - SourceMods: this is where you can add local source code changes.
> - Tools: a few utilities (we won’t use them directly)
> - Buildconf: configuration for building each component
> For this tutorial, we  wish to have a “cold” start as we are mostly interested in setting up our model.
> When ready to run in production, the model needs to be spin-up (run for several centuries until it reaches some kind of equilibrium).
>
> We will first make a short simulation (6 months):
>
> ```
> %%bash
> source activate fates
>
> cd $HOME/ctsm_cases/fates_alp1
> ./case.setup
> ./case.build
> ./xmlchange STOP_OPTION=nmonths            # set the simulation periods to "nmonths"
> ./xmlchange STOP_N=6                       # set the length of simulation, i.e, how many months
> ./case.submit > case_submit.out 2>&1
> ```
> The step above can take a lot of time because it needs to compile and run the FATES model.
> Therefore we suggest you make a break and come back later (or the following day) before you continue the
> tutorial.
{: .hands_on}

## Check your run

> <hands-on-title>check your simulation</hands-on-title>
>
> 1. From a new code cell:
>
> ```
> %%bash
>
> cd $HOME/work/fates_alp1
> ls -la
> ```
> You should see two folders:
> - bld: contains the object and CESM executable (called cesm.exe) for your configuration
> - run: this directory will be used during your simulation run to generate output files, etc.
>
> The **bld** folder contains the model executable (called `cesm.exe`) while **run** contains all the files used for running CLM-FATES (and not already archived).
> Once your run is terminated, many files are moved from the **run** folder to the **archive** folder:
>
> ```
> %%bash
> cd $HOME/archive/fates_alp1
> ls lnd/hist
> ```
>
> We are interested in the "history" files from the CLM-FATES model and these files are all located in `lnd/hist` folder.
> You can also check other model components in the archive directory (atm, etc.): in our case, it is not of a great interest as
> we are running the CLM-FATES component.
> We have run a very short simulation and get one file only, called `fates_alp1_t.clm2.h0.2000-01.nc`. The
> CLM-FATES model outputs are stored in netCDF format.
>
> > <comment-title>What is a netCDF file?</comment-title>
> >
> > Netcdf stands for “network Common Data Form”. It is self-describing, portable, metadata friendly, supported by many languages
> > (including python, R, fortran, C/C++, Matlab, NCL, etc.), viewing tools (like panoply, ncview/ncdump) and tool suites of file operators (in particular NCO and CDO).
> {: .comment}
>
> 2. Create a new Jupyter Notebook for analyzing your results:
>   - From the **File Menu** --> **New** --> **Notebook**:
>      - Rename your notebook to **check_analysis.ipynb**
>      - All the analysis of the 6 month FATES simulation will be done from this notebook
>
> 3. Get metadata
> In a Code cell:
>
> ```
> import os
> import xarray as xr
> xr.set_options(display_style="html")
> %matplotlib inline
>
> case = 'fates_alp1'
> path = os.path.join(os.getenv('HOME'), 'archive', case, 'lnd', 'hist')
> dset = xr.open_mfdataset(path + '/*.nc', combine='by_coords')
> dset
> ```
> As shown above, we are now using Python 3 for analyzing the results and [xarray](http://xarray.pydata.org/en/stable/) which
> is a Python package that can easily handle [netCDF](https://en.wikipedia.org/wiki/NetCDF) files.
> we opened all the history files we have produced and print metadata.
>
> 4. Plotting 1D variables (timeseries)
>
> You can select a variable by using its short name (see metadata above) and then calling the plot method:
> ```
> dset['AREA_TREES'].plot()
> ```
> As we ran 6 months only, we have very little points in our timeseries!
>
> To plot 2D variables such as **CANOPY_AREA_BY_AGE**, you can use the *col_wrap* option when plotting:
>
> ```
> dset['CANOPY_AREA_BY_AGE'].plot(aspect=3, size=6, col='fates_levage', col_wrap=1)
> ```
> In the plot above, we have one plot per row (*col_wrap=1*) and we will have a plot for each value of the *fates_levage* dimension.
> We also changed the aspect of the plot (aspect=3, size=6).
{: .hands_on}

## Customize your run

> <hands-on-title>Run 10 years</hands-on-title>
>
> ```
> %%bash
> source activate fates
> cd $HOME/ctsm_cases/fates_alp1
>
> ./xmlchange RUN_STARTDATE=0001-01-01      # set up the starting date of your simulation
> ./xmlchange STOP_OPTION=nyears            # set the simulation periods to "years"
> ./xmlchange STOP_N=5                      # set the length of simulation, i.e, how many years
> ./xmlchange CONTINUE_RUN=TRUE             # if you want to continue your simulation from restart file, set it to TRUE
> ./xmlchange RESUBMIT=1                    # set up how many times you want to resubmit your simulation.
>                                           # e.g, STOP_N=5, RESUBMIT=1, you will have simulation for 5+5*1=10
> ./xmlchange DATM_CLMNCEP_YR_START=1901    # set up the start year of the atmospheric forcing
> ./xmlchange DATM_CLMNCEP_YR_END=1910      # set up the end year of the atmospheric forcing
> ./case.submit > case_submit_sontinue_run.out 2>&1
> ```
> This step will take several hours.
{: .hands_on}

# Analysis

In this section, we will be able to analyze your 10 year simulation only when the run is terminated (note that data will be moved to the archive folder every 5 years).

## Analyzing FATES-CLM model outputs

> <hands-on-title>Open a new Python notebook</hands-on-title>
> - Create a notebook by clicking the `+` button in the file browser and then selecting a kernel in the new Launcher tab:
> - Rename your notebook to **analyse_case.ipynb**
> Get more information online at [JupyterLab notebooks](https://jupyterlab.readthedocs.io/en/stable/user/notebook.html).
{: .hands_on}

### Use `xarray` to read and plot

In this section, we give additional examples on how to visualize your results using `xarray`:

```
import xarray as xr
xr.set_options(display_style="html")
%matplotlib inline

case = 'fates_alp1'
path = os.path.join(os.getenv('HOME'), 'archive', case, 'lnd', 'hist')
dset = xr.open_mfdataset(path + '/*.nc', combine='by_coords')
dset
```
As you can see, we are now using `open_mfdataset` to read all the netCDF files available in the history folder.
The option `combine='by_coords')` is used to tell the method `open_mfdataset` how to combine the different files
together.

You can use the same `plot` method as before for plotting any variable. For instance:

```
dset['AREA_TREES'].plot(aspect=3, size=6)

```

For saving your plot, for instance in a *png* file format:

```
import matplotlib.pyplot as plt
fig = plt.figure(1, figsize=[14,7])
ax = plt.subplot(1, 1, 1)
dset['AREA_TREES'].plot(ax=ax)
ax.set_title(dset['AREA_TREES'].long_name)
fig.savefig('AREA_TREES.png')
```

![AREA TREES (10 years)](../../images/AREA_TREES_10yrs.png)

In the plot above, we create a figure (with specific dimension [14,7]) and one subplot with one row and one column.
The last argument of *subplot* is the index (1) of this particular subplot.

Finally, the resulting figure is saved in a file called 'AREA_TREES.png'.


To plot 2D variables and save the resulting plot in a png file, for instance **CANOPY_HEIGHT_DIST**:

```
p = dset['CANOPY_HEIGHT_DIST'].plot(aspect=3, size=6, col_wrap=1, col='fates_levheight')
p.fig.savefig('CANOPY_HEIGHT_DIST.png')
```

![CANOPY HEIGHT DIST (10 years)](../../images/CANOPY_HEIGHT_DIST_10yrs.png)

# Save your results to your Galaxy history

> <hands-on-title>Put your data to your Galaxy history</hands-on-title>
>
> ```
> %%bash
> cd $HOME
> tar cvf archive_emerald_fates_test.tar archive
> ```
> Then you are now ready to put your dataset into Galaxy. As it can be large, we recommend to use FTP:
>
> ```
> curl -T {"archive_emerald_fates_test.tar"} ftp://ftp.usegalaxy.eu --user USER:PASSWORD --ssl
> ```
> Where you replace `USER` by your galaxy username (what you used to login to Galaxy i.e. usually your email address
> and `PASSWORD` by your Galaxy login password.
>
> To get `archive_emerald_fates_test.tar` in your history:
>  - Open the Galaxy Upload Manager ({% icon galaxy-upload %} on the top-right of the tool panel)
>  - Click on **Choose FTP files** and select `archive_emerald_fates_test.tar` to import it into your history.
>
> And make sure to save all your notebooks to your Galaxy history too:
>
> ```
> %%bash
> put -p ipython_galaxy_notebook.ipynb
> put -p check_analysis.ipynb
> put -p analyse_case.ipynb
> put -p AREA_TREES.png
> put -p CANOPY_HEIGHT_DIST.png
> ```
{: .hands_on}

> <warning-title>Danger: You can lose data!</warning-title>
> If you do not copy data (FATES model results, jupyter notebooks, plots, etc.) before you stop your Galaxy climate
> JupyterLab tool, all your results will be lost!
{: .warning}


# Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

{% snippet faqs/galaxy/histories_sharing.md %}

> <hands-on-title>Share history</hands-on-title>
>
> 1. Share your history with your neighbour.
> 2. Find the history shared by your neighbour. Histories shared with specific users can be accessed by those users under their top masthead "User" menu under `Histories shared with me`.
{: .hands_on}


> <comment-title>Clone CLM-FATES release for Nordic site from github (advanced)</comment-title>
>
> You may also get the CLM-FATES release 2.0.1 directly from github:
>
> ```
> %%bash
> cd $HOME
> git clone -b release-emerald-platform2.0.1 https://github.com/NordicESMhub/ctsm.git
> cd ctsm
> ./manage_externals/checkout_externals
> ```
> This approach may be interesting if you wish to run another release or development version of CLM-FATES.
> All the tutorial shown can be done with your local version. In that case, you would need to use the
> local command such as `create__newcase` which then require the following steps:
> - locate the command on your local folder; for instance to locate `create_newcase`:
>
> ```
> %%bash
> cd $HOME/ctsm
> find . -name create_newcase
> ```
> The command above will give you the location of `create_newcase`:
> ```
> ./cime/scripts/create_newcase
> ```
> Be aware that it is a relative path. Then to create a new case:
>
> ```
> ./cime/scripts/create_newcase --case $HOME/ctsm_cases/fates_alp1_local --compset 2000_DATM%1PTGSWP3_CLM50%FATES_SICE_SOCN_MOSART_SGLC_SWAV --res 1x1_ALP1 --machine espresso --run-unsupported
>
> ```
> Finally, if you wish to make changes to the source code, we recommend first to add your changes in different folder and use the option `--user-mods-dir` when
> creating your case. In addition, you should make sure to use version control to save your changes.
> If you are not familiar with `git`, you could also save your changes in the corresponding Galaxy history.
{:  .comment}

# Conclusion

We have learnt to run single-point simulations with FATES-CLM through the Galaxy Climate JupyterLab.


topic: climate
tutorial: earth_system
title: Getting your hands-on earth data
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["earth-system", "ocean", "land", "atmosphere", "biodiversity"]
content_hash: aa3f17eee9d569dbc7c4ce6ea1ee9fcb
last_modified: 2026-05-27T09:01:47.874913
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/earth_system/tutorial.html

content:
This tutotrial aims at familiarzing you with Earth Science and discovering the earth data available on Galaxy. The target audience is not a scientist but
anyone interested in learning about Earth system.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

🌍🌏🌎

The Earth System is a complex and dynamic system that encompasses the interactions between the atmosphere, hydrosphere, geoshpere, and biosphere. Understanding and analyzing data from the Earth System Model is essential, for example to predict and mitigate the impacts of climate change. Currently, [Galaxy Earth System](https://earth-system.usegalaxy.eu/) uses 4 usecases to have an overview of an earth System: ocean, land, atmosphere, and biodiversity.
Of course, these components cross over each other thus, we have data that can fit two or more compartiments.

![Graphic visualization of an Earth System](../../images/earth_system/earth_system_galaxy.png)

Next we will see for each compartiment which kind of data (and how to get them) are availabe on Galaxy Earth System


# 🌊 Ocean 🌊

The ocean is a key component of the Earth’s climate system. Thus, it needs continuous real-time monitoring to help scientists better understand its dynamics and predict its evolution. 

## The Argo program
All around the world, oceanographers have managed to join their efforts and set up a Global Ocean Observing System among which Argo is a key component. Argo is an international program that collects information from inside the ocean using a fleet of robotic instruments that drift with the ocean currents and move up and down between the surface and a mid-water level.

The data available here are from the Argo gliders network. They contain the folowing variables: 
- latitude, longitude, and time
- physcical parameters: water temperature, pressure, and salinity
- biogeochemical parameters: chlorophyll, oxygen, and nitrate

To access those data you can use the ({% tool [Argo data access](toolshed.g2.bx.psu.edu/repos/ecology/argo_getdata/argo_getdata/0.1.15+galaxy0) %})tool. 

For more information follow the tutorial related to it called [Analyse Argo data]({% link topics/climate/tutorials/argo_pangeo/tutorial.md %}) or use the workflow Full Analyse Argo data `https://earth-system.usegalaxy.eu/u/marie.josse/w/full-analyse-argo-data`


## Copernicus Data Space Ecosystem

The Copernicus Data Space Ecosystem provides access to Earth observation data.
The goal of the **Sentinel-3 mission** is to accurately and reliably measure sea surface topography, sea and land surface temperature, and ocean and land surface color. This data will support ocean forecasting systems, environmental monitoring, and climate monitoring.

To access those data you can use the [Copernicus Data Space Ecosystem](https://usegalaxy.eu/root?tool_id=interactive_tool_copernicus_notebook) tool. 

For more information follow the tutorial related to it called [Sentinel 5P data visualisation]({% link topics/climate/tutorials/sentinel5_data/tutorial.md %}). 

Once you get into the Jupyterlab go in the left pannel in `samples > openeo > Sentinel_3.ipynb `. Yhis way you have an example on how to get Sentinel 3 data.

## EMODnet Chemistry

[EMODnet Chemistry](https://emodnet.ec.europa.eu/en/chemistry) provides easy access to marine chemical data, standardised, harmonised and validated data collections and data products for visualisation. These are relevant for the European marine environmental policies to assess the status and trends of ecosystems. 
Numerous substances are considered, most of which are invisible to the naked eye and are monitored using specialised sensors or laboratory analyses. This evidence-based information supports action against environmental changes that pose a threat to marine ecosystems and human health.

### A focus on ODV (Ocean Data View) collections

This [webODV](https://emodnet-chemistry.webodv.awi.de/) service facilitates to explore, subset, visualize, and extract data sets in multiple formats from the harmonized, standardized, validated data collections that EMODnet Chemistry is regularly producing and publishing for all European sea basins for eutrophication and contaminants. 
You can also find these collections on the [Sextant catalogue](https://sextant.ifremer.fr/eng/Data/Catalogue#/search?isTemplate=n&from=1&to=30&sortBy=dateStamp&sortOrder=desc&languageStrategy=searchInDetectedLanguage&any=EMODnet%20Chemistry)

You can analize thes kind of data through the [Ocean Data View (ODV)](https://usegalaxy.eu/root?tool_id=interactive_tool_odv) tool. 

For more information follow the related tutorials and workflows:
 - [Ocean's variables study]({% link topics/climate/tutorials/ocean-variables/tutorial.md %}) and the corresponding workflow  `https://usegalaxy.eu/u/marie.josse/w/oceans-var-v2`
 - [Ocean Data View (ODV)]({% link topics/climate/tutorials/ocean-data-view/tutorial.md %})


## Copernicus Marine Data Store

The [Copernicus Marine Data Store](https://marine.copernicus.eu/access-data) offers a ocean product catalogue, to download or visualize data accross nearly 15variables, including hindcast, current and forecast data.

A tool is currently being develop to enable easy access to this data. More information to come.



# 🏜️ Land 🏞️

This component comprehend all from the solid part of earth land, rock, volcano activities to the deragation of the land surface. This compartiment amis at assessing land and soil degradation, such as erosion, loss of organic matter and soil biodiversity, compaction, salinization, landslides, contamination, sealing, desertification and climate change.

## Copernicus Data Space Ecosystem

The Copernicus Data Space Ecosystem provides access to Earth observation data.

The Copernicus Sentinel-2 mission consists of two polar-orbiting satellites that are positioned in the same sun-synchronous orbit, with a phase difference of 180°.

It aims to monitor changes in land surface conditions. The satellites have a wide swath width (290 km) and a high revisit time. This capability will support monitoring of changes on the Earth's surface.

The Sentinel-2 twin satellites provide image data that largely contribute to existing and ongoing multispectral observations.

These satellites support a variety of services and applications offered by Copernicus, including:
- Land monitoring
- Agriculture
- Emergency management
- Risk mapping
-  Security
- Forestry
- Climate change
- Disaster control
- Marine
- Humanitarian relief operations

The Scene Classification algorithm uses the reflective properties of scene features to establish the presence or absence of clouds in a scene. It is based on a series of threshold tests that use as input: TOA reflectance of several Sentinel-2 spectral bands, band ratios and indexes like Normalised Difference Vegetation Index (NDVI) and Normalised Difference Snow and Ice Index (NDSI). For each of these threshold tests, a level of confidence is associated. It produces at the end of the processing chain a probabilistic cloud mask quality indicator and a snow mask quality indicator. The most recent version of the SCL algorithm includes also morphological operations, usage of auxiliary data like Digital Elevation Model and Land Cover information and exploit the parallax characteristics of Sentinel-2 MSI instrument to improve its overall classification accuracy.

The NDVI ratio can be determined from the contribution of visible wavelength and near-infra-red wavelengths. Strong and well-nourished vegetation will absorb most of the visible wavelengths that it receives and will reflect back a large proportion of the near-infra-red light, whereas poor condition vegetation or thin areas, will reflect more visible wavelength light and less near-infra-red light.

To access those data you can use the [Copernicus Data Space Ecosystem](https://usegalaxy.eu/root?tool_id=interactive_tool_copernicus_notebook) tool. 

For more information follow the tutorial related to it called [From NDVI data with OpenEO to time series visualisation with Holoviews]({% link topics/ecology/tutorials/ndvi_openeo/tutorial.md %}). 


## QGIS (Geographical Information System)

Based on a [QGIS official tutorial](https://docs.qgis.org/2.18/en/docs/training_manual/online_resources/wfs.html), you can learn how to access, filter and import GIS data through WFS web service using [QGIS Galaxy interactive tool](https://usegalaxy.eu/root?tool_id=interactive_tool_qgis)

In the Geographical Information System landscape, there is existing standards to help users deal with remote data. The most common web services are Web Map Services (WMS) and Web Feature Services (WFS). If WMS allows users only to access and display maps stored remotely, WFS is giving access to the features of data so you can modify it and create your own data and maps.

To learn more on how to do that folow [QGIS Web Feature Services]({% link topics/ecology/tutorials/QGIS_Web_Feature_Services/tutorial.md %})

# ☁️ Atmosphere 🌫️

This component represents the layer of gas around the Earth.

## Copernicus Data Space Ecosystem

The Copernicus Data Space Ecosystem provides access to Earth observation data. 
The Copernicus **Sentinel-5 Precursor** mission is the first Copernicus mission that focuses on monitoring our atmosphere. It is the result of a close partnership between ESA, the European Commission, the Netherlands Space Office, industry, data users and scientists.

To access those data you can use the [Copernicus Data Space Ecosystem](https://usegalaxy.eu/root?tool_id=interactive_tool_copernicus_notebook) tool. 

### Volcano 🌋

You can get atmospheric data on volcanic activity by taking a subset of Sentinel 5P L2 data  for instance,  from the 1st of April to the 30th of may 2021 of the Antilles. Especially of the La Soufriere Saint Vincent (Antilles) where a volcaninc erruption occured 9th of April. This dataset is focused on the dioxide sulfur (SO2) and Aerosol index (AI) spread out. Indeed, the knowledge of volcanic activity at a high temporal resolution is crucial for robustly determining large-scale impacts of volcanoes on atmosphere (air quality, air traffic) and climate. As such, this platform will be also of interest for scientists involved in the field of volcanic impacts at large, including institutions in charge of the monitoring of air quality and aviation safety.
 
For more information follow the tutorial related to it called [Sentinel 5P data visualisation]({% link topics/climate/tutorials/sentinel5_data/tutorial.md %}). 


## Climate Data Store

Dive into this wealth of information about the Earth's past, present and future climate with the ({% tool [Climate Data Store](toolshed.g2.bx.psu.edu/repos/climate/c3s/c3s/0.2.0) %}). With this tool you can have acces to atmospheric data such as ERA5 data. ERA5 provides a snapshot of the atmosphere, land surface and ocean waves for each hour from 1959 onwards (for 1950 to 1958 a preliminary version is available). It includes an uncertainty estimate which highlights the considerable evolution of the observing system, on which reanalysis products rely.


# 🐙 Biodiversity 🐿️

This englobes the part of the land, ocean, and athmosphere in which organisms are able to live.

## Marine biodiversity 🐳

European marine microbial biodiversity is a poorly understood and under-exploited component of continental biodiversity. This component aims to provide a set of tools for the analysis
of spatial- and time-comparable marine microbial metagenomics data sets for the exploration of biodiversity and its correlations with environmental quality.

### OBIS Ocean Biodiversity Information System

OBIS is a global open-access data and information clearing-house on marine biodiversity for science, conservation and sustainable development. In order to get occurences of marine biodiversity from OBIS you can use the ({% tool [OBIS occurences](toolshed.g2.bx.psu.edu/repos/ecology/obis_data/obis_data/0.0.2) %}) tool.

For more information follow the tutorial related to it called [OBIS marine indicators]({% link topics/ecology/tutorials/obisindicators/tutorial.md %}) or use the workflow Marine omics visualisation `https://usegalaxy.eu/u/marie.josse/w/marine-omics-visualisation`

## Land biodiversity 🦤

GBIF (Global Biodiversity Information Facility, www.gbif.org) is for sure THE most remarkable biodiversity data aggregator worldwide giving access to more than 1 billion records across all taxonomic groups. The data provided via these sources are highly valuable for research. However, some issues exist concerning data heterogeneity, as they are obtained from various collection methods and sources (OBIS is part of GBIF).



To access those data you can use the ({% tool [Get species occurrences data](toolshed.g2.bx.psu.edu/repos/ecology/spocc_occ/spocc_occ/0.9.0) %}) tool. 

For more information follow the tutorial related to it called [Cleaning GBIF data for the use in Ecology]({% link topics/ecology/tutorials/gbif_cleaning/tutorial.md %}).



# Conclusion 
Here you have an overview of what's is currently available in terms of data accessibility in Galaxy Earth System. This an on going work more is to come !

# Extra information

Coming up soon even more tutorials on and other Earth-System related trainings. Keep an {% icon galaxy-eye %} open if you are interested!

tools_json:
["Argo data access", "toolshed.g2.bx.psu.edu/repos/ecology/argo_getdata/argo_getdata/0.1.15+galaxy0", "Climate Data Store", "toolshed.g2.bx.psu.edu/repos/climate/c3s/c3s/0.2.0", "OBIS occurences", "toolshed.g2.bx.psu.edu/repos/ecology/obis_data/obis_data/0.0.2", "Get species occurrences data", "toolshed.g2.bx.psu.edu/repos/ecology/spocc_occ/spocc_occ/0.9.0"]


topic: climate
tutorial: argo_pangeo
title: Analyse Argo data
difficulty: intermediate
time_estimation: 1H
requirements_json: ["type: \"internal\""]
tags_json: ["earth-system", "ocean", "pangeo", "argo"]
content_hash: 788ef0e0d740ff4ac6d17b757654163c
last_modified: 2026-05-27T09:01:47.874691
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/argo_pangeo/tutorial.html

content:
The ocean is a key component of the Earth's climate system. It thus needs continuous real-time monitoring to help scientists better understand its dynamics and predict its evolution.
All around the world, oceanographers have managed to join their efforts and set up a Global Ocean Observing System among which Argo is a key component.
Argo is an international program that collects information from inside the ocean using a fleet of robotic instruments that drift with the ocean currents and move up and down between the surface and a mid-water level. 

The data used in this tutorial are from the Argo gliders network. We are interested in the following variables: water temperature, latitude, longitude and time.
Our main objective is to plot the water temperature with respect to time. For this, we will be using the netCDF xarray tools available in the Galaxy Europe (or your favourite Galaxy Instance) server.


These tools are part of the Pangeo ecosystem in which the next generation of open-source analysis tools for ocean, atmosphere and climate science can be developed, distributed,
and sustained. These tools must be scalable to meet the current and future challenges of big data, and these solutions should leverage the existing expertise outside of
the geoscience community.

This tutorial is part of a set of tutorials on the [Galaxy Earth System](https://earth-system.usegalaxy.eu/) supported by the [EOSC FAIR-EASE project](https://fairease.eu/).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <details-title>Short introduction on how Galaxy works</details-title>
>
> You can come back to where you left off the tutorial anytime by clicking {% icon level %}.
>
> > <hands-on-title>Log in to Galaxy</hands-on-title>
> > 1. Open your favorite browser (Chrome, Safari or Firefox as your browser, not Internet Explorer!)
> > 2. Browse to your [Galaxy instance](https://earth-system.usegalaxy.eu/)
> > 3. On the top go to **Login or Register**
> >
> >
> {: .hands_on}
>
> The Galaxy homepage is divided into three panels:
> * Tools on the left
> * Viewing panel in the middle
> * History of analysis and files on the right
>
> ![Screenshot of the Galaxy interface, the tools panel is on the left, the main panel is in the center, and the history is on the right.]({% link shared/images/galaxy_interface.png %} "Galaxy interface explanation")
>
> The first time you use Galaxy, there will be no files in your history panel.
{: .details}
---


# Argo gliders data
Argo is a global network of nearly 4000 autonomous probes measuring pressure, temperature and salinity from the surface to 2000m depth every 10 days.
The localisation of these probes is nearly random between the 60th parallels. All probes data are collected by satellite in real-time, processed by several data
centers and finally merged in a single dataset (collecting more than 2 million vertical profiles) made freely available to anyone through an FTP server or monthly zip snapshots.

Each Argo probe is an autonomous, free drifting, profiling float, i.e. a probe that can’t control its trajectory but can control its buoyancy and thus move up and down the water
column as it wishes. Argo floats continuously operate the same program, or cycle, illustrated in the figure below. After 9 to 10 days of free drift at a parking depth of about 1000m,
a typical Argo float dives down to 2000m and then shoals back to the surface while measuring pressure, temperature and salinity. Once it reaches the surface,
the float sends by satellite its measurements to a data center where they are processed in real-time and made freely available on the web in less than 24h00.

![Schema of a typical 10 days program, cycle, of an Argo float](../../images/argo_pangeo/argofloats_cycle.png "10 days program, cycle, of an Argo float")

Here we will focus on the Caribbean Sea surrounding the Antilles during April and May. On the 9th of April 2021 an eruption of the volcano La Soufriere Saint Vincent (Antilles) occurred. 
Another tutorial on this event is available on the [Galaxy Training Network]({% link topics/climate/tutorials/sentinel5_data/tutorial.md %}).

## Get Argo data

> <tip-title> History set up on Galaxy </tip-title>
> > <hands-on-title> History management </hands-on-title>
> >
> > Create a new history for this tutorial and give it a name (example: “Argo data with Pangeo”) for you to find it again later if needed.
> > 
> > {% snippet faqs/galaxy/histories_create_new.md box_type="none" %}
> {: .hands_on}
{: .tip}

> <hands-on-title> Arge data fetching </hands-on-title>
>
>
> 1. {% tool [Argo data access](toolshed.g2.bx.psu.edu/repos/ecology/argo_getdata/argo_getdata/0.1.15+galaxy0) %} with the following parameters:
>    - {% icon param-select %} *"We have preconfigured some mode of operations for you. What mode do you want to use?"*: `🏊 standard mode simplifies the dataset, removes most of its jargon and returns a priori good data` 
>    - {% icon param-select %} *"How do you want to select your data of interest ?"*: `🗺 For a space/time domain` 
>        - *"Input longitude min (+east/-west)"*: `-75.0` 
>        - *"Input longitude max (+east/-west)"*: `-45.0` 
>        - *"Input latitude min (+north/-south)"*: `20.0` 
>        - *"Input latitude max (+north/-south)"*: `30.0`
>        - *"Input pressure min (db)"*: `0.0` 
>        - *"Input pressure max (db)"*: `10.0` 
>        - *"Input starting date"*: `2021-04` 
>        - *"Input ending date"*: `2021-06` 
>    - {% icon param-select %} *"Which kind of dataset do you want ?"*: `Physical parameters: temperature, salinity, pressure` 
>
> 2. **Run Tool**
> 3. After a couple of minutes, an `Argo data` output will appear green in your history.
> 4. Check that your data are in netcdf format with {% icon galaxy-pencil %}, it should be `netcdf`
>
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="nectdf" box_type="tip" %}
{: .hands_on}

# Argo data analysed, managed, and visualised by the Pangeo tools

## Xarray tools

First, we'll use 2 xarray netcdf tools. xarray, formerly known as xray, is a Python package which enables us to play with gridded data.
This package shares most of its features from numpy, but in a more convenient manner by keeping track of labels in arrays.
The gridded data is mainly available in netCDF data format. Thus xarray comes in very handy while dealing with netCDF files.

## Knowing more about hour data
After fetching the required Argo data, the following stage is to obtain the meta info or metadata of the file.
The very purpose of these steps is to obtain information about dimensions, variables, global attributes, etc. The coordinate info helps to
know about the actual data entries present under the various variables.

## Get metadata

> <hands-on-title> NetCDF dataset with Xarray metadata Galaxy Tool </hands-on-title>
>
> 1. {% tool [NetCDF xarray Metadata Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/2022.3.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `output_argo` (output of **Argo data access** {% icon tool %})
>
{: .hands_on}

In the info output file, we can identify 4 different sections:

1. Dimensions: name of dimensions and corresponding number of elements;
2. Coordinates: contains coordinate arrays (longitude, latitude, level and time) with their values.
3. Data variables: contains all the variables available in the dataset. Here, we only have one variable. For each variable, we get information on its shape and values.
4. Global Attributes: at this level, we get the global attributes of the dataset. Each attribute has a name and a value.

## Coordinates information

> <hands-on-title> Get Coordinate information with Xarray Coordinate </hands-on-title>
>
> 1. {% tool [NetCDF xarray Coordinate Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/2022.3.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `output_argo` (output of **Argo data access** {% icon tool %})
>
{: .hands_on}

View {% icon galaxy-eye %} the 5 generated outputs:

1. latitude: a tabular file containing all the latitude values of our Xarray dataset;
2. longitude: a tabular file containing all the longitudes values;
3. level: this file contains information on all the different levels (here, we have surface data so level=0 meter);
4. time: this tabular file contains all the forecast times. In our case, these are relative to December 22, 2021;
5. version: this is a text file returning the Xarray package version. It is useful when publishing your Galaxy workflow.

![History with the newly generated outputs](../../images/argo_pangeo/argo_history.png "History with the newly generated outputs")

## Timeseries visualisation

> <hands-on-title> NetCDF timeseries Extractor </hands-on-title>
>
> 1. {% tool [NetCDF timeseries Extractor](toolshed.g2.bx.psu.edu/repos/ecology/timeseries_extraction/timeseries_extraction/2022.3.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input netcdf file"*: `output_argo` (output of **Argo data access** {% icon tool %})
>    - {% icon param-file %} *"Tabular of variables"*: `output` (output of **NetCDF xarray Metadata Info** {% icon tool %})
>    - *"Choose the variable to plot"*: ``
>    - *"Datetime selection"*: `No`
>    - In *"Advanced Plotting Options"*:
>        - *"Plot title"*: `Temperature`
>        - *"Label for x-axis"*: `Time`
>        - *"Label for y-axis"*: `Temperature`
>
>
{: .hands_on}

# Visualisation mapping with a Galaxy Earth System's tool

The Earth System is a complex and dynamic system that encompasses the interactions between the atmosphere, oceans, land, and biosphere.
Understanding and analyzing data from the Earth System Model (ESM) is essential, for example, to predict and mitigate the impacts of climate change.
The ESM that the project tries to implement includes coastal water dynamics, ocean bio-geochemical in-situ data, marine omics observations, volcano activities and land degradation.

To know more keep an {% icon galaxy-eye %} open for all related blogs on the [Galaxy Community Hub](https://galaxyproject.org/news/) or go try the tools directly on [Galaxy Earth System](https://earth-system.usegalaxy.eu/). 

![Galaxy Earth System welcome page image](../../images/argo_pangeo/earth_system_galaxy.png "Galaxy Earth System welcome page image")

## Ocean Data View (ODV)
Ocean Data View (ODV) is a software package for the interactive exploration, analysis and visualization of oceanographic and other geo-referenced profile, time-series, trajectory or sequence data. To know more about ODV go check the [official page](https://odv.awi.de/).

ODV is now integrated with Galaxy as an interactive tool. This kind of tool works differently than classical tools as it allows the user to interact with a dedicated graphical interface. Those tools are used to give access to Jupyter Notebooks, RStudio or R Shiny apps for example.

> <hands-on-title> Launch ODV</hands-on-title>
>
> 1. {% tool [ODV](interactive_tool_odv) %} with the following parameters:
>    - *"Select if you are using a ODV collection in a zip folder or if you have your own raw data"*: `The data you are using are Netcdf or tabular text files`
>        - {% icon param-file %} *"Netcdf or tabular text file. For text file, odv format is recommanded."*: `timeseries_tabular` (output of **NetCDF timeseries Extractor** {% icon tool %})
>
> 2. Click on **Run Tool**
>
> {% snippet faqs/galaxy/interactive_tools_open.md tool="ODV" %}
>
> > <tip-title>ODV - Disconnected</tip-title>
> > If at one point your ODV interface becomes grey with a red panel on the top "X ODV - Disconnected", do NOT panic ;) you just need to reload your tab (circular arrow on top left)
> {: .tip}
>
> > <tip-title>Copy pasting between computer and ODV</tip-title>
> > You can expand the ODV left panel (where there are 3 dots, vertically) to access the "clipboard" menu, and paste the content you want to paste on an ODV form. From there you can copy-paste everything from one side to the other. Then, click outside of this panel to collapse it.
> >
> > ![Image showing in transparent on the left of the ODV interface the clipboard](../../images/argo_pangeo/clipboard.png)
> {: .tip}
{: .hands_on}


> <hands-on-title> Visualise with ODV </hands-on-title>
>
> In the new tab in ODV.
> 1. Click on close of the pop-up screen for the check for Updates
>
> 2. Go the top left and click on **File**, then on **Open...**
>
> 3. On the pop-up screen on the left panel select **ODV**, then the folder **galaxy**, then **data**. 
> You should see a folder open it (double clicking)
>
> 4. In the bottom of the pop-up window in **Files of type** select **All Files (*)**
>
> 5. Click on **Open** in the bottom right
>
> 6. On the left smaller map right click and select **Zoom**
>
> 7. Then move your cursor on the map you should see a red rectangle moving along
>
> 8. Reduce the rectangular to have the selection you want on the map. It can be something similar to the following image (no need to be exactly the same).
>
> 9. Once you're happy with your selection click on **Enter** on your keyboard.
>
> There your data should be opening and you can now visualise them!
>
{: .hands_on}

![Visualistation on a ODV map](../../images/argo_pangeo/argo_odv_open.png "ODV map of the Argo data")

> <hands-on-title> Visualise the temperature on a map with ODV </hands-on-title>
> 1. Go on the big **+** on the top left
> 2. Select "1 SURFACE Window"
> 3. On the central white rectangular next to the map click right and and select **Properties...**
> 4. Go to the **Data** tab of the pop-up window 
> 5. Select for **Z-axis** the variable **TEMP @ N POINTS=first** as the following image.
> ![Going to the properties and modify the data to visualise the temperature](../../images/argo_pangeo/odv_properties.png)
> 6. Then, click on **OK**.
> 
> > <tip-title>Change your visualisation properties</tip-title>
> > 1. Go to the central map 
> > 2. Click right and select **Properties...**
> > 3. For example, make your data dots bigger in "Display Style" increase the number below "Symbol Size" to 30, and click **OK**
> >
> > You can now see bigger dots representing your data.
> {: .tip}
{: .hands_on}

> <hands-on-title>Save your visualisation map</hands-on-title>
> 1. Click right on the map select **Save Plot As...**
> 2. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 3. In **File name** rename your view (for example subset_temp_argo_data)
> 4. In **Files of type** select `PNG (*.png *.PNG)` and **Save** then **OK** and **OK**.
> Once you're finished with ODV :
> 5. On th top left click on **File** select **Exit**
> 6. If you want to save the other window also click on **Yes**. Here we don't need it so click **No**.
{: .hands_on}


![ODV result map](../../images/argo_pangeo/odv_result.png)

# Conclusion

Awesome! You now know how to get Argo data, then get metadata and other information within the Pangeo ecosystem and finally visualise these data with an Earth System tool,
Ocean Data View.
![The full workflow](../../images/argo_pangeo/workflow.png)


# Extra information
Coming up soon even more tutorials on and other Earth-System related trainings. Keep an {% icon galaxy-eye %} open if you are interested!

tools_json:
["Argo data access", "toolshed.g2.bx.psu.edu/repos/ecology/argo_getdata/argo_getdata/0.1.15+galaxy0", "NetCDF xarray Metadata Info", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/2022.3.0+galaxy0", "NetCDF xarray Coordinate Info", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/2022.3.0+galaxy0", "NetCDF timeseries Extractor", "toolshed.g2.bx.psu.edu/repos/ecology/timeseries_extraction/timeseries_extraction/2022.3.0+galaxy0", "ODV", "interactive_tool_odv"]


topic: climate
tutorial: panoply
title: Visualize Climate data with Panoply netCDF viewer
url: https://training.galaxyproject.org/training-material/topics/climate/tutorials/panoply/tutorial.html
difficulty: intermediate
time_estimation: 1H
tools_json: ["Panoply", "interactive_tool_panoply"]
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: d6e2bcec2651d03bec781ce5d7b7349f
last_modified: 2026-05-27T09:01:47.877072
zenodo_link: https://doi.org/10.5281/zenodo.3695482
workflows_json: []
content:
The practical aims at familiarzing you with the [Panoply](https://www.giss.nasa.gov/tools/panoply/) Galaxy interactive environment. Panoply is among the most popular tool to visualize geo-referenced data stored in [Network Common Data Form](https://en.wikipedia.org/wiki/NetCDF) (netCDF). It provides a graphical interface for inspecting (show metadata) and visualizing netCDF data. It supports many features to customize your plots and we will introduce some of them in this lesson.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Panoply documentation](https://www.giss.nasa.gov/tools/panoply/help/) ["Quick View Data with Panoply"](https://disc.gsfc.nasa.gov/information/howto?title=Quick%20View%20Data%20with%20Panoply) section.
>
{: .comment}

In this tutorial, you will learn to:
- Plot geo-referenced latitude-longitude, latitude-vertical, longitude-vertical, time-latitude or time-vertical arrays.
- Use any of numerous color tables for the scale colorbar
- Save plots to GIF, JPEG, PNG or TIFF bitmap images or as PDF or PostScript graphics files.
- Export animations as MOV or AVI videos


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
>There are many online services to get climate data, and it is often difficult to know which ones are up-to date and which resources to trust.
> Different services provide different Application Programming Interfaces (API), use different terminologies, different file formats etc., which make it difficult for new users to master them all.
> Therefore in this tutorial, we will be focusing on the usage of Climate data in [Network Common data Form](https://en.wikipedia.org/wiki/NetCDF) (netCDF) because it is the most common data format for storing Climate data.
> We will be using a freely available dataset containing Essential Climate Variables (sea ice area fraction, surface temperature) from [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/#!/home). We will learn to use panoply to visualize the sea ice area fraction over the poles (southern and northern poles) and surface temperatures for two different years (1979 and 2018).
{:  .comment}

## NetCDF format

[NetCDF](https://en.wikipedia.org/wiki/NetCDF) data format is a binary format and to be able to read or visualize it, we would need to use dedicated software or libraries that can handle this "special" format. It is self-describing and machine-independent data format that supports the creation, access, and sharing of array-oriented scientific data. NetCDF files usually have the extension *.nc* or *.netcdf*.

For climate and forecast data stored in NetCDF format there are (non-mandatory) conventions on metadata ([CF Convention](http://cfconventions.org/)).

In this tutorial, we will be using data from the [Copernicus Climate Data Store](https://zenodo.org/record/3695482/files/era5-land.nc?download=1) and more precisely a [reanalysis ERA5-Land monthly averaged dataset](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-land-monthly-means?tab=overview) for 2019. Data from Copernicus Climate Data Store is freely available but may require [free registration](https://cds.climate.copernicus.eu/user/register?destination=%2Fcdsapp%23!%2Fdataset%2Freanalysis-era5-land-monthly-means%3Ftab%3Doverview) and license agreement.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *Panoply* for example...
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the file from [Zenodo](https://doi.org/10.5281/zenodo.3697454) or from the shared data library
>
>    ```
>    https://zenodo.org/record/3697454/files/ecv_1979.nc
>    https://zenodo.org/record/3697454/files/ecv_2018.nc
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check that the datatype is **netcdf**
>
>    Files you uploaded are in netcdf format. In Galaxy, Datatypes are, by default, automatically guessed. Here, as netcdf is a derivative of the h5 format, Galaxy automatically affect the h5 datatype to netcdf files. To cope with that, one can change the datatype manually, once datasets uploaded (as shown below) OR you can directly specify datatype on the upload tool form so Galaxy will not try to automatically guess it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Rename Datasets
>
>    As "https://zenodo.org/record/3697454/files/ecv_1979.nc" is not a beautiful name and can give errors for some tools, it is a good practice to change the dataset name by something more meaningfull. For example by removing `https://zenodo.org/record/3697454/files/` to obtain `ecv_1979.nc` and `ecv_2018.nc`, respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Add a tag to the dataset corresponding to `copernicus`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Panoply

## Opening up Panoply

> <hands-on-title>Launch Panoply</hands-on-title>
>
>  Panoply is available as a Galaxy interactive environment and may not be available on all Galaxy servers.
>
> > <tip-title>Launch Panoply in Galaxy</tip-title>
> > Currently Panoply in Galaxy is available on useGalaxy.eu instance, on the "Interactive tools" tool panel section or, as all interactive tools, from the dedicated usGalaxy.eu subdomain: [Live.useGalaxy.eu](https://live.usegalaxy.eu)
> >
> > 1. Open the {% tool [Panoply](interactive_tool_panoply) %} [on UseGalaxy.eu](https://live.usegalaxy.eu/?tool_id=interactive_tool_panoply)
> > 2. Check **ecv_1979.nc** dataset selected in the netcdf input field
> > 3. Click Run Tool
> > 4. The tool will start running and will stay running permanently
> > 5. Click on the "User" menu at the top and go to "Active Interactive Tools" and locate the Panoply instance you started.
> > 6. Click on your Panoply instance
> >    ![Panoply dataset selection](../../images/select_dataset.png "Select dataset")
> > 7. Click on **ecv_1979.nc** dataset
> {: .tip}
{: .hands_on}

## Inspect metadata

> <hands-on-title>Inspect dataset</hands-on-title>
>
> 1. Inspect dataset content
>
>    Here you can look at the dataset (ecv_1979.nc) and related variables (`latitude`, `longitude`, `number`, `siconc`, etc)
>
>    > <question-title></question-title>
>    >
>    > what is the unit of the `sea ice area fraction` variable (**siconc**) variable?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The unit of **siconc** is (0-1) e.g. a real value between 0 (no ice) and 1 (fully coverted by ice).
>    > {: .solution}
>    {: .question}
>
>
> 2. Inspect the surface temperature (**t2m**) variable
>
>    > <question-title></question-title>
>    >
>    > what is the unit of this variable and its shape?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The unit of **t2m** is Kelvin (K). **t2m** is a 3D array (time, latitude, longitude).
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

## Create Geo-referenced Longitude-Latitude plot

> <hands-on-title>geographical map</hands-on-title>
>
> 1. Double click on the variable **t2m** and click on **Create**
>    ![Panoply create Latitude longitude map](../../images/panoply_t2m_map_default.png "Create map")
>
>    > <question-title></question-title>
>    >
>    > 1. What does it show?
>    > 2. What is the date of the generated plot?
>    > 3. Can you plot other dates?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The plot represent the surface temperature over the entire world.
>    > >    ![Panoply Latitude longitude map](../../images/panoply_geomap.png "Plot map")
>    > >
>    > > 2. The date of the default plot is 1st January 1979 at 00:00:00.
>    > >
>    > > 3. To plot another date, change either:
>    > > - Initial time of forecast (give a value between 1 and 12, corresponding to each month of year 1979.
>    > > - Click on the date and scroll down to select the date of your choice.
>    > {: .solution}
>    {: .question}
>
> 2. Save your plot
>    - Click on the tab **File** (from your plot window) to store your plot by selecting **Save Image As**
>    - Double click on the folder *outputs* to enter this folder and save your plot.
>    You need to make sure to save all your plot in the *outputs* folder otherwise you can loose all your plots once to close panoply.
>
> 3. Change colormap
>    **Always make sure you use color blind friendly palettes.**
>    - To change the default colormap, click on tab "**Scale**" (bottom of your plot window) and select another "**Color Table**" (you can scroll down to go through all the different available colormap).
>    - Save your plot using **Save Image As** and make sure to choose another name to avoid overwritting your preceding plot.
>    ![Panoply colormap](../../images/panoply_colormap.png "Plot colormap")
{: .hands_on}


## Changing map projection

> <hands-on-title>Change projection</hands-on-title>
>
> 1. From your previous plot window, click on Tab **Map** and change **Projection**. Try a few of them and save each of your plot with **File** --> **Save Image As**.
>
>    ![Panoply change projection](../../images/panoply_robinson.png "Change projection")
>
> 2. Create another plot window for sea ice area fraction (**siconc**) and make a new geo-referenced map
>
>    > <question-title></question-title>
>    >
>    > 1. What kind of colormap could you use to highlight the extent of sea-ice?
>    > 2. What projection would be best to use for showing the extent of sea-ice over the two poles?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Any colormap that shows low values (close to 0) in light color so we can focus on values that are close to 1. For instance, **CP_PuBu_08.cpt**.
>    > >    ![Panoply sea-ice colormap](../../images/panoply_sea-ice_colormap.png "Sea-ice colormap")
>    > > 2. Using *Orthographic* projection is best for showing the northern and southern poles. One advantage is that you can choose to center the plot over 90 degrees latitude. To have both the northern and southern poles at the same time, choose **Stereographic (Two hemispheres)**.
>    > >
>    > > ![Panoply ortho plot](../../images/panoply_sea-ice_ortho.png "Plot sea-ice using orthographic projection")
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Export Animation

> <hands-on-title>Export animation</hands-on-title>
>
> 1. From your previous plot window, click on **File** and select **Export Animation**. Save your plot using either **MOV** or **AVI** format.
>  It goes through each plot e.g. for each month and create an animation where you can see the evolution of sea-ice extent from January 1979 to December 1979.
> You will be able to download the resulting movie from Galaxy once you quit Panoply.
{: .hands_on}

## Create timeseries

> <hands-on-title>Create 1D plot</hands-on-title>
>
> 1. Double click on the variable **t2m**, click on **Create** and select **Create horizontal line plot along time axis** (make sure to switch to **time**).
> ![Panoply create 1D plot](../../images/panoply_create_1D.png "Create 1D plot")
>
>    > <question-title></question-title>
>    >
>    > 1. What was the maximum temperature in Oslo (latitude: 60 degrees North, longitude: 10.75 East) in 1979?
>    > 2. Which month was the warmest in Oslo?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The maximum temperature is about 288 K so about 15 degrees Celsius (`288 - 273.15`).
>    > > 2. The warmest month in 1979 was July.
>    > > 3. Note: The y-axis scale was set based on the first dataset. If you select a different location or time, you may need to manually adjust the scale under the Grid tab by modifying the Bottom and/or Top values to ensure the plot displays correctly.
>    > > ![Panoply 1D plot](../../images/panoply_t2m_oslo.png "Plot surface temperature Oslo")
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Quit Panoply


> <hands-on-title>Quit Panoply to keep your plots</hands-on-title>
>
> 1. To make sure all your plots stored in **outputs** folder get exported to Galaxy, you need to quit panoply:
>  **File** --> **Quit Panoply**.
> 2. Go back to your current Galaxy history and you should find **Panoply outputs**
>  ![Panoply outputs](../../images/panoply_outputs.png "Panoply outputs in Galaxy")
>
{: .hands_on}

# Conclusion


We have now learnt how to analyze climate data using Panoply. We only use one of the two datasets so we strongly encourage you to do the same exercises with the second dataset **ecv_2018.nc**. Please note that when comparing surface temperature or sea-ice area fraction from 1979 and 2018, you would not be able to conclude anything regarding climate change. For any climate studies, long term timeseries (between 20 to 30 years) are necessary to establish climate trends.


topic: climate
tutorial: sentinel5_data
title: Sentinel 5P data visualisation
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["earth-system", "volcano", "satellite data"]
content_hash: 66ac0de1c466aa171ecf152312d4ee70
last_modified: 2026-05-27T09:01:47.877187
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/sentinel5_data/tutorial.html

content:
Through this tutorial you will learn here how to access and download Copernicus Data Space Ecosystem (CDSE) data through a jupyterlab Galaxy interactive tool :

This tool enables you to leverage the Copernicus Data Space Ecosystem services and access data effortlessly.
The JupyterLab service allows you to dive into data exploration, visualization, and analysis without the hassle of installing dependencies or downloading large data sets.

Then, you can visualise these data and process them with Panoply (plots geo-referenced and other arrays from netCDF, HDF, GRIB, and other datasets).

The purpose of this tutorial is to propose a user-friendly, interactive and efficient way to explore and jointly analyze remote sensing observations, from both the atmosphere and solid earth communities, for the monitoring of the volcanic activity in case of eruption and the multi-scale impact of volcanic emissions on the atmosphere.

> <details-title>What about the data used here</details-title>
> Dataset of Sentinel 5P L2 from the 1st of April to the 30th of may 2021 of the Antilles. Especially of the La Soufriere Saint Vincent (Antilles) where a volcaninc erruption occured 9th of April. This dataset is focused on the dioxide sulfur (SO2) and Aerosol index (AI) spread out. Indeed, the knowledge of volcanic activity at a high temporal resolution is crucial for robustly determining large-scale impacts of volcanoes on atmosphere (air quality, air traffic) and climate. As such, this platform will be also of interest for scientists involved in the field of volcanic impacts at large, including institutions in charge of the monitoring of air quality and aviation safety.
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Context on Galaxy
Interactive tools are working differently than classical tools as it allows the user to interact with a dedicated graphical interface. This kind of tools is used to give access to Jupyter notebooks, RStudio or R Shiny apps for example. 

> <details-title>Short introduction on how Galaxy works</details-title>
>
> You can come back to where you left off the tutorial anytime by clicking {% icon level %}.
>
> > <hands-on-title>Log in to Galaxy</hands-on-title>
> > 1. Open your favorite browser (Chrome, Safari or Firefox as your browser, not Internet Explorer!)
> > 2. Browse to your [Galaxy instance](https://earth-system.usegalaxy.eu/)
> > 3. On the top pannel go to **Login or Register**
> >
> >
> {: .hands_on}
>
> The Galaxy homepage is divided into three panels:
> * Tools on the left
> * Viewing panel in the middle
> * History of analysis and files on the right
>
> ![Screenshot of the Galaxy interface, the tools panel is on the left, the main panel is in the center, and the history is on the right.]({% link shared/images/galaxy_interface.png %} "Galaxy interface explanation")
>
> The first time you use Galaxy, there will be no files in your history panel.
{: .details}
---


# Copernicus Data Space Ecosystem

> <hands-on-title> Launch the interactive tool </hands-on-title>
>
> 1. {% tool [Copernicus Data Space Ecosystem](interactive_tool_copernicus) %} with the following parameters:
>    - *"Do you already have a notebook?"*: `Start with a fresh notebook`
>    - *"Include data into the environment - optional "*: You don't need anything
> 2. Click on **Run Tool**
>
> 3. {% snippet faqs/galaxy/interactive_tools_open.md tool="Copernicus Data Space Ecosystem" %}
{: .hands_on}

> <hands-on-title> Navigate the jupyterlab </hands-on-title>
> 1. Go in the **notebooks** folder and open a Bash window 
> 2. Write the following: 
> > <code-in-title>Download the notebook</code-in-title>
> > ```bash
> > wget https://raw.githubusercontent.com/fair-ease/Copernicus-Data-Space-Ecosystem/main/Sentinel_5.ipynb
> > ```
> {: .code-in}
> 3. Open the new notebook that appeared on the left
> 4. You can start running your notebook. To do so you can go go on the pannel displaying button right above the notebook and click on the {% icon workflow-run %}. 
>    Everytime you press this button you can excecute a one by one the cells. 
> 5. In the **Setup** section when you execute the connection cell you'll have some actions to conduct.
> 6. First click ont the link appearing, you're notebook should look like the following :
>      ![Begining of the Sentinel 5 notebook  with the connection link visible](../../images/sentinel_5/notebook_begin.png)
> 7. Once you clicked a new window opens. If you don't have a copernicus account please create one and follow the instructions given. 
> 8. If you have an account sign in and then press the green **YES**
>      ![Image of how does the connection page looks like](../../images/sentinel_5/connection.png)
> 9. You'll be redirected to a page letting you know the connection was a success
>      ![Image of the successful connection](../../images/sentinel_5/connection_ok.png)
> 10. Now, you can go back on your jupyterlab and execute the rest of the notebook.
> 11. Once you executed the entire notebook you should have a new file in the left pannel named **sentinel5.nc**.
> 12. Select the file there and click right, and then copy
>      ![The copy command on the 3 netcdf files](../../images/sentinel_5/copy.png)
> 13. Then go back to the root of your path and go in the **ouputs** folder. There you can click right and paste the file. 
>      ![The resulting file in the outputs folder](../../images/sentinel_5/outputs.png)
> 14. This part is now finished you just have to correctly close this notebook. On the top left click on **files** and then on **Shut down**.
>      ![The shut down button to correctly close the jupyterlab](../../images/sentinel_5/shut_down.png)
>
> > <details-title> Some complementary info on the notebooks </details-title>
> > In the jupyterlab go to :
> >   - **notebooks**
> >   - Then, **openeo** 
> >   - Open the notebook **Sentinel_3.ipynb**
> > You can see it's relatively similar to the one you just worked on. You can adapt it as you want to retrieve Sentinel data of all kind.
> {: .details}
{: .hands_on}

Go back on your Galaxy window. After a couple minutes you should see your outputs turning green in your history.

> <hands-on-title>Clean your data</hands-on-title>
> Firstly, you'll can change the name of your new items by adding the extension at the end **sentinel5_SO2** of your history into **sentinel5_SO2.nc** (for the 2 others **sentinel5_aerosol_354.nc** and **sentinel5_aerosol_340.nc**). 
> 
> {% snippet  faqs/galaxy/datasets_rename.md %}
> 
> Check that your data are in netcdf format with {% icon galaxy-pencil %}, it should be `netcdf`
> 
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="nectdf" %}
> 
{: .hands-on} 


# Visualise Sentinel 5

## Global view 
> <hands-on-title> Visualise Sentinel 5 data with Panoply </hands-on-title>
>
> 1. {% tool [Panoply](interactive_tool_panoply) %} with the following parameters:
>    - *"netcdf"*: select `sentinel5_SO2.nc`, `sentinel5_aerosol_340.nc` and `sentinel5_aerosol_354.nc`
> 2. **Run Tool**
> 3. Access Panoply
> 
> {% snippet faqs/galaxy/interactive_tools_open.md tool="Panoply" %}
> 
> > <tip-title>Panoply - Disconnected</tip-title>
> > If at one point your Panoply interface becomes blank, do NOT panic ;) you just need to reload your tab (circular arrow top left)
> {: .tip}
> 
> Once in the Panoply interface :
> 5. In the pop-up window go select the 3 netcf files `sentinel5_SO2.nc`, `sentinel5_aerosol_340.nc` and `sentinel5_aerosol_354.nc` and then Open.
> 6. In the bottom of this window you have the possibilty to select what you want to **Show** go there and select "Georeferenced variables"
>      ![How to select Georeferenced variables](../../images/sentinel_5/georef_var.png)
> 7. Select one of the 3 possible data and then on the top left press **Create Plot**
> 8. In the pop-up window stay on the default choice "Create georeferenced Longitude-Latitude plot" and press **Create**
>      ![Creation of a georeferenced plot](../../images/sentinel_5/create_georef.png)
> 
> You will obtain a world map like the folowing
>      ![World map in panoply](../../images/sentinel_5/1st_map.png)
> You should see a colored spot in the Atlantic.
> 9. On your keyboard press `ctrl` (or `cmd` on Mac) and select a rectangle around the colored spot to zoom in.
> 10. In the "Overlay" tab in "Overlay 1:" select **MWDB_Coasts_Countries_1.cnob**, in order to see the islands delimitation on your map.
>      ![Zoomed-in map of the colored data](../../images/sentinel_5/final_visu.png)
> 11. Save your plot. Go on the top left to "File" then "Save Image As ..." go in the **output** folder and save.
{: .hands_on}

You can do the same plot for each data subset.

> <tip-title>Switch from one graph to another</tip-title>
> In the top of your graph there is a tab **Window** click on it there you should see the other graph made.
> An example here with the SO2 graph selected but you could switch to the AER_AI ones by checking one of their box.
> ![Image of how to switch graph](../../images/sentinel_5/switch_graph.png)
{: .tip}

> <hands-on-title> Visualise data through out time </hands-on-title>
> ## Georeferenced plot through time
> - On you graph window in the "Array(s)" you can select the day you want too see.
> ![Selection of another date for your map](../../images/sentinel_5/change_date.png)
> Some examples with changing the date :
> ![Example of the SO2 the 29th of May 2021](../../images/sentinel_5/change_date_result.png)
> ![Example of the AER AI 354 the 29th of May 2021](../../images/sentinel_5/05-29.png)
> ![Example of the AER AI 340 the 26th of May 2021](../../images/sentinel_5/05-26.png)
> 
> ## Create an animation through time
> From one of your previous plot window (the S02 one for instance), click on **File** and select **Export Animation**. Save your plot in "outputs" using either **MOV** or **AVI** format. It goes through each plot e.g. for each month and create an animation where you can see the evolution of SO2 variable from the 1st of April to the 30th of May 2021. You will be able to download the resulting movie from Galaxy once you quit Panoply.
>
> ## Create a timeserie plot
> 1. In the first tabular select the "SO2" variable and click on **Create Plot**
> 2. There you need to check the box "Create horizontal line plot along t axis" and create
> Here, you should see  timeserie as for the georeferenced plot you can swith from one day to another and also extract it as an animation through time.
{: .hands_on}

> <hands-on-title> Quit Panoply correctly </hands-on-title>
> 1. Go on the top left in **File** 
> 2. Select **Quit Panoply**
> 3. Go back to your current Galaxy history and you should find Panoply outputs
>      ![The new history with panoply outputs](../../images/sentinel_5/panoply_outputs.png)
> 4. On the top of the galaxy page click on the {% icon galaxy-scratchbook %} to activate the multi view.
> 5. Then in the panoply outputs click on the {% icon galaxy-eye %} of each of the png images.
> You should be able to rearrange the windows to see each outputs
>      ![Multi-view of the panoply plots](../../images/sentinel_5/pano_outputs.png)
> 6. For the animated file go on {% icon galaxy-save %}
> Then, in your downloads you should have a small video that you can visualise on your computer.
>      ![Video of the sulfur dioxide (SO2) evolution from april to may](../../images/sentinel_5/film.png)
{: .hands_on}

# Conclusion
Now you have finished this tutorial. You leearned how to use a jupyterlab tool with OpenEO technologies to download data and how to visualise them with Panoply.

tools_json:
["Copernicus Data Space Ecosystem", "interactive_tool_copernicus", "Panoply", "interactive_tool_panoply"]


topic: climate
tutorial: ocean-data-view
title: Ocean Data View (ODV)
difficulty: intermediate
time_estimation: 1H
tools_json: ["dedicated form", "interactive_tool_odv", "ODV instance"]
requirements_json: []
content_hash: a89ef08b3f698d9e9ff6ed8aec8b3a37
last_modified: 2026-05-27T09:01:47.875503
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/ocean-data-view/tutorial.html

content:
Through this tutorial you will learn here how to access, filter and import netCDF data through using ODV Galaxy interactive tool:

Ocean Data View (ODV) is a software package for the interactive exploration, analysis and visualization of oceanographic and other geo-referenced profile, time-series, trajectory or sequence data. To know more about ODV go check the [official page](https://odv.awi.de/)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Managing ODV Galaxy interactive tool
ODV is now integrated in Galaxy as an interactive tool. This kind of tools is working differently than classical tools as it allows the user to interact with a dedicated graphical interface. This kind of tools is used to give access to Jupyter notebooks, RStudio or R Shiny apps for example. 

To use ODV, you need to use the {% tool [dedicated form](interactive_tool_odv) %}, you can specify input datasets from your hisrtory you want to use in ODV, then press the **execute** button to launch a ODV instance. When the graphical user interface of ODV is ready to be used, a URL will be displayed at the top of the Galaxy center panel. If you don't see it, you can see and access it through the "Active InteractiveTools" space of the "User" menu or you can click on {% icon galaxy-eye %} on the tool in the history.

Once you finished your work on ODV, if you want to retrieve data and/or entire project, you need to save files in ODV/galaxy/outputs, then quit ODV properly through the "Project" Menu tab.

> <details-title>Short introduction on how Galaxy works</details-title>
>
> You can come back to where you left off the tutorial anytime by clicking {% icon level %}.
>
> > <hands-on-title>Log in to Galaxy</hands-on-title>
> > 1. Open your favorite browser (Chrome, Safari or Firefox as your browser, not Internet Explorer!)
> > 2. Browse to your [Galaxy instance](https://earth-system.usegalaxy.eu/)
> > 3. On the top pannel go to **Login or Register**
> >
> >
> {: .hands_on}
>
> The Galaxy homepage is divided into three panels:
> * Tools on the left
> * Viewing panel in the middle
> * History of analysis and files on the right
>
> ![Screenshot of the Galaxy interface, the tools panel is on the left, the main panel is in the center, and the history is on the right.]({% link shared/images/galaxy_interface.png %} "Galaxy interface explanation")
>
> The first time you use Galaxy, there will be no files in your history panel.
{: .details}


> <hands-on-title>Deploy your own ODV instance</hands-on-title>
>
> 1. Create a new history for this tutorial nd give it a name (example: “ODV for Argo”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
> 
> 2. Import netCDF data from amazonaws with this link an rename it "1900392_prof.nc" instead of "New File"
>
>    ```
>    https://argo-gdac-sandbox.s3-eu-west-3.amazonaws.com/pub/dac/coriolis/1900392/1900392_prof.nc
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}  
>
> 3. *"Select if you are using a ODV collection in a zip folder or if you have your own raw data"*: `The data you are using are a Netdcdf or tabular text files`
> 4. Select your new data before deploying {% tool [ODV instance](interactive_tool_odv) %} by pressing **execute**
> 5. Access ODV 
>
> {% snippet faqs/galaxy/interactive_tools_open.md tool="ODV" %}
>
{: .hands_on}

> <details-title> Some complementary information about your data </details-title>
> Argo is an international program to observe the interior of the ocean with a fleet of profiling floats drifting in the deep ocean currents (check it [out on the argo site](https://argo.ucsd.edu)). Argo GDAC is a dataset of 5 billion in situ ocean observations from 18.000 profiling floats (4.000 active) which started 20 years ago. Argo GDAC dataset is a collection of 18.000 NetCDF files. It is a major asset for ocean and climate science, a contributor to IOCCP reports. {% cite argodata %}
>
> These data are available on AWS registry of Open Data. This registry exists to help people discover and share datasets that are available via AWS resources. [See recent additions and learn more about sharing data on AWS](https://registry.opendata.aws/).
{: .details}

# Ocean Data View
## Map your data

> <tip-title>Copy pasting between computer and ODV</tip-title>
> You can expand the ODV left panel (where there are 3 dots, vertically) to access the "clipboard" menu, and paste the content you want to paste on a ODV form. From there you can copy paste everything from one side to the other. Then, click outside of this panel to collapse it.
> 
> ![Image showing in transparent on the left of the ODV interface the clipboard](../../images/coastal_water_dyn/clipboard.png)
{: .tip}

> <tip-title>ODV - Disconnected</tip-title>
> If at one point your ODV interface becomes grey with a red panel "X ODV - Disconnected", do NOT panic ;) you just need to reload your tab (circular arrow top left)
{: .tip}

> <hands-on-title>Loading Argo data</hands-on-title>
>
> 1. Go to the top left of ODV and click on "Open..."
> 2. Click on "1900392_prof.nc" of the pop-up screen and on  **Open**
> 3. Press **Next >** for "Select Dimensions (Step 1 of 4)"
> 4. Press **Next >** for "Associate Meta Variables (Step 2 of 4)"
> 5. Press **Next >** for "Select Primary Variable (Step 3 of 4)"
> 6. Press **Finish** for "Subset Dimensions (Step 4 of 4)"
> 7. "Enter netCDF Setup Name:" write "argo_prof_data", then, press **OK**
{: .hands_on}
![Map of all the stations of the Argo glider](../../images/coastal_water_dyn/argo_stations.png)

> <question-title></question-title>
>
> 1. What are the longitude and latitude of the red dot ?
>
> > <solution-title></solution-title>
> >
> > 1. On the to right window you can read Longitude 7°E and Latitude 21°S.
> >
> {: .solution}
>
{: .question}

## Create profiles curves

> <hands-on-title>Create a multiple windows view</hands-on-title>
> 1. On the top left press **+**
> 2. Click on **2 Station Windows**
>    ![Image showing where to go to have the 2 STATION Windows](../../images/coastal_water_dyn/2_station.png)
> 3. Two more slots appear follow the indication and press + on your keyboard (or SHIFT and +).
>
> Here you have created a multiple view on ODV
{: .hands_on}

> <hands-on-title>Create profiles</hands-on-title>
> 1. On the biggest window click on the right side of your mousse
> 2. Select **X-Variable**, press **TEMP [degree_Celsius]** and OK.
>    ![Image of the selection of the pression](../../images/coastal_water_dyn/xvar_temp.png)
> 3. Select **Y-Variable**, press **PRES [decibar]** and OK. Your profile is now uspide down. 
> 4. To correct that click right again go on **Y-Variable** and on the bottom of the pop-up screen check the box "Reverse range" and OK.
>    ![ODV interface with the map and the 1st profile pression temperature](../../images/coastal_water_dyn/1st_profile.png)
> 5. For the smallest window do the same but for the abscisse select **TEMP_ADJUSTED [degree_Celsius]**. This can be usefull to compare the 2 curves.
>    ![Resulting image of the 2 windows configurated](../../images/coastal_water_dyn/2_profiles.png)
>
> You now know how to create multiple profiles curves on ODV from netCDF data.
{: .hands_on}

> <question-title></question-title>
>
> How do you do if you want to compare the profiles with another station ?
>
> > <solution-title></solution-title>
> >
> > With your keyboard you can use the arrows left and right and go through the different stations one-by-one.
> > ![See the profiles of another station](../../images/coastal_water_dyn/2_curves.png)
> >
> {: .solution}
>
{: .question}

> <hands-on-title>Change the station of reference</hands-on-title>
> 1. Go on the map on the bottom left 
> 2. Click right and select **Station Filter**
> ![Image on how to select the option Station Filter](../../images/coastal_water_dyn/station_filter.png)
> 3. Then, select and click on **Customize**
> 4. In "Station ID" write the number of the new station of reference (both boxes)
> 
>    Everything has dissapeared !!! Don't panik it's totally normal ;)
>    - Press **+** on your keyboard (or SHIFT and +)
> 
>    The curves reapeared ! And they correspond to your knew station of reference !
> 
>    To make the other stations reapear, to allow you to navigate in the other stations while conserving your new station of reference, go on the map once again.
> 6. Click right and select **Station Filter**
> 7. This time select and click on **Relax**
> 
> And now you're back with all your stations visible !
{: .hands_on}

## Create one single sum-up profile curve 
> <hands-on-title>Create a scatter plot</hands-on-title>
> 1. On the top left press **+**
> 2. Select **1 SCATTER Window** another tab opens (you now have 3 of them)
> ![Image of the new open tab for the preparation of a scatter view](../../images/coastal_water_dyn/scatter_prep.png)
> 1. On the window click on the right side of your mousse
> 4. Select **X-Variable**, press **TEMP [degree_Celsius]** and OK.
> 5. Select **Y-Variable**, press **PRES [decibar]**, at the bottom of the pop-up check the box "Reverse range" and OK.
> ![Selection of the Pression and check of the Reverse range button on the pop-up image](../../images/coastal_water_dyn/yvar_pres.png)
> You now have the final image with all the curves for each stationside by side. You can navigate in them by using the arrows left/right on your keyboard with the red curve being the station you're on. 
> ![Resulting image of the scatter plot](../../images/coastal_water_dyn/final_scatter.png)
{: .hands_on}

> <hands-on-title>Save your analysis</hands-on-title>
> 1. Click right on the scatter image and select **Save plot as**
> 2. In the pop-up select and go in the **ODV/galaxy/outputs** folder
> 3. Once you're in the folder you can rename your scatter plot in "File Name:" by "1900392_prof_scatter"
> 4. You can also change the "Files of type:" by "PNG (*.png *.PNG) 
> 5. Click on **Save**, then OK and OK.
> 6. On the map click right and then **Save Map As...**
> 7. Still in the ODV/galaxy/outputs folder, rename it "1900392_prof_map" check that the "Files of type:" is "PNG (*.png *.PNG) 
> 8. Click on **Save**, then OK and OK
>
> Now, if you have finished with your analysis you can exit ODV. To do so you need to do it properly. 
>
> 9. On th top left click on **File** select **Exit**
> 10. If you want to save the other window also click on **Yes**. Here we don't need it so click **No**.
>
> You can now go back on your Galaxy instance.
{: .hands_on}

Now, after waiting for everything to turn green in your history, you can see 2 new outputs "1900392_prof_scatter" and "1900392_prof_map"

> <hands-on-title>View in Galaxy</hands-on-title>
> In the history panel click on the {% icon galaxy-eye %} (eye) icon of your output.
>
> You can now visualize the outputs in galaxy middle pannel. 
{: .hands_on}

![Image in the middle pannel of "1900392_prof_scatter"](../../images/coastal_water_dyn/scatter.png)
![Image in the middle pannel of "1900392_prof_map"](../../images/coastal_water_dyn/map.png)


# Conclusion

Great you now know how to use argo floater data to create profiles curves in Ocean Data View (ODV) within Galaxy.

# Extra information
Coming up soon follow ups tutorials on Coastal Water Dynamics workflow (and other Earth-System related trainings). Keep an {% icon galaxy-eye %} open if you are interested !

tags_json:
["earth-system", "ocean", "geographical information system", "ODV", "netcdf data", "maps", "marine data"]


topic: climate
tutorial: ocean-variables
title: Ocean's variables study
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["earth-system", "ocean", "netcdf data"]
content_hash: 9ce619ce1e81e4ec15a3f5b8a831240a
last_modified: 2026-05-27T09:01:47.875648
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/ocean-variables/tutorial.html

content:
Through this tutorial, you will learn in the first part how to import, visualise, and extract data from an ODV collection by using the ODV Galaxy interactive tool.
In the second part, you will learn how to use DIVAnd using the inputs the outputs from ODV.

Ocean Data View (ODV) is a software package for the interactive exploration, analysis and visualization of oceanographic and other geo-referenced profile, time-series, trajectory,
or sequence data. To know more about ODV go check the [official page](https://odv.awi.de/)

DIVAnd (Data-Interpolating Variational Analysis in n dimensions) performs an n-dimensional variational analysis/gridding of arbitrarily located observations.
Observations will be interpolated/analyzed on a curvilinear grid in 1, 2, 3 or more dimensions. See the [official page](https://gher-uliege.github.io/DIVAnd-presentation/#1)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <details-title> Some complementary information about your data </details-title>
> The data here are Mediterranean Sea - Eutrophication and Acidity aggregated datasets
> EMODnet Chemistry aims to provide access to marine chemistry datasets and derived data products concerning eutrophication, acidity, and contaminants. The importance of the selected substances and other parameters relates to the Marine Strategy Framework Directive (MSFD). This aggregated dataset contains all unrestricted EMODnet Chemistry data on eutrophication and acidity and covers the Mediterranean Sea. Data were aggregated and quality controlled by the 'Hellenic Centre for Marine Research, Hellenic National Oceanographic Data Centre (HCMR/HNODC)' in Greece. To get more ODV collection and other data please  [search the sextant catalogue](https://sextant.ifremer.fr/eng/Data/Catalogue#/search?isTemplate=n&from=1&to=30&sortBy=dateStamp&sortOrder=desc&languageStrategy=searchInDetectedLanguage&any=EMODnet%20Chemistry) or the [EMODnet Chemistry catalogue](https://emodnet-chemistry.webodv.awi.de/).
>
>  ITS-90 water temperature and water body salinity variables have also been included ('as are') to complete the eutrophication and acidity data. If you use these variables for calculations, please refer to SeaDataNet for the quality flags: https://www.seadatanet.org/Products/Aggregated-datasets.
>
> Regional datasets concerning eutrophication and acidity are automatically harvested, and the resulting collections are aggregated and quality-controlled using ODV Software and following a common methodology for all sea regions {% cite emodnet %}. Parameter names are based on P35 vocabulary, which relates to EMODnet Chemistry aggregated parameter names and is available at: https://vocab.nerc.ac.uk/search_nvs/P35/.
>
> When not present in the original data, water body nitrate plus nitrite was calculated by summing all nitrate and nitrite parameters. The same procedure was applied for water body dissolved inorganic nitrogen (DIN), which was calculated by summing all nitrate, nitrite, and ammonium parameters. Concentrations per unit mass were converted to a unit volume using a constant density of 1.025 kg/L. {% cite hcmrdata %}
>
{: .details}

{% include _includes/cyoa-choices.html option1="Tools" option2="Workflow" default="tools" disambiguation="tutorialstyle"
       text="Do you want to run the workflow or to discover the tools one by one ?" %}

<div class="Workflow" markdown="1">

# Manage your data

> <hands-on-title>Prepare your history</hands-on-title>
> 1. Create a new history for this tutorial and give it a name (for example “Ocean's variables”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import an ODV collection as a zip folder.
>
>    Use this link `https://github.com/fair-ease/Data-for-Galaxy-/raw/main/Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44.zip` in the folowing instructions.
>    And **Name** it **Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44.zip**
>
>    {% snippet  faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Get the Bathymetry
> First you need to download a bathymetry file you can use this link <https://dox.ulg.ac.be/index.php/s/U0pqyXhcQrXjEUX/download> with the folowing Hands-on box where you can name this dataset **bathymetry.nc**
>
> {% snippet faqs/galaxy/datasets_import_via_link.md box_type="hands-on" %}
>
> Check that the file is in netcdf !
>
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="nectdf" box_type="tip" %}
>
{: .hands_on}

# Import and launch the workflow
> <hands-on-title>Import the workflow</hands-on-title>
>   - Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
>   - Option 1: use the URL
>   	- Click on {% icon galaxy-upload %} Import at the top-right of the screen
>   	- Paste the URL of the workflow into the box labelled “Archived Workflow URL” `https://earth-system.usegalaxy.eu/u/marie.josse/w/oceans-var-v2`
>   - Option 2: use the workflow name
>   	- Click on **Public workflows** at the top-right of the screen
>       - Search for **Ocean's variables 2.0**
>       - In the workflow preview box click on {% icon galaxy-upload %} Import
>   - Click the Import workflow button
{: .hands_on}

> <hands-on-title>Run the workflow</hands-on-title>
>    - Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
>    - Click on the {% icon workflow-run %} (Run workflow) button next to your workflow
>    - /!\ Select **Yes** for **Workflow semi automatic**
>    - Configure the workflow as needed with the 2 datasets you uploaded right before (**Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44.zip** and **gebco_30sec_8.nc**
>    - Click the Run Workflow button at the top-right of the screen
>    - You may have to refresh your history to see the queued jobs
{: .hands_on}

# Workflow human in the loop

Now you've got your workflow running, however this workflow is composed of both interactive and non interactive tools. Thus, for the interactive tools we need you !

## Ocean Data view: Managing an interactive tool
ODV is now integrated in Galaxy as an interactive tool. This kind of tool works differently than classical tools as it allows the user to interact interactively with your data.
This kind of tool is used to give access to Jupyter Notebooks, RStudio or R Shiny apps for example.

To use ODV, you need to use the {% tool [dedicated form](interactive_tool_odv) %}, you can specify input datasets from your history you want to use in ODV,
then press the **execute** button to launch an ODV instance. When the graphical user interface of ODV is ready to be used, a URL will be displayed at the top of the
Galaxy center panel. If you don't see it, you can see and access it through the "Active InteractiveTools" space of the "User" menu or you can click on {% icon galaxy-eye %} on
the tool in the history.

Once you finish your work on ODV, if you want to retrieve data and/or the entire project, you need to save files in ODV/galaxy/outputs, then quit ODV properly through
the "Project" Menu tab.

To get in the interactive tool do the folowing

{% snippet faqs/galaxy/interactive_tools_open.md tool="ODV" %}

### Visualise your Data

> <tip-title>Copy pasting between computer and ODV</tip-title>
> You can expand the ODV left panel (where there are 3 dots, vertically) to access the "clipboard" menu and paste the content you want to paste on an ODV form. From there you can copy-paste everything from one side to the other. Then, click outside of this panel to collapse it.
>
> ![Image showing in transparent on the left of the ODV interface the clipboard](../../images/coastal_water_dyn/clipboard.png)
{: .tip}

> <tip-title>ODV - Disconnected</tip-title>
> If at one point your ODV interface becomes grey with a red panel on the top "X ODV - Disconnected", do NOT panic ;) you just need to reload your tab (circular arrow top left)
{: .tip}

> <hands-on-title>Loading data</hands-on-title>
>
> 1. Click on close of the pop-up screen for the check for Updates
> 2. Go the top left and click on **File**, then on **Open...**
> 3. On the pop-up screen on the left panel select **ODV**, then the folder **galaxy**, then **data**.
> You should see a folder open it (double clicking)
> 4. Select the file with a .odv extension
> 	![Screenshot of what your pop-up screen should be like when selecting the right data](../../images/ocean_var/select_data.png)
> 5. Click on **Open** in the bottom right
>
> There your data should be opening an you can now visualise them!
{: .hands_on}
![Visualistation on an ODV map of the ODV collection](../../images/ocean_var/visualise_data.png)

> <question-title></question-title>
>
> 1. What are the longitude and latitude of the red dot?
>
> > <solution-title></solution-title>
> >
> > 1. On the to right window you can read Longitude 34°E and Latitude 32.332°N.
> >
> {: .solution}
>
{: .question}

### Subset Data

> <hands-on-title>Create a subset</hands-on-title>
> 1. On the left smaller map right click and select **Zoom**
> 2. Then move your cursor on the map you should see a red rectangle moving along
> 3. Reduce the rectangular to have the selection you want on the map. It can be something similar to the following image (no need to be exactly the same)
>	![Selection on the map of the part of the data that you want to analyse](../../images/ocean_var/select_subset.png)
> 4. Once you're happy with your selection click on **Enter** on your keyboard.
>	![Result of the subsetting and how the maps should look now](../../images/ocean_var/subset.png)
>
> Here you have created a a subset of your data.
{: .hands_on}

> <tip-title>Change your visualisation properties</tip-title>
> 1. Go to the central map
> 2. Click right and select **Properties...**
> 3. For example, make your data dots bigger in "Display Style" increase the number below "Symbol Size" to 50, and click **OK**
> ![Image on how to change the size of your dots](../../images/ocean_var/size_dots.png)
>
> You can now see bigger dots representing your data.
> ![Image of your maps after the increase of the dots' size](../../images/ocean_var/big_dots.png)
>
> If you already saved it once you just have to click left on the save icon (top left) when it's red.
{: .tip}

### Save Data

{% include _includes/cyoa-choices.html option1="xview" option2="png" default="png" disambiguation="savedata"
       text="Here you can choose if you want to save your view as an ODV view in xview format (you will not able to directly visualise it on Galaxy) or if you want to save it in png which you can visualise on Galaxy." %}

<div class="xview" markdown="1">
> <hands-on-title>Save your subset view</hands-on-title>
> 1. On the top left of your screen, you can see a red save button. Right-click on it.
> 2. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 3. In **File name** rename your view (for example Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44), and **Save**.
{: .hands_on}
</div>
<div class="png" markdown="1">
> <hands-on-title>Save your subset map</hands-on-title>
> 1. Click right on the map and select **Save Plot As...**
> 2. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 3. In **File name** rename your view (for example Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44)
> 4. In **Files of type** select `PNG (*.png *.PNG)` and **Save** then **OK** and **OK**.
{: .hands_on}
</div>


> <hands-on-title>Extract your variables in netcdf data</hands-on-title>
> Now we want to extract and save the right parameters of your data in netcdf format.
> 1. Go to the the left and click on **Export**, **Data** and **NetCDF File...**
> 2. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 3. Click **Save**
> 4. A new pop-up window opens "Select Extended Metadata Variables for Export" Let the 56 items selected and click **OK**
> 5. "Select Data Variables for Export"  `here you need to select 1: Depth[m]`, ̀`4: Water body dissolved oxygen concentration [umol/l]`, `6: Water body phosphate [umol/l]` and click **OK**
>	![Screenshot of the variables one needs to select](../../images/ocean_var/select_var.png)
> 6. "NetCDF File Properties" change the **Longitude range** to `[-180 ... 180] degrees_E`, then select `Export metadata quality flags` and `Export data quality flags` and **OK**.
> 7. And **OK** again
>
> You now know how to export and save the right variables on ODV to netCDF data.
{: .hands_on}

Now, if you have finished with your analysis you can exit ODV. To do so you need to do it properly.

> <hands-on-title>Exit ODV and go back on Galaxy</hands-on-title>
> 1. On the top left click on **File** select **Exit**
> 2. If you want to save the other window also click on **Yes**. Here we don't need it so click **No**.
>
> You can now go back to your Galaxy instance.
{: .hands_on}

## DIVAnd: Data-Interpolating Variational Analysis in n dimensions
You can see now in your Galaxy history that ODV turned green and that the folowing step launched itself.
This tool DIVAnd is not interactive. It will create a climatology in netcdf that will then be used as input in the next step which is ODV again.
Once, DIVAnd finished ODV get started and there we need you again !


## BONUS: Even more ODV

Do you want to go even further and visualise your new netcdf data? (Because this tutorial is fascinating and you want to keep reading ;))

> <hands-on-title>Open your Netcdf data in ODV</hands-on-title>
>
> {% snippet faqs/galaxy/interactive_tools_open.md tool="ODV" %}
>
> 1. Click on close of the pop-up screen for the check for Updates
> 2. Go the top left and click on **File**, then on **Open...**
> 3. On the pop-up screen on the left panel select **ODV**, then the folder **galaxy**, then **data**.
> You should see a file open it (double clicking)
> 4. Click on **Open** in the bottom right. Then click on **Next** until you can click on **Finish** you can keep everything by default.
>
> There your data should be opening and you can now visualise them!
> ![Image of what the netcdf files look like when opening with ODV](../../images/ocean_var/med_view.png)
>
{: .hands-on}

> <hands-on-title>Visualise your Phosphate variable</hands-on-title>
> 1. Go on the big **+** on the top left
> 2. Select "1 SURFACE Window"
>	![Create a surface plot for the phosphate](../../images/ocean_var/surface_window.png)
> 3. Press right on the new plot and select **Properties...**
>	![Going to the properties of the plot](../../images/ocean_var/properties.png)
> 4. In the tab "Display Style" increase the "Symbol size" to 22 for instance
> 5. In the tab "Color Mapping" increase the level of the "Median" part and lower the "Nonlinearity" such as in the following picture
> 	![Balance the colors of the map to better visualise the Phosphate repartition](../../images/ocean_var/colors.png)
> 6. Then, click on **OK**, and you can see the Phosphate repartition.
>
> 7. Click right on the map and select **Save Plot As...**
> 8. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 9. In **Files of type** select `PNG (*.png *.PNG)` and **Save** then **OK** and **OK**.
>
> If you want to have a visualisation of the Phosphate according to the depth, you can do the following.
> 10. Go on the big **+** on the top left
> 11. Select "1 STATION Window" in the new tab, in the plot press enter on your keyboard.
> 12. Same as before to save it, click right on the map and select **Save Plot As...**
> 13. In **Files of type** select `PNG (*.png *.PNG)` and **Save** then **OK** and **OK**.
>
> You can now safely exit ODV.
> 14. On the top left click on **File** select **Exit**
> 15. If you want to save the other window also click on **Yes**. Here we don't need it so click **No**.
{: .hands-on}
</div>

<div class="Tools" markdown="1">
# Managing ODV Galaxy interactive tool

ODV is now integrated in Galaxy as an interactive tool. This kind of tool works differently than classical tools as it allows the user to interact interactively with your data.
This kind of tool is used to give access to Jupyter Notebooks, RStudio or R Shiny apps for example.

To use ODV, you need to use the {% tool [dedicated form](interactive_tool_odv) %}, you can specify input datasets from your history you want to use in ODV,
then press the **execute** button to launch an ODV instance. When the graphical user interface of ODV is ready to be used, a URL will be displayed at the top of the
Galaxy center panel. If you don't see it, you can see and access it through the "Active InteractiveTools" space of the "User" menu or you can click on {% icon galaxy-eye %} on
the tool in the history.

Once you finish your work on ODV, if you want to retrieve data and/or the entire project, you need to save files in ODV/galaxy/outputs, then quit ODV properly through
the "Project" Menu tab.

> <details-title>Short introduction on how Galaxy works</details-title>
>
> You can come back to where you left off the tutorial anytime by clicking {% icon level %}.
>
> > <hands-on-title>Log in to Galaxy</hands-on-title>
> > 1. Open your favorite browser (Chrome, Safari or Firefox as your browser, not Internet Explorer!)
> > 2. Browse to your [Galaxy instance](https://earth-system.usegalaxy.eu/)
> > 3. On the top panel go to **Login or Register**
> >
> >
> {: .hands_on}
>
> The Galaxy homepage is divided into three panels:
> * Tools on the left
> * Viewing panel in the middle
> * History of analysis and files on the right
>
> ![Screenshot of the Galaxy interface, the tools panel is on the left, the main panel is in the center, and the history is on the right.]({% link shared/images/galaxy_interface.png %} "Galaxy interface explanation")
>
> The first time you use Galaxy, there will be no files in your history panel.
{: .details}


> <hands-on-title>Deploy your own ODV instance</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (for example “Ocean's variables”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import an ODV collection as a zip folder.
>
>    Use this link `https://github.com/fair-ease/Data-for-Galaxy-/raw/main/Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44.zip` in the folowing instructions.
>    And **Name** it **Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44.zip**
>
>    {% snippet  faqs/galaxy/datasets_import_via_link.md %}
>
>
> 4. {% tool [Ocean Data View](interactive_tool_odv) %} with the following parameters:
>    - *"Select if you are using an ODV collection in a zip folder or if you have your own raw data"*: `The data you are using are an ODV collection in a zip folder`
>    - *"ODV collection in a zip folder."*: `Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44.zip`
>
>	![Screenshot of what parameters to input in ODV before running the tool](../../images/ocean_var/launching_odv.png)
>
> 4. Click on **Run Tool**
>
> {% snippet faqs/galaxy/interactive_tools_open.md tool="ODV" %}
>
{: .hands_on}

# Ocean Data View
## Visualise your Data

> <tip-title>Copy pasting between computer and ODV</tip-title>
> You can expand the ODV left panel (where there are 3 dots, vertically) to access the "clipboard" menu and paste the content you want to paste on an ODV form. From there you can copy-paste everything from one side to the other. Then, click outside of this panel to collapse it.
>
> ![Image showing in transparent on the left of the ODV interface the clipboard](../../images/coastal_water_dyn/clipboard.png)
{: .tip}

> <tip-title>ODV - Disconnected</tip-title>
> If at one point your ODV interface becomes grey with a red panel on the top "X ODV - Disconnected", do NOT panic ;) you just need to reload your tab (circular arrow top left)
{: .tip}

> <hands-on-title>Loading data</hands-on-title>
>
> 1. Click on close of the pop-up screen for the check for Updates
> 2. Go the top left and click on **File**, then on **Open...**
> 3. On the pop-up screen on the left panel select **ODV**, then the folder **galaxy**, then **data**.
> You should see a folder open it (double clicking)
> 4. Select the file with a .odv extension
> 	![Screenshot of what your pop-up screen should be like when selecting the right data](../../images/ocean_var/select_data.png)
> 5. Click on **Open** in the bottom right
>
> There your data should be opening an you can now visualise them!
{: .hands_on}
![Visualistation on an ODV map of the ODV collection](../../images/ocean_var/visualise_data.png)

> <question-title></question-title>
>
> 1. What are the longitude and latitude of the red dot?
>
> > <solution-title></solution-title>
> >
> > 1. On the to right window you can read Longitude 34°E and Latitude 32.332°N.
> >
> {: .solution}
>
{: .question}

## Subset Data

> <hands-on-title>Create a subset</hands-on-title>
> 1. On the left smaller map right click and select **Zoom**
> 2. Then move your cursor on the map you should see a red rectangle moving along
> 3. Reduce the rectangular to have the selection you want on the map. It can be something similar to the following image (no need to be exactly the same)
>	![Selection on the map of the part of the data that you want to analyse](../../images/ocean_var/select_subset.png)
> 4. Once you're happy with your selection click on **Enter** on your keyboard.
>	![Result of the subsetting and how the maps should look now](../../images/ocean_var/subset.png)
>
> Here you have created a a subset of your data.
{: .hands_on}

> <tip-title>Change your visualisation properties</tip-title>
> 1. Go to the central map
> 2. Click right and select **Properties...**
> 3. For example, make your data dots bigger in "Display Style" increase the number below "Symbol Size" to 50, and click **OK**
> ![Image on how to change the size of your dots](../../images/ocean_var/size_dots.png)
>
> You can now see bigger dots representing your data.
> ![Image of your maps after the increase of the dots' size](../../images/ocean_var/big_dots.png)
>
> If you already saved it once you just have to click left on the save icon (top left) when it's red.
{: .tip}

## Save Data

{% include _includes/cyoa-choices.html option1="xview" option2="png" default="png" disambiguation="savedata2"
       text="Here you can choose if you want to save your view as an ODV view in xview format (you will not able to directly visualise it on Galaxy) or if you want to save it in png which you can visualise on Galaxy." %}

<div class="xview" markdown="1">
> <hands-on-title>Save your subset view</hands-on-title>
> 1. On the top left of your screen, you can see a red save button. Right-click on it.
> 2. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 3. In **File name** rename your view (for example Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44), and **Save**.
{: .hands_on}
</div>
<div class="png" markdown="1">
> <hands-on-title>Save your subset map</hands-on-title>
> 1. Click right on the map and select **Save Plot As...**
> 2. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 3. In **File name** rename your view (for example Eutrophication_Med_profiles_2022_unrestricted_SNAPSHOT_2023-10-24T16-39-44)
> 4. In **Files of type** select `PNG (*.png *.PNG)` and **Save** then **OK** and **OK**.
{: .hands_on}
</div>


> <hands-on-title>Extract your variables in netcdf data</hands-on-title>
> Now we want to extract and save the right parameters of your data in netcdf format.
> 1. Go to the the left and click on **Export**, **Data** and **NetCDF File...**
> 2. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 3. Click **Save**
> 4. A new pop-up window opens "Select Extended Metadata Variables for Export" Let the 56 items selected and click **OK**
> 5. "Select Data Variables for Export"  `here you need to select 1: Depth[m]`, ̀`4: Water body dissolved oxygen concentration [umol/l]`, `6: Water body phosphate [umol/l]` and click **OK**
>	![Screenshot of the variables one needs to select](../../images/ocean_var/select_var.png)
> 6. "NetCDF File Properties" change the **Longitude range** to `[-180 ... 180] degrees_E`, then select `Export metadata quality flags` and `Export data quality flags` and **OK**.
> 7. And **OK** again
>
> You now know how to export and save the right variables on ODV to netCDF data.
{: .hands_on}

Now, if you have finished with your analysis you can exit ODV. To do so you need to do it properly.

> <hands-on-title>Exit ODV and go back on Galaxy</hands-on-title>
> 1. On the top left click on **File** select **Exit**
> 2. If you want to save the other window also click on **Yes**. Here we don't need it so click **No**.
>
> You can now go back to your Galaxy instance.
> Now, after waiting for everything to turn green in your history, you can see 2 new outputs
> ![Screenshot of the 2 new outputs present in your galaxy History](../../images/ocean_var/history.png)
>
> In the history panel in the output **ODV all outputs** click on the {% icon galaxy-eye %} (eye) icon of your output.
>
> You can now visualize the outputs in Galaxy middle panel.
>
> ![Screenshot of the text output showing the variables selected](../../images/ocean_var/text.png)
> ![Image in the middle pannel of the map](../../images/ocean_var/galaxy_output.png)
{: .hands_on}

# DIVAnd: Data-Interpolating Variational Analysis in n dimensions

> <tip-title>Check Datatype</tip-title>
> Go to your output 'data_from_Eutrophication_Med_profiles_2022_unrestricted'
>
> Check if the datatype is **netcdf** if not :
>
> {% snippet  faqs/galaxy/datasets_change_datatype.md %}
{: .tip}

{% include _includes/cyoa-choices.html option1="New-user" option2="Experimented-user" default="New-user" disambiguation="userlevel"
       text="If you want the easiest tutorial possible choose New user. If you feel that you need more flexibility on the DIVAnd parameters choose Experimented user." %}

<div class="New-user" markdown="1">

You chose New user ! Thus, you'll be running a batch tool, you just have to fill in the input like explained below.

## Get the Bathymetry
First you need to download a bathymetry file you can use this link <https://dox.ulg.ac.be/index.php/s/U0pqyXhcQrXjEUX/download> with the folowing Hands-on box where you can name this dataset **bathymetry.nc**

{% snippet faqs/galaxy/datasets_import_via_link.md box_type="hands-on" %}

Check that the file is in netcdf !

{% snippet faqs/galaxy/datasets_change_datatype.md datatype="nectdf" box_type="tip" %}

## Use DIVAnd
> <hands-on-title> Create a climatology </hands-on-title>
>
> 1. {% tool [DIVAnd](toolshed.g2.bx.psu.edu/repos/ecology/divand_full_analysis/divand_full_analysis/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input your netcdf data"*: `ODV netcdf outputs` (output of **Ocean Data View** {% icon tool %})
>    - {% icon param-file %} *"Input your bathymetry netcdf file"*: `bathymetry.nc`
>    - *"Input longitude min (+east/-west)"*: `19.0`
>    - *"Input longitude max (+east/-west)"*: `30.0`
>    - *"Input latitude min (+north/-south)"*: `32.0`
>    - *"Input latitude max (+north/-south)"*: `38.0`
>    - *"Input starting date"*: `1950-01-01`
>    - *"Input ending date"*: `2017-12-31`
>    - *"Write the name of the variable of the analysis"*: `phosphate`
>    - *"Minimum of the salinity"*: `0`
>    - *"Maximum of the salinity"*: `100`
>
> 2. **Run Tool**
> 3. After a couple minutes an `DIVAnd netcdf output` output will appear green in your history.
> 4. Check that your data are in netcdf format with {% icon galaxy-pencil %}, it should be `netcdf`
>
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="nectdf" box_type="tip" %}
{: .hands_on}
</div>

<div class="Experimented-user" markdown="1">

You chose Experimented user ! Thus, you'll be launching a Jupyterlab and you'll need to interact with it.

## Launch DIVAnd

Use ODV outputs as DIVAnd input.
> <hands-on-title>Run DIVANnd</hands-on-title>
>
> 1. Use {% tool [DIVAnd](interactive_tool_divand) %} with the following parameters:
>    - *"Do you already have a notebook"*: `Start with a fresh notebook`
>    - *"Include data into the environment"*: `ODV netcdf outputs` (output of **Ocean Data View** {% icon tool %})
> 2. **Run tool**
>
>    {% snippet faqs/galaxy/interactive_tools_open.md tool="DIVAnd" %}
{: .hands_on}

> <hands-on-title>Navigate the jupyterlab</hands-on-title>
> 1. Once in the Jupyterlab, go to the **Notebooks** folder.
> 2. There open a terminal window and write :
>    ```bash
>    wget https://raw.githubusercontent.com/fair-ease/Divand-Galaxy-ext/main/90-analysis-shorten.ipynb
>    ```
> 3. **Run the entire notebook.**
>
>    In the first 'outputs' folder (at the root of the Jupyterlab folders), you should now have one netcdf file "Water_body_Phosphate_Mediterranean".
>
>    So, you can now properly close the Jupyter notebook.
>
> 4. On the top left click on **files** and then on **Shut down**.
> 5. Go back to your Galaxy instance.
{: .hands-on}

After a couple of minutes, your outputs should appear in your Galaxy history.
</div>

# BONUS: Even more ODV

Do you want to go even further and visualise your new netcdf data? (Because this tutorial is fascinating and you want to keep reading ;))

> <hands-on-title>Open your Netcdf data in ODV</hands-on-title>
>  {% tool [Ocean Data View](interactive_tool_odv) %} with the following parameters:
>    - *"Select if you are using an ODV collection in a zip folder or if you have your own raw data"*: `The data you are using are Netcdf or text tabular files`
>    - *"Netcdf or tabular text file. For text file, odv format is recommanded."*: `DIVAnd netcdf outputs` (output of **DIVAnd** {% icon tool %})
>
> Click on **Run Tool**
>
> {% snippet faqs/galaxy/interactive_tools_open.md tool="ODV" %}
>
> 1. Click on close of the pop-up screen for the check for Updates
> 2. Go the top left and click on **File**, then on **Open...**
> 3. On the pop-up screen on the left panel select **ODV**, then the folder **galaxy**, then **data**.
> You should see a file open it (double clicking)
> 4. Click on **Open** in the bottom right. Then click on **Next** until you can click on **Finish** you can keep everything by default.
>
> There your data should be opening and you can now visualise them!
> ![Image of what the netcdf files look like when opening with ODV](../../images/ocean_var/med_view.png)
>
{: .hands-on}

> <hands-on-title>Visualise your Phosphate variable</hands-on-title>
> 1. Go on the big **+** on the top left
> 2. Select "1 SURFACE Window"
>	![Create a surface plot for the phosphate](../../images/ocean_var/surface_window.png)
> 3. Press right on the new plot and select **Properties...**
>	![Going to the properties of the plot](../../images/ocean_var/properties.png)
> 4. In the tab "Display Style" increase the "Symbol size" to 22 for instance
> 5. In the tab "Color Mapping" increase the level of the "Median" part and lower the "Nonlinearity" such as in the following picture
> 	![Balance the colors of the map to better visualise the Phosphate repartition](../../images/ocean_var/colors.png)
> 6. Then, click on **OK**, and you can see the Phosphate repartition.
>
> 7. Click right on the map and select **Save Plot As...**
> 8. In the pop-up screen go to the folder **ODV**, **galaxy**, **outputs**.
> 9. In **Files of type** select `PNG (*.png *.PNG)` and **Save** then **OK** and **OK**.
>
>    If you want to have a visualisation of the Phosphate according to the depth, you can do the following.
> 10. Go on the big **+** on the top left
> 11. Select "1 STATION Window" in the new tab, in the plot press enter on your keyboard.
> 12. Same as before to save it, click right on the map and select **Save Plot As...**
> 13. In **Files of type** select `PNG (*.png *.PNG)` and **Save** then **OK** and **OK**.
>
>     You can now safely exit ODV.
> 14. On the top left click on **File** select **Exit**
> 15. If you want to save the other window also click on **Yes**. Here we don't need it so click **No**.
{: .hands-on}

</div>

# Conclusion

Great you now know how to extract ocean variables from an ODV collection and use these extracted data in DIVAnd.
From there you can grid the phosphate variable on the subset extracted create a netcdf file and visualise it on ODV again.
We finished with the last results visible on the galaxy instance in the output **ODV all outputs** :
![Resulting map of the Phoshate variable repartition in the Mediterranean sea](../../images/ocean_var/phosphate_map.png)
![Resulting graph of the phosphate variable according to the depth](../../images/ocean_var/phosphate_graph.png)

# Extra information

Coming up soon even more tutorials on and other Earth-System related trainings. Keep an {% icon galaxy-eye %} open if you are interested!

tools_json:
["dedicated form", "interactive_tool_odv", "Ocean Data View", "DIVAnd", "toolshed.g2.bx.psu.edu/repos/ecology/divand_full_analysis/divand_full_analysis/0.1.0+galaxy0", "interactive_tool_divand"]


topic: climate
tutorial: pangeo
title: Pangeo ecosystem 101 for everyone - Introduction to Xarray Galaxy Tools
url: https://training.galaxyproject.org/training-material/topics/climate/tutorials/pangeo/tutorial.html
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["pangeo"]
content_hash: 29f41a85f302dc5388f2d86f90ead40e
last_modified: 2026-05-27T09:01:47.876943
zenodo_link: https://doi.org/10.5281/zenodo.5805953
workflows_json: []
content:
[Pangeo](https://pangeo.io/) is a project that effectively began in 2016 with a workshop at Columbia University. The mission for Pangeo developed at that workshop is still valid nowadays:

*Our mission is to cultivate an ecosystem in which the next generation of open-source analysis tools for ocean, atmosphere and climate science can be developed, distributed, and sustained. These tools must be scalable in order to meet the current and future challenges of big data, and these solutions should leverage the existing expertise outside of the geoscience community.*

In this tutorial, you will learn how to manipulate [netCDF](https://en.wikipedia.org/wiki/NetCDF) data files using [Xarray](https://xarray.pydata.org/en/stable/) Galaxy Tools.  NetCDF stands for **network Common Data Form** and is one of the most popular file format in climate science. It is used for storing multidimensional scientific data variables such as temperature or humidity, and metadata can be added to facilitate sharing of netCDF data. netCDF is widely used outside the Climate Science community and each community has its own set of conventions, especially for metadata. The [Climate and Forecast metadata convention](https://cfconventions.org/), also called CF-convention is used by the Climate community and is designed to promote the processing and sharing of netCDF files.


> <comment-title>Xarray and Earth Science</comment-title>
> Xarray works with labelled multi-dimensional arrays and can be used for a very wide range of data and data formats. In this training material, we focus on the usage of Xarray for Earth Science data following the [CF-Convention](https://cfconventions.org/). However, some Galaxy Tools also work for non Earth Science datasets, and if needed current Xarray Galaxy Tools could be extended to accommodate new usage.
{: .comment}

In this tutorial, we will be using data from [Copernicus Atmosphere Monitoring Service](https://ads.atmosphere.copernicus.eu/) (CAMS).

CAMS produces daily [European air quality forecasts](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-europe-air-quality-forecasts?tab=overview) over Europe at a resolution of 0.1 degrees (which corresponds approximately to 10km). 
It is produced from an ensemble of nine air quality forecasting models across Europe: the nine models can be combined together and the spread between these models are used to provide an extimate of the forecast uncertainty. The analysis combines model data with observations provided by the [European Environment Agency](https://www.eea.europa.eu/) (EEA).

Several variables are generated and we will be using PM2.5 ([Particle Matter < 2.5 μm](https://en.wikipedia.org/wiki/Particulates#Size,_shape_and_solubility_matter)) 4 days forecast from December, 22 2021. Particle Matter < 2.5 μm (PM2.5) are fine particules that remain suspended for a long time and exposure to high concentration of PM2.5 (yearly mean values greater than 25 µg/m3) can have multiple short term and long term health impacts such as eye irritation, asthma and chronic bronchitis. Information on air quality standards in Europe can be found at the [European Commission's air quality standards webpage](https://ec.europa.eu/environment/air/quality/standards.htm).
The dataset we will be using in this tutorial is very small and there is no need to parallelize our data analysis. Parallel data analysis with Pangeo is not covered in this tutorial. 

 
> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Create a history

> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you start from an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to be meaningful and easy to find. For instance, you can choose **Pangeo 101 for everyone - Xarray** as the name of your new history.
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}


##  Upload CAMS PM2.5 data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/5805953/files/CAMS-PM2_5-20211222.netcdf
>    ```
>
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. If needed rename the datasets to `CAMS-PM2_5-20211222.netcdf`
> 3. Check that the datatype is `netcdf`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Add a tag corresponding to ads (for Atmosphere Data Service)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


# Understanding our dataset


> <details-title>More information about CAMS PM2.5 forecast datasets</details-title>
>
> Our CAMS PM2.5 forecast dataset is in [netCDF](https://en.wikipedia.org/wiki/NetCDF) format. You could find the same dataset in different formats such as [GRIdded Binary or General Regularly-distributed Information in Binary form (GRIB)](https://en.wikipedia.org/wiki/GRIB) or [geoTIFF](https://en.wikipedia.org/wiki/GeoTIFF). The same Xarray Tools can be used with these other data formats. More information about this particular data set can be found on the [CAMS European air quality forecast webpage](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-europe-air-quality-forecasts?tab=overview). As mentioned earlier, we use netCDF data format because it is the most popular among climate scientists.
{: .details}


To understand what is contained in our dataset, we will first use Xarray metadata Galaxy Tool. That will give us all the metadata information about the dataset.


## Get metadata

### Global metadata information

> <hands-on-title>netCDF dataset with Xarray metadata Galaxy Tool</hands-on-title>
>
> 1. {% tool [NetCDF xarray Metadata Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1) %} with the following parameters:
>
>    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222.netcdf`
> 2. View {% icon galaxy-eye%} the two generated outputs:
>    - `Metadata infos` is a `tabular` providing the list of variables, their dimension names and number of elements per dimension. This file is used by other Xarray Tools. 
>    - The second file `info file` provide a summary of the **Xarray Dataset** contained in your netCDF file.
{: .hands_on}

In `info file` output file, we can identify 4 different sections:
1. **Dimensions**: name of dimensions and corresponding number of elements;
2. **Coordinates**: contains coordinate arrays (longitude, latitude, level and time) with their values.
3. **Data variables**: contains all the variables available in the dataset. Here, we only have one variable. For each variable, we get information on its shape and values.
4. **Global Attributes**: at this level, we get the global attributes of the dataset. Each attribute has a name and a value. 


> <question-title>CAM PM2.5 Dataset</question-title>
>
> What is the name of the variable for Particle matter < 2.5 μm and its physical units?
>
> > <solution-title></solution-title>
> > 1. Information about variable names and units can be found in **info file** that was generated by Xarray metadata Galaxy Tool. 
> >      - Variable name: `mass_concentration_of_pm2p5_ambient_aerosol_in_air`
> >      - Units: `µg/m3`
> >
> > 
> > > <code-out-title></code-out-title>
> > > ```bash
> > > xarray.Dataset {
> > > dimensions:
> > > 	latitude = 400 ;
> > > 	level = 1 ;
> > > 	longitude = 700 ;
> > > 	time = 97 ;
> > > 
> > > variables:
> > > 	float32 longitude(longitude) ;
> > > 		longitude:long_name = longitude ;
> > > 		longitude:units = degrees_east ;
> > > 	float32 latitude(latitude) ;
> > > 		latitude:long_name = latitude ;
> > > 		latitude:units = degrees_north ;
> > > 	float32 level(level) ;
> > > 		level:long_name = level ;
> > > 		level:units = m ;
> > > 	timedelta64[ns] time(time) ;
> > > 		time:long_name = FORECAST time from 20211222 ;
> > > 	float32 pm2p5_conc(time, level, latitude, longitude) ;
> > > 		pm2p5_conc:species = PM2.5 Aerosol ;
> > > 		pm2p5_conc:units = µg/m3 ;
> > > 		pm2p5_conc:value = hourly values ;
> > > 		pm2p5_conc:standard_name = mass_concentration_of_pm2p5_ambient_aerosol_in_air ;
> > > 
> > > // global attributes:
> > > 	:title = PM25 Air Pollutant FORECAST at the Surface ;
> > > 	:institution = Data produced by Meteo France ;
> > > 	:source = Data from ENSEMBLE model ;
> > > 	:history = Model ENSEMBLE FORECAST ;
> > > 	:FORECAST = Europe, 20211222+[0H_96H] ;
> > > 	:summary = ENSEMBLE model hourly FORECAST of PM25 concentration at the Surface from 20211222+[0H_96H] on Europe ;
> > > 	:project = MACC-RAQ (http://macc-raq.gmes-atmosphere.eu) ;
> > > }
> > > ```
> > {: .code-out}
> {: .solution }
{: .question }

### Coordinates information

> <hands-on-title>Get Coordinate information with Xarray Coordinate</hands-on-title>
>
> 1. {% tool [NetCDF xarray Coordinate Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.18.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222.netcdf`
>
> 2. View {% icon galaxy-eye%} the 5 generated outputs:
>    - `latitude`: a tabular file containing all the latitude values of our Xarray dataset;
>    - `longitude`: a tabular file containing all the longitudes values;
>    - `level`: this file contains information on all the different levels (here, we have surface data so level=0 meter);
>    - `time`: this tabular file contains all the forecast times. In our case, these are relative to December 22, 2021;
>    - `version`: this is a text file returning the Xarray package version. It is useful when publishing your Galaxy workflow.
>
>    > <comment-title></comment-title>
>    >
>    > This tool returns as many tabular files as the number of coordinate variables present in your input file. The values are decoded from the netCDF input file and no further processing is done. So units for instance for latitudes, longitudes, level and time may vary from one file to another depending on how it was coded in the original input file.
>    {: .comment}
>
{: .hands_on}


> <question-title>Understanding PM2.5 forecast coordinates</question-title>
>
>  1. What is the unit of the `time` coordinate?
>  2. What is the frequency of PM2.5 forecasts? 
>  3. What is the range of values for latitudes and longitudes?
>
> > <solution-title></solution-title>
> > 1. `info file` tells us that `time` is coded as `timedelta64[ns]` e.g. as differences in times (here in nanoseconds). Here the reference time is December 22, 2021. If we look at the tabular file named `time` (generated by `NetCDF xarray Coordinate Info`), we see that these times are automatically converted to human readable time format when printed:
> >
> > > <code-out-title></code-out-title>
> > > ```bash
> > > 0	0 days 00:00:00
> > > 1	0 days 01:00:00
> > > 2	0 days 02:00:00
> > > 3	0 days 03:00:00
> > > 4	0 days 04:00:00
> > > ```
> > {: .code-out}
> > This tells us that we have hourly forecast data. The last forecast time is `4 days 00:00:00` which means that the last forecast is in 4 days at 00:00 UTC (from December 22, 2021).
> {: .solution }
{: .question }

# Plotting our dataset on a geographical map

> <hands-on-title>Map plot</hands-on-title>
>  We will use Xarray mapplot Galaxy Tool to plot PM2.5 on December 22, 2021.
>
> 1. {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222.netcdf`
>    - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222.netcdf`
>    - *"Choose the variable to plot"*: `pm2p5_conc`
>    - *"Name of latitude coordinate"*: `latitude`
>    - *"Name of longitude coordinate"*: `longitude`
>    - *"Datetime selection"*: `Yes`
>        - {% icon param-file %} *"Tabular of time values"*: `time` 
>        - *"Choose the times to plot"*: `0 days 00:00:00`
>    - *"Shift longitudes [0,360] --> [-180,180]"*: `Yes`
>    - *"Range of values for plotting e.g. minimum value and maximum value (minval,maxval) (optional)"*: `0,35`
>    - *"Add country borders with alpha value [0-1] (optional)"*: `0.2`
>    - *"Add coastline with alpha value [0-1] (optional)"*: `0.5`
>    - *"Specify which colormap to use for plotting (optional)"*: `roma_r`
>    - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{'proj': 'Mercator', 'central_longitude': 12.0}`
>
>   ![CAMS PM2.5 December, 22th 2021 at 00:00 UTC](../../images/PM2_5_galaxy_20211222-00UTC.png)
>
>    > <comment-title>Why shifting longitudes?</comment-title>
>    >
>    > Longitudes are coded from 0 to 360 degrees. As we do not have global data but only covering Europe, we need to shift longitudes so that `NetCDF xarray map plotting` can plot properly our dataset. 
>    {: .comment}
>
> 
{: .hands_on}


> <question-title>Visualize and Compare</question-title>
>
> Make a plot to Visualize the forecast for December, 24th 2021 at 12:00 UTC. Do you see any obvious differences with the plot from December 22, 2021 at 00:00 UTC?
>
> > <solution-title></solution-title>
> > 
> > Data starts on December, 22nd 2021 at 00:00 UTC so we need to add 2 days and 12 hours to select the correct time index. We reuse the same `NetCDF xarray map plotting` with a different selection for time:
> >
> > {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0) %} with the following parameters:
> >    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222.netcdf`
> >   - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222.netcdf`
> >   - *"Choose the variable to plot"*: `pm2p5_conc`
> >   - *"Name of latitude coordinate"*: `latitude`
> >   - *"Name of longitude coordinate"*: `longitude`
> >   - *"Datetime selection"*: `Yes`
> >       - {% icon param-file %} *"Tabular of time values"*: `time` 
> >       - *"Choose the times to plot"*: `2 days 12:00:00`
> >   - *"Shift longitudes [0,360] --> [-180,180]"*: `Yes`
> >   - *"Range of values for plotting e.g. minimum value abd maximum value (minval,maxval) (optional)"*: `0,35`
> >   - *"Add country borders with alpha value [0-1] (optional)"*: `0.2`
> >   - *"Add coastline with alpha value [0-1] (optional)"*: `0.5`
> >   - *"Specify which colormap to use for plotting (optional)"*: `roma_r`
> >   - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{'proj': 'Mercator', 'central_longitude': 12.0}`
> >   ![CAMS PM2.5 December, 24th 2021 at 12:00 UTC](../../images/PM2_5_galaxy_20211224-12UTC.png)
> {: .solution }
{: .question }

# Select / Subset from coordinates


> <hands-on-title>NetCDF xarray operations manipulate xarray from netCDF and save back to netCDF</hands-on-title>
>
> 1. {% tool [NetCDF xarray operations](toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222.netcdf`
>    - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222.netcdf`
>    - *"Choose the variable to extract"*: `pm2p5_conc`
>    - In *"additional filter"*:
>        - {% icon param-repeat %} *"Insert additional filter"*
>            - *"Dimensions"*: `time`
>            - *"Comparator"*: `slice(threshold1,threshold2)`
>                - *"Choose the start value for slice"*: `0 days 00:00:00`
>                - *"Choose the end value for slice"*: `1 days 00:00:00`
> 2. Rename the output dataset to `CAMS-PM2_5-20211222_fc0-23h.netcdf`
> 3.  Add a tag corresponding to `0-23h` (do not forget to add `#` in front of the tag)
> 4. {% tool [NetCDF xarray Coordinate Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.18.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222_fc0-23h.netcdf`
> 5. Check the generated outputs and in particular `time`. We see that the tabular file `time` only contains **24** lines with times from `0 days 00:00:00` to `0 days 23:00:00`
>
>    > <comment-title>slice <i>threshold2</i> not included in selection</comment-title>
>    >
>    > You may have noticed already but when selecting a range with `slice` the upper limit (here `1 days 00:00:00`) is not included.
>    {: .comment}
>  
>
{: .hands_on}


> <question-title>PM2.5 over Italy region</question-title>
>
> Using a selection and making plots of PM2.5 over Italy (latitudes: 43.N, 40.N and longitudes: 11.E,15.E), can you tell us if the forecasted PM2.5 will increase or decrease during the next 24 hours between 10:00 UTC and 17:00 UTC? Over which town in Italy do you see high values?
>
> > <solution-title></solution-title>
> > 1. {% tool [NetCDF xarray operations](toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0) %} with the following parameters:
> >   - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222.netcdf`
> >   - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222.netcdf`
> >   - *"Choose the variable to extract"*: `pm2p5_conc`
> >   - In *"additional filter"*:
> >       - {% icon param-repeat %} *"Insert additional filter"*
> >           - *"Dimensions"*: `time`
> >           - *"Comparator"*: `slice(threshold1,threshold2)`
> >               - *"Choose the start value for slice"*: `0 days 10:00:00`
> >               - *"Choose the end value for slice"*: `0 days 18:00:00`
> >       - {% icon param-repeat %} *"Insert additional filter"*
> >           - *"Dimensions"*: `latitude`
> >           - *"Comparator"*: `slice(threshold1,threshold2)`
> >               - *"Choose the start value for slice"*: `43.05`
> >               - *"Choose the end value for slice"*: `40.05`
> >       - {% icon param-repeat %} *"Insert additional filter"*
> >           - *"Dimensions"*: `longitude`
> >           - *"Comparator"*: `slice(threshold1,threshold2)`
> >               - *"Choose the start value for slice"*: `11.05`
> >               - *"Choose the end value for slice"*: `15.05`
> > 2. Rename the output dataset to `CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
> > 3.  Add a tag corresponding to `0-23h-Italy`
> > 4. {% tool [NetCDF xarray Metadata Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1) %} with the following parameters:
> >   - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
> > 5. {% tool [NetCDF xarray Coordinate Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.18.2+galaxy0) %} with the following parameters:
> >   - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
> > 6.  {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0) %} with the following parameters:
> >   - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
> >   - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
> >   - *"Choose the variable to plot"*: `pm2p5_conc`
> >   - *"Name of latitude coordinate"*: `latitude`
> >   - *"Name of longitude coordinate"*: `longitude`
> >   - *"Datetime selection"*: `Yes`
> >   - {% icon param-file %} *"Tabular of time values"*: `time`
> >   - *"Choose the times to plot"*: **Tick Select all**
> >   - *"Shift longitudes [0,360] --> [-180,180]"*: `No`
> >   - *"Range of values for plotting e.g. minimum value abd maximum value (minval,maxval) (optional)"*: `0,35`
> >   - *"Add country borders with alpha value [0-1] (optional)"*: `0.2`
> >   - *"Add coastline with alpha value [0-1] (optional)"*: `0.5`
> >   - *"Specify which colormap to use for plotting (optional)"*: `roma_r` 
> >   - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{'proj': 'Mercator', 'central_longitude': 12.0}`
> > 8. {% tool [Image Montage](toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1) %} with the following parameters:
> >   - {% icon param-files %} *"Images"*: `Map plots`
> >   - {% icon param-text %} *"# of images wide"*: `4`
> > 
> > ![CAMS PM2.5 Italy 10:00 - 17:00 December 22, 2021](../../images/PM2_5_galaxy_20211222_Italy-10-17UTC.png)
> >
> > From the plot there is no obvious trend over this entire region of Italy. However, we clearly see that PM2.5 is always higher over Naples and tends to spread in the South-East direction by the end of the day (on that particular date).
> {: .solution }
{: .question }

> <comment-title> `latitude=slice(43.05, 40.05)` and not `latitude=slice(40.05, 43.05)`</comment-title>
> Why did we slice latitudes with `latitude=slice(43.05, 40.05)` and not `latitude=slice(40.05, 43.05)`?
> - because when using slice, you need to specify values using the same order as in the coordinates. Latitudes are specified in 
> decreasing order for CAMS.
>
{: .comment}

# Masking with Where statement

- Sometimes we may want to make more complex selections with criteria on the values of a given variable and not only on its coordinates. For this we use `where`.
- For instance, we may want to only keep PM2.5 if values are greater than a chosen threshold.

> <hands-on-title>Plot where PM2.5 is greater than 30 μm.m<sup>-3</sup></hands-on-title>
> 1.  {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0) %} with the following parameters:
>   - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
>   - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
>   - *"Choose the variable to plot"*: `pm2p5_conc`
>   - *"Name of latitude coordinate"*: `latitude`
>   - *"Name of longitude coordinate"*: `longitude`
>   - *"Datetime selection"*: `Yes`
>   - {% icon param-file %} *"Tabular of time values"*: `time` 
>   - *"Choose the times to plot"*: `0 days 10:00:00` 
>   - *"Shift longitudes [0,360] --> [-180,180]"*: `No`
>   - *"Range of values for plotting e.g. minimum value and maximum value (minval,maxval) (optional)"*: `0,35`
>   - *"Do not plot values below this threshold (optional)"*: `30`
>   - *"Add country borders with alpha value [0-1] (optional)"*: `0.2`
>   - *"Add coastline with alpha value [0-1] (optional)"*: `0.5`
>   - *"Specify which colormap to use for plotting (optional)"*: `roma_r`
>   - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{'proj': 'Mercator', 'central_longitude': 12.0}`
>    ![CAMS PM2.5 Italy 10:00  December 22, 2021](../../images/PM2_5_galaxy_20211222_Italy-10UTC-30.png)
> Now we clearly see that values of PM2.5 >  30 μm.m<sup>-3</sup> are only found over Naples on December 22, 10:00 UTC.
>
{: .hands_on}



> <question-title>PM2.5 over Italy over 30 μm.m<sup>-3</sup></question-title>
>
> Using the same geographical region over Italy, can you tell us if the forecasted PM2.5 will exceed
> 30 μm.m<sup>-3</sup> 
> between 10:00 UTC and 17:00 UTC on december 22, 2021? 
>
> > <solution-title></solution-title>
> > 1.  {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0) %} with the following parameters:
> >   - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
> >   - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222_fc10-17h_Italy.netcdf`
> >   - *"Choose the variable to plot"*: `pm2p5_conc`
> >   - *"Name of latitude coordinate"*: `latitude`
> >   - *"Name of longitude coordinate"*: `longitude`
> >   - *"Datetime selection"*: `Yes`
> >         - {% icon param-file %} *"Tabular of time values"*: `time` 
> >         - *"Choose the times to plot"*: **Tick Select all** 
> >   - *"Shift longitudes [0,360] --> [-180,180]"*: `No`
> >   - *"Range of values for plotting e.g. minimum value and maximum value (minval,maxval) (optional)"*: `0,35`
> >   - *"Do not plot values below this threshold (optional)"*: `30`
> >   - *"Add country borders with alpha value [0-1] (optional)"*: `0.2`
> >   - *"Add coastline with alpha value [0-1] (optional)"*: `0.5`
> >   - *"Specify which colormap to use for plotting (optional)"*: `roma_r`
> >   - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{'proj': 'Mercator', 'central_longitude': 12.0}`
> >   
> > 7. {% tool [Image Montage](toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1) %} with the following parameters:
> >   - {% icon param-files %} *"Images"*: Browse the dataset and manually select all images (png files) 
> >   - {% icon param-text %} *"# of images wide"*: `4`
> > 
> > ![CAMS PM2.5 Italy 10:00 - 17:00 December 22, 2021 with PM2.5 > 30 μm.m<sup>-3</sup>](../../images/PM2_5_galaxy_20211222_Italy-10-17UTC_30.png)
> > Using thresholds, we can clearly identify areas if anywhere there are "high" values of PM2.5. On that particular day there are a few pixels where PM2.5 values exceed 30 μm.m<sup>-3</sup>.
> {: .solution }
{: .question }

# From Xarray to Tabular Data

> <hands-on-title>Xarray selection</hands-on-title>
> 
> We will select a single location: Naples (40.8518° N, 14.2681° E) and select the grid point that is closest to Naples.
> 1. {% tool [NetCDF xarray operations](toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222.netcdf`
>    - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from CAMS-PM2_5-20211222.netcdf`
>    - *"Choose the variable to extract"*: `pm2p5_conc`
>    - In *"additional filter"*:
>        - {% icon param-repeat %} *"Insert additional filter"*
>            - *"Dimensions"*: `latitude`
>            - *"Comparator"*: `slice(threshold1,threshold2)`
>                - *"Choose the start value for slice"*: `40.95`
>                - *"Choose the end value for slice"*: `40.85`
>        - {% icon param-repeat %} *"Insert additional filter"*
>            - *"Dimensions"*: `longitude`
>            - *"Comparator"*: `slice(threshold1,threshold2)`
>                - *"Choose the start value for slice"*: `14.25`
>                - *"Choose the end value for slice"*: `14.35`
> 2. Rename the output dataset to `CAMS-PM2_5-20211222_Naples.netcdf`
> 3.  Add a tag corresponding to `Naples`
> 4. {% tool [NetCDF xarray Selection](toolshed.g2.bx.psu.edu/repos/ecology/xarray_select/xarray_select/0.15.1) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `CAMS-PM2_5-20211222_Naples.netcdf`
>    - *"Choose the variable to extract"*: `pm2p5_conc`
>    - *"Source of coordinates"*: `Manually enter coordinates`
>        - *"Geographical area"*: `Whole available region`
>    - In *"Select Time series"*:
>        - *"Datetime selection"*: `No`
>
> 5. Rename the output dataset to `CAMS-PM2_5-20211222_Naples.tabular`
> 6. View {% icon galaxy-eye%} the generated file. It is a tabular with timeseries of PM2.4 concentrations over Naples. The total number of lines is 97 but we only print the first 5 lines.
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > 	time	level	latitude	longitude	pm2p5_conc
>    > 	0	0 days 00:00:00.000000000	0.0	40.95000076293945	14.25	24.00212
>    > 	1	0 days 01:00:00.000000000	0.0	40.95000076293945	14.25	23.5767
>    > 	2	0 days 02:00:00.000000000	0.0	40.95000076293945	14.25	21.383186
>    > 	3	0 days 03:00:00.000000000	0.0	40.95000076293945	14.25	20.04839
>    > 	4	0 days 04:00:00.000000000	0.0	40.95000076293945	14.25	18.347801
>    > ```
>    {: .code-out}
>
{: .hands_on}



> <question-title>PM2.5 at Naples over the 4 forecasted days</question-title>
>
> From a qualitative point of view, can you say if PM2.5 may increase or decrease over the 4 forecasted days?
>
> > <solution-title></solution-title>
> >  We can make a simple plot using Scatterplot with ggplot2 ot climate stripes:
> > 1. {% tool [Scatterplot with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/2.2.1+galaxy2) %} with the following parameters:
> >   - {% icon param-file %} *Input in tabular format"*: `CAMS-PM2_5-20211222_Naples.tabular`
> >   - *"Column to plot on x-axis"*: `1`
> >   - *"Column to plot on y-axis"*: `6`
> >   - *"Label for x axis"*: `Forecast time (hour) from December, 22 2021`
> >   - *"Label for y axis"*: `Particule Matter < 2.5 μm.m-3`
> >   - In *"Advanced Options"*:
> >       - *"Type of plot"*: `Lines only `
> >       - *"Data point options"*: `User defined point options`
> >           - *"Transparency of points (On a scale of 0-1; 0=transparent, 1=default)"*: `0.7`
> >       - *"Plotting multiple groups"*: `No thanks - just plot the data as one group`
> >       - *"Axis title options"*: `Default`
> >       - *"Axis text options"*: `Default`
> >       - *"Plot title options"*: `Default`
> >       - *"Axis scaling"*: `Automatic axis scaling`
> >     ![CAMS PM2.5 Naples](../../images/PM2_5_galaxy_20211222_Naples_ggplot2.png)
> > 2. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1) %} with the following parameters:
> >   - {% icon param-file %} *"Select cells from"*: `CAMS-PM2_5-20211222_Naples.tabular`
> >   - *"using column"*: `c2`
> >   - In *"Check"*:
> >       - {% icon param-repeat %} *"Insert Check"*
> >           - *"Find Regex"*: `0 days`
> >           - *"Replacement"*: `20211222`
> >       - {% icon param-repeat %} *"Insert Check"*
> >           - *"Find Regex"*: `1 days`
> >           - *"Replacement"*: `20211223`
> >       - {% icon param-repeat %} *"Insert Check"*
> >           - *"Find Regex"*: `2 days`
> >           - *"Replacement"*: `20211224`
> >       - {% icon param-repeat %} *"Insert Check"*
> >           - *"Find Regex"*: `3 days`
> >           - *"Replacement"*: `20211225`
> >       - {% icon param-repeat %} *"Insert Check"*
> >           - *"Find Regex"*: `4 days`
> >           - *"Replacement"*: `20211226`
> > 3. Rename your dataset to `CAMS-PM2_5-20211222_Naples_with_dates.tabular`
> > 4. {% tool [climate stripes](toolshed.g2.bx.psu.edu/repos/climate/climate_stripes/climate_stripes/1.0.1) %} with the following parameters:
> >   - *"column name to use for plotting"*: `pm2p5_conc`
> >   - *"plot title"*: `PM2.5 4 days forecast from December 22 2021 over Naples`
> >   - In *"Advanced Options"*:
> >       - *"column name to use for x-axis"*: `time`
> >       - *"format for input date/time column"*: `%Y%m%d %H:%M:%S.%f`
> >       - *"format for plotting dates on the x-axis"*: `%d %b %H hours`
> >       - *""*: `winter`
> >  ![CAMS PM2.5 Naples Stripes](../../images/PM2_5_galaxy_20211222_Naples_stripes.png)
> > From December 24, 2021 at 00:00 UTC and onwards, PM2-5 concentration are much lower than at the beginning. This is bothe visible on the 1D plot and stripes.
> {: .solution }
{: .question }


# Conclusion


{% icon trophy %} Well done! In this tutorial, Xarray Galaxy Tools have been introduced and we learned to use these tools on a real dataset from Copernicus Atmosphere Monitoring Service. We encourage you to try with your own datasets.

tools_json:
["NetCDF xarray Metadata Info", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1", "NetCDF xarray Coordinate Info", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.18.2+galaxy0", "NetCDF xarray map plotting", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0", "NetCDF xarray operations", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0", "Image Montage", "toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1", "NetCDF xarray Selection", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_select/xarray_select/0.15.1", "Scatterplot with ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/2.2.1+galaxy2", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1", "climate stripes", "toolshed.g2.bx.psu.edu/repos/climate/climate_stripes/climate_stripes/1.0.1"]


topic: climate
tutorial: ocean_qcv_analysis
title: Nitrate DMQC for autonomous platforms such as Argo floats
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["earth-system", "ocean", "biogeochemical"]
content_hash: 7dc3dff899dcc66d0efb9d9ab5d4f729
last_modified: 2026-05-27T09:01:47.876394
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/climate/tutorials/ocean_qcv_analysis/tutorial.html

content:
The next sections explain how to qualify and calibrate the nitrate sensor mounted on the Argo float **4903881** either tool by tool or by running a pre-defined Galaxy workflow.
This requires two preliminary steps to prepare the environment.

For your information, execution time for running tools depends on the number of files, their size and/or the number of people working on Galaxy.

> <warning-title>Do not change file name</warning-title>
> DO NOT change file names to ensure that any changes you make with ODV will be carried over. However, it is possible to change the name of Galaxy collections to make them easier to find.
{: .warning}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prerequisites

## Galaxy
> <details-title>Galaxy account</details-title>
> Create or login to your [Galaxy Europe account](https://earth-system.usegalaxy.eu/login/start).
{: .details}

> <details-title>Galaxy tips and tricks</details-title>
> Once you are logged in, Galaxy is divided into 4 vertical panels:
> - Far left: **Galaxy action list** (Upload, Tools, Interactive Tools…)
> - Center: **Main panel**
> - Far right: **History** section
>
> ![Galaxy organisation](../../images/bgc_calib/interface_1.png)
>
> All symbols have a tooltip when you hover them. These names will be indicated with *{}* in this tutorial.
>
> The history section stores all your job results with the following color code:
> - Grey = cancelled
> - Orange = running
> - Green = finished with success
> - Red = failed
>
> It is possible to organise your history with sub-directories:
> - **New history**: click on **+** at the top-right of history, edit the name with the pencil, save
> - **Switch history**: click on the **⇄** icon and select a history
> - **Copy datasets**: click on **Operations** → *Copy datasets* and choose source/destination histories
>
> ![Galaxy operation symbol](../../images/bgc_calib/interface_2.png)
{: .details}

> <details-title>Galaxy execution constraints</details-title>
> - **Quota per user account**: check [your quota](https://galaxyproject.org/support/account-quotas/).
> Free space by clicking the database icon in the history panel and “Review and clear” GB.
> (Note: deleting a dataset in the history does not free quota until you purge it.)
>
> ![Galaxy quota info](../../images/bgc_calib/galaxy_storage_handling.png)
>
> - **Time limit for interactive tools**: 24h. After that, ODV closes automatically.
> - **Avoid weekends**: upgrades are performed and can stop tools/workflows.
> - **Max interactive tools simultaneously**: 2.
{: .details}

## Nitrate calibration

> <comment-title>Oxygen prerequisite</comment-title>
> Because oxygen concentration is implied in the estimation of the *nitrate adjustment from neural network method*,
> platforms to be calibrated must have **good-quality oxygen** (adjusted in real time or delayed mode for Argo floats or gliders).
{: .comment}

> <details-title>World Ocean Atlas reference</details-title>
> Nitrate calibration needs a potential reference dataset such as the **WOA annual climatology**.
> You can [download it here](https://www.ncei.noaa.gov/thredds-ocean/catalog/woa23/DATA/nitrate/netcdf/all/1.00/catalog.html?dataset=woa23/DATA/nitrate/netcdf/all/1.00/woa23_all_n00_01.nc) and upload it into Galaxy.
{: .details}

# Manage your data

## Get your data

> <hands-on-title>Get your data from the S3 server</hands-on-title>
> - Click on **Upload** in the vertical panel on the left
> - In the pop-up window, click on **Choose from repository**
> - In the top search bar, search for `argo`
> - In the result list, select **Argo marine floats data and metadata from Global Data Assembly Centre (Argo GDAC)**
> - Navigate to `pub/dac/coriolis/4903881`
> - Tick the following:
>   - `4903881_meta.nc`
>   - All files under `profiles/`
> - Click on **Select** (bottom-right)
> - Remove the synthetic files `SR4903881_*.nc` (by clicking the trash icon)
> - Click on **Start**
> - Wait until all files are stored in your history on the right (green color)
>
> ![S3 upload interface](../../images/bgc_calib/s3_upload.png)
{: .hands_on}

> <tip-title>Do it your way: uploading from your computer</tip-title>
> > <hands-on-title>Get your data from your computer</hands-on-title>
> > - Click on **Upload** in the vertical panel on the left
> > - In the pop-up, click on **Choose local file**
> > - Select the files on your system (Argo float or WOA)
> > - Click on **Open** then **Start**
> > - Wait until all files are stored in your history (green color)
> >
> > ![Local upload interface](../../images/bgc_calib/local_upload.png)
> {: .hands_on}
{: .tip}

## Organize your data

{% snippet faqs/galaxy/collections_autobuild_list.md box_type="hands-on" datasets_description="n files of interest" name="4903881" %}

> <hands-on-title>Change file extension (for WOA file)</hands-on-title>
>
> 1. Change the datatype of the WOA file to `NetCDF`
>
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="NetCDF" %}
>
>
> ![Change extension](../../images/bgc_calib/change_ext.png)
{: .hands_on}



{% include _includes/cyoa-choices.html option1="Tools" option2="Workflow" default="Tools" disambiguation="tutorialstyle"
       text="Do you want to run a workflow 'human in the loop' or to discover the tools one by one?" %}


<div class="Workflow" markdown="1">

# Run a predefined workflow

> <hands-on-title>Run a public workflow</hands-on-title>
>
> {% snippet faqs/galaxy/workflows_run.md box_type="none" workflows_run="Public Workflows" name="Argo-Glider Nitrate QCV" %}
>
>
> - When you click **Run**, a panel asks you for the input files:
>   - Select the Argo files and climatology uploaded during the [data management phase](#manage-your-data)
>   - Once filled in, click **Run** again
>
> ![Select workflow](../../images/bgc_calib/bgc_workflow.png)
>
> > <tip-title>Manually uploading the workflow</tip-title>
> > Is the workflow not available on your Galaxy instance? You can upload [the workflow](./workflows/Galaxy-Workflow-Argo-Glider_Nitrate_QCV.ga) yourself as follows:
> >
> > {% snippet faqs/galaxy/workflows_import.md  %}
> >
> {: .tip}
>
{: .hands_on}

## Workflow human in the loop

Now that your workflow is running, remember it contains both non-interactive and interactive steps. For the interactive tools (ODV), user action is required.

- When the workflow arrives at the **ODV interactive step for qualification**:
  - Open ODV
  - Apply QC following
  - Export history

- Once the qualification step is finished and the history exported, close ODV. The workflow will then continue automatically.

- When the workflow arrives at the **ODV interactive step for validation**:
  - Open ODV again
  - Follow the validation phase as described in

> <tip-title>Copy pasting between computer and ODV</tip-title>
> Expand the ODV left panel (3 vertical dots) to access the clipboard. Paste text into ODV forms via this clipboard. Click outside the panel to collapse it.
>
> ![ODV clipboard](../../images/coastal_water_dyn/clipboard.png)
{: .tip}

> <tip-title>ODV - Disconnected</tip-title>
> If ODV becomes grey with the red banner *ODV - Disconnected*, refresh the tab.
{: .tip}

> <warning-title>ODV execution limit</warning-title>
> ODV has a 24h time limit. If you launch a workflow and wait longer than 1 day before opening ODV, the workflow will continue automatically without your qualification step.
{: .warning}

## Available public workflows

> <details-title>Argo-Glider Nitrate QCV</details-title>
> Workflow to perform **Qualification**, **Nitrate Calibration**, and **Validation** of one Argo float or one Glider.
> Uses Neural Network + Climatology.
> Copy it before running and adjust calibration parametrizations if necessary.
{: .details}

</div>

<div class="Tools" markdown="1">

# Tool by tool

> <warning-title>Before starting</warning-title>
> Before running the QCV procedure tool by tool, be sure that your environment (see Prerequisites and Manage your data) is ready!
{: .warning}

## Harmonize your data

The original files must be pre-processed in order to use the same tools regardless of their origin. This is performed by the tool `QCV harmonizer`.

> <details-title>QCV harmonizer details</details-title>
>
> This tool:
> - harmonizes vocabulary and format
> - aggregates multiple files from the same platform into one single file
>
> Its outputs:
> - a single NetCDF file named `###_harm.nc` (since version 3.0)
>
> This tool is currently operational for Argo floats, World Ocean Atlas, and Gliders.
> It should be run as many times as there are datasets/platforms.
{: .details}

> <hands-on-title>Run QCV harmonizer</hands-on-title>
> 1. Finding your tool
>    - Click on **Tools** just below **Upload** on the left panel
>    - Search for **QCV harmonizer** in the search bar
>    - Select the tool. Its configuration page opens.
>
> 2. {% tool [QCV harmonizer](toolshed.g2.bx.psu.edu/repos/ecology/harmonize_insitu_to_netcdf/harmonize_insitu_to_netcdf/3.0+galaxy1) %} with the following parameters:
>    - In *Input the NetCDF data files*:
>      - If using a dataset collection (e.g. 4903881):
>        Select **Dataset collection** → click on *Select Value* → choose *4903881*
>      - If using WOA:
>        Select **Single dataset** → click on *Select Value* → choose *woa###.nc*
>    - Click **Run**
>
> ![QCV harmonizer tool](../../images/bgc_calib/qcv_harmonizer.png)
{: .hands_on}

> <details-title>Tip: drag and drop</details-title>
> If your file is not listed in the selection window, you can **drag and drop** it from the history panel into the *Select value* box.
{: .details}

## Qualify your data

### Create the ODV collection

The visualization tools for the qualification, the validation and the extraction or reporting of the user actions include **ODV software**, the **ODV collection manager** and the **ODV history manager**.

> <details-title>ODV collection manager details</details-title>
>
> This tool creates:
> - ODV spreadsheet collections from harmonized file(s)
> - ODV view for nitrate qualification
>
> Capabilities:
> - maps input QC flags with expected output QC flag scale
> - subsets reference datasets (option: 1=subset, 0=no, -1=inverse)
>
> Default parameters:
> - operator name = anonymous
> - QC convention = ARGO QC flag scale
> - Subsetting = 1 (yes, even if no reference data)
> - plt = 0 (no plot)
>
> Outputs:
> - `odv_collection.txt`
> - `qualification_startingPoint_nitrate.xview`
> - log file (`YYYY-MM-DDTHHMM_galaxy_odv-coll-manager_QV.log`)
{: .details}

> <hands-on-title>Run ODV collection manager</hands-on-title>
> 1. Finding your tool
>    - Click on **Tools** in the left panel
>    - Search for **ODV collection manager**
>    - Select the tool. Its configuration page opens
>
> 2. {% tool [ODV collection manager](toolshed.g2.bx.psu.edu/repos/ecology/tool_odv/tool_odv/1.3+galaxy2) %} with the following parameters:
>    - For *Input raw data*:
>      - Select **Multiple Datasets** → click *Select Value* → choose *4903881_harm.nc*
>    - (Optional) Add a harmonized WOA reference dataset
>    - (Optional) Change default parameters: operator name, QC convention, subsetting, plt
>    - Click **Run**
>
> Outputs:
> - ODV spreadsheet collection `odv_collection.txt`
> - View `qualification_startingPoint_nitrate.xview`
> - Log file with actions
>
> ![ODV collection manager](../../images/bgc_calib/odv_coll.png)
{: .hands_on}

### Launch ODV automatically

> <details-title>ODV</details-title>
>
> ODV is a software for visualizing and/or qualifying scientific data.
> It can be used locally or directly on the Galaxy {% tool [Ocean Data View](interactive_tool_odv) %}.
{: .details}

> <hands-on-title>Run ODV interactive tool</hands-on-title>
> 1. Finding your interactive tool
>    - Click on **Tools** or **Interactive Tools** in the left panel
>    - Search **ODV** interactive tool
>    - Select the tool. Its configuration page opens
>
> 2. {% tool [Ocean Data View](interactive_tool_odv) %} with the following parameters:
>    - Ensure the tool version is **v5.8_1** (change via the 3 cubes on the left if needed)
>    - Set parameters:
>      - *Load data automatically*: **Yes**
>      - *NetCDF or tabular text file*: select `ODV collection manager output` or drag and drop `odv_collection.txt`
>      - *Do you have a view?*: **Yes** → drag and drop `qualification_startingPoint_nitrate.xview`
>    - Click **Run**
>
> ![ODV launch](../../images/bgc_calib/odv_launch.png)
{: .hands_on}

> <details-title>How to open ODV</details-title>
> - When ODV is ready, a red dot appears on **Interactive Tools**
> - Click it → open the ODV panel → click the expand symbol
> - ODV opens in a new window with your selected view
>
> ![ODV interactive tool](../../images/bgc_calib/odv_launch_2.png)
> ![ODV nitrate view](../../images/bgc_calib/odv_view.png)
{: .details}

> <tip-title>ODV inactive</tip-title>
> After some inactivity, the ODV window may close. Just refresh the browser tab.
{: .tip}

> <tip-title>Copy/paste in ODV</tip-title>
> Open the clipboard (3 dots on the left panel) to copy-paste text.
> ![Clipboard in ODV](../../images/coastal_water_dyn/clipboard.png)
{: .tip}

### Qualify the dataset with ODV

Use ODV features to apply QC flags and edits as needed. Refer to the [ODV guide](https://odv.awi.de/fileadmin/user_upload/odv/docs/ODV_guide.pdf).

### Export history

> <hands-on-title>Export history</hands-on-title>
> 1. Export to your Galaxy history
>    - In ODV: **Export** > **History**
>    - Save as default name `history_from_odv_collection.txt` in `working/Documents/ODV/galaxy/outputs`
>    - Do not change the default filename
>    - Close ODV (**File** > **Exit**)
>
> Outputs:
> - Galaxy collection *ODV all outputs* (zip, history, odv_collection)
> - File *ODV history extracted*
{: .hands_on}

### (optional) Report QC & Data changes

Run **ODV history manager** to report QC flags and/or data changes into the harmonized dataset before calibrating the nitrate sensor.

> <details-title>ODV history manager details</details-title>
>
> This tool extracts QC changes (EDITFLAGS) and data edits (EDITDATA) from the ODV history and reports them into the harmonized file(s).
>
> It manages 2 types of ODV history files:
> - *synthetic* (manual export)
> - *extended* (automatic export, not yet available in Galaxy)
>
> Default parameters:
> - QC convention = ARGO QC flag scale
> - ODV convention changes: EDITFLAGS, EDITDATA
>
> Outputs:
> - `###_harm_qced.nc` (extended NetCDF files including changes)
> - `###_harm_history.csv` (summary of user changes)
> - log file (`YYYY-MM-DDTHHMM_galaxy_odv-history-manager_history.log`)
{: .details}

> <hands-on-title>Run ODV history manager</hands-on-title>
> 1. Find your tool
>    - Click on **Tools** just below **Upload** on the left panel
>    - Search **ODV history manager** in the search bar
>    - Select the tool
>    - Its configuration page opens in the center panel
>
> 2. {% tool [ODV history manager](toolshed.g2.bx.psu.edu/repos/ecology/tool_odv_history/tool_odv_history/1.2+galaxy2) %} with the following parameters:
>
>    Depending on your ODV spreadsheet file, select the appropriate inputs:
>
>    **Case 1: using ODV collection manager output**
>    - *Input NetCDF data*: harmonized file(s) `###_harm.nc` created by QCV harmonizer
>    - *Input history text file*: `ODV history extracted` (txt file exported from ODV after changes in **Export history**)
>    - *Input ODV file*: `ODV collection manager output` (the ODV spreadsheet collection created by ODV collection manager)
>
>    **Case 2: using odv_collection.txt**
>    - *Input NetCDF data*: harmonized file(s) `###_harm.nc` created by QCV harmonizer
>    - *Input history text file*: `history_from_odv_collection.txt` (txt file inside *ODV all outputs*)
>    - *Input ODV file*: `odv_collection.txt` (spreadsheet collection created by ODV collection manager and opened in ODV for QC changes)
>
>    (Optional) In **Tool Parameters**, you may change the default configuration:
>    - *QC convention for the ODV output file*: select one of the [QC flag scales understood by ODV](https://odv.awi.de/fileadmin/user_upload/odv/docs/ODV_quality_flag_sets.pdf)
>    - *ODV convention regarding changes performed on the ODV collection*:
>      - **EDITFLAGS** = report QC flag changes from history to the data collection
>      - **EDITDATA** = report data edits from history to the data collection
>
> ![ODV history manager parametrization](../../images/bgc_calib/odv_history.png)
>
{: .hands_on}

> <tip-title>ODV edits parametrization</tip-title>
> You can unselect either *EDITFLAGS* or *EDITDATA* if you want to report only QC changes or only data changes.
> However, you cannot unselect both. If you do, the backend will actually apply **both** edit types.
{: .tip}

> <warning-title>Check the logs</warning-title>
> At the end of the process, Galaxy distributes the tool outputs into 3 collections:
> - **ODV history manager netcdf collection** (extended harmonized NetCDFs, `###_harm_qced.nc`)
> - **ODV history manager csv collection** (CSV summaries, `###_harm_history.csv`)
> - **ODV history manager log files** (with `YYYY-MM-DDTHHMM_galaxy_odv-history-manager_history.log`)
>
> Always check the log file (click the {% icon galaxy-eye %} icon) to ensure that changes were correctly reported.
> If issues persist, make sure filenames match the step **Export history** conventions.
{: .warning}

All these outputs will appear in your **History** (green status). They are now ready for the next step: **Biogeochemical Calibration**.

## Calibrate your data

### Nitrate

> <hands-on-title>(optional) Choose Reference dataset</hands-on-title>
> - Upload WOA nitrate climatology (NetCDF) from your computer
> - Harmonize it with QCV harmonizer as in the step **Export history**
{: .hands_on}

Currently, the tool **Biogeochemical calibration** is operational for **nitrate**.

> <hands-on-title>Run Biogeochemical calibration</hands-on-title>
> 1. Find your tool
>   - Click on **Tools** → search **Biogeochemical calibration**
>
> 2. {% tool [BioGeoChemical calibration](toolshed.g2.bx.psu.edu/repos/ecology/tool_biogeochemical_calibration/tool_biogeochemical_calibration/2.1+galaxy2) %} with the following parameters:
>    - *Input harmonized netcdf data*: `###_harm_qced.nc` (created by QCV harmonizer + ODV history manager)
>    - *Input reference data* (optional): WOA harmonized NetCDF file if you want to estimate reference nitrate from climatology
>    - (Optional) Change default QC flag configuration (ARGO QC flag scale by default)
>    - (Optional) Add calibration parametrizations:
>      - Single regression
>      - CanyonB (default)
>      - Climatology / WOA
>      - Custom P value with user-defined breakpoints
>    - Run the tool
>
> ![Calibration tool](../../images/bgc_calib/bgc_calibration.png)
{: .hands_on}

> <details-title>Default calibration details</details-title>
> By default, the tool calibrates nitrate using:
> - Automatic evaluation of reference pressure
> - CanyonB as reference method
> - Linear regression for correction
> - ARGO QC flag convention
>
> Default QC flag mapping:
> - Bad Data QC = 4
> - Raw QC value to replace = 3
> - Replacement QC value = 1 (adjusted value)
>
> Possible configurations:
> - User-defined reference pressure
> - Canyon-med or WOA as reference method
> - Multi-linear regression with automatic or user-defined breakpoints
>
> Outputs per platform/configuration:
> - NetCDF file(s): `###_C{param}-nitrate.nc`
> - Diagnostic figures (up to 4):
>   - `###_..._LinearModel.png`
>   - `###_..._Rsquared.png`
>   - `###_..._Zvar_ref_detailed.png`
>   - `###_..._Zvar_ref_results.png`
> - Log file: `YYYY-MM-DDTHHMM_galaxy_odv-calibration-methods_nitrate_c.log`
{: .details}

> <details-title>Examples of parametrization</details-title>
> - Reproduce default method: set **Multiple linear regression** → *Single linear regression*
> - Add canyon B automatic: leave default
> - Add custom method:
>   - *Bypass*: user defines P → enter 1000 dbar
>   - *Multiple linear regression*: N = number of breakpoints
>   - *Reference method*: Climatology / WOA / Annual
> (Ensure WOA harmonized climatology is included in inputs)
{: .details}

> <details-title>DMfiller option</details-title>
> If activated, the tool also updates original Argo NetCDF files with DM information and produces BD Argo files.
> - Requires:
>   - Original raw NetCDF data (dataset collection 4903881)
>   - CSV file from ODV history manager
>   - Institution, BGC contact point, DM operator info
> - If info is missing or inconsistent with original files, BD files are not delivered.
>
> Outputs:
> - `###_C{param}-nitrate.tar.gz` (all BD files ready for DAC)
{: .details}

## Validate your data

Compare adjusted values with ODV:

> <hands-on-title>Validation with ODV</hands-on-title>
> 1. Re-run **ODV collection manager** including all calibrated NetCDFs from the step **(optional) Report QC & Data changes**
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
>
> 2. Launch **ODV interactive tool** again for visualization
> 3. Inspect calibrated vs. raw/reference values
{: .hands_on}

You may validate in two ways:

**Case 1: without external datasets**
- Create ODV collection with calibrated NetCDFs
- Launch ODV without a view
- ODV opens with new collection ready for validation

**Case 2: with external datasets**
- Add an external dataset (ODV spreadsheet, ODV collection, or NetCDF supported by ODV)
- Create ODV collection with calibrated + external data
- Launch ODV without automatic load and without view
- Import datasets manually via *File > New*
- Map metadata and variables when prompted
- Repeat for each external dataset
- ODV is now ready for side-by-side comparison

> <tip-title>Advice</tip-title>
> Import first the ODV spreadsheet collection created by **ODV collection manager**.
> Then import external datasets such as [webODV Glodap](https://explore.webodv.awi.de/ocean/hydrography/glodap/).
{: .tip}

</div>

# Extra information

Coming up soon even more tutorials on and other Earth-System related trainings. Keep an {% icon galaxy-eye %} open if you are interested!

tools_json:
["QCV harmonizer", "toolshed.g2.bx.psu.edu/repos/ecology/harmonize_insitu_to_netcdf/harmonize_insitu_to_netcdf/3.0+galaxy1", "ODV collection manager", "toolshed.g2.bx.psu.edu/repos/ecology/tool_odv/tool_odv/1.3+galaxy2", "Ocean Data View", "interactive_tool_odv", "ODV history manager", "toolshed.g2.bx.psu.edu/repos/ecology/tool_odv_history/tool_odv_history/1.2+galaxy2", "BioGeoChemical calibration", "toolshed.g2.bx.psu.edu/repos/ecology/tool_biogeochemical_calibration/tool_biogeochemical_calibration/2.1+galaxy2"]


topic: ecology
tutorial: PAMPA-toolsuite-tutorial
title: Compute and analyze biodiversity metrics with PAMPA toolsuite
difficulty: intermediate
time_estimation: 2H
requirements_json: []
content_hash: 59ad382ca4982f1a8c3dd77acbfb22d2
last_modified: 2026-05-27T09:01:48.393882
zenodo_link: https://doi.org/10.5281/zenodo.4264936
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/PAMPA-toolsuite-tutorial/tutorial.html

content:
This tutorial aims to present the PAMPA Galaxy workflow, how to use it to compute common
biodiversity metrics from species abundance data and analyse it through generalized
linear (mixed) models (GLM and GLMM). This workflow made up of 5 tools will allow you to process
temporal series data that include at least year, location and species sampled along with
abundance value and, finally, generate article-ready data products.
![PAMPA logo](../../images/PAMPA-toolsuite-tutorial/PAMPA.jpg){: style="float: right"}

The PAMPA workflow is an ecological analysis workflow, like so, it is divided as to do a Community
analysis and a Population analysis separately in accordance with the biodiversity level described.
Only the last tool to create a plot from your GLM results is common to the two parts of this workflow.
Thus, in this tutorial, we'll start by the Community analysis followed by the Population analysis.

> <details-title>Ecological definitions of Population and Community</details-title>
>
> Population: group of individuals of the same species interacting with each other.
>
> Community: group of several species interacting with each other.
>
{: .details}

![PAMPA toolsuite workflow](../../images/PAMPA-toolsuite-tutorial/PAMPA_workflow.png "PAMPA toolsuite workflow")

Each part of this workflow has the same elementary steps :
 - **A first tool** to compute metrics:
   - Community metrics ({% tool [Calculate community metrics](toolshed.g2.bx.psu.edu/repos/ecology/pampa_communitymetrics/pampa_communitymetrics/0.0.2) %}):
     Total abundance, Species richness, Shannon index, Simpson index, Hill index and Pielou index
   - Population metrics ({% tool [Calculate presence absence table](toolshed.g2.bx.psu.edu/repos/ecology/pampa_presabs/pampa_presabs/0.0.2) %}):
   Abundance, Presence-absence
 - **A second tool** to compute Generalized Linear (Mixed) Models, testing the effect of ```'site'```, ```'year'``` and/or
   ```'habitat'``` on a metric selected by the user:
   - Community analysis ({% tool [Compute GLM on community data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmcomm/pampa_glmcomm/0.0.2) %}):
     The GLM(M) is computed on the whole dataset and, if a separation factor is selected by the user, GLM(M)s are computed on subsets
     of the dataset (example: one GLM(M) per ecoregion)
   - Population analysis ({% tool [Compute GLM on population data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmsp/pampa_glmsp/0.0.2) %}):
     One GLM(M) is computed on each species separately
 - **A third tool** common for Community and Population analyses with at least ```'year'``` as a fixed effect to
   create time-series plots from your GLM(M) results ({% tool [Create a plot from GLM data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_plotglm/pampa_plotglm/0.0.2) %}).
   Two plots will be created for each GLM(M): one from the *"Estimate"* values of the GLM(M) and one from raw values from metrics file.

> <details-title>Details about Generalized Linear (Mixed) Models</details-title>
>
> Generalized Linear Models (GLMs) and Generalized Linear Mixed Models (GLMMs) are two extensions of Linear Models (LMs).
> As LMs uses only Gaussian distribution (for continuous variables), GLMs permits to analyse a wide spectrum of data
> with various probability distributions (including Poisson distribution for count data and binomial distribution for
> presence-absence data). To go further, GLMMs permits to handle pseudo-replication generated by non-independant
> repeated observations by defining one or more random effect in your model.
> Here, GLM(M) tools allows you to define ```'year'``` and/or ```'site'``` as random effects to take account of temporal
> and/or spatial replicas. The other effects taken into account in the model that aren't random are defined as 'fixed'.
>
{: .details}

In this tutorial, we'll be working on data extracted from the data portal DATRAS (Database of Trawls Surveys) of the
International Council for the Exploration of the Sea (ICES). After pre-processing to fit the input format of the tools,
we'll see how to calculate biodiversity metrics and construct properly, easily and with good practice, several GLMMs
to test the effect of ```'year'``` and ```'site'``` on the species richness of each survey studied and on the abundance
of each species.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Upload and pre-processing of the data

This first step consist of downloading and properly prepare the data to use it in PAMPA toolsuite.

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (example: "DATRAS data analysis through PAMPA workflow")
>    for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the CSV files from [Zenodo](https://doi.org/10.5281/zenodo.4264936) via link with the three following links
>    ```
>    https://zenodo.org/record/4264936/files/CPUE%20per%20length%20per%20area_A.csv?download=1
>    https://zenodo.org/record/4264936/files/CPUE%20per%20length%20per%20area_B.csv?download=1
>    https://zenodo.org/record/4264936/files/CPUE%20per%20length%20per%20area_C.csv?download=1
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    You may as well use the shared data library or directly download the data from the DATRAS data portal but the previous
>    method is preferred.
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    The files used for this tutorial have been imported from DATRAS data portal in October 2020. Those files are
>    often updated on the data portal. So if you choose to download files directly from DATRAS there may be
>    some differences with the files on Zenodo and the final results of this tutorial.
>
>    > <tip-title>Importing data from DATRAS data portal</tip-title>
>    >
>    > * Go to link: https://datras.ices.dk/Data_products/Download/Download_Data_public.aspx
>    > * In the field 'Data products' choose 'CPUE per length per area'
>    > * In the field 'Survey' choose the survey of interest. Here, we chose to use 'EVHOE', 'BITS'
>    >   and 'SWC-IBTS' surveys but you can't select the three surveys at once in this field.
>    >   So, you'll have to select one survey at a time ('EVHOE', 'BITS' and 'SWC-IBTS') and then
>    >   repeat the six following steps for each of the three surveys.
>    > * In the field 'Quarter(s)' we chose to tick 'All'
>    > * In the field 'Year(s)' we chose to tick 'All'
>    > * In the field 'Area(s)' we chose to tick 'All'
>    > * In the field 'Species' we chose to tick 'Standard species' and 'All'
>    > * Click on 'Submit'
>    > * Tick 'I accept the documents and their associated restrictions and agree to acknowledge the data source'
>    > * Open the Galaxy Upload Manager ({% icon galaxy-upload %} on the top-right of the tool panel)
>    > * Drop files on the dedicated area or click 'Choose local files'
>    > * Click on 'Start'
>    > * Click on 'Close' to return to Galaxy tools interface
>    >
>    {: .tip}
>
> 3. Make sure that each datasets has the following header ```"Survey","Year","Quarter","Area","AphiaID",
>    "Species","LngtClass","CPUE_number_per_hour"``` and that separator are ","
>
> 4. In order to avoid mixing up the data files, we'll add a tag to each file:
>       - tag `#EVHOE` to the tabular data file of the EVHOE survey : "CPUE per length per area_A.csv"
>       - tag `#SWCIBTS` to the tabular data file of the SWC-IBTS survey : "CPUE per length per area_B.csv"
>       - tag `#BITS` to the tabular data file of the BITS survey : "CPUE per length per area_C.csv"
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    If tags doesn't display directly on your history, refresh the page.
>
> 5. Convert datatype CSV to tabular for each data file
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert CSV to tabular" %}
>
{: .hands_on}

These three data files gather Catch Per Unit Effort (CPUE) observations, a relative abundance measure mostly used
to assess exploited fish species populations :
 - EVHOE ({% cite EVHOE %}) is a French survey on Southern Atlantic Bottom Trawl, the extracted file available on Zenodo goes from
   1997 to 2018 and records CPUE for seven exploited fish species
 - BITS ({% cite BITS %}) is an international survey on Trawl in Baltic sea, the extracted file available on Zenodo goes from 1991
   to 2020 and records CPUE for three exploited fish species
 - SWC-IBTS ({% cite SWCIBTS %}) is a Scottish survey on bottom trawl in the west coast, the extracted file available on Zenodo goes
   from 1985 to 2010 and records CPUE for six exploited fish species

## Prepare the data

Before starting the preparation of the data, we need to identify which inputs we need for the tools and their format
(same for Community or Population analyses except for a few details):
 - Input for the tools to compute metrics: Observation data file containing at least ```'year'``` and "location" sampled
   that might be represented in one field "observation.unit", "species.code" to name the species sampled,
   and ```'number'``` as the abundance value
 - Supplementary input for the tools to compute a GLM(M) analysis: Observation unit data file containing at least
   "observation.unit" and ```'year'```. It might also contain ```'habitat'```, ```'site'``` and any other information
   about the location at a given ```'year'``` that seems useful (examples: "temperature", "substrate type", "plant cover")

> <details-title>What is the 'observation.unit' field?</details-title>
>
> The 'observation.unit' field is a primary key linking the observation data file and the metrics data file to an observation
> unit data file. An unique 'observation.unit' identifier represents an unique sampling event at a given year and location.
> Such a file permits to reduce the size of data files, if all informations about the sampling event (examples:
> "temperature","substrate type","plant cover") are gathered in the same dataframe as the observations data
> ("species sampled","count") same informations will be unnecessarily repeated in many lines.
>
{: .details}

> <tip-title>The 'observation.unit' nomenclature</tip-title>
>
> For this workflow, two nomenclatures are advised :
> * The PAMPA file nomenclature:
>   ```[Two first letters of 'site'][Two last numbers of 'year'][Identifier of the precise sampled location in four numbers]```
>
>   The fields ```'year'``` and "location" will be automatically infered from this nomenclature in the first tools to
>   compute metrics
>
> * The automatic nomenclature: ```['year' in four numbers]_[Identifier of the precise sampled location in any character chain]```
>
>   This nomenclature will be automatically constructed in the first tools to compute metrics if the input file doesn't already
>   have a "observation.unit" field but only ```'year'``` and "location"
>
> If you already have your own nomenclature for the primary key linking your files, it will remain untouched by the tools but the
> output data table from the first tools to compute metrics won't contain ```'year'``` and "location" fields.
> In this case and if you want to add such informations to your metrics data file, you can use the tool
> {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2) %}
> with the metrics data file and the observation unit data file.
>
{: .tip}

> <details-title>The difference between fields ```'site'``` and "location"?</details-title>
>
> Those two fields both inform a geographical indication.
>
> "location" is the finest geographical level and ```'site'``` may represent a wider geographical range but can also
> represent the same geographical level as "location". This choice is up to your discretion, depending which geographical
> level you want to take into account in your statistical analysis.
>
{: .details}

### Concatenation of data files

> <hands-on-title>Data files concatenation</hands-on-title>
>
> 1. {% tool [Concatenate datasets tail-to-head (cat)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.0) %} select the three `#BITS`, `#EVHOE` and `#SWCIBTS`
>    datasets in {% icon param-files %} *"Datasets to concatenate"*
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
> 2. Suppress the `#BITS`, `#EVHOE` and `#SWCIBTS` tags and add the `#Concatenate` tag to the Concatenated data file
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 3. Verify if the three header lines have been included in the concatenate with {% tool [Count occurrences of each record](Count1) %}
>    and following parameters :
>    - {% icon param-file %} *"from dataset"*: `#Concatenate` Concatenated data file
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column: 1`
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-select %} *"How should the results be sorted?"*: `With the rarest value first` so multiple occurences can be seen directly at first lines of the resulting file
>
> 4. If the value `Survey` has more than `1` occurrence use {% tool [Filter data on any column using simple expressions](Filter1) %}
>    with following parameters :
>    - {% icon param-file %} *"Filter"*: `#Concatenate` Concatenated data file
>    - {% icon param-text %} *"With following condition"*: `c1!='Survey'`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
>    > <question-title></question-title>
>    > How many header line(s) is left?
>    >
>    > Use {% tool [Count occurrences of each record](Count1) %} and following parameters :
>    >   - {% icon param-file %} *"from dataset"*: `#Concatenate` Filtered data file
>    >   - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column: 1`
>    >   - {% icon param-select %} *"Delimited by"*: `Tab`
>    >   - {% icon param-select %} *"How should the results be sorted?"*: `With the rarest value first`
>    >
>    > **Tip:** Hit the rerun button {% icon galaxy-refresh %} on the output of the previous run of {% tool [Count occurrences of each record](Count1) %},
>    > and change the input dataset.
>    >
>    > > <solution-title></solution-title>
>    > > One header line is left as the value `Survey` has only `1` occurrence in the first column.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}

### Formating of data files

Here we have data from the three surveys in one datatable (Filtered data file) with the following fields ```"Survey",
"Year","Quarter","Area", "AphiaID","Species","LngtClass","CPUE_number_per_hour"```. We now need to define the proper
columns needed for the PAMPA workflow considering there is no primary key close to "observation.unit" :
 - "location" from "Area" associated with "Survey" to avoid mixing up areas from different surveys
 - ```'year'``` from "Year"
 - "species.code" from "Species"
 - ```'number'``` from "CPUE_number_per_hour"

This `#Concatenate` Filtered data file containing the three surveys will be used to create the observation unit data file and
compute Community metrics as the Community analysis can be automatically processed on each survey separately.
However, this file can't be used as it is to compute Population metrics as it may contain data for several
populations of the same species that do not interact with each other. Therefore, we have to work on the three
`#BITS`, `#EVHOE` and `#SWCIBTS` original tabular files for the Population analysis.

Hence, the following manipulations will be applied not only on `#Concatenate` Filtered data file but also on
the three `#BITS`, `#EVHOE` and `#SWCIBTS` original tabular files used for the concatenation.



> <hands-on-title>Create the "location" field</hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.0) %} with following parameters:
>    - {% icon param-files %} *"Select cells from"*: `#Concatenate` Filtered data file +
>      three `#BITS`, `#EVHOE` and `#SWCIBTS` tabular data files
>    - {% icon param-select %} *"using column"*: `Column: 1`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `([A-Z]{3}[A-Z]+)`
>         - {% icon param-text %} *"Replacement"*: `\1-`
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    > <question-title></question-title>
>    > What did we do with this tool?
>    > > <solution-title></solution-title>
>    > > We searched for every character chain matching the extended regular expression (Regex) `([A-Z]{3}[A-Z]+)`
>    > > in the first column "Survey" meaning every character chain of 4 or more capital letters. With the replacement `\1-`
>    > > we indicated to add `-` after the bracketed character chain. This manipulation is made in order to merge
>    > > properly columns "Area" and "Survey" with a separation.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>
> 2. {% tool [Merge Columns](toolshed.g2.bx.psu.edu/repos/devteam/merge_cols/mergeCols1/1.0.2) %} with following parameters :
>    - {% icon param-files %} *"Select data"*: four `#Concatenate`, `#BITS`, `#EVHOE` and `#SWCIBTS`
>      Column Regex Find And Replace data files
>    - {% icon param-select %} *"Merge column"*: `Column: 1`
>    - {% icon param-select %} *"with column"*: `Column: 4`
>
>    Check if the output data files has a supplementary column called "SurveyArea" and values with the survey ID and area ID
>    separated with `-`.
>
{: .hands_on}

> <hands-on-title>Change column names</hands-on-title>
>
> {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.0) %} with following parameters :
>    - {% icon param-files %} *"Select lines from"*: four `#Concatenate`, `#BITS`, `#EVHOE` and `#SWCIBTS` Merged data files
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `Year`
>         - {% icon param-text %} *"Replacement"*: `year`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `Species`
>         - {% icon param-text %} *"Replacement"*: `species.code`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `CPUE_number_per_hour`
>         - {% icon param-text %} *"Replacement"*: `number`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `SurveyArea`
>         - {% icon param-text %} *"Replacement"*: `location`
>
> Check if the columns are corresponding to ```"Survey","year","Quarter","Area","AphiaID","species.code","LngtClass",
> "number","location"``` (pay attention to case).
>
{: .hands_on}

> <details-title>Other tools that could be useful to pre-process data</details-title>
>
> {% tool Add input name as column %}; {% tool Compute an expression on every row %}
>; {% tool Table Compute %}; {% tool Join two files on column allowing a small difference %}
>; {% tool Join two files %}; {% tool Filter Tabular %}; {% tool melt %}
>; {% tool Unfold columns from a table %}; {% tool Remove columns by heading %}
>; {% tool Cut columns from a table %}; {% tool Filter data on any column using simple expressions %}
>; {% tool Transpose rows/columns in a tabular file %}; ...
>
{: .details}

# Compute Community and Population metrics

In this part of the tutorial, we're starting to use tools from the PAMPA toolsuite to compute Community
and Population metrics. Then, we'll be creating the observation unit file.

> <hands-on-title>Compute Community metrics</hands-on-title>
>
> 1. {% tool [Calculate community metrics](toolshed.g2.bx.psu.edu/repos/ecology/pampa_communitymetrics/pampa_communitymetrics/0.0.2) %} with following parameters:
>    - {% icon param-file %} *"Input file"*: `#Concatenate` Regex Find And Replace data file
>    - {% icon param-select %} *"Choose the community metrics you want to compute"*: `All`
>
>    Check if the output Community metrics data file has the following column names ```"year","location","number",
>    "species_richness","simpson","simpson_l","shannon","pielou","hill","observation.unit"```.
>
> 2. Tag the output Community metrics with `#Community`
>
{: .hands_on}

> <hands-on-title>Compute Population metrics</hands-on-title>
>
> {% tool [Calculate presence absence table](toolshed.g2.bx.psu.edu/repos/ecology/pampa_presabs/pampa_presabs/0.0.2) %} with {% icon param-files %} *"Input file"*: `#EVHOE`, `#BITS` and `#SWCIBTS` Regex Find And Replace data files
>
> Check if the outputs Population metrics data files has the following column names ```"year","location",
> "species.code","number","presence_absence","observation.unit"```.
>
{: .hands_on}

## Creation of the observation unit file


Now that we have the "observation.unit" fields automatically created in the Community and Population
metrics files we can create the observation unit file from the `#Concatenate` Regex Find And Replace data file with the same
nomenclature for the "observation.unit" fields, namely:

```['year' in four numbers]_[Identifier of the precise sampled location in any character chain]```

> <hands-on-title>Create the observation unit file</hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.0) %} with following parameters:
>    - {% icon param-files %} *"Select cells from"*: `#Concatenate` Regex Find And Replace data file
>    - {% icon param-select %} *"using column"*: `Column: 2`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `([0-9]{4})`
>         - {% icon param-text %} *"Replacement"*: `\1_`
>
> 2. {% tool [Merge Columns](toolshed.g2.bx.psu.edu/repos/devteam/merge_cols/mergeCols1/1.0.2) %} with following parameters :
>    - {% icon param-files %} *"Select data"*: `#Concatenate` Column Regex Find And Replace data file
>    - {% icon param-select %} *"Merge column"*: `Column: 2`
>    - {% icon param-select %} *"with column"*: `Column: 9`
>
>    Check if the output data file has a supplementary column called "yearlocation" and values with the ```'year'``` and "location" ID
>    separated with `_`.
>
>
> 3. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.0) %} with following parameters:
>    - {% icon param-files %} *"Select cells from"*: `#Concatenate` Merge Columns data file
>    - {% icon param-select %} *"using column"*: `Column: 2`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `([0-9]{4})_`
>         - {% icon param-text %} *"Replacement"*: `\1`
>
>    > <question-title></question-title>
>    > What did we do with this tool?
>    >
>    > > <solution-title></solution-title>
>    > > We removed the `_` we added in the first part of this "Hands_on" from the values of the second column
>    > > ```'year'``` to get proper numeric values back.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 4. {% tool [Cut columns from a table (cut)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %}
>    with following parameters :
>    - {% icon param-files %} *"File to cut"*: `#Concatenate` Column Regex Find And Replace data file
>    - {% icon param-select %} *"Operation"*: `Keep`
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-select %} *"Cut by"*: `fields`
>      - {% icon param-select %} *"List of Fields"*: `Column: 1` `Column: 2` `Column: 3` `Column: 4` `Column: 9` `Column: 10`
>
>    Check if the output `#Concatenate` Advanced Cut data file has the following column names ```"Survey","year","Quarter",
>    "Area","location","yearlocation"```.
>
>
> 5. {% tool [Sort data in ascending or descending order](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1) %}
>     with following parameters :
>    - {% icon param-files %} *"Sort Query"*: `#Concatenate` Advanced Cut data file
>    - {% icon param-text %} *"Number of header lines"*: `1`
>    - *"1: Column selections"*
>      - {% icon param-select %} *"on column"*: `Column: 6`
>      - {% icon param-check %} *"in"*: `Ascending order`
>      - {% icon param-check %} *"Flavor"*: `Alphabetical sort`
>    - {% icon param-select %} *"Output unique values"*: `Yes`
>    - {% icon param-select %} *"Ignore case"*: `No`
>
>    The output `#Concatenate` Sort data file must have fewer lines than the `#Concatenate` Advanced Cut data file.
>
>
> 6. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.0) %} with following parameters :
>    - {% icon param-files %} *"Select lines from"*: `#Concatenate` Sort data file
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `yearlocation`
>         - {% icon param-text %} *"Replacement"*: `observation.unit`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `location`
>         - {% icon param-text %} *"Replacement"*: `site`
>
>    Here, we define ```'site'``` represents the same geographical level as "location".
>
>
> 7. Tag the output `#Concatenate` Regex Find And Replace data file with `#unitobs`
>
>    Check if the output `#Concatenate #unitobs` data file has the following column names ```"Survey","year","Quarter",
>    "Area","site","observation.unit"```.
>
{: .hands_on}

# GLMMs and plots on Community and Population metrics

Now that we have all our input files for GLM(M) tools ready, we can start computing statistical models to make
conclusions on our surveys. But before starting to use the {% tool [Compute GLM on community data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmcomm/pampa_glmcomm/0.0.2) %}
and the {% tool [Compute GLM on population data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmsp/pampa_glmsp/0.0.2) %} tools, we have to think about the model we want to compute.
The tools allows to test the effects ```Year'```, ```Site``` and/or ```Habitat``` on one interest variable at a time.
As we don't have data about the ```Habitat``` we can't test its effect. It is always better to take a temporal and a
geographical variable into account in an ecological model so we'll test the effect of ```Year``` and ```Site``` with
a random effect of ```Site``` to take geographical pseudo-replication into account.

## Compute GLMMs and create plots on Community metrics

For the Community analysis we have the choice to test the effect of ```Year``` and ```Site``` on:
 - "Total abundance"
 - or "Species richness"
 - or "Simpson index"
 - or "1 - Simpson index"
 - or "Shannon index"
 - or "Pielou index"
 - or "Hill index"

As a GLM(M) permits to take into account only one interest variable at a time, we choose to take "Species richness"
as the interest variable, namely the quantity of different species at a given time and location.
This metric is located on the fourth column of the `#Concatenate #Community` metrics data file.

However, if you're interested to test the effects of ```Year``` and ```Site``` on another metric you may compute
the GLM(M) tool again and choose any column of the `#Concatenate #Community` metrics data file containing one of
the metrics cited above.

As said above, analyses will be separated by the field "Survey" (first column) to avoid bias.

> <hands-on-title>Compute GLMM on Community metrics</hands-on-title>
>
> 1. {% tool [Compute GLM on community data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmcomm/pampa_glmcomm/0.0.2) %} with following parameters:
>     - {% icon param-file %} *"Input metrics file"*: `#Concatenate #Community` metrics data file
>     - {% icon param-file %} *"Unitobs informations file"*: `#Concatenate #unitobs` data file
>     - {% icon param-select %} *"Interest variable from metrics file"*: `Column: 4`
>     - {% icon param-select %} *"Separation factor of your analysis from unitobs file"*: `Column: 1`
>     - {% icon param-select %} *"Response variables"*: `Year` `Site`
>     - {% icon param-select %} *"Random effect?"*: `Site`
>     - {% icon param-select %} *"Specify advanced parameters"*: `No, use program defaults`
>
> The three outputs must have three tags `#Concatenate #Community #unitobs` and the first one named 'GLM - results
> from your community analysis' must contain five lines. This file is formated to have one line per GLM processed,
> it has been formated this way to make GLM results easier to represent and modify if needed. However, this format is
> often not suited to read directly the results on Galaxy as Galaxy handles better tabular files with a lot of lines
> rather than files with a lot of columns. So, if your file doesn't display properly on Galaxy as a data table, we advise
> to do the following optional step.
>
>
> 2. (optional) {% tool  [Transpose rows/columns in a tabular file](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0) %} with `#Concatenate #Community #unitobs` 'GLM - results from your community analysis' data file
>
{: .hands_on}

> <details-title>Details on advanced parameters of GLM tools</details-title>
>
> For now, there are two advanced parameters you can setup in the GLM tools:
>  - {% icon param-select %} *"Distribution for model"* permits you to choose a probability distribution for your
>    model. When `Auto` is selected, the tool will automatically set the distribution on `Gaussian`, `Binomial`
>    or `Poisson` depending on the type of your interest variable but it may exist a better probability distribution
>    to fit your model. Mind that if you choose to use a random effect, "quasi" distribution can't be used for your model.
>
>  - {% icon param-select %} *"GLM object(s) as .Rdata output?"* permits you to get .Rdata file of your GLM(M)s if you
>    want to use it in your own R console.
>
{: .details}

> <details-title>How to choose a proper distribution family for your model?</details-title>
>
> The distribution family for a model is mainly selected by looking at the interest variable of the model:
>  - If your interest variable is a continuous numeric variable (weigth, size, mean, ...) it follows a
>    `Gaussian` distribution so you should select `Gaussian`.
>  - If your interest variable is a discrete numeric variable (count) it follows a `Poisson` distribution
>    so you should select `Poisson`.
>  - If your interest variable is a binary variable (0-1, presence-absence) it follows a `Binomial` distribution
>    so you should select `Binomial`.
>
> These three distributions are the most commonly used, the other distributions are used when the interest variable
> , the data and/or the model has specifics:
>  - When the model is over-dispersed, use a `Quasi-Poisson` (when interest variable is discrete) or a `Quasi-Binomial`
>    (when interest variable is binary) distribution.
>  - If your model doesn't fit well and your interest variable is a **positive** continuous variable, you may try
>    with `Gamma` or `Inverse-Gaussian` distribution.
>
{: .details}

### Read and interpret raw GLMM outputs

The GLM tool has three outputs:
 - The first output, 'GLM - results from your community analysis' data file, contains the summary of each GLMMs
   on each lines.
 - The second output, 'Simple statistics on chosen variables' file, contains descriptive statistics (mean, median, ...)
   of the interest variable for the whole input data table and for each selected factor levels combinations.
   Here, each ```Site``` at each ```Year```.
 - The third output, 'Your analysis rating file', contains a notation for all analyses performed and notations for
   each GLMMs processed. This rate is based on nine criterias commonly used in ecology to estimate wether a Linear
   Model has been constructed properly or not. At the end of the file, red flags and advices to improve your model
   are displayed also depending on the same criterias. This file is really useful to look at the bigger picture

> <details-title>Details on criterias used in 'Your analysis rating file'</details-title>
>
> For now, there are nine criterias taken into account to rate GLM(M)s in the Galaxy tools:
>  - *Plan completeness*: If part of factor levels combinations are inexistant, bias can be induced. (0.5)
>  - *Plan balance*: If factor levels combinations aren't present in same quantity, bias can be induced. (0.5)
>  - *NA proportion*: Lines with one or more NA value are ignored in Linear Models. If many NA values, there are
>    less observations taken in account in the model than expected. (1)
>  - *Residuals dispersion* (DHARMa package): If the dispersion of residuals isn't regular, the model is either under- or
>    over-dispersed. The distribution selected for the model isn't right. (1.5)
>    To know if it is under- or over-dispersion, compute *"deviance"*/*"df.resid"* : if the result is > 1 it is
>    over-dispersion, if the result is < 1 it is under-dispersion.
>  - *Residuals uniformity* (DHARMa package): If the residuals aren't uniform, the distribution selected for the model
>    isn't right. (1)
>  - *Outliers proportion* (DHARMa package): If too much outliers, the distribution selected for the model isn't right
>    or data is biased. (0.5)
>  - *Zero-inflation* (DHARMa package): If too much zeros in data, bias can be induced. (1)
>  - *Observations / factors ratio*: If the number of factors taken into account in the model exceeds 10% of the number
>    of observations (ex: 10 factors for 90 observations), bias can be induced. (1)
>  - *Number of levels of the random effect*: If no random effect in the model or if the factor selected as random effect
>    has less than 10 levels. (1)
>
{: .details}

#### First output: 'GLM - results from your community analysis'

First, on the 'GLM - results from your community analysis' data file, we see four analyses: `global` on the whole dataset
and one analysis for each of the three surveys `BITS`, `EVHOE` and `SWC-IBTS`. We also see the `Poisson` distribution has been
automatically selected by the tool. It seems right as the interest variable "Species richness" is a count.

Then, we see five fields that gives indexes to evaluate the quality of a model in comparison to another model performed
on the same data, so they can't be used to compare the four models performed here :
 - *"AIC"* Akaike information criterion. It is based on the number of factors. The lower AIC is better.
 - *"BIC"* Bayesian information criterion, is a derivate of AIC. It is based on the number of factors selected and on the
   sample size. The lower BIC is better.
 - *"logLik"* log-likelihood, is an index that estimates the likelihood of obtaining the sample of observations used
   for the model in the selected distribution. The higher logLik is better.
 - *"deviance"*, evaluates the model's goodness of fit. The lower deviance is better.
 - *"df.resid"*, degrees of freedom in residuals used to calculate the adjusted R-square that indicates the fit quality.

Three statistics about the random effect ```Site``` are given in this first file: standard deviation, number of observations taken
into account in the model and number of levels in the random effect.
As ```Site``` have been set as a random effect, pseudo-replication caused by the factor ```Site``` variable is taken into account to
fit the model as well as its effects on the interest variable but it is not tested so we can't make any conclusion regarding
its effects on species richness.

At this point on, elements on fixed effects are displayed :
 - *"Estimate"* is the mean value of the interest variable estimated by the model for a given factor.
 - *"Std.Err"* standard error, is the standard deviation of the *"Estimate"*. It measures uncertainty of the *"Estimate"*.
 - *"Zvalue"* (or *"Tvalue"* depending on the distribution) is the *"Estimate"* divided by the *"Std.Err"*.
 - *"Pvalue"* is the probability that the interest variable is not impacted by a given factor. If this probability is under a
   certain value (commonly 5% in ecology so a p-value under 0.05), we determine the factor has a significant effect on the
   interest variable.
 - *"IC_up"* and *"IC_inf"* are the upper and lower values of the confidence interval at 95% of the *"Estimate"*. The confidence
   interval is the range of values that your *"Estimate"* may take 95% of the time if you re-sample your data.
 - *"signif"* significativity, "yes" if the *"Pvalue"* is inferior to 0.05. Tells you wether the tested factor has a significant
   effect on your interest variable.

If the factor used as fixed effect is defined as a qualitative variable, those seven fields will be provided for each factor's level
and if it is a continuous variable they will be provided only once for the whole factor.

Here, the factor ```'year'``` may be considered a continuous variable as well as a qualitative variable. Only, it doesn't provide the same
informations :
 - If we consider ```'year'``` as a continuous variable the model will inform on the trend of the interest variable on the whole time-series.
   Making it more interesting to know about the global behaviour of the interest variable over time.
 - If we consider ```'year'``` as a qualitative variable the model will inform on the relative mean of interest variable for each year
   individually. Making it more interesting to know about each year individually and plot a detailed curve.

As both these solutions are interesting, the tool is computing two GLMMs when ```Year``` is selected as a fixed effect:
 - one GLMM with ```Year``` as a continuous variable with results on fields *"year Estimate"*, *"year Std.Err"*, *"year Zvalue"*,
   *"year Pvalue"*, *"year IC_up"*, *"year IC_inf"* and *"year signif"*.
 - one GLMM with ```Year``` as a qualitative variable in the same fields with each ```Year```'s levels as *"YYYY Estimate"*,
   *"YYYY Std.Err"*, *"YYYY Zvalue"*, *"YYYY Pvalue"*, *"YYYY IC_up"*, *"YYYY IC_inf"* and *"YYYY signif"*.

The *"(Intercept)"* value represents the reference value, it is rarely used for interpretations.
We see no significant effect on the 'global' model which seems logical as it considers the three surveys that are in
three geographical areas so three distinct communities. However, we observe the same thing with models on each of
the three surveys separately.

> <question-title></question-title>
> What does this result with no significant effect found mean? How can we explain it?
>
> > <solution-title></solution-title>
> > This result would mean that the communities sampled in this dataset haven't change in the whole
> > time series. However, this result can't be really trusted because the interest variable we selected is
> > the species richness and for the need of this tutorial we downloaded only part of the dataset with
> > a limited number of species. Hence, this dataset isn't really suited to perform a community analysis
> > as only a little part of the community is considered.
> >
> {: .solution}
>
{: .question}

#### Second output: 'Simple statistics on chosen variables'

The second output file doesn't tell much about the statistic models, it contains details for you to have an outlook on
the interest variable of the input data file containing the community metrics. We can see in the 'Base statistics'
part that the maximum species richness in the data file is 7 which is very low for trawl surveys and confirms this dataset
isn't suited to perform community analyses.

#### Third output: 'Your analysis rating file'

Here, we already pointed out a major issue with the original data file for community analyses and we know they
aren't robust. The third output will help us get a supplementary layer of understanding and take a step back on
the quality of the models performed.
We see the global rate is 3.5 out of 10 so it seems the analyses aren't of good quality but we need to see
each analysis individually to make real conclusions. The best rate is for the analysis on `EVHOE` survey 4 out
of 8 which is a medium rate, the model isn't necessarily bad but it can be improved. Indeed, data for this model
contains few NA values and it has a complete plan but not balanced. However, two very important criterias aren't
checked: residuals haven't a regular dispersion and aren't uniform so it seems Poisson distribution doesn't fit.
Besides, there isn't enough factor's levels in ```'site'``` to use it as a random effect. To get a relevant model we
should redo the model without ```Site``` as a random effect but in order to shorten this tutorial we won't develop this
manipulation here.

You can see in the last part of the file 'Red flags - advice', some advices to improve your models but the issues
detected in models can come from different reasons (like here, the data isn't representative enough of the whole
community) and may be hard to correct. Hence, you can try with another probability distribution but this issue may
come from the fact that the data isn't suited to make an analysis on species richness.
Overall, the four models aren't really trust-worthy.

#### Summary

With all this interpretation made from raw results, we still don't really visualize these results clearly. For this, it
could be nice to get a graphical representation of the GLMMs results and the interest variable.
Here, for these models, we know these plots won't be trust-worthy but it can still be interesting.


### Create and read plots

> <hands-on-title>Create plots from Community metrics file and GLMM results</hands-on-title>
>
> {% tool [Create a plot from GLM data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_plotglm/pampa_plotglm/0.0.2) %} with following parameters:
>  - {% icon param-file %} *"Input glm results file"*: `#Concatenate #Community #unitobs` 'GLM - results
>    from your community analysis' data file
>  - {% icon param-file %} *"Data table used for glm"*: `#Concatenate #Community` metrics data file
>  - {% icon param-file %} *"Unitobs table used for glm"*: `#Concatenate #unitobs` data file
>
> The output must be a data collection with four PNG files.
>
{: .hands_on}

![Community analysis plots](../../images/PAMPA-toolsuite-tutorial/Community.png "Community analysis plots")

Each PNG file contains two plots :
 - In the upper plot (blue), the line represents the *"Estimate"* values for each year and the confidence interval
   is the light blue ribbon. There is no significantly different year here but, usually, it is represented by a white
   dot inside the blue point of the *"Estimate"* value of the year. The writing on top of the plot is the *"Estimate"*
   value for the whole time-series (year as continuous variable), if nothing else is written it isn't significant.
 - In the lower plot (purple), the line represents the observed mean of the species richness for each year (calculated
   directly from the input metrics data table). If the interest variable had been a presence-absence, it would have been
   a presence proportion and if it had been an abundance it would have been a raw abundance (sum of all abundances each year).

We'll ignore the 'global' analysis plot as it associate data from different communities. We see on both plots of each of the
three surveys that, as predicted, species richness haven't varied much as few species are taken into account in the analysed
data set and these species are very commonly found as we selected only 'Standard species' when downloading the datasets.
Hence, mean species richness are globally around the maximum number of species taken into account in the surveys (7 for
`SWC-IBTS`, 6 for `EVHOE` and 3 for `BITS`). Moreover, confidence intervals are very wide and global trends are around 1.

## Compute GLMMs and create plots on Population metrics

For the Population analysis we have the choice to test the effect of ```Year``` and ```Site``` on:
 - "Abundance"
 - or "Presence-absence"

As a GLM(M) permits to take into account only one interest variable at a time, we choose to take
"Abundance" as the interest variable, namely the quantity of a given species at a given time and location.
This metric is located on the fourth column of the `#EVHOE`, `#BITS` and `#SWCIBTS` Population metrics data files.

However, if you're interested to test the effects of ```Year``` and ```Site``` on "Presence-absence" you may compute
the GLM(M) tool again and choose the fifth column of the `#EVHOE`, `#BITS` and `#SWCIBTS` Population metrics data
files.


> <hands-on-title>Compute GLMM on Population metrics</hands-on-title>
>
> 1. {% tool [Compute GLM on population data](toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmsp/pampa_glmsp/0.0.2) %} with following parameters:
>     - {% icon param-files %} *"Input metrics file"*: `#EVHOE`, `#BITS` and `#SWCIBTS` metrics (here presence absence) data files
>     - {% icon param-file %} *"Unitobs informations file"*: `#Concatenate #unitobs` data file
>     - {% icon param-select %} *"Interest variable from metrics file"*: `Column: 4`
>     - {% icon param-select %} *"Response variables"*: `Year` `Site`
>     - {% icon param-select %} *"Random effect?"*: `Site`
>     - {% icon param-select %} *"Specify advanced parameters"*: `No, use program defaults`
>
>    The three outputs for each survey must have three tags `#Concatenate #SurveyID #unitobs`.
>
>    The first files of each survey named 'GLM - results from your population analysis' must contain one line
>    per GLM processed, it has been formated this way to make GLM results easier to represent and modify if needed.
>    However, this format is often not suited to read directly the results on Galaxy as Galaxy handles better tabular
>    files with a lot of lines rather than files with a lot of columns. So, if your files doesn't display properly
>    on Galaxy as a data table, we advise to do the following optional step.
>
>
> 2. (optional) {% tool [Transpose rows/columns in a tabular file](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0) %}
>    with `#Concatenate #EVHOE #unitobs` `#BITS #Concatenate #unitobs` `#Concatenate #SWCIBTS #unitobs`
>    'GLM - results from your population analysis' data files
>
{: .hands_on}

### Read and interpret raw GLM outputs

The three outputs for each time the population GLM tool runs are the same as in the community GLM tool described in
the previous part 3. 1. 1. Read and interpret raw GLM outputs.
For the need of this tutorial, we'll be analyzing only the `BITS` survey analysis results.

#### First output: 'GLM - results from your population analysis'

First, on the `#BITS #Concatenate #unitobs` 'GLM - results from your population analysis' data file, we see three analyses
one for each species in the `BITS` survey: `Gadus morhua` (Atlantic cod), `Platichthys flesus` (European flounder) and
`Pleuronectes platessa` (European plaice).

![BITS survey standard fish species](../../images/PAMPA-toolsuite-tutorial/Fish-BITS.png "BITS survey standard fish species")

We also see the `Gaussian` distribution has been automatically selected by the tool. It seems right even if the interest variable
"Abundance" is usually a count, here it is a Catch Per Unit Effort abundance so it isn't a round value.

Then, the details of what contains each field in this file has been developped in the previous part on community analyses 3. 1. 1. 1. First output: 'GLM - results from your community analysis'.

Three statistics about the random effect ```Site``` are given in this first file: standard deviation, number of observations taken
into account in the model and number of levels in the random effect.
As ```'site'``` have been set as a random effect, pseudo-replication caused by the factor ```Site``` variable is taken into account to
fit the model as well as its effects on the "Abundance" but it is not tested so we can't make any conclusion regarding
its effects on CPUE abundance.

As said above, the tool is computing two GLMMs when ```Year``` is selected as a fixed effect:
 - one GLMM with ```Year``` as a continuous variable with results on fields *"year Estimate"*, *"year Std.Err"*, *"year Zvalue"*,
   *"year Pvalue"*, *"year IC_up"*, *"year IC_inf"* and *"year signif"*.
 - one GLMM with ```Year``` as a qualitative variable in the same fields with each ```Year```'s levels as *"YYYY Estimate"*,
   *"YYYY Std.Err"*, *"YYYY Zvalue"*, *"YYYY Pvalue"*, *"YYYY IC_up"*, *"YYYY IC_inf"* and *"YYYY signif"*.

The *"(Intercept)"* value represents the reference value, it is rarely used for interpretations.

Regarding the significance of the effect ```Year``` as continuous variable:
 - It isn't significant in the `Gadus morhua` model.
 - It is significant in the `Platichthys flesus` model.
 - It is significant in the `Pleuronectes platessa` model.

> <question-title></question-title>
> What does these results on the effect ```Year``` as continuous variable mean?
>
> > <solution-title></solution-title>
> > The question behind the test of this effect is: "Is the global temporal trend different from 1?"
> > When it isn't significant as in the `Gadus morhua` model, it means the *"year Estimate"* value isn't
> > significantly different from 1 so the interest variable "Abundance" hasn't varied much through time.
> > When it is significant as in the two other models, the *"year Estimate"* value is significantly different
> > from 1 so the interest variable "Abundance" has varied through time.
> >
> {: .solution}
>
{: .question}

These results are consistent with the results on the significance of the effects ```Year``` as qualitative variable:
 - Only one year out of 30 considered significantly different in the `Gadus morhua` model. On year 1995.
 - 14 years out 

tools_json:
["Calculate community metrics", "toolshed.g2.bx.psu.edu/repos/ecology/pampa_communitymetrics/pampa_communitymetrics/0.0.2", "Calculate presence absence table", "toolshed.g2.bx.psu.edu/repos/ecology/pampa_presabs/pampa_presabs/0.0.2", "Compute GLM on community data", "toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmcomm/pampa_glmcomm/0.0.2", "Compute GLM on population data", "toolshed.g2.bx.psu.edu/repos/ecology/pampa_glmsp/pampa_glmsp/0.0.2", "Create a plot from GLM data", "toolshed.g2.bx.psu.edu/repos/ecology/pampa_plotglm/pampa_plotglm/0.0.2", "Join", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2", "Concatenate datasets tail-to-head (cat)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.0", "Count occurrences of each record", "Count1", "Filter data on any column using simple expressions", "Filter1", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.0", "Merge Columns", "toolshed.g2.bx.psu.edu/repos/devteam/merge_cols/mergeCols1/1.0.2", "Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.0", "Cut columns from a table (cut)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "Sort data in ascending or descending order", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1", "Transpose rows/columns in a tabular file", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0"]

tags_json:
["species population EBV class", "community composition EBV class", "EBV dataset", "EBV workflow", "modeling", "biodiversity"]


topic: ecology
tutorial: x-array-map-plot
title: Visualization of Climate Data using NetCDF xarray Map Plotting
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["pangeo"]
content_hash: 2fb3a5ad54dfa855f2d89bfadd478834
last_modified: 2026-05-15T22:28:12.116640
zenodo_link: https://doi.org/10.5281/zenodo.6621460
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/x-array-map-plot/tutorial.html

content:
The data used in this tutorial is [ECMWF Reanalysis 5 hourly data on single levels](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview). We are interested in the following variables: air temperature at 2 metres, latitude, longitude and time. Our main objective is to plot the global air temperature at 2 metres with respect to time. For this we will be using the netCDF xarray tool available in the Galaxy Europe (or your favourite Galaxy Instance) server.

It will be a fun learning experience for anyone who loves visualization !

><comment-title></comment-title>
>
> The tutorial aims for establishing a good knowledge about meaningful visualization of climate data.
>
{: .comment}

It is beginner friendly and does not require much knowledge about the tool.

> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
> According to [UN](https://www.un.org/en/climatechange/what-is-climate-change) , Climate is the long term shift in temperature and weather patterns which may be due to natural or artificial causes. To learn more about climate, refer this [tutorial]({% link topics/climate/tutorials/climate-101/tutorial.md %}) from the GTN. Due to the frequently changing nature of the weather patterns, the size of the collected data is huge.
> The climate data is mainly represented in these three categories : [NetCDF](https://en.wikipedia.org/wiki/NetCDF) (Network Common Data Form), [HDF](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) (Hierarchical Data Format) , [GRIB](https://en.wikipedia.org/wiki/GRIB) (GRIdded Binary or General Regularly-distributed Information in Binary form).
>
> The NetCDF file format is basically used for storing multidimensional data which generally consists of variables such as temperature, precipitation, wind direction, etc. The variation of climate variables over a period of time is suitably plotted using this dataset. The entire earth is divided into both horizontal as well as vertical coordinates which makes plotting of the variables such as the ocean temperatures possible.
>
> The coordinate system, types of projections and colormaps are some of the very important considerations in achieving the most suitable visualization option.
{: .comment}



# Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial. Name it as per your choice. My suggestions : *ECMWF_Reanalysis*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Import remote files**
>    - location: `ECMWF ERA5 Reanalysis` ->`2022` -> `05` -> `data` -> `air_temperature_at_2_metres.nc`
>
>    {% snippet faqs/galaxy/datasets_import_from_remote_files.md location="ECMWF ERA4 Reanalysis -> 2022 -> 05 -> data -> air_temperature_at_2_metres.nc" %}
>
>    **or**, if this is not available, you can import from [Zenodo]({{page.zenodo_link}}):
>
>    ```
>    https://zenodo.org/record/6621460/files/air_temperature_at_2_metres.netcdf
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. **Check the datatype** of uploaded data, it should be `netCDF`.
>    -  If it is not `netCDF`, make sure to convert it using the Galaxy built-in format converters.
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert h5 to netCDF" %}
>
> 4. **Rename Datasets** {% icon galaxy-pencil %} if you find the name to be too long or you have something more meaningful in your mind.
>
> 5. **Tag Datasets** {% icon galaxy-tags %} with the tag `#ecmwf`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# What is xarray?

[xarray](https://docs.xarray.dev/en/stable/), formerly known as xray, is a python package which enables us to play with gridded data. This package shares most of its features from [numpy](https://numpy.org/), but in a more convenient manner by keeping track of labels in arrays. The gridded data is mainly available in netCDF data format. Thus **xarray comes very handy while dealing with netCDF files.**


## What is Visualisation in xarray?

xarray uses **[Cartopy](https://scitools.org.uk/cartopy/docs/latest/)** and **[Matplotlib](https://matplotlib.org/)** as the two main tools for creating detailed and informative plots.
Cartopy is a python package used for geospatial data analysis. In the Python library, matplotlib is the most used scientific plotting library.
For a multidimensional data consisting of latitudes and longitudes along with the other variables, xarray has the capability of appling cartopy map projections.

# Splitting the dataset  using  seltimestep, splithour and plotting

After loading the required data, the following stage is to obtain the meta info or meta data of the file. The very purpose of these steps are to obtain the information about dimensions, variables, global attributes, etc. The coordinate info helps to know about the actual data entries present under the various variables.

Follow the below steps:

## NetCDF xarray Metadata Info

> <hands-on-title>netCDF dataset with xarray metadata Galaxy Tool</hands-on-title>
>
> 1. In the tools search for {% tool [NetCDF xarray Metadata Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1) %} with the following parameter:
>    - {% icon param-file %} *"Netcdf file"*: `air_temperature_at_2_metres.nc`
>  and click on `Execute`.
>
> 2. View {% icon galaxy-eye%} the two generated outputs:
>    - `Metadata infos` is a `tabular` providing the list of variables, their dimension names and number of elements per dimension. This file is used by other xarray Tools.
>    - The second file `info file` provides a summary of the **xarray Dataset** contained in your netCDF file.
{: .hands_on}




> <question-title></question-title>
>
> 1. What is the name of the variable for air temperature at 2 metres. What are its physical units?
>
>
> > <solution-title></solution-title>
> > 1. Information about variable names and units can be found in **info file** that was generated by xarray metadata Galaxy Tool.
> >      - Variable name: `air_temperature_at_2_metres`
> >      - Units: `K`
> >
> >
> > > <code-out-title></code-out-title>
> > > ```bash
> > > xarray.Dataset {
> > > dimensions:
> > > lat = 721 ;
> > > lon = 1440 ;
> > > time0 = 595 ;
> > >
> > >
> > > variables:
> > >   float32 lon(lon) ;
> > >     lon:standard_name = longitude ;
> > >     lon:long_name = longitude ;
> > >     lon:units = degrees_east ;
> > >
> > >   float32 lat(lat) ;
> > >     lat:standard_name = latitude ;
> > >     lat:long_name = latitude ;
> > >     lat:units = degrees_north ;
> > >
> > >   datetime64[ns] time0(time0) ;
> > >  		time0:standard_name = time ;
> > >
> > >  	float32 air_temperature_at_2_metres(time0, lat, lon) ;
> > >  		air_temperature_at_2_metres:standard_name = air_temperature ;
> > >  		air_temperature_at_2_metres:units = K ;
> > >  		air_temperature_at_2_metres:long_name = 2 metre temperature ;
> > >  		air_temperature_at_2_metres:nameECMWF = 2 metre temperature ;
> > >  		air_temperature_at_2_metres:shortNameECMWF = 2t ;
> > >  		air_temperature_at_2_metres:nameCDM = 2_metre_temperature_surface;
> > >  		air_temperature_at_2_metres:product_type = analysis ;
> > >
> > >
> > > // global attributes:
> > >
> > >   :source = Reanalysis ;
> > >   :institution = ECMWF ;
> > > 	:title = ERA5 forecasts ;
> > > }
> > > ```
> > {: .code-out}
> {: .solution }
{: .question }

## Sub-step with **NetCDF xarray Coordinate Info**

> <hands-on-title> Get Coordinate information with xarray Coordinate</hands-on-title>
>
> 1. {% tool [NetCDF xarray Coordinate Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.20.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `air_temperature_at_2_metres.nc`.
>2. View {% icon galaxy-eye%} the 4 generated outputs:
>    - `lat`: a tabular file containing all the latitude values of our Xarray dataset;
>    - `lon`: a tabular file containing all the longitudes values;
>    - `time0`: this tabular file contains all the times extracted from the netCDF file. In our case, these are relative to 01/05/2022, 00:00:00 UTC;
>    - `version`: this is a text file returning the xarray package version. It is useful when publishing your Galaxy workflow.
>
{: .hands_on}

> <comment-title></comment-title>
>
>The number of tabular files returned by this programme is proportional to the number of coordinate variables in your input file. No further processing is done after decoding the values from the netCDF input file. As a result, depending on how the original input file was coded, units for latitudes, longitudes, (level, if this dimension was present, which is not the case in this particular case) and time may differ from one file to the next.
>
{: .comment}




> <question-title>Understanding air temperature coordinates at 2 metres</question-title>
>
>
> 1. What is the format of time coordinate?
> 2. What is the range of values for latitude and longitude?
>
> > <solution-title></solution-title>
> > 1. The `info file` tells us that `time0` is coded as `timedelta64[ns]` i.e. the time differences are in ns (here in nanoseconds). The format of time is `yy-mm-dd  hh:mm:ss`. If we look at the tabular file named `time0` (generated by `NetCDF xarray Coordinate Info`), we see that these times are automatically converted to human readable time format when printed:
> >
> > > <code-out-title></code-out-title>
> > > ```
> > > 0  2022-05-01 00:00:00
> > > 1  2022-05-01 01:00:00
> > > 2  2022-05-01 02:00:00
> > > 3  2022-05-01 03:00:00
> > > 4  2022-05-01 04:00:00
> > > ```
> > >
> > {: .code-out}
> >
> > This tells us that we have hourly forecast data.
> >
> > 2.If we look at the tabular file named `lat` and `lon` (generated by `NetCDF xarray Coordinate Info`), we see that these files are displayed with two columns e.g. index numbers and corresponding values. From these, we infer that the range of values for `lat` is  `90.0`  to `-90.0`and for `lon` is `0` to `359.75`.
> >
> {: .solution }
{: .question }


## Operations on Climate data using **CDO Operations**

We have hourly data. In order to plot it, we must first extract the hours from the bigger dataset. This is done using the `seltimestep` and the `splithour` options available in the `CDO Operations` tool. `splithour` is used when we wish to plot more than an hour. Our main aim is to plot the last hour data on the last day present in the dataset.

> <hands-on-title>Defining a particular time range using seltimestep </hands-on-title>
>
> 1. {% tool [CDO Operations](toolshed.g2.bx.psu.edu/repos/climate/cdo_operations/cdo_operations/2.0.0+galaxy0) %} with the following parameters:
>    - In *"CDO Operators"*:
>        - {% icon param-repeat %} *"Insert CDO Operators"*
>            - *"Select cdo operator"*: `seltimestep (Select timesteps)`
>            - *"Timesteps for selection"*: `744/744`
>            - {% icon param-file %} *"Additional input file"*: `air_temperatures_at_2_metres.nc`
>
>
>
> > <comment-title></comment-title>
> >
> > The syntax of using the `seltimestep` is `(initial data number / final data entry)`. An important thing to pay attention to is how the data entries are numbered: are they numbered starting from 0 or 1. Accordingly we can add or skip adding 1 to the data number to achieve the desired result.
> >
> > Although we are not using `splithour` here, you can find below the syntax for future uses.
> >
> > 1. {% tool [CDO Operations](toolshed.g2.bx.psu.edu/repos/climate/cdo_operations/cdo_operations/2.0.0+galaxy0) %} with the following parameters:
> >
> >    - In *"CDO Operators"*:
> >    - {% icon param-repeat %} *"Insert CDO Operators"*
> >    - *"Select cdo operator"*: `splithour (Split hours)`
> >    - {% icon param-file %} *"Additional input file"*: `outfile.netcdf` generated from the previous step.
> >
> > This step generates that `N` number of `outfiles.netcdf` files where `N` is the range of selection.
> > Suppose your selected range was `744/744` for the `seltimestep` , then it will generate `2` files which can be plotted further.
> >
> {: .comment}
>
{: .hands_on}






## Finding the  **NetCDF xarray Metadata Info**

> <hands-on-title>netCDF dataset with xarray metadata Galaxy Tool for the hourly plots</hands-on-title>
>
> 1. {% tool [NetCDF xarray Metadata Info](toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `outfile.netcdf`. .
>
> 2. View {% icon galaxy-eye%} the two generated outputs:
>    - `Metadata infos` is a `tabular` providing the list of variables, their dimension names and number of elements per dimension. This file is used by other xarray Tools.
>    - The second file `info file` provide a summary of the **xarray Dataset** contained in your netCDF file.
{: .hands_on}

# Map Plotting using **NetCDF xarray map plotting**

> <hands-on-title>Plotting the data of the last hour of the day</hands-on-title>
> The air temperatures corresponding to the 744th time step from the original netCDF file, namely `23:00:00` for 31st May 2022 is plotted here :
>
> 1. {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.20.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Netcdf file"*: `outfile.netcdf`
>    - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from outfile.netcdf` (output of **NetCDF xarray Metadata Info** >{% icon tool %})
>    - *"Choose the variable to plot"*: `air_temperature_at_2_metres`
>    - *"Name of latitude coordinate"*: `lat`
>    - *"Name of longitude coordinate"*: `lon`
>    - *"Datetime selection"*: `No`
>    - *"Range of values for plotting e.g. minimum value and maximum value (minval,maxval) (optional)"*: `220,320`
>    - *"Add country borders with alpha value [0-1] (optional)"*: `1.0`
>    - *"Add coastline with alpha value [0-1] (optional)"*: `1.0`
>    - *"Add ocean with alpha value [0-1] (optional)"*: `1.0`
>    - *"Specify plot title (optional)"*: `Air_temperature_at_2_metres___23:00:00_UTC`
>    - *"Specify which colormap to use for plotting (optional)"*: `lajolla`
>    - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{'proj': 'Mercator'}`
>
> ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 UTC](../../images/x-array-map-plot/plot1.png)
>
>
{: .hands_on}

> <details-title>Syntax for Projections</details-title>
> **Xarray Map Plotting** comes with a variety of projections. There are basically four types of projections available namely **Azimuthal, Conic, Cylindrical and Pseudo-cylindrical.**
>
>The **Azimuthal projection** depicts the surface of the earth with the help of a flat plane. It is also known as '**Plane Projection**.' This simple projection type forms a family of projections by considering the poles in "normal aspect." Another form of this projection is **Azimuthal Equidistant Projection**. It preserves both directions and distance from the central point of the earth. The **stereographic projection** which is a type of azimuthal projection was created before 150AD and deployed while mapping areas over the poles.
> The **conic type** is based on the concept of projecting the surface of the Earth on a conical surface which is unrolled into a plane  surfaced map. Some of the examples of conic type of projections are : **Albers Equal Area Conic, Equidistant Conic, Lambert Conformal Conic, and Polyconic**. This type of projection basically has a big implementation in aviation navigation.
>
>
> The **cylindrical type** is based on the concept of plotting the geographical features on the surface of the cylinder and then it is unrolled to present as a **flat projection**. Some examples of cylindrical projections are : **Cylindrical Equal Area, Behrmann Cylindrical Equal-Area , Stereographic Cylindrical, Peters, Mercator, and Transverse Mercator**. **Mercator** is one of the most famous and preferred type of projection. The straight lines (Rhumb Lines) are best for the navigation process.
>
> In the **pseudo cylindrical** type of projection , the meridians are straight instead of being curved. These kinds of projections are characterised by straight horizontal lines for parallels of latitude and (usually) equally-spaced curved meridians of longitude. **Oval projections** pinched or flattened at the poles are its identifying features.
>
>
> **Projections**:
>
> There are about **30** different kinds of projections available in the proj option. The basic knowledge of projections is very essential in plotting out the best fit map. Below is the list of plots deployed using the various kinds of projections supported by the NetCDF xarray mapplotting tool.
>
>
> **PlateCarree**
>**Syntax: {"proj":"PlateCarree"}**
>![PlateCarree](../../images/x-array-map-plot/projections/PlateCarree.png)
>
> **EquidistantConic**
>**Syntax: {"proj":"EquidistantConic", "central_longitude": 20.0, "central_latitude": 70.0 }**
>![EquidistantConic](../../images/x-array-map-plot/projections/EquidistantConic.png)
>
> **AlbersEqualArea**
> **Syntax: {"proj":"AlbersEqualArea", "central_longitude": 20.0, "central_latitude": 70.0 }**
> ![AlbersEqualArea](../../images/x-array-map-plot/projections/AlbersEqualArea.png)
>
> **EuroPP**
> **Syntax: {"proj":"EuroPP"}**
> ![EuroPP](../../images/x-array-map-plot/projections/EuroPP.png)
>
> **LambertConformal**
> **Syntax:{"proj":"LambertConformal"  }**
> ![LambertConformal](../../images/x-array-map-plot/projections/LambertConformal.png)
>
> **AzimuthalEquidistant**
> **Syntax: {"proj":"AzimuthalEquidistant" }**
> ![AzimuthalEquidistant](../../images/x-array-map-plot/projections/AzimuthalEquidistant.png)
>
> **LambertCylindrical**
> **Syntax:{"proj":"LambertCylindrical"}**
> ![LambertCylindrical](../../images/x-array-map-plot/projections/LambertCylindrical.png)
>
> **Mercator**
> **Syntax: {"proj":"Mercator" }**
> ![Mercator](../../images/x-array-map-plot/projections/Mercator.png)
>
> **Miller**
> **Syntax: {"proj":"Miller"  }**
> ![Miller](../../images/x-array-map-plot/projections/Miller.png)
>
> **Mollweide**
> **Syntax: {"proj":"Mollweide" }**
> ![Mollweide](../../images/x-array-map-plot/projections/Mollweide.png)
>
> **Orthographic**
> **Syntax:{"proj":"Orthographic"  }**
> ![Orthographic](../../images/x-array-map-plot/projections/Orthographic.png)
>
> **Robinson**
> **Syntax: {"proj":"Robinson"  }**
> ![Robinson](../../images/x-array-map-plot/projections/Robinson.png)
>
> **Sinusoidal**
> **Syntax: {"proj":"Sinusoidal"  }**
> ![Sinusoidal](../../images/x-array-map-plot/projections/sinusoidal.png)
>
> **Stereographic**
> **Syntax: {"proj":"Stereographic"  }**
> ![Stereographic](../../images/x-array-map-plot/projections/Stereographic.png)
>
> **TransverseMercator**
> **Syntax:{"proj":"TransverseMercator"  }**
> ![TransverseMercator](../../images/x-array-map-plot/projections/TransverseMercator.png)
>
> **InterruptedGoodeHomolosine**
> **Syntax: {"proj":"InterruptedGoodeHomolosine"  }**
> ![InterruptedGoodeHomolosine](../../images/x-array-map-plot/projections/InterruptedGoodeHomolosine.png)
>
> **RotatedPole**
> **Syntax: {"proj":"RotatedPole"  }**
> ![RotatedPole](../../images/x-array-map-plot/projections/RotatedPole.png)
>
> **OSGB**
> **Syntax:{"proj":"OSGB"  }**
> ![OSGB](../../images/x-array-map-plot/projections/OSGB.png)
>
> **Geostationary**
> **Syntax: {"proj":"Geostationary"  }**
> ![Geostationary](../../images/x-array-map-plot/projections/Geostationary.png)
>
> **NearsidePerspective**
> **Syntax:{"proj":"NearsidePerspective" }**
> ![NearsidePerspective](../../images/x-array-map-plot/projections/NearsidePerspective.png)
>
> **EckertI**
> **Syntax: {"proj":"EckertI" }**
> ![EckertI](../../images/x-array-map-plot/projections/EckertI.png)
>
> **EckertII**
> **Syntax: {"proj":"EckertII" }**
> ![EckertII](../../images/x-array-map-plot/projections/EckertII.png)
>
> **EckertIII**
> **Syntax: {"proj":"EckertIII" }**
> ![EckertIII](../../images/x-array-map-plot/projections/EckertIII.png)
>
> **EckertIV**
> **Syntax: {"proj":"EckertIV" }**
> ![EckertIV](../../images/x-array-map-plot/projections/EckertIV.png)
>
> **EckertV**
> **Syntax: {"proj":"EckertV" }**
> ![EckertV](../../images/x-array-map-plot/projections/EckertV.png)
>
> **EckertVI**
> **Syntax: {"proj":"EckertVI" }**
> ![EckertVI](../../images/x-array-map-plot/projections/EckertVI.png)
>
> **EqualEarth**
> **Syntax:{"proj":"EqualEarth" }**
> ![EqualEarth](../../images/x-array-map-plot/projections/EqualEarth.png)
>
> **Gnomonic**
> **Syntax: {"proj":"Gnomonic" }**
> ![Gnomonic](../../images/x-array-map-plot/projections/Gnomonic.png)
>
> **LambertAzimuthalEqualArea**
> **Syntax: {"proj":"LambertAzimuthalEqualArea" }**
> ![LambertAzimuthalEqualArea](../../images/x-array-map-plot/projections/LambertAzimuthalEqualArea.png)
>
> **NorthPolarStereo**
> **Syntax: {"proj":"NorthPolarStereo" }**
> ![NorthPolarStereo](../../images/x-array-map-plot/projections/NorthPolarStereo.png)
>
> **OSNI**
> **Syntax: {"proj":"OSNI" }**
> ![OSNI](../../images/x-array-map-plot/projections/OSNI.png)
>
> **SouthPolarStereo**
> **Syntax: {"proj":"SouthPolarStereo" }**
> ![SouthPolarStereo](../../images/x-array-map-plot/projections/SouthPolarStereo.png)
>
{: .details}


> <question-title></question-title>
>
> What are the different kinds of projections that can be used?
>
> > <solution-title></solution-title>
> >
> > There are many projections which can be used in the `NetCDF xarray map plotting` tool. Different projections have different purposes and need to be carefully chosen.
> >
> > Plotting different projections using **NetCDF xarray map plotting**:
> > > <hands-on-title>Plotting the major projections using the xarray tool.</hands-on-title>
> > >
> > > 1. {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.20.2+galaxy0) %} with the following parameters:
> > >    - {% icon param-file %} *"Netcdf file"*: `outfile.netcdf`
> > >    - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from outfile.netcdf` (output of **NetCDF xarray Metadata Info** {% icon tool %})
> > >    - *"Choose the variable to plot"*: `air_temperature_at_2_metres`
> > >    - *"Name of latitude coordinate"*: `lat`
> > >    - *"Name of longitude coordinate"*: `lon`
> > >    - *"Datetime selection"*: `No`
> > >    - *"Range of values for plotting e.g. minimum value and maximum value (minval,maxval) (optional)"*: `220,320`
> > >    - *"Add country borders with alpha value [0-1] (optional)"*: `1.0`
> > >    - *"Add coastline with alpha value [0-1] (optional)"*: `1.0`
> > >    - *"Add ocean with alpha value [0-1] (optional)"*: `1.0`
> > >    - *"Specify plot title (optional)"*: `Air_temperature_at_2_metres___23:00:00_UTC_InterruptedGoodeHomolosine`
> > >    - *"Specify which colormap to use for plotting (optional)"*: `lajolla`
> > >    - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{"proj":"InterruptedGoodeHomolosine" }`
> > >
> > >
> > >
> > > The final plot is shown below:
> > >
> > > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 : Interrupted_GoodeHomolosine](../../images/x-array-map-plot/Interrupted_GoodeHomolosine.png)
> > >
> > >
> > >
> > > Some other potentially interesting types of projections can be found below :
> > >
> > > *{"proj":"LambertCylindrical" }*
> > > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 : LambertCylindrical](../../images/x-array-map-plot/Lam.png)
> > > *{"proj":"Orthographic"  }*
> > > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 : Orthographic](../../images/x-array-map-plot/Ortho.png)
> > > *{"proj":"Sinusoidal" }*
> > > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 : Sinusoidal](../../images/x-array-map-plot/sinu.png)
> > > *{"proj":"EquidistantConic"}*
> > > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 : EquidistantConic](../../images/x-array-map-plot/Equidi.png)
> > > *{"proj":"LambertConformal" }*
> > > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 : LambertConformal](../../images/x-array-map-plot/Lamcon.png)
> > > *{"proj":"AzimuthalEquidistant" }*
> > > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 at 23:00:00 : AzimuthalEquidistant](../../images/x-array-map-plot/Azi.png)
> > {: .hands_on}
> {: .solution}
{: .question}


> <details-title>Available ColorMaps</details-title>
> There are about **150 different kinds of maps available in the  colormaps** options. This document contains a list of all the types of projections and colormaps available to the users while using the Xarray Map Plotting. These depict the global air temperature at 2 metres on a Kelvin Scale of range 220 K TO 320 K. Let's get started!
> Below is the complete list for reference.
>
> **ACCENT**
> ![Accent](../../images/x-array-map-plot/colormaps/Accent.png)
>  **BLUES**
> ![Blues](../../images/x-array-map-plot/colormaps/Blues.png)
>  **BrBg**
> ![BrBg](../../images/x-array-map-plot/colormaps/BrBG.png)
>  **BuGn**
> ![BuGn](../../images/x-array-map-plot/colormaps/BuGn.png)
>  **BuPu**
> ![BuPu](../../images/x-array-map-plot/colormaps/BuPu.png)
>  **CMRmap**
> ![CMRmap](../../images/x-array-map-plot/colormaps/CMRmap.png)
>  **Dark2**
> ![Dark2](../../images/x-array-map-plot/colormaps/Dark2.png)
>  **Acton**
> ![Acton](../../images/x-array-map-plot/colormaps/acton.png)
>  **Acton_r**
> ![Acton_r](../../images/x-array-map-plot/colormaps/acton_r.png)
>  **Afmhot**
> ![Afmhot](../../images/x-array-map-plot/colormaps/afmhot.png)
>  **Autumn**
> ![Autumn](../../images/x-array-map-plot/colormaps/autumn.png)
>  **Bam**
> ![BamO](../../images/x-array-map-plot/colormaps/bam.png)
>  **BamO**
> ![BamO](../../images/x-array-map-plot/colormaps/bam0.png)
>  **BamO_r**
> ![bamO_r](../../images/x-array-map-plot/colormaps/bamO_r.png)
>  **Bamako**
> ![Accent](../../images/x-array-map-plot/colormaps/bamako.png)
>  **Bamako_r**
> ![bamako_r](../../images/x-array-map-plot/colormaps/bamako_r.png)
>  **Batlow**
> ![Batlow](../../images/x-array-map-plot/colormaps/batlow.png)
>  **BatlowK**
> ![BatlowK](../../images/x-array-map-plot/colormaps/batlowK.png)
>  **BatlowK_r**
> ![BatlowK_r](../../images/x-array-map-plot/colormaps/batlowK_r.png)
>  **BatlowW**
> ![batlowW](../../images/x-array-map-plot/colormaps/batlowW.png)
>  **BatlowW_r**
> ![batlowW_r](../../images/x-array-map-plot/colormaps/batlowW_r.png)
>  **Batlow_r**
> ![Batlow_r](../../images/x-array-map-plot/colormaps/batlow_r.png)
>  **Berlin**
> ![Berlin](../../images/x-array-map-plot/colormaps/berlin.png)
>  **berlin_r**
> ![Berlin_r](../../images/x-array-map-plot/colormaps/berlin_r.png)
>  **Bilbao**
> ![bilbao](../../images/x-array-map-plot/colormaps/bilbao.png)
>  **Bilbao_r**
> ![Bilbao_r](../../images/x-array-map-plot/colormaps/bilbao_r.png)
>  **Binary**
> ![Binary](../../images/x-array-map-plot/colormaps/binary.png)
>  **Bone**
> ![Bone](../../images/x-array-map-plot/colormaps/bone.png)
>  **Brg**
> ![Brg](../../images/x-array-map-plot/colormaps/brg.png)
>  **Broc**
> ![Broc](../../images/x-array-map-plot/colormaps/broc.png)
>  **BrocO**
> ![BrocO](../../images/x-array-map-plot/colormaps/brocO.png)
>  **BrocO_r**
> ![BrocO_r](../../images/x-array-map-plot/colormaps/brocO_r.png)
>  **Broc_r**
> ![Broc_r](../../images/x-array-map-plot/colormaps/broc_r.png)
>  **Buda**
> ![Buda](../../images/x-array-map-plot/colormaps/buda.png)
>  **Buda_r**
> ![Buda_r](../../images/x-array-map-plot/colormaps/buda_r.png)
>  **Bukavu**
> ![Bukavu](../../images/x-array-map-plot/colormaps/bukavu.png)
>  **Bukavu_r**
> ![Bukavu_r](../../images/x-array-map-plot/colormaps/bukavu_r.png)
>  **Bwr**
> ![Bwr](../../images/x-array-map-plot/colormaps/bwr.png)
>  **Cool**
> ![Cool](../../images/x-array-map-plot/colormaps/cool.png)
>  **Coolwarm**
> ![Coolwarm](../../images/x-array-map-plot/colormaps/coolwarm.png)
>  **Copper**
> ![Copper](../../images/x-array-map-plot/colormaps/copper.png)
>  **Cork**
> ![Cork](../../images/x-array-map-plot/colormaps/cork.png)
>  **CorkO**
> ![CorkO](../../images/x-array-map-plot/colormaps/corkO.png)
>  **CorkO_r**
> ![CorkO_r](../../images/x-array-map-plot/colormaps/corkO_r.png)
>  **Cork_r**
> ![Cork_r](../../images/x-array-map-plot/colormaps/cork_r.png)
>  **Cubehelix**
> ![Cubehelix](../../images/x-array-map-plot/colormaps/cubehelix.png)
>  **Davos**
> ![Davos](../../images/x-array-map-plot/colormaps/davos.png)
>  **GnBu**
> ![GnBu](../../images/x-array-map-plot/colormaps/GnBu.png)
>  **Greens**
> ![Greens](../../images/x-array-map-plot/colormaps/Greens.png)
>  **Greys**
> ![Greys](../../images/x-array-map-plot/colormaps/Greys.png)
>  **Imola**
> ![Imola](../../images/x-array-map-plot/colormaps/Imola.png)
>  **OrRd**
> ![OrRd](../../images/x-array-map-plot/colormaps/OrRd.png)
>  **Oranges**
> ![Oranges](../../images/x-array-map-plot/colormaps/Oranges.png)
>  **Paired**
> ![Paired](../../images/x-array-map-plot/colormaps/Paired.png)
>  **Pastel1**
> ![Pastel1](../../images/x-array-map-plot/colormaps/Pastel1.png)
>  **Pastel2**
> ![Pastel2](../../images/x-array-map-plot/colormaps/Pastel2.png)
>  **Davos_r**
> ![Davos_r](../../images/x-array-map-plot/colormaps/davos_r.png)
>  **Devon_r**
> ![Devon_r](../../images/x-array-map-plot/colormaps/devon_r.png)
>  **Devon**
> ![Devon](../../images/x-array-map-plot/colormaps/devon.png)
>  **Fes**
> ![Fes](../../images/x-array-map-plot/colormaps/fes.png)
>  **Fes_r**
> ![Fes_r](../../images/x-array-map-plot/colormaps/fes_r.png)
>  **Flag**
> ![Flag](../../images/x-array-map-plot/colormaps/flag.png)
>  **Gist_earth**
> ![Gist_earth](../../images/x-array-map-plot/colormaps/gist_earth.png)
>  **Gist_gray**
> ![Gist_gray](../../images/x-array-map-plot/colormaps/gist_gray.png)
>  **Gist_heat**
> ![Gist_heat](../../images/x-array-map-plot/colormaps/gist_heat.png)
>  **Gist_ncar**
> ![Gist_ncar](../../images/x-array-map-plot/colormaps/gist_ncar.png)
>  **Gist_rainbow**
> ![Gist_rainbow](../../images/x-array-map-plot/colormaps/gist_rainbow.png)
>  **Gist_stern**
> ![Gist_stern](../../images/x-array-map-plot/colormaps/gist_stern.png)
>  **Gist_yarg**
> ![Gist_yarg](../../images/x-array-map-plot/colormaps/gist_yarg.png)
>  **Gnuplot**
> ![Gnuplot](../../images/x-array-map-plot/colormaps/gnuplot.png)
>  **Gnuplot2**
> ![Gnuplot2](../../images/x-array-map-plot/colormaps/gnuplot2.png)
>  **Gray**
> ![Gray](../../images/x-array-map-plot/colormaps/gray.png)
>  **GrayC**
> ![GrayC](../../images/x-array-map-plot/colormaps/grayC.png)
>  **GrayC_r**
> ![GrayC_r](../../images/x-array-map-plot/colormaps/grayC_r.png)
>  **Hawaii**
> ![Hawaii](../../images/x-array-map-plot/colormaps/hawaii.png)
>  **Hawaii_r**
> ![Hawaii_r](../../images/x-array-map-plot/colormaps/hawaii_r.png)
>  **Hot**
> ![Hot](../../images/x-array-map-plot/colormaps/hot.png)
>  **Hsv**
> ![Hsv](../../images/x-array-map-plot/colormaps/hsv.png)
>  **Imola_r**
> ![Imola_r](../../images/x-array-map-plot/colormaps/imola_r.png)
>  **Jet**
> ![Jet](../../images/x-array-map-plot/colormaps/jet.png)
>  **Lajolla**
> ![Lajolla](../../images/x-array-map-plot/colormaps/lajolla.png)
>  **Lajolla_r**
> ![Lajolla_r](../../images/x-array-map-plot/colormaps/lajolla_r.png)
>  **Lapaz**
> ![Lapaz](../../images/x-array-map-plot/colormaps/lapaz.png)
>  **Lapaz_r**
> ![Lapaz_r](../../images/x-array-map-plot/colormaps/lapaz_r.png)
>  **Lisbon**
> ![Lisbon](../../images/x-array-map-plot/colormaps/lisbon.png)
>  **Lisbon_r**
> ![Lisbon_r](../../images/x-array-map-plot/colormaps/lisbon_r.png)
>  **Nipy_spectral**
> ![Nipy_spectral](../../images/x-array-map-plot/colormaps/nipy_spectral.png)
>  **Nuuk**
> ![Nuuk](../../images/x-array-map-plot/colormaps/nuuk.png)
>  **Nuuk_r**
> ![Nuuk_r](../../images/x-array-map-plot/colormaps/nuuk_r.png)
>  **Ocean**
> ![Ocean](../../images/x-array-map-plot/colormaps/ocean.png)
>  **Oleron**
> ![Oleron](../../images/x-array-map-plot/colormaps/oleron.png)
>  **Oslo**
> ![Oslo](../../images/x-array-map-plot/colormaps/oslo.png)
>  **Oslo_r**
> ![Oslo_r](../../images/x-array-map-plot/colormaps/oslo_r.png)
>  **Pink**
> ![Pink](../../images/x-array-map-plot/colormaps/pink.png)
>  **PRGn**
> ![PRGn](../../images/x-array-map-plot/colormaps/PRGn.png)
>  **PiYG**
> ![PiYG](../../images/x-array-map-plot/colormaps/PiYG.png)
>  **PuBu**
> ![PuBu](../../images/x-array-map-plot/colormaps/PuBu.png)
>  **PuBuGn**
> ![PuBuGn](../../images/x-array-map-plot/colormaps/PuBuGn.png)
>  **PuOr**
> ![PuOr](../../images/x-array-map-plot/colormaps/PuOr.png)
>  **PuRd**
> ![PuRd](../../images/x-array-map-plot/colormaps/PuRd.png)
>  **Purples**
> ![Purples](../../images/x-array-map-plot/colormaps/Purples.png)
>  **RdBu**
> ![RdBu](../../images/x-array-map-plot/colormaps/RdBu.png)
>  **RdBu_r**
> ![RdBu_r](../../images/x-array-map-plot/colormaps/RdBu_r.png)
>  **RdBu_r**
> ![RdBu_r](../../images/x-array-map-plot/colormaps/RdBu_r.png)
>  **RdGy_r**
> ![RdGy_r](../../images/x-array-map-plot/colormaps/RdGy_r.png)
>  **RdPu**
> ![RdPu](../../images/x-array-map-plot/colormaps/RdPu.png)
> **RdPu_r**
> ![RdPu_r](../../images/x-array-map-plot/colormaps/RdPu_r.png)
> **RdYIBu**
> ![RdYIBu](../../images/x-array-map-plot/colormaps/RdYIBu.png)
> **RdYIGn**
> ![RdYIGn](../../images/x-array-map-plot/colormaps/RdYIGn.png)
> **Reds**
> ![Reds](../../images/x-array-map-plot/colormaps/Reds.png)
> **Set1**
> ![Set1](../../images/x-array-map-plot/colormaps/Set1.png)
> **Set2**
> ![Set2](../../images/x-array-map-plot/colormaps/Set2.png)
> **Set3**
> ![Set3](../../images/x-array-map-plot/colormaps/Set3.png)
> **Spectral**
> ![Spectral](../../images/x-array-map-plot/colormaps/Spectral.png)
> **Wistia**
> ![Wistia](../../images/x-array-map-plot/colormaps/Wistia.png)
> **YIGn**
> ![YIGn](../../images/x-array-map-plot/colormaps/YIGn.png)
> **YIGnBu**
> ![YIGnBu](../../images/x-array-map-plot/colormaps/YIGnBu.png)
> **YIOrBr**
> ![YIOrBr](../../images/x-array-map-plot/colormaps/YIOrBr.png)
> **YIOrRd**
> ![YIOrRd](../../images/x-array-map-plot/colormaps/YIOrRd.png)
> **Prism**
> ![Prism](../../images/x-array-map-plot/colormaps/prism.png)
> **Rainbow**
> ![Rainbow](../../images/x-array-map-plot/colormaps/rainbow.png)
> **Roma**
> ![Roma](../../images/x-array-map-plot/colormaps/roma.png)
> **RomaO**
> ![RomaO](../../images/x-array-map-plot/colormaps/romaO.png)
> **RomaO_r**
> ![RomaO_r](../../images/x-array-map-plot/colormaps/romaO_r.png)
> **Roma_r**
> ![Roma_r](../../images/x-array-map-plot/colormaps/roma_r.png)
> **Seismic**
> ![Seismic](../../images/x-array-map-plot/colormaps/seismic.png)
> **Spring**
> ![Spring](../../images/x-array-map-plot/colormaps/spring.png)
> **Summer**
> ![Summer](../../images/x-array-map-plot/colormaps/summer.png)
> **Tab10**
> ![Tab10](../../images/x-array-map-plot/colormaps/tab10.png)
> **Tab20**
> ![Tab20](../../images/x-array-map-plot/colormaps/tab20.png)
> **Tab20b**
> ![Tab20b](../../images/x-array-map-plot/colormaps/tab20b.png)
> **Tab20c**
> ![Tab20c](../../images/x-array-map-plot/colormaps/tab20c.png)
> **Terrain**
> ![Terrain](../../images/x-array-map-plot/colormaps/terrain.png)
> **Tofino**
> ![Tofino](../../images/x-array-map-plot/colormaps/tofino.png)
> **Tofino_r**
> ![Tofino_r](../../images/x-array-map-plot/colormaps/tofino_r.png)
> **Tokyo**
> ![Tokyo](../../images/x-array-map-plot/colormaps/tokyo.png)
> **Tokyo_r**
> ![Tokyo_r](../../images/x-array-map-plot/colormaps/tokyo_r.png)
> **Turku**
> ![Turku](../../images/x-array-map-plot/colormaps/turku.png)
> **Turku_r**
> ![Turku_r](../../images/x-array-map-plot/colormaps/turku_r.png)
> **Vanimo**
> ![Vanimo](../../images/x-array-map-plot/colormaps/vanimo.png)
> **Vanimo_r**
> ![Vanimo_r](../../images/x-array-map-plot/colormaps/vanimo_r.png)
> **Vik**
> ![Vik](../../images/x-array-map-plot/colormaps/vik.png)
> **VikO**
> ![VikO](../../images/x-array-map-plot/colormaps/vikO.png)
> **VikO_r**
> ![VikO_r](../../images/x-array-map-plot/colormaps/vikO_r.png)
> **Vik_r**
> ![Vik_r](../../images/x-array-map-plot/colormaps/vik_r.png)
> **Winter**
> ![Winter](../../images/x-array-map-plot/colormaps/winter.png)
>
{: .details}

> <question-title></question-title>
>
> 1. What are the different kinds of colormaps that can be used?
>
> > <solution-title></solution-title>
> >
> >When it comes to conveying the correct information, through visualisation, colors play a major role. Suppose you want to display a cold region, its an obvious practice of using cooler tones such as a `blue`. Thus it is important to understand the choices we have.
> >
> >
> >## Plotting different colormaps using **NetCDF xarray map plotting**
 > > > <hands-on-title>Plotting the major colormaps using the xarray tool.</hands-on-title>
> > >
> > > 1. {% tool [NetCDF xarray map plotting](toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.20.2+galaxy0) %} with the following parameters:
> > >    - {% icon param-file %} *"Netcdf file"*: `outfile.netcdf`
> > >    - {% icon param-file %} *"Tabular of variables"*: `Metadata infos from outfile.netcdf` (output of **NetCDF xarray Metadata Info** {% icon tool %})
> > >    - *"Choose the variable to plot"*: `air_temperature_at_2_metres`
> > >    - *"Name of latitude coordinate"*: `lat`
> > >    - *"Name of longitude coordinate"*: `lon`
> > >    - *"Datetime selection"*: `No`
> > >    - *"Range of values for plotting e.g. minimum value and maximum value (minval,maxval) (optional)"*: `220,320`
> > >    - *"Add country borders with alpha value [0-1] (optional)"*: `1.0`
> > >    - *"Add coastline with alpha value [0-1] (optional)"*: `1.0`
> > >    - *"Add ocean with alpha value [0-1] (optional)"*: `1.0`
> > >    - *"Specify plot title (optional)"*: `Air_temperature_at_2_metres___23:00:00_UTC_batlow`
> > >    - *"Specify which colormap to use for plotting (optional)"*: `batlow`
> > >
> > >    - *"Specify the projection (proj4) on which we draw e.g. {"proj":"PlateCarree"} with double quote (optional)"*: `{'proj': 'Mercator'}`
> > >
> > >
> > > The final plot is shown below:
> > >    ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-25 >at 18:00:00](../../images/x-array-map-plot/bat.png)
> > >Some other important color variants of the same map can be found below :
> > > - *colormap : oslo*
> > >![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-25 >at 18:00:00](../../images/x-array-map-plot/os.png)
> > >  - *colormap : bamako*
> > >![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-25 >at 18:00:00](../../images/x-array-map-plot/bama.png)
> > >  - *colormap : vik*
> > >![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-25 >at 18:00:00](../../images/x-array-map-plot/vi.png)
> > > - *colormap : tokyo*
> > >![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-25 >at 18:00:00](../../images/x-array-map-plot/tok.png)
> > > - *colormap : hawaii*
> > >![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-25 >at 18:00:00](../../images/x-array-map-plot/hawa.png)
> > {: .hands_on}
> >
> >
> >
> {: .solution}
>
{: .question}



> <question-title></question-title>
>
> 1. What insights can be driven from the data and how is the data important on a local level?
> 2. What are some giveaways from the tutorial ?
>
> > <solution-title></solution-title>
> >
> > 1. Every piece of data recites a story. The air temperature at a certain height has a lot of significance in major commercial and day to day activities. Read this data-blog on the above analysis. [Click Here](https://quickbeasts51429.github.io/Outreachy_Galaxy_Community_contributor/).
> > 2. The tutorial has summed up a proper way of plotting data from a netcdf file. It has discussed everything from loading of data to its final display. Some other key points to keep in mind are :
> >
> >    1. It may take some time while plotting the maps. It depends on traffic / load on the Galaxy server. It is suggested to have a 64-bit processor with 8GB RAM storage. Be patient.
> >    2.  You can view as well as download the generated plots to use further.
> >    3. Plotting over global maps is very convinient as you saw above. But many a times,  you want to plot a specific region, it becomes very easy using CDO tool. Refer to [this tutorial]({% link topics/climate/tutorials/pangeo/tutorial.md %}) for more info.
> >
> > 3. If you wish to present all the plotted maps at one place for comparision or analysis. It is a short and simple step and can be doe using the tool name `Image Montage`.
> >
> > {% tool [Image Montage](toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1) %} with the following parameters:
> >   - {% icon param-files %} *"Images"*: `Map plots`
> >   - {% icon param-text %} *"# of images wide"*: `4`
> > Here you can see that all the projections we plotted above have been shown in a single image using `Image Montage`.
> > ![ECMWF Reanalysis Air temperature a 2 metres on 2022-05-31 Monatge for 23:00:00 ](../../images/x-array-map-plot/montage.png)
> {: .solution}
>
{: .question}




# Conclusion

We have learnt about the `xarray map plotting tool` dealing with the `netcdf data set`. The tutorial also discussed about the types of climate datasets. One of the tutorial is info about usage of different [colormaps](https://github.com/Quickbeasts51429/Xarray_ColorMaps/blob/main/index.md#color-maps) and [projections](https://github.com/Quickbeasts51429/xarray_projection) in xarray.

tools_json:
["NetCDF xarray Metadata Info", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1", "NetCDF xarray Coordinate Info", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.20.2+galaxy0", "CDO Operations", "toolshed.g2.bx.psu.edu/repos/climate/cdo_operations/cdo_operations/2.0.0+galaxy0", "NetCDF xarray map plotting", "toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.20.2+galaxy0", "Image Montage", "toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1"]


topic: ecology
tutorial: yolo_predict_deepsea
title: Object detection with YOLO
difficulty: intermediate
time_estimation: 45m
requirements_json: []
tags_json: ["object-detection", "deep-learning", "ecology", "biodiversity", "marine"]
content_hash: 11d2fd0b362e426e3cc09c4cb0b6c1c9
last_modified: 2026-05-27T09:01:48.402109
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/yolo_predict_deepsea/tutorial.html

content:
YOLO (You Only Look Once) is a fast, deep learning-based algorithm for real-time object detection. It predicts object classes and bounding boxes in a single pass over the image. YOLOv8 is a specific version, offering improved accuracy and speed.

In this tutorial, you will use Galaxy to run YOLOv8 model for **Object Detection** using real underwater images from the SEANOE dataset.

> <agenda-title></agenda-title>
>
> In this tutotial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Detection of marine species

## Input Dataset

We will use selected images from the SEANOE dataset {% cite lebeaud2024deepsea %}.

The [SEANOE](https://www.seanoe.org/data/00907/101899) collection features real underwater images captured by deep‑sea observatories as part of a citizen science initiative called Deep Sea Spy. These non‑destructive imaging stations continuously monitor marine ecosystems and provide snapshots of various fauna. In this dataset, multiple annotators, including trained scientists and enthusiastic citizen scientists, have manually labeled images with polygons, lines, or points highlighting marine organisms. These annotations were then cleaned and converted into bounding boxes to create a training-ready dataset for object detection with YOLOv8. Though the exact species vary, images often include deep-sea fish, species, making this dataset well-suited for practicing detection tasks.

![sample buccinid data](../../images/yolo/CAM-TEMPO.jpg){: style="width:40%; display:inline-block;"}
![sample bythongraede data](../../images/yolo/MOMAR.jpg){: style="width:40%; display:inline-block;"}
![sample buccinid2 data](../../images/yolo/CAM-TEMPO2.jpg){: style="width:40%; display:inline-block;"}
![sample buccinid3 data](../../images/yolo/CAM-TEMPO3.jpg){: style="width:40%; display:inline-block;"}


### Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. **Create a new history** for this tutorial
>    - Give it a name (example: “DeepSeaSpy Yolo tutorial”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. **Import** image data files and models from [SEANOE marine datawarehouse](https://www.seanoe.org/data/00907/101899/).
>
>    DeepSeaSpy image data files and models as a zip file:
>    ```
>    https://www.seanoe.org/data/00907/101899/data/115473.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} with the following parameters:
>    - *"Input file"*: `115473.zip`
>
>    This will create a data collection in your history where all archive files will be unzipped.
>
> 4. **Unhide** {% icon galaxy-show-hidden %} the models data files:
>    - In the history search bar, enter: `name:detection deleted:false visible:any`
>    - Then unhide {% icon galaxy-show-hidden %} the 2 model files
>      - "dataset_seanoe_101899_YOLOv8-weights-for-Bythograeidae-detection" and
>      - "dataset_seanoe_101899_YOLOv8-weights-for-Buccinidae-detection".
>
>
>    {% snippet faqs/galaxy/datasets_unhidden.md %}
>
> 5. Select a sample of 100 image files and create a dedicated data collection
>    - History search `extension:jpg deleted:false visible:any` then
>    - click on "Select" and "Select All".
>    - click on "All 3979 selected" and "Build Dataset List" (or "Auto build list" depending on the selection mode),
>    - select 100 files and give a name of the data collection, "DeepSeaSpy 100 images sample" for example.
>
>    > <tip-title> Filter history to only show 100 files </tip-title>
>    > To select only last 100 files, you can use the history search function and specify `extension:jpg deleted:false hid>XXXX visible:any` in the search bar where XXXX is the id of the last image dataset minus 100.
>    > For example `extension:jpg deleted:false hid>3885 visible:any` if you have images until the history dataset ID 3985.
>    {: .tip}
>
> 6. **Create class name file** "Buccinide_classnames", copying and pasting this content in the file uploader:    ***Optional step - Only if using under "Galaxy Version 8.3.0+galaxy5" tool version***
>
>    ```
>    Autre poisson
>    Couverture de moules
>    Couverture microbienne
>    Couverture vers tubicole
>    Crabe araignée
>    Crabe bythograeidé
>    Crevette alvinocarididae
>    Escargot buccinidé
>    Ophiure
>    Poisson Cataetyx
>    Poisson chimère
>    Poisson zoarcidé
>    Pycnogonide
>    Ver polynoidé
>    Vers polynoidés
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md %}
>
{: .hands_on}

## Model

This dataset provides two pretrained YOLOv8 detection models tailored for the marine species found in [SEANOE](https://www.seanoe.org/data/00907/101899). One model detects Buccinidae (a family of sea snails), and the other targets Bythograeidae (a family of deep-sea crabs). These models were trained on cleaned annotation sets that contain thousands of examples—for instance, the Buccinidae set includes over 14,900 annotations in total. For this tutorial, you’ll find two model files—`*.pt` files—each accompanied by the appropriate class_names.txt file. You can upload either or both to Galaxy to run detection experiments on your underwater images.

##  Run YOLOv8 in detect mode


> <hands-on-title> Detect Buccinid snails on images </hands-on-title>
>
> 1. {% tool [Perform YOLO image labeling](toolshed.g2.bx.psu.edu/repos/bgruening/yolo_predict/yolo_predict/8.3.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input images"*: `DeepSeaSpy 100 images sample` (click the {% icon param-collection %} button, then select "DeepSeaSpy 100 images sample" from the list)
>    - {% icon param-file %} *"YOLO class name file"*: `Buccinide_classnames` (input plain text file that lists the names of the classes the model can detect)
>    - {% icon param-file %} *"Model file"*: `dataset_seanoe_101899_YOLOv8-weights-for-Buccinidae-detection` (click the "…" button, then search for the name of the file or just filter with ".pt" statement)
>    - *"Prediction mode"*: `detect`
>    - *"Image size"*: `1000`
>    - *"Confidence"*: `0.25`
>    - *"IoU"*: `0.45`
>    - *"Max detections"*: `300`
>
>    > <warning-title> Models type </warning-title>
>    >
>    > The model is trained only for detection, not segmentation.
>    >
>    {: .warning}
>
> > <tip-title>IoU threshold parameter</tip-title>
> >
> >   Try changing the confidence and IoU thresholds (this is used for Non-Maximum Suppression (NMS), which removes overlapping detections) to see how detection results vary. It helps you find a good balance between sensitivity and accuracy.
> >
> >
> {: .tip}
>
> > <comment-title>Additional information on class names file and parameters</comment-title>
> >
> >    Concerning the class names file: Each class name must be on its own line, in the same order used during model training. So the class ID 0 corresponds to Buccinidae, and 1 to Bythograeidae.
> >    Concerning tool parameters:
> >    - *"Image size"*: Use 1000 (or a smaller number like 640 if processing speed is important). This controls how much the image is resized before prediction. Smaller values = faster but possibly less accurate.
> >    - *"Confidence"*: Set to 0.25 (25%). This controls how confident the model must be to report a detection. If you increase this value (e.g., 0.5), you’ll get fewer detections, but they’ll be more confident. If you lower it (e.g., 0.1), you may get more results, but possibly more false positives.
> >    - *"IoU"*: Set to 0.45. This is used for Non-Maximum Suppression (NMS), which removes overlapping detections. A higher IoU value (e.g., 0.7) keeps more overlapping boxes. A lower IoU (e.g., 0.3) removes more overlaps, which may help clean up crowded images.
> >    - *"Max detections"*: Set a reasonable cap like 300. This limits the number of objects detected per image.
> >
> {: .comment}
>
{: .hands_on}

##  Explore the Outputs

After running the tool, Galaxy will give you several output files for each image. Let’s go through what each one means and how to use them:

### Text files (*.txt)

These are plain text files containing the detection results. Each line in a file shows:

```
<class_id> <confidence_score> <x_center> <y_center> <width> <height>
```

For example: ```0 0.82 350 200 100 120```

> <question-title></question-title>
>
> 1. What is the meaning of these lines ?
>
> > <solution-title></solution-title>
> >
> > 1. Class ID 0 (in our case, Buccinidae)
> > 2. Detected with 82% confidence
> > 3. The bounding box is centered at (350, 200) and has a width of 100 and height of 120 (in pixels, relative to the image. You can use this file to do further analysis, like counting species or tracking locations over time
> >
> {: .solution}
>
{: .question}

### Overlay images (*.jpg)

These are your original images with colored boxes drawn around detected species. Each box also includes:
The class name and the confidence score. For example, you might see a box labeled:
`Esxcargot buccinidé 0.40`

![buccinid](../../images/yolo/buccinid.jpeg)

And if you reexecute the Yolo labeling tool specifying the "dataset_seanoe_101899_YOLOv8-weights-for-Bythograeidae-detection" model file, you will obtain detection result for crabs.
![bythog](../../images/yolo/bythog2.jpeg)


These images are useful for visually checking whether detections are correct or if something was missed.

> <comment-title> What's the `?` in the labels ?</comment-title>
> The `?` character you see in the annotation is due to an incompatibility between the class names in the pre-trained model and the character encoding used on the system where the model was originally trained. This typically happens when non-UTF-8 characters are not properly handled during training or export.
{: .comment}

### No masks or segmentation files
Since we used detect mode, this tool will not generate segmentation masks (like .tiff or polygon files). Those are only available in segment mode, which we'll cover in a separate tutorial.


## What to Look For

- Are species detected correctly?
- Any false positives or missed detections?
- What confidence levels do you observe?
- How many objects per image?


# Next Steps

Want to go further?

- Train your own model on SEANOE annotations using {% tool [YOLO training tool in Galaxy](toolshed.g2.bx.psu.edu/repos/bgruening/yolo_training/yolo_training/8.3.0+galaxy2) %}
- Get more information from [YOLOv8 training notebook](https://docs.ultralytics.com)

tools_json:
["Unzip", "toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0", "Perform YOLO image labeling", "toolshed.g2.bx.psu.edu/repos/bgruening/yolo_predict/yolo_predict/8.3.0+galaxy2", "YOLO training tool in Galaxy", "toolshed.g2.bx.psu.edu/repos/bgruening/yolo_training/yolo_training/8.3.0+galaxy2"]


topic: ecology
tutorial: phylodiversity_workflow
title: Phylodiversity analysis quick tutorial
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["ecology", "taxonomic data", "phylodiversity", "biodiversity"]
content_hash: 6f3b162acb6f90fcdaed3743359e6b4e
last_modified: 2026-05-27T09:01:48.401104
zenodo_link: https://zenodo.org/records/15601932
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/phylodiversity_workflow/tutorial.html

content:
This tutorial is designed to guide you through the Phylodiversity Galaxy workflow, demonstrating how to easily compute phylodiversity and create phyloregions from phylogeny, occupency and spatial files.

The tutorial will provide a detailed explanation of inputs, workflow steps, and outputs. This tutorial gives a practical example, highlighting a use case extract from souhtern sea actinos populations.

The primary goal of this workflow is to compute phylodiversity index and identify phyloregions. The project's objective is to offer accessible, reproducible and transparents solutions for analyse phylodiversity.

This workflow is composed of four tools:

- PhylOccuMatcher
- CRSConverter
- PhyloIndex
- EstimEndem

In this tutorial, we estimate your data are correctly formated.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Before starting

This part will present the type of data you need to run the ecoregionalization workflow. This data will be downloaded in the next part of the tutorial.

## phylogenic tree file
The first file needed for this workflow is the phylogenetic tree of your interested species.
In this example it'a a simplified phylogeny of the actinopterigy
This file must be at newick format.

## occupancy file
The second file is an occupancy file, each line is a species, the decimal separator must be "." and the column must be separated with "\t"(={Tabulation}). You need to have a column "grids" containing the cell of the grid you've seen your species and the name of the column with the species names must be "newscientificname".

| grids   | newscientificname   | ...     | ...   | ...   |
| ------- | ------------------- | ------- | ----- | ----- |
| ------- | ------------------- | ------- | ----- | ...   |
| ------- | ------------------- | ------- | ----- | ----- |
| ...     | ...                 | ...     | ...   | ...   |
| ------- | ------------------- | ------- | ----- | ----- |

## Shapefile
The last file is a spatial file in shapefile format. In Galaxy this type of file must be uploaded as a composite file of type shp.
This kind of file must have at least 3 file with the same name and 3 different extension : .shp, .shx end .dbf. you can have more file optionally like the .prj file.

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    For the `tabular` and `newick` datafiles
>    ```
>    https://zenodo.org/records/15601932/files/phylogeny_test
>    https://zenodo.org/records/15601932/files/grid_test.tabular
>    ```
>
>    For the composite `shp` datafile (you here need to download locally each file to upload it from the "Composite" menu of "Upload Files" tool, selecting `shp` datatype)
>    ```
>    https://zenodo.org/records/15601932/files/shapefile.dbf
>    https://zenodo.org/records/15601932/files/shapefile.prj
>    https://zenodo.org/records/15601932/files/shapefile.shx
>    https://zenodo.org/records/15601932/files/shapefile.shp
>    ```
>
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype of the phylogenic file is `newick` (often not automatically detected to this format but `json`),  occupancy file `tabular` and the spatial file a composite dataset of type `shapefile`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="newick" %}
>
> 5. A good pratice is also to add to each datafile a tag corresponding for example to the taxon, here `Actinopterygians` or other relevant information.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Data formatting

The first step is to be sure your data are well formated. If all your file are in good format and do have the needed column as specified before, you can move forward.

An example of occupancy file:

![Example occupancy datafile](../../images/phylodiversity/phylodiversity_format.png "Example occupancy datafile"){: style="width:50%"}

# Phylodiversity Workflow

## Match your phylogeny and occupancy with **PhylOccuMatcher**

> <hands-on-title> run PhylOccuMatcher </hands-on-title>
>
> 1. {% tool [PhylOccuMatcher](toolshed.g2.bx.psu.edu/repos/ecology/phylogenetic_occupancy_matcher/phylogenetic_occupancy_matcher/1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Phylogeny file (Newick format)"*: `phylogeny_test` (Input dataset)
>    - {% icon param-file %} *"Occupancy data (Tabular format)"*: `grid_test.tabular` (Input dataset)
>
>
>    > <comment-title> short description </comment-title>
>    >
>    > This tool is the simpliest, you, normally, don't have anything to change and just have to run it with your file as input.
>    {: .comment}
>
{: .hands_on}

## modifying the projection with **CRSconverter**

> <hands-on-title> run CRSConverter </hands-on-title>
>
> 1. {% tool [CRSconverter](toolshed.g2.bx.psu.edu/repos/ecology/crsconverter/crsconverter/1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"shapefile"*: `composite_dataset` (Input dataset)
>
>
> > <warning-title>Pay attention to output format</warning-title>
> >
> > This tool provide multiple possible outputs formats but only the shapefile format can be used for the workflow. The other output format are graphical representation for the user to visualize. If you want it you can rerun this tool outside of the workflow withe the same input and option.
> >
> {: .warning}
>
> > <warning-title>Pay attention to the tool version</warning-title>
> >
> > For the workflow to work you need to use the CRSConverter 1.1 not the 1.0. So be cautious it's the case because if you use the 1.0 version the workflow will crash during the last step.
> >
> {: .warning}
>
> > <comment-title> short description </comment-title>
> >
> > The main interest of using this tool is to modify the projection of your shapefile. To use it you'll have to select the parameter you need in the advanced option before running this tool.
> {: .comment}
>
{: .hands_on}

## Compute phylodiversity index with **PhyloIndex**

> <hands-on-title> run PhyloIndex </hands-on-title>
>
> 1. {% tool [PhyloIndex](toolshed.g2.bx.psu.edu/repos/ecology/phylo_index/phylo_index/1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Phylogeny file (Newick format)"*: `Phylogeny with occupancy data` (output of **PhylOccuMatcher** {% icon tool %})
>    - {% icon param-file %} *"Occupancy data (Tabular format)"*: `Matched output data` (output of **PhylOccuMatcher** {% icon tool %})
>
>
>    > <comment-title> short description </comment-title>
>    >
>    > This tool compute phylodiversity index, It include some randomness so, for reproducibility, you'll need to select a random seed. Moreover you'll need to select the way of modeling you want by choosing between 3 propositon:
>    > -“tipshuffle”: shuffles tip labels multiple times.
>    > -“rowwise”: shuffles sites (i.e., varying richness) and keeping species occurrence frequency constant.
>    > -"colwise”: shuffles species occurrence frequency and keeping site richness constant.
>    > The default value is the tipshuffle method
>    {: .comment}
>
{: .hands_on}

## Estimate the endemism with **EstimEndem**

> <hands-on-title> run EstimEndem </hands-on-title>
>
> 1. {% tool [EstimEndem](toolshed.g2.bx.psu.edu/repos/ecology/estimate_endem/estimate_endem/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Phylogeny file (Newick format)"*: `Phylogeny with occupancy data` (output of **PhylOccuMatcher** {% icon tool %})
>    - {% icon param-file %} *"Occupancy data (Tabular format)"*: `Matched output data` (output of **PhylOccuMatcher** {% icon tool %})
>    - {% icon param-file %} *"input_shapefile"*: `shapefile` (output of **CRSconverter** {% icon tool %})
>
>
>    > <comment-title> short description </comment-title>
>    >
>    > The output of this tool is a shapefile with the clusterisation done in function of the endemism. You'll have to choose a number of cluster you want and the clustering method you want.
>    {: .comment}
>    > <comment-title> More tips and info </comment-title>
>    >
>    > If you have no idea how many cluster you want, the tool start with an estimation of how many clusters are optimal between 0 to 30. So you can firstly run the tool with default value and go check the standard output to check the recommanded number. However keep in mind that this estimation is purely statistics and don't always have biologic reasons.
>    {: .comment}
>
{: .hands_on}

# Conclusion

Congratulation for successfully completed the Phylodiversity workflow. Here is the end of this quick tutorial. Don't hesitate to contact us if you have any questions or if you have ideas for improvment of this workflow.

tools_json:
["PhylOccuMatcher", "toolshed.g2.bx.psu.edu/repos/ecology/phylogenetic_occupancy_matcher/phylogenetic_occupancy_matcher/1.0+galaxy0", "CRSconverter", "toolshed.g2.bx.psu.edu/repos/ecology/crsconverter/crsconverter/1.1+galaxy0", "PhyloIndex", "toolshed.g2.bx.psu.edu/repos/ecology/phylo_index/phylo_index/1.0+galaxy0", "EstimEndem", "toolshed.g2.bx.psu.edu/repos/ecology/estimate_endem/estimate_endem/0.1.0+galaxy0"]


topic: ecology
tutorial: biodiversity-data-exploration
title: Biodiversity data exploration
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["taxonomic data", "data quality", "biodiversity"]
content_hash: 7ea03fe1924fb0d3d1dbdfb52ff76ccb
last_modified: 2026-05-27T09:01:48.396163
zenodo_link: https://zenodo.org/record/6107457
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/biodiversity-data-exploration/tutorial.html

content:
This tutorial will guide you on the exploration of biodiversity data having taxonomic, spatial and temporal informations.

We'll be using Reef Life Survey (RLS) data extracted from the Australian Ocean Data Network (AODN) portal. We'll use a subset done directly on the AODN data portal (https://portal.aodn.org.au/) on this dataset "IMOS - National Reef Monitoring Network Sub-Facility - Global reef fish abundance and biomass". We decided to use data only on the Mollusca phylum from the east coast of Australia between 2008 and 2021. We'll explore this dataset in the view of making statistical analyses so we will check the homoscedasticity and normality of the variables, see if some variables are correlated, how the data is distributed through space and time, etc ... And finally, we'll explore Beta diversity through the computation of the SCBD and LCBD (Species and Local Contribution to Beta Diversity).

> <details-title>Definitions of SCBD and LCBD</details-title>
>
> Species Contribution to Beta Diversity: degree of variation for individual species across the study area.
>
> Local Contribution to Beta Diversity: comparative indicators of the ecological uniqueness of the sites.
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data preparation

First step is to download biodiversity data on your Galaxy history. Here we will use a "classical" (containing taxonomic, spatial and temporal informations) biodiversity dataset from the well known ["Reef life survey" initiative](https://reeflifesurvey.com/).

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (example: "RLS for biodiversity data exploration tutorial")
>    for you to find it again later if needed.
> 2. Import the files from [Zenodo](https://zenodo.org/record/6107457/files/IMOS-National_Reef_Monitoring_Network_Sub-Facility-Global_reef_fish_abundance_and_biomass.csv?download=1)
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets "reef_life_molluscs" for example and preview your dataset
>
>    You can see that the dataset hasn't been detected to be a CSV dataframe, it is because RLS data directly puts the
>    metadata of the dataframe in the first lines before the dataframe so you'll have to remove these lines using the {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>        - {% icon param-text %} *"Remove first"*: `72`
>        - {% icon param-file %} *"from"*: reef_life_molluscs data file
>
>    Then, verify if your new file hasn't got hashtags in the first lines and then ask Galaxy to autodetect datatype (click on the pencil, then "Datatypes" then click on "Auto-detect" button). Galaxy will normally detect it as csv.
>
> 4. Convert datatype CSV to tabular
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="Convert CSV to tabular" %}
>
{: .hands_on}

## Customize your dataset

In order to clean unnecessary informations from the table we will now cut a few columns and change the format of time information.

> <hands-on-title>Clean your data</hands-on-title>
>
> 1. Use {% tool [Advanced cut columns from a table (cut)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %}
>    with following parameters :
>    - {% icon param-files %} *"File to cut"*: Convert CSV to tabular data file
>    - {% icon param-select %} *"Operation"*: `Keep`
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-select %} *"Cut by"*: `fields`
>      - {% icon param-select %} *"List of Fields"*: `Column: 8` `Column: 10` `Column: 11` `Column: 12` `Column: 25` `Column: 28`
>
> 2. Use {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.0) %} with following parameters:
>    - {% icon param-files %} *"Select cells from"*: Advanced Cut data file
>    - {% icon param-select %} *"using column"*: `Column: 4`
>    - {% icon param-repeat %} Click *"+ Insert Check"*:
>         - {% icon param-text %} *"Find Regex"*: `([0-9]{4})-[0-9]{2}-[0-9]{2}`
>         - {% icon param-text %} *"Replacement"*: `\1`
>
>
{: .hands_on}

# Data checking

## Homoscedasticity and normality analysis

> <hands-on-title>Here we will check homogeneity of variances (Levene test) for every species and represent it through multiple boxplots and the normal distribution (Kolmogorov-Smirnov test) represented by a distribution histogram and a Q-Q plot.</hands-on-title>
>
> 1. {% tool [Homoscedasticity and normality](toolshed.g2.bx.psu.edu/repos/ecology/ecology_homogeneity_normality/ecology_homogeneity_normality/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: Column Regex Find and Replace data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Select column containing temporal data (year, date, ...)"*: `c4`
>    - {% icon param-select %} *"Select column containing species"*: `c5`
>    - {% icon param-select %} *"Select column containing numerical values (like abundances)"*: `c6`
>
>    You have to get three outputs: the Levene Test for homoscedasticity dataset, the Kolmogrov-Smirnov test for normality  and 9 PNG files in a data collection representing the homogeneity of variances for each species at each time point of the study.
>    If the levene test is significant (P-value in column Pr < 0.5 and at least one * at the end of the 4th line), variances aren't homogeneous, the hypothesis of homoscedasticity is rejected.
>    If the K-S test is significant (p-value < 0.5), your numerical variable isn't normally distributed, the hypothesis of normality is rejected.
>    The two tests have to be significant so variances aren't homogenous and data isn't normally distributed.
>
{: .hands_on}

![Homoscedasticity and normality_example on Sepioteuthis australis](../../images/BiodivExplo/Homoscedasticity_and_normality_of_%20Sepioteuthis%20australis.png)

## Autocorrelation in your data

> <hands-on-title>Autocorrelation</hands-on-title>
>
> 1. {% tool [Variables exploration](toolshed.g2.bx.psu.edu/repos/ecology/ecology_link_between_var/ecology_link_between_var/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: Column Regex Find and Replace data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Variables links exploration"*: `Autocorrelation of one selected numerical variable`
>        - {% icon param-select %} *"Select columns containing numerical values"*: `c6`
>
>     You have to get two outputs, one text file containing the Autocorrelation function values and one PNG file in the data collection showing the autocorrelation for a variable.
>     If the bars of the histogram are strictly confined between the dashed lines (representing 95% confidence interval without white noise), there is auto-correlation.
>
>    Here, we don't see there is autocorrelation.
>
{: .hands_on}

![Variable_exploration_autocorrelation example](../../images/BiodivExplo/Variables_exploration_autocorrelation.png)

## Check collinearity in your data

> <hands-on-title>Collinearity between numerical variables</hands-on-title>
>
> 1. {% tool [Variables exploration](toolshed.g2.bx.psu.edu/repos/ecology/ecology_link_between_var/ecology_link_between_var/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: formatted biodiversity data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Variables links exploration"*: `Collinearity between selected numerical variables for each species`
>        - {% icon param-select %} *"Select column containing species"*: `c5`
>        - {% icon param-select %} *"Select columns containing numerical values"*: `c['4', '6']`
>
>    You have to get two outputs, one describing species we couldn't evaluate and one PNG file with one plot containing multiple correlation plots and the correlation values between each variables.
>
{: .hands_on}

![Variable_exploration_collinarity example](../../images/BiodivExplo/Variables_exploration_collinarity_of_Sepioteuthis%20australis.png)

# Data exploration
## Visualize abundance repartition through space

> <hands-on-title>Abundance map in the environment</hands-on-title>
>
> 1. {% tool [Presence-absence and abundance](toolshed.g2.bx.psu.edu/repos/ecology/ecology_presence_abs_abund/ecology_presence_abs_abund/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: formatted biodiversity data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Variables presence, absence and abundance"*: `Abundance map in the environment `
>        - {% icon param-select %} *"Select column containing latitudes "*: `c2`
>        - {% icon param-select %} *"Select column containing longitudes"*: `c3`
>        - {% icon param-text %} *"What do you study in this analysis ?"*: `Molluscs of Australian east coast`
>        - {% icon param-select %} *"Select column containing taxon "*: `c5`
>    - {% icon param-select %} *"Select column containing abundances "*: `c6`
>
>    You have to get two outputs, one with the map of the abundance through space with the coordinates and one text file to inform you about the geographical extent of your map.
>
{: .hands_on}

![Presence-absence-example](../../images/BiodivExplo/Presence-absence_and_abundance_mappy.png)

## Visualize the number of locations where each taxons are present

> <hands-on-title>Presence count of taxons (barplot)</hands-on-title>
>
> 1. {% tool [Presence-absence and abundance](toolshed.g2.bx.psu.edu/repos/ecology/ecology_presence_abs_abund/ecology_presence_abs_abund/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: formatted biodiversity data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Variables presence, absence and abundance"*: `Presence count of taxons (barplot)`
>        - {% icon param-select %} *"Select column containing your separation variable"*: `c1`
>        - {% icon param-select %} *"Select column containing taxon"*: `c5`
>    - {% icon param-select %} *"Select column containing abundances "*: `c6`
>
>    You have to get two outputs, one with 120 PNG files (one for each site) representing the number of locations where each taxons are present and one text file to inform you about the used locations.
>
{: .hands_on}


## Visualize the rarefaction curves of your species

> <hands-on-title>Rarefaction curve of species</hands-on-title>
>
> 1. {% tool [Presence-absence and abundance](toolshed.g2.bx.psu.edu/repos/ecology/ecology_presence_abs_abund/ecology_presence_abs_abund/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: formatted biodiversity data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Variables presence, absence and abundance"*: `Rarefaction curve of species`
>        - {% icon param-text %} *"Size of subsamples"*: `200`
>        - {% icon param-select %} *"Select column containing species"*: `c5`
>    - {% icon param-select %} *"Select column containing abundances "*: `c6`
>
>    You have to get two outputs, one data collection with one PNG files representing the rarefaction curves of each species in one graph and one tabular file with log informations.
>
{: .hands_on}


![Variable_exploration_rarefaction curves](../../images/BiodivExplo/Presence-absence_and_abundance_rarefaction.png)


## Visualize the dispersion of a numeric variable and the correlation between species absence

> <hands-on-title>Boxplot of dispersion and correlation of absence</hands-on-title>
>
> 1. {% tool [Statistics on presence-absence](toolshed.g2.bx.psu.edu/repos/ecology/ecology_stat_presence_abs/ecology_stat_presence_abs/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: formatted biodiversity data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Select a column containing numerical values (such as the abundance) "*: `c6`
>    - {% icon param-select %} *"Select the column of the x-axis : most commonly species"*: `c5`
>    - {% icon param-select %} *"Select column containing locations "*: `c1`
>    - {% icon param-select %} *"Select column containing temporal data (year, date, ...) "*: `c4`
>
>    You have to get two outputs, one PNG file with the boxplot and dispersion plot of the abundance and one plot representing wether the absence of several species is correlated. In the second plot if you see there is a cross on the round representation the two related species haven't got their absences correlated, the other are correlated and seem to be co-absent.
>
{: .hands_on}


![Boxplot dispersion_example](../../images/BiodivExplo/Boxplot_and_dispersion_plot.png)
![Absence correlation_example](../../images/BiodivExplo/Absence-correlation_plot.png)

# Beta diversity
## Local and Species Contribution to Beta Diversity (SCBD and LCBD)

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Local Contributions to Beta Diversity (LCBD)](toolshed.g2.bx.psu.edu/repos/ecology/ecology_beta_diversity/ecology_beta_diversity/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: formatted biodiversity data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Select column with abundances"*: `c6`
>    - {% icon param-select %} *"Select column with locations"*: `c1`
>    - {% icon param-select %} *"Select column containing taxon"*: `c5`
>    - {% icon param-select %} *"Select column containing dates"*: `c4`
>    - {% icon param-select %} *"Other LCBD : spatialized representation or xy plot."*: `Spatialized representation`
>        - {% icon param-select %} *"Select column containing latitudes in decimal degrees"*: `c2`
>        - {% icon param-select %} *"Select column containing longitudes in decimal degrees"*: `c3`
>
>    You have to get three outputs. Two text file containing a table with information on the beta diversity and one text file with the list of species that has a SCBD larger than the mean SCBD. One data collection with PNG files showing multiple plots according to one type of variable in order to vizualize the beta diversity.
>
>
{: .hands_on}


![Variable_exploration_example](../../images/BiodivExplo/Local_Contributions_to_Beta_Diversity_LCBD_Beta_diversity_through_space.png)
![Variable_exploration_example](../../images/BiodivExplo/Local_Contributions_to_Beta_Diversity_LCBD_LCBD_sites_time.png)
![Variable_exploration_example](../../images/BiodivExplo/Local_Contributions_to_Beta_Diversity_LCBD_Mean_LCBD_through_time.png)
![Variable_exploration_example](../../images/BiodivExplo/Local_Contributions_to_Beta_Diversity_LCBD_SCBD_Species_Radar_plot.png)


# Conclusion


Here, you just explored your biodiversity dataframe properly and you know a lot more about your data. You can now peacefully make your statiscal analyses as most of the red flags you can get have been revealed by this toolsuite ! Enjoy !

# Bonus: Want to spatially anoymize your data?

A step of this tutorial could be to show you how you can simply apply spatial coordinates anonymization if you want to share data without spatial context, particularly of interest if you want to share endangered species oriented data.

## Bonus! Spatial coordinates anonymization

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Spatial coordinates anonymization](toolshed.g2.bx.psu.edu/repos/ecology/tool_anonymization/tool_anonymization/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input table"*: Column Regex Find and Replace data file
>    - {% icon param-select %} *"First line is a header line"*: `Yes`
>    - {% icon param-select %} *"Select column containing latitudes in decimal degrees"*: `c2`
>    - {% icon param-select %} *"Select column containing longitudes in decimal degrees"*: `c3`
>
{: .hands_on}

tools_json:
["Remove beginning", "Remove beginning1", "Advanced cut columns from a table (cut)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.0", "Homoscedasticity and normality", "toolshed.g2.bx.psu.edu/repos/ecology/ecology_homogeneity_normality/ecology_homogeneity_normality/0.0.0", "Variables exploration", "toolshed.g2.bx.psu.edu/repos/ecology/ecology_link_between_var/ecology_link_between_var/0.0.0", "Presence-absence and abundance", "toolshed.g2.bx.psu.edu/repos/ecology/ecology_presence_abs_abund/ecology_presence_abs_abund/0.0.0", "Statistics on presence-absence", "toolshed.g2.bx.psu.edu/repos/ecology/ecology_stat_presence_abs/ecology_stat_presence_abs/0.0.0", "Local Contributions to Beta Diversity (LCBD)", "toolshed.g2.bx.psu.edu/repos/ecology/ecology_beta_diversity/ecology_beta_diversity/0.0.0", "Spatial coordinates anonymization", "toolshed.g2.bx.psu.edu/repos/ecology/tool_anonymization/tool_anonymization/0.0.0"]


