# tutorials

topic: ecology
tutorial: Ecoregionalization_tutorial
title: Ecoregionalization workflow tutorial
difficulty: intermediate
time_estimation: 2H
requirements_json: []
content_hash: 6e9b131ea7dedee1bb34a7caa9232fb0
last_modified: 2026-05-27T09:01:48.380561
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/Ecoregionalization_tutorial/tutorial.html

content:
This tutorial is designed to guide you through the Ecoregionalization Galaxy workflow, demonstrating how to create an ecoregionalization map from occurrences and environmental data using a boosted regression trees model for predictions.

The workflow, consisting of six tools, is intended for processing occurrence data, which should include latitude, longitude and species presence or absence. The tutorial will provide a detailed explanation of inputs, workflow steps, and outputs. This tutorial gives a practical example, highlighting a use case centered on the Dumont d'Urville sea region and benthic invertebrates.

The primary goal of this workflow is to generate species distribution maps and identify ecoregions within the study area. The project's objective is to offer accessible, reproducible, and transparent IT solutions for processing and analyzing species occurrence data.

This workflow is therefore composed of 6 tools:

- GeoNearestNeighbor
- BRT prediction tool
- TaxaSeeker
- ClusterEstimate
- ClaraClust
- EcoMap

And in this tutorial we will be using 5 more tools to format data before running the ecoregionalization workflow:

- Advanced cut
- Column Regex Find And Replace
- Filter Tabular
- Merge Columns
- Interactive JupyterLab and notebook

Let's delve into the tutorial, outlining each step and tool to manage the creation of ecoregionalization maps.

> <details-title>Some definitions to start</details-title>
>
> Ecoregionalization: The process by which a territory is classified into a category of area that respond to the same environmental factors taking into account species information.
>
> Occurrences data: Data showing the presence of a species at a particular location.
>
> Environmental data: Environmental data are any measurement or information that describe environmental processes, location, or conditions.
>
> Boosted regression trees (BRT): Boosted Regression Trees is a kind of regression methodology based on Machine Learning. Unlike conventional regression methods (GLMs, GAMs), BRTs combine numerous basic decision trees to enhance the predictive performance. BRTs can handle complex relationships and interactions among predictors, and it is considered a robust technique that can control outliers and nonlinearity.
>
> Clustering: Clustering is a machine learning method of grouping data points by similarity or distance.
>
> CLARA/PAM : CLARA (Clustering Large Applications), is an extension to k-medoids (PAM) methods to deal with data containing a large number of objects. PAM stands for "Partition Around Medoids", the PAM algorithm searches for k representative objects in a data set (k medoids) and then assigns each object to the closest medoid in order to create clusters.
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Before starting

This part will present the type of data you need to run the ecoregionalization workflow. This data will be downloaded in the next part of the tutorial.

## Environmental data

To run this workflow, you will first need environmental data. This workflow accepts several types of environmental parameters like temperature or soil type. However, there is a mandatory file format, the tabular format (.tsv), and each pixel of the study area must be described in this file by associating latitude and longitude with environmental parameters.

In the use case presented in this tutorial, seventeen abiotic and physical parameters of the Dumont D'Urville sea region are used. They were taken from oceanographic models and in situ measurements {% cite Hemery2011 %}. The physical oceanographic parameters are mean temperature and its standard deviation, mean salinity and its standard deviation, mean current magnitude and its standard deviation, maximum current bearing, maximum current magnitude and sea ice production. Bathymetric parameters are depth, slope, and rugosity. Finally, the seabed substrate composition was characterized by percentages of biogenic carbonate, biogenic silica, gravel, sand, and mud.

Here an example of environmental file input:


|------|------|---------|------|------|
| long | lat  |  Carbo  | Grav | ...  |
|------|------|---------|------|------|
|139.22|-65.57|   0.88  |28.59 | ...  |
|------|------|---------|------|------|
|139.25|-65.63|   0.88  |28.61 | ...  |
|------|------|---------|------|------|
| ...  | ...  |   ...   | ...  | ...  |
|------|------|---------|------|------|

## Occurrence data

The second data file you will need to run this workflow is an occurrences data file. As defined above, occurrences data are showing the presence (1) or absence (0) of a species at a particular location. This data file also needs to be in tabular format (.tsv) and need to be construct as following:

- latitude and longitude columns.

- One column per taxon where each box corresponding to a geographical point is marked 1 if the taxon is present or 0 if the taxon is absent.

Here an example of occurrences data file input:

|----------|-----------|------------------------|-----------|-----|
|   lat    |   long    |Acanthorhabdus_fragilis | Acarnidae | ... |
|----------|-----------|------------------------|-----------|-----|
|-65,9     |142,3      |1                       |0          | ... |
|----------|-----------|------------------------|-----------|-----|
|-66,3     |141,3      |0                       |1          | ... |
|----------|-----------|------------------------|-----------|-----|
|   ...    |   ...     |...                     |    ...    | ... |
|----------|-----------|------------------------|-----------|-----|

For this tutorial, occurrences data from the Dumont d'Urville sea region will be downloaded from the GBIF. These data were collected as part of the CEAMARC program (The Collaborative East Antarctic Marine Census {% cite Hosie_2011 %}) between December 2007 and January 2008 {% cite bea09 %}. Prior to its inclusion in GBIF, these data originated from collections at the Muséum national d’Histoire naturelle (MNHN – Paris). A GBIF filter was used to download only the data of interest, namely the data from the CEAMARC expedition from the Aurora Australis icebreaker. The selected occurrences are invertebrates. In the GBIF query, five collections were selected: the cnidarians collection (IK), the echinoderm collection (IE), the crustaceans collection (IU), the molluscs collection (IM), and the tunicates collection (IT), and only occurrences recorded by “IPEV-AAD-MNHN" which correspond to the CEAMARC expedition.

## Jupyter notebook for the interactive JupyerLab and notebook tool

To switch from the occurrence data download from GBIF to the occurrence data supported by the ecoregionalisation workflow, the final step of data preparation use the "Interactive JupyterLab and notebook" tool who needs a jupyter notebook to work. In this Jupyter notebook, we used the pivot_wider function of the tidyr R package to transform our data into a wider format and adapt it to subsequent analyses as part of the Galaxy workflow for ecoregionalization. This transformation allowed us to convert our data to a format where each taxon becomes a separate column. We also took care to fill in the missing values with zeros and to sum the individual counts in case of duplications. Then all data >= 1 are replaced by 1 to have only presence (1) or abscence (0) data.

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (example: “Ecoregionalization workflow”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import occurrence data file from [GBIF](https://www.gbif.org/occurrence/download/0030809-240506114902167), environment file from [InDoRES](https://data.indores.fr/file.xhtml?persistentId=doi:10.48579/PRO/LQMHX9/CYSKQG&version=1.0#) and Jupyter notebook file from [InDoRES](https://data.indores.fr/file.xhtml?persistentId=doi:10.48579/PRO/LQMHX9/GIEAQN&version=1.0#)
>
>    Occurrence data file as a zip file where you will find "occurrence.txt"
>    ```
>    https://api.gbif.org/v1/occurrence/download/request/0030809-240506114902167.zip
>    ```
>    Environemental data : "ceamarc_env.tab"
>    ```
>    https://data.indores.fr/api/access/datafile/9777
>    ```
>    Jupyter notebook : "pivot_wider_jupytool_notebook.ipynb"
>    ```
>    https://data.indores.fr/api/access/datafile/9756
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_remote_files.md %}
>
> 3. The GBIF link allows you to download a zip file containing multiple information files about the dataset. The file that you need is the 'occurrence.txt' file that we will specifically extract on Galaxy in the following step.
>
> 4. Use  {% tool [Unzip a file](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} to create a data collection in your history where all GBIF archive files will be unzipped
>
> 5. Unhide the "occurence.txt" data file then modify datatype to select the "tabular" one
>
>    {% snippet faqs/galaxy/datasets_unhidden.md %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 6. Rename the datasets if needed, notably "9756" by "pivot_wider_jupytool_notebook.ipynb" and "9777" by "ceamarc_env.tab".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 7. Check that the datatype of the environment file and occurrence file are tabular.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

# Data formatting

The first step of this tutorial is data formatting because the GBIF species occurrence file download needs to be in a specific format to be included inside the Ecoregionalization workflow.

## Keep data columns that we need with **Advanced Cut**

> <hands-on-title> Select the columns we need </hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `occurrence` (Input dataset)
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>    - *"Delimited by"*: `Tab`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `c['27', '98', '99', '158', '159', '160', '162', '166', '170']`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What are the kept columns ?
>
> > <solution-title></solution-title>
> >
> > 1. The columns we kept are : indivdualCount, decimalLatitude, decimalLongitude, phylum, class, order, family, genus, specificEpithet.
> >
> {: .solution}
>
{: .question}

## Replace empty space with NAs with **Column Regex Find And Replace**

This step is used to add NAs to replace empty space in the file. We will do that for the four first columns and then remove lines with NA to have a complete dataset.

> <hands-on-title> Replace empty space by NAs for the first column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `output` (output of **Advanced Cut** {% icon tool %})
>    - *"using column"*: `c1`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^$`
>            - *"Replacement"*: `NA`
>
> > <warning-title>Pay attention to the regex parameter</warning-title>
> >
> > The regular expression ("^$") is used to capture empty cells in the dataframe so that they can be replaced by NA (Not Available). It is not advised to modify this regular expression nor its replacement, as you will encounter problems while running the next parts of the workflow.
> >
> {: .warning}
>
{: .hands_on}

> <hands-on-title> Replace empty space by NAs for the second column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file1` (output of **Column Regex Find And Replace on the first column** {% icon tool %})
>    - *"using column"*: `c2`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^$`
>            - *"Replacement"*: `NA`
>
{: .hands_on}

> <hands-on-title> Replace empty space by NAs for the third column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file2` (output of **Column Regex Find And Replace on the second column** {% icon tool %})
>    - *"using column"*: `c3`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^$`
>            - *"Replacement"*: `NA`
>
{: .hands_on}

> <hands-on-title> Replace empty space by NAs for the fourth column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file3` (output of **Column Regex Find And Replace on the third column** {% icon tool %})
>    - *"using column"*: `c4`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^$`
>            - *"Replacement"*: `NA`
> 2. Check your output. At the end of this step you must have NA instead of empty space. See example below.
>
>![Output of Regex find and replace example](./Images/NA_example.png "Output of Regex find and replace example")
>
{: .hands_on}

## Remove lines with NAs with **Filter Tabular**

> <hands-on-title> Remove lines with NAs </hands-on-title>
>
> 1. {% tool [Filter Tabular](toolshed.g2.bx.psu.edu/repos/iuc/filter_tabular/filter_tabular/3.3.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular Dataset to filter"*: `out_file4` (output of **Column Regex Find And Replace on the fourth column** {% icon tool %})
>    - In *"Filter Tabular Input Lines"*:
>        - {% icon param-repeat %} *"Insert Filter Tabular Input Lines"*
>            - *"Filter By"*: `by regex expression matching`
>                - *"regex pattern"*: `NA`
>                - *"action for regex match"*: `exclude line if pattern found`
>
> 2. Check your output. All the lines with NAs must have been deleted.
>
{: .hands_on}

### Prepare column merging with **Column Regex Find And Replace**

This manipulation is made in order to merge properly columns “phylum”, “class”, "order", "family", "genus" and "specificEpithet" with a separation.

> <hands-on-title> Add "_" at the end of the phylum column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `output` (output of **Filter Tabular** {% icon tool %})
>    - *"using column"*: `c4`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `(.$)`
>            - *"Replacement"*: `\1_`
> 2. Check your output. All phylum must have "_" at the end.
>
{: .hands_on}

> <hands-on-title> Add "_" at the end of the class column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file1` (output of **Column Regex Find And Replace on phylum column** {% icon tool %})
>    - *"using column"*: `c5`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `(.$)`
>            - *"Replacement"*: `\1_`
> 2. Check your output. All class must have "_" at the end.
>
{: .hands_on}

> <hands-on-title> Add "_" at the end of the order column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file2` (output of **Column Regex Find And Replace on class column** {% icon tool %})
>    - *"using column"*: `c6`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `(.$)`
>            - *"Replacement"*: `\1_`
> 2. Check your output. All order must have "_" at the end.
>
{: .hands_on}

> <hands-on-title> Add "_" at the end of the family column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file3` (output of **Column Regex Find And Replace on order column** {% icon tool %})
>    - *"using column"*: `c7`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `(.$)`
>            - *"Replacement"*: `\1_`
> 2. Check your output. All family must have "_" at the end.
>
{: .hands_on}

> <hands-on-title> Add "_" at the end of the genus column </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file4` (output of **Column Regex Find And Replace on family column** {% icon tool %})
>    - *"using column"*: `c8`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `(.$)`
>            - *"Replacement"*: `\1_`
> 2. Check your output. All genus must have "_" at the end. See example below.
>
> ![Output of Regex Find And Replace example](./Images/2_example.png "Output of Regex Find And Replace example")
>
{: .hands_on}

## Merge taxa columns with **Merge Columns**

> <hands-on-title> Merge columns “phylum”, “class”, "order", "family", "genus" and "specificEpithet" </hands-on-title>
>
> 1. {% tool [Merge Columns](toolshed.g2.bx.psu.edu/repos/devteam/merge_cols/mergeCols1/1.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Column Regex Find And Replace on genus column** {% icon tool %})
>    - *"Merge column"*: `c4`
>    - *"with column"*: `c5`
>    - In *"Columns"*:
>        - {% icon param-repeat %} *"Insert Columns"*
>            - *"Add column"*: `c6`
>        - {% icon param-repeat %} *"Insert Columns"*
>            - *"Add column"*: `c7`
>        - {% icon param-repeat %} *"Insert Columns"*
>            - *"Add column"*: `c8`
>        - {% icon param-repeat %} *"Insert Columns"*
>            - *"Add column"*: `c9`
> 2. Check your output. Your table must have a new column filled with the concatenation of “phylum”, “class”, "order", "family", "genus" and "specificEpithet" columns.
>
{: .hands_on}

## Remove columns no longer needed with **Advanced Cut**

> <hands-on-title> Remove columns that we don't need anymore </hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `out_file` (output of **Merge Columns** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>    - *"Delimited by"*: `Tab`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `c['4', '5', '6', '7', '8', '9']`
> 2. Check your output. Columns “phylum”, “class”, "order", "family", "genus" and "specificEpithet" must have been deleted and your table must have four columns : "individualCount", "decimalLatitude", "DecimalLongitude" and "phylum_class_order_family_genus_specificEpithet". See example below.
>
> ![Output of Advanced cut example](./Images/advanced_out_example.png "Output of Advanced cut example")
>
{: .hands_on}

## Obtain final data file with **Interactive JupyterLab Notebook**

The interactive JupyterLab Notebook allows to create, run, and share custom Galaxy tools based upon Jupyter Notebooks. Galaxy offers you to use Jupyter Notebooks directly in Galaxy accessing and interacting with Galaxy datasets as you like.
Here you will load an existing jupyter notebook ("pivot_wider_jupytool_notebook.ipynb" that you downloaded earlier) and run the code in it to get the final file that is needed for ecoregionalization workflow.

> <hands-on-title> Pivot_wider with Jupytool </hands-on-title>
>
> 1. {% tool [Interactive JupyterLab and notebook](interactive_tool_jupyter_notebook) %} with the following parameters:
>    - *"Do you already have a notebook?"*: `Load a previous notebook`
>        - {% icon param-file %} *"IPython Notebook"*: `pivot_wider_jupytool_notebook.ipynb` (Input dataset)
>        - *"Execute notebook and return a new one."*: `Yes`
>    - In *"User inputs"*:
>        - {% icon param-repeat %} *"Insert User inputs"*
>            - *"Name for parameter"*: `Input file`
>            - *"Choose the input type"*: `Dataset`
>                - {% icon param-file %} *"Select value"*: `output` (output of previous step **Advanced Cut** {% icon tool %})
>
> > <warning-title>Pay attention to the tool version</warning-title>
> >
> > It is possible to have specific troubleshoots with using interactive tools as, for now, we can't redirect to a specific version of the tool (as for classical tool) from the tutorial page. Here, if you encounter errors, you maybe need to specify the 1.0.0 version (the one used for the tuto) !
> >
> {: .warning}
>
> > <tip-title>How to launch the notebook</tip-title>
> >
> >   - The first parameter allows you to specify that you want to use a previous notebook ("pivot_wider_jupytool_notebook.ipynb") rather than create a new one.
> >   - Then you need to insert your ready to use jupyter notebook ("pivot_wider_jupytool_notebook.ipynb").
> >   - The third parameter is important to specify that you just want to execute a notebook and not dive into the web frontend (in this case).
> >   - Then you need to clik on "+ Insert User inputs"" button to add the next parameters.
> >   - After, you need to give a name to your input and choose its type.
> >   - Finally you have to input the occurrence data file that will be treated in the Jupyter Notebook.
> >
> > ![Interactive JupyterLab Notebook parameters example](./Images/JLN_param_example.png "Interactive JupyterLab Notebook parameters example")
> >
>{: .tip}
>
> 2. Check your output ("Intercative JupyterLab Notebook on data (pivot_file)") in your galaxy history. You must have a tabular file with latitude and longitude in first two columns and the others columns must be taxa. See example below.
>
> ![Interactive JupyterLab Notebook output example](./Images/pivot_file_example.png "Interactive JupyterLab Notebook output example")
>
> >  <warning-title>Galaxy can't display properly file with too many columns</warning-title>
> >
> > Too avoid having web interface issues, Galaxy is not displaying tabulated files with "beautifull" columns visualization when number of clumns is big. That's why here we are showing the preview of the dataset from the history.
> >
> {: .warning}
> 
{: .hands_on}

> <question-title></question-title>
>
> 1. How many columns have your data file now?
>
> > <solution-title></solution-title>
> >
> > 1. It must have 202 columns.
> >
> {: .solution}
>
{: .question}

# Ecoregionalization workflow

Now you have all you need to run the ecoregionalization Workflow :
 - Occurrence formatted file
 - Environment file

## Merge environment and occurence file with **GeoNearestNeighbor**

### What it does ?

This Galaxy tool allows you to merge two data tables (tabular format only) according to their latitude and longitude coordinates (in WGS84 projection), finding the closest points. This tool is used in the Ecoregionalization workflow to filter the occurence data file so it can be used on the BRT prediction tool.

### How to use it ?

> <hands-on-title> Run GeoNearestNeighbor </hands-on-title>
>
> 1. {% tool [GeoNearestNeighbor](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_geonearestneighbor/ecoregion_GeoNearestNeighbor/0.1.0+galaxy0) %} with the following parameters:
>    - In *"Your environment file (or table 1)"*:
>        - {% icon param-file %} *"Input your environment data file (tabular format only)"*: `ceamarc_env.tsv` (Input dataset)
>        - *"Choose columns where your latitude is in your environment data file."*: `c2`
>        - *"Choose columns where your longitude is in your environment data file."*: `c1`
>    - In *"Your occurrence file (or table 2)"*:
>        - {% icon param-file %} *"Input your occurrence data file (tabular format only)"*: `pivot_file` (output of **Interactive JupyTool and notebook** {% icon tool %})
>        - *"Choose columns where your latitude is in your occurrence data file."*: `c1`
>        - *"Choose columns where your longitude is in your occurrence data file."*: `c2`
>
>    > <comment-title> Coords precision </comment-title>
>    >
>    > It is recommended that, for optimal precision, the latitude and longitude values in both files should be of the same precision level. And, for the sake of relevance, the geographical coordinates in both files should be as close as possible to apply the most accurate environmental parameters to the correct species occurrences
>    >
>    {: .comment}
>
> 2. Check your outputs. You must have two files:
>       - Information file containing the coordinates of occurrence data, the coordinates retains from environmental data and the distances between the two.
>       - Occurrence and Environment merge file containing occurrence data and environmental data cooresponding.
>
{: .hands_on}

## Predicting taxa distribution with **BRT tool prediction**

This step implements a commonly used approach in ecological studies, namely species distribution modelling (SDM). This allows to characterize the distribution of each taxon by giving an indicator of probability of taxon presence for each environmental layer pixel. Here, the boosted regression trees (BRT) method was used to adjust the relationship between the presence of a single taxon and the environmental conditions under which the taxon has been detected. BRT modelling is based on an automatic learning algorithm using iterative classification trees.

### What it does ?

Two treatments are performed in this tool: the creation of the taxa distribution model and the use of this model to obtain a prediction index. The prediction index obtained from each BRT model for each pixel of the environmental layers is an approximation of the probability of detection of the presence of the taxon.

This tool gives as output a file containing the predictions of the probability of the presence of each taxon for each "environment pixel" (latitude, longitude), a visualization of these pixels for each taxon and graphs showing the percentage of model explanation for each environmental parameter. We're gonna go back to this in the following steps.

### How to use it ?

> <hands-on-title> Run the BRT tool </hands-on-title>
>
> 1. {% tool [BRT tool prediction](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_brt_analysis/ecoregion_brt_analysis/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input your environment data file of your study area (tabular format only)"*: `ceamarc_env.tsv` (Input dataset)
>    - {% icon param-file %} *"Input your occurrences data file(s) containing also the environmental characteristics where the species has been observe (tabular format only)"*: `Merge table` (output of **GeoNearestNeighbor** {% icon tool %})
>
>      {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    - *"Choose column(s) where your abiotic parameter are in your environment data file."*: `c['3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19']`
>
>    > <warning-title>You maybe can here prepare a cup of tea ;)</warning-title>
>    >
>    > This step is a first "long" one, and you have to wait something like 18 minutes and 34 sec to obtain the result. It is maybe the good moment to have a break and go outside to admirate biodiversity around you?
>    >
>    {: .warning}
>
>
> 2. Check your outputs. You must have four outputs collections.
>
>    - Prediction files
>    - Validation files (Taxa, AUC, Tree complexity, Total deviance explained)
>    - Species distribution prediction maps
>    - Partial dependence plots
>
{: .hands_on}

In the 'Prediction files' collection there must be a file containing predictions of the probability of the presence of each taxon for each "environmental pixel" (latitude, longitude) for each occurrence file you entered.

In the 'Validation files' collection there must be a file containing for each taxon the validation metrics of the associated model.

In the 'Species distribution prediction maps' collection there must be for each taxon a map representing their probability of presence at each environmental layer pixel.
Here is an example:

![Species distribution map](./Images/BRT-Echinodermata_Crinoidea_Comatulida_Antedonidae_Florometra_mawsoni__pred_plot.png "Florometra mawsoni distribution from BRT")

In the 'Partial dependence plots' collection there should be graphs showing the percentage explanation of the model for each environmental parameter.
Here is an example:

![Partial dependence plots](./Images/BRT-Echinodermata_Crinoidea_Comatulida_Antedonidae_Florometra_mawsoni.pdf "Percentage of explanation of the model for each environmental parameter")

## Collecting the list of taxa with **TaxaSeeker**

### What it does ?

This tool does three things:

- It allows obtaining a summary file for each taxon indicating whether a BRT model was obtained and the number of occurrences per taxon.

- It provides a list of taxa that obtained cleaned BRT models (without "_", "_sp", etc.) to propose the list to WoRMS (World Register of Marine Species) or another taxonomic database and obtain more information about the taxa.

- It generates a list of taxa for which a BRT model was obtained, needed as input of following steps.
### How to use it ?

> <hands-on-title> Run TaxaSeeker </hands-on-title>
>
> 1. {% tool [TaxaSeeker](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_taxa_seeker/ecoregion_taxa_seeker/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Environment file (tabular format only)"*: `ceamarc_env.tsv` (Input dataset)
>    - {% icon param-file %} *"Occurences file(s) (tabular format only)"*: `Merged table` (output of **GeoNearestNeighbor** {% icon tool %})
>    - {% icon param-file %} *"Predictions file(s)"*: `Prediction files collection` (output of **BRT tool prediction** {% icon tool %})
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    2. Check your outputs. You must have three files:
>
>    - Summary of taxa model
>    - List of taxa
>    - List of taxa clean
>
{: .hands_on}

## Determine the optimal number of clusters with **ClusterEstimate**

### What it does ?

This tool enables the determination of the optimal number of clusters for partition-based clustering, along with generating files used in the following anlaysis steps.

The tool will produce three outputs. The first two files that will be used in the following steps of the workflow: a file containing four pieces of information (latitude, longitude, presence prediction and corresponding taxon), and a file containing the data to be partitioned.
The third output corresponds to the main information of the tool, a graph presenting the value of the SIH index according to the number of clusters. The silhouette index provides a measure of the separation between clusters and the compactness within each cluster. The silhouette index ranges from -1 to 1. Values close to 1 indicate that objects are well grouped and separated from other clusters, while values close to -1 indicate that objects are poorly grouped and may be closer to other clusters. A value close to 0 indicates a situation where objects are located at the border between two neighboring clusters. Thus the optimal number of clusters is the one that maximizes the value of the SIH index.

### How to use it ?

> <hands-on-title> Run ClusterEstimate </hands-on-title>
>
> 1. {% tool [ClusterEstimate](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_cluster_estimate/ecoregion_cluster_estimate/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Environment file (tabular format only)"*: `ceamarc_env.tsv` (Input dataset)
>    - {% icon param-file %} *"Taxa selected file (File 'List of taxa' from TaxaSeeker tool)"*: `List_of_taxa` (output of **TaxaSeeker** {% icon tool %})
>    - {% icon param-file %} *"Prediction files"*: `Prediction files collection` (output of **BRT tool prediction** {% icon tool %})
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    - *"Number of Cluster to test"*: `10` (You can choose any number, but remember that the more cluster numbers to test the longer it will take)
>
>    > <comment-title> Two other parameters </comment-title>
>    >
>    > The other two parameters can be left as is. If you need to change them here is a short description of what they do:
>    >
>    > The first one is metric used to calculate the dissimilarities between the observations: Manhattan (distance between two real-valued vectors), Euclidean (shortest distance between two
>    > points) and Jaccard (defined as the size of the intersection divided by the size of the union of the sample sets)
>    >
>    > The second one is the sample size that will be used to perform clustering.
>    > Indeed, the clara function is used to cluster large data using a representative sample rather than the entire data set. This will speed up the clustering process and make the calculation
>    > more efficient. A fairly high value representative of the data is recommended. It is important to note that using a too small sample size may result in loss of information compared to using the
>    > entire data set.
>    >
>    {: .comment}
>
> 2. Check your outputs. You must have three files:
>
>    - SIH index plot (See example below)
>    - Data to cluster (Containing the data to be partitioned)
>    - Data.bio table (Containing four pieces of information, latitude, longitude, presence prediction and corresponding taxon)
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the optimal number of clusters in this use case ?
>
> > <solution-title></solution-title>
> >
> > 1. The optimal number of cluster is seven because it maximizes the value of the SIH index.
> >
> {: .solution}
>
{: .question}

Here is the SIH index plot you must obtained :

![SIH index plot](./Images/SIH_index_plot.png "SIH index plot")

With this graph you will be able to determine the optimal number of clusters retained for the construction of ecoregions. As said before, the optimal number of cluster is the one that maximizes the SIH index. In this example, the number of clusters that optimizes the SIH index is seven.

## Build ecoregional clusters with **ClaraClust**

### What it does ?

After choosing an optimal number of clusters with the ClusterEstimate tool, we can now partition each latitude and longitude point according to their associated values of the BRT prediction index. Due to
the large size of the datasets, the Clara function of the Cluster package is used here to apply the Partitioning Around Medoids (PAM) algorithm on a representative sample of the data. This speeds up the
clustering process and makes the calculation more efficient.

### How to use it ?

> <hands-on-title> Run ClaraClust </hands-on-title>
>
> 1. {% tool [ClaraClust](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_clara_cluster/ecoregion_clara_cluster/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Environmental file (tabular only)"*: `ceamarc_env.tsv` (Input dataset)
>    - {% icon param-file %} *"Prediction matrix (file 'data to cluster' from Cluster Estimate tool)"*: `Data_to_cluster` (output of **ClusterEstimate** {% icon tool %})
>    - {% icon param-file %} *"Prediction table (file 'data.bio table' from Cluster Estimate tool)"*: `Data.bio_table` (output of **ClusterEstimate** {% icon tool %})
>    - *"Number of Cluster wanted"*: `7` (Number of cluster determined at the previous step of the workflow in "SIH index plot")
>
>    > <comment-title> Two other parameters </comment-title>
>    >
>    > The other two parameters can be left as is. If you need to change them here is a short description of what they do:
>    >
>    > The first one is metric used to calculate the dissimilarities between the observations: Manhattan (distance between two real-valued vectors), Euclidean (shortest distance between two
>    > points) and Jaccard (defined as the size of the intersection divided by the size of the union of the sample sets)
>    >
>    > The second one is the sample size that will be used to perform clustering.
>    > Indeed, the clara function is used to cluster large data using a representative sample rather than the entire data set. This will speed up the clustering process and make the calculation
>    > more efficient. A fairly high value representative of the data is recommended. It is important to note that using a too small sample size may result in loss of information compared to using the
>    > entire data set.
>    >
>    {: .comment}
>
> 2. Check your outputs. You must have three files:
>
>    - SIH index plot (See example below.)
>    - Cluster points (Contains the latitude and longitude of each "environmental pixel" and the associated cluster number. We will use it in the next step of the workflow)
>    - Cluster info (Contains all the information related to the clusters created, that is, in column: the latitude, the longitude, the corresponding cluster number and for each taxon the prediction value)
>
{: .hands_on}

The tool will produce a silhouette plot that you can see below. A silhouette graph is a representation used to visualize the silhouette index of each observation in a clustered data set. It makes it possible to
assess the quality of clusters and determine their coherence and separation. In a silhouette graph, each observation is represented by a horizontal bar whose length is proportional to its silhouette index. The longer
the bar, the better the consistency of the observation with its cluster and the separation from other clusters. As mentioned above, the silhouette index ranges from -1 to 1. Values close to 1 indicate that objects
are well grouped and separated from other clusters, while values close to -1 indicate that objects are poorly grouped and may be closer to other clusters. A value close to 0 indicates a situation where objects are
located at the border between two clusters. Here, in the graph below, there is a good distribution of the observations because the majority of the bars are above the average value of the silhouette index.

![SIH plot](./Images/SIH_plot.png "SIH plot")

## Build a ecoregionalization map with **EcoMap**

> <hands-on-title> Run EcoMap </hands-on-title>
>
> 1. {% tool [EcoMap](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_eco_map/ecoregion_eco_map/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Source file (File 'cluster points' from previous step)"*: `cluster_points` (output of **ClaraClust** {% icon tool %})
>
> This tool simply need the previous step "Cluster points" output file to generate a map representing ecoregions.
>
> 2. Check output. You must have one Map representing ecoregions.
>
{: .hands_on}

This is the output map with our seven clusters representing ecoregions that you must have obtained.

![Ecoregion map](./Images/Map.png "Ecoregion map")

# Conclusion

Congratulations! You have successfully completed the ecoregionalization workflow. Here is the end of this tutorial aiming to explain the purpose of the ecoregionalization workflow and how to use it. This workflow provides a systematic and reproducible approach to ecoregionalization, allowing researchers to identify distinct ecological regions based on species occurrences and environmental data. This tutorial shows how to use this workflow, step by step, or all in one using the Dumont D'Urville sea region use case with related datasets. It allows you to understand ecoregions construction. You learned the use of the BRT algorithm for modeling species distribution as well as the cluster construction with the k-medoid clustering algorithms (CLARA/PAM). Feel free to explore and adapt this workflow for your specific research needs. If you have any questions, or encounter issues during the workflow, refer to the provided documentation or seek assistance from the Galaxy community. Don't hesitate to contact us if you have any questions.

tools_json:
["Unzip a file", "toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3", "Filter Tabular", "toolshed.g2.bx.psu.edu/repos/iuc/filter_tabular/filter_tabular/3.3.0", "Merge Columns", "toolshed.g2.bx.psu.edu/repos/devteam/merge_cols/mergeCols1/1.0.2", "Interactive JupyterLab and notebook", "interactive_tool_jupyter_notebook", "GeoNearestNeighbor", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_geonearestneighbor/ecoregion_GeoNearestNeighbor/0.1.0+galaxy0", "BRT tool prediction", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_brt_analysis/ecoregion_brt_analysis/0.1.0+galaxy0", "TaxaSeeker", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_taxa_seeker/ecoregion_taxa_seeker/0.1.0+galaxy0", "ClusterEstimate", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_cluster_estimate/ecoregion_cluster_estimate/0.1.0+galaxy0", "ClaraClust", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_clara_cluster/ecoregion_clara_cluster/0.1.0+galaxy0", "EcoMap", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_eco_map/ecoregion_eco_map/0.1.0+galaxy0"]

tags_json:
["ecology", "taxonomic data", "EBV workflow", "modeling", "gbif", "ocean", "earth-system", "interactive-tools", "biodiversity"]


topic: ecology
tutorial: ndvi_openeo
title: From NDVI data with OpenEO to time series visualisation with Holoviews
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["earth-system", "land degradation", "NDVI", "copernicus", "holoviews"]
content_hash: 9ede7a3c4a25e96af942cb14dc51dac1
last_modified: 2026-05-15T22:28:12.066218
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/ndvi_openeo/tutorial.html

content:
Through this tutorial you will learn here how to access and download Copernicus Data Space Ecosystem (CDSE) data through a jupyterlab Galaxy interactive tool :

This tool enables you to leverage the Copernicus Data Space Ecosystem services and access data effortlessly.
The JupyterLab service allows you to dive into data exploration, visualization, and analysis without the hassle of installing dependencies or downloading large data sets.

You will also apprehend how to use the Holoviz ecosysytem by plotting and visualising NDVI (from the CDSE) timeseries :
HoloViz provides high-level Python tools that are designed to work together to solve the entire problem of visualization, from conducting exploratory data analysis to deploying complex dashboards.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Context on Galaxy
Interactive tools work differently than classical tools as it allows the user to interact with a dedicated graphical interface. This kind of tools is used to give access to Jupyter notebooks, RStudio or R Shiny apps for example. 

> <details-title>Short introduction on how Galaxy works</details-title>
>
> You can come back to where you left off the tutorial anytime by clicking {% icon level %}.
>
> > <hands-on-title>Log in to Galaxy</hands-on-title>
> > 1. Open your favorite browser (Chrome, Safari or Firefox as your browser, not Internet Explorer!)
> > 2. Browse to your [Galaxy instance](https://earth-system.usegalaxy.eu/)
> > 3. On the top pannel go to **Login or Register**
> >
> >
> {: .hands_on}
>
> The Galaxy homepage is divided into three panels:
> * Tools on the left
> * Viewing panel in the middle
> * History of analysis and files on the right
>
> ![Screenshot of the Galaxy interface, the tools panel is on the left, the main panel is in the center, and the history is on the right.]({% link shared/images/galaxy_interface.png %} "Galaxy interface explanation")
>
> The first time you use Galaxy, there will be no files in your history panel.
{: .details}

# Copernicus Data Space Ecosystem

> <hands-on-title> Launch the interactive tool </hands-on-title>
>
> 1. {% tool [Copernicus Data Space Ecosystem](interactive_tool_copernicus_notebook) %} with the following parameters:
>    - *"Do you already have a notebook?"*: `Start with a fresh notebook`
>    - *"Include data into the environment - optional "*: You don't need anything
> 2. Click on **Run Tool**
>
> 3. {% snippet faqs/galaxy/interactive_tools_open.md tool="Copernicus Data Space Ecosystem" %}
{: .hands_on}

> <hands-on-title> Navigate the jupyterlab </hands-on-title>
> 1. Once you are in the jupyterlab go to :
>    - **notebooks**
>    - Then, **openeo** 
>    - Open the notebook **NDVI_Timeseries.ipynb**
> 2. You can start running your notebook. To do so you can go on the pannel displaying button right above the notebook and click on the {% icon workflow-run %}. 
>    Everytime you press this button you can excecute a one by one the cells. 
> 3. In the **Setup** section when you execute the connection cell you'll have some actions to conduct.
> 4. First click ont the link appearing, you're notebook should look like the following :
> ![Begining of the NDVI notebook  with the connection link visible](../../images/ndvi_openeo_holoviz/ndvi_notebook.png)
> 5. Once you clicked a new window opens. If you don't have a copernicus account please create one and follow the instructions given. 
> 6. If you have an account sign in and then press the green **YES**
> ![Image of how does the connection page looks like](../../images/ndvi_openeo_holoviz/connection.png)
> 7. You'll be redirected to a page letting you know the connection was a success
> ![Image of the successful connection](../../images/ndvi_openeo_holoviz/connection_ok.png)
> 8. Now, you can go back on your jupyterlab and execute the rest of the notebook.
> 9. Once you executed the entire notebook you should have a new folder in the left pannel named **ndvi-results** go in it.
> 10. Select the 3 files (timeseries-basic.csv, timeseries-masked.csv, timeseries-smoothed.csv) there and click right, and then copy
> ![The copy command on the 3 files in the ndvi-results folder](../../images/ndvi_openeo_holoviz/copy_paste.png)
> 11. Then go back to the root of your path and go in the **ouputs** folder. There you can click right and paste the 3 NDVI files (timeseries-basic.csv, timeseries-masked.csv, timeseries-smoothed.csv). 
> ![The resulting 3 files in the outputs folder](../../images/ndvi_openeo_holoviz/outputs.png)
> 12. This part is now finished you just have to correctly close this notebook. On the top left click on **files** and then on **Shut down**.
> ![The shut down button to correctly close the jupyterlab](../../images/ndvi_openeo_holoviz/shut_down.png)
{: .hands_on}

Go back on your Galaxy window. After a couple minutes you should see your outputs turning green in your history.

Check that your data are in csv format. 

> <tip-title> Check datatype </tip-title>
>
> {% icon galaxy-pencil %}, it should be `csv`
>
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="csv" %}
{: .tip}


# Holoviz

> <hands-on-title> Launch Holoviz </hands-on-title>
>
> 1. {% tool [Holoviz](interactive_tool_holoviz) %} with the following parameters:
>    - *"Do you already have a notebook?"*: `Start with a fresh notebook`
>    - *"Include data into the environment - optional "*: Select `timeseries-basic`, `timeseries-masked` and `timeseries-smoothed`
> 2. Click on **Run Tool**
>
> 3. {% snippet faqs/galaxy/interactive_tools_open.md tool="Holoviz" %}
> 
> 4. Once, you are in the jupyterlab go in the the **notebooks** folder
> 5. Open a terminal in bash and write the following
> > <code-in-title>Download the notebook</code-in-title>
> > ```bash
> > wget https://raw.githubusercontent.com/fair-ease/Holoviz/main/Tabular_Datasets_NDVI.ipynb
> > ```
> {: .code-in}
> Now that you have your notebook you can just follow the instructions.
{: .hands_on}

Enjoy discovering the rest of the notebook. 

# Conclusion
The tutorial ends there and with that you have the basis on how to use jupyterlab tools and more particularly the copernicus and holoviz ones.

tools_json:
["Copernicus Data Space Ecosystem", "interactive_tool_copernicus_notebook", "Holoviz", "interactive_tool_holoviz"]


topic: ecology
tutorial: Metashrimps_tutorial
title: Creating FAIR Quality assessment reports and draft of Data Papers from EML
difficulty: intermediate
time_estimation: 30M
tools_json: ["MetaShRIMPS", "interactive_tool_metashrimps"]
requirements_json: []
tags_json: ["metadata", "EML", "FAIR", "data paper", "biodiversity"]
content_hash: 98651b167556177a648083ad2b0d4b96
last_modified: 2026-05-27T09:01:48.390866
zenodo_link: https://doi.org/10.5281/zenodo.8130567
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/Metashrimps_tutorial/tutorial.html

content:
This tutorial aims to teach how to use the interactive tool MetaShRIMPS, available on Galaxy Ecology,
to produce Data Papers drafts and FAIR quality assessment reports from metadata using EML
standard.
This tutorial purpose is also to explain why it is needed to improve the overall FAIR quality
of metadata and how this tool responds to this necessity.

> <comment-title>What does FAIR mean?</comment-title>
> [FAIR](https://www.go-fair.org/fair-principles/) stand for **Findable, Accessible, Interoperable, Reusable**. 
>
>![FAIR Data Principles](./Images/FAIR_data_principles.jpg){:width="500"}
>These principles were to improve the access and usabiliy of data by the machine and to help making data reusable and shareable for users.
>Metadata is the data used to describe and explain all the context behind the production of data. It is necessary to produce a rich and FAIR metadata in order 
>to permit external users to understand and reuse data for their own studies.
{:  .comment}
> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# How can this tool improve the metadata quality?

The purpose of this tool is to help the user improve their metadata quality in order to increase its value to the scientific community and to help highlighting
the work of all the producers of the data.
To respond to this objective, this tool aims to give an easy access to a quality assessment report of EML metadata which could guide the producers of
data/metadata to the production of a metadata with high degree of FAIRness.
The other objective of this tool is to highlight the work of all of the people that helped in producing this data by giving the access of a draft of Data Paper
that is reuseable either in a non editable HTML file, that can represent the metadata in a more ergonomic way facilitating its understanding and shareability,
or in an editable docx file. Having an editable Data Paper draft will allow the producer to complete/modify the draft of Data Paper so that it could become
publishable as a real Data Paper giving recognition to all the people that helped producing the data.

> <comment-title>What is a Data Paper?</comment-title>
> According to the [GBIF](https://www.gbif.org/data-papers) (Global Biodiversity Information Facility), 
> A data paper is a peer reviewed document describing a dataset, published in a peer reviewed journal. It takes effort to prepare, curate and describe data. 
> Data papers provide recognition for this effort using a scholarly article.
{:  .comment}
# Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import this metadata file from [Zenodo]({{ page.zenodo_link }}) to test it
>     -> Training Data for "Creating Quality FAIR assessment reports and draft of Data Papers from EML metadata with MetaShRIMPS":
>    ```
>    https://zenodo.org/record/8130567/files/Kakila_database_marine_mammal.xml
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
{: .hands_on}


# Upload Data in MetaShRIMPS

When opening {% tool [MetaShRIMPS](interactive_tool_metashrimps) %}, you will have an interface looking like this :

![Interface of MetaShRIMPS](./Images/upload_1.png){:width="500"}

To upload data on MetaShRIMPS, if you was selecting the xml file as input of the tool, it is already launched, if not, you have to click on the browse button and select in your local folders, the file
you want to use. 
> <warning-title>Select the right format</warning-title>
> The file uploaded in this tool must be a metadata in XML format using EML metadata standard.
{: .warning}

![Interface when file is selected](./Images/upload_1.png){:width="500"}

After uploading the file, or if you have indicated it as input data in the tool form, you just have to click on **Execute** to launch MetaShRIMPS jobs with the file.

# Outputs

After clicking the **Execute** button, 2 new tabs called "Draft of Data Paper" and "Fair Assessment"  will appear.
You can access all of the tool outputs by clicking on each tab (it can take a little time for your results to be displayed).

## Draft of Data Paper

By clicking on the "Draft of Data Paper" tab, you will have access to the draft of Data Paper presented in an HTML format.
You can either navigate through the Data Paper with the tabs or with the scrollbar on the right and access different elements.

You can at the top of the page download the draft in either an HTML format or an editable docx format.
![Download in HTML](./Images/Download_HTML.png""Download in HTML")
![Download in docx](./Images/Download_docx.png "Download in docx")

## Fair Quality Assessment report

By clicking on the "Fair Assessment" tab, you will access the FAIR Quality report of the metadata uploaded.
You will have access to different figures such as a table displaying the results of all checks tested for your metadata.
![Table of results](./Images/Fairscore_tab.png "Example of the table displaying the results of the Quality Checks")

You will also have acces to a graph presenting scores of Quality for each of the FAIR principles tested (Findable,
Acessible, Interoperable, Reusable) on a 100 point scale.

![FAIR scores](./Images/Fairscore_bar.png "Example of a FAIR score")

# Conclusion

Here is the end of this short tutorial aiming in explaining the purpose of MetaShRIMPS and how to use it.
Don't hesitate to contact us if you have any questions :)


topic: ecology
tutorial: openrefine_gbif
title: Cleaning GBIF data using OpenRefine
difficulty: introductory
time_estimation: 0h45m
requirements_json: []
tags_json: ["biodiversity", "data cleaning"]
content_hash: 32f8fbaf5cb4c53a177f6f8dcdfd928e
last_modified: 2026-05-27T09:01:48.400641
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/openrefine_gbif/tutorial.html

content:
In this tutorial we will use OpenRefine tool to clean occurrence records retrieved from GBIF (Global Biodiversity Information Facility).

This tutorial is based on the GBIF {% cite GBIFtraining2021 %} "Biodiversity data mobilization course" accessible on [GBIF website](https://docs.gbif.org/course-data-mobilization/en/) on Use case 1 ["Herbarium Specimens"](https://docs.gbif.org/course-data-mobilization/en/exercise-3a-c.html) using file from Exercise 3C, at the bottom of the page, reachable through this direct [URL](https://docs.gbif.org/course-data-mobilization/exercise-data/UC1-3c-open-refine.csv).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Retrieve data from GBIF "Biodiversity data mobilization course"


> <hands-on-title> Data import and project creation </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (example: “OpenRefine tutorial”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import data file from GBIF {% cite GBIFtraining2021 %} "Biodiversity data mobilization course" accessible on [GBIF website](https://docs.gbif.org/course-data-mobilization/en/) on Use case 1 ["Herbarium Specimens"](https://docs.gbif.org/course-data-mobilization/en/exercise-3a-c.html) using file from Exercice 3C, at the bottom of the page
>
>    "Herbarium Specimens" data file as a csv file named "UC1-3c-open-refine.csv"
>    ```
>    https://docs.gbif.org/course-data-mobilization/exercise-data/UC1-3c-open-refine.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `UC1-3c-open-refine.csv`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `;`
>            - *"Replacement"*: `\t`
>    - click button "run tool"   
> 4. Change the datatype from "txt" to "tabular"
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 6. Check your output. All columns must be separated by a tabulation now. See example below.
>
>    ![Output of Regex Find And Replace example](../../images/openrefine/1_openrefine.png "Output of Regex Find And Replace example"){: style="width:50%"}
>
>
> 7. Rename the datasets if needed, for example with "UC1-3c-open-refine-tabular" to keep orginal name and specify the tabular format.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
>
{: .hands_on}


# Deploy an OpenRefine instance and push your data in


> <hands-on-title> Deploy an OpenRefine instance and import data from your Galaxy history </hands-on-title>
>
> 1. {% tool [OpenRefine tool](interactive_tool_openrefine) %} with the following parameters:
>    - *"Input file in tabular format"*: `UC1-3c-open-refine-tabular`
>    - click the button "run tool"
> You will then be redirected. A notification "Interactive Tools" shows up on the left-hand side, indicating you have an interactive tool running.
> Click on the notification, click on the empty box in front of the text saying "Openrefine".
> Then, click on the text "Openrefine" to open the tool in a new tab. It might take a moment until your cursor changes and the text becomes a hyperlink.
>
> 2. Open the OpenRefine instance
>    - go to "Open Project" space
>    - select the project named "Galaxy file"
> 3. You will now have your project with first 10 lines of your file displayed
> ![OpenRefine project start](../../images/openrefine/openrefine_start.PNG)
>
{: .hands_on}


# Faceting

Faceting is a feature that will allow us to get a big picture overview of the data, and to filter down to just the subset of rows
that we want to change or view in bulk. It facilitates the use and analysis of data and can be done with cells containing any
kind of text, numbers and dates.

> <hands-on-title> Faceting and mass editing </hands-on-title>
>
> 1. Go to column "kingdom", and then click on the column menu and follow the route to `Text facet`.
>    - On the left a window with the name of the column will appear, that is the facet.
>    ![OpenRefine facet kingdom](../../images/openrefine/openrefine_facet_kingdom.PNG)
> 2. Click on count to sort by count, then click on name to sort alphabetically
> 3. Fix the spelling mistakes (Plante -> Plantae). Place the cursor over the text in the window and click on `edit`, then fix the error in the text box, and to save click on `apply`.
>    - All the values will be fixed automatically.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many Plantae occurences you now have?
>
> > <solution-title></solution-title>
> >
> > 1. 736
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Faceting and white space 1 </hands-on-title>
>
> 1. Go to "Country col." column and click on column menu and perform a `Text Facet`
>    - On a quick view, the country appears to be spelled correctly, but the facet shows three different values due to the extra spaces at the end of the text.
> 2. Fix the error from the column menu on "Country col." column, following the route by clicking the dropdown menu, `Edit Cells > Common transforms > Trim leading and trailing whitespace`.
>    - You will see a notification message "Text transform on 38 cells in column Country col.: value.trim()" and the possibility to come back on previous state clicking on the `undo` hyperlink.
> 3. Now check the facet window; only one value will remain.
>
{: .hands_on}

> <hands-on-title> {% icon hands_on %} Faceting and white space 2 </hands-on-title>
>
> 1. Go to column "Full name" and click on the dropdown menu, then go to `Text facet`. Then click on `count`.
>    - *Guzmania lingulata* is the first item in the list with 25 specimens, but it is also present in the 3rd position with 20 specimens.
> 2. Fix the error from the "Full name" column menu, `Edit Cells > Common transforms > Collapse consecutive whitespaces`.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many *Guzmania lingulata* records only appear in the list finally?
>
> > <solution-title></solution-title>
> >
> > 1. 45
> >
> {: .solution}
>
{: .question}

> <hands-on-title> {% icon hands_on %} Faceting and duplicates </hands-on-title>
>
> 1. Go to column catalog in "Cat. Numb", and follow the route `Facet > Customized facets > Duplicates facet`.
>    - The facet will show 4 duplicates
> 2. Click on true, and you'll see the values in the main window
> 3. After a check with the specimens labels, fix the values clicking edit directly on the cell with the correct catalogue numbers
>    - UWP:122470 for Vargas P
>    - UWP:122471 for Vargas I
>    - UWP:157351 for Betancur H
>    - UWP:157339 for Betancur J
>    
>     click `apply` after you finish each edit
{: .hands_on}

# Filtering

> <hands-on-title> {% icon hands_on %} Basic filter </hands-on-title>
>
> 1. Go again to "Full name" column menu and perform a `Text facet` to visualize the values
> 2. Then go again to the column menu and click on `Text filter`
> 3. Perform the following filters and fix them as described below:
>      - search for "sp1" entries -> Then remove it and obtain "Cyperus", clicking on `Edit` directly in the cell
>      - search for "SP2" entries, check `case sensitive` -> To remove it and obtain "Cyperus", you can `Edit` directly in the cell
>      - search for "spp".
>         - Go to "Full name" column menu, then click `Edit cells > Transform`
>         - In the text box paste the formula `value.replace(" spp.", "")` and click `ok`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Do you think you can use also `Edit cells > Transform` formula to apply the changes you made manually for the two first terms searched ("sp1" and "SP2")?
> 2. Why can you easily edit these two first entries directly but not the third one?
>
> > <solution-title></solution-title>
> >
> > 1. Yes, using respectively `value.replace(" sp1", "")` and `value.replace(" SP2", "")` formulas
> > 2. Because the first two searches only concern one entry for each when the third one returns several.
> >
> {: .solution}
>
{: .question}

> <hands-on-title> {% icon hands_on %} Advance filter 1 </hands-on-title>
>
> 1. Go to column "genus" and perform a `Text filter`.
> 2. Check `regular expression` and `case sensitive` , then paste the expression "^[a-z]"
>    - This regular expression filters the strings in which the first letter is lowercase.
> 3. Perform a correction since the genus should be capitalized.
>
>
>    > <comment-title></comment-title>
>    >
>    > If you want to know more about regular expressions, you can consult the [Openrefine regular expressions documentation](https://openrefine.org/docs/manual/expressions#regular-expressions) and specifically the [documentation for GREL function](https://openrefine.org/docs/manual/grelfunctions)
>    {: .comment}
>
>    {% snippet faqs/galaxy/analysis_regular_expressions.md %}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Try to find a regular expression to apply the same kind of `Text filter` operation as in the previous exercise. Which regular expression can work?
>
> > <solution-title></solution-title>
> >
> > 1. On my side, I found this formla `value.replace(value.substring(0,1),toUppercase(value.substring(0,1)))`
> >
> {: .solution}
>
{: .question}

> <hands-on-title> {% icon hands_on %} Advance filter 2 </hands-on-title>
>
> 1. Go to column "Full name" and perform a `Text filter`.
> 2. Check `regular expression` and `case sensitive`, then paste the expression `^[A-Z].*\s[A-Z]`
>    - This regular expression filters the strings that start with a capital letter followed by 0 or more characters, then a space, then a capital letter.
> 3. Perform a correction since the second word of the name should be lowercase.
>
{: .hands_on}


# Clustering

> <hands-on-title> Basic clustering </hands-on-title>
>
> 1. Go to "County" and perform a `Text facet`.
>    - Keep in mind that the correct counties are: "Flores", "La Libertad", "Melchor de Mencos", "San Andres" and "San Jose".
> 2. On the top right of the facet window click on `Cluster`, a new window will appear.
> 3. Click on the `Cluster` button from this new window.
> 4. Now you can see information about the clusters:
>    - **"Cluster size":** the number of different versions that the clustering algorithm believes to be the same.
>    - **"Row count":** the number of records with any of the cluster values.
>    - **"Values in cluster":** the actual values that the algorithm believes to be the same. There is also the number of records with each particular value, and the possibility to browse the contents of the cluster in a different tab.
>    - **"Merge?":** check if values are to be merged into a single standard value.
>    - **"New cell value":** the value to be applied to every record in the cluster. By default, it is the value with most records. You can also click on any value to apply that to the New cell value.
>
>
>    > <comment-title></comment-title>
>    >
>    > If you want to know more about clustering have a look at the [manual](https://openrefine.org/docs/manual/cellediting#cluster-and-edit)
>    {: .comment}
>
> 5. Click on `Select All` and then on `Merge Selected & close`
>    - You will see a notification message "Mass edit 119 cells in column County".
> 6. To fix the remaining counties go again to Cluster in the facet window of Count.
> 7. In the *Cluster and edit* window, go to `Keying Function`, then select `ngram-fingerprint`
>    - set "1" as the value in `n-Gram Size`.
> 8. Press the `Cluster` button, you normally see a cluster about "San Andres" of size "4".
> 9. Click on `Select All` and then on `Merge Selected & close`, you will see a notification message "Mass edit 360 cells in column County".
>    - Your counties are now fixed! Congratulation!
{: .hands_on}


# Exporting

> <hands-on-title> {% icon hands_on %} Exporting cleaned file into your Galaxy history </hands-on-title>
>
> 1. On the upper right corner click on `Export` and select `Galaxy exporter`.
>    - A notification message as "Dataset has been exported to Galaxy, please close this tab" is displayed.
>    - You normally have your resulting data file exported on your Galaxy history as "openrefine-Galaxt file.tsv" dataset.
>
>    - When you want to close OpenRefine, go back to the Galaxy window. Go to the middle window, "Active Interactive Tools". Below `Openrefine` click `stop`.
>    - The interactive tool is then stopped and the respective task shows red in your history. This does not affect your data, which is safely stored in the "openrefine-Galaxt file.tsv" dataset.
>    - You can continue working with it in Galaxy for the following tasks.
>
>    > <comment-title></comment-title>
>    >
> > You can also download the file directly on your computer, choosing `Export` and `Custom tabular exporter`, allowing you to select notable columns and rows you want to download.
>    {: .comment}
>
{: .hands_on}


# API use

Reconciliation matches the information in one of your columns to an outside database. This is particularly helpful when it
comes to name validation, as it proves the name you have exists somewhere else. This is a really useful service but can be
time-consuming. In this case, we will go through the process with only three records using the API from GBIF.
To open your dataset again from Galaxy in OpenRefine, follow the earlier steps from this tutorial "Deploy an OpenRefine instance and push your data in".

> <hands-on-title> {% icon hands_on %} Higher taxonomy </hands-on-title>
>
> 1. In your opened dataset in OpenRefine, go to the "Collector" column, then make a `Text facet`. Select the collector "Elsa P".
> 2. Under "Full name", click on column menu and then `Edit column > Add column by fetching URLs…`
>    - call the new column "Api_name"
> 3. Change the Throttle Delay to 250 and paste the expression `"http://api.gbif.org/v1/species/match?verbose=true&name="+escape(value,'url')`
> 4. Click ok and wait, this might take some time depending on internet connection and the number of taxa.
> 5. Go to "Api_name", click on column menu and then `Edit column > Add column based on this column...`.
>    - Call the new column "higherClassification" and paste the expression:
>
>    ```
>    value.parseJson().get("kingdom")+
>    ", "+value.parseJson().get("phylum")+
>    ", "+value.parseJson().get("class")+
>    ", "+value.parseJson().get("order")+
>    ", "+value.parseJson().get("family")
>    ```
>   click `ok`. 
>   You will see the Kingdom, Phylum, Class, Order and family of each taxon.
> 6. Under "higherClassification" follow the route `Edit column > Split into several columns…`, leave the initial settings and click `ok`.
> 7. Now you know how to obtain the taxonomic categories of a given taxon if this is available in the GBIF API. Column names can be edited in `Edit column > Rename this column`.
> 8. For the purpose of the original GBIF workshop, the columns created in this exercise (Higher taxonomy) must be deleted.
>    -  Under All, which is the first column, go to `Edit columns > Re-order / remove columns…`.
>    - Remove columns "Api_name", "higherClassification 1", "higherClassification 2", "higherClassification 3", "higherClassification 4" and "higherClassification 5".
> 9. No need to export this file as it normally comes back to the previous version you already exported.
>
{: .hands_on}



# Conclusion

Here you learned how to use OpenRefine tool from Galaxy platform to clean Biodiversity data. This tutorial notably allowed you to apply some basic but powerful functionalities of OpenRefine to clean your data.

tools_json:
["Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3", "OpenRefine tool", "interactive_tool_openrefine"]


topic: ecology
tutorial: genetic-map-rad-seq
title: RAD-Seq to construct genetic maps
difficulty: intermediate
time_estimation: 8h
tools_json: []
requirements_json: []
tags_json: ["RAD-seq", "genetic composition EBV class", "EBV dataset", "EBV workflowz", "biodiversity"]
content_hash: 2dc5027ed23fbbcbfc833caf95b02bcd
last_modified: 2026-05-27T09:01:48.400158
zenodo_link: https://doi.org/10.5281/zenodo.1219888
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/genetic-map-rad-seq/tutorial.html

content:
This tutorial is based on the analysis described in [publication](http://www.genetics.org/content/188/4/799).
Further information about the pipeline is available from [the official STACKS website](http://catchenlab.life.illinois.edu/stacks).
The authors developed a genetic map in the spotted gar and presented data from a single linkage group.
The gar genetic map is an F1 pseudotest cross between two parents and 94 of their F1 progeny. They took the markers that
appeared in one of the linkage groups and worked backwards to provide the raw reads from all of the stacks contributing to that linkage group.

This tutorial re-analyzes these data through to genotype determination. These data do not require demultiplexing and do not need processing though `Process Radtags tool`.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pretreatments

## Data upload

The original data is available at [STACKS website](http://catchenlab.life.illinois.edu/stacks/) and the subset used here is findable on [Zenodo](https://zenodo.org/record/1219888#.WtZlK5c6-00).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RAD-seq exercise.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import Fasta files from parents and 20 progeny.
>
>    > <comment-title></comment-title>
>    > If you are using the [GenOuest Galaxy instance](https://galaxy.genouest.org), you can load the dataset using 'Shared Data' -> 'Data Libraries' -> '1 Galaxy teaching folder' -> 'EnginesOn' -> 'RADseq' -> 'Genetic map'
>    {: .comment}
>
>    All of the data for this tutorial is on Zenodo:
>
>    ```
>    https://zenodo.org/record/1219888/files/female
>    https://zenodo.org/record/1219888/files/male
>    https://zenodo.org/record/1219888/files/progeny_1
>    https://zenodo.org/record/1219888/files/progeny_2
>    https://zenodo.org/record/1219888/files/progeny_3
>    https://zenodo.org/record/1219888/files/progeny_4
>    https://zenodo.org/record/1219888/files/progeny_5
>    https://zenodo.org/record/1219888/files/progeny_6
>    https://zenodo.org/record/1219888/files/progeny_7
>    https://zenodo.org/record/1219888/files/progeny_8
>    https://zenodo.org/record/1219888/files/progeny_9
>    https://zenodo.org/record/1219888/files/progeny_10
>    https://zenodo.org/record/1219888/files/progeny_11
>    https://zenodo.org/record/1219888/files/progeny_12
>    https://zenodo.org/record/1219888/files/progeny_13
>    https://zenodo.org/record/1219888/files/progeny_14
>    https://zenodo.org/record/1219888/files/progeny_15
>    https://zenodo.org/record/1219888/files/progeny_16
>    https://zenodo.org/record/1219888/files/progeny_17
>    https://zenodo.org/record/1219888/files/progeny_18
>    https://zenodo.org/record/1219888/files/progeny_19
>    https://zenodo.org/record/1219888/files/progeny_20
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    As default, Galaxy takes the link as name. It does not link the dataset to a database or a reference genome.
>
{: .hands_on}

# Building loci using STACKS

Run `Stacks: De novo map` Galaxy tool. This program will run `ustacks`, `cstacks`, and `sstacks` on each individual, accounting for the alignments of each read.

> <comment-title></comment-title>
>
> Information on `denovo_map.pl` and its parameters can be found online: https://creskolab.uoregon.edu/stacks/comp/denovo_map.php.
{: .comment}


> <hands-on-title>Stacks: De novo map</hands-on-title>
> **Stacks: De novo map** {% icon tool %}: Run Stacks selecting the Genetic map usage.
>  - *"Select your usage"*: `Genetic map`
>  - *"Files containing parent sequences"*: `female` and `male`
>  - *"Files containing progeny sequences"*: all 20 progeny files
>  - *"Cross type"*: `CP(F1 cross)`
>  - Click on `Assembly options`
>    - *"Minimum number of identical raw reads required to create a stack"*: `3`
>    - *"Minimum number of identical, raw reads required to create a stack in 'progeny' individuals"*: `3`
>    - *"Number of mismatches allowed between loci when building the catalog"*: `3`
>    - *"Remove, or break up, highly repetitive RAD-Tags in the ustacks program"*: `Yes`
>
>    Once Stacks has completed running, you will see 5 new data collections and 8 datasets.
>
>    ![The output of de novo map](../../images/RAD2_Genetic_Map/denovomap_out.png)
>
>    Investigate the output files: `result.log` and `catalog.*` (snps, alleles and tags).
>
>    Looking at the first file, `denovo_map.log`, you can see the command line used and the start as end execution time.
>
>    ![De novo map log file](../../images/RAD2_Genetic_Map/denovo_map_log_top.png)
>
>    Then are the different STACKS steps:
>
>    `ustacks`
>
>    ![De novo map:ustacks log](../../images/RAD2_Genetic_Map/denovo_map_log_ustacks.png)
>
>    `cstacks`
>
>    ![De novo map: cstacks](../../images/RAD2_Genetic_Map/denovo_map_log_cstacks.png)
>
>
>    > <question-title></question-title>
>    >
>    > 1. Can you identify the meanning of the number 425?
>    > 2. Looking at the catalog.tags file, identify specific and shared loci from each individual. Count the number of catalog loci coming from the first individual, from the second, and find on both parents.
>    >
>    > > <solution-title></solution-title>
>    > > 1. Here, the catalog is made with 459 tags, 425 coming from the "reference individual", a female. Some of these 425 can be shared with the other parent.
>    > > 2. 35 / 34 / 390
>    > {: .solution }
>    {: .question}
>    `sstacks`
>
>    ![De novo map: sstacks log](../../images/RAD2_Genetic_Map/denovo_map_log_sstacks.png)
>
>
>    Lastly, `genotypes` is executed. It searches for markers identified on the parents and the associate progenies' haplotypes. If the first parent have a GA (ex: aatggtgtGgtccctcgtAc) and AC (ex: aatggtgtAgtccctcgtCc) haplotypes, and the second parent only a GA (ex: aatggtgtGgtccctcgtAc) haplotype, STACKS declares an ab/aa marker for this locus. Genotypes program then associate GA to a and AC to b and then scan progeny to determine which haplotype is found on each of them.
>
>    ![De novo map: genotypes log](../../images/RAD2_Genetic_Map/denovo_map_log_genotypes1.png)
>
>
>    ![De novo map: genotypes log](../../images/RAD2_Genetic_Map/denovo_map_log_genotypes2.png)
>
>
>    ![De novo map: genotypes log](../../images/RAD2_Genetic_Map/denovo_map_log_end.png)
>
>    Finally, 447 loci, markers, are kept to generate the `batch_1.genotypess_1.tsv` file. 459 loci are stored on the observed haplotype file `batch_1.haplotypes_1.tsv`.
>
{: .hands_on}

## Matches files

Here are `sample1.snps` (left) and `sample2.snps` (right)

![De novo map matches files](../../images/RAD2_Genetic_Map/denovo_map_matches1.PNG)

Catalog_ID (= catalog Stacks_ID) is composed by the `Stack_ID` from the "reference" individual, here sample1, but number is different from sample2 `Stack_ID`. Thus, in the `catalog.alleles.tsv`, the `Stack_ID` number 3 corresponds to the `Stack_ID` number 16 from sample2!

You can inspect the matches files (you maybe have to change the tsv datatype to a tabular one to correctly display the datasets).

![Male and female matches files](../../images/RAD2_Genetic_Map/denovo_map_matches2.png)

Consider catalog SNPs 27 & 28, on the 302 catalog locus:

![De novo map matches considering catalog SNPs](../../images/RAD2_Genetic_Map/denovo_map_matches_snps.png)

nd the corresponding catalog haplotypes, 3 on the 4 possible (AA, AT, GT but no GA):

![De novo map matches considering catalog haplotypes](../../images/RAD2_Genetic_Map/denovo_map_matches_alleles_haplotypes.png)

heterozygosity is observed on each parent (one ab, the other ac) and there are 19 genotypes for the 22 individuals.

![De novo map macthes: markers](../../images/RAD2_Genetic_Map/denovo_map_matches_markers.png)

We can then see that Stack_ID 330 for female corresponds to the 39 for male:


![De novo map matches: male and female](../../images/RAD2_Genetic_Map/denovo_map_matches_alleles_male_female.png)

# Genotypes determination

> <hands-on-title>Stacks: Genotypes</hands-on-title>
> **Stacks: genotypes** {% icon tool %}: Re-Run the last step of `Stacks: De novo map` pipeline specifying more options as:
>
>    1. The genetic map type (ie F1, F2 (left figure, F1xF1), Double Haploid, Back Cross (F1xF0), Cross Pollination (right figure, F1 or F2 but resulting from the cross of pure homozygous parents))
>
>       ![The genetic map type F2](../../images/RAD2_Genetic_Map/Genetic_map_F2.png)    ![The genetic map CrossPollination](../../images/RAD2_Genetic_Map/Genetic_map_CrossPollination.png)
>
>
>    2. Genotyping options output file type for input in genetic mapper tools (ie JoinMap, R/qtl, ...). Observe that the R/qtl format for an F2 cross type can be an input for MapMaker or Carthagene.
>
>    3. Thresholds concerning a minimal number of progeny and/or minimum stacks depth to consider a locus
>
>    4. Make Automated Corrections to the Data. This option allows the user to have the program automatically correct some types of errors. This setting can correct errors with the homozygous tags verification in the progeny by confirming the presence or absence of the SNP. If SNP detection model can't identify a site as heterygous or homozygous, that site is temporarily tagged as homozygous to facilitate the search, by sstacks, in concordance with the loci catalog. If a second allele is detected on the catalog (ie, in parents) and is found on a progeny with a weak frequency (<10% of a stack reads number), the genotypes program can correct the genotype. Additionally, it will delete a homozygous genotype on a particular individual if locus genotype is supported by less than 5 reads. Corrected genotypes are marked uppercase.
>
>    Here is an example of a locus originally marked as homozygous before automatic correction because an allele is supported by less than 5 reads. After correction, this locus is marked as heterozygous.
>
>    ![Automatic correction of genotypes](../../images/RAD2_Genetic_Map/genotypes_automatic_correction.png)
>
>    You can re-run **Stacks: genotypes** {% icon tool %}: modifying the number of genotyped progeny to consider a marker and thus be more or less stringent. Compare results.
>
{: .hands_on}

## Genotypes.tsv files

One line by locus, one column by individual (aa, ab, AB if automatic correction applied, bb, bc, ...) with observed genotype for each locus:

![Genotypes.tsv file overview](../../images/RAD2_Genetic_Map/genotypes_tsv.png)

## Genotypes.txt files

One line by individual, and for each individual, for each catalog locus, genotype:

![Genotypes.txt file overview](../../images/RAD2_Genetic_Map/genotypes_txt.png)

## Haplotypes.tsv files

One line by locus, one column by individual (aa, ab, AB if automatic correction applied, bb, bc, ...) with observed genotype for each locus:

![Haplotypes.tsv file overview](../../images/RAD2_Genetic_Map/haplotypes_tsv.png)

> <question-title></question-title>
>
> 1. The use of the deleverage algorithm allows to not consider loci obtained from merging more than 3 stacks. Why 3 if biologically, you are waiting something related to 2 for diploid organisms?
> 2. Re-execute **Stacks: De novo map** pipeline modifying the p-value treshold for the SNP model. What is the difference regarding to unverified haplotypes ?
>
> > <solution-title></solution-title>
> > 1. This value of 3 is important to use if we don't want to blacklist loci for whom 99.9% of individuals have one and/or the alt allele and 0.01% have a third one, resulting of a sequencing error.
> > 2. We see a moficiation of the number of unverified haplotypes
> {: .solution }
{: .question}


# Conclusion


In this tutorial, we have analyzed real RAD sequencing data to extract useful information, such as genotypes and haplotypes to generate input files for downstream genetic map creation. This approach can be summarized with the following scheme:

![The genetic map tutorial workflow](../../images/genetic_map_workflow.PNG)


topic: ecology
tutorial: remote-sensing
title: Sentinel 2 biodiversity
difficulty: intermediate
time_estimation: 48H
requirements_json: []
tags_json: ["Remote sensing", "biodiversity"]
content_hash: 6fb841a9ead2118f17a4867414197426
last_modified: 2026-05-27T09:01:48.401711
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/remote-sensing/tutorial.html

content:
This tutorial will guide you on getting Sentinel 2 data and processing them in order to calculate and visualize biodiversity indicators. This workflow made of 6 tools will allow you to explore Sentinel 2 data in the view of making biodiversity analyses.

Spatial diversity measurements should not replace in situ biodiversity data, but rather complement existing data and approaches. Spatial diversity estimates are currently based on long time scales, allowing more general predictions about rates of change in diversity. In practice, spatial data incorporate information on surface properties, including functional aspects, taxonomy, phylogeny and genetic diversity. 

The tools explained here are useful for observing variations in spatial and temporal ecosystem properties given the intrinsic relationship between spatial variations in ecosystems and pixel values of spectral signals. A single measurement cannot provide a complete description of all the different aspects of ecosystem heterogeneity. Therefore, the combination of multiple tools in a Galaxy-Ecology workflow offers multiple approaches to unravel the complexity of ecosystem heterogeneity in space and time.

So, we will compute biodiversity and spectral indices mainly using reflectance information.

> <details-title>Definition of reflectance</details-title>
>
> The reflectance is a proportion on reflected light on an area. It's the ratio between the electromagnetic incident wave on the area and the reflected wave. It's often a percentage between reflected intensity and incident intensity assumed as energy quantity.
>
{: .details}

![Sentinel 2 toolsuite workflow: from the dowload of the data to the production of biodiversity indicators](../../images/remote_sensing/sentinel_2.png "Sentinel 2 toolsuite workflow")

Each part of this workflow has elementary steps :
 - **A first step** to preprocess Sentinel 2 data:
   - Preprocess Sentinel 2 data
 - **A second step** to compute biodiversity indicators
   - Global overview
   - For Canopy
 - **A third step** to compute spectral indices:
   - Spectral indices
   - EBV

> <details-title>Details about spectral indices</details-title>
>
> Spectral indices are used to highlight particular features or properties of the earth's surface, e.g. vegetation, soil, water. They are developed on the basis of the spectral properties of the object of interest.
>
> Knowledge of the leaf cell, plant structure, state, condition and spectral properties is essential to perform vegetation analysis using remote sensing data.  
> Spectral indices dedicated to vegetation analysis are developed on the basis that healthy vegetation reflects strongly in the near infrared (NIR) spectrum while absorbing strongly in the visible red.  
>
{: .details}


In this tutorial, we'll be working on Sentinel 2 data extracted from the Theia Land portal.  First those data will be prepared. After pre-processing to fit the input format of the tools, we'll see how to calculate biodiversity metrics.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <warning-title>It can take a bit of time</warning-title>
> As your processing heavy data some of the steps can take some time. Notably, the step **Create Biodiversity maps** can run for 2 days. We advise to let it run and proceed with the tutorial. Once, the **pre-processing** part is done all the steps can be done separately. Thus, no need for you to wait the end of each tool before running another one.
{: .warning}

## Upload and pre-processing of the data

This first step consist of downloading and properly prepare the data to use it in Sentinel 2 toolsuite.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (example: “Sentinel 2 data for biodiversity tutorial”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>
> 2. Download the files from [Scihub](https://www.sentinel-hub.com/explore/copernicus-data-space-ecosystem/), [PEPS](https://peps.cnes.fr/rocket/#/search?maxRecords=50&page=1) or [Theia](https://catalogue.theia-land.fr) :
>
>    You will have to to create an account for either of these platform.
>    Select Reflectance, Sentinel 2 and "Niveau 2A" (level 2A).
>
>    ![Theia portal welcome page](../../images/remote_sensing/theia.png "Theia portal")
>
>    This an example of the Theia land portal. You need to download a zip folder. Keep it that way.
>
> 3. Upload the zip folder
>
>    {% snippet faqs/galaxy/datasets_upload.md %}
>
> 4. You can **rename** {% icon galaxy-pencil %} the dataset, `sentinel_2_data.zip` for example, and to keep informations about the original name `SENTINEL2A.....zip`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Preparing data

Using preprocS2 R package, this step provides a unique tool to read, crop, resample the original image directory, and write it as a raster stack

## Preprocessing sentinel 2 data

> <hands-on-title>Preprocess</hands-on-title>
>
> 1. {% tool [Preprocessing sentinel 2 data](toolshed.g2.bx.psu.edu/repos/ecology/srs_preprocess_s2/srs_preprocess_s2/0.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `sentinel_2_data.zip` (Input dataset)
>    - {% icon param-select %} *"Where does your data come from ?"*: 'From Theia'
>
> 2. Click on **Execute**
>
>    > <comment-title></comment-title>
>    >
>    > The interesting output is the ENVI image format which is a binary raster file with an accompanying header file. The data are stored as a binary stream of bytes in a BIL file without extension and the metadata are stored in the .hdr file. These data are in the output **Reflectance**. You can directly use the output **Reflectance** for the rest of the analysis.
>    {: .comment}
>
{: .hands_on}

![Preprocessing outputs 3 of them : the cloud mask folder, the reflectance folder and a file with the mission source](../../images/remote_sensing/preproc.png "Preprocessing outputs")


> <question-title></question-title>
>
> 1. What are the files you are interested in for the following tools ?
>
> > <solution-title></solution-title>
> >
> > 1. The 2 files in the **Reflectance** folder that finish by "_Refl" and "_Refl.hdr"
> >
> {: .solution}
>
{: .question}

# Producing biodiversity indicators

You can choose to compute spectral and biodiversity indicators either for global remote sensing data or for a canopy.

{% include _includes/cyoa-choices.html option1="Global" option2="Canopy" default="Global"
       text="Here you can choose which tutorial you want to folllow according to if your are more interested about studying canopy remote sensing data or more global ones" %}


<!-- Choose your own adventure path -->
<div class="Global" markdown="1">

## Compute a PCA

> <hands-on-title>Principal components analysis for remote sensing data</hands-on-title>
>
> 1. {% tool [Compute a PCA](toolshed.g2.bx.psu.edu/repos/ecology/srs_pca/srs_pca/0.0.1) %} with the following parameters:
>    - {% icon param-select %} *"In which format are your data ?"*: 'The data you are using are in a zip folder Reflectance'
>      - {% icon param-file %} *"Input data"*: `Reflectance` (output of **Preprocessing sentinel 2 data** {% icon tool %})
>    - {% icon param-select %} *"Do you want to do a PCA or a SPCA ?"*: 'PCA'
>
>
>    > <tip-title>If you want to use your own files ENVI BIL</tip-title>
>    >
>    > Check that the "Input raster" datatype is bil and that "Input raster header" datatype is hdr
>    >
>    > * Go on your raster data
>    > * Click on {% icon galaxy-pencil %} to edit it
>    > * Click on {% icon galaxy-chart-select-data %} Datatypes
>    > * On "New Type" **Select** bil
>    > * Press **Save**
>    >
>    >
>    >    > <comment-title></comment-title>
>    >    >
>    >    > Do the same for the raster header with the datatype hdr
>    >    {: .comment}
>    >
>    {: .tip}
>
{: .hands_on}

## Compute biodiversity indices

> <hands-on-title>Biodiversity indicators for global remote sensing data</hands-on-title>
>
> 1. {% tool [Compute biodiversity indices](toolshed.g2.bx.psu.edu/repos/ecology/srs_global_indices/srs_global_indices/0.0.1) %} with the following parameters:
>    - {% icon param-select %} *"In which format are your data ?"*: 'The data you are using are in a zip folder Reflectance'
>      - {% icon param-file %} *"Input data"*: `Reflectance` (output of **Preprocessing sentinel 2 data** {% icon tool %})
>    - {% icon param-text %} *"Write a number of the value of alpha"*: '1'
>
>
>    > <tip-title>If you want to use your own files ENVI BIL</tip-title>
>    >
>    > Check that the "Input raster" datatype is bil and that "Input raster header" datatype is hdr
>    >
>    > * Go on your raster data
>    > * Click on {% icon galaxy-pencil %} to edit it
>    > * Click on {% icon galaxy-chart-select-data %} Datatypes
>    > * On "New Type" **Select** bil
>    > * Press **Save**
>    >
>    >
>    >    > <comment-title></comment-title>
>    >    >
>    >    > Do the same for the raster header with the datatype hdr
>    >    {: .comment}
>    >
>    {: .tip}
>
>
>    > <tip-title>You can use the output of **Compute a PCA**</tip-title>
>    >
>    > - {% icon param-select %} *"In which format are your data ?"*: 'Your already have the files in ENVI BIL format'
>    > - {% icon param-file %} *"Input raster"*: `PCA raster` (output of **Compute a PCA** {% icon tool %})
>    > - {% icon param-file %} *"Input header"*: `PCA header` (output of **Compute a PCA** {% icon tool %})
>    > - {% icon param-text %} *"Write a number of the value of alpha"*: '1'
>    >
>    >
>    > Here again check that the "Input raster" datatype is bil and that "Input raster header" datatype is hdr
>    >
>    {: .tip}
>
{: .hands_on}

![Global biodiveristy tabular : a table summary with a column for each indicators calculated, the longitude and the latitude](../../images/remote_sensing/rasterdivtab.png "Global biodiveristy tabular")
![Global biodiveristy graph : angraph example for the shannon indice shown according to the longitude and latitude](../../images/remote_sensing/rasterdiv.png "Global biodiveristy graph")

These 2 pictures are only exemple it is normal if you don't have exactly the same output.


> <question-title></question-title>
>
> 1. How many biodiversity indicators do you have ?
>
> > <solution-title></solution-title>
> >
> > 1. You should have 7 of them (Shannon, Renyi, Prao, Pielou, Hill, CRE, Berger-Parker). If you have only 5 of them, no problem, it just means you data are too small to compute CRE and Pielou but you can still continue your analysis.
> >
> {: .solution}
>
{: .question}

</div> <!-- end CYOA path -->


<!-- Choose your own adventure path -->
<div class="Canopy" markdown="1">

## Compute a PCA

> <hands-on-title>Principal components analysis for remote sensing data</hands-on-title>
>
> 1. {% tool [Compute a PCA](toolshed.g2.bx.psu.edu/repos/ecology/srs_pca/srs_pca/0.0.1) %} with the following parameters:
>    - {% icon param-select %} *"In which format are your data ?"*: 'The data you are using are in a zip folder Reflectance'
>      - {% icon param-file %} *"Input data"*: `Reflectance` (output of **Preprocessing sentinel 2 data** {% icon tool %})
>    - {% icon param-select %} *"Do you want to do a PCA or a SPCA ?"*: 'PCA'
>
>
>    > <tip-title>If you want to use your own files ENVI BIL</tip-title>
>    >
>    > Check that the "Input raster" datatype is bil and that "Input raster header" datatype is hdr
>    >
>    > * Go on your raster data
>    > * Click on {% icon galaxy-pencil %} to edit it
>    > * Click on {% icon galaxy-chart-select-data %} Datatypes
>    > * On "New Type" **Select** bil
>    > * Press **Save**
>    >
>    >
>    >    > <comment-title></comment-title>
>    >    >
>    >    > Do the same for the raster header with the datatype hdr
>    >    {: .comment}
>    >
>    {: .tip}
>
>
{: .hands_on}

## Create Biodiversity maps

> <hands-on-title>Biodiversity indicators for canopy remote sensing data</hands-on-title>
>
> 1. {% tool [Mapping diversity](toolshed.g2.bx.psu.edu/repos/ecology/srs_diversity_maps/srs_diversity_maps/0.0.1) %} with the following parameters:
>    - {% icon param-select %} *"In which format are your data ?"*: 'The data you are using are in a zip folder Reflectance'
>      - {% icon param-file %} *"Input data"*: `Reflectance` (output of **Preprocessing sentinel 2 data** {% icon tool %})
>    - {% icon param-select %} *"Alpha, beta, functional diversity and comparison plot and map"*: 'All of the above'
>
>
>    > <tip-title>If you want to use your own files ENVI BIL</tip-title>
>    >
>    > Check that the "Input raster" datatype is bil and that "Input raster header" datatype is hdr
>    >
>    > * Go on your raster data
>    > * Click on {% icon galaxy-pencil %} to edit it
>    > * Click on {% icon galaxy-chart-select-data %} Datatypes
>    > * On "New Type" **Select** bil
>    > * Press **Save**
>    >
>    >
>    >    > <comment-title></comment-title>
>    >    >
>    >    > Do the same for the raster header with the datatype hdr
>    >    {: .comment}
>    >
>    {: .tip}
>
{: .hands_on}

![A graph showing the beta diversity according to the longitude and the latitude](../../images/remote_sensing/beta.png "Beta diversity map")
![A tabular with a column for the beta diversity, the longitude and the latitude](../../images/remote_sensing/betatab.png "Beta diversity tabular")

## Processing remote sensing data

> <hands-on-title>Comparing biodiversity indicators for canopy</hands-on-title>
> 1. {% tool [Comparing remote sensing data](toolshed.g2.bx.psu.edu/repos/ecology/srs_process_data/srs_process_data/0.0.1) %} with the following parameters:
>    - {% icon param-select %} *"In which format are your data ?"*: 'The data you are using are in a zip folder Reflectance'
>      - {% icon param-file %} *"Input data"*: `Reflectance` (output of **Preprocessing sentinel 2 data** {% icon tool %})
>    - {% icon param-file %} *"Plots folder zip"*: `output` (Input dataset)
>
>    > <comment-title>Shapefiles</comment-title>
>    >
>    > Here you must provide your folder of shapefiles (at least 2 in order to have the beta diversity).
>    {: .comment}
>
>
>    > <comment-title>Check datatype if you use your own ENVI BIL files</comment-title>
>    >
>    > Same as the compute spectral indices make sure you have the right datatypes bil and hdr.
>    {: .comment}
>
{: .hands_on}

![Biodiversity comparison graph allowing to locate the different types of vegetation in the PCoA space](../../images/remote_sensing/comp_bdmr.png "Biodiversity comparison graph")
![Bray curtis table for the types of vegetation](../../images/remote_sensing/braycu.png "Bray curtis table")


> <question-title></question-title>
>
> 1. What kind of data do you need to use these tools ?
> 2. Do you need a shapefile for mapping the diversity ?
> 3. Why do you need multiple locations for comparing biodiversity ?
>
> > <solution-title></solution-title>
> >
> > 1. This analisys is for data on forest, it's a canopy study.
> > 2. No, only for the comparison with in situ data.
> > 3. The Bray curtis table compare the diversity between locations.
> >
> {: .solution}
>
{: .question}

</div> <!-- end CYOA path -->


# Spectral indices

> <hands-on-title>Compute spectral indices</hands-on-title>
>
> 1. {% tool [Compute spectral indices](toolshed.g2.bx.psu.edu/repos/ecology/srs_spectral_indices/srs_spectral_indices/0.0.1) %} with the following parameters:
>    - {% icon param-select %} *"In which format are your data ?"*: 'The data you are using are in a zip folder Reflectance'
>      - {% icon param-file %} *"Input data"*: `Reflectance` (output of **Preprocessing sentinel 2 data** {% icon tool %})
>
>    - {% icon param-select %} *"Input the type of indice you want"*: 'NDVI'
>
>    - {% icon param-select %} *"Do you want the raster layer of the indice as an output ?"*: 'No'
>
>
>    > <tip-title>If you want to use your own files ENVI BIL</tip-title>
>    >
>    > Check that the "Input raster" datatype is bil and that "Input raster header" datatype is hdr
>    >
>    > * Go on your raster data
>    > * Click on {% icon galaxy-pencil %} to edit it
>    > * Click on {% icon galaxy-chart-select-data %} Datatypes
>    > * On "New Type" **Select** bil
>    > * Press **Save**
>    >
>    >
>    >    > <comment-title></comment-title>
>    >    >
>    >    > Do the same for the raster header with the datatype hdr
>    >    {: .comment}
>    >
>    {: .tip}
>
>    > <comment-title></comment-title>
>    >
>    > You can choose whichever indice you want
>    >
>    {: .comment}
>
{: .hands_on}


![A tabular with columns for the NDVI index, the longitude and the latitude](../../images/remote_sensing/ndvi.png "Normalized different vegetation index")


> <details-title>Spectral indices and Essential Biodiversity Variables</details-title>
>
> Remotely sensed diversity is consistent with most of the essential spatially constrained biodiversity variables proposed by Skidmore et al. (2015). This highlights the need for increased dialogue and collaboration between the biodiversity monitoring community and the remote sensing community to make satellite remote sensing a tool of choice for conservation. Increased dialogue is also essential within the biodiversity monitoring community to achieve this. From this point of view multiple Satellite Remote Sensing EBV (SRS EBV) were created.
> Some of the indices proposed here will allow you to compute  SRS EBV.
> For instance it allows you to compute one of GEO BON EBV [Canopy Chlorophyll Content](https://portal.geobon.org/ebv-detail?id=13). This EBV is computed by GEO BON on the Netherlands, here you can compute it on which ever Sentinel 2 data you want by chosing to calculate the indice CCCI.
{: .details}

![A tabular showing the Canopy Chlorophyl Content index according to the longitude and latitude](../../images/remote_sensing/ccci.png "Canopy Chlorophyl Content index tabular")

> <question-title></question-title>
>
> 1. What's the difference between biodiversity indicators and spectral indices ?
>
> > <solution-title></solution-title>
> >
> > 1. Biodiversity indicators give us informations on the heterogeneity of the landscape whereas spectral indices inform us on the well being of the vegetation.
> >
> {: .solution}
>
{: .question}


# Conclusion

You are now all set to use your remote sensing data in order to do a biodiversity analysis. Before you go on one last reflexion.

> <question-title></question-title>
>
> 1. Should remote sensing replace in-situ data ?
>
> > <solution-title></solution-title>
> >
> > 1. NO ! remote sensing and in-situ data should come and complete one another to have the most complete view of the state of biodiversity.
> >
> {: .solution}
>
{: .question}

tools_json:
["Preprocessing sentinel 2 data", "toolshed.g2.bx.psu.edu/repos/ecology/srs_preprocess_s2/srs_preprocess_s2/0.0.1", "Compute a PCA", "toolshed.g2.bx.psu.edu/repos/ecology/srs_pca/srs_pca/0.0.1", "Compute biodiversity indices", "toolshed.g2.bx.psu.edu/repos/ecology/srs_global_indices/srs_global_indices/0.0.1", "Mapping diversity", "toolshed.g2.bx.psu.edu/repos/ecology/srs_diversity_maps/srs_diversity_maps/0.0.1", "Comparing remote sensing data", "toolshed.g2.bx.psu.edu/repos/ecology/srs_process_data/srs_process_data/0.0.1", "Compute spectral indices", "toolshed.g2.bx.psu.edu/repos/ecology/srs_spectral_indices/srs_spectral_indices/0.0.1"]


topic: ecology
tutorial: ecoregiolifetraits
title: Life Traits Ecoregionalization workflow
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: ["biodiversity", "ecoregions", "life traits"]
content_hash: 19436ce89c279aac7babf28275f48bf6
last_modified: 2026-05-27T09:01:48.399541
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/ecoregiolifetraits/tutorial.html

content:
This tutorial is designed to guide you through the Ecoregionalization Galaxy workflow, demonstrating how to create a life traits ecoregionalization map from occurrences and environmental data using a boosted regression tree model for predictions.

The workflow, consisting of six tools, is intended for processing occurrence data, which should include latitude, longitude, species presence and scientific name or life traits (if life traits are absent they will be deduced automatically with scientific names). The tutorial will provide a detailed explanation of inputs, workflow steps, and outputs. This tutorial gives a practical example, highlighting a use case centered on the Kerguelen plateau and Echonoids species.

The primary goal of this workflow is to generate life traits distribution maps and identify ecoregions within the study area. The project’s objective is to offer accessible, reproducible, and transparent IT solutions for processing and analyzing life traits occurrence data. 

Note: if you're data already contains life traits, use {% link topics/ecology/tutorials/Ecoregionalization_tutorial/tutorial.md %} instead, the process is the same with taxa and life traits. You can also use this one skipping the two "WoRMS" dedicated steps.

This workflow is therefore composed of 6 tools:
 - WormsMeasurments
 - GeoNearestNeighbor
 - BRT prediction tool
 - TaxaSeeker
 - ClaraGuess
 - EcoMap

And in this tutorial we will be using 2 more tools to format data before running the ecoregionalization workflow:
 - Advanced cut
 - Filter


Let’s delve into the tutorial, outlining each step and tool to manage the creation of ecoregionalization maps.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
{: .agenda}

# Before starting

This part will present the type of data you need to run the life traits ecoregionalization workflow. This data will be downloaded in the next part of the tutorial.

## Environmental data

To run this workflow, you will first need environmental data. This workflow accepts several types of environmental parameters like temperature or soil type. However, there is a mandatory file format, the tabular format (.tsv or .tabular), and each pixel of the study area must be described in this file by associating latitude and longitude with environmental parameters.

In the use case presented in this tutorial, seventeen abiotic and physical parameters of the Dumont D’Urville sea region are used. They were taken from oceanographic models and in situ measurements Hemery et al. 2011. The physical oceanographic parameters are mean temperature and its standard deviation, mean salinity and its standard deviation, mean current magnitude and its standard deviation, maximum current bearing, maximum current magnitude and sea ice production. Bathymetric parameters are depth, slope, and rugosity. Finally, the seabed substrate composition was characterized by percentages of biogenic carbonate, biogenic silica, gravel, sand, and mud.

Here an example of environmental file input:

| long   | lat    | Carbo | Grav  | … |
|--------|--------|-------|-------|---|
| 139.22 | -65.57 | 0.88  | 28.59 | … |
| 139.25 | -65.63 | 0.88  | 28.61 | … |
| …      | …      | …     | …     | … |

## Occurrence data

The second data file you will need to run this workflow is an occurrences data file. As defined above, occurrences data are showing the presence of a species or life trait at a particular location. This data file also needs to be in tabular format (.tsv or tabular) and need to be constructed as following:
 - latitude and longitude columns
 - scientific name column
You're occurrence data shouldn't contain absence of a species, if so, you have to filter the absence. If you're occurrences contain abundance data, it won't be taken into account.

Here an example of occurrences data file input:
| lat   | long  | Planktotrophic | Lecitotrophic | Size |
|-------|-------|----------------|---------------|------|
| -65,9 | 142,3 | 1              | 0             | 3    |
| -66,3 | 141,3 | 0              | 1             | 2    |
| …     | …     | …              | …             | …    |

For this tutorial, occurrences data from the Kerguelen plateau will be downloaded from the GBIF. These data were collected as part of the PROTEKER program (Protecting the biodiversity of the Kerguelen Plateau) led by the Institut Paul-Émile Victor (IPEV, program 1044) between 2013 and 2015. Prior to their publication in the PANGAEA database and archiving in scientific repositories, these data originated from standardized sampling campaigns conducted around the Kerguelen Islands. The dataset includes occurrence records and associated environmental measurements such as temperature, salinity, and substrate type. Sampling focused on benthic macrofauna in shallow coastal habitats. Only occurrences identified to at least the genus level and sampled by scuba diving or grab sampling were retained for this tutorial. The original dataset is published under open license and fully documented in Eléaume *et al.* (2017).

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (example: “Ecoregionalization workflow”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import occurrence data file from [GBIF](https://api.gbif.org/v1/occurrence/download/request/0012077-250515123054153.zip), environment file from [InDoRES](https://data.indores.fr/file.xhtml?persistentId=doi:10.48579/PRO/LQMHX9/CYSKQG&version=1.0#)
>
>    Occurrence data file as a zip file where you will find "occurrence.txt"
>    ```
>    https://api.gbif.org/v1/occurrence/download/request/0012077-250515123054153.zip
>    ```
>    Environemental data : "ceamarc_env.tab"
>    ```
>    https://data.indores.fr/api/access/datafile/9777
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_remote_files.md %}
>
> 3. The GBIF link allows you to download a zip file containing multiple information files about the dataset. The file that you need is the 'occurrence.txt' file that we will specifically extract on Galaxy in the following step.
>
> 4. Use  {% tool [Unzip a file](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} to create a data collection in your history where all GBIF archive files will be unzipped
>
> 5. Unhide the "occurence.txt" data file then modify datatype to select the "tabular" one
>
>    {% snippet faqs/galaxy/datasets_unhidden.md %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 6. Rename the datasets if needed, notably "9777" by "ceamarc_env.tab".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 7. Check that the datatype of the environment file and occurrence file are tabular.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

# Data formatting

The first step of this tutorial is data formatting because the GBIF species occurrence file download needs to be in a specific format to be included inside the Ecoregionalization workflow.

## Keep data columns we need with Advanced Cut

> <hands-on-title> Select the columns we need </hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `occurrence` (Input dataset)
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>    - *"Delimited by"*: `Tab`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `c['98, '99', '149]`
>
>    > <comment-title> Issue with the column numbers </comment-title>
>    >
>    > It is possible the column numbers are changing in the occurence file, so don't hesitate to check the name of the column header, and if relevant, use the search functionnality to find the columns containing "latitude", "longitude" and "scientific name" informations.
>    >
>    {: .comment}
> 
> 
{: .hands_on}

## Clean scientific names for WoRMS API request

This step is used tobe sure scientific names are written in the manner WoRMS API is able to manage it (only genus and species name or genus alone).

> <hands-on-title> Clean taxons names for WoRMS API </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `output` (output of **Advanced Cut** {% icon tool %})
>    - *"using column"*: `c3`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `^([A-Z][^A-Z(]+)((?<!^).*)$`
>            - *"Replacement"*: `\1`
>    - In the second *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: ` $`
>            - *"Replacement"*: ``
>
>
{: .hands_on}


## Get life traits from WoRMS
> <hands-on-title>Life traits acquisition</hands-on-title>
>
> 1. {% tool [Enrich dataset with WoRMS](toolshed.g2.bx.psu.edu/repos/ecology/wormsmeasurements/WormsMeasurements/0.1.1) %}: add life traits to the dataset by
>    - *"occurrence data"*: a `occurrence` tabular file that fills the requirements mentionned earlier
>    - *"list of measurement types"*: a list of the life traits of interest like `Development`. Make they have the same spelling as on the WoRMS and with a capital letter. 
>    - *"scientific name column name"*: look at your occurrence file, if the name of the column of scientific names is different from `scientificName` replace by the correct one.
>    - *"include attributes inherited from parent taxon"*: WoRMS doesn't have all life traits informations and some Echinoid species have very limited informations. To avoid having too many missing values you can switch to on, if you don't know, run your workflow once, check the output of the next tool and if the size of the dataset decreased too much switch to yes. Warning, you need to have an idea of how many different life trait values you're going to have, the workflow will generate one map per life trait value, the number of life trait values will impact calculation time.
>    - *"one hot encoding on the measurement types"*: make sure it's enabled.
{: .hands_on}

## Remove lines with NAs or empty values with Filter

> <hands-on-title> Remove lines with NAs </hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"With following condition"*: enable and paste thix expression `c1!='' and c2!='' and c3!='' and c4!='' and c1!='NA' and c2!='NA' and c3!='NA' and c4!='NA'`. It removes lines with missing value corresponding with latitude, longitude, scientificName, Development_planktotrophic, if your dataset has a special value for missing values like -9999, add ` and c1!='missingvalue' and c2!='missingvalue' {% icon tool %})
>    - In *"Number of header lines to skip"* enter `1`
> 2. Check your output. All the lines with missing values must have been deleted.
> 3. Rename your file "corrected_occurence.tsv"
>
{: .hands_on}

# Ecoregionalization workflow

Now you have all you need to run the ecoregionalization Workflow :
 - Occurrence formatted file
 - Environment file

## Merge environment file and occurrence file with GeoNearestNeighbor

### What it does ?

This Galaxy tool allows you to merge two data tables (tabular format only) according to their latitude and longitude coordinates (in WGS84 projection), finding the closest points. This tool is used in the Ecoregionalization workflow to filter the occurence data file so it can be used on the BRT prediction tool.

### How to use it ?

> <hands-on-title> Run GeoNearestNeighbor </hands-on-title>
>
> 1. {% tool [GeoNearestNeighbor](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_geonearestneighbor/ecoregion_GeoNearestNeighbor/0.1.0+galaxy0) %} with the following parameters:
>    - In *"Your environment file (or table 1)"*:
>        - {% icon param-file %} *"Input your environment data file (tabular format only)"*: `ceamarc_env.tsv` (Input dataset)
>        - *"Choose columns where your latitude is in your environment data file."*: `c2`
>        - *"Choose columns where your longitude is in your environment data file."*: `c1`
>    - In *"Your occurrence file (or table 2)"*:
>        - {% icon param-file %} *"Input your occurrence data file (tabular format only)"*: `corrected_occurence.tsv` (output of previous **Filter** step {% icon tool %})
>        - *"Choose columns where your latitude is in your occurrence data file."*: `c1`
>        - *"Choose columns where your longitude is in your occurrence data file."*: `c2`
>
>    > <comment-title> Coords precision </comment-title>
>    >
>    > It is recommended that, for optimal precision, the latitude and longitude values in both files should be of the same precision level. And, for the sake of relevance, the geographical coordinates in both files should be as close as possible to apply the most accurate environmental parameters to the correct species occurrences.
>    >
>    {: .comment}
>
> 2. Check your outputs. You must have two files:
>       - Information file containing the coordinates of occurrence data, the coordinates retains from environmental data and the distances between the two.
>       - Occurrence and Environment merge file containing occurrence data and environmental data corresponding.
>
{: .hands_on}

## Clean merged table file

To use next tool, BRT, one need to have a clean merged table datafile. To do so, you can here remove the "scientificName" column, as BRT tool merged data input must only have numeric values columns.



> <hands-on-title> Run the BRT tool </hands-on-title>
>
> 1. {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file"*: `Merged table` (Input dataset)
>    - {% icon param-file %} *"Select Columns"*: `scientificName` (output of **GeoNearestNeighbor** {% icon tool %})
>
>
{: .hands_on}





## Predicting life traits with BRT tool prediction

This step implements a commonly used approach in ecological studies, namely species distribution modelling (SDM). This allows to characterize the distribution of each life traits by giving an indicator of probability of life trait presence for each environmental layer pixel. Here, the boosted regression trees (BRT) method was used to adjust the relationship between the presence of a single life trait and the environmental conditions under which the life trait has been detected. BRT modelling is based on an automatic learning algorithm using iterative classification trees.

### What it does ?

Two treatments are performed in this tool: the creation of the taxa distribution model and the use of this model to obtain a prediction index. The prediction index obtained from each BRT model for each pixel of the environmental layers is an approximation of the probability of detection of the presence of the taxon.

This tool gives as output a file containing the predictions of the probability of the presence of each taxon for each "environment pixel" (latitude, longitude), a visualization of these pixels for each taxon and graphs showing the percentage of model explanation for each environmental parameter. We're gonna go back to this in the following steps.

### How to use it ?

> <hands-on-title> Run the BRT tool </hands-on-title>
>
> 1. {% tool [BRT tool prediction](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_brt_analysis/ecoregion_brt_analysis/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input your environment data file of your study area (tabular format only)"*: `ceamarc_env.tsv` (Input dataset)
>    - {% icon param-file %} *"Input your occurrences data file(s) containing also the environmental characteristics where the species has been observe (tabular format only)"*: `Merged table` (output of **GeoNearestNeighbor** cleaned for extra column with **Remove column** tool if relevant {% icon tool %})
>
>      {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
>    - *"Choose column(s) where your abiotic parameter are in your environment data file."*: `c['3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19']`
>
>    > <warning-title>You maybe can here prepare a cup of tea ;)</warning-title>
>    >
>    > This step is a first "long" one, and you have to wait something like 18 minutes and 34 sec to obtain the result. It is maybe the good moment to have a break and go outside to admirate biodiversity around you?
>    >
>    {: .warning}
>
>
> 2. Check your outputs. You must have four outputs collections.
>
>    - Prediction files
>    - Validation files (life traits, AUC, Tree complexity, Total deviance explained)
>    - Species distribution prediction maps
>    - Partial dependence plots
>
{: .hands_on}

In the 'Prediction files' collection there must be a file containing predictions of the probability of the presence of each life trait for each "environmental pixel" (latitude, longitude) for each occurrence file you entered.

In the 'Validation files' collection there must be a file containing for each life traits the validation metrics of the associated model.

In the 'Species distribution prediction maps' collection there must be for each life traits a map representing their probability of presence at each environmental layer pixel.

![Life traits distribution map](./Images/BRT_Development_planktotrophic _pred_plot.png "Development planktotrophic distribution from BRT")

In the 'Partial dependence plots' collection there should be graphs showing the percentage explanation of the model for each environmental parameter.
Here is an example:

![Partial dependence plots](./Images/BRT_Development_direct.development.pdf "Percentage of explanation of the model for each environmental parameter")


## Collecting the list of life traits with TaxaSeeker

You may wonder "but shouldn't this tool be used only with taxon data?" it's commonly used for taxa but it works also with life traits.

### What it does ?

This tool does three things:

- It allows obtaining a summary file for each taxon indicating whether a BRT model was obtained and the number of occurrences per taxon.

- It provides a list of taxa that obtained cleaned BRT models (without "_", "_sp", etc.) to propose the list to WoRMS (World Register of Marine Species) or another taxonomic database and obtain more information about the taxa.

- It generates a list of taxa for which a BRT model was obtained, needed as input of following steps.

### How to use it ?

> <hands-on-title> Run TaxaSeeker </hands-on-title>
>
> 1. {% tool [TaxaSeeker](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_taxa_seeker/ecoregion_taxa_seeker/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Environment file (tabular format only)"*: `ceamarc_env.tsv` (Input dataset)
>    - {% icon param-file %} *"Occurences file(s) (tabular format only)"*: `Merged table` (output of **GeoNearestNeighbor** {% icon tool %})
>    - {% icon param-file %} *"Predictions file(s)"*: `Prediction files collection` (output of **BRT tool prediction** {% icon tool %})
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    2. Check your outputs. You must have three files:
>
>    - Summary of taxa model
>    - List of taxa
>    - List of taxa clean
>
{: .hands_on}

## Determine the optimal number of cluster and build ecoregional clusters with ClaraGuess

### What it does ?
This tool applies the CLARA clustering method to identify environmental clusters based on:
- BRT model predictions (a collection of tabular files),
- environmental variables (tabular),
- a list of taxa (tabular, from TaxaSeeker).
The tool enables the determination of the optimal number of clusters for partition-based clustering (if automatic mode is selected), along with generating files used in the subsequent ecoregionalization workflow.


> <hands-on-title> Run ClaraGuess </hands-on-title>
>
> 1. {% tool [ClaraGuess](toolshed.g2.bx.psu.edu/repos/ecology/claraguess/ClaraGuess/0.1.2) %} with the following parameters:
>    - {% icon param-file %} *"Environmental data (tabular)"*: `ceamarc_env.tsv` (Input dataset)
>    - {% icon param-file %} *"BRT prediction files (collection of tabular)"*: `Prediction files` (output of **BRT tool** {% icon tool %})
>    - {% icon param-file %} *"List of taxas (from TaxaSeeker)"*: `List of taxa` (output of **TaxaSeeker** {% icon tool %})
>    - *"k is ..."*: either the number of clusters of the model (the number of ecoregions) or if you don't how many cluster/ecoregions you want, the tool will find the more optimal number of clusters. The first time, this parameter should be set to automatic so you can see the plot of silhouette scores and determine yourself what is the best number of clusters to choose. The best number of clusters according to silhouette score isn't always equal the number of ecoregions.
>    - {% icon param-file %} *"Value of k (collection of tabular)"*: a number greater than 2
>
>    > <comment-title> Two other parameters </comment-title>
>    >
>    > The other two parameters can be left as is. If you need to change them here is a short description of what they do:
>    >
>    > The first one is metric used to calculate the dissimilarities between the observations: Manhattan (distance between two real-valued vectors), Euclidean (shortest distance between two
>    > points) and Jaccard (defined as the size of the intersection divided by the size of the union of the sample sets)
>    >
>    > The second one is the sample size that will be used to perform clustering.
>    > Indeed, the clara function is used to cluster large data using a representative sample rather than the entire data set. This will speed up the clustering process and make the calculation
>    > more efficient. A fairly high value representative of the data is recommended. It is important to note that using a too small sample size may result in loss of information compared to using the
>    > entire data set.
>    >
>    {: .comment}
>
> 2. Check your outputs. You must have three files:
>
>    - SIH index plot (See example below.)
>    - Cluster points (Contains the latitude and longitude of each "environmental pixel" and the associated cluster number. We will use it in the next step of the workflow)
>    - Cluster info (Contains all the information related to the clusters created, that is, in column: the latitude, the longitude, the corresponding cluster number and for each taxon the prediction value)
>
{: .hands_on}

The tool will produce a silhouette plot that you can see below. A silhouette graph is a representation used to visualize the silhouette index of each observation in a clustered data set. It makes it possible to
assess the quality of clusters and determine their coherence and separation. In a silhouette graph, each observation is represented by a horizontal bar whose length is proportional to its silhouette index. The longer
the bar, the better the consistency of the observation with its cluster and the separation from other clusters. As mentioned above, the silhouette index ranges from -1 to 1. Values close to 1 indicate that objects
are well grouped and separated from other clusters, while values close to -1 indicate that objects are poorly grouped and may be closer to other clusters. A value close to 0 indicates a situation where objects are
located at the border between two clusters. Here, in the graph below, there is a good distribution of the observations because the majority of the bars are above the average value of the silhouette index.

![SIH_index_plot](./Images/SIH_index_plot.png "SIH index plot")

![SIH plot](./Images/SIH_plot.png "SIH plot")

# Build a ecoregionalization map with **EcoMap**

> <hands-on-title> Run EcoMap </hands-on-title>
>
> 1. {% tool [EcoMap](toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_eco_map/ecoregion_eco_map/0.1.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Source file (File 'cluster points' from previous step)"*: `cluster assignements` (output of **ClaraGuess** {% icon tool %})
>
> This tool simply need the previous step "cluster assignements (lat, long, cluster)" output file to generate a map representing ecoregions.
>
> 2. Check output. You must have one Map representing ecoregions.
>
{: .hands_on}

This is the output map with our seven clusters representing ecoregions that you must have obtained.

![Ecoregion map](./Images/Map.png "Ecoregion map")

# Conclusion

Congratulations! You have successfully completed the life traits ecoregionalization workflow. Here is the end of this tutorial aiming to explain the purpose of the ecoregionalization workflow and how to use it. This workflow provides a systematic and reproducible approach to ecoregionalization, allowing researchers to identify distinct ecological regions based on species occurrences and environmental data. This tutorial shows how to use this workflow, step by step, or all in one using the Dumont D'Urville sea region use case with related datasets. It allows you to understand ecoregions construction. You learned the use of the BRT algorithm for modeling species distribution as well as the cluster construction with the k-medoid clustering algorithms (CLARA/PAM). Feel free to explore and adapt this workflow for your specific research needs. If you have any questions, or encounter issues during the workflow, refer to the provided documentation or seek assistance from the Galaxy community. Don't hesitate to contact us if you have any questions.

tools_json:
["Unzip a file", "toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3", "Enrich dataset with WoRMS", "toolshed.g2.bx.psu.edu/repos/ecology/wormsmeasurements/WormsMeasurements/0.1.1", "Filter", "Filter1", "GeoNearestNeighbor", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_geonearestneighbor/ecoregion_GeoNearestNeighbor/0.1.0+galaxy0", "Remove columns", "toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0", "BRT tool prediction", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_brt_analysis/ecoregion_brt_analysis/0.1.0+galaxy0", "TaxaSeeker", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_taxa_seeker/ecoregion_taxa_seeker/0.1.0+galaxy0", "ClaraGuess", "toolshed.g2.bx.psu.edu/repos/ecology/claraguess/ClaraGuess/0.1.2", "EcoMap", "toolshed.g2.bx.psu.edu/repos/ecology/ecoregion_eco_map/ecoregion_eco_map/0.1.0+galaxy0"]


topic: ecology
tutorial: panoply_ebv
title: Visualize EBV cube data with Panoply netCDF viewer
difficulty: intermediate
time_estimation: 1H
tools_json: ["Panoply", "interactive_tool_panoply"]
requirements_json: []
tags_json: ["interactive tools", "EBV cube", "data visualization"]
content_hash: a5d9c96aef8fe69beccb3bea00cbc4fb
last_modified: 2026-05-27T09:01:48.400760
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/panoply_ebv/tutorial.html

content:
The practical aims at familiarzing you with the [Panoply](https://www.giss.nasa.gov/tools/panoply/) Galaxy interactive tool. Panoply is among the most popular tool to visualize geo-referenced data stored in [Network Common Data Form](https://en.wikipedia.org/wiki/NetCDF) (netCDF). It provides a graphical interface for inspecting (show metadata) and visualizing netCDF data. It supports many features to customize your plots and we will introduce some of them in this lesson.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Panoply documentation](https://www.giss.nasa.gov/tools/panoply/help/) and the [Galaxy climate dedicated tutorial]({% link topics/climate/tutorials/panoply/tutorial.md %}).
>
{: .comment}


In this tutorial, you will learn to:
- Plot EBV cube arrays.
- Use any of numerous color tables for the scale colorbar
- Save plots to GIF, JPEG, PNG or TIFF bitmap images or as PDF or PostScript graphics files.
- Export animations as MOV or AVI videos


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
> In this tutorial, we will be focusing on the usage of Biodiversity data in [Network Common data Form](https://en.wikipedia.org/wiki/NetCDF) (netCDF) because it is the data format used to store data on the EBV data portal.
> We will be using a freely available dataset representing Essential Biodiversity Variables from [GEO BON data portal](https://portal.geobon.org/home). We will learn to use panoply to visualize the Local bird diversity for last century.
{:  .comment}

## NetCDF format

[NetCDF](https://en.wikipedia.org/wiki/NetCDF) data format is a binary format and to be able to read or visualize it, we would need to use dedicated software or libraries that can handle this "special" format. It is self-describing and machine-independent data format that supports the creation, access, and sharing of array-oriented scientific data. NetCDF files usually have the extension *.nc* or *.netcdf*.

For climate and forecast data stored in NetCDF format there are (non-mandatory) conventions on metadata ([CF Convention](http://cfconventions.org/)).

In this tutorial, we will be using data from the [GEO BON EBV data portal](https://portal.geobon.org/).

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *Panoply* for example...
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the file from [EBV data portal](https://portal.geobon.org/data/upload/1/public/martins_comcom_id1_20220208_v1.nc)
>
>    ```
>    https://portal.geobon.org/data/upload/1/public/martins_comcom_id1_20220208_v1.nc
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Check that the datatype is **netcdf**
>
>    Files you uploaded are in netcdf format. In Galaxy, Datatypes are, by default, automatically guessed. Here, as netcdf is a derivative of the h5 format, Galaxy automatically affect the h5 datatype to netcdf files. To cope with that, one can change the datatype manually, once datasets uploaded (as shown below) OR you can directly specify datatype on the upload tool form so Galaxy will not try to automatically guess it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. **Rename Datasets** {% icon galaxy-pencil %}
>
>    As the original name `martins_comcom_id1_20220208_v1.nc` can be not so good to use, don't hesitate to modify it in `Local bird diversity (cSAR/BES-SIM) martins dataset` for example.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. **Add a tag** {% icon galaxy-tags %} to the dataset corresponding to `#EBV`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Panoply

## Opening up Panoply

> <hands-on-title>Launch Panoply</hands-on-title>
>
>  Panoply is available as a Galaxy interactive tool and may not be available on all Galaxy servers.
>
> > <tip-title>Launch Panoply in Galaxy</tip-title>
> > Currently Panoply in Galaxy is available on useGalaxy.eu instance, on the "Interactive tools" tool panel section.
> >
> > 1. Open the {% tool [Panoply](interactive_tool_panoply) %} by clicking [on the Panoply interactive tool shortcut](https://usegalaxy.eu/?tool_id=interactive_tool_panoply)
> > 2. Check **martins_comcom_id1_20220208_v1.nc** dataset selected in the netcdf input field
> > 3. Click Run Tool
> > 4. The tool will start running and will stay running permanently
> > 5. Click on the "User" menu at the top and go to "Active Interactive Tools" and locate the Panoply instance you started.
> > 6. Click on your Panoply instance
> >    ![Panoply dataset selection](../../images/select_dataset.png "Select dataset")
> > 7. Click on **martins_comcom_id1_20220208_v1.nc** dataset and **open**.
> {: .tip}
{: .hands_on}

## Inspect metadata

> <hands-on-title>Inspect dataset</hands-on-title>
>
> 1. Inspect dataset content
>
>    Here you can look at the dataset (martins_comcom_id1_20220208_v1.nc) and related variables (`crs`, `entity`, `lat`, `lon`, `metric_1`, `ebv_cube`, `time`)
>
>    > <question-title></question-title>
>    >
>    > what is the unit of the `ebv_cube` variable of `metric_1`and its shape?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The unit of **ebv_cube** is "Percentage points". **ebv_cube** is a 4D array (entity, time, latitude, longitude).
>    > {: .solution}
>    {: .question}
>
>
> 2. Take a look at the general properties of the dataset
>
>    > <question-title></question-title>
>    >
>    > Can you find the title, summary, EBV class and EBV name informations?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > Double click on "martins_comcom_id1_2022..." element of the "Name" column to display these general information
>    > > Title: Local bird diversity (cSAR/BES-SIM).
>    > > Summary: Changes in bird diversity at 1-degree resolution caused by land use, estimated by the cSAR model for 1900-2015 using LUH2.0 historical reconstruction of land-use.
>    > > EBV Class: Community composition.
>    > > EBV name: Taxonomic and phylogenetic diversity.
>    > {: .solution}
>    {: .question}
>
>
>    > <question-title></question-title>
>    >
>    > Can you find the biodviersity metrics names?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > metric_1: Relative change in the number of species (%)
>    > > metric_2: Absolute change in the number of species
>    > {: .solution}
>    {: .question}
{: .hands_on}

## Create Geo-referenced Longitude-Latitude plot

> <hands-on-title>geographical map</hands-on-title>
>
> 1. Double click on the variable **ebv_cube** from `metric_1` and click on **Create**
>    ![Panoply create Latitude longitude map](../../images/map0.PNG "Create map")
>
>    > <question-title></question-title>
>    >
>    > 1. What does it shows?
>    > 2. What is the date of the generated plot?
>    > 3. Can you plot other dates?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The plot represent the relative change in the number of species (%).
>    > >    ![Panoply Latitude longitude map](../../images/map1.PNG "Plot map")
>    > >
>    > > 2. The date of the default plot is 1st January 1900 at 00:00.
>    > >
>    > > 3. To plot another date, change either:
>    > > - Initial time of forecast (give a value between 1 and 12, corresponding to years between 1900 and 2010.
>    > > - Click on the date and scroll down to select the date of your choice.
>    > {: .solution}
>    {: .question}
>
> 2. Save your plot
>    - Click on the tab **File** (from your plot window) to store your plot by selecting **Save Image As**
>    - Double click on the folder *outputs* to enter this folder and save your plot.
>    You need to make sure to save all your plot in the *outputs* folder otherwise you can loose all your plots once to close panoply.
>
> 3. Change colormap
>    **Always make sure you use color blind friendly palettes.**
>    - To change the default colormap, click on tab "**Scale**" (bottom of your plot window) and select another "**Color Table**" (you can scroll down to go through all the different available colormap).
>    - Save your plot using **Save Image As** and make sure to choose another name to avoid overwritting your preceding plot.
>    ![Panoply colormap](../../images/map2.PNG "Plot colormap")
{: .hands_on}


## Changing map projection

> <hands-on-title>Change projection</hands-on-title>
>
> From your previous plot window, click on Tab **Map** and change **Projection**. Try a few of them and save each of your plot with **File** --> **Save Image As**.
>
>    ![Panoply change projection](../../images/map3.PNG "Change projection")
>
{: .hands_on}

## Export Animation

> <hands-on-title>Export animation</hands-on-title>
>
> 1. From your previous plot window, click on **File** and select **Export Animation**. Save your plot using either **MOV** or **AVI** format.
>  It goes through each plot e.g. for each month and create an animation where you can see the evolution of sea-ice extent from January 1979 to December 1979.
> You will be able to download the resulting movie from Galaxy once you quit Panoply.
{: .hands_on}


## Quit Panoply


> <hands-on-title>Quit Panoply to keep your plots</hands-on-title>
>
> 1. To make sure all your plots stored in **outputs** folder get exported to Galaxy, you need to quit panoply:
>  **File** --> **Quit Panoply**.
> 2. Go back to your current Galaxy history and you should find **Panoply outputs**
>  ![Panoply outputs](../../images/map4.PNG "Panoply outputs in Galaxy")
>
{: .hands_on}

# Conclusion


We have now learnt how to visualize EBV cube data using Panoply. We only use one of the two datasets so we strongly encourage you to do the same exercises with others datasets from EBV data portal.


topic: ecology
tutorial: QGIS_Web_Feature_Services
title: QGIS Web Feature Services
difficulty: intermediate
time_estimation: 0H30M
tools_json: ["dedicated form", "interactive_tool_qgis", "QGIS instance"]
requirements_json: []
content_hash: a39dbc40a1f950b8fb247c38b4c831f6
last_modified: 2026-05-27T09:01:48.394016
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/QGIS_Web_Feature_Services/tutorial.html

content:
Based on this [QGIS official tutorial](https://docs.qgis.org/2.18/en/docs/training_manual/online_resources/wfs.html), you will learn here how to access, filter and import GIS data through WFS web service using QGIS Galaxy interactive tool:

In the Geographical Information System landscape, there is existing standards to help users deal with remote data. The most common web services are Web Map Services (WMS) and Web Feature Services (WFS). If WMS allows users only to access and display maps stored remotely, WFS is giving access to the features of data so you can modify it and create your own data and maps.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Managing QGIS Galaxy interactive tool

QGIS is now integrated in Galaxy as an interactive tool. This kind of tool works differently than classical tools as it allows the user to interact with a dedicated graphical interface.

To use QGIS, you need to use the {% tool [dedicated form](interactive_tool_qgis) %}, you can specify input datasets from your history you want to use in QGIS, or not ;), then press the execute button to launch a QGIS instance. When the graphical user interface of QGIS is ready to be used, a URL will be displayed at the top of the Galaxy center panel. If you don't see it, you can see and access it through the "Active InteractiveTools" space of the "User" menu or you can click on {% icon galaxy-eye %} on the tool in the history.

Once you finish your work on QGIS, if you want to reuse data and/or the entire project, you need to save files in the "output" folder (which you can find in the "working" directory). 
Then, quit QGIS properly through the "Project" Menu tab top left and click on "Exit QGIS".

> <hands-on-title>Deploy your own QGIS instance</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Deploy a {% tool [QGIS instance](interactive_tool_qgis) %}
> 3. Access QGIS
> 4. Save your project in /working/output folder
>
{: .hands_on}

# Web Feature Services
## Loading WFS layer

> <hands-on-title>Loading WFS layer</hands-on-title>
>
> 1. Go to the top left of QGIS and click on "Layer"
> 2. Click on the 1st option "Data Source Manager" 
> 3. Select "WFS / OGC API - Features" on the left part of the pop-up panel
> 4. Click the "New" button
> 5. In the dialog that appears, enter the Name as `nsidc` and the URL as `http://nsidc.org/cgi-bin/atlas_south?version=1.1.0`.
>
>    > <tip-title>copy pasting from computer to QGIS</tip-title>
>    >
>    > You can expand the QGIS left panel (where there are 3 dots, vertically) to access the "clipboard" menu and paste the content you want to paste on a QGIS form. Then, click outside of this panel to collapse it, and you can click for example on the `url` field to paste the URL from your clipboard
>    >
>    > ![Image of QGIS interface with on the left the clipboard panel in transparent where you can use the copy paste option.](../../images/QGIS/qgistuto1.PNG)
>    {: .tip}
>
> 5. Click OK, then you can create the connection with the "connect" button to see a list of available layers
> 6. Find and select the `antarctica_country_border` (Antartica border) layer
> 7. Find and select, holding the "CTRL" button, the `south_poles_wfs` (South border) layer
> 9. Unselect "only request features overlapping the view extent" option then click "add"
>   ![Image of QGIS interface to help you see which layers to select in the WFS / OGC API features](../../images/QGIS/qgistuto3.PNG)
> 10. You now have Antarctica border displayed with a symbol showing the south pole
>   ![Image of QGIS interface showing the resulting map from the selection of your layers](../../images/QGIS/qgistuto4.PNG)
> 11. You can double click each layer at the bottom left to modify symbology, notably color for "south_poles_wfs" modifying the symbol by selecting for example "effect drop shadow"  click on "Save Symbol..." and add a label `Geographic South Pole`
>   ![Image of QGIS interface to show you to click on the button "effect drop shadow" to make the south pole more visible on your map.](../../images/QGIS/qgistuto5.PNG)
>   ![Image of QGIS interface to show you how to add a title and change the color of your south pole marker](../../images/QGIS/qgistuto6.PNG)
> 12. And to "antarctica_country_border".
>   ![Image of QGIS interface showing how to change the background or color of the geographical zone here Antarctica](../../images/QGIS/qgistuto7.PNG)
> 
{: .hands_on}

## Querying a WFS layer

Even if you can select, download and display entire WFS layers, it is often more efficient to interrogate a layer before loading it to QGIS. This is a major interest of the use of such web services, as you can save internet bandwidth by selecting only parts of layer you want in your own QGIS instance.

> <hands-on-title>Querying a WFS layer</hands-on-title>
>
> 1. In the WFS server already created in the first step of this tutorial, there is a layer called `countries (excluding Antarctica)`. If we want to know where is South Africa related to the `south_poles_wfs`, there are several manners to operate. One can load the entire layer of `countries (excluding Antarctica)` and then use it locally, or we can save bandwidth and only load locally the needed information concerning South Africa. We will here use the second manner, querying the WFS layer to obtain only information we will use in our QGIS instance.
> 2. Go to the top left of QGIS and click on "Layer"
> 3. Click on the 1st option "Data Source Manager" 
> 4. Select "WFS / OGC API - Features" then the already connected `nsidc` server
> 5. Select the `countries (excluding Antarctica)` layer and click "Build query" button
>    ![Image of QGIS interface to help you see which layers to select in the WFS / OGC API features](../../images/QGIS/qgistuto8.PNG)
> 6. On the new dialog box, you can copy/paste this query: `SELECT * FROM country_borders_excluding_antarctica WHERE "Countryeng" = 'South Africa'` using the clipboard functionality from the left QGIS panel to make a "bridge" between your local clipboard and the virtualized QGIS one.
>    ![Image of QGIS interface to help you see what to write in the "SQL query composer"](../../images/QGIS/qgistuto9.PNG)
> 7. Clicking OK, you can now see the SQL query on a dedicated column of the layers table.
>    ![Image of QGIS interface showing you the results of the SQL command on the selection of layers in the WFS / OGC API features](../../images/QGIS/qgistuto10.PNG)
> 9. Clicking the "Add" button, you now have the South Africa layer added and displayed on QGIS.
>    ![Image of QGIS interface showing the resulting map of all your changes since the beginning of the tutorial](../../images/QGIS/qgistuto11.PNG)
{: .hands_on}

> <question-title></question-title>
>
> 1. Are you seeing how QGIS shows the fact that a layer is resulting from a query made on a larger layer?
>
> > <solution-title></solution-title>
> >
> > 1. There is a dedicated "filter" icon next to the name of the layer.
> >
> {: .solution}
>
{: .question}


# Conclusion

You just did a classical GIS operation, accessing remote data using WFS webservice through QGIS, retrieving features of interest you can share broadly through Galaxy!

tags_json:
["earth-system", "GIS", "Geographical Information System", "WFS", "spatial data", "maps", "OGC", "biodiversity"]


topic: ecology
tutorial: Obitools-metabarcoding
title: Metabarcoding/eDNA through Obitools
difficulty: intermediate
time_estimation: 1H
requirements_json: []
content_hash: 5e898668c654f25d5c511d5a681911a2
last_modified: 2026-05-27T09:01:48.393540
zenodo_link: https://zenodo.org/record/5932108
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/Obitools-metabarcoding/tutorial.html

content:
Based on this [OBITools official tutorial](https://pythonhosted.org/OBITools/wolves.html), you will learn here how to analyze DNA metabarcoding data produced on Illumina sequencers using:
 * the OBITools on Galaxy
 * some classical Galaxy tools

The data used in this tutorial correspond to the analysis of four wolf scats, using the protocol published in {% cite Shehzad2012 %} for assessing carnivore diet. After extracting DNA from the faeces, the DNA amplifications were carried out using the primers TTAGATACCCCACTATGC and TAGAACAGGCTCCTCTAG amplifying the 12S-V5 region ({% cite Riaz2011 %}), together with a wolf blocking oligonucleotide.

It is always a good idea to have a look at the intermediate results or to evaluate the best parameter for each step. Some commands are designed for that purpose, for example you can use :

- obicount to count the number of sequence records in a file
- obihead and obitail to view the first or last sequence records of a file
- obistat to get some basic statistics (count, mean, standard deviation) on the attributes (key=value combinations) in the header of each sequence record (see The [extended OBITools fasta format in the fasta format description](https://pythonhosted.org/OBITools/fasta.html))
- any Galaxy tools corresponding to classical unix command such as less, awk, sort, wc to check your files.

The OBITools programs imitate Unix standard programs because they usually act as filters. The main difference with classical Unix programs is that text files are not analyzed line per line but sequence record per sequence record (see below for a detailed description of a sequence record). Compared to packages for similar purposes like mothur ({% cite Schloss2009 %}) or QIIME ({% cite Caporaso2010 %}), the OBITools mainly rely on filtering and sorting algorithms. This allows users to set up versatile data analysis pipelines

![OBITools pipeline](../../images/Obitools/fig-Pipeline.png)

Most of the OBITools commands read sequence records from a file or from the stdin, make some computations on the sequence records and output annotated sequence records. As inputs, the OBITools are able to automatically recognize the most common sequence file formats (i.e. FASTA, FASTQ, EMBL, and GenBank). They are also able to read ecoPCR ({% cite Ficetola2010 %}) result files and ecoPCR/ecoPrimers formatted sequence databases ({% cite Riaz2011 %}) as ordinary sequence files. File format outputs are more limited. By default, sequences without and with quality information are written in FASTA and FASTQ formats, respectively. However, dedicated options allow enforcing the output format, and the OBITools are also able to write sequences in the ecoPCR/ecoPrimers database format, to produce reference databases for these programs. In FASTA or FASTQ format, the attributes are written in the header line just after the id, following a key=value; format.

![OBITools attributes records](../../images/Obitools/fig-Record.png)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Manage input data
The data needed to run the tutorial are the following:

- FASTQ files resulting from a GA IIx (Illumina) paired-end (2 x 108 bp) sequencing assay of DNA extracted and amplified from four wolf faeces:
    - wolf_F.fastq
    - wolf_R.fastq
- the file describing the primers and tags used for all samples sequenced:
    - wolf_diet_ngsfilter.txt
- the tags correspond to short and specific sequences added on the 5’ end of each primer to distinguish the different samples
- the file containing the reference database in a fasta format:
    -db_v05_r117.fasta This reference database has been extracted from the release 117 of EMBL using ecoPCR

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the zip archive containing input files from [Zenodo](https://zenodo.org/record/5932108/files/wolf_tutorial.zip)
>
>    ```
>    https://zenodo.org/record/5932108/files/wolf_tutorial.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the dataset, here a zip archive, if needed
> 4. Check that the datatype is `zip`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

## **Unzip** the downloaded archive

> <hands-on-title>Unzip the downladed .zip archive and prepare unzipped files to be used by OBITools</hands-on-title>
>
> 1. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/0.2) %} with the following parameters:
>    - *"Extract single file"*: `All files`
>
>    > <comment-title></comment-title>
>    >
>    > To work properly, this unzip Galaxy tool is waiting "simple" archive as input, this means without sub directory.
>    {: .comment}
>
> 2. Add to each datafile a tag and/or modify names (*optional*)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 3. Unhide all dataset from the resulting data collection so you can use these files independently.
>
> 4. Modify datatype from txt to tabular for the `wolf_diet_ngsfilter` dataset
{: .hands_on}


> <question-title></question-title>
>
> 1. Why do we need to unhide manually datasets from the data collection?
>
> > <solution-title></solution-title>
> >
> > 1. Data collection is a functionality often used to deal with multiple datasets on the same format who can be analysed in batch mode. Here, the data collection is populated with heterogenous datafiles, coming from an archive. We thus need to treat separately each dataset of the collection, and to do so, we need to unhide corresponding datasets from the history, as datasets inside collections are just like "symbolic link" to "classical" history datasets hidden by default.
> >
> {: .solution}
>
{: .question}




# Use OBITools

OBITools ({% cite Boyer2015 %}) is a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context, taking into account taxonomic information. It is distributed as an open source software available on the following website: [http://metabarcoding.org/obitools](http://metabarcoding.org/obitools).

The OBITools commands consider a sequence record as an entity composed of five distinct elements. Two of them are mandatory, the identifier (id) and the DNA or protein sequence itself. The id is a single word composed of characters, digits, and other symbols like dots or underscores excluding spaces. Formally, the ids should be unique within a dataset and should identify each sequence record unambiguously, but only a few OBITools actually rely on this property. The three other elements composing a sequence record are optional. They consist in a sequence definition, a quality vector, and a set of attributes. The last element is a set of attributes qualifying the sequence, each attribute being described by a key=value pair. The set of attributes is the central concept of the OBITools system. When an OBITools command is run on the sequence records included in a dataset, the result of the computation often consist in the addition of new attributes completing the annotation of each sequence record. This strategy of sequence annotation allows the OBITools to return their results as a new sequence record file that can be used as the input of another OBITools program, ultimately creating complex pipelines (source: [OBITools Welcome](https://pythonhosted.org/OBITools/welcome.html)).


## Micro assembly of paired-end sequences with **illuminapairedend**

When using the result of a paired-end sequencing assay with supposedly overlapping forward and reverse reads, the first step is to recover the assembled sequence.

The forward and reverse reads of the same fragment are at the same line position in the two FASTQ files obtained after sequencing. Based on these two files, the assembly of the forward and reverse reads is done with the **illuminapairedend** utility that aligns the two reads and returns the reconstructed sequence.

> <hands-on-title>Recover consensus sequences from overlapping forward and reverse reads.</hands-on-title>
>
> 1. {% tool [illuminapairedend](toolshed.g2.bx.psu.edu/repos/iuc/obi_illumina_pairend/obi_illumina_pairend/1.2.13) %} with the following parameters:
>    - *"Read from file"*: `wolf_F` for the 3p file
>    - *"Read from file"*: `wolf_R` for the 5p file
>    - *"minimum score for keeping aligment"*: `40.0`
>
>    > <comment-title></comment-title>
>    >
>    > Sequence records corresponding to the same read pair must be in the same order in the two files !
>    >
>    > If the alignment score is below the defined score, here 40, the forward and reverse reads are not aligned but concatenated, and the value of the mode attribute in the sequence header is set to joined instead of alignment
>    {: .comment}
>
{: .hands_on}



## Remove unaligned sequence records with **obigrep**


In this step we are going to use the value of the mode attribute in the sequence header of the illuminapairedend output file to discard sequences indicated as "joined", so not assembled ("alignment") (see explanation about this mode on the previous step)

> <hands-on-title>Remove unaligned sequence records</hands-on-title>
>
> 1. {% tool [obigrep](toolshed.g2.bx.psu.edu/repos/iuc/obi_grep/obi_grep/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `ilumimnapairedend fastq groomer output file`
>    - *"Choose the sequence record selection option"*: `predicat`
>        - *"Python boolean expression to be evaluated for each sequence record."*: `mode!="joined"`
>
>
>    > <tip-title>Verifying FastQ format and converting it</tip-title>
>    >
>    > * You can use {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} to look at the format of the sequencing encoding score then {% tool [FASTQ GROOMER](toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.1.5) %} to specify the guessed sequencing encoding score and create a fastqsanger file.
>    >
>    {: .tip}
>
>    > <comment-title></comment-title>
>    >
>    > The obigrep command is in some way analog to the standard Unix grep command. It selects a subset of sequence records from a sequence file.
>    >
>    > A sequence record is a complex object composed of an identifier, a set of attributes (key=value), a definition, and the sequence itself.
>    >
>    > Instead of working text line by text line as the standard Unix tool, selection is done sequence record by sequence record. A large set of options allows refining selection on any of the sequence record elements.
>    >
>    > Moreover obigrep allows specifying simultaneously several conditions (that take the value TRUE or FALSE) and only the sequence records that fulfill all the conditions (all conditions are TRUE) are selected. You can refer to [https://pythonhosted.org/OBITools/scripts/obigrep.html](https://pythonhosted.org/OBITools/scripts/obigrep.html) for more details
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How do you verify the operation is successful?
> 2. How many sequences are kept? Discarded?
>
> > <solution-title></solution-title>
> >
> > 1. you can search in the input file content the presence of `mode=joined` and same on the output file (just clicking the eye to visualize the content of each file and typing CTRL+C for example to search `mode=joined` in the file, or using a regex Galaxy tool for example). You can also at least look at the size of the output file, if smaller than input file, this is a first good indication.
> > 2. You can use a Galaxy tool like `Line/Word/Character count of a dataset` to count the number of lines of each dataset (input and output of obigrep) and divided by 4 (as in a FastQ file, each sequence is represented by a block of 4 lines). 45 276 sequences for input file. 44 717 for output file. Thus 559 sequences discarded.
> >
> {: .solution}
>
{: .question}

## Assign each sequence record to the corresponding sample/marker combination with **NGSfilter**

> <hands-on-title>Assigns sequence records to the corresponding experiment/sample based on DNA tags and primers</hands-on-title>
>
>
>    > <comment-title></comment-title>
>    >
>    > Each sequence record is assigned to its corresponding sample and marker using the data provided in a text file (here wolf_diet_ngsfilter.txt). This text file contains one line per sample, with the name of the experiment (several experiments can be included in the same file), the name of the tags (for example: aattaac if the same tag has been used on each extremity of the PCR products, or aattaac:gaagtag if the tags were different), the sequence of the forward primer, the sequence of the reverse primer, the letter T or F for sample identification using the forward primer and tag only or using both primers and both tags, respectively.
>    {: .comment}
>
> 1. {% tool [NGSfilter](toolshed.g2.bx.psu.edu/repos/iuc/obi_ngsfilter/obi_ngsfilter/1.2.13) %} with the following parameters:
>    - *"Parameter file"*: `wolf_diet_ngsfilter`
>    - *"Read from file"*: `obigrep output`
>    - *"Number of errors allowed for matching primers"*: `2`
>    - *"Output data type"*: `fastq`
>
>
>    > <tip-title>Be sure the text file is in tabular datatype</tip-title>
>    >
>    > * If you are sure the format is compatible with a tabular datatype, as this is the case here ;), you can manually change it, clicking on the eye of the "wolf_diet_ngsfilter.txt" dataset, then selecting the "Datatypes" section then affecting manually tabular and saving the operation
>    >
>    {: .tip}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many sequences are not assigned?
>
> > <solution-title></solution-title>
> >
> > 1. 1391
> >
> {: .solution}
>
{: .question}

## Dereplicate reads into uniq sequences with **obiuniq**

> <hands-on-title>Groups together sequence records</hands-on-title>
>
>
>    > <comment-title></comment-title>
>    >
>    > The same DNA molecule can be sequenced several times. In order to reduce both file size and computations time, and to get easier interpretable results, it is convenient to work with unique sequences instead of reads. To dereplicate such reads into unique sequences, we use the obiuniq command.
>    > Definition: Dereplicate reads into unique sequences
>    > * compare all the reads in a data set to each other
>    > * group strictly identical reads together
>    > * output the sequence for each group and its count in the original dataset (in this way, all duplicated reads are removed)
>    > *Definition adapted from Seguritan and Rohwer (2001)*
>    {: .comment}
>
> 1. {% tool [obiuniq](toolshed.g2.bx.psu.edu/repos/iuc/obi_uniq/obi_uniq/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `Trimmed and annotated file by NGSfilter`
>    - *"Attribute to merge"*: `sample`
>    - *"Use specific option"*: `merge`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many sequences you had and how many you finally obtain?
>
> > <solution-title></solution-title>
> >
> > 1. From 43 326 to 3 962
> >
> {: .solution}
>
{: .question}


## Limit number of informations with **obiannotate**

> <hands-on-title>Adds/Edits sequence record annotations</hands-on-title>
>
>    > <comment-title></comment-title>
>    >
>    > obiannotate is the command that allows adding/modifying/removing annotation attributes attached to sequence records.
>    > Once such attributes are added, they can be used by the other OBITools commands for filtering purposes or for statistics computing.
>    >
>    > Here, the goal is to keep only `count` and `merged_sample` key=value attributes!
>    {: .comment}
>
> 1. {% tool [obiannotate](toolshed.g2.bx.psu.edu/repos/iuc/obi_annotate/obi_annotate/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `obiuniq output file`
>    - In *"Keep only attribute with key"*:
>        - *"key"*: `count`
>        - *"if you want to specify a second key"*: `merged_sample`
>
{: .hands_on}


## Computes basic statistics for attribute values with **obistat**

> <hands-on-title>Computes basic statistics for attribute values</hands-on-title>
>
>
>    > <comment-title></comment-title>
>    >
>    > stats computes basic statistics for attribute values of sequence records. The sequence records can be categorized or not using one or several -c options. By default, only the number of sequence records and the total count are computed for each category. Additional statistics can be computed for attribute values in each category, like:
>    > * minimum value (-m option)
>    > * maximum value (-M option)
>    > * mean value (-a option)
>    > * variance (-v option)
>    > * standard deviation (-s option)
>    >
>    > The result is a contingency table with the different categories in rows, and the computed statistics in columns.
>    {: .comment}
>
> 1. {% tool [obistat](toolshed.g2.bx.psu.edu/repos/iuc/obi_stat/obi_stat/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `obiannotate output file`
>    - In *"Category attribute"*:
>        - {% icon param-repeat %} *"Insert Category attribute"*
>            - *"How would you specify the category attribute key?"*: `simply by a key of an attribute`
>                - *"Attribute used to categorize the sequence records"*: `count`
>    - *"Use a specific option"*: `no`
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Can you use this result to say how many sequences occuring only once? You would need to use Galaxy tools like `Sort data in ascending or descending order` and ` Select first lines from a dataset` to answer the question
>
> > <solution-title></solution-title>
> >
> > 1. 3131 sequences are occuring once.
> >
> {: .solution}
>
{: .question}

## Filtering sequances by count and length with **obigrep**

In this step, we are going to use *obigrep* in order to keep only the sequences having a count greater or equal to 10 and a length shorter than 80 bp.

> <hands-on-title>filter sequences with <b>obigrep</b></hands-on-title>
>
> 1. {% tool [obigrep](toolshed.g2.bx.psu.edu/repos/iuc/obi_grep/obi_grep/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `obiannotate output file`
>    - *"Choose the sequence record selection option"*: `predicat`
>        - *"Python boolean expression to be evaluated for each sequence record."*: `count>=10`
>
> 2. {% tool [obigrep](toolshed.g2.bx.psu.edu/repos/iuc/obi_grep/obi_grep/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `obigrep output file`
>    - *"Choose the sequence record selection option"*: `lmin`
>        - *"lmin"*: `80`
>
>    > <comment-title></comment-title>
>    > Based on the previous observation, we set the cut-off for keeping sequences for further analysis to a count of 10
>    > Based on previous knowledge we also remove sequences with a length shorter than 80 bp (option -l) as we know that the amplified 12S-V5 barcode for vertebrates must have a length around 100bp
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many sequences are kept following the "count" filter?
> 2. How many sequences are kept following the "length" filter?
>
> > <solution-title></solution-title>
> >
> > 1. 178
> > 2. 175
> >
> {: .solution}
>
{: .question}

## Clean the sequences for PCR/sequencing errors (sequence variants) with **obiclean**

> <hands-on-title>Clean the sequences for PCR/sequencing errors (sequence variants)</hands-on-title>
>
> As a final denoising step, using the obiclean program, we keep the head sequences that are sequences with no variants with a count greater than 5% of their own count
>
>
> 1. {% tool [obiclean](toolshed.g2.bx.psu.edu/repos/iuc/obi_clean/obi_clean/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `obigrep output file`
>    - *"attribute containing sample definition"*: `merged_sample`
>    - *"Maximum numbers of differences between two variant sequences (default: 1)"*: `1`
>    -  *"Threshold ratio between counts (rare/abundant counts) of two sequence records so that the less abundant one is a variant of the more abundant (default: 1, i.e. all less abundant sequences are variants)"*: `0.05`
>    - *"Do you want to select only sequences with the head status in a least one sample?"*: `Yes`
>
>
{: .hands_on}


## Taxonomic assignment of sequences with **NCBI BLAST+ blastn**

> <hands-on-title>Search nucleotide database with nucleotide query sequence(s) from OBITools treatments</hands-on-title>
>
> Once denoising has been done, the next step in diet analysis is to assign the barcodes to the corresponding species in order to get the complete list of species associated to each sample. Taxonomic assignment of sequences requires a reference database compiling all possible species to be identified in the sample. Assignment is then done based on sequence comparison between sample sequences and reference sequences. We here propose to use BLAST+ blastn.
>
> 1. {% tool [NCBI BLAST+ blastn](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.10.1+galaxy0) %} with the following parameters:
>    - *"Nucleotide query sequence(s)"*: `obiclean output file`
>    - *"Subject database/sequences"*: `FASTA file from your history`
>    - *"Nucleotide FASTA subject file to use instead of a database"*: `db_v05_r117`
>    - *"Set expectation value cutoff"*: `0.0001`
>    - *"Output format"*: `Tabular (extended 25 columns)`
>    - *"Advanced Options"*: `Show Advanced Options`
>    - *"Maximum hits to consider/show": `1`
>
>
>    > <comment-title></comment-title>
>    >
>    > Here we directly use the `db_v05_r117` fasta file proposed on the [original obitools tutorial](https://pythonhosted.org/OBITools/wolves.html#step-by-step-analysis). One can mention you can create such a fasta file using same obitools workflow describe before (using obigrep/obiuniq/obigrep/obiannotate) on downloaded EMBL datrabases and taxonomy treated by obitools ecoPCR tool.
>    {: .comment}
>
{: .hands_on}


## Filter database and query sequences by ID to re associate informations with **Filter sequences by ID**

> <hands-on-title>Filter Blast results</hands-on-title>
>
> > <comment-title></comment-title>
> >
> > This tool allows you to re-associate all the reference sequences information, notably the `species_name` one so you can see which species are potentially seen on the sample.
> > We will also use it to re-associate all the query sequences information, notably the `merged_sample` and `obiclean_count` attributes so we can better evaluate quality of the results.
> {: .comment}
>
> 1. {% tool [Filter sequences by ID](toolshed.g2.bx.psu.edu/repos/peterjc/seq_filter_by_id/seq_filter_by_id/0.2.7) %} with the following parameters:
>    - *"Sequence file to be filtered"*: `db_v05_r117`
>    - *"Filter using the ID list from"*: `tabular file`
>    - *"Tabular file containing sequence identifiers"*: `megablast on obiclean output`
>    - *"Column(s) containing sequence identifiers"*: `Column 2`
>    - *"Output positive matches, negative matches, or both?"*: `just positive match`
>
>
> 2. {% tool [Filter sequences by ID](toolshed.g2.bx.psu.edu/repos/peterjc/seq_filter_by_id/seq_filter_by_id/0.2.7) %} with the following parameters:
>    - *"Sequence file to be filtered"*: `obiclean output data`
>    - *"Filter using the ID list from"*: `tabular file`
>    - *"Tabular file containing sequence identifiers"*: `megablast on obiclean output`
>    - *"Column(s) containing sequence identifiers"*: `Column 1`
>    - *"Output positive matches, negative matches, or both?"*: `just positive match`
>
>
>
{: .hands_on}





## From FASTA to tabular with **Obitab**

> <hands-on-title>Convert fasta filtered files in tabular ones</hands-on-title>
>
> 1. {% tool [obitab](toolshed.g2.bx.psu.edu/repos/iuc/obi_tab/obi_tab/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `db_v05_r117 with matched ID`
>
>    > <comment-title></comment-title>
>    >
>    > This tool allows you to convert a fasta file into a tabular one so it is easier to read sequences definitions.
>    {: .comment}
>
> 2. {% tool [obitab](toolshed.g2.bx.psu.edu/repos/iuc/obi_tab/obi_tab/1.2.13) %} with the following parameters:
>    - *"Input sequences file"*: `obiclean on data 61 with matched ID`
>
>
{: .hands_on}

## create a final synthesis as a tabular file

> <hands-on-title>Join blast and obitab files then cut relevant column and apply filters</hands-on-title>
>
> 1. {% tool [Join two datasets side by side on a specified field](join1) %} with the following parameters:
>    - *"Join"*: `obitab on obiclean output file`
>    - *"using column"*: `Column 1`
>    - *"with"*: `megablast output file`
>    - *"using column"*: `Column 1`
>    - *"Fill empty columns"*: `Yes`
>    - *"Fill Columns by"*: `Single fill value`
>    - *"Fill value"*: `NA`
>
> 2. {% tool [Join two datasets side by side on a specified field](join1) %} with the following parameters:
>    - *"Join"*: last `Join two Datasets output file`
>    - *"using column"*: `Column 26`
>    - *"with"*: `obitab on db_v05_r117 with matched ID output file`
>    - *"using column"*: `Column 1`
>    - *"Fill empty columns"*: `Yes`
>    - *"Fill Columns by"*: `Single fill value`
>    - *"Fill value"*: `NA`
>
>
>    > <comment-title></comment-title>
>    >
>    > To have something easier to read and understand, we create a tabular file containing only columns with important informations (c1: query sequences names / c3-7: query counts / c50: reference sequences names / c54: family / c59: genus / c51: reference annotations).
>    {: .comment}
>
>
> 3. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c3,c4,c5,c6,c7,c50,c54,c59,c51`
>    - *"From"*: last `Join two Datasets output file`
>
> 4. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters:
>    - *"Filter"*: `Cut output file`
>    - *"With following condition"*: `c3>1000 or c4>1000 or c5>1000 or c6>1000`
>    - *"Number of header lines to skip"*: `1`
>
>    > <comment-title></comment-title>
>    >
>    > To keep only data with significative counts.
>    {: .comment}
{: .hands_on}


> <question-title></question-title>
>
> 1. How many species are identified? You can use `Cut columns from a table` and `unique occurences of each record` to isolate the `species name` column of obitab results.
> 2. Can you deduce the diet of each sample? You can use tools like `obitab` and `Join two Datasets side by side on a specified field` to join megablast results to obigrep one and `db_v05_r117 with matched ID`
>
> > <solution-title></solution-title>
> >
> > 1. 1O
> > 2. If we remove human sequences ;), some squirrel (sample:26a_F040644), deer (sample:15a_F730814 + sample:29a_F260619), stag (sample:29a_F260619 + sample:13a_F730603), marmot (sample:26a_F040644), doe (sample:29a_F260619), Grimm's duiker (sample:29a_F260619).
> >
> {: .solution}
>
{: .question}


# Conclusion


You just did a ecological analysis, finding diet from wolves faeces ! So now you know how to preprocess metabarcoding data on Galaxy, producing quantitative informations with quality checks and filtering results to interpret it and to have a synthesis table you can share broadly!

tools_json:
["Unzip", "toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/0.2", "illuminapairedend", "toolshed.g2.bx.psu.edu/repos/iuc/obi_illumina_pairend/obi_illumina_pairend/1.2.13", "obigrep", "toolshed.g2.bx.psu.edu/repos/iuc/obi_grep/obi_grep/1.2.13", "FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "FASTQ GROOMER", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.1.5", "NGSfilter", "toolshed.g2.bx.psu.edu/repos/iuc/obi_ngsfilter/obi_ngsfilter/1.2.13", "obiuniq", "toolshed.g2.bx.psu.edu/repos/iuc/obi_uniq/obi_uniq/1.2.13", "obiannotate", "toolshed.g2.bx.psu.edu/repos/iuc/obi_annotate/obi_annotate/1.2.13", "obistat", "toolshed.g2.bx.psu.edu/repos/iuc/obi_stat/obi_stat/1.2.13", "obiclean", "toolshed.g2.bx.psu.edu/repos/iuc/obi_clean/obi_clean/1.2.13", "NCBI BLAST+ blastn", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.10.1+galaxy0", "Filter sequences by ID", "toolshed.g2.bx.psu.edu/repos/peterjc/seq_filter_by_id/seq_filter_by_id/0.2.7", "obitab", "toolshed.g2.bx.psu.edu/repos/iuc/obi_tab/obi_tab/1.2.13", "Join two datasets side by side on a specified field", "join1", "Cut columns from a table", "Cut1", "Filter data on any column using simple expressions", "Filter1"]

tags_json:
["genetic composition EBV class", "community composition EBV class", "EBV dataset", "EBV workflow", "eDNA", "metabarcoding", "biodiversity"]


topic: ecology
tutorial: de-novo-rad-seq
title: RAD-Seq de-novo data analysis
difficulty: intermediate
time_estimation: 8h
tools_json: []
requirements_json: []
content_hash: d9ff1171f10c0f7b8b9114954543d477
last_modified: 2026-05-27T09:01:48.396494
zenodo_link: https://doi.org/10.5281/zenodo.1134547
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/de-novo-rad-seq/tutorial.html

content:
In the study of [Hohenlohe *et al.* 2010](http://journals.plos.org/plosgenetics/article?id=10.1371/journal.pgen.1000862), a genome scan of nucleotide diversity and differentiation in natural populations of threespine stickleback *Gasterosteus aculeatus* was conducted. Authors used Illumina-sequenced RAD tags to identify and type over 45,000 single nucleotide polymorphisms (SNPs) in each of 100 individuals from two oceanic and three freshwater populations.

![Abstract of the paper on Population Genomics](../../images/RAD4_Population_Genomics/Hohenlohe_et_al_2010.png)

We here proposed to re-analyze these data at least until the population genomics statistics calculation step using STACKS pipeline. Existing *Gasterosteus aculeatus* draft genome will not be used here so the analysis will be performed de novo. In a de novo RAD-seq data analysis, the reads are aligned one on each other to create stacks and then clustered to build loci. A reference approach can also be conducted (see [ref_based tutorial]({% link topics/ecology/tutorials/ref-based-rad-seq/tutorial.md %}), allowing to work on existing assembled loci).


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pretreatments

## Data upload

The original data is available at NCBI SRA ENA under accession number [SRR034310](https://trace.ncbi.nlm.nih.gov/Traces/sra/?run=SRR034310) as part of the NCBI SRA ENA study accession number [SRP001747](https://trace.ncbi.nlm.nih.gov/Traces/sra/?study=SRP001747).

We will look at the first run SRR034310 out of seven which includes 16 samples from 2 populations, 8 from Bear Paw (freshwater) and 8 from Rabbit Slough (oceanic). We will download the reads directly from SRA and the remaining data (i.e reference genome, population map file, and barcodes file) from [Zenodo](https://zenodo.org/record/1134547#.WkugUjfjJXI).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a **new history** for this RAD-seq exercise. If you are not inspired, you can name it *STACKS RAD: population genomics without reference genome* for example...
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **EBI SRA** {% icon tool %} with the following parameters:
>    - Select the Run from the results of the search for `SRR034310` (which will present you 1 Experiment (SRX015877) and 1 Run (SRR034310)).
>    - Click the link in the column **FASTQ files (Galaxy)** of the results table
>    - This will redirect to the Galaxy website and start the download.
>
> 3. Upload remaining training data from Zenodo:
>    ```
>    https://zenodo.org/record/1134547/files/Barcode_SRR034310.txt
>    https://zenodo.org/record/1134547/files/Details_Barcode_Population_SRR034310.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Make sure your fasta files are of datatype `fastqsanger`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger" %}
>
{: .hands_on}

The sequences are raw sequences from the sequencing machine, without any pretreatments. They need to be demultiplexed. To do so, we can use the Process Radtags tool from STACKS.

## Demultiplexing reads

For demultiplexing, we use the Process Radtags tool from [STACKS](https://www.g3journal.org/content/1/3/171.full) .

> <hands-on-title>Demultiplexing reads</hands-on-title>
>
> 1. **Process Radtags** {% icon tool %} to demultiplex the reads:
>  - *"Single-end or paired-end reads files"*: `Single-end files`
>  - *"singles-end reads infile(s)"*: `SRR034310.fastq(.gz)`
>  - *"Barcode file"*: `Barcodes_SRR034310.tabular`
>  - *"Number of enzymes"*: `One`
>  - *"Enzyme"*: `sbfI`
>  - *"Capture discarded reads to a file*": `Yes`
>  - *"Output format:"* `fastq`
>
>    > <question-title></question-title>
>    >
>    > 1. How many reads where on the original dataset?
>    > 2. How many are kept?
>    > 3. Can you try to explain the reason why we loose a lot of reads here?
>    > 4. What kind of information does this result give concerning the upcoming data analysis and the barcodes design in general?
>    >
>    > > <solution-title></solution-title>
>    > > The informations can be found in the results log file:
>    > >
>    > >  1. 8895289 total reads
>    > >  2. 8139531 retained reads
>    > >  3. There are no sequences filtered because of low quality. This is because radtags didn't apply quality related filtering since the corresponding advanced option (Discard reads with low quality scores) has not been enabled. So here, all not retained sequences are removed because of an ambiguous barcode (`626265`) or an ambiguous RAD-Tag (`129493`). This means that some barcodes are not exactly what was specified on the barcode file and that sometimes, no SbfI restriction enzyme site was found. This can be due to some sequencing problems but here, this is also due to the addition, in the original sequencing library, of RAD-seq samples from another study. This strategy is often used to avoid having too much sequences beginning with the exact same nucleotide sequence which may cause Illumina related issues during sequencing and cluster analysis
>    > >  4. Sequencing quality is essential! Each time your sequencing quality decreases, you loose data and thus essential biological information!
>    > >
>    > > In addition to the overall statistics the numbers of retained and removed reads are also given for each bar code sequence.
>    > {: .solution}
>    {: .question}
>
>    Next we will play around with the parameters a bit, to see the impact they have on the results.
>
> 2. **Process Radtags** {% icon tool %}: Re-Run playing with parameters
>    - Section **Advanced**
>      - *"Discard reads with low quality scores"*: `Yes`
>      - *"score limit"*: `20` (for example; play with this)
>      - *"Set the size of the sliding window as a fraction of the read length, between 0 and 1"*: `0.30` (for example; play with this)
>
> This sliding window parameter allows notably the user to deal with the declining quality at the 3' end of reads.
>
{: .hands_on}

> <hands-on-title></hands-on-title>
>
> You can use the `Charts` functionality through the Visualize button reachable on the `Radtags logs` file you just generated.
>
> If like me you don't have payed attention to the organization of you file for the graphical representation you obtain a non optimal bars diagram with a not intelligent X-axis ordering. There is a lot of different manner to fix this. You can use a copy/paste "bidouille" or you can use Galaxy tools to manipulate the `radtags logs` file to generate a better graph. For example, you can use `Select lines that match an expression` tool to select rows then use the `Concatenate datasets tail-to-head` tool to reorganize these lines in a new file.
> Then we generate a graphical display of the changes:
>
> First we cut the interesting lines of each `result.log with Stacks: process radtags`
>
> 1. **Select lines that match an expression** applying `^R1.fq.gz` on the log files and then
> 2. **Replace Text in entire line** on the resulting data sets finding `R1.fq.gz` and replacing with `NoScoreLimit` or `Score10` or `Score20` depending of the case
> 3. **Select lines that match an expression** applying `File\tRetained Reads` on one of the log file to obtain you futur header
> 4. **Replace Text in entire line** on the resulting data set finding `File` and replacing with `#` to have a best display
> 5. **Concatenate datasets tail-to-head** on the resulting data sets stating from the header you just created
> 6. Finally **Convert delimiters to TAB** on the resulting data set converting all Tabs to be sure having a well formatted tabular file at the end
>
> Alternatively just copy/paste these lines on the Galaxy upload tool using Paste/fetch data section and modifying the File header by sample and filename by Score 10 / Score 20 and Lowquality for example... Before Starting the upload, you can select the `Convert spaces to tabs` option through the `Upload configuration` wheel. If you did not pay attention to the order you can just sort the file using the first column.
{: .hands_on}

You can use the `Charts` functionality through the Visualize button to plot the data.

And you obtain a file like this one, ready to generate a beautiful and smart bar stacked!

```
#		Retained Reads	Low Quality	Ambiguous Barcodes	Ambiguous RAD-Tag	Total
NoScoreLimit	8139531		0		626265			129493			8895289
Score10		7373160		766371		626265			129493			8895289
Score20		2980543		5158988		626265			129493			8895289
```

![The chart on the sorted file](../../images/RAD4_Population_Genomics/Process_radtags_charts_end.PNG){:width="50%"}

You can further test using a filter like `clean data, remove any read with an uncalled base` and see that here, this has only little impact on the number of retained reads.

The demultiplexed sequences are raw sequences from the sequencing machine, without any pretreatments. They need to be controlled for their quality.

We propose to continue the tutorial using the dataset collection containing the demultiplexed reads obtained with Process Radtag execution made with a quality score of 10 and with the `Discard reads with low quality scores` parameter set to Yes (so containing 7373160 retained reads).

## Quality control

For quality control, we use similar tools as described in [NGS-QC tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}): [Falco](https://falco.readthedocs.io/en/latest/).

Falco is an efficiency-optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).

> <hands-on-title>Quality control</hands-on-title>
>
> 1. **Falco** {% icon tool %}: Run Falco on FastQ files to control the quality of the reads. Warning! Don't forget you are working on data collections....
>
>    > <question-title></question-title>
>    >
>    > 1. What is the read length?
>    >
>    > > <solution-title></solution-title>
>    > > The read length is 32 bp
>    > {: .solution }
>    {: .question}
>
> 2. **MultiQC** {% icon tool %}: Run MultiQC on Falco results to better see quality information over samples. For "Which tool was used generate logs?" choose `FastQC` no matter whether FastQC or Falco was used.
{: .hands_on}

![MultiQC output](../../images/RAD4_Population_Genomics/multiqc.PNG)

# SNP calling from radtags

Run `Stacks: De novo map` Galaxy tool. This program will run ustacks, cstacks, and sstacks on the members of the population, accounting for the alignments of each read.

> <comment-title></comment-title>
>
> Information on denovo_map.pl and its parameters can be found online: https://creskolab.uoregon.edu/stacks/comp/denovo_map.php.
{: .comment}


> <hands-on-title>Stacks: De novo map</hands-on-title>
>
> 1. First we will create a population map file with the following contents:
>
>    ```
>    sample_CCCC	1
>    sample_CCAA	1
>    sample_CCTT	1
>    sample_CCGG	1
>    sample_CACA	1
>    sample_CAAC	1
>    sample_CATG	1
>    sample_CAGT	1
>    sample_CTCT	2
>    sample_CTAG	2
>    sample_CTTC	2
>    sample_CTGA	2
>    sample_GGGG	2
>    sample_GGAA	2
>    sample_GGTT	2
>    sample_GGCC	2
>    ```
>    {% snippet faqs/galaxy/datasets_create_new_file.md %}
>
> 2. **Stacks: De novo map** {% icon tool %}: with the following parameters:
>    - *"Select your usage"*: `Population`
>    - *"Files containing an individual sample from a population"*: `The demultiplexed reads (collection)`
>    - *"Specify a population map"*: `the population map you just created`
>    - Section **Assembly options**
>      - *"Minimum number of identical raw reads required to create a stack"*: `3`
>      - *"Number of mismatches allowed between loci when building the catalog"*: `3`
>      - *"Remove, or break up, highly repetitive RAD-Tags in the ustacks program"*: `Yes`
>
> 3. Once Stacks has completed running, investigate the output files: `result.log` and `catalog.` (snps, alleles and tags).
>
>    ![The output of de novo map](../../images/RAD4_Population_Genomics/denovo/denovo_out.png)
>
> > <comment-title>Data formatting</comment-title>
> >
> > If you are using a file presenting population information and individual name in a different manner than expected by STACKS, you can use Galaxy tools like `Replace Text` (for example to replace `Rabbit Slough` by a population number like `2`, `Add column` (for example to add `sample_`) or `Cut columns from a table` (to put the new `sample_` column af the first place) and finally `Regex replace` (replacing `(sample_)\t` by `\1`) to generate it...
>    {: .comment}
>
>
{: .hands_on}

# Calculate population genomics statistics

> <hands-on-title>Calculate population genomics statistics</hands-on-title>
>
> 1. **Stacks: populations** {% icon tool %} with the following parameters:
>    - Section **Data filtering options**
>      - *"Minimum percentage of individuals in a population required to process a locus for that population"*: `0.75`
>    - Section **Output options** (VCF and Structure) and enabling SNP and haplotype-based F statistics calculation.
>      - *"Output results in Variant Call Format (VCF)"*: `Yes`
>      - *"Output results in Structure Format"*: `Yes`
>    - *"Enable SNP and haplotype-based F statistics"*: `Yes`
>
>    ![The output of the populations tool](../../images/RAD4_Population_Genomics/denovo/populations_log.png)
>
> 2. Now look at the output in the file `batch_1.sumstats` nammed `SNP and Haplotype-based F statistics with Stacks: populations ...` on your history. There are a large number of statistics calculated at each SNP, so use Galaxy tools like filter, cut, and sort to focus on some.
>
> > <question-title></question-title>
> >
> > 1. What is the maximum value of FST' at any SNP? Don't hesitate to look at the [STACKS manual](http://catchenlab.life.illinois.edu/stacks/manual/#files) to find where this parameter is
> > 2. What is the meaning of this FST' value compared a a classical FST one? Once again, the [STACKS manual](http://catchenlab.life.illinois.edu/stacks/manual/#files) can help you ;)
> > 3. How many SNPs reach this FST' value?
> >
> > > <solution-title></solution-title>
> > > 1. 1
> > > 2. FST’ is a haplotype measure of FST that is scaled to the theoretical maximum FST value at this locus. Depending on how many haplotypes there are, it may not be possible to reach an FST of 1, so this method will scale the value to 1
> > > 2. 114
> > {: .solution }
> {: .question}
{: .hands_on}

You can now for example filter this dataset to only keep FST'=1 loci for further analysis...

# Conclusion


In this tutorial, we have analyzed real RAD sequencing data to extract useful information, such as which loci are candidate regarding the genetic differentiation between freshwater and oceanic Stickelback populations. To answer these questions, we analyzed RAD sequence datasets using a de novo RAD-seq data analysis approach. This approach can be sum up with the following scheme:


![The de novo tutorial workflow](../../images/denovo_based_workflow.PNG)

tags_json:
["RAD-seq", "genetic composition EBV class", "species populations EBV class", "EBV dataset", "EBV workflow", "biodiversity"]


topic: ecology
tutorial: phylogeny-data-prep
title: Preparing genomic data for phylogeny reconstruction
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["phylogeny", "data handling", "functional annotation", "biodiversity"]
content_hash: 8136ece09b13e76ccdc5bc9d74ae281a
last_modified: 2026-05-27T09:01:48.401320
zenodo_link: https://zenodo.org/record/6610704
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/phylogeny-data-prep/tutorial.html

content:
A robust and well-resolved phylogenetic classification is essential to understand genetic relationships within and between species and the evolution of their phenotypic diversity. In the last decade the genomic revolution has represented a drastic change in the amount of data used for phylogenetic inference. The single-gene approach using universal phylogenetic markers for the different lineages across the tree of life, is now being replaced by the assembly of taxon-rich and genome-scale data matrices, the so called phylogenomic approach.

Molecular sequence data can be used to construct a phylogeny by comparing differences between nucleotide or amino acid sequences across species or strains, a technique called phylogenomics. {% cite Young2019 %} have written a comprehensive review on the topic of phylogenomics.

In this tutorial we prepare genetic sequence data for phylogenetic reconstruction, using sequences from chromosome 5 of five strains of the yeast *Saccharomyces cerevisiae*.

To prepare the data, we will:
1. Predict protein coding genes from the genome using Funannotate
2. Find the proteins that are present in more than one genome, called orthologs, using Proteinortho and extract a subset with orthologs that are present in all samples.
4. Align each set of orthologs using ClustalW.

The resulting dataset is ready to be used for phylogenetic reconstruction.


**If you are starting from sequence reads, please follow
[An Introduction to Genome Assembly]({% link topics/assembly/tutorials/general-introduction/tutorial.md %}), and the appropriate genome assembly training for your sequencing technology from GTN's [Assembly]({% link topics/assembly/index.md %}) section**

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data
For this training we will use a subset of the genome (chromosome 5) from four strains of *S. cerevisiae*. The GenBank annotated sequenced were produced using 'funannotate predict annotation' (Galaxy Version 1.8.9+galaxy2) on the nucleotide sequences sequences.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{page.topic_name}}`
>     -> `{{page.title}}`):
>
>    ```
>    https://zenodo.org/record/6610704/files/BK006939.2.genbank
>    https://zenodo.org/record/6610704/files/BK006939.2.genome.fasta
>    https://zenodo.org/record/6610704/files/CM000925.1.genbank
>    https://zenodo.org/record/6610704/files/CM000925.1.genome.fasta
>    https://zenodo.org/record/6610704/files/CM005043.2.genbank
>    https://zenodo.org/record/6610704/files/CM005043.2.genome.fasta
>    https://zenodo.org/record/6610704/files/CM005299.1.genbank
>    https://zenodo.org/record/6610704/files/CM005299.1.genome.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. Optional: Rename each dataset to its accession number followed by '.genome.fasta' or '.genbank'.
> 4. Group the datasets into [collections]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}). These will ease data handling and help minimize the clutter in your history. Make a collection of nucleotide sequences and another of protein sequences.
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
>
{: .hands_on}

# Protein coding gene prediction

## Mask repetitive sequences

Before we can annotate the genome, we will prepare the data by masking repetitive sequences in the genome.
Repeat-rich regions can interfere with genome annotation tools. In this step we find and soft-mask repetitive regions in the genome. The annotation tool can then take this information into account ([see RepeatMasker tutorial for more details]({% link topics/genome-annotation/tutorials/repeatmasker/tutorial.md %})).
We use **RepeatMasker**
{% cite RepeatMasker %}, a program that screens DNA sequences for interspersed repeats and low complexity DNA sequences.
This program has historically made use of [RepBase](https://www.girinst.org/repbase/update/index.html) ({%cite Kohany2006-ks %}), a service of the Genetic Information Research Institute, but this database in no longer open access. Instead, we will use [Dfam](https://www.dfam.org/home) ({%cite Storer2021 %}) an open collection of Transposable Element DNA sequence alignments,  HMMs derived from Repbase sequences and consensus sequences. For this reason, the annotation of repetitive sequences might be incomplete.

RepeatMasker will only accept compact fasta headers. Before we can mask repetitive regions with RepeatMasker we must trim the NCBI long header (`BK006939.2 TPA_inf: Saccharomyces cerevisiae S288C chromosome V, complete sequence`) to leave only the accession number (`>BK006939.2`) by using a regular expression.

> <hands-on-title> Mask repetitive sequences </hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} with the following parameters:
>    - {% icon param-collection %} *"File to process"*: `output` (Input dataset collection)
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `(>[^ ]+).+`
>            - *"Replace with:"*: `\1`
>
>               > <comment-title></comment-title>
>               >
>               >`\1` replaces the text in the header with the first matched text in the header, the accession number in this case.
>               {: .comment}
>
> 2. {% tool [RepeatMasker](toolshed.g2.bx.psu.edu/repos/bgruening/repeat_masker/repeatmasker_wrapper/4.1.2-p1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Genomic DNA"*: `outfile` (output of **Replace Text** {% icon tool %})
>    - *"Repeat library source"*: `DFam (curated only, bundled with RepeatMasker)`
>        - *"Select species name from a list?"*: `No`
>            - *"Repeat source species"*: `"Saccharomyces cerevisiae"`
>
>    > <comment-title></comment-title>
>    >
>    > If you don't select the species from the list, you must provide a species name ("Saccharomyces cerevisiae" in this example). In principal, all unique clade names occurring in [NCBI taxonomy database](https://www.ncbi.nlm.nih.gov/Taxonomy/taxonomyhome.html) can be used for species. Capitalization is ignored, multiple words need to bound by apostrophes. Not all "common" English names occur in the taxonomy database. Using Latin names is always safest.
>    {: .comment}
>
>3. Inspect the 'RepeatMasker masked sequence on data' output file. Scroll down and you will find stretches of 'N' on the location of repetitive sequences. The file is now ready for annotation with Funannotate.
{: .hands_on}

## Annotate with Funannotate

We will predict protein-coding genes from genomic sequences using [Funannotate](https://funannotate.readthedocs.io/) ({% cite Young2019 %}), which collects evidence from different ab-initio gene predictors as well as from RNA-seq or ESTs data. Funannotate has been developed for Fungi but it works with any Eukaryotic genome. The output of Funannotate is a list of ORFs and their translation in GenBank format.

> <comment-title></comment-title>
>
>If you would like to learn about genome annotation in more depth, the GTN has a [section]({% link topics/genome-annotation/index.md %}) dedicated to training on genome annotation, including a hands-on tutorial on [Funannotate]({% link topics/genome-annotation/tutorials/funannotate/tutorial.md %}).
{: .comment}

> <warning-title> Slow Step Ahead! </warning-title>
> Even for a small dataset, Funannotate can take a very long time to run. You can skip this step and use the Genbank files downloaded from Zenodo for the following step. These were generated using Funannotate as described in the hands-on below.
{: .warning}

> <hands-on-title> Annotate genome </hands-on-title>
>
> 1. {% tool [Funannotate predict annotation](toolshed.g2.bx.psu.edu/repos/iuc/funannotate_predict/funannotate_predict/1.8.9+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Assembly to annotate"*: `output_masked_genome` (output of **RepeatMasker** {% icon tool %})
>    - In *"Organism"*:
>        - *"Name of the species to annotate"*: `Saccharomyces cerevisiae`
>        - *"Is it a fungus species?"*: `Yes`
>        - *"Ploidy of assembly"*: `1`
>    - In *"Evidences"*:
>        - *"Select protein evidences"*: `Use UniProtKb/SwissProt (from selected Funannotate database)`
>
>    > <tip-title></tip-title>
>    >
>    > If available, include mRNA and/or ESTs in this evidence section to increase sensitivity of predictions.
>    {: .tip}
>
>    - In *"Busco"*:
>        - *"BUSCO models to align"*: `saccharomycetes`
>        - *"Initial Augustus species training set for BUSCO alignment"*: `saccharomyces`
>    - In *"Augustus settings (advanced)"*:
>        - *"Minimum number of models to train Augustus"*: `15`
>
>
>
>    > <comment-title></comment-title>
>    >
>    > When annotating full genomes, increase the number of *'Minimum number of models to train Augustus'* to an appropriate value. For the small sample dataset used here, values larger than 15 will result in failure.
>    {: .comment}
> 2. Inspect the output GenBank file. The FEATURES section contains the genome annotation of protein predictions, their location and their translation. Each predicted protein is given a unique ID, which will become the FASTA header in the next step.
{: .hands_on}

## Extract ORFs into FASTA files

In this step, we extract the protein sequences from the GenBank files into multi-FASTA files. Additionally, we modify the headers to include the accession number of the sample. This creates an unique accession-proteinID header for each predicted protein, and will allow us to retrieve them after we combine all predicted proteins into one multi-FASTA file.

So far, we have kept sequence files in a collection. Now we will combine all predicted proteins from all samples into one big multi-fasta file. Later, we will retrieve ortholog sequences from this file.

> <hands-on-title> Extract ORFs </hands-on-title>
>
> 1. {% tool [Extract ORF](toolshed.g2.bx.psu.edu/repos/bgruening/glimmer_gbk_to_orf/glimmer_gbk_to_orf/3.02) %} with the following parameters:
>    - {% icon param-file %} *"gene bank file"*: `annot_gbk` (output of **Funannotate predict annotation** {% icon tool %} - or the genbank files downloaded from Zenodo)
>
> 2. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `aa_output` (output of **Extract ORF** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `>([^ ]+).+`
>            - *"Replacement"*: `>#{input_name}_\1`
>
> 1. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/4.2) %} with the following parameters:
>    - {% icon param-file %} *"Collection of files to collapse into single dataset"*: `out_file1` (output of **Regex Find And Replace** {% icon tool %})
>    - *"Prepend File name"*: `Yes`
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Is the number of predicted ORFs the same across the samples?
>
> > <solution-title></solution-title>
> >
> > 1. No, the number of predicted ORFs ranges from 193 to 199.
> >
> {: .solution}
>
{: .question}

# Find orthologs

Orthologs are genes in different species evolved from a common ancestral gene by a speciation (lineage-splitting) event and contain the information needed for building phylogenies. The input for this step is the collection of multi-FASTA extracted from the GenBank file and modified to have unique headers.
The result file 'orthology-groups' contains one row per orthogroup and one column per sample.

Next, we select orthogroups where all the species are represented by only one protein, 1:1 single copy orthologs (a total of 4 proteins per orthogroup for this dataset).

> <hands-on-title> Find and filter orthologs </hands-on-title>
>
> 1. {% tool [Proteinortho](toolshed.g2.bx.psu.edu/repos/iuc/proteinortho/proteinortho/6.0.14+galaxy2.9.1) %} with the following parameters:
>    - {% icon param-file %} *"Select the input fasta files (>2)"*: `out_file1` (output of **Regex Find And Replace** {% icon tool %})
>    - *"Activate synteny feature (POFF)"*: `no`
>
> 2. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `proteinortho` (output of **Proteinortho** {% icon tool %})
>    - *"With following condition"*: `c1==4 and c2==4`
>
> 3. Inspect the 'orthology-groups' tabular file.
{: .hands_on}


> <question-title></question-title>
>
> 1. Inspect the the 'orthology-groups' tabular file.
>
>    1. Do any samples contain more than one gene for any given orthogroup?
>    2. What is the name given to these homologous genes within the same genome?
>
>    > <solution-title></solution-title>
>    >
>    > 1. Yes, *CM000925.1* contains two genes on the first orthogroup.
>    > 2. Paralogs. These are gene copies created by a duplication event within the same genome.
>    >
>    {: .solution}
>
> 2. Look at the filtered orthogroups.
>     1. How many orthogroups are represented once only in all four samples?
>
>     > <solution-title></solution-title>
>     >
>     > 1. 173
>     >
>     {: .solution}
>
{: .question}




## Quality control with **Busco**

Busco is a dataset of nearl-universal, single-copy orthologs.
Here we use it for quality control of the assembly and annotation produced above.
It outputs a proxy of completeness, duplication and fragmentation of the annotation (Busco can also be used to assess the completeness of a genome assembly).

> <hands-on-title> QC with Busco </hands-on-title>
>
> 1. {% tool [Busco](toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/4.1.4) %} with the following parameters:
>    - {% icon param-file %} *"Sequences to analyse"*: `out_file1` (output of **Regex Find And Replace** {% icon tool %})
>    - *"Mode"*: `Proteome`
>    - *"Lineage"*: `Saccharomycetes`
>    - In *"Advanced Options"*:
>        - *"Augustus species model"*: `Use the default species for selected lineage`
>
>    > <tip-title></tip-title>
>    > Make sure you select the 'lineage' closest to the species(s) you are analyzing.
>    >
>    {: .tip}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. The [Complete ('C') metric](https://busco.ezlab.org/busco_userguide.html#complete) stands for complete Busco genes identified. Look at the Busco output file 'Short summary' for sample BK006939.2. What is the 'C' number?
> 2. Why is it so low?
>
> > <solution-title></solution-title>
> >
> > 1. 4.0%
> > 2. Our dataset contains only chromosome 5 of the yeast genome.
> >
> {: .solution}
>
{: .question}

## Extract proteins with Proteinortho

Next we extract 1:1 single copy orthologs and generate one multi fasta file per ortholog.

> <hands-on-title> Extract protein sequences </hands-on-title>
>
> 1. {% tool [Proteinortho grab proteins](toolshed.g2.bx.psu.edu/repos/iuc/proteinortho_grab_proteins/proteinortho_grab_proteins/6.0.14+galaxy2.9.1) %} with the following parameters:
>    - {% icon param-file %} *"Select the input fasta files"*: `output` (output of **Collapse Collection** {% icon tool %})
>    - *"Query type"*: `orthology-groups output file`
>        - {% icon param-file %} *"A orthology-groups file"*: `out_file1` (output of **Filter** {% icon tool %})
>
>
{: .hands_on}

The output is a collection of multi-fasta ortholog files. All species are represented in each file and are ready to be aligned.

# Align ortholog sequences

Alignment of sequences allows cross-species (or other taxonomic level, strain, taxa) comparison. An alignment is a hypothesis of positional homology between bases/amino acids of different sequences. A correct alignment is crucial for phylogenetic inference. Here we use ClustaW, a well-established pairwise sequence aligner.

First we modify the headers of the multi-fasta file, such that only the sample name is retained. This is important for future conatenation of alignment into a supermatrix (see Maximum likelihood GTN training)

> <hands-on-title> Align orthologs with ClustalW </hands-on-title>
>
> 1. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `listproteinorthograbproteins` (output of **Proteinortho grab proteins** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `(>[^_]+).+`
>            - *"Replacement"*: `\1`
>
> 2. {% tool [ClustalW](toolshed.g2.bx.psu.edu/repos/devteam/clustalw/clustalw/2.1) %} with the following parameters:
>    - {% icon param-file %} *"FASTA file"*: `out_file1` (output of **Regex Find And Replace** {% icon tool %})
>    - *"Data type"*: `Protein sequences`
>    - *"Output alignment format"*: `FASTA format`
>    - *"Output complete alignment (or specify part to output)"*: `Complete alignment`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Open the ClustalW output 'queryOrthoGroup121.fasta' and its corresponding multifasta input. You can compare them side by side activating the 'Scratchbook' on the top panel. What is the main difference between the sequences in the unaligned multifasta (input) and the ClustalW output multifasta?
>
> > <solution-title></solution-title>
> >
> > 1. Two of the sequences from this orthogroup are truncated (early stop codon). The alignment program inserts '-' to represent indels in the alignment.
> >
> {: .solution}
>
{: .question}



# Conclusion
In this tutorial, you have prepared genome sequence data for phylogenetic analysis. First, you have extracted information from these in the form of predicted protein sequences. You then grouped the predicted proteins into orthogroups and aligned them. These aligned sequences can now be used for reconstructing a phylogeny and building a phylogenetic tree.

tools_json:
["Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2", "RepeatMasker", "toolshed.g2.bx.psu.edu/repos/bgruening/repeat_masker/repeatmasker_wrapper/4.1.2-p1+galaxy0", "Funannotate predict annotation", "toolshed.g2.bx.psu.edu/repos/iuc/funannotate_predict/funannotate_predict/1.8.9+galaxy2", "Extract ORF", "toolshed.g2.bx.psu.edu/repos/bgruening/glimmer_gbk_to_orf/glimmer_gbk_to_orf/3.02", "Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.1", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/4.2", "Proteinortho", "toolshed.g2.bx.psu.edu/repos/iuc/proteinortho/proteinortho/6.0.14+galaxy2.9.1", "Filter", "Filter1", "Busco", "toolshed.g2.bx.psu.edu/repos/iuc/busco/busco/4.1.4", "Proteinortho grab proteins", "toolshed.g2.bx.psu.edu/repos/iuc/proteinortho_grab_proteins/proteinortho_grab_proteins/6.0.14+galaxy2.9.1", "ClustalW", "toolshed.g2.bx.psu.edu/repos/devteam/clustalw/clustalw/2.1"]


topic: ecology
tutorial: NEAL_tutorial
title: Audio data annotation with NEAL (Nature + Energy Audio labeler)
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["ecology", "biodiversity", "sound", "media annotation"]
content_hash: 44e48ad5096362c8ae87f2bca628d4ec
last_modified: 2026-05-27T09:01:48.393384
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/NEAL_tutorial/tutorial.html

content:
This tutorial is designed to guide you through the annotation of audio data via the NEAL interactive tool. 

NEAL (Nature+Energy Audio Labeller) {% cite Gibbons2023 %} is an open-source interactive audio data annotation tool. Built using R and the associated R Shiny framework, the tool provides a reactive environment where users can quickly annotate audio files and adjust settings which automatically change the corresponding elements of the user interface. The app has been designed with the goal of having both expert birders and citizen scientists contribute to acoustic annotation projects {% cite githubsurvey2022 %}.

The data we will work with in this tutorial is a 0:43min recording of a common blackbird (Turdus merula) recorded in Armenia above Alaverdi town. For more information, you can see metadata on the [xeno-canto site](https://xeno-canto.org/explore?query=nr:1008421).   

In this tutorial, you will learn how to import an audio file into Galaxy, run the NEAL interactive tool, annotate the spectrogram, and export the resulting annotation file.  

Many of you may be new to Galaxy or to audio annotation, so we will go step by step. If you're new to Galaxy, feel free to check out other tutorials like [A short introduction to Galaxy]({% link topics/introduction/tutorials/galaxy-intro-short/tutorial.md %}) or [Galaxy Basics for everyone]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.md %}).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

So, the first step is to your audio file into Galaxy. 
In this tutorial, we will use the recording mentioned earlier, but feel free to use your own file(s). The NEAL interactive tool allows working on multiple files, so you can upload the number of files you want.

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name, e.g., "NEAL tutorial" for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
> 
> 2. Import the example audio file from [xeno-canto](https://xeno-canto.org/1008421/download)
>    Audio file :
>    
>    ```
>    https://xeno-canto.org/1008421/download
>    ```
>    
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_remote_files.md %}
>
> 3. Rename the dataset (default name: *Download*) to something more explicit, e.g., **1008421.wav** to have a name linked to the xeno-canto identifier.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>    
> > <tip-title>Provide a dataset name during import</tip-title>
> >
> >   Note that you also can provide the **1008421.wav** filename on the "Paste/Fetch data" upload module to avoid renaming after importing the file.
> >
>{: .tip}
>    
> 4. Make sure the dataset's datatype is `.wav`, as the tool only supports this format.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 5. You can add a tag to your dataset if you need to distinguish multiple datasets in your history.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


# Run NEAL interactive tool

All the work will be done inside the NEAL interactive tool, so let's run it.

> <hands-on-title> Run NEAL interactive tool</hands-on-title>
>
> 1. {% tool NEAL %} with the following parameters:
>    - {% icon param-file %} *"Input sound data file(s) in wav format"*: 1008421.wav
>    - {% icon param-file %} *"Do you already have a labels file?"*: Start with a fresh labels file
>
> 2. Wait for the tool to deploy it. It may take few a minutes, but you will see a message like this: 
>
>      ![Wait status](./images/wait_status.png "Wait status example")
>
> 3. You can now display the interactive interface. You can display it with different methods:
>
>    1. You can open it clicking ont the status message "Open"
>
>      ![Open status](./images/open_status.png "Open status example")
>    
>    2. You can open it by clicking on the link next to the red square in the "Launch and manage interactive tool" panel.
>
>      ![Manage interactive tool](./images/manage_IT.png "Manage interactive tools panel")
>    
>    3. You can also click on the eye icon on the "yellow" "NEAL on dataX" history dataset.
>
>      ![History status](./images/history_status.png "History status")
>
{: .hands_on}

# Inside the NEAL interactive tool

Congratulations 🎉 You are now inside the NEAL interface. Let’s explore the main features.

## Start session

To start annotating your audio file, you will need to start the session.

> <hands-on-title> Start session </hands-on-title>
>
> 1. Click on the **user icon** (top left).
>
> 2. Click the green **Start labelling** button.  
>     ![Green button](./images/green_button.png "Green button example")  
>
> 3. The **sonogram** of the audio file will appear.  
>     ![Sonogram](./images/sonogram.png "Sonogram example")
>
{: .hands_on}

## Listen to the audio

> <hands-on-title></hands-on-title>
>
> Use the **play button** below the sonogram to listen to the audio recording.  
> You can pause, stop, or replay as needed. And, adjust playback speed if necessary.
>
> > <warning-title></warning-title>
> >
> > As you can see, regardless of the total duration of your file, NEAL lets you listen to the audio in **15-second segments**.
> >
> {: .warning}
>
{: .hands_on}


## Annotate the audio file 

### Add a label

If this is your first annotation, you will need to **create a label** (e.g. "Blackbird song", "Noise", etc.). 

> <hands-on-title> Add a label </hands-on-title>
>  
>  1. Type the label name (e.g. *Turdus merula*).  
>  2. Click the **Add** button in the *Category buttons* section.  
> 
>     ![Add labels](./images/turdus_merula.png "Add labels")
>
{: .hands_on}

For this tutorial, you will only need the **Turdus merula** label you just added and the **Insect Noise** label, already present in the *Class List: Richfield*.

![Class list](./images/class_list.png "Class list")

### Select a time/frequency region

> <hands-on-title></hands-on-title>
>
> 1. Click and drag on the sonogram to select a **time–frequency region**.
>
>    The selected area will be highlighted. Here we can try to two different frequencies ranges : 
>
>      - One selection between 1 kHz and 10 kHz:
>       ![Selection on sonogram](./images/select.png "Selection on sonogram")
>
>      - One selection between 0kHz and 1kHz:
>       ![Selection on sonogram second example](./images/select_insect.png "Selection on sonogram second example")
>
> > <question-title></question-title>
> >
> > 1. Do you hear different sounds between the two frequencies we select? 
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are two sounds: 
> > >   - In the higher frequencies, you can hear *Turdus merula* sounds
> > >   - In the lower frequencies, you can hear a flying insect noise.
> > >
> > {: .solution}
> {: .question}
{: .hands_on}

### Annotate the selection

After selection of an area, you can assign it a label. 

> <hands-on-title> Add a label to the selected area </hands-on-title>
>
> 1. After selecting a region, choose the label that matches what you hear in the **Class list**.
>
> 2. Click on the **Save selection** button 
>       - This will link the time–frequency selection to a label. 
>       - You can now annotate the *Turdus merula* sound...
>         
>     ![Merula noise](./images/merula_noise.png "Merula Noise")
> 
>       - ... and the insect noise we selected in the previous step.
> 
>     ![Insect noise](./images/insect_noise.png "Insect Noise")
>
{: .hands_on}

## Download the label file

Once you have finished annotating your file you can save and download your annotations. 

> <hands-on-title></hands-on-title>
>
> 1. Export the labels by downloading the CSV file containing your annotations.
>    Simply click the **download** button in the **Label buttons** area:
>
>    ![Labels save](./images/labels_save.png "Labels save")
>
>    This CSV file can be re-used later to resume annotation.
>
{: .hands_on}

## Exit the interactive tool

After saving your file you can exit the NEAL interactive tool. 

> <hands-on-title></hands-on-title>
>
> 1. Click on the **End Labelling** button to end labelling
>
>    Your sonogram will disappear. 
>
> 2. Click on the **log out** button before leaving the NEAL interface.  
>    ![Log out example](./images/log_out.png "Log out example")
>
{: .hands_on}

## Shutdown the tool

Back on Galaxy, you can now Shutdown the NEAL interactive tool to free resources. 

> <hands-on-title></hands-on-title>
>
> To do so you can :
>
> 1. Click on the square on the tool running in your galaxy history.
>
>  ![Shutdown example](./images/history_status.png "Shutdown example")
> 
> 2. Click on the red square in the "launch and manage interactive tools" panel.  
>
>  ![Shutdown example 2](./images/manage_IT.png "Shutdown example 2")
>
> > <warning-title></warning-title>
> >
> > Don't worry if the NEAL tool appears in red in your history after shutting it down.
> > This is due to how Galaxy handles the termination of interactive tools and won't impact your work.  
> >
>{: .warning}
{: .hands_on}

# Rerun NEAL interactive tool with a previous labels file

If you need to continue your work on this audio file later you can restart the NEAL interactive tool using the CSV file containing your previous labels. 

> <hands-on-title> Rerun NEAL interactive tool </hands-on-title>
>
> 1. {% tool NEAL %} with the following parameters:
>    - {% icon param-file %} *"Input sound data file(s) in wav format"*: 1008421.wav
>    - {% icon param-file %} *"Do you already have a labels file?"*: Load a previous labels file
>    - {% icon param-file %} *"Input sound data labels file in csv format"*: my_labels_file.csv
>
{: .hands_on}

You can now continue your work from where you left off.  

> <warning-title></warning-title>
>
> Using it into Galaxy, NEAL does not automatically save labels in its **Class List**.  
> Labels are stored only inside the sonogram annotations.  
> This means you may need to recreate/re-add labels before adding new annotations.  
>
{: .warning}

# Conclusion

👏 Congratulations! You have completed the NEAL tutorial.  

You have learned how to:

  - Import audio data into Galaxy  
  - Launch and use the NEAL interactive tool  
  - Annotate audio files  
  - Export and re-use annotation files  

If you have questions or encounter issues, please check the documentation or contact the Galaxy community for support.


topic: ecology
tutorial: ENA_Biodiv_submission
title: Data submission using ENA upload Tool
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: ["biodiversity"]
content_hash: ff51c7b98e43f0d71db1cf87e8246198
last_modified: 2026-05-27T09:01:48.376381
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/ENA_Biodiv_submission/tutorial.html

content:
This tutorial will guide you through the necessary steps to manage and prepare sequencing files (ab1, FASTQ, FASTA) for submission to the genomic database ENA.
This workflow will take you from raw sequences in AB1 format through all the necessary steps to integrate these sequences into the ENA genomic database. We will convert the files into FASTQ and FASTA formats after performing quality control.
Additionally, we will perform alignments with the NCBI database to ensure the accuracy of your sequences.You will then need to fill a metadata Excel template to use the ENA upload Tool.
The worklow is made of 17 Galaxy tools, we will  present them and explain what they do.
The goal is to present an accessible and reproductible workflow for data submission.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare raw data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. **Create a new history** for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Import** the raw sequences files.
>
>    ```
>    https://data.indores.fr/api/access/datafile/3673
>    https://data.indores.fr/api/access/datafile/3609
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} your datafiles
>    - `3673` becomes `A2_RC_8F2_B.pl_HCOI.ab1`
>    - `3609` becomes `A12_RC_9G4_B.md_HCOI.ab1`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. **Check the datatype**
>    - Make sure it is `ab1`, and change it if not.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 5. **Build a Collection** containing these two files, you can ame i "ab1" for example
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}


## Tools used in the "Prepare Data submission" Workflow

Following steps take as input ab1 sequences files and produce filtered FastQ and Fasta files so sequences passing the quality checks are compared to NCBI nucleotidic database using Blastn operation.

### Converting Ab1 files to FASTQ

> <hands-on-title> ab1 to FASTQ converter </hands-on-title>
>
> 1. {% tool [ab1 to FASTQ converter](toolshed.g2.bx.psu.edu/repos/ecology/ab1_fastq_converter/ab1_fastq_converter/1.20.0) %} with the following parameters:
>    - {% icon param-collection %} *"Input ab1 file"*: `ab1` data collection created at the previous step
>
{: .hands_on}


### Quality Control

We are doing a first Quality control on the raw files using [Falco](https://falco.readthedocs.io/en/latest/) and MultiQC. Falco is an efficiency optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).

> <hands-on-title> Falco </hands-on-title>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Raw read data from your current history"*: `ab1.fastq` data collection created at the previous step
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `FastQC` (no matter whether FastQC or Falco was used)
>    - In *"FastQC output"*:
>         - {% icon param-file %} *"RawData FastQC output"*: `Falco on collection X:` data collection created at the previous step
>
> 3. *Check on the HTML files the general quality statistics of your sequences*
>
{: .hands_on}


> <question-title> Question </question-title>
>
> 1. What is the quality of your sequences?
> 2. Do you have adapters?
>
> > <solution-title></solution-title>
> >
> > 1. Quality is quite good looking at the "status checks" section of MultiQC. As expected (because here we only have one sequence by file) "Per base sequence Content" and "Overrepresented sequences" Sections are "bad" for both sequences files. "adapter content" section also show a "bad" result for A2_RC_8F2_B.pl_HCOI.ab1 file.
> >
> > 2. A2_RC_8F2_B.pl_HCOI.ab1 file seems to have adapters in it.
> >
> {: .solution}
{: .question}

# Cleaning the Data

## Cutadapt

Cutadapt enables the removal of adapters, polyA tails, and other artifacts from sequences. The tool also filters reads based on quality.

> <hands-on-title> Cutadapt </hands-on-title>
>
> 1. {% tool [Cutadapt](toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/4.8+galaxy0) %} with the following parameters:
>     - {% icon param-collection %} *"FASTQ/A file"*: the collection with your data (output of {% icon tool %} **ab1 to FastQ converter**)
>     - **"Single-end or Paired-end reads?"**: `Single-end`
>     - In **"Other Read Trimming Options"**:
>         - **"Quality cutoff(s) (R1)"**: `30`
>         - **"Shortening reads to a fixed length"**: `Disabled`
>
> > <comment-title> Suggestions </comment-title>
> >
> > You may consider changing these parameters depending on the quality of your dataset.
> {: .comment}
>
{: .hands_on}

> <comment-title> Quality Control </comment-title>
>
> We do a second quality control similar to the first one to check the quality of the sequences after cleaning them.
{: .comment}


## Quality Control with Falco and MultiQC

> <hands-on-title> Falco </hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Raw read data from your current history"*: output from {% icon tool%} **Cutadapt**
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>        - *"Which tool was used generate logs?"*: `FastQC` (no matter whether FastQC or Falco was used)
>        - {% icon param-repeat %} *"Insert FastQC output"*
>          - {%  icon param-collection %} *"FastQC output"*: the `raw` output from {% icon tool %} **Falco**
>
>    > <comment-title> Comment </comment-title>
>    >
>    > You should notice an improvement on the quality of your sequences.
>    {: .comment}
>
{: .hands_on}

## Filtering the collection


> <hands-on-title> Filter empty datasets </hands-on-title>
>
> 1. {% tool [Filter empty datasets](__FILTER_EMPTY_DATASETS__) %} with the following parameters
>    - {% icon param-collection %} *"Input Collection"*: output collection from Cutadapt step
>
> 2. {% tool [FASTQ Groomer](toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.1.5+galaxy2) %} with the following parameters:
>    - {% icon param-collection %} *"File to groom"* : output collection from the {% icon tool %} **Filter empty datasets**
>
>    This step is notably there to produce "standardized" fastqsanger sequences files so we can then use other tools accepting only such data format.
>
> 3. {% tool [Filter FASTQ](toolshed.g2.bx.psu.edu/repos/devteam/fastq_filter/fastq_filter/1.1.5) %} with the following parameters:
>    - *"FASTQ File"*: output collecton from  {% icon tool %} **FastQ Groomer**
>    - *"Minimum size"*: `300`
>
>    > <comment-title> Comment </comment-title>
>    >
>    > Here we descide to only keep sequences of 300bp or above, you may change this parameter depending on your dataset
>    {: .comment}
>
{: .hands_on}


### Changing files names

> <hands-on-title>Extract element identifiers and remove extensions</hands-on-title>
>
> 1. {% tool [Extract element identifiers](toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2) %}
>    - {% icon param-collection %} *"Dataset collection"*: output from the previous step
>
> 2. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3) %} with the following parameters:
>    - *"Select lines from"*: output of the previous step
>    - In *"Check"*:
>      - {% icon param-repeat %} *"Insert Check"*
>        - *"Find Regex"*: `.ab1`
>        - *"Replacement"*: ``
>
>    > <comment-title> Comment </comment-title>
>    >
>    > This is to ensure that all your files names end with .fastq.gz
>    {: .comment}
>
> 3. {% tool [Paste](Paste1) %} with the following parameters:
>    - {% icon param-file %} *"Paste"*: the file from {% icon tool %} **Extract element identifiers**
>    - {% icon param-file %} *"and"*: the file from {% icon tool %} **Regex Find And Replace**
>    - {% icon param-select %} *"Delimited by"*: Tab
>
> 4. **Check the datatype**
>    - should be 'tabular'. If not, change it now.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
{: .hands_on}



> <hands-on-title> Relabel identifiers </hands-on-title>
>
> 1. {% tool [Relabel identifiers](__RELABEL_FROM_FILE__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: output from {% icon tool %} **Filter FastQ**
>    - *"How should the new labels be specified?"*: `Map original identifiers to new ones using a two column table.`
>
{: .hands_on}


## Alignments on NCBI database

> <hands-on-title> NCBI BLAST alignment </hands-on-title>
>
> 1. {% tool [FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5) %} with the following parameters:
>    - {% icon param-collection %} *"Input FASTQ File"*: output collection from {% icon tool %} **Relabel Identifiers**
>
> 2. {% tool [NCBI BLAST+ blastn](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.14.1+galaxy2) %} with the following parameters:
>    - {% icon param-collection %} Nucleotide query sequence(s): output from the previous step
>    - *"Subject database/sequences"*: `Locally installed BLAST database`
>        - *"Nucleotide BLAST database"*: `NCBI NT (01 Sep 2023)`
>    - *"Output format"*: `Tabular (extended 25 columns)`
>    - *"Advanced Options"*: `Hide Advanced Options`
{: .hands_on}


> <hands-on-title> Extracting best hits </hands-on-title>
>
> 1. {% tool [Unique](toolshed.g2.bx.psu.edu/repos/bgruening/unique/bg_uniq/0.3) %} with the following parameters:
>    - {% icon param-collection %} *"File to scan for unique values"*: output from the previous step
>    - *"Advanced Options"*: `Show Advanced Options`
>        - *"Column start"*: `c1`
>        - *"Column end"*: `c1`
>
{: .hands_on}

## Workflow Outputs

1. **Collection of raw FASTQ files:** Input AB1 files converted into FASTQ files.

2. **Collection of FASTQ files (after quality control)**: Renamed Fastq files ready for submission after quality control and filtering.

3. **Collection of FASTA files**: FASTQ files converted into FASTA format. Used for conducting BLAST alignments.

4. **Falco Quality Control Results** before and after cleaning: Both raw Falco results and HTML reports are created

5. **MultiQC Quality Control Results** before and after cleaning: Both raw MultiQC statistics and HTML report are created

6. **Raw Blast Results**: Results of BLAST alignments conducted on our sequences. Columns names are:

   | Column | NCBI name    | Description                                    |
   |--------|--------------|------------------------------------------------|
   | 1      | qaccver      | Query accession dot version                    |
   | 2      | saccver      | Subject accession dot version (database hit)   |
   | 3      | pident       | Percentage of identical matches                |
   | 4      | length       | Alignment length                               |
   | 5      | mismatch     | Number of mismatches                           |
   | 6      | gapopen      | Number of gap openings                         |
   | 7      | qstart       | Start of alignment in query                    |
   | 8      | qend         | End of alignment in query                      |
   | 9      | sstart       | Start of alignment in subject (database hit)   |
   | 10     | send         | End of alignment in subject (database hit)     |
   | 11     | evalue       | Expectation value (E-value)                    |
   | 12     | bitscore     | Bit score                                      |
   | 13     | sallseqid    | All subject Seq-id(s), separated by a ';'      |
   | 14     | score        | Raw score                                      |
   | 15     | nident       | Number of identical matches                    |
   | 16     | positive     | Number of positive-scoring matches             |
   | 17     | gaps         | Total number of gaps                           |
   | 18     | ppos         | Percentage of positive-scoring matches         |
   | 19     | qframe       | Query frame                                    |
   | 20     | sframe       | Subject frame                                  |
   | 21     | qseq         | Aligned part of query sequence                 |
   | 22     | sseq         | Aligned part of subject sequence               |
   | 23     | qlen         | Query sequence length                          |
   | 24     | slen         | Subject sequence length                        |
   | 25     | salltitles   | All subject title(s), separated by a '<>'      |

7. **Filtered Blast Results**
Files containing the closest homologous sequences.

8. **Collection of Fastq files**
Contains filtered sequences.

# How to use ENA upload Tool

## Adding ENA "Webin" credentials to your Galaxy user information

> <comment-title> Having an ENA Submission Account </comment-title>
>
>  Make sure you have a submission account with the European Nucleotide Archive (ENA). You will need the identifier and the password, available through https://www.ebi.ac.uk/ena/submit/webin/login.
>
{: .comment}


>  <hands-on-title> Add your "WEBIN" credentials to your Galaxy account </hands-on-title>
>  **Instructions:**
>	- From the Menu, click on "User" > "Preferences". Click on "Manage Information". Scroll down to "Your ENA Webin account details" and enter your ENA "Webin" identifier and password.
>  ![Adding ENA Webin credentials](./images/imageI.png)
{: .hands_on}

## Submitting using a metadata template file

For this tutorial we will use the ENA default sample checklist.

![Excel Metadata template](./images/image2.png)

**Note:** It is crucial to fill in all the fields marked "Mandatory" and ensure that the sequence names match exactly those indicated in the Excel file.

> <comment-title> ENA Metadata Templates </comment-title>
>
> You can find metadata templates for each checklist in the [ELIXIR-Belgium GitHub repository](https://github.com/ELIXIR-Belgium/ENA-metadata-templates)
>
> 1. Direct download link of the [ENA default sample checklist]( https://github.com/ELIXIR-Belgium/ENA-metadata-templates/raw/main/templates/ERC000011/metadata_template_ERC000011.xlsx)
>
> 2. Direct download link of the [ENA default sample checklist filled with elements for the training](https://github.com/galaxyproject/training-material/raw/24776cf161e38ac0449755749d23e851400020aa/topics/ecology/tutorials/ENA_Biodiv_submission/metadata_GdBqCOI_ERC000011_Test.xlsx)
>
> You will need to import this file into your Galaxy history. Then, use the ENA Upload Tool to proceed with the submission.
>
{: .comment}



> <hands-on-title> Excel Metadata Template </hands-on-title>
>
> 1. Import the ENA default sample checklist file.
>
>    ```
>    https://github.com/galaxyproject/training-material/raw/24776cf161e38ac0449755749d23e851400020aa/topics/ecology/tutorials/ENA_Biodiv_submission/metadata_GdBqCOI_ERC000011_Test.xlsx
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. {% tool [ENA Upload tool](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} with the following parameters:
>    - *"Action to execute"*: `Add new (meta)data`
>    - *"Select the metadata input method"*: `Excel file`
>    - *"Select the ENA sample checklist"*: `ENA default sample checklist (ERC000011)`
>    - *"Select Excel file based on template"*: `metadata_GdBqCOI_ERC000011_Test.xlsx`
>    - *"Select input data"*: `Dataset or dataset collection`
>    - *"Add .fastq (.gz, .bz2) extension to the Galaxy dataset names to match the ones described in the input tables?"*: `Yes`
>
>    > <comment-title> Datatype </comment-title>
>    >
>    > The ENA upload tool will then automatically compress fastq sequences files into .fastq.gz format before submission
>    {: .comment}
>
>    > <warning-title> Danger: Submit to ENA test server! </warning-title>
>    > We suggest you first submit to the [ENA test server](https://wwwdev.ebi.ac.uk/ena/submit/webin/) before making a public submission! Submission can be seen in `Dashboard/Study Report`
>    {: .warning}
>
> ![ENA Upload tool](./images/image3.png)
>
{: .hands_on}



# Conclusion

This tutorial guides you through quality check and preparing raw data files for ENA submission. You can then verify that your sequences have been successfully sent by logging into the Test ENA portal (https://wwwdev.ebi.ac.uk/ena/submit/webin/login) and navigating to the Study Report section.

tools_json:
["ab1 to FASTQ converter", "toolshed.g2.bx.psu.edu/repos/ecology/ab1_fastq_converter/ab1_fastq_converter/1.20.0", "Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1", "Cutadapt", "toolshed.g2.bx.psu.edu/repos/lparsons/cutadapt/cutadapt/4.8+galaxy0", "Filter empty datasets", "__FILTER_EMPTY_DATASETS__", "FASTQ Groomer", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_groomer/fastq_groomer/1.1.5+galaxy2", "Filter FASTQ", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_filter/fastq_filter/1.1.5", "Extract element identifiers", "toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2", "Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3", "Paste", "Paste1", "Relabel identifiers", "__RELABEL_FROM_FILE__", "FASTQ to FASTA", "toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5", "NCBI BLAST+ blastn", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.14.1+galaxy2", "Unique", "toolshed.g2.bx.psu.edu/repos/bgruening/unique/bg_uniq/0.3", "ENA Upload tool"]


topic: ecology
tutorial: MetaShARK_tutorial
title: Creating metadata using Ecological Metadata Language (EML) standard
difficulty: intermediate
time_estimation: 30M
tools_json: ["MetaShARK", "interactive_tool_metashark"]
requirements_json: []
tags_json: ["metadata", "EML", "ecology", "biodiversity", "FAIR", "data paper"]
content_hash: c2f8ff04b5451791f7b9c9697a38ec58
last_modified: 2026-05-27T09:01:48.388659
zenodo_link: https://zenodo.org/records/10663465
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/MetaShARK_tutorial/tutorial.html

content:
<p align="justify">This tutorial aims to teach <b>how to use functionalities of the EML Assembly Line R package to produce rich metadata</b> using the Ecological Metadata Language (EML) international metadata standard. Here, we will notably propose a concrete example on how to use <b>Galaxy Ecology tools to create, evaluate and modify EML metadata</b> content using both EML Assemby Line metadata template tabular files, easily readable and editable by humans, and XML file, devoted to machine.</p>

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# 1] How can EML Assembly Line functionalities help producing rich metadata easily?

<p align="justify"> A major gap when a researcher is writing metadata documents is the fact that metadata international <b>standards often use formats not really human readable</b> and/or editable as XML or JSON. To answer this issue, <b>[Environmental Data Initiative](https://edirepository.org/) (EDI)</b> through the <b>EML Assembly Line R package</b> propose to generate intermediate metadata template files using classical tabular text format.
Another major issue regarding <b>metadata fill in</b>, is the fact that one <b>need to take a lot of time to write</b>, and often rewrite, metadata elements who can be already filled using automatic inferences or use of webservices. Here again, Environmental Data Initiaitve (EDI) through the <b>EML Assembly Line R</b> package propose to <b>generate automatically information</b> related to data attributes, geographic coverage, taxonomic coverage, using the content of provided datafiles.
<br><br>
Finally, through the MetaShARK R Shiny app, an R Shiny app in beta version for test, created by the french biodiversity data hub research infrastructure (Pôle national de données de Biodiversité (<b>PNDB</b>)), user can use a graphical user interface to apply the EML Assembly Line workflow and benefit from some additionnal functionnalities as:</p>

- Capacity to associate terminological resources terms coming from Bioportal ontologies to data attributes as keywords using CEDAR API
- Automatic fill in of personal information using ORCID API
- Automatic production of a data paper draft

<br>

> <comment-title>What is a <b>Data Paper</b> ?</comment-title>
> According to the [GBIF](https://www.gbif.org/data-papers) (Global Biodiversity Information Facility), 
> A data paper is a peer reviewed document describing a dataset, published in a peer reviewed journal. It takes effort to prepare, curate and describe data. 
> Data papers provide recognition for this effort by means of a scholarly article.
{: .comment}



# 2] Get data to describe 💾📂

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial. You can name it "EML assembly Line tutorial" for example
> 2. Download all files on your local computer from Zenodo: https://zenodo.org/api/records/10663465/files-archive. It is neccessary as for now, MetaShARK is deployed from Galaxy but without having a possibility to directly populate MetaShARK app with datafiles from Galaxy. You will thus then have to upload manually data files from your local computer to MetaShARK.
> 3. Unzip the donwloaded archive so you can access each file independently
> 4. Import tsv, netcdf and geotiff data files directly from [Zenodo]({{ page.zenodo_link }}) so it can be used on some steps of the tutorial.
>     -> Training Data for "Creating metadata using Ecological Metadata Language (EML) standard with EML Assembly Line functionalities" Galaxy tutorial:
>    ```
>    https://zenodo.org/records/10663465/files/datafile_1.tsv
>    https://zenodo.org/records/10663465/files/datafile_2.tsv
>    https://zenodo.org/records/10663465/files/LakeGeneva_phytoplankton_1974-2004.nc
>    https://zenodo.org/records/10663465/files/Present.Surface.pH.tif
>    ```
>    ![Upload datafiles](./Images/0_Upload_datafiles.png)
> 6. Import shapefile related files into Galaxy using the Galaxy upload tool, then on the "composite" tab, specifying "shp" composite type, then uploading .dbf, .shp and .shx files on the dedicated spaces.
>    ![Upload shapefile](./Images/1_upload_shapefile.png)
{: .hands_on}

# 3] MetaShARK 🦈 : Write rapidly an EML document through a work in progress interactive app 

To deploy a MetaShARK app, you can go to the Galaxy tool {% tool [MetaShARK](interactive_tool_metashark) %} and click "Execute". Then, you have to wait the launch of the app, and when ready to be used, you will see the message "There is an InteractiveTool result view available," with an hyperlink on the "Open" statement allowing you to reach the app clicking on it.

> <comment-title>WARNING</comment-title>
> 🚧 Note that MetaShARK R shiny app is in beta version. You can thus encounter issues using it!
{: .comment}

When oppening MetaShARK, you will have an interface looking like this :

![MetaShARK interface](./Images/2_MetaShARK.png)

To start creating metadata, you need to reach the **Fill in EML** module, then specify or complete the automatic `Data package name` and mention the `Dataset title`. Here a title can be "Manage heterogeneous data files through EML". Finally, you can choose an open licence between CC-BY-4.0, default, or CC0 then click on "Create". 

![Create EML Assembly Line "data package"](./Images/3_MetaShARK_create.png)

Then you can upload datafiles. Here, you can import these files from the downloaded Zenodo archive (link : https://zenodo.org/api/records/10663465/files-archive):

📁 List of datasets : 
 - datafile_1.**tsv**
 - datafile_2.**tsv**
 - LakeGeneva_phytoplankton_1974-2004.**nc**
 - Present.Surface.pH.**tif**
 - 02_Ref.**shp**
 - 02_Ref.**shx**
 - 02_Ref.**dbf**

> <comment-title>NOTE</comment-title>
> The folder contains many files with different extensions but MetaShark will normally recognise several types of file extension (notably .tsv for tabulated text files, shapefiles, .geotiff and .geojson)!
{: .comment}

![Upload datafiles on MetaShARK](./Images/4_MetaShARK_upload.png)

<p align="justify">MetaShARK will normally guess that the three `02_Ref` files are representing a uniq shapefile. MetaShARK will normally guess each data type and infer list of attributes for each file but the geotiff `Present.Surface.pH.tif` one. So now you need to select this datafile and upload the `attributes_Present.Surface.pH.txt` metadata template file so MetaShARK can fill attributes of this file (here the attribute is named "Present.Surface.pH").</p>

![Upload metadata template for geotiff attributes](./Images/5_MetaShARK_geotiffattributes.png)

<p align="justify">Then you can provide a description for this attribute, for example "Present surface pH", then look at each attribute information of each data file so you can click on the "Next" button and go to the next step, to give informations on categorical variables!</p>

![Categorical variables](./Images/6_MetaShARK_catvars.png)

<p align="justify">Clicking "Next" button will then allows you to fill spatial informations about all GIS recognized datafiles, here the `Present.Surface.pH.tif` geotiff raster file and the `02_Ref` shapefile vector file. Geotiff is in pixel, accuracy can be set to unknown and shapefile is in Point, both are in `GCS_WGS_1984`spatial reference.</p>

![Spatial information](./Images/7_MetaShARK_spatialinfo.png)

<p align="justify">Next step is devoted to specifying geographic coverage.<br><br> You can use a method between "columns" or "custom". "Custom" allows you to create one to several geographical sites using a map widget where you can draw limits of each site or enter directly latitude and longitude coordinates. "Columns" method, used here, allows you to specify an attribute containing site names then associated latitude and longitudes attributes.</p>

![Geographic coverage](./Images/8_MetaShARK_geocov.png)

<p align="justify">Now geographic coverage is set, one can specific taxonomic coverage.<br><br> To do so, you can select a data attribute containing taxonomic information then select kind of notation you want to have and finally on which taxonomic authority (or authorities) information will be compared. Note that this can take a while if you have a lot of taxons and time is duplicated for each selected additional authority.</p>

![Taxonomic coverage](./Images/9_MetaShARK_taxcov.png)

<p align="justify">Now we can fill personal informations.<br><br> To do so, the easiest way is to provide ORCID identifiers for each individual person involved as creator, contact and/or PI. Depending on the information filled in ORCID by each individual and on the level of accessibility of each, all field can be automatically filled. If "PI" is selected, you can specify a project name, funder name and related funding number.</p>

![Personal informations](./Images/10_MetaShARK_personal.png)

Then you can add final elements as:

- An abstract (writing directly on the dedicated field or uploading a text file containing the abstract)

![Abstract](./Images/11_MetaShARK_abstract.png)

- Methods (writing directly on the dedicated field or uploading a text file, can be in markdown, containing details of the methods used to create data files)

![Methods](./Images/12_MetaShARK_methods.png)

- Keywords, who can be linked to keyword thesaurus. This allows you to create "group" of keywords and/or refer to existing terms classifications as we can find in terminological resources such as ontologies or thesaurus.

![Keywords](./Images/13_MetaShARK_keywords.png)

> <comment-title>You can add Semantic Annotations</comment-title>
> **You can add [semantic annotations](https://eml.ecoinformatics.org/semantic-annotation-primer)!**
> 
>  
> <p align="justify"> To do so, you need to reach the MetaShARK parameters (upper right icon) then enter your CEDAR token. To create a CEDAR account, you can 1/ register here http://cedar.metadatacenter.org/ then 2/ go on the "profile" on https://cedar.metadatacenter.org/ and there 3/ you can find the API key.</p>
> 
>  API key format to enter is something like: 
> ```
> apiKey 205b1e521f2eaf0ad4a361c438b638a68348bea313caeae310839f919629525d
> ```
> 
> <p align="justify">You then can  use the `+` button on the keyword space to **Add keyword with dataset annotation**. You will have to choose a "predicate", from IAO ontology, then an "object" from ontologies coming from Bioportal to add information concerning a "subject", the ‘thing’ being annotated, here, regarding keyword, "dataset", but you can also apply the same to datafiles "attributes".</p>
> 
> ![Add semantic annotation predicate](./Images/17_MetaShARK_annotations_predicate.png)
> 
> ![Add semantic annotation object](./Images/18_MetaShARK_annotations_object.png)
> 
{: .comment}

<p align="justify">Finally, you can specify a temporal coverage and go to the last step of this MetaShARK workflow: Generat an EML metadata file! If everything is ok, you will have creation of an EML metadata file.</p>

![Produce EML metadata file](./Images/14_MetaShARK_makeeml.png)

<p align="justify">Once EML written, you can <b>download the data package through the button</b> "Download Data Package". This will allow you to <b>download a zip archive</b> you can unzip on your local computer. Resulting files are organized through 2 main folders : </p>

- **A main folder with data_objects**
  -  `all datafiles ` you uploaded into MetaShARK
  -  `eml ` which is the EML metadata file written in XML format
  -  `metadata_templates ` with all metadata files written in text format, column separated by tabulations
  
- A **second folder called "emldown"** where a **draft of data paper** written in html format can be accessed

<hr>
<p align="center"><b>👏 Congratulations! You've just produced your first EML yourself!👏</b></p>
<hr>

> <warning-title>MetaShARK can Freeze</warning-title>
> **At this final stage, MetaShARK can freeze and show like a "front grey filter"**
> 
> If this is the case, you can refresh MetaShARK app pressing F5 button of your keyboard for example. To dowload the resutling data package, you can go to the "Fill In" module, select the data package you created and click on "Download.zip".
{: .warning}

> <tip-title>Returning on your MetaShARK instance</tip-title>
> If you close MetaShARK navigator window, you can reopen it clicking on Menu > User > Active InteractiveTools.
> You can here click on "metashark visualization" hyperlink. You can then go on the Fill In module, select 
> the data package you created and click **Load**.
{: .tip}


# 4] MetaShRIMPS 🦐 : Easily FAIRness assessment and data paper sketches creation

To evaluate and modify metadata elements you have created, you can upload the EML xml file on the Galaxy history you created and all MetaShARK metadata templates files.

![Use MetaShRIMPS](./Images/15_MetaShRIMPS.png)

Open the [MetaShRIMPS interactive tool form](https://ecology.usegalaxy.eu/root?tool_id=interactive_tool_metashrimps) and select the EML xml file you just generate with MetaShARK. Clicking on Execute will launch the MetaShRIMPS R Shiny app, the message `There is an InteractiveTool result view available, waiting for view to become active... ` is displayed until the app will be ready to use. After some time, the message `There is an InteractiveTool result view available` confirm the app is deployed and you can access it clicking on **Open**. You will then have an interface looking like this:

![MetaShRIMPS use](./Images/upload_1.png)

Click on the **Execute** Button, 2 new tabs called "Draft of Data Paper" and "Fair Assessment"  will appear.
You can access all of the tool outputs by clicking on each tab (it can take a little time for your results to be displayed).

# 5] Draft of Data Paper 📝

<p align="justify">By clicking on the <b>"Draft of Data Paper"</b> tab, you will have access to the draft of Data Paper presented in an HTML format. You can either navigate through the Data Paper with the tabs or with the scrollbar on the right and access different elements.</p>

- You can at the top of the page **download the draft** in either an _HTML format_ .... :

![MetaShRIMPS data paper download in HTLM format](./Images/Download_HTML.png)

- .... or an editable _docx format_ :

![MetaShRIMPS editable data paper download in docx format](./Images/Download_docx.png)

<hr>
Voir comment mettre les deux photos côtés à côtes ou un truc mieux
<hr>

# 6] FAIR Quality Assessment report 📊✅

[FAIR](https://www.go-fair.org/fair-principles/) stand for **Findable, Accessible, Interoperable, Reusable**. 
<br> <br>

![FAIR principles](./Images/FAIR_data_principles.jpg)

 <br>

<p align="justify">These principles were to <b>improve the access and usabiliy of data</b> by the machine and to help <b>making data reusable and shareable for users</b>. It covers the whole concept of why it is necessary to produce a rich and described metadata in order to permit external users to <b>understand and reuse data</b> for their own studies.
  
There are several ways of computing the FAIR index, for each letter of the word is associated with a degree of FAIRitude of the data.</p>

> <comment-title>Metadata Quality</comment-title>
> The aim is to propose a metadata quality index and to see how it is possible to improve, not to punish the user 👍
{: .comment}

By clicking on the `Fair Assessment` tab, you will access the FAIR Quality report of the metadata uploaded. 

You will have access to different figures such as a **table displaying the results of all checks tested for your metadata**:

![Table of results](./Images/Fairscore_tab2.png "Example of the table displaying the results of the Quality Checks")

You will also have acces to a graph presenting scores of Quality for each of the FAIR principles tested (Findable,
Acessible, Interoperable, Reusable) on a 100 point scale.

![FAIR scores](./Images/Fairscore_tab.png "Overall FAIR score")

<hr>
Image un peu laide, je l'a modifierai un peu sur Paint pour optimiser sa taille vis à vis du vide/blanc occupé par certaines parties
<hr>

> <comment-title>How you can improve your FAIR score?</comment-title>
> 
> You can look at the lines of the table for FAILURE (red) and WARNING (yellow) status. Two WARNING 
> status lines  are related to an abstract content too short and an attribute definition too small.
> 
> ![Warning status lines](./Images/WARNING.png "Overall FAIR score") 
{: .comment}

> <comment-title>Update metadata content to elevate FAIR score</comment-title>
>
> 1. Search "abstract" and "attributes_Present.Surface.pH.txt" metadarta files on your history
>
> 2. Modify each file using the Galaxy included text editor
>
>    To do so, you can go on the "visualize" functionnality of each datafile, clicking on the name of the dataset on your history then on the "visualize" button at the bottom of the dataset description. you can then select "Editor / Manually edit text" to update the content and generate a new version of the dataset.
>
> 3. Rename each datafile as original names ("abstract" and "attributes_Present.Surface.pH.txt"), you can add a "modified" tag / label so you can better remember in the future the modification state.
>
> 4. Recreate a data collection with all metadata template files, taking the new "abstract" and "attributes_Present.Surface.pH.txt" files instead of old ones.
>
> 5. You can recreate an EML metadata file with [**Make EML**](https://ecology.usegalaxy.eu/root?tool_id=toolshed.g2.bx.psu.edu/repos/ecology/makeeml/makeeml/0.1.1+galaxy1) EAL tool and then redeploy a metashrimps tool on the new EML. Before executing [**Make EML**](https://ecology.usegalaxy.eu/root?tool_id=toolshed.g2.bx.psu.edu/repos/ecology/makeeml/makeeml/0.1.1+galaxy1) EAL tool, one need to pay attention to the rename composite datafiles (as here the shapefile one) as named originally (so `02_Ref`) and then create a data collection gathering all "dataTable" datafiles (both .tsv files and .nc one), a data collection gathering all "spatialRaster" datafiles (here `Present.Surface.pH.tif`) and a data collection gathering all "spatialVector" datafiles (here `LakeGeneva_phytoplankton_1974-2004.nc`).
>
{: .comment}

# 7] Conclusion

Here is the end of this short tutorial aiming in explaining the purpose of MetaShARK and how to use it.

Don't hesitate to contact us if you have any questions ☺️


topic: ecology
tutorial: champs-blocs
title: Champs blocs indicators
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["ecosystem EBV class", "EBV dataset", "EBV workflow", "marine ecosystems", "biodiversity"]
content_hash: dbb159c3edd566497405caf838b6b296
last_modified: 2026-05-27T09:01:48.396328
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/champs-blocs/tutorial.html

content:
This tutorial will allow you to better apprehend the workflow for indicators on boulder fields. This will help you understand the effects of global and local changes on marine habitats, and the effectiveness of the management measures adopted, requires standardized monitoring and robust and sensitive indicators reflecting the state of the habitats.

The "mediolittoral boulder fields" habitat harbors a high diversity of micro-habitats and species due to its high structural heterogeneity and its intermediate position on the foreshore, which makes it one of the most diversified mediolittoral habitats and of high ecological interest along the French Channel-Atlantic coast. It is also a very attractive habitat for recreational fishing, which, through the reworking of the blocks, can impact the communities.
Thus, the "mediolittoral boulder fields" habitat has been the subject of several national and local initiatives (including LIFE+ "Experimentation for a sustainable and concerted management of recreational foot fishing in France" 2013-2017) to better assess its status and relate it to the pressure of foot fishing with a view to adapting local management, particularly through the network of Marine Protected Areas (Natura 2000, PNM, PNR etc.).

In particular, these projects have made it possible to develop a network of involved field players and managers and tools for evaluating the ecological status and pressure of fishing on foot:

- the Visual Boulder Turning Indicator (VTI), which is similar to a "landscape" indicator to assess fishing pressure based on architectural criteria;
- the Boulder Field Ecological Quality Index (BFEQ) - the subject of this report - based on biotic and abiotic variables that respond to the disturbance "boulder turning".

Here we will go over the different steps in order to get those 2 indicators and more.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <details-title>Short introduction on how Galaxy works</details-title>
>
> You can leave the tutorial and go bask on the main screen by clicking outside of the tutorial screen.
> You can come back to where you left off the tutorial anytime by clicking {% icon level %}.
>
> > <hands-on-title>Log in to Galaxy</hands-on-title>
> > 1. Open your favorite browser (Chrome, Safari or Firefox as your browser, not Internet Explorer!)
> > 2. Browse to your Galaxy instance
> > 3. Log in or register
> >
> > ![Screenshot of Galaxy interface with the register or login button highlighted](../../images/champbloc/galaxy_homepage.png "Galaxy homepage")
> >
> > This is an image of Galaxy Ecology, located at [usegalaxy.eu](https://ecology.usegalaxy.eu/)
> >
> {: .hands_on}
>
> The Galaxy homepage is divided into three panels:
> * Tools on the left
> * Viewing panel in the middle
> * History of analysis and files on the right
>
> ![Screenshot of the Galaxy interface, the tools panel is on the left, the main panel is in the center, and the history is on the right.](../../images/champbloc/galaxy_interface.png "Galaxy interface explanation")
>
> The first time you use Galaxy, there will be no files in your history panel.
{: .details}

Now let's focus on our workflow on boulder field ecological state

![Workflow for boulder field ecological state analysis](../../images/champbloc/workflow.png "Workflow")

# Get data

{% include _includes/cyoa-choices.html option1="Yes" option2="No" default="Yes" text="Are your ESTAMP data ready ?" %}

<div class="No" markdown="1">

1. Download your data on ESTAMP [estamp.ofb.fr](https://estamp.ofb.fr/) website, clicking on "Accédez aux données" at the bottom of the page. You will get a zip folder.
2. Unzip your folder. In the folder three files .csv will interest us :

   - champbloc_ivr.csv
   - champbloc_qecb.csv
   - ficheterrain.csv

</div>

<div class="Yes" markdown="1">
</div>

> <hands-on-title>Data upload</hands-on-title>
> 1. Import the data into Galaxy
>    
>  * Open the Galaxy Upload Manager {% icon galaxy-upload %}
>  * Select **Choose local files**
>    ![Choose local files](../../images/champbloc/upload.png "upload your data")
>  * Browse in your computer and get your ESTAMP files (select all three of them : champbloc_ivr.csv, champbloc_qecb.csv and ficheterrain.csv)
>   
>  * Press **Start**
>    ![Start charging your data](../../images/champbloc/start.png "Start")
>  * Press **Close**
>    ![Close upload your data](../../images/champbloc/close.png "Close")
>  You need to wait for the data to become green on the right panel
>    ![Data ready](../../images/champbloc/after_before.png "Data ready")
>   
> > <tip-title>Create a new history</tip-title>
> > Create a new history for this tutorial and give it a name (example: “Champs blocs indicators”) for you to find it again later if needed.
> > {% snippet faqs/galaxy/histories_create_new.md box_type="none" %}
> {: .tip}
{: .hands_on}


# Computing the Visual Boulder turning Indicator for each site

Based on the proportion of "turned" and "not turned" blocs this indicator vary from 0 to 5 and can be quickly used.

> <tip-title>How to get your tool ?</tip-title>
>
> > <hands-on-title>Use a tool</hands-on-title>
> > 1.  click on for example the {% tool [IVR](toolshed.g2.bx.psu.edu/repos/ecology/cb_ivr/cb_ivr/0.0.0) %} tool or type **ivr** in the tools panel search box (top left)
> > The tool form will be displayed in the central Galaxy panel.
> {: .hands_on}
{: .tip}


## IVR

Go on the tool **IVR**

> <hands-on-title>Calculating IVR</hands-on-title>
>
> 1. {% tool [IVR](toolshed.g2.bx.psu.edu/repos/ecology/cb_ivr/cb_ivr/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input champbloc_ivr.csv"*: `input` (champbloc_ivr.csv)
>    - {% icon param-file %} *"Input ficheterrain.csv"*: `input` (ficheterrain.csv)
>
> 2. Press **Run Tool** (it can take some minutes to process it's normal if you have to wait a bit especially if your internet connection is weak)
>
> 3. Three outputs will appear in your history panel on the right.
>
> 4. View graphs
>  * Once it's green, click on **IVR plot**
>  * Then, click on the {% icon galaxy-eye %} (eye) icon of your output in the history panel.
> 
>  The information is displayed in the central panel
> 
>   ![IVR](../../images/champbloc/ivr.png "IVR for Bilfot"){:width="620px"}
>
> > <tip-title>Search for your site results</tip-title>
> >
> > 1. In the top of the History panel (on the right) go on the search box
> > 2. Click on the double arrows to display more filter options
> > 3. Type the name of your site (for instance "Bilfot") on the "Filter by name:" criteria then click on "No" for the "Visible:" criteria, as plots are created in a data collection and not directly visible on the history
> >
> >  ![Bilfot filter](../../images/champbloc/filter_data.jpg "Filter IVR for Bilfot"){:width="620px"}
> >
> {: .tip}
> **For your reports you need to download the ones you want. You won't be able to vizualise them directly on Galaxy center panel !**
>
> 5. Download results
> 
>  * Click on the output you are interested in for instance **Reports**
>  * Click on {% icon galaxy-save %} (download)
{: .hands_on}

# Computing dissimilarity coefficient for each site

## Dissimilarity
Cleaning out your data in order and then compute de dissimilarity coefficient.
Get the {% tool [Dissimilarity](toolshed.g2.bx.psu.edu/repos/ecology/cb_dissim/cb_dissim/0.0.0) %} tool

> <hands-on-title>Calculate dissimilarity index</hands-on-title>
>
> 1. {% tool [Dissimilarity](toolshed.g2.bx.psu.edu/repos/ecology/cb_dissim/cb_dissim/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input champbloc_qecb.csv"*: `input` (champbloc_qecb.csv)
>    - {% icon param-file %} *"Input ficheterrain.csv"*: `input` (ficheterrain.csv)
>
>    - {% icon param-select %} *"Do you have data after the year 2021 ?"*: `No`
>    - {% icon param-text %} *"Until when do you have data (write only the YEAR) ?"*: `2021`
>
>
> 2. Press **Run Tool** (it can take some minutes to process it's normal if you have to wait a bit especially if your internet connection is weak)
>
> 3. View graphs
>  * Once it's green, click on type of plot you want to see
>  * Then, click on the {% icon galaxy-eye %} (eye) icon of your output in the history panel.
> 
>  The information is displayed in the central panel
>
> ![Dissim fs fi](../../images/champbloc/fs_fi.png "face sup and inf dissimilarity"){:width="620px"}
>
> ![Dissim bf bm](../../images/champbloc/bf_bm.png "bloc fixed and mobile dissimilarity"){:width="620px"}
>
> 3. Download your results
> 
>  * Click on the output you are interested in
>  * Click on {% icon galaxy-save %} (download)
>
{: .hands_on}

## Diversity indicators
Taking as input cleaned data from the Dissimilarity step tocompute diversity indicators.
Get the {% tool [Diversity indicators](toolshed.g2.bx.psu.edu/repos/ecology/cb_div/cb_div/0.0.0) %} tool

> <hands-on-title>Calculate diversity indicators</hands-on-title>
>
> 1. {% tool [Diversity indicators](toolshed.g2.bx.psu.edu/repos/ecology/cb_div/cb_div/0.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Input clean data"*: `input` (Clean Rdata)
>
> 2. Press **Execute** 
>
> 3. View graphs
>  * Once it's green, click on type of plot you want to see
>  * Then, click on the {% icon galaxy-eye %} (eye) icon of your output in the history panel.
> 
>  The information is displayed in the central panel
>
> ![Div ](../../images/champbloc/simpson.jpg "Simpson indicator"){:width="620px"}
>
>
> 3. Download your results
> 
>  * Click on the output you are interested in
>  * Click on {% icon galaxy-save %} (download)
>
{: .hands_on}

# Conclusion
Here you have finished your study on your boulder fields biodiversity state.

tools_json:
["IVR", "toolshed.g2.bx.psu.edu/repos/ecology/cb_ivr/cb_ivr/0.0.0", "Dissimilarity", "toolshed.g2.bx.psu.edu/repos/ecology/cb_dissim/cb_dissim/0.0.0", "Diversity indicators", "toolshed.g2.bx.psu.edu/repos/ecology/cb_div/cb_div/0.0.0"]


topic: ecology
tutorial: regionalGAM
title: Regional GAM
difficulty: intermediate
time_estimation: 2h30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: adebf1585bca2fd00e80942296b20793
last_modified: 2026-05-27T09:01:48.401586
zenodo_link: https://zenodo.org/record/1324204
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/regionalGAM/tutorial.html

content:
This tutorial will show you how to study species abundance through the computation of a flight curve, abundance index, and trends. It will explain you how to use different [regionalGAM](https://github.com/RetoSchmucki/regionalGAM) tools on Galaxy allowing you to deal with datasets containing abundance information for various species per site and per date through a couple of years.

After a certain number of steps, you will be able to extract single species data and study related abundance through the years. The goal of this exercise is to be able to create abundance trend over time and biodiversity indicators. Following these indicators allow to follow trends in terms of population dynamics. You could for example try to predict the occurrences of one specific species in a certain type of environment using the prediction model of climate evolution. Based on charts that you will generate, you could try to explain the evolution of a species with environmental data (temperatures variations, modifications of the environmental conditions).

You will basically learn how to create a file on the basis of which you can create a visual material that can be quite easily understood and therefore be efficient for a large audience.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Step 1: Pre-processing

The goal of the first step is to upload and prepare the file so that it will be usable for the *regional GAM* analysis.

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a proper name like `regionalGAM tutorial`
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the CSV dataset file with only one species from [Zenodo](https://zenodo.org/record/1324204#.W2BmRn7fNE4) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1324204/files/regional%20GAM%20data.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Check that the file contains a header corresponding to: ```"SPECIES","SITE","YEAR","MONTH","DAY","COUNT"```, and that all the non numeric content are between double quotes as "x" and that separators are ","
>
{: .hands_on}

This dataset gathers years of records of the presence of butterfly species per site and per day. Columns indicates species names ("SPECIES"), observation site ("SITE"), date of the observation ("YEAR","MONTH","DAY") and number of individuals ("COUNT"). It compiles counts for the Gatekeeper (Pironia tithonus) collected between 2003 and 2012 and extracted from five European BMSs (United Kingdom, Netherlands, France, Germany, and Catalonia-Spain). The Gatekeeper has one generation per year and feeds mainly on Poaceae. It is found in the Central and Southern Europe, North Africa Anatolia, and Caucasus.

![Gatekeeper (Pironia tithonus)](../../images/regionalGAM/Gatekeeper-Pyronia_tithonus-male.jpg "Gatekeeper (Pironia tithonus) Charles J Sharp")


## Prepare the data

The current dataset contains a lot of data (exact site names for 5 butterfly monitoring scheme). Processing the file in this condition would require time and for the purpose of this tutorial, we will reduce the number of sites. The column with header `SITE` of the dataset you are using is really long and the `SITES` are classified into sub-sites (like `ESBMS.12`, `ESBMS.28`, `ESBMS.55`, etc).

Here, we will only keep the sites that are in the Netherlands (NLBMS.XX). We want to create a down-sampled file, by selecting the lines where NLBMS is found.

> <hands-on-title>Downsample and hide some information</hands-on-title>
> 1. **Text reformatting with awk** {% icon tool %} with the following parameters:
>       - {% icon param-file %} *"Select cells from"*: Uploaded CSV dataset file {% icon tool %}
>       - *"AWK Program"*:
>
>
>              NR == 1 { print }
>              /NLBMS/ { print }
>
>
> The first line will skip and print the header and the second will print all the lines where NLBMS is found
> Warning: depending on the Galaxy instance your are using, this tool can not appear on the tool panel. Another solution can be to use **Select lines that match an expression** {% icon tool %} to select lines matching 'NLBMS' pattern in addition to **Select first lines from a dataset** {% icon tool %} specifying keeping the first line, so the header, then **Concatenate datasets tail-to-head** {% icon tool %} to recreate a CSV file from concatenating the header and NLBMS lines of the original csv file.
>
{: .hands_on}

> <question-title></question-title>
>
> How many sites do you have before and after removing the sites from outside the Netherlands?
>
>
> You may need to use a tool like **Count occurrences of each record** {% icon tool %}. If you want to run the same tool with same parameters to several input files, you can directly specify the {% icon param-files %} **Multiple datasets** option on the tool form for the *"from dataset"* parameter).
>
> {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>
> > <solution-title></solution-title>
> > The dataset contains 280 sites now against 1143 before down-sampling.
> {: .solution}
{: .question}

> <details-title>If your original data is in RData format</details-title>
>
> > <hands-on-title>Data upload.</hands-on-title>
> > 1. Import the RData
> >
> >    For example, you can upload:
> >
> >    ```
> >    https://zenodo.org/record/1324204/files/gatekeeper_CM%20.RData
> >    ```
> >
> > 2. **RData binary file reader** {% icon tool %} with the following parameters:
> >    -  {% icon param-file %} *"Rdata binary file to explore"*: imported RData
> >
> > 2. **RData parser** {% icon tool %} with the following parameters
> >    -  {% icon param-file %} *"Rdata file to explore"*: imported RData
> >    -  {% icon param-file %} *"File with .Rdata content details"*: output of **RData binary file reader** {% icon tool %}
> >    -  *"Select which attribute(s) you want to extract"*: select everything but "trend"
> >    -  *"Bind variables in a single tabular when its possible"*: `Yes`
> {: .hands_on}
>
> If the tool **RData parser** {% icon tool %} don't succeed to create a single tabular file, it generates separate files, each of them containing one column. The file with the `TREND` header can be let aside as we don't need it for what will follow.
>
> > <question-title></question-title>
> >
> > If Rdata parser fails to generate a single unified tabular file, can you propose a way to regenerate such a dataset ?
> >
> > > <solution-title></solution-title>
> > > You can do that using:
> > > 1. **Paste two files side by side tool** {% icon tool %} with the following parameters:
> > >   - {% icon param-file %} *"paste"*: output from **RData parser** {% icon tool %} headed with "SPECIES"
> > >   - {% icon param-file %}*"and"*: output from **RData parser** {% icon tool %} with headed with "SITE"
> > > 2. Repeat **Paste two files side by side** {% icon tool %} executions as many times as there are separated files in order to create a final dataset with all the columns:
> > >     1. Repeat **Paste two files side by side tool** {% icon tool %} to paste the file containing 2 columns with the one headed by `YEAR`
> > >     1. Repeat **Paste two files side by side tool** {% icon tool %} to paste the file containing 3 columns with the one headed by `MONTH`
> > >     1. Repeat **Paste two files side by side tool** {% icon tool %} to paste the file containing 4 columns with the one headed by `DAY`
> > >     1. Repeat **Paste two files side by side tool** {% icon tool %} to paste the file containing 5 columns with the one headed by `COUNT`
> > {: .solution}
> {: .question}
{: .details}

## Check that the dataset contains only one species

The second step of any Regional GAM data analysis is making sure to have a dataset of only one specific species that you will then be able to use. If you want to create a graph showing abundance evolution by years of several species, you will have to superimpose the graphs on one another.

As the dataset is quite big and may contain heterogeneous information, we need to know whether the data are about one species or more.

> <hands-on-title>How many species are taken into account in this dataset?</hands-on-title>
>
> 1. **Count occurrences of each record** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"from dataset"*: output of **Text reformatting with awk**
>    - *"Count occurrences of values in column(s)"*: `Column: 1`, the `SPECIES` column
>    - *"Delimited by"*: `Tab`
>    - *"How should the results be sorted?"*: `With the most common values first`
> 2. Inspect the generated file by clicking on the {% icon galaxy-eye %} (eye) icon (**View data**)
> 3. Check that the dataset is for one species only
{: .hands_on}

> <details-title>Datasets containing information about more than one species</details-title>
>
> If your dataset contains information about more than one species, you can apply the previous steps and then run an extra-step to select one specific species and show all the data corresponding to it.
>
> As the dataset is quite big and contains heterogeneous information, you want to know whether the data are about one species or more. So the first step consists to count how many species are taken into account in this dataset.
>
>
> To test these steps, you can use the following dataset:
>
> ```
> https://zenodo.org/record/1324204/files/Dataset%20multispecies%20Regional%20GAM.csv
> ```
>
> > <question-title></question-title>
> >
> > 1. How many species does your initial dataset take into account ?
> > 2. What are their names ?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. The dataset contains information on 2 different species
> > > 2. Their names are "Pyronia tithonus" and "Aglais io".
> > >
> > {: .solution}
> {: .question}
>
> We now need to create a new file containing the data for only one species
>
> > <hands-on-title>Creating a new file containing only the data from one species</hands-on-title>
> > 1. Copy the name of the species you are interested in (for example: "Aglais io").
> > 2. **Filter data on any column using simple expressions** {% icon tool %}
> >      - {% icon param-file %} *"Filter"*: output of **Text reformatting with awk** {% icon tool %}
> >      - *"With following condition"*: `c1=='"Aglais io"'` or (another species name)
> >      - *"Number of header lines to skip"*: `1`
> >
> >    You can repeat this set of actions as much as necessary, changing only the name of the species taken into account. By doing this, you will obtain separated dataset, each of them containing a different species.
> >
> > 4. Repeat the last step on all the different `outputs` from **Filter data on any column using simple expressions** {% icon tool %} that you have, one by species
> > 5. Tag your new dataset with an explicit tags as "Count" and "Aglais io" and/or rename this dataset like "Aglais io count file".
> {: .hands_on}
>
> If you want to create a graph showing abundance evolution by years of several species, you will have to superimpose the graphs on one another.
{: .details}

# Step 2: Analyze phenology of a species through the years

## Compute and visualize the phenology

[Phenology](https://en.wikipedia.org/wiki/Phenology), as described in Wikipedia, is the study of periodic plant and animal life cycle events and how these are influenced by seasonal and inter-annual variations in climate, as well as habitat factors (such as elevation).

Now you have a file containing all the data on the species of interest. The main goal of this step is to model one phenology per year using a general additive model (GAM) with time and sites as explanatory variables. This model will be used to predict values for missing counts at local sites in order to improve the quality of the abundance indices.

> <hands-on-title>Phenology</hands-on-title>
> 1. **Flight curve** {% icon tool %} with the following parameters
>    - {% icon param-file %} *"Count file"*: output file you just generated with the **Text reformatting with awk** or **Filter data on any column using simple expressions** if your file was containing more than one species {% icon tool %}
>
> 2. Generate the chart using the visualization
>    1. Inspect and expand the output data from **Flight curve** {% icon tool %}
>    2. Click on the {% icon galaxy-barchart %} (**Visualize**) icon
>    3. Select **Line chart (NVD3)** as visualization type
>    4. Give it a proper name, e.g. `Pyronia tithonus phenology raw simple vizu`
>    5. On **Select data** area, specify:
>       - *"Provide a label"*: `Pyronia tithonus phenology from 2003 to 2012` for example
>       - *"Pick a series color"*: Choose a color for the line
>       - *"Data point labels"*: `Column 1`
>       - *"Values for x-axis"*: `Column 2`
>       - *"Values for y-axis"*: `Column 6`
>    6. On **Change settings**, specify:
>       - *"X-Axis label"*: `Year`
>       - *"Y-Axis label"*: `nm values`
>    7. Click on **Save**
>
{: .hands_on}


![Phenology chart](../../images/regionalGAM/phenology_year.png)

This graph displays the modeled flight curve of *Pyronia tithonus* over the years. But the first year is "null", probably a left-over of the 1st line with the header, and the week number is not shown.

> <hands-on-title>Create a new column of the dataset containing the week and the year</hands-on-title>
> 1. **Count occurrences of each record** {% icon tool %} with the following parameters
>    - {% icon param-file %} *"from dataset"*: output from **Flight curve**
>    - *"Select"*: `Column: 2` (the column with the year)
>    - *"Delimited by"*: `Tab`
>    - *"How should the results be sorted?"*: `By the values being counted`
> 2. Inspect and expand the output data from **Count occurrences of each record** {% icon tool %}
>
>    > <question-title></question-title>
>    >
>    > What can you see in this file?
>    >
>    > > <solution-title></solution-title>
>    > > A list of the years and the number of occurrence for each year which should match the number of days for each year.
>    > {: .solution }
>    {: .question}
>
> 3. **Column Regex Find And Replace** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: output file from **flight curve**.
>    - *"in column"*: `Column 2` (the column with the year)
>    - Click on *"Insert Check"*
>    - *"Find pattern"*: `(20[0-9][0-9])`
>    - *"Replace with"*: `-\1`
> 4. Inspect the file and check if all the years are now written with a `-` before the digits.
> 5. **Merge Columns together** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: output from the last **Column Regex Find And Replace**.
>    - *"Merge column"*: `Column: 3` (the column with the week)
>    - *"with column"*: `Column: 2` (the column with the year)
> 6. **Column Regex Find And Replace** {% icon tool %} to recreate the original content for the year column:
>    - {% icon param-file %} *"File to process"*: output file from **Merge Columns together**.
>    - *"in column"*: `Column 2` (the column with the year)
>    - *"Find pattern"*: `-(20[0-9][0-9])`
>    - *"Replace with"*: `\1`
> 7. **Remove beginning of a file** {% icon tool %} to remove first line (to avoid header in the visualization)
>
>    > <comment-title>Dataset with information about more than one species</comment-title>
>    > If your dataset contains information about more than one species, you can apply the previous steps and then run an extra-step to select one specific species and show all the data corresponding to it.
>    {: .comment}
>
> 8. Generate the chart using the visualization with the x-axis corresponding to your column `"week""year"`.
>    1. Inspect and expand the output data from **Remove beginning of a file** {% icon tool %}
>    2. Click on the {% icon galaxy-barchart %} (**Visualize**) icon
>    3. Select **Line chart (NVD3)** as visualization type
>    4. Give it a proper name, e.g. `Pyronia tithonus phenology simple vizu`
>    5. On **Select data** area, specify:
>       - *"Provide a label"*: `Pyronia tithonus phenology from 2003 to 2012` for example
>       - *"Pick a series color"*: Choose a color for the line
>       - *"Data point labels"*: `Column 6` (the nm column) or another one
>       - *"Values for x-axis"*: `Column 7` (the "week-year" column)
>       - *"Values for y-axis"*: `Column 6` (the nm column)
>    6. On **Customize**, specify:
>       - *"X-Axis label"*: `Week-Year`
>       - *"Y-Axis label"*: `nm values`
>    7. Click on **Save**
>
{: .hands_on}

![Phenology chart](../../images/regionalGAM/Pyronia_tithonus_phenology_explicit_ID.png)

This shows the abundance of *Pyronia tithonus*, over the weeks. We can see there is a peak every year as expected and we can notice some differences in the values between the weeks and years, but not how the weeks overlap between the years. We will plot a stalked version of this visualization to check if there are no unexpected changes and have an idea of the quality of the reconstructed phenology.

> <hands-on-title>Visualize the years on a stacked graph</hands-on-title>
> 1. **Scatterplot w ggplot2** {% icon tool %} with the following parameters
>    - {% icon param-file %} *"from dataset"*: output from **Flight curve**
>    - *"Column to plot on x-axis"*: `4` (the column with the year)
>    - *"Column to plot on y-axis"*: `6`
>    - *"Plot title"*: add a meaningful title (e.g. `Pyronia tithonus phenology stacked visualization`)
>    - *"Label for x axis"*: add a meaningful label (e.g. `Day number`)
>    - *"Label for y axis"*: add a meaningful label  (e.g. `nm value`)
>    - *"Type of plot"*: `Line only`
>    - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot` to superimpose years
>    - *"column differentiating the different groups"*: `2` to affect one coulor by year
>    - *"Color schemes to differentiate your groups"*: `set 3` or any set of color with more than 10 colors as there are 10 years in our dataset.
>
{: .hands_on}

![Stacked Phenology chart](../../images/regionalGAM/Pyronia_tithonus_phenology_stacked_explicit_ID.png)

With this graph, we can see that *Pyronia tithonus* records are always around the same weeks every year. We can also see small differences between years but overall the phenology is quite stable and the results of this step seem good and allow us to continue with the calculation of the abundance index.

> <details-title>Working with more than one species</details-title>
>
> If you are working with more than one species, you should follow the next steps
>
> > <hands-on-title></hands-on-title>
> > 1. **Paste two files side by side** {% icon tool %} with the following parameters:
> >     -  *"Paste"*: `the output` from **Merge Columns together** (with the dataset for species 1)
> >     -  *"and"*: `the output` from **Merge Columns together** (with the dataset for species 2)
> >     -  *"Delimited by"*: tabulation
> >
> >   >
> >   > <comment-title>You can add other species.</comment-title>
> >   > In order to do so you will have to do as explained below:
> >   > 1. **Paste two files side by side** {% icon tool %} with the following parameters:
> >   >    -  *"Paste"*: the `output` from **Paste two files side by side** (with the dataset containing species 1 and 2)
> >   >    -  *"and"*: `the output` from **Merge Columns together** (with the dataset containing species 3)
> >   >    -  *"Delimited by"*: tabulation
> >   > 2. Repeat **Paste two files side by side** {% icon tool %} with the output from **Paste two files side by side** (with the data containing species 1, 2 and 3) and with the output from **Merge Columns together** (with the dataset containing species 4) and so on.
> >   {: .comment}
> {: .hands_on}
>
> If your input dataset contains information about more than one species, you can now generate a chart for the multi-species dataset:
>
> > <hands-on-title></hands-on-title>
> > 1. Inspect and expand the output data from **flight curve** {% icon tool %}
> > 2. Click on the {% icon galaxy-barchart %} (**Visualize**) icon
> > 3. Select a visualization: `line chart (NVD3)`
> > 4. Give it a proper name like `Aglais io & Pyronia tithonus phenology`
> > 5. Select data
> >     -  *"Provide a label"*: The name of the first species, for example `Aglais io`
> >     -  *"Pick a series color"*: Choose a color
> >     -  *"Data point labels"*: `Column corresponding to the name of the species 1`
> >     -  *"Values for x-axis"*: `Column corresponding to the "week and year" of the species 1`
> >     -  *"Values for y-axis"*: `Column corresponding to nm of the species 1`
> > 6. Insert data series:
> >     -  *"Provide a label": he name of the second species, for example `Pyronia tithonus`
> >     -  *"Pick a series color"*: Choose a different color
> >     -  *"Data point labels"*: `Column corresponding to the name of the species 2`
> >     -  *"Values for x-axis"*: `Column corresponding to the "week and year" of the species 2`
> >     -  *"Values for y-axis"*: `Column corresponding to nm of the species 2`
> > 7. You may repeat "Insert data series" as many times as needed depending on the number of different species you want to represent on your chart.
> > 8. Click on **Customize**
> >     -  *"X-Axis label"*: `Week and Year`
> >     -  *"Y-Axis label"*: `nm values`
> >     -  *"Use multi-panels"*: click on `No`(or you will have separated charts, one for each species)
> > 9. Click on **Save** if you are willing to keep it
> >
> {: .hands_on}
>
{: .details}

## Compute Abundance Index across sites and years

We now would like to create a file showing the abundance index per year of a chosen species in a certain site.

> <hands-on-title>Generate an abundance index</hands-on-title>
> 1. **Abundance index** {% icon tool %} with the following parameters:
>     -  *"Count file"*: output from **Text reformatting with awk**, that should be named `Counting file` and/or tagged `Count`
>     -  *"Flight curve output"*: output from **flight curve**
{: .hands_on}

**Abundance index** {% icon tool %} generates a table with 5 columns:
- Site
- Species
- Year
- Regional GAM
- prop_pheno_sampled


## Compute a collated index for each year and estimates the temporal trend

The expected temporal trend allows you to have an overview of the evolution of a species in a certain type of environment in the future.

> <hands-on-title>Expected temporal trend</hands-on-title>
> 1. **Expected temporal trend** {% icon tool %} with the following parameters:
>    -  *"Tabular file generated by the ab_index tool"*: output of **abundance index**
{: .hands_on}

**Expected temporal trend** {% icon tool %} generates 2 files"
1. The graph like:

    ![Expected temporal trend](../../images/regionalGAM/Expected_temporal_trend.png)

2. A table with values of "x", the collated index value by year

Sometimes the expected temporal trend can't be done on dataset. If you want this action to work, the occurrences on your dataset must lie between the month of April and the end of the month of September.

## Model temporal trend

We would like to know if the year has an influence on the abundance of a species. We will use a linear regression to do that.


> <details-title>More details about the statistics</details-title>
> The model fitted to the data is:
> $$ Y_i = \alpha + \beta x_i + \epsilon_i $$
> with $$Y_i$$ = value of the dependent variable for the unit $$i$$
> $$\alpha$$ = intercept
> $$\beta$$ = slope
> $$x_i$$ = value of the explanatory variable for the unit $$i$$
> $$e_i$$ = residual for the unit $$i$$
>
{: .details}

> <hands-on-title>Model temporal trend with linear regression</hands-on-title>
> 1. **Model temporal trend with a simple linear regression** {% icon tool %}
>    - {% icon param-file %} *"File generated by the glmmpql/Expected temporal trend tool"*: tabular output of **temporal trend**
>    - {% icon param-file %} *"File generated by the ab_index tool"*: output from **abundance index**
{: .hands_on}

Have a look at the text file result. This is the output of the linear model.

![Output from the "model temporal trend with simple linear regression" tool ](../../images/regionalGAM/output-simple-linear-regression.png)

Details about the output from the tool

1. The formula of the model following this format: "Model : dependent variable ~ explanatory variable"
2. The estimated values of the coefficients
3. The standard error of the coefficients
4. The value of the "hypothesis test statistic"
5. The probability value.

> <question-title></question-title>
>
> 1. What are the estimates of the regression coefficients?
> 2. Can we use this model to make good predictions?
> 3. Is the test p-value significant?
>
> > <solution-title></solution-title>
> >
> > 1. Intercept is 183.33852 and the slope (Year) is -0.08952.
> > 2. No, the residual standard error is high.
> > 3. Yes (0.0282 < 0.05).
> >
> {: .solution}
{: .question}

We can also test for autocorrelation in the data.

> <hands-on-title>Check if there is autocorrelation of residuals</hands-on-title>
> 1. **Autocorrelation test check for temporal autocorrelation in the residuals** {% icon tool %} with the following parameters.
>    - {% icon param-file %} *"gls model"*: tabular output of **Model temporal trend**
{: .hands_on}

![Autocorrelation test graph](../../images/regionalGAM/Autocorrelation_test.png)


Have a look at this plot. We can see that one of the value is above the threshold (blue dashed line), therefore we observe autocorrelation in the data. We will now take into account the autocorrelation in the next model.

## Take into account the autocorrelation

> <hands-on-title>Model temporal trend taking into account autocorrelation of residuals</hands-on-title>
> 1. **Linear regression ajusted for autocorrelation in the residuals** {% icon tool %} with the following parameters.
>    - {% icon param-file %} *"File generated by the glmmpql/Expected temporal trend tool"*: tabular output of **temporal trend**
>    - {% icon param-file %} *"File generated by the ab_index tool"*: output from **abundance index**
{: .hands_on}

For this step you can use the scratchbook function in order to see the results of both models at the same time.

> <details-title>How to use the scratchbook</details-title>
> You can follow this tutorial if you want to learn how to use the scratchbook.
> https://usegalaxy.org/tours/core.scratchbook
>
{: .details}

![Plot of temporal trend with trend line](../../images/regionalGAM/output-both-models.png)

Here you can compare the models using the AIC for Akaike information criterion. We can see that the AIC is lower in the model taking the autocorrelation into account. A lower AIC value is associated with less information loss.
We will therefore, based on the AIC, select the model taking into account the autocorrelation.
We can now represent the data with the trend line from the model.

> <hands-on-title>Plot the data and the trend line from the model</hands-on-title>
> 1. **Plot abundance with trend line** {% icon tool %} with the following parameters.
>    - {% icon param-file %} *"File generated by the ab_index tool"*: output from **abundance index**
>    - {% icon param-files %} *"gls model"*: outputs of **Model temporal trend with a simple linear regression**.
{: .hands_on}

![Plot of temporal trend with trend line](../../images/regionalGAM/trend_simple.png)

You can see the trends is an apparent decrease. Here results are statistically significant so we can say that there is a significant decrease of the abundance.

# Conclusions


In this tutorial, you have analyzed regional GAM data to extract useful information in order to be able to show different tendencies of a chosen species. Therefore, you are now able to treat the dataset so that it shows only the data for one specific species of your choice. From there, you can show the phenology of this species through the years first on a dataset and then on a visual chart. You have also learned how to represent on a single chart the phenology of various species. Afterwards, we have shown you how to create a dataset containing the information on the abundance of a species per year and per site. Based on which you can henceforth visually represent the annual abundance trend on a chart. Thereafter, you have the possibility of showing the expected temporal trend, based on which you will be able to assess the abundance of species. The last part of this tutorial has shown you how to calculate the linear regression allowing you to determinate whether the year has an influence on the abundance of a species or not.


topic: ecology
tutorial: species-distribution-modeling
title: Species distribution modeling
difficulty: intermediate
time_estimation: 1h
tools_json: ["Wallace", "interactive_tool_wallace"]
requirements_json: []
tags_json: []
content_hash: 8298c61287a4dbe02fc284c3f648165f
last_modified: 2026-05-27T09:01:48.401870
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/species-distribution-modeling/tutorial.html

content:
Species Distribution Modeling (SDM) can help understand a species distribution in relation to its environment. It can also attempt to quantify the impact of climate change on the species' habitat, direct conservation efforts and predict invasive species distributions. This is acheived by associating species occurrence (observations) data with a set of environmental data (such as temperature and precipitation).

The goal of this tutorial is to model a theoretical ecological niche and predict species distribution in a future climate scenario by using an SDM with the Wallace interactive environment on Galaxy. We'll use the data occurrences of *Chrysemys picta* ([Painted turtle](https://en.wikipedia.org/wiki/Painted_turtle)) in the US, North America region.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Step 1: Import data from public databank

In this study the datasets are all imported from the [GBIF](https://www.gbif.org/) databanks. It is also possible to import from [BISON](https://bison.usgs.gov/), [iNaturalist](https://www.inaturalist.org/) and others.


> <hands-on-title>Import the data</hands-on-title>
> 1. Create a new history for this tutorial and give it a proper name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. **Get species occurrences data** {% icon tool %} with the following parameters
>    - *"Scientific name"*: `Chrysemys picta`
>    - *"Data source"*: `gbif`
>    - *"Number of records to return"*: `10000`
>
> 3. Inspect the generated file with the Scratchbook
{: .hands_on}

We now have a tabular file with 10,001 lines and many columns.

> <question-title></question-title>
> 1. What do the rows in the files represent?
> 2. What are the columns 1, 2, 3 and 54 of the generated file?
>
> > <solution-title></solution-title>
> > 1. Each row represents a *Chrysemys picta* observation, its location, and other metadata
> > 2. Column 1 is the name, column 2 the longitude of the observation of *Chrysemys picta*, column 3 the latitude and column 54 the country code of the location
> {: .solution }
{: .question }

We will extract the 4 columns mentioned above and keep only the occurrence records from the US.

> <hands-on-title>Import the data</hands-on-title>
> 1. **Cut columns** {% icon tool %} with the following parameters
>    - *"Cut columns"*: `c1,c2,c3,c54`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: output of **Get species occurrences data**
>
> 2. Inspect the generated file and check the 4 columns
> 3. **Filter data** {% icon tool %} with the following parameters
>    - {% icon param-file %} *"Filter"*: output of **Cut columns**
>    - *"With following condition"*: `c4=='US'`
>    - *"Number of header lines to skip"*: `1`
> 3. Inspect the generated files
>
>    > <question-title></question-title>
>    >
>    > How many occurrences were kept?
>    >
>    > > <solution-title></solution-title>
>    > > 63.06% of the occurrences (6,306) are conserved
>    > {: .solution }
>    {: .question}
>
> 4. **Tabular to CSV** {% icon tool %}
>    - Click on the **Edit Attributes** {% icon galaxy-pencil %} option for the output of **Filter**.
>    - On the central panel click on the {% icon galaxy-gear %} **Convert** tab.
>    - Check the  **Target data** is csv.
>    - Click on **Create Dataset**
>
> 5. Due to an issue related to the GBIF format conversion, each `"` sign is triplicated on the resulting file. You need to replace `"""` for `"`, using for example the **Replace** {% icon tool %} parts of text tool.
{: .hands_on}

# Step 2: Development and evaluation of the SDM using Wallace

[Wallace](https://wallaceecomod.github.io/) is an R Shiny app integrated into Galaxy. It provides an interactive environment for the rapid and effective development and evaluation of an SDM, it integrates data download, cleaning, partitioning, modeling, visualisation and prediction.

## Transmit occurrence data from Galaxy to Wallace

Here, you can either upload your Galaxy data or you can download data directly in Wallace. Let's use the data from your Galaxy history:

> <hands-on-title>Launch Wallace</hands-on-title>
> 1. Verify your current history has the filtered occurences data.
> 2. Open {% tool [Wallace](interactive_tool_wallace) %} in Galaxy or click here to [open the tool](https://ecology.usegalaxy.eu/root?tool_id=interactive_tool_wallace) and click on execute
> 3. Go to User > Active InteractiveTools, then click on the active InteractiveTool named `Wallace visualisation`.
> 4. In Wallace
>    0. Due to an issue related to the last Wallace Docker update, a manual fix is required. First go to the **5 Partition Occs**, select *"Spatial Partition"* then for *"Options available"* select `Checkerboard 2 (k=4)` then click on the **Partition** button. This will allow the *"Aggregation Factor"* option to be displayed. Sorry for the inconvenience.
>    1. Go to **1 Occ Data**
>    2. In **Obtain Occurrence Data**
>       - *"Modules Available"*: `Galaxy History User`
>       - *"From Galaxy History User csv file select"*: output of **Tabular to CSV** (or the **Replace** output if you used that tool).
>       - Click on **Load Occurrences**
{: .hands_on}

The data will load on the main panel of Wallace. It can now be inspected using the map, the table, etc.

You now have your occurrence records on Wallace!

<!--
> <question-title></question-title>
>
> *TODO: Add a questio<F4>n related to the data there*
>
> What happens if you click on a data point on the map?
>
> > <solution-title></solution-title>
> >
> > The metadata associated to the point is displayed. In our case, because we filtered the information we will get a lot of NAs.
> {: .solution }
{: .question}
-->

## Process occurrence data

For the modeling, we need to choose the occurrences we want to use. There is 4 ways to do this in Wallace in the **2 Process Occ** panel:

1. Selecting occurrences on the map by delimiting an interesting geographic area
2. Removing occurrences by ID
3. Delimiting a spatial thin

   This allows selecting occurrences by setting a minimum distance (in km) between the different occurrences. For example, by typing 30km, we will get all the occurrences on the map which are at a minimum of 30km from each other.

4. Considering all the occurrences

We want to work on the data from the US, we will select the occurrences with the 1st option

> <hands-on-title>Select interesting occurrences</hands-on-title>
> 1. Go to **2 Process Occ**
> 2. In **Process Occurrence Data**
>    - *"Modules Available"*: ` Select Occurrences On Map`
> 3. In the middle panel,
>    1. Click on the polygon icon on the map
>    2. Draw a polygon to delimit the area
> 4. On the left panel, click on **Select Occurrences**
{: .hands_on}


## Obtain Environmental Data

Now, we would like to obtain biologically relevant environmental data for our model. The [Bioclimatic variables](http://www.worldclim.org/bioclim) consist of such original and derived variables. We will associate them with the occurrence data, using the WorldClim Bioclims module.

This module provides a raster with environmental variables from online sources. The raster is composed of environmental information. Each layer of the raster contains a climatic variable: starting from BIO1 = Annual mean temperature, to BIO19 = Precipitation of Coldest Quarter.

> <hands-on-title>Obtain Environmental Data</hands-on-title>
> 1. Go to **3 Env Data**
> 2. In **Obtain Environmental Data**
>    - *"Modules Available"*: `WorldClim Bioclims`
>
>      > <comment-title>Using your own raster</comment-title>
>      > If you prefer to load your own raster:
>      > 1. Import it into Galaxy in the GeoTIFF format
>      > 2. In **Obtain Environmental Data**
>      >    - *"Modules Available"*: `Galaxy History User`
>      >    - *"Select from your Galaxy History User csv file"*: imported GeoTIFF file
>      >    Note that if you input your own raster, you don't have the option for time prediction later on.
>      {: .comment}
>
>    - *"Select WorldClim bioclimatic variable resolution"*: `10 arcmin`
>
>      This is the lowest resolution.
>
> 3. Click on the **Load Env Data**
{: .hands_on}

<!-- **TODO: comment the output / add a question to explain why the next step** -->

## Process environmental Data

Wallace will now associate environmental data and occurrences data to train a model.

1. It first creates a buffer zone around the occurrences

    We can choose the size of the buffer zone to control the area we will work with and on which a map of suitability will be made.

    Several background extents can be used:
    - Bounding box defines an area centered on the occurrences
    - Minimum convex polygon creates a convex area around the occurrences placement
    - Point buffers will use occurrence localities to build a buffer zone around each occurrence

2. It samples Background Points

> <hands-on-title>Obtain Environmental Data</hands-on-title>
> 1. Go to **4 Process Envs**
> 2. In **Process Environmental Data**
>    - *"Modules Available"*: `Select Study Region`
> 3. In **Step 1: Choose Background Extent**
>    - *"Background Extents"*: `Minimum convex polygon`
>    - *"Study region buffer distance (degree)"*: `1`
> 4. Click on **Select**
> 5. In **Step 2: Sample Background Points**
>    - *"No. of background points"*: `100000`
> 6. Click on **Sample**
{: .hands_on}

<!-- **TODO: comment the output / add a question to explain why the next step** -->

## Partition Occurrence Data

By partitioning data, a dataset is divided into subsets (i.e. bins). A model is built on each of the subsets but one, it is used to test the model (assuming that all the groups are independent).

There is 2 possible ways to partition data:

1. **Non-spatial Partition**: a partition used when there is no bias due to space, time or sampling method
    1. Jacknife (k=n): each occurrence in the dataset is considered as equal to a bin. This is usually used when you have a small dataset with no known bias
    2. Random k-fold: partition the data randomly in a number of bins set by the user with the option `Number of Folds`
2. **Spatial Partition**: a partition used when there could be bias due to time, space or sampling method
    1. Block (k=4): the area is divided in four and the different occurrences are put equally into four bins
    2. Checkerboard 1 (k=2): two bins are used according to the position of the occurrences on the grid
    3. Checkerboard 2 (k=4): four bins are used according to the position of the occurrences on the grid. This requires an aggregation factor, which is the size of a second grid placed on the first one. For example, with a factor 4, the grid's size will be 4x4

        ![Checkerboard 2](../../images/species-distribution-modeling/Checkerboard.png "Example of checkerboard (k=4) with a factor 4. Source: doi:10.0.4.87/2041-210X.12261")

For both of these techniques the number of occurrences in each bin may vary.

> <hands-on-title>Partition Occurrence Data</hands-on-title>
> 1. Go to **5 Partition Occs**
> 2. In **Partition Occurrence Data**
>    - *"Modules Available"*: `Spatial Partition`
>    - *"Options Available:"*: `Checkerboard 2 (k=4)`
>    - *"Aggregation Factor"*: `6`
> 4. Click on **Partition**
{: .hands_on}

<!-- **TODO: comment the output / add a question to explain why the next step** -->

## Build and Evaluate Niche Model

Wallace can build different models using either:

1. The presence-only approach BIOCLIM (Module BIOCLIM) ({% cite booth2018 %})
2. The presence-background (presence-pseudo absence) algorithm Maxent (Module Maxent)

To evaluate these models, Wallace computes the performance on a hold-out dataset (data not used for training) and provides evaluation metrics such as the AUC (Area Under the Curve) mean. As a rule of thumb, an AUC of 0.75 and above is considered good, and closer to 1 is better.

> <hands-on-title>Build and evaluate the niche model</hands-on-title>
> 1. Go to **6 Model**
> 2. In **Build and Evaluate Niche Model**
>    - *"Modules Available"*: `BIOCLIM`
> 4. Click on **Run**
{: .hands_on}

<!-- **TODO: comment the output / add a question to explain why the next step** -->

## Visualize Model Results

We would now like to visualize the model of the theoretical niche. First we will make a chart to simulate an ecological niche:
- X-axis: temperature
- Y-axis: the annual precipitation

> <hands-on-title>Visualize Model Results</hands-on-title>
> 1. Go to **7 Visualize**
> 2. In **Visualize Model Results**
>    - *"Modules Available"*: `BIOCLIM Envelope Plots`
>    - *"Axis 1"*: `1`
>    - *"Axis 1"*: `12`
>    - *"Set threshold"*: `0.75`
{: .hands_on}

![temperature vs annual precipitation](../../images/species-distribution-modeling/bioclim_enveloppe_plots.png)

The optimum environmental parameters for this species (represented in green) are between 5°C and 15°C (on the graph, values are x10) with an annual precipitation between approximately 700mm and 1250mm.

To visualize the suitability on the map, we can use the map prediction module.

> <hands-on-title>Visualize Model Results</hands-on-title>
> 2. In **Visualize Model Results**
>    - *"Modules Available"*: `Map Prediction`
>    - *"Set threshold"*: `No threshold`
>
>      It will display a gradient of predicted presence or use. If you prefer a map with the predicted presence and predicted absence, select `minimum training presence`.
>
> 2. Click on **Plot**
{: .hands_on}

![map prediction](../../images/species-distribution-modeling/map_prediction.png)

<!-- **TODO: comment the output / add a question to explain why the next step** -->

## Project Model

Wallace can use the trained model to predict possible species distributions in a different areas, outside of the sampled one. Here, we will try it for Canada.

> <hands-on-title>Visualize Model Results</hands-on-title>
> 1. Go to **8 Project**
> 2. In **Project Model**
>    - *"Modules Available"*: `Project to New Extent`
> 3. In the middle panel,
>    1. Click on the polygon icon on the map
>    2. Draw a polygon around a part of Canada
> 2. In **Project Model**
>    - *"Set threshold"*: `Minimum Training Presence`
> 4. Click on **Project**
{: .hands_on}

We can predict not only for a different area, but also for a different time and climate. We use the [Global Circulation Models (GCM)](https://en.wikipedia.org/wiki/General_circulation_model#Atmospheric_and_oceanic_models) to predict atmospheric fluctuation and then evaluate climate change. Each model is different and uses several parameters such as ocean atmosphere. The prediction also uses RCP scenarios, Representative Concentration Pathways take average enegy requirements across the planet (2.6, 4.5, 6.0 and 8.5 watts per meter squared) to model the concentration of greenhouse gases emitted in the near future and their effect on climate. These scenarios may produce varying predicted presence models.

We would like to predict the presence of *Chrysemys Picta* in Canada in 2050, given the model [CCSM4](http://www.cesm.ucar.edu/models/ccsm4.0/), a US model based on earth circulation, and an RCP value of 2.6 <!--(**TODO: what does that represent?**)-->

> <hands-on-title>Visualize Model Results with a 2.6 RCP</hands-on-title>
> 2. In **Project Model**
>    - *"Modules Available"*: `Project to New Time`
>    - *"Select time period"*: `2050`
>    - *"Select global circulation model"*: `CCSM4`
>    - *"Select RCP"*: `2.6`
>    - *"Select threshold"*: `Minimum Training Presence`
> 4. Click on **Project**
{: .hands_on}

![CCM4 with a 2.6 RCP](../../images/species-distribution-modeling/GCM_CCSM4_RCP_2.6.png)

Let's compare the prediction for the RCP value of 8.5. RCP 8.5 describes a pathway for the current carbon emission without additional constraints, it projects a 4.3˚C temperature increase by 2100.

> <hands-on-title>Visualize Model Results with a 8.5 RCP</hands-on-title>
> 2. In **Project Model**
>    - *"Modules Available"*: `Project to New Time`
>    - *"Select time period"*: `2050`
>    - *"Select global circulation model"*: `CCSM4`
>    - *"Select RCP"*: `8.5`
>    - *"Select threshold"*: `Minimum Training Presence`
> 4. Click on **Project**
{: .hands_on}

![CCM4 with a 8.5 RCP](../../images/species-distribution-modeling/GCM_CCSM4_RPC_8.5.png)

<!-- **TODO: comment the output / add a question to compare the 2 predictions**


**TODO: add a section on how to save / download** -->

# Conclusion

Following this tutorial, we have loaded a dataset of occurrences to the shiny app Wallace and modeled the distribution of *Chrysemys picta* (Painted turtle) with the Species Distribution Modeling (SDM) method. This allowed us to visualize it’s ecological niche and how climate change can influence it’s future distribution in North America. The saved project can be used for similar future studies.

> <details-title>Some useful references</details-title>
>
> - Predicting species distributions for conservation decisions ({% cite guisan2013 %}).
> - BIOCLIM: the first species distribution modelling package, its early applications and relevance to most current MAXENT studies. Diversity and Distributions ({% cite booth2018 %}).
> - ENMeval: An R package for conducting spatially independent evaluations and estimating optimal model complexity for MAXENT ecological niche models. ({% cite muscarella2014 %}).
>
> [GCM Guidelines](http://www.ipcc-data.org/guidelines/pages/gcm_guide.html) for information on Global Circulation Model (GCM), how it's done, what is taken in consideration, and more.
{: .details}


topic: ecology
tutorial: gbif_cleaning
title: Cleaning GBIF data for the use in Ecology
difficulty: intermediate
time_estimation: 0H30M
tools_json: []
requirements_json: []
tags_json: ["gbif", "data management", "data cleaning", "biodiversity"]
content_hash: 9f37dd405cb3c34222f080868dcb3532
last_modified: 2026-05-27T09:01:48.400031
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/gbif_cleaning/tutorial.html

content:
GBIF (Global Biodiversity Information Facility, www.gbif.org) is for sure THE most remarkable biodiversity data aggregator worldwide giving access to more than 1 billion records across all taxonomic groups. The data provided via these sources are highly valuable for research. However, some issues exist concerning data heterogeneity, as they are obtained from various collection methods and sources.

In this tutorial we will propose a way to clean occurrence records retrieved from GBIF.

This tutorial is based on the Ropensci {% cite zizka2018 %} tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. Download data from GBIF
> 2. Clean data
> 3. Convert text data into GIS format
> 4. Visualize spatialized data
> {:toc}
>
{: .agenda}

# Retrive data from GBIF

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from GBIF: **Get species occurrences data** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Scientific name of the species"*: write the scientific name of something you are interested on, for example `Loligo vulgaris`
>    - *"Data source to get data from"*: `Global Biodiversity Information Facility : GBIF`
>    - *"Number of records to return"*: `999999` is a minimum value
>
>    > <comment-title></comment-title>
>    >
>    > The spocc Galaxy tool allows you to search species occurrences across a single or many data sources (GBIF, eBird, iNaturalist, EcoEngine, VertNet, BISON). Changing the number of records to return allows you to have all or limited numbers of occurrences. Specifying more than one data source will change the manner the output dataset is formatted.
>    >
>    {: .comment}
>
>
> 3. **Check the datatype** {% icon galaxy-pencil %}, it should be `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 4. **Add tags** {% icon galaxy-tags %} to the dataset
>    - make them propagating tags (tags starting with `#`)
>    - make a tag corresponding to the species (`#LoligoVulgaris` for example here)
>    - and another tag mentioning the data source (#GBIF for example here).
>
>    Tagging dataset like this is good practice in Galaxy, and will help you 1/ finding content of particular interest (using the filtering option on the history search form for example) and 2/ visualizing rapidly (notably thanks to the propagated tags) which dataset is associated to which content.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


## Where do the records come from?

Here we propose to investigate the content of the dataset looking notably at the "basisOfRecord" attribute to know more about heterogeneity related to the data collection origin.

> <hands-on-title>"basisOfRecord" filtering</hands-on-title>
>
> 1. **Count** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"from dataset"*: `output` (output of **Get species occurrences data** {% icon tool %})
>    - *"Count occurrences of values in column(s)"*: `c[17]`
>
>
>    > <comment-title></comment-title>
>    >
>    > This tool is one of the important "classical" Galaxy tool who allows you to better synthesize information content of your data. Here we apply this tool to the 17th column (corresponding to the basisOfRecord attribute) but don't hesitate to investigate others attributes!
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many different types of data collection origin are there?
> 2. What is your assumption regarding this heterogeneity?
>
> > <solution-title></solution-title>
> >
> > 1. 5
> > 2. each basisOfRecord type is related to different collection method so different data quality
> >
> {: .solution}
>
{: .question}



## Filtering data based on the data origin

> <hands-on-title>Filter data on basisOfRecord GBIF attribute</hands-on-title>
>
> 1. **Filter** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `output` (output of **Get species occurrences data** {% icon tool %})
>    - *"With following condition"*: `c17=='HUMAN_OBSERVATION' or c17=='OBSERVATION' or c17=='PRESERVED_SPECIMEN'`
>    - *"Number of header lines to skip"*: `1`
>
>    > <comment-title></comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
>
>    > <question-title></question-title>
>    >
>    > 1. How many records are kept and what is the percentage of filtered data?
>    > 2. Why are we keeping only these 3 types of data collection origin?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 470 and 8.79% of records were drop out
>    > > 2. These data collection methods are the most relevant
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 3. Add to the output dataset a propagating tag corresponding to the filtering criteria adding `#basisOfRecord` string for example
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


## Have a look at the number of counts per record

Here we propose to have a look at the number of counts by record to know if there is some possible records with errors.

> <hands-on-title>Summary statistics of count</hands-on-title>
>
> 1. **Summary Statistics** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Summary statistics on"*: `out_file1` (output of **Filter** {% icon tool %})
>    - *"Column or expression"*: `c72`
>
> 2. Add to the output dataset a propagating tag corresponding to the filtering criteria adding `#individualCount` string for example
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the min and max of individual counts?
>
> > <solution-title></solution-title>
> >
> > 1. From 1 to 100
> >
> {: .solution}
>
{: .question}



## **Filtering** data on individual counts

> <hands-on-title>Filter data on individualCount GBIF attribute</hands-on-title>
>
> 1. **Filter** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `out_file1` (output of **Filter** {% icon tool %})
>    - *"With following condition"*: `c72>0 and c72<99`
>    - *"Number of header lines to skip"*: `1`
>
>
> > <question-title></question-title>
> >
> > 1. How many records are kept and what is the percentage of filtered data?
> > 2. How can you explain this result?
> > 3. Which propagated tag you can propose to add here?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 50 and 89.29% o records were drop out
> > > 2. An important percentage of data were drop out because of many records whithout any value for this individual count field
> > > 3. As for the previous "count" step you are dealing with the `individualCount` column, you can add a to the output dataset a  `#individualCount` tag for example
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}


## Have a look at the age of records

> <hands-on-title>Here we propose to have a look at the age of records, through the `year` GBIF attribute to know if there is some ancient data to maybe not consider.</hands-on-title>
>
> 1. **Summary Statistics** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Summary statistics on"*: `out_file1` (output of **Filter** {% icon tool %})
>    - *"Column or expression"*: `c41`
>
> 2. Add to the output dataset a propagating tag corresponding to the filtering criteria adding `#ageOfRecord` string for example
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the year of the older and younger records?
> 2. Why do you think of interest to treat differently ancient and recent records?
>
> > <solution-title></solution-title>
> >
> > 1. From 1903 to 2018
> > 2. We can assume ancient records are not made in the same way than recent one so keeping ancient records can enhance heterogeneity of our dataset.
> >
> {: .solution}
>
{: .question}


## Filtering data based on the age of records

> <hands-on-title>Filter data on ageOfRecord GBIF attribute</hands-on-title>
>
> 1. **Filter** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `out_file1` (output of **Get species occurrences data** {% icon tool %})
>    - *"With following condition"*: `c41>1945`
>    - *"Number of header lines to skip"*: `1`
>
>    > <comment-title></comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
>
>    > <question-title></question-title>
>    >
>    > 1. How many records are kept and what is the percentage of filtered data?
>    > 2. Why are we keeping only data from 1945?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 44 and 11.76% of records were drop out
>    > > 2. This arbitrary date allow to have only quite recent records, but you can specify another year.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 3. Add to the output dataset a propagating tag corresponding to the filtering criteria adding `#ageOfRecord` string for example
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

## Taxonomic investigation

> <hands-on-title>Investigate the taxonomic coverage, at the family level</hands-on-title>
>
> 1. **Count** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"from dataset"*: `out_file1` (output of **Filter** {% icon tool %})
>    - *"Count occurrences of values in column(s)"*: `c[31]`
>
>
>    > <comment-title></comment-title>
>    >
>    > This column allows us to look at the different families associated to records. Normally, looking at a unique species, we will obtain only one family
>    {: .comment}
>
{: .hands_on}

## Filtering

> <hands-on-title>Filter data on family attribute</hands-on-title>
>
> 1. **Filter** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `out_file1` (output of **Filter** {% icon tool %})
>    - *"With following condition"*: `c31=='Loliginidae'`
>    - *"Number of header lines to skip"*: `1`
>
>    > <comment-title></comment-title>
>    >
>    > We here select only records with the family of interest, Loliginidae
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Is the filtering here of interest ?
> 2. Why keeping this step can be of interest?
>
> > <solution-title></solution-title>
> >
> > 1. No, because 100% of records are kept
> > 2. Because this is an important step we have to take into account in such a GBIF data treatment, and if your goal is to create your own workflow you plan to use on others species, this can be of interest to keep this step
> >
> {: .solution}
>
{: .question}

## Sub-step with **OGR2ogr**

> <hands-on-title>Convert occurrence dataset to GIS one for visualization</hands-on-title>
>
> 1. **OGR2ogr** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gdal supported input file"*: `out_file1` (output of **Filter** {% icon tool %})
>    - *"Conversion format"*: `GEOJSON`
>    - *"Specify advanced parameters"*: `Yes, see full parameter list.`
>        - In *"Add an input dataset open option"*:
>            - {% icon param-repeat %} *"Insert Add an input dataset open option"*
>                - *"Input dataset open option"*: `X_POSSIBLE_NAMES=longitude`
>            - {% icon param-repeat %} *"Insert Add an input dataset open option"*
>                - *"Input dataset open option"*: `Y_POSSIBLE_NAMES=latitude`
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Did you have access to standard output and error of the original R script?
> 2. What kind of information you can retrieve here in the standard output and/or error?
>
> > <solution-title></solution-title>
> >
> > 1. Yes, of course ;) A previsualization of stdout is visible when clicking on the history output dataset and full report accessible through the information button, then stdout or stderr (here you can see warnings on the stderr)
> > 2. The stderr is showing several warning related to automatic variable name mapping from GBIF to OGR plus information about application of a truncate process on a particularly long GeoJSON value
> >
> {: .solution}
>
{: .question}


## Visualize your data on a GIS oriented visualization

From your GeoJSON Galaxy history dataset, you can launch GIS visualization.

> <hands-on-title>Launch OpenLayers to visualize a map with your filtered records</hands-on-title>
>
> 1. Click on the *Visualize* tab on the upper menu and select `Create Visualization`
> 2. Click on the OpenLayers icon
> 3. Select the GeoJSON file from your history
> 4. Click on `Create Visualization`
> 5. Select Openlayers
>
> > <question-title></question-title>
> >
> > 1. You don't see Opebnlayers? Did you know why?
> >
> > > <solution-title></solution-title>
> > >
> > > 1.If you don't see Openlayers but others visualization types like Cytoscape, this means your datatype is JSON, not geojson. You have to change the datafile manually before visualizing it
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}


# Conclusion

In this tutorial we learned how to get occurrence records from GBIF and several steps to filter these data to be ready to analyze it! So now, let's go for the show!


topic: ecology
tutorial: ref-based-rad-seq
title: RAD-Seq Reference-based data analysis
difficulty: intermediate
time_estimation: 8h
tools_json: []
requirements_json: []
content_hash: 2efee13e416165580f57a91a0790d7b6
last_modified: 2026-05-27T09:01:48.401419
zenodo_link: https://doi.org/10.5281/zenodo.1134547
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/ref-based-rad-seq/tutorial.html

content:
In the study of [Hohenlohe *et al.* 2010](http://journals.plos.org/plosgenetics/article?id=10.1371/journal.pgen.1000862), a genome scan of nucleotide diversity and differentiation in natural populations of threespine stickleback *Gasterosteus aculeatus* was conducted. Authors used Illumina-sequenced RAD tags to identify and type over 45,000 single nucleotide polymorphisms (SNPs) in each of 100 individuals from two oceanic and three freshwater populations.

![The abstract of the paper on Population Genomics](../../images/RAD4_Population_Genomics/Hohenlohe_et_al_2010.png)

We here proposed to re-analyze these data at least until the population genomics statistics calculation step using STACKS pipeline. *Gasterosteus aculeatus* draft genome will be used here as reference genome. In a reference-based RAD-seq data analysis, the reads are aligned (or mapped) against a reference genome to constrain our analysis to focus on already discovered loci. A de-novo approach can also be conducted (see [de_novo tutorial]({% link topics/ecology/tutorials/de-novo-rad-seq/tutorial.md %}), enhancing discoverability of new loci of interest but also of false positive one).


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pretreatments

## Data upload

The original data is available at NCBI SRA ENA under accession number [SRR034310](https://trace.ncbi.nlm.nih.gov/Traces/sra/?run=SRR034310) as part of the NCBI SRA ENA study accession number [SRP001747](https://trace.ncbi.nlm.nih.gov/Traces/sra/?study=SRP001747).

We will look at the first run SRR034310 out of seven which includes 16 samples from 2 populations, 8 from Bear Paw (freshwater) and 8 from Rabbit Slough (oceanic). We will download the reads directly from SRA and the remaining data (i.e reference genome, population map file, and barcodes file) from [Zenodo](https://zenodo.org/record/1134547#.WkugUjfjJXI).


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RAD-seq exercise. If you are not inspired, you can name it "STACKS RAD: population genomics with reference genome" for example...
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **EBI SRA** {% icon tool %} import files from SRA:
>    - Search for `SRR034310`
>    - Under results for *Run*, click on *view all 1 result*
>    - In the next menu, click on the result `SRR034310`
>    - In the table at the bottom, click the link in the column **FASTQ files (Galaxy)**
>
>    This will redirect to the Galaxy website and start the download.
>
> 3. Upload remaining training data from Zenodo:
>    ```
>    https://zenodo.org/record/1134547/files/Barcode_SRR034310.txt
>    https://zenodo.org/record/1134547/files/Details_Barcode_Population_SRR034310.txt
>    https://zenodo.org/record/1134547/files/Reference_genome_11_chromosomes.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Make sure the fastq.gz files you imported are of datatype `fastqsanger.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger.gz" %}
>
>    Next, we need to tell Galaxy about the reference genome associated with this.
>
> 5. We will define a custom reference build** for this tutorial:
>
>    {% snippet faqs/galaxy/analysis_add_custom_build.md name="Stickleback" dbkey="stickleback" fasta="Reference_genome_11_chromosomes.fasta" %}
>
> 6. Next, we set the `dbkey` of our imported fastq files to our custom build key
>
>    {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="stickleback" %}
>
> 7. **Rename datasets** according to the sample names
>
{: .hands_on}

The sequences are raw sequences from the sequencing machine, without any pretreatments. They need to be demultiplexed. To do so, we can use the Process Radtags tool from STACKS.

## Demultiplexing reads

For demultiplexing, we use the Process Radtags tool from [STACKS](https://www.g3journal.org/content/1/3/171.full).

> <hands-on-title>Demultiplexing reads</hands-on-title>
>
> 1. **Process Radtags** {% icon tool %} with the following parameters:
>    - *"Single-end or paired-end reads files"*: `Single-end files`
>      - *"Singles-end reads infile(s)"*: `SRR034310.fastq(.gz)`
>    - *"Barcode file"*: `Barcodes_SRR034310.tabular`
>    - *"Number of enzymes"*: `One`
>    - *"Enzyme"*: `sbfI`
>    - *"Capture discarded reads to a file"*: `Yes`
>    - *"Output format"*: `fastq`
>
>    > <question-title></question-title>
>    >
>    > 1. How many reads where on the original dataset?
>    > 2. How many are kept?
>    > 3. Can you try to explain the reason why we loose a lot of reads here?
>    > 4. What kind of information does this result give concerning the upcoming data analysis and the barcodes design in general?
>    >
>    > > <solution-title></solution-title>
>    > > The informations can be found in the results log file:
>    > >
>    > >  1. 8895289 total reads
>    > >  2. 8139531 retained reads
>    > >  3. There are no sequences filtered because of low quality. This is because radtags didn't apply quality related filtering since the corresponding advanced option (Discard reads with low quality scores) has not been enabled. So here, all not retained sequences are removed because of an ambiguous barcode (626265) or an ambiguous RAD-Tag (129493). This means that some barcodes are not exactly what was specified on the barcode file and that sometimes, no SbfI restriction enzyme site was found. This can be due to some sequencing problems but here, this is also due to the addition, in the original sequencing library, of RAD-seq samples from another study. This strategy is often used to avoid having too much sequences beginning with the exact same nucleotide sequence which may cause Illumina related issues during sequencing and cluster analysis
>    > >  4. Sequencing quality is essential! Each time your sequencing quality decreases, you loose data and thus essential biological information!
>    > >
>    > > In addition to the overall statistics the numbers of retained and removed reads are also given for each barcode sequence.
>    > {: .solution}
>    {: .question}
>
> 2. **Process Radtags** {% icon tool %}: Re-Run playing with parameters
>    - Section **Advanced**
>      - *"Discard reads with low quality scores"*: `Yes`
>      - *"score limit"*: `20` (for example; play with this)
>      - *"Set the size of the sliding window as a fraction of the read length, between 0 and 1"*: `0.30` (for example; play with this)
>
> This sliding window parameter allows notably the user to deal with the declining quality at the 3' end of reads.
>
{: .hands_on}


> <hands-on-title></hands-on-title>
>
> You can use the `Charts` functionality through the Visualize button reachable on the `Radtags logs` file you just generated.
>
> If like me you don't have payed attention to the organization of you file for the graphical representation you obtain a non optimal bars diagram with a not intelligent X-axis ordering. There is a lot of different manner to fix this. You can use a copy/paste "bidouille" or you can use Galaxy tools to manipulate the `radtags logs` file to generate a better graph. For example, you can use `Select lines that match an expression` tool to select rows then use the `Concatenate datasets tail-to-head` tool to reorganize these lines in a new file.
> Then we generate a graphical display of the changes:
>
> First we cut the interesting lines of each `result.log with Stacks: process radtags`
>
> 1. **Select lines that match an expression** applying `^R1.fq.gz` on the log files and then
> 2. **Replace Text in entire line** on the resulting data sets finding `R1.fq.gz` and replacing with `NoScoreLimit` or `Score10` or `Score20` depending of the case
> 3. **Select lines that match an expression** applying `File\tRetained Reads` on one of the log file to obtain you futur header
> 4. **Replace Text in entire line** on the resulting data set finding `File` and replacing with `#` to have a best display
> 5. **Concatenate datasets tail-to-head** on the resulting data sets stating from the header you just created
> 6. Finally **Convert delimiters to TAB** on the resulting data set converting all Tabs to be sure having a well formatted tabular file at the end
>
> Alternatively just copy/paste these lines on the Galaxy upload tool using Paste/fetch data section and modifying the File header by sample and filename by Score 10 / Score 20 and Lowquality for example... Before Starting the upload, you can select the `Convert spaces to tabs` option through the `Upload configuration` wheel. If you did not pay attention to the order you can just sort the file using the first column.
{: .hands_on}

And you obtain a file like this one, ready to generate a beautiful and smart bar stacked!

```
#		Retained Reads	Low Quality	Ambiguous Barcodes	Ambiguous RAD-Tag	Total
NoScoreLimit	8139531		0		626265			129493			8895289
Score10		7373160		766371		626265			129493			8895289
Score20		2980543		5158988		626265			129493			8895289
```

You can use the `Charts` functionality through the Visualize button to plot the data.

![The chart on the sorted file](../../images/RAD4_Population_Genomics/Process_radtags_charts_end.PNG){: width="75%"}

You can further test using a filter like `clean data, remove any read with an uncalled base` and see that here, this has only little impact on the number of retained reads.

The demultiplexed sequences are raw sequences from the sequencing machine, without any pretreatments. They need to be controlled for their quality.

We propose to continue the tutorial using the dataset collection containing the demultiplexed reads obtained with Process Radtag execution made with a quality score of 10 and with the `Discard reads with low quality scores` parameter set to Yes (so containing 7373160 retained reads).

## Quality control

For quality control, we use similar tools as described in [NGS-QC tutorial]({{site.baseurl}}/topics/sequence-analysis/tutorials/quality-control/tutorial.html): [Falco](https://falco.readthedocs.io/en/latest/)

Falco is an efficiency-optimized rewrite of [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/).

> <hands-on-title>Quality control</hands-on-title>
>
> 1. **Falco** {% icon tool %} to check the quality of the reads:
>    - {% icon param-collection %} *"Short read data from your current history"*: the fastq files (collection)
>
>    > <question-title></question-title>
>    >
>    > 1. What is the read length?
>    >
>    > > <solution-title></solution-title>
>    > > The read length is 32 bp
>    > {: .solution }
>    {: .question}
>
> 2. **MultiQC** {% icon tool %} with the following parameters:
>    - *"Which tool was used generate logs?"*: `FastQC` (no matter whether FastQC or Falco was used)
>    - *"Type of FastQC output?"*: `raw data`
>    - {% icon param-collection %} *"FASTQC output"*: the `raw data` output of Falco (collection)
{: .hands_on}

![MultiQC output](../../images/RAD4_Population_Genomics/multiqc.PNG)

Note the quality drop at bases 5-10 which are the cut site of the RADSeq
protocol (TGCAGG). This is caused by the extremely uneven distribution the
sequencer observes TGCAGG in lockstep on all spots/clusters, i.e. all clusters
light up at the same time. Therefore the sequencer can not be sure that each
spot/cluster is correct which is expressed by the lower quality score. Usually, to reduce this problem,
PhiX is added, OR, as it is the case here, samples with heterogeneous barcodes lengths are mixed.
Still, approximately ~99% of the cases match nucleotide expected for the cut
site sequence.



As there is a draft genome for *Gasterosteus aculeatus*, we can use this information and map the sequences on this genome to identify polymorphism.

# Mapping

To make sense of the reads, their positions within *Gasterosteus aculeatus* genome must be determined. This process is known as aligning or 'mapping' the reads to the reference genome.

> <comment-title></comment-title>
>
> Do you want to learn more about the principles behind mapping? Follow our [training]({{site.baseurl}}/topics/sequence-analysis/)
{: .comment}

Here we will use BWA. BWA is a fast light-weighted tool that aligns relatively short sequences (queries) to a sequence database (large), such as the human reference genome. It is developed by Heng Li at the Sanger Institute.

*[Li et Durbin, Bioinformatics, 2009](https://www.ncbi.nlm.nih.gov/pubmed/19451168)*


> <hands-on-title>Map with BWA</hands-on-title>
>
> 1. **Map with BWA - map short reads (< 100 bp) against reference genome** {% icon tool %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>    - *"Use the following dataset as the reference sequence"*: `Reference_genome_11_chromosomes.fasta`
>    - *"Select input type"*: `Single Fastq`
>    - *"Select fastq dataset"*: One dataset collection containing the demultiplexed reads obtained with Process Radtag execution made with a quality score of 10 and with the `Discard reads with low quality scores` parameter set to Yes (so containing 7373160 retained reads).
>
{: .hands_on}

**BWA** generates BAM files with the mapped reads.

We next want to run Stacks on the freshwater and oceanic populations.

# SNP calling from radtags

Run `Stacks: Reference map` Galaxy tool. This program will run pstacks, cstacks, and sstacks using a perl scrip (ref_map.pl) on the members of the population, accounting for the alignments of each read.

> <comment-title></comment-title>
>
> Information on ref_map.pl and its parameters can be found online: https://catchenlab.life.illinois.edu/stacks/comp/ref_map.php.
{: .comment}

> <hands-on-title>Stacks: Reference map</hands-on-title>
>
> 1. Create a new file with our population map and name it `Population_map.txt`:
>
>    ```
>    sample_CCCC	1
>    sample_CCAA	1
>    sample_CCTT	1
>    sample_CCGG	1
>    sample_CACA	1
>    sample_CAAC	1
>    sample_CATG	1
>    sample_CAGT	1
>    sample_CTCT	2
>    sample_CTAG	2
>    sample_CTTC	2
>    sample_CTGA	2
>    sample_GGGG	2
>    sample_GGAA	2
>    sample_GGTT	2
>    sample_GGCC	2
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md %}
>
> 2. **Stacks: Reference map** {% icon tool %}:
>    - *"Select your usage"*: `Population`
>    - *"Files containing an individual sample from a population"*: Choose the mapped reads (data collection)
>    - *"pecify a population map"*: `Population_map.txt` created above
>    - *"Minimum depth of coverage"*: `3`
>
>    > <comment-title></comment-title>
>    > If you are using a file presenting population information and individual name in a different manner than expected by STACKS, you can use Galaxy tools like `Replace Text` (for example to replace `Rabbit Slough` by a population number like `2`, `Add column` (for example to add `sample_`) or `Cut columns from a table` (to put the new `sample_` column af the first place) and finally `Regex replace` (replacing `(sample_)\t` by `\1`) to generate it...
>    {: .comment}
>
> Once Stacks has completed running, investigate the output files: `ref_map.log`, `Summary from Stacks: reference map on data...` and `Catalog (*)` (snps, alleles and tags).
> Notice that each locus now has a chromosome/base pair specified in each of the `*tags.tsv` files and in the catalog files.
>
{: .hands_on}

# Calculate population genomics statistics

> <hands-on-title>Calculate population genomics statistics</hands-on-title>
>
> 1. **Stacks: populations** {% icon tool %} with the following parameters:
>    - *"Input type"*: `Stacks output`
>    - *"Output from previous Stacks pipeline steps (e.g. denovo_map or refmap)"*: Full output from ref_map
>    - *"Specify a population map"*: `Population_map.txt`
>    - Section **Data filtering options**
>      - *"Minimum percentage of individuals in a population required to process a locus for that population"*: `0.75`
>    - Section **Output options** (VCF and Structure) and enabling SNP and haplotype-based F statistics calculation.
>      - *"Output results in Variant Call Format (VCF)"*: `Yes`
>      - *"Output results in Structure Format"*: `Yes`
>    - *"Enable SNP and haplotype-based F statistics"*: `Yes`
>
> 2. Now look at the output in the file `batch_1.sumstats` named `SNP and Haplotype-based F statistics with Stacks: populations ...` on your history. This file is also reachable on the data collection nammed `Full output from ref_map .....` with his original name `batch_1.sumstats`. There are a large number of statistics calculated at each SNP, so use Galaxy tools like filter, cut, and sort to focus on some.
>
> > <question-title></question-title>
> >
> > 1. What is the maximum value of FST' at any SNP? Don't hesitate to look at the [STACKS manual](http://catchenlab.life.illinois.edu/stacks/manual/#files) to find where this parameter is
> > 2. What is the meaning of this FST' value compared a a classical FST one? Once again, the [STACKS manual](http://catchenlab.life.illinois.edu/stacks/manual/#files) can help you ;)
> > 3. How many SNPs reach this FST' value?
> >
> > > <solution-title></solution-title>
> > > 1. 1
> > > 2. FST’ is a haplotype measure of FST that is scaled to the theoretical maximum FST value at this locus. Depending on how many haplotypes there are, it may not be possible to reach an FST of 1, so this method will scale the value to 1
> > > 2. 78
> > {: .solution }
> {: .question}
> You can now for example filter this dataset to only keep FST'=1 loci for further analysis...
{: .hands_on}


# Conclusion


In this tutorial, we have analyzed real RAD sequencing data to extract useful information, such as which loci are candidate regarding the genetic differentiation between freshwater and oceanic Stickelback populations. To answer these questions, we analyzed RAD sequence datasets using a reference-based RAD-seq data analysis approach. This approach can be sum up with the following scheme:


![Ref based tutorial workflow](../../images/ref_based_workflow.PNG)

tags_json:
["RAD-seq", "genetic composition EBV class", "species population EBV class", "EBV dataset", "EBV workflow", "biodiversity"]


topic: ecology
tutorial: marine_omics_bgc
title: Marine Omics identifying biosynthetic gene clusters
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["earth-system", "ocean", "marine omics", "biodiversity"]
content_hash: de5057f2f6c2a8780dd07f1e68b192e2
last_modified: 2026-05-27T09:01:48.400282
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/marine_omics_bgc/tutorial.html

content:
# Introduction
Through this tutorial, you will learn in the first part how to produce a protein fasta file from a nucleotide fasta file using Prodigal (it is a tool that predicts protein-coding genes from DNA sequences).

Then, you'll be using InterProscan to create a tabular. Interproscan is a batch tool to query the InterPro database. It helps identify and predict the functions of proteins by comparing them to known databases.

And finally, you will discover SanntiS both to build genbank and especially to detect and annotate biosynthetic gene clusters (BGCs).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data
The FASTA file used in this example is, intentionally, a very small fraction of a genome (spanning exactly 1 BGC), and it's main purpose is to quickly check that all the pieces of the workflow are working.

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name (for example “Marine Omics: SMBGC annotation”) for you to find it again later if needed.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the file with this link `https://figshare.com/ndownloader/files/48574534`and name it BGC0001472.fna
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. Rename the datasets BGC0001472.fna
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

{% include _includes/cyoa-choices.html option1="Tools" option2="Workflow"
       text="Do you want to run the workflow or to discover the tools one by one ?" %}

<div class="Workflow" markdown="1">

# Import and launch the workflow
> <hands-on-title>Import the workflow</hands-on-title>
>   - Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
>   - Option 1: use the URL
>   	- Click on {% icon galaxy-upload %} Import at the top-right of the screen
>   	- Paste the URL of the workflow into the box labelled “Archived Workflow URL” `https://earth-system.usegalaxy.eu/u/marie.josse/w/marine-omics-identifying-biosynthetic-gene-clusters`
>   - Option 2: use the workflow name
>   	- Click on **Public workflows** at the top-right of the screen
>       - Search for **Marine Omics identifying biosynthetic gene clusters**
>       - In the workflow preview box click on {% icon galaxy-upload %} Import
>   - Click the Import workflow button
{: .hands_on}

> <hands-on-title>Run the workflow</hands-on-title>
>    - Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
>    - Click on the {% icon workflow-run %} (Run workflow) button next to your workflow
>    - /!\ Select **Yes** for **Workflow semi automatic**
>    - Configure the workflow as needed with the 2 datasets you uploaded right before (**BGC0001472.fna**)
>    - Click the Run Workflow button at the top-right of the screen
>    - You may have to refresh your history to see the queued jobs
{: .hands_on}

Now you don't have to do anything else. You should see all the different steps of the workflow appear in your history.
When the workflow is fully completed you should have the following history.

![Image of the history with all the steps of the workflow](../../images/marineomics/history.png "History"){: style="width:50%"}

> <tip-title>Extract a RO-Crate</tip-title>
> Workflows are a powerful Galaxy feature that allows you to scale up your analysis by performing an end-to-end analysis with a single click of a button. In order to keep provenance of the workflow invocation (an invocation of a workflow means one run or execution of the workflow) it can be exported from Galaxy in the form of a Workflow Run Crate RO-Crate profile.
>
>
> After the workflow has completed, we can export the RO-Crate. The crate does not appear in your history, but can be accessed from the User -> Workflow Invocations menu on the top bar.
>
> > <hands-on-title> Extract a RO-Crate </hands-on-title>
> >
> > 1. In the top right of your history, go to {% icon galaxy-history-options %} -> Show Invocations
> >
> > ![Image of the history options](../../images/marineomics/showinvoc.png "History options")
> >
> > Our latest workflow run should be listed at the top.
> >
> > 2. Click on it to expand it:
> >
> > ![Image of the workflow invocation](../../images/marineomics/workflowinvoc.png "Workflow invocation")
> >
> > 3. Click on the Export tab in the expanded view of the workflow invocation.
> > 	You should see a page that contains three download options:
> >		- Research Object Crate (RO-Crate)
> >		- BioCompute Object
> >		- File
> > 4. Click on the **Generate** {% icon galaxy-download %} option of the RO-Crate box (1st box)
> >
> > ![Image of the RO-Crate download](../../images/marineomics/rocrate.png "RO-Crate")
> {: .hands-on}
>
>
> Great work! You have created a Workflow Run Crate. This makes it easy to track the provenance of the executed workflow.
{: .tip}

</div>


<div class="Tools" markdown="1">
# Prodigal Gene Predictor: generate a protein fasta file



## Prodigal Gene Predictor

Prodigal is a tool that predicts protein-coding genes from DNA sequences. It takes a nucleotide FASTA file as input and identifies regions that are likely to code for proteins. The output is a protein FASTA file where each sequence represents a predicted protein. Some of these sequences end with an asterisk (*), which marks the end of a complete protein sequence identified by Prodigal. This asterisk is added when Prodigal detects a full protein-coding region that ends with a stop codon. Sequences without an asterisk either represent partial proteins or do not end in a typical stop codon.

> <hands-on-title> Run prodigal </hands-on-title>
>
> 1. {% tool [Prodigal Gene Predictor](toolshed.g2.bx.psu.edu/repos/iuc/prodigal/prodigal/2.6.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Specify input file"*: `BGC0001472.fna` (Input the nucleotide fasta file)
>    - *"Specify mode"*: `Meta : Anonymous sequences, analyze using preset training files, ideal for metagenomic data or single short sequences`
>
>    You don't need to change any other parameters leave them on the default input.
>
> 2. Click on **Run Tool**
>
{: .hands_on}

You should have 4 new outputs appearing in your history. In these outputs you should have **Prodigal Gene Predictor on data 1 : protein translations file**.
You can click on it and then click on the {% icon galaxy-eye %} (eye).

![Image of the protein fasta file from prodigal and of the new history with all the new outputs](../../images/marineomics/prodigal.png "Prodigal outputs")

You can notice here that at each end of the sequence there's a *. Later on we will need to remove this star. But, first we are going to use this protein file to build the Genbank that SanntiS need to make a SMBGC annotation.

## SanntiS for building a Genbank file

This step combines the original nucleotide sequences with the cleaned protein sequences to create a GenBank format file. This format is widely used for storing and organising information about DNA sequences and their annotations. In this step, the DNA sequences and their corresponding coding regions are transformed into a format that is suitable for SanntiS.

> <hands-on-title> Build Genbank </hands-on-title>
>
> 1. {% tool [SanntiS biosynthetic gene clusters](toolshed.g2.bx.psu.edu/repos/ecology/SanntiS_marine/SanntiS_marine/0.9.3.5+galaxy1) %} with the following parameters:
>    - *"Do you want to build a genbank or to make a SMBGC Annotation?"*: `Build genbank`
>        - {% icon param-file %} *"Input a nucleotide fasta file"*: `BGC0001472.fna` (Input the nucleotide fasta file)
>        - {% icon param-file %} *"Input a protein fasta file"*: `Prodigal Gene Predictor on data 1 : protein translations file` (output of **Prodigal Gene Predictor** {% icon tool %})
> 2. Click on **Run Tool**
>
{: .hands_on}

![Image of the Genbank file produces by SanntiS](../../images/marineomics/genbank.png "Genbank file")

## Regex Find And Replace
Remember earlier we noticed the star * in the protein fasta file ?

Now is the time to remove it !

The asterisks at the end of some protein sequences are informative but can cause issues with some analysis tools. In this step, we remove these asterisks to produce a clean protein FASTA file, making it ready for further analysis.

> <hands-on-title> Remove * </hands-on-title>
>
> 1. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `Prodigal Gene Predictor on data 1 : protein translations file` (output of **Prodigal Gene Predictor** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `\*$`
>            - *"Replacement"*: `` (leave an empty box there)
>
> 2. Click on **Run Tool**
>
{: .hands_on}

Check if the * were well removed.

# InterProScan

InterProScan is a tool that helps identify and predict the functions of proteins by comparing them to known databases. This tool analyses the protein sequences and produces an output file containing detailed information about the possible functions, domains, and families associated with these proteins.

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [InterProScan](toolshed.g2.bx.psu.edu/repos/bgruening/interproscan/interproscan/5.59-91.0+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Protein FASTA File"*: `Regex Find And Replace on data **` (output of **Regex Find And Replace** {% icon tool %})
>    - *"Use applications with restricted license, only for non-commercial use?"*: `No`
> You can leave all the other parameters on the default input.
>
> 2. Click on **Run Tool**
{: .hands_on}

![Image of InterProScan output in tabular (tsv) in the history](../../images/marineomics/interproscan.png "InterProScan output"){: style="width:50%"}

# SanntiS for annotating biosynthetic gene clusters

SanntiS is a tool specifically designed to detect and annotate biosynthetic gene clusters (BGCs). It uses neural networks trained on InterPro signatures to achieve high accuracy in identifying BGCs in both genomic and metagenomic datasets 1. A significant benefit of using SanntiS is that your results will be comparable with a large number of datasets, including over 5,000 marine metagenomic assemblies archived in the [MGnify](https://www.ebi.ac.uk/metagenomics) resource. This tool provides valuable insights into the biosynthetic potential of organisms or environmental samples. The final output is a GFF file containing detailed BGC annotations, which can be used for further analyses or applications.

> <hands-on-title> Identify biosynthetic gene clusters </hands-on-title>
>
> 1. {% tool [SanntiS biosynthetic gene clusters](toolshed.g2.bx.psu.edu/repos/ecology/sanntis_marine/sanntis_marine/0.9.3.5+galaxy1) %} with the following parameters:
>    - *"Do you want to build a genbank or to make a SMBGC Annotation?"*: `Run SanntiS`
>        - {% icon param-file %} *"Input the tabular file from InterProScan"*: `InterProScan on data **` (output of **InterProScan** {% icon tool %})
>        - {% icon param-file %} *"Input a Genbank file"*: `SanntiS output data genbank` (output of **SanntiS biosynthetic gene clusters** {% icon tool %})
>
> 2. Click on **Run Tool**
{: .hands_on}

</div>

Finally, you should have one gff3 file in your history under **SanntiS output data**
![Image of SanntiS output](../../images/marineomics/sanntis.png "SanntiS output")


# Conclusion
Here you now know how to conduct a Marine Omics analysis

# Extra information

Coming up soon even more tutorials on and other Earth-System related trainings. Keep an {% icon galaxy-eye %} open if you are interested!

tools_json:
["Prodigal Gene Predictor", "toolshed.g2.bx.psu.edu/repos/iuc/prodigal/prodigal/2.6.3+galaxy0", "SanntiS biosynthetic gene clusters", "toolshed.g2.bx.psu.edu/repos/ecology/SanntiS_marine/SanntiS_marine/0.9.3.5+galaxy1", "Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.3", "InterProScan", "toolshed.g2.bx.psu.edu/repos/bgruening/interproscan/interproscan/5.59-91.0+galaxy3", "toolshed.g2.bx.psu.edu/repos/ecology/sanntis_marine/sanntis_marine/0.9.3.5+galaxy1"]


topic: ecology
tutorial: eDNA-taxonomic-analysis
title: Taxonomic Analysis of eDNA
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["biodiversity", "metabarcoding"]
content_hash: adc39b162bfde378cd341022a459bd3f
last_modified: 2026-05-27T09:01:48.397351
zenodo_link: https://zenodo.org/records/15367390
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/eDNA-taxonomic-analysis/tutorial.html

content:
Environmental DNA (eDNA) analysis has emerged as a powerful tool in ecological and conservation research. It involves the extraction and analysis of DNA shed by organisms into their surrounding environment without the need to directly observe or capture the species. This non-invasive approach allows us to gain valuable insights into biodiversity, species distribution, and community composition in a wide range of ecosystems, including marine environments, from which the eDNA in this tutorial originates.

This tutorial will guide you through a basic workflow for the taxonomic analysis of unknown eDNA sequence data. We will start with raw sequence reads and go through the steps of quality control, sequence alignment against a reference database, and the extraction of potential taxonomic information.

In this tutorial, we will address the following biological question:

- Can we assign taxonomic information to unknown DNA sequences from environmental samples?

By the end of this tutorial, you will be able to:

- Upload raw sequencing data to the Galaxy platform.
- Apply read trimming tools to improve the quality of eDNA sequence data.
- Convert sequence data between FASTQ and FASTA formats.
- Perform a BLASTN search against a nucleotide database to identify potential taxonomic matches.
- Extract potential genus and species names from the BLAST output table.

While this tutorial focuses on eDNA, the fundamental steps of sequence quality control and identification are applicable to many other types of biological sequence data.

> <agenda-title>Taxonomic Analysis of eDNA Workflow</agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Uploading the eDNA Sequence Data

In this section, you will learn how to bring your raw sequencing data into the Galaxy platform. Galaxy is a powerful, web-based platform that provides access to a wide range of bioinformatics tools and computational resources, all within an accessible interface. Think of it as your workbench for biological data analysis.

Let's briefly discuss what kind of data we'll be working with. Environmental DNA (eDNA) sequencing typically generates a large number of short DNA sequences, often called "reads." These reads represent fragments of genetic material present in the environmental sample (in our case, from a marine environment). Our goal is to take these raw, unannotated reads and, through a series of bioinformatics steps, identify the organisms they might have originated from.

The first step in any Galaxy-based analysis is getting your data into the platform. Galaxy offers several ways to upload data, including importing files either from a web link (if the data is hosted on Zenodo), from a shared data library within your Galaxy instance, or your local device.

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. **Create a new history** for this tutorial.
>    - Give it a meaningful name, such as "eDNA Taxonomic Analysis."
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Import the files** for this tutorial from Zenodo:
>
>    ```
>    https://zenodo.org/records/15367390/files/HP3_S9_L001_R1_001.fastq.gz
>    https://zenodo.org/records/15367390/files/HP3_S9_L001_R2_001.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Rename the datasets** once they are uploaded into your history. Click on the pencil icon next to each dataset name in the history panel (on the right side of the Galaxy interface) and give them informative names.
> 4. **Check that the datatype** for each of your renamed datasets, ensure the datatype is set to `fastqsanger` or `fastqsanger.gz`. To check or change the datatype, click on the pencil icon next to the dataset name, then look for the "Datatypes" field. If it's incorrect, select the appropriate datatype from the dropdown menu and click "Save".
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. **(Optional) Add tags to your datasets.** Tags are keywords that can help you organize and search your history. For each dataset, click the add tags line and enter your desired tag(s) in the field.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

#  Data Quality Control

After successfully uploading our raw eDNA sequence reads into Galaxy, the next step is to assess and improve the quality of the data. Raw sequencing data can contain errors introduced during the sequencing process, such as incorrect base calls. These errors can negatively impact analyses, including taxonomic identification, so performing quality control is essential to obtain reliable results.

The goal of quality control is to identify and remove (trim) low-quality reads or portions of reads. This involves evaluating various metrics, such as the Phred quality scores, the presence of adapter sequences, and the overall length distribution of the reads.

By performing quality control, we aim to:

  * **Improve the accuracy of downstream analyses:** High-quality data leads to more reliable taxonomic assignments.
  * **Reduce computational resources:** Removing low-quality data can decrease the size of the dataset and speed up subsequent steps.
  * **Increase the sensitivity of analysis:** By removing noise, we can potentially identify less abundant organisms.

In the following hands-on step, we will use the `fastp` tool, a fast and comprehensive preprocessor for FASTQ files, to perform quality control on our eDNA sequence data.

> <details-title> More about Sequencing Errors and Quality Scores </details-title>
>
> Sequencing technologies are not perfect and can introduce errors. These errors are often base substitutions, insertions, or deletions. The Phred quality score (Q) is a widely used metric to represent the probability of a base being called incorrectly. It is calculated using the formula:
>
> $$Q = -10 \log_{10}(P)$$
>
> where (P) is the probability of an incorrect base call.
>
> Adapter sequences are short, synthetic DNA molecules added to the ends of DNA fragments during library preparation. These adapters facilitate the sequencing process but are not part of the original environmental DNA. It's crucial to remove them as their presence can lead to incorrect alignments and taxonomic assignments.
>
{: .details}

## Concurrent Quality Control Steps with **fastp**

> <hands-on-title> Quality Control with fastp </hands-on-title>
>
> 1.  {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.24.1+galaxy0) %} for HP3 R1 (Default Parameters):
>
>       - *"Single-end or paired reads"*: `Single-end`
>           - {% icon param-file %} *"Input 1"*: `output` (Input dataset for HP3 R1)
>
>
> 2.  {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.24.1+galaxy0) %} for HP3 R2 (Default Parameters):
>
>       - *"Single-end or paired reads"*: `Single-end`
>           - {% icon param-file %} *"Input 1"*: `output` (Input dataset for HP3 R2)
>
>  > <comment-title> Running `fastp` </comment-title>
>  >
>  > In this step, we are running `fastp` on both the HP3 R1 and HP3 R2 files using its default parameter settings. We are applying `fastp` to the datasets that were uploaded in the previous step:
>  > * `HP3_S9_L001_R1_001.fastq.gz`
>  > * `HP3_S9_L001_R2_001.fastq.gz`
>  >
>  > **Important:** Galaxy will provide default names for the output files. For better organization and recognition between the results for HP3 R1 and HP3 R2, **it is recommended to rename the outputs** in your Galaxy history. For example, you could rename the reports to `fastp report on HP3_R1` and `fastp report on HP3_R2`, and the filtered reads to `fastp output HP3_R1` and `fastp output HP3_R2`.
>  > **Expected Outputs (for each input dataset):**
>  > * `fastp on data X: JSON report`
>  > * `fastp on data X: HTML report`
>  > * `fastp on data X: Read 1 output` (FASTQ format containing the quality-filtered sequences from your input)
>  >
>  > While we are using the default settings here, it's important to remember that these might not always be optimal for all types of data, as different datasets can have varying quality issues or adapter contamination. Running it in single-end mode treats each input file independently, for simplicity of the tutorial. Depending on your datasets, or what information you are trying to gather, the Trimmomatic tool is useful for handling pair-ended sequences.
>  {: .comment}
>
{: .hands_on}


> <question-title> Considering Default Parameters </question-title>
>
> 1.  Are the default parameters of a quality control tool like `fastp` always the most appropriate choice for every sequencing dataset? Why or why not?
>
> > <solution-title> Solution </solution-title>
> >
> > 1.  No, default parameters might not always be optimal. Sequencing data can vary in quality, adapter contamination levels, and other characteristics depending on the library prep and platform. It is often necessary to adjust parameters to suit the specific needs of your data for appropriate levels of quality control.
> >
> {: .solution}
>
{: .question}

#  FASTQ to FASTA Conversion

## Converting `fastp` Outputs to FASTA Format

> <hands-on-title> Task description </hands-on-title>
>
> 1.  {% tool [FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastq_to_fasta/cshl_fastq_to_fasta/1.0.2+galaxy2) %} for HP3 R1:
>
>       - {% icon param-file %} *"Input FASTQ file"*: `fastp output HP3_R1` (the renamed filtered reads output of **fastp** {% icon tool %} for HP3 R1)
>       - *"Discard sequences with unknown (N) bases"*: `Yes`
>       - *"Rename sequence names in output file (reduces file size)"*: `Yes`
>       - *"Compress output FASTA"*: `No`
>
> 2.  {% tool [FASTQ to FASTA](toolshed.g2.bx.psu.edu/repos/devteam/fastq_to_fasta/cshl_fastq_to_fasta/1.0.2+galaxy2) %} for HP3 R2:
>
>       - {% icon param-file %} *"Input FASTQ file"*: `fastp output HP3_R2` (the renamed filtered reads output of **fastp** {% icon tool %} for HP3 R2)
>       - *"Discard sequences with unknown (N) bases"*: `Yes`
>       - *"Rename sequence names in output file (reduces file size)"*: `Yes`
>       - *"Compress output FASTA"*: `No`
>
>  > <comment-title> Converting to FASTA for NCBI BLAST+ blastn </comment-title>
>  >
>  > In this step, we are converting the quality-controlled FASTQ files into FASTA format with the following options: discarding any sequences that contain unknown ('N') bases and renaming the sequence identifiers to reduce file size. The FASTA format, which contains only the sequence information, is a required input format for **NCBI BLAST+ blastn**, which will be used in the subsequent section for sequence similarity searching and potential taxonomic identification.
>  {: .comment}
>
{: .hands_on}

> <question-title> Understanding FASTQ to FASTA Conversion </question-title>
>
> 1.  What is the purpose of converting sequencing data from FASTQ format to FASTA format?
>
> > <solution-title> Solution </solution-title>
> >
> > 1.  The purpose of converting from FASTQ to FASTA is to obtain a file containing only the nucleotide sequences.
> >
> {: .solution}
>
{: .question}

#  Sequence Alignment for Taxonomic Identification using BLAST

## Performing the BLASTn Search with **NCBI BLAST+ blastn**

> <hands-on-title> Performing the BLASTn Search </hands-on-title>
>
> 1.  Execute the {% tool [NCBI BLAST+ blastn](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.16.0+galaxy0)%} with the following parameters:
>
> **For HP3 R1:**
>
>   - {% icon param-file %} *"Nucleotide query sequence(s)"*: `output` (output of **FASTQ to FASTA** {% icon tool %} for HP3 R1)
>   - *"Subject database/sequences"*: `Locally installed BLAST database`
>   - *"Nucleotide BLAST database"*: `NCBI NT (15 Aug 2024)`
>   - *"Type of BLAST"*: `blastn - Traditional BLASTN requiring an exact match of 11, for somewhat similar sequences`
>   - *"Set expectation value cutoff"*: `0.0001`
>   - *"Output format"*: `Tabular (extended 25 columns)`
>   - *"Advanced Options"*: `Show Advanced Options`
>       - *"Maximum hits to consider/show"*: `1`
>       - *"Restrict search of database to a given set of ID's"*: `No restriction, search the entire database`
>
> **For HP3 R2:**
>
>   - Repeat the **NCBI BLAST+ blastn** tool for your other FASTA file (e.g., HP3 R2) using the exact same parameters.
>
>  > <comment-title> Understanding BLAST Parameters and Database </comment-title>
>  >
>  > In this step, we are using BLASTn to search your eDNA sequences against the **NCBI NT database as it was on August 15, 2024**. This is a comprehensive collection of nucleotide sequences. The parameters are set to find relatively similar sequences and output the results in a detailed tabular format, keeping only the top hit for each query. **Choosing a specific database is crucial because different databases contain different sets of sequences, which can significantly affect the outcome of the search.**
>  {: .comment}
>
{: .hands_on}

> <question-title> Interpreting BLAST Database Choice </question-title>
>
> 1.  Why is it important to specify the database (like NCBI NT) when performing a BLAST search?
>
> > <solution-title> Answer </solution-title>
> >
> > 1.  Specifying the database ensures that your sequences are compared against a defined and relevant collection of sequences. Different databases have different content and scope, which can significantly impact the results and the taxonomic assignments you might obtain. Knowing the database also helps in understanding the context and limitations of your search.
> >
> {: .solution}
>
{: .question}

#  Extracting Taxonomic Information from the BLAST table

## Counting Potential Taxa using **Count**

The BLAST output (in tabular format with extended 25 columns) contains a wealth of information for each aligned sequence. Importantly, **column 25** of this output typically provides the **"subject scientific name"**, which often includes the genus and species identification of the top hit in the database. In this step, we will use the `Count` tool in Galaxy to extract these scientific names and determine how many times each unique name appears in our BLAST results for each of your input files (HP3 R1 and HP3 R2). This will give us a preliminary overview of the potential taxa present in your eDNA samples based on the top BLAST hits.

> <hands-on-title> Counting Top Taxonomic Assignments </hands-on-title>
>
> 1.  Execute the {% tool [Count](Count1) %} with the following parameters:
>
> **For HP3 R1 BLAST Results:**
>
>   - {% icon param-file %} *"from dataset"*: `output` (output of **NCBI BLAST+ blastn** {% icon tool %} for HP3 R1)
>   - {% icon param-file %} *"Count occurrences of values in column(s)"*: `c['25']`
>
> **For HP3 R2 BLAST Results:**
>
>   - Repeat the {% tool [Count](Count1) %} tool with the following parameters:
>     - {% icon param-file %} *"from dataset"*: `output` (output of **NCBI BLAST+ blastn** {% icon tool %} for HP3 R2)
>     - {% icon param-file %} *"Count occurrences of values in column(s)"*: `c['25']`
>
>  > <comment-title> Interpreting the Count Output </comment-title>
>  >
>  > The output of the **Count** tool will be a table showing each unique scientific name found in the 25th column of your BLAST output and the number of times it appeared. Since we set BLAST to only report the top hit for each eDNA sequence, this count represents the frequency of the most likely taxonomic assignment for those sequences.
>  > This output can be used for visualization such as a pie chart.
>  {: .comment}
>
{: .hands_on}

> <question-title> Understanding Top Hit Counting </question-title>
>
> 1.  Why are we using the `Count` tool on the 25th column of the BLAST output? What information does this column typically contain?
> 2.  What does the number of times a specific scientific name appears in the `Count` output roughly indicate in this analysis?
>
> > <solution-title> Answers </solution-title>
> >
> > 1.  We are using the `Count` tool on the 25th column because this column in the extended tabular BLAST output format typically contains the "subject scientific name" of the top hit from the database.
> > 2.  The number of times a scientific name appears roughly indicates how many of your original eDNA sequences had that specific organism as their best match in the database.
> >
> {: .solution}
>
{: .question}



# Conclusion

In this tutorial, you've successfully completed a basic bioinformatics workflow for the taxonomic analysis of environmental DNA (eDNA) sequences, starting with raw sequencing reads and progressed through several key steps including:

![Workflow of eDNA Taxonomic Analysis](images/Workflow.png)
![Galaxy History of eDNA Taxonomic Analysis](images/History.png)

* **Data Upload:** how to import sequence data into the Galaxy environment
* **Quality Control:** utilizing the `fastp` tool to assess and improve the quality of your reads by trimming potentially erroneous sequences
* **FASTQ to FASTA Conversion:** converting quality-controlled reads from FASTQ to FASTA format
* **Sequence Alignment with BLAST:** performning a BLASTN search against the NCBI NT nucleotide database to find similar sequences and  potential taxonomic identities of your eDNA sequences
* **Extracting Taxonomic Information:** applying the `Count` tool to extract and quantify the top unique taxonomic assignments from the BLAST output and can be used for further analysis and visualization

![Indetified Species Piecharts](images/HP3_R1_Species.png)
![Identified Species Piecharts](images/HP3_R2_Species.png)

By completing these steps, you have addressed the initial biological question of: **Can we assign taxonomic information to unknown DNA sequences from environmental samples?** This tutorial has demonstrated that, through the utilization of bioinformatics tools like BLAST, we can gain insights into the potential origins of DNA sequences source from non-invassive environmental samples.

You have also gained experience with several key bioinformatics techniques crucial for various types of DNA analysis: data quality control, sequence format conversion, sequence alignment, and basic data extraction. These skills have broad applications in biological research.

This tutorial can serve as a valuable starting point for further exploration into the fascinating field of environmental DNA analysis.

## Acknowledgments
I would like to thank both Dr. Gorden Ober and Dr. Jessica Kaufman for thier assitance and guidance during the course of the project that inspired this tutorial, as well as for their asssitance in sample collection and operation of the Illumina sequencer respectively.

tools_json:
["fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.24.1+galaxy0", "FASTQ to FASTA", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_to_fasta/cshl_fastq_to_fasta/1.0.2+galaxy2", "NCBI BLAST+ blastn", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/2.16.0+galaxy0", "Count", "Count1"]


topic: ecology
tutorial: SAM3
title: SAM3 – AI-based Semantic Segmentation of marine biodiversity Images and Videos
url: https://training.galaxyproject.org/training-material/topics/ecology/tutorials/SAM3/tutorial.html
difficulty: intermediate
time_estimation: 20M
requirements_json: []
tags_json: ["Marine ecosystems", "Biodiversity", "AI segmentation"]
content_hash: e0368e1e8746d8786c193e82a244c6de
last_modified: 2026-05-27T09:01:48.394552
workflows_json: []
content:
This tutorial will guide you through using the {% tool [SAM3](toolshed.g2.bx.psu.edu/repos/ecology/sam3_semantic_segmentation/sam3_semantic_segmentation/1.0.1+galaxy4) %} (Segment Anything Model 3) tool on Galaxy. SAM3 can automatically detect and segment objects in images or videos using text prompts, with no specific training required.

We will work through two concrete examples from the Moorev project:
1. A **photograph of a jellyfish** (*Pelagia noctiluca*)
2. A **video of shrimps**

> <agenda-title>In this tutorial, we will cover:</agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <details-title>Quick introduction to how Galaxy works</details-title>
>
> **Log in to Galaxy**
> 1. Open your preferred web browser 
> 2. Go to your Galaxy instance (please verify that the Galaxy instance you want to use propose SAM3 tool as Galaxy Europe is doing) 
> 3. Log in or create an account
>
> ![Screenshot of the Galaxy interface with the login button highlighted](../../images/SAM3/galaxy_homepage.png)
>
> This screenshot shows the Galaxy Ecology instance, available at [usegalaxy.eu](https://ecology.usegalaxy.eu/)
>
> The Galaxy home page is divided into 3 panels:
> * **Tools** on the left
> * The **visualisation panel** in the centre
> * The **history** of analyses and files on the right
>
> ![Screenshot of the Galaxy interface showing the three panels](../../images/champbloc/galaxy_interface.png)
>
> The first time you use Galaxy, there will be no files in your history panel.
{: .details}

# Loading data into Galaxy

Before running SAM3 Galaxy tool, you need to import the following files into Galaxy:  
- The jellyfish photo: `https://zenodo.org/records/19890809/files/Moorev-jellyfish.jpg`
- The shrimp video: `https://zenodo.org/records/19891364/files/2024-09-20-PorzBreign-shrimps.mp4`

{% snippet faqs/galaxy/datasets_import_via_link.md %}

> <warning-title>File format not recognised?</warning-title>
>
> If you want to try with other files, make sure the file extension is correct before uploading, as Galaxy may not detect it automatically. In that case, two options:
> - During upload, specify the format using the **Type (set all)** field
> - From the history, click the {% icon galaxy-pencil %} **pencil** icon, go to the **Datatype** tab and search for your extension
>
{: .warning}

# Segmenting an image: the jellyfish photograph

In this first section, we will run SAM3 Galaxy tool on the photo `Moorev-jellyfish.jpg` to detect and segment the jellyfish.  

> <tip-title>How to access the SAM3 tool?</tip-title>
>
> Type **SAM3** in the tool search bar at the top left, then click on the tool in the results.
>
> ![SAM3 tool search interface](../../images/SAM3/interface-tools.png "Search for SAM3 in Galaxy"){: style="width:50%; display:block; margin:auto;"}
>
{: .tip}

## Configuring SAM3 for the image

> <hands-on-title>Segment the jellyfish in the photo</hands-on-title>
>
> 1. {% tool [SAM3 Semantic Segmentation](toolshed.g2.bx.psu.edu/repos/ecology/sam3_semantic_segmentation/sam3_semantic_segmentation/1.0.1+galaxy4) %} with these parameters:
>
>    - {% icon param-file %} *"Model data"*: `Segment Anything Model 3 (SAM 3)` (default)
>    - {% icon param-select %} *"Input type"*: `One or more images` (default)
>    - {% icon param-file %} *"Input images"*: `Moorev-jellyfish.jpg`
>    - {% icon param-select %} *"Output formats"*: `COCO`
>    - {% icon param-text %} *"Text prompt"*: `jellyfish`
>    - {% icon version%} *"Confidence threshold"*: `0.5`
>    - {% icon version %} *"Video frame stride"*: `5` (default)
>    - {% icon param-toggle %} *"Show bounding boxes on annotated output"*: `Yes` (default)
>    - {% icon param-toggle %} *"Normalize outputs?"*: `No` (default)
>
>    > <tip-title>How to write a good prompt?</tip-title>
>    >
>    > The text prompt should describe the object to segment in **English**, using simple and precise terms.
>    > To detect multiple classes at once, separate them with commas:
>    > `jellyfish, shrimp, fish`
>    >
>    > Avoid overly vague descriptions like `animal` if you are specifically looking for a jellyfish.
>    > You can also use more descriptive prompt like `small blue fish`, but results may vary depending on the objects you want to detect.
>    {: .tip}
>
> 2. Click **Run Tool**
>
>    > <comment-title>Processing time</comment-title>
>    >
>    > Processing may take a few minutes depending on the image size and the resources available on the server. Wait until the outputs appear in green in the history.
>    >
>    {: .comment}
>
> 3. Once processing is complete, the following outputs appear in your history:
>    - **COCO Annotation**: the `annotations.json` file containing the segmentation masks
>    - **Annotated Outputs**: the collection of annotated images with overlaid masks
>
> 4. Viewing the annotated result
>
>    You should see the jellyfish outlined with a coloured mask and a bounding box.
>
>    > <tip-title>Display the image in the history</tip-title>
>    >
>    > Click on **Annotated Outputs** in the history panel:
>    > ![Click on Annotated Outputs](../../images/SAM3/history-annoted-outputs.png){: style="width:50%; display:block; margin:auto;"}
>    >
>    > Then use the {% icon galaxy-eye %} icon to display the image in the central panel:  
>    > ![Click the eye icon to display](../../images/SAM3/history-clic-eye.png){: style="width:50%; display:block; margin:auto;"}
>    >
>    > Or click {% icon galaxy-save %} to download the file directly.
>    {: .tip}
>
>    ![SAM3 segmentation mask on the jellyfish photo](../../images/SAM3/display_picture.png "Segmentation result"){: style="width:75%; display:block; margin:auto;"}
>
> 5. Exploring the COCO file
>    - Look at the content of your **COCO Annotation** file in your history  
>    - Use {% icon galaxy-eye %} to view the JSON, or {% icon galaxy-save %} to download it
>
>    The file contains the `images`, `annotations`, and `categories` fields. Each annotation includes:
>    - `segmentation`: the polygon coordinates of the mask
>    - `bbox`: the bounding box `[x, y, width, height]`
>    - `category_id`: the identifier of the detected class (`1` = `jellyfish`)
>
>    > <details-title>Export to YOLO format (optional)</details-title>
>    >
>    > If you need to train a YOLO model with your annotations, you can export results in YOLO format in addition to COCO.
>    > In the {% icon param-select %} *"Output formats"* parameter, select `COCO` **and/or** `YOLO segmentation masks` **and/or** `YOLO bounding boxes`.
>    >
>    > > <details-title>YOLO segmentation format</details-title>
>    > >
>    > > Each line in a YOLO segmentation label file follows this format:
>    > > ```
>    > > <class_id> <x1> <y1> <x2> <y2> ... <xn> <yn>
>    > > ```
>    > > Coordinates are normalised between 0 and 1 relative to the image dimensions.
>    > > Example for a jellyfish (class 0): `0 0.423 0.312 0.456 0.298 ...`
>    > >
>    > {: .details}
>    >
>    {: .details}
>
{: .hands_on}

# Segmenting a video: the shrimp video

In this second section, we will execute SAM3 tool to the video `2024-09-20-PorzBreign-shrimps.mp4`. SAM3 model analyses the video frame by frame, tracking the shrimps over time.  

## Configuring SAM3 for the video

> <hands-on-title>Segment the shrimps in the video</hands-on-title>
>
> 1. {% tool [SAM3 Semantic Segmentation](toolshed.g2.bx.psu.edu/repos/ecology/sam3_semantic_segmentation/sam3_semantic_segmentation/1.0.1+galaxy4) %} with these parameters:
>    - {% icon param-file %} *"Model data"*: `Segment Anything Model 3 (SAM 3)` (default)
>    - {% icon param-select %} *"Input type"*: `One video`
>    - {% icon param-file %} *"Input video file"*: `2024-09-20-PorzBreign-shrimps.mp4`
>    - {% icon param-select %} *"Video quality"*: `"2000k" = video bitrate 2000 kbps (480p~720p)`
>    - {% icon param-select %} *"COCO output mode"*: `Annotate the video — one COCO entry per frame, referencing the video file` (default)
>    - {% icon param-text %} *"Text prompt"*: `shrimp`
>    - {% icon version %} *"Confidence threshold"*: `0.25` (default)
>    - {% icon version %} *"Video frame stride"*: `5` (default)
>    - {% icon param-toggle %} *"Show bounding boxes on annotated output"*: `Yes` (default)
>    - {% icon param-toggle %} *"Normalize outputs?"*: `No` (default)
>
>    > <tip-title>Understanding the video parameters</tip-title>
>    >
>    > {% icon version %} *"Video frame stride"*: determines how often frames are analysed. A stride of `5` means one frame in every five is processed.
>    > - **Low stride (1–3)**: more precise analysis, but longer processing time
>    > - **High stride (10–30)**: faster processing, useful for long videos where objects move slowly
>    >
>    > {% icon param-select %} *"Video quality"*: controls the quality of the annotated output video, with no impact on processing speed or annotations.
>    >
>    > {% icon param-select %} *"COCO output mode"*: controls how COCO annotations are generated.
>    > - `Annotate the video`: one COCO entry per frame, referencing the video file (default)
>    > - `Annotate extracted frames`: saves frames as JPGs with one COCO entry per image — useful for pre-processing, for example with the {% tool [AnyLabeling Interactive](interactive_tool_anylabeling) %} tool as shown in the Moorev tutorial
>    >
>    {: .tip}
>
> 2. Click **Execute**
>
>    > <comment-title>Video processing time</comment-title>
>    >
>    > Video processing takes significantly longer than processing a single image. For a video of a few minutes, expect between 5 and 20 minutes depending on the server and the stride chosen.
>    >
>    {: .comment}
>
> 3. The following outputs appear in your history:
>    - **COCO Annotation**: the JSON file with annotations for each processed frame
>    - **Annotated Outputs**: the annotated video with segmentation masks overlaid frame by frame
>
> 4. Viewing the annotated video
>
>    - Click on **Annotated Outputs** in the history panel
>    - Click {% icon galaxy-eye %}
>    - Click {% icon galaxy-visualise %}
>    - Select **Media Player**
>
>    > <tip-title>Display the video in Galaxy</tip-title>
>    >
>    > ![Display the annotated video in Galaxy](../../images/SAM3/display_video.png){: style="width:75%; display:block; margin:auto;"}
>    >
>    > > <warning-title>Video not loading?</warning-title>
>    > >
>    > > The video may not load in Galaxy for several reasons:
>    > > - The file is too large for your internet connection
>    > > - The {% icon param-select %} *"Video quality"*: `Original quality (copy)` setting makes in-browser playback unavailable
>    > >
>    > > In that case, use {% icon galaxy-save %} to download the video and play it locally with your usual media player.  
>    > {: .warning}
>    >
>    {: .tip}
>
>    You will see the shrimps tracked with a coloured segmentation mask throughout the video.
>
>    ![SAM3 segmentation of shrimps](../../images/SAM3/2024-09-20-PorzBreign-shrimps.gif){: style="width:50%; display:block; margin:auto;"}
>
> 5. Downloading the annotated video
>    - Click the **Annotated Outputs** collection
>    - Use {% icon galaxy-save %} to download the `.mp4` video
>
>    > <comment-title>Limitations of SAM3 and pre-processing</comment-title>
>    >
>   > SAM3 tool is a first attempt to propose prompt-based  Galaxy tool. As it is using SAM3 model, you can have highly heterogenous results in term of quality depending on the objects you are searching to segment, notably if such kind of object can be on data used to train the SAM3 model. Adjusting the confidence threshold can help, but it does not solve everything. Pre-processing your images or videos is often necessary to improve results.  
>    > To learn more, check out the dedicated tutorial: **Tuto Moorev**
>    >
>    {: .comment}
>
{: .hands_on}

# Conclusion

You now know how to use SAM3 Galaxy tool to:
- Segment objects in an **image** using a simple text prompt
- Segment objects in a **video** frame by frame with temporal tracking
- Export results in **COCO** format (for annotation and evaluation tools) or **YOLO** format (for model training)

tools_json:
["SAM3", "toolshed.g2.bx.psu.edu/repos/ecology/sam3_semantic_segmentation/sam3_semantic_segmentation/1.0.1+galaxy4", "SAM3 Semantic Segmentation", "AnyLabeling Interactive", "interactive_tool_anylabeling"]


topic: ecology
tutorial: obisindicators
title: Obis marine indicators
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["earth-system", "ocean", "marine omics", "biodiversity"]
content_hash: 8bc32d8cd44092468d41c967feeb452d
last_modified: 2026-05-27T09:01:48.400402
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/ecology/tutorials/obisindicators/tutorial.html

content:
OBIS is a global open-access data and information clearing-house on marine biodiversity for science, conservation and sustainable development. In order to visualize their marine data OBIS created the package obisindicators.

Obisindicators is an R library developed during the 2022 IOOS Code Sprint. The purpose was to create an ES50 diversity index within hexagonal grids following the diversity indicators notebook by Pieter Provoost linked above. The package includes several examples, limited to 1M occurrences, that demonstrate uses of the package.
This tutorial will guide you on getting obis marine data and processing them in order to calculate and visualize multiple indicators. 

This tool for obisindicators is composed of 5 indicators : Number of record, Shannon, Simpson, Es50 and Hill which will be explained in more details later on.

In this tutorial, highly based on [OBIS indicators documentation](https://obis.org/indicators/documentation/) we'll be working on OBIS data extracted from [OBIS mapper](https://mapper.obis.org/). First those data will be prepared. 


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Upload of the data

This first step consist of downloading and uploading obis data onto galaxy.

> <hands-on-title>Data upload</hands-on-title>
> 
> Create a new history for this tutorial and give it a name (example: “Obisindicators tutorial”) for you to find it again later if needed.
> 
> {% snippet faqs/galaxy/histories_create_new.md %}
{: .hands_on}

{% include _includes/cyoa-choices.html option1="Tool" option2="Portal" default="Tool" text="You can use a tool to download the data you want to study or you can download them from the Obis portal." %}

<div class="Tool" markdown="1">
## Use the galaxy tool
> <hands-on-title>Download your data</hands-on-title>
> 
> 1. {% tool [OBIS occurences](toolshed.g2.bx.psu.edu/repos/ecology/obis_data/obis_data/0.0.2) %} with the following parameters:
>    - *"Scientific name of the species"*: Empty
>    - *"Taxon ID"*: Empty
>    - *"Input latitude min (+north/-south)"*: `41.5`
>    - *"Input latitude max (+north/-south)"*: `45.0`
>    - *"Input longitude min (+east/-west)"*: `7.5`
>    - *"Input longitude max (+east/-west)"*: `10.0`
> 2. **Run Tool**
{: .hands_on}

## Clean data Advanced Cut

> <hands-on-title>Clean your data</hands-on-title>
> 
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %} with the following parameters:
>    -*"File to cut"*: `Species occurences`
>    - *"Cut by"*: `fields`
>        - *"List of Fields"*: `basisOfRecord` `decimalLatitude` `decimalLongitude` `species` `individualCount`
> You should always have at least the 4 folowing columns: decimalLatitude, decimalLongitude, species and individualCount
{: .hands_on}
</div>

<div class="Portal" markdown="1">
## Get to the Portal
1. Download the files from [Obis](https://mapper.obis.org/) 

![Obis portal welcome page](../../images/obisindicators/portal.png "Obis portal")
> <details-title>Download data from the obis portal</details-title>
> * Go on the right panel and enter the criteria of your choise here in "Area" write down **Mediterranean** and select "France: Mediterranean Sea".
> * Click on save on the top right
> ![Guide to download obis data](../../images/obisindicators/download.png "Obis download")
> * Then on the 3 green lines top right press download
> * Enter your email on the pop-up screen and press **Yes, procced**
> ![The obis portal while you wait for the download](../../images/obisindicators/waiting.png "Waiting download")
> * The download can take a while depending on the size of your dataset (here less than 15min) 
> * Then click on **Download ZIP file**
> * Don't forget to unzip your file on your machine.
{: .details}

In the downloaded folder you should have your data either csv format (Occurence.csv) and you must have at least 4 columns containing: latitude, longitude, species and record.

2. Upload obis data

> <tip-title>Importing data from your computer</tip-title>
> 
> * Open the Galaxy Upload Manager {% icon galaxy-upload %}
> * Select **Choose local files**
> * Browse in your computer and get the downloaded zip folder
> 
> * Press **Start** (it can take a few seconds to get ready)
{: .tip}

3. Rename the datasets “obis data" for example and preview your dataset

4. Check the datatype must be csv or tabular

{% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}

## Convert data csv-to-tabular

> <hands-on-title>Convert your data</hands-on-title>
> 
> 1. On your data in your history pannel click on {% icon param-text %}
> 2. In the top click on {% icon galaxy-gear %} Convert
> 3. Press {% icon exchange %} Create Dataset
> 
>    
{: .hands_on}

## Clean data Advanced Cut

> <hands-on-title>Clean your data</hands-on-title>
> 
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy0) %} with the following parameters:
>    -*"File to cut"*: `obis data`
>    - *"Cut by"*: `fields`
>      - *"Delimited by"*: `Tab`
>      -*"Is there a header for the data's columns ?"*: `Yes`
>        - *"List of Fields"*: `basisOfRecord` `decimalLatitude` `decimalLongitude` `species` `individualCount`
> You should always have at least the 4 folowing columns: decimalLatitude, decimalLongitude, species and individualCount
{: .hands_on}
</div>

You are now all set to use your obis data in order to do a diversity analysis. 

## Ocean biodiversity indicators

> <hands-on-title>Ocean biodiversity indicators</hands-on-title>
>
> 1. {% tool [Ocean biodiversity indicators](toolshed.g2.bx.psu.edu/repos/ecology/obisindicators/obisindicators/0.0.1) %} with the following parameters:
>    - *"Input table"*: The result tabular of the Advanced Cut tool
>    - *"What character is the separator in your data? (Mostlikely a comma for a csv file and t for a tabular)"*: `Tabulator (\t)`
>    - *"Select column containing the decimal value of the longitude "*: `c3`
>    - *"Select column containing the decimal value of the latitude "*: `c2`
>    - *"Select column containing the species "*: `c4`
>    - *"Select column containing the number of records"*: `c5`
>    - *"Type of projection for the map : select your coordinate reference system (CRS)"*: Robinson Projection
>    - *"Choose a resolution for the discrete global grid"*: 9
>
> Note that you can rerun the tool and modify the resolution of the maps you want to create.
>
> 2. Click on **Execute** 
> 3. You will see 5 outputs appear on the history pannel. one for each of the indicators
>
{: .hands_on}

## Number of records

![Map of the number of records](../../images/obisindicators/records.png "Records map")
In the table index the records correspond to n.

## Shannon

The Shannon index expresses the uncertainty associated with the prediction of the species the next sampled individual belongs to. It assumes that individuals are randomly sampled from an infinitely large community, and that all species are represented in the sample.

Warning: OBIS uses records as a proxy for individuals and sampling is generally not random, the community is not infinitely large and not all species are represented in the sample.

The Shannon diversity index, also known as the Shannon-Wiener diversity index, is defined in OBIS as the sum over all species of $-fi*log(fi)$ with fi defined as $n/ni$ with n as the total number of records in the raster cell and ni as the total number of records for the ith-species in the raster cell.
![Map showing the shannon indice](../../images/obisindicators/shannon.png "Shannon map")


## Simpson
The measure equals the probability that two entities taken at random from the dataset of interest represent the same type. It equals:

where is richness (the total number of types in the dataset) and is the proportional abundances of the types of interest.

Simpson’s index expresses the probability that any two individuals drawn at random from an infinitely large community belong to the same species. Note that small values are obtained in cells of high diversity and large values in cells of low diversity. This counterintuitive behavior is adressed with the Hill 2 number, which is the inverse of the Simpson index.

The Simpson biodiversity index is defined in OBIS as the sum over all species of $(ni/n)^2$ with n as the total number of records in the cell and ni the total number of records for the ith species.

Warning: The Simpson index has the same assumptions as the Shannon index.
![Map showing the simpson indice](../../images/obisindicators/simpson.png "Simpson map")

## ES50

The expected number of marine species in a random sample of 50 individuals (records) is an indicator on marine biodiversity richness.

The ES50 is defined in OBIS as the $$ sum(esi) $$ over all species of the following per species calculation:

    when $$ n - ni$ <= 50 $$ (with n as the total number of records in the cell and ni the total number of records for the ith-species)
        $$ esi = 1 - exp(lngamma(n-ni+1) + lngamma(n-50+1) - lngamma(n-ni-50+1) - lngamma(n+1)) $$
    when $$ n >= 50 $$
        $$ esi = 1 $$
    else
        $$ esi = NULL $$

Warning: ES50 assumes that individuals are randomly distributed, the sample size is sufficiently large, the samples are taxonomically similar, and that all of the samples have been taken in the same manner.
![Map showing the ES50 indice](../../images/obisindicators/es50.png "ES50 map")


## Maxp

Maxp is the maximum of the total number of records for the ith-species ni divided by the total number of records in the cell n, ie, $$ Maxp = max(ni / n) $$.
![Map showing the Maxp indice](../../images/obisindicators/maxp.png "Maxp map")

## Hill

### Hill 1

The Hill biodiversity index accounts for species’ relative abundance (number of records in OBIS) and Hill1 can be roughly interpreted as the number of species with “typical” abundances, and is a commonly used indicator for marine biodiversity richness. It is defined as:

Warning: The Simpson index has the same assumptions as the Shannon index.

### Hill 2

The Hill biodiversity index accounts for species’ relative abundance (number of records in OBIS) and discounts rare species, so Hill2 can be interpreted as the equivalent to the number of more dominant species and so is less sensitive to sample size than Hill1. The Hill index is a commonly used indicator for marine biodiversity richness. It is defined as:

Warning: The Simpson index has the same assumptions as the Shannon index.

They are calculated as shown below:
     - $$ hill_1   = exp(shannon) $$
     - $$ hill_2   = 1 / simpson $$
     - $$ hill_inf = 1 / maxp $$

## Index file

You also have a tabular file that sums up each indicators.
![Summary tabular of all the indices](../../images/obisindicators/index.png "Tabular")
Nb: the column sp is the count of the number of observations in a dataset. It is  the number of records in the dataset.

# Conclusion

You here learn how to select and download OBIS dataset for your region of interest, to handle data to finally compute diversity indicators and display it in maps.

tools_json:
["OBIS occurences", "toolshed.g2.bx.psu.edu/repos/ecology/obis_data/obis_data/0.0.2", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy0", "Ocean biodiversity indicators", "toolshed.g2.bx.psu.edu/repos/ecology/obisindicators/obisindicators/0.0.1"]


