# tutorials

topic: statistics
tutorial: flexynesis_survival
title: Identifing Survival Markers of Brain tumor with Flexynesis
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: 730e94de954629c1258d48c8d6278cf8
last_modified: 2026-05-27T09:01:49.827962
zenodo_link: https://zenodo.org/records/16287482
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/flexynesis_survival/tutorial.html

content:
Here, we use Flexynesis tool suite on a multi-omics dataset of 506 Brain Lower Grade Glioma (LGG) and 288 Glioblastoma Multiforme (GBM) samples with matching mutation and copy number alteration. This data were downloaded from the [cBioPortal](https://www.cbioportal.org) ({% cite cbioportal-website %}). The data was split into train (70% of the samples) and test (30% of the samples).

We will work on CNA (copy number alteration) and MUT (mutation data which is a binary matrix of genexsample) data.

This training is inspired from the original flexynesis analysis notebook: [survival_subtypes_LGG_GBM.ipynb](https://github.com/BIMSBbioinfo/flexynesis/blob/main/examples/tutorials/survival_subtypes_LGG_GBM.ipynb).

> <comment-title> Getting data from cBioPortal </comment-title>
>
> If you want to download data directly from cBioPortal, you can go through "**Prepare data from CbioPortal for Flexynesis integration**" training
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <warning-title>LICENSE</warning-title>
>
>Flexynesis is only available for NON-COMMERCIAL use. Permission is only granted for academic, research, and educational purposes. Before using, be sure to review, agree, and comply with the license.
>For commercial use, please review the flexynesis license on GitHub and contact the [copyright holders](https://github.com/BIMSBbioinfo/flexynesis)
{: .warning}

# Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/records/16287482/files/train_mut_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/train_clin_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/test_cna_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/test_mut_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/test_clin_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/train_cna_lgggbm.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each database a tag corresponding to their modality (#mut, #cna, and #clinical)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Train Flexynesis model with a survival task

Now it is time import train and test datasets. We rank genes by Laplacian Scores and pick top 10% of the genes, while removing highly redundant genes with a correlation score threshold of 0.8 and a variance threshold of 50%. We will also use the intermediate fusion of omic layers.
For hyperparameter optimization we use the following parameters.


*`Model class`: We pick DirectPred (a fully connected network)
* `Column name in the train clinical data to use as survival event` and `Column name in the train clinical data to use as survival time`. Survival status (consists of 0s and 1s) and time since last followup. It is **important** that the clinical data contains both variables as ***numerical*** values.
* `Column name in the train clinical data to use for predictions, multiple targets are allowed`: We can concurrently train the same network to be able to predict other variables such as histological diagnosis, however, here we just focus on the survival endpoints, so we pass an empty list.
* `Number of iterations for hyperparameter optimization.`: We do 5 iterations of hyperparameter optimization. This is a reasonable number of demonstration purposes, but it could be beneficial to increase this value in order to discover even better models.
* `How many epochs to wait when no improvements in validation loss are observed. `: If a training does not show any signs of improving the performance on the validation part of the train_dataset for at least 10 epochs, we stop the training. This not only significantly decreases the amount spent on training by avoiding unnecessary continuation of unpromising training runs, but also helps avoid over-fitting the network on the training data.


> <hands-on-title> Survival task </hands-on-title>
>
> 1. {% tool [Flexynesis](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Type of Analysis"*: `Supervised training`
>        - {% icon param-file %} *"Training clinical data"*: `train_clin_lgggbm.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_clin_lgggbm.tabular`
>        - {% icon param-file %} *"Training omics data"*: `train_mut_lgggbm.tabular`
>        - {% icon param-file %} *"Test omics data"*: `test_mut_lgggbm.tabular`
>        - *"What type of assay is your input?"*: `mut`
>        - In *"Multiple omics layers?"*:
>            - {% icon param-repeat %} *"Insert Multiple omics layers?"*
>                - {% icon param-file %} *"Training omics data"*: `train_cna_lgggbm.tabular`
>                - {% icon param-file %} *"Test omics data"*: `test_cna_lgggbm.tabular`
>                - *"What type of assay is your input?"*: `cna`
>        - *"Model class"*: `DirectPred`
>        - *"Column name in the train clinical data to use as survival event"*: `c8`
>        - *"Column name in the train clinical data to use as survival time"*: `c7`
>        - In *"Advanced Options"*:
>            - *"Fusion method"*: `intermediate`
>            - *"Variance threshold (as percentile) to drop low variance features."*: `0.5`
>            - *"Correlation threshold to drop highly redundant features."*: `0.8`
>            - *"Minimum number of features to retain after feature selection."*: `1000`
>            - *"Top percentile features (among the features remaining after variance filtering and data cleanup) to retain after feature selection."*: `10.0`
>            - *"Number of iterations for hyperparameter optimization."*: `5`
>        - In *"Visualization"*:
>            - *"Generate embeddings plot?"*: `No`
>            - *"Generate kaplan meier curves plot?"*: `Yes`
>            - *"Generate hazard ratio plot?"*: `Yes`
>                - *"Omics layer to use for cox input"*: `mut`
>                - *"Number of top important features to include in Cox model"*: `5`
>                - *"Performs K-fold cross-validation?"*: `No`
>            - *"Generate scatter plot?"*: `No`
>            - *"Generate concordance heatmap plot?"*: `No`
>            - *"Generate precision-recall curves plot?"*: `No`
>            - *"Generate ROC curves plot?"*: `No`
>            - *"Generate boxplot?"*: `No`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What are main outputs of Flexynesis?
> 2. What does the generated plots show?
> 3. What information is in `job.stats` output?
> 4. What are top 10 features in `job.feature_importance.GradientShap`?
>
> > <solution-title></solution-title>
> >
> > 1. The results collection contains the following data:
> > * job.embeddings_test (latent space of test data)
> > * job.embeddings_train (latent space of train data)
> > * job.feature_importance.GradientShap (feature importance calculated by GradientShap method)
> > * job.feature_importance.IntegratedGradients (feature importance calculated by IntegratedGradients method)
> > * job.feature_logs.cna
> > * job.feature_logs.mut
> > * job.predicted_labels (Prediction of the created model)
> > * job.stats
> >
> > 2. There are two figures in plot collection:
> > * The first plot is the Kaplan Meier Curves of the risk subtypes.
> > * The second plot is a forest plot of calculated Cox-PH model of top 5 markers.
> >
> > 3. It shows that we achieved a reasonable performance of Harrell's Concordance Index (C-index) on test data.
> >
> > 4. *IDH2*, *IDH1*, *HIVEP3*, *PIK3CA*, *EGFR*, *RELN*, *ATRX*, *INSRR*, *TP53*, *COL6A3*. *IDH1*, *TP53*, and *ATRX* are extensively studied and been shown to be relevant in gliomas ({% cite Koschmann2016 %}).
> >
> {: .solution}
>
{: .question}

We can also visualize the high and low risk groups with the embeddings.

# Survival-risk subtypes

## Extract predicted labels and calculate median

Let's group the samples by predicted survival risk scores into 2 groups and visualize the sample embeddings colored by risk subtypes.

> <hands-on-title> Calculate median of predicted survival score</hands-on-title>
>
> First we should extract the `job.predicted_labels` and then calculate the median.
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.predicted_labels`
>
> 2. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `job.predicted_labels` (output of **Extract dataset** {% icon tool %})
>    - *"Input file has a header line"*: `Yes`
>    - In *"Operation to perform on each group"*:
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Median`
>            - *"On column"*: `Column: 6` (*"predicted_label)
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the median?
>
> > <solution-title></solution-title>
> >
> > 1. -0.66496020555496
> >
> {: .solution}
>
{: .question}

Now we add a column to our `job.predicted_labels` data indicating high and low risk groups (samples with predicted_label > median are high risk groups)

## Assign high and low risk groups

> <hands-on-title> Assign high/low groups </hands-on-title>
>
> 1. {% tool [Compute](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `job.predicted_labels` (output of **Extract dataset** {% icon tool %})
>    - *"Input has a header line with column names?"*: `Yes`
>        - In *"Expressions"*:
>            - {% icon param-repeat %} *"Insert Expressions"*
>                - *"Add expression"*: `float(c6)>=-0.66496020555496`
>                - *"The new column name"*: `risk_groups`
>    - In *"Error handling"*:
>        - *"Autodetect column types"*: `No`
>
>
{: .hands_on}

For better visualization, let's change True to High_risk and False to Low_risk

> <hands-on-title> Rename labels </hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `table` (output of **Compute** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `Column: 9`
>            - *"Find pattern"*: `True`
>            - *"Replace with"*: `High_risk`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `Column: 9`
>            - *"Find pattern"*: `False`
>            - *"Replace with"*: `Low_risk`
>
{: .hands_on}

Now we visualize the embedding.

> <hands-on-title> Dimension reduction plot </hands-on-title>
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.embeddings_test`
>
> 2. {% tool [Flexynesis plot](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis plot"*: `Dimensionality reduction`
>        - {% icon param-file %} *"Predicted labels"*: `table` (output of **Replace Text** {% icon tool %})
>        - {% icon param-file %} *"Embeddings"*: `job.embeddings_test` (output of **Extract dataset** {% icon tool %})
>        - *"Column in the labels file to use for coloring the points in the plot"*: `Column: 9`
>
{: .hands_on}

# Conclusion

In this tutorial, we have successfully applied the Flexynesis framework to analyze survival markers in brain tumors using multi-modal genomic data from 506 Lower Grade Glioma (LGG) and 288 Glioblastoma Multiforme (GBM) samples. Through hands-on analysis of mutation and copy number alteration data, we have separated our samples to high and low risk groups and ranked the genes by their importance from which some are already known to affect survival in glioma.

tools_json:
["Flexynesis", "toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3", "Extract dataset", "__EXTRACT_DATASET__", "Datamash", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.9+galaxy0", "Compute", "toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2", "Flexynesis plot", "toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3"]


topic: statistics
tutorial: fruit_360
title: Image classification in Galaxy with fruit 360 dataset
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: 12673ed38f2b5dac95f6888ce165037d
last_modified: 2026-05-15T22:28:17.019978
zenodo_link: https://zenodo.org/record/5702887
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/fruit_360/tutorial.html

content:
The classification of fruits and vegetables offers many useful applications such as
automated harvesting by robots, building up stocks for supermarkets, effective detection
of specific defects, and determining fruit ripeness ({% cite Duong2020 %},
{% cite NaranjoTorres2020 %}, {% cite Iswari2017 %}). Machine Learning (ML) techniques
such as Deep Learning (DL) are commonly used for image classification problems in various
domains, including in agriculture ({% cite Kamilaris2018 %}). DL is a technique inspired
by how a human brain operates. Due to the increased availability of compute capacity and
training data, DL techniques have become very popular in recent years. In this tutorial,
we will use Galaxy's ML toolkit to build a DL model to classify fruit and vegetable
images. Our DL model is trained and evaluated on Fruit 360 dataset ({% cite Murean2018 %})

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}
## Overview of convolutional neural networks (CNN)

Here we provide a brief overview of CNN. For a more in depth discussion, please refer to the CNN tutorial cited in the requirements
section. CNN were inspired by how the visual cortex of the brain processes visual information ({% cite HubelWiesel %}). There are
two types of cells in our visual cortex: **simple cells** detect objects at certain angles/locations, and **complex** cells,
which receive inputs from multiple simple cells, and detect movement. In 1980, inspired by hierarchical structure of complex and
simple cells, Fukushima proposed *Neocognitron* ({% cite Fukishima %}), a hierarchical neural network used for handwritten Japanese
character recognition. In 1989, LeCun et. al. ({% cite LeCunEtAl %}) proposed a CNN that could be trained by backpropagation
algorithm. CNN gained immense popularity when they outperformed other models at ImageNet Challenge, a competition in object
classification and detection on hundreds of object categories and millions of images.

## Architecture of CNN

A typical CNN has the following 4 layers ({% cite OSheaEtAl %})

1. Input layer
2. Convolution layer
3. Pooling layer
4. Fully connected layer

Please note that we will explain a 2 dimensional (2D) CNN here. But the same concepts apply to a 1 (or 3) dimensional CNN as well.

### Input layer

The input layer represents the input to the CNN. An example input, could be a 28 by 28 pixel grayscale image. We do not
"flatten" the input to a 1D vector. This makes capturing spatial relationships easier.

### Convolution layer

The convolution layer is composed of multiple **filters** (also called **kernels**). Filters for a 2D image are also 2D. Suppose
we have a 28 by 28 pixel grayscale image. Each pixel is represented by a number between 0 and 255, where 0 represents the color
black, 255 represents the color white, and the values in between represent different shades of gray. Suppose we have a 3 by 3
filter (9 values in total), and the values are randomly set to 0 or 1. Convolution is the process of placing the 3 by 3 filter
on the top left corner of the image, multiplying filter values by the pixel values and adding the results, moving the filter to
the right one pixel at a time and repeating this process (Figure 1). When we get to the top right corner of the image, we simply
move the filter down one pixel and restart from the left. This process ends when we get to the bottom right corner of the image.

![A 3 by 3 filter applied to a 4 by 4 image, resulting in a 2 by 2 image](../../images/Conv_no_padding_no_strides.gif "A 3 by 3 filter applied to a 4 by 4 image, resulting in a 2 by 2 image ({% cite DumoulinVisin %})")

Covolution operator has several parameters.

1. Filter size
2. Padding
3. Stride
4. Dilation
5. Activation function

Filter size can be 5 by 5, 3 by 3, and so on. Larger filter sizes should be avoided as more weights need to be learned (more
compute capacity, more training time, more chance of overfitting). Also, odd sized filters are preferred to even sized filters,
due to the nice geometric property of all the input pixels being around the output pixel.

If you look at Figure 1 you see that after applying a 3 by 3 filter to a 4 by 4 image, we end up with a 2 by 2 image -- the
size of the image has gone down. If we want to keep the image size the same, we can use *padding* (Figure 2). We pad the input
in every direction with 0's before applying the filter. If the padding is 1 by 1, then we add 1 zero in evey direction. If its
2 by 2, then we add 2 zeros in every direction, and so on.

![A 3 by 3 filter applied to a 5 by 5 image, with padding of 1, resulting in a 5 by 5 image](../../images/Conv_same_padding_no_strides.gif "A 3 by 3 filter applied to a 5 by 5 image, with padding of 1, resulting in a 5 by 5 image ({% cite DumoulinVisin %})")

As mentioned before, we start the convolution by placing the filter on the top left corner of the image, and after multiplying
filter and image values (and adding them), we move the filter to the right and repeat the process. How many pixels we move to
the right (or down) is the *stride*. In figure 1 and 2, the stride of the filter is 1. We move the filter one pixel to the right
(or down). But we could use a different stride. Figure 3 shows an example of using stride of 2.

![A 3 by 3 filter applied to a 5 by 5 image, with stride of 2, resulting in a 2 by 2 image](../../images/Conv_no_padding_strides.gif "A 3 by 3 filter applied to a 5 by 5 image, with stride of 2, resulting in a 2 by 2 image ({% cite DumoulinVisin %})")

When we apply a, say 3 by 3, filter to an image, our filter's output is affected by pixels in a 3 by 3 subset of the image. If we
like to have a larger *receptive field* (portion of image that affect filter's output), we could use *dilation*. If we set the
dilation to 2 (Figure 4), instead of a contiguous 3 by 3 subset of the image, every other pixel of a 5 by 5 subset of the image
affects the filter's output.

![A 3 by 3 filter applied to a 7 by 7 image, with dilation of 2, resulting in a 3 by 3 image](../../images/Conv_dilation.gif "A 3 by 3 filter applied to a 7 by 7 image, with dilation of 2, resulting in a 3 by 3 image ({% cite DumoulinVisin %})")

After the filter scans the whole image, we apply an activation function to filter output to introduce non-linearlity. The preferred
activation function used in CNN is ReLU ({% cite NwankpaEtAl %}). ReLU leaves pixels with positive values in filter output as is,
and replaces negative values with 0. Figure 5 shows the results of applying ReLU activation function to a filter output.

![Two matrices representing filter output before and after ReLU activation function is applied](../../images/Conv_ReLU.png "Applying ReLU activation function to filter output")

![One matrix representing an input and another matrix representing a filter, along with calculation for single input channel two dimensional convolution operation](../../images/Conv_single_input_channel.png "Illustration of single input channel two dimensional convolution")

Figure 6 illustrates the calculations for a convolution operation, via a 3 by 3 filter on a single channel 5 by 5 input
(5 x 5 x 1). Figure 7 illustrates the calculations when the input has 3 channels. To show this in 2 dimensions, we are
displaying each channel in input and filter separately. Figure 9 shows a sample multi-channel 2D convolution in 3 dimensions.

![Three matrices representing an input and another three matrices representing a filter, along with calculation for multiple input channel two dimensional convolution operation ](../../images/Conv_multiple_input_channel.png "Illustration of multiple input channel two dimensional convolution")

As Figures 7 and 8 show the output of a multi-channel 2D filter is a single channel 2D image. Applying *multiple* filters to the
input image results in a multi-channel 2D image for the output. For example, if the input image is 28 by 28 by 3
(rows x columns x channels), and we apply a 3 by 3 filter with 1 by 1 padding, we would get a 28 by 28 by 1 image. If we apply 15
filters to the input image, our output would be 28 by 28 by 15. Hence, the number of filters in a convolution layer allows us to
increase or decrease the channel size.

![Multiple cubes representing input, filter, and output in a 3 channel 2 dimensional convolution operation](../../images/Conv_multiple_channel_3d.gif "Three dimensional illustration of multiple input channel two dimensional convolution (Source: https://thomelane.github.io/convolutions/2DConvRGB.html)")

### Pooling layer

The pooling layer performs down sampling to reduce the spatial dimensionality of the input. This decreases the number of parameters,
which in turn reduces the learning time and computation, and the likelihood of overfitting. The most popular type of pooling is
*max pooling*. Its usually a 2 by 2 filter with a stride of 2 that returns the maximum value as it slides over the input data,
similar to convolution filters.

### Fully connected layer

The last layer in a CNN is a fully connected layer. We connect all the nodes from the previous layer to this fully connected layer,
which is responsible for classification of the image.

![A convolutional neural network with 3 convolution layers followed by 3 pooling layers](../../images/Conv_CNN.png "A convolutional neural network with 3 convolution layers followed by 3 pooling layers ({% cite OSheaEtAl %})")

As shown in Figure 9, a typical CNN usually has more than one convolution plus pooling layer. Each convolution plus pooling layer
is responsible for feature extraction at a different level of abstraction. For example, the filters in the first layer could detect
horizontal, vertical, and diagonal edges. The filters in the next layer could detect shapes, and the filters in the last layer could
detect collection of shapes. Filter values are randomly initialized and are learned by the learning algorithm. This makes CNN very
powerful as they not only do classification, but can also automatically do feature extraction. This distinguishes CNN from other classification techniques (like Support Vector Machines), which cannot do feature extraction.

## Fruit 360 dataset

Fruit 360 is a dataset with 90380 images of 131 fruits and vegetables
(https://www.kaggle.com/moltean/fruits). Images are 100 pixel by 100 pixel and are color
(RGB) images (Hence, 3 values for each pixel). There are 67,692 images in the training
dataset and 22,688 images in the test dataset. The dataset we use for this tutorial is a
subset of fruit 360 dataset, containing only 10 fruits/vegetables (Strawberry,
Apple_Red_Delicious, Pepper_Green, Corn, Banana, Tomato_1, Potato_White, Pineapple,
Orange, and Peach). We selected a subset of fruits/vegetables, so the dataset size is
smaller and the neural network can be trained faster. Our training dataset has 5,015 images
and our testing dataset has 1,679 images.

The utilities used to create the subset dataset, along with step by step instructions, can
be found here: https://github.com/kxk302/fruit_dataset_utilities. First, we created feature
vectors for each image. Images are 100 pixel by 100 pixel and are color (RGB) images
(3 values for each pixel). Hence, each image can be represented by 30,000 values
(100 X 100 X 3). Second, we selected a subset of 10 fruit/vegetable images. Training and
testing dataset sizes go from 7 GB and 2.5 GB for 131 fruits/vegetables to 500 MB and
177 MB for 10 fruits/vegetables, respectively. Third, we created separate files for feature
vectors and labels. Finally, we mapped the labels for the 10 selected fruits/vegetables to
a range of 0 to 9. Full dataset labels are in the 0 to 130 range, as the full dataset
includes 131 fruits/vegetables. The 10 labels for out dataset are as follows: Strawberry:0,
Apple_Red_Delicious:1, Pepper_Green:2, Corn:3, Banana:4, Tomato_1:5, Potato_White:6, Pineapple:7,
Orange:8, Peach:9.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Make sure you have an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to make it easy to recognize
>
>    > <tip-title>Rename a history</tip-title>
>    >
>    > * Click on the title of the history (by default the title is `Unnamed history`)
>    >
>    >   ![Renaming history]({% link shared/images/rename_history.png %})
>    >
>    > * Type `Galaxy Introduction` as the name
>    > * Press <kbd>Enter</kbd>
>    >
>    {: .tip}
>
>
> 3. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/train_X_10.tsv
>    {{ page.zenodo_link }}/files/train_y_10.tsv
>    {{ page.zenodo_link }}/files/test_X_10.tsv
>    {{ page.zenodo_link }}/files/test_y_10.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Rename the datasets as `train_X_10`, `train_y_10`, `test_X_10`, and `test_y_10` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Check that the datatype of all the three datasets is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

## Classification of fruit 360 dataset images with CNN

In this section, we define a CNN and train it using fruit 360 dataset training data. The
goal is to learn a model such that given an image of a fruit/vegetable, we can predict
what fruit/vegetable it is (Labels are in the range of 0 to 9). We then evaluate the trained
CNN on the test dataset and plot the confusion matrix.

In order to train the CNN, we must have the One-Hot Encoding (OHE) representation of the training
labels. This is needed to calculate the categorical cross entropy loss function. OHE encodes labels
as a **one-hot** numeric array, where only one element is 1 and the rest are 0's. For example, if
we had 3 fruits (apple, orange, banana) and their labels were 1, 2, and 3, the OHE
represntation of apple would be (1,0,0), the OHE representation of orange would be (0,1,0), and the
OHE representation of banana would be (0,0,1). For apple with label 1, the first element of array
is 1 (and the rest are 0's); For Orange with label 2, the second element of the array is 1 (and the
rest are 0's); And for Banana with label 3, the third element of the array is 1 (and the rest are 0's).
We have 10 fruits/vegetables in our dataset and we would just have an array of size 10, where only one
element is 1, corresponding to fruit/vegetable label, and the rest are 0's.


In order to calculate the OHE of labels, we must first extract the labels column from train_y_10 file.
train_y_10 file has 3 columns: Label_name (string representation of label), file_name (name of fruit/vegetable
image file), and Label (integer representation of label). We extract Label from train_y_10, and then calcuate
its OHE representation.

### Extract the Label column from train_y_10

> <hands-on-title>Advanced Cut</hands-on-title>
>
> - {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %}
>    - *"File to cut"* : Select `train_y_10`
>    - *"Operation"* : Select `Keep`
>    - *"Delimited by"*: Select `Tab`
>    - *"Cut by"*: Select `fields`
>    - *"List of fields"*: Select `Column: 3`
>    - Click *"Run Tool"*
>
{: .hands_on}

### Create One-Hot Encoding (OHE) representation of training labels

> <hands-on-title>One-Hot Encoding</hands-on-title>
>
> - {% tool [To categorical](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_to_categorical/sklearn_to_categorical/1.0.8.3) %}
>    - *"Input file"* : Select the output of the previous step.
>    - *"Does the dataset contain header?"* : Select `Yes`
>    - *"Total number of classes"*: Select `10`
>    - Click *"Run Tool"*
>
{: .hands_on}

### Create a deep learning model architecture

> <hands-on-title>Model config</hands-on-title>
>
> - {% tool [Create a deep learning model architecture](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/0.5.0) %}
>    - *"Select keras model type"*: `sequential`
>    - *"input_shape"*: `(30000,)`
>    - In *"LAYER"*:
>        - {% icon param-repeat %} *"1: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Reshape`
>                - *"target_shape"*: `(100,100,3)`
>        - {% icon param-repeat %} *"2: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `16`
>                - *"kernel_size"*: `5`
>                - *"Activation function"*: `relu`
>                - *"Type in key words arguments if different from the default"*: `input_shape=(100, 100, 3)`
>        - {% icon param-repeat %} *"3: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"4: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `32`
>                - *"kernel_size"*: `5`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"5: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"6: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `64`
>                - *"kernel_size"*: `5`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"7: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"8: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Flatten`
>        - {% icon param-repeat %} *"9: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `256`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"10: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `10`
>                - *"Activation function"*: `softmax`
>    - Click *"Run Tool"*
{: .hands_on}

Each image is passed in as a vector of size 30,000 (100 x 100 X 3 = 30,000). The reshape
layer reshapes it into (100, 100, 3) dimensions -- 100 rows (image height), 100 columns
(image width), and 3 channels. Channel size is 3 since the image is color (RGB) and each
color pixel can be represented by 3 integers, representing the Red, Green, and Blue
primary colors. Our CNN then has 3 convolution + pooling layers. The first convolution layer
has 16 filters (output channel size would be 16), and filter size is 5 x 5. The second convolutional
layer has 32 filters (output channel size would be 32), and filter size is 5 x 5. The third
convolutional layer has 64 filters (output channel size would be 64), and filter size is 5 x 5. All
3 pooling layers are MaxPool layers with pool size of 2 x 2. Afterwards, we flatten the previous layer's
output (every row/column/channel would be an individual node), then add a fully connected layer with 256
nodes and relu activation function. Finally, we add a fully connected layers with 10 nodes, and use
softmax activation function to get the probability of each fruit/vegetable. Fruit/vegetable with the
highest probability is predicted by CNN. The model config can be downloaded as a JSON file.

### Create a deep learning model

> <hands-on-title>Model builder (Optimizer, loss function, and fit parameters)</hands-on-title>
>
> - {% tool [Create deep learning model](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/0.5.0) %}
>    - *"Choose a building mode"*: `Build a training model`
>    - *"Select the dataset containing model configuration"*: Select the *Keras Model Config* from the previous step.
>    - *"Do classification or regression?"*: `KerasGClassifier`
>    - In *"Compile Parameters"*:
>        - *"Select a loss function"*: `categorical_crossentropy`
>        - *"Select an optimizer"*: `Adam - Adam optimizer `
>        - *"Select metrics"*: `acc/accuracy`
>    - In *"Fit Parameters"*:
>        - *"epochs"*: `40`
>        - *"batch_size"*: `50`
>    - Click *"Run Tool"*
{: .hands_on}

A loss function measures how different the predicted output is from the expected output. For multi-class classification problems,
we use *categorical cross entropy* as loss function. Epochs is the number of times the whole training data is used to train the
model. Setting *epochs* to 40 means each training example in our dataset is used 40 times to train our model. If we update network
weights/biases after all the training data is feed to the network, the training will be very slow (as we have 5014 training examples
in our dataset). To speed up the training, we present only a subset of the training examples to the network, after which we update
the weights/biases. *batch_size* decides the size of this subset. The model builder can be downloaded as a zip file.

### Deep learning training and evaluation

> <hands-on-title>Training the model</hands-on-title>
>
> - {% tool [Deep learning training and evaluation](toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.8.2) %}
>    - *"Select a scheme"*: `Train and Validate`
>    - *"Choose the dataset containing pipeline/estimator object"*: Select the *Keras Model Builder* from the previous step.
>    - *"Select input type:"*: `tabular data`
>        - *"Training samples dataset"*: Select `train_X_10` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>        - *"Dataset containing class labels or target values"*: Select the OHE representation of `train_y_10` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
>
{: .hands_on}

The training step generates 3 datasets. 1) accuracy of the trained model, 2) the trained model, downloadable as a zip file, and
3) the trained model weights, downloadable as an hdf5 file. These files are needed for prediction in the next step.

### Model Prediction

> <hands-on-title>Testing the model</hands-on-title>
>
> - {% tool [Model Prediction](toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.8.2) %}
>    - *"Choose the dataset containing pipeline/estimator object"* : Select the trained model from the previous step.
>    - *"Choose the dataset containing weights for the estimator above"* : Select the trained model weights from the previous step.
>    - *"Select invocation method"*: `predict`
>    - *"Select input data type for prediction"*: `tabular data`
>    - *"Training samples dataset"*: Select `test_X_10` dataset
>    - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
{: .hands_on}

The prediction step generates 1 dataset. It's a file that has predictions (0 to 9 for the predicted fruit/vegetable) for every image
in the test dataset.

### Machine Learning Visualization Extension

> <hands-on-title>Creating the confusion matrix</hands-on-title>
>
> - {% tool [Machine Learning Visualization Extension](toolshed.g2.bx.psu.edu/repos/bgruening/ml_visualization_ex/ml_visualization_ex/1.0.8.2) %}
>    - *"Select a plotting type"*: `Confusion matrix for classes`
>    - *"Select dataset containing the true labels"*": `test_y_10`
>    - *"Does the dataset contain header:"*: `Yes`
>    - *"Choose how to select data by column:"*: `Select columns by column header name(s)`
>    	- *"Type header name(s):"*: `Label`
>    - *"Select dataset containing the predicted labels"*": Select `Model Prediction` from the previous step
>    - *"Does the dataset contain header:"*: `Yes`
>    - Click *"Run Tool"*
>
{: .hands_on}

**Confusion Matrix** is a table that describes the performance of a classification model. It lists the number of examples that were
correctly classified by the model, True positives (TP) and true negatives (TN). It also lists the number of examples that were
classified as positive that were actually negative (False positive, FP, or Type I error), and the number of examples that were
classified as negative that were actually positive (False negative, FN, or Type 2 error). Given the confusion matrix, we can
calculate **precision** and **recall** {% cite TatbulEtAl  %}. Precision is the fraction of predicted positives that are true
positives (Precision = TP / (TP + FP)). Recall is the fraction of true positives that are predicted (Recall = TP / (TP + FN)).
One way to describe the confusion matrix with just one value is to use the **F score**, which is the harmonic mean of precision
and recall

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} $$

![Confusion matrix for fruit 360 image classification problem](../../images/Fruit_confusion_matrix.png "Fruits/vegetables image classification confusion matrix")

Figure 1 is the resultant confusion matrix for our image classification problem. The first row in the table represents the *true*
fruit/vegetable with 0 as class label, which is strawberry (we have 164 strawberry images with 0 as class label). The second row
represents the *true* fruit/vegetable with 1 as class label, which is Apple_Red_Delicious (We have 166 Apple_Red_Delicious images
with 1 as class label). Similarly, you can count the true class labels for fruits/vegetables with class label of 2 to 9 by adding
up the numbers in the corresponding row. The first column from the left represents the *predicted* fruit/vegetable with 0 as class
label, which is strawberry (Our CNN predicted 164 images as being strawberry, and having class label 0). The second column from
the left represents the *predicted* fruit/vegetable with 1 as class label, which is Apple_Red_Delicious (Our CNN predicted 166
images as being Apple_Red_Delicious, and having class label 1). Similarly, you can count the predicted class labels for
fruits/vegetables with class labels 2 to 9 by adding up the numbers in the corresponding column.

For label 3, which is corn, looking at the green cell in the 4th row and 4th column, we see that our CNN has correctly
predicted 118 images as being a corn image (True positives). Adding the numbers in the other rows in column 4, we see that
our CNN has incorrectly predicted 16 images as being corn (False positives). Adding the numbers on the 4th row besides the
True positives, we see that our CNN has incorrectly predicted 32 corn images as being label Potato_White (False negatives).
Given these numbers we can calculate Precision, Recall, and the F score for digit 0 as follows:

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} = \frac{118}{118 + 16} = 0.88 $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} = \frac{118}{118 + 32} = 0.78 $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} = \frac{2 * 0.88 * 0.78}{0.88 + 0.78} = 0.82 $$

You can calculate the Precision, Recall, and F score for other digits in a similar manner.

# Conclusion


In this tutorial, we briefly described convolutional neural networks (CNN) and their application to image classification problems.
We then used Galaxy's ML toolkit to solve an image classification problem using CNN on fruit 360 dataset.

tools_json:
["Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "To categorical", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_to_categorical/sklearn_to_categorical/1.0.8.3", "Create a deep learning model architecture", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/0.5.0", "Create deep learning model", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/0.5.0", "Deep learning training and evaluation", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.8.2", "Model Prediction", "toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.8.2", "Machine Learning Visualization Extension", "toolshed.g2.bx.psu.edu/repos/bgruening/ml_visualization_ex/ml_visualization_ex/1.0.8.2"]


topic: statistics
tutorial: genomic-llm-finetuning
title: Fine-tuning a LLM for DNA Sequence Classification
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml", "Large Language Model"]
content_hash: 976ff7259b55d04122771cc66ef21c92
last_modified: 2026-05-27T09:01:49.829192
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/genomic-llm-finetuning/tutorial.html

content:
After preparing, training, and utilizing a language model for DNA sequences, we can now fine-tune a pre-trained Large Language Model (LLM) for specific DNA sequence classification tasks. Here, we will use a pre-trained model from Hugging Face, specifically the [Mistral-DNA-v1-17M-hg38](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-17M-hg38), and adapt it to classify DNA sequences based on their biological functions. Our objective is to classify sequences according to whether they bind to transcription factors.

> <comment-title>Transcription factors</comment-title>
>
> Transcription factors are proteins that play a crucial role in regulating gene expression by binding to specific DNA sequences, known as enhancers or promoters. These proteins act as molecular switches, turning genes on or off in response to various cellular signals and environmental cues. By binding to DNA, transcription factors either promote or inhibit the recruitment of RNA polymerase, the enzyme responsible for transcribing DNA into RNA, thereby influencing the rate of transcription.
> 
> ![Diagram illustrating DNA binding with CTCF. The left panel, outlined in red, shows a DNA sequence 'CCACCAGGGGGCGC' labeled as 'DNA binding CTCF,' with an oval labeled 'CTCF' above it. The right panel, outlined in blue, shows a different DNA sequence 'GTGGCTAGTAGGTAG' labeled as 'DNA not binding CTCF,' indicating that this sequence does not interact with CTCF.](images/two_dna_sequences.png "Two types of DNA sequences. On the left, a DNA sequence that binds the transcription factor CTCF. On the right, a DNA sequence that does not bind CTCF.")
>
> Transcription factors are essential for numerous biological processes, including cell differentiation, development, and response to external stimuli. Their ability to recognize and bind specific DNA sequences allows them to orchestrate complex gene expression programs, ensuring that the right genes are expressed at the right time and in the right place within an organism. Understanding the function and regulation of transcription factors is vital for deciphering the molecular mechanisms underlying health and disease, and it opens avenues for developing targeted therapeutic interventions.
>
{: .comment}

This classification task is crucial for understanding gene regulation, as transcription factors play a vital role in controlling which genes are expressed in a cell. By training a model to predict whether a DNA sequence binds to a transcription factor, we can gain insights into regulatory mechanisms and potentially identify novel binding sites or understand the impact of genetic variations on transcription factor binding.

By fine-tuning the model, we aim to leverage its pre-trained knowledge of DNA sequences to achieve high accuracy in this classification task. This tutorial will guide you through the necessary steps, from data preparation to model evaluation, ensuring you can apply these techniques to your own research or projects.

We will use [`Mistral-DNA-v1-17M-hg38`](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-1M-hg38), a mixed model that was pre-trained on the entire Human Genome. It contains approximately 17 million parameters and was trained using the Human Genome assembly GRCh38 on sequences of 10,000 bases (10K):

```python
model_name="RaphaelMourad/Mistral-DNA-v1-17M-hg38"
```

> <comment-title>Pretraining a LLM</comment-title>
>
> To learn how to pretrain a LLM on DNA, please follow the dedicated ["Pretraining a Large Language Model (LLM) from Scratch on DNA Sequences"]({% link topics/statistics/tutorials/genomic-llm-pretraining/tutorial.md %}) tutorial
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare resources

## Install dependencies

The first step is to install the required dependencies:

```python
!pip install accelerate==1.1.0
!pip install peft==0.13.2
!pip install torch==2.5.0
!pip install transformers -U
!pip install progressbar
!pip install bitsandbytes
```

> <question-title></question-title>
>
> 1. What is `accelerate`?
> 2. What is `peft`?
> 3. What is `torch`?
> 4. What is `transformers`?
>
> > <solution-title></solution-title>
> >
> > 1. `accelerate` is a library by [Hugging Face](https://huggingface.co/) -- a platform that provides tools and resources for building, training, and deploying machine learning models -- designed to simplify the process of training and deploying machine learning models across different hardware environments. It provides tools to optimize performance on GPUs, TPUs, and other accelerators, making it easier to scale models efficiently.
> >
> > 2. The PEFT (Parameter-Efficient Fine-Tuning) Python library, developed by Hugging Face, is a tool designed to efficiently adapt large pretrained models to various downstream tasks without the need to fine-tune all of the model's parameters. By focusing on a small subset of parameters, PEFT significantly reduces computational and storage costs, making it feasible to fine-tune large language models (LLMs) on consumer-grade hardware. The library integrates seamlessly with the Hugging Face ecosystem, including Transformers, Diffusers, and Accelerate, enabling streamlined model training and inference. PEFT supports techniques like LoRA (Low-Rank Adaptation) and prompt tuning, and it can be combined with quantization to further optimize resource usage. Its open-source nature fosters collaboration and accessibility, allowing developers to customize models for specific applications quickly and efficiently.
> > 
> > 3. `torch`, also known as PyTorch, it is an open-source machine learning library developed by Facebook's AI Research lab. It provides a flexible platform for building and training neural networks, with a focus on tensor computations and automatic differentiation.
> > 
> > 4. `transformers` is a library by Hugging Face that provides implementations of state-of-the-art transformer models for natural language processing (NLP). It includes pre-trained models and tools for fine-tuning, making it easier to apply transformers to various NLP tasks.
> >
> {: .solution}
>
{: .question}

## Import Python libraries

Let's now import them.

```python
import os

import accelerate
import flash_attn
import numpy as np
import pandas as pd
import torch
import transformers
from accelerate import FullyShardedDataParallelPlugin, Accelerator
from pathlib import Path
from peft import (
    LoraConfig,
    get_peft_model,
    get_peft_model_state_dict,
    prepare_model_for_kbit_training,
)
from progressbar import ProgressBar
from random import randrange
from torch.utils.data import TensorDataset, DataLoader
from torch.distributed.fsdp.fully_sharded_data_parallel import FullOptimStateDictConfig, FullStateDictConfig
from transformers import (
    AutoTokenizer,
    AutoModel,
    BitsAndBytesConfig,
    EarlyStoppingCallback,
    set_seed,
)
```

> <comment-title>Versions</comment-title>
>
> This tutorial has been tested with following versions:
> - `numpy` = 1.19 (and not 1.2)
> - `transformers` > 4.47.1
> 
> You can check the versions with:
>
> ```python
> np.__version__
> transformers.__version__
> ```
{: .comment}

# Configure fine-tuning

## Check and configure available resources

We select the appropriate device (CUDA-enabled GPU if available) for running PyTorch operations

```python
torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

Let's check the GPU usage and RAM:

```python
!nvidia-smi
```

We now set an environment variable that configures how PyTorch manages CUDA memory allocations

```python
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"
```

## Specify settings for quantization

Quantization is a technique used in machine learning and signal processing to reduce the precision of numerical values, typically to decrease memory usage and computational requirements. This process is particularly useful when working with large models as it allows them to be deployed on hardware with limited resources without significantly sacrificing performance.

Here, we use `BitsAndBytesConfig` to configure a 4-bit quantization. Using 4-bit precision reduces the memory footprint of the model, which is particularly useful for very large models that might not fit into GPU memory otherwise:

```python
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `load_in_4bit=True`
> 2. `bnb_4bit_use_double_quant=True`
> 3. `bnb_4bit_compute_dtype=torch.bfloat16`
>
> > <solution-title></solution-title>
> >
> > 1. `load_in_4bit=True`: Specifies that the model should be loaded with 4-bit quantization. Using 4-bit precision reduces the memory footprint of the model, which is particularly useful for very large models that might not fit into GPU memory otherwise.
> >
> > 2. `bnb_4bit_use_double_quant=True`: enables double quantization, which means that the quantization constants from the first quantization are quantized again. This further reduces the memory footprint, although it may introduce additional computational overhead.
> >
> > 3. `bnb_4bit_compute_dtype=torch.bfloat16`: sets the compute data type to bfloat16 (Brain Floating Point 16-bit format). Using bfloat16 can provide a good balance between computational efficiency and numerical stability, especially on hardware that supports this format, such as certain GPUs and TPUs.
> >
> {: .solution}
>
{: .question}

## Configure Accelerate

Now, we will configure the [Hugging Face Accelerate library](https://huggingface.co/docs/accelerate/en/index) to optimize the training process for large models using Fully Sharded Data Parallel (FSDP). This setup is crucial for efficiently utilizing GPU resources and enabling distributed training across multiple devices.

First, we need to configure the FSDP plugin, which will manage how model parameters and optimizer states are sharded across GPUs. This configuration helps in reducing memory usage and allows for the training of larger models.

```python
fsdp_plugin = FullyShardedDataParallelPlugin(
    state_dict_config=FullStateDictConfig(offload_to_cpu=True, rank0_only=False),
    optim_state_dict_config=FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=False),
)
```
> <question-title></question-title>
>
> What do the parameters?
> 
> 1. `state_dict_config=FullStateDictConfig(offload_to_cpu=True, rank0_only=False)`?
> 2. `optim_state_dict_config=FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=False)`?
>
> > <solution-title></solution-title>
> >
> > 1. `state_dict_config=FullStateDictConfig(offload_to_cpu=True, rank0_only=False)`
> >    - `FullStateDictConfig`: Configures how the model's state dictionary (parameters) is managed.
> >    - `offload_to_cpu=True`: Specifies that the model's parameters should be offloaded to CPU memory when not in use. This helps free up GPU memory, especially useful when working with large models.
> >    - `rank0_only=False`: Indicates that the state dictionary operations (like saving and loading) are not restricted to the rank 0 process. This allows all processes to participate in these operations, which can be beneficial for distributed training setups.
> >
> > 2. `optim_state_dict_config=FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=False)`
> >    - `FullOptimStateDictConfig`: Configures how the optimizer's state dictionary is managed.
> >    - `offload_to_cpu=True`: Similar to the model's state dictionary, this setting offloads the optimizer states to CPU memory when not in use, further reducing GPU memory usage.
> >    - `rank0_only=False`: Allows all processes to handle the optimizer state dictionary operations, ensuring that the optimizer states are managed efficiently across the distributed setup.
> >
> {: .solution}
>
{: .question}

Next, we initialize the Accelerator from the Hugging Face Accelerate library, integrating the FSDP plugin for seamless distributed training:

```python
accelerator = Accelerator(fsdp_plugin=fsdp_plugin)
```

By passing the FSDP plugin to the `Accelerator`, we enable sharded data parallelism, which efficiently manages model and optimizer states across multiple GPUs.

With this configuration, the `Accelerator` will handle the complexities of distributed training, allowing us to focus on developing and experimenting with our models. This setup is particularly beneficial when working with large-scale models and limited GPU resources, as it optimizes memory usage and enables faster training times.

## Configure LoRA for Parameter-Efficient Fine-Tuning

We will configure the LoRA (Low-Rank Adaptation) settings for parameter-efficient fine-tuning of a large language model. LoRA is a technique that allows us to fine-tune only a small number of additional parameters while keeping the original model weights frozen, making it highly efficient for adapting large models to specific tasks.

We use the `LoraConfig` class to define the settings for LoRA. This configuration specifies how the low-rank adaptations are applied to the model.

```python
peft_config = LoraConfig(
    r=16,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type="SEQ_CLS",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]
)
```

> <question-title></question-title>
>
> What do the parameters?
> 
> 1. `r=16`?
> 2. `lora_alpha=16`?
> 3. `lora_dropout=0.05`?
> 4. `bias="none"`?
> 5. `task_type="SEQ_CLS"`?
> 6. `target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]`?
>
> > <solution-title></solution-title>
> >
> > 1. `r=16`: This parameter specifies the rank of the low-rank matrices used in the adaptation. A higher rank allows the model to capture more complex patterns but also increases the number of trainable parameters.
> >
> > 2. `lora_alpha=16`: This scaling factor controls the magnitude of the updates applied by the low-rank matrices. It helps balance the influence of the adaptations relative to the original model weights.
> >
> > 3. `lora_dropout=0.05`: Dropout is applied to the low-rank matrices during training to prevent overfitting. A dropout rate of 0.05 means that 5% of the elements are randomly set to zero during each training step.
> >
> > 4. `bias="none"`: This setting specifies that no bias parameters are added to the low-rank adaptations. Other options include "all" to add biases to all layers or "lora_only" to add biases only to the LoRA layers.
> >
> > 5. `task_type="SEQ_CLS"`: This indicates that the model is being fine-tuned for a sequence classification task. Other task types might include "CAUSAL_LM" for causal language modeling or "SEQ_2_SEQ_LM" for sequence-to-sequence tasks.
> >
> > 6. `target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]`: This list specifies the modules within the model architecture to which the LoRA adaptations will be applied. These modules are typically the attention layers in transformer models:
> >    - `"q_proj"`: query projections
> >    - `"k_proj"`: key projections
> >    - `"v_proj"`: value projections
> >    - `"o_proj"`: output projections
> >    - `"gate_proj"`: gating projections in some architectures.
> >
> {: .solution}
>
{: .question}

By configuring LoRA in this way, we can efficiently adapt a large pretrained model to a specific task with minimal computational overhead, making it feasible to fine-tune on consumer-grade hardware. This approach is particularly useful for tasks like text classification, sentiment analysis, or any other application where we need to specialize a general-purpose language model.

## Configure Training Arguments

Let's now set up the training arguments using the `TrainingArguments` class from the Hugging Face Transformers library. These arguments define the training configuration, including hyperparameters and settings for saving and evaluating the model.

```python
training_args = transformers.TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=1e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    weight_decay=0.01,
    bf16=True,
    report_to="none",
    load_best_model_at_end = True,
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `output_dir="./results"`
> 2. `evaluation_strategy="epoch"`
> 3. `save_strategy="epoch"`
> 4. `learning_rate=1e-5`
> 5. `per_device_train_batch_size=16`
> 6. `per_device_eval_batch_size=16`
> 7. `num_train_epochs=5`
> 8. `weight_decay=0.01`
> 9. `bf16=True`
> 10. `report_to="none"`
> 11. `load_best_model_at_end=True`
> 
> > <solution-title></solution-title>
> >
> > 1. `output_dir="./results"`: Specifies the directory where the model predictions and checkpoints will be saved.
> >
> > 2. `evaluation_strategy="epoch"`: The model will be evaluated at the end of each epoch. This allows for monitoring the model's progress and adjusting the training process as needed.
> >
> > 3. `save_strategy="epoch"`: The model checkpoints will be saved at the end of each epoch.  This ensures that checkpoints are available for each complete pass through the dataset.
> >
> > 4. `learning_rate=1e-5`: Sets the initial learning rate for the optimizer. This rate determines how much the model's weights are updated during training.
> >
> > 5. `per_device_train_batch_size=16`: The number of samples per device (e.g., GPU) to load for training. 
> >
> > 6. `per_device_eval_batch_size=16`: The number of samples per device to load for evaluation.
> >
> > 7. `num_train_epochs=5`: The total number of training epochs. An epoch is one complete pass through the training dataset.
> >
> > 8. `weight_decay=0.01`: Applies L2 regularization to the model weights to prevent overfitting.
> >
> > 9. `bf16=True`: Enables mixed precision training using bfloat16, which can speed up training and reduce memory usage on compatible hardware.
> >
> > 10. `report_to="none"`: Disables reporting to external services like WandB or TensorBoard. If you want to track metrics, you can set this to "wandb", "tensorboard", etc.
> >
> > 11. `load_best_model_at_end=True`: Ensures that the best model based on evaluation metrics is loaded at the end of training.
> >
> {: .solution}
>
{: .question}

These settings provide a balanced configuration for training a model efficiently while ensuring that the best version of the model is saved and can be used for further evaluation or deployment. Adjust these parameters based on your specific use case and available computational resources.



# Prepare the tokenizer

We will now set up the tokenizer to convert DNA sequences into numerical tokens that the model can process. The tokenizer is a crucial component in preparing the data for model training and inference: it transforms raw text into a format that can be processed by machine learning models. 

We use the `AutoTokenizer` class from the Hugging Face Transformers library to load a pre-trained tokenizer. We specify the pre-trained model from which to load the tokenizer. This should match the model you plan to use for training or inference. This tokenizer will be configured to handle DNA sequences efficiently.

```python
tokenizer = transformers.AutoTokenizer.from_pretrained(
    model_name,
    model_max_length=200,
    padding_side="right",
    use_fast=True,
    trust_remote_code=True,
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `model_max_length=200`
> 2. `padding_side="right"`
> 3. `use_fast=True`
> 4. `trust_remote_code=True`
> 
> > <solution-title></solution-title>
> >
> > 1. `model_max_length=200`: Sets the maximum length of the tokenized sequences. Sequences longer than this will be truncated, and shorter ones will be padded.
> > 
> > 2. `padding_side="right"`: Specifies that padding should be added to the right side of the sequences. This ensures that all sequences in a batch have the same length.
> >
> > 3. `use_fast=True`: Enables the use of the fast tokenizer implementation, which is optimized for speed and is suitable for most use cases.
> >
> > 4. `trust_remote_code=True`: Allows the tokenizer to execute custom code from the model repository, which may be necessary for some models that require specific preprocessing steps.
> >
> {: .solution}
>
{: .question}

By configuring the tokenizer in this way, we ensure that our DNA sequences are properly tokenized and formatted for input into the model. This step is essential for preparing our data for efficient and effective model training and evaluation.

Let's now tailor the tokenizer to better suit our specific use case, ensuring that the model processes sequences accurately and efficiently. Special tokens play a crucial role in defining how sequences are processed and interpreted by the model. Here, we sets:
- the end-of-sequence (EOS) token, which indicates the end of a sequence. It is essential for tasks where the model needs to generate sequences or understand where a sequence ends.
- the padding (PAD) token, which is used to pad sequences to a uniform length within a batch. Padding ensures that all sequences in a batch have the same length, which is necessary for efficient processing during training and inference

```
tokenizer.eos_token = "[EOS]"
tokenizer.pad_token = "[PAD]"
```

# Prepare data

To finetune the model, we must provide a dataset to train the model. We will the data with the 1st transcription factor (`tf0`) in mouse from {% cite zhou2024dnabert2efficientfoundationmodel %}. The data is stored on [GitHub](https://github.com/raphaelmourad/Mistral-DNA).

## Get data

Let's get the data for from GitHub:

```python
!git clone https://github.com/raphaelmourad/Mistral-DNA.git
```

We now need to uncompress the labeled data:

```python
!tar -xf Mistral-DNA/data/GUE.tar.xz -C Mistral-DNA/data/
```

We change the current working directory to the `Mistral-DNA` folder.

```python
os.chdir("Mistral-DNA/")
print(os.getcwd())
```

Let's define experience and path to data variables

```python
expe = "tf/0"
data_path = f"data/GUE/{ expe }" 
```




## Prepare Datasets for Training and Validation

We now need to set up the datasets required for training and validating. Properly preparing these datasets is crucial for ensuring that the model finetunes effectively and generalizes well to new data.

We will use the files `data_path` folder we just defined:
- `train.csv` for training
- `dev.csv` for validation

> <question-title></question-title>
>
> How is the content of each file?
>
> 1. `train.csv`
> 2. `dev.csv`
> 
> > <solution-title></solution-title>
> >
> > The 2 files are CSV files with 2 columns (`sequence` and `label`) and different number of rows:
> > 1. `train.csv`: 32,379 rows. 
> > 2. `dev.csv`: 1,000 rows
> > 
> > Values in `label` are:
> > - `0`: The DNA sequence in `sequence` column does not bind to the 1st transcription factor.
> > - `1`: The DNA sequence in `sequence` column binds to the transcription factor.
> >
> {: .solution}
>
{: .question}


Before we proceed we import some classes and functions from `scriptPython/function.py`:

```python
### LOAD FUNCTIONS MODULE
import sys
sys.path.append("scriptPython/")
from functions import *
```

We use the `SupervisedDataset` class to load and prepare the datasets. This class handles the tokenization and formatting of the data, making it ready for model training and evaluation.

```python
train_dataset = SupervisedDataset(
    tokenizer=tokenizer,
    data_path=Path(data_path) / "train.csv",
    kmer=-1,
)
val_dataset = SupervisedDataset(
    tokenizer=tokenizer,
    data_path=Path(data_path) / "dev.csv",
    kmer=-1,
)
```

> <question-title></question-title>
>
> What does `kmer=-1`?
>
> > <solution-title></solution-title>
> >
> > This parameter is used to specify the length of k-mers (substrings of length k) to be considered in the dataset. A value of -1 typically means that no k-mer splitting is applied, and the sequences are processed as they are.
> >
> {: .solution}
>
{: .question}

## Configure Data Collation

A data collator ensures that sequences are properly padded and formatted, which is crucial for optimizing the training process.

We'll use the `DataCollatorForSupervisedDataset` class to handle the collation of tokenized data. This collator will manage padding and ensure that all sequences in a batch are of uniform length.

```python
data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer)
```

# Load and Configure the Model for Sequence Classification

Let's now load the pre-trained model, a model originally trained for large language modeling tasks, not specifically for classification. To adapt it for our binary classification task, we will add a new classification head on top of the existing architecture. This head will consist of a single neuron that connects to the output of the language model, enabling it to classify whether a DNA sequence binds to a transcription factor (label `1`) or not (label `0`).

This additional layer, or **classification head**, is a simple neural network layer that takes the high-level features extracted by the language model and maps them to our binary classification output. It learns to weigh these features appropriately to make accurate predictions for our specific task.

We use the `AutoModelForSequenceClassification` class from the Hugging Face Transformers library to load the pre-trained model and set it up for our specific classification task:

```python
model=transformers.AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2,
    output_hidden_states=False,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `num_labels=2`
> 2. `output_hidden_states=False`
> 3. `quantization_config=bnb_config`
> 4. `device_map="auto"`
> 5. `trust_remote_code=True`
> 
> > <solution-title></solution-title>
> >
> > 1. `num_labels=2`: Sets the number of output labels to 2, corresponding to the binary classification task (binding or not binding to transcription factors).
> >
> > 2. `output_hidden_states=False`: Indicates that the model should not output hidden states. This is typically set to False unless you need access to the intermediate representations for further analysis.
> > 
> > 3. `quantization_config=bnb_config`: Applies predefined quantization configuration to the model, which helps reduce memory usage and enables efficient training on consumer-grade hardware.
> > 
> > 4. `device_map="auto"`: Automatically determines the best device placement for the model's layers, optimizing for available hardware (e.g., GPUs). If it finds a GPU, it will use a GPU. If there's no GPU, it will not use the GPU
> > 
> > 5. `trust_remote_code=True`: Allows the model to execute custom code from the model repository, which may be necessary for certain architectures or preprocessing steps.
> >
> {: .solution}
>
{: .question}

To ensure that the model correctly handles padding tokens, we need to align the padding token configuration between the model and the tokenizer. This step is crucial for maintaining consistency during training and inference, especially when dealing with sequences of varying lengths:

```python
model.config.pad_token_id = tokenizer.pad_token_id
```

# Initialize the Trainer

We can now set up the `Trainer` to manage the training and evaluation process of our model. The `Trainer` class simplifies the training loop, handling many of the complexities involved in training deep learning models.

We first need to attach the LoRA adapter to the model:

```python
model.add_adapter(peft_config, adapter_name="lora_1")
```

Let's now set up the `Trainer`:

```python
trainer = transformers.Trainer(
    model=model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    data_collator=data_collator,
    callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]
)
```

> <question-title></question-title>
>
> What do the `callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]` parameter?
> 
> > <solution-title></solution-title>
> >
> > It adds an early stopping mechanism to the training process. This mechanism is designed to halt training when the model's performance on the validation set stops improving, helping to prevent overfitting and conserve computational resources.
> > 
> > How Early Stopping Works?
> > 
> > **Purpose**: The primary goal of early stopping is to capture the model parameters when the loss reaches its minimum value during training. This is crucial because, after a certain point, continued training may lead to overfitting, where the model starts to perform worse on unseen data.
> > 
> > **Patience Parameter**: The `early_stopping_patience=3` setting specifies that training should continue for three additional epochs after the model's performance on the validation set stops improving. This "patience" period helps mitigate the effects of noise in the training process. Noise can cause temporary fluctuations in the loss, making it seem like the model has reached a local minimum when further training might yield better results.
> >
> > **Process**: During training, the loss is monitored at each epoch. If the loss does not decrease for three consecutive epochs, training is stopped. However, if a better model with a lower loss is found within those three epochs, training continues. This approach ensures that the model has truly reached a robust local minimum, rather than being prematurely halted due to noise.
> >
> > By incorporating early stopping with a patience of three epochs, you balance the need to find an optimal model with the risk of overfitting, ultimately leading to more efficient and effective training outcomes.
> > 
> {: .solution}
>
{: .question}

Ffor distributed training, where multiple GPUs or nodes are used to accelerate the training process, it is essential to do:

```python
trainer.local_rank=training_args.local_rank
```

The `local_rank` parameter identifies the rank of the current process within its local node, enabling coordinated communication and synchronization between processes. This setup is crucial for managing tasks such as gradient synchronization and data partitioning, ensuring that each process operates on the correct portion of the model or dataset. By assigning the local rank from `training_args` to the `Trainer`, we facilitate efficient and scalable training, leveraging the full computational power of multi-GPU environments.

# Start the training

Let's start the training process for our model using the trainer.train() method:

```python
trainer.train()
```

After launching `trainer.train()`, we can notice that the training process is significantly faster compared to training a model from scratch seen in ["" tutorial]({% link topics/statistics/tutorials/genomic-llm-pretraining/tutorial.md %}). This efficiency is due to the use of a pre-trained model, which has already undergone extensive training on large datasets using powerful computational resources. For example, pre-training a model on even a small portion of the human genome can take dozens of hours, but fine-tuning this model on a specific task, such as classifying DNA sequences, is much quicker. Fine-tuning leverages the pre-trained model's foundational knowledge, allowing you to adapt it to new tasks with a smaller, labeled dataset. This approach not only saves time but also reduces the need for extensive computational power. By downloading a pre-trained model from platforms like Hugging Face and fine-tuning it on a local machine with a modest GPU, we can achieve high performance with minimal overhead, making advanced modeling techniques accessible for a wide range of applications.

# Evaluate Model Performance

After successfully training the model, the next essential step is to evaluate its performance on a test dataset. This evaluation process is crucial for understanding how well the model generalizes to new, unseen data and for assessing its readiness for real-world applications.

> <comment-title></comment-title>
>
> If finetuning is too long, you can stop the training.
>
{: .comment}

The test data is stored in `data_path/test.csv`, we prepare it as for training and validation data.

```python
test_dataset = SupervisedDataset(
    tokenizer=tokenizer,
    data_path=Path(data_path) / "test.csv",
    kmer=-1,
)
```

We then use the `trainer.evaluate()` method. This methods is designed to assess the model's performance on a specified dataset, typically the test dataset, which contains data that the model has not encountered during training.

```python
results = trainer.evaluate(eval_dataset=test_dataset)
```

The method computes various evaluation metrics, such as accuracy, precision, recall, and F1 score, depending on the task and the configuration specified in `compute_metrics`. These metrics provide a comprehensive view of the model's performance, highlighting its strengths and weaknesses.

The Trainer uses the `data_collator` to ensure that the test data is properly formatted and padded, maintaining consistency with the training process. This consistency is crucial for accurate evaluation.

The evaluation results are stored in the `results` variable, which contains the computed metrics. We can analyze these `results` to gain insights into the model's performance and make informed decisions about further improvements or deployment.

> <question-title></question-title>
>
> What is stored in `results`? How do you interpret this information?
> 
> > <solution-title></solution-title>
> >
> > `results` provides a comprehensive overview of the model's performance on the evaluation dataset with:
> > 1. **eval_loss (0.424961)**: This metric represents the loss value calculated on the evaluation dataset. Lower values indicate better model performance.
> >
> >    A loss of 0.425 suggests that the model is reasonably well-fitted to the data, though the specific interpretation depends on the context and the loss function used (e.g., cross-entropy for classification tasks).
> > 
> > 2. **eval_accuracy (0.804000)**: Accuracy measures the proportion of correctly predicted instances out of the total instances.
> > 
> >    An accuracy of 80.4% indicates that the model correctly predicted the class for 80.4% of the samples in the evaluation dataset.
> > 
> > 4. **eval_f1 (0.800838)**: The F1 score is the harmonic mean of precision and recall, providing a single metric that balances both concerns.
> >    
> >    An F1 score of 0.801 suggests a good balance between precision and recall, indicating that the model performs well in both identifying positive cases and minimizing false positives and negatives.
> > 
> > 4. **eval_matthews_correlation (0.628276)**: The Matthews Correlation Coefficient (MCC) is a measure of the quality of binary classifications, taking into account true and false positives and negatives.
> >    
> >    An MCC of 0.628 indicates a moderate to strong correlation between the predicted and actual classes, suggesting the model is performing better than random guessing.
> > 
> > 6. **eval_precision (0.824614)**: Precision is the ratio of correctly predicted positive observations to the total predicted positives.
> > 
> >    A precision of 82.5% means that out of all the instances predicted as positive, 82.5% were actually positive.
> > 
> > 6. **eval_recall (0.804000)**: Recall (or sensitivity) is the ratio of correctly predicted positive observations to all observations in the actual class.
> >    
> >    A recall of 80.4% indicates that the model correctly identified 80.4% of all actual positive cases.
> > 
> > 8. **eval_runtime (6.548800)**: The total time taken to evaluate the model on the dataset.
> >    
> >    A runtime of 6.55 seconds provides insight into the computational efficiency of the evaluation process.
> > 
> > 8. **eval_samples_per_second (152.699000)**: The number of samples processed per second during evaluation.
> > 
> >    Processing 152.7 samples per second indicates the efficiency of the evaluation pipeline.
> > 
> > 9. **eval_steps_per_second (9.620000)**: The number of evaluation steps completed per second.
> >     
> >    Completing 9.62 steps per second reflects the speed of the evaluation process.
> > 
> > 11. **epoch (3.000000)**: The number of training epochs completed before this evaluation.
> > 
> >    The evaluation was conducted after 3 epochs of training, providing context for the model's learning progress.
> > 
> {: .solution}
>
{: .question}


# Conclusion

In this tutorial, we explored the process of fine-tuning a large language model (LLM) for DNA sequence classification. By following the steps outlined, you have learned how to leverage pre-trained models to achieve efficient and effective classification of DNA sequences, specifically focusing on their binding affinity to transcription factors.

We began by configuring the fine-tuning process, ensuring that available computational resources were optimally utilized. This included specifying settings for quantization, configuring Accelerate for distributed training, and implementing LoRA for parameter-efficient fine-tuning. These steps were crucial for maximizing performance and minimizing computational overhead.

Next, we prepared the tokenizer and data, ensuring that DNA sequences were properly tokenized and formatted for model input. We created datasets for training and validation, and configured data collation to handle batch processing efficiently.

We then loaded and configured the model for sequence classification, adding a classification head to adapt the pre-trained model to our specific task. With the model and data prepared, we initialized the `Trainer`, which streamlined the training process by managing the training loop, evaluation, and checkpointing.

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python", "deep-learning-without-gai-with-python", "genomic-llm-pretraining"]


topic: statistics
tutorial: genomic-llm-sequence-optimization
title: Optimizing DNA Sequences for Biological Functions using a DNA LLM
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml", "Large Language Model"]
content_hash: a20b18f8802487d9b96dc7fba97bf47d
last_modified: 2026-05-27T09:01:49.830276
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/genomic-llm-sequence-optimization/tutorial.html

content:
# Prepare resources 

## Install dependencies


```python
!pip install datasets==3.0.1
!pip install torch==2.5.0
!pip install transformers -U
!pip install accelerate==1.1.0
!pip install peft==0.13.2
!pip install bitsandbytes==0.44.1
!pip install flash-attn==2.6.3
!pip install Bio==1.7.1
!pip install orfipy
```

## Import Python libraries

- `torch`
- `flash_attn`
- `numpy`
- `transformers`
  - `AutoTokenizer`
  - `EarlyStoppingCallback`
  - `Trainer`
  - `TrainingArguments`
  - `AutoModelForCausalLM`
  - `AutoConfig`
  - `DataCollatorForLanguageModeling`
- `datasets`
- `accelerate`


```python
# 
# During the class, 
import os

import sys
import time
from os import path
import gc


import flash_attn
import torch
import numpy as np
import pandas as pd
import scipy as sp
import matplotlib.pyplot as plt

import transformers

from transformers import AutoTokenizer
from transformers import EarlyStoppingCallback, Trainer, TrainingArguments
from transformers import AutoModelForCausalLM, AutoConfig
from transformers import DataCollatorForLanguageModeling
from datasets import load_dataset

import accelerate
```

## Check versions

Numpy version > 1.26.4

```python
np.__version__
```

transformers version > 4.47.1

```python
transformers.__version__
```

flash_attn > 2.6.0.post1 and 2.7.0.post2

```python
flash_attn.__version__
```

accelerate > 0.32.1

```python
# Tested with accelerate==0.32.1

accelerate.__version__
```

## Prepare GPU


```python
# CHECK GPU
# We can see how many VRAM is used and how much the GPU is used.
!nvidia-smi
```

    Thu Feb  6 07:59:23 2025       
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
    |-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  Tesla T4                       Off |   00000000:00:04.0 Off |                    0 |
    | N/A   36C    P8              9W /   70W |       0MiB /  15360MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
                                                                                             
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |  No running processes found                                                             |
    +-----------------------------------------------------------------------------------------+


```python
# LOOK AT GPU USAGE AND RAM
!nvidia-smi
```

    Thu Feb  6 16:49:41 2025       
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
    |-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  Tesla T4                       Off |   00000000:00:04.0 Off |                    0 |
    | N/A   54C    P8             10W /   70W |       2MiB /  15360MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
                                                                                             
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |  No running processes found                                                             |
    +-----------------------------------------------------------------------------------------+



Control the use of ram by CUDA

```python
torch.backends.cudnn.benchmark=True
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32 "
```

```python
# Check GPU
import torch
torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

    device(type='cuda')

# Get the model 

Mistral-DNA-v0.1 was derived from Mixtral-8x7B for the human genome. Mixtral-8x7B was simplified for DNA: the number of layers and the hidden size were reduced. The model was pretrained using the human genome hg38 with 200B DNA sequences.

The model can be downloaded on HuggingFace: https://huggingface.co/RaphaelMourad/Mistral-DNA-v0.1

```python
!git clone https://github.com/raphaelmourad/Mistral-DNA.git
```


```python
!ls
```

    Mistral-DNA  sample_data


```python
# SET DIRECTORY
os.chdir("Mistral-DNA/")
print(os.getcwd())
```

/content/Mistral-DNA

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python", "deep-learning-without-gai-with-python", "genomic-llm-pretraining", "genomic-llm-finetuning", "genomic-llm-zeroshot-prediction", "genomic-llm-sequence-generation"]


topic: statistics
tutorial: RNN
title: Deep Learning (Part 2) - Recurrent neural networks (RNN)
url: https://training.galaxyproject.org/training-material/topics/statistics/tutorials/RNN/tutorial.html
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: 7523a953c73d078ea7312aa2d4472641
last_modified: 2026-05-27T09:01:49.813195
zenodo_link: https://zenodo.org/record/4477881
workflows_json: []
content:
Artificial neural networks are a machine learning discipline roughly inspired by how neurons in a
human brain work. In the past decade, there has been a huge resurgence of neural networks thanks
to the vast availability of data and enormous increases in computing capacity (successfully
training complex neural networks in some domains requires lots of data and compute capacity). There
are various types of neural networks (feedforward, recurrent, etc). In this tutorial, we discuss
recurrent neural networks (RNN), which model sequential data, and have been successfully applied to language
generation, machine translation, speech recognition, image description, and text summarization
({% cite wen-etal-2015-semantically %}, {% cite cho-etal-2014-learning %}, {% cite LimEtAl %},
{% cite karpathyEtAl %}, {% cite li-etal-2017-deep %}). We start by explaining how RNN differ from
feedforward networks (FNN), describe various RNN architectures, and solve a sentiment analysis problem
using RNN in Galaxy.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Feedforward neural networks (FNN)

In feedforward neural networks (FNN) a single training example is presented to the network,
after which the network generates an output. For example, a lung X-ray image is passed
to a FNN, and the network predicts tumor or no tumor.

## Single layer FNN

![Neurons forming the input and output layers of a single layer feedforward neural network](../../images/FFNN_no_hidden.png "Single layer feedforward neural network")

Figure 1 shows a single layer FNN, where the input is 3 dimensional. Each input field is multiplied by a
weight. Afterwards, the results are summed up, along with a bias, and passed to an activation function.

![An activation function (such as Sigmoid, Tanh, etc.) applied to the input of the only neuron in the output layer of a feedforward neural network](../../images/activation.gif "Activation of the output neuron o1. Activation function f could be Sigmoid, Tanh, ReLU, etc.")

The activation function can have many forms (sigmoid, tanh, ReLU, linear, step function, sign function, etc.).
Output layer neurons usually have *sigmoid* or *tanh* functions. For more information on the listed activation
functions, please refer to {% cite nwankpaEtAl %}.

![Mathmatical formula for Sigmoid activation function](../../images/sigmoid.gif "Sigmoid activation function")

## Multi-layer FNN

Minsky and Papert showed that a single layer FNN cannot solve problems in which the data is not linearly separable,
such as the XOR problem ({% cite Newell780 %}). Adding one (or more) hidden layers to FNN enables it to solve problems
in which data is non-linearly separable. Per Universal Approximation Theorem, a FNN with one hidden layer can represent
any function ({% cite Cybenko1989 %}), although in practice training such a model is very difficult (if not impossible),
hence, we usually add multiple hidden layers to solve complex problems.

![Neurons forming the input, output, and hidden layers of a multi-layer feedforward neural network](../../images/FFNN.png "Feedforward neural network with a hidden layer. Biases to hidden/output layer neurons are omitted for clarity")

## Learning algorithm

In supervised learning, we are given a set of input-output pairs, called the *training set*. Given the training set, the learning algorithm
(iteratively) adjusts the model parameters, so that the model can accurately map inputs to outputs. We usually have another set of input-output
pairs, called the *test set*, which is not used by the learning algorithm. When the learning algorithm completes, we assess the learned model by
providing the test set inputs to the model and comparing the model outputs to test set outputs. We need to define a **loss function** to objectively
measure how much the model output is off of the expected output. For classification problems we use the **cross entropy** loss function.

![Mathematical formula for calculating the cross entropy loss function, which quantifies the difference between the predicted and desired output of a neural network](../../images/CrossEntropy.gif "Cross entropy loss function")

The loss function is calculated for each input-output pair in the training set. The average of the calculated loss functions for all training
set input-output pairs is called the **Cost function**. The goal of the learning algorithm is to minimize the cost function. The cost function
is a function of network weights and biases of all neurons in all layers. The **backpropagation** learning algorithm {% cite Rumelhart1986 %}
iteratively computes the gradient of cost function relative to each weight and bias, then updates the weights and biases in the opposite
direction of the gradient, to find the local minimum.

![Mathematical formula for calcuating the cross entropy cost function, which is just the average of cross entropy loss functions for training samples](../../images/CostFunction.gif "Cross entropy cost function")

# Recurrent neural networks

Unlike FNN, in RNN the output of the network at time *t* is used as network input
at time *t+1*. In RNN, a training example is a sequence, which is presented to the
network one at a time. For example, a sequence of English words is passed to a
RNN, one at a time, and the network generates a sequence of Persian words, one
at a time. RNN handle sequential data, whether its temporal or ordinal.

## Possible RNN inputs/outputs

There are 4 possible input/output combinations for RNN and each have a specific application. *One-to-one* is basically a FNN. *One-to-many* is
where we have one input and a variable number of outputs. One example application is image captioning, where a single image is provided
as input and a variable number of words (which caption the image) is returned as output (see Figure 7).

![Neurons forming a one-to-many recurrent neural network](../../images/RNN_1_to_n.png "One-to-many RNN")

*Many-to-one* RNN combination has a variable number of inputs and a single output. One example application is document sentiment
classification, where a variable number of words in a document are presented as input, and a single output predicts whether the document
has a positive or negative sentiment regarding a topic (see Figure 8).

![Neurons forming a many-to-one recurrent neural network](../../images/RNN_n_to_1.png "Many-to-one RNN")

Finally there is many-to-many RNN, which have two types: one in which the number of inputs and outputs match, e.g., in labeling the video frames the number
of frames matches the number of labels, and the other in which the number of inputs and outputs do not match, e.g., in language translation
we pass in *n* words in English and get *m* words in Italian (see Figure 9).

![Neurons forming a many-to-many recurrent neural network](../../images/RNN_n_to_m.png "Many-to-many RNN")

## RNN architectures

Mainly, there are three types of RNN: 1) Vanilla RNN, 2) LSTM ({% cite hochreiter1997long %}), and 3) GRU ({% cite cho-etal-2014-learning %}).
A Vanilla RNN, simply combines the state information from the previous timestamp with the input from the current timestamp to generate the
state information and output for current timestamp. A problem with Vanilla RNN is that training deep RNN networks is impossible due to the
**vanishing gradient** problem. Basically, weights/biases are updated according to the gradient of the loss functions relative to
the weights/biases. The gradients are calculated recursively from the output layer towards the input layer (hence, the name *backpropagation*).
The gradient of the input layer is the product of the gradient of the subsequent layers. If those gradients are small, the gradient of the input
layer (which is the product of multiple small values) will very small, resulting in very small updates to weights/biases of the initial layers
of the RNN, effectively halting the learning process.

LSTM and GRU are two RNN architectures that address vanishing gradient problem. Full description of LSTM/GRU is beyond the scope of this
tutorial (please refer to {% cite hochreiter1997long %} and {% cite cho-etal-2014-learning %}), but in a nutshell both LSTM and GRU use **gates** such that the weights/biases updates in previous
layers are calculated via a series of additions (not multiplications). Hence, these architectures can learn even when the RNN has hundreds or
thousands of layers.

# Text representation schemes

In this tutorial we perform sentiment analysis on IMDB (https://www.imdb.com/) movie reviews dataset ({% cite maas-EtAl %}). We train our RNN on
the training dataset, which is made up of 25,000 movie reviews, some positive and some negative. We then test our RNN on the test set, which is
also made up of 25,000 movie reviews, again some positive and some negative. The training and test sets have no overlap. Since we are dealing with
text data, it's a good idea to review various mechanisms for representing text data. Before that, we are going to briefly discuss how to preprocess
text documents.

## Text preprocessing

The first step is to tokenize a document, i.e., break it down into words. Next, we remove the punctuation marks, URLs, and stop words -- words like
'a', 'of', 'the', etc. that happen frequently in all documents and do not have much value in discriminating between documents. Next, we normalize
the text, e.g., replace 'brb' with 'Be right back', etc. Then, We then run the spell checker to fix typos and also make all words lowercase. Next, we perform stemming or lemmatization. Namely, if we have words like 'organizer', 'organize', 'organized', and 'organization' we want to reduce all of them to a single word. Stemming cuts the end of these words to come up with a single root (e.g., 'organiz'). The root may not be an actual word.
Meanwhile, lemmatization is smarter in that it reduces the word variants to a root that is actually a word (e.g., 'organize'). All of these steps help reduce
the number of features in feature vector of a document and should make the training of our model faster/easier.

For this introductory tutorial, we do minimal text preprocessing. We ignore the top 50 words in IMDB reviews (mostly stop words) and include
the next 10,000 words in our dataset. Reviews are limited to 500 words. They are trimmed if they are longer and padded if they are shorter.

## Bag of words and TF-IDF

If you don't care about the order of the words in a document, you can use bag of words (BoW) or term frequency inverse document frequency (TF-IDF).
In these models we have a 2 dimensional array. The rows represent the documents (in our example, the movie reviews) and the columns
represent the words in our vocabulary (all the unique words in all the documents). If a word is not present in a document, we have a zero
at the corresponding row and column as the entry. If a word is present in the document, we have a one as the entry -- Alternatively, we could use
the word count or frequency.

![Table showing a bag-of-words representation of sample documents](../../images/BoW.png "Bag of words (BoW) representation")

Suppose we have the following 2 documents: 1) Magic passed the basketball to Kareem, and 2) Lebron stole the basketball from Curry. The BoW
representation of these documents is given in Figure 10.

BoW's advantage is its simplicity, yet it does not take into account the rarity of a word across documents, which unlike common words are
important for document classification.

In TF-IDF, similar to BoW we have an entry for each document-word pair. In TD-IDF, the entry is the product of 1) term frequency, the
frequency of a word in a document, and 2) inverse document frequency, the inverse of the number of documents that have the word divided
by the total number of documents (we usually use logarithm of the IDF).

TF-IDF takes into account the rarity of a word across documents, and like BoW, it also does not capture word order or word meaning in documents. BoW
and TF-IDF are suitable representations for when word order is not important. They are used in document classification problems, like spam detection.

## One hot encoding (OHE)

OHE is a technique to convert categorical variables such as words into a vector. Suppose our vocabulary has 3 words: orange, apple, banana.
Each word for this vocabulary is represented by a vector of size 3. Orange is represented by a vector whose first element is 1 and other
elements are 0; apple is represented by a vector whose second element is 1 and other elements are 0; and banana is represented by a
vector whose third element is 1 and other elements are 0. As you can see only one element in the vector is 1 and the rest are 0's. The same
concept applies if the size of the vocabulary is N.

![Mathematical vectors representing one-hot-encoding representation of words orange, apple, and banana](../../images/OHE.gif "One hot encoding (OHE) representation")

The problem with OHE is that for very large vocabulary sizes (say, 100,000 words) it requires tremendous amount of storage. Also, it has no
concept of word similarity.

## Word2Vec

In Word2Vec, each word is represented as an *n* dimensional vector (*n* being much smaller than vocabulary size), such that the words that have
similar meanings are closer to each other in the vector space, and words that don't have a similar meaning are farther apart. Words are
considered to have a similar meaning if they co-occur often in documents. There are 2 Word2Vec architectures, one that predicts the probability
of a word given the surrounding words (continuous BoW), and one that given a word predicts the probability of the surrounding words (continuous
skip-gram).

In this tutorial, we find an *n* dimensional representation of the IMDB movie review words, not based on word meanings, but based on how they
improve the sentiment classification task. The *n* dimensional representation is learned by the learning algorithm, simply by reducing the
cost function via backpropagation.

# Get Data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://zenodo.org/record/4477881) and choose the type of data as `tabular`
>
>    ```
>    https://zenodo.org/record/4477881/files/X_test.tsv
>    https://zenodo.org/record/4477881/files/X_train.tsv
>    https://zenodo.org/record/4477881/files/y_test.tsv
>    https://zenodo.org/record/4477881/files/y_train.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. Rename the datasets as `X_test`, `X_train`, `y_test`, and `y_train` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype of all the four datasets is `tabular`. If not, change the dataset's datatype to tabular.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

# Sentiment Classification of IMDB movie reviews with RNN

In the section, we define a RNN and train it using IMDB movie reviews training dataset. The goal is to learn a model such that given the
words in a review we can predict whether the review was positive or negative. We then evaluate the trained RNN on the test dataset
and plot the confusion matrix.

## Create a deep learning model architecture

> <hands-on-title>Model config</hands-on-title>
>
> - {% tool [Create a deep learning model architecture](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/1.0.10.0) %}
>    - *"Select keras model type"*: `sequential`
>    - *"input_shape"*: `(500,)`
>    - In *"LAYER"*:
>        - {% icon param-repeat %} *"1: LAYER"*:
>            - *"Choose the type of layer"*: `Embedding -- Embedding`
>                - *"input_dim"*": `10000`
>                - *"output_dim"*": `32`
>        - {% icon param-repeat %} *"2: LAYER"*:
>            - *"Choose the type of layer"*: `Recurrent -- LSTM`
>                - *"units"*": `100`
>        - {% icon param-repeat %} *"3: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*: `1`
>                - *"Activation function"*: `sigmoid`
>    - Click *"Run Tool"*
{: .hands_on}

Input is a movie review of size 500 (longer reviews were trimmed and shorter ones padded). Our neural network has 3 layers. The first layer is
an embedding layer, that transforms each review words into a 32 dimensional vector (*output_dim*). We have 10,000 unique words in our IMDB dataset
(*input_dim*). The second layer is an *LSTM* layer, which is a type of RNN and it learns dependencies between time steps. We set the output size of the LSTM layer to *100*. The third layer is a
*Dense* layer, which is a fully connected layer (all 100 output neurons in LSTM layer are connected to a single neuron in this layer). It has a
*sigmoid* activation function, that generates an output between 0 and 1. Any output greater than 0.5 is considered a predicted positive review,
and anything less than 0.5 a negative one. The model config can be downloaded as a JSON file for future reuse.

## Create a deep learning model

> <hands-on-title>Model builder (Optimizer, loss function, and fit parameters)</hands-on-title>
>
> - {% tool [Create deep learning model](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/1.0.10.0) %}
>    - *"Choose a building mode"*: `Build a training model`
>    - *"Select the dataset containing model configuration"*: Select the *Keras Model Config* from the previous step.
>    - *"Do classification or regression?"*: `KerasGClassifier`
>    - In *"Compile Parameters"*:
>        - *"Select a loss function"*: `binary_crossentropy`
>        - *"Select an optimizer"*: `Adam - Adam optimizer `
>        - *"Select metrics"*: `acc/accuracy`
>    - In *"Fit Parameters"*:
>        - *"epochs"*: `2`
>        - *"batch_size"*: `128`
>    - Click *"Run Tool"*
{: .hands_on}

A loss function measures how different the predicted output is versus the expected output. For binary classification problems, we use
*binary cross entropy* as loss function. Epochs is the number of times the whole training data is used to train the model. Setting *epochs* to 2
means each training example in our dataset is used twice to train our model. If we update network weights/biases after all the training data is
feed to the network, the training will be very slow (as we have 25,000 training examples in our dataset). To speed up the training, we present
only a subset of the training examples to the network, after which we update the weights/biases. *batch_size* decides the size of this subset.
The model builder can be downloaded as a zip file.

## Deep learning training and evaluation

> <hands-on-title>Training the model</hands-on-title>
>
> - {% tool [Deep learning training and evaluation](toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.10.0) %}
>    - *"Select a scheme"*: `Train and Validate`
>    - *"Choose the dataset containing pipeline/estimator object"*: Select the *Keras Model Builder* from the previous step.
>    - *"Metrics for evaluation - Select the primary metric (scoring)"*: `Classification -- 'f1'`
>    - *"Select input type:"*: `tabular data`
>        - *"Training samples dataset"*: Select `X_train` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>        - *"Dataset containing class labels or target values"*: Select `y_train` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
>
{: .hands_on}

The training step generates 2 datasets. 1) accuracy of the trained model, 2) the trained model, in *h5mlm* format. These files are needed for prediction in the next step.

## Model Prediction

> <hands-on-title>Testing the model</hands-on-title>
>
> - {% tool [Model Prediction](toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.10.0) %}
>    - *"Choose the dataset containing pipeline/estimator object"* : Select the trained model from the previous step.
>    - *"Select invocation method"*: `predict`
>    - *"Select input data type for prediction"*: `tabular data`
>    - *"Training samples dataset"*: Select `X_test` dataset
>    - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
{: .hands_on}

The prediction step generates 1 dataset. It's a file that has predictions (1 or 0 for positive or negative movie reviews) for every review in
the test dataset.

## Machine Learning Visualization Extension

> <hands-on-title>Creating the confusion matrix</hands-on-title>
>
> - {% tool [Machine Learning Visualization Extension](toolshed.g2.bx.psu.edu/repos/bgruening/ml_visualization_ex/ml_visualization_ex/1.0.10.0) %}
>    - *"Select a plotting type"*: `Confusion matrix for classes`
>    - *"Select dataset containing the true labels"*": `y_test`
>    - *"Choose how to select data by column:"*: `All columns`
>    - *"Select dataset containing the predicted labels"*": Select `Model Prediction` from the previous step
>    - *"Does the dataset contain header:"*: `Yes`
>    - Click *"Run Tool"*
>
{: .hands_on}

**Confusion Matrix** is a table that describes the performance of a classification model. It lists the number of positive and negative examples
that were correctly classified by the model, true positives (TP) and true negatives (TN), respectively. It also lists the number of examples that
were classified as positive that were actually negative (false positive, FP, or type I error), and the number of examples that were classified
as negative that were actually positive (false negative, FN, or type 2 error). Given the confusion matrix, we can calculate **precision** and
**recall** {% cite TatbulEtAl  %}. Precision is the fraction of predicted positives that are true positives (precision = TP / (TP + FP)). Recall
is the fraction of true positives that are predicted (recall = TP / (TP + FN)). One way to describe the confusion matrix with just one value is
to use the **F score**, which is the harmonic mean of precision and recall.

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} $$

![Confusion matrix for our sentiment analysis problem](../../images/ConfusionMatrix.png "Sentiment analysis confusion matrix")

Figure 12 is the resultant confusion matrix for our sentiment analysis problem (note that your numbers in the matrix may differ, and that is expected). The top row in the figure represents the *true* 0 (or negative sentiment)
class labels (we have 10,397 + 2,103 = 12,500 reviews with negative sentiment). The bottom row represents the *true* 1 (or positive sentiment) class labels
(Again, we have 1,281 + 11,219 = 12,500 reviews with positive sentiment). The left column represents the *predicted* negative sentiment class labels (Our RNN
predicted 10,397 + 1,281 = 11,678 reviews as having a negative sentiment). The right column represents the *predicted* positive class labels (Our RNN
predicted 11,219 + 2,103 = 13,322 reviews as having a positive sentiment).Looking at the bottom right cell, we see that our RNN has correctly predicted 11,219
reviews as having a positive sentiment (true positives). Looking at the top right cell, we see that our RNN has incorrectly predicted 2,103 reviews as having
a positive (false positives). Similarly, looking at the top left cell, we see that our RNN has correctly predicted 10,397 reviews as having negative sentiment
(true negative). Finally, looking at the bottom left cell, we see that our RNN has incorrectly predicted 1,281 reviews as negative (false negative). Given
these numbers we can calculate Precision, Recall, and the F score as follows:

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} = \frac{11,219}{11,219 + 2,102} = 0.84 $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} = \frac{11,219}{11,219 + 1,281} = 0.89 $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} = \frac{2 * 0.84 * 0.89}{0.84 + 0.89} = 0.86 $$

# Conclusion


In this tutorial, we briefly reviewed feedforward neural networks, explained how recurrent neural networks are different, and discussed various
RNN input/outputs and architectures. We also discussed various text representation and preprocessing schemes and used Galaxy to solve a sentiment
classification problem using RNN on IMDB movie reviews dataset.

tools_json:
["Create a deep learning model architecture", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/1.0.10.0", "Create deep learning model", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/1.0.10.0", "Deep learning training and evaluation", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.10.0", "Model Prediction", "toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.10.0", "Machine Learning Visualization Extension", "toolshed.g2.bx.psu.edu/repos/bgruening/ml_visualization_ex/ml_visualization_ex/1.0.10.0"]


topic: statistics
tutorial: neural-networks-with-python
title: Neural networks using Python
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: ["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python"]
tags_json: ["elixir", "ai-ml"]
content_hash: d41d8cd98f00b204e9800998ecf8427e
last_modified: 2026-05-27T09:01:49.841500
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/neural-networks-with-python/tutorial.html


topic: metabolomics
tutorial: lcms
title: Mass spectrometry: LC-MS analysis
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: []
content_hash: 611afab3536cbadb68e446946efc04a3
last_modified: 2026-05-27T09:01:49.251738
zenodo_link: https://zenodo.org/record/3244991
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/lcms/tutorial.html

content:
You may already know that there are different types of *-omic* sciences; out of these, metabolomics is most closely related to phenotypes.
Metabolomics involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants. It  focuses on studying the very small molecules
which are called *metabolites*, to better understand matters linked to the metabolism. However, studying metabolites is not a piece of cake
since it requires several critical steps which still have some major bottlenecks. Metabolomics is still quite a young science, and has many
kinds of specific challenges.

One of the three main technologies used to perform metabolomic analysis is **Liquid-Chromatography Mass Spectrometry** (LC-MS). Data analysis
for this technology requires a large variety of steps, ranging from extracting information from the raw data, to statistical analysis
and annotation. To be able to perform a complete LC-MS analysis in a single environment, the [Wokflow4Metabolomics](http://workflow4metabolomics.org/)
team provides Galaxy tools dedicated to metabolomics. This tutorial explains the main steps involved in untargeted LC-MS data processing
for metabolomic analysis, and shows how to conduct metabolomic data analysis from preprocessing to annotation using Galaxy.

To illustrate this approach, we will use data from {% cite Thvenot2015 %}. The objectives of this paper was to analyze
the influence of age, body mass index, and gender on the urine metabolome. To do so, the authors collected samples
from 183 employees from the French Alternative Energies and Atomic Energy Commission (CEA) and performed LC-HRMS LTQ-Orbitrap
(negative ionization mode) analyses.

Since the original dataset takes a few hours to be processed, we chose to take a limited subset of individuals for this tutorial.
This will allow you to perform an example of metabolomic workflow, from pre-processing to annotation, in a limited time, even though
the results obtained may not be reliable from a scientific point of view due to the small sample size. Nevertheless,
the chosen diversity of sample will allow you to explore the basics of a metabolomic workflow.

We chose a subset of 9 samples, composed of 6 biological samples and 3 quality-control pooled samples (QC pools - mix of all
biological samples).

To analyze these data, we will then follow a light version of the [LC-MS workflow](http://workflow4metabolomics.org/the-lc-ms-workflow),
developed by the [Wokflow4metabolomics group](http://workflow4metabolomics.org/) ({% cite Giacomoni2014 %}, {% cite Guitton2017 %}).
The workflow is composed of 4 main parts:
  1. **Preprocessing** extracts ions from raw data
  2. **Data processing** checks the quality of data and transforms it to something relevant
  3. **Statistical Analysis** highlights interesting information inside the data
  4. **Annotation** puts a name on selected variables

[![The full tutorial workflow](../../images/tutorial-lcms-workflow.png)](../../images/tutorial-lcms-workflow.png)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Preprocessing with XCMS

The first step in the workflow is the pre-processing of the raw data with **XCMS** ({% cite Smith2006 %}).

**XCMS** {% icon tool %} is a free and open source software dedicated to pre-processing of any type of mass spectrometry acquisition files from low to
high resolution, including FT-MS data coupled with different kind of chromatography (liquid or gas). This software is
used worldwide by a huge community of specialists in metabolomics using mass spectrometry methods.

This software is based on different algorithms that have been published, and is provided and maintained using R software.

**MSnbase readMSData** {% icon tool %}, prior to **XCMS** {% icon tool %} is able to read files with open format as `mzXML`, `mzMl`, `mzData` and `netCDF`, which are independent of the constructors' formats. The **XCMS** {% icon tool %} package itself is composed of R functions able to extract, filter, align and fill gap, with the possibility to annotate isotopes,
adducts and fragments using the R package CAMERA ({% cite CAMERA %}). This set of functions gives modularity, and thus is particularly well
adapted to define workflows, one of the key points of Galaxy.


## Importing the LC/MS data into Galaxy

In metabolomics studies, the number of samples can vary a lot (from a handful to several hundreds). Thus, extracting your
data from the raw files can be very fast, or take quite a long time. To optimise the computation time as much as possible, the W4M core team chose to develop tools that can run single raw files for the first steps of
pre-processing in parallel, since the initial actions in the extraction process treat files independently.

Since the first steps can be run on each file independently, the use of **Dataset collections** in Galaxy is recommended, to avoid
having to launch jobs manually for each sample. You can start using the dataset collection option from the very beginning of your analysis, when uploading your data into Galaxy.

> <hands-on-title>Data upload the mzXML with <b>Get data</b></hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 9 `mzXML` files into a collection named `sacurine`
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from Zenodo using the URLs given below
>
>    ```
>    https://zenodo.org/record/3244991/files/HU_neg_048.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_090.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_123.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_157.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_173.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_192.mzML
>    https://zenodo.org/record/3244991/files/QC1_002.mzML
>    https://zenodo.org/record/3244991/files/QC1_008.mzML
>    https://zenodo.org/record/3244991/files/QC1_014.mzML
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="sacurine" renaming=false %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md astype="as a Collection" %}
>
> 3. Make sure your data is in a **collection**. Make sure it is named `sacurine`
>    - If you forgot to select the collection option during import, you can create the collection now:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
{: .hands_on}

You should have in your history a green Dataset collection (`sacurine`) with 9 datasets in mzml format.

Their size can be checked by clicking on the information icon {% icon galaxy-info %} on the individual datasets

## Data preparation for XCMS: *MSnbase readMSData*

This first step is only meant to read your `mzXML` file and generate an object usable by **XCMS** {% icon tool %}.

**MSnbase readMSData** {% icon tool %} takes as input your raw files and prepares `RData` files for the first XCMS step.

> <hands-on-title>MSnbase readMSData</hands-on-title>
>
> 1. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy2) %} with the following parameters:
>    - *"File(s) from your history containing your chromatograms"*: the `sacurine` dataset collection
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>
>  > <question-title></question-title>
>  >
>  > What do you get as output?
>  >
>  > > <solution-title></solution-title>
>  > >
>  > > 1. A **Dataset collection** containing 9 dataset.
>  > > 2. The datasets are some RData objects with the **rdata.msnbase.raw** datatype.
>  > >
>  > {: .solution}
>  >
>  {: .question}
{: .hands_on}

Now that you have prepared your data, you can begin with the first XCMS extraction step: peakpicking. However, before beginning to
extract meaningful information from your raw data, you may be interested in visualising your chromatograms. This can be of particular
interest if you want to check whether you should consider discarding some range of your analytical sequence (some scan or *retention
time* (RT) ranges).

To do so, you can use a tool that is called **xcms plot chromatogram** {% icon tool %} that will plot each sample's chromatogram (see dedicated section
further). However, to use this tool, you may need additional information about your samples for colouring purpose. Thus, you may need
to upload into Galaxy a table containing metadata of your samples (a *sampleMetadata* file).


## Importing a sample metadata file

What we referenced here as a *sampleMetadata* file corresponds to a table containing information about your samples (= sample metadata).

A sample metadata file contains various information for each of your raw files:
- **Classes** which will be used during the preprocessing steps
- **Analytical batches** which will be useful for a batch correction step, along with **sample types** (pool/sample) and **injection order**
- Different **experimental conditions** which can be used for statistics
- Any information about samples that you want to keep, in a *column* format

The content of your sample metadata file has to be filled by you, since it is not contained in your raw data.
Note that you can either:
- Upload an existing metadata file
- Use a template to create one (because it can be painful to get the sample list without misspelling or omission)
  1. Generate a template with the **xcms get a sampleMetadata file** {% icon tool %} tool
  2. Fill it using your favorite table editor (Excel, LibreOffice etc.)
  3. Upload it within Galaxy

> <tip-title>Optional: Generate the right template with <b>xcms get a sampleMetadata file</b></tip-title>
>
> In the case of this tutorial, we already prepared a *sampleMetadata* file with all the necessary information. Below is an optional hands-on explaining how to get a template to fill, with the two following advantages:
>
> 1. You will have the exact list of the samples you used in Galaxy, with the exact identifiers (*i.e.* exact sample names)
> 2. You will have a file with the right format (tabulation-separated text file) that only needs to be filled with the information you want.
>
> > <hands-on-title>xcms get a sampleMetadata file</hands-on-title>
> >
> > 1. {% tool [xcms get a sampleMetadata file](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy1) %} with the following parameters:
> >    - {% icon param-collection %} *"RData file"*: the `sacurine.raw.RData` collection output from **MSnbase readMSData** {% icon tool %}
> >
> {: .hands_on}
>
> An easy step for an easy sampleMetadata filling!
>
> From this tool, you will obtain a `tabular` file (meaning a tab-separated text file) with a first column of identifiers and a
> second column called *class* which is empty for the moment (only '.' for each sample). You can now download this file by clicking on the {% icon galaxy-save %} icon.
>
{: .tip}


### Prepare your sampleMetadata file

The sampleMetadata file is a tab-separated table, in text format. This table has to be filled by the user. You can use any
software you find appropriate to construct your table, as long as you save your file in a compatible format. For example, you can
use a spreadsheet software such as Microsoft Excel or LibreOffice.

> <warning-title>Important: Save your table in the correct format</warning-title>
>
> The file has to be a `.txt`or a `.tsv` (tab-separated values). Neither `.xlsx` nor `.odt` are supported.
> If you use a spreadsheet software, be sure to change the default format to **Text (Tab delimited)** or equivalent.
{: .warning}

Once your sampleMetadata table is ready, you can proceed to the upload. In this tutorial we already prepared the table for you ;)

> <tip-title>Optional: Filling the <i>sampleMetadata</i> using the template obtained from Galaxy</tip-title>
>
> For this tutorial, we already provide the *sampleMetadata* file, so you only have to upload it to Galaxy. Below we
> explain how we filled this file from the template we generated in Galaxy.
>
> First, we used **xcms get a sampleMetadata file** {% icon tool %} as mentioned in the previous tip box.
>
> We obtained the following table:
>
> sample_name | class
> --- | ---
> QC1_014 | .
> QC1_008 | .
> QC1_002 | .
> HU_neg_192 | .
> HU_neg_173 | .
> HU_neg_157 | .
> HU_neg_123 | .
> HU_neg_090 | .
> HU_neg_048 | .
>
> We used a spreadsheet software to open the file. First, we completed the class column. You will see in further XCMS steps that this
> second column matters.
>
> sample_name | class
> --- | ---
> QC1_014 | QC
> QC1_008 | QC
> QC1_002 | QC
> HU_neg_192 | sample
> HU_neg_173 | sample
> HU_neg_157 | sample
> HU_neg_123 | sample
> HU_neg_090 | sample
> HU_neg_048 | sample
>
> With this column, we will be able to colour the samples depending on the sample type (QC or sample).
> Next, we added columns with interesting or needed information, as following:
>
> sample_name | class | polarity | sampleType | injectionOrder | batch | osmolality | sampling | age | bmi | gender
> --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
> QC1_014 | QC | 0 | pool | 185 | ne1 | NA | NA | NA | NA | NA
> QC1_008 | QC | 0 | pool | 105 | ne1 | NA | NA | NA | NA | NA
> QC1_002 | QC | 0 | pool | 27 | ne1 | NA | NA | NA | NA | NA
> HU_neg_192 | sample | 0 | sample | 165 | ne1 | 1184 | 8 | 31 | 24.22 | Male
> HU_neg_173 | sample | 0 | sample | 148 | ne1 | 182 | 7 | 55 | 20.28 | Female
> HU_neg_157 | sample | 0 | sample | 137 | ne1 | 504 | 7 | 43 | 21.95 | Female
> HU_neg_123 | sample | 0 | sample | 100 | ne1 | 808 | 5 | 49 | 24.39 | Male
> HU_neg_090 | sample | 0 | sample | 75 | ne1 | 787 | 4 | 46 | 19.79 | Male
> HU_neg_048 | sample | 0 | sample | 39 | ne1 | 997 | 3 | 39 | 19.49 | Female
>
> In particular, the `batch`, `sampleType` and `injectionOrder` columns are mandatory to correct the data from signal drift (see later in
> the tutorial).
> Once we completed the table filling, we saved the file, minding to stick with the original format. Then, our *sampleMetadata* was ready to
> be uploaded into Galaxy.
>
{: .tip}


### Upload the sampleMetada file with 'Get data'

> <hands-on-title>Upload the sampleMetada</hands-on-title>
>
> 1. Import the `sampleMetadata_completed.tsv` file from Zenodo or from a shared data library (ask your instructor)
>    ```
>    https://zenodo.org/record/3244991/files/sampleMetadata_completed.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Check the data type of your imported files.
>    - The datatype should be `tabular`, if this is not the case, please change the datatype now
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
>    > <comment-title></comment-title>
>    >
>    > Here we provided the sampleMetadata file so we know that the upload led to a 'tabular' file. But from experience we also know that
>    > it can happen that, when uploading a sampleMetadata table, a user obtains other inappropriate types of data. This is generally due to the file
>    > not following all the requirements about the format (*e.g.* wrong separator, or lines with different numbers of columns).
>    > Thus, we highly recommend that you always take a second to check the data type after the upload. This way you can handle the problem
>    > right away if you happen to get one of these obvious issues.
>    {: .comment}
>
> 3. Rename your sampleMetadata file with a shorter name 'sampleMetadata_completed.tsv'
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many columns should I have in my sampleMetadata file?
> 2. What kind of class can I have?
>
> > <solution-title></solution-title>
> >
> > 1. At least 2, with the identifiers and the class column. But as many as you need to describe the potential variability of your samples
> >    (*e.g.* the person in charge of the sample preparation, the temperature...). The statistical analysis will expose the relevant parameters.
> > 2. Sample, QC, blank... The class (the 2nd column) is useful for the preprocessing step with XCMS to detect the metabolite across the samples.
> >    So it can be important to separate very different types of samples, as biological ones and blank ones for example. If you don't have any specific class
> >    that you want to consider in XCMS preprocessing, just fill everywhere with `sample` or a dot `.` for example.
> >
> {: .solution}
>
{: .question}


## Getting an overview of your samples' chromatograms

You may be interested in getting an overview of what your samples' chromatograms look like, for example to see if some of
your samples have distinct overall characteristics, *e.g.* unexpected chromatographic peaks or huge overall intensity.

You can use the *sampleMedata* file we previously uploaded to add some group colours to your samples when visualising your chromatograms.
The tool automatically takes the second column as colour groups when a file is provided.

Note that you can also check the chromatograms at any moment during the workflow, in particular at the following steps:
 - After **MSnbase readMSData** {% icon tool %} to help you to define retention time ranges that you may want to discard from the very beginning (*"Specta Filters"* in **findChromPeaks** {% icon tool %})
 - After **adjustRtime** {% icon tool %} to check the result of the correction (and potentially rerun *adjustRtime* with other settings)

> <hands-on-title>xcms plot chromatogram</hands-on-title>
>
> 1. {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `sacurine.raw.RData` (collection)
>    - *"Sample metadata file"*: `sampleMetadata_completed.tsv` you uploaded previously
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    > <comment-title></comment-title>
>    >
>    > If you use this tool at a later step of XCMS workflow while having provided, in the Merger step (see further in this tutorial), a sampleMetadata with a second column containing groups,
>    > you will get colouring according to these groups even without providing a sampleMetadata file as a 'plot chromatogram' parameter.
>    {: .comment}
>
{: .hands_on}

This tool generates Base Peak Intensity Chromatograms (BPIs) and Total Ion Chromatograms (TICs). If you provide groups as we do here, you obtain two plots:
one with colours based on provided groups, one with one colour per sample.

![Base Peak Intensity Chromatograms](../../images/lcms_BPC_9samp.png)


## First XCMS step: *peak picking*

Now that your data is ready for XCMS processing, the first step is to extract peaks from each of your data files
independently. The idea here is, for each peak, to proceed to chromatographic peak detection.

The XCMS solution provides two different algorithms to perform chromatographic peak detection: *matchedFilter* and
*centWave*. The matchedFilter strategy is the first one provided by the XCMS R package. It is compatible with any
LC-MS device, but was developed at a time when high resolution mass spectrometry was not common standard yet. On the
other side, the **centWave** algorithm ({% cite Tautenhahn2008 %}) was specifically developed for high resolution mass spectrometry, dedicated to
data in centroid mode. In this tutorial, you will practice using the centWave algorithm.


> <comment-title>How the centWave algorithm works</comment-title>
>
> Remember that these steps are performed for each of your data files independently.
>  - Firstly, the algorithm detects series of scans with close values of m/z. They are called 'region of interest' (ROI).
> The m/z deviation is defined by the user. The tolerance value should be set according to the mass spectrometer accuracy.
>  - On these regions of interest, a second derivative of a Gaussian model is applied to these consecutive scans in order to define
> the extracted ion chromatographic peak. The Gaussian model is defined by the peak width which corresponds to the standard deviation
> of the Gaussian model. Depending on the shape, the peak is added to the peak list of the current sample.
>
> At the end of the algorithm, a list of peaks is obtained for each sample. This list is then considered to represent the content
> of your sample; if an existing peak is not considered a peak at this step, then it can not be considered in the next steps of
> pre-processing.
{: .comment}

Let's try performing the peakpicking step with the **xcms findChromPeaks (xcmsSet)** {% icon tool %}


> <hands-on-title>xcms findChromPeaks (xcmsSet)</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy1) %} with the following parameters:
>   - *"RData file"*: `sacurine.raw.RData` (collection)
>   - *"Extraction method for peaks detection"*: `CentWave - chromatographic peak detection using the centWave method`
>     - *"Max tolerated ppm m/z deviation in consecutive scans in ppm"*: `3`
>     - *"Min,Max peak width in seconds"*: `5,20`
>     - In **Advanced Options**:
>       - *"Prefilter step for for the first analysis step (ROI detection)"*: `3,5000`
>       - *"Noise filter"*: `1000`
>
> You can leave the other parameters with their default values.
>
>    > <comment-title></comment-title>
>    >
>    > Along with the parameters used in the core centWave algorithm, XCMS provides other filtering options allowing you to get
>    > rid of ions that you don't want to consider. For example, you can use *Spectra Filters* allowing you to discard some RT or M/z
>    > ranges, or *Noise filter* (as in this hands-on) not to use low intensity measures in the ROI detection step.
>    {: .comment}
>
{: .hands_on}

At this step, you obtain a dataset collection containing one `RData` file per sample, with independent lists of ions. Although this
is already a nice result, what you may want now is to get all this files together to identify which ions are shared between samples.
To do so, XCMS provides a function that is called *groupChromPeaks* (or group). But before proceeding to this grouping step, first you
need to group your individual RData files into a single one.


## Gathering the different samples in one Rdata file

A dedicated tool exists to merge the different `RData` files into a single one: **xcms findChromPeaks Merger** {% icon tool %}. Although you can simply take as
input your dataset collection alone, the tool also provides de possibility to take into account a sampleMetadata file. Indeed,
depending of your analytical sequence, you may want to treat part of your samples a different way when proceeding to the grouping step using **xcms groupChromPeaks (group)** {% icon tool %}.

This can be the case for example if you have in your analytical sequence some blank samples (your injection solvent) that you want to
extract along with your biological samples to be able to use them as a reference for noise estimation and noise filtering. The fact that
these blank samples have different characteristics compared to your biological samples can be of importance when setting parameters of
your grouping step. You will see what this is all about in the 'grouping' section of this tutorial, but in the workflow order, it is
at this step that you need to provide the needed information if you want distinction in your grouping step.

**In the case of our tutorial data**, we do not want to separate the samples according to groups, so we do not provide the sampleMetadata when executing
the Merger tool.


> <hands-on-title>xcms findChromPeaks Merger</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `sacurine.raw.xset.RData` (collection)
>    - *"Sample metadata file"*: `Nothing selected`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
{: .hands_on}

The tool generates a single `RData` file containing information from all the samples in your dataset collection input.

## Second XCMS step: *determining shared ions across samples*

The first peak picking step gave us lists of ions for each sample. However, what we want now is a single matrix of ions intensities for all samples.
To obtain such a table, we need to determine, among the individual ion lists, which ions are the same. This is the aim of the present step, called
'grouping'.

The group function aligns ions extracted with close retention time and close m/z values in the different samples. In order to define this
similarity, we have to define on one hand a m/z window and on the other hand a retention time window. A binning is then performed in the
mass domain. The size of the bins is called width of overlapping m/z slices. You have to set it according to your mass spectrometer resolution.

Then, a kernel density estimator algorithm is used to detect region of retention time with high density of ions. This algorithm uses a Gaussian
model to group together peaks with similar retention time.

The inclusion of ions in a group is defined by the standard deviation of the Gaussian model, called bandwidth. This parameter has a large weight
on the resulting matrix. It must be chosen according to the quality of the chromatography. To be valid, the number of ions in a group must be greater
than a given number of samples. Either a percentage of the total number of samples or an absolute value of samples can be given. This is defined by the user.

> <hands-on-title>xcms groupChromPeaks (group)</hands-on-title>
>
> 1. {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy1) %} with the following parameters:
>   - *"RData file"*: `xset.merged.RData`
>   - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>     - *"Bandwidth"*: `5.0`
>     - *"Width of overlapping m/z slices"*: `0.01`
>
{: .hands_on}


This grouping step is very important because it defines the data matrix which will be used especially for the statistical analyses.
User has to check the effect of parameter values on the result.

In order to check the result of the grouping function, a pdf file is created. It provides one plot per m/z slice found in the data. Each picture
represents the peak density across samples, plotting the corresponding Gaussian model which width is defined by the bandwidth parameter. Each red
dot corresponds to a sample. The plot allows to assess the quality of alignment. The grey areas' width is associated with the bandwidth parameter.

Here is an example of two m/z slides obtained from the hands-on:

![plotChromPeakDensity.pdf](../../images/lcms_group_9samp.png)


> <question-title></question-title>
>
> 1. Look at the `283.1127 - 283.1163` m/z slice. How many peak groups are considered? Can you explain why some peaks are not affected to peak groups?
> 2. Look at the `284.1198 - 284.1253` m/z slice. What do you think could have happened if you had used a smaller bandwidth value?
>
> > <solution-title></solution-title>
> >
> > 1. There are 3 peak groups in this m/z slice. The two peaks that are not assigned to peak groups are alone in their retention time area. Thus,
> >    the number of samples under the corresponding density peaks does not reach the minimum fraction of samples set by the user (0.5) to consider a peak group.
> > 2. If the bandwidth value had been set to a smaller value, the density peak width would have been smaller. With a small-enough bandwidth value,
> >    there could have been two density peaks instead of one under the current first density peak. Thus, the sample in line 5 would have been out of the
> >    previous peak group, thus not assigned to any peak group due to the 0.5 minimum fraction limit.
> >
> {: .solution}
>
{: .question}

When looking at the plots from plotChromPeakDensity.pdf, we can notice that in some cases there seems to be a small drift of retention time for some samples.
This phenomenon is well known with LC-MS techniques. To be able to attribute correct groups for peaks, it may be needed to perform some retention time
correction accross samples. Thus, the idea is (when needed) to apply a retention time strategy on the output of your grouping step, then to perform
a second grouping step on the corrected data.

## Optional XCMS step: *retention time correction*

Sometimes with LC-MS techniques, a deviation in retention time occurs from a sample to another. In particular, this is likely to be observed when you
inject large sequences of samples.

This optional step aims to correct retention time drift for each peak among samples. The correction is based on what is called *well behaved peaks*,
that are peaks found in all samples or at least in most of the samples.

Sometimes it is difficult to find enough peaks present in all samples. The user can define a percentage of the total number of samples in which
a peak should be found to be considered a well behaved peak. This parameter is called *minimum required fraction of samples*.

On the contrary, you may have peak groups with more detected peaks than the total number of samples. Those peaks are called *additional peaks*.
If you do not want to consider peak groups with too much additional peaks as 'well behaved peaks', you can use the 'maximal number of additional
peaks' parameter to put them aside.

The algorithm uses statistical smoothing methods. You can choose between linear or loess regression.


> <hands-on-title>xcms adjustRtime (retcor)</hands-on-title>
>
> 1. {% tool [xcms adjustRtime (retcor)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.RData`
>    - *"Method to use for retention time correction"*: `PeakGroups - retention time correction based on aligment of features (peak groups) present in most/all samples.`
>        - *"Minimum required fraction of samples in which peaks for the peak group were identified"*: `0.8299`
>
> You can leave the other parameters to default values.
>
>    > <comment-title></comment-title>
>    >
>    > If you have a very large number of samples (*e.g.* a thousand), it might be impossible to find peaks that are present in 100% of your samples.
>    > If that is the case and you still set a very high value for the minimum required fraction of samples, the tool can not complete successfully the retention
>    > time correction. A special attention should also be given to this parameter when you expect a large number of peaks not to be present in part of your samples
>    > (*e.g.* when dealing with some blank samples).
>    {: .comment}
>
{: .hands_on}


This tool generates a plot output that you can use to visualise how retention time was applied across the samples and along the chromatogram.
It also allows you to check whether the well behaved peaks were distributed homogeneously along the chromatogram.

> <tip-title>Check the impact of RT correction using 'xcms plot chromatogram'</tip-title>
>
> Apart from the plots generated by the adjustRtime tool, you can check the impact of the retention time
> correction by comparing the chromatogram you obtained previously to a new one generated after correction.
>
> > <hands-on-title>xcms plot chromatogram</hands-on-title>
> >
> > 1. {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy1) %} with the following parameters:
> >    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
> >
> >    > <comment-title></comment-title>
> >    >
> >    > As in the previous 'plot chromatogram', you can use your completed sampleMetadata file to get colours.
> >    {: .comment}
> >
> {: .hands_on}
>
>
{: .tip}


The retention time correction step is not mandatory. However, when it is used retention time are modified.
Consequently, applying this step on your data requires to complete it with an additional 'grouping' step using the
**xcms groupChromPeaks (group)** {% icon tool %} tool again.

Parameters for this second group step are expected to be similar to the first group step. Nonetheless,
since retention times are supposed to be less variable inside a same peak group now, in some cases it can be relevant to
lower a little the bandwidth parameter.

> <hands-on-title>second 'xcms groupChromPeaks (group)'</hands-on-title>
>
> 1. {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `5.0`
>        - *"Width of overlapping m/z slices"*: `0.01`
>    - *"Get the Peak List"*: `Yes`
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>        - *"Replace the remain NA by 0 in the dataMatrix"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > When performing this second grouping (or at the first one if you do not plan to perform retention time correction),
>    > you can take this opportunity to check how your peak table looks like at this point of the XCMS extraction. For this, you can
>    > set the 'Get the Peak List' option to `Yes`.
>    {: .comment}
>
{: .hands_on}

It is possible to use the retention time correction and grouping step in an iterative way if needed. Once you perform your
last adjustRtime step and thus your last grouping step, you will obtain your final peak list (*i.e.* final list of ions).

> <question-title></question-title>
>
> 1. How many ions did you obtained with the final grouping step?
> 2. Open the dataMatrix file you obtained with the final grouping. This table corresponds to intensities for each ion and each
>    sample. What do you notice when looking at the intensity of the first ion regarding the first sample?
>
> > <solution-title></solution-title>
> >
> > 1. The final grouping step led to 5815 ions.
> > 2. The first ion (M58T69) has a 'NA' value for the first sample (QC1_014). This is also the case for several other ions and samples.
> >
> {: .solution}
>
{: .question}

At this point of the XCMS extraction workflow, the peak list may contain NA when peaks where not considered peaks in only some
of the samples in the first 'findChromPeaks' step. This does not necessary means that no peak exists for these samples. For example,
sometimes peaks are of very low intensity for some samples and were not kept as peaks because of that in the first 'findChromPeaks'
step.

To be able to get the information that may actually exist behind NAs, there is an additional XCMS step that is called *fillChromPeaks*.

> <comment-title></comment-title>
>
> Before performing the 'fillChromPeaks' step, it is highly recommended to first have a look at your data concerning the distribution
> of NAs in your data. Indeed, this will allow you to check whether your results are consistent with your expectations; if not you
> may want to go back to some of your parameter choices in previous XCMS steps.
> To perform your NA diagnosis, you can use the variableMetadata file and dataMatrix file that you obtained with the last grouping step
> with the 'Get the Peak List' option to `Yes`. The variableMetadata file contains information about your ions: you will find information
> about the number of peaks detected for each ion. The dataMatrix files contains the intensities for each ion and each sample; you can get
> an overview of it using the **Intensity Check** {% icon tool %} module.
{: .comment}


## Final XCMS step: *integrating areas of missing peaks*

The idea of the XCMS step is to integrate signal in the mz-rt area of an ion (chromatographic peak group) for samples in which no
chromatographic peak for this ion was identified.

> <hands-on-title>xcms fillChromPeaks (fillPeaks)</hands-on-title>
>
> 1. {% tool [xcms fillChromPeaks (fillPeaks)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.*.RData`
>    - In *"Peak List"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>
>    > <comment-title></comment-title>
>    >
>    > The *"Reported intensity values"* parameter is important here. It defines how the intensity will be computed. You have three choices:
>    > - into : integration of peaks (*i.e.* areas under the peaks)
>    > - maxo : maximum height of peaks
>    > - intb : integration of peaks with baseline subtraction
>    {: .comment}
>
{: .hands_on}

With this 'fillChromPeaks' step, you obtain your final intensity table. At this step, you have everything mandatory to begin analysing
your data:
 - A *sampleMetadata* file (if not done yet, to be completed with information about your samples)
 - A *dataMatrix* file (with the intensities)
 - A *variableMetadata* file (with information about ions such as retention times, m/z)

Nonetheless, before proceeding with the next step in the workflow (processing and filtering of your data), you can add an optional step with the
**CAMERA.annotate** {% icon tool %} module. This tool uses the CAMERA R package to perform a first annotation of your data based on XCMS outputs.


## Annotation with CAMERA [Optional]

This last step provides annotation of isotopes, adducts and neutral losses. It gives also some basic univariate statistics in case you
considered several groups for your XCMS extraction.

There is a huge number of parameters that will not be detailed in this short tutorial. However most of the default values can be kept here
for a first attempt to run this function. Nevertheless, a few parameters have to be set at each run:
 - The polarity has to be set since it affects annotation.
 - For statistical analysis, you have to define if you have two or more conditions to compare. These conditions had to be defined in the
 sample metadata uploaded previously and taken into account in the XCMS workflow.
 - You can define how many significant ions will be used for extracted ions chromatogram (EIC) plot. These plots will be included in a pdf file.

 Apart from the PDF file, the main three outcomes from **CAMERA.annotate** {% icon tool %} are three columns added in the variableMetadata file:
 - **isotopes:** the name says everything
 - **adduct:** same here; this column is filled only in the 'All functions' mode
 - **pcgroup:** this stands for Pearson's correlation group; it corresponds to groups of ions that match regarding retention time and intensity
 correlations, leading to think that maybe they could come from the same original metabolite.

> <hands-on-title>CAMERA.annotate</hands-on-title>
>
> 1. {% tool [CAMERA.annotate](toolshed.g2.bx.psu.edu/repos/lecorguille/camera_annotate/abims_CAMERA_annotateDiffreport/2.2.7+camera1.48.0-galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.*.fillChromPeaks.RData`
>    - In *"Annotate Isotopes [findIsotopes]"*:
>        - *"Max. ion charge"*: `2`
>    - *"Mode"*: `Only groupFWHM and findIsotopes functions [quick]`
>    - In *"Export options"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>
>    > <comment-title></comment-title>
>    >
>    > As said previously, there are quite a few parameters in this tool, some of them having very high impact on your annotations.
>    > In particular, the **Mode** parameter will influence a lot your results regarding pcgroups, and adducts (that will not be computed otherwise).
>    {: .comment}
>
{: .hands_on}

The information given by this tool is not mandatory for the next step of the metabolomic workflow. Commonly, annotation is considered a later
step in the pipeline, but since CAMERA uses the outputs of XCMS, if you want to use it you better do it at this step, allowing you to have the
corresponding information in your variableMetadata file for later use.


# Stopover: debriefing and preparation for next steps

At the end of the Preprocessing step, you should have three tabulation-separated tables:
 - A **sampleMetadata** file given and completed by the user
 - A **dataMatrix** file from XCMS.fillChromPeaks
 - A **variableMetadata** file from either XCMS.fillChromPeaks or CAMERA.annotate

Concerning the **sampleMetadata** file, for the next steps of the workflow, there are four columns that are mandatory to go through all the analysis:
 - `injectionOrder`: a numerical column of injection order
 - `sampleType`: specifies if a QC pool or a sample (coded `pool` or `sample`)
 - `batch`: a categorical column indicating the batches of analysis (if only one, must be a constant)
 - your variable(s) of interest: here we will consider as an example the Body Mass Index (`bmi`)

> <comment-title></comment-title>
>
> The preprocessing part of this analysis can be quite time-consuming, and already corresponds to quite a few number of steps, depending of your analysis.
> It can also generate several versions of your 3 tables, with only one of interest for each at the end of the extraction process. We highly recommend,
> at this step of the metabolomic workflow, to split your analysis by beginning a new Galaxy history with only the 3 tables you need. This will help
> you in limiting selecting the wrong dataset in further analyses, and bring a little tidiness for future review of your analysis process.
>
> To begin a new history with the 3 tables from your current history, you can use the functionality 'copy dataset' and copy it into a new history.
>
> We also recommend you to rename your 3 tables before proceeding with the next steps of the metabolomic workflow. Indeed, you may have notice that
> the XCMS tools generate output names that contain the different XCMS steps you used, allowing easy traceability while browsing your history.
> However, knowing that the next steps of analysis are also going to extend the 3 tables' names, if you keep the original names it will become very
> long and thus may reduce the names' readability. Hence, we highly recommend you to rename them with something short, *e.g.* 'sampleMetadata',
> 'variableMetadata' and 'dataMatrix', or anything not too long that you may find convenient.
{: .comment}




> <hands-on-title>Copying the 3 tables into a new history and renaming them</hands-on-title>
>
> 1. Create a new history with the 3 tables
>
>    {% snippet faqs/galaxy/histories_copy_dataset.md history_name="Sacurine Processing" %}
>
> 2. Rename the 3 tables with shorter names:
>    - `xset.merged.groupChromPeaks.adjustRtime.groupChromPeaks.fillpeaks.dataMatrix.tsv` -> **dataMatrix.tsv**
>    - `xset.merged.groupChromPeaks.*.fillChromPeaks.annotate.variableMetadata.tsv` -> **variableMetadata.tsv**
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Data processing: quality checks, normalisation, data filtering

In the previous step of the LC-MS workflow, you saw how to extract features from your acquisition files. This data is
shaped in a format allowing the use of various standard statistical methods. However, being able to perform a
statistical analysis does not mean necessarily being able to highlight relevant information. Indeed, data are often affected
by various sources of unwanted variability. It can limit the effectiveness of statistical methods, leading sometimes to
difficulties in revealing investigated effects. Identifying such variability can help analyzing your data at its full potential.

In this tutorial, we chose to limit the data processing to 3 steps:
 - overview of the variability in the data
 - signal drift correction
 - filtering of unreliable variables based on coefficients of variation

> <comment-title></comment-title>
> To get a little more information regarding data processing, do not hesitate to visit the usemetabo.oc platform:
> [Link to LC-MS processing step](https://usemetabo.org/courses/w4mlc-ms-processing)
{: .comment}


## Step 1: global variability in the data

Commonly, LC-MS analyses generate a significant number of variables (hundreds to thousands). Getting a complete view of
such dataset may not be an easy task, but getting a glimpse of it is possible using some common unsupervised multivariate
analysis. One of the most commonly used method is the **Principal Components Analysis** (PCA). You can get a basic PCA along with
over useful information using the **Quality Metrics** {% icon tool %} tool.

> <hands-on-title>Using <b>Quality Metrics</b> to get an overview of your data</hands-on-title>
>
> 1. {% tool [Quality Metrics](toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8) %} with the following parameters:
>    - *"Data matrix file"*: `dataMatrix.tsv`
>    - *"Sample metadata file"*: `sampleMetadata_completed.tsv`
>    - *"Variable metadata file"*: `variableMetadata.tsv`
>
{: .hands_on}

For a first overview of your data, you can focus on the graphical output of this tool: **Quality_Metrics_figure.pdf**.
It provides a variety of useful information:
 - Summary of the intensities in the dataMatrix file (information.txt file and plot top center paragraph)
 - View of these intensities with a color scale (plot bottom right panel)
 - 2-components PCA score plot to check for clusters or outliers (plot top left panel)
 - Sum of intensities per sample according to injection order to check the presence of signal drift or batch effect (plot top center panel)
 - Z-scores for intensity distribution and proportion of missing values (plot bottom left panel)
 - Ions' standard deviation (sd) and mean values (plot top right panel)

![Quality_Metrics_figure.pdf](../../images/lcms_QM_9samp_raw.png)

> <question-title>cross-referencing information</question-title>
>
> Look at the 'Quality_Metrics_figure.pdf' file.
> 1. Look at the proportion of ions with a pool coefficient of variation <30%. Knowing that pools are identical samples, and that
>    values with a CV > 30% can not be considered stables, what can you conclude about your dataset regarding the possibility to
>    compare intensities between samples?
> 2. Look at the sum of intensities per sample according to injection order. What major information do you observe on the plot?
>    Can this observation help you understand the CV results you just looked at?
> 3. Now look at the PCA plot. Can you explain the first component (t1)? Use the previous plot to help you.
>
> > <solution-title></solution-title>
> >
> > 1. You can read on the plot that *pool CV < 30%: 26%*, meaning that the pool values are stable only for a quarter of the ions
> >    in your dataset. If the pooled samples are not stable, this means that you can observe differences between samples even when
> >    there are no biological differences. Thus, comparing samples becomes difficult and has high risk of being unreliable.
> >    Consequently, with only a quarter of the ions being stable regarding pool intensities, performing statistical analyses on
> >    this full dataset would probably lead to unreliable results.
> > 2. We can see in the figure that the global intensity of samples seems to decrease with the injection order. In particular,
> >    the fact that the pooled samples' intensities decrease leads us to suspect a signal drift due to the clogging effect of successive
> >    injection of samples.
> >    This signal drift could be the reason why so many ions in the dataset led to high CV values for pools, since it prevents
> >    at least part of the ions to be stable regarding pools' intensities.
> > 3. If we look closely at the samples' identifiers on the plot, it seems that the lowest numbers in IDs are at the right side
> >    of the first component. Knowing that these numbers correspond to an order in the injection sequence, we can link it to the
> >    previous picture's samples. Then, what we can observe is that the order of samples in the first component of PCA from right to left
> >    corresponds approximately to the decreasing order of sums of intensities. Thus, we can conclude that the main variability in
> >    the dataset may be due to the signal drift.
> >
> {: .solution}
>
{: .question}

## Step 2: handling the signal drift observed all through the analytical sequence

It is known that when injecting successively a large number of samples, the system tends to get dirty, and this may cause a measure drift.
To prevent inability to catch signal anymore, in case of large injection series, the sequence is generally divided into several batches
and the source is cleaned between batches. Unfortunately, these signal drift and batch design can add significant variability in data,
making sample comparison complicated. In case data is impacted by these effects, it is highly recommended to normalise the data to get
rid of these unwanted variabilities.

In our case study, we saw that the data seemed to be affected by signal drift. Thus, we will use the **Batch_correction** {% icon tool %} tool
to get rid of it.

> <hands-on-title>Data normalisation</hands-on-title>
>
> 1. {% tool [Batch_correction](toolshed.g2.bx.psu.edu/repos/melpetera/batchcorrection/Batch_correction/3.0.0) %} with the following par

tools_json:
["MSnbase readMSData", "toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy2", "xcms get a sampleMetadata file", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy1", "xcms plot chromatogram", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy1", "xcms findChromPeaks (xcmsSet)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy1", "xcms findChromPeaks Merger", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy1", "xcms groupChromPeaks (group)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy1", "xcms adjustRtime (retcor)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy1", "xcms fillChromPeaks (fillPeaks)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy1", "CAMERA.annotate", "toolshed.g2.bx.psu.edu/repos/lecorguille/camera_annotate/abims_CAMERA_annotateDiffreport/2.2.7+camera1.48.0-galaxy1", "Quality Metrics", "toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8", "Batch_correction", "toolshed.g2.bx.psu.edu/repos/melpetera/batchcorrection/Batch_correction/3.0.0", "Generic_Filter", "toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01", "Univariate", "toolshed.g2.bx.psu.edu/repos/ethevenot/univariate/Univariate/2.2.4", "HMDB MS search", "toolshed.g2.bx.psu.edu/repos/fgiacomoni/hmdb_ms_search/wsdl_hmdb/1.7.6"]


topic: metabolomics
tutorial: lcms-preprocessing
title: Mass spectrometry: LC-MS preprocessing with XCMS
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: []
content_hash: b3f0d5326ae0a4c7d1054132b590ccc1
last_modified: 2026-05-15T22:28:13.787721
zenodo_link: https://zenodo.org/record/3757956
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html

content:
Metabolomics is a *-omic* science known for being one of the most closely related to phenotypes.
It involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants.
It focuses on studying the very small molecules which are called *metabolites*, to better understand matters linked to the metabolism.

One of the three main technologies used to perform metabolomic analyses is **Liquid-Chromatography Mass Spectrometry** (LC-MS). Data analysis
for this technology requires a large variety of steps, ranging from extracting information from the raw data to statistical analysis and annotation.
To be able to perform a complete LC-MS analysis in a single environment, the [Wokflow4Metabolomics](http://workflow4metabolomics.org/)
team provides Galaxy tools dedicated to metabolomics. This tutorial details the steps involved in the first part of untargeted LC-MS
data processing: extracting information from raw data to obtain what is called a peak table. This step is commonly refered to as
the preprocessing step. This tutorial will show you how to perform such a step using the Galaxy implementation of the XCMS software.

To illustrate this approach, we will use data from {% cite Thvenot2015 %}. The objectives of this paper was to analyze
the influence of age, body mass index, and gender on the urine metabolome. To do so, the authors collected samples
from 183 employees from the French Alternative Energies and Atomic Energy Commission (CEA) and performed LC-HRMS LTQ-Orbitrap
(negative ionization mode).

Since the original dataset takes a few hours to be processed, we chose to take a limited subset of individuals for this tutorial.
This will enable you to perform an example of metabolomic preprocessing workflow in a limited time, even though
the results obtained may not be reliable enough for biological interpretation due to the small sample size.
Nevertheless, the chosen diversity of sample will allow you to explore the basics of a preprocessing workflow.

We chose a subset of 12 samples, composed of 6 biological samples, 3 quality-control pooled samples ('QC pools' - mix of all
biological samples) and 3 blank samples ('blanks' - solvent injection).

To analyze these data, we will then follow a Galaxy workflow
developed by the [Wokflow4metabolomics group](http://workflow4metabolomics.org/) ({% cite Giacomoni2014 %}, {% cite Guitton2017 %}).

[![The full tutorial workflow represented as boxes linked with arrows, each box being labeled with the corresponding galaxy tool name](../../images/lcmspreproc_wf.png "The tutorial workflow")](../../images/lcmspreproc_wf.png)

> <comment-title>Workflow4Metabolomics public history</comment-title>
>
> This training material can be followed running it on any Galaxy instance holding the Galaxy modules needed.
> Nonetheless, if you happen to be a W4M user and do not want to run the hands-on yourself, please note that
> you can find the entire history in the 'published histories' section:
> [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms)
>
{: .comment}

The preprocessing steps presented in this tutorial are built around the solutions provided through the R package **XCMS**.

**XCMS** ({% cite Smith2006 %}) is a free and open source software dedicated to pre-processing of any type of mass spectrometry acquisition files from low to
high resolution, including FT-MS data coupled with different kind of chromatography (liquid or gas). This software is
used worldwide by a huge community of specialists in metabolomics using mass spectrometry methods.

This software is based on different algorithms that have been published, and is provided and maintained using R software.

Since sometimes a couple of pictures is worth a thousand words, you will find in the following slides some material to help
you understand how XCMS works:
[link to slides](../../tutorials/lcms-preprocessing/slides.html).
This document is refered to as "Check the next X slides" in the present training material.
As an example, [check the next 2 slides](../../tutorials/lcms-preprocessing/slides.html#raw-to-matrix)
for complementary material about XCMS R package.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data preparation for XCMS

The **XCMS** {% icon tool %} R package is composed of R functions able to extract, filter, align and fill gap, with the possibility to annotate isotopes,
adducts and fragments using the CAMERA R package ({% cite CAMERA %}). This set of functions gives modularity, and thus is particularly well
adapted to define workflows, one of the key points of Galaxy.

The first Galaxy module, **MSnbase readMSData** {% icon tool %}, is meant to read files with open format as `mzXML`, `mzMl`, `mzData` and `netCDF`,
which are independent of the constructors' formats.
First of all, you need to upload your data (that must be in one of the cited open formats) into Galaxy.


## Importing the LC-MS data into Galaxy

In metabolomics studies, the number of samples can vary a lot (from a handful to several hundreds). Thus, extracting your
data from the raw files can be very fast, or take quite a long time. To optimise the computation time as much as possible,
the W4M core team chose to develop tools that can run single raw files in parallel for the first steps of
pre-processing, since the initial actions in the extraction process treat files independently.

Since the first steps can be run on each file independently, the use of **Dataset collections** in Galaxy is recommended, to avoid
having to launch jobs manually for each sample. You can start using the dataset collection option from the very beginning of your analysis, when uploading your data into Galaxy.

> <hands-on-title>Data upload the mzXML with <b>Get data</b></hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 12 `mzXML` files into a collection named `mzML`
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from Zenodo using the URLs given below
>
>      [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3757956.svg)](https://doi.org/10.5281/zenodo.3757956)
>    ```
>    https://zenodo.org/record/3757956/files/Blanc05.mzML
>    https://zenodo.org/record/3757956/files/Blanc10.mzML
>    https://zenodo.org/record/3757956/files/Blanc16.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_048.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_090.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_123.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_157.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_173.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_192.mzML
>    https://zenodo.org/record/3757956/files/QC1_002.mzML
>    https://zenodo.org/record/3757956/files/QC1_008.mzML
>    https://zenodo.org/record/3757956/files/QC1_014.mzML
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="mzML" renaming=false %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md astype="as a Collection" %}
>
> 3. Make sure your data is in a **collection**. Make sure it is named `mzML`
>    - If you forgot to select the collection option during import, you can create the collection now:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > If you happen to be a W4M user, please note that you can find at the following link a ready-to-start history:
> > [GTN_LCMSpreprocessing_mzML](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingmzml).
> > We highly recommend to get started by importing this history.
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > this step corresponds to the dataset collection number 13.
> {: .tip}
>
{: .hands_on}

You should have in your history a green Dataset collection (`mzML`) with 12 datasets in mzml format.

Their size can be checked by clicking on the information icon {% icon galaxy-info %} on the individual datasets.

## First Galaxy module: *MSnbase readMSData*

This first step is only meant to read your `mzXML` file and generate an object usable by **XCMS** {% icon tool %}.

**MSnbase readMSData** {% icon tool %} takes as input your raw files and prepares `RData` files for the first XCMS step.

> <hands-on-title>MSnbase readMSData</hands-on-title>
>
> 1. Run {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} with the following parameters:
>    - *"File(s) from your history containing your chromatograms"*: the `mzML` dataset collection
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>  > <tip-title>Comment to W4M users</tip-title>
>  >
>  > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
>  > this step corresponds to the dataset collection number 14.
>  {: .tip}
>
>  > <question-title></question-title>
>  >
>  > What do you get as output?
>  >
>  > > <solution-title></solution-title>
>  > >
>  > > 1. A **Dataset collection** containing 12 datasets.
>  > > 2. The datasets are some RData objects with the **rdata.msnbase.raw** datatype.
>  > >
>  > {: .solution}
>  >
>  {: .question}
{: .hands_on}

Now that you have prepared your data, you can begin with the first XCMS extraction step: peakpicking. However, before beginning to
extract meaningful information from your raw data, you may be interested in visualising your chromatograms. This can be of particular
interest if you want to check whether you should consider discarding some range of your analytical sequence (some scans or *retention
time* (RT) ranges).

To do so, you can use a tool that is called **xcms plot chromatogram** {% icon tool %} that will plot each sample's chromatogram (see dedicated section
further). However, to use this tool, you may need additional information about your samples for colouring purpose. Thus, you may need
to upload into Galaxy a table containing metadata of your samples (a *sampleMetadata* file).


## Importing a sample metadata file

What we referenced here as a *sampleMetadata* file corresponds to a table containing information about your samples (= sample metadata).

A sample metadata file contains various information for each of your raw files:
- **Classes** which will be used during the preprocessing steps
- **Analytical batches** which will be useful for a batch correction step, along with **sample types** (pool/sample) and **injection order**
- Different **experimental conditions** which can be used for statistics
- Any information about samples that you want to keep, in a *column* format

The content of your sample metadata file has to be filled by you, since it is not contained in your raw data.
Note that you can either:
- Upload an existing metadata file
- Use a template to create one (because it can be painful to get the sample list without misspelling or omission)
  1. Generate a template with the **xcms get a sampleMetadata file** {% icon tool %} tool
  2. Fill it using your favorite table editor (Excel, LibreOffice)
  3. Upload it within Galaxy

> <tip-title>Optional: Generate the right template with <b>xcms get a sampleMetadata file</b></tip-title>
>
> In the case of this tutorial, we already prepared a *sampleMetadata* file with all the necessary information.
> Below is an optional hands-on explaining how to get a template to fill, with the two following advantages:
>   1. You will have the exact list of the samples you used in Galaxy, with the exact identifiers (*i.e.* exact sample names)
>   2. You will have a file with the right format (tabulation-separated text file) that only needs to be filled with the information you want.
>
> > <hands-on-title>xcms get a sampleMetadata file</hands-on-title>
> >
> > {% tool [xcms get a sampleMetadata file](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy0) %} with the following parameters:
> >    - {% icon param-collection %} *"RData file"*: the `mzML.raw.RData` collection output from **MSnbase readMSData** {% icon tool %}
> >
> > > <tip-title>Comment to W4M users</tip-title>
> > >
> > > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > > this step corresponds to the dataset number 27.
> > {: .tip}
> >
> {: .hands_on}
>
> An easy step for an easy sampleMetadata filling!
>
> From this tool, you will obtain a `tabular` file (meaning a tab-separated text file) with a first column of identifiers and a
> second column called *class* which is empty for the moment (only '.' for each sample). You can now download this file by clicking on the {% icon galaxy-save %} icon.
>
{: .tip}


### Prepare your sampleMetadata file

The sampleMetadata file is a tab-separated table, in text format. This table has to be filled by the user. You can use any
software you find appropriate to construct your table, as long as you save your file in a compatible format. For example, you can
use a spreadsheet software such as Microsoft Excel or LibreOffice.

> <warning-title> Save your table in the correct format</warning-title>
>
> The file has to be a `.txt` or a `.tsv` (tab-separated values). Neither `.xlsx` nor `.odt` are supported.
> If you use a spreadsheet software, be sure to change the default format to **Text (Tab delimited)** or equivalent.
{: .warning}

Once your sampleMetadata table is ready, you can proceed to the upload. In this tutorial we already prepared the table for you ;)

> <tip-title>Optional: Filling the <i>sampleMetadata</i> using the template obtained from Galaxy</tip-title>
>
> For this tutorial, we already provide the *sampleMetadata* file, so you only have to upload it to Galaxy. Below we
> explain how we filled this file from the template we generated in Galaxy.
>
> First, we used **xcms get a sampleMetadata file** {% icon tool %} as mentioned in the previous tip box.
>
> We obtained the following table:
>
> sample_name | class
> --- | ---
> QC1_014 | .
> QC1_008 | .
> QC1_002 | .
> HU_neg_192 | .
> HU_neg_173 | .
> HU_neg_157 | .
> HU_neg_123 | .
> HU_neg_090 | .
> HU_neg_048 | .
> Blanc16 | .
> Blanc10 | .
> Blanc05 | .
>
> We used a spreadsheet software to open the file. First, we completed the class column. You will see in further XCMS steps that this
> second column matters.
>
> sample_name | class
> --- | ---
> QC1_014 | QC
> QC1_008 | QC
> QC1_002 | QC
> HU_neg_192 | sample
> HU_neg_173 | sample
> HU_neg_157 | sample
> HU_neg_123 | sample
> HU_neg_090 | sample
> HU_neg_048 | sample
> Blanc16 | blk
> Blanc10 | blk
> Blanc05 | blk
>
> With this column, we will be able to colour the samples depending on the sample type (QC, sample, blk).
> Next, we added columns with interesting or needed information, as following:
>
> sample_name | class | sampleType | injectionOrder | batch | osmolality | sampling | age | bmi | gender
> --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
> QC1_014 | QC | pool | 185 | ne1 | NA | NA | NA | NA | NA
> QC1_008 | QC | pool | 105 | ne1 | NA | NA | NA | NA | NA
> QC1_002 | QC | pool | 27 | ne1 | NA | NA | NA | NA | NA
> HU_neg_192 | sample | sample | 165 | ne1 | 1184 | 8 | 31 | 24.22 | Male
> HU_neg_173 | sample | sample | 148 | ne1 | 182 | 7 | 55 | 20.28 | Female
> HU_neg_157 | sample | sample | 137 | ne1 | 504 | 7 | 43 | 21.95 | Female
> HU_neg_123 | sample | sample | 100 | ne1 | 808 | 5 | 49 | 24.39 | Male
> HU_neg_090 | sample | sample | 75 | ne1 | 787 | 4 | 46 | 19.79 | Male
> HU_neg_048 | sample | sample | 39 | ne1 | 997 | 3 | 39 | 19.49 | Female
> Blanc16 | blk | blank | 173 | ne1 | NA | NA | NA | NA | NA
> Blanc10 | blk | blank | 94 | ne1 | NA | NA | NA | NA | NA
> Blanc05 | blk | blank | 29 | ne1 | NA | NA | NA | NA | NA
>
> In particular, the `batch`, `sampleType` and `injectionOrder` columns are mandatory to correct the data from signal drift during later quality processing
> (see the [Mass spectrometry: LC-MS analysis](../../tutorials/lcms/tutorial.html)
> Galaxy training material for more information).
> Once we completed the table filling, we saved the file, minding to stick with the original format. Then, our *sampleMetadata* was ready to
> be uploaded into Galaxy.
>
{: .tip}

> <warning-title> The class column </warning-title>
>
> Depending on further choices, the sampleMetadata file can be decisive.
> It can be used to colour some plots, but also for ion selection (see further in the tutorial).
> Please note that the information needed for these steps **should be given as the second column of the sampleMetadata file**,
> the first one being the samples' identifiers.
{: .warning}


### Upload the sampleMetada file with 'Get data'

> <hands-on-title>Upload the sampleMetada</hands-on-title>
>
> 1. Import the `sampleMetadata_completed.tsv` file from your computer, from Zenodo or from a shared data library
>    ```
>    https://zenodo.org/record/3757956/files/sampleMetadata_12samp_completed.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Check the data type of your imported files.
>    - The datatype should be `tabular`, if this is not the case, please change the datatype now.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
>    > <comment-title></comment-title>
>    >
>    > Here we provided the sampleMetadata file so we know that the upload led to a 'tabular' file. But from experience we also know that
>    > it can happen that, when uploading a sampleMetadata table, user obtained other inappropriate types of data. This is generally due to the file
>    > not following all the requirements about the format (*e.g.* wrong separator, or lines with different numbers of columns).
>    > Thus, we highly recommend that you always take a second to check the data type after the upload. This way you can handle the problem
>    > right away if you appear to get one of these obvious issues.
>    {: .comment}
>
> 3. Rename your sampleMetadata file with a shorter name 'sampleMetadata_completed.tsv'
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > this step corresponds to the dataset number 28.
> {: .tip}
>
>
> > <question-title></question-title>
> >
> > 1. How many columns should I have in my sampleMetadata file?
> > 2. What kind of class can I have?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. At least 2, with the identifiers and the class column. But as many as you need to describe the potential variability of your samples
> > >    (*e.g.* the person in charge of the sample preparation, the temperature...). This will allow later statistical analysis to expose the relevant parameters.
> > > 2. Sample, QC, blank... The class (the 2nd column) is useful for the preprocessing step with XCMS to detect the metabolite across the samples.
> > >    Thus, it can be important to separate very different types of samples, as biological ones and blank ones for example. If you don't have any specific class
> > >    that you want to consider in XCMS preprocessing, just fill everywhere with `sample` or a dot `.` for example.
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

## Getting an overview of your samples' chromatograms

You may be interested in getting an overview of what your samples' chromatograms look like, for example to see if some of
your samples have distinct overall characteristics, *e.g.* unexpected chromatographic peaks or huge overall intensity.

You can use the *sampleMedata* file we previously uploaded to add some group colours to your samples when visualising your chromatograms.
The tool automatically takes the second column as colour groups when a file is provided.

Note that you can also check the chromatograms at any moment during the workflow, in particular at the following steps:
 - After **MSnbase readMSData** {% icon tool %} to help you to define retention time ranges that you may want to discard from the very beginning
(*"Spectra Filters"* section in **xcms findChromPeaks (xcmsSet)** {% icon tool %} parameters)
 - After **adjustRtime** {% icon tool %} to check the result of the correction (and potentially rerun *adjustRtime* with other settings)

> <hands-on-title>xcms plot chromatogram</hands-on-title>
>
> 1. {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `mzML.raw.RData` (collection)
>   - *"Sample metadata file"*: `sampleMetadata_completed.tsv` that you uploaded previously
>
>    > <comment-title></comment-title>
>    >
>    > If you use this tool at a later step of XCMS workflow and provided in the Merger step a sampleMetadata with a second column containing groups
>    > (see further in this tutorial), you will get colouring according to these groups even without providing a sampleMetadata file as a 'plot chromatogram' parameter.
>    {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > this step corresponds to the datasets number 29 and 30.
> {: .tip}
>
{: .hands_on}

This tool generates Base Peak Intensity Chromatograms (BPIs) and Total Ion Chromatograms (TICs). If you provide groups as we do here, you obtain two plots:
one with colours based on provided groups, one with one colour per sample.

![Screenshot of the PDF output corresponding to the Base Peak Intensity Chromatograms. It is composed of the layering of BPIs: one drawn line per sample.](../../images/lcmspreproc_BPC12samp.png "Base Peak Intensity Chromatograms")

How BPIs and TICs look like is dependant of the kind of data you have: LC-MS technology used, study design, type of samples, events that may have occured during the analysis...
It can vary a lot from one experiment to another due to these characteristics.
Sometimes you can see (un)expected effects on these plots as retention time variations accros samples, overall sample intensity differences due to analytical or biological
matters, specific peak areas... Generally, it is recommended to first focus on QC pooled samples if you have some, since they are not supposed to be affected by
biological variabilities: it helps you to focus on aspects uncorrelated to individual sample effects. Evaluation of blank samples (if any) can also be a good start:
"Are all blanks globally less intense than real samples?" is a question that can generally be raised.

# First XCMS step: *peak picking*

Now that your data is ready for XCMS processing, the first step is to extract peaks from each of your data files
independently. The idea here is, for each peak, to proceed to chromatographic peak detection.

The XCMS solution provides two different algorithms to perform chromatographic peak detection: *matchedFilter* and
*centWave*. The matchedFilter strategy is the first one provided by the XCMS R package. It is compatible with any
LC-MS device, but was developed at a time when high resolution mass spectrometry was not common standard yet. On the
other side, the **centWave** algorithm ({% cite Tautenhahn2008 %}) was specifically developed for high resolution mass spectrometry, dedicated to
data in centroid mode. In this tutorial, you will practice using the centWave algorithm.


> <comment-title>How the centWave algorithm works</comment-title>
>
> [Check the next 7 slides](../../tutorials/lcms-preprocessing/slides.html#findchrompeaks)
> to help you understand how the centWave algorithm works.
> Remember that these steps are performed for each of your data files independently.
>  - Firstly, the algorithm detects series of scans with close values of m/z. They are called 'region of interest' (ROI).
> The m/z deviation is defined by the user. The tolerance value should be set according to the mass spectrometer accuracy.
>  - On these regions of interest, a second derivative of a Gaussian model is applied to these consecutive scans in order to define
> the extracted ion chromatographic peak. The Gaussian model is defined by the peak width which corresponds to the standard deviation
> of the Gaussian model. Depending on the shape, the peak is added to the peak list of the current sample.
>
> At the end of the algorithm, a list of peaks is obtained for each sample. This list is then considered to represent the content
> of your sample; if an existing peak is not considered a peak at this step, then it can not be considered in the next steps of
> pre-processing.
>
{: .comment}

Let's try performing the peakpicking step with the **xcms findChromPeaks (xcmsSet)** {% icon tool %} tool.


> <hands-on-title>xcms findChromPeaks (xcmsSet)</hands-on-title>
>
> {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `mzML.raw.RData` (collection)
>   - *"Extraction method for peaks detection"*: `CentWave - chromatographic peak detection using the centWave method`
>     - *"Max tolerated ppm m/z deviation in consecutive scans in ppm"*: `3`
>     - *"Min,Max peak width in seconds"*: `5,20`
>     - In **Advanced Options**:
>       - *"Prefilter step for for the first analysis step (ROI detection)"*: `3,5000`
>       - *"Noise filter"*: `1000`
>
> You can leave the other parameters with their default values.
>
> > <comment-title></comment-title>
> >
> > Along with the parameters used in the core centWave algorithm, XCMS provides other filtering options allowing you to get
> > rid of ions that you don't want to consider. For example, you can use *Spectra Filters* allowing you to discard some RT or m/z
> > ranges, or *Noise filter* (as in this hands-on) not to use low intensity measures in the ROI detection step.
> {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the dataset collections number 31 and 32.
> {: .tip}
>
{: .hands_on}

At this step, you obtain a dataset collection containing one `RData` file per sample, with independent lists of ions. Although this
is already a nice result, what you may want now is to get all this files together to identify which are the shared ions between samples.
To do so, XCMS provides a function that is called *groupChromPeaks* (or group). But before proceeding to this grouping step, first you
need to group your individual RData files into a single one.

# Gathering the different samples in one Rdata file

A dedicated tool exists to merge the different `RData` files into a single one: **xcms findChromPeaks Merger** {% icon tool %}. Although you can simply take as
input your dataset collection alone, the tool also provides the possibility to take into account a sampleMetadata file. Indeed,
depending of your analytical sequence, you may want to treat part of your samples a different way when proceeding to
the grouping step using **xcms groupChromPeaks (group)** {% icon tool %}.

This can be the case for example if you have in your analytical sequence some blank samples (your injection solvent) that you want to
extract along with your biological samples to be able to use them as a reference for additional noise estimation and noise filtering. The fact that
these blank samples have different characteristics compared to your biological samples can be of importance when setting parameters of
your grouping step. You will see what this is all about in the 'grouping' section of this tutorial, but in the workflow order, it is
at this step that you need to provide the needed information if you want distinction in your grouping step.

In the case of our tutorial data, we have three sample types: the original biological samples ('samples'), quality-control pooled samples ('pools')
corresponding to a mix of all biological samples, and blank samples ('blanks') constituted of injection solvent.
We will consider these three classes for the grouping step. Thus, we need to provide the sampleMetadata file during the merging step,
with the second column defining theses classes.


> <hands-on-title>xcms findChromPeaks Merger</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `mzML.raw.xset.RData` (collection)
>   - *"Sample metadata file"*: `sampleMetadata_completed.tsv`
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the dataset number 57.
> {: .tip}
>
{: .hands_on}

The tool generates a single `RData` file containing information from all the samples in your dataset collection input.

# Second XCMS step: *determining shared ions across samples*

The first peak picking step gave us lists of ions for each sample. However, what we want now is a single matrix of ions' intensities for all samples.
To obtain such a table, we need to determine, among the individual ion lists, which ions are the same. This is the aim of the present step, called
'grouping'.

Various methods are available to proceed to the grouping step. In this tutorial, we will focus on the 'PeakDensity' method
that performs peak grouping based on time dimension peak densities.

[Check the next 8 slides](../../tutorials/lcms-preprocessing/slides.html#groupchrompeaks),
you will find additional material to help you understand the grouping algorithm.

The group function aligns ions extracted with close retention time and close m/z values in the different samples. In order to define this
similarity, we have to define on one hand a m/z window and on the other hand a retention time window. A binning is then performed in the
mass domain. The size of the bins is called width of overlapping m/z slices. You have to set it according to your mass spectrometer resolution.

Then, a kernel density estimator algorithm is used to detect region of retention time with high density of ions. This algorithm uses a Gaussian
model to group together peaks with similar retention time.

The inclusion of ions in a group is defined by the standard deviation of the Gaussian model, called bandwidth. This parameter has a large weight
on the resulting matrix. It must be chosen according to the quality of the chromatography. To be valid, the number of ions in a group must be greater
than a given number of samples. Either a percentage of the total number of samples or an absolute number of samples can be given. This is defined by the user.

> <hands-on-title>xcms groupChromPeaks (group)</hands-on-title>
>
> {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `xset.merged.RData`
>   - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>     - *"Bandwidth"*: `5.0`
>     - *"Minimum fraction of samples"*: `0.9`
>     - *"Width of overlapping m/z slices"*: `0.01`
>   - *"Get the Peak List"*: `Yes`
>     - *"If NA values remain, replace them by 0 in the dataMatrix"*: `No`
>
> > <comment-title>Minimum fraction of samples</comment-title>
> >
> > This parameter sets the minimum proportion of samples in a class where a peak should be found to keep the corresponding ion in the peaktable.
> > The idea is to look inside **each class (*i.e.* each group of samples defined in the second column of the sampleMetadata file)** and to keep
> > an ion if there is at least one class where the ion is found in at least the specified proportion of samples.
> >
> > The way to define what kind of classes would be relevant is not straightforward. It is a somehow complex combination of your study objectives,
> > the kind of biological matrix you analyse, assumptions you may have on the presence or absence of some compounds...
> > What could typically be considered is whether you have radically different kind of samples (distinct parts of an organism, distinct kind of
> > sample preparation...) and whether you expect huge composition differences between specific biological groups of individuals.
> > For example, if you study human blood samples in order to distinguish specificities between women and men, even if most of the metabolites
> > present may be expected to be find in both groups (at various levels but always present), some may be specific to one such as hormones.
> > Thus, you may consider putting men and women samples in distinct classes.
> >
> > In the Sacurine study, no specific a priori groups were defined. Thus, no specific assumption was taken into consideration regarding biological
> > classes. Thus, in this tutorial we considered only three classes: one for samples, one for pools and one for blanks.
> > Since pools are a mix of all biological samples,
> > we may want to consider an ion only if it is found in every pools. Thus, we may consider setting the 'Minimum fraction of samples' parameter to 1.
> > Considering that it sometimes happens that one sample can be a little out of the box for various reasons (pools do not make exception),
> > we may consider to lower a little this threshold, for example using a proportion of 0.9.
> >
> > Please note that in our example, since the dataset contains only 3 pools, this would make no difference to 1.0 concerning the 'pools' class.
> {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the datasets number 58 to 61.
> {: .tip}
>
{: .hands_on}


This grouping step is very important because it defines the data matrix which will be used especially for the statistical analyses.
User has to check the effect of parameter values on the result.

In order to check the result of the grouping function, a pdf file is created. It provides one plot per m/z slice found in the data. Each picture
represents the peak density across samples, plotting the corresponding Gaussian model which width is defined by the bandwidth parameter. Each dot
corresponds to a sample, with colours corresponding to defined classes if any. The plot allows to assess the quality of alignment.
The grey areas' width is associated with the bandwidth parameter.

Here is an example of two m/z slices obtained from the hands-on:

![Picture with two plots extracted from the plotChromPeakDensity.pdf output, one representing the 283.1127-283.1163 m/z slice and the other the 284.1198-284.1253 m/z slice](../../images/lcmspreproc_group12samp.png "Extract from plotChromPeakDensity.pdf")


> <question-title></question-title>
>
> 1. Look at the `283.1127 - 283.1163` m/z slice. How many peak groups are considered? Can you explain why some peaks are not affected to peak groups?
> 2. Look at the `284.1198 - 284.1253` m/z slice. What do you think could have happened if you had used a smaller bandwidth value?
>
> > <solution-title></solution-title>
> >
> > 1. There are 3 peak groups in this m/z slice. The two peaks that are not assigned to peak groups are alone in their retention time area. Thus,
> >    the number of samples under the corresponding density peaks does not reach the minimum fraction of samples set by the user (0.9) to consider a peak group,
> > in any specified class.
> > 2. If the bandwidth value had been set to a smaller value, the density peak width would have been smaller. With a small-enough bandwidth value,
> >    there could have been two density peaks instead of one under the current first density peak. Thus, the sample in line 5 would have been out of the
> >    previous peak group, thus not assigned to any peak group due to the 0.9 minimum fraction limit.
> >
> {: .solution}
>
{: .question}

When performing a grouping step, you construct a peak table. Even if this step may not be the final extraction step, you can take this opportunity
to check how your peak table looks at this point of the XCMS extraction. For this, you need to set the 'Get the Peak List' option to `Yes`
as done in this tutorial hands-on. This option generates two additional tables:
- a data matrix (xset.merged.group.dataMatrix.tsv) with intensities for each ion and each sample;
- a variable metadata file (xset.merged.group.variableMetadata.tsv) with information concerning the ions.

The variable metadata file contains various information that can be of interest:
- information about the m/z ratio of each ion (`mz`, `mzmin`, `mzmax` columns);
- information about the retention time of each ion (`rt`, `rtmin`, `rtmax` columns);
- the total number of peaks that were found for each ion (`npeaks` column);
- the number of samples in which each ion has been found, for each class (one column per class).

The data matrix contains the intensities for each ion and each sample. When no peak was found in a sample for a specific ion, value is given as NA
(you can choose to get a '0' value instead, using the 'If NA values remain, replace them by 0 in the dataMatrix' option). You can get a summary of NA proportions
using the **Intensity Check** {% icon tool %} module.

> <tip-title>Optional: Getting an overview of the proportion of NA in your data</tip-title>
>
> When you choose to export the peak list while using the **xcms groupChromPeaks (group)** {% icon tool %} module, you can have a first level of
> information regarding the proportion of NA by looking at the columns of classes you specified (or the `.` column when no class were given) in
> the variable metadata file ('variableMetadata').
> However, since the number of ions can be of hundreds or thousands, it can be difficult to evaluate the overall proportion in the dataset.
> Thus, one way to go is to use the variableMetadata to check specific ions (ones you may have chosen *a priori* and/or ones you spotted due to
> outstanding behaviour), and to use the **Intensity Check** {% icon tool %} module to get an overview of the whole dataset.
>
> > <hands-on-title>Intensity Check</hands-on-title>
> >
> > {% tool [Intensity Check](toolshed.g2.bx.psu.edu/repos/melpetera/intensity_checks/intens_check/1.2.8) %} with the following parameters:
> >   - *"Data matrix file"*: `xset.merged.group.dataMatrix.tsv`
> >   - *"Sample metadata file"*: `sampleMetadata_completed.tsv`
> >   - *"Variable metadata file"*: `xset.merged.group.variableMetadata.tsv`
> >   - *"Computation method"*: `For each class of samples`
> >     - *"Class column"*: `c2: class`
> >     - *"Statistics"*: tick `Missing values`
> >     - *"Calculate the mean fold change"*: tick `No`
> >
> > > <tip-title>Comment to W4M users</tip-title>
> > >
> > > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > > the datasets number 62 and 63.
> > {: .tip}
> >
> {: .hands_on}
>
> In the pdf file you will get as output, you can find a plot displaying proportions of NA for each class you specified.
> With the example provided in this tutorial, you can notice that the blk class containing blanks has a high number of NA with 4394 ions
> that have not been found, in any blank.
>
> ![Picture representing the proportions of NA in the 3 groups of the class column (blk, QC and sample). It is displayed as number of ions having given proportions of NA - proportions being grouped from 0 to 100 percents by classes of 20 percents.](../../images/lcmspreproc_ICgp1.png "Content of IC_Graphs.pdf")
>
> > <question-title></question-title>
> >
> > How many ions in the table have been found in all the pools?
> >
> > > <solution-title></solution-title>
> > >
> > > Knowing that there are 3 pools in the QC class, we can determine there are 4746 ions that have been found in all pools.
> > >
> > {: .solution}
> >
> {: .question}
>
{: .tip}

When looking back at the plots from plotChromPeakDensity.pdf, we can notice that in some cases there seems to be a small drift of retention time for some samples.
This phenomenon is well known with LC-MS techniques. To be able to attribute correct groups for peaks, it may be needed to perform some retention time
correction accross samples. Thus, the idea is (when needed) to apply a retention time strategy on the output of your grouping step, then to perform
a second grouping step on the corrected data.

# Optional XCMS step: *retention time correction*

Sometimes with LC-MS techniques, a deviation in retention time occurs from a sample to another. In particular, this is likely to be observed when you
inject large sequences of samples.

This optional step aims at correcting retention time drift for each peak among samples. The XCMS package provides two algorithms to do so.
[Check the next 4 slides](../../tutorials/lcms-preprocessing/slides.html#adjustrtime),
you will find additional material to help you understand the retention time correction algorithms.
In this training material we will focus on the "PeakGroups" method.

This correction is based on what is called *well behaved peaks*.
One characteristic of these peaks is that they are found in all samples or at least in most of the samples.
Sometimes it is difficult to find enough peaks present in all samples. The user can define a percentage of the total number of samples in which
a peak should be found to be considered a well behaved peak. This parameter is called *minimum required fraction of samples*.

On the contrary, you may have peak groups with more detected peaks than the total number of samples. Those peaks are called *additional peaks*.
If you do not want to consider peak groups with too much additional peaks as 'well behaved peaks', you can use the 'maximal number of additional
peaks' parameter to put them aside.

The algorithm uses statistical smoothing methods. You can choose between linear or loess regression.


> <hands-on-title>xcms adjustRtime (retcor)</hands-on-title>
>
> {% tool [xcms adjustRtime (retcor)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy0) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.RData`
>    - *"Method to use for retention time correction"*: `PeakGroups - retention time correction based on aligment of features (peak groups) present in most/all samples.`
>        - *"Minimum required fraction of samples in which peaks for the peak group were identified"*: `0.7`
>
> You can leave the other parameters to default values.
>
>    > <comment-title></comment-title>
>    >
>    > If you have a very large number of samples (*e.g.* a thousand), it might be impossible to find peaks that are present in 100% of your samples.
>    > If that is the case and you still set a very high value for the minimum required fraction of samples, the tool can not complete successfully the retention
>    > time correction.
>    > A special attention should also be given to this parameter when you expect a large number of peaks not to be present in part of your samples
>    > (*e.g.* when dealing with some blank samples).
>    {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the datasets number 64 and 65.
> {: .tip}
>
{: .hands_on}


This tool generates a plot output that you can use to visualise how retention time correction was applied across the samples and along the chromatogram.
It also allows you to check whether the well behaved peaks were distributed homogeneously along the chromatogram.

> <tip-title>Check the impact of RT correction using 'xcms plot chromatogram'</tip-title>
>
> Apart from the plots generated by the adjustRtime tool, you can check the impact of the retention time
> correction by comparing the chromatogram you obtained previously to a new one generated after correction.
>
> > <hands-on-title>xcms plot chromatogram</hands-on-title>
> >
> > {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy0) %} with the following parameters:
> >    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
> >
> >    > <comment-title></comment-title>
> >    >
> >    > As in the previous 'plot chromatogram', you can use your completed sampleMetadata file to get colours.
> >    {: .comment}
> >
> > > <tip-title>Comment to W4M users</tip-title>
> > >
> > > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > > the datasets number 66 and 67.
> > {: .tip}
> >
> {: .hands_on}
>
>
{: .tip}


The retention time correction step is not mandatory. However, when it is used retention time values are modified.
Consequently, applying this step on your data requires to complete it with an additional 'grouping' step using the
**xcms groupChromPeaks (group)** {% icon tool %} tool again.

Parameters for this second group step are expected to be similar to the first group step. Nonetheless,
since retention times are supposed to be less variable inside a same peak group now, in some cases it can be relevant to
lower a little the bandwidth parameter.

[Check the next slide](../../tutorials/lcms-preprocessing/slides.html#beforeafterrt)
to get an illustration of grouping before/after retention time correction.

> <hands-on-title>second 'xcms groupChromPeaks (group)'</hands-on-title>
>
> {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `5.0`
>        - *"Minimum fraction of samples"*: `0.9`
>        - *"Width of overlapping m/z slices"*: `0.01`
>    - *"Get the Peak List"*: `Yes`
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>        - *"If NA values remain, replace them by 0 in the dataMatrix"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > When performing this second grouping, similarly to the first grouping you can take this opportunity to check how your peak table
>    > looks at this point of the XCMS extraction, setting the 'Get the Peak List' option to `Yes`. As previously explained, you can
>    > look at your variableMetadata file as well as perform an NA diagnostic using the **Intensity Check** {% icon tool %} module.
>    {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the datasets from number 68 to number 71.
> {: .tip}
>
{: .hands_on}

It is possible to use the retention time correction and grouping step in an iterative way if needed. Once you perform your
last adjustRtime step and thus your last grouping step, you will obtain your final peak list (*i.e.* final list of ions).

> <question-title></question-title>
>
> 1. How many ions did you obtained with the final grouping step?
> 2. Open the dataMatrix file you obtained with the final grouping. This table corresponds to intensities for each ion and each
>    sample. What do you notice when looking at the intensity of the fourth ion regarding the first sample?
>
> > <solution-title></solution-title>
> >
> > 1. The final grouping step led to 5100 ions.
> > 2. The fourth ion (M74T317) has a 'NA' value for the first sample (QC1_014). This is also the case for several other ions
> >    and samples.
> >
> {: .solution}
>
{: .question}

At this point of the XCMS extraction workflow, the peak list may contain NA when peaks were not considered peaks in only some
of the samples in the first 'findChromPeaks' step. This does not necessary means that no peak exists for these samples. For example,
sometimes peaks are of very low intensity for some samples and were not kept as peaks because of that in the first 'findChromPeaks'
step.

To be able to get the information that may actually exist behind NAs, there is an additional XCMS step that is called *fillChromPeaks*.

> <comment-title></comment-title>
>
> Before performing the 'fillChromPeaks' step, it is highly recommended to first have a look at your data concerning the distribution
> of NAs in your data. Indeed, this will allow you to check whether your results are consistent with your expectations; if not you
> may want to go back to some of your parameter choices in previous XCMS steps.
> To perform your NA diagnosis, you can use the variableMetadata file and dataMatrix file that you obtained with the last grouping step
> with the 'Get the Peak List' option to `Yes`. The variableMetadata file contains information about your ions: you will find information
> about the number of peaks detected for each ion. The dataMatrix files contains the intensities

tools_json:
["MSnbase readMSData", "toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0", "xcms get a sampleMetadata file", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy0", "xcms plot chromatogram", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy0", "xcms findChromPeaks (xcmsSet)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0", "xcms findChromPeaks Merger", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0", "xcms groupChromPeaks (group)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0", "Intensity Check", "toolshed.g2.bx.psu.edu/repos/melpetera/intensity_checks/intens_check/1.2.8", "xcms adjustRtime (retcor)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy0", "xcms fillChromPeaks (fillPeaks)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy0", "CAMERA.annotate", "toolshed.g2.bx.psu.edu/repos/lecorguille/camera_annotate/abims_CAMERA_annotateDiffreport/2.2.6+camera1.48.0-galaxy0", "xcms process history", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_summary/abims_xcms_summary/3.12.0+galaxy0", "ID choice", "toolshed.g2.bx.psu.edu/repos/melpetera/idchoice/idchoice/19.12"]


topic: metabolomics
tutorial: msi-analyte-distribution
title: Mass spectrometry imaging: Examining the spatial distribution of analytes
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: 203d83071aaf6af63fb2baa7be2268ee
last_modified: 2026-05-15T22:28:13.789097
zenodo_link: https://zenodo.org/record/484496
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/msi-analyte-distribution/tutorial.html

content:
Mass spectrometry imaging (MSI) is applied to measure the spatial distribution of hundreds of biomolecules in a sample. A mass spectrometer scans over the entire sample and collects a mass spectrum every 5-200 µm. This results in thousands of spots (or pixels) for each of which a mass spectrum is acquired. Each mass spectrum consists of hundreds of analytes that are measured by their mass-to-charge (m/z) ratio. For each analyte the peak intensity in the mass spectra of every pixel is known and can be set together to map the spatial distribution of the analyte in the sample.

The technique has a broad range of applications as it is able to measure many different kinds of analytes such as peptides, proteins, metabolites or chemical compounds in a large variety of samples such as cells, tissues and liquid biopsies. Application areas include pharmacokinetic studies, biomarker discovery, molecular pathology, forensic studies, plant research and material sciences. The strength of MSI is the simultaneous analysis of hundreds of analytes in an unbiased, untargeted, label-free, fast and affordable measurement while maintaining morphological information.

Depending on the analyte of interest and the application, different mass spectrometers are used. A mass spectrometer measures the analytes by ionizing, evaporating and sorting them by their mass-to-charge (m/z) ratio. Put simply, a mass spectrometer consists basically of three parts: an ionization source, a mass analyzer and a detector. The most common ionization sources for MSI are MALDI (Matrix Assisted Laser Desorption/Ionization), DESI (Desorption Electrospray Ionization) and SIMS (Secondary Ion Mass Spectrometry).

![MSI measurement](../../images/msi_distribution_maldi_tof_imaging.png "MALDI TOF imaging of a mouse kidney")

One common type of mass spectrometer for MSI is a MALDI Time-Of-Flight (MALDI-TOF) device. During MALDI ionization, a laser is fired onto the sample, which has been covered with a special matrix that absorbs the laser energy and transfers it to the analytes. This process vaporizes and ionizes the analytes. As they are now charged, they can be accelerated in an electric field towards the TOF tube. The time of flight through the tube to the detector is measured, which allows calculation of the mass over charge (m/z) of the analyte, as both mass and charge are correlated with time of flight. During measurement, complete mass spectra with hundreds of m/z - intensity pairs are acquired in thousands of sample plots, leading to large and complex datasets. Each mass spectrum is annotated with coordinates (x,y) that define its location in the sample. This allows visualization of the intensity distribution of each m/z feature in the sample as a heatmap.

Depending on the analyte of interest, the sample type and the mass spectrometer, the sample preparation steps as well as the properties of the acquired data vary. Apart from these differences, the preparation and measurement of the sample is normally straightforward, while the analysis of the large and complex dataset is the main bottleneck of a MSI experiment.

In this tutorial we will determine analytes with a localized distribution in a chilli cross section. The data analysis steps can be transferred to any other application in which the morphological distribution of an analyte in a sample is analyzed. A common application is studying the distribution of a drug and its metabolites in an animal model to better understand pharmacokinetic properties of the drug.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Chilli dataset

In this tutorial we will use the chilli dataset generated in the lab of Robert Winkler at the CINVESTAV institute in Irapuato (Mexico) and deposited at [Zenodo](https://zenodo.org/record/484496). The data was acquired to demonstrate their self-made low-cost 'plug and play' MSI system as well as their open source R scripts and to show they allow the direct spatially resolved detection of volatile organic compounds (VOCs) from plant tissue ({% cite Maldonado_Torres_2014 %} and {% cite Gamboa_Becerra_2015 %}).

The mass spectrometer consisted of a low-temperature plasma (LTP) ionization source coupled to a quadrupole mass analyzer. This ionization source allows measurement of semi-volatile organic compunds directly from fresh material under ambient conditions without destruction of the sample. To get an idea how this set up and the chilli looks, have a look at their [video](https://www.youtube.com/watch?v=NbZ1QqTZvXM) and [website](http://lababi.bioprocess.org/). The chilli (*Capsicum annuum*) was purchased from a local market in Irapuato, Mexico and longitudinally cut into a slice of 80 x 35 mm and 4 mm thickness and laid onto a glass slide. Mass spectra were acquired with a spatial resolution of 1 mm and in a mass range from 15 – 2000 m/z to measure semi-volatile and volatile organic compounds (VOCs).

We will check the properties of the dataset and then perform a similar analysis to that described by {% cite Gamboa_Becerra_2015 %}: plotting an average mass spectrum, filtering the dataset for the relevant m/z range, automatical generation of m/z images to find analytes that correspond to morphological features and generating an image that overlays three different analytes.

![Chilli](../../images/msi_distribution_chilli.png "Cross section of a chilli in a similar orientation as in the publication by Gamboa-Becerra")

## Get data

The imzML composite file consists of two files. The first file contains the metadata in an XML file and has the extension .imzML. The second file contains the mass spectra data, is saved as a binary file and has the extension .ibd. To be valid, both files must have the same filename before the extension. More information about the imzML file structure can be found here: [ms-imaging.org](https://ms-imaging.org/wp/imzml/data-structure/). Galaxy provides a composite upload option for files consisting of several components.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. **Create a new history** and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Upload the data from [Zenodo](https://zenodo.org/record/484496) via the **composite** option
>
>    > <tip-title>Upload via the composite option</tip-title>
>    > - Open the Galaxy Upload Manager({% icon galaxy-upload %} on the top-right of the tool panel)
>    > - Click on **Composite** on the top
>    > - Set **Composite Type** to `imzml`
>    > - Expand the first **Select** button (for the imzML metadata component)
>    > - Select **Paste/Fetch data** and paste
>    >    ```
>    > https://zenodo.org/record/484496/files/ltpmsi-chilli.imzML
>    >    ```
>    > - Expand the second **Select** button (for the mass spectral data component)
>    > - Select **Paste/Fetch data** and paste
>    >    ```
>    > https://zenodo.org/record/484496/files/ltpmsi-chilli.ibd
>    >    ```
>    > - Press **Start**
>    > - **Close** the window
>    {: .tip}
>
>    ![Upload imzML files](../../images/msi_distribution_upload_imzml.png "Upload an imzML file via the composite upload")
>
>    > <tip-title>FTP upload for large files</tip-title>
>    > * In case one subfile is larger than 2 GB the uploading needs to be done via ftp.
>    > * The necessary steps are explained in this tutorial [Getting data into Galaxy]({{ site.baseurl }}/topics/galaxy-interface/tutorials/get-data/slides.html)
>    {: .tip}
>
>    > <tip-title>Uploading an Analyze7.5 file</tip-title>
>    > * `Analyze7.5` files are also supported by Galaxy.
>    > * The file consists of three components and is therefore uploaded via the 'composite' function, analogously to the imzML upload.
>    > * The files to select in the `composite` tab are the header file `.hdr`, the m/z values file `.t2m` and the spectra file `.img`.
>    {: .tip}
>
> 3. Rename the data into `ltpmsi-chilli`
>
>    > <tip-title>Rename a dataset</tip-title>
>    > - Click on the {% icon galaxy-pencil %} **pencil icon** for the dataset to edit its attributes
>    > - In the central panel, change the **Name** field to `ltpmsi-chilli`
>    > - Click the **Save** button
>    {: .tip}
{: .hands_on}


# Exploring and filtering the chilli dataset

## Quality control of the data

Before starting any analysis it is important to check the characteristics and quality of the MSI data. The MSI quality control tool creates a comprehensive PDF report that contains multiple descriptive visualizations of different data attributes. Here we will use the MSI quality control to get an idea about the m/z range and the shape of the chilli section. More details about the MSI quality control tool can be found in the tutorial [Mass spectrometry imaging 1: Loading and exploring MSI data]({{site.baseurl}}/topics/proteomics/tutorials/mass-spectrometry-imaging-loading-exploring-data/tutorial.html).

> <hands-on-title>Quality control </hands-on-title>
>1. {% tool [MSI_Qualitycontrol](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_quality_report/cardinal_quality_report/2.6.0.1) %}
>    - {% icon param-file %} *"MSI data"*: `ltpmsi-chilli.imzML` (Input dataset)
>    - *"Centroided input"*: `yes`
>    - *"Processed imzML file"*: `yes`
>        - *"Mass accuracy to which the m/z values will be binned"*: `0.2`
>        - *"Unit of the mass accuracy"*: `mz`
>    - *"PCA with 2 components"*: `Yes`
>    - Press **Run Tool**
>
>    > <comment-title>Properties of the imzML file</comment-title>
>    > To set the parameters for the mass spectrometry imaging tools correctly the following three parameters should be known about the dataset: 1) is the imzML file type processed or continuous, 2) are the spectra in profile or centroided mode and 3) the accuracy of the mass spectrometer. For the chilli dataset the publication states that it is a processed imzML type in centroided mode. This information can also be extracted by opening the local imzML component of the file in a text editor or web browser. Line 10 and 11 state:
>    >
>    >      <cvParam cvRef="MS" accession="MS:1000128" name="profile spectrum" value=""/>
>    >      <cvParam cvRef="IMS" accession="IMS:1000031" name="processed" value=""/>
>    >
>    > Starting from line 105 the "run" and "spectrumList" elements describe the acquired spectra in more detail:
>    >
>    >      <run defaultInstrumentConfigurationRef="IC1" id="_x0031_30704_IMGCHJ2">
>    >      <spectrumList count="4166" defaultDataProcessingRef="pwiz_Reader_conversion">
>    >        <spectrum defaultArrayLength="11127" id="controllerType=0 controllerNumber=1 scan=1" index="0" dataProcessingRef="pwiz_Reader_conversion">
>    >          <cvParam cvRef="MS" accession="MS:1000511" name="ms level" value="1"/>
>    >          <cvParam cvRef="MS" accession="MS:1000579" name="MS1 spectrum" value=""/>
>    >          <cvParam cvRef="MS" accession="MS:1000127" name="centroid spectrum" value=""/>
>    >          <cvParam cvRef="MS" accession="MS:1000528" name="lowest observed m/z" value="15"/>
>    >          <cvParam cvRef="MS" accession="MS:1000527" name="highest observed m/z" value="2000"/>
>    >          <cvParam cvRef="MS" accession="MS:1000504" name="base peak m/z" value="80.0377"/>
>    >          <cvParam cvRef="MS" accession="MS:1000505" name="base peak intensity" value="3.72813e+006"/>
>    >          <cvParam cvRef="MS" accession="MS:1000285" name="total ion current" value="2.42458e+008"/>
>    >
>    > This means that the file consists of 4166 spectra (count="4166") and that the imzML type is 'processed' as the first spectrum (scan="1") has 11127 m/z values (defaultArrayLength="11127) while the following spectra (not shown here) have different numbers of m/z values. In processed imzML files each spectrum has an individual m/z axis, in contrast to the continuous imzML type, where all spectra have the same m/z axis. In many software tools processed imzML files are not or only partly supported. The MSI tools in Galaxy are based on [Cardinal](http://cardinalmsi.org/) and therefore support imzML files, but require a binning of the m/z values while reading the file. The bin size should be chosen according to the m/z accuracy of the mass spectrometer. For the chilli dataset we therefore use 0.1 m/z, as this results in m/z bins of 0.2 m/z which corresponds to the m/z step size that was used in the publication.
>    >
>    > Regarding the spectra, line 10 states that the spectra are in profile mode while the 'run' section defines spectrum 1 as a centroided spectrum. As the 'run' section is directly linked to the spectra in the ibd file, this is the information to trust. Furthermore, the 'dataProcessingList' element gives a hint that peak picking was performed, which is confirmed in the publication.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. In which m/z range were the data acquired?
> 2. How many spectra (pixels) were measured?
> 3. In which images can you guess the shape of the chilli?
>
> > <solution-title></solution-title>
> >
> > 1. The measured m/z range was m/z 15 - 2000.
> > 2. 4166 spectra were measured.
> > 3. The rough shape of the chilli section is visible in the "Number of peaks per spectrum"; "Total Ion Current" and "PCA result" images.
> >
> {: .solution}
>
{: .question}

Open the quality report with the eye button and check the summary table on the first page to answer question 1 and 2. The shape of the chilli can roughly be seen in the images showing the number of peaks per spectrum and the total ion current, with higher values in the middle part of the fruit that roughly corresponds to the placenta. In the PCA image the complete chilli, except for the lowest part of the fruit, is visible in light colours compared to the dark background.

To further investigate mass spectra of different chilli compartments we select one pixel for each tissue area (pericarp, placenta and seeds) from the total ion current image. Knowing the exact shape of the chilli tissue from Figure 4 of the publication helps to find the corresponding areas. This are the pixels we have chosen for the tutorial:

seeds: x=39 y=53, placenta: x=50 y=44, pericarp: x=25 y=60

The relevant m/z range for VOCs was not known before the measurement and therefore chosen quite liberally. The average mass spectra plots give a hint about the relevant m/z range of the sample with most peaks below 750 m/z and no distinct peaks above 750 m/z.

![quality report](../../images/msi_distribution_qc.png "Example plots from the quality report")


## Plotting average mass spectra

We will follow up on the average mass spectra plots from the quality control report as well as on the differences between mass spectra from different chilli compartments. First, we generate more zoomed in mass spectra plots to get an idea about the m/z range that is relevant for VOCs. Next, we will plot and compare mass spectra that derive from different regions of the chilli.

> <hands-on-title>Average mass spectra</hands-on-title>
>
> 1. {% tool [MSI plot spectra](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_spectra_plots/cardinal_spectra_plots/2.6.0.3) %}
>    - {% icon param-file %} *"MSI data"*: `ltpmsi-chilli.imzML` (Input dataset)
>    - *"Centroided input"*: `yes`
>    - *"Processed imzML file"*: `yes`
>        - *"Mass accuracy to which the m/z values will be binned"*: `0.2`
>        - *"Unit of the mass accuracy"*: `mz`
>    - *"Choose spectra"*: `Plot mean spectra based on all spectra`
>        - In *"zoomed in m/z range"*: `Manual input`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"1: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `10`
>                - *"upper m/z boundary for plotting window"*: `200`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"2: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `200`
>                - *"upper m/z boundary for plotting window"*: `500`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"3: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `500`
>                - *"upper m/z boundary for plotting window"*: `1000`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"4: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `1000`
>                - *"upper m/z boundary for plotting window"*: `1500`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"5: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `1500`
>                - *"upper m/z boundary for plotting window"*: `2000`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the approximate m/z of the heaviest peak visible in the average spectra plot of the complete m/z range?
> 2. What is the approximate m/z of the peak with the highest average intensity?
>
> > <solution-title></solution-title>
> >
> > 1. There is a tiny peak at m/z ~760.
> > 2. The highest peak is at ~80 m/z, which can best be seen in the mass spectrum between 15 and 200 m/z.
> >
> {: .solution}
>
{: .question}

In the average spectrum of the complete m/z range there are no visible peaks above 800 m/z. Most of the high intensity peaks are below 300 Th, which corresponds to the typical masses of VOCs. Even in the zoomed mass spectra between 1500 and 2000 m/z only a few distinct peaks can be identified in the noise. In accordance with {% cite Gamboa_Becerra_2015 %}, we will restrict further analysis to the m/z range between 15 and 1000.

![Average plots](../../images/msi_distribution_average_spectra.png "Complete and zoomed in average mass spectra")

We will rerun the MSI plot spectra tool and add annotation to the pixels that correspond to different chilli compartments.

> <hands-on-title>Mass spectra of different chilli compartments</hands-on-title>
>
> 1. Create a tabular file with the coordinates of interest and rename it to 'annotations':
>
>    ```
>    x     y     compartment
>    39    53    seeds
>    50    44    placenta
>    25    60    pericarp
>    ```
>
>    > <tip-title>Creating a new file</tip-title>
>    >
>    > * Open the Galaxy Upload Manager
>    > * Select **Paste/Fetch Data**
>    > * Paste the file contents into the text field
>    > * From the settings menu {% icon galaxy-gear %} select **Convert spaces to tabs**
>    > * Press **Start** and **Close** the window
>    {: .tip}
>
> 2. {% tool [MSI plot spectra](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_spectra_plots/cardinal_spectra_plots/2.6.0.3) %}
>    - Press the rerun button, changing only the following:
>    - *"Choose spectra"*: `Plot single spectra (separate or overlaid)`
>        - {% icon param-file %} *"Load tabular file with pixel coordinates"*: `annotations` (Input dataset)
>        - *"Column with x values"*: `column:1`
>        - *"Column with y values"*: `column:2`
>        - *"Tabular file contains a header line"*: `Yes`
>        - *"Separate plot per spectrum or overlaid plot with average spectra per annotation group"*: `Overlaid spectra plots`
>        - *"Select column with annotations"*: `column:3`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which compartment does the highest intensity peak belong to?
> 2. Which compartment does the third-highest intensity peak belong to?
>
> > <solution-title></solution-title>
> >
> > 1. The highest intensity peak at around 80 m/z belongs to the spectrum that is located in the pericarp of the chilli.
> > 2. The peak at around 60 m/z belongs to the spectrum that is located in the seeds.
> {: .solution}
>
{: .question}

![Annotated plots](../../images/msi_distribution_overlay_plot.png "Mass spectra for single pixels of different compartments")

The single spectra that derive from different chilli compartments show some major differences. You should keep in mind that these are single spectra and therefore the intensities can be different to the average intensity of the compartment. Looking at the m/z range between 1000 and 2000, we realize that there are a number of distinct peaks which were hidden in the noise of the average spectrum, which contained all background spectra as well as the spectra of the chilli. In accordance with the original publication, and to speed up computation time, we focus on the low m/z range, but for a deeper analysis it could also be worth analyzing the higher m/z range.

## Filtering for the relevant m/z range

> <hands-on-title>filtering for a m/z range</hands-on-title>
>
> 1. {% tool [MSI filtering](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_filtering/cardinal_filtering/2.6.0.0) %}
>    - {% icon param-file %} *"MSI data"*: `ltpmsi-chilli.imzML` (Input dataset)
>    - *"Centroided input"*: `yes`
>    - *"Processed imzML file"*: `yes`
>        - *"Mass accuracy to which the m/z values will be binned"*: `0.2`
>        - *"Unit of the mass accuracy"*: `mz`
>    - *"Select m/z feature filtering option"*: `m/z range (manually)`
>        - *"Minimum value for m/z"*: `15.0`
>        - *"Maximum value for m/z"*: `1000.0`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many m/z features are left in the output dataset?
> 2. How many m/z features were removed?
>
> > <solution-title></solution-title>
> >
> > 1. After filtering 4926 m/z features are left.
> > 2. The dataset originally had 9926 m/z features; after filtering 4926 are left, which means that 5000 were removed.
> >
> {: .solution}
>
{: .question}

The MSI tools are only able to write outputs as continuous imzML format; therefore, from now on the dataset is in continuous imzML format and "processed imzML file" can be set to 'no' in the following tools.

# Multiple and overlayed analyte images

## Automatic generation of analyte images

The main question for the chilli dataset is which m/z features have a localized distribution in the fruit. This question can be addressed by automatically generating distribution images for all analytes (m/z features) and then visually identifying which features are localized in specific compartments of the fruit. {% cite Gamboa_Becerra_2015 %} scan the m/z range with a step size of 0.2 m/z and a tolerance of 0.4 Th, which resulted in more than 2000 images which they visually explored for localized features. In this training we will only generate images for the two mass ranges where we have already seen high intensity peaks in the spectra plots: around 60 and 80 m/z.

This requires three steps. Firstly, all m/z features of the dataset with the MSI data exporter are extracted. Secondly, all m/z features are filtered in the m/z ranges that we are interested in: between 55 and 65 Th, as well as between 75 and 85 m/z (this step can be skipped to obtain distribution images for all features). Thirdly, the MSI m/z image tool is used to automatically generate distribution images for each of the m/z from the second step. As the data is already binned to 0.2 Th, only the m/z tolerance of 0.4 m/z has to be specified. The large tolerance was chosen to obtain images from overlapping m/z ranges to make the signal intensity more robust against small m/z inaccuracies in individual mass spectra.

> <hands-on-title>Generation of multiple analyte images</hands-on-title>
> 1. {% tool [MSI data exporter](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_data_exporter/cardinal_data_exporter/2.6.0.0) %}
>    - {% icon param-file %} *"MSI data"*: `filtered imzML` (output of **MSI filtering** {% icon tool %})
>    - *"Centroided input"*: `yes`
>    - *"Multiple output files can be selected"*: `mz feature output`
>
> 2. {% tool [Filter data on any column using simple expressions](Filter1) %}
>    - {% icon param-file %} *"Filter"*: `MSI data exporter on data 6: features` (output of **MSI data exporter** {% icon tool %})
>    - *"With following condition"*: `c2>=55 and c2<=65 or c2>=75 and c2<=85`
>    - *"Number of header lines to skip"*: `1`
>
> 3. {% tool [MSI mz images](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_mz_images/cardinal_mz_images/2.6.0.2) %}
>    - {% icon param-file %} *"MSI data"*: `filtered imzML` (output of **MSI filtering** {% icon tool %})
>    - *"Centroided input"*: `yes`
>    - {% icon param-file %} *"m/z of interest (e.g. internal Calibrants)"*: `Filter on data` (output of **Filter** {% icon tool %})
>    - *"Column with m/z values"*: `column:2`
>    - *"Column with name of m/z values"*: `column:1`
>    - *"Tabular file contains a header line"*: `Yes`
>    - *"M/z range"*: `0.4`
>    - *"Contrast enhancement"*: `suppression`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which are the two analyte images that show the most localized distribution?
> 2. Which compartments are the two features localized in?
>
> > <solution-title></solution-title>
> >
> > 1. Defining the exact m/z with the best localization is quite tricky. Between 61.8 and 62.6 m/z there is a localized feature, as well as between 84 and 84.4 m/z. We can choose 62.2 and 84.2 to get the best images (in the publication it was 62.1 and 84.1).
> > 2. The comparison of the feature localization with the chilli image shows that the 62.2 m/z feature is located in the seeds and the 84.2 m/z feature more uniformly in the placenta and pericarp.
> >
> {: .solution}
>
{: .question}

![best images](../../images/msi_distribution_auto_images.png "Distribution images for the analytes with the most localized distribution")

In this tutorial we only evaluated 102 features, while the authors of this study evaluated more than 2000 images visually. This process is time-consuming and biased by the judgment of the scientist, but seems to still be used sometimes. Several attempts have been made to circumvent this process and obtain m/z features with localized distribution automatically, for example by {% cite Alexandrov_2013 %} and {% cite Inglese_2018 %}

After m/z features with a localized distribution have been found, it is interesting to identify them. {% cite Gamboa_Becerra_2015 %} performed additional gas and liquid chromatography mass spectrometry (GC-MS and LC-MS) which enabled the identification of several features with high certainty. Unfortunately the highly localized features at 62.2 and 84.2 m/z could not be identified, but the defined localization implies their biological importance. Several capsaicinoids could be identified, amongst them capsaicin (306.6 m/z).

> <comment-title>Identification of m/z features</comment-title>
> The identification of m/z features in MSI experiments is not always necessary, but can increase the confidence in the analysis and the molecular understanding. m/z features are either matched to databases or for more confidence another experiment is performed on the same sample to restrict the identification possibilities to analytes that are actually present in the sample. In this study other mass spectrometry techniques were applied, allowing identification of several m/z features, but at the same time features with defined localization could not be identified (e.g. 62.2 m/z and 84.2 m/z). These molecules may be fragments, metabolites or contaminants that are not available with the used mass spectrometry techniques.
> Targeted analysis can be performed if the m/z of the molecule of interest is already known (e.g. when studying the tissue distribution of a drug and its known metabolites). In this case distribution images for the molecules of interest are a good starting point and scanning through all m/z features might not be necessary.
{: .comment}

## Overlay image for three analyte features

To get an idea about the distribution of capsaicin in the chilli we will plot its distribution in an overlay image with the two unknown but localized features 62.2 m/z and 84.2 m/z.

> <hands-on-title>Overlay image for several m/z features</hands-on-title>
>
> 1. Create a tabular file with the m/z of interest and rename it to 'mz features':
>
>    ```
>    62.2
>    84.2
>    306.6
>    ```
>
>    > <tip-title>Creating a new file</tip-title>
>    >
>    > * Open the Galaxy Upload Manager
>    > * Select **Paste/Fetch Data**
>    > * Paste the file contents into the text field
>    > * Set Type to **tabular** instead of auto-detect
>    > * Press **Start** and **Close** the window
>    {: .tip}
>
> 2. {% tool [MSI mz images](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_mz_images/cardinal_mz_images/2.6.0.2) %}
>    - {% icon param-file %} *"MSI data"*: `filtered imzML` (output of **MSI filtering** {% icon tool %})
>    - *"Centroided input"*: `yes`
>    - {% icon param-file %} *"m/z of interest (e.g. internal Calibrants)"*: `mz features` (Input dataset)
>    - *"Column with m/z values"*: `column:1`
>    - *"Column with name of m/z values"*: `column:1`
>    - *"Tabular file contains a header line"*: `No`
>    - *"M/z range"*: `0.4`
>    - *"Contrast enhancement"*: `suppression`
>    - *"Alpha Power"*: `1.5`
>    - *"Linear normalization of image"*: `Yes`
>    - *"Overlay image"*: `Yes`
>        - *"1: Colours for the plots"*: `blue`
>        - *"2: Colours for the plots"*: `green`
>        - *"3: Colours for the plots"*: `red`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which compartment is capsaicin located in?
>
> > <solution-title></solution-title>
> >
> > 1. Capsaicin is located in the middle of the placenta, as well as partly in the seeds.
> >
> {: .solution}
>
{: .question}

![overlay images](../../images/msi_distribution_overlay_images.png "Distribution image for capsaicin and overlay of the three analytes")

# Conclusion

This tutorial covered the steps to explore the properties of a dataset and visualize the morphological distribution of molecules.

![workflow](../../images/msi_distribution_workflow.png "Workflow that was used during this tutorial.")

tools_json:
["MSI_Qualitycontrol", "toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_quality_report/cardinal_quality_report/2.6.0.1", "MSI plot spectra", "toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_spectra_plots/cardinal_spectra_plots/2.6.0.3", "MSI filtering", "toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_filtering/cardinal_filtering/2.6.0.0", "MSI data exporter", "toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_data_exporter/cardinal_data_exporter/2.6.0.0", "Filter data on any column using simple expressions", "Filter1", "MSI mz images", "toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_mz_images/cardinal_mz_images/2.6.0.2"]


topic: metabolomics
tutorial: msi-finding-nglycans
title: Mass spectrometry imaging: Finding differential analytes
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 294bf7be8c42e55114a300b986a195d8
last_modified: 2026-05-15T22:28:13.790466
zenodo_link: https://zenodo.org/record/2628280
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/msi-finding-nglycans/tutorial.html

content:
Mass spectrometry imaging (MSI) is applied to measure the spatial distribution of hundreds of biomolecules in a sample. A mass spectrometer scans over the entire sample and collects a mass spectrum every 5-200 µm. This results in thousands of spots (or pixels) for each of which a mass spectrum is acquired. Each mass spectrum consists of hundreds of analytes that are measured by their mass-to-charge (m/z) ratio. For each analyte the peak intensity in the mass spectra of every pixel is known and can be set together to map the spatial distribution of the analyte in the sample.

The technique has a broad range of applications as it is able to measure many different kinds of analytes such as peptides, proteins, metabolites or chemical compounds in a large variety of samples such as cells, tissues and liquid biopsies. Application areas include pharmacokinetic studies, biomarker discovery, molecular pathology, forensic studies, plant research and material sciences. The strength of MSI is the simultaneous analysis of hundreds of analytes in an unbiased, untargeted, label-free, fast and affordable measurement while maintaining morphological information.

Depending on the analyte of interest and the application, different mass spectrometers are used. A mass spectrometer measures the analytes by ionizing, evaporating and sorting them by their mass-to-charge (m/z) ratio. Put simply, a mass spectrometer consists basically of three parts: an ionization source, a mass analyzer and a detector. The most common ionization sources for MSI are MALDI (Matrix Assisted Laser Desorption/Ionization), DESI (Desorption Electrospray Ionization) and SIMS (Secondary Ion Mass Spectrometry).

![MSI measurement](../../images/msi_distribution_maldi_tof_imaging.png "MALDI TOF imaging of a mouse kidney")

One common type of mass spectrometer for MSI is a MALDI Time-Of-Flight (MALDI-TOF) device. During MALDI ionization, a laser is fired onto the sample, which has been covered with a special matrix that absorbs the laser energy and transfers it to the analytes. This process vaporizes and ionizes the analytes. As they are now charged, they can be accelerated in an electric field towards the TOF tube. The time of flight through the tube to the detector is measured, which allows calculation of the mass over charge (m/z) of the analyte, as both mass and charge are correlated with time of flight. During measurement, complete mass spectra with hundreds of m/z - intensity pairs are acquired in thousands of sample plots, leading to large and complex datasets. Each mass spectrum is annotated with coordinates (x,y) that define its location in the sample. This allows visualization of the intensity distribution of each m/z feature in the sample as a heatmap.

Depending on the analyte of interest, the sample type and the mass spectrometer, the sample preparation steps as well as the properties of the acquired data vary. Human tissues are often stained with hematoxylin and hemalaun (H&E) after mass spectrometry imaging. This allows better visualization of tissue histology and can be used to compare the MSI heatmaps with the real morphology. The stained image of the tissue can also be used to define regions of interest that can be extracted from the MSI data and can be subjected to supervised statistical analysis.

As MSI measures only unfractionated molecules (MS1), identification of the resulting m/z features requires matching them to a list of known m/z features. The easiest approach is to match the MSI m/z features to existing databases such as UniProt for proteins or lipidmaps for lipids. More accurate identification is obtained by performing additional mass spectrometry experiments (either on the same or on a similar sample, e.g. adjacent slice from a tissue) that also measure the fragmented molecules (MS2) and thereby also allow accurate identification of isobaric molecules. This tandem mass spectrometry approach can either be done in situ directly on the sample or by transferring the sample to a tube and bringing it into solution for liquid-chromatography tandem mass spectrometry (LC-MS/MS).

In this tutorial we will determine and identify N-linked glycans by differential analysis of a control (untreated) murine kidney section and a section treated with PNGase F. The data analysis steps can be transferred to any other application that requires finding differential analytes, such as for biomarker discovery.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# N-linked glycans in murine kidney dataset

In this tutorial we will use the [murine kidney N-glycan dataset](https://doi.org/10.1016/j.dib.2018.08.186) generated in the lab of Peter Hoffmann (Adelaide, Australia). N-linked glycans are carbohydrates (consisting of several sugar molecules) that are attached as a post-translational modification to the carboxamide side chain of an asparagine residue of a protein. Changes in N-linked glycans have been observed in several diseases. The Hoffmann lab acquired the presented dataset to show that their automated sample preparation method for N-glycan MALDI imaging successfully analyses N-glycans from formalin-fixed paraffin-embedded murine kidney tissue ([Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7)). The datasets generated in this study were made publicly available via the [PRIDE Archive](https://www.ebi.ac.uk/pride/archive/projects/PXD009808).

Three 6 µm sections of formalin-fixed paraffin-embedded murine kidney tissue were prepared for MALDI imaging. To release N-linked glycans from the proteins, PNGase F was printed onto two kidney sections. In the third kidney section, one area was printed with buffer to serve as a control, and another was covered with N-glycan calibrants [Figure 4a-c](https://media.springernature.com/original/springer-static/image/art%3A10.1007%2Fs00216-014-8293-7/MediaObjects/216_2014_8293_Fig4_HTML.gif). 2,5-DHB matrix was sprayed onto the tissue sections and MALDI imaging was performed with 100 µm intervals using a MALDI-TOF/TOF instrument.

To make computation times feasible for this training, we reduced all datasets to a m/z range of 1250 – 2310 and resampled the m/z values with a step size of 0.1. The main part of the training will be performed with the control and one PNGase F treated kidney file, which were both restricted to representative pixels (Figure 2). To test the results on the complete dataset we also provide a file in which all four (m/z reduced and resampled) files (control, calibrants, treated kidney 1 and treated kidney 2) are combined and TIC-normalized. All size-reduced training datasets are available via [Zenodo](https://zenodo.org/record/2628280). We will combine the cropped control and the cropped treated1 file and perform similar steps to those described by [Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7). These steps include combining files, preprocessing, unsupervised and supervised analysis, identification of N-glycans and generation of analyte distribution heatmaps.

![file_dimensions](../../images/msi-diff-abundance-file-dimensions.png "Top image: All four files after combining them. Black rectangles show the orientation of the cropped control and cropped treated1 files that are used in the major part of the training. Bottom: Cropped training files after combining them.")


## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://zenodo.org/record/2628280). Upload all three datasets (one by one) to Galaxy via the **composite** option and rename them to 'control', 'treated1' and 'all_files'.
>    ```
>    https://zenodo.org/record/2628280/files/conrol.imzml
>    https://zenodo.org/record/2628280/files/control.ibd
>    https://zenodo.org/record/2628280/files/treated1.imzml
>    https://zenodo.org/record/2628280/files/treated1.ibd
>    https://zenodo.org/record/2628280/files/all_files.imzml
>    https://zenodo.org/record/2628280/files/all_files.ibd
>    ```
>
> 3. Paste the link of the tabular file with the N-glycan identities into the regular upload and rename it to 'Glycan IDs':
>    ```
>    https://zenodo.org/record/2628280/files/Glycan_IDs.tabular
>    ```
>
> 4. Create a tabular file with the three glycan calibrants that were used in the study and rename it to 'calibrants':
>    ```
>    M+Na+    composition
>    1257.4225    Man5GlcNAc2
>    1542.5551    Man3GlcNAc5
>    2393.8457    Man3Gal4GlcNAc6
>    ```
>
> 5. Add a tag to each imzML file ('control', 'treated1' and 'all_files') and to the tabular file ('Glycans'):
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Preparing the files for statistical analysis

## Combining control and treated1 files

> <hands-on-title>Combining files</hands-on-title>
>
> 1. **MSI combine** {% icon tool %} with the following parameters:
>    - {% icon param-files %} *"MSI data"*: `control` (uploaded dataset) and `treated1` (uploaded dataset)
>    - *"Output format"*: `imzML`
>
>    > <comment-title>on the combine tool</comment-title>
>    >
>    > Two combining options are available: 'Automatic combination' combines the selected files into a grid with 5 pixel space between them. The order of the files in the history determines their orientation in the combined file and the file name is used to annotate the corresponding pixels. For custom orientation of the files the option 'shift xy coordinates with a tabular file' can be applied. For now, the **MSI combine** {% icon tool %} only allows combining files with the same m/z axis. Different m/z axes can be adjusted by filtering for the common m/z range and then binning with the same bin size (**MSI filtering** {% icon tool %} and **MSI preprocessing** {% icon tool %}.
>    {: .comment}
>
{: .hands_on}

## Exploring the combined file

To get an overview of the combined dataset we will use the **MSI Qualitycontrol** {% icon tool %}. We will use the annotation file that was generated by the **MSI combine** {% icon tool %} to inform the **MSI Qualitycontrol** {% icon tool %} which spectra belong to which subfile - this will lead to several extra figures where the data properties for both subfiles are plotted separately.

> <hands-on-title>Quality control with spectrum annotations</hands-on-title>
>
> 1. **MSI Qualitycontrol** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `combined imzML` (output of **MSI combine** {% icon tool %})
>    - *"Use pixel annotation from tabular file for QC plots"*: `use pixel annotation from a tabular file`
>        - {% icon param-file %} *"Tabular file with pixel coordinates and annotation"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with pixel annotations"*: `Column: 3`
>        - *"Tabular file contains a header line"*: `Yes`
>    - {% icon param-file %} *"m/z of interest (e.g. internal Calibrants)"*: `calibrants` (uploaded dataset)
>    - *"Column with m/z values"*: `column:1`
>    - *"Column with name of m/z values"*: `column:2`
>    - *"Tabular file contains a header line"*: `Yes`
>    - *"PCA with 2 components"*: `Yes`
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How many m/z values does the combined dataset contain?
> 2. How many spectra does the combined dataset contain?
> 3. How many input calibrants were valid?
> 4. Which of the two subfiles shows higher overall intensities?
>
> > <solution-title></solution-title>
> >
> > 1. 10581
> > 2. 1935
> > 3. Two calibrants are valid, which means that they lie in the (cropped) m/z range of our files.
> > 4. Several plots show that the treated subfile has higher intensities than the control subfile.
> >
> {: .solution}
>
{: .question}

The quality report gives a broad overview of the properties of both files. The pixel order seems to be random and does not correspond to the measurement in the MALDI. This means that the plots where the 'spectra index' is plotted on the x-axis do not represent the acquisition process. While the PNGase F treated kidney file shows higher intensities in several plots (e.g. median intensity per spectrum and intensity boxplots), the number of peaks is similar in both subfiles. The most abundant m/z feature plot of the treated kidney roughly corresponds to the morphology of the kidney while in the control file there are no visible kidney structures. The first principal componant in the PCA analysis roughly differentiates the spectra of the two subfiles which is a good sign for finding differences in the treated vs. control kidney files. For the following preprocessing steps, it is important to know the rough number of data points that one peak consists of. From page 27 on, the average mass spectra of the two glycan calibrants are shown. Due to our harsh resampling to 0.1 m/z, the calibrant 1257.4225 (Man5GlcNAc2) shows quite an angular peak shape. The complete peak consists of about 6 data points, which corresponds to a width of 0.5 m/z. On page 28 the average spectrum for each subfile is plotted, showing a nice peak in the treated subfile and only basal intensity in the control subfile. Page 29 and 30 show the average mass spectra for the 1542.5551 (Man3GlcNAc5) calibrant. The intensities of this calibrant are relatively low and only slightly higher in the treated than in the control subfile. Furthermore, the split up peak shape is not useful for defining the peak width properly.

## Preprocessing of the combined file

Preprocessing of the MSI raw spectra is crucial to reduce experimental (electrical, chemical and mechanical) noise and variance within and between the datasets before statistical analysis. Depending on the ionization technique, the mass analyzer, the sample and the statistical analysis, different preprocessing steps have to be applied. Here, we will perform similar preprocessing steps to those described by [Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7).

  1. **Intensity normalization**:
Signal intensities in MALDI datasets are affected by heterogeneous matrix spray or crystallization, salt and pH gradients and contamination of the ion source over time. Therefore intensity normalization is important to ensure spectra that were acquired at different positions of the sample are comparable. The most common method is the normalization to the total ion current (TIC). This means that each intensity value of a spectrum is divided by the sum of all intensities of the spectrum (equal to the total ion current, TIC). More information on normalization can be found in the publications by [Gibb and Strimmer](https://arxiv.org/pdf/1607.03180.pdf) and [Deininger et al.](https://link.springer.com/article/10.1007%2Fs00216-011-4929-z).

  2. **Spectra smoothing**:
Smoothing reduces noise in the spectra and thereby improves peak picking. Smoothing is an optional preprocessing step, especially for high resolution data. The critical parameter for smoothing is the window size. Larger window sizes lead to stronger smoothing, which not only reduces noise but also the intensities of real peaks. Therefore it is important to set the window size large enough to remove noise but small enough to preserve the peaks. We will use the 'gaussian' smoothing method that applies a weighted moving average filter and set the window size to 4 (resampling was done with 0.1, multiplied by 4 this gives a 0.4 m/z window which is slightly smaller than the complete peak width that we found to be roughly 0.5 m/z). The standard deviation is set to the default setting: window/4 = 1. As the smoothing changes the peak width, we will quickly examine the 1257.42 calibrant peak width afterwards and use the information to define the correct window sizes in further preprocessing steps.

  3. **Baseline removal**:
Especially for MALDI datasets it is common to perform a baseline subtraction to remove the background intensity that is generated by chemical noise. Baseline removal could also be performed before smoothing but as smoothing might slightly increase the baseline we perform the baseline removal afterwards. Similar to the smoothing step, the window size in which the noise is estimated is crucial. The window should be large enough to obtain a representative number of data points but small enough to avoid removing parts of the peaks as well. Here, the window is obtained by dividing the m/z range by the number of blocks. Recommended window sizes range from 1 - 5 m/z. We will divide the m/z axis into 500 blocks, giving a window size of about 2 m/z.


> <hands-on-title>Preprocessing part I</hands-on-title>
>
> 1. **MSI preprocessing** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `combined imzML` (output of **MSI combine** {% icon tool %})
>    - In *"Preprocessing"*:
>        - Click on *"Insert Preprocessing"*:
>        - In *"1: Preprocessing"*:
>            - *"Preprocessing methods"*: `Intensity Normalization (TIC)`
>       - Click on *"Insert Preprocessing"*:
>        - In *"2: Preprocessing"*:
>            - *"Preprocessing methods"*: `Peak smoothing`
>                - *"Smoothing method"*: `gaussian`
>                    - *"The standard deviation for the Gaussian kernel (window/sd)"*: `1`
>                - *"Window size"*: `4.0`
>        - In *"3: Preprocessing"*:
>            - *"Preprocessing methods"*: `Baseline Reduction`
>    - *"Output format"*: `imzML`
>
> 2. **MSI plot spectra** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `processed imzML` (output of **MSI preprocessing** {% icon tool %})
>    - *"Select if you want to plot the mass spectrum of a single pixel or the average spectrum of all pixels of a sample"*: `Average spectrum for each sample`
>        - *"Use pixel annotation from tabular file for spectra plots"*: `group pixels according to annotations`
>        - {% icon param-file %} *"Tabular file with pixel coordinates and annotation"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with pixel annotations"*: `Column: 3`
>        - *"Tabular file contains a header line"*: `Yes`
>        - In *"Zoomed in plots with m/z min and m/z max to define the plot window"*:
>            - Click on *"Insert Zoomed in plots with m/z min and m/z max to define the plot window"*:
>            - In *"1: Zoomed in plots with m/z min and m/z max to define the plot window"*:
>                - *"lower m/z boundary for plotting window"*: `1257`
>                - *"upper m/z boundary for plotting window"*: `1259`
>
{: .hands_on}

The QC report of the **MSI preprocessing** {% icon tool %} shows the changes during preprocessing for four random spectra. TIC normalization only changes the dimension of the y-axis. Smoothing decreases the peak height and thus the intensity values on the y-axis shrink as well. Baseline removal shifts the complete spectrum towards the bottom.

The **MSI plot spectra** {% icon tool %} is a quick way to control peak shape after smoothing. The zoomed in spectrum of the 1257.42 calibrant peaks shows that the peaks become broader than before. Most of the underlying data points can be seen as angles of the peak. The peak width ranges from 1257.1 to 1257.9 m/z = 0.8 m/z width. We will set the following window sizes to 9 data points (half the window size of 4).

We can then continue with the second part of the preprocessing:

  1. **m/z alignment**:
Despite regular m/z calibration of the mass spectrometer, m/z values vary slightly during data acquisition. To make m/z features comparable between different spectra or samples, they need to have the same m/z value. Furthermore, if identification is desired the m/z values should also be as accurate as possible. For high resolution data, binning of the m/z axis is often sufficient to group similar m/z values to a common m/z bin. For TOF instruments, re-calibration of the m/z values is commonly applied with the help of known, ubiquitous analytes such as internal calibrants or matrix peaks. Unfortunately, no internal calibrants were applied and the DHB matrix peaks occur more frequently below 1000 m/z. Therefore, the m/z values are not re-calibrated with known analytes but instead aligned to the average picked mass spectrum to ensure that similar m/z features obtain the same m/z value. This common m/z value can still be slightly shifted from its correct position - this should be kept in mind for the identification step that we will perform later. m/z alignment or binning could also be done after peak picking, but as we will perform peak picking on the average spectrum it is important that the m/z shifts are already minimized. We will remove spectra in which less than 2 reference m/z were found by setting them to zero and then removing all empty spectra. To account for the m/z shifts we will export the processed imzML file, which means that each spectrum has its own m/z axis.

  2. **Peak picking and binning**:
The key step during preprocessing is peak detection as it reduces the dimensionality of the data and bases the following statistical analysis on potentially relevant peaks rather than on noise. A peak is a local maximum above a user-defined noise threshold. The higher the signal-to-noise ratio is set, the more true positive peaks are picked, but at the cost of lowered sensitivity. Therefore, we will set the signal-to-noise ratio to 5, which should remove most of the noise while keeping enough sensitivity. The window should ideally have about the size of the peak width (recalling we set half window size to 4). We will perform peak picking and monoisotopic picking on the average mass spectra of each subfile, which are again defined with the annotation tabular file. The removal of isotopic peaks reduces the number of m/z features of the dataset, which is not only beneficial for computation time but also for many statistical approaches that are hindered by highly correlating variables. To ensure comparability of the picked m/z features, binning of the m/z axis is performed.

  3. **Extracting peaks**:
The average peak picking has removed all spatial information and kept only mean intensity values for the two subfiles. Therefore, the 'peaks' function is used to retrieve the picked peaks from all pixels of the normalized, smoothed and baseline reduced dataset.

For further reading on preprocessing we recommend [Gibb and Strimmer](https://arxiv.org/pdf/1607.03180.pdf) and [Norris et al.](https://doi.org/10.1016/j.ijms.2006.10.005).

> <hands-on-title>Preprocessing part II</hands-on-title>
>
> 1. **MALDIquant preprocessing** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML` (output of **MSI preprocessing** {% icon tool %})
>    - In *"Method"*:
>        - In *"1: Method"*:
>            - *"Select a method"*: `Align spectra`
>                - *"Tolerance"*: `0.0002`
>                - *"Half window size"*: `4`
>                - *"Signal-to-noise-ratio"*: `3`
>                - *"Allow no matches"*: `Yes`
>                - *"Empty no matches"*: `Yes`
>                - *"Remove empty spectra"*: `Yes`
>                - *"minFrequency"*: `0.8`
>                - *"Export processed imzML"*: `Yes`
>
> 2. **MALDIquant peak detection** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `MALDIquant preprocessing imzML` (output of **MALDIquant preprocessing** {% icon tool %})
>    - *"Spectra annotations"*: `yes`
>        - *"Use annotations from tabular file"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with pixel annotations"*: `Column: 3`
>        - *"File contains a header line"*: `Yes`
>    - In *"Method"*:
>        - In *"1: Method"*:
>            - *"Select a method"*: `Peak detection`
>                - *"Half window size"*: `4`
>                - *"Signal to noise"*: `5`
>                - *"Detect peaks on average mass spectra"*: `Yes`
>        - Click on *"Insert Method"*:
>        - In *"2: Method"*:
>            - *"Select a method"*: `Monoisotopic peaks`
>                - *"Minimal correlation"*: `0.4`
>                - *"Tolerance"*: `0.0002`
>                - *"Size"*: `2:5`
>        - Click on *"Insert Method"*:
>        - In *"3: Method"*:
>            - *"Select a method"*: `Binning`
>                - *"Tolerance"*: `0.0002`
>
> 3. **MSI preprocessing** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML from first step` (output of **MSI preprocessing** {% icon tool %})
>    - In *"Preprocessing"*:
>        - Click on *"Insert Preprocessing"*:
>        - In *"1: Preprocessing"*:
>            - *"Preprocessing methods"*: `Data reduction`
>                - *"Reduction method"*: `peaks`
>                - *"Tabular file with m/z features to extract from input file"*: `MALDIquant peak detection intensity matrix` (output of **MALDIquant peak detection** {% icon tool %})
>                - *"Tabular file contains a header line"*: `Yes`
>    - *"Output format"*: `imzML`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many m/z values are left after preprocessing?
> 2. What is the lowest m/z feature that is left after preprocessing?
>
> > <solution-title></solution-title>
> >
> > 1. 125 m/z features are left
> > 2. 1257.47 (min m/z)
> >
> {: .solution}
>
{: .question}

> <details-title>How to find good preprocessing parameters</details-title>
> Before starting with preprocessing it is important to know the properties of the data. Internal calibrants can help to estimate the quality of the data and to control the preprocessing steps. The **MSI Qualitycontrol** {% icon tool %} shows the extent of a noise baseline, the level of noise in the mass spectra, intensity distributions and m/z accuracy as described in the ([Mass spectrometry imaging: Loading and exploring MSI data]({{site.baseurl}}/topics/proteomics/tutorials/mass-spectrometry-imaging-loading-exploring-data/tutorial.html)) tutorial. Most preprocessing steps are applied to single spectra and therefore allow optimization of the preprocessing steps on a small subset of the original dataset that can be obtained with the **MSI filtering** {% icon tool %}. To make sure that the preprocessing steps and parameters are chosen properly it is recommended to perform them step by step and check the outcome after every step. The **MSI Qualitycontrol** {% icon tool %}, the **MSI spectra plot** {% icon tool %}, and the **MSI mz images** {% icon tool %} are helpful to track the changes during preprocessing. The first two tools are able to overlay MSI data with different annotations, making it possible to combine files before and after the preprocessing step, as done in step 1 and 2 of this tutorial, and compare the introduced changes directly. The usage of annotations in the **MSI spectra plot** {% icon tool %} is explained in the tutorial [Mass spectrometry imaging: Examining the spatial distribution of analytes]({{ site.baseurl }}/topics/metabolomics/tutorials/msi-analyte-distribution/tutorial.html).
{: .details}

The QC table of the **MALDIquant preprocessing** {% icon tool %} reveals that 19 spectra (pixel) were removed during the alignment. The empty spectra are visible in the QC file of **MALDIquant peak detection** {% icon tool %}. This QC furthermore shows the baseline in blue and the picked (monoisotopic) peaks in green. After peak extraction the mass spectra are not in profile mode anymore (one peak consists of many datapoints) but centroided (one peak is represented by one datapoint). Therefore, in further steps 'centroided input' must be set to 'Yes'.


# Statistical analysis and identification of N-glycans

## Unsupervised statistical analysis

Unsupervised analysis can be applied at any time to any dataset because no prior knowledge of the data is required. The clustering or segmentation algorithms (that perform unsupervised analysis) separate regions of the data that have the most distinct differences in their mass spectra. We will perform principal component analysis (PCA) with 4 clusters to visualize the largest differences in the dataset.

> <hands-on-title>Segmentation with PCA</hands-on-title>
>
> 1. **MSI segmentation** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML (peaks)` (output of **MSI preprocessing** {% icon tool %})
>    - *"Centroided input"*: `Yes`
>    - *"Select the tool for spatial clustering"*: `pca`
>        - *"The number of principal components to calculate"*: `4`
>    - In *"Colours for the plots"*:
>        - Click on *"Insert Colours for the plots"*:
>        - In *"1: Colours for the plots"*:
>            - *"Colours"*: `dark_blue`
>        - Click on *"Insert Colours for the plots"*:
>        - In *"2: Colours for the plots"*:
>            - *"Colours"*: `orange`
>        - Click on *"Insert Colours for the plots"*:
>        - In *"3: Colours for the plots"*:
>            - *"Colours"*: `green`
>        - Click on *"Insert Colours for the plots"*:
>        - In *"4: Colours for the plots"*:
>            - *"Colours"*: `red`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which principal components correlate with the morphology of the kidney?
> 2. What are the approximate m/z values with the highest loadings for each of those principal components?
>
> > <solution-title></solution-title>
> >
> > 1. PC1, PC3 and PC4
> > 2. PC1: 1900; PC3: 1900; PC4: 2300 (from PCA plot)
> >
> {: .solution}
>
{: .question}

![PCA](../../images/msi-diff-abundance-pca.png "Distribution of the principal components")

The PCA clustering reveals that the N-glycan profile is different in different kidney compartments. To find the m/z features that correspond to these compartments, the feature tabular file can further be analyzed by sorting the file for the principal components of interest. This reveals the discrete m/z values for the features with the highest loadings: PC1 and PC3: 1905.72, PC4: 2304.91. Later, we will visualize the morphological distribution of those features on the complete dataset.


## Supervised statistical analysis

Supervised analysis can only be performed if the regions of interest that should be compared are known. For this dataset we know which spectra were treated or not treated with PNGase F. With this information we can use the spatial shrunken centroids algorithm, which in addition to classification is able to find the m/z features that best distinguish each class ([Bemis et al.](https://doi.org/10.1074/mcp.O115.053918)). We expect those discriminating m/z features to be the N-linked glycans that were only released in the treated kidney section but not in the control section.

> <hands-on-title>Spatial shrunken centroids classification</hands-on-title>
>
> 1. **MSI classification** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML (peaks)` (output of **MSI preprocessing** {% icon tool %})
>    - *"Centroided input"*: `Yes`
>    - *"Analysis step to perform"*: `training`
>        - {% icon param-file %} *"Load tabular file with pixel coordinates and their classes"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with response (condition) values"*: `Column: 3`
>        - *"Tabular files contain a header line"*: `Yes`
>        - *"Select the method for classification"*: `spatial shrunken centroids`
>            - *"Analysis step to perform"*: `spatial shrunken centroids analysis`
>            - *"The sparsity thresholding parameter by which to shrink the t-statistics (s)"*: `0`
>
{: .hands_on}


![SSC](../../images/msi-diff-abundance-classification.png "T-statistics plot for the classification and distribution of the most discriminating m/z feature and class prediction for all spectra.")

High accuracy, sensitivity and specificity indicate that the control and PNGase F treated kidney sections can be discriminated quite well. The prediction shows the location of the few misclassified spectra towards the border of the tissue. As our aim is not to build and test a classifier for accurate prediction, we move on to investigate the m/z features that were most discriminative for the two groups more deeply. Looking at the feature tabular file, we spot the 2 m/z features that had the highest loadings in the PCA, amongst the top features with the highest t-statistics. Furthermore, this list can be compared to the m/z features that were determined as potential N-linked glycans in [Gustafsson et al. Figure S6](https://static-content.springer.com/esm/art%3A10.1007%2Fs00216-014-8293-7/MediaObjects/216_2014_8293_MOESM1_ESM.pdf). We found nine out of twelve m/z features. Two m/z features were outside our cropped m/z range and could therefore not be found. The 1079.408 m/z feature is missing in our feature list.


## Identification of N-glycans

We filter the m/z feature output from the **MSI classification** {% icon tool %} for an adjusted p-value below 0.05 and a positive t-statistic in the 'treated1' spectra to consider only the most discriminative features that are more abundant in the treated than in the control spectra. Those m/z features are potentially N-glycans that were released by PNGase F treatment from the kidney section but not from the control section. To verify that the m/z features are N-linked glycans and to elucidate their identity and structure [Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7) performed an additional LC-MS/MS experiment on N-linked glycans released from another mouse kidney section. They identified [31 N-linked glycans](https://media.springernature.com/original/springer-static/image/art%3A10.1007%2Fs00216-014-8293-7/MediaObjects/216_2014_8293_Fig3_HTML.gif) and calculated the m/z for the sodium N-glycan ions which can be matched to the imaging m/z features. Protonated glycans do not ionize well, but sodium adducts are the best detected glycan form in positive mode mass spectrometry. We match the differential m/z features from the imaging experiment to the m/z features from the LC-MS/MS results with a tolerance of 300 ppm and keep all matches that occur within this distance. We then calculate the average ppm error for the matches.


> <hands-on-title>Identification of N-glycans</hands-on-title>
>
> 1. **Filter data on any column using simple expressions** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `classification features` (output of **MSI classification** {% icon tool %})
>    - *"With following condition"*: `c5=="treated1" and c7>0 and c9<0.05`
>    - *"Number of header lines to skip"*: `1`
>
> 2. **Join two files on column allowing a small difference** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"1st file"*: `Filter data` (output of **Filter** {% icon tool %})
>    - *"Column to use from 1st file"*: `Column: 1`
>    - {% icon param-file %} *"2nd file"*: `Glycan IDs` (uploaded dataset)
>    - *"Column to use from 2nd file"*: `Column: 1`
>    - *"Does the input files contain a header line"*: `Yes`
>    - *"Add an addional column with the calculated distance."*: `Yes`
>    - *"Choose the mode of merging."*: `All matches within the defined distance`
>    - *"Choose the metrics of your distance"*: `Distance in ppm`
>    - *"Allowed distance between the two values that will trigger a merge"*: `300.0`
>
> 3. **Summary Statistics for any numerical column**  {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Summary statistics on"*: `Join two files` (output of **Join two files** {% icon tool %})
>    - *"Column or expression"*: `c12`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many m/z features are filtered by our criteria?
> 2. How many m/z features could be identified?
> 3. How many duplicated matches occured?
> 4. What was the average m/z error for the matching?
>
> > <solution-title></solution-title>
> >
> > 1. The filtered file has 32 lines minus one header line, i.e. 31 m/z features.
> > 2. 19 m/z features were identified.
> > 3. 0
> > 4. 28.98 ppm
> {: .solution}
>
{: .question}

31 m/z features of the imaging dataset have a p-value below 0.05 and are potentially N-linked glycans. To ensure the identity of those features, they were matched to the results of an additional LC-MS/MS experiment. This step allows us to identify 19 m/z features as N-linked glycans. 15 of those m/z features were also found and identified in the [original publication](https://doi.org/10.1007/s00216-014-8293-7), which means that we found 4 additional N-glycans: 1704.7, 1542.6, 2101.8 and 1955.7. Those are the 4 m/z features with the lowest t-statistic. One hint to their validity is their m/z error. The m/z errors of 5, 24, 32 and 64 ppm seem reasonable. The average m/z error was 28.98 ppm compared to 38.7 ppm in the original publication. Without resampling of the m/z axis (which was mainly done to speed up computation time) or with additional internal calibrants for m/z re-calibration this error would probably further decrease.
Next, we visually explore the abundance distribution for all identified N-glycans in the complete, TIC-normalized dataset (consisting of the subfiles: control, calibrant, treated1 and treated2).



## N-linked glycan distribution in the complete dataset

> <hands-on-title>Distribution images for the identified N-glycans</hands-on-title>
>
> 1. **MSI mz images** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `all_files` (uploaded dataset)
>    - {% icon param-file %} *"m/z of interest"*: `Join two files` (output of **Join two files** {% icon tool %})
>    - *"Column with m/z values"*: `Column: 1`
>    - *"Column with name of m/z values"*: `Column: 11`
>    - *"Tabular file contains a header line"*: `Yes`
>    - *"Contrast enhancement"*: `suppression`
{: .hands_on}

> <question-title></question-title>
>
> 1. Which N-linked glycans were used as calibrants?
>
> > <solution-title></solution-title>
> >
> > 1. (Hex)2+(Man)3(GlcNAc)2 (1257.5) and (HexNAc)3+(Man)3(GlcNAc)2 (1542.6) (higher intensity in the calibrant file)
> >
> {: .solution}
>
{: .question}

Despite the classification being done on a small part of the complete dataset, the discovered N-glycans are more abundant in both treated kidney sections than in the control and calibrant regions. The four additional N-glycans that we could identify are more abundant in the treated kidney than in the control files, but compared to the other N-glycans their intensities are lower.


## N-linked glycans with a distinct localization to kidney compartments

During PCA analysis we found N-glycans that are localized in specific kidney compartments. We will filter the identified N-glycan file for the specific m/z features to plot their distribution to different compartments as overlay image.

> <comment-title>Regular expressions to filter text</comment-title>
>
> Regular expressions are syntax that can be used to describe a certain 'text'. They are extremely helpful to filter any kind of text and are therefore utilized by several Galaxy tools. The tools help section provides more details about how to use regular expressions. Here, we want to filter for three m/z features, but we are unsure where the exact decimal places are. Therefore we filter only for the m/z feature without decimal places but include the point '.' to make sure that we do not get matches to any other numbers in the tabular file. As the point '.' has a special meaning in regular expressions, it is necessary to 'escape' it by using the backslash '\' - this allows searching for the point as well. For example '1663\\.' will filter for '1663.' in the text. The pipe '\|' separates alternate possibilities and allows the user to run the tool only once while filtering for three different m/z features.
{: .comment}

> <hands-on-title>Overlayed distribution image for 3 N-glycans</hands-on-title>
>
> 1. **Select lines that match an expression** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `Join two files` (output of **Join two files** {% icon tool %})
>    - *"the pattern"*: `(1663\.)|(1905\.)|(2304\.)`
>
> 2. **MSI mz images** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `all_files` (uploaded dataset)
>    - {% icon param-file %} *"m/z of interest"*: `Select on data` (output of **Select** {% icon tool %})
>    - *"Column with m/z values"*: `Column: 1`
>    - *"Column with name of m/z values"*: `Column: 11`
>    - *"Contrast enhancement"*: `suppression`
>    - *"Overlay image"*: `yes`
>        - In *"Colours for the plots"*:
>            - Click on *"Insert Colours for the plots"*:
>            - In *"1: Colours for the plots"*:
>                - *"Colours"*: `dark_blue`
>            - Click on *"Insert Colours for the plots"*:
>            - In *"2: Colours for the plots"*:
>                - *"Colours"*: `green`
>            - Click on *"Insert Colours for the plots"*:
>            - In *"3: Colours for the plots"*:
>                - *"Colours"*: `red`
{: .hands_on}

![overlay_images](../../images/msi-diff-abundance-mz-images.png "Distribution for three analytes that are localized in different kidney compartments and their overlay.")


# Conclusion

This tutorial covered combining datasets, data preprocessing and supervised as well as unsupervised statistical analysis.

![workflow](../../images/msi-diff-abundance-workflow.png "Workflow that was used during this tutorial.")


topic: metabolomics
tutorial: qcxms-predictions
title: Predicting EI+ mass spectra with QCxMS
difficulty: introductory
time_estimation: 1H
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: e693fc2621edc361bdf09967b21a9a85
last_modified: 2026-05-15T22:28:13.791831
zenodo_link: https://zenodo.org/record/13327051
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/qcxms-predictions/tutorial.html

content:
Mass spectrometry (MS) is a powerful analytical technique used in many fields, including proteomics, metabolomics, drug discovery and many more areas relying on compound identifications. Even though nowadays MS is a standard and popular method, there are many compounds which lack experimental spectra. In those cases, predicting mass spectra from the chemical structure can reveal useful information, help in compound identification and expand the spectral databases, improving the accuracy and efficiency of database search. [{% cite Zhu2023 %}, {% cite Allen2016Computational %}]. 
There have been several methods developed to predict mass spectra, which can be classified as either first-principles physical-based simulation or data-driven statistical methods [{% cite Zhu2023 %}]. To the first category we can assign purely statistical theories (quasi-equilibrium theory (QET) or Rice–Ramsperger–Kassel–Marcus (RRKM) theories) [{% cite Vetter1994 %}], as well as QCxMS [{% cite Koopman2021 %}] and semiempirical GFNn-xTB [{% cite Koopman2019 %}] which use Born–Oppenheimer molecular dynamics (MD) combined with fragmentation pathways. Data-driven statistical methods - forming the second category - reach back to 1960s when the DENDRAL project (using rule-based heuristic programming) was started by early artificial intelligence (AI) scientists [{% cite Lindsay1980ApplicationsOA %}]. More recently, CFM-ID has been introduced [{% cite Allen2014 %}, {% cite Allen2014metabolomics %}, {% cite Allen2016Computational %}, {% cite DjoumbouFeunang2019 %}, {% cite Wang2021 %}], which uses rule-based fragmentation and employs machine learning methods. Current advancements in machine learning led to recent work using deep neural networks that allow predicting spectra from molecular graphs or fingerprints [{% cite Wei2019 %}].

You will be able to check out how QCxMS works in practice since we are going to use Galaxy tool suite based on this method [{% cite Grimme2013 %}, {% cite Bauer2014 %}, {% cite Bauer2016 %}, {% cite Koopman2021 %}]. Beforehand, we will generate conformers of the query molecule with [RDKit](http://www.rdkit.org) and we will use xTB for molecular optimisation [{% cite Bannwarth2020 %}].
But first things first, let’s get some toy data to play with and crack on! 

> <question-title></question-title>
>
> What does QCEIMS stand for?
>
>   > <tip-title>Hint</tip-title>
>   > With a little knowledge of chemistry, you’ll be able to work it out yourself!
>   >
>   > Look at the acronym in the following way: **QC-EI-MS**.
>   {: .tip}
> > <solution-title></solution-title>
> >
> > QC = quantum chemical 
> >
> > EI = electron ionisation
> >
> > MS = mass spectrometry
> >
> > Hence QCEIMS = quantum-chemical electron ionisation mass spectrometry
> >
> {: .solution}
>
{: .question}

> <details-title>What is QCxMS?</details-title>
> QCxMS is a successor of QCEIMS, where the *EI* part is replaced by *x* to take into account other ionisation methods and improve the applicability of the program. In QCEIMS, *EI* stands for *electron ionisation*, while in QCxMS, *x* refers to *EI* or *CID (collision-induced dissociation)* [{% cite Koopman2021 %}]. Currently, only *EI* simulations are supported - using *CID* with the Galaxy tool wrappers is still under development.
{: .details}

 


# Importing data and pre-processing

In this tutorial, you can choose whether you want to predict the mass spectrum for one molecule only, or if you want to do it for multiple molecules at once. The pre-processing steps will slightly differ depending on your choice. If you are completing this tutorial just to see how the QCxMS tools work, feel free to follow the instructions for one molecule to skip some pre-processing steps.  

In both cases, we start from molecule’s SMILES, and then we convert it to SDF, so if you already have SDF files to work with, simply jump in the relevant place in the workflow and carry on from there.

> <details-title>What is SMILES?</details-title>
>
> SMILES (.smi) - the simplified molecular-input line-entry system (SMILES) is a specification in the form of a line notation for describing the structure of a chemical species using short ASCII strings. It is a linear text format which can describe the connectivity and chirality of a molecule [{% cite Weininger1988 %}]. Even though many different forms of SMILES exist, the differences are not relevant for us in this application.
> ![Depiction of a molecular structure and the corresponding SMILES string. Starts of rings are denoted with a number after the corresponding element (i.e. C1). Non-organic elements need to be written in brackets and isotopic forms need to be denoted before the atom and in brackets as well. Atoms with charges need to be in brackets as well and the charge needs to be denoted behind the atom or the functional group. Hydrogens inside brackets need to be stated explicitly. Double bonds are denoted with a `=` and triple bonds with a `#`.](../../images/qcxms_smiles_explanation.png "A quick explanation of the SMILES system.")
> Image credit: Helge Hecht, License: MIT
>
{: .details}

<!---
consider the figure with SMILES and the corresponding structure
-->


{% include _includes/cyoa-choices.html option1="Predict MS for a single molecule" option2="Predict MS for multiple molecules at once" default="Predict MS for a single molecule"
       text="Choose below if you just want to follow the pipeline for prediting the spectrum for only one molecule or multiple molecules at once!" disambiguation="single_vs_multiple" %}

<div class="Predict-MS-for-a-single-molecule" markdown="1">

## Upload data onto Galaxy

Working on a single molecule means that we will work on a dataset and not on a collection of datasets. To simulate the spectrum, we will use ethanol (C<sub>2</sub>H<sub>5</sub> OH) as an example, but you can choose any other molecule that you want, but be aware that the more complex structure you choose, the more time it will take to complete the analysis since the workflow involves generating conformers, semiempirical methods and molecular optimisation.
We will simply start with molecule’s SMILES. 
You have three options for uploading the data. The first two - importing via history and Zenodo link will give a file specific to this tutorial, while the last one – “Paste data uploader" gives you more flexibility in terms of the compounds you would like to test with this workflow. 

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. You can simply import [this history](https://usegalaxy.eu/u/j.jakiela/h/input-file-ei-mass-spectra-prediction-using-qcxms-1) with the input file. 
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the history to your name of choice.
>
{: .hands_on}

><hands-on-title>Option 2: Data upload - Add to history via Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the input table from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/records/13327051/files/ethanol_SMILES.smi
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

> <hands-on-title> Option 3: Data Upload  - paste data </hands-on-title>
> 
> 1. Create a new history for this tutorial
> 2. 
>    - Click {% icon galaxy-upload %} **Upload Data** at the top of the tool panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Paste the SMILES into the text field:  
> ```
> CCO
> ```
>    - Change **Type** from "Auto-detect" to `smi`
>    - Press **Start** and **Close** the window
> 3. You can then rename the dataset as you wish (here we use `ethanol_SMILES`)
> 4. Check that the datatype is `smi`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="smi" %}
>
{: .hands_on}

## SMILES to SDF

> <hands-on-title> Convert SMILES to SDF </hands-on-title>
>
> {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: `ethanol_SMILES` (your SMILES dataset from your history)
>    - *"Output format"*: `MDL MOL format (sdf, mol)`
>    - *"Append the specified text after each molecule title"*: `ethanol`
>
{: .hands_on}

We now have an SDF file, containing the atoms' coordinates and the investigated molecule's name. 

# 3D Conformer generation & optimization

## Generate conformers
The next step involves generating three-dimensional (3D) conformers for our molecule. It crteaes the actual 3D topology of the molecule based on electromagnetic forces. This process might seem trivial for very small and simplistic (meaning no complex structure) molecules, but this can be more challenging for larger molecules with a more flexible geometry. This concerns for example P containing biomolecules, where P often forms a rotational center of the molecule. The number of conformers to generate can be specified as an input parameter, with a default value of 1 if not provided. This process is crucial for exploring the possible shapes and energies that a molecule can adopt. The output of this step is a file containing the generated 3D conformers.

> <details-title> What are conformers? </details-title>
>
> Conformers are different spatial arrangements of a molecule that result from rotations around single bonds. They have different potential energies and hence some are more favourable (local minima on the potential energy surface) than others. 
> ![Newman projections of butane conformations & their relative energy differences (not total energies). Conformations form when butane rotates about one of its single covalent bond. Torsional/dihedral angle is shown on x-axis.](../../images/qcxms_predictions_conformers.svg "Conformers of butane and their relative energy differences.")
> Image credit: [Keministi](https://commons.wikimedia.org/wiki/File:Butane_conformations_and_relative_energies.svg), License: Creative Commons CC0 1.0. 
>
{: .details}


> <hands-on-title> Generate conformers </hands-on-title>
>
>  {% tool [Generate conformers](toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_conformers/ctb_im_conformers/1.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of **Convert to sdf** {% icon tool %}
>    - *"Number of conformers to generate"*: `1`
>
{: .hands_on}

Now - once again format conversion! This time we will convert the generated conformers from the SDF format to Cartesian coordinate (XYZ) format. The XYZ format lists the atoms in a molecule and their respective 3D coordinates, which is a common format used in computational chemistry for further processing and analysis.

> <hands-on-title> Molecular Format Conversion </hands-on-title>
>
> 1. {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: output (sdf) of **Generate conformers** {% icon tool %}
>    - *"Output format"*: `XYZ cartesian coordinates format`
>    - *"Add hydrogens appropriate for pH"*: `7.0`
>
> 2. Check that the datatype is `xyz`. If it's not, just change it – below is the tip how to do it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="xyz" %}
>
{: .hands_on}


## Molecular optimization

As shown in the image in the {% icon details %} *Details box* above, different conformers have different energies. Therefore, our next step will optimize the geometry of the molecules to find the lowest energy conformation. This is crucial to achieve convergence in the next steps. If the input geometry for the QCxMS method is too crude, the ground state neutral run will not converge and we won't be able to sample the geometry to calculate individual trajectories. We will perform semi-empirical optimization on the molecules using the [Extended Tight-Binding (xTB)](https://github.com/grimme-lab/xtb) method. The level of optimization accuracy to be used can be specified as an input parameter, *"Optimization Levels"*. The default quantum chemical method is GFN2-xTB.

> <hands-on-title> Molecular optimisation with xTB </hands-on-title>
>
> {% tool [xtb molecular optimization](toolshed.g2.bx.psu.edu/repos/recetox/xtb_molecular_optimization/xtb_molecular_optimization/6.6.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Atomic coordinates file"*: output of **Convert to xyz** {% icon tool %})
>    - *"Optimization Levels"*: `tight`
>    - *"Keep molecule name"*: {% icon galaxy-toggle %}  `Yes`
>
{: .hands_on}


# QCxMS Spectra Prediction 

## Neutral and production runs

Finally, let’s predict the spectra for our molecule. As mentioned, we will use [QCxMS](https://github.com/qcxms/QCxMS) for this purpose. First, we need to prepare the necessary input files for the QCxMS production runs. These files are required for running the QCxMS simulations, which will predict the mass spectrum of the molecule. This step typically formats the optimized molecular data into a format that can be used for the production simulations. This step performs the ground state calculations. The resulting geometry trajectory is then sampled and one representation is used for each trajectory.


> <hands-on-title> QCxMS neutral run </hands-on-title>
>
>  {% tool [QCxMS neutral run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_neutral_run/qcxms_neutral_run/5.2.1+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Molecule 3D structure [.xyz]"*: output of **xtb molecular optimization** {% icon tool %}
>    - *"QC Method"*: `GFN2-xTB`
>
{: .hands_on}

The outputs of the above step are as follows:
-	.in output: Input file for the QCxMS production run 
-	.start output: Start file for the QCxMS production run 
-	.xyz output: Cartesian coordinate file for the QCxMS production run

We can now use those files as input for the next tool which calculates the mass spectra using QCxMS. This simulation generates .res files, which contain the raw results of the mass spectra calculations. These results are essential for predicting how the molecules will appear in mass spectrometry experiments.

> <hands-on-title> QCxMS production run </hands-on-title>
>
> 1. {% tool [QCxMS production run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_production_run/qcxms_production_run/5.2.1+galaxy3) %} with the following parameters:
>    - {% icon param-collection %} *Dataset collection* *"in files [.in]"*: `input in files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"start files [.start]"*: `input start files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"xyz files [.xyz]"*: `input xyz files` generated by **QCxMS neutral run** {% icon tool %}
>
{: .hands_on}


## Filter failed datasets

It might be the case that some runs might have failed, therefore it is crucial to filter out any failed runs from the dataset to ensure only successful results are processed further. This step is important to maintain the integrity and quality of the data being analyzed in subsequent steps. The output is a file containing only the successful mass spectra results.

> <hands-on-title> Filter failed datasets </hands-on-title>
>
> 1. {% tool [Filter failed datasets](__FILTER_FAILED_DATASETS__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: `res files generated by QCxMS` (output of **QCxMS production run** {% icon tool %} )
>
{: .hands_on}


## Get MSP spectra

The filtered collection contains .res files from the QCxMS production run. This final step converts the .res files into simulated mass spectra in MSP (Mass Spectrum Peak) file format. The MSP format is widely used for storing and sharing mass spectrometry data, enabling easy comparison and analysis of the results, for example by comparing the spectra using the [matchms](https://github.com/matchms/matchms) package.

> <hands-on-title> QCxMS get MSP results </hands-on-title>
>
> 1. {% tool [QCxMS get results](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_getres/qcxms_getres/5.2.1+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Molecule 3D structure [.xyz]"*: `Convert to xyz` (output of **Compound conversion** {% icon tool %})
>    - *"res files [.res]"*: output of **Filter failed datasets** {% icon tool %} (if the collection doesn't appear in the drop-down list, simply drag and drop it from the history panel to the input box)
>
{: .hands_on}

> <details-title>MSP files </details-title>
> MSP (Mass Spectrum Peak) file is a text file structured according to the NIST MSSearch spectra format. MSP is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programmes (MS-DIAL, NIST MS Search, AMDIS, matchms, etc.). It can contain one or more mass spectra, these are split by an empty line. The individual spectra essentially consist of two sections: metadata (such as name, spectrum type, ion mode, retention time, and the number of m/z peaks) and peaks, consisting of m/z and intensity tuples.
{: .details}

You can now {% icon dataset-save %} download the MSP file and open it in your spectra processing software for further investigation! 

To give you some insight into how well QCxMS can perform, below is the mass spectrum of ethanol resulting from our workflow compared with an [experimental spectrum](https://hmdb.ca/spectra/c_ms/28442). Both spectra were compiled using an [online mass spectrum generator](https://www.sisweb.com/mstools/spectrum.htm) which requires only m/z values and intensities – so the values that you can get from our MSP file! As you can see, the predicted peaks nicely correspond to experimental ones. But be careful - there might be slight deviations for molecules with more structural complexity! 

![Upper panel shows the experimental spectrum of ethanol, while the lower panel shows analogical spectrum but predicted with the current workflow. The predicted peaks correspond well to the experimental ones. Intensities of simulated peaks have not been predicted perfectly, but the most important trends are preserved.](../../images/qcxms_predictions_ms_exp_pred.png "Comparison between experimental (upper panel) and predicted (lower panel) mass spectra of ethanol.")


# Conclusion
{% icon trophy %} Well done, you’ve simulated the mass spectrum! You might want to consult your results with the [key history](https://usegalaxy.eu/u/j.jakiela/h/ms-prediction-qcxms-single-molecule). If you would like to process multiple molecules at once, you can simply use [the workflow](https://usegalaxy.eu/u/hechth/h/end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms-1) or move to “Predict MS for multiple molecules at once” tab of this tutorial to learn how the pipeline differs from the one that we’ve just covered. 


</div>

<div class="Predict-MS-for-multiple-molecules-at-once" markdown="1">

## Upload data onto Galaxy

We will work on two simple molecules – ethanol (C<sub>2</sub>H<sub>5</sub> OH) and ethylene (C<sub>2</sub>H<sub>4</sub>). Of course, you might add more and choose any other molecules that you want, but be aware that the more complex structure you choose, the more time it will take to complete the analysis since the workflow involves generating conformers, semiempirical methods and molecular optimisation.
We will start with a table with the first column being molecule names and the second one – corresponding SMILES. 
You have three options for uploading the data. The first two - importing via history and Zenodo link will give a file specific to this tutorial, while the last one – “Paste data uploader" gives you more flexibility in terms of the compounds you would like to test with this workflow. 

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. You can simply import [this history](https://usegalaxy.eu/u/j.jakiela/h/input-file-end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms) with the input table. 
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the history to your name of choice.
>
{: .hands_on}

><hands-on-title>Option 2: Data upload - Add to history via Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the input table from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/records/13327051/files/qcxms_prediction_input.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

> <hands-on-title> Option 3: Data Upload  - paste data </hands-on-title>
> 
> 1. Create a new history for this tutorial
> 2. 
>    - Click {% icon galaxy-upload %} **Upload Data** at the top of the tool panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Paste the contents into the text field, separated by space. First, enter the name of the molecule, then its SMILES. Please note that we are not using headers here. For this tutorial, we’ll use the example of ethanol and ethylene, but feel free to use your own examples.  
> ```
> ethanol CCO
> ethylene C=C
> ```
>    - Change **Type** from "Auto-detect" to `tabular`
>    - Find the gear symbol ({% icon galaxy-gear %}), deselect any ticked options and select only ({% icon galaxy-gear %}) **Convert spaces to tabs**
>    - Press **Start** and **Close** the window
> 3. You can then rename the dataset as you wish.
> 4. Check that the datatype is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

## Input pre-processing

Once your dataset is uploaded, we can do some simple pre-processing to prepare the file for downstream analysis. Let’s start with ‘cutting’ the table into two columns – one with SMILES, the other with the molecule name – and then separating each entry to create a dataset collection, followed by parsing out the text information. 

> <hands-on-title> Cutting out name column </hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: the tabular file in your history with the compound name and SMILES
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>    - *"Delimited by"*: `Tab`
>    - *"Is there a header for the data’s columns"*: `No`
>    - *"List of Fields"*: `Column: 1`
>
> 2. You can now rename the resulting dataset or just add a tag in order not to confuse it with subsequent outputs:
>    - {% icon galaxy-tags %} Add tag: #names
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

> <hands-on-title> Creating dataset collection (molecule name) </hands-on-title>
>
> {% tool [Split file to dataset collection](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.2) %} with the following parameters:
>    - {% icon param-file %} *"Select the file type to split"*: `Tabular`
>    - *"Tabular file to split"*: output of **Advanced Cut** {% icon tool %} (with #names tag)
>    - *"Number of header lines to transfer to new files"*: `0`
>    - *"Split by row or by a column?"*: `By row`
>    - *"Specify number of output files or number of records per file?"*: `Number of records per file (‘chunk mode’)`
>    - *"Chunk size"*: `1`
>    - *"Base name for new files in collection"*: `split_file`
>    - *"Method to allocate records to new files"*: `Maintain record order`
>
{: .hands_on}

> <hands-on-title> Parsing out name info </hands-on-title>
>
>  {% tool [Parse parameter value](param_value_from_file) %} with the following parameters:
>    - *"Input file containing parameter to parse out of"*: * click on {% icon param-collection %} *Dataset collection* and select output of **Split file** {% icon tool %}
>    - *"Select type of parameter to parse"*: `Text`
>    - *"Remove newlines ?"*: {% icon galaxy-toggle %}  `Yes`
>
>   > <tip-title>Can't find "Parse parameter value" tool? </tip-title>
>   > If you have any problems with accessing **Parse parameter value** tool, you can open [the tool](https://usegalaxy.eu/root?tool_id=param_value_from_file) directly using a given link.
>   {: .tip}
>
{: .hands_on}



We will repeat the first two steps, but processing SMILES this time. 


> <hands-on-title> Cutting out SMILES column </hands-on-title>
>
>  1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: the tabular file in your history with the compound name and SMILES
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>    - *"Delimited by"*: `Tab`
>    - *"Is there a header for the data’s columns"*: `No`
>    - *"List of Fields"*: `Column: 2`
>
> 2. You can now rename the resulting dataset or just add a tag in order not to confuse it with subsequent outputs:
>    - {% icon galaxy-tags %} Add tag: #SMILES
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


> <hands-on-title>  Creating dataset collection (SMILES) </hands-on-title>
>
>  1. {% tool [Split file to dataset collection](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.2) %} with the following parameters:
>    - {% icon param-file %} *"Select the file type to split"*: `Tabular`
>    - *"Tabular file to split"*: output of the latest **Advanced Cut** {% icon tool %} (higher number in your history, with #SMILES tag)
>    - *"Number of header lines to transfer to new files"*: `0`
>    - *"Split by row or by a column?"*: `By row`
>    - *"Specify number of output files or number of records per file?"*: `Number of records per file (‘chunk mode’)`
>    - *"Chunk size"*: `1`
>    - *"Base name for new files in collection"*: `split_file`
>    - *"Method to allocate records to new files"*: `Maintain record order`
>
> 2. Check that the datatype is `smi`. If it's not, just change it! 
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="smi" %}
>
{: .hands_on}

Now, onto format conversion. Let’s convert our SMILES to SDF (Structure Data File) and append the molecule’s name that we have already extracted. 

> <hands-on-title> Convert SMILES to SDF </hands-on-title>
>
> {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - *"Molecular input file"*: click on {% icon param-collection %} *Dataset collection* and select output of **Split file** {% icon tool %} on SMILES
>    - *"Output format"*: `MDL MOL format (sdf, mol)`
>
>   > <tip-title>Add name of the molecule</tip-title>
>   > Another parameter of this tool, *"Append the specified text after each molecule title"*, allows you to add name of the molecule to the title of the generated file. 
>   > If you are working on a single molecule file, you can simply type the name of that compound into the parameter box.
>   > However, if your input is a collection of SMILES (like in this case), then you have to add the names (output of **Parse parameter value** {% icon tool %}) at the level of the workflow editor. 
>   {: .tip}
>
{: .hands_on}

<!---
- *"Append the specified text after each molecule title"*: click on {% icon param-collection %} *Dataset collection* and select output of **Parse parameter value** {% icon tool %} which contains the extracted molecules' names
-->


We now have two SDF files, each containing the coordinates of the atoms and the name of the investigated molecule. Let’s combine them to make life easier and work on just one file. 

> <hands-on-title> Concatenating the files </hands-on-title>
>
> {% tool [Concatenate datasets tail-to-head (cat)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Datasets to concatenate"*: 
>    * if nothing pops up when you click on {% icon param-collection %} *Dataset collection*, then click on "switch to column select" 
>    * from the datasets list, find the output of **Compound conversion** {% icon tool %} (should have the highest number in your history) and select it
>    * {% icon workflow-run %} Run tool! 
>
{: .hands_on}

# 3D Conformer generation & optimization

## Generate conformers
The next step involves generating three-dimensional (3D) conformers for each molecule from the generated SDF. It crteaes the actual 3D topology of the molecules based on electromagnetic forces. This process might seem trivial for very small and simplistic (meaning no complex structure) molecules, but this can be more challenging for larger molecules with a more flexible geometry. This concerns for example P containing biomolecules, where P often forms a rotational center of the molecule. The number of conformers to generate can be specified as an input parameter, with a default value of 1 if not provided. This process is crucial for exploring the possible shapes and energies that a molecule can adopt. The output of this step is a file containing the generated 3D conformers.

> <details-title> What are conformers? </details-title>
>
> Conformers are different spatial arrangements of a molecule that result from rotations around single bonds. They have different potential energies and hence some are more favourable (local minima on the potential energy surface) than others. 
> ![Newman projections of butane conformations & their relative energy differences (not total energies). Conformations form when butane rotates about one of its single covalent bond. Torsional/dihedral angle is shown on x-axis.](../../images/qcxms_predictions_conformers.svg "Conformers of butane and their relative energy differences.")
> Image credit: [Keministi](https://commons.wikimedia.org/wiki/File:Butane_conformations_and_relative_energies.svg), License: Creative Commons CC0 1.0. 
>
{: .details}


> <hands-on-title> Generate conformers </hands-on-title>
>
>  {% tool [Generate conformers](toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_conformers/ctb_im_conformers/1.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of **Concatenate datasets** {% icon tool %}
>    - *"Number of conformers to generate"*: `1`
>
{: .hands_on}

Now - once again format conversion! This time we will convert the generated conformers from the SDF format to Cartesian coordinate (XYZ) format. The XYZ format lists the atoms in a molecule and their respective 3D coordinates, which is a common format used in computational chemistry for further processing and analysis.

> <hands-on-title> Molecular Format Conversion </hands-on-title>
>
> 1. {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: output of **Generate conformers** {% icon tool %}
>    - *"Output format"*: `XYZ cartesian coordinates format`
>    - *"Split multi-molecule files into a collection"*: {% icon galaxy-toggle %} `Yes`
>    - *"Add hydrogens appropriate for pH"*: `7.0`
>
> 2. Check that the datatype is `xyz`. If it's not, just change it! 
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="xyz" %}
>
{: .hands_on}


## Molecular optimization

As shown in the image in the {% icon details %} *Details box* above, different conformers have different energies. Therefore, our next step will optimize the geometry of the molecules to find the lowest energy conformation. This is crucial to achieve convergence in the next steps. If the input geometry for the QCxMS method is too crude, the ground state neutral run will not converge and we won't be able to sample the geometry to calculate individual trajectories. We will perform semi-empirical optimization on the molecules using the [Extended Tight-Binding (xTB)](https://github.com/grimme-lab/xtb) method. The level of optimization accuracy to be used can be specified as an input parameter, *"Optimization Levels"*. The default quantum chemical method is GFN2-xTB.

> <hands-on-title> Molecular optimisation with xTB </hands-on-title>
>
> {% tool [xtb molecular optimization](toolshed.g2.bx.psu.edu/repos/recetox/xtb_molecular_optimization/xtb_molecular_optimization/6.6.1+galaxy1) %} with the following parameters:
>    - *"Atomic coordinates file"*: click on {% icon param-collection %} *Dataset collection* and select `Prepared ligands` (output of **Compound conversion** {% icon tool %})
>    - *"Optimization Levels"*: `tight`
>    - *"Keep molecule name"*: {% icon galaxy-toggle %}  `Yes`
>
{: .hands_on}


# QCxMS Spectra Prediction 

## Neutral and production runs

Finally, let’s predict the spectra for our molecules. As mentioned, we will use [QCxMS](https://github.com/qcxms/QCxMS) for this purpose. First, we need to prepare the necessary input files for the QCxMS production runs. These files are required for running the QCxMS simulations, which will predict the mass spectra of the molecules. This step typically formats the optimized molecular data into a format that can be used for the production simulations. This step performs the ground state calculations. The resulting geometry trajectory is then sampled and one representation is used for each trajectory.


> <hands-on-title> QCxMS neutral run </hands-on-title>
>
>  {% tool [QCxMS neutral run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_neutral_run/qcxms_neutral_run/5.2.1+galaxy3) %} with the following parameters:
>    - *"Molecule 3D structure [.xyz]"*: click on {% icon param-collection %} *Dataset collection* and select output of **xtb molecular optimization** {% icon tool %}
>    - *"QC Method"*: `GFN2-xTB`
>
{: .hands_on}

The outputs of the above step are as follows:
-	.in output: Input file for the QCxMS production run 
-	.start output: Start file for the QCxMS production run 
-	.xyz output: Cartesian coordinate file for the QCxMS production run

We can now use those files as input for the next tool which calculates the mass spectra for each molecule using QCxMS (Quantum Chemistry and Mass Spectrometry). This simulation generates .res files, which contain the raw results of the mass spectra calculations. These results are essential for predicting how the molecules will appear in mass spectrometry experiments.

> <hands-on-title> QCxMS production run </hands-on-title>
>
> 1. {% tool [QCxMS production run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_production_run/qcxms_production_run/5.2.1+galaxy3) %} with the following parameters:
>    - {% icon param-collection %} *Dataset collection* *"in files [.in]"*: `input in files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"start files [.start]"*: `input start files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"xyz files [.xyz]"*: `input xyz files` generated by **QCxMS neutral run** {% icon tool %}
>
{: .hands_on}


## Filter failed datasets

It might be the case that some runs might have failed, therefore it is crucial to filter out any failed runs from the dataset to ensure only successful results are processed further. This step is important to maintain the integrity and quality of the data being analyzed in subsequent steps. The output is a file containing only the successful mass spectra results.

> <hands-on-title> Filter failed datasets </hands-on-title>
>
> 1. {% tool [Filter failed datasets](__FILTER_FAILED_DATASETS__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: output of **QCxMS production run** {% icon tool %}
>
{: .hands_on}


## Get MSP spectra

The filtered collection contains .res files from the QCxMS production run. This final step converts the .res files into simulated mass spectra in MSP (Mass Spectrum Peak) file format. The MSP format is widely used for storing and sharing mass spectrometry data, enabling easy comparison and analysis of the results, for example by comparing the spectra using the [matchms](https://github.com/matchms/matchms) package.

> <hands-on-title> QCxMS get MSP results </hands-on-title>
>
> 1. {% tool [QCxMS get results](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_getres/qcxms_getres/5.2.1+galaxy2) %} with the following parameters:
>    - *"Molecule 3D structure [.xyz]"*: click on {% icon param-collection %} *Dataset collection* and select `Prepared ligands` (output of **Compound conversion** {% icon tool %})
>    - {% icon param-file %} *"res files [.res]"*: output of **Filter failed datasets** {% icon tool %} (if the collection doesn't appear in the drop-down list, simply drag and drop it from the history panel to the input box)
>
{: .hands_on}

The output of this step is a collection of two MSP files - one per each molecule. However, if you want, you can combine those two into one MSP file. 

> <hands-on-title> Combine MSP files into one </hands-on-title>
>
> 1. {% tool [Concatenate datasets tail-to-head (cat)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Datasets to concatenate"*: 
>    * if nothing pops up when you click on {% icon param-collection %} *Dataset collection*, then click on "switch to column select" 
>    * from the datasets list, find the output of **QCxMS get results** {% icon tool %} (should be the latest output dataset) and select it
>    * {% icon workflow-run %} Run tool! 
>
{: .hands_on}

> <details-title>MSP files </details-title>
> MSP (Mass Spectrum Peak) file is a text file structured according to the NIST MSSearch spectra format. MSP is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programmes (MS-DIAL, NIST MS Search, AMDIS, matchms, etc.). It can contain one or more mass spectra, these are split by an empty line. The individual spectra essentially consist of two sections: metadata (such as name, spectrum type, ion mode, retention time, and the number of m/z peaks) and peaks, consisting of m/z and intensity tuples.
{: .details}

You can now {% icon dataset-save %} download the MSP file and open it in your spectra processing software for further investigation! 

To give you some insight into how well QCxMS can perform, below is the mass spectrum of ethanol resulting from our workflow compared with [experimental spectrum](https://hmdb.ca/spectra/c_ms/28442). Both spectra were compiled using an [online mass spectrum generator](https://www.sisweb.com/mstools/spectrum.htm) which requires only m/z values and intensities – so the values that you can get from our MSP file! As you can see, the predicted peaks nicely correspond to experimental ones. But be careful - there might be slight deviations for molecules with more structural complexity! 

![Upper panel shows the experimental spectrum of ethanol, while the lower panel shows analogical spectrum but predicted with the current workflow. The predicted peaks correspond well to the experimental ones. Intensities of simulated peaks have not been predicted perfectly, but the most important trends are preserved.](../../images/qcxms_predictions_ms_exp_pred.png "Comparison between experimental (upper panel) and predicted (lower panel) mass spectra of ethanol.")


# Conclusion
{% icon trophy %} Well done, you’ve simulated mass spectra! You might want to consult your results with the [key history](https://usegalaxy.eu/u/hechth/h/end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms-1) or use [the workflow](https://usegalaxy.eu/u/hechth/h/end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms-1) associated with this tutorial. 

</div>

The prediction of mass spectra might be very useful, particularly for compounds that lack experimental data. Simulating the spectra can also save time and resources. This field has been developing quite rapidly, and recent advancements in new algorithms and packages have led to more and more accurate results. However, one cannot forget that this kind of software should be used to deepen our chemical understanding of the structures of studied compounds and not as a replacement for practical experiments.

tools_json:
["Compound conversion", "toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0", "Generate conformers", "toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_conformers/ctb_im_conformers/1.1.4+galaxy0", "xtb molecular optimization", "toolshed.g2.bx.psu.edu/repos/recetox/xtb_molecular_optimization/xtb_molecular_optimization/6.6.1+galaxy1", "QCxMS neutral run", "toolshed.g2.bx.psu.edu/repos/recetox/qcxms_neutral_run/qcxms_neutral_run/5.2.1+galaxy3", "QCxMS production run", "toolshed.g2.bx.psu.edu/repos/recetox/qcxms_production_run/qcxms_production_run/5.2.1+galaxy3", "Filter failed datasets", "__FILTER_FAILED_DATASETS__", "QCxMS get results", "toolshed.g2.bx.psu.edu/repos/recetox/qcxms_getres/qcxms_getres/5.2.1+galaxy2", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1", "Split file to dataset collection", "toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.2", "Parse parameter value", "param_value_from_file", "Concatenate datasets tail-to-head (cat)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/9.3+galaxy1"]


topic: metabolomics
tutorial: gcms
title: Mass spectrometry: GC-MS analysis with the metaMS package
difficulty: introductory
time_estimation: 2H
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: c4033be37f6f5b48d1b8256e1a8e0998
last_modified: 2026-05-27T09:01:49.251265
zenodo_link: https://zenodo.org/records/16538501
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/gcms/tutorial.html

content:
You may already know that there are different types of *-omic* sciences; out of these, metabolomics is most closely related to phenotypes. Metabolomics involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants. It  focuses on studying the very small molecules which are called *metabolites*, to better understand matters linked to the metabolism. However, studying metabolites is not a piece of cake since it requires several critical steps which still have some major bottlenecks. Metabolomics is still quite a young science, and has many kinds of specific challenges.

One of the three main technologies used to perform metabolomic analysis is **Gas-Chromatography Mass Spectrometry** (GC-MS). Data analysis for this technology requires a large variety of steps, ranging from extracting information from the raw data, to statistical analysis and annotation. Many packages in R/Python are available for the analysis of GC-MS or LC-MS (Liquid-Chromatography Mass Spectrometry) data - for more details see the reviews by {% cite Stanstrup2019 %} and {% cite Misra2021 %}.

This tutorial explains the main steps involved in untargeted **GC-MS** data processing. To do so we focus on some open-source solutions integrated within the Galaxy framework, namely **XCMS** and **metaMS**. The selected tools and functionalities only covers a small portion of available tools but allow to **perform a complete GC-MS analysis** in a single environment.
In this tutorial, we will learn how to (1) extract features from the raw data using **XCMS** ({% cite Smith2006 %}), (2) deconvolute the detected features into spectra with **metaMS** ({% cite Wehrens2014 %}) and (3) annotate unknow spectra using spectral database comparison tools.

To illustrate this approach, we will use data from {% cite Dittami2012 %}. Due to time constraints in processing the original dataset, a limited subset of samples was used to illustrate the workflow. This subset (see details below) demonstrates the key steps of metabolomics analysis, from pre-processing to annotation. Although the results derived from this reduced sample size may not be scientifically robust, they provide insight into essential methodological foundations of GC-MS data-processing workflow.

> <details-title> About the data - Algae samples </details-title>
>
> The objective of the study conducted by Dittami *et al.* was to investigate the adaptation mechanisms of the brown algae *Ectocarpus* to low-salinity environments. The research focused on examining physiological tolerance and metabolic changes in freshwater and marine strains of *Ectocarpus*. Using transcriptomic (gene expression profiling) and metabolic analyses, the authors identified significant, reversible changes occurring in the freshwater strain when exposed to seawater. Both strains exhibited similarities in gene expression under identical conditions; however, substantial differences were observed in metabolite profiles.
> 
> The study utilized a **freshwater strain** of Ectocarpus and a **marine strain** for comparative analysis. The algae were cultured in media with varying salinities, prepared by diluting natural seawater or adding NaCl. The algae were acclimated to these conditions before extraction.
> 
> The six samples used in this training were analyzed by GC-MS (low resolution instrument). A marine strain raised in sea water media (2 replicates) and freshwater strains raised in either 5% or 100% sea water media (2 replicates each). The training dataset is [available on Zenodo](https://zenodo.org/records/16538501)
>
{: .details}

To process the GC-MS data, we can use several tools. One of these is  **XCMS** ({% cite Smith2006 %}), a general R package for untargeted metabolomics profiling. It can be used for any type of mass spectrometry acquisition (centroid and profile) and resolution (from low to high resolution), including FT-MS data coupled with a different kind of chromatography (liquid or gas). Because of the generality of packages like **XCMS**, several other packages have been developed to use the functionalities of **XCMS** for optimal performance in a particular context. The R package called **metaMS** ({% cite Wehrens2014 %}) does so for the field of GC-MS untargeted metabolomics. One of the goals of **metaMS** was to set up a simple system with few user-settable parameters, capable of handling untargeted metabolomics experiments.
In this tutorial we use **XCMS** to detect chromatographic peaks within our samples. Once we have detected them, they need to be deconvoluted into mass spectra representing chemical compounds. For that, we use **metaMS** functions. To normalize the retention time of deconvoluted spectra in our sample, we compute the retention index using Alkane references and a dedicated function of **metaMS**. Finally, we identify detected spectra by aligning them with a database of known compounds. This can be achieved using an in-house built database in the common MSP format (`.msp`) (used in the NIST MS search program for example), resulting in a table of annotated compounds.


> <comment-title></comment-title>
> In Galaxy other GC-MS data processing workflows are available and may be of interest for more advanced Galaxy users (see the [Metabolomics section]({% link /topics/metabolomics/ %})).
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data preparation and prepocessing

Before we can start with the actual analysis pipeline, we first need to download and prepare our dataset. Many of the preprocessing steps can be run in parallel on individual samples. Therefore, we recommend using the Dataset collections in Galaxy. This can be achieved by using the dataset collection option from the beginning of your analysis when uploading your data into Galaxy.


## Import the data into Galaxy

> <hands-on-title> Upload data </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) into a collection:
>
>    ```
>    https://zenodo.org/records/16538501/files/alg11.mzML
>    https://zenodo.org/records/16538501/files/alg2.mzML
>    https://zenodo.org/records/16538501/files/alg3.mzML
>    https://zenodo.org/records/16538501/files/alg7.mzML
>    https://zenodo.org/records/16538501/files/alg8.mzML
>    https://zenodo.org/records/16538501/files/alg9.mzML
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="input" renaming=false %}
>
> 3. Make sure your data is in a **collection**. You can always manually create the collection from separate files:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
>    In the further steps, this dataset collection will be referred to as `input` (and we recommend naming this collection like that to avoid confusion).
>
> 4. Import the following extra files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/record/16538501/files/reference_alkanes.csv
>    https://zenodo.org/record/16538501/files/W4M0004_database_small.msp
>    https://zenodo.org/record/16538501/files/sampleMetadata.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
>    > <comment-title> The extra files </comment-title>
>    >
>    > The three additional files contain the **reference_alkanes**, the **W4M0004_database_small**, and the **sampleMetadata**. Those files are auxiliary inputs used in the data processing and contain either extra information about the samples or serve as reference data for indexing and identification.
>    > 
>    > The **reference_alkanes** (`.tsv` or `.csv`) with retention times and carbon number or retention index is used to compute the retention index of the deconvoluted peaks. The alkanes should be measured in the same batch as the input sample collection.
>    > 
>    > The **W4M0004_database_small** (`.msp`) is a reference database used for the identification of spectra. It contains the recorded and annotated mass spectra of chemical standards, ideally from a similar instrument. The unknown spectra which can be detected in the sample can then be confirmed via comparison with this library. The specific library is an in-house library of metabolite standards extracted with **metaMS** .
>    > 
>    > The **sample metadata** (`.csv` or `.tsv`) is a table containing information about our samples. In particular, the tabular file contains for each sample its associated sample name, class (SW, FWS, etc.). It is possible to add more columns to include additional details about the samples (e.g : batch number, injection order...).
>    {: .comment}
>
{: .hands_on}

As a result of this step, you should have in our history a green Dataset collection {% icon param-collection %} with all 6 samples `.mzML` files as well as three separate files with reference alkanes, reference spectral library, and sample metadata.

## Create the XCMS object

The first part of data processing is using the **XCMS** tool to detect peaks in the MS signal. For that, we first need to take the `.mzML` files and create a format usable by the **XCMS** tool. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} ({% cite gatto2012msnbase %}. {% cite gatto2020msnbase %}) takes as input our files and prepares `RData` files for the first **XCMS** step.


> <comment-title></comment-title>
> The **MSnbase readMSData** {% icon tool %} function, prior to **XCMS**, is able to read files with open format as `mzXML`, `mzML`, `mzData` and `netCDF`, which are independent of the manufacturers' proprietary formats. Working with open MS data file format allows users to us tools developed outside of the MS instrument provider. This set of packages/functions gives modularity, and thus is particularly well adapted to define workflows, one of the key points of Galaxy.
> 
{: .comment}

> <hands-on-title> Create the XCMS object </hands-on-title>
>
> 1. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"File(s) from your history containing your chromatograms"*: `input`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    The output should be `input.raw.RData`
>    
>    This dataset is a collection of `rdata.msnbase.raw` files. That `Rdata` file is necessary in the next step of the workflow.  It contains an internal R representation of **XCMS** objects.
>    
{: .hands_on}

As a result of this step, you should have in your history a new green dataset collection called `input.raw.RData` that will serve as input for the next processing step: **peak picking**.

# Peak detection using XCMS

The first step in the workflow is to detect the peaks in our data using **XCMS** functions. This part, however, is covered by a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html). Although the tutorial is dedicated to LC-MS data, it can also be followed for our GC-MS data. Therefore, in this section, we do not explain this part of the workflow in detail but rather refer the reader to the dedicated tutorial. Please also pay attention to the parameter values for individual Galaxy tools, as these can differ from the referred tutorial and are adjusted to our GC-MS dataset.

> <tip-title>Short-cut for Experts </tip-title>
> Since this step is already covered in a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html), it is possible to skip it. Instead, you can go directly to [Peak deconvolution]({{ site.baseurl }}/topics/metabolomics/tutorials/gcms/tutorial.html#processing-with-metams-option-1) step using a preprocessed **XCMS** object file prepared for you.
> >
> > <hands-on-title> Upload data </hands-on-title>
> > > 1. Import the following files from [Zenodo]({{ page.zenodo_link }}):
> > >
> > >    ```
> > >    https://zenodo.org/records/16538501/files/xset.merged.RData
> > >    ```
> > >
> > >    {% snippet faqs/galaxy/datasets_import_via_link.md %}
> > > 
> > >     The format of uploaded file containing **XCMS** object should be `.RData`.
> > > 
> > >    {% snippet faqs/galaxy/datatypes_understanding_datatypes.md %}
> {: .hands_on}
{: .tip}


The first step (*called peak picking*) is to extract peaks from each of your data files independently. For this purpose, we use the _MatchedFilter_ chromatographic peak detection algorithm implemented in {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %}.

 One Galaxy Training material already explains how to act with MS data. We encourage you to **follow this link and complete the corresponding tutorial**: [Mass spectrometry: LC-MS preprocessing with XCMS]({% link topics/metabolomics/tutorials/lcms-preprocessing/tutorial.md %}). 
For GC-MS analysis you **don't really need to follow all of this previous tutorial** but for a better understanding of your data, it is recommended to try it with their test dataset.
Concerning the current GC-MS tutorial, you **just have to compute the following steps and specific parameters** described in the hands-on part below (please follow the parameter values below to obtain the same results during the training).



> <hands-on-title>Peak picking of GC-MS data with XCMS</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.RData` (the output collection of files from the **MSnbase readMSData** {% icon tool %} step)
>    - *"Extraction method for peaks detection"*: `MatchedFilter - peak detection in chromatographic space`
>        - *"Full width at half maximum of matched filtration gaussian model peak"*: `5`
>        - *"Step size to use for profile generation"*: `0.5`
>        - In *"Advanced Options"*:
>            - *"Maximum number of peaks that are expected/will be identified per slice"*: `500`
>            - *"Signal to Noise ratio cutoff"*: `2`
>            - *"Minimum difference in m/z for peaks with overlapping Retention Times"*: `0.5`
>
>
>    > <comment-title></comment-title>
>    >
>    > For GC-MS data in profile mode, use the *MatchedFilter* algorithm instead of the *CentWave* algorithm used in the LC-MS tutorial.
>    > For low-resolution GC-MS data, a larger m/z difference (0.5 Da) can be used to separate overlapping peaks. If you have high-resolution data (from GC-ToF or GC-Orbitrap), you can use a smaller m/z difference (0.01 Da).
>    {: .comment}
>
> 2. {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.xset.RData` (output collection of files from the **xcms findChromPeaks (xcmsSet)** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file "*: `sampleMetadata.tsv` (One of the uploaded files from Zenodo)
>
>    > <comment-title></comment-title>
>    >
>    > To merge your data, you need to **input a sampleMetadata file** containing filenames and their metadata informations like their class for example.
>    > If you don't add a sampleMetadata file, the **xcms findChromPeaks Merger** {% icon tool %} tool will **group all your files together**. 
>    > You can also **create your sampleMetadata file** with W4M Galaxy tool {% tool [xcms get a sampleMetadata file](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy3) %} with the following parameters: {% icon param-collection %} *"RData file"*: `input.raw.RData` output from  **MSnbase readMSData** {% icon tool %}. 
>    > Here is an example of the minimum expectations about a sampleMetadata file (**important**: remove file extension from the sample names, file1.mzML should be file1):
>    >
>    >
>    > | sample_name |  class  | 
>    > |:-----------:|:-------:|
>    > |    file1    |   marine strain   |
>    > |    file2    |  pool  |
>    > |    file3    |   freshwater strain  |
>    >
>    {: .comment}
>
{: .hands_on}

The output from **xcms findChromPeaks Merger** {% icon tool %} is an *.RData* file required for the next steps of the process.

There are two available options: 
- using the metaMS strategy specifically designed for GC-MS data deconvolution and annotation
- using a full XCMS process for GC-MS data processing 

Although this training material is dedicated to GC-MS analysis with the metaMS package, the two options are illustrated in this tutorial. Indeed, as mentioned in the introduction, metaMS is based on XCMS functions, with a specific adaptation to GC-MS data. Thus, one can be interested in comparing results that would be obtained with a standard XCMS processing. Consequently, the two options on the same dataset are available here.  


{% include _includes/cyoa-choices.html option1="Deconvolution and annotation using metaMS" option2="Process GC-MS data with XCMS function directly" default="Deconvolution and annotation using metaMS" text="Choose below if you just want to follow the pipeline using **metaMS** or **XCMS** for GC-MS deconvolution and annotation" disambiguation="gcms_metams_vs_xcms" %}

<div class="Deconvolution-and-annotation-using-metaMS" markdown="1">

# Processing with metaMS (option 1)

**metaMS** is an R package for MS-based metabolomics data. It was made to ease GC-MS data deconvolution and alignment steps using functions from **XCMS** and **CAMERA** packages. In its Galaxy implementation, the two main outputs of **metaMS** are: (1) a table of feature intensities in all samples, which can be analyzed with multivariate methods immediately, and (2) an MSP (`.msp`) file containing GC-MS spectra in a common spectral database format.
The biggest difference between **XCMS** only workflow ([option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2)) or **XCMS + metaMS** GC-MS data processing (option 1) is that rather than a feature-based analysis with individual peaks, as in the [option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2) case, **metaMS** performs a pseudospectrum-based analysis and use it to align compound between samples. One other advantage is that **metaMS** allows creation of MSP (`.msp`) spectra export files ready for annotation.

> <comment-title></comment-title>
> > Not all **metaMS** R package functions have been made available in Galaxy.
When run in R, the **metaMS** package offers a lot of possibilities. For more information on the full set of metaMS functions, visit the [metaMS Bioconductor page](https://www.bioconductor.org/packages/release/bioc/html/metaMS.html).
{: .comment}
 
During this part of the tutorial we are interested in GC-MS analysis with **metaMS**, so we will use the *runGC* function of **metaMS** and describe it in detail to understand all the capabilities of that function.
The standard workflow of **metaMS** for GC-MS data is the following: 

![Workflow diagram of metaMS for GC-MS data analysis](../../images/tuto_gcms_workflow_2options.png "Workflow of metaMS for GC-MS data")

The *runGC* function is implemented in **metaMS.runGC {% icon tool %}** tool in Galaxy. It takes as inputs an {% icon param-collection %} *.RData* file after **XCMS** peak picking  and optionally for annotation purposes an alkane reference file (in either `.csv` or `.tsv` format) for RI calculation and/or a spectral database in `.msp` format.


## Deconvolution and Alignment with metaMS

The peak picking is performed by the usual **XCMS** functions and the output file in `.RData` is used for deconvolution and Alignment steps with *runGC* function.

> <hands-on-title> metaMS.runGC </hands-on-title>
> 
> 1.{% tool [metaMS.runGC](toolshed.g2.bx.psu.edu/repos/yguitton/metams_rungc/metams_runGC/3.0.0+metaMS1.24.0-galaxy0) %}  with the following parameters: 
>   - {% icon param-file %} *"Rdata from xcms and merged"*: `xset.merged.RData` (output of the **xcms findChromPeaks Merger** {% icon tool %} step)
>   - *"Settings"*: `user_defined`
>   - *"RT range option"*: `hide` (If set to *show* you can limit the range of RT processed, for example remove solvant delays)
>   - *"RT_Diff"*: `0.05` (Max time deviation in minute to cluster unknown pseudo-spectra between samples)
>   - *"Min_Features*: `5` (Minimal number of features required to have a valid pseudo-spectrum, compound with less ions will be discarded)
>   - *"similarity_threshold"*: `0.7` (Minimum cosine similarity between pseudo-spectra to be considers as equal)
>   - *"min.class.fract"*: `0.5` (Minimal fraction of samples in which a pseudo-spectrum should be present to be kept) 
>   - *"min.class.size"*: `2` (Minimum number of samples in which a pseudo-spectrum should be find) 
>   - *"Use Personnal DataBase option"*: `show` (This activates the *"DB file"* selector) 
>     - {% icon param-file %} *"DB file"*:  `W4M0004_database_small.msp` (The file download from Zenodo; if not available set the *"Use Personnal DataBase option"* to *hide*)
>   - *"Use RI option*: `show` (Choose *hide* if you want to skip RI calculation)
>     - {% icon param-file %} *"RI file"*: `reference_alkanes.csv` (Format should be strictly respected)
>   - *"Use RI as filter"*: `FALSE` (If set to TRUE only unknown spectra with close RI as those in database will be kept)
>   - *"RIshift"*: *"not used"*
>
>    > <comment-title></comment-title>
>    >
>    > For faster processing keep annotation modules *off* by setting *"Use Personnal DataBase option"*: `hide` and *"Use RI option*: `hide`
>    {: .comment}
> 
>    > <tip-title>Export MSP file to external databases </tip-title>
>    > >You can {% icon dataset-save %} download the MSP file and open it in your favorite spectra processing software or online database for further investigation!
>    {: .tip}
> 
{: .hands_on}

> <details-title> Definitions </details-title>
>    > <details-title> Pseudo-spectra </details-title>
>    > The biggest difference between **XCMS** only or **XCMS + metaMS** GC-MS data processing is that rather than a feature-based analysis with individual peaks, as it is the case with **XCMS**, **metaMS** performs a pseudospectrum-based analysis. So, the basic entity is a set of m/z values showing a chromatographic peak at the same retention time. The idea behind that is that Electron Ionization (EI), which is the most widely used ionization mode in GC-MS analysis, generates a lot more ions for the same molecule than Electrospray Ionisation used in LC-MS. The *runGC* function from **metaMS** is able to group all ions belonging to a molecule into one single cluster that will be used for statistical analysis. For each compound found by **metaMS**, a list of grouped m/z and their intensity is exported as pseudospectrum and this will be used for annotation purpose. For that, the MSP file format is used, which is a common format for mass spectra databases. The pseudospectra are created by grouping all m/z values of a chromatographic peak at the same retention time into one single entry, and then exporting this information in the `.msp` format.
>    > 
>    > ![Example of visualisation of pseudospectrum information](../../images/tuto_gcms_eic.png "Example of cluster of ions through their EIC (left) and the associated pseudospectra (right)")
>    > 
>    > This choice is motivated by several considerations. First of all, **in GC the amount of overlap is much less than in LC**: peaks are much narrower. This means that even a one- or two-second difference in retention time can be enough to separate the corresponding mass spectra. Secondly, EI MS spectra for many compounds are **available in extensive libraries like the [NIST library](http://www.nist.gov/srd/nist1a.cfm "NIST library")** or other online ones like [Golm Metabolome library](http://gmd.mpimp-golm.mpg.de/)
>    {: .details}
>
>    > <details-title>MSP files </details-title>
>    > MSP (Mass Spectrum Peak) file is a text file structured according to the NIST MSSearch spectra format. MSP is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programs (MS-DIAL, NIST MS Search, AMDIS, matchms, etc.). It can contain one or more mass spectra, which are split by an empty line. The individual spectra essentially consist of two sections: metadata (such as name, spectrum type, ion mode, retention time, and the number of m/z peaks) and peaks, consisting of m/z and intensity tuples.
>    > 
>    > **Example of an MSP file entry:**
>    > ```
>    >  Name: Biomarker1
>    >  Retention_time: 5.23
>    >  Num Peaks: 3
>    >  43  100
>    >  57  80
>    >  71  60
>    >  ```
>    >  
>    >  For more details, you can view an [example MSP file](https://zenodo.org/records/16538501/files/W4M0004_database_small.msp).
>    {: .details}
{: .details}

## Alignment

Once **metaMS** has created the pseudo-spectra for each unknown compound in each file, we can start the alignment process. This is done by *comparing each pseudospectrum* to each other in order to group/align similar MS spectra between samples. As a similarity measure, the weighted dot product is used as it is fast, simple, and gives good results ({% cite Stein1994 %}). The first step in the comparison is based on retention, since a comparison of either retention time or retention index is much faster than a spectral comparison. Since the weighted dot product uses scaled mass spectra, the scaling of the database is done once, and then used in all comparisons. If a pseudo-spectra Y from sample A is similar to pseudo-spectra X in sample B and they have close retention (time or index), the two pseudo-spectra are considered as corresponding to the same compound. This process will create the *dataMatrix* and *variableMetadata* outputs, where aligned pseudo-spectra for different samples will belong to the same line in the final *variableMetadata* and will be considered as Unknown compound X. 


![Illustration of two GC-MS spectra aligned for visual comparison](../../images/tuto_gcms_match_spec.png "Best match between an experimental pseudospectrum (red) and a database entry (blue)")

If an MSP database has been added to the *runGC* function inputs then the function returns a table where all patterns that have a match with a DB entry are shown with their name. The other non-matching pseudo-spectra will be named UnknownX in the first column of the *variableMetadata* and *dataMatrix*.

Once **metaMS** have created the pseudo-spectra for each unknown compound in each file, we can start the alignment process. This is done by *comparing every pseudospectrum* to each others in order to group/align similar MS spectra between samples. As a similarity measure, the weighted dot product is used as it is fast, simple, and gives good results ({% cite Stein1994 %}). The first step in the comparison is based on retention, since a comparison of either retention time or retention index is much faster than a spectral comparison. Since the weighted dot product uses scaled mass spectra, the scaling of the database is done once, and then used in all comparisons. If a pseudo-spectra Y from sample A is similar to pseudo-spectra X in sample B and they have close retention (time or index), the two pseudo-spectra are considered as corresponding to the same compound. This process will create the *dataMatrix* and *variableMetadata* outputs, where aligned pseudo-spectra for different samples will belong to the same line in the final *variableMetadata* and will be considered as Unknown compound X. 

## Unknowns research

An important aspect of untargeted metabolomics is the definition of unknowns - features that occur repeatedly in a minimum number or fraction of samples (as defined by the `min.class.fract` and `min.class.size` parameters in the metaMS settings), but for which no annotation has been found. In **metaMS**, these unknown features are found by comparing all patterns (i.e. pseudo-spectra which are groups of features) within a certain retention time (or retention index) difference on their spectral characteristics.
If an MSP database has been added to the *runGC* function inputs, then the function returns a table where all patterns that have a match with a DB entry are shown with their name. The other non-matching pseudo-spectra will be named UnknownX in the first column of the *variableMetadata* and *dataMatrix*.
One strenght of **metaMS** is its ability to use pseudo-spectra (1) for alignment of unknows between samples and (2) to compare unknown experimental pseudo-spectra to previously created in-house spectra databse (in MSP format). By doing so **metaMS** *runGC* function can serve as an annotation tool. You just have to set  - *"Use Personnal DataBase option"* : `show` and add you in-house database file as input.

The *runGC* process will always create an MSP file as output (either with only unknown spectra or with a mix of annotated ones and unknowns). That MSP file can be used for database search online (as Golm ({% cite Kopka2005 %}) and MassBank ({% cite Horai2010 %})) or locally (NIST MSSEARCH) for NIST search (as shown in the following [PDF tutorial](https://workflow4metabolomics.org/sites/default/files/fichiers/documents/w4m_HowToUseNIST_V01.pdf).

For large numbers of samples, this process can take quite some time (it scales quadratically), especially if the allowed difference in retention time is large. The result now is a list of two elements: the first is the annotation table that we also saw after the comparison with the database, and the second is a list of pseudo-spectra corresponding to unknowns. 
An important aspect of untargeted metabolomics is the definition of unknowns — features that occur repeatedly in a minimum number or fraction of samples (as defined by the `min.class.fract` and `min.class.size` parameters in the metaMS settings), but for which no annotation has been found. In **metaMS**, these unknown features are found by comparing all patterns (i.e. pseudo-spectra which are groups of features) within a certain retention time (or retention index) difference on their spectral characteristics.
## Outputs and results

At this stage, all elements are complete: we have the list of pseudo-spectra with an annotation, either as a chemical standard from the database, or an unknown occurring in a sizeable fraction of the injections. The only thing left to do is to calculate relative intensities for the pseudo-spectra, and to put the results in an easy-to-use table. This table consists in two parts. The first part is the information on the “features”, which here are the pseudo-spectra. The second part of the table contains the intensities of these features in the individual injections. 


![Screenshot of peaktable.tsv output of metaMS.runGC](../../images/tuto_gcms_finale_table.png "Final table with unknowns and compounds found during the metaMS processus")

The first eight lines are the standards, and the next ones are the unknowns that are identified by the pipeline. For each sample, "relative intensities" calculated by **metaMS** are given.
In the manual interpretation of this kind of data, the intensities of one or two “highly specific” features (called quantifiers) are often used to achieve relative quantitation. In **metaMS** the reported intensities are calculated differently.

In metaMS, the main measurement for each compound (pseudospectrum) works differently than you might expect. Instead of simply adding up the intensities of different ions belonging to one compound, the package compares what it observes to a "reference model" - think of this as a known fingerprint from a database or a standard sample.
This comparison uses a statistical method called robust linear regression. Here is what this means in simple terms: the software looks for the best scaling factor that makes the observed spectrum match the reference model as closely as possible, while automatically reducing the influence of any unusual or problematic peaks that doesn't fit the pattern.

This method provides a more reliable estimate that is less affected by random fluctuations in the data and variations in specific peaks or generated by a particularly intense spectra that might throw off simpler calculations.

In practical terms, rather than relying on just one or two ions to quantify a compound (which could be unreliable if those specific ions have problems), metaMS considers the entire ion pattern of the pseudospectrum. It calculates an overall "area" using this regression factor.
This approach minimizes the impact of noisy or problematic peaks while making use of all available information - giving you a more robust and trustworthy measurement for each compound in your GC-MS analysis.

In both cases, the result is a list containing a set of patterns corresponding with the compounds that have been found (either annotated or unknown), the relative intensities of these patterns in the individual annotations, and possibly the xcmsSet object for further inspection. In practice, the *runGC* function is all that users need to use.

To recap your [option 1](#processing-with-metams-option-1) journey, here are some questions to check the outcomes of your hands-on: 


> <question-title>getting an overview of your GC-MS processing steps</question-title>
> 
>  **1** - How many known compounds have been detected by metaMS?
> > <solution-title></solution-title>
> > 
> > The answer is: **8**. Here you have to open the {% icon param-file %} `peaktable.tsv` and count.
> > 
> > 
> {: .solution}
> <br>
>  **2** - What is the relative intensity of D-Mannitol (6 TMS) in sample alg7?
> > <solution-title></solution-title>
> > 
> > You should find 74 020 498 in the {% icon param-file %} `peaktable.tsv`.
> > 
> > 
> {: .solution}
> <br>
> **3** - How many Unknowns were detected?
> > <solution-title></solution-title>
> > 
> > The answer is **198**. To be able to see the number of Unknown compounds detected you have to open the {% icon param-file %} `peaktable.tsv` file and go to the last line.
> > 
> > 
> {: .solution}
{: .question}

</div>

<div class="Process-GC-MS-data-with-XCMS-function-directly" markdown="1">
 
# Process GC-MS data with a full XCMS workflow (option 2)

This option follows the standard **XCMS** workflow with GC-MS data as start to obtain in the end a *dataMatrix* file and its corresponding *variableMetadata* file. The main difference with the [option 1](#processing-with-metams-option-1) is that the *dataMatrix* file will contain individual peaks rather than pseudo-spectra, and the *variableMetadata* file will contain information about each peak, such as its retention time, m/z, and intensity. **No** `.msp` file will be generated in this case, as the peaks are not grouped into pseudo-spectra so the annotation process will be different.

> <hands-on-title>Example untargeted GC-MS data processing with the standard XCMS workflow</hands-on-title>
>
> 1. {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.RData` (output of the **xcms findChromPeaks Merger** {% icon tool %} job)
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `5.0`
>        - *"Width of overlapping m/z slices"*: `0.5`
>
> 2. {% tool [xcms fillChromPeaks (fillPeaks)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.RData` (output of the **xcms groupChromPeaks (group)** {% icon tool %} job)
>    - In *"Peak List"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>        - *"Reported intensity values"*: `maxo`
>
> > <comment-title></comment-title>
> > After the **XCMS** processing, the extraction of the MSP-formated spectra can be done using **RAMclustR**{% icon tool %} tool (See this [GTN on GC-MS data processing with XCMS, RAMClustR, RIAssigner, and matchms]({% link topics/metabolomics/tutorials/gc_ms_with_xcms/tutorial.md %})).
> {: .comment}
{: .hands_on}

The outputs of this strategy are similar to the ones described in the LC-MS tutotial mentioned previously. 


To recap your [option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2) journey, here are some questions to check the outcomes of your hands-on: 


> <question-title>getting an overview of your GC-MS processing steps</question-title>
> 
>  **1** - What are the XCMS steps you made in order to obtain your final file RData file?
> > <solution-title></solution-title>
> > 
> > Here are the different steps made through this tutorial: 
> >  - **(Not with XCMS)** import of the data into the Galaxy instance
> >  - **MSNbase readMSData** {% icon tool %} to read and prepare the MS data for the extraction step
> >  - XCMS peak picking with the **xcms findChromPeaks (xcmsSet)** {% icon tool %} tool
> >  - (Not directly XCMS processing, but necessary in the Galaxy tool suit) merge my data into one file with **xcms findChromPeaks Merger** {% icon tool %} tool
> >  - XCMS grouping with the **xcms groupChromPeaks (group)** {% icon tool %} tool
> >  - XCMS integration of missing peaks with **xcms fillChromPeaks (fillPeaks)** {% icon tool %} tool
> > 
> {: .solution}
> <br>
>  **2** - What is the complete name of your final RData file?
> > <solution-title></solution-title>
> > 
> > During each step of the XCMS pre-processing, the name of the file which is processed is completed by the name of the step you used. So, finally your file should be named `xset.merged.groupChromPeaks.fillChromPeaks.RData`. That means (as seen in the previous question) you have run the findChromPeaks (xcmsSet) step, then a merge, a grouping and finally the filling of missing data.
> > 
> > 
> {: .solution}
> <br>
> > 
{: .question}

</div>
   
># Verify your data after the pre-processing and clean datasets
>
>The {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy3) %} part allows users to display the TIC (Total Ion Chromatogram) and the BPC (Base Peak Chromatogram). 

If you separated your samples into different classes, this tool can constructs TICs and BPCs one class against one class, in a `pdf` file (see bellow). 

![The image is a capture of an example of result picture of a TIC one can obtain when comparing two groups.](../../images/tuto_gcms_tic.png "Example of a TIC comparing 2 classes")

 The {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy3) %} part allows users to display the TIC (Total Ion Chromatogram) and the BPC (Base Peak Chromatogram). 

>When you have processed **all or only needed** steps described before, you can continue the processing of your data with statistics or annotation tools. 
Don't forget to always check your files' format for compatibility with further analysis! 
>
>The pre-processing part of this analysis can be **quite time-consuming**, and already corresponds to quite a few number of steps, depending of your analysis. If you plan to proceed with further steps in Galaxy (statistics for example), we highly recommend, at this step of the GC-MS workflow, to split your analysis by beginning a new Galaxy history with **only the files you need** for further steps (that would be the final `.tsv` matrices - sampleMetadata, variableMetadata, dataMatrix - and the `.msp` spectral database). This will help you in limiting the chance to select the wrong dataset in further analysis, and bring a little **tidiness** for future review of your analysis process. This would also enable you to make alternative extractions in the future (by adjusting peak picking parameters) in the same history, without drowning your statistical analysis steps far bellow.
>
> To begin a new history with the files from your current history, you can **use the functionality ‘copy dataset’** and copy it into a new history.
> 
> {% snippet faqs/galaxy/histories_copy_dataset.md %}
> 
> 
> You may have notice that the XCMS tools generate **output names that contain the different XCMS steps you used**, allowing easy traceability while browsing your history. Hence, when begining further processing steps (as statistics), we highly recommend you (in particular if you have used the [option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2)) to first rename your datasets **with something short**, e.g. "dataMatrix", "variableMetadata", or anything not too long that you may find convenient.
>
> {% snippet faqs/galaxy/datasets_rename.md %}
>
> > <warning-title> Be careful of the file format</warning-title>
> > During each step of pre-processing, your dataset has its formats changed and can have also its name changed. To be able to proceed with GC-MS processing with **metaMS**, you need to identify which datasets can be used with which tool. For example to be able to use *runGC* function you need to have an RData object which is at least **merged** (output from **xcms findChromPeaks Merger** {% icon tool %}). 
After following the option 1 steps you obtained tabular files that can be used in a large variety of analyses, as well as an MSP file (if option 1 was used). Always check the format needed for further tools, as these format are 'text format' but with characteristics that may matter in further steps. 
With option 2, if you want to process your data with XCMS or other tools you may also have to **align** them with **xcms groupChromPeaks (group)** {% icon tool %}. It means that you should have at least a file named `xset.merged.RData` to be able to continue XCMS processing.
> {: .warning} 

# Conclusion 
{% icon trophy %} Well done, you’ve processed GC-MS data with [metaMS (option 1)](#processing-with-metams-option-1) or [all with XCMS (option 2)](#process-gc-ms-data-with-a-full-xcms-workflow-option-2)! You might want to explore other [Galaxy trainings for Metabolomics]({% link topics/metabolomics/ %})  
<br>
You might want to consult your results with the [key history option 1](https://workflow4metabolomics.usegalaxy.fr/u/yguitton44/h/gtn-gcms-with-metams) or use [the workflow for option 1](https://workflow4metabolomics.usegalaxy.fr/u/yguitton44/w/workflow-for-gtn-gcms-with-metams-option1) associated with this tutorial.

tools_json:
["MSnbase readMSData", "toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0", "xcms findChromPeaks (xcmsSet)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0", "xcms findChromPeaks Merger", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0", "xcms get a sampleMetadata file", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy3", "metaMS.runGC", "toolshed.g2.bx.psu.edu/repos/yguitton/metams_rungc/metams_runGC/3.0.0+metaMS1.24.0-galaxy0", "xcms groupChromPeaks (group)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0", "xcms fillChromPeaks (fillPeaks)", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy0", "xcms plot chromatogram", "toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy3"]


topic: metabolomics
tutorial: mfassignr
title: Molecular formula assignment and mass recalibration with MFAssignR package
difficulty: intermediate
time_estimation: 2H
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: eba97ab3e580b0f56fde1324a0c70130
last_modified: 2026-05-27T09:01:49.258441
zenodo_link: https://zenodo.org/records/13768009
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/mfassignr/tutorial.html

content:
This training covers the multi-element molecular formula (MF) assignment using the MFAssignR tool. It was originally developed by {% cite Schum2020 %} for the analysis of untargeted mass spectrometry data coming from complex environmental mixtures. The package contains several functions including noise assessment, isotope filtering, internal mass recalibration, and formula assignment. 

The **MFAssignR workflow** is composed of several steps:

1. Run **KMDNoise** to determine the noise level for the data.
2. Check the effectiveness of the S/N threshold using **SNplot**.
3. Use **IsoFiltR** to identify potential 13C and 34S isotope masses.
4. Using the signal-to-noise (S/N) threshold, and the two data frames output from IsoFiltR(), run **MFAssignCHO** to assign MF with C, H, and O to assess the mass accuracy.
5. Use **RecalList** to generate a list of the potential recalibrant series.
6. Use **FindRecalSeries** to find the most suitable recalibrant series.
7. After choosing recalibrant series, use **Recal** to recalibrate the mass lists.
8. Assign MF to the recalibrated mass list using **MFAssign**.
9. Check the output plots from MFAssign() to evaluate the quality of the assignments.

We can illustrate the workflow also on the following scheme:
![workflow_graphical](images/mfassignr_scheme.png)

Let's dive now into the individual steps and explain all the inputs, parameter settings, and outputs.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data import

At the very beginning, we need to import the dataset we will be using. MFAssignR requires on input a table in a **tabular format**, where:

- the first column is the mass to charge ratio (m/z),
- the second column is intensity,
- the optional third column is retention time.

In our case, we will use the model data from the MFAssignR package, which contains the negative ions from acetonitrile extract of wildfire-influenced atmospheric organic aerosol, collected at the Pacific Northwest National Laboratory (PNNL). The detection m/z range of the instrument was 100-800 and the ions were analyzed by electrospray ionization (ESI) on Orbitrap Elite MS and recorded as Xcalibur.raw files. In the MFAssignR package, this dataset is named Raw_Neg_ML in .rds format, which we saved as a tabular file and we will use this file as our input.

> <hands-on-title> Upload data </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/records/13768009/files/mfassignr_input.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Set the format of the dataset to 'tabular'. 
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 4. We can now view our dataset, and ensure, that it has a correct input format.
>
>
{: .hands_on}

# Noise assessment

Having our input data in the workspace, we can now start with the analysis!

The first step is the noise assessment, which allows us to avoid both false positives (if noise is underestimated) and false negatives (if noise is overestimated). For this purpose, we will be using the **KMDNoise** function, and visualize the result using **SNplot** function. MFAssignR provides an additional function, **HistNoise**, but often the data distribution prerequisites are not fulfilled and especially when the analyte signal tapers into the noise, HistNoise fails to separate the distributions. Therefore, using KMDNoise is strongly preferred.

## KMDNoise

KMDNoise function takes advantage of the calculation of **Kendrick Mass Defect (KMD)**.

> <details-title>Kendrick Mass Defect</details-title>
>
>The **Kendrick mass (KM)** serves to simplify classification and
>identification of repeating units in molecules, typically 
>homologous CH2 series, which differ only by the number of base 
>units (-CH2 groups in this case). It sets the mass of the molecular fragment to 
>integer value in atomic mass units (amu).
>
>The KM of a compound can be therefore calculated as the known m/z (given the ionization) of the compound multiplied by a ratio (rounded
>(CH2_mass)/exact(CH2_mass)), which is (14/14.01565) = 0.9988834.
>
>$$\text{KM} = \text{m/z} \times \frac{\text{14.00000}}{\text{14.01565}}$$
>
>
>The **Kendrick Mass Defect (KMD)** is then defined as the difference 
>between rounded KM values and KM.
>$$\text{KMD} = \text{nominal KM} - {\text{KM}}$$
>
>Homologous series, meaning compounds differing only in the number of 
>repeating units (e.g. alkylation series), will always have the same KMD.
>
>{% cite kendrickmass_mzmine %} {% cite kendrickmass_wiki %}
{: .details}


When the KMD is calculated for all peaks in the mass spectrum, there will be a clear separation of more intense analyte peaks and low-intense noise peaks. To isolate the noise region, we can use the KMD limits of chemically feasible molecular formulas in conjunction with the calculation of the slope of a KMD plot using a linear equation $$y = 0.1132x + b$$, where *y* is the KMD value, *0,1132* was an empirically derived integer, *x* the measured ion mass and *b* an y-intercept. To provide a more accurate assessment, two lines with different y-intercepts are selected (we will set this lower and upper y-limit below). Once the noise region is isolated, a noise level will be estimated as the average intensity of peaks within that region.

![KMD plot](images/KMDplot_explained.png)

When running the function, we can stick with the default values: the upper limit for the y-intercept is set to 0.2, so that it does not interact with any potentially double-charged peaks, the lower limit of the y-intercept value is set to 0.05 to ensure no analyte peaks are incorporated into the noise estimation. Both upper and lower x-intercept limits are optional and will be set to minimum and maximum mass in the data if not specified. 

> <hands-on-title> Noise assessment using KMDNoise </hands-on-title>
>
> 1. {% tool [MFAssignR KMDNoise](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_histnoise/mfassignr_kmdnoise/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `mfassignr_input.txt` (Input dataset)
>    - {% icon param-file %} *"upper limit for the y intercept"*: `0.2`
>    - {% icon param-file %} *"lower limit for the y intercept"*: `0.05`
>   
>
{: .hands_on}

The function outputs a **KMD plot**, where the noise area is separated between red lines and a **noise estimate**. The noise estimate we can then multiply with a multiplier, typically a value between 3 and 10 in order to remove low-intensity m/z values. Multiplication with 10 means we will be very stringent, the signal-to-noise ratio will be high and close to the limit of quantitation (LOQ), whereas multiplication with 3 means the signal-to-noise ratio will be low and we are close to the limit of detection (LOD). The optimal is therefore to start with e.g. 6 and check the results using SNplot, as we will do further.

> <question-title></question-title>
>
> 1. What is the noise level estimated by KMDNoise function?
>
> > <solution-title></solution-title>
> >
> > 1. Using our model data, the estimated noise is **346.0706**.
> >
> >
> {: .solution}
>
{: .question}

## SNplot

We can now check the effectiveness of the S/N threshold using **SNplot**, which plots the mass spectrum with the masses below and above the chosen threshold, where the noise is indicated by red.

The `cut` parameter can be computed as an estimated noise level * multiplier, so if we get 346.0706 as a noise level from KMDnoise, we can multiply it by 6, which gives us 2076.

- `Mass` - parameter defines a centerpoint to look at the mass spectrum.
- `Parameter window.x` - sets the +/- range around the mass centerpoint, default is 0.5.
- `Parameter window.y` - is used for setting the limit of the y-axis in the plot. Parameter window.y multiplied the `cut` parameter value, and the multiplication result is used as the y-axis limit. The default is 10.

> <hands-on-title> Plot the SNplot </hands-on-title>
>
> 1. {% tool [MFAssignR SNplot](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_snplot/mfassignr_snplot/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `mfassignr_input.txt` (Input dataset)
>    - *"SN ratio"*: `6.0`
>    - *"Estimated noise"* `346.0706`
>    - *"mass"*: `301.0`
>    - *"window.x"*: `50.0`
>    - *"window.y"*: `10.0`
{: .hands_on}

![SNplot](images/SNplot.png)

Based on the SNplot, we can see that the noise - indicated in red - forms a uniform background and is effectively separated by the horizontal line (indicating the S/N threshold), meaning we can further use the multiplication of 346 and 6. If the noise was higher than the S/N threshold, it would mean we need to increase the value of the multiplier.


# Isotope filtering

The next step is the identification of probable isotopic ion masses containing 1-2 13C or 34S from monoisotopic masses in order to prevent incorrect interpretation of the molecular composition. For this, we will use the **IsoFiltR** function.

Isotope filtering has 4 steps:

1. Firstly, the mass list is transformed to pairs of ions, which have a mass difference of 1.003355 Da for 13C and 1.995797 Da for 34S with some flexibility for a user-defined error tolerance (by default 5 ppm).
2. For specific isotopes, KMD values are calculated using the mass difference between 12C and 13C (1.003355 Da) and the mass difference between 32S and 34S (1.995797 Da).
3. Isotope pairs are further refined using *Resolution Enhanced KMD* adapted from {% cite Zheng2019 %}, in which the Kendrick base is adjusted by an experimentally derived integer (For 34S, the integer is 12 (determined by trial and error) and for 13C the integer is 21 based on {% cite Zheng2019 %}) in order to obtain a desired separation.
4. Finally, abundance ratios are evaluated, where the abundance ratio for 13C should be < 0.6 and the abundance ratio for 34S should be < 0.3.

After performing isotopic filtering, two tables are outputted, one containing the isotopic masses and one containing the monoisotopic masses and all masses that did not have a matching isotopic mass. In complex mixtures, a mass can be classified as both monoisotopic and isotopic; in those cases, it is included in both output tables and classified after the MF assignment.

We will change only the S/N ratio parameter and estimated noise (based on the KMDNoise function), otherwise, we can follow with the default settings.

> <hands-on-title> Isotope filtering using IsoFiltR </hands-on-title>
>
> 1. {% tool [MFAssignR IsoFiltR](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_isofiltr/mfassignr_isofiltr/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input Peak Data"*: `mfassignr-input.txt` (Input dataset)
>    - *"SN ratio"*: `6.0`
>    - *"Estimated noise"* `346.0706`
>
>
{: .hands_on}

# Preliminary molecular formula assignment

Once we have denoised the data and have identified the potential 13C and 34S masses, we can start with the preliminary MF assignment, using the **MFAssignCHO** function. This function assigns MF only with C, H, and O elements to assess the mass accuracy, and therefore it's much quicker than the main MFAssign function. It is based on the CHOFIT algorithm developed by {% cite Green2015 %}.

Firstly, the initial formula assignment with the CHOFIT algorithm is performed along with the quality checks. Then, KMD and z* values are calculated for all masses with the CH2 Kendrick base, and masses are sorted into CH2 homologous series. Subsequently, 1-3 members of each CH2 homologous series below the user-defined threshold are selected and assigned a MF. The ambiguous MFs are returned within the _Unassigned_ list, whereas the unambiguous MF are further re-assigned using CH2, O, H2, H2O, and CH2O extensions. 

Let's use the MFAssignCHO function. On the input, we will need the output of the IsoFiltR function, a data frame of monoisotopic masses, and also (optionally) the data frame containing isotopic masses. Furthermore, we select in which ion mode we measured the data (positive or negative) and we set the signal-to-noise threshold - based on the noise estimate we obtained from the **KMDNoise** or **HistNoise** functions multiplied by the multiplier. Finally, based on the acquisition range, we set the lowMW and highMW, and also the allowed ppm error. Other parameters we can leave in default settings, although in the 'Advanced' tab, many additional parameters can be set by user, including amounts of elements, ratios (e.g. minimum allowed O/C ratio) and many others. 

On the output, there are two dataframes, `Ambiguous` and `Unambigous` provided.

> <hands-on-title> Preliminary MF assignment with MFAssignCHO </hands-on-title>
>
>
> 1. {% tool [MFAssignR MFAssignCHO](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_mfassigncho/mfassignr_mfassignCHO/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Data frame of monoisotopic masses"*: `mono_out` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - {% icon param-file %} *"Data frame of isotopic masses"*: `iso_out` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - *"ppm_err"*: `3`
>    - *"Ion mode"*: `negative`
>    - *"SN ratio"*: `6`
>    - *"Estimated noise"*: `346.0706`
>    - *"Lower limit of molecular mass to be assigned"*: `50.0`
>    - *"Upper limit of molecular mass to be assigned"*: `1000.0`
>
>
{: .hands_on}

On the output, we get several data frames: unambiguous assignments, ambiguous assignments, and a data frame containing unassigned masses. Additionally, several plots are available, such as MSAssign showing a mass spectrum highlighting assigned and unassigned masses, errorMZ showing the relationship between absolute error (in ppm) and ion mass, van Krevelen plot, and a mass spectrum colored by molecular group.

![errorMZ](images/errorMZ.png)
This plot shows us the absolute error, which increases with the increasing mass as expected. Actually, it is good to see a trend here, meaning there is a systematic error that will be corrected during recalibration, a problem would be if there was a random error that we can't correct.

![Van Krevelen plot](images/vankrevelen.png)

> <details-title>Van Krevelen plot</details-title>
>The Van Krevelen plot was originally introduced in 1950 {% cite krevelen1950graphical %} to illustrate 
>the coal formation processes. It typically depicts the atomic ratios of hydrogen-to-carbon (H/C) 
>ratio on the y-axis and oxygen-to-carbon (O/C) on the x-axis. Certain molecules occupy different regions of
>the van Krevelen plot, e.g. alkanes usually have high H/C and low O/C ratios, aromatic compounds have
>lower H/C ratios and carbohydrates usually have high O/C and moderate H/C ratios.
>
{: .details}

Van Krevelen plot visualizes the chemical composition of complex chemical mixtures and it characterizes the data quality and how the data is distributed. Compounds with similar structures and functional groups will cluster together on the plot.

![MS groups](images/MSgroups.png)
The MSgroups plot shows the abundance distribution of unambiguous molecular formula assignments. 

![msassign](images/msassign.png)
Assignment mass spectrum then shows what was assigned: green are the assigned compounds, red are unassigned ones (eventually, if applicable, isotopes are in blue). It is useful to check whether there are any gaps around a particular m/z, which would mean there is a problem with the assignment.

# Recalibration
The next step is recalibration, which ensures that we will have an accurate mass list before formula assignment and any systemic bias is removed. In MFAssignR, recalibration was adapted from {% cite Savory2011 %} and {% cite Kozhinov2013 %}.

There are three consecutive functions, **RecalList**, **FindRecalSeries** and **Recal**, which we will use. 

## RecalList

Firstly, we will generate a table containing potential recalibrant CH2 homologous series using the RecalList function. 
As in input, we will use the `Unambig` data frame which we generated in the MFAssignCHO function:

> <hands-on-title> Finding recalibrant series </hands-on-title>
>
> 1. {% tool [MFAssignR RecalList](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_recallist/mfassignr_recallist/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `Unambig` (output of **MFAssignR MFAssignCHO** {% icon tool %})
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many series were returned?
>
> > <solution-title></solution-title>
> >
> > 1. When we click on the results ('Recalibration series by MFAssignR RecalList on data 10'), we can
> >see there are 226 lines (1 being the column names), meaning 225 series which we can select from for the recalibration.
> >
> {: .solution}
>
{: .question}

On the output, we get a data frame containing CH2 homologous series that contain more than 3 members. Let's dive more into what metrics are available:

- **Series** - reports the homologous series according to class, adduct, and DBE. The format is "class_adduct_DBE", for example, a homologous series with class = "O6, adduct of Na+, and DBE = 4" would be "O6_Na_4".
- **Number Observed** - reports the number of members of each homologous series.
- **Series Index** - represents the order of the series when ordered by length of homologous series.
- **Mass Range** - reports the minimum and maximum mass for the compounds within a homologous series.
- **Tall Peak** - reports the mass of the most abundant peak in each series.
- **Abundance Score** - reports the percentage difference between the mean abundance of a homologous series and the median abundance within the mass range the "Tall Peak" falls in (for example m/z 200-300). A higher score is generally better.
- **Peak Score** - This column compares the intensity of the tallest peak in a given series to the second tallest peak. This comparison is calculated by $$\text{log10} \times \frac{\text{Max Peak Intensity}}{\text{Second Peak Intensity}}$$ The closer to 0 this value is the better, in general.
- **Peak Distance** - This column shows the number of CH2 units between the tallest and second tallest peak in each series. In general, the value should be as close to 1 as possible.
- **Series Score** - This column compares the number of actual observations in each series to the theoretical maximum number based on the CH2 homologous series. The closer to one this value is, the better.

## FindRecalSeries

Which are the best series for recalibration? Combined, these series should cover the full mass spectral range to provide the best overall recalibration. The best series to choose are generally long and combined have a "Tall Peak” at least every 100 m/z.

Currently, it is up to the user to choose the most suitable recalibrant series that will be used for the recalibration in the Recal step. It is possible to choose up to 10 series and they should indeed span over the whole range of m/z - often an error is thrown to add more series in case there are gaps, or the chosen series would have a good score but would be way too alike. Therefore, we provide the **FindRecalSeries** function, which attempts to select the best series which fulfill the abovementioned criteria.

We will only provide the `Recalibration series` dataframe from RecalList function, and the best series will be chosen automatically. Although a list of more than 10 series might be provided on the output, only first 10 best series will be used by the Recal function in the next step.

> <hands-on-title> Selecting most suitable series </hands-on-title>
>
> 1. {% tool [MFAssignR FindRecalSeries](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_findrecalseries/mfassignr_findRecalSeries/1.1.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `recal_series` (output of **MFAssignR RecalList** {% icon tool %})
>
{: .hands_on}

## Recal

The Recal function is used for the internal mass recalibration. It takes the output from MFAssignCHO and outputs from IsoFiltrR, the `Mono` and `Iso` data frames. Finally, it takes the series for recalibration, either chosen by the user or selected by the FindRecalSeries function. 

A common error points to increasing the `MzRange` parameter, which sets the recalibration segment length and has a default value of 30 - sometimes it is needed to increase it to even values as 80. Other parameters can be left to their defaults.

> <hands-on-title> Internal mass recalibration </hands-on-title>
>
> 1. {% tool [MFAssignR Recal](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_recal/mfassignr_recal/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data (Output from MFAssign)"*: `Unambig` (output of **MFAssignR MFAssignCHO** {% icon tool %})
>    - {% icon param-file %} *"Calibration series (Output from RecalList)"*: `Final recalibration series` (output of **MFAssignR FindRecalSeries** {% icon tool %})
>    - {% icon param-file %} *"Peaks dataframe (Mono from IsoFiltR)"*: `Monoisotopic masses` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - {% icon param-file %} *"Isopeaks dataframe (Iso from IsoFiltR)"*: `iso_out` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - *"Ion mode"*: `negative`
>    - *"SN ratio"*: `6`
>    - *"Estimated noise"*: `346.0706`
>    - *"Mass windows used for the segmented recalibration"*: `50.0`
>
>
{: .hands_on}

# Molecular formula assignment

The last step of the workflow is the actual assignment of molecular formulas with 12C, 1H, and 16O and a variety of heteroatoms and isotopes, including 2H, 13C, 14N, 15N, 31P, 32S, 34S, 35Cl, 37Cl, 19F, 79Br, 81Br, and 126I. It can also assign Na+ adducts, common in positive ion mode. We have already described the principles of the MFAssignCHO function, which works the very same way as MFAssign (only being limited to C, H, and O elements), so let's directly use the MFAssign function for the formula assignment. 

> <hands-on-title> MF assignment using MFAssign </hands-on-title>
>
> 1. {% tool [MFAssignR MFAssign](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_mfassign/mfassignr_mfassign/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Data frame of monoisotopic masses"*: `Mono` (output of **MFAssignR Recal** {% icon tool %})
>    - {% icon param-file %} *"Data frame of isotopic masses"*: `Iso` (output of **MFAssignR Recal** {% icon tool %})
>    - *"ppm_err*: `3`
>    - *"Ion mode"*: `negative`
>    - *"SN ratio"*: `6`
>    - *"Estimated noise"*: `346.0706`
>    - *"Lower limit of molecular mass to be assigned"*: `50.0`
>    - *"Upper limit of molecular mass to be assigned"*: `1000.0`
>
{: .hands_on}

Finally, we obtain assigned formulas on rearranged data. Similarly to the MFAssignCHO function, data frames of unassigned masses, ambiguous assignments and unambiguous assignments are provided, as well as quality check plots. 


# Conclusion

In this tutorial, we showed how can we assign molecular formulas using the MFAssignR package. We learned  several key steps, including noise estimation, isotope filtering, recalibration, and finally the formula assignment. We described how parameters of individual functions are used and how they should be adjusted if needed. 

![workflow overview](images/workflow_overview.png)

tools_json:
["MFAssignR KMDNoise", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_histnoise/mfassignr_kmdnoise/1.1.2+galaxy0", "MFAssignR SNplot", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_snplot/mfassignr_snplot/1.1.2+galaxy0", "MFAssignR IsoFiltR", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_isofiltr/mfassignr_isofiltr/1.1.2+galaxy0", "MFAssignR MFAssignCHO", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_mfassigncho/mfassignr_mfassignCHO/1.1.2+galaxy0", "MFAssignR RecalList", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_recallist/mfassignr_recallist/1.1.2+galaxy0", "MFAssignR FindRecalSeries", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_findrecalseries/mfassignr_findRecalSeries/1.1.2+galaxy1", "MFAssignR Recal", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_recal/mfassignr_recal/1.1.2+galaxy0", "MFAssignR MFAssign", "toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_mfassign/mfassignr_mfassign/1.1.2+galaxy0"]


