# tutorials

topic: statistics
tutorial: genomic-llm-sequence-optimization
title: Optimizing DNA Sequences for Biological Functions using a DNA LLM
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml", "Large Language Model"]
content_hash: a20b18f8802487d9b96dc7fba97bf47d
last_modified: 2026-02-26T21:12:18.199019
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/genomic-llm-sequence-optimization/tutorial.html

content:
# Prepare resources 

## Install dependencies


```python
!pip install datasets==3.0.1
!pip install torch==2.5.0
!pip install transformers -U
!pip install accelerate==1.1.0
!pip install peft==0.13.2
!pip install bitsandbytes==0.44.1
!pip install flash-attn==2.6.3
!pip install Bio==1.7.1
!pip install orfipy
```

## Import Python libraries

- `torch`
- `flash_attn`
- `numpy`
- `transformers`
  - `AutoTokenizer`
  - `EarlyStoppingCallback`
  - `Trainer`
  - `TrainingArguments`
  - `AutoModelForCausalLM`
  - `AutoConfig`
  - `DataCollatorForLanguageModeling`
- `datasets`
- `accelerate`


```python
# 
# During the class, 
import os

import sys
import time
from os import path
import gc


import flash_attn
import torch
import numpy as np
import pandas as pd
import scipy as sp
import matplotlib.pyplot as plt

import transformers

from transformers import AutoTokenizer
from transformers import EarlyStoppingCallback, Trainer, TrainingArguments
from transformers import AutoModelForCausalLM, AutoConfig
from transformers import DataCollatorForLanguageModeling
from datasets import load_dataset

import accelerate
```

## Check versions

Numpy version > 1.26.4

```python
np.__version__
```

transformers version > 4.47.1

```python
transformers.__version__
```

flash_attn > 2.6.0.post1 and 2.7.0.post2

```python
flash_attn.__version__
```

accelerate > 0.32.1

```python
# Tested with accelerate==0.32.1

accelerate.__version__
```

## Prepare GPU


```python
# CHECK GPU
# We can see how many VRAM is used and how much the GPU is used.
!nvidia-smi
```

    Thu Feb  6 07:59:23 2025       
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
    |-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  Tesla T4                       Off |   00000000:00:04.0 Off |                    0 |
    | N/A   36C    P8              9W /   70W |       0MiB /  15360MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
                                                                                             
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |  No running processes found                                                             |
    +-----------------------------------------------------------------------------------------+


```python
# LOOK AT GPU USAGE AND RAM
!nvidia-smi
```

    Thu Feb  6 16:49:41 2025       
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
    |-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  Tesla T4                       Off |   00000000:00:04.0 Off |                    0 |
    | N/A   54C    P8             10W /   70W |       2MiB /  15360MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
                                                                                             
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |  No running processes found                                                             |
    +-----------------------------------------------------------------------------------------+



Control the use of ram by CUDA

```python
torch.backends.cudnn.benchmark=True
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32 "
```

```python
# Check GPU
import torch
torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

    device(type='cuda')

# Get the model 

Mistral-DNA-v0.1 was derived from Mixtral-8x7B for the human genome. Mixtral-8x7B was simplified for DNA: the number of layers and the hidden size were reduced. The model was pretrained using the human genome hg38 with 200B DNA sequences.

The model can be downloaded on HuggingFace: https://huggingface.co/RaphaelMourad/Mistral-DNA-v0.1

```python
!git clone https://github.com/raphaelmourad/Mistral-DNA.git
```


```python
!ls
```

    Mistral-DNA  sample_data


```python
# SET DIRECTORY
os.chdir("Mistral-DNA/")
print(os.getcwd())
```

/content/Mistral-DNA

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python", "deep-learning-without-gai-with-python", "genomic-llm-pretraining", "genomic-llm-finetuning", "genomic-llm-zeroshot-prediction", "genomic-llm-sequence-generation"]


topic: statistics
tutorial: RNN
title: Deep Learning (Part 2) - Recurrent neural networks (RNN)
url: https://training.galaxyproject.org/training-material/topics/statistics/tutorials/RNN/tutorial.html
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 7523a953c73d078ea7312aa2d4472641
last_modified: 2026-02-26T21:12:18.157309
zenodo_link: https://zenodo.org/record/4477881
workflows_json: []
content:
Artificial neural networks are a machine learning discipline roughly inspired by how neurons in a
human brain work. In the past decade, there has been a huge resurgence of neural networks thanks
to the vast availability of data and enormous increases in computing capacity (successfully
training complex neural networks in some domains requires lots of data and compute capacity). There
are various types of neural networks (feedforward, recurrent, etc). In this tutorial, we discuss
recurrent neural networks (RNN), which model sequential data, and have been successfully applied to language
generation, machine translation, speech recognition, image description, and text summarization
({% cite wen-etal-2015-semantically %}, {% cite cho-etal-2014-learning %}, {% cite LimEtAl %},
{% cite karpathyEtAl %}, {% cite li-etal-2017-deep %}). We start by explaining how RNN differ from
feedforward networks (FNN), describe various RNN architectures, and solve a sentiment analysis problem
using RNN in Galaxy.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Feedforward neural networks (FNN)

In feedforward neural networks (FNN) a single training example is presented to the network,
after which the network generates an output. For example, a lung X-ray image is passed
to a FNN, and the network predicts tumor or no tumor.

## Single layer FNN

![Neurons forming the input and output layers of a single layer feedforward neural network](../../images/FFNN_no_hidden.png "Single layer feedforward neural network")

Figure 1 shows a single layer FNN, where the input is 3 dimensional. Each input field is multiplied by a
weight. Afterwards, the results are summed up, along with a bias, and passed to an activation function.

![An activation function (such as Sigmoid, Tanh, etc.) applied to the input of the only neuron in the output layer of a feedforward neural network](../../images/activation.gif "Activation of the output neuron o1. Activation function f could be Sigmoid, Tanh, ReLU, etc.")

The activation function can have many forms (sigmoid, tanh, ReLU, linear, step function, sign function, etc.).
Output layer neurons usually have *sigmoid* or *tanh* functions. For more information on the listed activation
functions, please refer to {% cite nwankpaEtAl %}.

![Mathmatical formula for Sigmoid activation function](../../images/sigmoid.gif "Sigmoid activation function")

## Multi-layer FNN

Minsky and Papert showed that a single layer FNN cannot solve problems in which the data is not linearly separable,
such as the XOR problem ({% cite Newell780 %}). Adding one (or more) hidden layers to FNN enables it to solve problems
in which data is non-linearly separable. Per Universal Approximation Theorem, a FNN with one hidden layer can represent
any function ({% cite Cybenko1989 %}), although in practice training such a model is very difficult (if not impossible),
hence, we usually add multiple hidden layers to solve complex problems.

![Neurons forming the input, output, and hidden layers of a multi-layer feedforward neural network](../../images/FFNN.png "Feedforward neural network with a hidden layer. Biases to hidden/output layer neurons are omitted for clarity")

## Learning algorithm

In supervised learning, we are given a set of input-output pairs, called the *training set*. Given the training set, the learning algorithm
(iteratively) adjusts the model parameters, so that the model can accurately map inputs to outputs. We usually have another set of input-output
pairs, called the *test set*, which is not used by the learning algorithm. When the learning algorithm completes, we assess the learned model by
providing the test set inputs to the model and comparing the model outputs to test set outputs. We need to define a **loss function** to objectively
measure how much the model output is off of the expected output. For classification problems we use the **cross entropy** loss function.

![Mathematical formula for calculating the cross entropy loss function, which quantifies the difference between the predicted and desired output of a neural network](../../images/CrossEntropy.gif "Cross entropy loss function")

The loss function is calculated for each input-output pair in the training set. The average of the calculated loss functions for all training
set input-output pairs is called the **Cost function**. The goal of the learning algorithm is to minimize the cost function. The cost function
is a function of network weights and biases of all neurons in all layers. The **backpropagation** learning algorithm {% cite Rumelhart1986 %}
iteratively computes the gradient of cost function relative to each weight and bias, then updates the weights and biases in the opposite
direction of the gradient, to find the local minimum.

![Mathematical formula for calcuating the cross entropy cost function, which is just the average of cross entropy loss functions for training samples](../../images/CostFunction.gif "Cross entropy cost function")

# Recurrent neural networks

Unlike FNN, in RNN the output of the network at time *t* is used as network input
at time *t+1*. In RNN, a training example is a sequence, which is presented to the
network one at a time. For example, a sequence of English words is passed to a
RNN, one at a time, and the network generates a sequence of Persian words, one
at a time. RNN handle sequential data, whether its temporal or ordinal.

## Possible RNN inputs/outputs

There are 4 possible input/output combinations for RNN and each have a specific application. *One-to-one* is basically a FNN. *One-to-many* is
where we have one input and a variable number of outputs. One example application is image captioning, where a single image is provided
as input and a variable number of words (which caption the image) is returned as output (see Figure 7).

![Neurons forming a one-to-many recurrent neural network](../../images/RNN_1_to_n.png "One-to-many RNN")

*Many-to-one* RNN combination has a variable number of inputs and a single output. One example application is document sentiment
classification, where a variable number of words in a document are presented as input, and a single output predicts whether the document
has a positive or negative sentiment regarding a topic (see Figure 8).

![Neurons forming a many-to-one recurrent neural network](../../images/RNN_n_to_1.png "Many-to-one RNN")

Finally there is many-to-many RNN, which have two types: one in which the number of inputs and outputs match, e.g., in labeling the video frames the number
of frames matches the number of labels, and the other in which the number of inputs and outputs do not match, e.g., in language translation
we pass in *n* words in English and get *m* words in Italian (see Figure 9).

![Neurons forming a many-to-many recurrent neural network](../../images/RNN_n_to_m.png "Many-to-many RNN")

## RNN architectures

Mainly, there are three types of RNN: 1) Vanilla RNN, 2) LSTM ({% cite hochreiter1997long %}), and 3) GRU ({% cite cho-etal-2014-learning %}).
A Vanilla RNN, simply combines the state information from the previous timestamp with the input from the current timestamp to generate the
state information and output for current timestamp. A problem with Vanilla RNN is that training deep RNN networks is impossible due to the
**vanishing gradient** problem. Basically, weights/biases are updated according to the gradient of the loss functions relative to
the weights/biases. The gradients are calculated recursively from the output layer towards the input layer (hence, the name *backpropagation*).
The gradient of the input layer is the product of the gradient of the subsequent layers. If those gradients are small, the gradient of the input
layer (which is the product of multiple small values) will very small, resulting in very small updates to weights/biases of the initial layers
of the RNN, effectively halting the learning process.

LSTM and GRU are two RNN architectures that address vanishing gradient problem. Full description of LSTM/GRU is beyond the scope of this
tutorial (please refer to {% cite hochreiter1997long %} and {% cite cho-etal-2014-learning %}), but in a nutshell both LSTM and GRU use **gates** such that the weights/biases updates in previous
layers are calculated via a series of additions (not multiplications). Hence, these architectures can learn even when the RNN has hundreds or
thousands of layers.

# Text representation schemes

In this tutorial we perform sentiment analysis on IMDB (https://www.imdb.com/) movie reviews dataset ({% cite maas-EtAl %}). We train our RNN on
the training dataset, which is made up of 25,000 movie reviews, some positive and some negative. We then test our RNN on the test set, which is
also made up of 25,000 movie reviews, again some positive and some negative. The training and test sets have no overlap. Since we are dealing with
text data, it's a good idea to review various mechanisms for representing text data. Before that, we are going to briefly discuss how to preprocess
text documents.

## Text preprocessing

The first step is to tokenize a document, i.e., break it down into words. Next, we remove the punctuation marks, URLs, and stop words -- words like
'a', 'of', 'the', etc. that happen frequently in all documents and do not have much value in discriminating between documents. Next, we normalize
the text, e.g., replace 'brb' with 'Be right back', etc. Then, We then run the spell checker to fix typos and also make all words lowercase. Next, we perform stemming or lemmatization. Namely, if we have words like 'organizer', 'organize', 'organized', and 'organization' we want to reduce all of them to a single word. Stemming cuts the end of these words to come up with a single root (e.g., 'organiz'). The root may not be an actual word.
Meanwhile, lemmatization is smarter in that it reduces the word variants to a root that is actually a word (e.g., 'organize'). All of these steps help reduce
the number of features in feature vector of a document and should make the training of our model faster/easier.

For this introductory tutorial, we do minimal text preprocessing. We ignore the top 50 words in IMDB reviews (mostly stop words) and include
the next 10,000 words in our dataset. Reviews are limited to 500 words. They are trimmed if they are longer and padded if they are shorter.

## Bag of words and TF-IDF

If you don't care about the order of the words in a document, you can use bag of words (BoW) or term frequency inverse document frequency (TF-IDF).
In these models we have a 2 dimensional array. The rows represent the documents (in our example, the movie reviews) and the columns
represent the words in our vocabulary (all the unique words in all the documents). If a word is not present in a document, we have a zero
at the corresponding row and column as the entry. If a word is present in the document, we have a one as the entry -- Alternatively, we could use
the word count or frequency.

![Table showing a bag-of-words representation of sample documents](../../images/BoW.png "Bag of words (BoW) representation")

Suppose we have the following 2 documents: 1) Magic passed the basketball to Kareem, and 2) Lebron stole the basketball from Curry. The BoW
representation of these documents is given in Figure 10.

BoW's advantage is its simplicity, yet it does not take into account the rarity of a word across documents, which unlike common words are
important for document classification.

In TF-IDF, similar to BoW we have an entry for each document-word pair. In TD-IDF, the entry is the product of 1) term frequency, the
frequency of a word in a document, and 2) inverse document frequency, the inverse of the number of documents that have the word divided
by the total number of documents (we usually use logarithm of the IDF).

TF-IDF takes into account the rarity of a word across documents, and like BoW, it also does not capture word order or word meaning in documents. BoW
and TF-IDF are suitable representations for when word order is not important. They are used in document classification problems, like spam detection.

## One hot encoding (OHE)

OHE is a technique to convert categorical variables such as words into a vector. Suppose our vocabulary has 3 words: orange, apple, banana.
Each word for this vocabulary is represented by a vector of size 3. Orange is represented by a vector whose first element is 1 and other
elements are 0; apple is represented by a vector whose second element is 1 and other elements are 0; and banana is represented by a
vector whose third element is 1 and other elements are 0. As you can see only one element in the vector is 1 and the rest are 0's. The same
concept applies if the size of the vocabulary is N.

![Mathematical vectors representing one-hot-encoding representation of words orange, apple, and banana](../../images/OHE.gif "One hot encoding (OHE) representation")

The problem with OHE is that for very large vocabulary sizes (say, 100,000 words) it requires tremendous amount of storage. Also, it has no
concept of word similarity.

## Word2Vec

In Word2Vec, each word is represented as an *n* dimensional vector (*n* being much smaller than vocabulary size), such that the words that have
similar meanings are closer to each other in the vector space, and words that don't have a similar meaning are farther apart. Words are
considered to have a similar meaning if they co-occur often in documents. There are 2 Word2Vec architectures, one that predicts the probability
of a word given the surrounding words (continuous BoW), and one that given a word predicts the probability of the surrounding words (continuous
skip-gram).

In this tutorial, we find an *n* dimensional representation of the IMDB movie review words, not based on word meanings, but based on how they
improve the sentiment classification task. The *n* dimensional representation is learned by the learning algorithm, simply by reducing the
cost function via backpropagation.

# Get Data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://zenodo.org/record/4477881) and choose the type of data as `tabular`
>
>    ```
>    https://zenodo.org/record/4477881/files/X_test.tsv
>    https://zenodo.org/record/4477881/files/X_train.tsv
>    https://zenodo.org/record/4477881/files/y_test.tsv
>    https://zenodo.org/record/4477881/files/y_train.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. Rename the datasets as `X_test`, `X_train`, `y_test`, and `y_train` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype of all the four datasets is `tabular`. If not, change the dataset's datatype to tabular.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

# Sentiment Classification of IMDB movie reviews with RNN

In the section, we define a RNN and train it using IMDB movie reviews training dataset. The goal is to learn a model such that given the
words in a review we can predict whether the review was positive or negative. We then evaluate the trained RNN on the test dataset
and plot the confusion matrix.

## Create a deep learning model architecture

> <hands-on-title>Model config</hands-on-title>
>
> - {% tool [Create a deep learning model architecture](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/1.0.10.0) %}
>    - *"Select keras model type"*: `sequential`
>    - *"input_shape"*: `(500,)`
>    - In *"LAYER"*:
>        - {% icon param-repeat %} *"1: LAYER"*:
>            - *"Choose the type of layer"*: `Embedding -- Embedding`
>                - *"input_dim"*": `10000`
>                - *"output_dim"*": `32`
>        - {% icon param-repeat %} *"2: LAYER"*:
>            - *"Choose the type of layer"*: `Recurrent -- LSTM`
>                - *"units"*": `100`
>        - {% icon param-repeat %} *"3: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*: `1`
>                - *"Activation function"*: `sigmoid`
>    - Click *"Run Tool"*
{: .hands_on}

Input is a movie review of size 500 (longer reviews were trimmed and shorter ones padded). Our neural network has 3 layers. The first layer is
an embedding layer, that transforms each review words into a 32 dimensional vector (*output_dim*). We have 10,000 unique words in our IMDB dataset
(*input_dim*). The second layer is an *LSTM* layer, which is a type of RNN and it learns dependencies between time steps. We set the output size of the LSTM layer to *100*. The third layer is a
*Dense* layer, which is a fully connected layer (all 100 output neurons in LSTM layer are connected to a single neuron in this layer). It has a
*sigmoid* activation function, that generates an output between 0 and 1. Any output greater than 0.5 is considered a predicted positive review,
and anything less than 0.5 a negative one. The model config can be downloaded as a JSON file for future reuse.

## Create a deep learning model

> <hands-on-title>Model builder (Optimizer, loss function, and fit parameters)</hands-on-title>
>
> - {% tool [Create deep learning model](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/1.0.10.0) %}
>    - *"Choose a building mode"*: `Build a training model`
>    - *"Select the dataset containing model configuration"*: Select the *Keras Model Config* from the previous step.
>    - *"Do classification or regression?"*: `KerasGClassifier`
>    - In *"Compile Parameters"*:
>        - *"Select a loss function"*: `binary_crossentropy`
>        - *"Select an optimizer"*: `Adam - Adam optimizer `
>        - *"Select metrics"*: `acc/accuracy`
>    - In *"Fit Parameters"*:
>        - *"epochs"*: `2`
>        - *"batch_size"*: `128`
>    - Click *"Run Tool"*
{: .hands_on}

A loss function measures how different the predicted output is versus the expected output. For binary classification problems, we use
*binary cross entropy* as loss function. Epochs is the number of times the whole training data is used to train the model. Setting *epochs* to 2
means each training example in our dataset is used twice to train our model. If we update network weights/biases after all the training data is
feed to the network, the training will be very slow (as we have 25,000 training examples in our dataset). To speed up the training, we present
only a subset of the training examples to the network, after which we update the weights/biases. *batch_size* decides the size of this subset.
The model builder can be downloaded as a zip file.

## Deep learning training and evaluation

> <hands-on-title>Training the model</hands-on-title>
>
> - {% tool [Deep learning training and evaluation](toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.10.0) %}
>    - *"Select a scheme"*: `Train and Validate`
>    - *"Choose the dataset containing pipeline/estimator object"*: Select the *Keras Model Builder* from the previous step.
>    - *"Metrics for evaluation - Select the primary metric (scoring)"*: `Classification -- 'f1'`
>    - *"Select input type:"*: `tabular data`
>        - *"Training samples dataset"*: Select `X_train` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>        - *"Dataset containing class labels or target values"*: Select `y_train` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
>
{: .hands_on}

The training step generates 2 datasets. 1) accuracy of the trained model, 2) the trained model, in *h5mlm* format. These files are needed for prediction in the next step.

## Model Prediction

> <hands-on-title>Testing the model</hands-on-title>
>
> - {% tool [Model Prediction](toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.10.0) %}
>    - *"Choose the dataset containing pipeline/estimator object"* : Select the trained model from the previous step.
>    - *"Select invocation method"*: `predict`
>    - *"Select input data type for prediction"*: `tabular data`
>    - *"Training samples dataset"*: Select `X_test` dataset
>    - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
{: .hands_on}

The prediction step generates 1 dataset. It's a file that has predictions (1 or 0 for positive or negative movie reviews) for every review in
the test dataset.

## Machine Learning Visualization Extension

> <hands-on-title>Creating the confusion matrix</hands-on-title>
>
> - {% tool [Machine Learning Visualization Extension](toolshed.g2.bx.psu.edu/repos/bgruening/ml_visualization_ex/ml_visualization_ex/1.0.10.0) %}
>    - *"Select a plotting type"*: `Confusion matrix for classes`
>    - *"Select dataset containing the true labels"*": `y_test`
>    - *"Choose how to select data by column:"*: `All columns`
>    - *"Select dataset containing the predicted labels"*": Select `Model Prediction` from the previous step
>    - *"Does the dataset contain header:"*: `Yes`
>    - Click *"Run Tool"*
>
{: .hands_on}

**Confusion Matrix** is a table that describes the performance of a classification model. It lists the number of positive and negative examples
that were correctly classified by the model, true positives (TP) and true negatives (TN), respectively. It also lists the number of examples that
were classified as positive that were actually negative (false positive, FP, or type I error), and the number of examples that were classified
as negative that were actually positive (false negative, FN, or type 2 error). Given the confusion matrix, we can calculate **precision** and
**recall** {% cite TatbulEtAl  %}. Precision is the fraction of predicted positives that are true positives (precision = TP / (TP + FP)). Recall
is the fraction of true positives that are predicted (recall = TP / (TP + FN)). One way to describe the confusion matrix with just one value is
to use the **F score**, which is the harmonic mean of precision and recall.

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} $$

![Confusion matrix for our sentiment analysis problem](../../images/ConfusionMatrix.png "Sentiment analysis confusion matrix")

Figure 12 is the resultant confusion matrix for our sentiment analysis problem (note that your numbers in the matrix may differ, and that is expected). The top row in the figure represents the *true* 0 (or negative sentiment)
class labels (we have 10,397 + 2,103 = 12,500 reviews with negative sentiment). The bottom row represents the *true* 1 (or positive sentiment) class labels
(Again, we have 1,281 + 11,219 = 12,500 reviews with positive sentiment). The left column represents the *predicted* negative sentiment class labels (Our RNN
predicted 10,397 + 1,281 = 11,678 reviews as having a negative sentiment). The right column represents the *predicted* positive class labels (Our RNN
predicted 11,219 + 2,103 = 13,322 reviews as having a positive sentiment).Looking at the bottom right cell, we see that our RNN has correctly predicted 11,219
reviews as having a positive sentiment (true positives). Looking at the top right cell, we see that our RNN has incorrectly predicted 2,103 reviews as having
a positive (false positives). Similarly, looking at the top left cell, we see that our RNN has correctly predicted 10,397 reviews as having negative sentiment
(true negative). Finally, looking at the bottom left cell, we see that our RNN has incorrectly predicted 1,281 reviews as negative (false negative). Given
these numbers we can calculate Precision, Recall, and the F score as follows:

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} = \frac{11,219}{11,219 + 2,102} = 0.84 $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} = \frac{11,219}{11,219 + 1,281} = 0.89 $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} = \frac{2 * 0.84 * 0.89}{0.84 + 0.89} = 0.86 $$

# Conclusion


In this tutorial, we briefly reviewed feedforward neural networks, explained how recurrent neural networks are different, and discussed various
RNN input/outputs and architectures. We also discussed various text representation and preprocessing schemes and used Galaxy to solve a sentiment
classification problem using RNN on IMDB movie reviews dataset.


topic: statistics
tutorial: neural-networks-with-python
title: Neural networks using Python
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: ["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python"]
tags_json: ["elixir", "ai-ml"]
content_hash: d41d8cd98f00b204e9800998ecf8427e
last_modified: 2026-02-26T21:12:18.218963
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/neural-networks-with-python/tutorial.html


topic: metabolomics
tutorial: lcms
title: Mass spectrometry: LC-MS analysis
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 611afab3536cbadb68e446946efc04a3
last_modified: 2026-02-26T21:12:17.162451
zenodo_link: https://zenodo.org/record/3244991
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/lcms/tutorial.html

content:
You may already know that there are different types of *-omic* sciences; out of these, metabolomics is most closely related to phenotypes.
Metabolomics involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants. It  focuses on studying the very small molecules
which are called *metabolites*, to better understand matters linked to the metabolism. However, studying metabolites is not a piece of cake
since it requires several critical steps which still have some major bottlenecks. Metabolomics is still quite a young science, and has many
kinds of specific challenges.

One of the three main technologies used to perform metabolomic analysis is **Liquid-Chromatography Mass Spectrometry** (LC-MS). Data analysis
for this technology requires a large variety of steps, ranging from extracting information from the raw data, to statistical analysis
and annotation. To be able to perform a complete LC-MS analysis in a single environment, the [Wokflow4Metabolomics](http://workflow4metabolomics.org/)
team provides Galaxy tools dedicated to metabolomics. This tutorial explains the main steps involved in untargeted LC-MS data processing
for metabolomic analysis, and shows how to conduct metabolomic data analysis from preprocessing to annotation using Galaxy.

To illustrate this approach, we will use data from {% cite Thvenot2015 %}. The objectives of this paper was to analyze
the influence of age, body mass index, and gender on the urine metabolome. To do so, the authors collected samples
from 183 employees from the French Alternative Energies and Atomic Energy Commission (CEA) and performed LC-HRMS LTQ-Orbitrap
(negative ionization mode) analyses.

Since the original dataset takes a few hours to be processed, we chose to take a limited subset of individuals for this tutorial.
This will allow you to perform an example of metabolomic workflow, from pre-processing to annotation, in a limited time, even though
the results obtained may not be reliable from a scientific point of view due to the small sample size. Nevertheless,
the chosen diversity of sample will allow you to explore the basics of a metabolomic workflow.

We chose a subset of 9 samples, composed of 6 biological samples and 3 quality-control pooled samples (QC pools - mix of all
biological samples).

To analyze these data, we will then follow a light version of the [LC-MS workflow](http://workflow4metabolomics.org/the-lc-ms-workflow),
developed by the [Wokflow4metabolomics group](http://workflow4metabolomics.org/) ({% cite Giacomoni2014 %}, {% cite Guitton2017 %}).
The workflow is composed of 4 main parts:
  1. **Preprocessing** extracts ions from raw data
  2. **Data processing** checks the quality of data and transforms it to something relevant
  3. **Statistical Analysis** highlights interesting information inside the data
  4. **Annotation** puts a name on selected variables

[![The full tutorial workflow](../../images/tutorial-lcms-workflow.png)](../../images/tutorial-lcms-workflow.png)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Preprocessing with XCMS

The first step in the workflow is the pre-processing of the raw data with **XCMS** ({% cite Smith2006 %}).

**XCMS** {% icon tool %} is a free and open source software dedicated to pre-processing of any type of mass spectrometry acquisition files from low to
high resolution, including FT-MS data coupled with different kind of chromatography (liquid or gas). This software is
used worldwide by a huge community of specialists in metabolomics using mass spectrometry methods.

This software is based on different algorithms that have been published, and is provided and maintained using R software.

**MSnbase readMSData** {% icon tool %}, prior to **XCMS** {% icon tool %} is able to read files with open format as `mzXML`, `mzMl`, `mzData` and `netCDF`, which are independent of the constructors' formats. The **XCMS** {% icon tool %} package itself is composed of R functions able to extract, filter, align and fill gap, with the possibility to annotate isotopes,
adducts and fragments using the R package CAMERA ({% cite CAMERA %}). This set of functions gives modularity, and thus is particularly well
adapted to define workflows, one of the key points of Galaxy.


## Importing the LC/MS data into Galaxy

In metabolomics studies, the number of samples can vary a lot (from a handful to several hundreds). Thus, extracting your
data from the raw files can be very fast, or take quite a long time. To optimise the computation time as much as possible, the W4M core team chose to develop tools that can run single raw files for the first steps of
pre-processing in parallel, since the initial actions in the extraction process treat files independently.

Since the first steps can be run on each file independently, the use of **Dataset collections** in Galaxy is recommended, to avoid
having to launch jobs manually for each sample. You can start using the dataset collection option from the very beginning of your analysis, when uploading your data into Galaxy.

> <hands-on-title>Data upload the mzXML with <b>Get data</b></hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 9 `mzXML` files into a collection named `sacurine`
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from Zenodo using the URLs given below
>
>    ```
>    https://zenodo.org/record/3244991/files/HU_neg_048.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_090.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_123.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_157.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_173.mzML
>    https://zenodo.org/record/3244991/files/HU_neg_192.mzML
>    https://zenodo.org/record/3244991/files/QC1_002.mzML
>    https://zenodo.org/record/3244991/files/QC1_008.mzML
>    https://zenodo.org/record/3244991/files/QC1_014.mzML
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="sacurine" renaming=false %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md astype="as a Collection" %}
>
> 3. Make sure your data is in a **collection**. Make sure it is named `sacurine`
>    - If you forgot to select the collection option during import, you can create the collection now:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
{: .hands_on}

You should have in your history a green Dataset collection (`sacurine`) with 9 datasets in mzml format.

Their size can be checked by clicking on the information icon {% icon galaxy-info %} on the individual datasets

## Data preparation for XCMS: *MSnbase readMSData*

This first step is only meant to read your `mzXML` file and generate an object usable by **XCMS** {% icon tool %}.

**MSnbase readMSData** {% icon tool %} takes as input your raw files and prepares `RData` files for the first XCMS step.

> <hands-on-title>MSnbase readMSData</hands-on-title>
>
> 1. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy2) %} with the following parameters:
>    - *"File(s) from your history containing your chromatograms"*: the `sacurine` dataset collection
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>
>  > <question-title></question-title>
>  >
>  > What do you get as output?
>  >
>  > > <solution-title></solution-title>
>  > >
>  > > 1. A **Dataset collection** containing 9 dataset.
>  > > 2. The datasets are some RData objects with the **rdata.msnbase.raw** datatype.
>  > >
>  > {: .solution}
>  >
>  {: .question}
{: .hands_on}

Now that you have prepared your data, you can begin with the first XCMS extraction step: peakpicking. However, before beginning to
extract meaningful information from your raw data, you may be interested in visualising your chromatograms. This can be of particular
interest if you want to check whether you should consider discarding some range of your analytical sequence (some scan or *retention
time* (RT) ranges).

To do so, you can use a tool that is called **xcms plot chromatogram** {% icon tool %} that will plot each sample's chromatogram (see dedicated section
further). However, to use this tool, you may need additional information about your samples for colouring purpose. Thus, you may need
to upload into Galaxy a table containing metadata of your samples (a *sampleMetadata* file).


## Importing a sample metadata file

What we referenced here as a *sampleMetadata* file corresponds to a table containing information about your samples (= sample metadata).

A sample metadata file contains various information for each of your raw files:
- **Classes** which will be used during the preprocessing steps
- **Analytical batches** which will be useful for a batch correction step, along with **sample types** (pool/sample) and **injection order**
- Different **experimental conditions** which can be used for statistics
- Any information about samples that you want to keep, in a *column* format

The content of your sample metadata file has to be filled by you, since it is not contained in your raw data.
Note that you can either:
- Upload an existing metadata file
- Use a template to create one (because it can be painful to get the sample list without misspelling or omission)
  1. Generate a template with the **xcms get a sampleMetadata file** {% icon tool %} tool
  2. Fill it using your favorite table editor (Excel, LibreOffice etc.)
  3. Upload it within Galaxy

> <tip-title>Optional: Generate the right template with <b>xcms get a sampleMetadata file</b></tip-title>
>
> In the case of this tutorial, we already prepared a *sampleMetadata* file with all the necessary information. Below is an optional hands-on explaining how to get a template to fill, with the two following advantages:
>
> 1. You will have the exact list of the samples you used in Galaxy, with the exact identifiers (*i.e.* exact sample names)
> 2. You will have a file with the right format (tabulation-separated text file) that only needs to be filled with the information you want.
>
> > <hands-on-title>xcms get a sampleMetadata file</hands-on-title>
> >
> > 1. {% tool [xcms get a sampleMetadata file](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy1) %} with the following parameters:
> >    - {% icon param-collection %} *"RData file"*: the `sacurine.raw.RData` collection output from **MSnbase readMSData** {% icon tool %}
> >
> {: .hands_on}
>
> An easy step for an easy sampleMetadata filling!
>
> From this tool, you will obtain a `tabular` file (meaning a tab-separated text file) with a first column of identifiers and a
> second column called *class* which is empty for the moment (only '.' for each sample). You can now download this file by clicking on the {% icon galaxy-save %} icon.
>
{: .tip}


### Prepare your sampleMetadata file

The sampleMetadata file is a tab-separated table, in text format. This table has to be filled by the user. You can use any
software you find appropriate to construct your table, as long as you save your file in a compatible format. For example, you can
use a spreadsheet software such as Microsoft Excel or LibreOffice.

> <warning-title>Important: Save your table in the correct format</warning-title>
>
> The file has to be a `.txt`or a `.tsv` (tab-separated values). Neither `.xlsx` nor `.odt` are supported.
> If you use a spreadsheet software, be sure to change the default format to **Text (Tab delimited)** or equivalent.
{: .warning}

Once your sampleMetadata table is ready, you can proceed to the upload. In this tutorial we already prepared the table for you ;)

> <tip-title>Optional: Filling the <i>sampleMetadata</i> using the template obtained from Galaxy</tip-title>
>
> For this tutorial, we already provide the *sampleMetadata* file, so you only have to upload it to Galaxy. Below we
> explain how we filled this file from the template we generated in Galaxy.
>
> First, we used **xcms get a sampleMetadata file** {% icon tool %} as mentioned in the previous tip box.
>
> We obtained the following table:
>
> sample_name | class
> --- | ---
> QC1_014 | .
> QC1_008 | .
> QC1_002 | .
> HU_neg_192 | .
> HU_neg_173 | .
> HU_neg_157 | .
> HU_neg_123 | .
> HU_neg_090 | .
> HU_neg_048 | .
>
> We used a spreadsheet software to open the file. First, we completed the class column. You will see in further XCMS steps that this
> second column matters.
>
> sample_name | class
> --- | ---
> QC1_014 | QC
> QC1_008 | QC
> QC1_002 | QC
> HU_neg_192 | sample
> HU_neg_173 | sample
> HU_neg_157 | sample
> HU_neg_123 | sample
> HU_neg_090 | sample
> HU_neg_048 | sample
>
> With this column, we will be able to colour the samples depending on the sample type (QC or sample).
> Next, we added columns with interesting or needed information, as following:
>
> sample_name | class | polarity | sampleType | injectionOrder | batch | osmolality | sampling | age | bmi | gender
> --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
> QC1_014 | QC | 0 | pool | 185 | ne1 | NA | NA | NA | NA | NA
> QC1_008 | QC | 0 | pool | 105 | ne1 | NA | NA | NA | NA | NA
> QC1_002 | QC | 0 | pool | 27 | ne1 | NA | NA | NA | NA | NA
> HU_neg_192 | sample | 0 | sample | 165 | ne1 | 1184 | 8 | 31 | 24.22 | Male
> HU_neg_173 | sample | 0 | sample | 148 | ne1 | 182 | 7 | 55 | 20.28 | Female
> HU_neg_157 | sample | 0 | sample | 137 | ne1 | 504 | 7 | 43 | 21.95 | Female
> HU_neg_123 | sample | 0 | sample | 100 | ne1 | 808 | 5 | 49 | 24.39 | Male
> HU_neg_090 | sample | 0 | sample | 75 | ne1 | 787 | 4 | 46 | 19.79 | Male
> HU_neg_048 | sample | 0 | sample | 39 | ne1 | 997 | 3 | 39 | 19.49 | Female
>
> In particular, the `batch`, `sampleType` and `injectionOrder` columns are mandatory to correct the data from signal drift (see later in
> the tutorial).
> Once we completed the table filling, we saved the file, minding to stick with the original format. Then, our *sampleMetadata* was ready to
> be uploaded into Galaxy.
>
{: .tip}


### Upload the sampleMetada file with 'Get data'

> <hands-on-title>Upload the sampleMetada</hands-on-title>
>
> 1. Import the `sampleMetadata_completed.tsv` file from Zenodo or from a shared data library (ask your instructor)
>    ```
>    https://zenodo.org/record/3244991/files/sampleMetadata_completed.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Check the data type of your imported files.
>    - The datatype should be `tabular`, if this is not the case, please change the datatype now
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
>    > <comment-title></comment-title>
>    >
>    > Here we provided the sampleMetadata file so we know that the upload led to a 'tabular' file. But from experience we also know that
>    > it can happen that, when uploading a sampleMetadata table, a user obtains other inappropriate types of data. This is generally due to the file
>    > not following all the requirements about the format (*e.g.* wrong separator, or lines with different numbers of columns).
>    > Thus, we highly recommend that you always take a second to check the data type after the upload. This way you can handle the problem
>    > right away if you happen to get one of these obvious issues.
>    {: .comment}
>
> 3. Rename your sampleMetadata file with a shorter name 'sampleMetadata_completed.tsv'
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many columns should I have in my sampleMetadata file?
> 2. What kind of class can I have?
>
> > <solution-title></solution-title>
> >
> > 1. At least 2, with the identifiers and the class column. But as many as you need to describe the potential variability of your samples
> >    (*e.g.* the person in charge of the sample preparation, the temperature...). The statistical analysis will expose the relevant parameters.
> > 2. Sample, QC, blank... The class (the 2nd column) is useful for the preprocessing step with XCMS to detect the metabolite across the samples.
> >    So it can be important to separate very different types of samples, as biological ones and blank ones for example. If you don't have any specific class
> >    that you want to consider in XCMS preprocessing, just fill everywhere with `sample` or a dot `.` for example.
> >
> {: .solution}
>
{: .question}


## Getting an overview of your samples' chromatograms

You may be interested in getting an overview of what your samples' chromatograms look like, for example to see if some of
your samples have distinct overall characteristics, *e.g.* unexpected chromatographic peaks or huge overall intensity.

You can use the *sampleMedata* file we previously uploaded to add some group colours to your samples when visualising your chromatograms.
The tool automatically takes the second column as colour groups when a file is provided.

Note that you can also check the chromatograms at any moment during the workflow, in particular at the following steps:
 - After **MSnbase readMSData** {% icon tool %} to help you to define retention time ranges that you may want to discard from the very beginning (*"Specta Filters"* in **findChromPeaks** {% icon tool %})
 - After **adjustRtime** {% icon tool %} to check the result of the correction (and potentially rerun *adjustRtime* with other settings)

> <hands-on-title>xcms plot chromatogram</hands-on-title>
>
> 1. {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `sacurine.raw.RData` (collection)
>    - *"Sample metadata file"*: `sampleMetadata_completed.tsv` you uploaded previously
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    > <comment-title></comment-title>
>    >
>    > If you use this tool at a later step of XCMS workflow while having provided, in the Merger step (see further in this tutorial), a sampleMetadata with a second column containing groups,
>    > you will get colouring according to these groups even without providing a sampleMetadata file as a 'plot chromatogram' parameter.
>    {: .comment}
>
{: .hands_on}

This tool generates Base Peak Intensity Chromatograms (BPIs) and Total Ion Chromatograms (TICs). If you provide groups as we do here, you obtain two plots:
one with colours based on provided groups, one with one colour per sample.

![Base Peak Intensity Chromatograms](../../images/lcms_BPC_9samp.png)


## First XCMS step: *peak picking*

Now that your data is ready for XCMS processing, the first step is to extract peaks from each of your data files
independently. The idea here is, for each peak, to proceed to chromatographic peak detection.

The XCMS solution provides two different algorithms to perform chromatographic peak detection: *matchedFilter* and
*centWave*. The matchedFilter strategy is the first one provided by the XCMS R package. It is compatible with any
LC-MS device, but was developed at a time when high resolution mass spectrometry was not common standard yet. On the
other side, the **centWave** algorithm ({% cite Tautenhahn2008 %}) was specifically developed for high resolution mass spectrometry, dedicated to
data in centroid mode. In this tutorial, you will practice using the centWave algorithm.


> <comment-title>How the centWave algorithm works</comment-title>
>
> Remember that these steps are performed for each of your data files independently.
>  - Firstly, the algorithm detects series of scans with close values of m/z. They are called 'region of interest' (ROI).
> The m/z deviation is defined by the user. The tolerance value should be set according to the mass spectrometer accuracy.
>  - On these regions of interest, a second derivative of a Gaussian model is applied to these consecutive scans in order to define
> the extracted ion chromatographic peak. The Gaussian model is defined by the peak width which corresponds to the standard deviation
> of the Gaussian model. Depending on the shape, the peak is added to the peak list of the current sample.
>
> At the end of the algorithm, a list of peaks is obtained for each sample. This list is then considered to represent the content
> of your sample; if an existing peak is not considered a peak at this step, then it can not be considered in the next steps of
> pre-processing.
{: .comment}

Let's try performing the peakpicking step with the **xcms findChromPeaks (xcmsSet)** {% icon tool %}


> <hands-on-title>xcms findChromPeaks (xcmsSet)</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy1) %} with the following parameters:
>   - *"RData file"*: `sacurine.raw.RData` (collection)
>   - *"Extraction method for peaks detection"*: `CentWave - chromatographic peak detection using the centWave method`
>     - *"Max tolerated ppm m/z deviation in consecutive scans in ppm"*: `3`
>     - *"Min,Max peak width in seconds"*: `5,20`
>     - In **Advanced Options**:
>       - *"Prefilter step for for the first analysis step (ROI detection)"*: `3,5000`
>       - *"Noise filter"*: `1000`
>
> You can leave the other parameters with their default values.
>
>    > <comment-title></comment-title>
>    >
>    > Along with the parameters used in the core centWave algorithm, XCMS provides other filtering options allowing you to get
>    > rid of ions that you don't want to consider. For example, you can use *Spectra Filters* allowing you to discard some RT or M/z
>    > ranges, or *Noise filter* (as in this hands-on) not to use low intensity measures in the ROI detection step.
>    {: .comment}
>
{: .hands_on}

At this step, you obtain a dataset collection containing one `RData` file per sample, with independent lists of ions. Although this
is already a nice result, what you may want now is to get all this files together to identify which ions are shared between samples.
To do so, XCMS provides a function that is called *groupChromPeaks* (or group). But before proceeding to this grouping step, first you
need to group your individual RData files into a single one.


## Gathering the different samples in one Rdata file

A dedicated tool exists to merge the different `RData` files into a single one: **xcms findChromPeaks Merger** {% icon tool %}. Although you can simply take as
input your dataset collection alone, the tool also provides de possibility to take into account a sampleMetadata file. Indeed,
depending of your analytical sequence, you may want to treat part of your samples a different way when proceeding to the grouping step using **xcms groupChromPeaks (group)** {% icon tool %}.

This can be the case for example if you have in your analytical sequence some blank samples (your injection solvent) that you want to
extract along with your biological samples to be able to use them as a reference for noise estimation and noise filtering. The fact that
these blank samples have different characteristics compared to your biological samples can be of importance when setting parameters of
your grouping step. You will see what this is all about in the 'grouping' section of this tutorial, but in the workflow order, it is
at this step that you need to provide the needed information if you want distinction in your grouping step.

**In the case of our tutorial data**, we do not want to separate the samples according to groups, so we do not provide the sampleMetadata when executing
the Merger tool.


> <hands-on-title>xcms findChromPeaks Merger</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `sacurine.raw.xset.RData` (collection)
>    - *"Sample metadata file"*: `Nothing selected`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
{: .hands_on}

The tool generates a single `RData` file containing information from all the samples in your dataset collection input.

## Second XCMS step: *determining shared ions across samples*

The first peak picking step gave us lists of ions for each sample. However, what we want now is a single matrix of ions intensities for all samples.
To obtain such a table, we need to determine, among the individual ion lists, which ions are the same. This is the aim of the present step, called
'grouping'.

The group function aligns ions extracted with close retention time and close m/z values in the different samples. In order to define this
similarity, we have to define on one hand a m/z window and on the other hand a retention time window. A binning is then performed in the
mass domain. The size of the bins is called width of overlapping m/z slices. You have to set it according to your mass spectrometer resolution.

Then, a kernel density estimator algorithm is used to detect region of retention time with high density of ions. This algorithm uses a Gaussian
model to group together peaks with similar retention time.

The inclusion of ions in a group is defined by the standard deviation of the Gaussian model, called bandwidth. This parameter has a large weight
on the resulting matrix. It must be chosen according to the quality of the chromatography. To be valid, the number of ions in a group must be greater
than a given number of samples. Either a percentage of the total number of samples or an absolute value of samples can be given. This is defined by the user.

> <hands-on-title>xcms groupChromPeaks (group)</hands-on-title>
>
> 1. {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy1) %} with the following parameters:
>   - *"RData file"*: `xset.merged.RData`
>   - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>     - *"Bandwidth"*: `5.0`
>     - *"Width of overlapping m/z slices"*: `0.01`
>
{: .hands_on}


This grouping step is very important because it defines the data matrix which will be used especially for the statistical analyses.
User has to check the effect of parameter values on the result.

In order to check the result of the grouping function, a pdf file is created. It provides one plot per m/z slice found in the data. Each picture
represents the peak density across samples, plotting the corresponding Gaussian model which width is defined by the bandwidth parameter. Each red
dot corresponds to a sample. The plot allows to assess the quality of alignment. The grey areas' width is associated with the bandwidth parameter.

Here is an example of two m/z slides obtained from the hands-on:

![plotChromPeakDensity.pdf](../../images/lcms_group_9samp.png)


> <question-title></question-title>
>
> 1. Look at the `283.1127 - 283.1163` m/z slice. How many peak groups are considered? Can you explain why some peaks are not affected to peak groups?
> 2. Look at the `284.1198 - 284.1253` m/z slice. What do you think could have happened if you had used a smaller bandwidth value?
>
> > <solution-title></solution-title>
> >
> > 1. There are 3 peak groups in this m/z slice. The two peaks that are not assigned to peak groups are alone in their retention time area. Thus,
> >    the number of samples under the corresponding density peaks does not reach the minimum fraction of samples set by the user (0.5) to consider a peak group.
> > 2. If the bandwidth value had been set to a smaller value, the density peak width would have been smaller. With a small-enough bandwidth value,
> >    there could have been two density peaks instead of one under the current first density peak. Thus, the sample in line 5 would have been out of the
> >    previous peak group, thus not assigned to any peak group due to the 0.5 minimum fraction limit.
> >
> {: .solution}
>
{: .question}

When looking at the plots from plotChromPeakDensity.pdf, we can notice that in some cases there seems to be a small drift of retention time for some samples.
This phenomenon is well known with LC-MS techniques. To be able to attribute correct groups for peaks, it may be needed to perform some retention time
correction accross samples. Thus, the idea is (when needed) to apply a retention time strategy on the output of your grouping step, then to perform
a second grouping step on the corrected data.

## Optional XCMS step: *retention time correction*

Sometimes with LC-MS techniques, a deviation in retention time occurs from a sample to another. In particular, this is likely to be observed when you
inject large sequences of samples.

This optional step aims to correct retention time drift for each peak among samples. The correction is based on what is called *well behaved peaks*,
that are peaks found in all samples or at least in most of the samples.

Sometimes it is difficult to find enough peaks present in all samples. The user can define a percentage of the total number of samples in which
a peak should be found to be considered a well behaved peak. This parameter is called *minimum required fraction of samples*.

On the contrary, you may have peak groups with more detected peaks than the total number of samples. Those peaks are called *additional peaks*.
If you do not want to consider peak groups with too much additional peaks as 'well behaved peaks', you can use the 'maximal number of additional
peaks' parameter to put them aside.

The algorithm uses statistical smoothing methods. You can choose between linear or loess regression.


> <hands-on-title>xcms adjustRtime (retcor)</hands-on-title>
>
> 1. {% tool [xcms adjustRtime (retcor)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.RData`
>    - *"Method to use for retention time correction"*: `PeakGroups - retention time correction based on aligment of features (peak groups) present in most/all samples.`
>        - *"Minimum required fraction of samples in which peaks for the peak group were identified"*: `0.8299`
>
> You can leave the other parameters to default values.
>
>    > <comment-title></comment-title>
>    >
>    > If you have a very large number of samples (*e.g.* a thousand), it might be impossible to find peaks that are present in 100% of your samples.
>    > If that is the case and you still set a very high value for the minimum required fraction of samples, the tool can not complete successfully the retention
>    > time correction. A special attention should also be given to this parameter when you expect a large number of peaks not to be present in part of your samples
>    > (*e.g.* when dealing with some blank samples).
>    {: .comment}
>
{: .hands_on}


This tool generates a plot output that you can use to visualise how retention time was applied across the samples and along the chromatogram.
It also allows you to check whether the well behaved peaks were distributed homogeneously along the chromatogram.

> <tip-title>Check the impact of RT correction using 'xcms plot chromatogram'</tip-title>
>
> Apart from the plots generated by the adjustRtime tool, you can check the impact of the retention time
> correction by comparing the chromatogram you obtained previously to a new one generated after correction.
>
> > <hands-on-title>xcms plot chromatogram</hands-on-title>
> >
> > 1. {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy1) %} with the following parameters:
> >    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
> >
> >    > <comment-title></comment-title>
> >    >
> >    > As in the previous 'plot chromatogram', you can use your completed sampleMetadata file to get colours.
> >    {: .comment}
> >
> {: .hands_on}
>
>
{: .tip}


The retention time correction step is not mandatory. However, when it is used retention time are modified.
Consequently, applying this step on your data requires to complete it with an additional 'grouping' step using the
**xcms groupChromPeaks (group)** {% icon tool %} tool again.

Parameters for this second group step are expected to be similar to the first group step. Nonetheless,
since retention times are supposed to be less variable inside a same peak group now, in some cases it can be relevant to
lower a little the bandwidth parameter.

> <hands-on-title>second 'xcms groupChromPeaks (group)'</hands-on-title>
>
> 1. {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `5.0`
>        - *"Width of overlapping m/z slices"*: `0.01`
>    - *"Get the Peak List"*: `Yes`
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>        - *"Replace the remain NA by 0 in the dataMatrix"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > When performing this second grouping (or at the first one if you do not plan to perform retention time correction),
>    > you can take this opportunity to check how your peak table looks like at this point of the XCMS extraction. For this, you can
>    > set the 'Get the Peak List' option to `Yes`.
>    {: .comment}
>
{: .hands_on}

It is possible to use the retention time correction and grouping step in an iterative way if needed. Once you perform your
last adjustRtime step and thus your last grouping step, you will obtain your final peak list (*i.e.* final list of ions).

> <question-title></question-title>
>
> 1. How many ions did you obtained with the final grouping step?
> 2. Open the dataMatrix file you obtained with the final grouping. This table corresponds to intensities for each ion and each
>    sample. What do you notice when looking at the intensity of the first ion regarding the first sample?
>
> > <solution-title></solution-title>
> >
> > 1. The final grouping step led to 5815 ions.
> > 2. The first ion (M58T69) has a 'NA' value for the first sample (QC1_014). This is also the case for several other ions and samples.
> >
> {: .solution}
>
{: .question}

At this point of the XCMS extraction workflow, the peak list may contain NA when peaks where not considered peaks in only some
of the samples in the first 'findChromPeaks' step. This does not necessary means that no peak exists for these samples. For example,
sometimes peaks are of very low intensity for some samples and were not kept as peaks because of that in the first 'findChromPeaks'
step.

To be able to get the information that may actually exist behind NAs, there is an additional XCMS step that is called *fillChromPeaks*.

> <comment-title></comment-title>
>
> Before performing the 'fillChromPeaks' step, it is highly recommended to first have a look at your data concerning the distribution
> of NAs in your data. Indeed, this will allow you to check whether your results are consistent with your expectations; if not you
> may want to go back to some of your parameter choices in previous XCMS steps.
> To perform your NA diagnosis, you can use the variableMetadata file and dataMatrix file that you obtained with the last grouping step
> with the 'Get the Peak List' option to `Yes`. The variableMetadata file contains information about your ions: you will find information
> about the number of peaks detected for each ion. The dataMatrix files contains the intensities for each ion and each sample; you can get
> an overview of it using the **Intensity Check** {% icon tool %} module.
{: .comment}


## Final XCMS step: *integrating areas of missing peaks*

The idea of the XCMS step is to integrate signal in the mz-rt area of an ion (chromatographic peak group) for samples in which no
chromatographic peak for this ion was identified.

> <hands-on-title>xcms fillChromPeaks (fillPeaks)</hands-on-title>
>
> 1. {% tool [xcms fillChromPeaks (fillPeaks)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.*.RData`
>    - In *"Peak List"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>
>    > <comment-title></comment-title>
>    >
>    > The *"Reported intensity values"* parameter is important here. It defines how the intensity will be computed. You have three choices:
>    > - into : integration of peaks (*i.e.* areas under the peaks)
>    > - maxo : maximum height of peaks
>    > - intb : integration of peaks with baseline subtraction
>    {: .comment}
>
{: .hands_on}

With this 'fillChromPeaks' step, you obtain your final intensity table. At this step, you have everything mandatory to begin analysing
your data:
 - A *sampleMetadata* file (if not done yet, to be completed with information about your samples)
 - A *dataMatrix* file (with the intensities)
 - A *variableMetadata* file (with information about ions such as retention times, m/z)

Nonetheless, before proceeding with the next step in the workflow (processing and filtering of your data), you can add an optional step with the
**CAMERA.annotate** {% icon tool %} module. This tool uses the CAMERA R package to perform a first annotation of your data based on XCMS outputs.


## Annotation with CAMERA [Optional]

This last step provides annotation of isotopes, adducts and neutral losses. It gives also some basic univariate statistics in case you
considered several groups for your XCMS extraction.

There is a huge number of parameters that will not be detailed in this short tutorial. However most of the default values can be kept here
for a first attempt to run this function. Nevertheless, a few parameters have to be set at each run:
 - The polarity has to be set since it affects annotation.
 - For statistical analysis, you have to define if you have two or more conditions to compare. These conditions had to be defined in the
 sample metadata uploaded previously and taken into account in the XCMS workflow.
 - You can define how many significant ions will be used for extracted ions chromatogram (EIC) plot. These plots will be included in a pdf file.

 Apart from the PDF file, the main three outcomes from **CAMERA.annotate** {% icon tool %} are three columns added in the variableMetadata file:
 - **isotopes:** the name says everything
 - **adduct:** same here; this column is filled only in the 'All functions' mode
 - **pcgroup:** this stands for Pearson's correlation group; it corresponds to groups of ions that match regarding retention time and intensity
 correlations, leading to think that maybe they could come from the same original metabolite.

> <hands-on-title>CAMERA.annotate</hands-on-title>
>
> 1. {% tool [CAMERA.annotate](toolshed.g2.bx.psu.edu/repos/lecorguille/camera_annotate/abims_CAMERA_annotateDiffreport/2.2.7+camera1.48.0-galaxy1) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.*.fillChromPeaks.RData`
>    - In *"Annotate Isotopes [findIsotopes]"*:
>        - *"Max. ion charge"*: `2`
>    - *"Mode"*: `Only groupFWHM and findIsotopes functions [quick]`
>    - In *"Export options"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>
>    > <comment-title></comment-title>
>    >
>    > As said previously, there are quite a few parameters in this tool, some of them having very high impact on your annotations.
>    > In particular, the **Mode** parameter will influence a lot your results regarding pcgroups, and adducts (that will not be computed otherwise).
>    {: .comment}
>
{: .hands_on}

The information given by this tool is not mandatory for the next step of the metabolomic workflow. Commonly, annotation is considered a later
step in the pipeline, but since CAMERA uses the outputs of XCMS, if you want to use it you better do it at this step, allowing you to have the
corresponding information in your variableMetadata file for later use.


# Stopover: debriefing and preparation for next steps

At the end of the Preprocessing step, you should have three tabulation-separated tables:
 - A **sampleMetadata** file given and completed by the user
 - A **dataMatrix** file from XCMS.fillChromPeaks
 - A **variableMetadata** file from either XCMS.fillChromPeaks or CAMERA.annotate

Concerning the **sampleMetadata** file, for the next steps of the workflow, there are four columns that are mandatory to go through all the analysis:
 - `injectionOrder`: a numerical column of injection order
 - `sampleType`: specifies if a QC pool or a sample (coded `pool` or `sample`)
 - `batch`: a categorical column indicating the batches of analysis (if only one, must be a constant)
 - your variable(s) of interest: here we will consider as an example the Body Mass Index (`bmi`)

> <comment-title></comment-title>
>
> The preprocessing part of this analysis can be quite time-consuming, and already corresponds to quite a few number of steps, depending of your analysis.
> It can also generate several versions of your 3 tables, with only one of interest for each at the end of the extraction process. We highly recommend,
> at this step of the metabolomic workflow, to split your analysis by beginning a new Galaxy history with only the 3 tables you need. This will help
> you in limiting selecting the wrong dataset in further analyses, and bring a little tidiness for future review of your analysis process.
>
> To begin a new history with the 3 tables from your current history, you can use the functionality 'copy dataset' and copy it into a new history.
>
> We also recommend you to rename your 3 tables before proceeding with the next steps of the metabolomic workflow. Indeed, you may have notice that
> the XCMS tools generate output names that contain the different XCMS steps you used, allowing easy traceability while browsing your history.
> However, knowing that the next steps of analysis are also going to extend the 3 tables' names, if you keep the original names it will become very
> long and thus may reduce the names' readability. Hence, we highly recommend you to rename them with something short, *e.g.* 'sampleMetadata',
> 'variableMetadata' and 'dataMatrix', or anything not too long that you may find convenient.
{: .comment}




> <hands-on-title>Copying the 3 tables into a new history and renaming them</hands-on-title>
>
> 1. Create a new history with the 3 tables
>
>    {% snippet faqs/galaxy/histories_copy_dataset.md history_name="Sacurine Processing" %}
>
> 2. Rename the 3 tables with shorter names:
>    - `xset.merged.groupChromPeaks.adjustRtime.groupChromPeaks.fillpeaks.dataMatrix.tsv` -> **dataMatrix.tsv**
>    - `xset.merged.groupChromPeaks.*.fillChromPeaks.annotate.variableMetadata.tsv` -> **variableMetadata.tsv**
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Data processing: quality checks, normalisation, data filtering

In the previous step of the LC-MS workflow, you saw how to extract features from your acquisition files. This data is
shaped in a format allowing the use of various standard statistical methods. However, being able to perform a
statistical analysis does not mean necessarily being able to highlight relevant information. Indeed, data are often affected
by various sources of unwanted variability. It can limit the effectiveness of statistical methods, leading sometimes to
difficulties in revealing investigated effects. Identifying such variability can help analyzing your data at its full potential.

In this tutorial, we chose to limit the data processing to 3 steps:
 - overview of the variability in the data
 - signal drift correction
 - filtering of unreliable variables based on coefficients of variation

> <comment-title></comment-title>
> To get a little more information regarding data processing, do not hesitate to visit the usemetabo.oc platform:
> [Link to LC-MS processing step](https://usemetabo.org/courses/w4mlc-ms-processing)
{: .comment}


## Step 1: global variability in the data

Commonly, LC-MS analyses generate a significant number of variables (hundreds to thousands). Getting a complete view of
such dataset may not be an easy task, but getting a glimpse of it is possible using some common unsupervised multivariate
analysis. One of the most commonly used method is the **Principal Components Analysis** (PCA). You can get a basic PCA along with
over useful information using the **Quality Metrics** {% icon tool %} tool.

> <hands-on-title>Using <b>Quality Metrics</b> to get an overview of your data</hands-on-title>
>
> 1. {% tool [Quality Metrics](toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8) %} with the following parameters:
>    - *"Data matrix file"*: `dataMatrix.tsv`
>    - *"Sample metadata file"*: `sampleMetadata_completed.tsv`
>    - *"Variable metadata file"*: `variableMetadata.tsv`
>
{: .hands_on}

For a first overview of your data, you can focus on the graphical output of this tool: **Quality_Metrics_figure.pdf**.
It provides a variety of useful information:
 - Summary of the intensities in the dataMatrix file (information.txt file and plot top center paragraph)
 - View of these intensities with a color scale (plot bottom right panel)
 - 2-components PCA score plot to check for clusters or outliers (plot top left panel)
 - Sum of intensities per sample according to injection order to check the presence of signal drift or batch effect (plot top center panel)
 - Z-scores for intensity distribution and proportion of missing values (plot bottom left panel)
 - Ions' standard deviation (sd) and mean values (plot top right panel)

![Quality_Metrics_figure.pdf](../../images/lcms_QM_9samp_raw.png)

> <question-title>cross-referencing information</question-title>
>
> Look at the 'Quality_Metrics_figure.pdf' file.
> 1. Look at the proportion of ions with a pool coefficient of variation <30%. Knowing that pools are identical samples, and that
>    values with a CV > 30% can not be considered stables, what can you conclude about your dataset regarding the possibility to
>    compare intensities between samples?
> 2. Look at the sum of intensities per sample according to injection order. What major information do you observe on the plot?
>    Can this observation help you understand the CV results you just looked at?
> 3. Now look at the PCA plot. Can you explain the first component (t1)? Use the previous plot to help you.
>
> > <solution-title></solution-title>
> >
> > 1. You can read on the plot that *pool CV < 30%: 26%*, meaning that the pool values are stable only for a quarter of the ions
> >    in your dataset. If the pooled samples are not stable, this means that you can observe differences between samples even when
> >    there are no biological differences. Thus, comparing samples becomes difficult and has high risk of being unreliable.
> >    Consequently, with only a quarter of the ions being stable regarding pool intensities, performing statistical analyses on
> >    this full dataset would probably lead to unreliable results.
> > 2. We can see in the figure that the global intensity of samples seems to decrease with the injection order. In particular,
> >    the fact that the pooled samples' intensities decrease leads us to suspect a signal drift due to the clogging effect of successive
> >    injection of samples.
> >    This signal drift could be the reason why so many ions in the dataset led to high CV values for pools, since it prevents
> >    at least part of the ions to be stable regarding pools' intensities.
> > 3. If we look closely at the samples' identifiers on the plot, it seems that the lowest numbers in IDs are at the right side
> >    of the first component. Knowing that these numbers correspond to an order in the injection sequence, we can link it to the
> >    previous picture's samples. Then, what we can observe is that the order of samples in the first component of PCA from right to left
> >    corresponds approximately to the decreasing order of sums of intensities. Thus, we can conclude that the main variability in
> >    the dataset may be due to the signal drift.
> >
> {: .solution}
>
{: .question}

## Step 2: handling the signal drift observed all through the analytical sequence

It is known that when injecting successively a large number of samples, the system tends to get dirty, and this may cause a measure drift.
To prevent inability to catch signal anymore, in case of large injection series, the sequence is generally divided into several batches
and the source is cleaned between batches. Unfortunately, these signal drift and batch design can add significant variability in data,
making sample comparison complicated. In case data is impacted by these effects, it is highly recommended to normalise the data to get
rid of these unwanted variabilities.

In our case study, we saw that the data seemed to be affected by signal drift. Thus, we will use the **Batch_correction** {% icon tool %} tool
to get rid of it.

> <hands-on-title>Data normalisation</hands-on-title>
>
> 1. {% tool [Batch_correction](toolshed.g2.bx.psu.edu/repos/melpetera/batchcorrection/Batch_correction/3.0.0) %} with the following par


topic: metabolomics
tutorial: lcms-preprocessing
title: Mass spectrometry: LC-MS preprocessing with XCMS
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: []
content_hash: b3f0d5326ae0a4c7d1054132b590ccc1
last_modified: 2026-02-26T21:12:17.160700
zenodo_link: https://zenodo.org/record/3757956
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html

content:
Metabolomics is a *-omic* science known for being one of the most closely related to phenotypes.
It involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants.
It focuses on studying the very small molecules which are called *metabolites*, to better understand matters linked to the metabolism.

One of the three main technologies used to perform metabolomic analyses is **Liquid-Chromatography Mass Spectrometry** (LC-MS). Data analysis
for this technology requires a large variety of steps, ranging from extracting information from the raw data to statistical analysis and annotation.
To be able to perform a complete LC-MS analysis in a single environment, the [Wokflow4Metabolomics](http://workflow4metabolomics.org/)
team provides Galaxy tools dedicated to metabolomics. This tutorial details the steps involved in the first part of untargeted LC-MS
data processing: extracting information from raw data to obtain what is called a peak table. This step is commonly refered to as
the preprocessing step. This tutorial will show you how to perform such a step using the Galaxy implementation of the XCMS software.

To illustrate this approach, we will use data from {% cite Thvenot2015 %}. The objectives of this paper was to analyze
the influence of age, body mass index, and gender on the urine metabolome. To do so, the authors collected samples
from 183 employees from the French Alternative Energies and Atomic Energy Commission (CEA) and performed LC-HRMS LTQ-Orbitrap
(negative ionization mode).

Since the original dataset takes a few hours to be processed, we chose to take a limited subset of individuals for this tutorial.
This will enable you to perform an example of metabolomic preprocessing workflow in a limited time, even though
the results obtained may not be reliable enough for biological interpretation due to the small sample size.
Nevertheless, the chosen diversity of sample will allow you to explore the basics of a preprocessing workflow.

We chose a subset of 12 samples, composed of 6 biological samples, 3 quality-control pooled samples ('QC pools' - mix of all
biological samples) and 3 blank samples ('blanks' - solvent injection).

To analyze these data, we will then follow a Galaxy workflow
developed by the [Wokflow4metabolomics group](http://workflow4metabolomics.org/) ({% cite Giacomoni2014 %}, {% cite Guitton2017 %}).

[![The full tutorial workflow represented as boxes linked with arrows, each box being labeled with the corresponding galaxy tool name](../../images/lcmspreproc_wf.png "The tutorial workflow")](../../images/lcmspreproc_wf.png)

> <comment-title>Workflow4Metabolomics public history</comment-title>
>
> This training material can be followed running it on any Galaxy instance holding the Galaxy modules needed.
> Nonetheless, if you happen to be a W4M user and do not want to run the hands-on yourself, please note that
> you can find the entire history in the 'published histories' section:
> [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms)
>
{: .comment}

The preprocessing steps presented in this tutorial are built around the solutions provided through the R package **XCMS**.

**XCMS** ({% cite Smith2006 %}) is a free and open source software dedicated to pre-processing of any type of mass spectrometry acquisition files from low to
high resolution, including FT-MS data coupled with different kind of chromatography (liquid or gas). This software is
used worldwide by a huge community of specialists in metabolomics using mass spectrometry methods.

This software is based on different algorithms that have been published, and is provided and maintained using R software.

Since sometimes a couple of pictures is worth a thousand words, you will find in the following slides some material to help
you understand how XCMS works:
[link to slides](../../tutorials/lcms-preprocessing/slides.html).
This document is refered to as "Check the next X slides" in the present training material.
As an example, [check the next 2 slides](../../tutorials/lcms-preprocessing/slides.html#raw-to-matrix)
for complementary material about XCMS R package.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data preparation for XCMS

The **XCMS** {% icon tool %} R package is composed of R functions able to extract, filter, align and fill gap, with the possibility to annotate isotopes,
adducts and fragments using the CAMERA R package ({% cite CAMERA %}). This set of functions gives modularity, and thus is particularly well
adapted to define workflows, one of the key points of Galaxy.

The first Galaxy module, **MSnbase readMSData** {% icon tool %}, is meant to read files with open format as `mzXML`, `mzMl`, `mzData` and `netCDF`,
which are independent of the constructors' formats.
First of all, you need to upload your data (that must be in one of the cited open formats) into Galaxy.


## Importing the LC-MS data into Galaxy

In metabolomics studies, the number of samples can vary a lot (from a handful to several hundreds). Thus, extracting your
data from the raw files can be very fast, or take quite a long time. To optimise the computation time as much as possible,
the W4M core team chose to develop tools that can run single raw files in parallel for the first steps of
pre-processing, since the initial actions in the extraction process treat files independently.

Since the first steps can be run on each file independently, the use of **Dataset collections** in Galaxy is recommended, to avoid
having to launch jobs manually for each sample. You can start using the dataset collection option from the very beginning of your analysis, when uploading your data into Galaxy.

> <hands-on-title>Data upload the mzXML with <b>Get data</b></hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 12 `mzXML` files into a collection named `mzML`
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from Zenodo using the URLs given below
>
>      [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3757956.svg)](https://doi.org/10.5281/zenodo.3757956)
>    ```
>    https://zenodo.org/record/3757956/files/Blanc05.mzML
>    https://zenodo.org/record/3757956/files/Blanc10.mzML
>    https://zenodo.org/record/3757956/files/Blanc16.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_048.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_090.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_123.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_157.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_173.mzML
>    https://zenodo.org/record/3757956/files/HU_neg_192.mzML
>    https://zenodo.org/record/3757956/files/QC1_002.mzML
>    https://zenodo.org/record/3757956/files/QC1_008.mzML
>    https://zenodo.org/record/3757956/files/QC1_014.mzML
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="mzML" renaming=false %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md astype="as a Collection" %}
>
> 3. Make sure your data is in a **collection**. Make sure it is named `mzML`
>    - If you forgot to select the collection option during import, you can create the collection now:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > If you happen to be a W4M user, please note that you can find at the following link a ready-to-start history:
> > [GTN_LCMSpreprocessing_mzML](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingmzml).
> > We highly recommend to get started by importing this history.
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > this step corresponds to the dataset collection number 13.
> {: .tip}
>
{: .hands_on}

You should have in your history a green Dataset collection (`mzML`) with 12 datasets in mzml format.

Their size can be checked by clicking on the information icon {% icon galaxy-info %} on the individual datasets.

## First Galaxy module: *MSnbase readMSData*

This first step is only meant to read your `mzXML` file and generate an object usable by **XCMS** {% icon tool %}.

**MSnbase readMSData** {% icon tool %} takes as input your raw files and prepares `RData` files for the first XCMS step.

> <hands-on-title>MSnbase readMSData</hands-on-title>
>
> 1. Run {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} with the following parameters:
>    - *"File(s) from your history containing your chromatograms"*: the `mzML` dataset collection
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>  > <tip-title>Comment to W4M users</tip-title>
>  >
>  > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
>  > this step corresponds to the dataset collection number 14.
>  {: .tip}
>
>  > <question-title></question-title>
>  >
>  > What do you get as output?
>  >
>  > > <solution-title></solution-title>
>  > >
>  > > 1. A **Dataset collection** containing 12 datasets.
>  > > 2. The datasets are some RData objects with the **rdata.msnbase.raw** datatype.
>  > >
>  > {: .solution}
>  >
>  {: .question}
{: .hands_on}

Now that you have prepared your data, you can begin with the first XCMS extraction step: peakpicking. However, before beginning to
extract meaningful information from your raw data, you may be interested in visualising your chromatograms. This can be of particular
interest if you want to check whether you should consider discarding some range of your analytical sequence (some scans or *retention
time* (RT) ranges).

To do so, you can use a tool that is called **xcms plot chromatogram** {% icon tool %} that will plot each sample's chromatogram (see dedicated section
further). However, to use this tool, you may need additional information about your samples for colouring purpose. Thus, you may need
to upload into Galaxy a table containing metadata of your samples (a *sampleMetadata* file).


## Importing a sample metadata file

What we referenced here as a *sampleMetadata* file corresponds to a table containing information about your samples (= sample metadata).

A sample metadata file contains various information for each of your raw files:
- **Classes** which will be used during the preprocessing steps
- **Analytical batches** which will be useful for a batch correction step, along with **sample types** (pool/sample) and **injection order**
- Different **experimental conditions** which can be used for statistics
- Any information about samples that you want to keep, in a *column* format

The content of your sample metadata file has to be filled by you, since it is not contained in your raw data.
Note that you can either:
- Upload an existing metadata file
- Use a template to create one (because it can be painful to get the sample list without misspelling or omission)
  1. Generate a template with the **xcms get a sampleMetadata file** {% icon tool %} tool
  2. Fill it using your favorite table editor (Excel, LibreOffice)
  3. Upload it within Galaxy

> <tip-title>Optional: Generate the right template with <b>xcms get a sampleMetadata file</b></tip-title>
>
> In the case of this tutorial, we already prepared a *sampleMetadata* file with all the necessary information.
> Below is an optional hands-on explaining how to get a template to fill, with the two following advantages:
>   1. You will have the exact list of the samples you used in Galaxy, with the exact identifiers (*i.e.* exact sample names)
>   2. You will have a file with the right format (tabulation-separated text file) that only needs to be filled with the information you want.
>
> > <hands-on-title>xcms get a sampleMetadata file</hands-on-title>
> >
> > {% tool [xcms get a sampleMetadata file](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy0) %} with the following parameters:
> >    - {% icon param-collection %} *"RData file"*: the `mzML.raw.RData` collection output from **MSnbase readMSData** {% icon tool %}
> >
> > > <tip-title>Comment to W4M users</tip-title>
> > >
> > > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > > this step corresponds to the dataset number 27.
> > {: .tip}
> >
> {: .hands_on}
>
> An easy step for an easy sampleMetadata filling!
>
> From this tool, you will obtain a `tabular` file (meaning a tab-separated text file) with a first column of identifiers and a
> second column called *class* which is empty for the moment (only '.' for each sample). You can now download this file by clicking on the {% icon galaxy-save %} icon.
>
{: .tip}


### Prepare your sampleMetadata file

The sampleMetadata file is a tab-separated table, in text format. This table has to be filled by the user. You can use any
software you find appropriate to construct your table, as long as you save your file in a compatible format. For example, you can
use a spreadsheet software such as Microsoft Excel or LibreOffice.

> <warning-title> Save your table in the correct format</warning-title>
>
> The file has to be a `.txt` or a `.tsv` (tab-separated values). Neither `.xlsx` nor `.odt` are supported.
> If you use a spreadsheet software, be sure to change the default format to **Text (Tab delimited)** or equivalent.
{: .warning}

Once your sampleMetadata table is ready, you can proceed to the upload. In this tutorial we already prepared the table for you ;)

> <tip-title>Optional: Filling the <i>sampleMetadata</i> using the template obtained from Galaxy</tip-title>
>
> For this tutorial, we already provide the *sampleMetadata* file, so you only have to upload it to Galaxy. Below we
> explain how we filled this file from the template we generated in Galaxy.
>
> First, we used **xcms get a sampleMetadata file** {% icon tool %} as mentioned in the previous tip box.
>
> We obtained the following table:
>
> sample_name | class
> --- | ---
> QC1_014 | .
> QC1_008 | .
> QC1_002 | .
> HU_neg_192 | .
> HU_neg_173 | .
> HU_neg_157 | .
> HU_neg_123 | .
> HU_neg_090 | .
> HU_neg_048 | .
> Blanc16 | .
> Blanc10 | .
> Blanc05 | .
>
> We used a spreadsheet software to open the file. First, we completed the class column. You will see in further XCMS steps that this
> second column matters.
>
> sample_name | class
> --- | ---
> QC1_014 | QC
> QC1_008 | QC
> QC1_002 | QC
> HU_neg_192 | sample
> HU_neg_173 | sample
> HU_neg_157 | sample
> HU_neg_123 | sample
> HU_neg_090 | sample
> HU_neg_048 | sample
> Blanc16 | blk
> Blanc10 | blk
> Blanc05 | blk
>
> With this column, we will be able to colour the samples depending on the sample type (QC, sample, blk).
> Next, we added columns with interesting or needed information, as following:
>
> sample_name | class | sampleType | injectionOrder | batch | osmolality | sampling | age | bmi | gender
> --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
> QC1_014 | QC | pool | 185 | ne1 | NA | NA | NA | NA | NA
> QC1_008 | QC | pool | 105 | ne1 | NA | NA | NA | NA | NA
> QC1_002 | QC | pool | 27 | ne1 | NA | NA | NA | NA | NA
> HU_neg_192 | sample | sample | 165 | ne1 | 1184 | 8 | 31 | 24.22 | Male
> HU_neg_173 | sample | sample | 148 | ne1 | 182 | 7 | 55 | 20.28 | Female
> HU_neg_157 | sample | sample | 137 | ne1 | 504 | 7 | 43 | 21.95 | Female
> HU_neg_123 | sample | sample | 100 | ne1 | 808 | 5 | 49 | 24.39 | Male
> HU_neg_090 | sample | sample | 75 | ne1 | 787 | 4 | 46 | 19.79 | Male
> HU_neg_048 | sample | sample | 39 | ne1 | 997 | 3 | 39 | 19.49 | Female
> Blanc16 | blk | blank | 173 | ne1 | NA | NA | NA | NA | NA
> Blanc10 | blk | blank | 94 | ne1 | NA | NA | NA | NA | NA
> Blanc05 | blk | blank | 29 | ne1 | NA | NA | NA | NA | NA
>
> In particular, the `batch`, `sampleType` and `injectionOrder` columns are mandatory to correct the data from signal drift during later quality processing
> (see the [Mass spectrometry: LC-MS analysis](../../tutorials/lcms/tutorial.html)
> Galaxy training material for more information).
> Once we completed the table filling, we saved the file, minding to stick with the original format. Then, our *sampleMetadata* was ready to
> be uploaded into Galaxy.
>
{: .tip}

> <warning-title> The class column </warning-title>
>
> Depending on further choices, the sampleMetadata file can be decisive.
> It can be used to colour some plots, but also for ion selection (see further in the tutorial).
> Please note that the information needed for these steps **should be given as the second column of the sampleMetadata file**,
> the first one being the samples' identifiers.
{: .warning}


### Upload the sampleMetada file with 'Get data'

> <hands-on-title>Upload the sampleMetada</hands-on-title>
>
> 1. Import the `sampleMetadata_completed.tsv` file from your computer, from Zenodo or from a shared data library
>    ```
>    https://zenodo.org/record/3757956/files/sampleMetadata_12samp_completed.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Check the data type of your imported files.
>    - The datatype should be `tabular`, if this is not the case, please change the datatype now.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
>    > <comment-title></comment-title>
>    >
>    > Here we provided the sampleMetadata file so we know that the upload led to a 'tabular' file. But from experience we also know that
>    > it can happen that, when uploading a sampleMetadata table, user obtained other inappropriate types of data. This is generally due to the file
>    > not following all the requirements about the format (*e.g.* wrong separator, or lines with different numbers of columns).
>    > Thus, we highly recommend that you always take a second to check the data type after the upload. This way you can handle the problem
>    > right away if you appear to get one of these obvious issues.
>    {: .comment}
>
> 3. Rename your sampleMetadata file with a shorter name 'sampleMetadata_completed.tsv'
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > this step corresponds to the dataset number 28.
> {: .tip}
>
>
> > <question-title></question-title>
> >
> > 1. How many columns should I have in my sampleMetadata file?
> > 2. What kind of class can I have?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. At least 2, with the identifiers and the class column. But as many as you need to describe the potential variability of your samples
> > >    (*e.g.* the person in charge of the sample preparation, the temperature...). This will allow later statistical analysis to expose the relevant parameters.
> > > 2. Sample, QC, blank... The class (the 2nd column) is useful for the preprocessing step with XCMS to detect the metabolite across the samples.
> > >    Thus, it can be important to separate very different types of samples, as biological ones and blank ones for example. If you don't have any specific class
> > >    that you want to consider in XCMS preprocessing, just fill everywhere with `sample` or a dot `.` for example.
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

## Getting an overview of your samples' chromatograms

You may be interested in getting an overview of what your samples' chromatograms look like, for example to see if some of
your samples have distinct overall characteristics, *e.g.* unexpected chromatographic peaks or huge overall intensity.

You can use the *sampleMedata* file we previously uploaded to add some group colours to your samples when visualising your chromatograms.
The tool automatically takes the second column as colour groups when a file is provided.

Note that you can also check the chromatograms at any moment during the workflow, in particular at the following steps:
 - After **MSnbase readMSData** {% icon tool %} to help you to define retention time ranges that you may want to discard from the very beginning
(*"Spectra Filters"* section in **xcms findChromPeaks (xcmsSet)** {% icon tool %} parameters)
 - After **adjustRtime** {% icon tool %} to check the result of the correction (and potentially rerun *adjustRtime* with other settings)

> <hands-on-title>xcms plot chromatogram</hands-on-title>
>
> 1. {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `mzML.raw.RData` (collection)
>   - *"Sample metadata file"*: `sampleMetadata_completed.tsv` that you uploaded previously
>
>    > <comment-title></comment-title>
>    >
>    > If you use this tool at a later step of XCMS workflow and provided in the Merger step a sampleMetadata with a second column containing groups
>    > (see further in this tutorial), you will get colouring according to these groups even without providing a sampleMetadata file as a 'plot chromatogram' parameter.
>    {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms) history,
> > this step corresponds to the datasets number 29 and 30.
> {: .tip}
>
{: .hands_on}

This tool generates Base Peak Intensity Chromatograms (BPIs) and Total Ion Chromatograms (TICs). If you provide groups as we do here, you obtain two plots:
one with colours based on provided groups, one with one colour per sample.

![Screenshot of the PDF output corresponding to the Base Peak Intensity Chromatograms. It is composed of the layering of BPIs: one drawn line per sample.](../../images/lcmspreproc_BPC12samp.png "Base Peak Intensity Chromatograms")

How BPIs and TICs look like is dependant of the kind of data you have: LC-MS technology used, study design, type of samples, events that may have occured during the analysis...
It can vary a lot from one experiment to another due to these characteristics.
Sometimes you can see (un)expected effects on these plots as retention time variations accros samples, overall sample intensity differences due to analytical or biological
matters, specific peak areas... Generally, it is recommended to first focus on QC pooled samples if you have some, since they are not supposed to be affected by
biological variabilities: it helps you to focus on aspects uncorrelated to individual sample effects. Evaluation of blank samples (if any) can also be a good start:
"Are all blanks globally less intense than real samples?" is a question that can generally be raised.

# First XCMS step: *peak picking*

Now that your data is ready for XCMS processing, the first step is to extract peaks from each of your data files
independently. The idea here is, for each peak, to proceed to chromatographic peak detection.

The XCMS solution provides two different algorithms to perform chromatographic peak detection: *matchedFilter* and
*centWave*. The matchedFilter strategy is the first one provided by the XCMS R package. It is compatible with any
LC-MS device, but was developed at a time when high resolution mass spectrometry was not common standard yet. On the
other side, the **centWave** algorithm ({% cite Tautenhahn2008 %}) was specifically developed for high resolution mass spectrometry, dedicated to
data in centroid mode. In this tutorial, you will practice using the centWave algorithm.


> <comment-title>How the centWave algorithm works</comment-title>
>
> [Check the next 7 slides](../../tutorials/lcms-preprocessing/slides.html#findchrompeaks)
> to help you understand how the centWave algorithm works.
> Remember that these steps are performed for each of your data files independently.
>  - Firstly, the algorithm detects series of scans with close values of m/z. They are called 'region of interest' (ROI).
> The m/z deviation is defined by the user. The tolerance value should be set according to the mass spectrometer accuracy.
>  - On these regions of interest, a second derivative of a Gaussian model is applied to these consecutive scans in order to define
> the extracted ion chromatographic peak. The Gaussian model is defined by the peak width which corresponds to the standard deviation
> of the Gaussian model. Depending on the shape, the peak is added to the peak list of the current sample.
>
> At the end of the algorithm, a list of peaks is obtained for each sample. This list is then considered to represent the content
> of your sample; if an existing peak is not considered a peak at this step, then it can not be considered in the next steps of
> pre-processing.
>
{: .comment}

Let's try performing the peakpicking step with the **xcms findChromPeaks (xcmsSet)** {% icon tool %} tool.


> <hands-on-title>xcms findChromPeaks (xcmsSet)</hands-on-title>
>
> {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `mzML.raw.RData` (collection)
>   - *"Extraction method for peaks detection"*: `CentWave - chromatographic peak detection using the centWave method`
>     - *"Max tolerated ppm m/z deviation in consecutive scans in ppm"*: `3`
>     - *"Min,Max peak width in seconds"*: `5,20`
>     - In **Advanced Options**:
>       - *"Prefilter step for for the first analysis step (ROI detection)"*: `3,5000`
>       - *"Noise filter"*: `1000`
>
> You can leave the other parameters with their default values.
>
> > <comment-title></comment-title>
> >
> > Along with the parameters used in the core centWave algorithm, XCMS provides other filtering options allowing you to get
> > rid of ions that you don't want to consider. For example, you can use *Spectra Filters* allowing you to discard some RT or m/z
> > ranges, or *Noise filter* (as in this hands-on) not to use low intensity measures in the ROI detection step.
> {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the dataset collections number 31 and 32.
> {: .tip}
>
{: .hands_on}

At this step, you obtain a dataset collection containing one `RData` file per sample, with independent lists of ions. Although this
is already a nice result, what you may want now is to get all this files together to identify which are the shared ions between samples.
To do so, XCMS provides a function that is called *groupChromPeaks* (or group). But before proceeding to this grouping step, first you
need to group your individual RData files into a single one.

# Gathering the different samples in one Rdata file

A dedicated tool exists to merge the different `RData` files into a single one: **xcms findChromPeaks Merger** {% icon tool %}. Although you can simply take as
input your dataset collection alone, the tool also provides the possibility to take into account a sampleMetadata file. Indeed,
depending of your analytical sequence, you may want to treat part of your samples a different way when proceeding to
the grouping step using **xcms groupChromPeaks (group)** {% icon tool %}.

This can be the case for example if you have in your analytical sequence some blank samples (your injection solvent) that you want to
extract along with your biological samples to be able to use them as a reference for additional noise estimation and noise filtering. The fact that
these blank samples have different characteristics compared to your biological samples can be of importance when setting parameters of
your grouping step. You will see what this is all about in the 'grouping' section of this tutorial, but in the workflow order, it is
at this step that you need to provide the needed information if you want distinction in your grouping step.

In the case of our tutorial data, we have three sample types: the original biological samples ('samples'), quality-control pooled samples ('pools')
corresponding to a mix of all biological samples, and blank samples ('blanks') constituted of injection solvent.
We will consider these three classes for the grouping step. Thus, we need to provide the sampleMetadata file during the merging step,
with the second column defining theses classes.


> <hands-on-title>xcms findChromPeaks Merger</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `mzML.raw.xset.RData` (collection)
>   - *"Sample metadata file"*: `sampleMetadata_completed.tsv`
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the dataset number 57.
> {: .tip}
>
{: .hands_on}

The tool generates a single `RData` file containing information from all the samples in your dataset collection input.

# Second XCMS step: *determining shared ions across samples*

The first peak picking step gave us lists of ions for each sample. However, what we want now is a single matrix of ions' intensities for all samples.
To obtain such a table, we need to determine, among the individual ion lists, which ions are the same. This is the aim of the present step, called
'grouping'.

Various methods are available to proceed to the grouping step. In this tutorial, we will focus on the 'PeakDensity' method
that performs peak grouping based on time dimension peak densities.

[Check the next 8 slides](../../tutorials/lcms-preprocessing/slides.html#groupchrompeaks),
you will find additional material to help you understand the grouping algorithm.

The group function aligns ions extracted with close retention time and close m/z values in the different samples. In order to define this
similarity, we have to define on one hand a m/z window and on the other hand a retention time window. A binning is then performed in the
mass domain. The size of the bins is called width of overlapping m/z slices. You have to set it according to your mass spectrometer resolution.

Then, a kernel density estimator algorithm is used to detect region of retention time with high density of ions. This algorithm uses a Gaussian
model to group together peaks with similar retention time.

The inclusion of ions in a group is defined by the standard deviation of the Gaussian model, called bandwidth. This parameter has a large weight
on the resulting matrix. It must be chosen according to the quality of the chromatography. To be valid, the number of ions in a group must be greater
than a given number of samples. Either a percentage of the total number of samples or an absolute number of samples can be given. This is defined by the user.

> <hands-on-title>xcms groupChromPeaks (group)</hands-on-title>
>
> {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>   - *"RData file"*: `xset.merged.RData`
>   - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>     - *"Bandwidth"*: `5.0`
>     - *"Minimum fraction of samples"*: `0.9`
>     - *"Width of overlapping m/z slices"*: `0.01`
>   - *"Get the Peak List"*: `Yes`
>     - *"If NA values remain, replace them by 0 in the dataMatrix"*: `No`
>
> > <comment-title>Minimum fraction of samples</comment-title>
> >
> > This parameter sets the minimum proportion of samples in a class where a peak should be found to keep the corresponding ion in the peaktable.
> > The idea is to look inside **each class (*i.e.* each group of samples defined in the second column of the sampleMetadata file)** and to keep
> > an ion if there is at least one class where the ion is found in at least the specified proportion of samples.
> >
> > The way to define what kind of classes would be relevant is not straightforward. It is a somehow complex combination of your study objectives,
> > the kind of biological matrix you analyse, assumptions you may have on the presence or absence of some compounds...
> > What could typically be considered is whether you have radically different kind of samples (distinct parts of an organism, distinct kind of
> > sample preparation...) and whether you expect huge composition differences between specific biological groups of individuals.
> > For example, if you study human blood samples in order to distinguish specificities between women and men, even if most of the metabolites
> > present may be expected to be find in both groups (at various levels but always present), some may be specific to one such as hormones.
> > Thus, you may consider putting men and women samples in distinct classes.
> >
> > In the Sacurine study, no specific a priori groups were defined. Thus, no specific assumption was taken into consideration regarding biological
> > classes. Thus, in this tutorial we considered only three classes: one for samples, one for pools and one for blanks.
> > Since pools are a mix of all biological samples,
> > we may want to consider an ion only if it is found in every pools. Thus, we may consider setting the 'Minimum fraction of samples' parameter to 1.
> > Considering that it sometimes happens that one sample can be a little out of the box for various reasons (pools do not make exception),
> > we may consider to lower a little this threshold, for example using a proportion of 0.9.
> >
> > Please note that in our example, since the dataset contains only 3 pools, this would make no difference to 1.0 concerning the 'pools' class.
> {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the datasets number 58 to 61.
> {: .tip}
>
{: .hands_on}


This grouping step is very important because it defines the data matrix which will be used especially for the statistical analyses.
User has to check the effect of parameter values on the result.

In order to check the result of the grouping function, a pdf file is created. It provides one plot per m/z slice found in the data. Each picture
represents the peak density across samples, plotting the corresponding Gaussian model which width is defined by the bandwidth parameter. Each dot
corresponds to a sample, with colours corresponding to defined classes if any. The plot allows to assess the quality of alignment.
The grey areas' width is associated with the bandwidth parameter.

Here is an example of two m/z slices obtained from the hands-on:

![Picture with two plots extracted from the plotChromPeakDensity.pdf output, one representing the 283.1127-283.1163 m/z slice and the other the 284.1198-284.1253 m/z slice](../../images/lcmspreproc_group12samp.png "Extract from plotChromPeakDensity.pdf")


> <question-title></question-title>
>
> 1. Look at the `283.1127 - 283.1163` m/z slice. How many peak groups are considered? Can you explain why some peaks are not affected to peak groups?
> 2. Look at the `284.1198 - 284.1253` m/z slice. What do you think could have happened if you had used a smaller bandwidth value?
>
> > <solution-title></solution-title>
> >
> > 1. There are 3 peak groups in this m/z slice. The two peaks that are not assigned to peak groups are alone in their retention time area. Thus,
> >    the number of samples under the corresponding density peaks does not reach the minimum fraction of samples set by the user (0.9) to consider a peak group,
> > in any specified class.
> > 2. If the bandwidth value had been set to a smaller value, the density peak width would have been smaller. With a small-enough bandwidth value,
> >    there could have been two density peaks instead of one under the current first density peak. Thus, the sample in line 5 would have been out of the
> >    previous peak group, thus not assigned to any peak group due to the 0.9 minimum fraction limit.
> >
> {: .solution}
>
{: .question}

When performing a grouping step, you construct a peak table. Even if this step may not be the final extraction step, you can take this opportunity
to check how your peak table looks at this point of the XCMS extraction. For this, you need to set the 'Get the Peak List' option to `Yes`
as done in this tutorial hands-on. This option generates two additional tables:
- a data matrix (xset.merged.group.dataMatrix.tsv) with intensities for each ion and each sample;
- a variable metadata file (xset.merged.group.variableMetadata.tsv) with information concerning the ions.

The variable metadata file contains various information that can be of interest:
- information about the m/z ratio of each ion (`mz`, `mzmin`, `mzmax` columns);
- information about the retention time of each ion (`rt`, `rtmin`, `rtmax` columns);
- the total number of peaks that were found for each ion (`npeaks` column);
- the number of samples in which each ion has been found, for each class (one column per class).

The data matrix contains the intensities for each ion and each sample. When no peak was found in a sample for a specific ion, value is given as NA
(you can choose to get a '0' value instead, using the 'If NA values remain, replace them by 0 in the dataMatrix' option). You can get a summary of NA proportions
using the **Intensity Check** {% icon tool %} module.

> <tip-title>Optional: Getting an overview of the proportion of NA in your data</tip-title>
>
> When you choose to export the peak list while using the **xcms groupChromPeaks (group)** {% icon tool %} module, you can have a first level of
> information regarding the proportion of NA by looking at the columns of classes you specified (or the `.` column when no class were given) in
> the variable metadata file ('variableMetadata').
> However, since the number of ions can be of hundreds or thousands, it can be difficult to evaluate the overall proportion in the dataset.
> Thus, one way to go is to use the variableMetadata to check specific ions (ones you may have chosen *a priori* and/or ones you spotted due to
> outstanding behaviour), and to use the **Intensity Check** {% icon tool %} module to get an overview of the whole dataset.
>
> > <hands-on-title>Intensity Check</hands-on-title>
> >
> > {% tool [Intensity Check](toolshed.g2.bx.psu.edu/repos/melpetera/intensity_checks/intens_check/1.2.8) %} with the following parameters:
> >   - *"Data matrix file"*: `xset.merged.group.dataMatrix.tsv`
> >   - *"Sample metadata file"*: `sampleMetadata_completed.tsv`
> >   - *"Variable metadata file"*: `xset.merged.group.variableMetadata.tsv`
> >   - *"Computation method"*: `For each class of samples`
> >     - *"Class column"*: `c2: class`
> >     - *"Statistics"*: tick `Missing values`
> >     - *"Calculate the mean fold change"*: tick `No`
> >
> > > <tip-title>Comment to W4M users</tip-title>
> > >
> > > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > > the datasets number 62 and 63.
> > {: .tip}
> >
> {: .hands_on}
>
> In the pdf file you will get as output, you can find a plot displaying proportions of NA for each class you specified.
> With the example provided in this tutorial, you can notice that the blk class containing blanks has a high number of NA with 4394 ions
> that have not been found, in any blank.
>
> ![Picture representing the proportions of NA in the 3 groups of the class column (blk, QC and sample). It is displayed as number of ions having given proportions of NA - proportions being grouped from 0 to 100 percents by classes of 20 percents.](../../images/lcmspreproc_ICgp1.png "Content of IC_Graphs.pdf")
>
> > <question-title></question-title>
> >
> > How many ions in the table have been found in all the pools?
> >
> > > <solution-title></solution-title>
> > >
> > > Knowing that there are 3 pools in the QC class, we can determine there are 4746 ions that have been found in all pools.
> > >
> > {: .solution}
> >
> {: .question}
>
{: .tip}

When looking back at the plots from plotChromPeakDensity.pdf, we can notice that in some cases there seems to be a small drift of retention time for some samples.
This phenomenon is well known with LC-MS techniques. To be able to attribute correct groups for peaks, it may be needed to perform some retention time
correction accross samples. Thus, the idea is (when needed) to apply a retention time strategy on the output of your grouping step, then to perform
a second grouping step on the corrected data.

# Optional XCMS step: *retention time correction*

Sometimes with LC-MS techniques, a deviation in retention time occurs from a sample to another. In particular, this is likely to be observed when you
inject large sequences of samples.

This optional step aims at correcting retention time drift for each peak among samples. The XCMS package provides two algorithms to do so.
[Check the next 4 slides](../../tutorials/lcms-preprocessing/slides.html#adjustrtime),
you will find additional material to help you understand the retention time correction algorithms.
In this training material we will focus on the "PeakGroups" method.

This correction is based on what is called *well behaved peaks*.
One characteristic of these peaks is that they are found in all samples or at least in most of the samples.
Sometimes it is difficult to find enough peaks present in all samples. The user can define a percentage of the total number of samples in which
a peak should be found to be considered a well behaved peak. This parameter is called *minimum required fraction of samples*.

On the contrary, you may have peak groups with more detected peaks than the total number of samples. Those peaks are called *additional peaks*.
If you do not want to consider peak groups with too much additional peaks as 'well behaved peaks', you can use the 'maximal number of additional
peaks' parameter to put them aside.

The algorithm uses statistical smoothing methods. You can choose between linear or loess regression.


> <hands-on-title>xcms adjustRtime (retcor)</hands-on-title>
>
> {% tool [xcms adjustRtime (retcor)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy0) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.RData`
>    - *"Method to use for retention time correction"*: `PeakGroups - retention time correction based on aligment of features (peak groups) present in most/all samples.`
>        - *"Minimum required fraction of samples in which peaks for the peak group were identified"*: `0.7`
>
> You can leave the other parameters to default values.
>
>    > <comment-title></comment-title>
>    >
>    > If you have a very large number of samples (*e.g.* a thousand), it might be impossible to find peaks that are present in 100% of your samples.
>    > If that is the case and you still set a very high value for the minimum required fraction of samples, the tool can not complete successfully the retention
>    > time correction.
>    > A special attention should also be given to this parameter when you expect a large number of peaks not to be present in part of your samples
>    > (*e.g.* when dealing with some blank samples).
>    {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the datasets number 64 and 65.
> {: .tip}
>
{: .hands_on}


This tool generates a plot output that you can use to visualise how retention time correction was applied across the samples and along the chromatogram.
It also allows you to check whether the well behaved peaks were distributed homogeneously along the chromatogram.

> <tip-title>Check the impact of RT correction using 'xcms plot chromatogram'</tip-title>
>
> Apart from the plots generated by the adjustRtime tool, you can check the impact of the retention time
> correction by comparing the chromatogram you obtained previously to a new one generated after correction.
>
> > <hands-on-title>xcms plot chromatogram</hands-on-title>
> >
> > {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy0) %} with the following parameters:
> >    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
> >
> >    > <comment-title></comment-title>
> >    >
> >    > As in the previous 'plot chromatogram', you can use your completed sampleMetadata file to get colours.
> >    {: .comment}
> >
> > > <tip-title>Comment to W4M users</tip-title>
> > >
> > > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > > the datasets number 66 and 67.
> > {: .tip}
> >
> {: .hands_on}
>
>
{: .tip}


The retention time correction step is not mandatory. However, when it is used retention time values are modified.
Consequently, applying this step on your data requires to complete it with an additional 'grouping' step using the
**xcms groupChromPeaks (group)** {% icon tool %} tool again.

Parameters for this second group step are expected to be similar to the first group step. Nonetheless,
since retention times are supposed to be less variable inside a same peak group now, in some cases it can be relevant to
lower a little the bandwidth parameter.

[Check the next slide](../../tutorials/lcms-preprocessing/slides.html#beforeafterrt)
to get an illustration of grouping before/after retention time correction.

> <hands-on-title>second 'xcms groupChromPeaks (group)'</hands-on-title>
>
> {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData`
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `5.0`
>        - *"Minimum fraction of samples"*: `0.9`
>        - *"Width of overlapping m/z slices"*: `0.01`
>    - *"Get the Peak List"*: `Yes`
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>        - *"If NA values remain, replace them by 0 in the dataMatrix"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > When performing this second grouping, similarly to the first grouping you can take this opportunity to check how your peak table
>    > looks at this point of the XCMS extraction, setting the 'Get the Peak List' option to `Yes`. As previously explained, you can
>    > look at your variableMetadata file as well as perform an NA diagnostic using the **Intensity Check** {% icon tool %} module.
>    {: .comment}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSpreprocessingXCMS](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmspreprocessingxcms), this step corresponds to
> > the datasets from number 68 to number 71.
> {: .tip}
>
{: .hands_on}

It is possible to use the retention time correction and grouping step in an iterative way if needed. Once you perform your
last adjustRtime step and thus your last grouping step, you will obtain your final peak list (*i.e.* final list of ions).

> <question-title></question-title>
>
> 1. How many ions did you obtained with the final grouping step?
> 2. Open the dataMatrix file you obtained with the final grouping. This table corresponds to intensities for each ion and each
>    sample. What do you notice when looking at the intensity of the fourth ion regarding the first sample?
>
> > <solution-title></solution-title>
> >
> > 1. The final grouping step led to 5100 ions.
> > 2. The fourth ion (M74T317) has a 'NA' value for the first sample (QC1_014). This is also the case for several other ions
> >    and samples.
> >
> {: .solution}
>
{: .question}

At this point of the XCMS extraction workflow, the peak list may contain NA when peaks were not considered peaks in only some
of the samples in the first 'findChromPeaks' step. This does not necessary means that no peak exists for these samples. For example,
sometimes peaks are of very low intensity for some samples and were not kept as peaks because of that in the first 'findChromPeaks'
step.

To be able to get the information that may actually exist behind NAs, there is an additional XCMS step that is called *fillChromPeaks*.

> <comment-title></comment-title>
>
> Before performing the 'fillChromPeaks' step, it is highly recommended to first have a look at your data concerning the distribution
> of NAs in your data. Indeed, this will allow you to check whether your results are consistent with your expectations; if not you
> may want to go back to some of your parameter choices in previous XCMS steps.
> To perform your NA diagnosis, you can use the variableMetadata file and dataMatrix file that you obtained with the last grouping step
> with the 'Get the Peak List' option to `Yes`. The variableMetadata file contains information about your ions: you will find information
> about the number of peaks detected for each ion. The dataMatrix files contains the intensities


topic: metabolomics
tutorial: msi-analyte-distribution
title: Mass spectrometry imaging: Examining the spatial distribution of analytes
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 203d83071aaf6af63fb2baa7be2268ee
last_modified: 2026-02-26T21:12:17.180592
zenodo_link: https://zenodo.org/record/484496
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/msi-analyte-distribution/tutorial.html

content:
Mass spectrometry imaging (MSI) is applied to measure the spatial distribution of hundreds of biomolecules in a sample. A mass spectrometer scans over the entire sample and collects a mass spectrum every 5-200 µm. This results in thousands of spots (or pixels) for each of which a mass spectrum is acquired. Each mass spectrum consists of hundreds of analytes that are measured by their mass-to-charge (m/z) ratio. For each analyte the peak intensity in the mass spectra of every pixel is known and can be set together to map the spatial distribution of the analyte in the sample.

The technique has a broad range of applications as it is able to measure many different kinds of analytes such as peptides, proteins, metabolites or chemical compounds in a large variety of samples such as cells, tissues and liquid biopsies. Application areas include pharmacokinetic studies, biomarker discovery, molecular pathology, forensic studies, plant research and material sciences. The strength of MSI is the simultaneous analysis of hundreds of analytes in an unbiased, untargeted, label-free, fast and affordable measurement while maintaining morphological information.

Depending on the analyte of interest and the application, different mass spectrometers are used. A mass spectrometer measures the analytes by ionizing, evaporating and sorting them by their mass-to-charge (m/z) ratio. Put simply, a mass spectrometer consists basically of three parts: an ionization source, a mass analyzer and a detector. The most common ionization sources for MSI are MALDI (Matrix Assisted Laser Desorption/Ionization), DESI (Desorption Electrospray Ionization) and SIMS (Secondary Ion Mass Spectrometry).

![MSI measurement](../../images/msi_distribution_maldi_tof_imaging.png "MALDI TOF imaging of a mouse kidney")

One common type of mass spectrometer for MSI is a MALDI Time-Of-Flight (MALDI-TOF) device. During MALDI ionization, a laser is fired onto the sample, which has been covered with a special matrix that absorbs the laser energy and transfers it to the analytes. This process vaporizes and ionizes the analytes. As they are now charged, they can be accelerated in an electric field towards the TOF tube. The time of flight through the tube to the detector is measured, which allows calculation of the mass over charge (m/z) of the analyte, as both mass and charge are correlated with time of flight. During measurement, complete mass spectra with hundreds of m/z - intensity pairs are acquired in thousands of sample plots, leading to large and complex datasets. Each mass spectrum is annotated with coordinates (x,y) that define its location in the sample. This allows visualization of the intensity distribution of each m/z feature in the sample as a heatmap.

Depending on the analyte of interest, the sample type and the mass spectrometer, the sample preparation steps as well as the properties of the acquired data vary. Apart from these differences, the preparation and measurement of the sample is normally straightforward, while the analysis of the large and complex dataset is the main bottleneck of a MSI experiment.

In this tutorial we will determine analytes with a localized distribution in a chilli cross section. The data analysis steps can be transferred to any other application in which the morphological distribution of an analyte in a sample is analyzed. A common application is studying the distribution of a drug and its metabolites in an animal model to better understand pharmacokinetic properties of the drug.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Chilli dataset

In this tutorial we will use the chilli dataset generated in the lab of Robert Winkler at the CINVESTAV institute in Irapuato (Mexico) and deposited at [Zenodo](https://zenodo.org/record/484496). The data was acquired to demonstrate their self-made low-cost 'plug and play' MSI system as well as their open source R scripts and to show they allow the direct spatially resolved detection of volatile organic compounds (VOCs) from plant tissue ({% cite Maldonado_Torres_2014 %} and {% cite Gamboa_Becerra_2015 %}).

The mass spectrometer consisted of a low-temperature plasma (LTP) ionization source coupled to a quadrupole mass analyzer. This ionization source allows measurement of semi-volatile organic compunds directly from fresh material under ambient conditions without destruction of the sample. To get an idea how this set up and the chilli looks, have a look at their [video](https://www.youtube.com/watch?v=NbZ1QqTZvXM) and [website](http://lababi.bioprocess.org/). The chilli (*Capsicum annuum*) was purchased from a local market in Irapuato, Mexico and longitudinally cut into a slice of 80 x 35 mm and 4 mm thickness and laid onto a glass slide. Mass spectra were acquired with a spatial resolution of 1 mm and in a mass range from 15 – 2000 m/z to measure semi-volatile and volatile organic compounds (VOCs).

We will check the properties of the dataset and then perform a similar analysis to that described by {% cite Gamboa_Becerra_2015 %}: plotting an average mass spectrum, filtering the dataset for the relevant m/z range, automatical generation of m/z images to find analytes that correspond to morphological features and generating an image that overlays three different analytes.

![Chilli](../../images/msi_distribution_chilli.png "Cross section of a chilli in a similar orientation as in the publication by Gamboa-Becerra")

## Get data

The imzML composite file consists of two files. The first file contains the metadata in an XML file and has the extension .imzML. The second file contains the mass spectra data, is saved as a binary file and has the extension .ibd. To be valid, both files must have the same filename before the extension. More information about the imzML file structure can be found here: [ms-imaging.org](https://ms-imaging.org/wp/imzml/data-structure/). Galaxy provides a composite upload option for files consisting of several components.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. **Create a new history** and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Upload the data from [Zenodo](https://zenodo.org/record/484496) via the **composite** option
>
>    > <tip-title>Upload via the composite option</tip-title>
>    > - Open the Galaxy Upload Manager({% icon galaxy-upload %} on the top-right of the tool panel)
>    > - Click on **Composite** on the top
>    > - Set **Composite Type** to `imzml`
>    > - Expand the first **Select** button (for the imzML metadata component)
>    > - Select **Paste/Fetch data** and paste
>    >    ```
>    > https://zenodo.org/record/484496/files/ltpmsi-chilli.imzML
>    >    ```
>    > - Expand the second **Select** button (for the mass spectral data component)
>    > - Select **Paste/Fetch data** and paste
>    >    ```
>    > https://zenodo.org/record/484496/files/ltpmsi-chilli.ibd
>    >    ```
>    > - Press **Start**
>    > - **Close** the window
>    {: .tip}
>
>    ![Upload imzML files](../../images/msi_distribution_upload_imzml.png "Upload an imzML file via the composite upload")
>
>    > <tip-title>FTP upload for large files</tip-title>
>    > * In case one subfile is larger than 2 GB the uploading needs to be done via ftp.
>    > * The necessary steps are explained in this tutorial [Getting data into Galaxy]({{ site.baseurl }}/topics/galaxy-interface/tutorials/get-data/slides.html)
>    {: .tip}
>
>    > <tip-title>Uploading an Analyze7.5 file</tip-title>
>    > * `Analyze7.5` files are also supported by Galaxy.
>    > * The file consists of three components and is therefore uploaded via the 'composite' function, analogously to the imzML upload.
>    > * The files to select in the `composite` tab are the header file `.hdr`, the m/z values file `.t2m` and the spectra file `.img`.
>    {: .tip}
>
> 3. Rename the data into `ltpmsi-chilli`
>
>    > <tip-title>Rename a dataset</tip-title>
>    > - Click on the {% icon galaxy-pencil %} **pencil icon** for the dataset to edit its attributes
>    > - In the central panel, change the **Name** field to `ltpmsi-chilli`
>    > - Click the **Save** button
>    {: .tip}
{: .hands_on}


# Exploring and filtering the chilli dataset

## Quality control of the data

Before starting any analysis it is important to check the characteristics and quality of the MSI data. The MSI quality control tool creates a comprehensive PDF report that contains multiple descriptive visualizations of different data attributes. Here we will use the MSI quality control to get an idea about the m/z range and the shape of the chilli section. More details about the MSI quality control tool can be found in the tutorial [Mass spectrometry imaging 1: Loading and exploring MSI data]({{site.baseurl}}/topics/proteomics/tutorials/mass-spectrometry-imaging-loading-exploring-data/tutorial.html).

> <hands-on-title>Quality control </hands-on-title>
>1. {% tool [MSI_Qualitycontrol](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_quality_report/cardinal_quality_report/2.6.0.1) %}
>    - {% icon param-file %} *"MSI data"*: `ltpmsi-chilli.imzML` (Input dataset)
>    - *"Centroided input"*: `yes`
>    - *"Processed imzML file"*: `yes`
>        - *"Mass accuracy to which the m/z values will be binned"*: `0.2`
>        - *"Unit of the mass accuracy"*: `mz`
>    - *"PCA with 2 components"*: `Yes`
>    - Press **Run Tool**
>
>    > <comment-title>Properties of the imzML file</comment-title>
>    > To set the parameters for the mass spectrometry imaging tools correctly the following three parameters should be known about the dataset: 1) is the imzML file type processed or continuous, 2) are the spectra in profile or centroided mode and 3) the accuracy of the mass spectrometer. For the chilli dataset the publication states that it is a processed imzML type in centroided mode. This information can also be extracted by opening the local imzML component of the file in a text editor or web browser. Line 10 and 11 state:
>    >
>    >      <cvParam cvRef="MS" accession="MS:1000128" name="profile spectrum" value=""/>
>    >      <cvParam cvRef="IMS" accession="IMS:1000031" name="processed" value=""/>
>    >
>    > Starting from line 105 the "run" and "spectrumList" elements describe the acquired spectra in more detail:
>    >
>    >      <run defaultInstrumentConfigurationRef="IC1" id="_x0031_30704_IMGCHJ2">
>    >      <spectrumList count="4166" defaultDataProcessingRef="pwiz_Reader_conversion">
>    >        <spectrum defaultArrayLength="11127" id="controllerType=0 controllerNumber=1 scan=1" index="0" dataProcessingRef="pwiz_Reader_conversion">
>    >          <cvParam cvRef="MS" accession="MS:1000511" name="ms level" value="1"/>
>    >          <cvParam cvRef="MS" accession="MS:1000579" name="MS1 spectrum" value=""/>
>    >          <cvParam cvRef="MS" accession="MS:1000127" name="centroid spectrum" value=""/>
>    >          <cvParam cvRef="MS" accession="MS:1000528" name="lowest observed m/z" value="15"/>
>    >          <cvParam cvRef="MS" accession="MS:1000527" name="highest observed m/z" value="2000"/>
>    >          <cvParam cvRef="MS" accession="MS:1000504" name="base peak m/z" value="80.0377"/>
>    >          <cvParam cvRef="MS" accession="MS:1000505" name="base peak intensity" value="3.72813e+006"/>
>    >          <cvParam cvRef="MS" accession="MS:1000285" name="total ion current" value="2.42458e+008"/>
>    >
>    > This means that the file consists of 4166 spectra (count="4166") and that the imzML type is 'processed' as the first spectrum (scan="1") has 11127 m/z values (defaultArrayLength="11127) while the following spectra (not shown here) have different numbers of m/z values. In processed imzML files each spectrum has an individual m/z axis, in contrast to the continuous imzML type, where all spectra have the same m/z axis. In many software tools processed imzML files are not or only partly supported. The MSI tools in Galaxy are based on [Cardinal](http://cardinalmsi.org/) and therefore support imzML files, but require a binning of the m/z values while reading the file. The bin size should be chosen according to the m/z accuracy of the mass spectrometer. For the chilli dataset we therefore use 0.1 m/z, as this results in m/z bins of 0.2 m/z which corresponds to the m/z step size that was used in the publication.
>    >
>    > Regarding the spectra, line 10 states that the spectra are in profile mode while the 'run' section defines spectrum 1 as a centroided spectrum. As the 'run' section is directly linked to the spectra in the ibd file, this is the information to trust. Furthermore, the 'dataProcessingList' element gives a hint that peak picking was performed, which is confirmed in the publication.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. In which m/z range were the data acquired?
> 2. How many spectra (pixels) were measured?
> 3. In which images can you guess the shape of the chilli?
>
> > <solution-title></solution-title>
> >
> > 1. The measured m/z range was m/z 15 - 2000.
> > 2. 4166 spectra were measured.
> > 3. The rough shape of the chilli section is visible in the "Number of peaks per spectrum"; "Total Ion Current" and "PCA result" images.
> >
> {: .solution}
>
{: .question}

Open the quality report with the eye button and check the summary table on the first page to answer question 1 and 2. The shape of the chilli can roughly be seen in the images showing the number of peaks per spectrum and the total ion current, with higher values in the middle part of the fruit that roughly corresponds to the placenta. In the PCA image the complete chilli, except for the lowest part of the fruit, is visible in light colours compared to the dark background.

To further investigate mass spectra of different chilli compartments we select one pixel for each tissue area (pericarp, placenta and seeds) from the total ion current image. Knowing the exact shape of the chilli tissue from Figure 4 of the publication helps to find the corresponding areas. This are the pixels we have chosen for the tutorial:

seeds: x=39 y=53, placenta: x=50 y=44, pericarp: x=25 y=60

The relevant m/z range for VOCs was not known before the measurement and therefore chosen quite liberally. The average mass spectra plots give a hint about the relevant m/z range of the sample with most peaks below 750 m/z and no distinct peaks above 750 m/z.

![quality report](../../images/msi_distribution_qc.png "Example plots from the quality report")


## Plotting average mass spectra

We will follow up on the average mass spectra plots from the quality control report as well as on the differences between mass spectra from different chilli compartments. First, we generate more zoomed in mass spectra plots to get an idea about the m/z range that is relevant for VOCs. Next, we will plot and compare mass spectra that derive from different regions of the chilli.

> <hands-on-title>Average mass spectra</hands-on-title>
>
> 1. {% tool [MSI plot spectra](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_spectra_plots/cardinal_spectra_plots/2.6.0.3) %}
>    - {% icon param-file %} *"MSI data"*: `ltpmsi-chilli.imzML` (Input dataset)
>    - *"Centroided input"*: `yes`
>    - *"Processed imzML file"*: `yes`
>        - *"Mass accuracy to which the m/z values will be binned"*: `0.2`
>        - *"Unit of the mass accuracy"*: `mz`
>    - *"Choose spectra"*: `Plot mean spectra based on all spectra`
>        - In *"zoomed in m/z range"*: `Manual input`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"1: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `10`
>                - *"upper m/z boundary for plotting window"*: `200`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"2: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `200`
>                - *"upper m/z boundary for plotting window"*: `500`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"3: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `500`
>                - *"upper m/z boundary for plotting window"*: `1000`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"4: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `1000`
>                - *"upper m/z boundary for plotting window"*: `1500`
>            - Click on *"Define min m/z and max m/z for the plotting window"*:
>            - In *"5: Define min m/z and max m/z for the plotting window"*:
>                - *"lower m/z boundary for plotting window"*: `1500`
>                - *"upper m/z boundary for plotting window"*: `2000`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the approximate m/z of the heaviest peak visible in the average spectra plot of the complete m/z range?
> 2. What is the approximate m/z of the peak with the highest average intensity?
>
> > <solution-title></solution-title>
> >
> > 1. There is a tiny peak at m/z ~760.
> > 2. The highest peak is at ~80 m/z, which can best be seen in the mass spectrum between 15 and 200 m/z.
> >
> {: .solution}
>
{: .question}

In the average spectrum of the complete m/z range there are no visible peaks above 800 m/z. Most of the high intensity peaks are below 300 Th, which corresponds to the typical masses of VOCs. Even in the zoomed mass spectra between 1500 and 2000 m/z only a few distinct peaks can be identified in the noise. In accordance with {% cite Gamboa_Becerra_2015 %}, we will restrict further analysis to the m/z range between 15 and 1000.

![Average plots](../../images/msi_distribution_average_spectra.png "Complete and zoomed in average mass spectra")

We will rerun the MSI plot spectra tool and add annotation to the pixels that correspond to different chilli compartments.

> <hands-on-title>Mass spectra of different chilli compartments</hands-on-title>
>
> 1. Create a tabular file with the coordinates of interest and rename it to 'annotations':
>
>    ```
>    x     y     compartment
>    39    53    seeds
>    50    44    placenta
>    25    60    pericarp
>    ```
>
>    > <tip-title>Creating a new file</tip-title>
>    >
>    > * Open the Galaxy Upload Manager
>    > * Select **Paste/Fetch Data**
>    > * Paste the file contents into the text field
>    > * From the settings menu {% icon galaxy-gear %} select **Convert spaces to tabs**
>    > * Press **Start** and **Close** the window
>    {: .tip}
>
> 2. {% tool [MSI plot spectra](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_spectra_plots/cardinal_spectra_plots/2.6.0.3) %}
>    - Press the rerun button, changing only the following:
>    - *"Choose spectra"*: `Plot single spectra (separate or overlaid)`
>        - {% icon param-file %} *"Load tabular file with pixel coordinates"*: `annotations` (Input dataset)
>        - *"Column with x values"*: `column:1`
>        - *"Column with y values"*: `column:2`
>        - *"Tabular file contains a header line"*: `Yes`
>        - *"Separate plot per spectrum or overlaid plot with average spectra per annotation group"*: `Overlaid spectra plots`
>        - *"Select column with annotations"*: `column:3`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which compartment does the highest intensity peak belong to?
> 2. Which compartment does the third-highest intensity peak belong to?
>
> > <solution-title></solution-title>
> >
> > 1. The highest intensity peak at around 80 m/z belongs to the spectrum that is located in the pericarp of the chilli.
> > 2. The peak at around 60 m/z belongs to the spectrum that is located in the seeds.
> {: .solution}
>
{: .question}

![Annotated plots](../../images/msi_distribution_overlay_plot.png "Mass spectra for single pixels of different compartments")

The single spectra that derive from different chilli compartments show some major differences. You should keep in mind that these are single spectra and therefore the intensities can be different to the average intensity of the compartment. Looking at the m/z range between 1000 and 2000, we realize that there are a number of distinct peaks which were hidden in the noise of the average spectrum, which contained all background spectra as well as the spectra of the chilli. In accordance with the original publication, and to speed up computation time, we focus on the low m/z range, but for a deeper analysis it could also be worth analyzing the higher m/z range.

## Filtering for the relevant m/z range

> <hands-on-title>filtering for a m/z range</hands-on-title>
>
> 1. {% tool [MSI filtering](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_filtering/cardinal_filtering/2.6.0.0) %}
>    - {% icon param-file %} *"MSI data"*: `ltpmsi-chilli.imzML` (Input dataset)
>    - *"Centroided input"*: `yes`
>    - *"Processed imzML file"*: `yes`
>        - *"Mass accuracy to which the m/z values will be binned"*: `0.2`
>        - *"Unit of the mass accuracy"*: `mz`
>    - *"Select m/z feature filtering option"*: `m/z range (manually)`
>        - *"Minimum value for m/z"*: `15.0`
>        - *"Maximum value for m/z"*: `1000.0`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many m/z features are left in the output dataset?
> 2. How many m/z features were removed?
>
> > <solution-title></solution-title>
> >
> > 1. After filtering 4926 m/z features are left.
> > 2. The dataset originally had 9926 m/z features; after filtering 4926 are left, which means that 5000 were removed.
> >
> {: .solution}
>
{: .question}

The MSI tools are only able to write outputs as continuous imzML format; therefore, from now on the dataset is in continuous imzML format and "processed imzML file" can be set to 'no' in the following tools.

# Multiple and overlayed analyte images

## Automatic generation of analyte images

The main question for the chilli dataset is which m/z features have a localized distribution in the fruit. This question can be addressed by automatically generating distribution images for all analytes (m/z features) and then visually identifying which features are localized in specific compartments of the fruit. {% cite Gamboa_Becerra_2015 %} scan the m/z range with a step size of 0.2 m/z and a tolerance of 0.4 Th, which resulted in more than 2000 images which they visually explored for localized features. In this training we will only generate images for the two mass ranges where we have already seen high intensity peaks in the spectra plots: around 60 and 80 m/z.

This requires three steps. Firstly, all m/z features of the dataset with the MSI data exporter are extracted. Secondly, all m/z features are filtered in the m/z ranges that we are interested in: between 55 and 65 Th, as well as between 75 and 85 m/z (this step can be skipped to obtain distribution images for all features). Thirdly, the MSI m/z image tool is used to automatically generate distribution images for each of the m/z from the second step. As the data is already binned to 0.2 Th, only the m/z tolerance of 0.4 m/z has to be specified. The large tolerance was chosen to obtain images from overlapping m/z ranges to make the signal intensity more robust against small m/z inaccuracies in individual mass spectra.

> <hands-on-title>Generation of multiple analyte images</hands-on-title>
> 1. {% tool [MSI data exporter](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_data_exporter/cardinal_data_exporter/2.6.0.0) %}
>    - {% icon param-file %} *"MSI data"*: `filtered imzML` (output of **MSI filtering** {% icon tool %})
>    - *"Centroided input"*: `yes`
>    - *"Multiple output files can be selected"*: `mz feature output`
>
> 2. {% tool [Filter data on any column using simple expressions](Filter1) %}
>    - {% icon param-file %} *"Filter"*: `MSI data exporter on data 6: features` (output of **MSI data exporter** {% icon tool %})
>    - *"With following condition"*: `c2>=55 and c2<=65 or c2>=75 and c2<=85`
>    - *"Number of header lines to skip"*: `1`
>
> 3. {% tool [MSI mz images](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_mz_images/cardinal_mz_images/2.6.0.2) %}
>    - {% icon param-file %} *"MSI data"*: `filtered imzML` (output of **MSI filtering** {% icon tool %})
>    - *"Centroided input"*: `yes`
>    - {% icon param-file %} *"m/z of interest (e.g. internal Calibrants)"*: `Filter on data` (output of **Filter** {% icon tool %})
>    - *"Column with m/z values"*: `column:2`
>    - *"Column with name of m/z values"*: `column:1`
>    - *"Tabular file contains a header line"*: `Yes`
>    - *"M/z range"*: `0.4`
>    - *"Contrast enhancement"*: `suppression`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which are the two analyte images that show the most localized distribution?
> 2. Which compartments are the two features localized in?
>
> > <solution-title></solution-title>
> >
> > 1. Defining the exact m/z with the best localization is quite tricky. Between 61.8 and 62.6 m/z there is a localized feature, as well as between 84 and 84.4 m/z. We can choose 62.2 and 84.2 to get the best images (in the publication it was 62.1 and 84.1).
> > 2. The comparison of the feature localization with the chilli image shows that the 62.2 m/z feature is located in the seeds and the 84.2 m/z feature more uniformly in the placenta and pericarp.
> >
> {: .solution}
>
{: .question}

![best images](../../images/msi_distribution_auto_images.png "Distribution images for the analytes with the most localized distribution")

In this tutorial we only evaluated 102 features, while the authors of this study evaluated more than 2000 images visually. This process is time-consuming and biased by the judgment of the scientist, but seems to still be used sometimes. Several attempts have been made to circumvent this process and obtain m/z features with localized distribution automatically, for example by {% cite Alexandrov_2013 %} and {% cite Inglese_2018 %}

After m/z features with a localized distribution have been found, it is interesting to identify them. {% cite Gamboa_Becerra_2015 %} performed additional gas and liquid chromatography mass spectrometry (GC-MS and LC-MS) which enabled the identification of several features with high certainty. Unfortunately the highly localized features at 62.2 and 84.2 m/z could not be identified, but the defined localization implies their biological importance. Several capsaicinoids could be identified, amongst them capsaicin (306.6 m/z).

> <comment-title>Identification of m/z features</comment-title>
> The identification of m/z features in MSI experiments is not always necessary, but can increase the confidence in the analysis and the molecular understanding. m/z features are either matched to databases or for more confidence another experiment is performed on the same sample to restrict the identification possibilities to analytes that are actually present in the sample. In this study other mass spectrometry techniques were applied, allowing identification of several m/z features, but at the same time features with defined localization could not be identified (e.g. 62.2 m/z and 84.2 m/z). These molecules may be fragments, metabolites or contaminants that are not available with the used mass spectrometry techniques.
> Targeted analysis can be performed if the m/z of the molecule of interest is already known (e.g. when studying the tissue distribution of a drug and its known metabolites). In this case distribution images for the molecules of interest are a good starting point and scanning through all m/z features might not be necessary.
{: .comment}

## Overlay image for three analyte features

To get an idea about the distribution of capsaicin in the chilli we will plot its distribution in an overlay image with the two unknown but localized features 62.2 m/z and 84.2 m/z.

> <hands-on-title>Overlay image for several m/z features</hands-on-title>
>
> 1. Create a tabular file with the m/z of interest and rename it to 'mz features':
>
>    ```
>    62.2
>    84.2
>    306.6
>    ```
>
>    > <tip-title>Creating a new file</tip-title>
>    >
>    > * Open the Galaxy Upload Manager
>    > * Select **Paste/Fetch Data**
>    > * Paste the file contents into the text field
>    > * Set Type to **tabular** instead of auto-detect
>    > * Press **Start** and **Close** the window
>    {: .tip}
>
> 2. {% tool [MSI mz images](toolshed.g2.bx.psu.edu/repos/galaxyp/cardinal_mz_images/cardinal_mz_images/2.6.0.2) %}
>    - {% icon param-file %} *"MSI data"*: `filtered imzML` (output of **MSI filtering** {% icon tool %})
>    - *"Centroided input"*: `yes`
>    - {% icon param-file %} *"m/z of interest (e.g. internal Calibrants)"*: `mz features` (Input dataset)
>    - *"Column with m/z values"*: `column:1`
>    - *"Column with name of m/z values"*: `column:1`
>    - *"Tabular file contains a header line"*: `No`
>    - *"M/z range"*: `0.4`
>    - *"Contrast enhancement"*: `suppression`
>    - *"Alpha Power"*: `1.5`
>    - *"Linear normalization of image"*: `Yes`
>    - *"Overlay image"*: `Yes`
>        - *"1: Colours for the plots"*: `blue`
>        - *"2: Colours for the plots"*: `green`
>        - *"3: Colours for the plots"*: `red`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which compartment is capsaicin located in?
>
> > <solution-title></solution-title>
> >
> > 1. Capsaicin is located in the middle of the placenta, as well as partly in the seeds.
> >
> {: .solution}
>
{: .question}

![overlay images](../../images/msi_distribution_overlay_images.png "Distribution image for capsaicin and overlay of the three analytes")

# Conclusion

This tutorial covered the steps to explore the properties of a dataset and visualize the morphological distribution of molecules.

![workflow](../../images/msi_distribution_workflow.png "Workflow that was used during this tutorial.")


topic: metabolomics
tutorial: msi-finding-nglycans
title: Mass spectrometry imaging: Finding differential analytes
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 294bf7be8c42e55114a300b986a195d8
last_modified: 2026-02-26T21:12:17.181701
zenodo_link: https://zenodo.org/record/2628280
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/msi-finding-nglycans/tutorial.html

content:
Mass spectrometry imaging (MSI) is applied to measure the spatial distribution of hundreds of biomolecules in a sample. A mass spectrometer scans over the entire sample and collects a mass spectrum every 5-200 µm. This results in thousands of spots (or pixels) for each of which a mass spectrum is acquired. Each mass spectrum consists of hundreds of analytes that are measured by their mass-to-charge (m/z) ratio. For each analyte the peak intensity in the mass spectra of every pixel is known and can be set together to map the spatial distribution of the analyte in the sample.

The technique has a broad range of applications as it is able to measure many different kinds of analytes such as peptides, proteins, metabolites or chemical compounds in a large variety of samples such as cells, tissues and liquid biopsies. Application areas include pharmacokinetic studies, biomarker discovery, molecular pathology, forensic studies, plant research and material sciences. The strength of MSI is the simultaneous analysis of hundreds of analytes in an unbiased, untargeted, label-free, fast and affordable measurement while maintaining morphological information.

Depending on the analyte of interest and the application, different mass spectrometers are used. A mass spectrometer measures the analytes by ionizing, evaporating and sorting them by their mass-to-charge (m/z) ratio. Put simply, a mass spectrometer consists basically of three parts: an ionization source, a mass analyzer and a detector. The most common ionization sources for MSI are MALDI (Matrix Assisted Laser Desorption/Ionization), DESI (Desorption Electrospray Ionization) and SIMS (Secondary Ion Mass Spectrometry).

![MSI measurement](../../images/msi_distribution_maldi_tof_imaging.png "MALDI TOF imaging of a mouse kidney")

One common type of mass spectrometer for MSI is a MALDI Time-Of-Flight (MALDI-TOF) device. During MALDI ionization, a laser is fired onto the sample, which has been covered with a special matrix that absorbs the laser energy and transfers it to the analytes. This process vaporizes and ionizes the analytes. As they are now charged, they can be accelerated in an electric field towards the TOF tube. The time of flight through the tube to the detector is measured, which allows calculation of the mass over charge (m/z) of the analyte, as both mass and charge are correlated with time of flight. During measurement, complete mass spectra with hundreds of m/z - intensity pairs are acquired in thousands of sample plots, leading to large and complex datasets. Each mass spectrum is annotated with coordinates (x,y) that define its location in the sample. This allows visualization of the intensity distribution of each m/z feature in the sample as a heatmap.

Depending on the analyte of interest, the sample type and the mass spectrometer, the sample preparation steps as well as the properties of the acquired data vary. Human tissues are often stained with hematoxylin and hemalaun (H&E) after mass spectrometry imaging. This allows better visualization of tissue histology and can be used to compare the MSI heatmaps with the real morphology. The stained image of the tissue can also be used to define regions of interest that can be extracted from the MSI data and can be subjected to supervised statistical analysis.

As MSI measures only unfractionated molecules (MS1), identification of the resulting m/z features requires matching them to a list of known m/z features. The easiest approach is to match the MSI m/z features to existing databases such as UniProt for proteins or lipidmaps for lipids. More accurate identification is obtained by performing additional mass spectrometry experiments (either on the same or on a similar sample, e.g. adjacent slice from a tissue) that also measure the fragmented molecules (MS2) and thereby also allow accurate identification of isobaric molecules. This tandem mass spectrometry approach can either be done in situ directly on the sample or by transferring the sample to a tube and bringing it into solution for liquid-chromatography tandem mass spectrometry (LC-MS/MS).

In this tutorial we will determine and identify N-linked glycans by differential analysis of a control (untreated) murine kidney section and a section treated with PNGase F. The data analysis steps can be transferred to any other application that requires finding differential analytes, such as for biomarker discovery.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# N-linked glycans in murine kidney dataset

In this tutorial we will use the [murine kidney N-glycan dataset](https://doi.org/10.1016/j.dib.2018.08.186) generated in the lab of Peter Hoffmann (Adelaide, Australia). N-linked glycans are carbohydrates (consisting of several sugar molecules) that are attached as a post-translational modification to the carboxamide side chain of an asparagine residue of a protein. Changes in N-linked glycans have been observed in several diseases. The Hoffmann lab acquired the presented dataset to show that their automated sample preparation method for N-glycan MALDI imaging successfully analyses N-glycans from formalin-fixed paraffin-embedded murine kidney tissue ([Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7)). The datasets generated in this study were made publicly available via the [PRIDE Archive](https://www.ebi.ac.uk/pride/archive/projects/PXD009808).

Three 6 µm sections of formalin-fixed paraffin-embedded murine kidney tissue were prepared for MALDI imaging. To release N-linked glycans from the proteins, PNGase F was printed onto two kidney sections. In the third kidney section, one area was printed with buffer to serve as a control, and another was covered with N-glycan calibrants [Figure 4a-c](https://media.springernature.com/original/springer-static/image/art%3A10.1007%2Fs00216-014-8293-7/MediaObjects/216_2014_8293_Fig4_HTML.gif). 2,5-DHB matrix was sprayed onto the tissue sections and MALDI imaging was performed with 100 µm intervals using a MALDI-TOF/TOF instrument.

To make computation times feasible for this training, we reduced all datasets to a m/z range of 1250 – 2310 and resampled the m/z values with a step size of 0.1. The main part of the training will be performed with the control and one PNGase F treated kidney file, which were both restricted to representative pixels (Figure 2). To test the results on the complete dataset we also provide a file in which all four (m/z reduced and resampled) files (control, calibrants, treated kidney 1 and treated kidney 2) are combined and TIC-normalized. All size-reduced training datasets are available via [Zenodo](https://zenodo.org/record/2628280). We will combine the cropped control and the cropped treated1 file and perform similar steps to those described by [Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7). These steps include combining files, preprocessing, unsupervised and supervised analysis, identification of N-glycans and generation of analyte distribution heatmaps.

![file_dimensions](../../images/msi-diff-abundance-file-dimensions.png "Top image: All four files after combining them. Black rectangles show the orientation of the cropped control and cropped treated1 files that are used in the major part of the training. Bottom: Cropped training files after combining them.")


## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://zenodo.org/record/2628280). Upload all three datasets (one by one) to Galaxy via the **composite** option and rename them to 'control', 'treated1' and 'all_files'.
>    ```
>    https://zenodo.org/record/2628280/files/conrol.imzml
>    https://zenodo.org/record/2628280/files/control.ibd
>    https://zenodo.org/record/2628280/files/treated1.imzml
>    https://zenodo.org/record/2628280/files/treated1.ibd
>    https://zenodo.org/record/2628280/files/all_files.imzml
>    https://zenodo.org/record/2628280/files/all_files.ibd
>    ```
>
> 3. Paste the link of the tabular file with the N-glycan identities into the regular upload and rename it to 'Glycan IDs':
>    ```
>    https://zenodo.org/record/2628280/files/Glycan_IDs.tabular
>    ```
>
> 4. Create a tabular file with the three glycan calibrants that were used in the study and rename it to 'calibrants':
>    ```
>    M+Na+    composition
>    1257.4225    Man5GlcNAc2
>    1542.5551    Man3GlcNAc5
>    2393.8457    Man3Gal4GlcNAc6
>    ```
>
> 5. Add a tag to each imzML file ('control', 'treated1' and 'all_files') and to the tabular file ('Glycans'):
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Preparing the files for statistical analysis

## Combining control and treated1 files

> <hands-on-title>Combining files</hands-on-title>
>
> 1. **MSI combine** {% icon tool %} with the following parameters:
>    - {% icon param-files %} *"MSI data"*: `control` (uploaded dataset) and `treated1` (uploaded dataset)
>    - *"Output format"*: `imzML`
>
>    > <comment-title>on the combine tool</comment-title>
>    >
>    > Two combining options are available: 'Automatic combination' combines the selected files into a grid with 5 pixel space between them. The order of the files in the history determines their orientation in the combined file and the file name is used to annotate the corresponding pixels. For custom orientation of the files the option 'shift xy coordinates with a tabular file' can be applied. For now, the **MSI combine** {% icon tool %} only allows combining files with the same m/z axis. Different m/z axes can be adjusted by filtering for the common m/z range and then binning with the same bin size (**MSI filtering** {% icon tool %} and **MSI preprocessing** {% icon tool %}.
>    {: .comment}
>
{: .hands_on}

## Exploring the combined file

To get an overview of the combined dataset we will use the **MSI Qualitycontrol** {% icon tool %}. We will use the annotation file that was generated by the **MSI combine** {% icon tool %} to inform the **MSI Qualitycontrol** {% icon tool %} which spectra belong to which subfile - this will lead to several extra figures where the data properties for both subfiles are plotted separately.

> <hands-on-title>Quality control with spectrum annotations</hands-on-title>
>
> 1. **MSI Qualitycontrol** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `combined imzML` (output of **MSI combine** {% icon tool %})
>    - *"Use pixel annotation from tabular file for QC plots"*: `use pixel annotation from a tabular file`
>        - {% icon param-file %} *"Tabular file with pixel coordinates and annotation"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with pixel annotations"*: `Column: 3`
>        - *"Tabular file contains a header line"*: `Yes`
>    - {% icon param-file %} *"m/z of interest (e.g. internal Calibrants)"*: `calibrants` (uploaded dataset)
>    - *"Column with m/z values"*: `column:1`
>    - *"Column with name of m/z values"*: `column:2`
>    - *"Tabular file contains a header line"*: `Yes`
>    - *"PCA with 2 components"*: `Yes`
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How many m/z values does the combined dataset contain?
> 2. How many spectra does the combined dataset contain?
> 3. How many input calibrants were valid?
> 4. Which of the two subfiles shows higher overall intensities?
>
> > <solution-title></solution-title>
> >
> > 1. 10581
> > 2. 1935
> > 3. Two calibrants are valid, which means that they lie in the (cropped) m/z range of our files.
> > 4. Several plots show that the treated subfile has higher intensities than the control subfile.
> >
> {: .solution}
>
{: .question}

The quality report gives a broad overview of the properties of both files. The pixel order seems to be random and does not correspond to the measurement in the MALDI. This means that the plots where the 'spectra index' is plotted on the x-axis do not represent the acquisition process. While the PNGase F treated kidney file shows higher intensities in several plots (e.g. median intensity per spectrum and intensity boxplots), the number of peaks is similar in both subfiles. The most abundant m/z feature plot of the treated kidney roughly corresponds to the morphology of the kidney while in the control file there are no visible kidney structures. The first principal componant in the PCA analysis roughly differentiates the spectra of the two subfiles which is a good sign for finding differences in the treated vs. control kidney files. For the following preprocessing steps, it is important to know the rough number of data points that one peak consists of. From page 27 on, the average mass spectra of the two glycan calibrants are shown. Due to our harsh resampling to 0.1 m/z, the calibrant 1257.4225 (Man5GlcNAc2) shows quite an angular peak shape. The complete peak consists of about 6 data points, which corresponds to a width of 0.5 m/z. On page 28 the average spectrum for each subfile is plotted, showing a nice peak in the treated subfile and only basal intensity in the control subfile. Page 29 and 30 show the average mass spectra for the 1542.5551 (Man3GlcNAc5) calibrant. The intensities of this calibrant are relatively low and only slightly higher in the treated than in the control subfile. Furthermore, the split up peak shape is not useful for defining the peak width properly.

## Preprocessing of the combined file

Preprocessing of the MSI raw spectra is crucial to reduce experimental (electrical, chemical and mechanical) noise and variance within and between the datasets before statistical analysis. Depending on the ionization technique, the mass analyzer, the sample and the statistical analysis, different preprocessing steps have to be applied. Here, we will perform similar preprocessing steps to those described by [Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7).

  1. **Intensity normalization**:
Signal intensities in MALDI datasets are affected by heterogeneous matrix spray or crystallization, salt and pH gradients and contamination of the ion source over time. Therefore intensity normalization is important to ensure spectra that were acquired at different positions of the sample are comparable. The most common method is the normalization to the total ion current (TIC). This means that each intensity value of a spectrum is divided by the sum of all intensities of the spectrum (equal to the total ion current, TIC). More information on normalization can be found in the publications by [Gibb and Strimmer](https://arxiv.org/pdf/1607.03180.pdf) and [Deininger et al.](https://link.springer.com/article/10.1007%2Fs00216-011-4929-z).

  2. **Spectra smoothing**:
Smoothing reduces noise in the spectra and thereby improves peak picking. Smoothing is an optional preprocessing step, especially for high resolution data. The critical parameter for smoothing is the window size. Larger window sizes lead to stronger smoothing, which not only reduces noise but also the intensities of real peaks. Therefore it is important to set the window size large enough to remove noise but small enough to preserve the peaks. We will use the 'gaussian' smoothing method that applies a weighted moving average filter and set the window size to 4 (resampling was done with 0.1, multiplied by 4 this gives a 0.4 m/z window which is slightly smaller than the complete peak width that we found to be roughly 0.5 m/z). The standard deviation is set to the default setting: window/4 = 1. As the smoothing changes the peak width, we will quickly examine the 1257.42 calibrant peak width afterwards and use the information to define the correct window sizes in further preprocessing steps.

  3. **Baseline removal**:
Especially for MALDI datasets it is common to perform a baseline subtraction to remove the background intensity that is generated by chemical noise. Baseline removal could also be performed before smoothing but as smoothing might slightly increase the baseline we perform the baseline removal afterwards. Similar to the smoothing step, the window size in which the noise is estimated is crucial. The window should be large enough to obtain a representative number of data points but small enough to avoid removing parts of the peaks as well. Here, the window is obtained by dividing the m/z range by the number of blocks. Recommended window sizes range from 1 - 5 m/z. We will divide the m/z axis into 500 blocks, giving a window size of about 2 m/z.


> <hands-on-title>Preprocessing part I</hands-on-title>
>
> 1. **MSI preprocessing** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `combined imzML` (output of **MSI combine** {% icon tool %})
>    - In *"Preprocessing"*:
>        - Click on *"Insert Preprocessing"*:
>        - In *"1: Preprocessing"*:
>            - *"Preprocessing methods"*: `Intensity Normalization (TIC)`
>       - Click on *"Insert Preprocessing"*:
>        - In *"2: Preprocessing"*:
>            - *"Preprocessing methods"*: `Peak smoothing`
>                - *"Smoothing method"*: `gaussian`
>                    - *"The standard deviation for the Gaussian kernel (window/sd)"*: `1`
>                - *"Window size"*: `4.0`
>        - In *"3: Preprocessing"*:
>            - *"Preprocessing methods"*: `Baseline Reduction`
>    - *"Output format"*: `imzML`
>
> 2. **MSI plot spectra** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `processed imzML` (output of **MSI preprocessing** {% icon tool %})
>    - *"Select if you want to plot the mass spectrum of a single pixel or the average spectrum of all pixels of a sample"*: `Average spectrum for each sample`
>        - *"Use pixel annotation from tabular file for spectra plots"*: `group pixels according to annotations`
>        - {% icon param-file %} *"Tabular file with pixel coordinates and annotation"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with pixel annotations"*: `Column: 3`
>        - *"Tabular file contains a header line"*: `Yes`
>        - In *"Zoomed in plots with m/z min and m/z max to define the plot window"*:
>            - Click on *"Insert Zoomed in plots with m/z min and m/z max to define the plot window"*:
>            - In *"1: Zoomed in plots with m/z min and m/z max to define the plot window"*:
>                - *"lower m/z boundary for plotting window"*: `1257`
>                - *"upper m/z boundary for plotting window"*: `1259`
>
{: .hands_on}

The QC report of the **MSI preprocessing** {% icon tool %} shows the changes during preprocessing for four random spectra. TIC normalization only changes the dimension of the y-axis. Smoothing decreases the peak height and thus the intensity values on the y-axis shrink as well. Baseline removal shifts the complete spectrum towards the bottom.

The **MSI plot spectra** {% icon tool %} is a quick way to control peak shape after smoothing. The zoomed in spectrum of the 1257.42 calibrant peaks shows that the peaks become broader than before. Most of the underlying data points can be seen as angles of the peak. The peak width ranges from 1257.1 to 1257.9 m/z = 0.8 m/z width. We will set the following window sizes to 9 data points (half the window size of 4).

We can then continue with the second part of the preprocessing:

  1. **m/z alignment**:
Despite regular m/z calibration of the mass spectrometer, m/z values vary slightly during data acquisition. To make m/z features comparable between different spectra or samples, they need to have the same m/z value. Furthermore, if identification is desired the m/z values should also be as accurate as possible. For high resolution data, binning of the m/z axis is often sufficient to group similar m/z values to a common m/z bin. For TOF instruments, re-calibration of the m/z values is commonly applied with the help of known, ubiquitous analytes such as internal calibrants or matrix peaks. Unfortunately, no internal calibrants were applied and the DHB matrix peaks occur more frequently below 1000 m/z. Therefore, the m/z values are not re-calibrated with known analytes but instead aligned to the average picked mass spectrum to ensure that similar m/z features obtain the same m/z value. This common m/z value can still be slightly shifted from its correct position - this should be kept in mind for the identification step that we will perform later. m/z alignment or binning could also be done after peak picking, but as we will perform peak picking on the average spectrum it is important that the m/z shifts are already minimized. We will remove spectra in which less than 2 reference m/z were found by setting them to zero and then removing all empty spectra. To account for the m/z shifts we will export the processed imzML file, which means that each spectrum has its own m/z axis.

  2. **Peak picking and binning**:
The key step during preprocessing is peak detection as it reduces the dimensionality of the data and bases the following statistical analysis on potentially relevant peaks rather than on noise. A peak is a local maximum above a user-defined noise threshold. The higher the signal-to-noise ratio is set, the more true positive peaks are picked, but at the cost of lowered sensitivity. Therefore, we will set the signal-to-noise ratio to 5, which should remove most of the noise while keeping enough sensitivity. The window should ideally have about the size of the peak width (recalling we set half window size to 4). We will perform peak picking and monoisotopic picking on the average mass spectra of each subfile, which are again defined with the annotation tabular file. The removal of isotopic peaks reduces the number of m/z features of the dataset, which is not only beneficial for computation time but also for many statistical approaches that are hindered by highly correlating variables. To ensure comparability of the picked m/z features, binning of the m/z axis is performed.

  3. **Extracting peaks**:
The average peak picking has removed all spatial information and kept only mean intensity values for the two subfiles. Therefore, the 'peaks' function is used to retrieve the picked peaks from all pixels of the normalized, smoothed and baseline reduced dataset.

For further reading on preprocessing we recommend [Gibb and Strimmer](https://arxiv.org/pdf/1607.03180.pdf) and [Norris et al.](https://doi.org/10.1016/j.ijms.2006.10.005).

> <hands-on-title>Preprocessing part II</hands-on-title>
>
> 1. **MALDIquant preprocessing** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML` (output of **MSI preprocessing** {% icon tool %})
>    - In *"Method"*:
>        - In *"1: Method"*:
>            - *"Select a method"*: `Align spectra`
>                - *"Tolerance"*: `0.0002`
>                - *"Half window size"*: `4`
>                - *"Signal-to-noise-ratio"*: `3`
>                - *"Allow no matches"*: `Yes`
>                - *"Empty no matches"*: `Yes`
>                - *"Remove empty spectra"*: `Yes`
>                - *"minFrequency"*: `0.8`
>                - *"Export processed imzML"*: `Yes`
>
> 2. **MALDIquant peak detection** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `MALDIquant preprocessing imzML` (output of **MALDIquant preprocessing** {% icon tool %})
>    - *"Spectra annotations"*: `yes`
>        - *"Use annotations from tabular file"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with pixel annotations"*: `Column: 3`
>        - *"File contains a header line"*: `Yes`
>    - In *"Method"*:
>        - In *"1: Method"*:
>            - *"Select a method"*: `Peak detection`
>                - *"Half window size"*: `4`
>                - *"Signal to noise"*: `5`
>                - *"Detect peaks on average mass spectra"*: `Yes`
>        - Click on *"Insert Method"*:
>        - In *"2: Method"*:
>            - *"Select a method"*: `Monoisotopic peaks`
>                - *"Minimal correlation"*: `0.4`
>                - *"Tolerance"*: `0.0002`
>                - *"Size"*: `2:5`
>        - Click on *"Insert Method"*:
>        - In *"3: Method"*:
>            - *"Select a method"*: `Binning`
>                - *"Tolerance"*: `0.0002`
>
> 3. **MSI preprocessing** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML from first step` (output of **MSI preprocessing** {% icon tool %})
>    - In *"Preprocessing"*:
>        - Click on *"Insert Preprocessing"*:
>        - In *"1: Preprocessing"*:
>            - *"Preprocessing methods"*: `Data reduction`
>                - *"Reduction method"*: `peaks`
>                - *"Tabular file with m/z features to extract from input file"*: `MALDIquant peak detection intensity matrix` (output of **MALDIquant peak detection** {% icon tool %})
>                - *"Tabular file contains a header line"*: `Yes`
>    - *"Output format"*: `imzML`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many m/z values are left after preprocessing?
> 2. What is the lowest m/z feature that is left after preprocessing?
>
> > <solution-title></solution-title>
> >
> > 1. 125 m/z features are left
> > 2. 1257.47 (min m/z)
> >
> {: .solution}
>
{: .question}

> <details-title>How to find good preprocessing parameters</details-title>
> Before starting with preprocessing it is important to know the properties of the data. Internal calibrants can help to estimate the quality of the data and to control the preprocessing steps. The **MSI Qualitycontrol** {% icon tool %} shows the extent of a noise baseline, the level of noise in the mass spectra, intensity distributions and m/z accuracy as described in the ([Mass spectrometry imaging: Loading and exploring MSI data]({{site.baseurl}}/topics/proteomics/tutorials/mass-spectrometry-imaging-loading-exploring-data/tutorial.html)) tutorial. Most preprocessing steps are applied to single spectra and therefore allow optimization of the preprocessing steps on a small subset of the original dataset that can be obtained with the **MSI filtering** {% icon tool %}. To make sure that the preprocessing steps and parameters are chosen properly it is recommended to perform them step by step and check the outcome after every step. The **MSI Qualitycontrol** {% icon tool %}, the **MSI spectra plot** {% icon tool %}, and the **MSI mz images** {% icon tool %} are helpful to track the changes during preprocessing. The first two tools are able to overlay MSI data with different annotations, making it possible to combine files before and after the preprocessing step, as done in step 1 and 2 of this tutorial, and compare the introduced changes directly. The usage of annotations in the **MSI spectra plot** {% icon tool %} is explained in the tutorial [Mass spectrometry imaging: Examining the spatial distribution of analytes]({{ site.baseurl }}/topics/metabolomics/tutorials/msi-analyte-distribution/tutorial.html).
{: .details}

The QC table of the **MALDIquant preprocessing** {% icon tool %} reveals that 19 spectra (pixel) were removed during the alignment. The empty spectra are visible in the QC file of **MALDIquant peak detection** {% icon tool %}. This QC furthermore shows the baseline in blue and the picked (monoisotopic) peaks in green. After peak extraction the mass spectra are not in profile mode anymore (one peak consists of many datapoints) but centroided (one peak is represented by one datapoint). Therefore, in further steps 'centroided input' must be set to 'Yes'.


# Statistical analysis and identification of N-glycans

## Unsupervised statistical analysis

Unsupervised analysis can be applied at any time to any dataset because no prior knowledge of the data is required. The clustering or segmentation algorithms (that perform unsupervised analysis) separate regions of the data that have the most distinct differences in their mass spectra. We will perform principal component analysis (PCA) with 4 clusters to visualize the largest differences in the dataset.

> <hands-on-title>Segmentation with PCA</hands-on-title>
>
> 1. **MSI segmentation** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML (peaks)` (output of **MSI preprocessing** {% icon tool %})
>    - *"Centroided input"*: `Yes`
>    - *"Select the tool for spatial clustering"*: `pca`
>        - *"The number of principal components to calculate"*: `4`
>    - In *"Colours for the plots"*:
>        - Click on *"Insert Colours for the plots"*:
>        - In *"1: Colours for the plots"*:
>            - *"Colours"*: `dark_blue`
>        - Click on *"Insert Colours for the plots"*:
>        - In *"2: Colours for the plots"*:
>            - *"Colours"*: `orange`
>        - Click on *"Insert Colours for the plots"*:
>        - In *"3: Colours for the plots"*:
>            - *"Colours"*: `green`
>        - Click on *"Insert Colours for the plots"*:
>        - In *"4: Colours for the plots"*:
>            - *"Colours"*: `red`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which principal components correlate with the morphology of the kidney?
> 2. What are the approximate m/z values with the highest loadings for each of those principal components?
>
> > <solution-title></solution-title>
> >
> > 1. PC1, PC3 and PC4
> > 2. PC1: 1900; PC3: 1900; PC4: 2300 (from PCA plot)
> >
> {: .solution}
>
{: .question}

![PCA](../../images/msi-diff-abundance-pca.png "Distribution of the principal components")

The PCA clustering reveals that the N-glycan profile is different in different kidney compartments. To find the m/z features that correspond to these compartments, the feature tabular file can further be analyzed by sorting the file for the principal components of interest. This reveals the discrete m/z values for the features with the highest loadings: PC1 and PC3: 1905.72, PC4: 2304.91. Later, we will visualize the morphological distribution of those features on the complete dataset.


## Supervised statistical analysis

Supervised analysis can only be performed if the regions of interest that should be compared are known. For this dataset we know which spectra were treated or not treated with PNGase F. With this information we can use the spatial shrunken centroids algorithm, which in addition to classification is able to find the m/z features that best distinguish each class ([Bemis et al.](https://doi.org/10.1074/mcp.O115.053918)). We expect those discriminating m/z features to be the N-linked glycans that were only released in the treated kidney section but not in the control section.

> <hands-on-title>Spatial shrunken centroids classification</hands-on-title>
>
> 1. **MSI classification** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `preprocessed imzML (peaks)` (output of **MSI preprocessing** {% icon tool %})
>    - *"Centroided input"*: `Yes`
>    - *"Analysis step to perform"*: `training`
>        - {% icon param-file %} *"Load tabular file with pixel coordinates and their classes"*: `combined annotations` (output of **MSI combine** {% icon tool %})
>        - *"Column with x values"*: `Column: 1`
>        - *"Column with y values"*: `Column: 2`
>        - *"Column with response (condition) values"*: `Column: 3`
>        - *"Tabular files contain a header line"*: `Yes`
>        - *"Select the method for classification"*: `spatial shrunken centroids`
>            - *"Analysis step to perform"*: `spatial shrunken centroids analysis`
>            - *"The sparsity thresholding parameter by which to shrink the t-statistics (s)"*: `0`
>
{: .hands_on}


![SSC](../../images/msi-diff-abundance-classification.png "T-statistics plot for the classification and distribution of the most discriminating m/z feature and class prediction for all spectra.")

High accuracy, sensitivity and specificity indicate that the control and PNGase F treated kidney sections can be discriminated quite well. The prediction shows the location of the few misclassified spectra towards the border of the tissue. As our aim is not to build and test a classifier for accurate prediction, we move on to investigate the m/z features that were most discriminative for the two groups more deeply. Looking at the feature tabular file, we spot the 2 m/z features that had the highest loadings in the PCA, amongst the top features with the highest t-statistics. Furthermore, this list can be compared to the m/z features that were determined as potential N-linked glycans in [Gustafsson et al. Figure S6](https://static-content.springer.com/esm/art%3A10.1007%2Fs00216-014-8293-7/MediaObjects/216_2014_8293_MOESM1_ESM.pdf). We found nine out of twelve m/z features. Two m/z features were outside our cropped m/z range and could therefore not be found. The 1079.408 m/z feature is missing in our feature list.


## Identification of N-glycans

We filter the m/z feature output from the **MSI classification** {% icon tool %} for an adjusted p-value below 0.05 and a positive t-statistic in the 'treated1' spectra to consider only the most discriminative features that are more abundant in the treated than in the control spectra. Those m/z features are potentially N-glycans that were released by PNGase F treatment from the kidney section but not from the control section. To verify that the m/z features are N-linked glycans and to elucidate their identity and structure [Gustafsson et al.](https://doi.org/10.1007/s00216-014-8293-7) performed an additional LC-MS/MS experiment on N-linked glycans released from another mouse kidney section. They identified [31 N-linked glycans](https://media.springernature.com/original/springer-static/image/art%3A10.1007%2Fs00216-014-8293-7/MediaObjects/216_2014_8293_Fig3_HTML.gif) and calculated the m/z for the sodium N-glycan ions which can be matched to the imaging m/z features. Protonated glycans do not ionize well, but sodium adducts are the best detected glycan form in positive mode mass spectrometry. We match the differential m/z features from the imaging experiment to the m/z features from the LC-MS/MS results with a tolerance of 300 ppm and keep all matches that occur within this distance. We then calculate the average ppm error for the matches.


> <hands-on-title>Identification of N-glycans</hands-on-title>
>
> 1. **Filter data on any column using simple expressions** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `classification features` (output of **MSI classification** {% icon tool %})
>    - *"With following condition"*: `c5=="treated1" and c7>0 and c9<0.05`
>    - *"Number of header lines to skip"*: `1`
>
> 2. **Join two files on column allowing a small difference** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"1st file"*: `Filter data` (output of **Filter** {% icon tool %})
>    - *"Column to use from 1st file"*: `Column: 1`
>    - {% icon param-file %} *"2nd file"*: `Glycan IDs` (uploaded dataset)
>    - *"Column to use from 2nd file"*: `Column: 1`
>    - *"Does the input files contain a header line"*: `Yes`
>    - *"Add an addional column with the calculated distance."*: `Yes`
>    - *"Choose the mode of merging."*: `All matches within the defined distance`
>    - *"Choose the metrics of your distance"*: `Distance in ppm`
>    - *"Allowed distance between the two values that will trigger a merge"*: `300.0`
>
> 3. **Summary Statistics for any numerical column**  {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Summary statistics on"*: `Join two files` (output of **Join two files** {% icon tool %})
>    - *"Column or expression"*: `c12`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many m/z features are filtered by our criteria?
> 2. How many m/z features could be identified?
> 3. How many duplicated matches occured?
> 4. What was the average m/z error for the matching?
>
> > <solution-title></solution-title>
> >
> > 1. The filtered file has 32 lines minus one header line, i.e. 31 m/z features.
> > 2. 19 m/z features were identified.
> > 3. 0
> > 4. 28.98 ppm
> {: .solution}
>
{: .question}

31 m/z features of the imaging dataset have a p-value below 0.05 and are potentially N-linked glycans. To ensure the identity of those features, they were matched to the results of an additional LC-MS/MS experiment. This step allows us to identify 19 m/z features as N-linked glycans. 15 of those m/z features were also found and identified in the [original publication](https://doi.org/10.1007/s00216-014-8293-7), which means that we found 4 additional N-glycans: 1704.7, 1542.6, 2101.8 and 1955.7. Those are the 4 m/z features with the lowest t-statistic. One hint to their validity is their m/z error. The m/z errors of 5, 24, 32 and 64 ppm seem reasonable. The average m/z error was 28.98 ppm compared to 38.7 ppm in the original publication. Without resampling of the m/z axis (which was mainly done to speed up computation time) or with additional internal calibrants for m/z re-calibration this error would probably further decrease.
Next, we visually explore the abundance distribution for all identified N-glycans in the complete, TIC-normalized dataset (consisting of the subfiles: control, calibrant, treated1 and treated2).



## N-linked glycan distribution in the complete dataset

> <hands-on-title>Distribution images for the identified N-glycans</hands-on-title>
>
> 1. **MSI mz images** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `all_files` (uploaded dataset)
>    - {% icon param-file %} *"m/z of interest"*: `Join two files` (output of **Join two files** {% icon tool %})
>    - *"Column with m/z values"*: `Column: 1`
>    - *"Column with name of m/z values"*: `Column: 11`
>    - *"Tabular file contains a header line"*: `Yes`
>    - *"Contrast enhancement"*: `suppression`
{: .hands_on}

> <question-title></question-title>
>
> 1. Which N-linked glycans were used as calibrants?
>
> > <solution-title></solution-title>
> >
> > 1. (Hex)2+(Man)3(GlcNAc)2 (1257.5) and (HexNAc)3+(Man)3(GlcNAc)2 (1542.6) (higher intensity in the calibrant file)
> >
> {: .solution}
>
{: .question}

Despite the classification being done on a small part of the complete dataset, the discovered N-glycans are more abundant in both treated kidney sections than in the control and calibrant regions. The four additional N-glycans that we could identify are more abundant in the treated kidney than in the control files, but compared to the other N-glycans their intensities are lower.


## N-linked glycans with a distinct localization to kidney compartments

During PCA analysis we found N-glycans that are localized in specific kidney compartments. We will filter the identified N-glycan file for the specific m/z features to plot their distribution to different compartments as overlay image.

> <comment-title>Regular expressions to filter text</comment-title>
>
> Regular expressions are syntax that can be used to describe a certain 'text'. They are extremely helpful to filter any kind of text and are therefore utilized by several Galaxy tools. The tools help section provides more details about how to use regular expressions. Here, we want to filter for three m/z features, but we are unsure where the exact decimal places are. Therefore we filter only for the m/z feature without decimal places but include the point '.' to make sure that we do not get matches to any other numbers in the tabular file. As the point '.' has a special meaning in regular expressions, it is necessary to 'escape' it by using the backslash '\' - this allows searching for the point as well. For example '1663\\.' will filter for '1663.' in the text. The pipe '\|' separates alternate possibilities and allows the user to run the tool only once while filtering for three different m/z features.
{: .comment}

> <hands-on-title>Overlayed distribution image for 3 N-glycans</hands-on-title>
>
> 1. **Select lines that match an expression** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `Join two files` (output of **Join two files** {% icon tool %})
>    - *"the pattern"*: `(1663\.)|(1905\.)|(2304\.)`
>
> 2. **MSI mz images** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"MSI data"*: `all_files` (uploaded dataset)
>    - {% icon param-file %} *"m/z of interest"*: `Select on data` (output of **Select** {% icon tool %})
>    - *"Column with m/z values"*: `Column: 1`
>    - *"Column with name of m/z values"*: `Column: 11`
>    - *"Contrast enhancement"*: `suppression`
>    - *"Overlay image"*: `yes`
>        - In *"Colours for the plots"*:
>            - Click on *"Insert Colours for the plots"*:
>            - In *"1: Colours for the plots"*:
>                - *"Colours"*: `dark_blue`
>            - Click on *"Insert Colours for the plots"*:
>            - In *"2: Colours for the plots"*:
>                - *"Colours"*: `green`
>            - Click on *"Insert Colours for the plots"*:
>            - In *"3: Colours for the plots"*:
>                - *"Colours"*: `red`
{: .hands_on}

![overlay_images](../../images/msi-diff-abundance-mz-images.png "Distribution for three analytes that are localized in different kidney compartments and their overlay.")


# Conclusion

This tutorial covered combining datasets, data preprocessing and supervised as well as unsupervised statistical analysis.

![workflow](../../images/msi-diff-abundance-workflow.png "Workflow that was used during this tutorial.")


topic: metabolomics
tutorial: qcxms-predictions
title: Predicting EI+ mass spectra with QCxMS
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: e693fc2621edc361bdf09967b21a9a85
last_modified: 2026-02-26T21:12:17.182862
zenodo_link: https://zenodo.org/record/13327051
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/qcxms-predictions/tutorial.html

content:
Mass spectrometry (MS) is a powerful analytical technique used in many fields, including proteomics, metabolomics, drug discovery and many more areas relying on compound identifications. Even though nowadays MS is a standard and popular method, there are many compounds which lack experimental spectra. In those cases, predicting mass spectra from the chemical structure can reveal useful information, help in compound identification and expand the spectral databases, improving the accuracy and efficiency of database search. [{% cite Zhu2023 %}, {% cite Allen2016Computational %}]. 
There have been several methods developed to predict mass spectra, which can be classified as either first-principles physical-based simulation or data-driven statistical methods [{% cite Zhu2023 %}]. To the first category we can assign purely statistical theories (quasi-equilibrium theory (QET) or Rice–Ramsperger–Kassel–Marcus (RRKM) theories) [{% cite Vetter1994 %}], as well as QCxMS [{% cite Koopman2021 %}] and semiempirical GFNn-xTB [{% cite Koopman2019 %}] which use Born–Oppenheimer molecular dynamics (MD) combined with fragmentation pathways. Data-driven statistical methods - forming the second category - reach back to 1960s when the DENDRAL project (using rule-based heuristic programming) was started by early artificial intelligence (AI) scientists [{% cite Lindsay1980ApplicationsOA %}]. More recently, CFM-ID has been introduced [{% cite Allen2014 %}, {% cite Allen2014metabolomics %}, {% cite Allen2016Computational %}, {% cite DjoumbouFeunang2019 %}, {% cite Wang2021 %}], which uses rule-based fragmentation and employs machine learning methods. Current advancements in machine learning led to recent work using deep neural networks that allow predicting spectra from molecular graphs or fingerprints [{% cite Wei2019 %}].

You will be able to check out how QCxMS works in practice since we are going to use Galaxy tool suite based on this method [{% cite Grimme2013 %}, {% cite Bauer2014 %}, {% cite Bauer2016 %}, {% cite Koopman2021 %}]. Beforehand, we will generate conformers of the query molecule with [RDKit](http://www.rdkit.org) and we will use xTB for molecular optimisation [{% cite Bannwarth2020 %}].
But first things first, let’s get some toy data to play with and crack on! 

> <question-title></question-title>
>
> What does QCEIMS stand for?
>
>   > <tip-title>Hint</tip-title>
>   > With a little knowledge of chemistry, you’ll be able to work it out yourself!
>   >
>   > Look at the acronym in the following way: **QC-EI-MS**.
>   {: .tip}
> > <solution-title></solution-title>
> >
> > QC = quantum chemical 
> >
> > EI = electron ionisation
> >
> > MS = mass spectrometry
> >
> > Hence QCEIMS = quantum-chemical electron ionisation mass spectrometry
> >
> {: .solution}
>
{: .question}

> <details-title>What is QCxMS?</details-title>
> QCxMS is a successor of QCEIMS, where the *EI* part is replaced by *x* to take into account other ionisation methods and improve the applicability of the program. In QCEIMS, *EI* stands for *electron ionisation*, while in QCxMS, *x* refers to *EI* or *CID (collision-induced dissociation)* [{% cite Koopman2021 %}]. Currently, only *EI* simulations are supported - using *CID* with the Galaxy tool wrappers is still under development.
{: .details}

 


# Importing data and pre-processing

In this tutorial, you can choose whether you want to predict the mass spectrum for one molecule only, or if you want to do it for multiple molecules at once. The pre-processing steps will slightly differ depending on your choice. If you are completing this tutorial just to see how the QCxMS tools work, feel free to follow the instructions for one molecule to skip some pre-processing steps.  

In both cases, we start from molecule’s SMILES, and then we convert it to SDF, so if you already have SDF files to work with, simply jump in the relevant place in the workflow and carry on from there.

> <details-title>What is SMILES?</details-title>
>
> SMILES (.smi) - the simplified molecular-input line-entry system (SMILES) is a specification in the form of a line notation for describing the structure of a chemical species using short ASCII strings. It is a linear text format which can describe the connectivity and chirality of a molecule [{% cite Weininger1988 %}]. Even though many different forms of SMILES exist, the differences are not relevant for us in this application.
> ![Depiction of a molecular structure and the corresponding SMILES string. Starts of rings are denoted with a number after the corresponding element (i.e. C1). Non-organic elements need to be written in brackets and isotopic forms need to be denoted before the atom and in brackets as well. Atoms with charges need to be in brackets as well and the charge needs to be denoted behind the atom or the functional group. Hydrogens inside brackets need to be stated explicitly. Double bonds are denoted with a `=` and triple bonds with a `#`.](../../images/qcxms_smiles_explanation.png "A quick explanation of the SMILES system.")
> Image credit: Helge Hecht, License: MIT
>
{: .details}

<!---
consider the figure with SMILES and the corresponding structure
-->


{% include _includes/cyoa-choices.html option1="Predict MS for a single molecule" option2="Predict MS for multiple molecules at once" default="Predict MS for a single molecule"
       text="Choose below if you just want to follow the pipeline for prediting the spectrum for only one molecule or multiple molecules at once!" disambiguation="single_vs_multiple" %}

<div class="Predict-MS-for-a-single-molecule" markdown="1">

## Upload data onto Galaxy

Working on a single molecule means that we will work on a dataset and not on a collection of datasets. To simulate the spectrum, we will use ethanol (C<sub>2</sub>H<sub>5</sub> OH) as an example, but you can choose any other molecule that you want, but be aware that the more complex structure you choose, the more time it will take to complete the analysis since the workflow involves generating conformers, semiempirical methods and molecular optimisation.
We will simply start with molecule’s SMILES. 
You have three options for uploading the data. The first two - importing via history and Zenodo link will give a file specific to this tutorial, while the last one – “Paste data uploader" gives you more flexibility in terms of the compounds you would like to test with this workflow. 

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. You can simply import [this history](https://usegalaxy.eu/u/j.jakiela/h/input-file-ei-mass-spectra-prediction-using-qcxms-1) with the input file. 
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the history to your name of choice.
>
{: .hands_on}

><hands-on-title>Option 2: Data upload - Add to history via Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the input table from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/records/13327051/files/ethanol_SMILES.smi
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

> <hands-on-title> Option 3: Data Upload  - paste data </hands-on-title>
> 
> 1. Create a new history for this tutorial
> 2. 
>    - Click {% icon galaxy-upload %} **Upload Data** at the top of the tool panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Paste the SMILES into the text field:  
> ```
> CCO
> ```
>    - Change **Type** from "Auto-detect" to `smi`
>    - Press **Start** and **Close** the window
> 3. You can then rename the dataset as you wish (here we use `ethanol_SMILES`)
> 4. Check that the datatype is `smi`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="smi" %}
>
{: .hands_on}

## SMILES to SDF

> <hands-on-title> Convert SMILES to SDF </hands-on-title>
>
> {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: `ethanol_SMILES` (your SMILES dataset from your history)
>    - *"Output format"*: `MDL MOL format (sdf, mol)`
>    - *"Append the specified text after each molecule title"*: `ethanol`
>
{: .hands_on}

We now have an SDF file, containing the atoms' coordinates and the investigated molecule's name. 

# 3D Conformer generation & optimization

## Generate conformers
The next step involves generating three-dimensional (3D) conformers for our molecule. It crteaes the actual 3D topology of the molecule based on electromagnetic forces. This process might seem trivial for very small and simplistic (meaning no complex structure) molecules, but this can be more challenging for larger molecules with a more flexible geometry. This concerns for example P containing biomolecules, where P often forms a rotational center of the molecule. The number of conformers to generate can be specified as an input parameter, with a default value of 1 if not provided. This process is crucial for exploring the possible shapes and energies that a molecule can adopt. The output of this step is a file containing the generated 3D conformers.

> <details-title> What are conformers? </details-title>
>
> Conformers are different spatial arrangements of a molecule that result from rotations around single bonds. They have different potential energies and hence some are more favourable (local minima on the potential energy surface) than others. 
> ![Newman projections of butane conformations & their relative energy differences (not total energies). Conformations form when butane rotates about one of its single covalent bond. Torsional/dihedral angle is shown on x-axis.](../../images/qcxms_predictions_conformers.svg "Conformers of butane and their relative energy differences.")
> Image credit: [Keministi](https://commons.wikimedia.org/wiki/File:Butane_conformations_and_relative_energies.svg), License: Creative Commons CC0 1.0. 
>
{: .details}


> <hands-on-title> Generate conformers </hands-on-title>
>
>  {% tool [Generate conformers](toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_conformers/ctb_im_conformers/1.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of **Convert to sdf** {% icon tool %}
>    - *"Number of conformers to generate"*: `1`
>
{: .hands_on}

Now - once again format conversion! This time we will convert the generated conformers from the SDF format to Cartesian coordinate (XYZ) format. The XYZ format lists the atoms in a molecule and their respective 3D coordinates, which is a common format used in computational chemistry for further processing and analysis.

> <hands-on-title> Molecular Format Conversion </hands-on-title>
>
> 1. {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: output (sdf) of **Generate conformers** {% icon tool %}
>    - *"Output format"*: `XYZ cartesian coordinates format`
>    - *"Add hydrogens appropriate for pH"*: `7.0`
>
> 2. Check that the datatype is `xyz`. If it's not, just change it – below is the tip how to do it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="xyz" %}
>
{: .hands_on}


## Molecular optimization

As shown in the image in the {% icon details %} *Details box* above, different conformers have different energies. Therefore, our next step will optimize the geometry of the molecules to find the lowest energy conformation. This is crucial to achieve convergence in the next steps. If the input geometry for the QCxMS method is too crude, the ground state neutral run will not converge and we won't be able to sample the geometry to calculate individual trajectories. We will perform semi-empirical optimization on the molecules using the [Extended Tight-Binding (xTB)](https://github.com/grimme-lab/xtb) method. The level of optimization accuracy to be used can be specified as an input parameter, *"Optimization Levels"*. The default quantum chemical method is GFN2-xTB.

> <hands-on-title> Molecular optimisation with xTB </hands-on-title>
>
> {% tool [xtb molecular optimization](toolshed.g2.bx.psu.edu/repos/recetox/xtb_molecular_optimization/xtb_molecular_optimization/6.6.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Atomic coordinates file"*: output of **Convert to xyz** {% icon tool %})
>    - *"Optimization Levels"*: `tight`
>    - *"Keep molecule name"*: {% icon galaxy-toggle %}  `Yes`
>
{: .hands_on}


# QCxMS Spectra Prediction 

## Neutral and production runs

Finally, let’s predict the spectra for our molecule. As mentioned, we will use [QCxMS](https://github.com/qcxms/QCxMS) for this purpose. First, we need to prepare the necessary input files for the QCxMS production runs. These files are required for running the QCxMS simulations, which will predict the mass spectrum of the molecule. This step typically formats the optimized molecular data into a format that can be used for the production simulations. This step performs the ground state calculations. The resulting geometry trajectory is then sampled and one representation is used for each trajectory.


> <hands-on-title> QCxMS neutral run </hands-on-title>
>
>  {% tool [QCxMS neutral run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_neutral_run/qcxms_neutral_run/5.2.1+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Molecule 3D structure [.xyz]"*: output of **xtb molecular optimization** {% icon tool %}
>    - *"QC Method"*: `GFN2-xTB`
>
{: .hands_on}

The outputs of the above step are as follows:
-	.in output: Input file for the QCxMS production run 
-	.start output: Start file for the QCxMS production run 
-	.xyz output: Cartesian coordinate file for the QCxMS production run

We can now use those files as input for the next tool which calculates the mass spectra using QCxMS. This simulation generates .res files, which contain the raw results of the mass spectra calculations. These results are essential for predicting how the molecules will appear in mass spectrometry experiments.

> <hands-on-title> QCxMS production run </hands-on-title>
>
> 1. {% tool [QCxMS production run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_production_run/qcxms_production_run/5.2.1+galaxy3) %} with the following parameters:
>    - {% icon param-collection %} *Dataset collection* *"in files [.in]"*: `input in files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"start files [.start]"*: `input start files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"xyz files [.xyz]"*: `input xyz files` generated by **QCxMS neutral run** {% icon tool %}
>
{: .hands_on}


## Filter failed datasets

It might be the case that some runs might have failed, therefore it is crucial to filter out any failed runs from the dataset to ensure only successful results are processed further. This step is important to maintain the integrity and quality of the data being analyzed in subsequent steps. The output is a file containing only the successful mass spectra results.

> <hands-on-title> Filter failed datasets </hands-on-title>
>
> 1. {% tool [Filter failed datasets](__FILTER_FAILED_DATASETS__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: `res files generated by QCxMS` (output of **QCxMS production run** {% icon tool %} )
>
{: .hands_on}


## Get MSP spectra

The filtered collection contains .res files from the QCxMS production run. This final step converts the .res files into simulated mass spectra in MSP (Mass Spectrum Peak) file format. The MSP format is widely used for storing and sharing mass spectrometry data, enabling easy comparison and analysis of the results, for example by comparing the spectra using the [matchms](https://github.com/matchms/matchms) package.

> <hands-on-title> QCxMS get MSP results </hands-on-title>
>
> 1. {% tool [QCxMS get results](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_getres/qcxms_getres/5.2.1+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Molecule 3D structure [.xyz]"*: `Convert to xyz` (output of **Compound conversion** {% icon tool %})
>    - *"res files [.res]"*: output of **Filter failed datasets** {% icon tool %} (if the collection doesn't appear in the drop-down list, simply drag and drop it from the history panel to the input box)
>
{: .hands_on}

> <details-title>MSP files </details-title>
> MSP (Mass Spectrum Peak) file is a text file structured according to the NIST MSSearch spectra format. MSP is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programmes (MS-DIAL, NIST MS Search, AMDIS, matchms, etc.). It can contain one or more mass spectra, these are split by an empty line. The individual spectra essentially consist of two sections: metadata (such as name, spectrum type, ion mode, retention time, and the number of m/z peaks) and peaks, consisting of m/z and intensity tuples.
{: .details}

You can now {% icon dataset-save %} download the MSP file and open it in your spectra processing software for further investigation! 

To give you some insight into how well QCxMS can perform, below is the mass spectrum of ethanol resulting from our workflow compared with an [experimental spectrum](https://hmdb.ca/spectra/c_ms/28442). Both spectra were compiled using an [online mass spectrum generator](https://www.sisweb.com/mstools/spectrum.htm) which requires only m/z values and intensities – so the values that you can get from our MSP file! As you can see, the predicted peaks nicely correspond to experimental ones. But be careful - there might be slight deviations for molecules with more structural complexity! 

![Upper panel shows the experimental spectrum of ethanol, while the lower panel shows analogical spectrum but predicted with the current workflow. The predicted peaks correspond well to the experimental ones. Intensities of simulated peaks have not been predicted perfectly, but the most important trends are preserved.](../../images/qcxms_predictions_ms_exp_pred.png "Comparison between experimental (upper panel) and predicted (lower panel) mass spectra of ethanol.")


# Conclusion
{% icon trophy %} Well done, you’ve simulated the mass spectrum! You might want to consult your results with the [key history](https://usegalaxy.eu/u/j.jakiela/h/ms-prediction-qcxms-single-molecule). If you would like to process multiple molecules at once, you can simply use [the workflow](https://usegalaxy.eu/u/hechth/h/end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms-1) or move to “Predict MS for multiple molecules at once” tab of this tutorial to learn how the pipeline differs from the one that we’ve just covered. 


</div>

<div class="Predict-MS-for-multiple-molecules-at-once" markdown="1">

## Upload data onto Galaxy

We will work on two simple molecules – ethanol (C<sub>2</sub>H<sub>5</sub> OH) and ethylene (C<sub>2</sub>H<sub>4</sub>). Of course, you might add more and choose any other molecules that you want, but be aware that the more complex structure you choose, the more time it will take to complete the analysis since the workflow involves generating conformers, semiempirical methods and molecular optimisation.
We will start with a table with the first column being molecule names and the second one – corresponding SMILES. 
You have three options for uploading the data. The first two - importing via history and Zenodo link will give a file specific to this tutorial, while the last one – “Paste data uploader" gives you more flexibility in terms of the compounds you would like to test with this workflow. 

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. You can simply import [this history](https://usegalaxy.eu/u/j.jakiela/h/input-file-end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms) with the input table. 
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the history to your name of choice.
>
{: .hands_on}

><hands-on-title>Option 2: Data upload - Add to history via Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the input table from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/records/13327051/files/qcxms_prediction_input.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

> <hands-on-title> Option 3: Data Upload  - paste data </hands-on-title>
> 
> 1. Create a new history for this tutorial
> 2. 
>    - Click {% icon galaxy-upload %} **Upload Data** at the top of the tool panel
>    - Select {% icon galaxy-wf-edit %} **Paste/Fetch Data** at the bottom
>    - Paste the contents into the text field, separated by space. First, enter the name of the molecule, then its SMILES. Please note that we are not using headers here. For this tutorial, we’ll use the example of ethanol and ethylene, but feel free to use your own examples.  
> ```
> ethanol CCO
> ethylene C=C
> ```
>    - Change **Type** from "Auto-detect" to `tabular`
>    - Find the gear symbol ({% icon galaxy-gear %}), deselect any ticked options and select only ({% icon galaxy-gear %}) **Convert spaces to tabs**
>    - Press **Start** and **Close** the window
> 3. You can then rename the dataset as you wish.
> 4. Check that the datatype is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

## Input pre-processing

Once your dataset is uploaded, we can do some simple pre-processing to prepare the file for downstream analysis. Let’s start with ‘cutting’ the table into two columns – one with SMILES, the other with the molecule name – and then separating each entry to create a dataset collection, followed by parsing out the text information. 

> <hands-on-title> Cutting out name column </hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: the tabular file in your history with the compound name and SMILES
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>    - *"Delimited by"*: `Tab`
>    - *"Is there a header for the data’s columns"*: `No`
>    - *"List of Fields"*: `Column: 1`
>
> 2. You can now rename the resulting dataset or just add a tag in order not to confuse it with subsequent outputs:
>    - {% icon galaxy-tags %} Add tag: #names
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

> <hands-on-title> Creating dataset collection (molecule name) </hands-on-title>
>
> {% tool [Split file to dataset collection](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.2) %} with the following parameters:
>    - {% icon param-file %} *"Select the file type to split"*: `Tabular`
>    - *"Tabular file to split"*: output of **Advanced Cut** {% icon tool %} (with #names tag)
>    - *"Number of header lines to transfer to new files"*: `0`
>    - *"Split by row or by a column?"*: `By row`
>    - *"Specify number of output files or number of records per file?"*: `Number of records per file (‘chunk mode’)`
>    - *"Chunk size"*: `1`
>    - *"Base name for new files in collection"*: `split_file`
>    - *"Method to allocate records to new files"*: `Maintain record order`
>
{: .hands_on}

> <hands-on-title> Parsing out name info </hands-on-title>
>
>  {% tool [Parse parameter value](param_value_from_file) %} with the following parameters:
>    - *"Input file containing parameter to parse out of"*: * click on {% icon param-collection %} *Dataset collection* and select output of **Split file** {% icon tool %}
>    - *"Select type of parameter to parse"*: `Text`
>    - *"Remove newlines ?"*: {% icon galaxy-toggle %}  `Yes`
>
>   > <tip-title>Can't find "Parse parameter value" tool? </tip-title>
>   > If you have any problems with accessing **Parse parameter value** tool, you can open [the tool](https://usegalaxy.eu/root?tool_id=param_value_from_file) directly using a given link.
>   {: .tip}
>
{: .hands_on}



We will repeat the first two steps, but processing SMILES this time. 


> <hands-on-title> Cutting out SMILES column </hands-on-title>
>
>  1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: the tabular file in your history with the compound name and SMILES
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>    - *"Delimited by"*: `Tab`
>    - *"Is there a header for the data’s columns"*: `No`
>    - *"List of Fields"*: `Column: 2`
>
> 2. You can now rename the resulting dataset or just add a tag in order not to confuse it with subsequent outputs:
>    - {% icon galaxy-tags %} Add tag: #SMILES
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


> <hands-on-title>  Creating dataset collection (SMILES) </hands-on-title>
>
>  1. {% tool [Split file to dataset collection](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.2) %} with the following parameters:
>    - {% icon param-file %} *"Select the file type to split"*: `Tabular`
>    - *"Tabular file to split"*: output of the latest **Advanced Cut** {% icon tool %} (higher number in your history, with #SMILES tag)
>    - *"Number of header lines to transfer to new files"*: `0`
>    - *"Split by row or by a column?"*: `By row`
>    - *"Specify number of output files or number of records per file?"*: `Number of records per file (‘chunk mode’)`
>    - *"Chunk size"*: `1`
>    - *"Base name for new files in collection"*: `split_file`
>    - *"Method to allocate records to new files"*: `Maintain record order`
>
> 2. Check that the datatype is `smi`. If it's not, just change it! 
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="smi" %}
>
{: .hands_on}

Now, onto format conversion. Let’s convert our SMILES to SDF (Structure Data File) and append the molecule’s name that we have already extracted. 

> <hands-on-title> Convert SMILES to SDF </hands-on-title>
>
> {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - *"Molecular input file"*: click on {% icon param-collection %} *Dataset collection* and select output of **Split file** {% icon tool %} on SMILES
>    - *"Output format"*: `MDL MOL format (sdf, mol)`
>
>   > <tip-title>Add name of the molecule</tip-title>
>   > Another parameter of this tool, *"Append the specified text after each molecule title"*, allows you to add name of the molecule to the title of the generated file. 
>   > If you are working on a single molecule file, you can simply type the name of that compound into the parameter box.
>   > However, if your input is a collection of SMILES (like in this case), then you have to add the names (output of **Parse parameter value** {% icon tool %}) at the level of the workflow editor. 
>   {: .tip}
>
{: .hands_on}

<!---
- *"Append the specified text after each molecule title"*: click on {% icon param-collection %} *Dataset collection* and select output of **Parse parameter value** {% icon tool %} which contains the extracted molecules' names
-->


We now have two SDF files, each containing the coordinates of the atoms and the name of the investigated molecule. Let’s combine them to make life easier and work on just one file. 

> <hands-on-title> Concatenating the files </hands-on-title>
>
> {% tool [Concatenate datasets tail-to-head (cat)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Datasets to concatenate"*: 
>    * if nothing pops up when you click on {% icon param-collection %} *Dataset collection*, then click on "switch to column select" 
>    * from the datasets list, find the output of **Compound conversion** {% icon tool %} (should have the highest number in your history) and select it
>    * {% icon workflow-run %} Run tool! 
>
{: .hands_on}

# 3D Conformer generation & optimization

## Generate conformers
The next step involves generating three-dimensional (3D) conformers for each molecule from the generated SDF. It crteaes the actual 3D topology of the molecules based on electromagnetic forces. This process might seem trivial for very small and simplistic (meaning no complex structure) molecules, but this can be more challenging for larger molecules with a more flexible geometry. This concerns for example P containing biomolecules, where P often forms a rotational center of the molecule. The number of conformers to generate can be specified as an input parameter, with a default value of 1 if not provided. This process is crucial for exploring the possible shapes and energies that a molecule can adopt. The output of this step is a file containing the generated 3D conformers.

> <details-title> What are conformers? </details-title>
>
> Conformers are different spatial arrangements of a molecule that result from rotations around single bonds. They have different potential energies and hence some are more favourable (local minima on the potential energy surface) than others. 
> ![Newman projections of butane conformations & their relative energy differences (not total energies). Conformations form when butane rotates about one of its single covalent bond. Torsional/dihedral angle is shown on x-axis.](../../images/qcxms_predictions_conformers.svg "Conformers of butane and their relative energy differences.")
> Image credit: [Keministi](https://commons.wikimedia.org/wiki/File:Butane_conformations_and_relative_energies.svg), License: Creative Commons CC0 1.0. 
>
{: .details}


> <hands-on-title> Generate conformers </hands-on-title>
>
>  {% tool [Generate conformers](toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_conformers/ctb_im_conformers/1.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: output of **Concatenate datasets** {% icon tool %}
>    - *"Number of conformers to generate"*: `1`
>
{: .hands_on}

Now - once again format conversion! This time we will convert the generated conformers from the SDF format to Cartesian coordinate (XYZ) format. The XYZ format lists the atoms in a molecule and their respective 3D coordinates, which is a common format used in computational chemistry for further processing and analysis.

> <hands-on-title> Molecular Format Conversion </hands-on-title>
>
> 1. {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: output of **Generate conformers** {% icon tool %}
>    - *"Output format"*: `XYZ cartesian coordinates format`
>    - *"Split multi-molecule files into a collection"*: {% icon galaxy-toggle %} `Yes`
>    - *"Add hydrogens appropriate for pH"*: `7.0`
>
> 2. Check that the datatype is `xyz`. If it's not, just change it! 
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="xyz" %}
>
{: .hands_on}


## Molecular optimization

As shown in the image in the {% icon details %} *Details box* above, different conformers have different energies. Therefore, our next step will optimize the geometry of the molecules to find the lowest energy conformation. This is crucial to achieve convergence in the next steps. If the input geometry for the QCxMS method is too crude, the ground state neutral run will not converge and we won't be able to sample the geometry to calculate individual trajectories. We will perform semi-empirical optimization on the molecules using the [Extended Tight-Binding (xTB)](https://github.com/grimme-lab/xtb) method. The level of optimization accuracy to be used can be specified as an input parameter, *"Optimization Levels"*. The default quantum chemical method is GFN2-xTB.

> <hands-on-title> Molecular optimisation with xTB </hands-on-title>
>
> {% tool [xtb molecular optimization](toolshed.g2.bx.psu.edu/repos/recetox/xtb_molecular_optimization/xtb_molecular_optimization/6.6.1+galaxy1) %} with the following parameters:
>    - *"Atomic coordinates file"*: click on {% icon param-collection %} *Dataset collection* and select `Prepared ligands` (output of **Compound conversion** {% icon tool %})
>    - *"Optimization Levels"*: `tight`
>    - *"Keep molecule name"*: {% icon galaxy-toggle %}  `Yes`
>
{: .hands_on}


# QCxMS Spectra Prediction 

## Neutral and production runs

Finally, let’s predict the spectra for our molecules. As mentioned, we will use [QCxMS](https://github.com/qcxms/QCxMS) for this purpose. First, we need to prepare the necessary input files for the QCxMS production runs. These files are required for running the QCxMS simulations, which will predict the mass spectra of the molecules. This step typically formats the optimized molecular data into a format that can be used for the production simulations. This step performs the ground state calculations. The resulting geometry trajectory is then sampled and one representation is used for each trajectory.


> <hands-on-title> QCxMS neutral run </hands-on-title>
>
>  {% tool [QCxMS neutral run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_neutral_run/qcxms_neutral_run/5.2.1+galaxy3) %} with the following parameters:
>    - *"Molecule 3D structure [.xyz]"*: click on {% icon param-collection %} *Dataset collection* and select output of **xtb molecular optimization** {% icon tool %}
>    - *"QC Method"*: `GFN2-xTB`
>
{: .hands_on}

The outputs of the above step are as follows:
-	.in output: Input file for the QCxMS production run 
-	.start output: Start file for the QCxMS production run 
-	.xyz output: Cartesian coordinate file for the QCxMS production run

We can now use those files as input for the next tool which calculates the mass spectra for each molecule using QCxMS (Quantum Chemistry and Mass Spectrometry). This simulation generates .res files, which contain the raw results of the mass spectra calculations. These results are essential for predicting how the molecules will appear in mass spectrometry experiments.

> <hands-on-title> QCxMS production run </hands-on-title>
>
> 1. {% tool [QCxMS production run](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_production_run/qcxms_production_run/5.2.1+galaxy3) %} with the following parameters:
>    - {% icon param-collection %} *Dataset collection* *"in files [.in]"*: `input in files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"start files [.start]"*: `input start files` generated by **QCxMS neutral run** {% icon tool %}
>    - {% icon param-collection %} *Dataset collection* *"xyz files [.xyz]"*: `input xyz files` generated by **QCxMS neutral run** {% icon tool %}
>
{: .hands_on}


## Filter failed datasets

It might be the case that some runs might have failed, therefore it is crucial to filter out any failed runs from the dataset to ensure only successful results are processed further. This step is important to maintain the integrity and quality of the data being analyzed in subsequent steps. The output is a file containing only the successful mass spectra results.

> <hands-on-title> Filter failed datasets </hands-on-title>
>
> 1. {% tool [Filter failed datasets](__FILTER_FAILED_DATASETS__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: output of **QCxMS production run** {% icon tool %}
>
{: .hands_on}


## Get MSP spectra

The filtered collection contains .res files from the QCxMS production run. This final step converts the .res files into simulated mass spectra in MSP (Mass Spectrum Peak) file format. The MSP format is widely used for storing and sharing mass spectrometry data, enabling easy comparison and analysis of the results, for example by comparing the spectra using the [matchms](https://github.com/matchms/matchms) package.

> <hands-on-title> QCxMS get MSP results </hands-on-title>
>
> 1. {% tool [QCxMS get results](toolshed.g2.bx.psu.edu/repos/recetox/qcxms_getres/qcxms_getres/5.2.1+galaxy2) %} with the following parameters:
>    - *"Molecule 3D structure [.xyz]"*: click on {% icon param-collection %} *Dataset collection* and select `Prepared ligands` (output of **Compound conversion** {% icon tool %})
>    - {% icon param-file %} *"res files [.res]"*: output of **Filter failed datasets** {% icon tool %} (if the collection doesn't appear in the drop-down list, simply drag and drop it from the history panel to the input box)
>
{: .hands_on}

The output of this step is a collection of two MSP files - one per each molecule. However, if you want, you can combine those two into one MSP file. 

> <hands-on-title> Combine MSP files into one </hands-on-title>
>
> 1. {% tool [Concatenate datasets tail-to-head (cat)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Datasets to concatenate"*: 
>    * if nothing pops up when you click on {% icon param-collection %} *Dataset collection*, then click on "switch to column select" 
>    * from the datasets list, find the output of **QCxMS get results** {% icon tool %} (should be the latest output dataset) and select it
>    * {% icon workflow-run %} Run tool! 
>
{: .hands_on}

> <details-title>MSP files </details-title>
> MSP (Mass Spectrum Peak) file is a text file structured according to the NIST MSSearch spectra format. MSP is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programmes (MS-DIAL, NIST MS Search, AMDIS, matchms, etc.). It can contain one or more mass spectra, these are split by an empty line. The individual spectra essentially consist of two sections: metadata (such as name, spectrum type, ion mode, retention time, and the number of m/z peaks) and peaks, consisting of m/z and intensity tuples.
{: .details}

You can now {% icon dataset-save %} download the MSP file and open it in your spectra processing software for further investigation! 

To give you some insight into how well QCxMS can perform, below is the mass spectrum of ethanol resulting from our workflow compared with [experimental spectrum](https://hmdb.ca/spectra/c_ms/28442). Both spectra were compiled using an [online mass spectrum generator](https://www.sisweb.com/mstools/spectrum.htm) which requires only m/z values and intensities – so the values that you can get from our MSP file! As you can see, the predicted peaks nicely correspond to experimental ones. But be careful - there might be slight deviations for molecules with more structural complexity! 

![Upper panel shows the experimental spectrum of ethanol, while the lower panel shows analogical spectrum but predicted with the current workflow. The predicted peaks correspond well to the experimental ones. Intensities of simulated peaks have not been predicted perfectly, but the most important trends are preserved.](../../images/qcxms_predictions_ms_exp_pred.png "Comparison between experimental (upper panel) and predicted (lower panel) mass spectra of ethanol.")


# Conclusion
{% icon trophy %} Well done, you’ve simulated mass spectra! You might want to consult your results with the [key history](https://usegalaxy.eu/u/hechth/h/end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms-1) or use [the workflow](https://usegalaxy.eu/u/hechth/h/end-to-end-ei-mass-spectra-prediction-workflow-using-qcxms-1) associated with this tutorial. 

</div>

The prediction of mass spectra might be very useful, particularly for compounds that lack experimental data. Simulating the spectra can also save time and resources. This field has been developing quite rapidly, and recent advancements in new algorithms and packages have led to more and more accurate results. However, one cannot forget that this kind of software should be used to deepen our chemical understanding of the structures of studied compounds and not as a replacement for practical experiments.


topic: metabolomics
tutorial: gcms
title: Mass spectrometry: GC-MS analysis with the metaMS package
difficulty: introductory
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: c4033be37f6f5b48d1b8256e1a8e0998
last_modified: 2026-02-26T21:12:17.144307
zenodo_link: https://zenodo.org/records/16538501
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/gcms/tutorial.html

content:
You may already know that there are different types of *-omic* sciences; out of these, metabolomics is most closely related to phenotypes. Metabolomics involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants. It  focuses on studying the very small molecules which are called *metabolites*, to better understand matters linked to the metabolism. However, studying metabolites is not a piece of cake since it requires several critical steps which still have some major bottlenecks. Metabolomics is still quite a young science, and has many kinds of specific challenges.

One of the three main technologies used to perform metabolomic analysis is **Gas-Chromatography Mass Spectrometry** (GC-MS). Data analysis for this technology requires a large variety of steps, ranging from extracting information from the raw data, to statistical analysis and annotation. Many packages in R/Python are available for the analysis of GC-MS or LC-MS (Liquid-Chromatography Mass Spectrometry) data - for more details see the reviews by {% cite Stanstrup2019 %} and {% cite Misra2021 %}.

This tutorial explains the main steps involved in untargeted **GC-MS** data processing. To do so we focus on some open-source solutions integrated within the Galaxy framework, namely **XCMS** and **metaMS**. The selected tools and functionalities only covers a small portion of available tools but allow to **perform a complete GC-MS analysis** in a single environment.
In this tutorial, we will learn how to (1) extract features from the raw data using **XCMS** ({% cite Smith2006 %}), (2) deconvolute the detected features into spectra with **metaMS** ({% cite Wehrens2014 %}) and (3) annotate unknow spectra using spectral database comparison tools.

To illustrate this approach, we will use data from {% cite Dittami2012 %}. Due to time constraints in processing the original dataset, a limited subset of samples was used to illustrate the workflow. This subset (see details below) demonstrates the key steps of metabolomics analysis, from pre-processing to annotation. Although the results derived from this reduced sample size may not be scientifically robust, they provide insight into essential methodological foundations of GC-MS data-processing workflow.

> <details-title> About the data - Algae samples </details-title>
>
> The objective of the study conducted by Dittami *et al.* was to investigate the adaptation mechanisms of the brown algae *Ectocarpus* to low-salinity environments. The research focused on examining physiological tolerance and metabolic changes in freshwater and marine strains of *Ectocarpus*. Using transcriptomic (gene expression profiling) and metabolic analyses, the authors identified significant, reversible changes occurring in the freshwater strain when exposed to seawater. Both strains exhibited similarities in gene expression under identical conditions; however, substantial differences were observed in metabolite profiles.
> 
> The study utilized a **freshwater strain** of Ectocarpus and a **marine strain** for comparative analysis. The algae were cultured in media with varying salinities, prepared by diluting natural seawater or adding NaCl. The algae were acclimated to these conditions before extraction.
> 
> The six samples used in this training were analyzed by GC-MS (low resolution instrument). A marine strain raised in sea water media (2 replicates) and freshwater strains raised in either 5% or 100% sea water media (2 replicates each). The training dataset is [available on Zenodo](https://zenodo.org/records/16538501)
>
{: .details}

To process the GC-MS data, we can use several tools. One of these is  **XCMS** ({% cite Smith2006 %}), a general R package for untargeted metabolomics profiling. It can be used for any type of mass spectrometry acquisition (centroid and profile) and resolution (from low to high resolution), including FT-MS data coupled with a different kind of chromatography (liquid or gas). Because of the generality of packages like **XCMS**, several other packages have been developed to use the functionalities of **XCMS** for optimal performance in a particular context. The R package called **metaMS** ({% cite Wehrens2014 %}) does so for the field of GC-MS untargeted metabolomics. One of the goals of **metaMS** was to set up a simple system with few user-settable parameters, capable of handling untargeted metabolomics experiments.
In this tutorial we use **XCMS** to detect chromatographic peaks within our samples. Once we have detected them, they need to be deconvoluted into mass spectra representing chemical compounds. For that, we use **metaMS** functions. To normalize the retention time of deconvoluted spectra in our sample, we compute the retention index using Alkane references and a dedicated function of **metaMS**. Finally, we identify detected spectra by aligning them with a database of known compounds. This can be achieved using an in-house built database in the common MSP format (`.msp`) (used in the NIST MS search program for example), resulting in a table of annotated compounds.


> <comment-title></comment-title>
> In Galaxy other GC-MS data processing workflows are available and may be of interest for more advanced Galaxy users (see the [Metabolomics section]({% link /topics/metabolomics/ %})).
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data preparation and prepocessing

Before we can start with the actual analysis pipeline, we first need to download and prepare our dataset. Many of the preprocessing steps can be run in parallel on individual samples. Therefore, we recommend using the Dataset collections in Galaxy. This can be achieved by using the dataset collection option from the beginning of your analysis when uploading your data into Galaxy.


## Import the data into Galaxy

> <hands-on-title> Upload data </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) into a collection:
>
>    ```
>    https://zenodo.org/records/16538501/files/alg11.mzML
>    https://zenodo.org/records/16538501/files/alg2.mzML
>    https://zenodo.org/records/16538501/files/alg3.mzML
>    https://zenodo.org/records/16538501/files/alg7.mzML
>    https://zenodo.org/records/16538501/files/alg8.mzML
>    https://zenodo.org/records/16538501/files/alg9.mzML
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="input" renaming=false %}
>
> 3. Make sure your data is in a **collection**. You can always manually create the collection from separate files:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
>    In the further steps, this dataset collection will be referred to as `input` (and we recommend naming this collection like that to avoid confusion).
>
> 4. Import the following extra files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/record/16538501/files/reference_alkanes.csv
>    https://zenodo.org/record/16538501/files/W4M0004_database_small.msp
>    https://zenodo.org/record/16538501/files/sampleMetadata.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
>    > <comment-title> The extra files </comment-title>
>    >
>    > The three additional files contain the **reference_alkanes**, the **W4M0004_database_small**, and the **sampleMetadata**. Those files are auxiliary inputs used in the data processing and contain either extra information about the samples or serve as reference data for indexing and identification.
>    > 
>    > The **reference_alkanes** (`.tsv` or `.csv`) with retention times and carbon number or retention index is used to compute the retention index of the deconvoluted peaks. The alkanes should be measured in the same batch as the input sample collection.
>    > 
>    > The **W4M0004_database_small** (`.msp`) is a reference database used for the identification of spectra. It contains the recorded and annotated mass spectra of chemical standards, ideally from a similar instrument. The unknown spectra which can be detected in the sample can then be confirmed via comparison with this library. The specific library is an in-house library of metabolite standards extracted with **metaMS** .
>    > 
>    > The **sample metadata** (`.csv` or `.tsv`) is a table containing information about our samples. In particular, the tabular file contains for each sample its associated sample name, class (SW, FWS, etc.). It is possible to add more columns to include additional details about the samples (e.g : batch number, injection order...).
>    {: .comment}
>
{: .hands_on}

As a result of this step, you should have in our history a green Dataset collection {% icon param-collection %} with all 6 samples `.mzML` files as well as three separate files with reference alkanes, reference spectral library, and sample metadata.

## Create the XCMS object

The first part of data processing is using the **XCMS** tool to detect peaks in the MS signal. For that, we first need to take the `.mzML` files and create a format usable by the **XCMS** tool. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} ({% cite gatto2012msnbase %}. {% cite gatto2020msnbase %}) takes as input our files and prepares `RData` files for the first **XCMS** step.


> <comment-title></comment-title>
> The **MSnbase readMSData** {% icon tool %} function, prior to **XCMS**, is able to read files with open format as `mzXML`, `mzML`, `mzData` and `netCDF`, which are independent of the manufacturers' proprietary formats. Working with open MS data file format allows users to us tools developed outside of the MS instrument provider. This set of packages/functions gives modularity, and thus is particularly well adapted to define workflows, one of the key points of Galaxy.
> 
{: .comment}

> <hands-on-title> Create the XCMS object </hands-on-title>
>
> 1. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"File(s) from your history containing your chromatograms"*: `input`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    The output should be `input.raw.RData`
>    
>    This dataset is a collection of `rdata.msnbase.raw` files. That `Rdata` file is necessary in the next step of the workflow.  It contains an internal R representation of **XCMS** objects.
>    
{: .hands_on}

As a result of this step, you should have in your history a new green dataset collection called `input.raw.RData` that will serve as input for the next processing step: **peak picking**.

# Peak detection using XCMS

The first step in the workflow is to detect the peaks in our data using **XCMS** functions. This part, however, is covered by a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html). Although the tutorial is dedicated to LC-MS data, it can also be followed for our GC-MS data. Therefore, in this section, we do not explain this part of the workflow in detail but rather refer the reader to the dedicated tutorial. Please also pay attention to the parameter values for individual Galaxy tools, as these can differ from the referred tutorial and are adjusted to our GC-MS dataset.

> <tip-title>Short-cut for Experts </tip-title>
> Since this step is already covered in a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html), it is possible to skip it. Instead, you can go directly to [Peak deconvolution]({{ site.baseurl }}/topics/metabolomics/tutorials/gcms/tutorial.html#processing-with-metams-option-1) step using a preprocessed **XCMS** object file prepared for you.
> >
> > <hands-on-title> Upload data </hands-on-title>
> > > 1. Import the following files from [Zenodo]({{ page.zenodo_link }}):
> > >
> > >    ```
> > >    https://zenodo.org/records/16538501/files/xset.merged.RData
> > >    ```
> > >
> > >    {% snippet faqs/galaxy/datasets_import_via_link.md %}
> > > 
> > >     The format of uploaded file containing **XCMS** object should be `.RData`.
> > > 
> > >    {% snippet faqs/galaxy/datatypes_understanding_datatypes.md %}
> {: .hands_on}
{: .tip}


The first step (*called peak picking*) is to extract peaks from each of your data files independently. For this purpose, we use the _MatchedFilter_ chromatographic peak detection algorithm implemented in {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %}.

 One Galaxy Training material already explains how to act with MS data. We encourage you to **follow this link and complete the corresponding tutorial**: [Mass spectrometry: LC-MS preprocessing with XCMS]({% link topics/metabolomics/tutorials/lcms-preprocessing/tutorial.md %}). 
For GC-MS analysis you **don't really need to follow all of this previous tutorial** but for a better understanding of your data, it is recommended to try it with their test dataset.
Concerning the current GC-MS tutorial, you **just have to compute the following steps and specific parameters** described in the hands-on part below (please follow the parameter values below to obtain the same results during the training).



> <hands-on-title>Peak picking of GC-MS data with XCMS</hands-on-title>
>
> 1. {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.RData` (the output collection of files from the **MSnbase readMSData** {% icon tool %} step)
>    - *"Extraction method for peaks detection"*: `MatchedFilter - peak detection in chromatographic space`
>        - *"Full width at half maximum of matched filtration gaussian model peak"*: `5`
>        - *"Step size to use for profile generation"*: `0.5`
>        - In *"Advanced Options"*:
>            - *"Maximum number of peaks that are expected/will be identified per slice"*: `500`
>            - *"Signal to Noise ratio cutoff"*: `2`
>            - *"Minimum difference in m/z for peaks with overlapping Retention Times"*: `0.5`
>
>
>    > <comment-title></comment-title>
>    >
>    > For GC-MS data in profile mode, use the *MatchedFilter* algorithm instead of the *CentWave* algorithm used in the LC-MS tutorial.
>    > For low-resolution GC-MS data, a larger m/z difference (0.5 Da) can be used to separate overlapping peaks. If you have high-resolution data (from GC-ToF or GC-Orbitrap), you can use a smaller m/z difference (0.01 Da).
>    {: .comment}
>
> 2. {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.xset.RData` (output collection of files from the **xcms findChromPeaks (xcmsSet)** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file "*: `sampleMetadata.tsv` (One of the uploaded files from Zenodo)
>
>    > <comment-title></comment-title>
>    >
>    > To merge your data, you need to **input a sampleMetadata file** containing filenames and their metadata informations like their class for example.
>    > If you don't add a sampleMetadata file, the **xcms findChromPeaks Merger** {% icon tool %} tool will **group all your files together**. 
>    > You can also **create your sampleMetadata file** with W4M Galaxy tool {% tool [xcms get a sampleMetadata file](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_export_samplemetadata/xcms_export_samplemetadata/3.12.0+galaxy3) %} with the following parameters: {% icon param-collection %} *"RData file"*: `input.raw.RData` output from  **MSnbase readMSData** {% icon tool %}. 
>    > Here is an example of the minimum expectations about a sampleMetadata file (**important**: remove file extension from the sample names, file1.mzML should be file1):
>    >
>    >
>    > | sample_name |  class  | 
>    > |:-----------:|:-------:|
>    > |    file1    |   marine strain   |
>    > |    file2    |  pool  |
>    > |    file3    |   freshwater strain  |
>    >
>    {: .comment}
>
{: .hands_on}

The output from **xcms findChromPeaks Merger** {% icon tool %} is an *.RData* file required for the next steps of the process.

There are two available options: 
- using the metaMS strategy specifically designed for GC-MS data deconvolution and annotation
- using a full XCMS process for GC-MS data processing 

Although this training material is dedicated to GC-MS analysis with the metaMS package, the two options are illustrated in this tutorial. Indeed, as mentioned in the introduction, metaMS is based on XCMS functions, with a specific adaptation to GC-MS data. Thus, one can be interested in comparing results that would be obtained with a standard XCMS processing. Consequently, the two options on the same dataset are available here.  


{% include _includes/cyoa-choices.html option1="Deconvolution and annotation using metaMS" option2="Process GC-MS data with XCMS function directly" default="Deconvolution and annotation using metaMS" text="Choose below if you just want to follow the pipeline using **metaMS** or **XCMS** for GC-MS deconvolution and annotation" disambiguation="gcms_metams_vs_xcms" %}

<div class="Deconvolution-and-annotation-using-metaMS" markdown="1">

# Processing with metaMS (option 1)

**metaMS** is an R package for MS-based metabolomics data. It was made to ease GC-MS data deconvolution and alignment steps using functions from **XCMS** and **CAMERA** packages. In its Galaxy implementation, the two main outputs of **metaMS** are: (1) a table of feature intensities in all samples, which can be analyzed with multivariate methods immediately, and (2) an MSP (`.msp`) file containing GC-MS spectra in a common spectral database format.
The biggest difference between **XCMS** only workflow ([option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2)) or **XCMS + metaMS** GC-MS data processing (option 1) is that rather than a feature-based analysis with individual peaks, as in the [option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2) case, **metaMS** performs a pseudospectrum-based analysis and use it to align compound between samples. One other advantage is that **metaMS** allows creation of MSP (`.msp`) spectra export files ready for annotation.

> <comment-title></comment-title>
> > Not all **metaMS** R package functions have been made available in Galaxy.
When run in R, the **metaMS** package offers a lot of possibilities. For more information on the full set of metaMS functions, visit the [metaMS Bioconductor page](https://www.bioconductor.org/packages/release/bioc/html/metaMS.html).
{: .comment}
 
During this part of the tutorial we are interested in GC-MS analysis with **metaMS**, so we will use the *runGC* function of **metaMS** and describe it in detail to understand all the capabilities of that function.
The standard workflow of **metaMS** for GC-MS data is the following: 

![Workflow diagram of metaMS for GC-MS data analysis](../../images/tuto_gcms_workflow_2options.png "Workflow of metaMS for GC-MS data")

The *runGC* function is implemented in **metaMS.runGC {% icon tool %}** tool in Galaxy. It takes as inputs an {% icon param-collection %} *.RData* file after **XCMS** peak picking  and optionally for annotation purposes an alkane reference file (in either `.csv` or `.tsv` format) for RI calculation and/or a spectral database in `.msp` format.


## Deconvolution and Alignment with metaMS

The peak picking is performed by the usual **XCMS** functions and the output file in `.RData` is used for deconvolution and Alignment steps with *runGC* function.

> <hands-on-title> metaMS.runGC </hands-on-title>
> 
> 1.{% tool [metaMS.runGC](toolshed.g2.bx.psu.edu/repos/yguitton/metams_rungc/metams_runGC/3.0.0+metaMS1.24.0-galaxy0) %}  with the following parameters: 
>   - {% icon param-file %} *"Rdata from xcms and merged"*: `xset.merged.RData` (output of the **xcms findChromPeaks Merger** {% icon tool %} step)
>   - *"Settings"*: `user_defined`
>   - *"RT range option"*: `hide` (If set to *show* you can limit the range of RT processed, for example remove solvant delays)
>   - *"RT_Diff"*: `0.05` (Max time deviation in minute to cluster unknown pseudo-spectra between samples)
>   - *"Min_Features*: `5` (Minimal number of features required to have a valid pseudo-spectrum, compound with less ions will be discarded)
>   - *"similarity_threshold"*: `0.7` (Minimum cosine similarity between pseudo-spectra to be considers as equal)
>   - *"min.class.fract"*: `0.5` (Minimal fraction of samples in which a pseudo-spectrum should be present to be kept) 
>   - *"min.class.size"*: `2` (Minimum number of samples in which a pseudo-spectrum should be find) 
>   - *"Use Personnal DataBase option"*: `show` (This activates the *"DB file"* selector) 
>     - {% icon param-file %} *"DB file"*:  `W4M0004_database_small.msp` (The file download from Zenodo; if not available set the *"Use Personnal DataBase option"* to *hide*)
>   - *"Use RI option*: `show` (Choose *hide* if you want to skip RI calculation)
>     - {% icon param-file %} *"RI file"*: `reference_alkanes.csv` (Format should be strictly respected)
>   - *"Use RI as filter"*: `FALSE` (If set to TRUE only unknown spectra with close RI as those in database will be kept)
>   - *"RIshift"*: *"not used"*
>
>    > <comment-title></comment-title>
>    >
>    > For faster processing keep annotation modules *off* by setting *"Use Personnal DataBase option"*: `hide` and *"Use RI option*: `hide`
>    {: .comment}
> 
>    > <tip-title>Export MSP file to external databases </tip-title>
>    > >You can {% icon dataset-save %} download the MSP file and open it in your favorite spectra processing software or online database for further investigation!
>    {: .tip}
> 
{: .hands_on}

> <details-title> Definitions </details-title>
>    > <details-title> Pseudo-spectra </details-title>
>    > The biggest difference between **XCMS** only or **XCMS + metaMS** GC-MS data processing is that rather than a feature-based analysis with individual peaks, as it is the case with **XCMS**, **metaMS** performs a pseudospectrum-based analysis. So, the basic entity is a set of m/z values showing a chromatographic peak at the same retention time. The idea behind that is that Electron Ionization (EI), which is the most widely used ionization mode in GC-MS analysis, generates a lot more ions for the same molecule than Electrospray Ionisation used in LC-MS. The *runGC* function from **metaMS** is able to group all ions belonging to a molecule into one single cluster that will be used for statistical analysis. For each compound found by **metaMS**, a list of grouped m/z and their intensity is exported as pseudospectrum and this will be used for annotation purpose. For that, the MSP file format is used, which is a common format for mass spectra databases. The pseudospectra are created by grouping all m/z values of a chromatographic peak at the same retention time into one single entry, and then exporting this information in the `.msp` format.
>    > 
>    > ![Example of visualisation of pseudospectrum information](../../images/tuto_gcms_eic.png "Example of cluster of ions through their EIC (left) and the associated pseudospectra (right)")
>    > 
>    > This choice is motivated by several considerations. First of all, **in GC the amount of overlap is much less than in LC**: peaks are much narrower. This means that even a one- or two-second difference in retention time can be enough to separate the corresponding mass spectra. Secondly, EI MS spectra for many compounds are **available in extensive libraries like the [NIST library](http://www.nist.gov/srd/nist1a.cfm "NIST library")** or other online ones like [Golm Metabolome library](http://gmd.mpimp-golm.mpg.de/)
>    {: .details}
>
>    > <details-title>MSP files </details-title>
>    > MSP (Mass Spectrum Peak) file is a text file structured according to the NIST MSSearch spectra format. MSP is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programs (MS-DIAL, NIST MS Search, AMDIS, matchms, etc.). It can contain one or more mass spectra, which are split by an empty line. The individual spectra essentially consist of two sections: metadata (such as name, spectrum type, ion mode, retention time, and the number of m/z peaks) and peaks, consisting of m/z and intensity tuples.
>    > 
>    > **Example of an MSP file entry:**
>    > ```
>    >  Name: Biomarker1
>    >  Retention_time: 5.23
>    >  Num Peaks: 3
>    >  43  100
>    >  57  80
>    >  71  60
>    >  ```
>    >  
>    >  For more details, you can view an [example MSP file](https://zenodo.org/records/16538501/files/W4M0004_database_small.msp).
>    {: .details}
{: .details}

## Alignment

Once **metaMS** has created the pseudo-spectra for each unknown compound in each file, we can start the alignment process. This is done by *comparing each pseudospectrum* to each other in order to group/align similar MS spectra between samples. As a similarity measure, the weighted dot product is used as it is fast, simple, and gives good results ({% cite Stein1994 %}). The first step in the comparison is based on retention, since a comparison of either retention time or retention index is much faster than a spectral comparison. Since the weighted dot product uses scaled mass spectra, the scaling of the database is done once, and then used in all comparisons. If a pseudo-spectra Y from sample A is similar to pseudo-spectra X in sample B and they have close retention (time or index), the two pseudo-spectra are considered as corresponding to the same compound. This process will create the *dataMatrix* and *variableMetadata* outputs, where aligned pseudo-spectra for different samples will belong to the same line in the final *variableMetadata* and will be considered as Unknown compound X. 


![Illustration of two GC-MS spectra aligned for visual comparison](../../images/tuto_gcms_match_spec.png "Best match between an experimental pseudospectrum (red) and a database entry (blue)")

If an MSP database has been added to the *runGC* function inputs then the function returns a table where all patterns that have a match with a DB entry are shown with their name. The other non-matching pseudo-spectra will be named UnknownX in the first column of the *variableMetadata* and *dataMatrix*.

Once **metaMS** have created the pseudo-spectra for each unknown compound in each file, we can start the alignment process. This is done by *comparing every pseudospectrum* to each others in order to group/align similar MS spectra between samples. As a similarity measure, the weighted dot product is used as it is fast, simple, and gives good results ({% cite Stein1994 %}). The first step in the comparison is based on retention, since a comparison of either retention time or retention index is much faster than a spectral comparison. Since the weighted dot product uses scaled mass spectra, the scaling of the database is done once, and then used in all comparisons. If a pseudo-spectra Y from sample A is similar to pseudo-spectra X in sample B and they have close retention (time or index), the two pseudo-spectra are considered as corresponding to the same compound. This process will create the *dataMatrix* and *variableMetadata* outputs, where aligned pseudo-spectra for different samples will belong to the same line in the final *variableMetadata* and will be considered as Unknown compound X. 

## Unknowns research

An important aspect of untargeted metabolomics is the definition of unknowns - features that occur repeatedly in a minimum number or fraction of samples (as defined by the `min.class.fract` and `min.class.size` parameters in the metaMS settings), but for which no annotation has been found. In **metaMS**, these unknown features are found by comparing all patterns (i.e. pseudo-spectra which are groups of features) within a certain retention time (or retention index) difference on their spectral characteristics.
If an MSP database has been added to the *runGC* function inputs, then the function returns a table where all patterns that have a match with a DB entry are shown with their name. The other non-matching pseudo-spectra will be named UnknownX in the first column of the *variableMetadata* and *dataMatrix*.
One strenght of **metaMS** is its ability to use pseudo-spectra (1) for alignment of unknows between samples and (2) to compare unknown experimental pseudo-spectra to previously created in-house spectra databse (in MSP format). By doing so **metaMS** *runGC* function can serve as an annotation tool. You just have to set  - *"Use Personnal DataBase option"* : `show` and add you in-house database file as input.

The *runGC* process will always create an MSP file as output (either with only unknown spectra or with a mix of annotated ones and unknowns). That MSP file can be used for database search online (as Golm ({% cite Kopka2005 %}) and MassBank ({% cite Horai2010 %})) or locally (NIST MSSEARCH) for NIST search (as shown in the following [PDF tutorial](https://workflow4metabolomics.org/sites/default/files/fichiers/documents/w4m_HowToUseNIST_V01.pdf).

For large numbers of samples, this process can take quite some time (it scales quadratically), especially if the allowed difference in retention time is large. The result now is a list of two elements: the first is the annotation table that we also saw after the comparison with the database, and the second is a list of pseudo-spectra corresponding to unknowns. 
An important aspect of untargeted metabolomics is the definition of unknowns — features that occur repeatedly in a minimum number or fraction of samples (as defined by the `min.class.fract` and `min.class.size` parameters in the metaMS settings), but for which no annotation has been found. In **metaMS**, these unknown features are found by comparing all patterns (i.e. pseudo-spectra which are groups of features) within a certain retention time (or retention index) difference on their spectral characteristics.
## Outputs and results

At this stage, all elements are complete: we have the list of pseudo-spectra with an annotation, either as a chemical standard from the database, or an unknown occurring in a sizeable fraction of the injections. The only thing left to do is to calculate relative intensities for the pseudo-spectra, and to put the results in an easy-to-use table. This table consists in two parts. The first part is the information on the “features”, which here are the pseudo-spectra. The second part of the table contains the intensities of these features in the individual injections. 


![Screenshot of peaktable.tsv output of metaMS.runGC](../../images/tuto_gcms_finale_table.png "Final table with unknowns and compounds found during the metaMS processus")

The first eight lines are the standards, and the next ones are the unknowns that are identified by the pipeline. For each sample, "relative intensities" calculated by **metaMS** are given.
In the manual interpretation of this kind of data, the intensities of one or two “highly specific” features (called quantifiers) are often used to achieve relative quantitation. In **metaMS** the reported intensities are calculated differently.

In metaMS, the main measurement for each compound (pseudospectrum) works differently than you might expect. Instead of simply adding up the intensities of different ions belonging to one compound, the package compares what it observes to a "reference model" - think of this as a known fingerprint from a database or a standard sample.
This comparison uses a statistical method called robust linear regression. Here is what this means in simple terms: the software looks for the best scaling factor that makes the observed spectrum match the reference model as closely as possible, while automatically reducing the influence of any unusual or problematic peaks that doesn't fit the pattern.

This method provides a more reliable estimate that is less affected by random fluctuations in the data and variations in specific peaks or generated by a particularly intense spectra that might throw off simpler calculations.

In practical terms, rather than relying on just one or two ions to quantify a compound (which could be unreliable if those specific ions have problems), metaMS considers the entire ion pattern of the pseudospectrum. It calculates an overall "area" using this regression factor.
This approach minimizes the impact of noisy or problematic peaks while making use of all available information - giving you a more robust and trustworthy measurement for each compound in your GC-MS analysis.

In both cases, the result is a list containing a set of patterns corresponding with the compounds that have been found (either annotated or unknown), the relative intensities of these patterns in the individual annotations, and possibly the xcmsSet object for further inspection. In practice, the *runGC* function is all that users need to use.

To recap your [option 1](#processing-with-metams-option-1) journey, here are some questions to check the outcomes of your hands-on: 


> <question-title>getting an overview of your GC-MS processing steps</question-title>
> 
>  **1** - How many known compounds have been detected by metaMS?
> > <solution-title></solution-title>
> > 
> > The answer is: **8**. Here you have to open the {% icon param-file %} `peaktable.tsv` and count.
> > 
> > 
> {: .solution}
> <br>
>  **2** - What is the relative intensity of D-Mannitol (6 TMS) in sample alg7?
> > <solution-title></solution-title>
> > 
> > You should find 74 020 498 in the {% icon param-file %} `peaktable.tsv`.
> > 
> > 
> {: .solution}
> <br>
> **3** - How many Unknowns were detected?
> > <solution-title></solution-title>
> > 
> > The answer is **198**. To be able to see the number of Unknown compounds detected you have to open the {% icon param-file %} `peaktable.tsv` file and go to the last line.
> > 
> > 
> {: .solution}
{: .question}

</div>

<div class="Process-GC-MS-data-with-XCMS-function-directly" markdown="1">
 
# Process GC-MS data with a full XCMS workflow (option 2)

This option follows the standard **XCMS** workflow with GC-MS data as start to obtain in the end a *dataMatrix* file and its corresponding *variableMetadata* file. The main difference with the [option 1](#processing-with-metams-option-1) is that the *dataMatrix* file will contain individual peaks rather than pseudo-spectra, and the *variableMetadata* file will contain information about each peak, such as its retention time, m/z, and intensity. **No** `.msp` file will be generated in this case, as the peaks are not grouped into pseudo-spectra so the annotation process will be different.

> <hands-on-title>Example untargeted GC-MS data processing with the standard XCMS workflow</hands-on-title>
>
> 1. {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.RData` (output of the **xcms findChromPeaks Merger** {% icon tool %} job)
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `5.0`
>        - *"Width of overlapping m/z slices"*: `0.5`
>
> 2. {% tool [xcms fillChromPeaks (fillPeaks)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.RData` (output of the **xcms groupChromPeaks (group)** {% icon tool %} job)
>    - In *"Peak List"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"Number of decimal places for retention time values reported in ions' identifiers."*: `2`
>        - *"Reported intensity values"*: `maxo`
>
> > <comment-title></comment-title>
> > After the **XCMS** processing, the extraction of the MSP-formated spectra can be done using **RAMclustR**{% icon tool %} tool (See this [GTN on GC-MS data processing with XCMS, RAMClustR, RIAssigner, and matchms]({% link topics/metabolomics/tutorials/gc_ms_with_xcms/tutorial.md %})).
> {: .comment}
{: .hands_on}

The outputs of this strategy are similar to the ones described in the LC-MS tutotial mentioned previously. 


To recap your [option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2) journey, here are some questions to check the outcomes of your hands-on: 


> <question-title>getting an overview of your GC-MS processing steps</question-title>
> 
>  **1** - What are the XCMS steps you made in order to obtain your final file RData file?
> > <solution-title></solution-title>
> > 
> > Here are the different steps made through this tutorial: 
> >  - **(Not with XCMS)** import of the data into the Galaxy instance
> >  - **MSNbase readMSData** {% icon tool %} to read and prepare the MS data for the extraction step
> >  - XCMS peak picking with the **xcms findChromPeaks (xcmsSet)** {% icon tool %} tool
> >  - (Not directly XCMS processing, but necessary in the Galaxy tool suit) merge my data into one file with **xcms findChromPeaks Merger** {% icon tool %} tool
> >  - XCMS grouping with the **xcms groupChromPeaks (group)** {% icon tool %} tool
> >  - XCMS integration of missing peaks with **xcms fillChromPeaks (fillPeaks)** {% icon tool %} tool
> > 
> {: .solution}
> <br>
>  **2** - What is the complete name of your final RData file?
> > <solution-title></solution-title>
> > 
> > During each step of the XCMS pre-processing, the name of the file which is processed is completed by the name of the step you used. So, finally your file should be named `xset.merged.groupChromPeaks.fillChromPeaks.RData`. That means (as seen in the previous question) you have run the findChromPeaks (xcmsSet) step, then a merge, a grouping and finally the filling of missing data.
> > 
> > 
> {: .solution}
> <br>
> > 
{: .question}

</div>
   
># Verify your data after the pre-processing and clean datasets
>
>The {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy3) %} part allows users to display the TIC (Total Ion Chromatogram) and the BPC (Base Peak Chromatogram). 

If you separated your samples into different classes, this tool can constructs TICs and BPCs one class against one class, in a `pdf` file (see bellow). 

![The image is a capture of an example of result picture of a TIC one can obtain when comparing two groups.](../../images/tuto_gcms_tic.png "Example of a TIC comparing 2 classes")

 The {% tool [xcms plot chromatogram](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_plot_chromatogram/xcms_plot_chromatogram/3.12.0+galaxy3) %} part allows users to display the TIC (Total Ion Chromatogram) and the BPC (Base Peak Chromatogram). 

>When you have processed **all or only needed** steps described before, you can continue the processing of your data with statistics or annotation tools. 
Don't forget to always check your files' format for compatibility with further analysis! 
>
>The pre-processing part of this analysis can be **quite time-consuming**, and already corresponds to quite a few number of steps, depending of your analysis. If you plan to proceed with further steps in Galaxy (statistics for example), we highly recommend, at this step of the GC-MS workflow, to split your analysis by beginning a new Galaxy history with **only the files you need** for further steps (that would be the final `.tsv` matrices - sampleMetadata, variableMetadata, dataMatrix - and the `.msp` spectral database). This will help you in limiting the chance to select the wrong dataset in further analysis, and bring a little **tidiness** for future review of your analysis process. This would also enable you to make alternative extractions in the future (by adjusting peak picking parameters) in the same history, without drowning your statistical analysis steps far bellow.
>
> To begin a new history with the files from your current history, you can **use the functionality ‘copy dataset’** and copy it into a new history.
> 
> {% snippet faqs/galaxy/histories_copy_dataset.md %}
> 
> 
> You may have notice that the XCMS tools generate **output names that contain the different XCMS steps you used**, allowing easy traceability while browsing your history. Hence, when begining further processing steps (as statistics), we highly recommend you (in particular if you have used the [option 2](#process-gc-ms-data-with-a-full-xcms-workflow-option-2)) to first rename your datasets **with something short**, e.g. "dataMatrix", "variableMetadata", or anything not too long that you may find convenient.
>
> {% snippet faqs/galaxy/datasets_rename.md %}
>
> > <warning-title> Be careful of the file format</warning-title>
> > During each step of pre-processing, your dataset has its formats changed and can have also its name changed. To be able to proceed with GC-MS processing with **metaMS**, you need to identify which datasets can be used with which tool. For example to be able to use *runGC* function you need to have an RData object which is at least **merged** (output from **xcms findChromPeaks Merger** {% icon tool %}). 
After following the option 1 steps you obtained tabular files that can be used in a large variety of analyses, as well as an MSP file (if option 1 was used). Always check the format needed for further tools, as these format are 'text format' but with characteristics that may matter in further steps. 
With option 2, if you want to process your data with XCMS or other tools you may also have to **align** them with **xcms groupChromPeaks (group)** {% icon tool %}. It means that you should have at least a file named `xset.merged.RData` to be able to continue XCMS processing.
> {: .warning} 

# Conclusion 
{% icon trophy %} Well done, you’ve processed GC-MS data with [metaMS (option 1)](#processing-with-metams-option-1) or [all with XCMS (option 2)](#process-gc-ms-data-with-a-full-xcms-workflow-option-2)! You might want to explore other [Galaxy trainings for Metabolomics]({% link topics/metabolomics/ %})  
<br>
You might want to consult your results with the [key history option 1](https://workflow4metabolomics.usegalaxy.fr/u/yguitton44/h/gtn-gcms-with-metams) or use [the workflow for option 1](https://workflow4metabolomics.usegalaxy.fr/u/yguitton44/w/workflow-for-gtn-gcms-with-metams-option1) associated with this tutorial.


topic: metabolomics
tutorial: mfassignr
title: Molecular formula assignment and mass recalibration with MFAssignR package
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: eba97ab3e580b0f56fde1324a0c70130
last_modified: 2026-02-26T21:12:17.174387
zenodo_link: https://zenodo.org/records/13768009
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/mfassignr/tutorial.html

content:
This training covers the multi-element molecular formula (MF) assignment using the MFAssignR tool. It was originally developed by {% cite Schum2020 %} for the analysis of untargeted mass spectrometry data coming from complex environmental mixtures. The package contains several functions including noise assessment, isotope filtering, internal mass recalibration, and formula assignment. 

The **MFAssignR workflow** is composed of several steps:

1. Run **KMDNoise** to determine the noise level for the data.
2. Check the effectiveness of the S/N threshold using **SNplot**.
3. Use **IsoFiltR** to identify potential 13C and 34S isotope masses.
4. Using the signal-to-noise (S/N) threshold, and the two data frames output from IsoFiltR(), run **MFAssignCHO** to assign MF with C, H, and O to assess the mass accuracy.
5. Use **RecalList** to generate a list of the potential recalibrant series.
6. Use **FindRecalSeries** to find the most suitable recalibrant series.
7. After choosing recalibrant series, use **Recal** to recalibrate the mass lists.
8. Assign MF to the recalibrated mass list using **MFAssign**.
9. Check the output plots from MFAssign() to evaluate the quality of the assignments.

We can illustrate the workflow also on the following scheme:
![workflow_graphical](images/mfassignr_scheme.png)

Let's dive now into the individual steps and explain all the inputs, parameter settings, and outputs.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data import

At the very beginning, we need to import the dataset we will be using. MFAssignR requires on input a table in a **tabular format**, where:

- the first column is the mass to charge ratio (m/z),
- the second column is intensity,
- the optional third column is retention time.

In our case, we will use the model data from the MFAssignR package, which contains the negative ions from acetonitrile extract of wildfire-influenced atmospheric organic aerosol, collected at the Pacific Northwest National Laboratory (PNNL). The detection m/z range of the instrument was 100-800 and the ions were analyzed by electrospray ionization (ESI) on Orbitrap Elite MS and recorded as Xcalibur.raw files. In the MFAssignR package, this dataset is named Raw_Neg_ML in .rds format, which we saved as a tabular file and we will use this file as our input.

> <hands-on-title> Upload data </hands-on-title>
>
> 1. Create a new history for this tutorial and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/records/13768009/files/mfassignr_input.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Set the format of the dataset to 'tabular'. 
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 4. We can now view our dataset, and ensure, that it has a correct input format.
>
>
{: .hands_on}

# Noise assessment

Having our input data in the workspace, we can now start with the analysis!

The first step is the noise assessment, which allows us to avoid both false positives (if noise is underestimated) and false negatives (if noise is overestimated). For this purpose, we will be using the **KMDNoise** function, and visualize the result using **SNplot** function. MFAssignR provides an additional function, **HistNoise**, but often the data distribution prerequisites are not fulfilled and especially when the analyte signal tapers into the noise, HistNoise fails to separate the distributions. Therefore, using KMDNoise is strongly preferred.

## KMDNoise

KMDNoise function takes advantage of the calculation of **Kendrick Mass Defect (KMD)**.

> <details-title>Kendrick Mass Defect</details-title>
>
>The **Kendrick mass (KM)** serves to simplify classification and
>identification of repeating units in molecules, typically 
>homologous CH2 series, which differ only by the number of base 
>units (-CH2 groups in this case). It sets the mass of the molecular fragment to 
>integer value in atomic mass units (amu).
>
>The KM of a compound can be therefore calculated as the known m/z (given the ionization) of the compound multiplied by a ratio (rounded
>(CH2_mass)/exact(CH2_mass)), which is (14/14.01565) = 0.9988834.
>
>$$\text{KM} = \text{m/z} \times \frac{\text{14.00000}}{\text{14.01565}}$$
>
>
>The **Kendrick Mass Defect (KMD)** is then defined as the difference 
>between rounded KM values and KM.
>$$\text{KMD} = \text{nominal KM} - {\text{KM}}$$
>
>Homologous series, meaning compounds differing only in the number of 
>repeating units (e.g. alkylation series), will always have the same KMD.
>
>{% cite kendrickmass_mzmine %} {% cite kendrickmass_wiki %}
{: .details}


When the KMD is calculated for all peaks in the mass spectrum, there will be a clear separation of more intense analyte peaks and low-intense noise peaks. To isolate the noise region, we can use the KMD limits of chemically feasible molecular formulas in conjunction with the calculation of the slope of a KMD plot using a linear equation $$y = 0.1132x + b$$, where *y* is the KMD value, *0,1132* was an empirically derived integer, *x* the measured ion mass and *b* an y-intercept. To provide a more accurate assessment, two lines with different y-intercepts are selected (we will set this lower and upper y-limit below). Once the noise region is isolated, a noise level will be estimated as the average intensity of peaks within that region.

![KMD plot](images/KMDplot_explained.png)

When running the function, we can stick with the default values: the upper limit for the y-intercept is set to 0.2, so that it does not interact with any potentially double-charged peaks, the lower limit of the y-intercept value is set to 0.05 to ensure no analyte peaks are incorporated into the noise estimation. Both upper and lower x-intercept limits are optional and will be set to minimum and maximum mass in the data if not specified. 

> <hands-on-title> Noise assessment using KMDNoise </hands-on-title>
>
> 1. {% tool [MFAssignR KMDNoise](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_histnoise/mfassignr_kmdnoise/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `mfassignr_input.txt` (Input dataset)
>    - {% icon param-file %} *"upper limit for the y intercept"*: `0.2`
>    - {% icon param-file %} *"lower limit for the y intercept"*: `0.05`
>   
>
{: .hands_on}

The function outputs a **KMD plot**, where the noise area is separated between red lines and a **noise estimate**. The noise estimate we can then multiply with a multiplier, typically a value between 3 and 10 in order to remove low-intensity m/z values. Multiplication with 10 means we will be very stringent, the signal-to-noise ratio will be high and close to the limit of quantitation (LOQ), whereas multiplication with 3 means the signal-to-noise ratio will be low and we are close to the limit of detection (LOD). The optimal is therefore to start with e.g. 6 and check the results using SNplot, as we will do further.

> <question-title></question-title>
>
> 1. What is the noise level estimated by KMDNoise function?
>
> > <solution-title></solution-title>
> >
> > 1. Using our model data, the estimated noise is **346.0706**.
> >
> >
> {: .solution}
>
{: .question}

## SNplot

We can now check the effectiveness of the S/N threshold using **SNplot**, which plots the mass spectrum with the masses below and above the chosen threshold, where the noise is indicated by red.

The `cut` parameter can be computed as an estimated noise level * multiplier, so if we get 346.0706 as a noise level from KMDnoise, we can multiply it by 6, which gives us 2076.

- `Mass` - parameter defines a centerpoint to look at the mass spectrum.
- `Parameter window.x` - sets the +/- range around the mass centerpoint, default is 0.5.
- `Parameter window.y` - is used for setting the limit of the y-axis in the plot. Parameter window.y multiplied the `cut` parameter value, and the multiplication result is used as the y-axis limit. The default is 10.

> <hands-on-title> Plot the SNplot </hands-on-title>
>
> 1. {% tool [MFAssignR SNplot](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_snplot/mfassignr_snplot/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `mfassignr_input.txt` (Input dataset)
>    - *"SN ratio"*: `6.0`
>    - *"Estimated noise"* `346.0706`
>    - *"mass"*: `301.0`
>    - *"window.x"*: `50.0`
>    - *"window.y"*: `10.0`
{: .hands_on}

![SNplot](images/SNplot.png)

Based on the SNplot, we can see that the noise - indicated in red - forms a uniform background and is effectively separated by the horizontal line (indicating the S/N threshold), meaning we can further use the multiplication of 346 and 6. If the noise was higher than the S/N threshold, it would mean we need to increase the value of the multiplier.


# Isotope filtering

The next step is the identification of probable isotopic ion masses containing 1-2 13C or 34S from monoisotopic masses in order to prevent incorrect interpretation of the molecular composition. For this, we will use the **IsoFiltR** function.

Isotope filtering has 4 steps:

1. Firstly, the mass list is transformed to pairs of ions, which have a mass difference of 1.003355 Da for 13C and 1.995797 Da for 34S with some flexibility for a user-defined error tolerance (by default 5 ppm).
2. For specific isotopes, KMD values are calculated using the mass difference between 12C and 13C (1.003355 Da) and the mass difference between 32S and 34S (1.995797 Da).
3. Isotope pairs are further refined using *Resolution Enhanced KMD* adapted from {% cite Zheng2019 %}, in which the Kendrick base is adjusted by an experimentally derived integer (For 34S, the integer is 12 (determined by trial and error) and for 13C the integer is 21 based on {% cite Zheng2019 %}) in order to obtain a desired separation.
4. Finally, abundance ratios are evaluated, where the abundance ratio for 13C should be < 0.6 and the abundance ratio for 34S should be < 0.3.

After performing isotopic filtering, two tables are outputted, one containing the isotopic masses and one containing the monoisotopic masses and all masses that did not have a matching isotopic mass. In complex mixtures, a mass can be classified as both monoisotopic and isotopic; in those cases, it is included in both output tables and classified after the MF assignment.

We will change only the S/N ratio parameter and estimated noise (based on the KMDNoise function), otherwise, we can follow with the default settings.

> <hands-on-title> Isotope filtering using IsoFiltR </hands-on-title>
>
> 1. {% tool [MFAssignR IsoFiltR](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_isofiltr/mfassignr_isofiltr/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input Peak Data"*: `mfassignr-input.txt` (Input dataset)
>    - *"SN ratio"*: `6.0`
>    - *"Estimated noise"* `346.0706`
>
>
{: .hands_on}

# Preliminary molecular formula assignment

Once we have denoised the data and have identified the potential 13C and 34S masses, we can start with the preliminary MF assignment, using the **MFAssignCHO** function. This function assigns MF only with C, H, and O elements to assess the mass accuracy, and therefore it's much quicker than the main MFAssign function. It is based on the CHOFIT algorithm developed by {% cite Green2015 %}.

Firstly, the initial formula assignment with the CHOFIT algorithm is performed along with the quality checks. Then, KMD and z* values are calculated for all masses with the CH2 Kendrick base, and masses are sorted into CH2 homologous series. Subsequently, 1-3 members of each CH2 homologous series below the user-defined threshold are selected and assigned a MF. The ambiguous MFs are returned within the _Unassigned_ list, whereas the unambiguous MF are further re-assigned using CH2, O, H2, H2O, and CH2O extensions. 

Let's use the MFAssignCHO function. On the input, we will need the output of the IsoFiltR function, a data frame of monoisotopic masses, and also (optionally) the data frame containing isotopic masses. Furthermore, we select in which ion mode we measured the data (positive or negative) and we set the signal-to-noise threshold - based on the noise estimate we obtained from the **KMDNoise** or **HistNoise** functions multiplied by the multiplier. Finally, based on the acquisition range, we set the lowMW and highMW, and also the allowed ppm error. Other parameters we can leave in default settings, although in the 'Advanced' tab, many additional parameters can be set by user, including amounts of elements, ratios (e.g. minimum allowed O/C ratio) and many others. 

On the output, there are two dataframes, `Ambiguous` and `Unambigous` provided.

> <hands-on-title> Preliminary MF assignment with MFAssignCHO </hands-on-title>
>
>
> 1. {% tool [MFAssignR MFAssignCHO](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_mfassigncho/mfassignr_mfassignCHO/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Data frame of monoisotopic masses"*: `mono_out` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - {% icon param-file %} *"Data frame of isotopic masses"*: `iso_out` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - *"ppm_err"*: `3`
>    - *"Ion mode"*: `negative`
>    - *"SN ratio"*: `6`
>    - *"Estimated noise"*: `346.0706`
>    - *"Lower limit of molecular mass to be assigned"*: `50.0`
>    - *"Upper limit of molecular mass to be assigned"*: `1000.0`
>
>
{: .hands_on}

On the output, we get several data frames: unambiguous assignments, ambiguous assignments, and a data frame containing unassigned masses. Additionally, several plots are available, such as MSAssign showing a mass spectrum highlighting assigned and unassigned masses, errorMZ showing the relationship between absolute error (in ppm) and ion mass, van Krevelen plot, and a mass spectrum colored by molecular group.

![errorMZ](images/errorMZ.png)
This plot shows us the absolute error, which increases with the increasing mass as expected. Actually, it is good to see a trend here, meaning there is a systematic error that will be corrected during recalibration, a problem would be if there was a random error that we can't correct.

![Van Krevelen plot](images/vankrevelen.png)

> <details-title>Van Krevelen plot</details-title>
>The Van Krevelen plot was originally introduced in 1950 {% cite krevelen1950graphical %} to illustrate 
>the coal formation processes. It typically depicts the atomic ratios of hydrogen-to-carbon (H/C) 
>ratio on the y-axis and oxygen-to-carbon (O/C) on the x-axis. Certain molecules occupy different regions of
>the van Krevelen plot, e.g. alkanes usually have high H/C and low O/C ratios, aromatic compounds have
>lower H/C ratios and carbohydrates usually have high O/C and moderate H/C ratios.
>
{: .details}

Van Krevelen plot visualizes the chemical composition of complex chemical mixtures and it characterizes the data quality and how the data is distributed. Compounds with similar structures and functional groups will cluster together on the plot.

![MS groups](images/MSgroups.png)
The MSgroups plot shows the abundance distribution of unambiguous molecular formula assignments. 

![msassign](images/msassign.png)
Assignment mass spectrum then shows what was assigned: green are the assigned compounds, red are unassigned ones (eventually, if applicable, isotopes are in blue). It is useful to check whether there are any gaps around a particular m/z, which would mean there is a problem with the assignment.

# Recalibration
The next step is recalibration, which ensures that we will have an accurate mass list before formula assignment and any systemic bias is removed. In MFAssignR, recalibration was adapted from {% cite Savory2011 %} and {% cite Kozhinov2013 %}.

There are three consecutive functions, **RecalList**, **FindRecalSeries** and **Recal**, which we will use. 

## RecalList

Firstly, we will generate a table containing potential recalibrant CH2 homologous series using the RecalList function. 
As in input, we will use the `Unambig` data frame which we generated in the MFAssignCHO function:

> <hands-on-title> Finding recalibrant series </hands-on-title>
>
> 1. {% tool [MFAssignR RecalList](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_recallist/mfassignr_recallist/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `Unambig` (output of **MFAssignR MFAssignCHO** {% icon tool %})
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many series were returned?
>
> > <solution-title></solution-title>
> >
> > 1. When we click on the results ('Recalibration series by MFAssignR RecalList on data 10'), we can
> >see there are 226 lines (1 being the column names), meaning 225 series which we can select from for the recalibration.
> >
> {: .solution}
>
{: .question}

On the output, we get a data frame containing CH2 homologous series that contain more than 3 members. Let's dive more into what metrics are available:

- **Series** - reports the homologous series according to class, adduct, and DBE. The format is "class_adduct_DBE", for example, a homologous series with class = "O6, adduct of Na+, and DBE = 4" would be "O6_Na_4".
- **Number Observed** - reports the number of members of each homologous series.
- **Series Index** - represents the order of the series when ordered by length of homologous series.
- **Mass Range** - reports the minimum and maximum mass for the compounds within a homologous series.
- **Tall Peak** - reports the mass of the most abundant peak in each series.
- **Abundance Score** - reports the percentage difference between the mean abundance of a homologous series and the median abundance within the mass range the "Tall Peak" falls in (for example m/z 200-300). A higher score is generally better.
- **Peak Score** - This column compares the intensity of the tallest peak in a given series to the second tallest peak. This comparison is calculated by $$\text{log10} \times \frac{\text{Max Peak Intensity}}{\text{Second Peak Intensity}}$$ The closer to 0 this value is the better, in general.
- **Peak Distance** - This column shows the number of CH2 units between the tallest and second tallest peak in each series. In general, the value should be as close to 1 as possible.
- **Series Score** - This column compares the number of actual observations in each series to the theoretical maximum number based on the CH2 homologous series. The closer to one this value is, the better.

## FindRecalSeries

Which are the best series for recalibration? Combined, these series should cover the full mass spectral range to provide the best overall recalibration. The best series to choose are generally long and combined have a "Tall Peak” at least every 100 m/z.

Currently, it is up to the user to choose the most suitable recalibrant series that will be used for the recalibration in the Recal step. It is possible to choose up to 10 series and they should indeed span over the whole range of m/z - often an error is thrown to add more series in case there are gaps, or the chosen series would have a good score but would be way too alike. Therefore, we provide the **FindRecalSeries** function, which attempts to select the best series which fulfill the abovementioned criteria.

We will only provide the `Recalibration series` dataframe from RecalList function, and the best series will be chosen automatically. Although a list of more than 10 series might be provided on the output, only first 10 best series will be used by the Recal function in the next step.

> <hands-on-title> Selecting most suitable series </hands-on-title>
>
> 1. {% tool [MFAssignR FindRecalSeries](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_findrecalseries/mfassignr_findRecalSeries/1.1.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input data"*: `recal_series` (output of **MFAssignR RecalList** {% icon tool %})
>
{: .hands_on}

## Recal

The Recal function is used for the internal mass recalibration. It takes the output from MFAssignCHO and outputs from IsoFiltrR, the `Mono` and `Iso` data frames. Finally, it takes the series for recalibration, either chosen by the user or selected by the FindRecalSeries function. 

A common error points to increasing the `MzRange` parameter, which sets the recalibration segment length and has a default value of 30 - sometimes it is needed to increase it to even values as 80. Other parameters can be left to their defaults.

> <hands-on-title> Internal mass recalibration </hands-on-title>
>
> 1. {% tool [MFAssignR Recal](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_recal/mfassignr_recal/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input data (Output from MFAssign)"*: `Unambig` (output of **MFAssignR MFAssignCHO** {% icon tool %})
>    - {% icon param-file %} *"Calibration series (Output from RecalList)"*: `Final recalibration series` (output of **MFAssignR FindRecalSeries** {% icon tool %})
>    - {% icon param-file %} *"Peaks dataframe (Mono from IsoFiltR)"*: `Monoisotopic masses` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - {% icon param-file %} *"Isopeaks dataframe (Iso from IsoFiltR)"*: `iso_out` (output of **MFAssignR IsoFiltR** {% icon tool %})
>    - *"Ion mode"*: `negative`
>    - *"SN ratio"*: `6`
>    - *"Estimated noise"*: `346.0706`
>    - *"Mass windows used for the segmented recalibration"*: `50.0`
>
>
{: .hands_on}

# Molecular formula assignment

The last step of the workflow is the actual assignment of molecular formulas with 12C, 1H, and 16O and a variety of heteroatoms and isotopes, including 2H, 13C, 14N, 15N, 31P, 32S, 34S, 35Cl, 37Cl, 19F, 79Br, 81Br, and 126I. It can also assign Na+ adducts, common in positive ion mode. We have already described the principles of the MFAssignCHO function, which works the very same way as MFAssign (only being limited to C, H, and O elements), so let's directly use the MFAssign function for the formula assignment. 

> <hands-on-title> MF assignment using MFAssign </hands-on-title>
>
> 1. {% tool [MFAssignR MFAssign](toolshed.g2.bx.psu.edu/repos/recetox/mfassignr_mfassign/mfassignr_mfassign/1.1.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Data frame of monoisotopic masses"*: `Mono` (output of **MFAssignR Recal** {% icon tool %})
>    - {% icon param-file %} *"Data frame of isotopic masses"*: `Iso` (output of **MFAssignR Recal** {% icon tool %})
>    - *"ppm_err*: `3`
>    - *"Ion mode"*: `negative`
>    - *"SN ratio"*: `6`
>    - *"Estimated noise"*: `346.0706`
>    - *"Lower limit of molecular mass to be assigned"*: `50.0`
>    - *"Upper limit of molecular mass to be assigned"*: `1000.0`
>
{: .hands_on}

Finally, we obtain assigned formulas on rearranged data. Similarly to the MFAssignCHO function, data frames of unassigned masses, ambiguous assignments and unambiguous assignments are provided, as well as quality check plots. 


# Conclusion

In this tutorial, we showed how can we assign molecular formulas using the MFAssignR package. We learned  several key steps, including noise estimation, isotope filtering, recalibration, and finally the formula assignment. We described how parameters of individual functions are used and how they should be adjusted if needed. 

![workflow overview](images/workflow_overview.png)


topic: metabolomics
tutorial: gc_ms_with_xcms
title: Mass spectrometry: GC-MS data processing (with XCMS, RAMClustR, RIAssigner, and matchms)
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 47dad6ee3e7298bf9a395e7269120b32
last_modified: 2026-02-26T21:12:17.142696
zenodo_link: https://zenodo.org/record/7890956
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/gc_ms_with_xcms/tutorial.html

content:
The study of metabolites in biological samples is routinely defined as metabolomics. Metabolomics' studies based on untargeted mass spectrometry provide the capability to investigate metabolism on a global and relatively unbiased scale in comparison to traditional targeted studies focused on specific pathways of metabolism and a small number of metabolites. The untargeted approach enables the detection of thousands of metabolites in hypothesis-generating studies and links previously unknown metabolites with biologically important roles {% cite Patti2012 %}. There are two major issues in contemporary mass spectrometry-based metabolomics: the first is enormous loads of signal generated during the experiments, and the second is the fact that some metabolites in the studied samples may not be known to us. These obstacles make the task of processing and interpreting the metabolomics data a cumbersome and time-consuming process {% cite Nash2019 %}.

Many packages are available for the analysis of GC-MS or LC-MS data - for more details see the reviews by {% cite Stanstrup2019 %} and {% cite Misra2021 %}. In this tutorial, we focus on open-source solutions integrated within the Galaxy framework, namely **XCMS**, **RAMClustR**, **RIAssigner**, and **matchms**. In this tutorial, we will learn how to (1) extract features from the raw data using **XCMS** ({% cite Smith2006 %}), (2) deconvolute the detected features into spectra with **RAMClustR** ({% cite broeckling2014ramclust %}), (3) compute retention indices with **RIAssigner** ({% cite hecht2022riassigner %}), and (4) identify the present compounds leveraging spectra and retention indices using **matchms** ({% cite Huber2020 %}). For demonstration, we use three GC-[EI+] high-resolution mass spectrometry data files generated from quality control seminal plasma samples.


> <details-title> Seminal plasma samples </details-title>
> 
> The seminal plasma samples were analyzed according to the standard operating procedure [(SOP) for metabolite profiling of seminal plasma via GC Orbitrap](https://zenodo.org/record/5734331).
> The 3 samples used in this training are pooled quality control (QC) samples coming from about 200 samples. The pooled samples were analyzed in dilution series to test the system suitability and the quality of the assay.
>
{: .details}

To process the data, we use several tools. **XCMS** ({% cite Smith2006 %}) is a general package for untargeted metabolomics profiling. It can be used for any type of mass spectrometry acquisition (centroid and profile) or resolution (from low to high resolution), including FT-MS data coupled with a different kind of chromatography (liquid or gas). We use it to detect chromatographic peaks within our samples. Once we have detected them, they need to be deconvoluted into spectra representing chemical compounds. For that, we use **RAMClustR** ({% cite broeckling2014ramclust %}) tool. To normalise the retention time of deconvoluted spectra in our sample, we compute the retention index using **RIAssigner** ({% cite hecht2022riassigner %}) by comparing the data to a well-defined list of reference compound (commonly alkanes) analyzed on the same GC column. Finally, we identify detected spectra by aligning them with a database of known compounds. This can be achieved using **matchms** ({% cite Huber2020 %}), resulting in a table of identified compounds weighted by a matching score.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data preparation and prepocessing

Before we can start with the actual analysis pipeline, we first need to download and prepare our dataset. Many of the preprocessing steps can be run in parallel on individual samples. Therefore, we recommend using the Dataset collections in Galaxy. This can be achieved by using the dataset collection option from the beginning of your analysis when uploading your data into Galaxy.

## Import the data into Galaxy

> <hands-on-title> Upload data </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) into a collection:
>
>    ```
>    https://zenodo.org/record/7890956/files/8_qc_no_dil_milliq.raw
>    https://zenodo.org/record/7890956/files/21_qc_no_dil_milliq.raw
>    https://zenodo.org/record/7890956/files/29_qc_no_dil_milliq.raw
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true format="mzml" collection_name="input" renaming=false %}
>
> 3. Make sure your data is in a **collection**. You can always manually create the collection from separate files:
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
>    In the further steps, this dataset collection will be referred to as `input` (and we recommend naming this collection like that to avoid confusion).
>
> 4. Import the following extra files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/record/7890956/files/reference_alkanes.csv
>    https://zenodo.org/record/7890956/files/reference_spectral_library.msp
>    https://zenodo.org/record/7890956/files/sample_metadata.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
> 
>    Please pay attention to the format of all uploaded files, and make sure they were correctly imported.
>
>    {% snippet faqs/galaxy/datatypes_understanding_datatypes.md %}
>
>    > <comment-title> The extra files </comment-title>
>    >
>    > The three additional files contain the **reference alkanes**, the **reference spectral library**, and the **sample metadata**. Those files are auxiliary inputs used in the data processing and contain either extra information about the samples or serve as reference data for indexing and identification.
>    > 
>    > The **list of alkanes** (`.tsv` or `.csv`) with retention times and carbon number or retention index is used to compute the retention index of the deconvoluted peaks. The alkanes should be measured in the same batch as the input sample collection.
>    > 
>    > The **reference spectral library** (`.msp`) is used for the identification of spectra. It contains the recorded and annotated mass spectra of chemical standards, ideally from a similar instrument. The unknown spectra which can be detected in the sample can then be confirmed via comparison with this library. The specific library is an in-house library of metabolite standards measured at RECETOX using a GC Orbitrap.
>    > 
>    > The **sample metadata** (`.csv` or `.tsv`) is a table containing information about our samples. In particular, the tabular file contains for each sample its associated sample name, class (QC, blank, sample, etc.), batch number, and injection order. It is possible to add more columns to include additional details about the samples.
>    {: .comment}
>
{: .hands_on}

As a result of this step, we should have in our history a green Dataset collection with three `.raw` files as well as three separate files with reference alkanes, reference spectral library, and sample metadata.

## Convert the raw data to mzML

Our input data are in `.raw` format, which is not suitable for the downstream tools in this tutorial. We use the tool {% tool [msconvert](toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2) %} to convert our samples to the appropriate format (`.mzML` in this case).

> <hands-on-title> Convert the raw data to mzML </hands-on-title>
>
> 1. {% tool [msconvert](toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.20287.2) %} with the following parameters:
>    - {% icon param-collection %} *"Input unrefined MS data"*: `input` (Input dataset collection)
>    - *"Do you agree to the vendor licenses?"*: `Yes`
>    - *"Output Type"*: `mzML`
>    - In *"Data Processing Filters"*:
>        - *"Apply peak picking?"*: `Yes` (This option will compute centroids in the m/z domain.)
>
>    > <comment-title> Centroids </comment-title>
>    >
>    > `msconvert` with selected parameters computes centroids in the m/z domain. MS instruments continuously sample and record signals. Therefore, a mass peak for a single ion in one spectrum consists of multiple intensities at discrete m/z values. Centroiding is the process of reducing these mass peaks to a single representative signal, the centroid. This results in much smaller file sizes without losing too much information. In the further steps, **XCMS** uses the _centWave_ chromatographic peak detection algorithm, which was designed for centroided data. That is the reason we perform the centroiding prior to chromatographic peak detection in this step.
>    {: .comment}
>
{: .hands_on}

## Create the XCMS object

The first part of data processing is using the **XCMS** tool to detect peaks in the MS signal. For that, we first need to take the `.mzML` files and create a format usable by the **XCMS** tool. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} ({% cite gatto2012msnbase %}. {% cite gatto2020msnbase %}) takes as input our files and prepares `RData` files for the first **XCMS** step.

> <hands-on-title> Create the XCMS object </hands-on-title>
>
> 1. {% tool [MSnbase readMSData](toolshed.g2.bx.psu.edu/repos/lecorguille/msnbase_readmsdata/msnbase_readmsdata/2.16.1+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"File(s) from your history containing your chromatograms"*: `input.mzML` (output of **msconvert** {% icon tool %})
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
>    > <comment-title> Output - `input.raw.RData` </comment-title>
>    >
>    > Collection of `rdata.msnbase.raw` files. `Rdata` file that is necessary in the next step of the workflow. These serve for an internal R representation of **XCMS** objects needed in the further steps.
>    {: .comment}
{: .hands_on}

# Peak detection using XCMS

The first step in the workflow is to detect the peaks in our data using **XCMS**. This part, however, is covered by a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html). Although the tutorial is dedicated to LC-MS data, it can also be followed for our GC-MS data. Therefore, in this section, we do not explain this part of the workflow in detail but rather refer the reader to the dedicated tutorial. Please also pay attention to the parameter values for individual Galaxy tools, as these can differ from the referred tutorial and are adjusted to our dataset.

> <details-title> Skip this step </details-title>
> 
> Since this step is already covered in a [separate tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-preprocessing/tutorial.html), it is possible to skip it. Instead, you can continue with [Peak deconvolution]({{ site.baseurl }}/topics/metabolomics/tutorials/gc_ms_with_xcms/tutorial.html#peak-deconvolution) step using a preprocessed **XCMS** object file prepared for you.
>
> > <hands-on-title> Upload data </hands-on-title>
> >
> > 1. Create a new history for this tutorial
> >
> >    {% snippet faqs/galaxy/histories_create_new.md %}
> >
> > 2. Import the following files from [Zenodo]({{ page.zenodo_link }}):
> >
> >    ```
> >    https://zenodo.org/record/7890956/files/XCMS_object.rdata.xcms.fillpeaks
> >    ```
> >
> >    {% snippet faqs/galaxy/datasets_import_via_link.md %}
> >
> >    The format of uploaded file containing **XCMS** object should be `rdata.xcms.fillpeaks`.
> >
> >    {% snippet faqs/galaxy/datatypes_understanding_datatypes.md %}
> >
> {: .hands_on}
>
{: .details}

## Peak picking

The first step is to extract peaks from each of your data files independently. For this purpose, we use the _centWave_ chromatographic peak detection algorithm implemented in {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %}.

> <hands-on-title> Peak picking </hands-on-title>
>
>  {% tool [xcms findChromPeaks (xcmsSet)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_xcmsset/abims_xcms_xcmsSet/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.RData` (output of **MSnbase readMSData** {% icon tool %})
>    - *"Extraction method for peaks detection"*: `CentWave - chromatographic peak detection using the centWave method`
>        - *"Max tolerated ppm m/z deviation in consecutive scans in ppm"*: `3.0`
>        - *"Min,Max peak width in seconds"*: `1,30`
>        - In *"Advanced Options"*:
>            - *"Prefilter step for for the first analysis step (ROI detection)"*: `3,500`
>            - *"Noise filter"*: `1000`
>  
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Determining shared ions

At this step, you obtain a dataset collection containing one `RData` file per sample, with independent lists of ions. Next, we want to identify the ions shared between samples. To do so, first, you need to group your individual `RData` files into a single one.

> <hands-on-title> Merging files </hands-on-title>
>
>  {% tool [xcms findChromPeaks Merger](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_merge/xcms_merge/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"RData file"*: `input.raw.xset.RData` (output of **xcms findChromPeaks (xcmsSet)** {% icon tool %})
>    - {% icon param-file %} *"Sample metadata file"*: `sample_metadata.tsv` (Input dataset)
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

Now we can proceed with the grouping and determining shared ions among samples. The aim of this step, called _grouping_, is to obtain a single matrix of ions’ intensities across all samples.

> <hands-on-title> Grouping peaks </hands-on-title>
>
>  {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.RData` (output of **xcms findChromPeaks Merger** {% icon tool %})
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `3.0`
>        - *"Minimum fraction of samples"*: `0.9`
>        - *"Width of overlapping m/z slices"*: `0.01`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Retention time correction

A deviation in retention time occurs from one sample to another, especially when you inject large sequences of samples. This step aims at correcting retention time drift for each peak among samples.

> <hands-on-title> Retention time correction </hands-on-title>
>
>  {% tool [xcms adjustRtime (retcor)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_retcor/abims_xcms_retcor/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.RData` (output of **xcms groupChromPeaks (group)** {% icon tool %})
>    - *"Method to use for retention time correction"*: `PeakGroups - retention time correction based on aligment of features (peak groups) present in most/all samples.`
>        - *"Minimum required fraction of samples in which peaks for the peak group were identified"*: `0.7`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Second round of determining shared ions

By applying retention time correction, the used retention time values were modified. Consequently, applying this step to your data requires completing it with an additional _grouping_ step.

> <hands-on-title> Grouping peaks </hands-on-title>
>
>  {% tool [xcms groupChromPeaks (group)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_group/abims_xcms_group/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.RData` (output of **xcms adjustRtime (retcor)** {% icon tool %})
>    - *"Method to use for grouping"*: `PeakDensity - peak grouping based on time dimension peak densities`
>        - *"Bandwidth"*: `3.0`
>        - *"Minimum fraction of samples"*: `0.9`
>        - *"Width of overlapping m/z slices"*: `0.01`
>    - *"Get the Peak List"*: `Yes`
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>        - *"If NA values remain, replace them by 0 in the dataMatrix"*: `No`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

## Integrating areas of missing peaks

At this point, the peak list may contain `NA` values when peaks were not considered in only some of the samples in the first peak-picking step. In this step, we will integrate the signal in the m/z-rt area of an ion (chromatographic peak group) for samples in which no chromatographic peak for this ion was identified.

> <hands-on-title> Integrating areas of missing peaks </hands-on-title>
>
>  {% tool [xcms fillChromPeaks (fillPeaks)](toolshed.g2.bx.psu.edu/repos/lecorguille/xcms_fillpeaks/abims_xcms_fillPeaks/3.12.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"RData file"*: `xset.merged.groupChromPeaks.adjustRtime.groupChromPeaks.RData` (output of **xcms groupChromPeaks (group)** {% icon tool %})
>    - In *"Peak List"*:
>        - *"Convert retention time (seconds) into minutes"*: `Yes`
>
>    You can leave the other parameters with their default values.
>
>    > <comment-title> Output </comment-title>
>    >
>    > After the `fillChromPeaks` step, you obtain your final intensity table. At this step, you have everything mandatory to begin analysing
>    > your data:
>    >  - A *sampleMetadata* file (if not done yet, to be completed with information about your samples)
>    >  - A *dataMatrix* file (with the intensities)
>    >  - A *variableMetadata* file (with information about ions such as retention times, m/z) 
>    {: .comment}
>
{: .hands_on}

# Peak deconvolution

The next step is deconvoluting the detected peaks in order to reconstruct the full spectra of the chemical compounds present in the sample, separated by the chromatography and ionized in the mass spectrometer. {% tool [RAMClustR](toolshed.g2.bx.psu.edu/repos/recetox/ramclustr/ramclustr/1.3.0+galaxy0) %} is used to group features based on correlations across samples in a hierarchy, focusing on consistency across samples. While each feature is typically derived from a single compound and represents a fragment, the whole mass spectrum can be used to more accurately identify the precursor compound or molecular ion. **RAMClustR** uses a novel grouping method that operates in an unsupervised manner to group peaks into spectra without relying on the predicting in-source phenomena (in-source fragments or adduct formation) or fragmentation mechanisms.

> <details-title> RAMClust method </details-title>
> 
> The RAMclust similarity scoring utilises a Gaussian function, allowing flexibility in tuning correlational and retention time similarity decay rates independently, based on the dataset and the acquisition instrumentation. The correlational relationship between two features can be described by either MS-MS, MS-idMS/MS, or idMS/MS-idMS/MS values, and we support several correlation methods (e.g. Pearson’s method) to calculate similarity. These are parametrised by `sigma_t` and `sigma_r`, Gaussian tuning parameters of retention time similarity and correlational score, respectively, between feature pairs.
> 
> Similarities are then converted to dissimilarities for clustering. The similarity matrix is then clustered using one of the available methods, such as average or complete linkage hierarchical clustering. The dendrogram is cut using the `cutreeDynamicTree` function from the package `dynamicTreeCut`. For this application, the minimum module size is set to 2, dictating that only clusters with two or more features are returned, as singletons are impossible to interpret intelligently.
> 
> Cluster membership, in conjunction with the abundance values from individual features in the input data, is used to create spectra. The mass to charge ratio is derived from the feature m/z, and the abundance for each m/z in the spectrum is derived from the weighted mean of the intensity values for that feature. These spectra are then exported as an `.msp` formatted file, which can be directly imported by **NIST MSsearch**, or used as input for **MassBank** or **NIST msPepSearch** batch searching.
>
{: .details}

> <hands-on-title> Peak deconvolution </hands-on-title>
>
> 1. {% tool [RAMClustR](toolshed.g2.bx.psu.edu/repos/recetox/ramclustr/ramclustr/1.3.0+galaxy0) %} with the following parameters:
>    - *"Choose input format:"*: `XCMS`
>        - In *"Input MS Data as XCMS"*:
>            - {% icon param-file %} *"Input XCMS"*: `xset.merged.groupChromPeaks.adjustRtime.groupChromPeaks.fillChromPeaks.RData` (output of **xcms fillChromPeaks (fillPeaks)** {% icon tool %})
>        - In *"General parameters"*:
>            - *"Sigma r"*: `0.7`
>            - *"Maximum RT difference"*: `10.0`
>    - In *"Clustering"*:
>        - *"Minimal cluster size"*: `5`
>        - *"Maximal tree height"*: `0.9`
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

The spectral data comes as an `.msp` file, which is a text file structured according to the **NIST MSSearch** spectra format. `.msp` is one of the generally accepted formats for mass spectral libraries (or collections of unidentified spectra, so called spectral archives), and it is compatible with lots of spectra processing programmes (MS-DIAL, NIST MS Search, AMDIS, etc.). Because `.msp` files are text-based, they can be viewed as simple `txt` files:

> <hands-on-title> Data Exploration </hands-on-title>
>
> Click *"View data"* {% icon galaxy-eye %} icon next to the dataset in the Galaxy history. The contents of the file would look like this:
> 
> 
> {% snippet faqs/galaxy/datasets_icons.md %}
> 
> 
> ```
> NAME:C001
> IONMODE:Negative
> SPECTRUMTYPE:Centroid
> RETENTIONTIME:383.27
> Num Peaks:231
> 217.1073 64041926
> 243.0865 35597866
> 257.1134 31831229
> 224.061 27258239
> 258.11 24996353
> 241.0821 23957171
> 315.1188 13756744
> ...
>
> NAME:C002
> IONMODE:Negative
> SPECTRUMTYPE:Centroid
> RETENTIONTIME:281.62
> Num Peaks:165
> 307.1573 299174880
> 147.0654 298860831
> 149.0447 287809889
> 218.1066 118274758
> 189.076 112486871
> 364.1787 75134143
> 191.0916 52526567
> 308.1579 52057158
> ...
> ```
>
> > <details-title> Negative ion mode </details-title>
> >
> > You might wonder how can the ionisation mode (_IONMODE_) for GC-MS data be negative when using electron impact (EI+) ionization. This is, of course, incorrect. This is actually just a default behaviour of **RAMClustR**. We can optionally change this by providing **RAMClustR** an experiment definition file. This file can be created manually or using the {% tool [RAMClustR define experiment](toolshed.g2.bx.psu.edu/repos/recetox/ramclustr_define_experiment/ramclustr_define_experiment/1.0.2) %} tool. There we can specify annotations such as what instrument we used or ionisation mode (which was EI+ in our case), and this will be transfered to the `.msp` file. Finally, we can provide such a file as an input to **RAMClustR** in the _Extras_ inputs section.
> >
> {: .details}
>
{: .hands_on}

`.msp` files can contain one or more mass spectra, these are split by an empty line. The individual spectra essentially consist of two sections: metadata and peaks. The metadata consists of compound name, spectrum type (which is centroid in this case), ion mode, retention time, and the number of *m/z* peaks. If the compound has not been identified, as in our case, the **NAME** can be any arbitrary string. It is best, however, if that string is unique within that `.msp` file. The metadata fields are usually unordered, so it is quite common for one `.msp` file to contain **NAME** as the first metadata key and for another `.msp` to have this key somewhere in the middle. The keys themselves also aren't strictly defined and can have different names (e.g., **compound_name** instead of **NAME**). The `msp` format is inherently not standardized and various flavours exist.

However, as we can observe, the metadata part is rather incomplete. We would like to gather more information about the detected spectra and identify the specific compounds corresponding to them.

The second output file is the so called **Spec Abundance** table, containing the expression of all deconvoluted spectra across all samples. It is the corresponding element to the peak intensity table obtained from **XCMS**, only for the whole deconvoluted spectrum. This file is used for further downstream processing and analysis of the data and eventual comparisons across different sample types or groups. For more information on this downstream data processing see this [tutorial]({{ site.baseurl }}/topics/metabolomics/tutorials/lcms-dataprocessing/tutorial.html).

# Retention index calculation

The retention index ({% cite van1963generalization %}) is a way how to convert equipment- and experiment-specific retention times into system-independent normalised constants for GC-based experiments. The retention index of a compound is computed from the retention time by interpolating between the retention times adjacent alkanes which are assigned a fixed retention index. This can be different for the individual chromatographic system, but the derived retention indices are independent and allow comparing values measured by different analytical laboratories.

We use the package {% tool [RIAssigner](toolshed.g2.bx.psu.edu/repos/recetox/riassigner/riassigner/0.3.4+galaxy1) %} to compute retention indices for files in the `.msp` format using an indexed reference list of alkanes in `.csv` or `.msp` format. The output follows the same format as the input but with added retention index values. These can be used at a subsequent stage to improve compound identification ({% cite kumari2011applying %}). Multiple computation methods (e.g. piecewise-linear or cubic spline) are supported by the tool.

> <hands-on-title> Retention index calculation </hands-on-title>
>
> 1. {% tool [RIAssigner](toolshed.g2.bx.psu.edu/repos/recetox/riassigner/riassigner/0.3.4+galaxy1) %} with the following parameters:
>    - In *"Query dataset"*:
>        - {% icon param-file %} *"Query compound list"*: `Mass spectra from RAMClustR` (output of **RAMClustR** {% icon tool %})
>    - In *"Reference dataset"*:
>        - {% icon param-file %} *"Reference compound list"*: `reference_alkanes.csv` (Input dataset)
>
>    You can leave the other parameters with their default values.
>
>    > <comment-title> Minutes vs. seconds </comment-title>
>    >
>    > You might notice that in the last **XCMS** step, we converted the retention times to minutes. However, in this step, we are using seconds again. The reason is that **RAMClustR** converted them internally again to seconds. Regarding the reference compound list, this database already has its retention times in seconds.
>    {: .comment}
>
>    > <details-title> Kováts method </details-title>
>    >
>    > The Kováts retention index (or Kováts index) of a compound is its retention time normalized to the retention times of adjacently eluting n-alkanes. It is based on the fact that the logarithm of the retention time converges to the number of carbon atoms in the alkane. For an isothermal chromatogram, you use the following equation to calculate the Kováts index: 
>    > 
>    > $$I = 100 \times \frac{\mathtt{log}~t_x - \mathtt{log}~t_n}{\mathtt{log}~t_{n+1} - \mathtt{log}~t_n}$$ 
>    > 
>    > where $$t_n$$ and $$t_{n+1}$$ are retention times of the reference n-alkane hydrocarbons eluting immediately before and after chemical compound *X* and $$t_x$$ is the retention time of compound *X*.
>    > 
>    > In practice, first, you run a chromatogram of a standard alkane mixture in the range of interest. Then you do a co-injection of your sample with the standard alkanes. The main advantage of this method is reproducibility.
>    >
>    {: .details}
>
{: .hands_on}

# Identification

To annotate and putatively identify the deconvoluted spectra, we compare them with a reference spectral library. This library contains spectra of standards measured on the same instrument for optimal comparability. The **matchms** package is used for spectral matching. It was built to import and apply different similarity measures to compare large numbers of spectra. This includes common cosine scores but can also easily be extended with custom similarity measures. For more information on how to use matchms from Python check out this [blog](https://blog.esciencecenter.nl/build-your-own-mass-spectrometry-analysis-pipeline-in-python-using-matchms-part-i-d96c718c68ee).

> <details-title> Reference spectral library </details-title>
> 
> Fragmentation patterns of ions (spectra) are widely used in the process of compound identification for EI+ ionization in complex mixtures. The number of identifiable compounds and associated data keeps growing with the increasing sensitivity and resolution of mass spectrometers. These data are clustered into collections of chemical structures and their spectra, also called spectral libraries ({% cite stein1995chemical %}, {% cite stein2012mass %}), which can be used for fast, reliable identifications for any compound whose fragmentation pattern is measured by the instrument. 
> 
> To identify the compounds after GC/MS analysis, the *library searching* is performed for every detected spectrum. This locates the most similar spectra in the reference library, providing a list of the potential identifications sorted by computed similarity. Indeed, a library search should also yield the confidence of compound identification, often expressed as a similarity score.
>
> Reference libraries' users usually expect their reliability to be quite high. However, there are many sources of error which we need to be aware of. A particularly error-prone aspect is annotation. Even an excellent spectrum becomes worthless for a wrongly annotated compound. A variety of computer-aided quality control measures can help to improve these issues, but libraries always require some level of expert human decision-making before a spectrum is added to the library. This is also important for redundant and unidentified spectra.
>
{: .details}


## Compute similarity scores

We use the cosine score with a greedy peak pairing heuristic to compute the number of matching ions with a given tolerance and the cosine scores for the matched peaks. Compared to more traditional methods of forward and reverse matching, **matchms** is performing matching on peaks present both in the measured sample and the reference data. This inherently yields higher scores than the forward and reverse scoring used by NIST.

> <details-title> Cosine score methods </details-title>
>
> One of the most common methods to compute the similarity score between two spectra is computing the cosine of the *angle* between them. Each spectrum can be represented as a vector with an axis along each of its *m/z* values. Then the cosine of the angle is basically a normalised *dot product* of these two vectors. The intensity and *m/z* weighting corrections have been used to optimise performance. For example, higher mass peaks can be given greater weight since they are more discriminating than lower mass peaks. *Reverse* scores assign no weight to peaks present only in the measured sample spectrum under the assumption that they arise from impurities.
>
{: .details}

> <hands-on-title> Compute similarity scores </hands-on-title>
>
> 1. {% tool [matchms similarity](toolshed.g2.bx.psu.edu/repos/recetox/matchms_similarity/matchms_similarity/0.20.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Queries spectra"*: `RI using kovats of Mass spectra from RAMClustR` (output of **RIAssigner** {% icon tool %})
>    - *"Symmetric"*: `No` (if we were to query our spectra agains itself, we would select `Yes`)
>        - {% icon param-file %} *"Reference spectra"*: `reference_spectral_library.msp` (downloaded file from Zenodo)
>    - In *"Algorithm Parameters"*:
>        - *"tolerance"*: `0.03`
>    - *"Apply RI filtering"*: `Yes` (we use this since our data is GC)
>
>    You can leave the other parameters with their default values.
>
>    > <comment-title> Used reference spectra </comment-title>
>    > The RECETOX Metabolome HR-[EI+]-MS library is a collection of mostly endogenous compounds from the MetaSci Human Metabolite Library. Analytes underwent methoximation/silylation prior to acquisition. Spectra were acquired at 70 eV on Thermo Fisher Q Exactive™ GC Orbitrap™ GC-MS/MS at 60000 resolving power.
>    {: .comment}
>
{: .hands_on}

> <details-title> Overview of the spectral similarity scores </details-title>
>
> ## Cosine Greedy
> The cosine score, also known as the dot product, is based on representing the similarity of two spectra through the cosine of an angle between the vectors that the spectra produce. Two peaks are considered as matching if their *m/z* values lie within the given tolerance. Cosine greedy looks up matching peaks in a "greedy" way, which does not always lead to the most optimal alignments.
>
> This score was among the first to be used for looking up matching spectra in spectral libraries and, to this day, remains one of the most popular scoring methods for both library matching and molecular networking workflows.
>
> ## Cosine Hungarian
> This method computes the similarities in the same way as the *Cosine Greedy* but with a difference in *m/z* peak alignment. The difference lies in that the Hungarian algorithm is used here to find matching peaks. This leads to the best peak pairs match but can take significantly longer than the "greedy" algorithm.
>
> ## Modified Cosine
> Modified Cosine is another, as its name states, representative of the family of cosine-based scores. This method aligns peaks by finding the best possible matches and considers two peaks a match if their *m/z* values are within tolerance before or after a mass-shift is applied. A mass shift is essentially a difference of precursor-*m/z* of two compared spectra. The similarity is then again expressed as a cosine of the angle between two vectors.
>
> ## Neutral Losses Cosine
> Neutral Loss metric works similarly to all described above with one major difference: instead of encoding the spectra as "intensity vs *m/z*" vector, it encodes it to an "intensity vs *Δm/z*", where delta is computed as an *m/z* difference between precursor and a fragment *m/z*. This, in theory, could better capture the underlying structural similarities between molecules.
>
{: .details}

## Format the output

The output of the previous step is a `json` file. This format is very simple to read for our computers, but not so much for us. We can use the {% tool [matchms scores formatter](toolshed.g2.bx.psu.edu/repos/recetox/matchms_formatter/matchms_formatter/0.20.0+galaxy0) %} to convert the data to a tab-separated file with a scores matrix.

The output table contains the scores and number of matched ions of the deconvoluted spectra with the spectra in the reference library. The raw output can be filtered to only contain the top matches (3 by default) and/or to contain only pairs with a score and number of matched ions larger than provided thresholds.

> <hands-on-title> Format the output </hands-on-title>
>
> 1. {% tool [matchms scores formatter](toolshed.g2.bx.psu.edu/repos/recetox/matchms_formatter/matchms_formatter/0.20.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Scores object"*: `CosineGreedy scores` (output of **matchms similarity** {% icon tool %})
>
>    You can leave the other parameters with their default values.
>
{: .hands_on}

> | query | reference | CosineGreedy_score | CosineGreedy_matches |
> |-------|-----------|---------|--------|
> | C044  | Guanine_3TMS    | 0.987    | 19    |
> | C068  | Tryptophan_3TMS | 0.981    | 9   |
> | C053  | Norleucine_2TMS | 0.980    | 12   |
> | ...   | ...             | ... | ...  |
{: .matrix}

At this stage, all steps are complete: we have the list of identified spectra corresponding to a compound from the reference database. Since this process is dependent on the parameters we used along the way, the result should be considered dependent on them. To enumerate this dependency, each potential compound has been assigned a confidence score.

# Conclusion

In this tutorial, we show how untargeted GC-MS data can be processed to obtain tentative identification of compounds contained in the analyzed samples.
Using **XCMS**, we obtain a peak and intensity table from our files which contains information about the detected ions (such as m/z and retention time) and their intensities across all samples.
Afterwards, we used **RAMClustR** to reconstruct the fragmentation spectra which we assume to originate from the analyzed compounds contained within our samples.
We use a correlation based approach which allows us to reconstruct also low-abundant features as long as their intensities are correlating across samples.
This method is different than the sample-wise deconvolution using **CAMERA** ({% cite Kuhl2012 %}).
To have more information available for compound identification, we computed the retention index using **RIAssigner** and added it to our deconvoluted spectra.
We then finally compare the deconvoluted spectra against the reference library using **matchms**, using a retention index threshold to pre-filter our matches and then computing the cosine similarity for the spectra within the retention index tolerance.

This workflow leverages the correlation of features across samples to improve the annotation performance, trying to capture even low abundant signals.
It is suitable for both high- and low-resolution GC-MS data acquired in either profile or centroid mode by adapting the individual tool parameters or in-/excluding individual steps which are specific to a certain data format.

Before further statistical analysis, the information from the scores table can be included in the library of deconvoluted spectra manually to add the putative identifications. The further processing can then leverage this information and the intensity information contained in the **SpecAbundance** output from **RAMclustR**.

Note that this tutorial doesn't cover all steps which are possible or compatible with this workflow. Other options include the {% tool [matchms filtering](toolshed.g2.bx.psu.edu/repos/recetox/matchms_filtering/matchms_filtering/0.17.0+galaxy0) %} tool with which you can normalize your spectra or remove low-abundant ions before matching.
Another option for normalization outside of the built-in normalization tools of **RAMClustR** is the {% tool [WaveICA](toolshed.g2.bx.psu.edu/repos/recetox/waveica/waveica/0.2.0+galaxy2) %} tool for correction of batch effects.


topic: metabolomics
tutorial: lcms-dataprocessing
title: Mass spectrometry: LC-MS data processing
difficulty: introductory
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: c0c2cd0623af5aab787af13f915f965e
last_modified: 2026-02-26T21:12:17.159819
zenodo_link: https://zenodo.org/record/5179809
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/metabolomics/tutorials/lcms-dataprocessing/tutorial.html

content:
Metabolomics is a *-omic* science known for being one of the most closely related to phenotypes.
It involves the study of different types of matrices, such as blood, urine, tissues, in various organisms including plants.
It focuses on studying the very small molecules which are called *metabolites*, to better understand matters linked to the metabolism.

Metabolomics analyses can be quite complex to conduct, especially when dealing with untargeted approaches.
**Liquid-Chromatography Mass Spectrometry** (LC-MS) is one of the three main technologies used to perform this kind of approach.
Data analysis for this technology requires a large variety of steps, ranging from extracting information from the raw data to statistical analysis and annotation.
One of these steps is called "data processing". It takes place after the pre-processing step (extraction of the peak list from raw data) and before any statistical analysis.
You can get an overview of a complete LC-MS untargeted metabolomic workflow by following [the dedicated training material]({% link topics/metabolomics/tutorials/lcms/tutorial.md %}).

After the pre-processing step, what you have at your disposal is a list of ions (hereafter in the form of a "variableMetadata" file) and the corresponding intensities ("dataMatrix" file).
What you may want now is to get some relevant information from your tables. However, your data may not be suitable yet for statistical analysis.
What should you do to ensure the quality of your tables? This tutorial will show you what the usual quality steps are.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Overview and data upload

Data processing covers a large range of actions. They can generally be described as transformation (*e.g.* normalisation) or filter (*e.g.* removal of unwanted ions).
In this tutorial we will focus on three main types of processing:
- Removing "trash" signals
- Correcting intensities
- Removing signals of insufficient quality

## Galaxy tools' workflow

The different Galaxy tools that will be used in this tutorial are given in the following workflow picture.

![The picture is composed of 3 main boxes linked by arrows from left to right. In each of the first and last boxes, labelled '3 tabulars', are found 3 files named sampleMetadata, variableMetadata and dataMatrix. In the middle box labelled 'Quality Control' is found the workflow used in this tutorial, represented as tiny boxes with tool names, linked with arrows following the tutorial order.](../../images/tutorial-lcms-proc-wf.png "The full tutorial workflow")

All these modules are part of the [Wokflow4Metabolomics](http://workflow4metabolomics.org/) tool suit ({% cite Giacomoni2014 %}, {% cite Guitton2017 %}).
They are compatible with the whole data analysis solution maintained by the W4M team.

> <comment-title>Workflow4Metabolomics public history</comment-title>
>
> This training material can be followed running it on any Galaxy instance holding the Galaxy modules needed.
> Nonetheless, if you happen to be a W4M user and do not want to run the hands-on yourself, please note that
> you can find the entire history in the 'published histories' section of usegalaxy.fr:
> [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing)
>
{: .comment}

## Dataset description

To illustrate the steps in this tutorial, a dataset has been built purposely.
It is composed of 3 files. They are text files of tables with tabulation as separator.

The *dataMatrix* file is a table containing the intensities of measured variables (ions, in lines) for every samples (in column).
The first column is for ions' identifiers while the first line is for samples' identifiers.

The *variableMetadata* file is a table containing information about the ions.
The first column is for ions' identifiers while the other columns gather information about m/z and retention time (rt).

The *sampleMetadata* file is a table containing information about the samples.
The first column is for samples' identifiers while the other columns gather analytical and biological information
such as the order of injection in the analytical sequence and the biological groups of interest for the supposed study.

The simulated design is composed of 30 biological samples (tagged "sample" in the *sampleType* column),
completed with 8 quality-control pooled samples (tagged "pool" in the *sampleType* column)
and 6 extraction solvent samples (tagged "blank" in the *sampleType* column).
The samples have been supposedly injected in two distinct sequences (tagged "B1" and "B2" in the *batch* column),
the injection order being given in the *injectionOrder* column.
Two sample characteristics are given:
- The *Group* column represents two groups "A" and "B", supposedly two biological groups (*e.g.* phenotypes, treatment groups...).
- The *Osmo* column represents a measurement of supposed osmolarity, imagining that the samples may be urine samples.

## Data upload

To perform the different exercices of this tutorial, you need to create a new history and upload the dedicated dataset.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the 3 starting files in your history. Two possibilities:
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from [Zenodo](https://zenodo.org/record/5179809) using the URLs given below:
>
>    ```
> https://zenodo.org/record/5179809/files/Dataprocessing_dataMatrix.txt
> https://zenodo.org/record/5179809/files/Dataprocessing_sampleMetadata.txt
> https://zenodo.org/record/5179809/files/Dataprocessing_variableMetadata.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets for their names to be shorter: "dataMatrix", "sampleMetadata", "variableMetadata".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype is "tabular". If not, you may change it.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> > <tip-title>Comment to W4M users</tip-title>
> >
> > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history,
> > this step corresponds to the datasets number 1 to 3.
> {: .tip}
>
{: .hands_on}

# Removing "trash" signals

Data are often affected by various sources of unwanted variability. This can be found in your tables in different ways,
two common ones being the presence of unwanted ions, and the effect of biological or analytical variables on intensity measures.
This unwanted information can limit the effectiveness of statistical methods, leading sometimes to difficulties in revealing investigated effects.
Thus, identifying such variability can help analysing your data at its full potential.
Yet, getting rid of such information may not be a trivial problem. It requires different elements to be completed successfully.

In this section we will adress the question of "trash" signal filtering.
By this we mean ions that are present in the extracted peak list we have, but that do not correspond to relevant compounds to analyse.
This can be for example noise, or ions from compounds that are not present in the original biological samples.

To make it clearer, we will illustrate this with two examples: a "simple" one first and a more advanced one in a second time.

## Filtering signals at given retention times

When using a chromatography column for MS analysis, you may want to exclude some time ranges where you know the information found there is not of interest.
For example, you may want to exclude the dead volume, a calibration zone at the begining or the end, or to exclude a column flush.

In this tutorial, let's suppose the data are from some LC-QTOF analysis with a dead volume between 0 and 0.4 minutes and a column flush from 16 minutes.
Then we may want to exclude ions that may be found at theses specific retention time (rt) ranges.
A quick check at the variableMetadata file reveals that a retention time column is available ("rt") with values in minutes.
We can then use this column to filter the dataset.

> <hands-on-title>Using <b>Generic_filter</b> to filter ions found at specific retention times</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: the `dataMatrix` file
>    - {% icon param-file %} *"Sample metadata file"*: the `sampleMetadata` file
>    - {% icon param-file %} *"Variable metadata file"*: the `variableMetadata` file
>    - *"Deleting samples and/or variables according to Numerical values"*: `yes`
>        - In *"Identify the parameter to filter "*:
>            - *"1: Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `rt`
>                - *"Interval of values to remove"*: `extremity`
>                    - *"Remove all values lower than"*: `0.4`
>                    - *"And upper than"*: `16.0`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `no`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history,
>    > this step corresponds to the datasets number 4 to 6.
>    {: .tip}
>
{: .hands_on}

The **Generic Filter** {% icon tool %} tool generates 3 tables. They correspond to the 3 original tables,
except the content has been filtered according to the specified parameters.
By "filtering", it means removing from the dataset some variables (ions) and/or samples according to the defined filters.

> <question-title></question-title>
>
> What have changed between the 3 input tables and the 3 output ones?
>
> > <solution-title></solution-title>
> >
> >  The sampleMetadata output is identical to the input one since no filter on samples has been applied.
> >  The variableMetadata output contains less lines than the input one: some ions has been removed from the table.
> > The same happened with the dataMatrix table, corresponding ions being removed from the file.
> > Thus, these two files went from 201 lines (header included) to 172 lines, meaning only 171 ions remained in the dataset.
> >
> {: .solution}
>
{: .question}

This was a relatively easy task to do. The key point to perform the filter was to know the ranges of rt that needed to be filter,
since the information about rt values was already explicitly found in the dataset.
However, sometimes filtering requires more steps to be able to perform the wanted processing.
This will be illustrated with the next example.

## Using blanks to filter noise signals

As mentioned before, measured signals using mass spectrometry may not always be all of interest.
Some can be noise, or compounds not characteristic of the analysed biological samples.
There are several ways to reduce there impact on gathered data.
But one key point is always, as a starting point, to identify the issue.

In the previous example we knew there were retention time ranges of values where signals were not relevant.
But they may also be some signals that represent noise, found at no specific retention times.
So a question can be "how do we identify these signals?".

One possible procedure is the use of blanks to estimate the noise, as a reference.
The idea is to compare blanks’ intensities with other samples’ intensities.
If there is no subtantial difference, we can assume that the concerned signal is noise.

Of course, to be able to do so, you need to inject reference blanks along with your biological samples to get these intensities in your dataMatrix.
Thus, you need to anticipate it when you define your injection sequence.
Ideally, the blanks to use are extraction blanks, but you can also use injection solvent depending on your protocols.

When blanks are available in your dataset, another thing to consider is "how do I formalise the information needed to be able to filter?".
One common way to compare may be to set a minimum difference between means or medians, or to test for significant difference with a statistical test.
In this tutorial, we will choose to calculate a mean fold change between blanks and non-blank samples, and to set a threshold value for filtering.

The mean fold change ("fold") for each ion can be calculated using the **Intensity Check** {% icon tool %} tool.

> <hands-on-title>Using <b>Intensity Check</b> to generate the information needed to filter</hands-on-title>
>
> 1. {% tool [Intensity Check](toolshed.g2.bx.psu.edu/repos/melpetera/intensity_checks/intens_check/1.2.8) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `GF_dataMatrix` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_sampleMetadata` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `GF_variableMetadata` (output of the previous **Generic Filter** {% icon tool %} job)
>    - *"Computation method"*: `Between one class and all the remaining samples`
>        - *"Class column"*: `c5: sampleType`
>        - *"Selected class"*: `blank`
>        - *"Calculate the mean fold change"*: `Yes`
>            - *"Where should the class be placed for the mean fold change calculation?"*: `Denominator (Bottom)`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history,
>    > this step corresponds to the datasets number 7 and 8.
>    {: .tip}
>
{: .hands_on}

This module generates two outputs: a pdf file for plots (that is not of interest in our example)
and a table corresponding to the variableMetadata file used as input, completed with new columns depending on the selected parameters.
In our case, it generated a column named *fold_Other_VS_blank* that we will use for filtering.

> <question-title></question-title>
>
> What does a value of "4" mean in the *fold_Other_VS_blank* column?
> Remember that we used as parameter *"Selected class"*=`blank` and
> *"Where should the class be placed for the mean fold change calculation?"*=`Denominator (Bottom)`.
>
> > <solution-title></solution-title>
> >
> > Since *"Selected class"*=`blank`, the samples are devided in two groups: the blank samples on one hand
> > and all the other samples on the other hand.
> > To calculate a mean fold change (*i.e.* a ratio of means) between the two classes, we need to define which mean will be used as numerator
> > and which one will be used as denominator. Since we defined that the selected class should be the denominator, the values we will get are
> > (mean of non-blank samples)/(mean of blank samples).
> > Thus, a value of "4" for a given ion means that the ion has a mean 4-times higher in non-blank samples compared to blank samples.
> >
> {: .solution}
>
{: .question}

The **Intensity Check** {% icon tool %} tool generates additional information about your dataset, but do not perform any filter.
We can now define a threshold for filtering, and use the **Generic Filter** {% icon tool %} tool again to remove noise signal.
Here we will use a threshold value of "4". What we want is to remove ions having a fold value lower than 4,
meaning that the mean values of biological samples' intensities for theses ions are not sufficiently high compared to blanks to be considered
to be resulting from relevant compounds.

> <hands-on-title>Using <b>Generic_filter</b> to filter ions with insuffisant mean contrast with blank samples</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `GF_dataMatrix` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_sampleMetadata` (output of the previous **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `IC_GF_variableMetadata` (output of the previous **Intensity Check** {% icon tool %} job)
>    - *"Deleting samples and/or variables according to Numerical values"*: `yes`
>        - In *"Identify the parameter to filter "*:
>            - *"1: Insert Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `fold_Other_VS_blank`
>                - *"Interval of values to remove"*: `lower`
>                    - *"Remove all values lower than"*: `4.0`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `no`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 9 to 11.
>    {: .tip}
>
{: .hands_on}

As for the previous use of **Generic Filter**, we now have a dataset filtered from noise signals.

> <question-title></question-title>
>
> What is the current stage of our training dataset?
>
> > <solution-title></solution-title>
> >
> > From a dataset containing originally 200 ions, the two successive filters lead to a dataset containing only 131 ions.
> > The number of samples did not change, so we still have 30 biological samples, 8 QC pools and 6 blanks.
> >
> {: .solution}
>
{: .question}

At that point we will not use the blank samples anymore. However, before removing them from the dataset, it is always interesting to
get an overview of the dataset including blanks.

## Filtered dataset overview

The idea here is to have a glance at what the dataset looks like at a macro scale.
For this tutorial, we consider that at this step what we have in our dataset are ions only resulting from compounds originally present in the biological samples.
Thus, there are some assumptions we can begin to make, that we can try to check using graphical tools.

Here, we will use a tool that is called **Quality Metrics** to have an overview of our dataset through the generation of a pdf file containing some plots.

> <hands-on-title>Using <b>Quality Metrics</b> to get an overview of the dataset</hands-on-title>
>
> 1. {% tool [Quality Metrics](toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `GF_GF_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_GF_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `GF_IC_GF_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Coefficient of Variation"*: `no`
>    - *"Advanced parameters"*: `Use default`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 12 to 15.
>    {: .tip}
>
{: .hands_on}

This tool generates several outputs (some of them are going to be used in a later use of this module), however for now only the pdf output is of interest.

![Screenshot of the PDF output from Quality Metrics. It is a composition of several distinct plots. In particular there are a PCA plot and a total intensity per sample plot.](../../images/lcmsproc_QM.png "PDF output from Quality Metrics")

On the top left of the picture, we can see the two first components of a Principal Component Analysis (PCA) with the projection of samples on it, colored by sample type.
We can see that the main variability in the dataset distinguishes the blank samples (on the left in black) from the other ones (red for pools and green for samples).
This is awaited since blanks are supposed to have very low intensities compared to biological samples.
This observation is consistant with the top middle plot which represents the sum of intensities for each sample (plotting according to the injection order),
where blank samples have very low values.

This kind of plots is a good way to detect samples that may have abnormally low profiles:
if a sample is positioned at the same area as the blanks, it is suspicious and a special attention should be given to the concerned sample.

In this tutorial dataset, no atypical sample is observed, so no special attention needs to be paid on specific samples.

Since the blank samples are of no use anymore, we can remove them from the dataset.
Again, this can be done running the **Generic Filter** {% icon tool %} tool, using the *sampleType* column of the sampleMetadata table.

> <hands-on-title>Using <b>Generic_filter</b> to remove blank samples from the dataset</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `GF_GF_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `GF_GF_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `GF_IC_GF_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Deleting samples and/or variables according to Numerical values"*: `no`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `yes`
>        - In *"Removing a level in factor"*:
>            - *"1: Insert Removing a level in factor"*
>                - *"Name of the column to filter"*: `sampleType`
>                - *"Remove factor when"*: `blank`
>
>    > <comment-title></comment-title>
>    >
>    > The tabular outputs of the previous **Quality Metrics** job are not needed here.
>    > This is why what we use as input here are the outputs from the last **Generic Filter** job.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 16 to 18.
>    {: .tip}
>
{: .hands_on}

This step leads to a dataset containing 131 ions and 38 samples (from which 8 pools).

You may have noticed that from module to module, output names tend to become longer and longer.
To prevent very long and not-so-informative names due to successive use of modules, we highly recommand to regularly rename the outputs.
Here, we completed a first row of data processing with the filter of trash signals. Now is a good time for a little renaming.

> <hands-on-title>Rename the last 3 tables</hands-on-title>
>
> 1. Rename the tables for their names to be shorter: "RT_blank_Filter_dataMatrix", "RT_blank_Filter_sampleMetadata", "RT_blank_Filter_variableMetadata".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
{: .hands_on}

Now we are ready to investigate another type of data processing: correcting intensities.

# Signal drift and batch effect correction

Here we will illustrate an example of data processing that is not about filtering, but about correcting.
Indeed, when it comes to performing statistics comparing biological samples, it is crucial for the intensity values used to be reflecting
relevant variability between samples.

In untargeted Metabolomics studies, we manipulate measures that are relative abundancies.
Although we have no unit attached to the intensities, we at least assume that, for a given ion, an intensity value for one sample being higher
than the one from another sample means that the compound from which the ion is generated is originally found in higher abundance in the biological sample from
the first sample compared to the other.
This assumption may seem trivial, but truth is it is not when dealing with LC-MS data.

It is known that when injecting successively a large number of samples, the LC-MS system tends to get dirty. This may cause a measure drift.
To prevent inability to catch signal anymore, in case of large injection series, the sequence is generally divided into several batches and the source is cleaned between batches.
Unfortunately, these signal drift and batch design can add significant variability in the data.
It makes sample comparison complicated, since the assumption stated previously about abundance in biological samples may not be true anymore.
In case data is impacted by these effects, we need a way to normalise the data to obtain something reliable for statistical analysis.

## [Optional step] Checking batch effect on data

You may have noticed in the **Quality Metrics** output PDF that appart from blanks, samples seemed to be seperated in two groups.
Truth is it is indeed. However, the best would be to confirm it, in particular by finding out what these groups are linked to.
Here dices are already rolled, and given the fact that in this section we are adressing signal drift and batch effects,
one could suppose that the groups may have something to do with it.

Let's confirm it by performing a PCA, with the specificity to colour the sample projections according to the supposed effect: the batch information.
This can be done using the **Multivariate** {% icon tool %} tool.

> <hands-on-title>Using <b>Multivariate</b> to get a coloured score plot from a PCA</hands-on-title>
>
> 1. {% tool [Multivariate](toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `RT_blank_Filter_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `RT_blank_Filter_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `RT_blank_Filter_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Number of predictive components"*: `2`
>    - *"Advanced graphical parameters"*: `Full parameter list`
>        - *"Sample colors"*: `batch`
>        - *"Amount by which plotting text should be magnified relative to the default"*: `0.4`
>    - *"Advanced computational parameters"*: `Use default`
>
>    > <comment-title></comment-title>
>    >
>    > Changing the `0.8` value to `0.4` makes the text size of labels on the score plot smaller. Since the plot box size by default is tiny, this enhances the readability of the plot.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 19 to 22.
>    {: .tip}
>
{: .hands_on}

The output of **Multivariate** we will be interested in here is only the PDF file (Multivariate_figure.pdf).
Among the four plots displayed in the file, we can see at the bottom left corner that the first component clearly reveals two distinct groups.
These groups perfectly match the batch information we coloured the sample projections with.

This confirms that we indeed have at least a batch effect in the data.
To note, this comment is relevant thanks to the fact that supposedly we randomised the injection sequence.
This is essential to be able to efficiently separate any analytical effect from known biological variables when searching for effects.
If you look at the sampleMetadata file we originally imported, you can notice that the biological group (*A* and *B*) are equally found in the two batches,
with alternation all through the injection order.

## Performing the correction process

We previously saw that in data processing issues, it is sometimes crucial to anticipate what needs to be done before defining the injection sequence.
This is even more the case when dealing with batch effects and signal drifts.

There is a strategy, described initially by Van Der Kloet in 2009 ({% cite VdK2009 %}), that has made its way to nowadays procedures ({% cite Dunn2011 %}).
The idea is to model the signal drift and the batch effect level by using a reference that can be representative for every ions in the dataset.
Indeed, signal drift and batch effects can have very different impacts accross ions in a same dataset, so it is crucial to have a reference that is reliable
for each ion independantly.

Here the "universal" reference is obtained by using samples that are made by pooling together an extract of every samples in study.
Thus, these Quality-control pooled samples ('pools') contain all the compounds that are originally found in the samples, being a reference for a very wide range of ions.
By injecting these pools all through the injection sequence of the study samples, we obtain a reference for which the main variability observed is composed of
the analytical effects we want to correct, since biologically they are supposed to be identical.

> <details-title>More details about the theory</details-title>
>
> The procedure that is used to correct data in this tutorial is the following.
>
> For each ion independently, the normalisation process works as described in the following picture:
>
> ![An example plot with the normalisation formula](../../images/lcms_BC_theo.png "How this works")
>
> You can see a plot representing 6 sample measures (blue points) in a batch for a given extracted ion.
> The yellow line represents a model for the signal drift that can be used to normalise the data.
> This line is determined using the pools only (red squares).
> With the given formula, we can correct the signal drift.
>
> This work has to be done for each batch.
> Thus, if your sequence is divided into several batches, the idea is to obtain something similar to the following picture:
>
> ![A before/after plot showing an example of intensities before correction, with clear signal drift and batch effects, and after correction, where the effects have been erased thanks to the correction process](../../images/lcms_BC_theo2.png "Before/after picture")
>
{: .details}

In this tutorial, we have all the information we need to perform the correction:
- Pools representing the molecule diversity have been injected regularly all through the sequences.
- They are numerous enough in each batch for the regression to be reliable (well, at least we have enough of them to perform a linear modeling).
- The dataset contains the mandatory information needed in the sampleMetadata file: the injection order, the batches of analysis and the sample type (pool or sample).

We can then use the **Batch correction** {% icon tool %} tool to perform the correction.

> <hands-on-title>Using <b>Batch correction</b> to correct the data from signal drift and batch effet</hands-on-title>
>
> 1. {% tool [Batch_correction](toolshed.g2.bx.psu.edu/repos/melpetera/batchcorrection/Batch_correction/3.0.0) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `RT_blank_Filter_dataMatrix` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `RT_blank_Filter_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `RT_blank_Filter_variableMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - *"Type of regression model "*: `linear`
>        - *"Factor of interest "*: `Group`
>        - *"Level of details for plots "*: `complete`
>
>    > <comment-title></comment-title>
>    >
>    > The tabular outputs of the previous **Multivariate** job are not needed here.
>    > This is why what we use as input here are the outputs from the last **Generic Filter** job.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 23 to 25.
>    {: .tip}
>
{: .hands_on}

The tool generates 3 outputs:
- a dataMatrix file containing the corrected intensities
- a variableMetadata file corresponding to the input one completed with additional columns
- a pdf file with graphics to visualise the impact of the correction process

Depending on the regression model option chosen (linear/lowess/loess/all_loess_pool/all_loess_sample), the output details differ slightly.
Indeed, the previously described algorithm has been adapted following two strategies: one for the linear/lowess/loess (3L) options,
another for the all_loess_pool/all_loess_sample (all_loess) ones.

The main differences between the two strategies are the following:
- one part allows regression type choice and applies the normalisation depending on internal quality metrics
- the other part offers the possibility to apply the correction without pool samples.
For more information, do not hesitate to read the help section provided with the Galaxy tool.

In this tutorial example, we used the *linear* option which is part of the 3L strategy.
We could not use any of the other options because the other ones would need more pools per batch than what we have here.
When using the 3L strategy, in the PDF file you can have various graphics depending on the *Level of details for plots* option you choose.
Here we generated all the plots that are available. This means that the PDF file contains first one page per ion with before/after plots,
plus 3 additional pages at the end that display overall before/after indicators (sum of intensities, PCA plots, pools' coefficients of variation).

> <question-title></question-title>
>
> Look at the penultimate page of the PDF output (page n°133).
> This page represents before/after PCA plots with various colouring.
> We will only focus on the two first lines, in particular looking at the first column.
> The two corresponding plots represent PCA score plots before (first line) and after (second line) the correction process.
> The two plots are coloured according to the batch information (B1 and B2).
> What can you say about the batch effect after the correction process compared to before?
>
> > <solution-title></solution-title>
> >
> > Compared to the batch effect that was highlighted as the main effect in the data before the correction process,
> > we can see that after correction, the batch effect is no longer explaining the first PCA component.
> > This result is awaited since the reduction of the batch effect is one of the aim of the Batch correction step.
> >
> {: .solution}
>
{: .question}

Now that we corrected the data from the signal drift and batch effect, we consider that differences in intensity between samples
can be attributed mostly to differences in concentration of original compounds.
However, we know that what we converted to intensity values in our table is initially signal measurements, thus we suspect that
intensities may be noisy at some point.

# Filtering signals of insufficient quality

Similarly to many measuring devices, values that are generated by LC-MS devices are subject to noise.
In the specific case of untargeted Metabolomics, the values are relative and can have, in a single dataset, a very large range of scales.
It is difficult to quantify the noise, even so it is crucial to have a minimum confidence when searching for subtile differences between groups of individuals.
Thus, having a way to put appart variables (ions) that may be of insufficient quality is key to limit the risk of wrong conclusion while analysing the data.

Beyond the issue of noise, the quality of the variables in one's dataset is a key question. In particular, in untargeted Metabolomics the data has particularities that
make the endpoint analyses complicated, such as the disproportion between sample size and variable number, the complex redundancy and the noise issue already mentioned.
When quality indicators can be computed before any statistical analysis, it is a real advantage: it can improve the efficiency of later statistical analysis and
also help to select good candidates in biomarker approaches.

One common and powerful way to assess measurements quality is the use of pools.
Indeed, now that the data is corrected for signal drift, we expect to have stable intensities within pools since there is no biological variability among them.
We can evaluate the remaining variability in pools for each ion, and consider it to be noise and/or analytical effects that we may not have managed to remove for specific ions.
This enables the use of quality thresholds to get rid of ions we consider not to be sufficiently reliable regarding intensity values.

In this tutorial, we will illustrate the use of pools by computing coefficients of variation (CV).
This metric allows to evaluate the variability accross samples independantly of the unit scale.
This advantage enables the definition of a unique threshold value that can be applied to every ions whatever the scales.
CV are computed for each ion in your dataset, so you can filter ions using the chosen threshold.
Note: the greater the CV value, the greater the variability.

The first use of CV we will illustrate here is the computation of CVs for the pools only. The idea here is to grasp the pool variability.
It is expected that pools do not vary much since they correspond to the same biological content.
Thus, if your pools vary too much, you can suspect that the corresponding ion is not reliable enough.

The second use of CV we will use here is a comparison between pool CVs and sample CVs.
Indeed, we expect sample CVs to be higher than pool CVs since they share the same noise and analytical variability but also carry biological variability.

The **Quality Metrics** {% icon tool %} tool provides different CV indicators depending on what is in your sample list.
In particular, in the present case-study, it computes pool CVs along with a ratio between pool CVs and sample CVs.
These two indicators are automatically calculated as long as the input datasets provide the needed information
(a *sampleType* column in the sampleMetadata file with the information of *pool* and *sample*).
Thus, for these two indicators the tool can be run without specific parameters to set.

> <hands-on-title>Using <b>Quality Metrics</b> to compute the CV indicators</hands-on-title>
>
> 1. {% tool [Quality Metrics](toolshed.g2.bx.psu.edu/repos/ethevenot/qualitymetrics/quality_metrics/2.2.8) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `BC_linear_RT_blank_Filter_dataMatrix` (output of the **Batch_correction** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `RT_blank_Filter_sampleMetadata` (output of the last **Generic Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `BC_linear_RT_blank_Filter_variableMetadata` (output of the **Batch_correction** {% icon tool %} job)
>    - *"Coefficient of Variation"*: `no`
>    - *"Advanced parameters"*: `Use default`
>
>
>    > <comment-title></comment-title>
>    >
>    > You may have noticed that we already used this tool previously in the tutorial.
>    > Although CV columns were already available at that time, here you need to use this tool again since this time indicators will be computed on intensities
>    > corrected from the signal drifts and batch effects.
>    > Note that what we are going to use this time is the tabular output, but while you are at it you can always check the pdf file if you want.
>    {: .comment}
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 26 to 29.
>    {: .tip}
>
{: .hands_on}

The tool provides a variableMetadata tabular output, containing all the computed CV values.
For now no filter has been applied on the dataset. To remove the ions of insufficient quality, you will need the **Generic Filter** {% icon tool %} tool again.

But first, you need to determine the thresholds you want to use.
Here we have two indicators we want to use. One is the pool CVs, the other is the ratio between pool CVs and sample CVs.
This means that we will need to set two distinct thresholds, and to add two numerical filters in **Generic Filter** {% icon tool %}.

> <hands-on-title>Using <b>Generic Filter</b> to filter the dataset</hands-on-title>
>
> 1. {% tool [Generic_Filter](toolshed.g2.bx.psu.edu/repos/melpetera/generic_filter/generic_filter/2020.01) %} with the following parameters:
>    - {% icon param-file %} *"Data Matrix file"*: `BC_linear_RT_blank_Filter_dataMatrix` (output of the **Batch_correction** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `QM_RT_blank_Filter_sampleMetadata` (output of the last **Quality Metrics** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `QM_BC_linear_RT_blank_Filter_variableMetadata` (output of the last **Quality Metrics** {% icon tool %} job)
>    - *"Deleting samples and/or variables according to Numerical values"*: `yes`
>        - In *"Identify the parameter to filter "*:
>            - *"1: Insert Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `poolCV_over_sampleCV`
>                - *"Interval of values to remove"*: `upper`
>                    - *"Remove all values upper than"*: `1.0`
>            - {% icon param-repeat %} *"2: Insert Identify the parameter to filter "*
>                - *"On file"*: `Variable metadata`
>                - *"Name of the column to filter"*: `pool_CV`
>                - *"Interval of values to remove"*: `upper`
>                    - *"Remove all values upper than"*: `0.3`
>    - *"Deleting samples and/or variables according to Qualitative values"*: `yes`
>        - In *"Removing a level in factor"*:
>            - *"1: Insert Removing a level in factor"*
>                - *"Name of the column to filter"*: `sampleType`
>                - *"Remove factor when"*: `pool`
>
>
>    > <comment-title></comment-title>
>    >
>    > Here, we took the opportunity of this filter to remove the pools from the dataset.
>    > Indeed, pools will be no longer used in this tutorial's steps, so we can remove them from the dataset.
>    {: .comment}
>
>
> 2. Rename the output tables for their names to be shorter: "Filtered_dataMatrix", "Filtered_sampleMetadata", "Filtered_variableMetadata".
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 30 to 32.
>    {: .tip}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What does the *0.3* threshold mean in the hands-on exercise you just executed?
> 2. What does the *1.0* threshold mean in the hands-on exercise you just executed?
> 3. How many variables are left in your dataset? How many samples?
>
> > <solution-title></solution-title>
> >
> > 1. The *0.3* value corresponds to the maximum value kept in the dataset ('Interval of values to remove: *upper*') regarding the
> >    *pool_CV* column in your *Variable metadata* file.
> >    Thus, ions with pool CV values superior to 0.3 are excluded from the dataset.
> >    To note, pool CV values are commonly considered as reflecting unstable ions when superior to 0.3.
> > 2. The *1.0* value corresponds to the maximum value kept in the dataset ('Interval of values to remove: *upper*') regarding the
> >    *poolCV_over_sampleCV* column in your *Variable metadata* file. This means that any ion with a pool CV / sample CV ratio above 1
> >    (*i.e.* a pool CV greater than the sample CV) is discarded from the dataset.
> > 3. Filtering led to 102 ions and 30 samples.
> >
> {: .solution}
>
{: .question}

Presented threshold values here are common thresholds in Metabolomics studies.
Please note that thresholds should be chosen consistently regarding the LC-MS device caracteristics and the study objectives.
Thus you may need to be more strict in some cases and less strict in others.
However, being less strict than usual should always be appropriately justified.

# Study-specific data processing: example normalising data according to a non-analytical effect

The previous steps presented in this tutorial illustrated categories of data processing that need to be consider whatever the study in
untargeted LC-MS Metabolomics projects. In some cases the decision may be not to take actions, but in all cases the issue has to be considered
to apprehend the reliability of the data for further analyses.

Nonetheless, each study is unique and specific issues can arise that would need some additional data processing steps that are very specific to the study.
Thus, sticking to a standard data processing workflow may not be sufficient in some cases.
For example, one commonly considered data processing step in biomarker discovery studies is the reduction of redundancy,
which is a complex question that can be addressed using various strategies depending on the study objectives.

In this section, we will illustrate an example of study-specific data processing that involves a normalisation of the dataset.
This will follow the standard *Identify->Formalise->Handle* design that is common to any data processing approach.

## Identifying unwanted variability

First thing is of course to identify an issue in the data that needs to be handled.
In the previous steps, this was the "easy" part since we already had knowledge about what needed to be handled.
We adressed well-known problems, for which solutions may not always be easy, but at least we know there is something.

In study-specific data processing, it may not be that simple.
Usually, to see something you need to search for it. This means you have hypotheses prior to the data analysis.
However, sometimes unwanted variability is unexpected, making the search process difficult.

Fortunately, whether you have hypotheses or not, there is a common starting point in Metabolomics that can help you see whether an issue seems to need special attention.
This is the use of PCA, that we already saw in previous steps to highlight awaited effects.

In this tutorial, we constructed the dataset so we already know what is the variability to identify.
In a real project it can also be the case, or at least we can have hypotheses about effects we would not be supprised to observe.
In this tutorial example, we considered samples as being urine samples.
It is known that urine samples can vary a lot in total solute concentration, potentially limiting the possibility to detect specific biological effects.
Thus, even if it may not be the case, we can wonder whether the variability of concentration may have a huge effect on the metabolomic dataset we have.
This is why, while making a PCA, we may want to plot the first components and see whether the variability displayed seems to match the concentration.

In the sampleMetadata file provided in this tutorial, you can find a column named *Osmo* that is meant to correspond to measures of osmolality,
one way to represent the overall concentration of urine samples.
Let's try to colour the PCA score plots according to that variable. For this, we will use the **Multivariate** {% icon tool %} Galaxy module.

> <hands-on-title>Using <b>Multivariate</b> to visualise the two first components of a PCA</hands-on-title>
>
> 1. {% tool [Multivariate](toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10) %} with the following parameters:
>    - {% icon param-file %} *"Data matrix file"*: `Filtered_dataMatrix` (output of the last **Generic_Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Sample metadata file"*: `Filtered_sampleMetadata` (output of the last **Generic_Filter** {% icon tool %} job)
>    - {% icon param-file %} *"Variable metadata file"*: `Filtered_variableMetadata` (output of the last **Generic_Filter** {% icon tool %} job)
>    - *"Number of predictive components"*: `2`
>    - *"Advanced graphical parameters"*: `Full parameter list`
>        - *"Sample colors"*: `Osmo`
>        - *"Amount by which plotting text should be magnified relative to the default"*: `0.4`
>    - *"Advanced computational parameters"*: `Use default`
>
>
>    > <tip-title>Comment to W4M users</tip-title>
>    >
>    > In the [GTN_LCMSprocessing](https://workflow4metabolomics.usegalaxy.fr/u/peteram/h/gtnlcmsdataprocessing) history, this step corresponds to the datasets number 33 to 36.
>    {: .tip}
>
{: .hands_on}

Among the four output files generated by the tool, the one we are interested in is the *Multivariate_figure.pdf* file.
The PCA scores plot representing the projection of samples on the two first components is located at the bottom left corner:

![The picture represents the PCA scores plot of components t1 (32% of total variability) and t2 (9% of total variability) with sample labels coloured according to the values from the Osmo column of the sampleMetadata file. We observe a gradation in the colours of samples following the first bisector of the graphic.](../../images/lcmsproc_osmo.png "PCA scores plot")

Taking into account that the PCA was computed on Unit-Variance-scaled data (see the **Multivariate** {% icon tool %} tool's help section for 


