# tutorials

topic: statistics
tutorial: classification_regression
title: Machine learning: classification and regression
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 6c2fe5c9b9d2f97ee3c8f6b2fad13b9d
last_modified: 2026-02-26T21:12:18.166093
zenodo_link: https://zenodo.org/record/2579649
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/classification_regression/tutorial.html

content:
Machine learning is a subset of artificial intelligence (AI) that provides machines with the ability to automatically learn from data without being explicitly programmed. It is a combined field of computer science, mathematics and statistics to create a predictive model by learning patterns in a dataset. The dataset may have an output field which makes the learning process supervised. The [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning) methods in machine learning have outputs (also called as targets or classes or categories) defined in the datasets in a column. These targets can either be  integers or real (continuous) numbers. When the targets are integers, the learning task is known as classification. Each row in the dataset is a sample and the classification is assigning a class label/target to each sample. The algorithm which is used for this learning task is called a classifier. When the targets are real numbers, the learning task is called regression and the algorithm which is used for this task is called a regressor. We will go through classification first and look at regression later in this tutorial.

> <question-title></question-title>
>
> What are features and outputs/targets in a dataset?
>
> > <solution-title></solution-title>
> >
> > Features and targets of a breast cancer dataset:
> > ![Breast_cancer_targets](images/breast_cancer_classification.png "The image shows a breast cancer dataset with 8 rows and 10 columns. Each column is a feature and encodes a specific information about breast cancer or detected tumor. For example, the feature "clump thickness" gives information about whether the cells are mono or multi-layered. Similarly, other columns/features provide different information about the tumor. The last column is the "target" column which identifies whether the tumor is benign (0) or malignant (1) for each row of the dataset. Here, the targets are discrete which makes the learning task classification. In another task, if these targets contain continuous values, the learning task will become regression.")
> >
> {: .solution}
>
{: .question}


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Classification

A [classification](https://en.wikipedia.org/wiki/Statistical_classification) task assigns a category/class to each sample by learning a decision boundary in a dataset. This dataset is called a training dataset and contains samples and desired class/category for each sample. The training dataset contains "features" as columns and a mapping between these features and the target is learned for each sample. The performance of mapping is evaluated using a test dataset (which is separate from training dataset). The test dataset contains only the feature columns and not the target column. The target column is predicted using the mapping learned on the training dataset. In this tutorial, we will use a classifier to train a model using a training dataset, predict the targets for test dataset and visualize the results using plots.

> <comment-title></comment-title>
> The terms like 'targets', 'classes', 'categories' or 'labels' have been used interchangeably for the classification part of this tutorial. They contain identical meaning. For regression, we will just use 'targets'.
{: .comment}

![classification](images/classification.png "Classification of samples belonging to different classes (green for no tumor and violet for tumor). The line creates a boundary between two sets of samples and is learned by a classifier. When a new sample comes for which we do not know whether it is a tumor or no tumor, we use this decision boundary to compute the actual class.")

In figure [2](#figure-2), the line is a boundary which separates a class from another class (for example from tumor to no tumor). The task of a classifier is to learn this boundary, which can be used to classify or categorize an unseen/new sample. The line is the decision boundary. There are different ways to learn this decision boundary. If the dataset is linearly separable, linear classifiers can produce good classification results. But, when the dataset is complex and requires non-linear decision boundaries, more powerful classifiers like `support vector machine` or  `tree` or `ensemble` based classifiers may prove to be beneficial. In the following part, we will perform classification on breast cancer dataset using a linear classifier and then will analyze the results with plots. Let's begin by uploading the necessary datasets.

## Data upload

The datasets to be used for classification contain 9 features. Each feature contains some unique information about breast cancer including the thickness of clump, cell-size, cell-shape and so on. More information about the dataset can be found here - [a](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)) and [b](https://sites.google.com/a/googlesciencefair.com/science-fair-2012-project-64a91af142a459cfb486ed5cb05f803b2eb41354-1333130785-87/observations). In addition to these features, the training dataset contains one more column as the `target`. It has a binary value (0 or 1) for each row. `0` indicates no breast cancer (benign) and `1` (malignant) indicates breast cancer. The test dataset does not contain the `target` column (which should be predicted by a classifier). The third dataset contains all the samples from the test dataset, this time including the `target` column which is needed to compare between real and predicted targets.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following datasets and choose the type of data as `tabular`.
>
>    ```
>    https://zenodo.org/record/3248907/files/breast-w_targets.tsv
>    https://zenodo.org/record/3248907/files/breast-w_test.tsv
>    https://zenodo.org/record/3248907/files/breast-w_train.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename datasets to `breast-w_train`, `breast-w_test` and `breast-w_targets`.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

## Learn using training dataset

The training dataset is used for learning the associations between features and the targets. The classifier learns general patterns in a dataset and saves a trained model. This model can be used for classifying a new sample. In this step, we will use `breast-w_train` as the training dataset and apply [SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC) (support vector machine) classifier. It will learn features from the dataset and maps them to the targets. This mapping is called a trained model. The training step produces a model file of type `zip`.

> <hands-on-title>Train the model</hands-on-title>
>
> 1. **Support vector machines (SVMs) for classification** {% icon tool %} with the following parameters to train the classifier on training dataset:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Classifier type"*: `Linear Support Vector Classification`
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `breast-w_train`
>             - *"Does the dataset contain header"*: `Yes`
>             - *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - *"Type header name(s)"*: `target`
>             - {% icon param-file %} *"Dataset containing class labels"*: `breast-w_train`
>             - *"Does the dataset contain header"*: `Yes`
>             - *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - *"Select target column(s)"*: `target`
> 2. Rename the generated file to `model`
{: .hands_on}

> <question-title></question-title>
>
> What is learned by the classifier?
>
> > <solution-title></solution-title>
> >
> > Two attributes **coef_** and **intercept_** are learned by the classifier using the training dataset. The **coef_** contains importance weight for each feature and **intercept_** is just a
> > constant scalar. However, for different classifiers, these attributes are different. The attributes shown here are specific to the **Linear support vector** classifier.
> > These attributes are stored in the trained model and can be accessed by reading this file.
> > 
> {: .solution}
>
{: .question}

## Predict categories of test dataset

After the training process is complete, we can see the trained model file (`zip` file) which contains information about patterns in the form of weights. The trained model is used to predict the classes of the test (`breast-w_test`) dataset. It assigns a class (either tumor or no tumor) to each row in the `breast-w_test` dataset.

> <hands-on-title>Predict classes using the trained model</hands-on-title>
>
> 1. **Support vector machines (SVMs) for classification** {% icon tool %} with the following parameters to predict classes of test dataset using the trained model:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `model` file (output of the previous step)
>       - {% icon param-file %} *"Data (tabular)"*: `breast-w_test` file
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `predicted_labels`.
{: .hands_on}

## Visualise the predictions

We should evaluate the quality of predictions by comparing them against the true targets. To do this, we will use another dataset (`breast-w_targets`). This is the same as the test dataset (`breast-w_test`) but contains an extra `target` column containing the true classes of the test dataset. With the predicted and true classes, the learned model is evaluated to verify how correct the predictions are. To visualise these predictions, a plotting tool is used. It creates three plots - confusion matrix, precision, recall and F1 and ROC and AUC. We will mainly analyze the precision and recall plot.

> <hands-on-title>Check and visualize the predictions</hands-on-title>
> 1. **Plot confusion matrix, precision, recall and ROC and AUC curves** {% icon tool %} with the following parameters to visualise the predictions:
>    - {% icon param-file %} *"Select input data file"*: `breast-w_targets`
>    - {% icon param-file %} *"Select predicted data file"*: `predicted_labels`
>    - {% icon param-file %} *"Select trained model"*: `model`
>
{: .hands_on}

We will analyze the following plots:

1. [Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix):

    ![confusion_matrix](images/confusion_matrix.png "Confusion matrix of the predictions as a heatmap. The horizontal axis (x-axis) shows the predicted labels and the vertical axis (y-axis) shows the true labels. Each rectangular box shows a count of samples falling into the four output combinations (true class, predicted class) - (1, 0), (1, 1), (0, 1) and (0, 0). For a good prediction, the diagonal running from top-left to bottom-right should contain less number of samples (because it shows the counts of incorrectly predicted samples). Hovering over each box in Galaxy shows the true and predicted class labels and the count of samples.")

2. [Precision, recall and F1 score](https://en.wikipedia.org/wiki/Precision_and_recall):

    ![prf1_scores](images/precision_recall_f1.png "Precision, recall and F1 score. These scores determine the robustness of classification. It shows 1 for both the classes (shown along the horizontal axis) which means that all the samples belonging to these classes have been classified correctly. In the 'breast-w_test' dataset, all the samples with predicted class '0' (no tumor) have the true class as '0' (precision = 1.0) while not all samples with the predicted class as '1' have the true class as '1' (precision < 1.0) (see precision curve). The recall curve is the opposite of the precision curve. Not all the samples with the true class as '0' are predicted as class '0' (recall < 1.0) and all the samples with the true class as '1' are predicted as class '1' (recall = 1.0) (see recall curve). It is important to analyze the plot for any classification task to verify the accuracy across different classes which provides more information about the balanced or imbalanced accuracy across multiple classes present in the dataset.")

3. [Receiver operator characteristics (ROC) and area under ROC (AUC)](https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5):

    ![roc_scores](images/roc.png "Receiver operator characteristics (ROC) and area under ROC (AUC). The ROC curve is shown in blue. For a good prediction, it should be more towards the top-left of this plot. For a bad prediction, it is close to the orange line (y = x).")

Using these plots, the robustness of classification can be visualized.


## Summary
By following these steps from data upload until plotting, we have learned how to do classification and visualise the predictions using Galaxy's machine learning and plotting tools. A similar analysis can be performed using a different dataset or by using a different classifier. This machine learning suite provides multiple classifiers from linear to complex ones suited for different classification tasks. For example for a binary class classification, `support vector machine` classifier may perform well. It is recommended to try out different classifiers on a dataset to find the best one.


# Regression

For classification, the targets are integers. However, when the targets in a dataset are real numbers, the machine learning task becomes [regression](https://en.wikipedia.org/wiki/Regression_analysis). Each sample in the dataset has a real-valued output or target. Figure [6](#figure-6) shows how a (regression) curve is fitted which explains most of the data points (blue balls). Here, the curve is a straight line (red). The regression task is to learn this curve which explains the underlying distribution of the data points. The target for a new sample will lie on the curve learned by the regression task. A regressor learns the mapping between the features of a dataset row and its target value. Inherently, it tries to fit a curve for the targets. This curve can be linear or non-linear. In this part of the tutorial, we will perform regression on [body density](https://rstudio-pubs-static.s3.amazonaws.com/65314_c0d1e5696cdd4e93a3784ea67f9e3d34.html) dataset.

![regression](images/regression.png "Regression fit through data points. These data points are the targets of the training dataset. A curve is learned which best explains all these points. The target for a new sample will lie along the curve learned by the regression task.")


## Data upload

The dataset contains information about human [body density](https://rstudio-pubs-static.s3.amazonaws.com/65314_c0d1e5696cdd4e93a3784ea67f9e3d34.html). It includes 14 features like underwater body density, age, weight, height, neck circumference and so on. The target is the percent body fat. The aim of the task is to learn a mapping between several body features and fat content inside the human body. Using this learning, the body fat percentage can be predicted using other features. To carry out this task, we will need training and test datasets. Again, we will also prepare another test dataset with targets included to evaluate the regression performance. `body_fat_train` dataset is used as the training dataset and `body_fat_test` as the test dataset. The dataset `body_fat_test_labels` contains the true targets for the test dataset (`body_fat_test`).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial.
> 2. Import the following datasets and choose the type of data as `tabular`.
>
>    ```
>    https://zenodo.org/record/3248907/files/body_fat_train.tsv
>    https://zenodo.org/record/3248907/files/body_fat_test_labels.tsv
>    https://zenodo.org/record/3248907/files/body_fat_test.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename datasets to `body_fat_train`, `body_fat_test_labels` and `body_fat_test`.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}


## Learn from training dataset

To learn the mapping between several features and the targets, we will apply a regressor which is called
the [Gradient boosting regressor](http://scikit-learn.org/stable/modules/ensemble.html#regression). It is an ensemble-based regressor because its prediction is the collective performance of multiple weak learners (e.g. decision trees). It learns features from training dataset (`body_fat_train`) and maps all the rows to their respective targets (real numbers). The process of mapping gives a trained model.

> <hands-on-title>Train a model</hands-on-title>
>
> 1. **Ensemble methods for classification and regression** {% icon tool %} with the following parameters to train the regressor:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Select an ensemble method"*: `Gradient Boosting Regressor`
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `body_fat_train`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `target`
>             - {% icon param-file %} *"Dataset containing class labels"*: `body_fat_train`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - {% icon param-text %} *"Select target column(s)"*: `target`
> 2. Rename the generated file to `model`.
{: .hands_on}

> <question-title></question-title>
>
> What is learned by the regressor?
>
> > <solution-title></solution-title>
> >
> > Unlike the **Linear support vector** classifier (used for classification in the first part of the tutorial) which learned only two attributes,
> > the **Gradient boosting** regressor learns multiple attributes such as **feature_importances_** (weight for each feature/column),
> > **oob_improvement_** (which stores incremental improvements in learning), **estimators_** (collection of weak learners) and a few more.
> > These attributes are used to predict the target for a new sample and are stored in the trained model. They can be accessed by reading this file.
> > 
> {: .solution}
>
{: .question}

## Predict using test dataset

After learning on the training dataset, we should evaluate the performance on the test dataset to know whether the algorithm learned general patterns from the training dataset or not. These patterns are used to predict a new sample and a similar accuracy is expected. Similar to the classification task, the trained model is evaluated on `body_fat_test` which predicts a target value for each row. The predicted targets are compared to the expected targets to measure the robustness of learning.

> <hands-on-title>Predict targets using the model</hands-on-title>
>
> 1. **Ensemble methods for classification and regression** {% icon tool %} with the following parameters to predict targets of test dataset using the trained model:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `model`
>       - {% icon param-file %} *"Data (tabular)"*: `body_fat_test`
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `predicted_data`.
{: .hands_on}

## Visualise the prediction

We will evaluate the predictions by comparing them to the expected targets.

> <hands-on-title>Check and visualize the predictions</hands-on-title>
> 1. **Plot actual vs predicted curves and residual plots** {% icon tool %} with the following parameters to visualise the predictions:
>    - {% icon param-file %} *"Select input data file"*: `body_fat_test_labels`
>    - {% icon param-file %} *"Select predicted data file"*: `predicted_data`
{: .hands_on}

The visualization tool creates the following plots:

1. True vs predicted targets curves:

    ![true_predicted](images/true_pred_curves.png "True vs predicted targets curves. The corresponding points in both these curves should be close to each other for a good regression performance. We can see that the plot shows this behaviour.")


2. [Scatter plot](https://towardsdatascience.com/everything-you-need-to-know-about-scatter-plots-for-data-visualisation-924144c0bc5) for true vs. predicted targets:

    ![scatter_plot](images/true_vs_pred_scatter.png "Scatter plot for true vs. predicted targets. The plot shows the performance of the regression task. The data points (blue) lie along the orange curve (y = x) which shows that the true and predicted values are close. The more points are aligned along the x = y line, the better the prediction. The R2 (coefficient of determination) score (0.98) is close to the best possible score of 1.0. The RMSE (root mean squared error) is also small (1.32) which also tells us that the predicted targets are close to the true targets.")

3. [Residual plot](http://docs.statwing.com/interpreting-residual-plots-to-improve-your-regression/) between residual (predicted - true) and predicted targets:

    ![residual_plot](images/residual_plot.png "Residual plot between residual (predicted - true) and predicted targets. The plot shows a random pattern of points. For a good regression performance, this plot should exhibit a random pattern and the points should be symmetrically distributed along the y=0 line.")

These plots are important to visualize the quality of regression and the true and predicted targets - how close or far they are from each other. The closer they are, the better the prediction.

## Summary
By following these steps, we learned how to perform regression and visualise the predictions using Galaxy's machine learning and plotting tools. The features of the training dataset are mapped to the real-valued targets. This mapping is used to make predictions on an unseen (test) dataset. The quality of predictions is visualised using a plotting tool. There are multiple other regression algorithms, few are simpler to use (with fewer parameters) and some are powerful, which can be tried out on this dataset and on other datasets as well.

# Conclusion
We learned how to perform classification and regression using different datasets and machine learning tools in Galaxy. Moreover, we visualized the results using multiple plots to ascertain the robustness of machine learning tasks. There are many other classifiers and regressors in the machine learning suite which can be tried out on these datasets to find how they perform. Different datasets can also be analysed using these classifiers and regressors. The classifiers and regressors have lots of parameters which can be altered while performing the analyses to see if they affect the prediction accuracy. It may be beneficial to perform hyperparameter search to tune these parameters of classifiers and regressors for different datasets. Some data pre-processors can also be used to clean the datasets.


topic: statistics
tutorial: intro-to-ml-with-python
title: Foundational Aspects of Machine Learning using Python
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: ["python-basics", "python-warmup-stat-ml"]
tags_json: ["elixir", "ai-ml"]
content_hash: 6d4861a5262635150bf857f32debd8aa
last_modified: 2026-02-26T21:12:18.211354
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/intro-to-ml-with-python/tutorial.html

content:
Machine Learning is a subset of artificial intelligence that involves training algorithms to learn patterns from data and make predictions or decisions without being explicitly programmed. It has revolutionized various fields, from healthcare and finance to autonomous vehicles and natural language processing.

This tutorial is designed to equip you with essential knowledge and skills in Machine Learning, setting a strong foundation for exploring advanced topics such as Deep Learning.

The objective of this tutorial is to introduce you to some foundational aspects of Machine Learning which are key to delve in other topics such as Deep Learning. Our focus will be on high-level strategies and decision-making processes related to data and objectives, rather than delving into the intricacies of specific algorithms and models. This approach will help you understand the broader context and applications of Machine Learning.

We will concentrate on supervised learning, where the goal is to predict a "target" variable based on input data. In supervised learning, the target variable can be:

- **Categorical**: In this case, we are performing **classification**, where the objective is to assign input data to predefined categories or classes.
- **Continuous**: Here, we are conducting **regression**, aiming to predict a continuous value based on the input data.

While classification and regression employ different models and evaluation metrics, many of the underlying strategies and principles are shared across both types of tasks.

In this tutorial, we will explore a practical example using a transcriptomics dataset to predict phenotypic traits in potatoes, specifically focusing on potato coloration. The dataset has been pre-selected and normalized to include the 200 most promising genes out of approximately 15,000, as proposed by {% cite acharjee2016integration %}. This real-world application will help you understand how to apply Machine Learning techniques to solve practical problems.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare resources

Let's start by preparing the environment

## Install dependencies

The first step is to install the required dependencies if they are not already installed:

```python
!pip install matplotlib
!pip install numpy
!pip install pandas
!pip install seaborn
!pip install scikit-learn
```

## Import tools

Let's now import them.

```python
import matplotlib.pylab as pylab
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
```

## Configure for plotting

For plotting, we will use Matplotlib, a popular plotting library in Python. We would like to customize the appearance of plots to enhance readability and ensure that all plots generated during the tutorial will have a consistent and professional appearance, making them easier to read and interpret:

```python
pylab.rcParams["figure.figsize"] = 5, 5
plt.rc("font", size=10)

plt.rc("xtick", color="k", labelsize="medium", direction="in")
plt.rc("xtick.major", size=8, pad=12)
plt.rc("xtick.minor", size=8, pad=12)

plt.rc("ytick", color="k", labelsize="medium", direction="in")
plt.rc("ytick.major", size=8, pad=12)
plt.rc("ytick.minor", size=8, pad=12)
```

> <question-title></question-title>
>
> What are the above commands doing?
>
> > <solution-title></solution-title>
> >
> > - `pylab.rcParams["figure.figsize"] = 5, 5`: Sets the default figure size to 5x5 inches. This ensures that all plots generated will have a consistent size unless otherwise specified.
> > - `plt.rc("font", size=10)`: Sets the default font size for all text elements in the plots to 10. This includes titles, labels, and tick marks.
> > - `plt.rc("xtick", color="k", labelsize="medium", direction="in")`: Configures the x-axis ticks:
> >      - `color="k"`: Sets the tick color to black.
> >      - `labelsize="medium"`: Sets the label size to medium.
> >      - `direction="in"`: Sets the direction of the ticks to point inward.
> > - `plt.rc("xtick.major", size=8, pad=12)`: Configures the major x-axis ticks:
> >      - `size=8`: Sets the length of the major ticks to 8 points.
> >      - `pad=12`: Sets the padding between the tick labels and the plot to 12 points.
> > - *`plt.rc("xtick.minor", size=8, pad=12)`: Configures the minor x-axis ticks with the same settings as the major ticks:
> >      - `size=8`: Sets the length of the minor ticks to 8 points.
> >      - `pad=12`: Sets the padding between the tick labels and the plot to 12 points.
> > - `plt.rc("ytick", color="k", labelsize="medium", direction="in")`: Configures the y-axis ticks with the same settings as the x-axis ticks:
> >      - `color="k"`: Sets the tick color to black.
> >      - `labelsize="medium"`: Sets the label size to medium.
> >      - `direction="in"`: Sets the direction of the ticks to point inward.
> > - `plt.rc("ytick.major", size=8, pad=12)`: Configures the major y-axis ticks:
> >      - `size=8`: Sets the length of the major ticks to 8 points.
> >      - `pad=12`: Sets the padding between the tick labels and the plot to 12 points.
> > - `plt.rc("ytick.minor", size=8, pad=12)`: Configures the minor y-axis ticks with the same settings as the major ticks:
> >      - `size=8`: Sets the length of the minor ticks to 8 points.
> >      - `pad=12`: Sets the padding between the tick labels and the plot to 12 points.
> {: .solution}
>
{: .question}

# Get data

Let's now get the data. As mentioned in the introduction, we will use data from {% cite acharjee2016integration %} where they used transcriptomics dataset to predict potato coloration. The dataset has been pre-selected and normalized to include the 200 most promising genes out of approximately 15,000.

First, let's import the metadata of the studied potatoes:

```python
file_metadata = "https://github.com/sib-swiss/statistics-and-machine-learning-training/raw/refs/heads/main/data/potato_data.phenotypic.csv"
df = pd.read_csv(file_metadata, index_col=0)
```

> <question-title></question-title>
>
> 1. How many potatoes and metadata do we have?
> 2. In which column is the color information?
>
> > <solution-title></solution-title>
> > 
> > 1. Let's get the dimension of the dataframe:
> >
> >    ```python
> >    df.shape
> >    ```
> > 
> >    The dimension of the dataframe is 86 rows and 8 columns. So there are 86 potatoes and 8 metadata for each.
> >
> > 2. `Flesh Colour` column
> >
> {: .solution}
>
{: .question}

For the sake of our story, we will imagine that out of the 86 potatoes in the data, we have only 73 at the time of our experiment. We put aside the rest for later.

```python
i1 = df.index[:73]
i2 = df.index[73:]
```

We are interested in the colors of the potatoes, stored in the `Flesh Colour` column:

```python
y = df.loc[i1, "Flesh Colour"]
```

> <question-title></question-title>
>
> ```python
> y.describe()
> ```
> 
> ```
> count    73.000000
>  mean     24.473845
>  std      12.437785
>  min       6.992000
>  25%      13.484500
>  50%      24.746500
>  75%      30.996200
>  max      57.035100
>  Name: Flesh Colour, dtype: float64
> ```
>
> 1. How many entries are non-null?
> 2. What is the average value of the dataset?
> 3. What is the standard deviation?
> 4. What is the minimum value?
> 5. What is the first quartile (25th percentile)?
> 6. What is the median (50th percentile)?
> 7. What is the third quartile (75th percentile)?
> 8. What is the maximum value?
>
> > <solution-title></solution-title>
> > 
> > 1. 73
> > 2. mean = 24.47
> > 3. std = 12.44
> > 4. min = 6.992000
> > 5. 25% = 13.484500
> > 6. 50% = 24.746500
> > 7. 75% = 30.996200
> > 8. max = 57.035100
> {: .solution}
>
{: .question}

Let's look at the distribution of the values:

```python
sns.histplot(y)
```

![Bar chart depicting the distribution of a dataset categorized by 'Flesh Colour.' The x-axis represents different ranges of flesh colour values, while the y-axis indicates the count of occurrences within each range. The chart shows the highest frequency in the 10-20 range, followed by a gradual decline in counts as the flesh colour values increase. The counts decrease more sharply after the 30-40 range, with the lowest frequencies observed in the 50 range.](images/outputs/output_8_1.png)
    

We can now import transcriptomic data for the 200 selected genes in the 86 potatoes:

```python
file_data = "https://github.com/sib-swiss/statistics-and-machine-learning-training/raw/refs/heads/main/data/potato_data.transcriptomic.top200norm.csv"
dfTT = pd.read_csv(file_data, index_col=0)
```

We keep only the 73 potatoes:

```python
X = dfTT.loc[i1, :]
```

> <question-title></question-title>
>
> How does the transcriptomic data look like?
>
> > <solution-title></solution-title>
> > 
> > ```python
> > X.head()
> > ```
> > 
> > | Genotype  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 |
> > | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
> > | CE017 | 0.086271 | -0.790631 | -0.445972 | 0.788895 | 0.510650 | 0.626438 | 0.829346 | 0.432200 | -1.344748 | 1.794652 | ... | -0.754008 | -0.013125 | 0.852473 | 1.067286 | 0.877670 | 0.537247 | 1.251427 | 1.052070 | -0.135479 | -0.526788 |
> > | CE069 | -0.540687 | 0.169014 | 0.282120 | -1.107200 | -1.200370 | 0.518986 | 1.027663 | -0.374142 | -0.937715 | 1.488139 | ... | -0.237367 | 0.684905 | 1.460319 | -1.570253 | 0.547969 | 0.635307 | 0.257955 | 1.043724 | 0.733218 | -1.768250 |
> > | CE072 | -1.713273 | -1.400956 | -1.543058 | -0.930367 | -1.058800 | -0.455020 | -1.302403 | -0.110293 | -0.332380 | -0.232460 | ... | -0.131733 | -0.070336 | 0.821996 | -1.566652 | 0.914053 | -1.707726 | 0.498226 | -1.500588 | 0.361168 | -1.020456 |
> > | CE084 | -0.096239 | -0.599251 | -1.499636 | -0.847275 | -1.171365 | -0.952574 | -1.347691 | 0.561542 | -0.335009 | -0.702851 | ... | -0.729461 | 0.135614 | 1.074398 | 0.629679 | -0.691100 | -1.247779 | 0.167965 | -1.525064 | 0.150271 | 0.105746 |
> > | CE110 | -0.712374 | -1.081618 | -1.530316 | -1.259747 | -1.109999 | -0.582357 | -1.233085 | 0.008014 | -0.915632 | -0.746339 | ... | -0.054882 | 0.363344 | 0.720155 | 0.465315 | 1.450199 | -1.706606 | 0.602451 | -1.507727 | -2.207455 | -0.139036 |
> >
> > It is a tabular file with 73 rows and 201 columns. For each gene (column) and potatoes (row), we get the expression value of the gene 
> >
> {: .solution}
>
{: .question}


# Linear regression

Linear regression is one of the most fundamental and widely used techniques in supervised Machine Learning. It is employed to model the relationship between a dependent variable (target) and one or more independent variables (features). The primary goal of linear regression is to fit a linear equation to observed data, enabling predictions and insights into the relationships between variables.

## Approach 1: Simple linear regression

Let's start by fitting a simple linear model with our gene expression values, and see what happens.

We start by importing elements from `sklearn`, a widely used library for Machine Learning in Python:

```python
from sklearn.linear_model import LinearRegression
```

> <details-title>About `from sklearn.linear_model import LinearRegression`</details-title>
>
> With `from sklearn.linear_model import LinearRegression`, we import the `LinearRegression` class from the `linear_model module` of `scikit-learn`. `LinearRegression` is used to create and train a linear regression model. It fits a linear equation to the observed data, allowing you to make predictions based on the relationship between the independent variables (features) and the dependent variable (target).
> 
{: .details}

We now:
1. Create an instance of the `LinearRegression` class that will be used to fit the linear regression model to our data
2. Train the linear regression model using the training data:
    - `X`: the input feature matrix, where each row represents a sample and each column represents a feature, here a gene and its expression
    - `y`: the target vector, containing the values you want to predict, i.e. the flest color
3. Use the trained linear regression model to make predictions on the same input data `X`

```python
lin_reg = LinearRegression()
lin_reg.fit(X, y)
y_pred = lin_reg.predict(X)
```

The next step is to **evaluate the prediction** using two evaluation metrics from the `metrics` module of `scikit-learn`:
- `r2_score`: This function calculates the R-squared (coefficient of determination) score, which indicates how well the independent variables explain the variance in the dependent variable. An R-squared value of 1 indicates a perfect fit, while a value of 0 indicates that the model does not explain any of the variance.
- `mean_squared_error`: This function calculates the mean squared error (MSE), which measures the average of the squares of the errors—that is, the average squared difference between the observed actual outcomes and the outcomes predicted by the model. Lower values of MSE indicate better model performance.

```python
from sklearn.metrics import r2_score, mean_squared_error
print(f"R-squared score: { r2_score(y ,y_pred ) :.2f}")
print(f"mean squared error: { mean_squared_error(y ,y_pred ) :.2f}")
```

> <question-title></question-title>
>
> 1. What is the value of the R-squared score?
> 2. What is the value of the mean squared error?
>
> > <solution-title></solution-title>
> > 1. `R-squared score: 1.00`
> > 2. `mean squared error: 0.00`
> {: .solution}
>
{: .question}

**Wow!!** this is a perfect fit. But if you know anything about biology, or data analysis, then you likely suspect something wrong is happening.


> <details-title>"Generic" `sklearn` usage </details-title>
>
> The main library we will be using for machine learning is scikit-learn.
> 
> It should go without saying that if you have any questions regarding its usage and capabilities, your first stop should be their [website](https://scikit-learn.org/stable/),especially since it provides plenty of [examples](https://scikit-learn.org/stable/auto_examples/ensemble/plot_voting_decision_regions.html#sphx-glr-auto-examples-ensemble-plot-voting-decision-regions-py), [guides](https://scikit-learn.org/stable/user_guide.html), and [tutorials](https://scikit-learn.org/stable/tutorial/index.html#tutorial-menu).
> 
> Nevertheless, we introduce here the most common behavior of `sklearn` object.
> 
> Indeed, `sklearn` implement machine learning algorithms (random forest, clustering algorithm,...), as well as all kinds of preprocessers (scalin, missing value imputation,...) with a fairly consistent interface.
> 
> Most methods must first be instanciated as an object from a specific class:
> 
> ```python
> ## import the class, here RandomForestClassifier
> from sklearn.ensemble import RandomForestClassifier
> 
> ## instanciate the class object:
> my_clf = RandomForestClassifier()
> ```
> 
> As it stands, the object is just a "naive" version of the algorithm.
> 
> The next step is then to feed the object data, so it can learn from it. This is done with the `.fit` method:
> 
> ```python
> my_clf.fit( X , y )
> ```
> > <comment-title></comment-title>
> > In this context, `X` is the data and `y` is the objective to attain. When the object is not an ML algorithm but a preprocessor, you only give the `X`
> {: .comment}
>
> Now that the object has been trained with your data, you can use it. For instance, to:
> * `.transform` your data (typically in the case of a preprocessor):
>
>   ```python
>   X_scaled = myScaler.transform(X)  # apply a transformation to the data
>   ```
>
> * `.predict` some output from data (typically in the case of an ML algorithm, like a classifier)
> 
>   ```python
>   y_predicted = clf.predict(X)  # predict classes of the training data
>   ```
> 
> Last but not least, it is common in example code to "fit and transform" a preprocesser in the same line using ` .fit_transform`
> 
> ```python
> X_scaled = myNaiveScaler.fit_transform(X)  # equivalent to myNaiveScaler.fit(X).transform(X)
> ```
> 
> That's the basics. You will be able to experiment at length with this and go well beyond it.
>
{: .details}

Back to our potato model. At the moment, our claim is that our model can predict flesh color perfectly (RMSE=0.0) from the normalized expression of these 200 genes. But, say we now have some colleagues who come to us with some new potato data (the leftover data points)

```python
Xnew = dfTT.loc[i2, :]
ynew = df.loc[i2, "Flesh Colour"]
```

We apply the model on the new data

```python
ynew_pred = lin_reg.predict(Xnew)
```

> <question-title></question-title>
>
> How is the prediction in term of R-squared score and mean squared error?
>
> > <solution-title></solution-title>
> > 
> > ```python
> > print(f"new data R-squared score: { r2_score( ynew , ynew_pred ) :.2f}")
> > print(f"new data mean squared error: { mean_squared_error( ynew , ynew_pred ) :.2f}")
> > ```
> > 
> > ```
> > new data R-squared score: 0.47
> > new data mean squared error: 130.47
> > ```
> >
> > The values are not really good.
> > 
> {: .solution}
>
{: .question}

To compare observed values to predicted values, we can generate a scatter plot that visualizes the relationship between observed and predicted values for both training and new data:

```python
plt.scatter(y, y_pred, label="training data")
plt.scatter(ynew, ynew_pred, label="new data")
plt.xlabel("observed values")
plt.ylabel("predicted values")
plt.legend()
```
    
![Scatter plot comparing predicted values to observed values. The x-axis represents observed values ranging from 0 to 60, and the y-axis represents predicted values ranging from -10 to 60. The plot includes two sets of data points: blue dots labeled as 'training data' and orange dots labeled as 'new data.' The blue training data points form a tight cluster along a diagonal line from the bottom-left to the top-right, indicating a strong linear relationship between observed and predicted values. The orange new data points are more scattered around the diagonal line, showing varying degrees of deviation from the perfect prediction line.](images/outputs/output_15_1.png)

As expected, the performance on the new data is not as good as with the data we used to train the model. We have **overfitted** the data.

Here, we could still use the model that we have created, but we would agree that reporting the perfect performance we had with our training data would be misleading. To honestly report the performance of our model, we measure it on a **set of data that has not been used at all to train it: the *test set*.**

To that end, we typically begin by dividing our data into:

* **train** set to find the best model
* **test** set to give an honest evaluation of how the model perform on completely new data.

![Diagram illustrating the division of data into train and test sets. The diagram is divided into two main sections labeled 'train set' and 'test set.' The 'train set' is represented by a larger red rectangle on the left, with an annotation pointing to it that reads 'Finding the data related parameters of a particular model.' The 'test set' is represented by a smaller blue rectangle on the right, with an annotation pointing to it that reads 'Checking the value of our model on unseen data.' Below both rectangles, there is a horizontal line labeled 'DATA,' indicating the entire dataset. The train set is used for training the model, while the test set is used for evaluating the model's performance on unseen data.](images/train_test.png)


```python
X_test = Xnew
y_test = ynew
```

## Approach 2: Adding regularization and validation set

In the case of a Least Square fit, the function we are minimizing corresponds to the sum of squared difference between the observation and the predictions of our model:

\\( \sum_i (y_i-f(\pmb X_i,\pmb{\beta}))^2 \\)

with $$y$$ the target vector, $$\pmb X$$ the input feature matrix and $$f$$ our model


**Regularization** is a way to reduce overfitting. In the case of the linear model
we do so by adding to this function a **penalization term which depends on coefficient weights**.
In brief, the stronger the coefficient, the higher the penalization. So only coefficients which bring more fit than penalization will be kept.

The formulas used in `scikit-learn` functions -other libraries may have a different parameterization, but the concepts stay the same- are:

- **L1 regularization** (Lasso): \\(\frac{1}{2n}\sum_i (y_i-f(\pmb X_i,\pmb{\beta}))^2 + \alpha\sum_{j}\lvert\beta_{j}\rvert\\), with  $$\alpha$$ being the weight that you put on that regularization

- **L2 regularization** (Ridge): \\(\sum_i (y_i-f(\pmb X_i,\pmb{\beta}))^2 + \alpha\sum_{j}\beta_{j}^{2}\\)

- **Elastic Net**: \\(\frac{1}{2n}\sum_i (y_i-f(\pmb X_i,\pmb{\beta}))^2 + \alpha\sum_{j}(\rho\lvert\beta_{j}\rvert+\frac{(1-\rho)}{2}\beta_{j}^{2})\\)


> <comment-title></comment-title>
> For a deeper understanding of those notions, you may look at :
> 
> * [DataCamp's "Regularization in R Tutorial: Ridge, Lasso and Elastic Net"](https://www.datacamp.com/community/tutorials/tutorial-ridge-lasso-elastic-net)
> 
> * [towards data science's "Understanding l1 and l2 Regularization"](https://towardsdatascience.com/regularization-in-machine-learning-76441ddcf99a)
{: .comment}

> <comment-title></comment-title>
> Regularization generalize to maximum likelihood contexts as well
{: .comment}

Let's try that on our data.

We start by importing the `SGDRegressor` class from the `linear_model` module of the scikit-learn library. SGDRegressor stands for Stochastic Gradient Descent Regressor. It is an implementation of linear regression that uses Stochastic Gradient Descent (SGD) to fit the model. SGD is an optimization algorithm that updates the model parameters iteratively based on one sample at a time (or a mini-batch of samples), making it efficient for large datasets. It supports various types of regularization (L1, L2, Elastic Net) to prevent overfitting.

```python
from sklearn.linear_model import SGDRegressor
```

We would like now to perform a grid search over 50 different values -- logarithmically spaced between $$10^{−2}$$ and $$10^2$$ -- of the regularization parameter `alpha` for an SGDRegressor with L1 penalty (Lasso regression). The results will be stored in a DataFrame for further analysis.

```python
%%time

logalphas = []

coef_dict = {
    "name": [],
    "coefficient": [],
    "log-alpha": [],
}
r2 = []

for alpha in np.logspace(-2, 2, 50):
    reg = SGDRegressor(penalty="l1", alpha=alpha)
    reg.fit(X , y)
    
    logalphas.append(np.log10(alpha))
    r2.append(r2_score(y, reg.predict(X)))
    
    coef_dict["name"] += list(X.columns)
    coef_dict["coefficient"] += list(reg.coef_)
    coef_dict["log-alpha"] += [np.log10(alpha)]* len(X.columns)

coef_df = pd.DataFrame(coef_dict)
```

Let's visualize:

```python
fig,ax = plt.subplots(1, 2, figsize=(14,7))

ax[0].plot(logalphas, r2)
ax[0].set_xlabel("log10( alpha )")
ax[0].set_ylabel("R2")

sns.lineplot(
    x="log-alpha",
    y="coefficient",
    hue="name",
    data=coef_df,
    ax=ax[1],
    legend = False,
)

fig.suptitle("Regression of potato data with an L1 regularization.")
```

![Figure showing the results of regression analysis on potato data with L1 regularization. The figure consists of two subplots. The left subplot is a line graph with the x-axis labeled 'log10(alpha)' ranging from -2.0 to 2.0, and the y-axis labeled 'R2' ranging from 0.0 to 1.0. It displays a blue curve that starts near the top left, remains high and relatively stable for lower values of log10(alpha), and then sharply drops to near zero as log10(alpha) increases, indicating a decrease in the R-squared score with higher alpha values. The right subplot is a line graph with the x-axis labeled 'log-alpha' ranging from -2.0 to 2.0, and the y-axis labeled 'coefficient' ranging from -2 to 2. It shows multiple colored lines, each representing the coefficient path of a different feature as log-alpha varies. The lines start spread out and converge towards zero as log-alpha increases, illustrating how coefficients shrink with increasing regularization. The title of the figure reads 'regression of potato data with an L1 regularization.'](images/outputs/output_21_1.png)

> <question-title></question-title>
>
> 1. What is in the left subplot?
> 2. How to iterpret the left subplot?
> 3. What is in the right subplot?
> 4. how to iterpret the right subplot?
>
> > <solution-title></solution-title>
> >
> > 1. The left subplot shows R-squared ($$R^2$$) vs. $$\log10(\alpha)$$
> > 2. Interpretation
> >    - **High $$R^2$$ at Low $$\alpha$$**: At lower values of $$\log10(\alpha)$$ (left side of the plot), the R-squared score is high, indicating that the model fits the data well. This is because the regularization is weak, allowing more features to contribute to the model.
> >    - **Sharp Drop in $$R^2$$**: As $$\log10(\alpha)$$ increases, the R-squared score drops sharply. This indicates that stronger regularization (higher alpha values) leads to a simpler model with fewer features, which may not capture the data's variance as well.
> >    - **Optimal Alpha**: The point where the R-squared score starts to drop significantly can be considered as the optimal alpha value, balancing model complexity and performance.
> > 3. The right subplot shows Coefficients vs. log-alpha
> > 4. Interpretation:
> >    - **Coefficient Paths**: Each colored line represents the path of a coefficient for a specific feature as log-alpha varies.
> >    - **Shrinkage to Zero**: As log-alpha increases, many coefficients shrink towards zero. This is the effect of L1 regularization, which penalizes the absolute size of the coefficients, leading to sparse models where many coefficients become exactly zero.
> >    - **Feature Selection**: Features with coefficients that shrink to zero first are less important for the model. Features whose coefficients remain non-zero for higher values of log-alpha are more important.
> >    - **Stability**: The spread of the lines at lower log-alpha values indicates that the model includes many features. As log-alpha increases, the lines converge, showing that the model becomes simpler and more stable.
> >
> {: .solution}
>
{: .question}

> <hands-on-title></hands-on-title>
>
> Adapt the code above to generate this plot with an L2 penalty.
>
> ![Figure showing the results of regression analysis on potato data with L2 regularization. The figure consists of two subplots. The left subplot is a line graph with the x-axis labeled 'log10(alpha)' ranging from -2.0 to 2.0, and the y-axis labeled 'R2' ranging from 0.3 to 1.0. It displays a blue curve that starts near the top left, remains relatively high and stable for lower values of log10(alpha), and then gradually decreases with some fluctuations as log10(alpha) increases, indicating a decrease in the R-squared score with higher alpha values. The right subplot is a line graph with the x-axis labeled 'log-alpha' ranging from -2.0 to 2.0, and the y-axis labeled 'coefficient' ranging from -2 to 2. It shows multiple colored lines, each representing the coefficient path of a different feature as log-alpha varies. The lines start spread out and gradually converge towards zero as log-alpha increases, illustrating how coefficients shrink with increasing regularization. The title of the figure reads 'regression of potato data with an L2 regularization.'](images/outputs/output_23_1.png)
>
> > <solution-title></solution-title>
> > 
> > ```python
> > ### correction
> > from sklearn.linear_model import SGDRegressor
> > 
> > logalphas = []
> > 
> > coef_dict = {
> >     "name": [],
> >     "coefficient": [],
> >     "log-alpha": [],
> > }
> > r2 = []
> > 
> > for alpha in np.logspace(-2, 2, 50):
> >     reg = SGDRegressor(penalty="l2", alpha=alpha)
> >     reg.fit(X, y)
> >  
> >     logalphas.append(np.log10(alpha))
> >     r2.append(r2_score(y, reg.predict(X)))
> >     
> >     coef_dict["name"] += list(X.columns)
> >     coef_dict["coefficient"] += list(reg.coef_)
> >     coef_dict["log-alpha"] += [np.log10(alpha)]* len(X.columns )
> > 
> > coef_df = pd.DataFrame(coef_dict)
> > 
> > fig,ax = plt.subplots(1, 2, figsize=(14,7))
> > 
> > ax[0].plot(logalphas, r2)
> > ax[0].set_xlabel("log10( alpha )")
> > ax[0].set_ylabel("R2")
> > 
> > sns.lineplot(
> >     x="log-alpha",
> >     y="coefficient",
> >     hue="name",
> >     data=coef_df,
> >     ax = ax[1],
> >     legend = False
> > )
> > 
> > fig.suptitle("regression of potato data with an L2 regularization.")
> > ```
> {: .solution}
>
>
> > <question-title></question-title>
> > 1. How do each type of regularization affect R-squared (R²)?
> > 2. How do each type of regularization affect Coefficients?
> >
> > 
> > > <solution-title></solution-title>
> > > 
> > > 1. We look at the left Subplot: R-squared (R²) vs. log10(alpha).
> > >   - L1 Regularization (Lasso): The R-squared score starts high and remains relatively stable for lower values of log10(alpha), then sharply drops to near zero as log10(alpha) increases. L1 regularization tends to drive many coefficients to exactly zero, effectively performing feature selection. As alpha increases, more features are excluded from the model, leading to a simpler model with fewer features and a sharp drop in R-squared score.
> > >   - L2 Regularization (Ridge): The R-squared score starts high and gradually decreases with some fluctuations as log10(alpha) increases. L2 regularization penalizes the squared magnitude of the coefficients, leading to a more gradual shrinkage of coefficients. This results in a smoother decrease in the R-squared score as alpha increases, without the sharp drop seen in L1 regularization.
> > >
> > > 2. Right Subplot: Coefficients vs. log-alpha
> > >   - L1 Regularization (Lasso): The coefficient paths show many lines shrinking to zero as log-alpha increases, indicating that many features are excluded from the model. L1 regularization performs feature selection by driving less important feature coefficients to zero. This results in a sparse model where only a few features have non-zero coefficients.
> > >   - L2 Regularization (Ridge): The coefficient paths show a more gradual shrinkage towards zero as log-alpha increases, with fewer coefficients becoming exactly zero.L2 regularization retains all features in the model but shrinks their coefficients. This results in a model where all features contribute, but their contributions are reduced.
> > >
> > {: .solution}
> {: .question}
{: .hands-on}

This is great, but how do we choose which level of regularization we want ?

It is a general rule that **as you decrease $$\alpha$$, the $$R^2$$ on the data used for the fit increase**, i.e. you risk overfitting. Consequently, we cannot choose the value of $$\alpha$$ parameter from the data used to fit alone. We call such a parameter an **hyper-parameter**.

> <question-title></question-title>
>
> What are other hyper-parameters we could optimize at this point?
>
> > <solution-title></solution-title>
> >
> > If we used an Elastic Net penalty, we would also have the **L1 ratio** to explore
> >
> > But we could also explore which normalization method we use, which imputation method (if applicable, here that is not the case), and whether we should account for interaction between samples, or degree 2,3,4,... polynomials.
> > 
> {: .solution}
{: .question}

In order to find the optimal value of an hyper-parameter, we separate our training data into:
* a **train set** used to fit the model
* a **validation set** used to evaluate how our model perform on new data 

Here we have 73 points. we will use 60 points to train the model and the rest to evaluate the model  

```python
I = list(range(X.shape[0]))
np.random.shuffle( I ) 

I_train = I[:60]
I_valid = I[60:]

X_train = X.iloc[I_train, : ] 
y_train = y.iloc[I_train]

X_valid = X.iloc[I_valid, : ] 
y_valid = y.iloc[I_valid]
```

We now to train the model with 200 `alpha` values logarithmically spaced between $$10^{−3}$$ and $$10^2$$ with L1 penalty (Lasso regression) and make prediction on both the train and validation sets:

```python
logalphas = []

r2_train = []
r2_valid = []

for alpha in np.logspace(-3, 2, 200):
    reg = SGDRegressor(penalty="l1", alpha=alpha)
    reg.fit(X_train, y_train)
    
    logalphas.append(np.log10(alpha))
    r2_train.append(r2_score(y_train, reg.predict(X_train)))
    r2_valid.append(r2_score(y_valid, reg.predict(X_valid)))
```

Let's visualize the $$R^2$$ for both the train and validation sets, as well as the $$\alpha$$ value with highest $$R^2$$ for the validation set:

```python
bestI = np.argmax(r2_valid)
bestLogAlpha = logalphas[bestI]
bestR2_valid = r2_valid[bestI]

fig,ax = plt.subplots(figsize=(10,5))
fig.suptitle(f"best alpha : {10**bestLogAlpha:.2f} - validation R2 : {bestR2_valid:.2f}")
ax.plot(logalphas, r2_train, label="train set")
ax.plot(logalphas, r2_valid, label="validation set")
ax.scatter([bestLogAlpha], [bestR2_valid], c="red")
ax.set_xlabel("log10( alpha )")
ax.set_ylabel("R2")
ax.set_ylim(-0.1, 1.1)
ax.legend()
```
    
![Line graph showing the relationship between the regularization parameter alpha and the R-squared (R²) scores for both the training set and the validation set. The x-axis is labeled 'log10(alpha)' and ranges from -3 to 2. The y-axis is labeled 'R2' and ranges from 0.0 to 1.0. The graph includes two lines: a blue line representing the R-squared scores for the training set and an orange line representing the R-squared scores for the validation set. The blue line starts high and remains relatively stable for lower values of log10(alpha), then gradually decreases and drops sharply as log10(alpha) increases. The orange line starts low, increases to a peak, and then decreases, with the peak marked by a red dot indicating the best alpha value with a validation R-squared score of 0.65. The title of the graph reads 'best alpha : 0.46 - validation R2 : 0.65'. The legend in the upper right corner distinguishes between the 'train set' and 'validation set' lines.](images/outputs/output_26_2.png)


So now, with the help of the validation set, we can clearly see different phases:
* **Underfitting**: for high $$\alpha$$, the performance is low for both the train and the validation set
* **Overfitting**: for low $$\alpha$$, the performance is high for the train set, and low for the validation set
 
We want the equilibrium point between the two where performance is ideal for the validation set. **But** if we run the code above several time, we will see that the optimal point varies due to the random assignation to train or validation set. 

There exists a myriad of possible strategies to deal with that problem, such as repeating the above many times and taking the average of the results for instance.

> <comment-title></comment-title>
> This problem gets less important as the validation set size increases.
{: .comment}

Anyhow, on top of our earlier regression model, we have added:

* an **hyper-parameter** : $$\alpha$$, the strength of the regularization term
* a **validation strategy** for our model in order to avoid overfitting

That's it, we are now in the world of Machine Learning. But how does the modified model perform on the test data?

> <hands-on-title></hands-on-title>
> 
> Check how the modified model performs on the test data
> 
> > <solution-title></solution-title>
> > 
> > ```python
> > reg = SGDRegressor(penalty="l1", alpha = 10**bestLogAlpha)
> > reg.fit( X , y )
> > 
> > y_pred = reg.predict( X )
> > print(f"train data R-squared score: { r2_score(y, y_pred ) :.2f}")
> > print(f"train data mean squared error: { mean_squared_error(y, y_pred ) :.2f}")
> > 
> > y_test_pred = reg.predict( X_test )
> > 
> > print(f"test data R-squared score: { r2_score(y_test, y_test_pred ) :.2f}")
> > print(f"test data mean squared error: { mean_squared_error(y_test, y_test_pred ) :.2f}")
> > 
> > plt.scatter(y, y_pred, label="training data" )
> > plt.scatter(y_test, y_test_pred, label="new data" )
> > plt.xlabel("observed values")
> > plt.ylabel("predicted values")
> > plt.legend()
> > ```
> {: .solution}
> 
> > <question-title></question-title>
> > What are the R-squared score and mean squared error for both training and test data?
> > 
> > > <solution-title></solution-title>
> > > 
> > > Data | R-squared score | mean squared error
> > > --- | --- | ---
> > > Training | 0.90 | 15.81
> > > Test | 0.73 | 65.83
> > {: .solution}
> {: .question}
{: .hands-on}

![Scatter plot comparing predicted values to observed values. The x-axis is labeled 'observed values' and ranges from 0 to 60. The y-axis is labeled 'predicted values' and ranges from 0 to 60. The plot includes two sets of data points: blue dots labeled as 'training data' and orange dots labeled as 'new data.' The blue training data points form a tight cluster along a diagonal line from the bottom-left to the top-right, indicating a strong linear relationship between observed and predicted values. The orange new data points are more scattered around the diagonal line, showing varying degrees of deviation from the perfect prediction line. The legend in the upper left corner distinguishes between the 'training data' and 'new data' points.](images/outputs/output_29_2.png)

Two things we can observe:
* Still better performance on the train data than on the test data (generally always the case)
* the performance on the test set has improved: our model is less overfit and more generalizable

## Approach 3: k-fold cross-validation

In the previous approach, we have split our training data into a train set and a validation set. This approach works well if you have enough data for your validation set to be representative. Often, we unfortunately do not have enough data for this.

Indeed, we have seen that if we run the code above several time, we see that the optimal point varies due to the random assignation to train or validation set. **k-fold cross validation** is one of the most common strategy to try to mitigate this randomness with a limited amount of data.

![Diagram illustrating the process of cross-validation. The data is divided into a train set and a test set. The train set is further split into multiple folds, with each fold used as a validation set while the remaining folds are used for training. The process is repeated for each fold, and the average score across all folds is calculated to evaluate the model's performance.](images/kfold.png)

In k-fold cross-validation, 
1. Data is split in $$k$$ subpart, called fold. 
2. $$k$$ models are trained for a given hyper-parameter values combination: each time a different fold is used for validation (and the remaining $$k-1$$ folds for training).
3. The average performance is computed across all fold : this is the **cross-validated performance**.

Let's do a simple k-fold manually once (with $$k = 5$$) to explore how it works:

```python
from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=True, random_state=734)
for i, (train_index, valid_index) in enumerate(kf.split(X)):
    print(f"Fold {i}:")
    print(f" Train: index={train_index}")
    print(f" Test: index={valid_index}")
```

> <question-title></question-title>
>
> 
> ```
> Fold 0:
>  Train: index=[ 0  1  2  3  4  5  6  7  9 10 11 12 13 14 15 17 18 19 20 21 22 24 25 26 27 28 29 30 31 33 34 35 39 40 41 45 46 47 48 50 51 53 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 72]
>  Test:  index=[ 8 16 23 32 36 37 38 42 43 44 49 52 54 70 71]
> Fold 1:
>  Train: index=[ 4  5  6  8  9 10 11 12 13 14 15 16 18 19 20 21 23 24 26 27 28 29 30 31 32 33 34 35 36 37 38 39 42 43 44 46 47 48 49 50 51 52 53 54 56 57 58 59 60 61 62 63 64 65 66 67 70 71]
>  Test:  index=[ 0  1  2  3  7 17 22 25 40 41 45 55 68 69 72]
> Fold 2:
>  Train: index=[ 0  1  2  3  4  6  7  8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 47 48 49 52 53 54 55 57 59 61 64 65 68 69 70 71 72]
>  Test:  index=[ 5  9 28 29 30 46 50 51 56 58 60 62 63 66 67]
> Fold 3:
>  Train: index=[ 0  1  2  3  5  6  7  8  9 12 14 16 17 19 20 22 23 25 28 29 30 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 49 50 51 52 54 55 56 57 58 59 60 61 62 63 64 66 67 68 69 70 71 72]
>  Test:  index=[ 4 10 11 13 15 18 21 24 26 27 31 48 53 65]
> Fold 4:
>  Train: index=[ 0  1  2  3  4  5  7  8  9 10 11 13 15 16 17 18 21 22 23 24 25 26 27 28 29 30 31 32 36 37 38 40 41 42 43 44 45 46 48 49 50 51 52 53 54 55 56 58 60 62 63 65 66 67 68 69 70 71 72]
>  Test:  index=[ 6 12 14 19 20 33 34 35 39 47 57 59 61 64]
> ```
>
> Do you have the same indexes for train and test sets in each fold as above? How is it possible?
> 
> > <solution-title></solution-title>
> > 
> > The indexes for train and test sets in each fold are the same because the random state was fixed.
> > 
> {: .solution}
>
{: .question}

In practice that it is mostly automatized with some of scikit-learn's recipes and objects:
1. Setting up the range of $$\alpha$$ values
2. Setting up k-fold cross-validation as above
3. Looping over $$\alpha$$ values and over each fold in the k-fold cross-validation:
    1. Splits the data into training and validation sets.
    2. Fits the `SGDRegressor` model with the current $$\alpha$$ value on the training set.
    3. Evaluates the model on the validation set using the $$R^{2}$$ score.
    4. Stores the $$R^{2}$$ score for the current fold and updates the average $$R^{2}$$ score across all folds.
4. Finding the best $$\alpha$$ value
     

```python
%%time
logalphas = np.linspace(-2, 1, 200)

kf = KFold(n_splits=5, shuffle=True, random_state=6581)

fold_r2s = [[] for i in range(kf.n_splits)] ## for each fold
cross_validated_r2 = [] # average across folds
   
for j,alpha in enumerate( 10**logalphas ) :
    cross_validated_r2.append(0)
    for i, (train_index, valid_index) in enumerate(kf.split(X)):
        ## split train and validation sets
        X_train = X.iloc[train_index, : ]
        X_valid = X.iloc[valid_index, : ]
        y_train = y.iloc[train_index]
        y_valid = y.iloc[valid_index]

        ## fit model for that fold
        reg = SGDRegressor(penalty="l1", alpha = alpha)
        reg.fit(X_train, y_train )

        ## evaluate for that fold
        fold_score = r2_score(y_valid, reg.predict(X_valid))
        
        ## keeping in the curve specific to this fold
        fold_r2s[i].append(fold_score)
        
        ## keeping a tally of the average across folds
        cross_validated_r2[-1] += fold_score/kf.n_splits

bestI = np.argmax(cross_validated_r2)
bestLogAlpha = logalphas[bestI] 
bestR2_valid = cross_validated_r2[bestI]
```

Let's now plot the cross-validated performance of an SGDRegressor model with L1 regularization ($$R^{2}$$) across the different $$\alpha$$ values:

```python
fig,ax = plt.subplots(figsize=(10,5))

ax.plot(logalphas, cross_validated_r2, label="cross-validated r2")
ax.scatter([bestLogAlpha] , [bestR2_valid], c="red")

for i,scores in enumerate(fold_r2s):
    ax.plot(logalphas, scores, label=f"fold {i}", linestyle="dotted")

ax.set_xlabel("log10( alpha )")
ax.set_ylabel("R2")
ax.set_ylim(-0.1, 1.1)
ax.legend()

fig.suptitle(f"best alpha : {10**bestLogAlpha:.2f} - cross-validated R2 : {bestR2_valid:.2f}")
```
    
![Plot illustrating the cross-validated performance of an SGDRegressor model with L1 regularization across different alpha values. The x-axis represents the logarithm of the alpha values (log10(alpha)), ranging from -2.0 to 1.0. The y-axis represents the R squared score, which measures the model's performance. The plot includes multiple lines, each corresponding to a different fold in the cross-validation process (fold 0 to fold 4), shown in various colors and line styles. The blue solid line represents the average cross-validated R squared score across all folds. The red dot highlights the best alpha value (0.52) with the highest average cross-validated R squared score of 0.54. The plot shows how the model's performance varies with different alpha values, indicating the optimal alpha for the best generalization to unseen data.](images/outputs/output_34_2.png)
    

> <hands-on-title></hands-on-title>
>
> Re-fit a model with the best $$\alpha$$ above ($$\alpha = 0.52$$) and check the performance with the *test* data
>
> > <solution-title></solution-title>
> >
> > ```python
> > reg = SGDRegressor(penalty="l1", alpha = 10**bestLogAlpha)
> > reg.fit(X, y )
> > 
> > y_pred = reg.predict(X)
> > print(f"train data R-squared score: { r2_score(y, y_pred ) :.2f}")
> > print(f"train data mean squared error: { mean_squared_error(y, y_pred ) :.2f}")
> > 
> > y_test_pred = reg.predict(X_test)
> > 
> > print(f"test data R-squared score: { r2_score(y_test, y_test_pred ) :.2f}")
> > print(f"test data mean squared error: { mean_squared_error(y_test, y_test_pred ) :.2f}")
> > 
> > plt.scatter(y, y_pred, label="training data" )
> > plt.scatter(y_test, y_test_pred, label="new data" )
> > plt.xlabel("observed values")
> > plt.ylabel("predicted values")
> > plt.legend()
> > ```
> >
> {: .solution}
> >
> > <question-title></question-title>
> >
> > 1. What are the R-squared score and mean squared error for both training and test data?
> > 2. What can we conclude from the plot?
> >
> > ![Scatter plot comparing predicted values against observed values for a machine learning model. The x-axis represents the observed values, while the y-axis represents the predicted values. Two types of data points are plotted: blue dots representing the training data used to train the model, and orange dots representing new, unseen data. The plot shows a general trend where the predicted values increase with the observed values, indicating that the model captures the underlying relationship in the data.](./images/outputs/output_36_2.png)
> > 
> > > <solution-title></solution-title>
> > > 1. R-squared score and mean squared error for both training and test data:
> > >
> > >    Data | R-squared score | mean squared error
> > >    --- | --- | ---
> > >    Training | 0.90 | 14.99
> > >    Test | 0.72 | 69.05
> > >
> > > 2. There are some deviations, especially for the new data points, suggesting areas where the model's predictions may need improvement.
> > {: .solution}
> > 
> {: .question}
{: .hands-on}

There, you can realize that now, for each possible value of our hyper-parameter we fit and evaluate not 1, but $k$ models, here 4. So, for 200 values of $$\alpha$$, that means $$ 200 x 5 = 1000$$ models to fit and evaluate.

Now, consider that we have other hyper-parameters, such as the type of regularization (L1 or L2), or how we perform scaling, or whether we consider interactions, and now you understand why Machine Learning can quickly become computationnaly intensive. 

## Approach 4: "classical" ML pipeline

Let's start back from scratch to recapitulate what we have seen and use `scikit-learn` to solve the potato problem:

1. Get the full dataset (86 potatoes instead of 73):

   ```python
   X = dfTT
   y = df["Flesh Colour"]
   ```

2. Split our data in a train and a test set using `train_test_split`  from `scikit-learn` with the idea that 20% of the data will be used for testing, and the remaining 80% will be used for training.

   ```python
   from sklearn.model_selection import train_test_split
   
   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
   
   print(f"train set size: {len(y_train)}")
   print(f"test set size: {len(y_test)}")
   ```

    > <question-title></question-title>
    >
    > What are the sizes of the train and test sets?
    >
    > > <solution-title></solution-title>
    > > 
    > > train set size: 68
    > >
    > > test set size: 18
    > > 
    > {: .solution}
    >
    {: .question}

3. Train a model while optimizing some hyper-parameters.

On top of what we've done before, we would like to add a scaling phase and test L1 or L2 penalties. The **scaling phase** is a crucial preprocessing step where the features of your dataset are transformed to a standard scale. This step is important for several reasons: it improves model performance, prevents the dominance of large-range features, and ensures convergence, especially for algorithms that use optimization techniques like gradient descent. Common scaling techniques include standardization (Z-score normalization), which scales the features to have a mean of 0 and a standard deviation of 1, and min-max scaling (normalization), which scales the features to a fixed range, usually between 0 and 1.

Scikit-learn's `GridSearchCV` is useful to explore these more "complex" hyper-parameter spaces. Let's see how to set up a machine learning pipeline with scaling and linear regression using `StandardScaler` and `SGDRegressor`, and how to use GridSearchCV to explore different hyperparameter combinations, after that `X_train` and `y_train` have been defined as training features and labels.

1. Create the pipeline with two steps: scaling (set mean of each feature at 0 and (optionally) standard dev at 1 ) and linear regression with regularization

   ```python
   from sklearn.preprocessing import StandardScaler
   from sklearn.pipeline import Pipeline

   pip_reg = Pipeline([
      ("scaler", StandardScaler()), # Standard scaling step
      ("model", SGDRegressor()),    # Linear regression model with regularization
   ])
   ```

2. Define the hyperparameters and their ranges to be tested

   ```python
   grid_values = {
      "scaler__with_std": [True, False], # Whether to scale to unit variance (standard deviation of 1)
      "model__penalty": ["l1", "l2"], # L1 or L2 regularization
      "model__alpha": np.logspace(-2, 2, 200), # Regularization strength
   }
   ```

    > <question-title></question-title>
    >
    > 1. What is the data structure to define the hyperparameters and their range?
    > 2. What is the 


topic: statistics
tutorial: CNN
title: Deep Learning (Part 3) - Convolutional neural networks (CNN)
url: https://training.galaxyproject.org/training-material/topics/statistics/tutorials/CNN/tutorial.html
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: internal", "intro_deep_learning", "FNN", "RNN"]
tags_json: []
content_hash: 5809b640d9e928f0358053ef1bbfa17d
last_modified: 2026-02-26T21:12:18.155461
zenodo_link: https://zenodo.org/record/4697906
workflows_json: []
content:
Artificial neural networks are a machine learning discipline that have been successfully applied to problems
in pattern classification, clustering, regression, association, time series prediction, optimiztion, and control {% cite JainEtAl %}.
With the increasing popularity of social media in the past decade, image and video processing tasks have become very
important. The previous neural network architectures (e.g. feedforward neural networks) could not scale up to handle
image and video processing tasks. This gave way to the development of convolutional neural networks that are specifically
tailored to image and video processing tasks. In this tutorial, we explain what convolutional neural networks are, discuss
their architecture, and solve an image classification problem using MNIST digit classification dataset using a CNN in Galaxy.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Limitations of feedforward neural networks (FNN) for image processing

In a fully connected FNN (Figure 1), all the nodes in a layer are connected to all the nodes in the next layer. Each connection has a weight
$$ w_{i,j} $$ that needs to be learned by the learning algorithm. Lets say our input is a 64 pixel by 64 pixel grayscale image. Each grayscale
pixel is represented by 1 value, usually between 0 to 255, where 0 represents black, 255 represents white, and the values in between represent
various shades of gray. Since each grayscale pixel can be represented by 1 value, we say the *channel* size is 1. Such an image can be represented
by 64 X 64 X 1 = 4,096 values (rows X columns X channels). Hence, the input layer of a FNN processing such an image has 4096 nodes.

Lets assume the next layer has 500 nodes. Since all the nodes in subsequent layers are fully connected, we will have 4,096 X 500 = 2,048,000 weights
between the input and the first hidden layer. For complex problems, we usually need multiple hidden layers in our FNN, as a simpler FNN may not be
able to learn the model mapping the inputs to outputs in the training data. Having multiple hidden layers compounds the problem of having many weights
in our FNN. Having many weights makes the learning process more difficult as the dimension of the search space is increased. It also makes the training
more time and resource consuming and increases the likelihood of overfitting. This problem is further compunded for color images. Unlike grayscale
images, each pixel in a color image is represented by 3 values, representing red, green, and blue colors (Called RGB color mode), where every color
can be represented by various combination of these primary colors. Since each color pixel can be represented by 3 values, we say the **channel** size
is 3. Such an image can be represented by 64 X 64 X 3 = 12,288 values (rows X columns X channels). The number of weights between the input layer and
the first hidden layer with 500 nodes is now 12,288 X 500 = 6,144,000. It is clear that a FNN cannot scale to handle larger images
({% cite OSheaEtAl %}) and that we need a more scalable architecture.

![Neurons forming the input, output, and hidden layers of a multi-layer feedforward neural network](../../images/FFNN.png "Feedforward neural network with a hidden layer. Biases to hidden/output layer neurons are omitted for clarity")

Another problem with using FNN for image processing is that a 2 dimensional image is represented as a 1 dimensional vector in the input layer,
hence, any spatial relationship in the data is ignored. CNN, on the other hand, maintains the spatial structure of the data, and is better suited
for finding spatial relationships in the image data.

## Inspiration for convolutional neural networks

In 1959 Hubel and Wiesel conducted an experiment to understand how the visual cortex of the brain processes visual information ({% cite HubelWiesel %}).
They recorded the activity of the neurons in the visual cortex of a cat while moving a bright line in front of the cat. They noticed that some cells fire
when the bright line is shown at a particular angle and a particular location (They called these **simple** cells). Other neurons fired when the bright
line was shown regardless of the angle/location and seemed to detect movement (They called these **complex** cells). It seemed complex cells receive
inputs from multiple simple cells and have an hierarchical structure. Hubel and Wiesel won the Noble prize for their findings in 1981.

In 1980, inspired by hierarchical structure of complex and simple cells, Fukushima proposed *Neocognitron* ({% cite Fukishima %}), a hierarchical neural
network used for handwritten Japanese character recognition. Neocognitron was the first CNN, and had its own training algorithm. In 1989, LeCun et. al.
({% cite LeCunEtAl %}) proposed a CNN that could be trained by backpropagation algorithm. CNN gained immense popularity when they outperformed other
models at ILSVRC (ImageNet Large Scale Visual Recognition Challenge). ILSVRC is a competition in object classification and detection on hundreds of
object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty
institutions ({% cite RussakovskyEtAl %}). Notable CNN architectures that won ILSVRC are AlexNet in 2012 ({% cite KrizhevskyEtAl %}), ZFNet in 2013 (
{% cite ZeilerEtAl %}), GoogLeNet and VGG in 2014 ({% cite SzegedyEtAl %}, {% cite SimonyanEtAl %}), and ResNet in 2015 ({% cite HeEtAl %}).

## Architecture of CNN

A typical CNN has the following 4 layers ({% cite OSheaEtAl %})

1. Input layer
2. Convolution layer
3. Pooling layer
4. Fully connected layer

Please note that we will explain a 2 dimensional (2D) CNN here. But the same concepts apply to a 1 (or 3) dimensional CNN as well.

### Input layer

The input layer represents the input to the CNN. An example input, could be a 28 pixel by 28 pixel grayscale image. Unlike FNN, we do not
"flatten" the input to a 1D vector, and the input is presented to the network in 2D as a 28 x 28 matrix. This makes capturing
spatial relationships easier.

### Convolution layer

The convolution layer is composed of multiple filters (also called kernels). Filters for a 2D image are also 2D. Suppose we have
a 28 pixel by 28 pixel grayscale image. Each pixel is represented by a number between 0 and 255, where 0 represents the color black,
255 represents the color white, and the values in between represent different shades of gray. Suppose we have a 3 by 3 filter (9
values in total), and the values are randomly set to 0 or 1. Convolution is the process of placing the 3 by 3 filter on the top left
corner of the image, multiplying filter values by the pixel values and adding the results, moving the filter to the right one pixel at
a time and repeating this process. When we get to the top right corner of the image, we simply move the filter down one pixel and
restart from the left. This process ends when we get to the bottom right corner of the image.

![A 3 by 3 filter applied to a 4 by 4 image, resulting in a 2 by 2 image](../../images/Conv_no_padding_no_strides.gif "A 3 by 3 filter applied to a 4 by 4 image, resulting in a 2 by 2 image ({% cite DumoulinVisin %})")

Covolution operator has the following parameters:

1. Filter size
2. Padding
3. Stride
4. Dilation
5. Activation function

Filter size can be 5 by 5, 3 by 3, and so on. Larger filter sizes should be avoided as the learning algorithm needs to learn filter values (weights),
and larger filters increase the number of weights to be learned (more compute capacity, more training time, more chance of overfitting). Also, odd
sized filters are preferred to even sized filters, due to the nice geometric property of all the input pixels being around the output pixel.

If you look at Figure 2 you see that after applying a 3 by 3 filter to a 4 by 4 image, we end up with a 2 by 2 image -- the size of the image has gone
down. If we want to keep the resultant image size the same, we can use *padding*. We pad the input in every direction with 0's before applying the
filter. If the padding is 1 by 1, then we add 1 zero in evey direction. If its 2 by 2, then we add 2 zeros in every direction, and so on.

![A 3 by 3 filter applied to a 5 by 5 image, with padding of 1, resulting in a 5 by 5 image](../../images/Conv_same_padding_no_strides.gif "A 3 by 3 filter applied to a 5 by 5 image, with padding of 1, resulting in a 5 by 5 image ({% cite DumoulinVisin %})")

As mentioned before, we start the convolution by placing the filter on the top left corner of the image, and after multiplying filter and image
values (and adding them), we move the filter to the right and repeat the process. How many pixels we move to the right (or down) is the *stride*.
In figure 2 and 3, the stride of the filter is 1. We move the filter one pixel to the right (or down). But we could use a different stride. Figure 4
shows an example of using stride of 2.

![A 3 by 3 filter applied to a 5 by 5 image, with stride of 2, resulting in a 2 by 2 image](../../images/Conv_no_padding_strides.gif "A 3 by 3 filter applied to a 5 by 5 image, with stride of 2, resulting in a 2 by 2 image ({% cite DumoulinVisin %})")

When we apply a, say 3 by 3, filter to an image, our filter's output is affected by pixels in a 3 by 3 subset of the image. If we like to have a
larger *receptive field* (portion of the image that affect our filter's output), we could use *dilation*. If we set the dilation to 2 (Figure 5),
instead of a contiguous 3 by 3 subset of the image, every other pixel of a 5 by 5 subset of the image affects the filter's output.

![A 3 by 3 filter applied to a 7 by 7 image, with dilation of 2, resulting in a 3 by 3 image](../../images/Conv_dilation.gif "A 3 by 3 filter applied to a 7 by 7 image, with dilation of 2, resulting in a 3 by 3 image ({% cite DumoulinVisin %})")

After the filter scans the whole image, we apply an activation function to filter output to introduce non-linearlity. The preferred activation function
used in CNN is ReLU or one its variants like Leaky ReLU ({% cite NwankpaEtAl %}). ReLU leaves pixels with positive values in filter output as is, and
replacing negative values with 0 (or a small number in case of Leaky ReLU). Figure 6 shows the results of applying ReLU activation function to a filter
output.

![Two matrices representing filter output before and after ReLU activation function is applied](../../images/Conv_ReLU.png "Applying ReLU activation function to filter output")

Given the input size, filter size, padding, stride and dilation you can calculate the output size of the convolution operation as below.

$$ \frac{(\text{input size} - \text{(filter size + (filter size -1)*(dilation - 1)})) + (2*padding)}{stride} + 1 $$

![One matrix representing an input vector and another matrix representing a filter, along with calculation for single input channel two dimensional convolution operation](../../images/Conv_single_input_channel.png "Illustration of single input channel two dimensional convolution")

Figure 7 illustrates the calculations for a convolution operation, via a 3 by 3 filter on a single channel 5 by 5 input vector (5 x 5 x 1). Figure 8
illustrates the calculations when the input vector has 3 channels (5 x 5 x 3). To show this in 2 dimensions, we are displaying each channel in input
vector and filter separately. Figure 9 shows a sample multi-channel 2D convolution in 3 dimensions.

![Three matrices representing an input vector and another three matrices representing a filter, along with calculation for multiple input channel two dimensional convolution operation ](../../images/Conv_multiple_input_channel.png "Illustration of multiple input channel two dimensional convolution")

As Figures 8 and 9 show the output of a multi-channel 2 dimensional filter is a single channel 2 dimensional image. Applying *multiple* filters to the
input image results in a multi-channel 2 dimensional image for the output. For example, if the input image is 28 by 28 by 3 (rows x columns x channels),
and we apply a 3 by 3 filter with 1 by 1 padding, we would get a 28 by 28 by 1 image. If we apply 15 filters to the input image, our output would be 28
by 28 by 15. Hence, the number of filters in a convolution layer allows us to increase or decrease the channel size.

![Multiple cubes representing input vector, filter, and output in a 3 channel 2 dimensional convolution operation](../../images/Conv_multiple_channel_3d.gif "Three dimensional illustration of multiple input channel two dimensional convolution (Source: https://thomelane.github.io/convolutions/2DConvRGB.html)")

### Pooling layer

The pooling layer performs down sampling to reduce the spatial dimensionality of the input. This decreases the number of parameters, which in turn
reduces the learning time and computation, and the likelihood of overfitting. The most popular type of pooling is *max pooling*. Its usually a 2 by 2
filter with a stride of 2 that returns the maximum value as it slides over the input data (similar to convolution filters).

### Fully connected layer

The last layer in a CNN is a fully connected layer. We connect all the nodes from the previous layer to this fully connected layer, which is responsible
for classification of the image.

![A convolutional neural network with 3 convolution layers followed by 3 pooling layers](../../images/Conv_CNN.png "A convolutional neural network with 3 convolution layers followed by 3 pooling layers ({% cite OSheaEtAl %})")

As shown in Figure 10, a typical CNN usually has more than one convolution layer plus pooling layer. Each convolution plus pooling layer is responsible
for feature extraction at a different level of abstraction. For example, the filters in the first layer could detect horizental, vertical, and diagonal
edges. The filters in the next layer could detect shapes, and the filters in the last layer could detect collection of shapes. Filter values are randomly
initialized and are learned by the learning algorithm. This makes CNN very powerful as they not only do classification, but can also automatically do
feature extraction. This distinguishes CNN from other classification techniques (like Support Vector Machines), which cannot do feature extraction.

## MNIST dataset

The MNIST database of handwritten digits ({% cite LeCunMnist %}) is composed of a training set of 60,000 images and a test set of 10,000 images. The digits
have been size-normalized and centered in a fixed-size image (28 by 28 pixels). Images are grayscale, where each pixel is represented by a number between
0 and 255 (0 for black, 255 for white, and other values for different shades of gray). MNIST database is a standard image classification dataset and is used
to compare various Machine Learning techniques.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Make sure you have an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to make it easy to recognize
>
>    > <tip-title>Rename a history</tip-title>
>    >
>    > * Click on the title of the history (by default the title is `Unnamed history`)
>    >
>    >   ![Renaming history]({% link shared/images/rename_history.png %})
>    >
>    > * Type `Galaxy Introduction` as the name
>    > * Press <kbd>Enter</kbd>
>    >
>    {: .tip}
>
>
> 3. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/X_train.tsv
>    {{ page.zenodo_link }}/files/y_train.tsv
>    {{ page.zenodo_link }}/files/X_test.tsv
>    {{ page.zenodo_link }}/files/y_test.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Rename the datasets as `X_train`, `y_train`, `X_test`, and `y_test` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Check that the datatype of all four datasets is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

## Classification of MNIST dataset images with CNN

In this section, we define a CNN and train it using MNIST dataset training data. The goal is to learn a model such that given an image
of a digit we can predict whether the digit (0 to 9). We then evaluate the trained CNN on the test dataset and plot the confusion matrix.

In order to train the CNN, we must have the One-Hot Encoding (OHE) representation of the training
labels. This is needed to calculate the categorical cross entropy loss function. OHE encodes labels
as a **one-hot** numeric array, where only one element is 1 and the rest are 0's. For example, if
we had 3 fruits (apple, orange, banana) and their labels were 1, 2, and 3, the OHE
represntation of apple would be (1,0,0), the OHE representation of orange would be (0,1,0), and the
OHE representation of banana would be (0,0,1). For apple with label 1, the first element of array
is 1 (and the rest are 0's); For Orange with label 2, the second element of the array is 1 (and the
rest are 0's); And for Banana with label 3, the third element of the array is 1 (and the rest are 0's).
We have 10 digits in our dataset and we would just have an array of size 10, where only one
element is 1, corresponding to the digit, and the rest are 0's.

### Create One-Hot Encoding (OHE) representation of training labels

> <hands-on-title>One-Hot Encoding</hands-on-title>
>
> - {% tool [To categorical](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_to_categorical/sklearn_to_categorical/1.0.10.0) %}
>    - *"Input file"* : Select `y_train`
>    - *"Does the dataset contain header?"* : Select `No`
>    - *"Total number of classes"*: Select `10`
>    - Click *"Run Tool"*
>
{: .hands_on}

### Create a deep learning model architecture

> <hands-on-title>Model config</hands-on-title>
>
> - {% tool [Create a deep learning model architecture](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/1.0.10.0) %}
>    - *"Select keras model type"*: `sequential`
>    - *"input_shape"*: `(784,)`
>    - In *"LAYER"*:
>        - {% icon param-repeat %} *"1: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Reshape`
>                - *"target_shape"*: `(28,28,1)`
>        - {% icon param-repeat %} *"2: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `64`
>                - *"kernel_size"*: `3`
>                - *"Activation function"*: `relu`
>                - *"Type in key words arguments if different from the default"*: `padding='same'`
>        - {% icon param-repeat %} *"3: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"4: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `32`
>                - *"kernel_size"*: `3`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"5: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"6: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Flatten`
>        - {% icon param-repeat %} *"7: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `10`
>                - *"Activation function"*: `softmax`
>    - Click *"Run Tool"*
{: .hands_on}

Each image is passed in as a 784 dimensional vector (28 x 28 = 784). The reshape layer reshapes it into (28, 28, 1) dimensions -- 28 rows (image height), 28 columns (image width), and
1 channel. Channel is 1 since the image is grayscale and each pixel can be represented by one integer. Color images are represented by 3 integers (RGB
values) and have channel size 3. Our CNN then has 2 convolution + pooling layers. First convolution layer has 64 filters (output would be 64 dimensional),
and filter size is 3 x 3. Second convolutional layer has 32 filters (output would be 32 dimensional), and filter size is 3 x 3. Both pooling layers are
MaxPool layers with pool size of 2 by 2. Afterwards, we flatten the previous layers output (every row/colum/channel would be an individual node). Finally,
we add a fully connected layer with 10 nodes and use a softmax activation function to get the probability of each digit. Digit with the highest
probability is predicted by CNN. The model config can be downloaded as a JSON file.

### Create a deep learning model

> <hands-on-title>Model builder (Optimizer, loss function, and fit parameters)</hands-on-title>
>
> - {% tool [Create deep learning model](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/1.0.10.0) %}
>    - *"Choose a building mode"*: `Build a training model`
>    - *"Select the dataset containing model configuration"*: Select the *Keras Model Config* from the previous step.
>    - *"Do classification or regression?"*: `KerasGClassifier`
>    - In *"Compile Parameters"*:
>        - *"Select a loss function"*: `categorical_crossentropy`
>        - *"Select an optimizer"*: `Adam - Adam optimizer `
>        - *"Select metrics"*: `acc/accuracy`
>    - In *"Fit Parameters"*:
>        - *"epochs"*: `2`
>        - *"batch_size"*: `500`
>    - Click *"Run Tool"*
{: .hands_on}

A loss function measures how different the predicted output is versus the expected output. For multi-class classification problems, we use
*categorical cross entropy* as loss function. Epochs is the number of times the whole training data is used to train the model. Setting *epochs* to 2
means each training example in our dataset is used twice to train our model. If we update network weights/biases after all the training data is
feed to the network, the training will be very slow (as we have 60000 training examples in our dataset). To speed up the training, we present
only a subset of the training examples to the network, after which we update the weights/biases. The *batch_size* decides the size of this subset.
The model builder can be downloaded as a zip file.

### Deep learning training and evaluation

> <hands-on-title>Training the model</hands-on-title>
>
> - {% tool [Deep learning training and evaluation](toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.11.0) %}
>    - *"Select a scheme"*: `Train and Validate`
>    - *"Choose the dataset containing pipeline/estimator object"*: Select the *Keras Model Builder* from the previous step.
>    - *"Select input type:"*: `tabular data`
>        - *"Training samples dataset"*: Select `X_train` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>        - *"Dataset containing class labels or target values"*: `To categorical on y_train` (the output of the first step)
>        - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
>
{: .hands_on}

The training step generates 3 datasets: 1) accuracy of the trained model, 2) the trained model architecture file, downloadable as a tabular file, and 3) the trained model weights, downloadable as an hdf5 (or h5mlm) file. These files are needed for prediction in the next step.

### Model Prediction

> <hands-on-title>Testing the model</hands-on-title>
>
> - {% tool [Model Prediction](toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.11.0) %}
>    - *"Choose the dataset containing pipeline/estimator object"* : Select the trained model from the previous step.
>    - *"Select invocation method"*: `predict`
>    - *"Select input data type for prediction"*: `tabular data`
>    - *"Training samples dataset"*: Select `X_test` dataset
>    - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
{: .hands_on}

The prediction step generates 1 dataset. It's a file that has predictions (0 to 9 for the predicted digits) for every image in the test dataset.

### Machine Learning Visualization Extension

> <hands-on-title>Creating the confusion matrix</hands-on-title>
>
> - {% tool [Machine Learning Visualization Extension](toolshed.g2.bx.psu.edu/repos/bgruening/ml_visualization_ex/ml_visualization_ex/1.0.11.0) %}
>    - *"Select a plotting type"*: `Confusion matrix for classes`
>    - *"Select dataset containing the true labels"*": `y_test`
>    - *"Choose how to select data by column:"*: `All columns`
>    - *"Select dataset containing the predicted labels"*": Select `Model Prediction` from the previous step
>    - *"Does the dataset contain header:"*: `Yes`
>    - Click *"Run Tool"*
>
{: .hands_on}

**Confusion Matrix** is a table that describes the performance of a classification model. It lists the number of examples that were correctly
classified by the model, True positives (TP) and true negatives (TN). It also lists the number of examples that were classified as positive that
were actually negative (False positive, FP, or Type I error), and the number of examples that were classified as negative that were actually
positive (False negative, FN, or Type 2 error). Given the confusion matrix, we can calculate **precision** and
**recall** {% cite TatbulEtAl  %}. Precision is the fraction of predicted positives that are true positives (Precision = TP / (TP + FP)). Recall
is the fraction of true positives that are predicted (Recall = TP / (TP + FN)). One way to describe the confusion matrix with just one value is
to use the **F score**, which is the harmonic mean of precision and recall

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} $$

![Confusion matrix for MNIST image classification problem](../../images/Conv_confusion_matrix_new_model.png "Image classification confusion matrix")

Figure 11 is the resultant confusion matrix for our image problem. The first row in the table represents the *true* digit 0 class labels (we have
541 + 1 + 2 + 1 + 1 + 5 + 9 + 1 + 3 + 4 = 568 digit 0 images). The second row represents the *true* digit 1 class labels (Again, we have 665 + 5 + 5 + 3 + 2 + 2 + 4 = 686 digit 1
images). Similarly, you can count the true class labels for digits 2 to 9 by adding up the numbers in the coresponding row. The first column from the
left represents the *predicted* digit 0 class labels (Our CNN predicted 541 + 1 + 3 + 2 + 7 + 1 + 15 = 570 images as being digit 0). The second column from
the left represents the *predicted* digit 1 class labels (Our CNN predicted 665 + 1 + 4 + 2 + 7 + 5 + 1 + 4 = 689 images as being digit 1). Similarly,
you can count the predicted class labels for digits 2 to 9 by adding up the numbers in the corresponding column.

For digit 0, looking at the top-left green cell, we see that our CNN has correctly predicted 541 images as digit 0 (True positives). Adding the numbers
in the left most column besides the True positives (1 + 3 + 2 + 7 + 1 + 15 = 29), we see that our CNN has incorrectly predicted 29 images as being digit 0
(False positives). Adding the numbers on the top row besides the True positives (1 + 2 + 1 + 1 + 5 + 9 + 1 + 3 + 4 = 27), we see that our CNN has incorrectly predicted
27 digits 0 images as not being digit 0 (False negatives). Given these numbers we can calculate Precision, Recall, and the F score for digit 0 as
follows:

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} = \frac{541}{541 + 29} = 0.949 $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} = \frac{541}{541 + 27} = 0.952 $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} = \frac{2 * 0.949 * 0.952}{0.949 + 0.952} = 0.950 $$

You can calculate the Precision, Recall, and F score for other digits in a similar manner.

# Conclusion


In this tutorial, we explained the motivation for convolutional neural networks, explained their architecture, and discussed convolution
operator and its parameters. We then used Galaxy to solve an image classification problem using CNN on MNIST dataset.


topic: statistics
tutorial: regression_machinelearning
title: Regression in Machine Learning
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 34207d0392b62ad7ce2a9556d3c66ea2
last_modified: 2026-02-26T21:12:18.221815
zenodo_link: https://zenodo.org/record/2545213
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/regression_machinelearning/tutorial.html

content:
In this tutorial you will learn how to use Galaxy tools to solve [regression](https://en.wikipedia.org/wiki/Regression_analysis) problems. First, we will introduce the concept of regression briefly, and then examine linear regression, which models the relationship between a target variable and some explanatory variables (also known as independent variables). Next, we will discuss gradient boosting regression, an more advanced regressor model which can model nonlinear relationships between variables. Then, we will show how to visualize the results in each step. Finally, we will discuss how to train our models by finding the values of their parameters that minimize a cost function. We will work through a real problem to learn how the models and learning algorithms work.

In this tutorial we will build a regression model for chronological age prediction, based on DNA methylation. This is based on the work of [Jana Naue et al. 2017](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub), in which biomarkers are examined to predict the chronological age of humans by analyzing the DNA methylation patterns. Different machine learning algorithms are used in this study to make an age prediction.

It has been recognized that within each individual, the level of [DNA methylation](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3174260/) changes with age. This knowledge is used to select useful biomarkers from DNA methylation datasets. The [CpG sites](https://en.wikipedia.org/wiki/CpG_site) with the highest correlation to age are selected as the biomarkers (and therefore features for building a regression model). In this tutorial, specific biomarkers are analyzed by machine learning algorithms to create an age prediction model.



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Regression

Regression analysis attempts to determine the relationship between one target variable and a series of independent variables. A regressor learns the mapping between the features of a dataset row (i.e. the values for each of the independent variables) and its target value. It tries to fit a curve for the targets, which can be linear or non-linear. When the targets in a dataset are real numbers, the machine learning task is known as regression and each sample in the dataset has a real-valued output or target. Figure [1](#figure-1) shows how a regression curve is fitted which best explains all these points. Here, the curve is a straight line, so this is an example of linear regression. The regression task is to learn this curve, which explains the underlying distribution of the data points. The curve can then be used to make predictions for new data points; the target for a new sample will lie on the curve learned by the regression task.

![regression](images/regression.png "Regression fit through data points.")

Linear regression is a technique used to analyze a linear relationship between input variables (independent variables) and a single target variable. A linear relationship means that the data points tend to follow a straight line. Simple linear regression involves only a single input variable. Figure 2 shows a dataset with a linear relationship.

![regression](images/linear_regression_generated_data.png "Data points with a linear relationship.")

In linear regression, our goal is to find the line that best models the path of the data points. Figure 3 shows the dataset we used in Figure 2 with a line of best fit through it. The position of the line is determined by certain fitting coefficients (in this simple case, the gradient and intercept) and linear regression helps us pick appropriate values for these coefficients. In this example we have only one input variable and the problem is therefore simple linear regression. Note that in real problems we have more than one input variable. In this case, we call it multiple linear regression. Adding extra input variables just means that we’ll need to find more weights.

![regression](images/linear_regression_regressor.png "Regression fit through data points.")


> <details-title>Cost function</details-title>
>
> Once we have a prediction in regression problems, we need some way to tell if it’s reasonable or not. A cost function helps us do this. The cost function compares all the predictions against their actual values and provides a single number that we can use as a measure. Two common cost functions are the error and squared error. The error is how far our prediction is away from the actual value (Figure 4).
>
>![regression](images/cost_function.png "Distance between true value and prediction.")
> We know an error above the actual value and an error below the actual value should be about as bad as each other. The squared error makes this clear because both positive and negative values of the error result in a positive squared error. We will use the Mean Squared Error (MSE) function as our cost function. This function finds the average squared error value for all of our data points. Cost functions are important to us because they measure how accurate our model is against the target values.
>
{: .details}


# Analyze DNA methylation dataset

As a benchmark, we will use the [DNA methylation dataset](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub) to predict the chronological age. One important reason to choose this dataset for an age prediction task is that DNA methylation changes with age and this change occurs at specific CpG sites in humans. 
It has been recognized that DNA methylation analysis, which mostly occurs in a CpG sequence context, can give additional information besides the DNA profile.  It has been shown that DNA methylation changes with age within each individual. This alteration in DNA methylation occurs at specific CpG sites in all individuals, but with individual differences in “speed”, showing more DNA methylation differences in older twins compared to young ones.

Epigenomic and phenotypic changes which are age-dependent are also contained in these cells. This knowledge is used to select useful biomarkers from the DNA methylation dataset. The CpG sites with the highest correlation to age are selected as biomarkers (features). In this study, specific biomarkers are analyzed by machine learning algorithms to create an age prediction model.

In this tutorial, we will apply a couple of ([scikit-learn](https://scikit-learn.org/stable/)) machine learning tools to DNA methylation datasets to predict the chronological age of humans.

## Get training and test datasets

Whole blood samples are collected from humans with their ages falling in the range 18-69 and the best age-correlated CpG sites in the genome are chosen as features. The dataset is divided into two parts - training and test sets. The training set is used to train a regressor and the test set is used to evaluate the performance of the trained model. We proceed with the analysis by uploading new datasets and creating a new history. 

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/record/2545213/files/train_rows.csv
>    https://zenodo.org/record/2545213/files/test_rows_labels.csv
>    https://zenodo.org/record/2545213/files/test_rows.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets as `train_rows`, `test_rows_labels` and `test_rows` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype of all the three datasets is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

The datasets from this study contain features (present as columns). The last column in the dataset refers to `Age`, which is used as labels/targets. Since the targets are real numbers, the machine learning task becomes a regression problem. Using these features and targets, a model is built which learns a mapping between these input features and targets. The training set contains `208` rows corresponding to individual samples and `13` features (age-correlated CpG sites in DNA methylation dataset). The last column is `Age`. The test set contains `104` rows and the same number of features as the training set. The `Age` column in the test set is predicted after training on the training set. Another dataset `test_rows_labels` contains the true age values of the test set which is used to compute scores between true and predicted age.
The `train_rows` contains a column `Age` which is the label or target. We will evaluate our model on `test_rows` and compare the predicted age with the true age in `test_rows_labels`
{: .comment}

> <details-title>Test-train data splitting data within Galaxy</details-title>
> For this task, test and training datasets are provided. However, if you want to analyze your own data, you can check out the **Split Dataset** tool, which can split a dataset into training and test subsets with a number of advanced options.
>
{: .details}

## Learn the linear model from training dataset

At the first step, to learn the mapping between several features and the targets, we will apply a linear regressor. It learns features from the training dataset and maps all the rows to their respective targets. The process of mapping gives a trained model.

The dataset is divided into two parts - training and test sets. The training set is used to train a regressor and the test set is used to evaluate the performance of the trained model.

> <hands-on-title>Train a model</hands-on-title>
>
> 1. {% tool [Generalized linear models for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_generalized_linear/sklearn_generalized_linear/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Select a linear method"*: `Linear Regression model`
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Age`
>             - {% icon param-file %} *"Dataset containing class labels or target values"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Age`
> 2. Rename the generated file to `LinearRegression_model`
{: .hands_on}

> <question-title></question-title>
>
> What is learned by the linear regressor?
>
> > <solution-title></solution-title>
> >
> > The linear regressor learns the coefficients of the best fit line to the data.
> > 
> {: .solution}
>
{: .question}

## Predict age using test dataset

After learning on the training dataset, we should evaluate the performance on the test dataset to know whether the model created by the learning algorithm from the training dataset is good or not. This model is used to predict a new sample and a similar accuracy is expected.

Now, we will predict age in the test dataset using this model in order to see if the model has learned important features which can be generalized on a new dataset. The test dataset (`test_rows`) contains the same number of features but does not contain the `Age` column; the age will instead be predicted using the trained model.


> <hands-on-title>Predict targets using the model</hands-on-title>
>
> 1. {% tool [Generalized linear models for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_generalized_linear/sklearn_generalized_linear/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `LinearRegression_model`
>       - {% icon param-file %} *"Data (tabular)"*: `test_rows`
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `predicted_data_linear`
{: .hands_on}

## Visualize the prediction

We will evaluate the predictions by comparing them to the expected targets. In the previous step, we generated predictions for the test dataset (`predicted_data_linear`). We have one more dataset (`test_rows_labels`) which contains the true age values of the test set. Using the true and predicted values of age in the test set, we will verify the performance by analyzing the plots. As you can see, `predicted_data_linear` has no header, so first we need to remove the header from `test_rows_labels` to perform the comparison.

> <hands-on-title>Remove the header</hands-on-title>
>
> 1. **Remove beginning of a file** {% icon tool %} with the following parameters:
>       - {% icon param-file %} *"Remove first"*: `1`
>       - {% icon param-file %} *"from"*: `test_rows_labels`
> 2. Rename the generated file to `test_rows_labels_without_header`
{: .hands_on}


Now we visualize and analyze the predictions using the **Plot actual vs predicted curves and residual plots** tool in Galaxy.
Hint: Please find the above tool in "Graph/Display data" tool section in Galaxy.

> <hands-on-title>Check and visualize the predictions</hands-on-title>
> {% tool [Plot actual vs predicted curves and residual plots](toolshed.g2.bx.psu.edu/repos/bgruening/plotly_regression_performance_plots/plotly_regression_performance_plots/0.1) %}:
>    - {% icon param-file %} *"Select input data file"*: `test_rows_labels_without_header`
>    - {% icon param-file %} *"Select predicted data file"*: `predicted_data_linear`
{: .hands_on}

The visualization tool creates the following plots:

1. True vs. predicted targets curves: In this plot the corresponding points in both these curves should be close to each other for a good regression performance. We can see that the plot shows this behaviour.

    ![true_predicted](images/true_pred_curves.png "True vs predicted targets curves. ")

2. [Scatter plot](https://towardsdatascience.com/everything-you-need-to-know-about-scatter-plots-for-data-visualisation-924144c0bc5) for true vs. predicted targets: We can see in the scatter plot (figure [6](#figure-6)) that most of the points lie along the x=y curve. It means that the true and predicted ages are close to each other. The root mean square error (`RMSE`) is `4.1` and the R2 score is `0.93`.

    ![scatter_plot](images/true_vs_pred_scatter.png "Scatter plot for true vs. predicted targets.")

3. [Residual plot](http://docs.statwing.com/interpreting-residual-plots-to-improve-your-regression/) between residual (predicted - true) and predicted targets: The residual plot shown in figure [7](#figure-7) is generated to see if there is any visible pattern between residual (predicted age - true age) and predicted age. For a good regression performance, this plot should exhibit a random pattern and the points should be symmetrically distributed along the y=0 line.

    ![residual_plot](images/residual_plot.png "Residual plot between residual (predicted - true) and predicted targets. The plot shows a random pattern of points.")

These plots are important to visualize the quality of regression and the true and predicted targets - how close or far they are from each other. The closer they are, the better the prediction.

> <details-title>R2 (coefficient of determination)</details-title>
> In both the parts, learning on datasets is done using cross-validation and [R2](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html) scoring metric is used to evaluate the performance of the trained model. The closer it is to 1.0, the better it is. If it is negative, then the trained model is not good. To infer how its values exhibit model performance, we can compare the figures [8](#figure-8) and [9](#figure-9). In both the plots, the true and predicted targets are plotted in a scatter plot. For a good model, most of the points should lie along the `x = y` line as the true and predicted targets are close to each other. In figure [8](#figure-8), we can see that the points are scattered and do not show any pattern. Therefore, the R2 score is `-0.06`. On the other hand, figure [9](#figure-9) shows a better pattern as most of the points lie along the line and the R2 score is almost `1.0`. For RNA-seq dataset, we will compute the cross-validated R2 score using the training set and for DNA methylation dataset, we will compute the R2 score for the test set.
>
> ![model_bad](../../images/age-prediction-with-ml/model_bad.png "This shows an example of a bad model as most of the points are scattered.")
>
>![model_good](../../images/age-prediction-with-ml/model_good.png "This shows an example of a good model as most of the points lie along the x = y line.")
>
{: .details}


> <question-title></question-title>
>
> Inspect the plots. What can you say about the predictions?
>
> > <solution-title></solution-title>
> >
> > Figures 5, 6 and 7 show that the prediction is acceptable and the predicted age lies about close to the true age, but the reults can be improved by using better algorithms such as ensemble-based regressors. 
> >
> {: .solution}
{: .question}


## Using ensemble methods for regression

Linear regression is a useful technique but isn’t always the right choice for our real problems. Linear regression is a good choice when there is a linear relationship between your independent and dependent variables and you are trying to predict continuous values.

It is not a good choice when the relationship between independent and dependent variables is more complicated. For example, Figure 10 shows a dataset that does not have a linear relationship so linear regression would not be a good choice.

![regression](images/nonlinear_regression_generated_data.png "Data points with nonlinear relationship. ")

To learn the mapping between several features and the targets, in the next step, we will apply the
[Gradient boosting regressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor). It is an ensemble-based regressor because its prediction is the collective performance of multiple weak learners (e.g. decision trees). It learns features from training dataset and maps all the rows to their respective targets (real numbers). The process of mapping gives a trained model.

> <comment-title></comment-title>
> An [*ensemble*](https://scikit-learn.org/stable/modules/ensemble.html#ensemble) method uses multiple learning models internally for better predictions and boosting is a method of converting weak learners into strong learners.
{: .comment}

[Jana Naue et al. (2017)](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub) used [Random Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor) (another ensemble-based regressor) as regressor and we can conclude from this study that an ensemble-based regressor works well on this DNA methylation dataset. Therefore, we will use gradient boosting to build a prediction model.

Like the random forest method, gradient boosting is an ensemble-based regressor, because it uses multiple decision tree regressors internally and makes predictions by taking the collective performances of predictions made by multiple decision trees. It has a good predictive power and is robust to outliers. It creates an ensemble of weak learners (decision trees) and iteratively minimizes errors. One disadvantage, which comes from its basic principle of boosting, is that it cannot be parallelized. *Hint:* Please find the following tool in "Machine learning" tool section in Galaxy.

> <hands-on-title>Train a model</hands-on-title>
>
> 1. {% tool [Ensemble methods for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_ensemble/sklearn_ensemble/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Select an ensemble method"*: `Gradient Boosting Regressor` (*Note:* choose `Gradient Boosting Regressor` not `Gradient Boosting Classifier`)
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Age`
>             - {% icon param-file %} *"Dataset containing class labels or target values"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Age`
> 2. Rename the generated file to `gradient_boosting_model`
{: .hands_on}

> <question-title></question-title>
>
> What is learned by the gradient boosting regressor?
>
> > <solution-title></solution-title>
> >
> > **Gradient boosting** regressor learns multiple attributes like **feature_importances_** (weight for each feature/column),
> > **oob_improvement_** (which stores incremental improvements in learning), **estimators_** (collection of weak learners) and a few more.
> > These attributes are used to predict the target for a new sample and are stored in the trained model. They can be accessed by using the **Estimator attributes** tool on the `gradient_boosting_model` dataset.
> > 
> {: .solution}
>
{: .question}

After learning on the training dataset, we should evaluate the performance on the test dataset.

> <hands-on-title>Predict targets using the model</hands-on-title>
>
> 1. {% tool [Ensemble methods for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_ensemble/sklearn_ensemble/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `gradient_boosting_model`
>       - {% icon param-file %} *"Data (tabular)"*: `test_rows`
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `predicted_data_gradient_boosting`
{: .hands_on}

Now we can visualize and analyze the predictions using the **Plot actual vs predicted curves and residual plots** tool.
> <hands-on-title>Check and visualize the predictions</hands-on-title>
> {% tool [Plot actual vs predicted curves and residual plots](toolshed.g2.bx.psu.edu/repos/bgruening/plotly_regression_performance_plots/plotly_regression_performance_plots/0.1) %}:
>    - {% icon param-file %} *"Select input data file"*: `test_rows_labels` (*Note:* use the `test_rows_labels` dataset, not the `test_rows_labels_without_header` one)
>    - {% icon param-file %} *"Select predicted data file"*: `predicted_data_gradient_boosting`
{: .hands_on}


 ![scatter_plot](images/true_vs_pred_scatter_ensemble.png "Scatter plot for true vs. predicted targets using gradient boosting regressor.")

R2 score is 0.93, the same as the linear model, but the RMSE is smaller (3.85), which tells us that the predicted targets are closer to the true targets.

## Create data processing pipeline

In the final step, we will create a pipeline learner with the **Pipeline builder** tool but this time, we just specify the regressor. The **Pipeline builder** tool will wrap this regressor and return a `h5mlm` file. This file also contains a list of tunable hyperparameters.

> <hands-on-title>Create pipeline</hands-on-title>
>
> {% tool [Pipeline builder](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_build_pipeline/sklearn_build_pipeline/1.0.11.0) %}:
>    - In *"Final Estimator"*:
>        - *"Choose the module that contains target estimator"*: `sklearn.ensemble`
>            - *"Choose estimator class"*: `GradientBoostingRegressor`
>            - *"Type in parameter settings if different from default"*: `random_state=42`
> 
>      `random_state` could be set to any arbitrary integer number; its purpose is to ensure a determistic and therefore reproducible result.
>
{: .hands_on}


### Search for the best values of hyperparameters

After the **New Pipeline/Estimator** dataset and its tunable hyperparameters are produced by the **Pipeline builder** tool, we will use the **Hyperparameter search** tool to find the best values for each hyperparameter. These values will lead us to create the best model based on the search space chosen for each hyperparameter. We use only one parameter `n_estimators` of `Gradient boosting` regressor for this task. This parameter specifies the number of boosting stages the learning process has to go through. The default value of `n_estimators` for this regressor is `100`. However, we are not sure if this gives the best accuracy. Therefore, it is important to try setting this parameter to different values to find the optimal one. We choose some values which are less than `100` and a few more than `100`. The hyperparameter search will look for the optimal number of estimators and gives the best-trained model as one of the outputs. This model is used in the next step to predict age in the test dataset.


> <details-title>5-fold cross-validation</details-title>
>
> This is a model validation technique which estimates the performance of a predictive model on an unseen dataset. A dataset is divided into `5` folds and these folds are categorized into training and validation sets. The idea of cross-validation is shown in figure [12](#figure-12). The complete dataset is divided into `5` equal parts. 4 out of the 5 parts are used for training and the remaining 1 part is used for validating the performance of training. This is done for `5` folds/iterations; each time the validation set (1/5 of the dataset) is different. In all five folds, the complete dataset is used for training and validation. The final validation performance is averaged over `5` folds.
>
> ![5fold_cv](../../images/age-prediction-with-ml/5fold_cv.png "5-fold cross validation. ")
>The image demonstrates how the 5-fold cross-validation works. The complete dataset is divided into 5 equal parts/folds. 4 parts (80%) of the data (training set shown in yellow) are used for training the model and the remaining one part is used for evaluating (validation set shown in blue) the trained model. This is repeated for 5 times till every part/fold is used as the validation set. The accuracies computed for different validation folds are averaged to give 5-fold cross-validation accuracy.
>
{: .details}

> <hands-on-title>Hyperparameter search</hands-on-title>
>
> {% tool [Hyperparameter search](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_searchcv/sklearn_searchcv/1.0.11.0) %}:
>    - *"Select a model selection search scheme"*: `GridSearchCV - Exhaustive search over specified parameter values for an estimator `
>        - {% icon param-files %} *"Choose the dataset containing pipeline/estimator object"*: `h5mlm` file (output of **Pipeline builder** {% icon tool %})
>        - *"Is the estimator a deep learning model?"*: `No`
>        - In *"Search parameters Builder"*:
>             - In *"Parameter settings for search"*:
>                 - {% icon param-repeat %} *"1: Parameter settings for search"*
>                    - *"Choose a parameter name (with current value)"*: `n_estimators: 100`
>                    - *"Search list"*: `[25, 50, 75, 100, 200]`
>        - In *"Advanced Options for SearchCV"*:
>            - *"Select the primary metric (scoring)"*: `Regression -- 'r2'`
>
>               A scoring metric can be set. In this tutorial, we use `Regression -- 'r2'`
>
>            - *"Select the cv splitter"*: `KFold`
>
>               There are different ways to split the dataset into training and validation sets. In our tutorial, we will use `KFold` which splits the dataset into `K` consecutive parts. It is used for cross-validation. It is set to `5` using another parameter `n_splits`.
>
>                - *"n_splits"*: `5`
>                - *"Whether to shuffle data before splitting"*: `Yes`
>                - *"Random seed number"*: `3111696`
>
>                   It is set to an integer and used to retain the randomness/accuracy when *"Whether to shuffle data before splitting"* is `Yes` across successive experiments.
>
>            - *"Raise fit error"*: `No`
>
>               While setting different values for a parameter during hyperparameter search, it can happen that wrong values are set, which may generate an error. To avoid stopping the execution of a regressor, it is set to `No` which means even if a wrong parameter value is encountered, the regressor does not stop running and simply skips that value.
>
>    - *"Select input type"*: `tabular data`
>        - {% icon param-files %} *"Training samples dataset"*: `train_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>            - *"Type header name(s)"*: `Age`
>        - {% icon param-files %} *"Dataset containing class labels or target values"*: `train_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `Select columns by column header name(s)`
>            - *"Type header name(s)"*: `Age`
>    - *"Whether to hold a portion of samples for test exclusively?"*: `Nope`
>    - *"Save best estimator?"*: `Fitted best estimator or Detailed cv_results_ from nested CV`
>
{: .hands_on}

> <question-title></question-title>
>
> What is the optimal number of estimators for the given dataset?
>
> Hint: Please look at the `mean_test_score` column in the tabular result from the **Hyperparameter search** tool.
>
> > <solution-title></solution-title>
> >
> > 75. (Even though the default value of the number of estimators for the gradient boosting regressor is `100`, `75` gives the best accuracy. That's why it is important to perform hyperparameter search to tune these parameters for any dataset). 50 estimators also give almost the same accuracy.
> >
> {: .solution}
>
{: .question}

Using the **Hyperparameter search** tool, we optimized our model, based on the training data. Now, we will predict the age from the test dataset using this model.

> <hands-on-title>Predict age</hands-on-title>
>
> 1. {% tool [Ensemble methods for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_ensemble/sklearn_ensemble/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>        - {% icon param-files %} *"Models"*: `h5mlm` file (output of **Hyperparameter search** {% icon tool %})
>        - {% icon param-files %} *"Data (tabular)"*: `test_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>
{: .hands_on}


Now we will verify the performance by creating and analyzing the plots.

> <hands-on-title>Create regression plots</hands-on-title>
>
> {% tool [Plot actual vs predicted curves and residual plots](toolshed.g2.bx.psu.edu/repos/bgruening/plotly_regression_performance_plots/plotly_regression_performance_plots/0.1) %}:
>    - {% icon param-files %} *"Select input data file"*: `test_rows_labels` tabular file
>    - {% icon param-files %} *"Select predicted data file"*: `tabular` file (output of **Ensemble methods for classification and regression** {% icon tool %})
>
{: .hands_on}


> <question-title></question-title>
>
> Inspect the plots. What can you say about the predictions?
>
> > <solution-title></solution-title>
> >
> > The figures show that the prediction is very good because the predicted age lies close to the true age. 
> {: .solution}
{: .question}

![Scatter plot](../../images/age-prediction-with-ml/scatter_plot.png "Scatter plot for true and predicted age for test set using hyperparameter search. ")

We can see in the scatter plot (figure [13](#figure-13)) that most of the points lie along the `x = y` curve. This tells us that the true and predicted ages are close to each other. The root mean square error (`RMSE`) is `3.74` and the R2 score is `0.94`.

![Residuals](../../images/age-prediction-with-ml/residual_plot.png "The plot shows the residuals (predicted age - true) age against the predicted age.")

As you can see in the figure, there is no visible pattern between the plotted points, which means our model is good.

![True vs predicted age](../../images/age-prediction-with-ml/true_vs_predicted_plot.png "True vs predicted age for all the samples in the test set. ")
We can see that the predicted values are close to the true values.


Figure [13](#figure-13) shows that we achieved an R2 score of `0.94` and root mean square score of `3.74` for the test set using Gradient boosting regressor. [Jana Naue et al. 2017](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub) mention a similar root mean square score (`3.93`) using the random forest regressor. The root mean square score shows the difference between the true and predicted age of humans. The R2 score (`0.94`) is close to the best achievable score of `1.0` which shows that the trained model is good. Overall, the second part of the analysis also shows that using the machine learning tools in Galaxy, we can achieve state-of-the-art predictions mentioned in the recent scientific studies.



# Conclusion
By following these steps, we learned how to perform regression and visualize the predictions using Galaxy's machine learning and plotting tools. The features of the training dataset are mapped to the real-valued targets. This mapping is used to make predictions on an unseen (test) dataset. 
The quality of predictions is visualized using multiple plots to ascertain the robustness of machine learning tasks. There are many other regressors in the machine learning suite which can be tried out on these datasets to find how they perform. Different datasets can also be analyzed using these regressors. The regressors have many parameters which can be altered while performing the analyses to see if they affect the prediction accuracy. It may be beneficial to perform a hyperparameter search to tune these parameters for different datasets.


topic: statistics
tutorial: loris_model
title: Galaxy Tabular Learner - Building a Model using Chowell clinical data
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["LORIS Score Model", "Machine Learning", "Tabular Learner"]
content_hash: e0fbcde410b1c1c4b44e2bd949f79fa0
last_modified: 2026-02-26T21:12:18.214153
zenodo_link: https://zenodo.org/records/13885908
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/loris_model/tutorial.html

content:
In this tutorial, we will build a new immunotherapy-response classifier with Galaxy Tabular Learner using a comprehensive dataset of patients treated with immune checkpoint blockade (ICB) and non-ICB-treated patients across 18 solid tumor types. The goal is to accurately predict patient responses to the treatment.

Rather than re-implementing the original LORIS pipeline, we treat the published LORIS LLR6 logistic regression model as a benchmark and use the Tabular Learner report to confirm model quality and to understand where and why results differ ({% cite Chang2024 %}).

We will follow three steps: (i) upload the preprocessed training and test tables to Galaxy, (ii) run Tabular Learner to train and select a best model, and (iii) compare discrimination, calibration, and threshold-dependent behavior against the LORIS LLR6 reference using the interactive report.

![schema of the whole process of training model and test.](../../images/loris_tutorial/tutorial_schema.png "Overview of the process steps to obtain the model from the Chowell dataset."){: style="width: 60%; display: block; margin-left: auto; margin-right: auto;"}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
> The [LORIS dataset](https://github.com/rootchang/LORIS/blob/main/02.Input/AllData.xlsx) contains clinical, pathologic, and genomic features from a diverse cohort of patients, providing a foundation for robust analysis and model development. The raw dataset (xlsx) includes 10 distinct cohorts:
>
> Chowell_train, 2) Chowell_test, 3) MSK1, 4) MSK2, 5) Kato_panCancer, 6) Shim_NSCLC, 7) Vanguri_NSCLC, 8) Ravi_NSCLC, 9) Pradat_panCancer, 10) MSK_nonICB.
>
> To keep this tutorial concise, we will focus on the Chowell_train cohort for model training and the Chowell_test cohort for testing the PanCancer model.
>
{:  .comment}

# Dataset composition (features) to train the model 

Before we begin the hands-on session, here’s a brief explanation of the features we’ll be using. These features were selected based on the findings of Chang et al. (2024), who identified them as the most important for training the model.

It's important to note that other features were tested in different combinations and model architectures. However, the authors concluded that the six features selected are the most valuable for achieving the best model performance.

## TMB (Tumor mutation burden)
Tumor mutation burden (TMB) is defined as the total number of somatic mutations within a specified region of the tumor genome. It has been recognized as a biomarker for predicting the efficacy of immune checkpoint blockade (ICB) in solid tumors. The [U.S. Food and Drug Administration](https://www.fda.gov/) (FDA) has approved a threshold of 10 mutations per megabase (mut/Mb) as a biomarker for response to ICB treatment.

In this dataset, TMB values range from 0 to over 368 mutations per megabase, with some extreme values, such as 368.6 and 93.5. To mitigate the influence of these outliers, TMB values will be truncated at 50 mut/Mb, meaning any value exceeding 50 will be capped at 50. This is crucial because extreme TMB values can disproportionately skew the model’s learning process, leading to unreliable predictions.

## Systemic Therapy History
This feature is a binary variable that indicates whether a patient received chemotherapy or targeted therapy prior to immunotherapy. It is coded as 1 if the patient had undergone such treatments before starting immunotherapy, and 0 if they had not.

## Albumin
The albumin feature represents the albumin levels measured in patients, which is an important biomarker often associated with nutritional status and liver function. The values are measured in grams per deciliter (g/dL) and typically range between 2.1 and 4.9 g/dL in the dataset. Higher levels of albumin are generally associated with better overall health and can serve as an indicator of a patient's ability to recover or respond to treatments like immunotherapy.

## Cancer Type
The CancerType feature represents the type of cancer diagnosed in each patient, which can vary significantly across the dataset. Common types include Non-Small Cell Lung Cancer (NSCLC), Small Cell Lung Cancer (SCLC), Melanoma, Endometrial cancer, and other cancer types such as Gastric, Colorectal, Renal, and Breast cancer. This feature is critical for understanding the heterogeneity of the patient cohort and may influence treatment decisions, response rates, and outcomes.

Incorporating this feature into a machine learning model requires translating the categorical CancerType into one-hot encoded variables. Each cancer type will be represented as a binary feature (0 or 1), with each type becoming a separate column in the dataset. This enables the model to interpret the presence or absence of a specific cancer type for each patient.

## NLR (blood neutrophil–lymphocyte ratio)
The neutrophil–lymphocyte ratio (NLR), a biomarker derived from the ratio of neutrophils to lymphocytes in the blood, is increasingly used in cancer research due to its association with inflammation and immune response. It can serve as a prognostic factor in various cancer types. Higher NLR values often indicate a poorer prognosis, potentially reflecting a more aggressive disease or impaired immune response.

In this dataset, NLR values range, for example, from 0.8 to 88, with several extreme outliers. To address this, NLR values will be truncated at 25, meaning any value above 25 will be capped at 25. This truncation is important for preventing extreme outliers from disproportionately influencing the machine learning model.

## Age
In predictive models for patient outcomes, age is a crucial feature because it is often correlated with various health factors and disease risks. As people age, their immune systems, metabolism, and ability to recover from illnesses may change, influencing how they respond to treatments, medications, or disease progression. Including age as a feature helps models account for the biological changes that occur over time and can improve the accuracy of predictions across different age groups.

However, there are limits to how predictive age might be, particularly for extreme values. For example, patients over a certain age may share similar health characteristics, and further increases in age may not significantly add predictive value. Truncating age to a maximum value (like 85) helps avoid overemphasizing small differences between very old patients, where the added predictive power might be negligible.

## Response
The Response feature is a categorical target variable indicating whether patients benefited from immune checkpoint blockade (ICB) therapy, classified as 0 (no benefit) or 1 (benefit). The model is trained using all previous features to predict patient outcomes when ICB is chosen as the treatment.

# Model Architecture Selection Process

In the article ({% cite Chang2024 %}), the model was trained using different architectures, including Decision Tree, Random Forest, Logistic Regression, and XGBoost, using the scikit-learn framework (full code is available here: [Chang et al., github repo](https://github.com/rootchang/LORIS/tree/main/code). In the end, the best model architecture was a Logistic LASSO (Least Absolute Shrinkage and Selection Operator) Regression model, fitting the six features discussed above. This model was named LLR6.

It is important to note that Tabular Learner has the capability of training different model architectures, just as was done by Chang et al., 2024, but automatically.There are two options:
1. **Tool-selected best model**: let Tabular Learner compare candidate classifiers and select the strongest performer under its evaluation protocol.
2. **Model specific**: if you want a model family, restrict the candidate list to specific algorithms via the tool option: `Only Select Classification Models if you don't want to compare all models,`.

Since the purpose of this tutorial is not to replicate the exact model, we will not use the model specific option. Instead, we will train all candidate models. We therefore expect to obtain results similar to those reported by Chang et al., with logistic regression performing best among the models evaluated.

# Prepare environment and get the data 
> <comment-title>Preprocessing the raw data</comment-title>
>
> The raw data published by ({% cite Chang2024 %}) can be found here: 
> [LORIS raw dataset](https://github.com/rootchang/LORIS/blob/main/02.Input/AllData.xlsx)
> 
> We preprocessed the raw data using a Python script to:
> 1) Extract the `Chowell_train` and `Chowell_test` tab from the excel file.
> 2) Select the 7 features (`TMB`, `Systemic Therapy History`, `Albumin`, `Cancer Type`, `NLR`, `Age`, and `Response`) important for building the model.
> 3) Truncate the values for `Age`, `NLR`, and `TMB`.
> 4) Encode `Cancer Type` using one-hot encoding. 
> 5) Save the dataset as a .tsv file.
>
> A Jupyter notebook for preprocessing can be found at Dockstore: [LORIS_preprocessing](https://dockstore.org/notebooks/github.com/paulocilasjr/pycaret-use-case/preprocessing:main?tab=info)
>
{:  .comment}

> <hands-on-title> Environment and Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *LORIS model classifier*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from Zenodo or from the shared data library
>
>    ```
>    https://zenodo.org/records/13885908/files/Chowell_train_Response.tsv
>    https://zenodo.org/records/13885908/files/Chowell_test_Response.tsv
>    ```
>>   <tip-title>Data Type</tip-title>
>>   Leave the `Type` field as `Auto-Detect` when uploading (it's the default).
>>
>    {: .tip}
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the data format assigned for the file is **tabular**.
>    If it is not, follow the Changing the datatype tip.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add a tag (`LORIS model dataset`) to the dataset corresponding to `Chowell_train_Response.tsv` and `Chowell_test_Response.tsv`
>    This is important to trace back on what dataset the model was built on.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Using Tabular Learner Tool
> <hands-on-title> Run 1: train a Tabular Learner model </hands-on-title>
>
> 1. {% tool [Tabular Learner](toolshed.g2.bx.psu.edu/repos/goeckslab/tabular_learner/tabular_learner/0.1.3) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset (CSV or TSV)"*: `Chowell_train_Response.tsv`
>    - {% icon param-file %} *"Test Dataset (CSV or TSV)"*: `Chowell_test_Response.tsv`
>    - {% icon param-file %} *"Select the target column"*: `C22: Response`
>    - {% icon param-file %} *"Task"*: `Classification`
>
> 2. Run the tool 
{: .hands_on}

> <hands-on-title> Run 2: Re-evaluate at a selected decision threshold </hands-on-title>
>
> Threshold-dependent metrics (accuracy, precision, recall, F1, MCC) change when the probability cutoff changes. For a transparent comparison to LORIS LLR6, do the following: 
>
> 1. {% tool [Tabular Learner](toolshed.g2.bx.psu.edu/repos/goeckslab/tabular_learner/tabular_learner/0.1.3) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset (CSV or TSV)"*: `Chowell_train_Response.tsv`
>    - {% icon param-file %} *"Test Dataset (CSV or TSV)"*: `Chowell_test_Response.tsv`
>    - {% icon param-file %} *"Select the target column"*: `C22: Response`
>    - {% icon param-file %} *"Task"*: `Classification`
>    - {% icon param-file %} *"Customize Default Settings?"*: `Yes`
>    - {% icon param-file %} *"Classification Probability Threshold"*: `0.25`
>
> 2. Run the tool 
{: .hands_on}


# Tool output files
After training and testing your model, you should see two new files in your history list:

- Tabular Learner Best Model: An HDF5 file (`.h5`) that stores the serialized best model (and preprocessing) so it can be reused without retraining.

- Tabular Learner Model Report: An interactive HTML report with sortable tables and plots summarizing setup, validation, test results, and feature importance.

A hidden CSV with the best model's hyperparameters (`best_model.csv`) is also produced; show hidden datasets if you want to download it.

For this tutorial, we will focus on the Tabular Learner Model Report.

# Tabular Learner Model Report 
The Tabular Learner HTML report provides a comprehensive, browser-ready overview of the trained model and its evaluation, including performance on the training, validation, and test data. The report consists of four tabs: Model Config Summary, Validation Summary, Test Summary, and Feature Importance.
![report tabs](../../images/loris_tutorial/report_tabs.png "Tabs in the Tabular Learner Report")

Below is a brief explanation of the content in each tab of the report.

> <tip-title>Model Config Summary Tab</tip-title>
>- Dataset Overview (classification only): Counts and percentages of each label in the train/validation/test splits.
>- Performance Summary (classification only): Train/validation/test metrics computed for the best model, using the selected probability threshold when applicable.
>- Setup Parameters: Records the key run settings (target column, train size, normalization, feature selection, cross validation, outlier/multicollinearity handling, polynomial features, class-imbalance handling, model list, and probability threshold).
>- Best Model Hyperparameters: The final parameter values for the selected model.
>
{: .tip}

> <tip-title>Validation Summary Tab</tip-title>
>- Train & Validation Summary: A sortable table comparing all candidate models on cross-validation metrics; includes a tuning summary when hyperparameter tuning is enabled.
>- Diagnostic Plots (vary by task and model availability): Learning curve, validation curve, calibration curve (classification), threshold plot (classification), and dimensionality-reduction or feature-selection visuals (e.g., t-SNE, manifold, RFE).
>
{: .tip}

> <tip-title>Test Summary Tab</tip-title>
>- Test Metrics Table: Holdout/test performance for the selected model.
>- Classification Plots: Confusion matrix, per-class report, ROC and PR curves (optionally annotated at the chosen threshold), lift curve, and cumulative precision.
>- Regression Plots: Residuals plot and prediction error distribution, plus a preview of true vs. predicted values.
>
{: .tip}

> <tip-title>Feature Importance Tab</tip-title>
>- Model-based Importance: Feature importance from the selected model (tree-based or coefficient-based when supported), with a SHAP summary plot to show global impact.
>- SHAP and Permutation Importance: Mean absolute SHAP impact and permutation importance derived from the explainer.
>- Partial Dependence Plots (PDPs): Per-feature effect plots to show how individual features influence predictions.
>- Feature Scope Table: Reports how many transformed features are used in SHAP when feature caps are applied.
>
{: .tip}

# Benchmarking Tabular Learner against the LORIS LLR6 model

The objective of this use case is to build a **new** model with Tabular Learner under a standardized Galaxy workflow and then use the published LORIS LLR6 model as a **benchmark baseline** ({% cite Chang2024 %}). Because both approaches use the same cohort split and predictor set, agreement in threshold-independent metrics (ROC-AUC, PR-AUC) supports model fidelity, while any differences in threshold-dependent metrics can often be explained by distinct probability cutoffs, calibration, or regularization choices recorded in the report.

> <tip-title>Robustness definition </tip-title>
>
> Some evidence is considered robust if it holds up across reasonable variations in analysis, while fragile evidence depends heavily on specific analytical choices, such as the inclusion or exclusion of certain observations or covariates.
>
> Robustness, therefore, refers to assessing the reliability of previous findings by applying different analytical approaches to the same dataset.>
>![table showing the difference of robustness to the others analysis](../../images/loris_tutorial/robustness.png  "Comparison of Robustness, Reproducibility, Generalizability, and Replicability Based on Data and Method Variations")
>
{: .tip}

## Classification Algorithms
A key feature of Tabular Learner is its capability to train and compare multiple models with minimal code. By default, Tabular Learner evaluates a diverse range of algorithms, including linear models, tree-based models, and ensemble methods, ranking these models based on their performance. The primary metric used to determine the best-performing model is the accuracy.

![Comparison results on the cross-validation set](../../images/loris_tutorial/model_comparison.png "Comparison of Model Performance Metrics Across Algorithms")

In many runs of this use case, logistic regression ranks among the top-performing models, consistent with the LORIS LLR6 baseline.

## Hyperparameters
The model from {% cite Chang2024 %} for the Pan-Cancer LLR6 model has the following hyperparameters set: C = 0.1, Class Weight = Balanced, L1 ratio = 1, max iter = 100, Penalty = Elasticnet, Solver = Saga.

The hyperparameter search performed by Galaxy-Tabular Learner resulted in slightly different settings for the model: C = 1.0, Class Weight = None, L1 ratio = None, max iter = 1000, Penalty = L2, Solver = LBFGS.

![Model hyperparameters table](../../images/loris_tutorial/hyperparameters.png "Table showing the hyperparameters set for the selected model")

The differences found in the hyperparameters guide the analysis towards testing the robustness of the LORIS Score proposed by {% cite Chang2024 %}.

> <comment-title>Hyperparameters meaning</comment-title>
>
> *Penalty* - Defines the type of regularization applied to the model to prevent overfitting. Options for linear and logistic regression are: `L1`, `L2`, and `Elasticnet`. Briefly, L1 (Lasso Regression) removes less important features, helping to overcome overfitting as well as performing dimension reduction. However, when most features (variables) in the model are useful, L2 (Ridge Regression) is used. Elasticnet regularization combines both L1 and L2, addressing multicollinearity while also enabling feature selection. When Elasticnet is selected, it opens the L1 Ratio parameter.
>
> *L1-Ratio* - Controls the balance between L1 and L2 penalties. A value of `1` uses purely L1 regularization, which encourages sparsity. A value of `0` uses purely L2.
>
> *C* - Is the inverse of the regularization strength. A smaller C (like `0.1`) implies stronger regularization, which prevents overfitting by penalizing large coefficients and also makes the model simpler (smaller coefficients). In contrast, higher values make the model more complex.
>
> *Solver* - Specifies the optimization algorithm used for fitting the model. `SAGA` is well-suited for large datasets and supports the Elasticnet penalty. It is also effective for sparse data and is fast for L1. `LBFGS` is a quasi-Newton optimization algorithm, efficient for smaller datasets, supports L2 regularization, but does not support Elasticnet.
>
> *Class Weight* - Is used to handle imbalanced classes by adjusting the weight associated with each class. Balanced adjusts these weights inversely proportional to class frequencies in the data, giving more weight to the minority class.
>
> *Max Iter* - Specifies the `maximum number` of iterations the solver will run before stopping. If convergence is achieved earlier, it will stop; if not, you may need to increase the value.
>
{:  .comment}

## Model Evaluation Metrics

When comparing Tabular Learner models to LORIS LLR6, separate metrics into two groups and compare both runs of Tabular Learner:

1. **Threshold-independent (primary for benchmarking)**  
   - **ROC-AUC** and **PR-AUC** summarize discrimination across all thresholds. In this use case, both Tabular Learner runs show stronger discrimination than LLR6 (AUC 0.76 vs 0.72; AUPRC 0.55 vs 0.53), and the two runs are identical on these metrics because the underlying model and data did not change.

2. **Threshold-dependent (report with the chosen cutoff)**  
   - **Accuracy and F1** shift when the probability cutoff changes. Moving from 0.50 (Run 1) to 0.25 (Run 2) lowers accuracy (0.79 → 0.67) but improves F1 (0.42 → 0.52).

The table below summarizes the key numbers:

| Model | Threshold | Accuracy | ROC-AUC | PR-AUC | F1 |
| --- | --- | --- | --- | --- | --- |
| LLR6 (reference) | 0.30 | 0.68 | 0.72 | 0.53 | 0.53 |
| Tabular Learner Run 1 | 0.50 | 0.80 | 0.76 | 0.55 | 0.42 |
| Tabular Learner Run 2 | 0.25 | 0.67 | 0.76 | 0.55 | 0.52 |

Use the following report components to interpret differences:

- **Model Config Summary**: confirm the split strategy, evaluation protocol, selected model, and tuned hyperparameters.
- **ROC and PR curves**: compare discrimination against the published baseline without relying on a specific threshold.
- **Calibration plot and probability diagnostics**: determine whether predicted probabilities are systematically over- or under-confident.
- **Threshold plot (precision/recall/F1 vs cutoff)**: select and justify an operating point (for example, maximize F1, or prioritize recall for clinical sensitivity).
- **Confusion matrix and classification report**: quantify how the chosen cutoff changes false positives and false negatives.

![Bar chart comparing LLR6 metrics vs. Tabular Learner (threshold = 0.25)](../../images/loris_tutorial/test_metrics_results.png "Comparison of LLR6 model metrics and Tabular Learner model metrics at a 0.25 threshold.")

- Accuracy
Accuracy measures the proportion of correct predictions among all predictions. In our experiments, Tabular Learner Run 1 achieved the highest accuracy (0.80), substantially higher than the published reference model LLR6 (0.68) and Run 2 (0.67). This indicates that, at its selected operating threshold, Run 1 produced more correct classifications overall. However, because accuracy depends on the chosen cutoff and can be influenced by class imbalance, it should be interpreted alongside threshold-dependent metrics (e.g., F1) and threshold-independent metrics (e.g., ROC-AUC, PR-AUC).

- Area Under the Curve (AUC)
ROC-AUC summarizes discrimination across all possible thresholds. Both Tabular Learner runs show higher ROC-AUC (0.76) than the published LLR6 baseline (0.72), indicating improved ability to rank positives above negatives irrespective of where the cutoff is set. Despite Run 1 having much higher accuracy than Run 2, their identical ROC-AUC suggests that the key difference is likely where the decision threshold was placed, rather than a large difference in underlying separability.

- Precision-Recall AUC (AUPRC)
Precision-Recall AUC (AUPRC or PR-AUC) is often more informative than ROC-AUC when positives are relatively rare. Both Tabular Learner runs have PR-AUC = 0.55, slightly higher than the reference (0.53), suggesting a modest improvement in performance in the precision–recall tradeoff over the baseline, independent of a single threshold choice.

- Threshold selection and F1 (Run 1 vs Run 2 vs reference)
The threshold determines the balance between false positives and false negatives. Although Run 1 attains higher accuracy, it has a lower F1 (0.42) than Run 2 (0.52) and the reference LLR6 (0.53). This pattern is consistent with Run 1 using a more conservative cutoff (0.50), which typically increases true negatives (improving accuracy) but can reduce recall and harm F1 if positives are missed. In contrast, Run 2 uses a lower threshold (0.25), which generally increases sensitivity and can improve F1 by capturing more positives, even if it introduces more false positives and lowers overall accuracy.The comparison between the Tabular Learner and Scikit-learn models demonstrates consistent performance across multiple evaluation metrics, such as accuracy, AUC, and F1 score. The slight differences in the results reflect the inherent flexibility of the LORIS Score, showing that a model built using two different frameworks—Tabular Learner and Scikit-learn—can still yield comparable performance.

Compared to the published model, our runs demonstrate that while discrimination (ROC-AUC/PR-AUC) is slightly improved, the choice of threshold materially changes the reported point metrics. Therefore, differences in accuracy and F1 across Run 1, Run 2, and LLR6 should be interpreted primarily as different operating points on similar underlying probability rankings, rather than as contradictions in model quality.

Overall, these results emphasize that (1) ROC-AUC and PR-AUC support comparability with, and slight improvement over, the published baseline, and (2) threshold selection must be explicitly justified because it can shift the balance between accuracy and clinically relevant sensitivity/precision, leading to substantially different conclusions from the same model outputs.

# Conclusion
In this tutorial, we demonstrated how to use the Galaxy-Tabular Learner Tool to build a machine learning model for ICB treatment patient selection using clinical data. We followed a structured approach consisting of:
- Uploading datasets (.tsv)
- Running the Tabular Learner
- Evaluating the model's performance using the metrics published by {% cite Chang2024 %} as the gold standard.

Throughout the process, we showcased how Tabular Learner simplifies the complexities of machine learning workflows, making them more accessible and efficient for users. Additionally, we explored the robustness of the LORIS Score by building a Logistic Regression model with different hyperparameters.

By the end of this tutorial, you should have a solid understanding of how to deploy a traditional machine learning model using tabular data and effectively interpret its results.


topic: statistics
tutorial: hyperdimensional_computing
title: Supervised Learning with Hyperdimensional Computing
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 464023e647043b1157c19194b27e5042
last_modified: 2026-02-26T21:12:18.200630
zenodo_link: https://doi.org/10.5281/zenodo.6467875
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/hyperdimensional_computing/tutorial.html

content:
`chopin2` ({% cite Cumbo2020 %}) implements a domain-agnostic supervised classification method based on the hyperdimensional (HD) computing paradigm. It is an open-source tool and its code is available on GitHub at [https://github.com/cumbof/chopin2](https://github.com/cumbof/chopin2).

In this tutorial, we are going to work on a dataset with microbial relative abundances (RA) and presence/absence information (BIN) computed on metagenomic samples collected from individuals affected by colorectal cancer (CRC) in a case/control scenario.
The main goal is to build a supervised classification model over the RA profiles with `chopin2` in order to discriminate case and control samples with a high accuracy.

The dataset with RA profiles has been produced with `MetaPhlAn3` ({% cite beghini2021integrating %}) and it is available through the `curatedMetagenomicData` package for R ({% cite pasolli2017accessible %}). The studies describing the analysis of the metagenomic samples are available in Nature Medicine ({% cite thomas2019metagenomic %} {% cite wirbel2019meta %}).
In particular, the dataset contains the profiles of 241 microbial species detected in 101 stool samples of patients affected by CRC, and 92 samples collected from the stool of healthy individuals.

The data we use in this tutorial is also available on [Zenodo](https://doi.org/10.5281/zenodo.6467875).

Please note that both the RA and BIN datasets in Zenodo have also been stratified according to the age category and sex of both case and control individuals. However, in this tutorial we are going to analyze the unstratified datasets, called `RA__ThomasAM__species.csv` and `BIN__ThomasAM__species.csv`.

> <agenda-title></agenda-title>
>
> A `chopin2` analysis is composed of three steps:
>
> 1. [Get the data](#get-the-data)
> 2. [Build a classification model](#build-a-classification-model)
> 3. [Feature selection](#feature-selection)
{: .agenda}

# Get the data

The first step consists in importing the RA and BIN datasets into a Galaxy history. As previously mentioned, RA refers to the dataset with microbial relative abundance profiles computed with `MetaPhlAn3`, while BIN refers to the dataset with presence/absence information about microbes in samples.

> <hands-on-title>Get the data</hands-on-title>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import datasets from [Zenodo](https://zenodo.org/record/7806264):
>    - RA dataset (`RA__ThomasAM__species.csv`)
>    - BIN dataset (`BIN__ThomasAM__species.csv`)
>
>    ```
>    https://zenodo.org/record/7806264/files/RA__ThomasAM__species.csv
>    https://zenodo.org/record/7806264/files/BIN__ThomasAM__species.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    When uploading the datasets, set the `type` to csv. Both datasets contain samples in the columns and microbial species in the rows. The RA dataset contains relative abundance profiles, thus values ranging from 0.0 to 100.0, while the BIN dataset contains presence/absence (i.e., 0 or 1) information computed on the relative abundance. It means that the value of the cell `ij` in the BIN matrix is 1 if the relative abundance of the species `i` in the sample `j` is greater than 0.0 in the RA matrix. Otherwise, it is 0.
>
> 3. Edit history item attributes {% icon galaxy-pencil %}
>    - Make sure dataset names are clear, like `RA__ThomasAM__species.csv` and `BIN__ThomasAM__species.csv`.
>    - If you did not previously set the datatype to csv, you can do that now under the convert tab.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
{: .hands_on}

# Build a classification model

Hyperdimensional computing is an emerging brain-inspired computing paradigm that deals with vectors in a high-dimensional space. Every kind of information is thus encoded into random 10,000-length vectors (usually binary or bipolar) that are combined together to represent complex concepts. The length of the vectors is usually in the order of 10,000 to guarantee their quasi-orthogonality.

There are usually three types of arithmetic operators that can be applied to combine vectors: bundling, binding, and permutation. They all have unique properties that must be taken into account when used ({% cite kanerva2009hyperdimensional %}).

For what concerns the bundling operator: (i) the resulting vector is similar to the input vectors, (ii) the more vectors that are involved in bundling, the harder it is to determine the component vectors, (iii) if several copies of any vector are included in bundling, the resulting vector is closer to the dominant vector than to the other components.

The binding operation is instead: (i) invertible (unbinding), (ii) it distributes over bundling, (iii) it preserves the distance, (iv) and the resulting vector is dissimilar to the input vectors.

Finally, the permutation operation is: (i) invertible, (ii) it distributes over bundling and any elementwise operation, (iii) it preserves the distance, and (iv) the resulting vector is dissimilar to the input vectors.

The set of vectors and arithmetic operators used for combining vectors is called vector-symbolic architecture.

`chopin2` {% icon tool %} implements a supervised classification model that works by encoding every observation in a dataset into high-dimensional vectors by combining values under their features. The model is built by collapsing the vector representation of the observations in the training set into a vector for each class of observations (e.g., in the case of the datasets retrieved in the previous step, classes are `CRC` and `control`). The classification model is then tested by computing the inner product between the vector representations of the observations in the test set against the two classes of vectors. Vectors are thus classified based on their closeness to the classes.

> <hands-on-title>Build a classification model with `chopin2`</hands-on-title>
> {% tool [chopin2](toolshed.g2.bx.psu.edu/repos/iuc/chopin2/chopin2/1.0.7+galaxy1) %} with the following configuration:
>    * {% icon param-file %} *"Select a dataset"*: `RA__ThomasAM__species.csv`;
>    * {% icon param-text %} *"Vector dimensionality"*: `10000`;
>    * {% icon param-text %} *"Levels"*: `100`;
>    * {% icon param-text %} *"Model retraining iterations"*: `10`;
>    * {% icon param-text %} *"Number of folds for cross-validation"*: `5`;
>    * {% icon param-select %} *"Enable feature selection"*: `Disabled`
>
> The number of levels is the number of random vectors that `chopin2` {% icon tool %} generates for encoding data. This must be carefully selected since the accuracy of the resulting model is strictly correlated to this parameter ({% cite Cumbo2020 %}).
>
> The tool will create a `summary` file in your history containing a few basic information about the generated classification model and, more importantly, its accuracy. You can open this file by clicking the {% icon galaxy-eye %} (eye) icon.
>
> We suggest that you repeat the same steps for building a classification model on the binary presence/absence profiles in `BIN__ThomasAM__species.csv`. In this case there is no need to use many levels, since the dataset contains only two possible values (i.e., 0 and 1). Thus, the number of levels must be changed to 2.
>
>    > <question-title></question-title>
>    >
>    > Compare the `summary` files generated by `chopin2` {% icon tool %} as the result of building a classification model over the `RA__ThomasAM__species.csv` and `BIN__ThomasAM__species.csv` datasets.
>    > 1. Is there a difference in the accuracy of the two models?
>    > 2. Does the difference in the number of levels have an impact on the running time and the final accuracy of the models?
>    >
>    >    > <solution-title></solution-title>
>    >    >
>    >    > 1. The accuracy of the model built on the `BIN__ThomasAM__species.csv` dataset is over 80%, while the accuracy of the model built on the `RA__ThomasAM__species.csv` dataset is around 75%.
>    >    > 2. The tool generates as many hyperdimensional vectors as the number of levels. In cases of a high number of levels, the tool could take a while to generate all the hyperdimensional vectors initially, but it does not affect its speed in the generation of the classification model. However, it could heavily affect the memory consumption. Additionally, the number of levels has an impact on the accuracy of the model. It is recommended to choose as many levels as the number of unique values in the datasets, by also taking into account the precision of the numerical values.
>    > {: .solution}
>    {: .question}
{: .hands_on}

# Feature selection

`chopin2` {% icon tool %} also implements a feature selection method based on the backward variable elimination strategy. It means that the tool will produce a classification model starting with the whole set of features in the dataset and iteratively remove those features that do not contribute to the accuracy of the model.

Be aware that this specific type of feature selection method could lead to the generation of thousands of classification models in order to determine the best features and discard those ones that do not significantly contribute to a good accuracy. However, despite the huge amount of computational resources required to run the algorithm, it can easily handle datasets with massive amounts of features.

> <hands-on-title>Identify the best features</hands-on-title>
> {% tool [chopin2](toolshed.g2.bx.psu.edu/repos/iuc/chopin2/chopin2/1.0.7+galaxy1) %} with the following configuration:
>    * {% icon param-file %} *"Select a dataset"*: `BIN__ThomasAM__species.csv`;
>    * {% icon param-text %} *"Vector dimensionality"*: `10000`;
>    * {% icon param-text %} *"Levels"*: `2`;
>    * {% icon param-text %} *"Model retraining iterations"*: `10`;
>    * {% icon param-text %} *"Number of folds for cross-validation"*: `5`;
>    * {% icon param-select %} *"Enable feature selection"*: `Enabled`
>
> We are going to focus on the `BIN__ThomasAM__species.csv` dataset only this time since `chopin2` {% icon tool %} produced a classification model with a better accuracy compared to that of the model built over the `RA__ThomasAM__species.csv` dataset.
>
> The tool will end up creating a `summary` file in your history (click on the {% icon galaxy-eye %} (eye) icon to check its content). It contains a line for each step of the backward variable elimination method. Every line corresponds to a classification model. Please note that the last column reports the features excluded in each step.
>
> A `selection` file will also appear in your history with the list of best features selected by `chopin2` {% icon tool %} (you can also inspect this file by clicking on the {% icon galaxy-eye %} (eye) icon). Some of them are well known to be actually linked in someway to the genesis and development of CRC as a simple search on the [Disbiome](https://disbiome.ugent.be/) ({% cite janssens2018disbiome %}) database can confirm. However, these features have been selected because they all contribute to define the best accuracy according to the classification model and the feature selection technique implemented in `chopin2` {% icon tool %}, and they do not necessarily have a biological relevance. Thus, this is not enough for identifying true possible biomarkers for CRC, and further statistical analyses must be performed to validate and support these findings.
>
>    > <question-title></question-title>
>    >
>    > Look at the list of microbial species (features) selected by `chopin2` {% icon tool %} reported in the `selection` file. Can you recognize any microbial species known to be linked in some way to the genesis and development of colorectal cancer?
>    >
>    >    > <solution-title></solution-title>
>    >    >
>    >    > _Clostridium symbiosum_, _Gemella morbillorum_, and _Parvimonas micra_, among the other selected species, are well known CRC-enriched bacteria. These species have all been proposed as biomarkers for an early detection of the disease ({% cite xie2017fecal %} {% cite yao2021new %} {% cite zhao2022parvimonas %}).
>    > {: .solution}
>    {: .question}
{: .hands_on}

> <details-title>Feature selection with Hyperdimensional Computing</details-title>
> `chopin2` {% icon tool %} is the first tool that implements a feature selection method based on the hyperdimensional computing paradigm.
{: .details}


topic: statistics
tutorial: flexynesis_classification
title: Modeling Breast Cancer Subtypes with Flexynesis
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 6fa94d26393861ea31950e6443218f3a
last_modified: 2026-02-26T21:12:18.179724
zenodo_link: https://zenodo.org/records/16287482
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/flexynesis_classification/tutorial.html

content:
Flexynesis represents a state-of-the-art deep learning framework specifically designed for multi-modal data integration in biological research ({% cite Uyar2024 %}). What sets Flexynesis apart is its comprehensive suite of deep learning architectures that can handle various data integration scenarios while providing robust feature selection and hyperparameter optimization.

Here, we use Flexynesis tool suite on a multi-omics dataset of Breast Cancer samples from the METABRIC consortium ({% cite metabric-website %}), one of the landmark breast cancer genomics studies available through cBioPortal ({% cite cbioportal-website %}). This dataset contains comprehensive molecular and clinical data from over 2,000 breast cancer patients, including gene expression profiles, copy number alterations, mutation data, and clinical outcomes. The data was downloaded from Cbioportal and randomly split into train (70% of the samples) and test (30% of the samples) data folders. The data files were processed to follow the same nomenclature.

This training is inspired from the original Flexynesis analysis notebook: [brca_subtypes.ipynb](https://github.com/BIMSBbioinfo/flexynesis/blob/main/examples/tutorials/brca_subtypes.ipynb).

> <warning-title>LICENSE</warning-title>
>
> Flexynesis is only available for NON-COMMERCIAL use. Permission is only granted for academic, research, and educational purposes. Before using, be sure to review, agree, and comply with the license.
> For commercial use, please review the Flexynesis license on GitHub and contact the [copyright holders](https://github.com/BIMSBbioinfo/flexynesis)
{: .warning}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

In this training we will use a multi-omics dataset from the METABRIC database downloaded through cBioPortal and split to 70% train and 30% test samples.

The multi-omics data consists of three data types:

- **Gene Expression (GEX)**: Measures the activity levels of genes by quantifying mRNA abundance. This data captures which genes are being actively transcribed in each sample.

- **Copy Number Alterations (CNA)**: Detects genomic regions that have been duplicated or deleted compared to the normal genome. Cancer cells often have chromosomal instabilities leading to gene amplifications (gains) or deletions (losses), which can drive tumor development and progression.

- **Clinical data (CLIN)**: Contains patient metadata including demographics, treatment history, tumor characteristics, and importantly for this tutorial, the breast cancer molecular subtypes (such as Luminal A, Luminal B, HER2-enriched, and Basal-like) subtypes based on the expression patterns of key biomarkers including claudin proteins, which are important for cell adhesion and tumor behavior.

These three data modalities provide complementary views of the same tumor samples - the clinical data provides the biological context and classification labels, while the molecular data (GEX and CNA) capture the underlying genomic alterations that drive these clinical phenotypes.

Flexynesis tries to learn the patterns in these molecular alterations and connect them to the phenotype to predict cancer subtypes based on the genomic and transcriptomic profiles.

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/16287482/files/train_cna_brca.tabular
>    https://zenodo.org/records/16287482/files/train_gex_brca.tabular
>    https://zenodo.org/records/16287482/files/train_clin_brca.tabular
>    https://zenodo.org/records/16287482/files/test_cna_brca.tabular
>    https://zenodo.org/records/16287482/files/test_gex_brca.tabular
>    https://zenodo.org/records/16287482/files/test_clin_brca.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each database a tag corresponding to the modality (#gex, #cna, #clin)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


# Classification task with Flexynesis (1 iteration)

Flexynesis automatically handles key preprocessing steps such as:

* Data processing:
  * removing uninformative features (e.g. features with near-zero-variation)
  * removing samples with too many NA values
  * removing features with too many NA values and impute remaining NA values using median imputation (replacing missing values with the median of each feature)

* Feature selection **only on training data** for each omics layer separately:
  * Features are sorted by Laplacian score
  * Features that make it in the `top_percentile`
  * Highly redundant features are further removed (for a pair of highly correlated features, keep the one with the higher laplacian score).

* Harmonize the training data with the test data

* Normalize the datasets (Subset the test data features to those that are kept for training data)

* Normalize training data (standard scaling) and apply the same scaling factors to the test data.

* Log transform the final matrices (Optional)

* Distinguish numerical and categorical variables in the clinical data file. For categorical variables, create a numerical encoding of the labels for training data. Use the same encoders to map the test samples to the same numerical encodings.

> <comment-title> No manual preprocessing is required </comment-title>
>
> Flexynesis performs all data cleaning internally as part of its pipeline.
{: .comment}

Now it's time to train model using Flexynesis:
* We choose whether we want to concatenate the data matrices (early integration) or not (intermediate integration) before running them through the neural networks.
* We want to apply feature selection and keep only top 10% of the features. In the end, we want to keep at least 1000 features per omics layer.
* We apply a variance threshold (for simplicity of demonstration, we want to keep a few most variable features). Setting this to 80, will remove 80% of the features with the lowest variation from each modality.
* We choose which model architecture to use:
  * DirectPred: a fully connected network (standard multilayer perceptron) with supervisor heads (one MLP for each target variable)
* `target_variables`: Column headers in clinical data.
* `hpo_iter`: How many hyperparameter search steps to implement:
  * This example runs 1 hyperparameter search step using DirectPred architecture and a hyperparameter configuration space defined for "DirectPred" with a supervisor head for "CLAUDIN_SUBTYPE" variable. At the end of the parameter optimization, the best model will be selected and returned.

Training a model longer than needed causes the model to overfit, yield worse validation performance, and also it takes a longer time to train the models, considering if we have to run a long hyperparameter optimization routine, not just for 1 step, but say more than 100 steps.

It is possible to set early stopping criteria in Flexynesis, which is basically a simple callback that is handled by `Pytorch Lightning`. This is regulated using the `early_stop_patience`. When set to e.g. 10, the training will stop if the validation loss has not been improved in the last 10 epochs.

> <hands-on-title> Flexynesis </hands-on-title>
>
> 1. {% tool [Flexynesis](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Type of Analysis"*: `Supervised training`
>        - {% icon param-file %} *"Training clinical data"*: `train_clin_brca.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_clin_brca.tabular`
>        - {% icon param-file %} *"Training omics data"*: `train_gex_brca.tabular`
>        - {% icon param-file %} *"Test omics data"*: `test_gex_brca.tabular`
>        - *"What type of assay is your input?"*: `gex`
>        - In *"Multiple omics layers?"*:
>            - {% icon param-repeat %} *"Insert Multiple omics layers?"*
>                - {% icon param-file %} *"Training omics data"*: `train_cna_brca.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_cna_brca.tabular`
>                - *"What type of assay is your input?"*: `cna`
>        - *"Model class"*: `DirectPred`
>        - *"Column name in the train clinical data to use for predictions, multiple targets are allowed"*: `Column: 16` (CLAUDIN_SUBTYPE)
>        - In *"Advanced Options"*:
>            - *"Variance threshold (as percentile) to drop low variance features."*: `0.8`
>            - *"Minimum number of features to retain after feature selection."*: `100`
>            - *"Top percentile features (among the features remaining after variance filtering and data cleanup) to retain after feature selection."*: `10.0`
>            - *"Number of iterations for hyperparameter optimization."*: `1`
>        - In *"Visualization"*:
>            - *"Generate embeddings plot?"*: `Yes`
>            - *"Generate precision-recall curves plot?"*: `Yes`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What are main outputs of Flexynesis?
> 2. What does the generated plots show?
>
> > <solution-title></solution-title>
> >
> > 1. The results collection contains the following data:
> > * job.embeddings_test (latent space of test data)
> > * job.embeddings_train (latent space of train data)
> > * job.feature_importance.GradientShap (feature importance calculated by GradientShap method)
> > * job.feature_importance.IntegratedGradients (feature importance calculated by IntegratedGradients method)
> > * job.feature_logs.cna
> > * job.feature_logs.gex
> > * job.predicted_labels (Prediction of the created model)
> > * job.stats
> >
> > 2. There are three figures in plot collection:
> > * A PCA plot of the Embeddings colored by known, true subtypes
> > ![PCA plot - colored by known subtypes](../../images/flexynesis_classification/niter1_pca_known.jpeg "PCA plot - colored by known subtypes")
> > * A PCA plot of the Embeddings colored by predicted subtypes
> > ![PCA plot - colored by predicted subtypes](../../images/flexynesis_classification/niter1_pca_predicted.jpeg "PCA plot - colored by predicted subtypes")
> > We can see that the subtypes were predicted by the model with a good approximation.
> > * A precision-recall curves plot, showing the accuracy of the prediction
> > ![PR-curve plot - showing prediction accuracy](../../images/flexynesis_classification/niter1_pr_curve.jpeg "PR-curve plot - showing prediction accuracy")
> > This plot also conforms that most of the subtypes were classified with a good precision (except Class 4 (LumB), in the PCA plot you can see some Her2 samples were classified as LumB for example)
> {: .solution}
>
{: .question}

## Generate UMAP plot of the embeddings

To generate a UMAP plot, we need to extract the embeddings and predicted table.

> <hands-on-title> Extract embeddings and prediction </hands-on-title>
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.embeddings_test`
>
> 2. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.predicted_labels`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What information do we have in predicted labels?
>
> > <solution-title></solution-title>
> >
> > 1. It is a tabular file which contains the probability of each subtype for each sample and a final predicted label based on the probability.
> >
> {: .solution}
>
{: .question}

As you have seen in predicted labels, each sample has a prediction value for each subtype. For UMAP plot we only need the final predicted label.

> <hands-on-title> Remove probabilities (duplicates) </hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - *"Sort Query"*: `job.predicted_labels` (output of **Extract dataset** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `Column: 1`
>            - *"Flavor"*: `Alphabetical sort `
>    - *"Output unique values"*: `Yes`
>
{: .hands_on}

And finally the UMAP plot.

> <hands-on-title> UMAP plot </hands-on-title>
>
> 1. {% tool [Flexynesis plot](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis plot"*: `Dimensionality reduction`
>        - {% icon param-file %} *"Predicted labels"*: `table` (output of **Sort** {% icon tool %})
>        - {% icon param-file %} *"Embeddings"*: `job.embeddings_test` (output of **Extract dataset** {% icon tool %})
>        - *"Column in the labels file to use for coloring the points in the plot"*: `Column: 5` (known_label)
>        - *"Transformation method"*: `UMAP`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What does the generated plot show?
>
> > <solution-title></solution-title>
> >
> > ![UMAP plot - colored by known subtypes](../../images/flexynesis_classification/niter1_umap_known.jpeg "UMAP plot - colored by known subtypes")
> >
> > This is a UMAP representation of the embeddings which shows a better speration between subtypes
> >
> {: .solution}
>
{: .question}

# Longer training

In reality, hyperparameter optimization should run for multiple steps so that the parameter search space is large enough to find a good set. However, for demonstration purposes, we only run it for 5 steps here:
- *"Number of iterations for hyperparameter optimization."*: `5`

> <hands-on-title> Flexynesis </hands-on-title>
>
> 1. {% tool [Flexynesis](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Type of Analysis"*: `Supervised training`
>        - {% icon param-file %} *"Training clinical data"*: `train_clin_brca.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_clin_brca.tabular`
>        - {% icon param-file %} *"Training omics data"*: `train_gex_brca.tabular`
>        - {% icon param-file %} *"Test omics data"*: `test_gex_brca.tabular`
>        - *"What type of assay is your input?"*: `gex`
>        - In *"Multiple omics layers?"*:
>            - {% icon param-repeat %} *"Insert Multiple omics layers?"*
>                - {% icon param-file %} *"Training omics data"*: `train_cna_brca.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_cna_brca.tabular`
>                - *"What type of assay is your input?"*: `cna`
>        - *"Model class"*: `DirectPred`
>        - *"Column name in the train clinical data to use for predictions, multiple targets are allowed"*: `Column: 16` (CLAUDIN_SUBTYPE)
>        - In *"Advanced Options"*:
>            - *"Variance threshold (as percentile) to drop low variance features."*: `0.8`
>            - *"Minimum number of features to retain after feature selection."*: `100`
>            - *"Top percentile features (among the features remaining after variance filtering and data cleanup) to retain after feature selection."*: `10.0`
>            - *"Number of iterations for hyperparameter optimization."*: `5`
>        - In *"Visualization"*:
>            - *"Generate embeddings plot?"*: `Yes`
>            - *"Generate precision-recall curves plot?"*: `Yes`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What does the generated plots show?
>
> > <solution-title></solution-title>
> >
> > 1. There are three figures in plot collection:
> > * A PCA plot of the Embeddings colored by known, true subtypes
> > ![PCA plot - colored by known subtypes](../../images/flexynesis_classification/niter5_pca_known.jpeg "PCA plot - colored by known subtypes")
> > * A PCA plot of the Embeddings colored by predicted subtypes
> > ![PCA plot - colored by predicted subtypes](../../images/flexynesis_classification/niter5_pca_predicted.jpeg "PCA plot - colored by predicted subtypes")
> > We can see that the subtypes were predicted by the model with a good approximation.
> > * A precision-recall curves plot, showing the accuracy of the prediction
> > ![PR-curve plot - showing prediction accuracy](../../images/flexynesis_classification/niter5_pr_curve.jpeg "PR-curve plot - showing prediction accuracy")
> > This plot also conforms that most of the subtypes were classified with a good precision. Also the precision has incresed (compare Class4 = 0.23 with 1 iteration to Class4 = 0.41 with 5 iterations)
> {: .solution}
>
{: .question}

## Generate UMAP plot of the embeddings

To generate a UMAP plot, we need to extract the embeddings and predicted table.

> <hands-on-title> Extract embeddings and prediction </hands-on-title>
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.embeddings_test`
>
> 2. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.predicted_labels`
>
{: .hands_on}

> <hands-on-title> Remove probabilities (duplicates) </hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - *"Sort Query"*: `job.predicted_labels` (output of **Extract dataset** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `Column: 1`
>            - *"Flavor"*: `Alphabetical sort `
>    - *"Output unique values"*: `Yes`
>
{: .hands_on}

And finally the UMAP plot.

> <hands-on-title> UMAP plot </hands-on-title>
>
> 1. {% tool [Flexynesis plot](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis plot"*: `Dimensionality reduction`
>        - {% icon param-file %} *"Predicted labels"*: `table` (output of **Sort** {% icon tool %})
>        - {% icon param-file %} *"Embeddings"*: `job.embeddings_test` (output of **Extract dataset** {% icon tool %})
>        - *"Column in the labels file to use for coloring the points in the plot"*: `Column: 5` (known_label)
>        - *"Transformation method"*: `UMAP`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What does the generated plot show?
>
> > <solution-title></solution-title>
> >
> > ![UMAP plot - colored by known subtypes](../../images/flexynesis_classification/niter5_umap_known.jpeg "UMAP plot - colored by known subtypes")
> >
> > This is a UMAP representation of the embeddings which shows a better speration between subtypes
> >
> {: .solution}
>
{: .question}

{% include _includes/cyoa-choices.html option1="Let's try TabPFN!" option2="I'll try later!" default="I'll try later!" text="We have another tool for the classification task called **TabPFN** which is a foundation model for tabular data. We can repeat the classification task using TabPFN and compare it with Flexynesis. Please note that TabPFN is slow on CPU and **it will take a lot of time to finish on this dataset**. What do you say? Should we continue with TabPFN?" disambiguation="tool"%}

<div class="Ill-try-later" markdown="1">

# Conclusion

In this tutorial, we demonstrated how Flexynesis can model breast cancer subtypes from multi omics data using a deep generative framework. The tool handles preprocessing, feature selection, and latent space learning automatically, making it efficient and robust.

By training on TCGA BRCA data, we:

  * Visualized meaningful subtype separation in the latent space

  * Identified subtype-specific genes

  * Showed how learned features relate to known clinical labels

Flexynesis provides an accessible way to explore complex omics datasets and uncover biological structure without extensive manual tuning.
</div>

<div class="Lets-try-TabPFN" markdown="1">

Alright, Let's see how TabPFN can predict the subtypes

# Prepare data for TabPFN

Currently, TabPFN supports up to **10,000** samples and **500** features (genes) in a tabular data.
We will filter our data by gene variance and will use top 500 variable genes as input for TabPFN.

The train and test tabular data for TabPFN should be transposed so the samples are in rows and genes in columns, the last column should contain the labels and train and test data should have **same** set of features in **same** order.

Since TabPFN does not support data integration, we should try gene expression and copy number alteration data separately.

## CNA data

First, let's filter the cna data by variance.

Here we will:
* Calculate the variance of each gene in train matrix
* Add the variance back to the matrix
* Sort the matrix by variance in descending order
* Filter the matrix by top 500 genes
* Sort the matrix by gene name


> <hands-on-title> Prepare CNA data </hands-on-title>
>
> 1. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy2) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `train_cna_brca.tabular`
>        - *"Type of table operation"*: `Compute expression across rows or columns`
>            - *"Calculate"*: `Variance`
>            - *"For each"*: `Row`
>
> 2. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `table` (output of **Table Compute** {% icon tool %})
>    - *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `train_cna_brca.tabular`
>    - *"and column"*: `Column: 1`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `Yes`
>
> 3. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `table` (output of **Join two Datasets** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `Column: 2`
>            - *"in"*: `Descending order`
>            - *"Flavor"*: `Fast numeric sort (-n) `
>
> 4. {% tool [Select first](Show beginning1) %} with the following parameters:
>    - *"Select first"*: `500`
>    - {% icon param-file %} *"from"*: `table` (output of **Sort** {% icon tool %})
>    - *"Dataset has a header"*: `Yes`
>
> 5. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `table` (output of **Select first** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `Column: 1`, `Column: 2`
>
> 6. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `table` (output of **Advanced Cut** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `Column: 1`
>            - *"Flavor"*: `Alphabetical sort `
>
> 7. Rename the output file to `train_cna_brca_500gene.tabular`
>
>    > <comment-title> Short explanation of steps: </comment-title>
>    > Here we will:
>    > * Extract the list of genes from the train matrix
>    > * Filter the test data by extracted genes
>    > * Sort the matrix by gene name
>    > * Transpose both train and test data
>    {: .comment}
>
> 8. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `train_cna_brca_500gene.tabular` (output of **Advanced Cut** {% icon tool %})
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `Column: 1`
>
> 9. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"1st file"*: `table` (output of **Advanced Cut** {% icon tool %})
>    - *"Column to use from 1st file"*: `Column: 1`
>    - {% icon param-file %} *"2nd File"*: `test_cna_brca.tabular`
>    - *"Column to use from 2nd file"*: `Column: 1`
>    - *"First line is a header line"*: `Yes`
>
> 10. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `output` (output of **Join** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `Column: 1`
>            - *"Flavor"*: `Alphabetical sort `
>
> 11. Rename the output file to `test_cna_brca_500gene.tabular`
>
> 12. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `train_cna_brca_500gene.tabular` (output of **Sort** {% icon tool %})
>
> 13. Rename the output file to `train_cna_brca_500gene_transposed.tabular`
>
> 14. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `test_cna_brca_500gene.tabular` (output of **Sort** {% icon tool %})
>
> 15. Rename the output file to `test_cna_brca_500gene_transposed.tabular`
>
>     > <comment-title> Short explanation of steps: </comment-title>
>     > Here we will:
>     > * Extract `sample_id` and `CLAUDIN_SUBTYPE` from the train and clinical data
>     > * Add the subtype to the train and test matrix
>     > * And finally remove the `sample_id` from the matrices.
>     {: .comment}
>
> 16. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `train_clin_brca.tabular` (Input dataset)
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `Column: 1`, `Column: 16`
>
> 17. Rename the output to `Train annotation`
>
> 18. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `test_clin_brca.tabular` (Input dataset)
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `Column: 1`, `Column: 16`
>
> 19. Rename the output to `Test annotation`
>
> 20. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"1st file"*: `train_cna_brca_500gene_transposed.tabular` (output of **Transpose** {% icon tool %})
>     - *"Column to use from 1st file"*: `Column: 1`
>     - {% icon param-file %} *"2nd File"*: `Train annotation` (output of **Advanced Cut** {% icon tool %})
>     - *"Column to use from 2nd file"*: `Column: 1`
>     - *"First line is a header line"*: `Yes`
>
> 21. Rename the output to `Annotated train matrix cna`
>
> 22. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"1st file"*: `test_cna_brca_500gene_transposed.tabular` (output of **Transpose** {% icon tool %})
>     - *"Column to use from 1st file"*: `Column: 1`
>     - {% icon param-file %} *"2nd File"*: `Test annotation` (output of **Advanced Cut** {% icon tool %})
>     - *"Column to use from 2nd file"*: `Column: 1`
>     - *"First line is a header line"*: `Yes`
>
> 23. Rename the output to `Annotated test matrix`
>
> 24. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"File to cut"*: `Annotated train matrix` (output of **Join** {% icon tool %})
>     - *"Operation"*: `Discard`
>     - *"Cut by"*: `fields`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>         - *"List of Fields"*: `Column: 1`
>
> 25. Rename the output to `TabPFN ready train data - CNA`
>
> 26. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"File to cut"*: `Annotated test matrix` (output of **Join** {% icon tool %})
>     - *"Operation"*: `Discard`
>     - *"Cut by"*: `fields`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>         - *"List of Fields"*: `Column: 1`
>
> 27. Rename the output to `TabPFN ready test data - CNA`
{: .hands_on}

Now the CNA data is ready for TabPFN. Let's do the same for GEX!

## GEX data

Here we will:
* Calculate the variance of each gene in train matrix
* Add the variance back to the matrix
* Sort the matrix by variance in descending order
* Filter the matrix by top 500 genes
* Sort the matrix by gene name


> <hands-on-title> Prepare GEX data </hands-on-title>
>
> 1. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy2) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `train_gex_brca.tabular`
>        - *"Type of table operation"*: `Compute expression across rows or columns`
>            - *"Calculate"*: `Variance`
>            - *"For each"*: `Row`
>
> 2. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `table` (output of **Table Compute** {% icon tool %})
>    - *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `train_gex_brca.tabular`
>    - *"and column"*: `Column: 1`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `Yes`
>
> 3. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `table` (output of **Join two Datasets** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `Column: 2`
>            - *"in"*: `Descending order`
>            - *"Flavor"*: `Fast numeric sort (-n) `
>
> 4. {% tool [Select first](Show beginning1) %} with the following parameters:
>    - *"Select first"*: `500`
>    - {% icon param-file %} *"from"*: `table` (output of **Sort** {% icon tool %})
>    - *"Dataset has a header"*: `Yes`
>
> 5. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `table` (output of **Select first** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `Column: 1`, `Column: 2`
>
> 6. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `table` (output of **Advanced Cut** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `Column: 1`
>            - *"Flavor"*: `Alphabetical sort `
>
> 7. Rename the output file to `train_gex_brca_500gene.tabular`
>
>    > <comment-title> Short explanation of steps: </comment-title>
>    > Here we will:
>    > * Extract the list of genes from the train matrix
>    > * Filter the test data by extracted genes
>    > * Sort the matrix by gene name
>    > * Transpose both train and test data
>    {: .comment}
>
> 8. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `train_gex_brca_500gene.tabular` (output of **Advanced Cut** {% icon tool %})
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `Column: 1`
>
> 9. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"1st file"*: `table` (output of **Advanced Cut** {% icon tool %})
>    - *"Column to use from 1st file"*: `Column: 1`
>    - {% icon param-file %} *"2nd File"*: `test_gex_brca.tabular`
>    - *"Column to use from 2nd file"*: `Column: 1`
>    - *"First line is a header line"*: `Yes`
>
> 10. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"Sort Query"*: `output` (output of **Join** {% icon tool %})
>     - *"Number of header lines"*: `1`
>     - In *"Column selections"*:
>       - {% icon param-repeat %} *"Insert Column selections"*
>         - *"on column"*: `Column: 1`
>           - *"Flavor"*: `Alphabetical sort `
>
> 11. Rename the output file to `test_gex_brca_500gene.tabular`
>
> 12. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.9+galaxy0) %} with the following parameters:
>     - {% icon param-file %} *"Input tabular dataset"*: `train_gex_brca_500gene.tabular` (output of **Sort** {% icon tool %})
>
> 13. Rename the output file to `train_gex_brca_500gene_transposed.tabular`
>
> 14. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `test_gex_brca_500gene.tabular` (output of **Sort** {% icon tool %})
>
> 15. Rename the output file to `test_gex_brca_500gene_transposed.tabular`
>
>     > <comment-title> Short explanation of steps: </comment-title>
>     > Here we will:
>     > * Extract `sample_id` and `CLAUDIN_SUBTYPE` from the train and clinical data
>     > * Add the subtype to the train and test matrix
>     > * And finally remove the `sample_id` from the matrices.
>     {: .comment}
>
> 16. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"File to cut"*: `train_clin_brca.tabular` (Input dataset)
>     - *"Operation"*: `Keep`
>     - *"Cut by"*: `fields`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>         - *"List of Fields"*: `Column: 1`, `Column: 16`
>
> 17. Rename the output to `Train annotation`
>
> 18. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"File to cut"*: `test_clin_brca.tabular` (Input dataset)
>     - *"Operation"*: `Keep`
>     - *"Cut by"*: `fields`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>         - *"List of Fields"*: `Column: 1`, `Column: 16`
>
> 19. Rename the output to `Test annotation`
>
> 20. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"1st file"*: `train_gex_brca_500gene_transposed.tabular` (output of **Transpose** {% icon tool %})
>     - *"Column to use from 1st file"*: `Column: 1`
>     - {% icon param-file %} *"2nd File"*: `Train annotation` (output of **Advanced Cut** {% icon tool %})
>     - *"Column to use from 2nd file"*: `Column: 1`
>     - *"First line is a header line"*: `Yes`
>
> 21. Rename the output to `Annotated train matrix - gex`
>
> 22. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"1st file"*: `test_gex_brca_500gene_transposed.tabular` (output of **Transpose** {% icon tool %})
>     - *"Column to use from 1st file"*: `Column: 1`
>     - {% icon param-file %} *"2nd File"*: `Test annotation` (output of **Advanced Cut** {% icon tool %})
>     - *"Column to use from 2nd file"*: `Column: 1`
>     - *"First line is a header line"*: `Yes`
>
> 23. Rename the output to `Annotated test matrix - gex`
>
> 24. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `Annotated train matrix` (output of **Join** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>      - *"Is there a header for the data's columns ?"*: `Yes`
>        - *"List of Fields"*: `Column: 1`
>
> 25. Rename the output to `TabPFN ready train data - GEX`
>
> 26. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>     - {% icon param-file %} *"File to cut"*: `Annotated test matrix` (output of **Join** {% icon tool %})
>     - *"Operation"*: `Discard`
>     - *"Cut by"*: `fields`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>         - *"List of Fields"*: `Column: 1`
>
> 27. Rename the output to `TabPFN ready test data - GEX`
{: .hands_on}

Now it is time to run TabPFN for CNA and GEX data.

# TabPFN

> <warning-title>High computation time</warning-title>
>
> TabPFN takes a lot of time to do the classification task on CPU (With our data it takes about 7 hours)
> You can instead import the output (data 55,56,59, and 60) from this [archived history](https://usegalaxy.eu/u/nilchia/h/final-modeling-breast-cancer-subtypes-tabpfn-3).
{: .warning}

> <hands-on-title> TabPFN on CNA </hands-on-title>
>
> 1. {% tool [Tabular data prediction using TabPFN](toolshed.g2.bx.psu.edu/repos/bgruening/tabpfn/tabpfn/2.0.9+galaxy0) %} with the following parameters:
>    - *"Select a machine learning task"*: `Classification`
>    - {% icon param-file %} *"Train data"*: `TabPFN ready train data - CNA` (output of **Advanced Cut** {% icon tool %})
>    - {% icon param-file %} *"test data"*: `TabPFN ready test data - CNA` (output of **Advanced Cut** {% icon tool %})
>    - *"Does test data contain labels?"*: `Yes`
>
{: .hands_on}

> <hands-on-title> TabPFN on GEX </hands-on-title>
>
> 1. {% tool [Tabular data prediction using TabPFN](toolshed.g2.bx.psu.edu/repos/bgruening/tabpfn/tabpfn/2.0.9+galaxy0) %} with the following parameters:
>    - *"Select a machine learning task"*: `Classification`
>    - {% icon param-file %} *"Train data"*: `TabPFN ready train data - GEX` (output of **Advanced Cut** {% icon tool %})
>    - {% icon param-file %} *"test data"*: `TabPFN ready test data - GEX` (output of **Advanced Cut** {% icon tool %})
>    - *"Does test data contain labels?"*: `Yes`
>
{: .hands_on}

To make comparison of TabPFN and Flexynesis fair, we should apply Flexynesis on the GEX and CNA with 500 features separately.

> <hands-on-title> Flexynesis on CNA </hands-on-title>
>
> 1. {% tool [Flexynesis](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Type of Analysis"*: `Supervised training`
>        - {% icon param-file %} *"Training clinical data"*: `train_clin_brca.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_clin_brca.tabular`
>        - {% icon param-file %} *"Training omics data"*: `train_cna_brca_500gene.tabular` (output of **Advanced Cut** {% icon tool %})
>        - {% icon param-file %} *"Test omics data"*: `test_cna_brca_500gene.tabular` (output of **Advanced Cut** {% icon tool %})
>        - *"What type of assay is your input?"*: `cna`
>        - *"Model class"*: `DirectPred`
>        - *"Column name in the train clinical data to use for predictions, multiple targets are allowed"*: `Column: 16` (CLAUDIN_SUBTYPE)
>        - In *"Advanced Options"*:
>            - *"Variance threshold (as percentile) to drop low variance features."*: `0.8`
>            - *"Minimum number of features to retain after feature selection."*: `100`
>            - *"Top percentile features (among the features remaining after variance filtering and data cleanup) to retain after feature selection."*: `10.0`
>            - *"Number of iterations for hyperparameter optimization."*: `5`
>        - In *"Visualization"*:
>            - *"Generate embeddings plot?"*: `Yes`
>            - *"Generate precision-recall curves plot?"*: `Yes`
>
{: .hands_on}

> <hands-on-title> Flexynesis on GEX </hands-on-title>
>
> 1. {% tool [Flexynesis](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Type of Analysis"*: `Supervised training`
>        - {% icon param-file %} *"Training clinical data"*: `train_clin_brca.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_clin_brca.tabular`
>        - {% icon param-file %} *"Training omics data"*: `train_gex_brca_500gene.tabular` (output of **Advanced Cut** {% icon tool %})
>        - {% icon param-file %} *"Test omics data"*: `test_gex_brca_500gene.tabular` (output of **Advanced Cut** {% icon tool %})
>        - *"What type of assay is your input?"*: `gex`
>        - *"Model class"*: `DirectPred`
>        - *"Column name in the train clinical data to use for predictions, multiple targets are allowed"*: `Column: 16` (CLAUDIN_SUBTYPE)
>        - In *"Advanced Options"*:
>            - *"Variance threshold (as percentile) to drop low variance features."*: `0.8`
>            - *"Minimum number of features to retain after feature selection."*: `100`
>            - *"Top percentile features (among the features remaining after variance filtering and data cleanup) to retain after feature selection."*: `10.0`
>            - *"Number of iterations for hyperparameter optimization."*: `5`
>        - In *"Visualization"*:
>            - *"Generate embeddings plot?"*: `Yes`
>            - *"Generate precision-recall curves plot?"*: `Yes`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Compare the PR-curve plots of both Flexynesis and TabPFN predictions, which one is more accurate?
>
> > <solution-title></solution-title>
> >
> > 1. You can see that the prediction of TabPFN was better with the CNA data, however we have got better prediction with Flexynesis on GEX, and with the power of data integration using both CNA and GEX, the prediction is further improved!!
> >![Precision-Recall curve plot of TabPFN on CNA](../../images/flexynesis_classification/pr_curve-tabpfn_cna.png "Precision-Recall curve plot of TabPFN on CNA")
> >![Precision-Recall curve plot of Flexynesis on CNA](../../images/flexynesis_classification/pr_curve-flexynesis_cna.jpeg "Precision-Recall curve plot of Flexynesis on CNA")
> >![Precision-Recall curve plot of TabPFN on GEX](../../images/flexynesis_classification/pr_curve-tabpfn_gex.png "Precision-Recall curve plot of TabPFN on GEX")
> >![Precision-Recall curve plot of Flexynesis on GEX](../../images/flexynesis_classification/pr_curve-flexynesis_gex.jpeg "Precision-Recall curve plot of Flexynesis on GEX")
> >![Precision-Recall curve plot of Flexynesis](../../images/flexynesis_classification/pr_curve-flexynesis.jpeg "Precision-Recall curve plot of Flexynesis")
> {: .solution}
>
{: .question}

> <comment-title> Try Flexynesis with more iteration </comment-title>
>
> In this training we only used 5 iterations to train our model. You can rerun the tool with higher number of iterations and see how the prediction improves.
>
 {: .comment}

# Conclusion

In this tutorial, we demonstrated how Flexynesis can model breast cancer subtypes from multi omics data using a deep generative framework. The tool handles preprocessing, feature selection, and latent space learning automatically, making it efficient and robust.

By training on TCGA BRCA data, we:

  * Visualized meaningful subtype separation in the latent space

  * Identified subtype-specific genes

  * Showed how learned features relate to known clinical labels

Flexynesis provides a fast and accessible way to explore complex omics datasets and uncover biological structure without extensive manual tuning.

We also compared Flexynesis with TabPFN and we saw although TabPFN predicted the subtypes with CNA data better, but it was not as good as Flexynesis on GEX data. It also provides fewer outputs compared to Flexynesis for example the integrated latent space embeddings, feature importance and a full information about the predictions.

</div>


topic: statistics
tutorial: galaxy-ludwig
title: Train and Test a Deep learning image classifier with Galaxy-Ludwig
difficulty: intermediate
time_estimation: 40M
tools_json: []
requirements_json: []
tags_json: ["MNIST", "Deep learning", "Ludwig"]
content_hash: b413650b9b0a2be76fc0904c1c0cfa7b
last_modified: 2026-02-26T21:12:18.183265
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/galaxy-ludwig/tutorial.html

content:
> <comment-title>Galaxy-Ludwig Tool</comment-title>
>
> The Ludwig tool described in this tutorial is only available at: 
> [Cancer-Galaxy](https://cancer.usegalaxy.org)
>
> Galaxy-ML tools > Ludwig Experiment
>
{:  .comment}

Using the MNIST image dataset of handwritten digits as input, we will build an image recognition model with the Galaxy-Ludwig tool. The goal is to classify the handwritten digit in each image.

To accomplish this, three steps are needed: (i) upload Ludwig files and image files to Galaxy (ii) Set up and running the Ludwig experiment function on Galaxy, and (iii) Evaluate the image classification model. As a bonus step, we'll also explore (iv) improving the model's classification performance (Figure 1).

![schema of the whole process of training model and test.](../../images/galaxy-ludwig/explain_model_schema.png "Overview of the steps process to obtain the handwritten classification model and testing it.")

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


> <comment-title>Background</comment-title>
>
> [MNIST dataset](https://yann.lecun.com/exdb/mnist/) provides the database of handwritten digits
> the database has a training set of 60,000 examples, and a test set of 10,000 examples.
> at our planet and its environment for the ultimate benefit of all European citizens.
> It is a subset of a larger set available from NIST.
> The digits have been size normalized and centered in a fixed-size image.
{:  .comment}

# Files Format
Before starting our hands-on, here is a brief explanation of the three files generated for the Ludwig Experiment tool.

## MNIST Train/Test Image Files

Since our model will learn directly from the PNG files containing the handwritten numbers, we need to upload these files as well. The images are stored in two main folders: one for training and one for testing. Another folder, named after the figure label, is used to store the figures (Figure 2).

Notice that we are going to work with compressed file (.zip), Galaxy-Ludwig knows how to decompress the files and have them ready.

![folder tree related to the hand-written digits picture](../../images/galaxy-ludwig/explain_images_file.png "File tree containing the images files used to training and testing the model")

## MNIST Tabular Structure 

The MNIST dataset consists of images and their corresponding labels. For the purpose of this tutorial, mnist_dataset.csv file is created and contains three columns: image_path, label and, split.

Briefly, the image_path column provides the file paths to the images that will be fed into the deep learning algorithm. The label column contains the correct classifications, ranging from 0 to 9, for the handwritten digits in the images. The split column indicates whether the data should be used for training (0) or testing (2) the model.

![Dataset.csv file format snapshot](../../images/galaxy-ludwig/explain_dataset_format.png "Dataset.csv file format snapshot. features in order: file_path, label, split."){: width="50%" height="10%"}
 
## Configuring Convolutional Neural Networks (CNNs) 

The config.yaml file is crucial as it defines the entire structure of your machine learning experiment. This configuration file tells Ludwig how to process your data, what model to use, how to train it, and what outputs to generate.

The rationale behind the construction of this configuration file for the dataset is as follows:

i) The model takes images as input and uses a stacked convolutional neural network (CNN) to extract features.
ii) It consists of two convolutional layers followed by a fully connected layer, with dropout applied to the second convolutional layer. The max pooling settings are defined by pool_size = 2 and pool_stride = 2, which are important for reducing the spatial dimensions of the input volume.
iii) The fully connected layer helps to reduce overfitting.
iv) The split configuration is used to divide the dataset based on the 'split' column rather than using a random split (which is the default). In this configuration, rows labeled with '0' are included in the training set, while rows labeled with '2' are included in the testing set. If a validation category is needed, rows are labeled with '1'.
v) The model is trained to classify images into categories (e.g., recognizing digits) and will train for 5 epochs.

![Config.yaml file snapshot](../../images/galaxy-ludwig/explain_config.png "Config.yaml file snapshot."){: width="50%" height="5%"}


# Galaxy-Ludwig Tool

Ludwig simplifies the complexities of machine learning by automating essential steps such as data preprocessing, model architecture selection, hyperparameter tuning, and device management. To properly adapt the configuration for different datasets, users should consider the following parameters:

- Input Features: Ensure the type reflects the nature of the data.
- Encoder: Depending on the input type, this may need to be changed. For example, for image data, consider varying `num_filters` and `filter_size` hyperparameters. For text data, you might switch to type: rnn or type: transformer.
- Output Features: Adjust the type according to the target variable. For instance, if it's a regression task, use type: numerical instead of category.
- Trainer Parameters: Users can adjust epochs, batch_size, and learning_rate based on the dataset size, complexity, data behavior, and available computational resources.

These are just a few of the many options that Ludwig provides out-of-the-box, allowing users to easily set and modify parameters. This is particularly beneficial for Galaxy users who are more interested in addressing their scientific questions than in navigating the intricacies of machine learning workflows.

## Prepare environment and get the data 

> <hands-on-title> Environment and Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *image classifier*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from Zenodo or from the shared data library
>
>    ```
>    https://zenodo.org/records/13292895/files/mnist_images.zip
>    https://zenodo.org/records/13357260/files/mnist_dataset.csv 
>    https://zenodo.org/records/13713095/files/config.yaml
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the data format assigned for each file is **zip**, **csv**, and **yaml**, respectively.
>    If it is not, follow the Changing the datatype tip.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add a tag (`Stacked CNN`) to the dataset corresponding to `config.yaml`
>    This is important to trace back on what configurations the model was built on.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

## Ludwig Experiment

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Ludwig Experiment](toolshed.g2.bx.psu.edu/repos/paulo_lyra_jr/ludwig_applications/ludwig_experiment/2024.0.10.3) %} with the following parameters:
>    - {% icon param-file %} *"Select the dataset containing model configuration"*: `config.yaml`
>    - {% icon param-file %} *"Input dataset"*: `mnist_dataset.csv`
>    - {% icon param-file %} *"Raw data"*: `mnist_images.zip`
> Run the tool 
{: .hands_on}

## Ludwig Experiment Outputs files

After your model is trained and tested, you should see three new files in your history list:

- Ludwig Experiment Report: An HTML file containing the evaluation report of the trained model.
 
- Ludwig Experiment Trained Model: A composite dataset of the trained Ludwig model.
 
- Ludwig Experiment Predictions CSV: A CSV file with the prediction results from the model evaluations.

To evaluate the model, the Ludwig Experiment Report contains all the necessary information.

# Ludwig Experiment Report

Before we begin, it's crucial to determine the most appropriate metric for a given machine learning problem. While accuracy is a useful metric, other performance measures might be more suitable depending on the specific goal. For instance, an overfitted model might exhibit high accuracy due to class imbalance in the dataset. Therefore, the purpose of building the model should guide the interpretation of various metrics.

In this section, we will demonstrate how to evaluate the results of an image classifier model, providing you with a foundational understanding of how to interpret the model's performance.

## Confusion matrix

A confusion matrix is powerful for evaluating the performance of a deep learning model, especially in classification tasks. It provides a granular view of model performance, helping to refine the model. It achieves that by showing the predicted labels with the actual labels in a structured table format, where rows are the actual values and columns the prediction from the model (Figure 5).

![figure illustrates the process of generating a confusion matrix from a machine learning model's predictions](../../images/galaxy-ludwig/explain_confusion_matrix_2.png "This figure illustrates the process of generating a confusion matrix from a machine learning model's predictions. The confusion matrix is a table used to evaluate the performance of a classification algorithm. It compares the actual labels (true values) with the predicted labels produced by the model.")

Understanding the confusion matrix allows to calculate and interpreting: Recall, Precision, Specificity, Accuracy, and other aspects involved such as bias.

The starting point is the confusion matrix generated after training and testing the model (Figure 6). As usual, the diagonal elements represent the correctly predicted samples, giving you a sense of how well your model predicted all possible labels before conducting a more in-depth analysis.

The confusion matrix plot:

![confusion matrix generated from the image classifier results](../../images/galaxy-ludwig/test1_confusion_matrix.png "Confusion matrix derivate from Image classifier model testing.")

For example, we can interpret the results for the handwritten digit '1' (Figure 7a) and for digit '7' (Figure 7b). Each analysis provides a snapshot of the model's performance for each tested digit.

![Analysis of the confusion matrix derived from image classifier model testing](../../images/galaxy-ludwig/explain_confusion_matrix.png "Analysis of the confusion matrix derived from image classifier model testing. A) Analysis of the digit '1' and its corresponding confusion matrix. B) Analysis of the digit '7' and its corresponding confusion matrix.")

Based on the analysis presented, we can conclude that (i) the model demonstrates a remarkable ability to identify the digits, (ii) with slightly better predictions for the digit '1' compared to the digit '7'. (iii) It's important to note that the confusion matrix also provides insight into the model's bias; for example, if the numbers along the diagonal are low and the off-diagonal numbers are high, it may indicate bias. The model shows a strong tendency to confuse the digit '1' with the digit '8' and the digit '7' with the digit '2' and '9'. 

## Entropy

In the context of a confusion matrix, entropy measures the uncertainty or disorder in the model's predictions (Figure 8). A confusion matrix with low entropy would have high values concentrated along the diagonal (indicating correct predictions) and low values elsewhere, suggesting that the model is confident and accurate in its classifications. Conversely, a high entropy confusion matrix would have more evenly distributed values, indicating more frequent misclassifications and greater uncertainty in the model's predictions. In simple terms, entropy helps quantify how mixed or "confused" the model is when making decisions, with higher entropy reflecting more confusion.

The confusion matrix plot:

![Confusion matrix entropy from image classifier results](../../images/galaxy-ludwig/test1_entropy.png "Confusion matrix entropy of the model trained. The figure visualizes how the model's performance varies across different digits based on the distribution of prediction probabilities.")

The lowest entropy value, observed for digit '0' (0.1147), indicates that the model is highly confident and accurate in predicting this digit, with fewer misclassifications and greater certainty. Conversely, the highest entropy value, seen for digit '8' (0.6066), suggests that the model experiences more uncertainty and confusion when predicting this digit, leading to more frequent misclassifications. Additionally, the entropy for digit '1' is lower than that for digit '7', confirming our earlier observation that the model classifies digit '1' more effectively. This variability in entropy can guide further improvements in model training and tuning, with the goal of reducing uncertainty and enhancing prediction accuracy across all digits.

## Combined Loss

The Loss vs. Epoch graph represents the learning progress during the training of a neural network. The x-axis shows the epochs, where one epoch means that each sample in the training dataset has had an opportunity to update the model’s internal parameters. The y-axis represents the loss, a numerical metric indicating how incorrect the model's predictions are. The objective of training is to minimize this loss, aiming to reduce it to the lowest possible value.

> <tip-title>Loss Function details</tip-title>
>In image classification tasks, cross-entropy loss, also known as log loss, is the most frequently used loss function. For binary classification (two classes), binary cross-entropy is used, while sparse categorical cross-entropy is applied when classifying three or more categories. The model produces a probability vector indicating the likelihood that the input image belongs to each predefined category. This predicted vector is then compared to the actual output, which is represented by a vector of the same size, where the correct category is assigned a probability of 1, and all other categories are assigned a probability of 0.
>
{: .tip}

In the graph (Figure 9), there is a training curve. When training a model, you'll often look at a loss curve to determine if the model has converged. It could also help in detecting overfitting and underfitting and can provide insights for implementing early stopping.

> <tip-title>Overfitting</tip-title>
>Overfitting occurs when the network fails to generalize well to unseen data because it begins to model the noise and randomness in the training dataset.
>![alt](../../images/galaxy-ludwig/explain_overfitting.png "Overfitting curve")
>
{: .tip}

> <tip-title>Underfitting</tip-title>
>Underfitting happens when the patterns in the training dataset are too complex for our model. It can’t capture them, so it has a poor performance on training and validation sets. 
>![alt](../../images/galaxy-ludwig/explain_underfitting.png "Underfitting curve")
>
{: .tip}

![loss vs epochs](../../images/galaxy-ludwig/test1_combined_loss.png "Loss versus Epochs during training. The graph illustrates the relationship between the training loss and the number of epochs during the training of the model.")

Based on the training curve (blue line), we can draw the following conclusions:

- CONSISTENT IMPROVEMENT:
    Training loss consistently decreases over the five epochs. This indicates that the model is effectively learning and improving its performance on the dataset.
- APPROACHING CONVERGENCE:
    By Epoch 5, the training loss has reduced to less than 0.5. The gradual reduction in losses suggests that the model is approaching convergence, where further training may yield diminishing returns in terms of performance improvement.
- ADDITIONAL INFORMATION: 
    Looking at the test statistics results {"combined": {"loss": 0.03800534829497337}}, the combined loss indicates that the model is generalizing well to unseen data after training.

## Accuracy

Plotting accuracy over epochs in deep learning is a common practice to visualize how well a model is learning to make correct predictions as training progresses. The x-axis represents the number of epochs, which are complete passes through the training dataset, while the y-axis shows the accuracy, typically expressed as a percentage.
As training progresses, the accuracy on the training data usually increases, indicating that the model is learning to correctly classify more examples. 

> <tip-title>Plotting accuracy and/or Loss</tip-title>
> Plotting accuracy and plotting loss are both essential for monitoring the performance of a deep learning model, but they serve different purposes and provide different insights. While accuracy provides a straightforward measure of model performance, loss offers deeper insights into the training process, making both plots valuable for different aspects of model evaluation.
{: .tip}

![alt](../../images/galaxy-ludwig/test1_label_accuracy.png "Accuracy versus Epochs. This graph depicts the accuracy of the model over the course of training. The training accuracy curve indicates how well the model is performing on the training data.")

Based on the training curve (blue line), we can draw the following conclusions:

- THE DATASET IS RELATIVELY EASY FOR THE MODEL TO LEARN:
    The model starts with a relatively high training accuracy of >0.8.
- CONSISTENT IMPROVEMENT:
    Training accuracies increase steadily across the epochs. This consistent upward trend indicates that the model is learning effectively and becoming more accurate in its predictions with each epoch.
- ADDITIONAL INFORMATION:
   The test statistics show an accuracy of {"label": {"accuracy": 0.9885247349739075}} for the model when applied to unseen images. This high accuracy is crucial because it indicates that the model can effectively classify new data that it has not encountered during training. Thus, validates the model's ability to generalize beyond the training data, ensuring that it can make reliable predictions in real-world scenarios. 

## Hits at K

This metric measures the proportion of times the correct item is found within the top-k predictions made by the model. X-axis (Epochs) represents the number of training cycles (epochs) completed. Y-axis (Hits at k) represents the hit rate, or the frequency with which the correct item appears in the top k predictions.

> <tip-title>Hits at K trend</tip-title>
> Upward Trend: An upward trend in the graph indicates that the model is improving, as it becomes more likely to rank the correct item within the top k predictions as training progresses. Plateauing: If the graph plateaus, it suggests that the model's performance has stabilized and further training may not yield significant improvements. Fluctuations: Fluctuations in the graph could indicate overfitting, underfitting, or instability in the model's training process.
{: .tip}

![alt](../../images/galaxy-ludwig/test1_label_hits_at_k.png "Hits at K metric across different epochs. The Hits at K metric evaluates the proportion of times the correct item is found within the top K predictions made by the model.")

Based on the training curve (blue line), we can draw the following conclusions:

- GOOD MODEL PERFORMANCE:
    The model shows consistent improvement over the epochs and Hits at k approaching near-perfect levels. 
- CONVERGENCE:
    Overall, the graph would depict a rapid increase in performance early on, followed by a plateau as the model approaches optimal performance. This indicates successful training with effective generalization.

# Changing model's configuration (Epoch = 15)

When designing a neural network, it is unfortunately not possible to determine the optimal number of neurons, layers, or other hyperparameters solely from a problem description. There is limited guidance available for selecting appropriate initial values, making it challenging to identify a starting point.

A widely adopted approach involves making an initial estimate based on prior experience with networks designed for similar problems. This experience could be derived from your own work or gathered from secondary sources, such as training courses, blogs, or research papers. From there, it is advisable to experiment with different configurations and carefully evaluate the performance of each before selecting the most effective architecture.

Since we have gathered insights from our recent model, we can now begin testing new configurations and comparing them to the previous results.

## Changing Epochs

For this new configuration, the file has been modified to adjust the Epoch value, increasing it from 5 to 15. Upload the file using the following link:

```
https://zenodo.org/records/13716147/files/config_test_2.yaml
```
If you are comfortable using the same history to run and obtain the results, that is fine. Click on the "Upload Data" button in the navigation bar on the left side, then select "Paste/Fetch Data" to paste the link.

Alternatively, you can create a new history to keep your experiment organized. Follow the steps below to start the new history.

> <tip-title>create new history and upload files</tip-title> 
>
>> <hands-on-title> Environment and Data Upload </hands-on-title>
>>
>> 1. Create a new history for this tutorial. If you are not inspired, you can name it *climate101*.
>>
>>    {% snippet faqs/galaxy/histories_create_new.md %}
>>
>> 2. Import the files from Zenodo or from the shared data library
>>
>>    ```
>>    https://zenodo.org/records/13292895/files/mnist_images.zip
>>    https://zenodo.org/records/13357260/files/mnist_dataset.csv 
>>    https://zenodo.org/records/13716147/files/config_test_2.yaml
>>    ```
>>
>>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>>
>> 3. Check that the data format assigned for each file is **zip**, **csv**, and **yaml**, respectively.
>>    If it is not, follow the Changing the datatype tip.
>>
>>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>>
>> 5. Add a tag (`Stacked CNN`) to the dataset corresponding to `config.yaml`
>>    This is important to trace back on what configurations the model was built on.
>>
>>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>>
>{: .hands_on}
>
{: .tip}

Run the experiment once again. If you need a refresher on how to set up the Ludwig experiment to run the tool, follow the instructions below.

><tip-title>Ludwig Experiment</tip-title>
>
>><hands-on-title> Task description </hands-on-title>
>>
>> 1. {% tool [Ludwig Experiment](toolshed.g2.bx.psu.edu/repos/paulo_lyra_jr/ludwig_applications/ludwig_experiment/2024.0.10.3) %} with the following parameters:
>> - {% icon param-file %} *"Select the dataset containing model configuration"*: `config.yaml`
>> - {% icon param-file %} *"Input dataset"*: `mnist_dataset.csv`
>> - {% icon param-file %} *"Raw data"*: `mnist_images.zip`
>>
>>> <comment-title> short description </comment-title>
>>>
>>> Once all the inputs are in place, press the 'RUN TOOL' button to start your job.
>>{: .comment}
>>
>{: .hands_on}
>
{: .tip}

## Ludwig Experiment Report

Here are the results from the new model we have trained and validated. Using your knowledge from the previous analysis, answer the following questions:

> <comment-title>Confusion Matrix</comment-title>
>
>![alt](../../images/galaxy-ludwig/test2_confusion_matrix.png "Confusion Matrix - Epoch = 15")
>> <question-title></question-title>
>>
>> 1. For which label(s) did the model increase the number of correct predictions?
>>
>>> <solution-title></solution-title>
>>>
>>> 1. The model improved prediction across all labels.
>>>
>>{: .solution}
>>
>{: .question}
>
{:  .comment}

> <comment-title>Entropy</comment-title>
>
>![alt](../../images/galaxy-ludwig/test2_entropy.png "Entropy - Epoch = 15")
>> <question-title></question-title>
>>
>> 1. For which label(s) did the model decrease the entropy? 
>>
>> 2. Why did the order of the labels change compared to the previous results?
>>
>>> <solution-title></solution-title>
>>>
>>> 1. The model decreased entropy across all labels. 
>>>
>>> 2. For some labels, the entropy drop was more pronounced, moving those labels toward the bottom of the list.
>>>
>>{: .solution}
>>
>{: .question}
>
{:  .comment}

> <comment-title>Loss</comment-title>
>
>![alt](../../images/galaxy-ludwig/test2_combined_loss.png "Combined Loss - Epoch = 15")
>> <question-title></question-title>
>>
>> 1. Did the increase in Epochs have a significant impact on the model's loss?
>>
>>> <solution-title></solution-title>
>>>
>>> 1. The model's training loss curve showed improvement, reaching a loss below 0.25 by the end of the epochs. 
>>>
>>{: .solution}
>>
>{: .question}
>
{:  .comment}

> <comment-title>Accuracy</comment-title>
>
>![alt](../../images/galaxy-ludwig/test2_accuracy.png "Accuracy - Epoch = 15")
>> <question-title></question-title>
>>
>> 1. Did the increase in Epochs have a significant impact on the model's accuracy 
>>
>>> <solution-title></solution-title>
>>>
>>> 1. In alignment with the data presented by the confusion matrix, the accuracy exhibited an improvement.
>>>
>>{: .solution}
>>
>{: .question}
>
{:  .comment}

> <comment-title>Hits at K</comment-title>
>
>![alt](../../images/galaxy-ludwig/test2_hits_at_k.png "Hits at K - Epoch = 15")
>> <question-title></question-title>
>>
>> 1. Can we infer that, overall, the model is making more accurate label predictions based on the hits at k?
>>
>>> <solution-title></solution-title>
>>>
>>> 1. Yes, the curve for training has approached to 1, indicating that the correct label is now present within the top-k predictions more frequently. 
>>>
>>{: .solution}
>>
>{: .question}
>
{:  .comment}

## One more graph - F1 Score vs Frequency

The F1 score is a metric used to evaluate the performance of a classification model, particularly in cases where the dataset is imbalanced, meaning some classes are more frequent than others. It combines two other important metrics: precision and recall, into a single score that balances them both.

> <tip-title>Precision definition</tip-title>
> Precision: Precision measures the accuracy of positive predictions. It is the ratio of correctly predicted positive observations to the total predicted positives. In other words, it answers the question, "Of all the instances the model predicted as positive, how many were actually positive?
>![precision formula](../../images/galaxy-ludwig/explain_sensitivity_formula.png)
>
{: .tip}

> <tip-title>Recall definition</tip-title>
> Recall, also known as sensitivity or true positive rate, measures how well the model identifies all the actual true cases. It is the ratio of correctly predicted positive observations to all observations in the actual class. It answers the question, "Of all the actual true, how many did the model correctly identify? 
>![recall formula](../../images/galaxy-ludwig/explain_recall_formula.png)
>
{: .tip}


> <tip-title>F1 score definition</tip-title>
> The F1 score is the harmonic mean of precision and recall. Unlike the simple average, the harmonic mean gives more weight to lower values, so the F1 score will be high only if both precision and recall are high.
>![F1 score formula](../../images/galaxy-ludwig/explain_f1_score.png)
>
{: .tip}

The F1 score vs. frequency label graph is a valuable tool for diagnosing how well a model handles different classes, especially in datasets where some classes are much more common than others. By analyzing this graph, you can identify areas where your model might need more attention, ensuring balanced and accurate predictions across all classes.


![F1 score vs frequency label](../../images/galaxy-ludwig/test2_f1_score.png "F1 Score vs. Frequency of Labels. This graph illustrates the relationship between the frequency of each class label in the dataset and the model's F1 score for that class. Higher F1 scores indicate better model performance.")

Digits like '1' (occurrences > 7800 and F1 score > 0.992) and '0' (occurrences > 6800 and F1 score > 0.980) suggest that the model has learned to accurately identify these digits, despite the difference when looking at the frequency of the digits. Digit '5', with a lower frequency, still achieves a strong F1 score. Although its frequency is the lowest among the digits, the F1 score remains high, indicating that the model maintains a good balance of precision and recall even with fewer examples. This reflects the robustness of the CNN model.

# Conclusion
In this tutorial, we demonstrated how to utilize the Galaxy-Ludwig tool to build an image recognition model using the MNIST dataset of handwritten digits. We followed a structured approach consisting of 
- uploading necessary files, 
- running the Ludwig experiment, and 
- evaluating the model's performance. 

Through the process, we have seen how Ludwig simplifies the complexities of machine learning workflows, making them accessible and efficient for users. Additionally, we explored ways to enhance model performance by adjusting configurations and analyzing detailed evaluation metrics. 

By the end of this tutorial, you should have a solid understanding of how to deploy a handwritten digit classification model and interpret its results effectively.


topic: statistics
tutorial: flexynesis_cbio_import
title: Prepare Data from CbioPortal for Flexynesis Integration
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 536be98893a5bb7ef663181d8f9278ac
last_modified: 2026-02-26T21:12:18.178837
zenodo_link: https://zenodo.org/records/16287482
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/flexynesis_cbio_import/tutorial.html

content:
The cBioPortal is an open-access web platform that provides intuitive access to large-scale cancer genomics datasets {% cite Gao2013 %} {% cite cbioportal-website %}. Originally developed to make complex molecular profiling data more accessible to the broader research community, cBioPortal hosts data from thousands of cancer studies and tens of thousands of tumor samples.

In this tutorial, we will work with data from the METABRIC consortium {% cite metabric-website %}, one of the landmark breast cancer genomics studies available through cBioPortal. This dataset contains comprehensive molecular and clinical data from over 2,000 breast cancer patients, including gene expression profiles, copy number alterations, mutation data, and clinical outcomes.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Import data from CbioPortal

> <hands-on-title> Import data from cBioPortal </hands-on-title>
>
> 1. {% tool [Flexynesis cBioPortal import](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_cbioportal_import/flexynesis_cbioportal_import/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"cBioPortal study ID"*: `brca_metabric`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What modalities are imported to Galaxy?
>
> > <solution-title></solution-title>
> >
> > 1. * Clinical data
> >    * Copy number alteration
> >    * Methylation
> >    * Gene expression
> >    * Mutation
> >
> {: .solution}
>
{: .question}


# Data cleanup

Now we need to clean up our data. This means removing comment lines in the matrix, removing duplicate samples and so on.

## Clinical data

> <hands-on-title> Prepare clinical data </hands-on-title>
>
> Here we'll First extract the clinical data from the collection, then we remove the comment lines, and finally we remove the extra index column.
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `datasets` (output of **Flexynesis cBioPortal import** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `data_clinical_patient`
>
>
> 2. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy2) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `data_clinical_patient` (output of **Extract dataset** {% icon tool %})
>        - In *"Advanced File Options "*:
>            - *"Header begins at line N"*: `4`
>        - *"Type of table operation"*: `No operation (just reformat on output)`
>
> 3. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `table` (output of **Table Compute** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `c1:`
>
> 4. Rename the data `clinical data - cleaned`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many samples are there in the clinical data?
>
> > <solution-title></solution-title>
> >
> > 1. Click on the data, you will see the data has **2510** lines, so **2509** samples.
> >
> {: .solution}
>
{: .question}

## Omics data

Time to prepare our omics data. We are interested in mutation and gene expression.

> <hands-on-title> Prepare gene expression data </hands-on-title>
>
> Here we'll First extract the clinical data from the collection, then we remove duplicate genes from the matrix, and finally we remove the ENTREZ Ids.
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `data` (output of **Flexynesis cBioPortal import** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `data_mrna_illumina_microarray`
>
> 2. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `data_mrna_illumina_microarray` (output of **Extract dataset** {% icon tool %})
>    - *"Number of header lines"*: `1`
>    - In *"Column selections"*:
>        - {% icon param-repeat %} *"Insert Column selections"*
>            - *"on column"*: `c1`
>    - *"Output unique values"*: `Yes`
>
> 3. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `output` (output of **Sort** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `c2: Entrez_Gene_Id`
>
> 4. Rename the data `expression data - cleaned`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many samples and genes are there in the gene expression data?
>
> > <solution-title></solution-title>
> >
> > 1. Click on the data, you will see the data has **20386** lines and 1981 columns, so **20385** genes and 1980 samples.
> >
> {: .solution}
>
{: .question}

> <hands-on-title> Prepare mutation data </hands-on-title>
>
> Here we'll First extract the clinical data from the collection, then we remove comment lines from the matrix, and remove the extra index column, and finally, we create a binarized matrix which indicates the number of mutations per genes.
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `data` (output of **Flexynesis cBioPortal import** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `data_mutations`
>
> 2. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy2) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `data_mutations` (output of **Extract dataset** {% icon tool %})
>        - In *"Advanced File Options "*:
>            - *"Header begins at line N"*: `1`
>        - *"Type of table operation"*: `No operation (just reformat on output)`
>
> 3. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `table` (output of **Table Compute** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"Is there a header for the data's columns ?"*: `Yes`
>            - *"List of Fields"*: `c1:`
>
> 4. {% tool [Flexynesis utils](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_utils/flexynesis_utils/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis utils"*: `Binarize mutation data`
>        - {% icon param-file %} *"Mutation data"*: `table` (output of **Advanced Cut** {% icon tool %})
>        - *"Column in the mutation file with genes"*: `Column: 1`
>        - *"Column in the mutation file with samples"*: `Column: 17`
>
> 5. Rename the data `mutation data - cleaned`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many samples and genes are there in the mutation data?
>
> > <solution-title></solution-title>
> >
> > 1. Click on the data, you will see the data has **174** lines and **2370** columns, so **174** genes and **2370** samples.
> >
> {: .solution}
>
{: .question}

# Split data to train and test

In the last step we split our clinical and omics data into train and test with ratio of 0.7 (70% as training and 30% as test)

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Flexynesis utils](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_utils/flexynesis_utils/0.2.20+galaxy2) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis utils"*: `Split data to train and test`
>        - {% icon param-file %} *"Clinical data"*: `clinical data - cleaned` (output of **Advanced Cut** {% icon tool %})
>        - {% icon param-files %} *"Omics data"*: `expression data - cleaned` (output of **Advanced Cut** {% icon tool %}), `mutation data - cleaned` (output of **Flexynesis utils** {% icon tool %})
>
{: .hands_on}

# Conclusion

In this tutorial, we showed how to download and prepare multi-modal cancer genomics data from cBioPortal for integration using Flexynesis. Working with the METABRIC breast cancer dataset, we covered the complete data preparation pipeline from raw data access to analysis-ready formats.


topic: statistics
tutorial: clustering_machinelearning
title: Clustering in Machine Learning
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: ab88e8596dadfe5d8e7303ab898562f1
last_modified: 2026-02-26T21:12:18.174767
zenodo_link: https://zenodo.org/record/3813447
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/clustering_machinelearning/tutorial.html

content:
The goal of [unsupervised learning](https://en.wikipedia.org/wiki/Unsupervised_learning) is to discover hidden patterns in any unlabeled data. One of the approaches to unsupervised learning is [clustering](https://en.wikipedia.org/wiki/Cluster_analysis). In this tutorial, we will discuss clustering, its types and a few algorithms to find clusters in data. Clustering groups data points based on their similarities. Each group is called a cluster and contains data points with high similarity and low similarity with data points in other clusters. In short, data points of a cluster are more similar to each other than they are to the data points of other clusters. The goal of clustering is to divide a set of data points in such a way that similar items fall into the same cluster, whereas dissimilar data points fall in different clusters. Further in this tutorial, we will discuss ideas on how to choose different metrics of similarity  between data points and use them in different clustering algorithms.

Clustering is crucial in multiple research fields in BioInformatics such as analyzing unlabeled data which can be gene expressions profiles, biomedical images and so on. For example, clustering is often used in gene expression analysis to find groups of genes with similar expression patterns which may provide a useful understanding of gene functions and regulations, cellular processes and so on. For more details, please refer to [ref1](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0171429) and [ref2](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5135122/).

We represent an observation/sample/data point as an n-dimensional vector and many such data points constitute a dataset. To show an example, let us assume that a dataset, shown in Figure 1, contains many samples and each sample has two dimensions each:

![data](images/data_before_clustering.png "Sample data before clustering")

Clustering reveals the following three groups, indicated by different colors:

![data](images/data_after_clustering.png "Sample data after clustering")


Clustering is divided into two subgroups based on the assignment of data points to clusters:

- Hard: Each data point is assigned to exactly one cluster. One example is [k-means](https://en.wikipedia.org/wiki/K-means_clustering) clustering.

- Soft: Each data point is assigned a probability or likelihood of being in a cluster. One example is [expectation-maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) (EM) algorithm.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Types of clustering algorithms

There are many algorithms available for data clustering which use different ways to establish similarity between data points. The clustering algorithms can be broadly divided into many categories such as connectivity model, centroid model, density model, distribution model, group model, graph-based model and so on. Some of these are discussed below:

 - Connectivity model: This model assigns higher similarity to data points which are closer in one or multi-dimensional space than those points which are farther away. There are two approaches - first, it categorises all data points into different clusters and then merges the data points in relation to the distances among them. Second, it categorises all data points into one single cluster and then partitions them into different clusters as the distance increases. This model is easy to understand but has problems in handling large datasets. One example is [hierarchical clustering](https://en.wikipedia.org/wiki/Hierarchical_clustering) and its variants.

 - Centroid model: It is an iterative clustering algorithm in which similarity is based on the proximity of a data point to the centroids of the clusters. [K-means](https://en.wikipedia.org/wiki/K-means_clustering) clustering is one example of this model. It needs a number of clusters before running and then divides data points into these many clusters iteratively. Therefore, to use k-means, users should acquire some prior knowledge about the dataset.

 - Density model: This model searches one or multi-dimensional space for dense regions (having a large number of data points in a small region). A popular example of a density model is [DBSCAN](https://en.wikipedia.org/wiki/DBSCAN).

In this tutorial, we will go through three clustering algorithms - hierarchical clustering, k-means, DBSCAN, and a comparison between these methods. Further, we will discuss their parameters and how to apply them to find clusters in the [iris flower dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) and a few other datasets.


# Clustering distance measures

Clustering groups similar data points and requires a metric or measure to compute a degree of similarity or dissimilarity of data points. Two main types of measures are distance and similarity. The smaller the distance between two objects, the more similar they are to each other. Moreover, the type of distance measure depends on the problem and one measure may not work with all kinds of problems.

Many clustering algorithms use distance measures to determine the similarity or dissimilarity between any pair of data points. A valid distance measure should be symmetric and obtains its minimum value (usually zero) in case of identical data points. By computing the distance or (dis)similarity between each pair of observations, a dissimilarity or distance matrix is obtained.

The choice of a distance measure is crucial in clustering. It defines how the similarity of two elements `(x, y)` is calculated as it influences the shape of the clusters. The classical distance measures are [euclidean](https://en.wikipedia.org/wiki/Euclidean_distance) and [manhattan](https://en.wikipedia.org/wiki/Taxicab_geometry) distances. For the most common clustering algorithms, the default distance measure is euclidean. If the euclidean distance is chosen, then observations having high magnitudes of their respective features will be clustered together. The same holds for the observations having low magnitudes of their respective features. In Figure 3, we group the cells using euclidean distance and their distance matrix.

![Distances](images/raceid_distance.svg "Euclidean distance between three points (R, P, V) across three features (G1, G2, G3)")


> <question-title></question-title>
>
> 1. Why are there zeroes along the diagonal of the above example distance matrix?
> 1. Is there any symmetry in this matrix?
>
> > <solution-title></solution-title>
> >
> > 1. The distance between a point to itself is zero.
> > 1. The distance between point *a* to point *b* is the same as the distance between point *b* to point *a* using the Euclidean distance metric.
> >
> {: .solution }
>
{: .question }

Other dissimilarity measures exist such as correlation-based distances, which are widely used for gene expression data analyses. Correlation-based distance considers two objects to be similar if their features are highly correlated, even though the observed values may be far apart in terms of euclidean distance. The distance between the two objects is 0 when they are perfectly correlated. [Pearson’s correlation](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is quite sensitive to outliers. This does not matter when clustering samples because the correlation is over thousands of genes. However, it is important to be aware of the possible impact of outliers. This can be mitigated by using [Spearman’s correlation](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) instead of Pearson’s correlation.

# Different clustering approaches

## Hierarchical clustering

Hierarchical clustering creates a hierarchy of clusters. It starts with all the data points assigned to clusters of their own. Then, the two nearest clusters are merged into the same cluster. In the end, the algorithm terminates when there is only one cluster left.

Following are the steps that are performed during hierarchical clustering:

1. In the beginning, every data point in the dataset is treated as a cluster which means that we have `N` clusters at the beginning of the algorithm for a dataset of size `N`.

2. The distance between all the points is calculated and two points closest to each other are merged together to form a new cluster.

3. Next, the point which is closest to the cluster formed in step 2, will be merged to the cluster.

4. Steps 2 and 3 are repeated until one large cluster is created.

5. Finally, this large cluster is divided into K small clusters with the help of dendrograms.

Let’s now see how dendrograms help in hierarchical clustering.

![data](images/Hierarchical_clustering_1.png "Hierarchical clustering")

All data points are chosen at the bottom and each one is assigned to a separate cluster. Then, the two closest clusters are merged till just one cluster is left at the top. From the dendrogram thus formed, the distance between two clusters can be determined by computing the height at which two clusters are merged.

By looking at the dendrogram, the clusters can be observed showing different groups in the best way. The optimal number of clusters is the number of vertical lines in the dendrogram cut by a horizontal line that can transverse maximum distance vertically without intersecting a cluster.

In the above example, the best choice of the number of clusters will be 4 as the red horizontal line in the dendrogram below covers maximum vertical distance AB. For more details, please read [this blogpost entitled *"Clustering | Introduction, Different Methods, and Applications"*](https://www.analyticsvidhya.com/blog/2016/11/an-introduction-to-clustering-and-different-methods-of-clustering/).
![data](images/Hierarchical_clustering_2.png "Hierarchical clustering")


This algorithm explained above uses the bottom-up approach. It is also possible to follow the top-down approach starting with all data points assigned in the same cluster and recursively performing splits till each data point is assigned a separate cluster. The decision of merging two clusters is taken based on the proximity of these clusters.

> <comment-title>Background of the iris dataset</comment-title>
> The iris flower dataset or Fisher’s iris dataset is a multivariate dataset introduced by the British statistician and biologist Ronald Fisher in his 1936 paper ({% cite Fisher1936 %}).
> Each row of the table represents an iris flower, including its species and dimensions of its botanical parts, sepal and petal, in centimeters.
> For more history of this dataset read here [Wikipedia](https://en.wikipedia.org/wiki/Iris_flower_data_set).
{: .comment}


At the first step, we should upload the iris dataset and two other datasets which will be used at the end of the tutorial.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. **Import** {% icon galaxy-upload %} the file `iris.csv` from [Zenodo](https://zenodo.org/record/3813447/files/iris.csv) or from the data library
>
>    ```
>    https://zenodo.org/record/3813447/files/iris.csv
>    https://zenodo.org/record/3813447/files/circles.csv
>    https://zenodo.org/record/3813447/files/moon.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 2. **Rename** {% icon galaxy-pencil %} the datasets to `iris`, `circles` and `moon` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Check the **datatype**
>    - Click on the history item to expand it to get more information.
>    - The datatype of the iris dataset should be `csv`.
>    - **Change** {% icon galaxy-pencil %} the datatype *if* it is different than `csv`.
>      - Option 1: Datatypes can be **autodetected**
>      - Option 2: Datatypes can be **manually set**
>
>    {% snippet faqs/galaxy/datasets_detect_datatype.md datatype="datatypes" %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="csv" %}
>
{: .hands_on}

Our objective is to categorise similar flowers in different groups (Figure 6). We know that we have **3** species of iris flowers (versicolor, virginica, setosa) with
**50** samples for each. These species look very much alike as shown in the figure below.

![3 species of iris flowers](images/iris_flowers.png "3 species of iris flowers")

In our dataset, we have the following features measured for each flower: [petal](https://en.wikipedia.org/wiki/Petal) length, petal width, [sepal](https://en.wikipedia.org/wiki/Sepal) length, sepal width

Figure 7 shows the dendrogram of these data.

![data](images/Hierarchical_iris.png "Iris data hierarchical clustering")


We will apply hierarchical clustering to the iris dataset to find clusters based on two features (of flowers) - sepal length and width.
**Hint**: Please find the `Numeric Clustering` tool in the `Statistics` tool section.

> <hands-on-title>Hierarchical clustering</hands-on-title>
>
> 1. **Numeric Clustering** {% icon tool %} with the following clustering parameters:
>    - *"Select the format of input data"*: `Tabular Format (tabular,txt)`
>        - {% icon param-file %} *"Data file with numeric values"*: `iris`
>        - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>        - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>            - {% icon param-text %} *"Type header name(s)"*: `Species`
>        - {% icon param-select %} *"Clustering Algorithm"*: `Hierarchical Agglomerative Clustering`
>        - In *"Advanced options"*
>            - {% icon param-text %} *"Number of clusters"*: `2`
>            - {% icon param-select %} *"Affinity"*: `Euclidean`
>            - {% icon param-select %} *"Linkage"*: `ward`
>
> 2. Rename the generated file to `Hierarchical clustering`
{: .hands_on}

If you view the result table, you can see the last column is the label for each cluster and as you see, all the setosa samples are grouped in one cluster and two other species (versicolor and virginica) are grouped in the second cluster. From Figure 6, it is obvious that versicolor and virginica are more similar to each other.

### Visualize hierarchical clustering

The resulting candidate clustering can be visualized using the `Scatterplot with ggplot2` tool. Each sample is color-coded based on its clustering for that sample.
Let's visualize the clustering results to see how groups have been built. **Hint**: Please find the `Scatterplot with ggplot2` tool in the `Graph/Display data` tool section.

> <hands-on-title>Visualize hierarchical clustering result</hands-on-title>
>
> 1. **Scatterplot with ggplot2** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: **Hierarchical clustering**
>    - *"Column to plot on x-axis"*: `1`
>    - *"Column to plot on y-axis"*: `2`
>    - *"Plot title"*: `Hierarchical clustering in iris data`
>    - *"Label for x axis"*: `Sepal length`
>    - *"Label for y axis"*: `Sepal width`
>    - In *"Advanced Options"*:
>        - *"Data point options"*: `User defined point options`
>            - *"relative size of points"*: `2.0`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>            - *"column differentiating the different groups"*: `6`
>            - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>    - In *"Output options"*:
>        - {% icon param-text %} *"width of output"*: `7.0`
>        - {% icon param-text %} *"height of output"*: `5.0`
>        - {% icon param-text %} *"dpi of output"*: `175.0`
>
> 2. **View** {% icon galaxy-eye%} the resulting plot
> 3. Rename to `Hierarchical scatter plot`
{: .hands_on}

![data](images/hierarchical_scatter.png "Hierarchical clustering scatter plot")


## K-means clustering

K-means clustering is the most commonly used unsupervised machine learning algorithm for partitioning a given dataset into a set of k clusters, where k represents the number of groups pre-specified by the user. In k-means clustering, each cluster is represented by its center or centroid which corresponds to the mean of points assigned to the cluster. The basic idea behind k-means clustering is to define clusters and their centroids such that the total intra-cluster variation is minimized.

K-means is popular because of its speed and scalability. Many variants of the k-means algorithm such as [Lloyd's algorithm](https://en.wikipedia.org/wiki/Lloyd%27s_algorithm), k-medians algorithms and so on are available. The standard algorithm defines the total within-cluster variation as the sum of squared Euclidean distances between items and the corresponding centroid. K is a hyperparameter of the algorithm and the k-means algorithm can be summarized as follows:

1. Specify the number of clusters (k) to be created (to be specified by users).

2. Select k data points randomly from the dataset as the initial cluster centers or means.

3. Assign each data point to their closest centroid, based on the euclidean distance between a data point and its centroid.

4. For each of the k clusters update cluster centroid by calculating the new mean values of all the data points in the cluster.

5. Iteratively minimize the total within the sum of squares: iterate steps 3 and 4 until the cluster assignments stop changing or the maximum number of iterations is reached.

The parameters that minimize the cost function are learned through an iterative process of assigning data points to clusters and then moving the clusters. A restriction for the k-means algorithm is that the dataset should be continuous.

> <hands-on-title>K-means clustering</hands-on-title>
>
> 1. **Numeric Clustering** {% icon tool %} with the following clustering parameters:
>    - *"Select the format of input data"*: `Tabular Format (tabular,txt)`
>        - {% icon param-file %} *"Data file with numeric values"*: `iris`
>        - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>        - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>            - {% icon param-text %} *"Type header name(s)"*: `Species`
>        - {% icon param-select %} *"Clustering Algorithm"*: `KMeans`
>        - In *"Advanced options"*
>            - {% icon param-text %} *"Number of clusters"*: `2`
> 2. Rename the generated file to `k-means clustering`
{: .hands_on}


### Visualize k-means clustering

> <hands-on-title>Visualize k-means clustering result</hands-on-title>
>
> 1. **Scatterplot with ggplot2** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: **k-means clustering**
>    - *"Column to plot on x-axis"*: `1`
>    - *"Column to plot on y-axis"*: `2`
>    - *"Plot title"*: `K-means clustering in iris data`
>    - *"Label for x axis"*: `Sepal length`
>    - *"Label for y axis"*: `Sepal width`
>    - In *"Advanced Options"*:
>        - *"Data point options"*: `User defined point options`
>            - *"relative size of points"*: `2.0`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>            - *"column differentiating the different groups"*: `6`
>            - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>    - In *"Output options"*:
>        - {% icon param-text %} *"width of output"*: `7.0`
>        - {% icon param-text %} *"height of output"*: `5.0`
>        - {% icon param-text %} *"dpi of output"*: `175.0`
> 2. **View** {% icon galaxy-eye%} the resulting plot
> 3. Rename to `k-means scatter plot`
{: .hands_on}

![data](images/k_means_scatter.png "K-means clustering scatter plot")


> <question-title></question-title>
>
> How to choose the right number of expected clusters (k)?
>
>
> > <solution-title></solution-title>
> >
> > Major difficulty found with k-means is the choice of the number of clusters. Different methods are proposed to solve this problem.
> > Here, we provide a simple solution. The idea is to compute k-means clustering using different values of clusters k. Next, the within sum of squares is drawn according to the number of clusters. The location of a bend (knee) in the plot is generally considered as an indicator of the appropriate number of clusters.
> > ![data](images/number_of_clusters.png "Optimal number of clusters")
> > The plot above represents the variance within the clusters. It decreases as k increases, but it can be seen as a bend (or “elbow”) at k = 4. This bend indicates that
> > additional clusters beyond the fourth have little value.
> {: .solution}
{: .question}


> <question-title></question-title>
>
> What are the differences between k-means and hierarchical clustering techniques
>
> > <solution-title></solution-title>
> >
> > 1. Hierarchical clustering has difficulties in handling large data well but k-means clustering can. This is because the time complexity (of Lloyd's variant) of k-means is linear (O(nkdi), n = number of data points, k = number clusters, d = data point dimensions and i = number of iterations) while the time-complexity of the optimal hierarchical clustering is quadratic (O(n2)).
> >
> > 2. K-means works well when the clusters are spherical (like circle in 2D, sphere in 3D) in shape. But, when the clusters are of arbitrary geometrical shapes, the performance suffers.
> >
> > 3. K-means clustering requires prior knowledge of the number of clusters. It does not learn the number of clusters from data. But, for the hierarchical clustering it is not necessary.
> >
> {: .solution }
>
{: .question }


## DBSCAN clustering

DBSCAN (Density-based spatial clustering of applications with noise) is a popular clustering algorithm and finds clusters as regions of high density followed by regions of low density. Clusters found by DBSCAN can be of any shape, as opposed to k-means which works well if the clusters are spherical in shape. The central component of the DBSCAN algorithm are the core samples which are present in the areas of high density. A cluster is, therefore, a set of core samples close to one other (measured by some distance measure) and a set of non-core samples that are close to core samples (but are not core samples themselves). There are two important parameters in DBSCAN algorithm - `min_samples` is the number of samples in a neighborhood for a point to be considered as a core point and `eps` is the maximum distance (between two samples) for a sample to be considered as in the neighborhood of the other. Higher the value of `min_samples` or lower the value of eps indicate higher density necessary to form a cluster. DBSCAN does not require one to specify the number of clusters in the data a priori, as opposed to k-means.

> <hands-on-title>DBSCAN clustering</hands-on-title>
>
> 1. **Numeric Clustering** {% icon tool %} with the following clustering parameters:
>    - *"Select the format of input data"*: `Tabular Format (tabular,txt)`
>        - {% icon param-file %} *"Data file with numeric values"*: `iris`
>        - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>        - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>            - {% icon param-text %} *"Type header name(s)"*: `Species`
>        - {% icon param-select %} *"Clustering Algorithm"*: `DBSCAN`
> 2. Rename the generated file to `DBSCAN clustering`
{: .hands_on}


### Visualise DBSCAN clustering

> <hands-on-title>Visualize DBSCAN clustering result</hands-on-title>
>
> 1. **Scatterplot with ggplot2** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: **DBSCAN clustering**
>    - *"Column to plot on x-axis"*: `1`
>    - *"Column to plot on y-axis"*: `2`
>    - *"Plot title"*: `DBSCAN clustering in iris data`
>    - *"Label for x axis"*: `Sepal length`
>    - *"Label for y axis"*: `Sepal width`
>    - In *"Advanced Options"*:
>        - *"Data point options"*: `User defined point options`
>            - *"relative size of points"*: `2.0`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>            - *"column differentiating the different groups"*: `6`
>            - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>    - In *"Output options"*:
>        - {% icon param-text %} *"width of output"*: `7.0`
>        - {% icon param-text %} *"height of output"*: `5.0`
>        - {% icon param-text %} *"dpi of output"*: `175.0`
> 2. **View** {% icon galaxy-eye%} the resulting plot:
> 3. Rename to `DBSCAN scatter plot`
{: .hands_on}


![data](images/dbscan_scatter.png "DBSCAN clustering scatter plot")


You will also notice that the green points (factor = -1) in the plot are not contained within any cluster. DBSCAN does not necessarily categorize every data point, and is therefore works very well with handling outliers in a dataset.

> <question-title></question-title>
>
> How can we evaluate the clustering results?
>
>
> > <solution-title></solution-title>
> >
> > Clustering is an unsupervised learning algorithm; there are no labels or ground truth to compare with the clusters. However, we can still evaluate the performance of the algorithm using intrinsic measures.
> > There is a performance measure for clustering evaluation which is called the [silhouette coefficient](https://en.wikipedia.org/wiki/Silhouette_(clustering)). It is a measure of the compactness and separation of the clusters.
> > It increases as the quality of the clusters increase; it is large for compact clusters that are far from each other and small for large, overlapping clusters. The silhouette coefficient is calculated per instance; for a set of instances, it is calculated as the mean of the individual sample score.
> {: .solution}
{: .question}

# Applying clustering algorithms on multiple datasets

We can apply the same steps on the other datasets such `moon` and `circles` datasets (already imported) which are generated using [scikit-learn](https://scikit-learn.org/stable/datasets/index.html#generated-datasets) methods.

## Visualise datasets

> <hands-on-title>Visualize scatter plot of data</hands-on-title>
>
> 1. **Scatterplot with ggplot2** {% icon tool %} with the following parameters:
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>    - {% icon param-file %} *"Input tabular dataset"*: `circles` and `moon` as **multiple datasets**
>    - *"Column to plot on x-axis"*: `1`
>    - *"Column to plot on y-axis"*: `2`
>    - *"Plot title"*: `Scatter Plot`
>    - *"Label for x axis"*: `X`
>    - *"Label for y axis"*: `Y`
>    - In *"Output options"*:
>        - {% icon param-text %} *"width of output"*: `7.0`
>        - {% icon param-text %} *"height of output"*: `5.0`
>        - {% icon param-text %} *"dpi of output"*: `175.0`
> 2. **View** {% icon galaxy-eye%} the resulting plots
{: .hands_on}

![data](images/circles_moon_scatter.png "Scatter plot of circles and moon datasets")


## Find clusters

Now you can find clusters in these datasets using the aforementioned algorithms.

> <hands-on-title>Hierarchical clustering of circles and moon datasets</hands-on-title>
>
> 1. **Numeric Clustering** {% icon tool %} with the following clustering parameters:
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>    - *"Select the format of input data"*: `Tabular Format (tabular,txt)`
>        - {% icon param-file %} *"Data file with numeric values"*: `circles` and `moon` as **multiple datasets**
>        - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>        - {% icon param-select %} *"Choose how to select data by column"*: `All columns`
>        - {% icon param-select %} *"Clustering Algorithm"*: `Hierarchical Agglomerative Clustering`
>        - In *"Advanced option"*
>            - {% icon param-text %} *"Number of clusters"*: `2`
>            - {% icon param-select %} *"Affinity"*: `Euclidean`
>            - {% icon param-select %} *"Linkage"*: `ward`
> 2. Rename the generated files to `circles hierarchical clustering` and `moon hierarchical clustering` respectively
{: .hands_on}

## Visualise clusters

Then, you can visualize the clustering results using the following steps:

> <hands-on-title>Visualize hierarchical clustering result on circles and moon datasets.</hands-on-title>
>
> 1. **Scatterplot with ggplot2** {% icon tool %} with the following parameters:
>
>    {% snippet faqs/galaxy/tools_select_multiple_datasets.md %}
>    - {% icon param-file %} *"Input tabular dataset"*: `circles hierarchical clustering` and `moon hierarchical clustering` as **multiple datasets**
>    - *"Column to plot on x-axis"*: `1`
>    - *"Column to plot on y-axis"*: `2`
>    - *"Plot title"*: `Hierarchical clustering`
>    - *"Label for x axis"*: `X`
>    - *"Label for y axis"*: `Y`
>    - In *"Advanced Options"*:
>        - *"Data point options"*: `User defined point options`
>            - *"relative size of points"*: `2.0`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>            - *"column differentiating the different groups"*: `3`
>            - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>    - In *"Output options"*:
>        - {% icon param-text %} *"width of output"*: `7.0`
>        - {% icon param-text %} *"height of output"*: `5.0`
>        - {% icon param-text %} *"dpi of output"*: `175.0`
> 2. **View** {% icon galaxy-eye%} the resulting plots
> 3. Rename the generated files to `Circles scatter plot` and `Moon scatter plot` respectively
{: .hands_on}

You can apply the other two algorithms (k-means and DBSCAN) to moon and circles datasets in the same way as explained above. In the k-means algorithm, please use `k=2` and for the DBSCAN algorithm, the parameters should not be the default ones as used earlier. They should be set as follows: for the circles dataset (`maximum neighborhood distance=0.2` and `minimal core point density=5`) and for the moon dataset (`maximum neighborhood distance=0.3` and `minimal core point density=4`). You can see the scatter plots of the clustering results for all three clustering algorithms in Figure 13 and 14.

![data](images/circles_clustering.png "Plot of clustering algorithms on circles dataset")

![data](images/moon_clustering.png "Plot of clustering algorithms on moon dataset")


# Conclusion

In this tutorial, we discussed 3 clustering algorithms which are used to discover structures or patterns in unlabeled data. You learned about the hierarchical, k-means and DBSCAN algorithms. By following steps specified for each clustering tool, we learned how to perform clustering and visualize results using clustering and plotting tools, respectively in Galaxy. There are many other clustering approaches which can be tried out on these datasets to find how they perform and how they compare to the 3 clustering algorithms explained in this tutorial. Different datasets can also be analysed using these algorithms. The clustering algorithms have some parameters which can be altered while performing the analyses to see if they affect the clustering or not. While using clustering algorithms, we need to take care of some important aspects like treating outliers in data and making sure each cluster has sufficient population. Some data pre-processors can also be used to clean the datasets.


topic: statistics
tutorial: gpu_jupyter_lab
title: A Docker-based interactive Jupyterlab powered by GPU for artificial intelligence in Galaxy
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
content_hash: 010d620f3cd41d6615103f589f3494f0
last_modified: 2026-02-26T21:12:18.199788
zenodo_link: https://zenodo.org/record/6091361
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/gpu_jupyter_lab/tutorial.html

content:
[Jupyterlab](https://jupyterlab.readthedocs.io/en/stable/) is a popular integrated development environment (IDE) for a variety of tasks in data science such as prototyping analyses, creating meaningful plots, data manipulation and preprocessing. Python is one of the most used languages in such an environment. Given the usefulness of Jupyterlab, more importantly in online platforms, a robust [Jupyterlab notebook application](https://usegalaxy.eu/root?tool_id=interactive_tool_ml_jupyter_notebook) has been developed that is powered by GPU acceleration and contains numerous packages such as Pandas, Numpy, Scipy, [Scikit-learn](https://scikit-learn.org/), [Tensorflow](https://www.tensorflow.org/), [ONNX](https://onnx.ai/) to support modern data science projects. It has been developed as an interactive Galaxy tool that runs on an isolated [docker container](https://github.com/anuprulez/ml-jupyter-notebook). The docker container has been built using [nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04](https://hub.docker.com/layers/anupkumar/docker-ml-jupyterlab/galaxy-integration-0.2/images/sha256-e2d7e28a2f975523db0f5ac29c2e2ce3c7a35b061072098ad388d5b42ee86fba?context=repo) as the base container. Moreover, a Galaxy [tool](https://github.com/bgruening/galaxytools/pull/1157) ( `run_jupyter_job`) can be executed using [Bioblend](https://bioblend.readthedocs.io/) which uses Galaxy's remote job handling for long-running machine learning and deep learning training. The training happens remotely on a Galaxy cluster and the outcome datasets such as the trained models, tabular files and so on are saved in a Galaxy history for further use.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Features of Jupyterlab
[Jupyterlab](https://jupyterlab.readthedocs.io/en/stable/) notebook has been augmented with several useful features and together they make it ready-to-use for quick prototyping and end-to-end artificial intelligence (AI) projects. Being able to **serve online** makes it convenient to share it with other researchers and users. Its features can be broadly classified into two categories - features that have been added solely as Python packages and those that have been added also as Python packages but have their respective user interfaces. We will briefly discuss these features and later, we will use some of those for building and using AI models.

### Features (Python packages)
Accessibility to many features has been made possible by adding several compatible Python packages, the most prominent ones are Bioblend, [CUDA](https://developer.nvidia.com/cuda-toolkit) and ONNX. Using Bioblend the notebook can be **connected to Galaxy** and its histories, tools and workflows can be accessed and operated using various commands from the notebook. GPUs have accelerated AI research, especially deep learning. Therefore, the backend of the Jupyterlab is **powered by GPU** to make long-running AI training programs finish faster by parallelizing matrix multiplications. CUDA acts as a bridge between GPU and AI algorithms. In addition, a standard AI model format, **open neural network exchange (ONNX)**, has been supported in the notebook to transform Scikit-learn and Tensorflow models to `onnx` files. These model files can be conveniently shared and used for inference. Galaxy also supports `onnx` file format which makes it easier to interoperate such models from a notebook to Galaxy and vice-versa. Galaxy tool for **remote training** can also run on GPU. There are several additional packages, suited for performing machine learning tasks such as Scikit-learn and Tensorflow for developing AI algorithms, [Open-CV](https://github.com/opencv/opencv) and [Scikit-Image](https://scikit-image.org/) for image processing, [NiBabel](https://nipy.org/nibabel/) package for processing images files have been made available to researchers.

### Features (Python packages with user interfaces)
There are other features, added as Python packages, that have user interface components. These include **[GPU utilization dashboards](https://github.com/rapidsai/jupyterlab-nvdashboard)** for monitoring the GPU usage and system memory utilization, **[Voila](https://github.com/voila-dashboards/voila)** for rendering output cells of a notebook in a separate tab hiding all code cells, interactive **[Bqplots](https://github.com/bqplot/bqplot)** for plotting, **[Git](https://github.com/jupyterlab/jupyterlab-git)** version control to clone, commit and maintain codebase directly from the notebook and **[Elyra AI](https://elyra.readthedocs.io/en/stable/)** pipelines for knitting together multiple notebooks to create a workflow. Many notebooks are created for processing datasets in different ways. These notebooks can be knit together to form one pipeline where each notebook transforms datasets taking a different form of data from its previous notebook and passing on the transformed datasets to its next notebook. An example workflow created using Elyra AI can be found by the name "METABRIC\_ML.pipeline" in the "elyra" folder of a typical Jupyterlab instance. These pipelines can also be executed remotely on multiple "runtimes" supported by docker containers. An example pipeline created in the notebook can be seen in Figure 1. These features, integrated into the Jupyterlab notebook, are useful for the efficient execution and management of data science projects.

![Elyra AI pipeline](../../images/elyra_ai_pipeline.png "An example ML pipeline created using Elyra AI in a notebook. Each connected block of the pipeline is a different notebook")

In the following sections, we will look at two examples of machine learning analyses - image segmentation and prediction of 3D structure of protein sequences. We will go through each step of how to perform such analyses in this Jupyterlab notebook.

## Image segmentation
In an image classification task, a label is assigned to each image. For example, an image containing a cat gets a label as "cat" ("cat" is given an integer to be used in a machine learning program). But, in an image segmentation task, each pixel of an image gets a label. For example, in an image containing a cat, all the pixels occupied by the cat in the image are denoted by violet color (see Figure 2). More information can be found in the [image segmentation](https://www.tensorflow.org/tutorials/images/segmentation) tutorial. Image segmentation is a widely used technique in many fields such as medical imaging to find out abnormal regions in images of tissues, self-driving cars to detect different objects on highways and many more.

![Image segmentation for a cat](../../images/cat_segmentation.png "Image segmentation for cat. The violet region of the middle plot denotes the cat and the outline of the violet section denoted by the yellow color is the cat's mask")


## COVID CT scan
Segmenting lung CT scans task to locate infected regions has been actively proposed to augment the RT-PCR testing for initial screening of COVID infection in humans. Deep learning has been used to predict these regions with high accuracy in studies such as {% cite InfNet %}, {% cite JCSWu2021 %} and many more. In Figure 3, the differences between the CT scans of a normal person and a person suffering from COVID-19 can be seen. The regions of the lungs marked by white patches are infected as discussed in {% cite SaeedizadehTVUnet %}.

![Normal and Covid CT scans](../../images/normal_covid_ct_scans.jpg "(Left) Lungs CT scan for a normal person and (right) lungs CT scan for a person having COVID-19")


## CT scans and masks
Figure 4 shows CT scans of infected lungs (top row) and borders around the infected regions have been drawn with red color (middle row). In the last row respective masks, from the CT scans, have been taken out denoting the infected regions. These masks are the "segmented" regions from the corresponding CT scans. While creating a dataset for training the deep learning model, Unet (briefly discussed in the following section), the CT scans are the data points and its "known" respective masks of infected regions become their labels ({% cite SaeedizadehTVUnet %}).

![Masks](../../images/covid_ct_scan_masks.png "Picture shows CT scans and their masks of infected regions that are drawn with red color in middle row plots")

### Unet neural network
Unet neural network is widely used for segmentation tasks in images ({% cite Ronnebergerunet %}). The name "Unet" resembles the shape of its architecture as "U" (see Figure 5). It has two parts - encoder and decoder. The left half of the "U" shape is the encoder that learns the feature representation (downsampling) at a lower level. The right half is the decoder that maps the low-resolution feature representation on the higher resolution (upsampling) pixel space.

![Unet architecture](../../images/Unet.jpg "Architecture of Unet neural network for image segmentation. The name "Unet" resembles the shape of its architecture as 'U'")

In this tutorial, we will use the dataset of CTs scans and their respective masks to train an Unet neural network model. The model learns to map the infected regions in the CT scans to their masks. For prediction, the trained model is given unseen CT scans and for each CT scan, it predicts infected regions or masks. For this experiment, we will use Jupyterlab for cloning the notebooks from Github that contain scripts for downloading data, creating and training Unet model and prediction tasks. Data (CT scans and also the trained model) required for the notebooks can be downloaded from Zenodo ({% cite anupkumar20226091361 %}). The model can either be trained in Jupyterlab or can be sent to Galaxy's cluster for remote processing. After remote processing, the created datasets such as the trained model become available in new Galaxy history.

## Protein 3D structure
Understanding the structure of proteins provides insights into their functions. But, only a few proteins have known structures out of billions of known proteins. To advance in the direction of predicting their 3D structures only from their sequences, AlphaFold2 ({% cite AlphaFold2 %}) has made a breakthrough to predict their structures with high accuracy. However, the databases that it uses are large, approximately 2 terabytes (TB), and are hard to store and manage. Therefore, to make it more accessible, a few approaches such as ColabFold ({% cite Mirdita2021 %}), have been developed that consume significantly less memory, much faster and produce 3D structures with similar accuracy. We will look at, in later sections, how we can use ColabFold to predict 3D structure of a protein sequence using the Jupyterlab infrastructure in Galaxy.

# Reproduce results from recent publications
First, we will discuss a few features of Jupyterlab to create and train a Unet deep learning model and then, predict segmented regions in COVID CT scans using the trained Unet model ({% cite SaeedizadehTVUnet %}). In addition, we will predict 3D structure of protein sequence using ColabFold ({% cite Mirdita2021 %}), a faster implementation of AlphaFold2. First, let's start the Jupyterlab instance on Galaxy Europe. The interactive tool can be found by searching "GPU enabled Interactive Jupyter Notebook for Machine Learning" name in the tool search.

## Open Jupyterlab

> <hands-on-title>GPU enabled Interactive Jupyter Notebook for Machine Learning</hands-on-title>
>
> - {% tool [GPU enabled Interactive Jupyter Notebook for Machine Learning](interactive_tool_ml_jupyter_notebook) %}
>    - *"Do you already have a notebook?"*: `Start with default notebooks`
>    - Click *"Run Tool"*
>
{: .hands_on}

Now, we should wait for a few minutes until Galaxy creates the required compute environment for opening a new Jupyterlab. Usually, this task takes around 10-15 minutes. The progress can be checked by clicking on the "User>Active Interactive tools". On "Active Interactive Tools" page, there is a list of all open interactive tools. When the job info shows "running" on the "Active Interactive Tools" page, then the name of the interactive tool gets associated with a URL to the running Jupyterlab. On clicking this URL, the running Jupyterlab can be opened. Several features of Jupyterlab running in Galaxy can be learned by going through the "home_page.ipynb" notebook. The folder "notebooks" contain several notebooks that show multiple use-cases of features and packages. Now, let's download the necessary notebooks from Github for performing image segmentation and predicting 3D structure of protein sequences.

## Clone Github repository
To use Git version control for cloning any codebase from GitHub, the following steps should be performed. Alternatively, the notebooks that showcase different usecases are also available by default at `<<root>>/usecases/`.

> <hands-on-title>Pull code</hands-on-title>
>
> 1. Create a new folder named `covid_ct_segmentation` alongside other folders such as "data", "outputs", "elyra" or you can use your favourite folder name.
> 2. Inside the created folder, clone a code repository by clicking on "Git" icon as shown in Figure 6.
> 3. In the shown popup, provide the repository path as shown below and then, click on "clone":
>    ```
>    https://github.com/anuprulez/gpu_jupyterlab_ct_image_segmentation
>    ```
> 4. The repository "anuprulez/gpu_jupyterlab_ct_image_segmentation" gets immediately cloned.
> 5. Move inside the created folder `gpu_jupyterlab_ct_image_segmentation`. A few notebooks can be found inside that are numbered.
>    ![Clone repository](../../images/git_clone.png "Clone a code repository using Git")
>
{: .hands_on}

Now, we have all the notebooks available for performing image segmentation and predicting 3D structures of protein sequences. The entire analysis of image segmentation can be completed in two ways - first by running 3 notebooks (numbered as 1, 2 and 3) in the Jupyterlab itself and second by using 4, 5 and 6 notebooks that start remote training on Galaxy cluster invoking another Galaxy tool from one of the notebooks. First, let's look at how we can run this analysis in the notebook itself.


## Use-case 1: Run image segmentation analysis

### Run notebooks in Jupyterlab

> <hands-on-title>Run notebooks in Jupyterlab</hands-on-title>
>
> 1. Download and save datasets in the notebook using **"1_fetch_datasets.ipynb"** notebook. It creates all the necessary folders and then, downloads two datasets - one "h5" file containing many matrices as sub-datasets belonging to training data, training labels, validation data, validation labels, test data and test labels. 
>
>    > <comment-title></comment-title>
>    >  Sub-datasets are stored as different variables after reading the original `h5` file once. For training, we only need these datasets/matrices - training data, training labels, validation data and validation labels. The matrices, test data and test labels, are used only for prediction. We use `h5` format for storing and retrieving datasets as all AI algorithms need input datasets in the form of matrices. Since, in the field of AI, there are many different types of datasets such as images, sequences, real numbers and so on, therefore, to converge all these different forms of datasets into one format, we use `h5` to store matrices. In any AI analysis, different forms of datasets can be stored as `h5` files. Therefore, for the image segmentation task, all the input datasets/matrices to the deep learning model are saved as sub-datasets in one `h5` file so that they can be easily created, stored, downloaded and used. This step may take a few minutes as it downloads around 450 MB of data from Zenodo ({% cite anupkumar20226091361 %}). Once these datasets are downloaded to the notebook, we can move to the next step to create a deep learning model and start training it.
>    >
>    {: .comment}
>
> 2. Create and train a deep learning Unet model using **"2_create_model_and_train.ipynb"** notebook. This will read the input `h5` dataset containing all images and train the model after creating deep learning model architecture.
>
>    > <comment-title></comment-title>
>    > This notebook first creates a deep learning architecture based on Unet including custom loss functions such as total variation and binary cross-entropy losses. After creating the deep learning architecture, all training datasets such as training data, training labels, validation data, validation labels are loaded from the combined `h5` file. In the next step, all the datasets and deep learning architecture are compiled together and training starts for 10 epochs (10 iterations over the entire training dataset). The training is fast as it runs on GPU and finishes in a few minutes and creates a trained model. In the last step, the trained model containing several files is converted to one `onnx` file. Once a trained model is ready, we can move to the next step to make predictions on unseen CT scan masks.
>    >
>    {: .comment}
>
> 3. Predict unseen masks using the trained model in **"3_predict_masks.ipynb"** notebook.
>
>    > <comment-title></comment-title>
>    > First, it reads test datasets from the combined "h5" file and then, loads the `onnx` model. Using this model, it predicts masks of unseen CT scans and then plots the ground truth and predicted masks in one plot (see Figure 7)
>    >
>    {: .comment}
>
> ![True and predicted masks of CTs scans](../../images/true_pred_masks.png "Ground truth and predicted masks of COVID CT scans. Predicted masks in the third and fourth columns (computed using slightly different loss functions) are very close to the corresponding original masks in the second column")
>
> Figure 7 shows original lungs CT scans in the first column. The second column shows the corresponding true infected regions in white. The third and fourth columns show the infected regions predicted with different loss functions, one is the binary cross-entropy loss and another is the combination of binary cross-entropy loss and total variation loss. Binary cross-entropy calculates loss between two corresponding pixels of true and predicted images. It measures how close the two corresponding pixels are. But, total variation loss measures the amount of noise in predicted images as noisy regions usually show large variations in their neighbourhood. The noise in the predicted images gets minimized and the connectivity of predicted masks improves as well when minimizing this loss.
>
{: .hands_on}


### Run Jupyterlab notebooks remotely on a Galaxy cluster

The training task completed in the notebook above can also be sent to a Galaxy cluster by executing a Galaxy tool in the notebook itself. Using [Bioblend](https://bioblend.readthedocs.io/) APIs, datasets in the form of `h5` files and notebook are uploaded to a Galaxy history and then a Galaxy tool `run_jupyter_job` executes the notebook using the uploaded dataset on a Galaxy cluster and creates a trained model in the Galaxy history. Let's look at how to execute a notebook remotely on a Galaxy cluster.

> <hands-on-title>Run Jupyterlab notebooks remotely on a Galaxy cluster</hands-on-title>
>
> 1. Download and save datasets in the notebook using **"1_fetch_datasets.ipynb"** notebook in the same way as before. Ignore this step if this notebook has already been executed.
> 2. Execute **"5_run_remote_training.ipynb"** notebook to dynamically execute code inside **"4_create_model_and_train_remote.ipynb"** notebook on a remote cluster using a different Galaxy tool (`run_jupyter_job`). The notebook **"5_run_remote_training.ipynb"** provides the path of a notebook to be executed remotely along with the datasets to the Galaxy tool by calling a custom function `run_script_job` which is part of the Jupyterlab notebook. 
>
>    > <comment-title></comment-title>
>    > Executing **"5_run_remote_training.ipynb"** uploads datasets and dynamic Python script, extracted from the **"4_create_model_and_train_remote.ipynb"** notebook, to a newly created Galaxy history (Figure 8). When the task of uploading dataset and dynamic code is finished, the Galaxy tool (`run_jupyter_job`) executes the dynamically uploaded script with the uploaded dataset on a remote Galaxy cluster which is similar to running any other Galaxy tool. When the Galaxy tool (`run_jupyter_job`) finishes its execution, the resulting models and other datasets appear in the created Galaxy history. While the job is running on the Galaxy cluster, the Jupyter notebook can be closed as the model training task gets decoupled from the notebook and is entirely transferred to the Galaxy cluster. Specific history used for this job can be accessed in Galaxy. The Jupyterlab method `run_script_job` has multiple input parameters - `script_path`: relative path to the script that is to be executed remotely, `data_dict`: list of input datasets as `h5` files, `server`: Galaxy server URL, `key`: Galaxy API key and `new_history_name`: the name of the Galaxy history that is created). 
>    >
>    {: .comment}
>
>    ![Galaxy history](../../images/finished_history_remote_ai.png "Galaxy history showing finished datasets after remote training on a Galaxy cluster")
> 
> **Note**: The training may take longer depending on how busy Galaxy's queueing is as it sends the training task to be done on a Galaxy cluster. Therefore, this feature should be used when the training task is expected to run for several hours. The training time is higher because a large Docker container is downloaded on the assigned cluster and only then, the training task can proceed.
>
{: .hands_on}


When the Galaxy job finishes, it creates a history having a few datasets such as input `h5` dataset, dynamic Python script extracted from the notebook, a trained model saved as `onnx` file, all saved lists and NumPy arrays and a zipped file containing all the intermediate files created and saved in the dynamic script. One such example of a Galaxy history is [available](https://usegalaxy.eu/u/kumara/h/ctsegmentationmarch18). Datasets from this history can be downloaded into the Jupyterlab notebook and further analyses can be performed. Using the notebook **"6_predict_masks_remote_model.ipynb"**, the trained model can be downloaded from the newly created Galaxy history to Jupyterlab notebook and then, it can be used to make predictions of unseen CT scan masks in the same way as shown in **"3_predict_masks.ipynb"** notebook.


## Use-case 2: Run ColabFold to predict 3D structure of protein

Google Deepmind’s AlphaFold2 has made a breakthrough in predicting the 3D structure of proteins with outstanding accuracy. However, due to their large database size (a few TB), it is not easily accessible to researchers. Therefore, a few approaches have been developed that replace the time-consuming steps of AlphaFold2 with slightly different steps to create input features and predict the 3D structure with comparable accuracy as AlphaFold2. One such approach is ColabFold which replaces the large database search in AlphaFold2 for finding homologous sequences by a significantly faster [MMseqs2 API](https://github.com/soedinglab/MMseqs2) call to generate input features based on the query protein sequence. ColabFold is approximately 16 times faster than AlphaFold2 in predicting 3D structures of protein sequences. To make it accessible in this Jupyterlab, it has been integrated into the Docker container by having two additional packages - ColabFold and the GPU enabled [JAX](https://github.com/google/jax) which is just-in-time compiler from Google used for mathematical transformations. In the following hands-on section, we will run a notebook and predict a 3D structure of a protein sequence. 

> <hands-on-title>Predict 3D structure of protein using ColabFold</hands-on-title>
>
> 1. Clone a Github repository by following the steps described in the previous hands-on section `Clone Github repository`. Ignore this step if already done.
> 2. Open **7_ColabFold_MMseq2.ipynb** notebook and execute it.
> 3. It needs a few minutes (around 5-10 minutes) to download AlphaFold2 weights (of size approximately 4 GB) and uses these weights for prediction. An example prediction is shown in figure 9. 
> !["3D structure of a protein sequence of length 300 with side-chains predicted by ColabFold"](../../images/3D_300_L_protein.png "3D structure of a protein sequence of length 300 with side-chains predicted by ColabFold")
>
{: .hands_on}


## Docker's security
For remote training, a dynamic Python script is sent to a Galaxy tool for execution on a Galaxy cluster that may pose security risks of containing malicious code. To minimize the security risks that come along with executing dynamic Python scripts, the Galaxy tool `run_jupyter_job` is executed inside a [Docker container](https://github.com/anuprulez/ml-jupyter-notebook). It is the same container inside which the Jupyterlab notebook runs. All the packages available inside the notebook are also available for the Galaxy tool. Programs run in isolated environments inside Docker containers and have their connections limited to necessary libraries, and network connections. Due to these restrictions, they have a reduced number of interactions with the host operating system.

# GPU Juyterlab tool in a Galaxy workflow

## Introduction

In addition to an interactive mode that opens a Jupyterlab in Galaxy, the GPU Jupyterlab tool can also be used as a tool in a Galaxy workflow. In this mode, it takes input datasets from a different tool along with an `ipynb` notebook file. It executes the `ipynb` file along with the input datasets and produces output datasets inside a dataset collection. This output dataset collection can then be used with other Galaxy tools. This feature enables it to be used in a Galaxy workflow like other Galaxy tools.

### Execute an existing notebook
In this mode, the GPU Jupyterlab tool executes the input `ipynb` file and produces output datasets (in a collection) if created in the notebook file. Input datasets can also be attached to the notebook which become available at `/galaxy_inputs/<<Name for parameter>>` folder. The parameter `Name for parameter` is set for each input dataset that is used as the folder name inside `/galaxy_inputs/`. More than one input datasets can be attached to the notebook. Another feature `Execute notebook and return a new one` controls whether webfrontend becomes available showing Jupyterlab or a notebook is just executed. When this parameter is set to `yes`, then the attached notebook is executed and output datasets if any become available in an output dataset collection. When this parameter is set to `no` (by default), then Jupyterlab infrastructure as a webfrontend is opened along with the attached notebook and input datasets if attached.

## Run as a tool in workflow
When the parameter `Execute notebook and return a new one` is set to `yes`, the GPU Jupyterlab tool can be used as a part of any workflow. In this mode, it requires an `ipynb` file/notebook that gets executed in Galaxy and output datasets if any become available in the Galaxy history. Along with a notebook, multiple input datasets can also be attached that become automatically available inside the notebook. They can be accessed inside the notebook and processed to produce desired output datasets. These output datasets can further be used with other Galaxy tools. The following image shows a sample workflow for illustration purposes. Similarly, high-quality workflows to analyse scientific datasets can be created.


!["A sample Galaxy workflow that uses GPU Jupyterlab as a tool"](../../images/workflow_gpu_jupyterlab.png "A sample Galaxy workflow that uses GPU Jupyterlab as a tool which takes input datasets from one tool, trains a machine learning model to predict classes and then the predicted datasets is used as input to another Galaxy tool.")

Let's look at how can this workflow be created in a step-wise manner. There are 3 steps - first, the training dataset is filtered using the `Filter` tool. The output of this tool along with 2 other datasets (`test_rows` and `test_rows_labels`), a sample IPython notebook is executed by the GPU Jupyterlab tool. The sample IPython notebook trains a simple machine learning model using the train dataset and creates a classification model using `RandomForestClassifier`. The trained model is then used to predict classes using the test dataset. The predicted classes is produced as a file in an output collection by the GPU Jupyterlab tool. As a last step, `Cut` tool is used to extract the first column of the output collection. Together, these steps showcase how the GPU Jupyterlab tool is used with other Galaxy tools in a workflow.

### Get data

Datasets such as train, test and sample IPython (`ipynb`) notebook files are downloaded from [Zenodo](https://zenodo.org/record/7534061) to import them into a Galaxy history. 

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Make sure you have an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to make it easy to recognize
>
>    > <tip-title>Rename a history</tip-title>
>    >
>    > * Click on the title of the history (by default the title is `Unnamed history`)
>    >
>    >   ![Renaming history]({% link shared/images/rename_history.png %})
>    >
>    > * Type `Galaxy Introduction` as the name
>    > * Press <kbd>Enter</kbd>
>    >
>    {: .tip}
>
>
> 3. Import the files from [Zenodo](https://zenodo.org/record/7534061)
>
>    ```
>    https://zenodo.org/record/7534061/files/sample_notebook.ipynb
>    https://zenodo.org/record/7534061/files/test_rows.csv
>    https://zenodo.org/record/7534061/files/train_rows.csv
>    https://zenodo.org/record/7534061/files/test_rows_labels.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Rename the datasets as `test_rows`, `train_rows` and `test_rows_labels` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Check that the datatype of all three tabular datasets is `csv`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}


### Filter training dataset
This tool showcases how a tool prior to using the GPU Jupyterlab tool (in a workflow) can be used and how its output can be used as input to the GPU Jupyterlab tool. 

> <hands-on-title>Filter</hands-on-title>
>
> - {% tool [Filter](Filter1) %}
>    - *"Filter *"* : Select `train_rows`
>    - *"With following condition"* :`c3==0`
>    - *"Number of header lines to skip"*: `1`
>    - Click *"Run Tool"*
>
{: .hands_on}

Rename the filtered dataset as `new_train_rows` which will be used as one of the inputs to the GPU Jupyterlab tool in the next step. The datatype should be `tabular`.

### Execute IPython notebook using GPU Jupyterlab tool
This tool showcases GPU Jupyterlab tool that executes a notebook using input datasets produced by different tools. Galaxy tool like features for this interactive tool has been adapted from another interactive Galaxy tool **Interactive JupyTool and notebook**.

> <hands-on-title>GPU Jupyterlab tool</hands-on-title>
>
> - {% tool [GPU Jupyterlab tool](interactive_tool_ml_jupyter_notebook) %}
>    - *"Do you already have a notebook?"* : Select `Load an existing notebook`
>    - *"IPython Notebook"* : Select `sample_notebook.ipynb`
>    - *"Execute notebook and return a new one"*: `Yes`
>    - In *"User inputs"*:
>        - {% icon param-repeat %} *"1: User inputs"*
>            - *"Name for parameter"*: `train`
>            - *"Choose the input type"*: `Dataset`
>            - *"Select value"*: Select `new_train_rows`
>        - {% icon param-repeat %} *"2: User inputs"*
>            - *"Name for parameter"*: `test`
>            - *"Choose the input type"*: `Dataset`
>            - *"Select value"*: Select `test_rows`
>        - {% icon param-repeat %} *"3: User inputs"*
>            - *"Name for parameter"*: `testlabels`
>            - *"Choose the input type"*: `Dataset`
>            - *"Select value"*: Select `test_rows_labels`
>
{: .hands_on}


### Cut a column from the output collection
This tool showcases how the output dataset collection produced by the GPU Jupyterlab tool can be used by a different Galaxy tool.

> <hands-on-title>Cut</hands-on-title>
>
> - {% tool [Cut](Cut1) %}
>    - *"Cut columns"* : `c1`
>    - *"Delimited by"* : `Comma`
>    - *"From"*: Choose dataset collection `GPU JupyterLab notebook output collection`
>    - Click *"Run Tool"*
>
{: .hands_on}


The above tool extracts the first column from dataset residing in the output dataset collection. All three steps explained above show how the GPU Jupyterlab tool can be used with other Galaxy tools to create a workflow.


# Conclusion
In this tutorial, we have discussed several features of the Jupyterlab notebook in Galaxy and used a few of them to perform an image segmentation task on COVID CT scans by training Unet, a deep learning model. In addition, we have used it for predicting 3D structure of a protein sequence which is a memory-intensive and time-consuming task. The ready-to-use infrastructure that we employed here for developing an AI algorithm and training a model shows that it is convenient for creating quick prototypes and end-to-end projects, accelerated by GPU, in the data science field without caring much about the storage and compute resources.

The possibility of remote execution of long-running training tasks obviates keeping the notebook or computer open till the training finishes and at the same time, the datasets and models stay safe in Galaxy history. The usage of `h5` files for input datasets to AI algorithms solves the problem of varying data formats across multiple fields that are used extensively for predictive tasks. Using `onnx` for creating, storing and sharing an AI model makes it easier to handle such it as one compact file. Git integration makes it easy to maintain an entire code repository directly in the notebook. The entire environment can be shared with other researchers only via sharing the URL of the Jupyterlab. Bioblend connects this infrastructure to Galaxy which makes its histories, workflows and tools accessible directly via the notebook. Overall, this infrastructure would prove useful for researchers having little access to large compute resources to develop and manage quick prototypes and end-to-end data science projects.

Feature to use GPU Jupyterlab as a regular Galaxy tool provides an ability to execute IPython notebooks in a workflow. The executed notebook is also available in the Galaxy history with the associated cell outputs.

tags_json:
["interactive-tools", "machine-learning", "deep-learning", "jupyter-lab", "image-segmentation", "protein-3D-structure"]


topic: statistics
tutorial: intro_deep_learning
title: Introduction to deep learning
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 2563536d69bc1a9e678c330ea8934123
last_modified: 2026-02-26T21:12:18.212319
zenodo_link: https://zenodo.org/record/3706539
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/intro_deep_learning/tutorial.html

content:
## Introduction

### Deep learning and neural networks
[Deep learning](https://en.wikipedia.org/wiki/Deep_learning), a branch of artificial intelligence, provides a collection of learning methods to model data with complex architectures to perform different non-linear transformations of data. Using these transformations, patterns are recognised in large volumes of data and new data can be categorised using these patterns extracted on existing data. These patterns are learned by computational models devised using different architectures of neural networks. In the recent years, the neural network architectures such as convolutional, long short-term memory networks, deep belief networks have become increasingly popular as machine learning tools in the fields of computer vision, image analysis, bioinformatics, speech recognition, natural language processing and so on achieving state-of-the-art performance, sometimes exceeding human performance. The availability of greater computational resources, more data, new algorithms for training deep models and easy to use libraries for implementation and training of neural networks are the drivers of this development. Deep learning works by approximating the mathematical function which maps data to its output and it has been shown that it can [approximate](https://arxiv.org/pdf/1910.03344.pdf) any function making it widely popular across multiple fields to analyse data. A neural network is a web of artificial neurons which are also called processing units. The idea of a neural network is inspired by [biological neural networks](https://en.wikipedia.org/wiki/Neural_circuit) where neuronal circuits are used to process information and learn. An artificial neural network is structured into multiple layers where each layer contains several neurons. The neurons from adjacent layers are interconnected ([feed-forward neural network](https://en.wikipedia.org/wiki/Feedforward_neural_network)) allowing the exchange of information between layers of neurons.

![data](../../images/neuron.svg "Structure of an artificial neuron. The input x (x1, x2, ..., xn) and its corresponding weight w (w1, w2, ..., wn) are vectors. The input and its weight are transformed to produce an output y")

An artificial neuron is shown in Figure 1. The neuron, shown in orange, takes input `x` (only `x1` and `x2` are shown for simplicity) and computes output `y`. The entities `w1`, `w2` are the weights of the connections (between inputs and neuron). The weights and inputs are combined following the basic principles of mathematics to produce output `y` (shown in Figures 2, 3 and 4).

![data](../../images/eq1.png "Transformation of a component (x1) of the input vector (x).")

![data](../../images/eq2.png "Transformation of a component (x2) of the input vector (x).")

Weights denote the significance of a particular input to produce the observed output. When it is large, the input is significant and when small, the input is less significant to produce the output. These weights can be initialised randomly and they are modified throughout the learning by a neural network. Using the updated inputs (as shown in the above equations), the output is computed:

![data](../../images/eq3.png "Computation of output y using input x, weight w and activation function f.")

where *f* is an activation function. An [activation function](https://keras.io/activations/) is a mathematical function which translates the combination of inputs to an output. The choices of these functions are many - sigmoid, linear, tanh, ReLU and so on. For example, sigmoid is:

![data](../../images/eq4.png "Sigmoid activation function.")

The above equation will return a real number between 0 and 1.

Rectified exponential linear unit (ReLU) is given by:

![data](../../images/eq5.png "Rectified exponential linear unit (ReLU) activation function.")

As discussed earlier, neurons make the building blocks of a neural network and are arranged in several layers and a usual neural network looks like as shown in Figure 7.

#### Input layer
In the neural network (Figure 7), the input layer is shown in green. This layer receives input data and passes it on to the next layer. The number of neurons in this layer depends on the number of dimensions of input data. For example, if input data (matrix) is of size (500, 10), 500 rows (samples) and 10 columns (features), then the number of neurons in the input layer should be 10. Each neuron in input layer is connected to all neurons in the next layer. All these connections have a separate weight (denoted by `w`).

#### Hidden layer
The next two layers after the input layer are called hidden layers. In the first hidden layer too, all the neurons are connected to all other neurons in the adjacent (hidden) layer. The number of hidden layers determines if the resulting neural network is deep (2 or more hidden layers) or shallow. When the number of hidden layers is 2 or more, the structure or architecture of the neural network is deep and overall learning is called deep learning. More the number of hidden layers, the more complex the architecture is. A complex architecture is beneficial for learning unique patterns from big data. But, complex architecture is prone to [overfitting](https://en.wikipedia.org/wiki/Overfitting) when a neural network starts memorising data without learning unique and general patterns.

![data](../../images/neural_network.svg "A neural network consisting of 4 layers - 1 input, 2 hidden and 1 output. The neurons in each layer are connected to all neurons in the adjacent layer. Each connection between a pair of neurons contains a weight.")

The number of hidden layers and the size of each hidden layer is not fixed as it completely depends on the data. If the dataset is small (say only 1,000 samples), then it is sufficient to choose a less complex architecture (fewer hidden layers) to avoid the danger of overfitting. However, if the dataset is large (say > 100,000 samples), more complex architecture can be chosen. In short, the architecture of the hidden layer is completely dependent on the nature and size of data.

#### Output layer
This layer collects output computed using input data and weights which are optimised during learning. An activation function is chosen to transform the combination of input and weight to an output. Some examples of activation functions have been discussed above.

#### Optimisation
Compted or predicted output, collected at the output layer, and the actual output are compared to find error (or loss). Neural network learning aims is to minimise this error so that the predicted output gets as close to the actual output as possible. This process of minimising error between predicted and actual output is called optimisation. There are several optimisers such as [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent), root mean square propagation ([RMSProp](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#RMSProp)), [adadelta](https://arxiv.org/pdf/1212.5701.pdf) and so on are available. These optimisers work by primarily adjusting the weights of connections so that the error is minimised. Once, a set of weights are achieved which provides the best accuracy or minimum error, the learning is terminated because the weights cannot be updated anymore which can further minimise the error.

#### Neural network training
Training is a process where input data is passed to a neural network at input layer and when finished, a trained model is created containing all the learned parameters such as weights of all connections in the neural network. Usually, a portion of data is extracted and saved as test data which is not used for training. It is used only for evaluating the trained model to get an unbiased estimate of learning and prediction strength. The partitioning of data into training and test can be set by deep learning practitioners. An example of partition can be - 70% training and 30% test data.

#### Batch and Epoch
While training a neural network, input data is passed in small batches. A batch is a subset of training data. An epoch is one iteration when all the training data is used for training in multiple batches. For example, if there is training data of size (500, 10) and batch size is fixed at 50, then there would be 10 batches (50 * 10 = 500) in each epoch. Each batch will have 50 samples and they are passed to the input layer of neural network. The loss computed at the output layer is propagated back and the weights are adjusted. The newly adjusted weights are used for the second batch of samples and so on. When all batches are finished, then one epoch of learning is done. The number of epochs and the size of a batch are parameters to be set by deep learning practitioners. These parameters depend on the size of data and should be tuned according to the data for optimum results.

![data](../../images/mse.png "Mean squared error loss function.")

#### Loss function
The error between the computed and actual output is calculated using a loss function which is necessary to evaluate the strength of learning. Learning is good when loss decreases with training epochs otherwise, training should be stopped and the architecture should be carefully adjusted. There are several choices of loss functions too. Functions such as root mean squared error (RMSE) and absolute error (AE) are used for regression problems while cross-entropy error functions such as binary cross-entropy and categorical cross-entropy are used in classification problems. An example of loss function is shown in Figure 8.

> <question-title></question-title>
>
> 1. What do you understand by an architecture of a neural network?
> 2. How does a neural network learn?
>
> > <solution-title></solution-title>
> >
> > 1. Architecture of a neural network consists of multiple layers such as input, hidden, convolutional, output and their number of respective neurons, optimiser, loss and activation functions etc.
> > 2. The learning happens by minimising the loss between the computed and actual output. The weights of all neuronal connections are adjusted (increased or decreased) to achieve the minimum loss. To ascertain the amount of change for weights, a technique known as backpropagation is used. Using this technique, the loss computed at the output layer is "propagated" back in the neural network (from output to input layer) and each neuronal connection is assigned a share of the total loss. In other words, how much each neuron is contributing to the total accumulated loss. For example, w1 is adjusted according to equation:
> > ![data](../../images/partial_derivative.png "Weight w1 is updated by computing a partial derivative of loss L with respect to weight. The derivative is multiplied with learning rate n.")
> > In the above equation, `L` is the total loss, `w1` is the weight of a connection between an input neuron and a hidden neuron. Similarly, all weights are adjusted and in the subsequent iteration, the updated weights are used to compute loss at the output layer. Parameter `n` is the learning rate which determines how small or big changes are needed in weights. It can either be a fixed quantity or a variable one. In case of a variable learning rate, it usually starts with a large number (say 1.0) and subsequently decays to a small number (say 0.001) as the training epochs proceed because initially a large learning rate helps to reach close to the minimum error quickly and then it is decayed to slow down the learning so that it stabilises at the minimum. [More on backpropagation can be read here](http://neuralnetworksanddeeplearning.com/chap2.html).
> >
> {: .solution}
>
{: .question}

### Relevance of deep learning in Bioinformatics
Deep learning is an established tool in finding patterns in big data for multiple fields of research such as computer vision, image analysis, drug response prediction, protein structure prediction and so on. Different research areas use different architectures of neural network which are suitable to their respective data. For example - in computer vision and image analysis, convolutional neural network (CNN) is popular, graph convolutional neural network is often used for drug response prediction, recurrent neural network is useful for identifying motifs in protein sequences and so on. The table below shows more examples of neural networks which are popular with different fields of bioinformatics. These use-cases of deep learning in bioinformatics prove that it is essential to explore deep learning algorithms to find patterns in big data in biology. More details can be found in [Deep learning in bioinformatics: Introduction, application, and perspective in the big data era](https://www.sciencedirect.com/science/article/pii/S1046202318303256).

![data](../../images/dl_bioinformatics.png "Different architectures of neural networks for different fields of bioinformatics.")

## Get training and test datasets
The datasets used for this tutorial contain gene expression profiles of humans suffering from two types of cancer - [acute myeloid leukemia (AML)](https://en.wikipedia.org/wiki/Acute_myeloid_leukemia) and [acute lymphoblastic leukemia (ALL)](https://en.wikipedia.org/wiki/Acute_lymphoblastic_leukemia). The tutorial aims to differentiate between these two cancer types, predicting a cancer type for each patient, by learning unique patterns in gene expression profiles of patients. The data is divided into 2 parts - one for training and another for prediction. Each part contains two datasets - one has the gene expression profiles and another has labels (the types of cancer). The size of the training data (`X_train`) is (38, 7129) where 38 is the number of patients and 7129 is the number of genes. The label dataset (`y_train`) is of size (38, 1) and contains the information of the type of cancer for each patient (label encoding is 0 for ALL and 1 for AML). The test dataset (`X_test`) is of size (34, 7129) and contains the same genes for 34 different patients. The label dataset for test is `y_test` and is of size (34, 1). The neural network, which will be formulated in the remaining part of the tutorial, learns on the training data and its labels to create a trained model. The prediction ability of this model is evaluated on the test data (which is unseen during training to get an unbiased estimate of prediction ability). These datasets are uploaded to Galaxy by following the steps defined below:

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/record/3706539/files/X_test.tsv
>    https://zenodo.org/record/3706539/files/X_train.tsv
>    https://zenodo.org/record/3706539/files/y_test.tsv
>    https://zenodo.org/record/3706539/files/y_train.tsv
>    ```
>
> 3. Rename the datasets as `X_test`, `X_train`, `y_test` and `y_train` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Check that the datatype is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}


## Neural network architecture
Defining a neural network architecture needs to ascertain the types and number of layers, the number of neurons for each layer, activation functions for all layers, type of optimiser and loss function. Choosing these parameters may require many experiments with data as there is no golden rule to choose the best combination of these parameters. The neural network used in this tutorial has an input layer, 2 hidden layers and one output layer. The input layer has a parameter `input_shape` which is set according to the number of dimensions of data. It is set to (7129,) which is the number of genes present in data. The hidden layers have 16 neurons (units) each and the output layer has only one because a scalar output is expected (0 or 1). This partial architecture (having input shape, types and size of layers, and activation functions) of the neural network is defined as follows:

### Create architecture: Choose layers

> <hands-on-title>Create a deep learning model architecture using Keras</hands-on-title>
>
> 1. **Create a deep learning model architecture using Keras** {% icon tool %} with the following parameters:
>    - *"Select keras model type"*: `Sequential`
>    - *"input_shape"*: `(7129, )`
>       
>    - In *"LAYER"*:
>        - {% icon param-repeat %} *"1: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*: `16`
>                - *"Activation function"*: `elu`
>        - {% icon param-repeat %} *"2: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*: `16`
>                - *"Activation function"*: `elu`
>        - {% icon param-repeat %} *"3: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*: `1`
>                - *"Activation function"*: `sigmoid`
> 
>
{: .hands_on}

The tool returns a JSON output file containing data about the neural network layers and their attributes like their types, number of units they have and their activation functions. This file is used as an input to the next step where the architecture of the neural network is completed by adding optimiser, loss function, and training parameters such as the number of epochs and batch size. The loss function is chosen as `binary_crossentropy` as the learning task is the classification of two labels (0 and 1).

### Create architecture: Add training parameters

> <hands-on-title>Create deep learning model with an optimizer, loss function and fit parameters</hands-on-title>
>
> 1. **Create deep learning model with an optimizer, loss function and fit parameters** {% icon tool %} with the following parameters:
>    - *"Choose a building mode"*: `Build a training model`
>    - *"Select the dataset containing model configurations (JSON)"*: `Keras model config` (output of **Create a deep learning model architecture using Keras** {% icon tool %})
>    - *"Do classification or regression?"*: `KerasGClassifier`
> 
>    `KerasGClassifier` is chosen because the learning task is classfication i.e. assigning each patient a type of cancer.
>    - In *"Compile Parameters"*:
>        - *"Select a loss function"*: `binary_crossentropy`
> 
>        The loss function is `binary_crossentropy` because the labels are discrete and binary (0 and 1).
>        - *"Select an optimizer"*: `RMSprop - RMSProp optimizer`
>    - In *"Fit Parameters"*:
>        - *"epochs"*: `10`
>        - *"batch_size"*: `4`
> 
>        The training data is small (only 38 patients). Therefore the number of epochs and batch size are also small.
>
{: .hands_on}

The tool returns a zipped file containing an object of the neural network architecture (define in the last two steps) which is used as a classifier to train it on data. Once the architecture is finalised, its associated object is used for training combining it with the training data as follows:

### Deep learning training
A neural network is trained on training data to learn hidden representations and mapping from features (genes) to both the types of cancer. As discussed earlier, the neural network minimises the error, which is given by the loss function, between actual and predicted labels while adjusting the weights of connections among neurons in multiple layers. Once the training is finished, the architecture and learned weights are saved. They are used to predict labels in test data. The deep learning training is set up as follows:

> <hands-on-title>Deep learning training and evaluation conduct deep training and evaluation either implicitly or explicitly</hands-on-title>
>
> 1. **Deep learning training and evaluation conduct deep training and evaluation** {% icon tool %} with the following parameters:
>    - *"Select a scheme"*: `Train and validate`
>    - *"Choose the dataset containing pipeline/estimator object"*: `Keras model builder` (output of **Create deep learning model** {% icon tool %})
>    - *"Select input type"*: `tabular data`
>    - *"Training samples dataset"*: `X_train`
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns`
>    - *"Dataset containing class labels or target values"*: `y_train`
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns`
>
{: .hands_on}

The tool gives 3 files as output - a tabular file containing output (accuracy of cross-validation) of training, a zipped file with the trained model (fitted estimator) and an H5 (HDF5) file containing the weights of neural network layers. The files containing the fitted estimator and weights are used to recreate the model and this recreated model is used to predict labels in test data.

### Prediction on test data
After training, the saved architecture (fitted estimator) and weights are used to predict labels for the test data. For each patient in the test data, a type of cancer is predicted using the trained model learned in the previous step.

> <hands-on-title>Model Prediction predicts on new data using a preffited model</hands-on-title>
> 
> 1. **Model Prediction predicts on new data using a preffited model** {% icon tool %} with the following parameters:
>    - *"Choose the dataset containing pipeline/estimator object"*: `Fitted estimator or estimator skeleton` (output of **Deep learning training and evaluation** {% icon tool %})
>    - *"Choose the dataset containing weights for the estimator above"*: `Weights trained` (output of **Create deep learning model** {% icon tool %})
>    - *"Select invocation method"*: `predict`
>    - *"Select input data type for prediction"*: `tabular data`
>        - *"Training samples dataset"*: `X_test`
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns`
> 
>
{: .hands_on}

The tool returns the predicted labels (0 for ALL and 1 AML) for test data in a tabular format. The size of this data is (34,1) where 34 is the number of cancer patients in test data.

## Visualisation
Visualising the results is important to ascertain the generalisation ability of the trained model on an unseen dataset. Using a dataset with the actual labels for the test data, the performance of the trained model is estimated by comparing the actual labels against the predicted labels using a confusion matrix plot.

> <hands-on-title>Machine Learning Visualization Extension includes several types of plotting for machine learning</hands-on-title>
> 
> 1. **Machine Learning Visualization Extension includes several types of plotting for machine learning** {% icon tool %} with the following parameters:
>    - *"Select a plotting type"*: `Confusion matrix for classes` 
>    - *"Select dataset containing true labels"*: `y_test`
>    - *"Does the dataset contain header"*: `Yes`
>    - *"Choose how to select data by column"*: `All columns`
>    - *"Select dataset containing predicted labels"*: `Model prediction` (output of **Model Prediction predicts on new data using a preffited model** {% icon tool %})
>    - *"Does the dataset contain header"*: `Yes`
> 
>
{: .hands_on}

> <comment-title></comment-title>
> Please note that your predictions could be different from the plot shown in Figure 11 because the training data is small and the predictions may vary. Stability in predictions can be achieved if the deep learning model is trained on large data. But, for this tutorial, it is kept small to reduce the training time as the aim is to showcase how to create a pipeline for deep learning training. Generally, deep learning models are trained on large data and may keep running for a few hours to a few days.
{: .comment}

The image below shows [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) which is a square matrix. It contains actual labels on the y-axis and predicted labels on the x-axis. Each cell in the matrix plot gives the number of cancer patients who got predicted correctly or incorrectly. For example, the number in the top-left cell (0, 0) denotes how many of these patients are predicted correctly for ALL (17/20). The higher the number in this cell, the better is the model for this cell. In the top-right cell, 3 patients who have ALL but they are predicted having AML. Similarly, the bottom-right cell denotes how many patients are predicted correctly for AML (10/14). In the bottom-left cell, 4 patients have AML but are predicted as ALL.

![data](../../images/confusion_matrix_dl.png "Confusion matrix for true and predicted classes")


## Conclusion
The tutorial presents a case-study to predict labels (ALL and AML) of 34 new cancer patients after learning gene expression profiles of 38 cancer patients through multiple steps of a deep learning pipeline. All these steps show how to create a neural network architecture using Galaxy's deep learning tools and analyse results using a confusion matrix visualisation. Similarly, multiple different architectures of neural networks can be created well-suited to datasets and aim of particular experiments. Moreover, it should be noted that one architecture of neural network giving promising results on a dataset may not work at all with another dataset. It is essential to perform multiple experiments with a dataset to formulate an optimal neural network architecture.


topic: statistics
tutorial: flexynesis_survival
title: Identifing Survival Markers of Brain tumor with Flexynesis
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 730e94de954629c1258d48c8d6278cf8
last_modified: 2026-02-26T21:12:18.180746
zenodo_link: https://zenodo.org/records/16287482
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/flexynesis_survival/tutorial.html

content:
Here, we use Flexynesis tool suite on a multi-omics dataset of 506 Brain Lower Grade Glioma (LGG) and 288 Glioblastoma Multiforme (GBM) samples with matching mutation and copy number alteration. This data were downloaded from the [cBioPortal](https://www.cbioportal.org) ({% cite cbioportal-website %}). The data was split into train (70% of the samples) and test (30% of the samples).

We will work on CNA (copy number alteration) and MUT (mutation data which is a binary matrix of genexsample) data.

This training is inspired from the original flexynesis analysis notebook: [survival_subtypes_LGG_GBM.ipynb](https://github.com/BIMSBbioinfo/flexynesis/blob/main/examples/tutorials/survival_subtypes_LGG_GBM.ipynb).

> <comment-title> Getting data from cBioPortal </comment-title>
>
> If you want to download data directly from cBioPortal, you can go through "**Prepare data from CbioPortal for Flexynesis integration**" training
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <warning-title>LICENSE</warning-title>
>
>Flexynesis is only available for NON-COMMERCIAL use. Permission is only granted for academic, research, and educational purposes. Before using, be sure to review, agree, and comply with the license.
>For commercial use, please review the flexynesis license on GitHub and contact the [copyright holders](https://github.com/BIMSBbioinfo/flexynesis)
{: .warning}

# Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```
>    https://zenodo.org/records/16287482/files/train_mut_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/train_clin_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/test_cna_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/test_mut_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/test_clin_lgggbm.tabular
>    https://zenodo.org/records/16287482/files/train_cna_lgggbm.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each database a tag corresponding to their modality (#mut, #cna, and #clinical)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Train Flexynesis model with a survival task

Now it is time import train and test datasets. We rank genes by Laplacian Scores and pick top 10% of the genes, while removing highly redundant genes with a correlation score threshold of 0.8 and a variance threshold of 50%. We will also use the intermediate fusion of omic layers.
For hyperparameter optimization we use the following parameters.


*`Model class`: We pick DirectPred (a fully connected network)
* `Column name in the train clinical data to use as survival event` and `Column name in the train clinical data to use as survival time`. Survival status (consists of 0s and 1s) and time since last followup. It is **important** that the clinical data contains both variables as ***numerical*** values.
* `Column name in the train clinical data to use for predictions, multiple targets are allowed`: We can concurrently train the same network to be able to predict other variables such as histological diagnosis, however, here we just focus on the survival endpoints, so we pass an empty list.
* `Number of iterations for hyperparameter optimization.`: We do 5 iterations of hyperparameter optimization. This is a reasonable number of demonstration purposes, but it could be beneficial to increase this value in order to discover even better models.
* `How many epochs to wait when no improvements in validation loss are observed. `: If a training does not show any signs of improving the performance on the validation part of the train_dataset for at least 10 epochs, we stop the training. This not only significantly decreases the amount spent on training by avoiding unnecessary continuation of unpromising training runs, but also helps avoid over-fitting the network on the training data.


> <hands-on-title> Survival task </hands-on-title>
>
> 1. {% tool [Flexynesis](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Type of Analysis"*: `Supervised training`
>        - {% icon param-file %} *"Training clinical data"*: `train_clin_lgggbm.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test_clin_lgggbm.tabular`
>        - {% icon param-file %} *"Training omics data"*: `train_mut_lgggbm.tabular`
>        - {% icon param-file %} *"Test omics data"*: `test_mut_lgggbm.tabular`
>        - *"What type of assay is your input?"*: `mut`
>        - In *"Multiple omics layers?"*:
>            - {% icon param-repeat %} *"Insert Multiple omics layers?"*
>                - {% icon param-file %} *"Training omics data"*: `train_cna_lgggbm.tabular`
>                - {% icon param-file %} *"Test omics data"*: `test_cna_lgggbm.tabular`
>                - *"What type of assay is your input?"*: `cna`
>        - *"Model class"*: `DirectPred`
>        - *"Column name in the train clinical data to use as survival event"*: `c8`
>        - *"Column name in the train clinical data to use as survival time"*: `c7`
>        - In *"Advanced Options"*:
>            - *"Fusion method"*: `intermediate`
>            - *"Variance threshold (as percentile) to drop low variance features."*: `0.5`
>            - *"Correlation threshold to drop highly redundant features."*: `0.8`
>            - *"Minimum number of features to retain after feature selection."*: `1000`
>            - *"Top percentile features (among the features remaining after variance filtering and data cleanup) to retain after feature selection."*: `10.0`
>            - *"Number of iterations for hyperparameter optimization."*: `5`
>        - In *"Visualization"*:
>            - *"Generate embeddings plot?"*: `No`
>            - *"Generate kaplan meier curves plot?"*: `Yes`
>            - *"Generate hazard ratio plot?"*: `Yes`
>                - *"Omics layer to use for cox input"*: `mut`
>                - *"Number of top important features to include in Cox model"*: `5`
>                - *"Performs K-fold cross-validation?"*: `No`
>            - *"Generate scatter plot?"*: `No`
>            - *"Generate concordance heatmap plot?"*: `No`
>            - *"Generate precision-recall curves plot?"*: `No`
>            - *"Generate ROC curves plot?"*: `No`
>            - *"Generate boxplot?"*: `No`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What are main outputs of Flexynesis?
> 2. What does the generated plots show?
> 3. What information is in `job.stats` output?
> 4. What are top 10 features in `job.feature_importance.GradientShap`?
>
> > <solution-title></solution-title>
> >
> > 1. The results collection contains the following data:
> > * job.embeddings_test (latent space of test data)
> > * job.embeddings_train (latent space of train data)
> > * job.feature_importance.GradientShap (feature importance calculated by GradientShap method)
> > * job.feature_importance.IntegratedGradients (feature importance calculated by IntegratedGradients method)
> > * job.feature_logs.cna
> > * job.feature_logs.mut
> > * job.predicted_labels (Prediction of the created model)
> > * job.stats
> >
> > 2. There are two figures in plot collection:
> > * The first plot is the Kaplan Meier Curves of the risk subtypes.
> > * The second plot is a forest plot of calculated Cox-PH model of top 5 markers.
> >
> > 3. It shows that we achieved a reasonable performance of Harrell's Concordance Index (C-index) on test data.
> >
> > 4. *IDH2*, *IDH1*, *HIVEP3*, *PIK3CA*, *EGFR*, *RELN*, *ATRX*, *INSRR*, *TP53*, *COL6A3*. *IDH1*, *TP53*, and *ATRX* are extensively studied and been shown to be relevant in gliomas ({% cite Koschmann2016 %}).
> >
> {: .solution}
>
{: .question}

We can also visualize the high and low risk groups with the embeddings.

# Survival-risk subtypes

## Extract predicted labels and calculate median

Let's group the samples by predicted survival risk scores into 2 groups and visualize the sample embeddings colored by risk subtypes.

> <hands-on-title> Calculate median of predicted survival score</hands-on-title>
>
> First we should extract the `job.predicted_labels` and then calculate the median.
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.predicted_labels`
>
> 2. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `job.predicted_labels` (output of **Extract dataset** {% icon tool %})
>    - *"Input file has a header line"*: `Yes`
>    - In *"Operation to perform on each group"*:
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Median`
>            - *"On column"*: `Column: 6` (*"predicted_label)
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the median?
>
> > <solution-title></solution-title>
> >
> > 1. -0.66496020555496
> >
> {: .solution}
>
{: .question}

Now we add a column to our `job.predicted_labels` data indicating high and low risk groups (samples with predicted_label > median are high risk groups)

## Assign high and low risk groups

> <hands-on-title> Assign high/low groups </hands-on-title>
>
> 1. {% tool [Compute](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `job.predicted_labels` (output of **Extract dataset** {% icon tool %})
>    - *"Input has a header line with column names?"*: `Yes`
>        - In *"Expressions"*:
>            - {% icon param-repeat %} *"Insert Expressions"*
>                - *"Add expression"*: `float(c6)>=-0.66496020555496`
>                - *"The new column name"*: `risk_groups`
>    - In *"Error handling"*:
>        - *"Autodetect column types"*: `No`
>
>
{: .hands_on}

For better visualization, let's change True to High_risk and False to Low_risk

> <hands-on-title> Rename labels </hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `table` (output of **Compute** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `Column: 9`
>            - *"Find pattern"*: `True`
>            - *"Replace with"*: `High_risk`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `Column: 9`
>            - *"Find pattern"*: `False`
>            - *"Replace with"*: `Low_risk`
>
{: .hands_on}

Now we visualize the embedding.

> <hands-on-title> Dimension reduction plot </hands-on-title>
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `Select by element identifier`
>        - *"Element identifier:"*: `job.embeddings_test`
>
> 2. {% tool [Flexynesis plot](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis plot"*: `Dimensionality reduction`
>        - {% icon param-file %} *"Predicted labels"*: `table` (output of **Replace Text** {% icon tool %})
>        - {% icon param-file %} *"Embeddings"*: `job.embeddings_test` (output of **Extract dataset** {% icon tool %})
>        - *"Column in the labels file to use for coloring the points in the plot"*: `Column: 9`
>
{: .hands_on}

# Conclusion

In this tutorial, we have successfully applied the Flexynesis framework to analyze survival markers in brain tumors using multi-modal genomic data from 506 Lower Grade Glioma (LGG) and 288 Glioblastoma Multiforme (GBM) samples. Through hands-on analysis of mutation and copy number alteration data, we have separated our samples to high and low risk groups and ranked the genes by their importance from which some are already known to affect survival in glioma.


topic: statistics
tutorial: fruit_360
title: Image classification in Galaxy with fruit 360 dataset
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 12673ed38f2b5dac95f6888ce165037d
last_modified: 2026-02-26T21:12:18.182430
zenodo_link: https://zenodo.org/record/5702887
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/fruit_360/tutorial.html

content:
The classification of fruits and vegetables offers many useful applications such as
automated harvesting by robots, building up stocks for supermarkets, effective detection
of specific defects, and determining fruit ripeness ({% cite Duong2020 %},
{% cite NaranjoTorres2020 %}, {% cite Iswari2017 %}). Machine Learning (ML) techniques
such as Deep Learning (DL) are commonly used for image classification problems in various
domains, including in agriculture ({% cite Kamilaris2018 %}). DL is a technique inspired
by how a human brain operates. Due to the increased availability of compute capacity and
training data, DL techniques have become very popular in recent years. In this tutorial,
we will use Galaxy's ML toolkit to build a DL model to classify fruit and vegetable
images. Our DL model is trained and evaluated on Fruit 360 dataset ({% cite Murean2018 %})

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}
## Overview of convolutional neural networks (CNN)

Here we provide a brief overview of CNN. For a more in depth discussion, please refer to the CNN tutorial cited in the requirements
section. CNN were inspired by how the visual cortex of the brain processes visual information ({% cite HubelWiesel %}). There are
two types of cells in our visual cortex: **simple cells** detect objects at certain angles/locations, and **complex** cells,
which receive inputs from multiple simple cells, and detect movement. In 1980, inspired by hierarchical structure of complex and
simple cells, Fukushima proposed *Neocognitron* ({% cite Fukishima %}), a hierarchical neural network used for handwritten Japanese
character recognition. In 1989, LeCun et. al. ({% cite LeCunEtAl %}) proposed a CNN that could be trained by backpropagation
algorithm. CNN gained immense popularity when they outperformed other models at ImageNet Challenge, a competition in object
classification and detection on hundreds of object categories and millions of images.

## Architecture of CNN

A typical CNN has the following 4 layers ({% cite OSheaEtAl %})

1. Input layer
2. Convolution layer
3. Pooling layer
4. Fully connected layer

Please note that we will explain a 2 dimensional (2D) CNN here. But the same concepts apply to a 1 (or 3) dimensional CNN as well.

### Input layer

The input layer represents the input to the CNN. An example input, could be a 28 by 28 pixel grayscale image. We do not
"flatten" the input to a 1D vector. This makes capturing spatial relationships easier.

### Convolution layer

The convolution layer is composed of multiple **filters** (also called **kernels**). Filters for a 2D image are also 2D. Suppose
we have a 28 by 28 pixel grayscale image. Each pixel is represented by a number between 0 and 255, where 0 represents the color
black, 255 represents the color white, and the values in between represent different shades of gray. Suppose we have a 3 by 3
filter (9 values in total), and the values are randomly set to 0 or 1. Convolution is the process of placing the 3 by 3 filter
on the top left corner of the image, multiplying filter values by the pixel values and adding the results, moving the filter to
the right one pixel at a time and repeating this process (Figure 1). When we get to the top right corner of the image, we simply
move the filter down one pixel and restart from the left. This process ends when we get to the bottom right corner of the image.

![A 3 by 3 filter applied to a 4 by 4 image, resulting in a 2 by 2 image](../../images/Conv_no_padding_no_strides.gif "A 3 by 3 filter applied to a 4 by 4 image, resulting in a 2 by 2 image ({% cite DumoulinVisin %})")

Covolution operator has several parameters.

1. Filter size
2. Padding
3. Stride
4. Dilation
5. Activation function

Filter size can be 5 by 5, 3 by 3, and so on. Larger filter sizes should be avoided as more weights need to be learned (more
compute capacity, more training time, more chance of overfitting). Also, odd sized filters are preferred to even sized filters,
due to the nice geometric property of all the input pixels being around the output pixel.

If you look at Figure 1 you see that after applying a 3 by 3 filter to a 4 by 4 image, we end up with a 2 by 2 image -- the
size of the image has gone down. If we want to keep the image size the same, we can use *padding* (Figure 2). We pad the input
in every direction with 0's before applying the filter. If the padding is 1 by 1, then we add 1 zero in evey direction. If its
2 by 2, then we add 2 zeros in every direction, and so on.

![A 3 by 3 filter applied to a 5 by 5 image, with padding of 1, resulting in a 5 by 5 image](../../images/Conv_same_padding_no_strides.gif "A 3 by 3 filter applied to a 5 by 5 image, with padding of 1, resulting in a 5 by 5 image ({% cite DumoulinVisin %})")

As mentioned before, we start the convolution by placing the filter on the top left corner of the image, and after multiplying
filter and image values (and adding them), we move the filter to the right and repeat the process. How many pixels we move to
the right (or down) is the *stride*. In figure 1 and 2, the stride of the filter is 1. We move the filter one pixel to the right
(or down). But we could use a different stride. Figure 3 shows an example of using stride of 2.

![A 3 by 3 filter applied to a 5 by 5 image, with stride of 2, resulting in a 2 by 2 image](../../images/Conv_no_padding_strides.gif "A 3 by 3 filter applied to a 5 by 5 image, with stride of 2, resulting in a 2 by 2 image ({% cite DumoulinVisin %})")

When we apply a, say 3 by 3, filter to an image, our filter's output is affected by pixels in a 3 by 3 subset of the image. If we
like to have a larger *receptive field* (portion of image that affect filter's output), we could use *dilation*. If we set the
dilation to 2 (Figure 4), instead of a contiguous 3 by 3 subset of the image, every other pixel of a 5 by 5 subset of the image
affects the filter's output.

![A 3 by 3 filter applied to a 7 by 7 image, with dilation of 2, resulting in a 3 by 3 image](../../images/Conv_dilation.gif "A 3 by 3 filter applied to a 7 by 7 image, with dilation of 2, resulting in a 3 by 3 image ({% cite DumoulinVisin %})")

After the filter scans the whole image, we apply an activation function to filter output to introduce non-linearlity. The preferred
activation function used in CNN is ReLU ({% cite NwankpaEtAl %}). ReLU leaves pixels with positive values in filter output as is,
and replaces negative values with 0. Figure 5 shows the results of applying ReLU activation function to a filter output.

![Two matrices representing filter output before and after ReLU activation function is applied](../../images/Conv_ReLU.png "Applying ReLU activation function to filter output")

![One matrix representing an input and another matrix representing a filter, along with calculation for single input channel two dimensional convolution operation](../../images/Conv_single_input_channel.png "Illustration of single input channel two dimensional convolution")

Figure 6 illustrates the calculations for a convolution operation, via a 3 by 3 filter on a single channel 5 by 5 input
(5 x 5 x 1). Figure 7 illustrates the calculations when the input has 3 channels. To show this in 2 dimensions, we are
displaying each channel in input and filter separately. Figure 9 shows a sample multi-channel 2D convolution in 3 dimensions.

![Three matrices representing an input and another three matrices representing a filter, along with calculation for multiple input channel two dimensional convolution operation ](../../images/Conv_multiple_input_channel.png "Illustration of multiple input channel two dimensional convolution")

As Figures 7 and 8 show the output of a multi-channel 2D filter is a single channel 2D image. Applying *multiple* filters to the
input image results in a multi-channel 2D image for the output. For example, if the input image is 28 by 28 by 3
(rows x columns x channels), and we apply a 3 by 3 filter with 1 by 1 padding, we would get a 28 by 28 by 1 image. If we apply 15
filters to the input image, our output would be 28 by 28 by 15. Hence, the number of filters in a convolution layer allows us to
increase or decrease the channel size.

![Multiple cubes representing input, filter, and output in a 3 channel 2 dimensional convolution operation](../../images/Conv_multiple_channel_3d.gif "Three dimensional illustration of multiple input channel two dimensional convolution (Source: https://thomelane.github.io/convolutions/2DConvRGB.html)")

### Pooling layer

The pooling layer performs down sampling to reduce the spatial dimensionality of the input. This decreases the number of parameters,
which in turn reduces the learning time and computation, and the likelihood of overfitting. The most popular type of pooling is
*max pooling*. Its usually a 2 by 2 filter with a stride of 2 that returns the maximum value as it slides over the input data,
similar to convolution filters.

### Fully connected layer

The last layer in a CNN is a fully connected layer. We connect all the nodes from the previous layer to this fully connected layer,
which is responsible for classification of the image.

![A convolutional neural network with 3 convolution layers followed by 3 pooling layers](../../images/Conv_CNN.png "A convolutional neural network with 3 convolution layers followed by 3 pooling layers ({% cite OSheaEtAl %})")

As shown in Figure 9, a typical CNN usually has more than one convolution plus pooling layer. Each convolution plus pooling layer
is responsible for feature extraction at a different level of abstraction. For example, the filters in the first layer could detect
horizontal, vertical, and diagonal edges. The filters in the next layer could detect shapes, and the filters in the last layer could
detect collection of shapes. Filter values are randomly initialized and are learned by the learning algorithm. This makes CNN very
powerful as they not only do classification, but can also automatically do feature extraction. This distinguishes CNN from other classification techniques (like Support Vector Machines), which cannot do feature extraction.

## Fruit 360 dataset

Fruit 360 is a dataset with 90380 images of 131 fruits and vegetables
(https://www.kaggle.com/moltean/fruits). Images are 100 pixel by 100 pixel and are color
(RGB) images (Hence, 3 values for each pixel). There are 67,692 images in the training
dataset and 22,688 images in the test dataset. The dataset we use for this tutorial is a
subset of fruit 360 dataset, containing only 10 fruits/vegetables (Strawberry,
Apple_Red_Delicious, Pepper_Green, Corn, Banana, Tomato_1, Potato_White, Pineapple,
Orange, and Peach). We selected a subset of fruits/vegetables, so the dataset size is
smaller and the neural network can be trained faster. Our training dataset has 5,015 images
and our testing dataset has 1,679 images.

The utilities used to create the subset dataset, along with step by step instructions, can
be found here: https://github.com/kxk302/fruit_dataset_utilities. First, we created feature
vectors for each image. Images are 100 pixel by 100 pixel and are color (RGB) images
(3 values for each pixel). Hence, each image can be represented by 30,000 values
(100 X 100 X 3). Second, we selected a subset of 10 fruit/vegetable images. Training and
testing dataset sizes go from 7 GB and 2.5 GB for 131 fruits/vegetables to 500 MB and
177 MB for 10 fruits/vegetables, respectively. Third, we created separate files for feature
vectors and labels. Finally, we mapped the labels for the 10 selected fruits/vegetables to
a range of 0 to 9. Full dataset labels are in the 0 to 130 range, as the full dataset
includes 131 fruits/vegetables. The 10 labels for out dataset are as follows: Strawberry:0,
Apple_Red_Delicious:1, Pepper_Green:2, Corn:3, Banana:4, Tomato_1:5, Potato_White:6, Pineapple:7,
Orange:8, Peach:9.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Make sure you have an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to make it easy to recognize
>
>    > <tip-title>Rename a history</tip-title>
>    >
>    > * Click on the title of the history (by default the title is `Unnamed history`)
>    >
>    >   ![Renaming history]({% link shared/images/rename_history.png %})
>    >
>    > * Type `Galaxy Introduction` as the name
>    > * Press <kbd>Enter</kbd>
>    >
>    {: .tip}
>
>
> 3. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/train_X_10.tsv
>    {{ page.zenodo_link }}/files/train_y_10.tsv
>    {{ page.zenodo_link }}/files/test_X_10.tsv
>    {{ page.zenodo_link }}/files/test_y_10.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Rename the datasets as `train_X_10`, `train_y_10`, `test_X_10`, and `test_y_10` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 5. Check that the datatype of all the three datasets is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

## Classification of fruit 360 dataset images with CNN

In this section, we define a CNN and train it using fruit 360 dataset training data. The
goal is to learn a model such that given an image of a fruit/vegetable, we can predict
what fruit/vegetable it is (Labels are in the range of 0 to 9). We then evaluate the trained
CNN on the test dataset and plot the confusion matrix.

In order to train the CNN, we must have the One-Hot Encoding (OHE) representation of the training
labels. This is needed to calculate the categorical cross entropy loss function. OHE encodes labels
as a **one-hot** numeric array, where only one element is 1 and the rest are 0's. For example, if
we had 3 fruits (apple, orange, banana) and their labels were 1, 2, and 3, the OHE
represntation of apple would be (1,0,0), the OHE representation of orange would be (0,1,0), and the
OHE representation of banana would be (0,0,1). For apple with label 1, the first element of array
is 1 (and the rest are 0's); For Orange with label 2, the second element of the array is 1 (and the
rest are 0's); And for Banana with label 3, the third element of the array is 1 (and the rest are 0's).
We have 10 fruits/vegetables in our dataset and we would just have an array of size 10, where only one
element is 1, corresponding to fruit/vegetable label, and the rest are 0's.


In order to calculate the OHE of labels, we must first extract the labels column from train_y_10 file.
train_y_10 file has 3 columns: Label_name (string representation of label), file_name (name of fruit/vegetable
image file), and Label (integer representation of label). We extract Label from train_y_10, and then calcuate
its OHE representation.

### Extract the Label column from train_y_10

> <hands-on-title>Advanced Cut</hands-on-title>
>
> - {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %}
>    - *"File to cut"* : Select `train_y_10`
>    - *"Operation"* : Select `Keep`
>    - *"Delimited by"*: Select `Tab`
>    - *"Cut by"*: Select `fields`
>    - *"List of fields"*: Select `Column: 3`
>    - Click *"Run Tool"*
>
{: .hands_on}

### Create One-Hot Encoding (OHE) representation of training labels

> <hands-on-title>One-Hot Encoding</hands-on-title>
>
> - {% tool [To categorical](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_to_categorical/sklearn_to_categorical/1.0.8.3) %}
>    - *"Input file"* : Select the output of the previous step.
>    - *"Does the dataset contain header?"* : Select `Yes`
>    - *"Total number of classes"*: Select `10`
>    - Click *"Run Tool"*
>
{: .hands_on}

### Create a deep learning model architecture

> <hands-on-title>Model config</hands-on-title>
>
> - {% tool [Create a deep learning model architecture](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/0.5.0) %}
>    - *"Select keras model type"*: `sequential`
>    - *"input_shape"*: `(30000,)`
>    - In *"LAYER"*:
>        - {% icon param-repeat %} *"1: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Reshape`
>                - *"target_shape"*: `(100,100,3)`
>        - {% icon param-repeat %} *"2: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `16`
>                - *"kernel_size"*: `5`
>                - *"Activation function"*: `relu`
>                - *"Type in key words arguments if different from the default"*: `input_shape=(100, 100, 3)`
>        - {% icon param-repeat %} *"3: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"4: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `32`
>                - *"kernel_size"*: `5`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"5: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"6: LAYER"*:
>            - *"Choose the type of layer"*: `Convolutional -- Conv2D`
>                - *"filters"*: `64`
>                - *"kernel_size"*: `5`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"7: LAYER"*:
>            - *"Choose the type of layer"*: `Pooling -- MaxPooling2D`
>                - *"pool_size"*: `(2,2)`
>        - {% icon param-repeat %} *"8: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Flatten`
>        - {% icon param-repeat %} *"9: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `256`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"10: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `10`
>                - *"Activation function"*: `softmax`
>    - Click *"Run Tool"*
{: .hands_on}

Each image is passed in as a vector of size 30,000 (100 x 100 X 3 = 30,000). The reshape
layer reshapes it into (100, 100, 3) dimensions -- 100 rows (image height), 100 columns
(image width), and 3 channels. Channel size is 3 since the image is color (RGB) and each
color pixel can be represented by 3 integers, representing the Red, Green, and Blue
primary colors. Our CNN then has 3 convolution + pooling layers. The first convolution layer
has 16 filters (output channel size would be 16), and filter size is 5 x 5. The second convolutional
layer has 32 filters (output channel size would be 32), and filter size is 5 x 5. The third
convolutional layer has 64 filters (output channel size would be 64), and filter size is 5 x 5. All
3 pooling layers are MaxPool layers with pool size of 2 x 2. Afterwards, we flatten the previous layer's
output (every row/column/channel would be an individual node), then add a fully connected layer with 256
nodes and relu activation function. Finally, we add a fully connected layers with 10 nodes, and use
softmax activation function to get the probability of each fruit/vegetable. Fruit/vegetable with the
highest probability is predicted by CNN. The model config can be downloaded as a JSON file.

### Create a deep learning model

> <hands-on-title>Model builder (Optimizer, loss function, and fit parameters)</hands-on-title>
>
> - {% tool [Create deep learning model](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/0.5.0) %}
>    - *"Choose a building mode"*: `Build a training model`
>    - *"Select the dataset containing model configuration"*: Select the *Keras Model Config* from the previous step.
>    - *"Do classification or regression?"*: `KerasGClassifier`
>    - In *"Compile Parameters"*:
>        - *"Select a loss function"*: `categorical_crossentropy`
>        - *"Select an optimizer"*: `Adam - Adam optimizer `
>        - *"Select metrics"*: `acc/accuracy`
>    - In *"Fit Parameters"*:
>        - *"epochs"*: `40`
>        - *"batch_size"*: `50`
>    - Click *"Run Tool"*
{: .hands_on}

A loss function measures how different the predicted output is from the expected output. For multi-class classification problems,
we use *categorical cross entropy* as loss function. Epochs is the number of times the whole training data is used to train the
model. Setting *epochs* to 40 means each training example in our dataset is used 40 times to train our model. If we update network
weights/biases after all the training data is feed to the network, the training will be very slow (as we have 5014 training examples
in our dataset). To speed up the training, we present only a subset of the training examples to the network, after which we update
the weights/biases. *batch_size* decides the size of this subset. The model builder can be downloaded as a zip file.

### Deep learning training and evaluation

> <hands-on-title>Training the model</hands-on-title>
>
> - {% tool [Deep learning training and evaluation](toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.8.2) %}
>    - *"Select a scheme"*: `Train and Validate`
>    - *"Choose the dataset containing pipeline/estimator object"*: Select the *Keras Model Builder* from the previous step.
>    - *"Select input type:"*: `tabular data`
>        - *"Training samples dataset"*: Select `train_X_10` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>        - *"Dataset containing class labels or target values"*: Select the OHE representation of `train_y_10` dataset
>        - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
>
{: .hands_on}

The training step generates 3 datasets. 1) accuracy of the trained model, 2) the trained model, downloadable as a zip file, and
3) the trained model weights, downloadable as an hdf5 file. These files are needed for prediction in the next step.

### Model Prediction

> <hands-on-title>Testing the model</hands-on-title>
>
> - {% tool [Model Prediction](toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.8.2) %}
>    - *"Choose the dataset containing pipeline/estimator object"* : Select the trained model from the previous step.
>    - *"Choose the dataset containing weights for the estimator above"* : Select the trained model weights from the previous step.
>    - *"Select invocation method"*: `predict`
>    - *"Select input data type for prediction"*: `tabular data`
>    - *"Training samples dataset"*: Select `test_X_10` dataset
>    - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
{: .hands_on}

The prediction step generates 1 dataset. It's a file that has predictions (0 to 9 for the predicted fruit/vegetable) for every image
in the test dataset.

### Machine Learning Visualization Extension

> <hands-on-title>Creating the confusion matrix</hands-on-title>
>
> - {% tool [Machine Learning Visualization Extension](toolshed.g2.bx.psu.edu/repos/bgruening/ml_visualization_ex/ml_visualization_ex/1.0.8.2) %}
>    - *"Select a plotting type"*: `Confusion matrix for classes`
>    - *"Select dataset containing the true labels"*": `test_y_10`
>    - *"Does the dataset contain header:"*: `Yes`
>    - *"Choose how to select data by column:"*: `Select columns by column header name(s)`
>    	- *"Type header name(s):"*: `Label`
>    - *"Select dataset containing the predicted labels"*": Select `Model Prediction` from the previous step
>    - *"Does the dataset contain header:"*: `Yes`
>    - Click *"Run Tool"*
>
{: .hands_on}

**Confusion Matrix** is a table that describes the performance of a classification model. It lists the number of examples that were
correctly classified by the model, True positives (TP) and true negatives (TN). It also lists the number of examples that were
classified as positive that were actually negative (False positive, FP, or Type I error), and the number of examples that were
classified as negative that were actually positive (False negative, FN, or Type 2 error). Given the confusion matrix, we can
calculate **precision** and **recall** {% cite TatbulEtAl  %}. Precision is the fraction of predicted positives that are true
positives (Precision = TP / (TP + FP)). Recall is the fraction of true positives that are predicted (Recall = TP / (TP + FN)).
One way to describe the confusion matrix with just one value is to use the **F score**, which is the harmonic mean of precision
and recall

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} $$

![Confusion matrix for fruit 360 image classification problem](../../images/Fruit_confusion_matrix.png "Fruits/vegetables image classification confusion matrix")

Figure 1 is the resultant confusion matrix for our image classification problem. The first row in the table represents the *true*
fruit/vegetable with 0 as class label, which is strawberry (we have 164 strawberry images with 0 as class label). The second row
represents the *true* fruit/vegetable with 1 as class label, which is Apple_Red_Delicious (We have 166 Apple_Red_Delicious images
with 1 as class label). Similarly, you can count the true class labels for fruits/vegetables with class label of 2 to 9 by adding
up the numbers in the corresponding row. The first column from the left represents the *predicted* fruit/vegetable with 0 as class
label, which is strawberry (Our CNN predicted 164 images as being strawberry, and having class label 0). The second column from
the left represents the *predicted* fruit/vegetable with 1 as class label, which is Apple_Red_Delicious (Our CNN predicted 166
images as being Apple_Red_Delicious, and having class label 1). Similarly, you can count the predicted class labels for
fruits/vegetables with class labels 2 to 9 by adding up the numbers in the corresponding column.

For label 3, which is corn, looking at the green cell in the 4th row and 4th column, we see that our CNN has correctly
predicted 118 images as being a corn image (True positives). Adding the numbers in the other rows in column 4, we see that
our CNN has incorrectly predicted 16 images as being corn (False positives). Adding the numbers on the 4th row besides the
True positives, we see that our CNN has incorrectly predicted 32 corn images as being label Potato_White (False negatives).
Given these numbers we can calculate Precision, Recall, and the F score for digit 0 as follows:

$$ Precision = \frac{\text{True positives}}{\text{True positives + False positives}} = \frac{118}{118 + 16} = 0.88 $$

$$ Recall = \frac{\text{True positives}}{\text{True positives + False negatives}} = \frac{118}{118 + 32} = 0.78 $$

$$ F score = \frac{2 * \text{Precision * Recall}}{\text{Precision + Recall}} = \frac{2 * 0.88 * 0.78}{0.88 + 0.78} = 0.82 $$

You can calculate the Precision, Recall, and F score for other digits in a similar manner.

# Conclusion


In this tutorial, we briefly described convolutional neural networks (CNN) and their application to image classification problems.
We then used Galaxy's ML toolkit to solve an image classification problem using CNN on fruit 360 dataset.


topic: statistics
tutorial: genomic-llm-finetuning
title: Fine-tuning a LLM for DNA Sequence Classification
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml", "Large Language Model"]
content_hash: 976ff7259b55d04122771cc66ef21c92
last_modified: 2026-02-26T21:12:18.195393
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/genomic-llm-finetuning/tutorial.html

content:
After preparing, training, and utilizing a language model for DNA sequences, we can now fine-tune a pre-trained Large Language Model (LLM) for specific DNA sequence classification tasks. Here, we will use a pre-trained model from Hugging Face, specifically the [Mistral-DNA-v1-17M-hg38](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-17M-hg38), and adapt it to classify DNA sequences based on their biological functions. Our objective is to classify sequences according to whether they bind to transcription factors.

> <comment-title>Transcription factors</comment-title>
>
> Transcription factors are proteins that play a crucial role in regulating gene expression by binding to specific DNA sequences, known as enhancers or promoters. These proteins act as molecular switches, turning genes on or off in response to various cellular signals and environmental cues. By binding to DNA, transcription factors either promote or inhibit the recruitment of RNA polymerase, the enzyme responsible for transcribing DNA into RNA, thereby influencing the rate of transcription.
> 
> ![Diagram illustrating DNA binding with CTCF. The left panel, outlined in red, shows a DNA sequence 'CCACCAGGGGGCGC' labeled as 'DNA binding CTCF,' with an oval labeled 'CTCF' above it. The right panel, outlined in blue, shows a different DNA sequence 'GTGGCTAGTAGGTAG' labeled as 'DNA not binding CTCF,' indicating that this sequence does not interact with CTCF.](images/two_dna_sequences.png "Two types of DNA sequences. On the left, a DNA sequence that binds the transcription factor CTCF. On the right, a DNA sequence that does not bind CTCF.")
>
> Transcription factors are essential for numerous biological processes, including cell differentiation, development, and response to external stimuli. Their ability to recognize and bind specific DNA sequences allows them to orchestrate complex gene expression programs, ensuring that the right genes are expressed at the right time and in the right place within an organism. Understanding the function and regulation of transcription factors is vital for deciphering the molecular mechanisms underlying health and disease, and it opens avenues for developing targeted therapeutic interventions.
>
{: .comment}

This classification task is crucial for understanding gene regulation, as transcription factors play a vital role in controlling which genes are expressed in a cell. By training a model to predict whether a DNA sequence binds to a transcription factor, we can gain insights into regulatory mechanisms and potentially identify novel binding sites or understand the impact of genetic variations on transcription factor binding.

By fine-tuning the model, we aim to leverage its pre-trained knowledge of DNA sequences to achieve high accuracy in this classification task. This tutorial will guide you through the necessary steps, from data preparation to model evaluation, ensuring you can apply these techniques to your own research or projects.

We will use [`Mistral-DNA-v1-17M-hg38`](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-1M-hg38), a mixed model that was pre-trained on the entire Human Genome. It contains approximately 17 million parameters and was trained using the Human Genome assembly GRCh38 on sequences of 10,000 bases (10K):

```python
model_name="RaphaelMourad/Mistral-DNA-v1-17M-hg38"
```

> <comment-title>Pretraining a LLM</comment-title>
>
> To learn how to pretrain a LLM on DNA, please follow the dedicated ["Pretraining a Large Language Model (LLM) from Scratch on DNA Sequences"]({% link topics/statistics/tutorials/genomic-llm-pretraining/tutorial.md %}) tutorial
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare resources

## Install dependencies

The first step is to install the required dependencies:

```python
!pip install accelerate==1.1.0
!pip install peft==0.13.2
!pip install torch==2.5.0
!pip install transformers -U
!pip install progressbar
!pip install bitsandbytes
```

> <question-title></question-title>
>
> 1. What is `accelerate`?
> 2. What is `peft`?
> 3. What is `torch`?
> 4. What is `transformers`?
>
> > <solution-title></solution-title>
> >
> > 1. `accelerate` is a library by [Hugging Face](https://huggingface.co/) -- a platform that provides tools and resources for building, training, and deploying machine learning models -- designed to simplify the process of training and deploying machine learning models across different hardware environments. It provides tools to optimize performance on GPUs, TPUs, and other accelerators, making it easier to scale models efficiently.
> >
> > 2. The PEFT (Parameter-Efficient Fine-Tuning) Python library, developed by Hugging Face, is a tool designed to efficiently adapt large pretrained models to various downstream tasks without the need to fine-tune all of the model's parameters. By focusing on a small subset of parameters, PEFT significantly reduces computational and storage costs, making it feasible to fine-tune large language models (LLMs) on consumer-grade hardware. The library integrates seamlessly with the Hugging Face ecosystem, including Transformers, Diffusers, and Accelerate, enabling streamlined model training and inference. PEFT supports techniques like LoRA (Low-Rank Adaptation) and prompt tuning, and it can be combined with quantization to further optimize resource usage. Its open-source nature fosters collaboration and accessibility, allowing developers to customize models for specific applications quickly and efficiently.
> > 
> > 3. `torch`, also known as PyTorch, it is an open-source machine learning library developed by Facebook's AI Research lab. It provides a flexible platform for building and training neural networks, with a focus on tensor computations and automatic differentiation.
> > 
> > 4. `transformers` is a library by Hugging Face that provides implementations of state-of-the-art transformer models for natural language processing (NLP). It includes pre-trained models and tools for fine-tuning, making it easier to apply transformers to various NLP tasks.
> >
> {: .solution}
>
{: .question}

## Import Python libraries

Let's now import them.

```python
import os

import accelerate
import flash_attn
import numpy as np
import pandas as pd
import torch
import transformers
from accelerate import FullyShardedDataParallelPlugin, Accelerator
from pathlib import Path
from peft import (
    LoraConfig,
    get_peft_model,
    get_peft_model_state_dict,
    prepare_model_for_kbit_training,
)
from progressbar import ProgressBar
from random import randrange
from torch.utils.data import TensorDataset, DataLoader
from torch.distributed.fsdp.fully_sharded_data_parallel import FullOptimStateDictConfig, FullStateDictConfig
from transformers import (
    AutoTokenizer,
    AutoModel,
    BitsAndBytesConfig,
    EarlyStoppingCallback,
    set_seed,
)
```

> <comment-title>Versions</comment-title>
>
> This tutorial has been tested with following versions:
> - `numpy` = 1.19 (and not 1.2)
> - `transformers` > 4.47.1
> 
> You can check the versions with:
>
> ```python
> np.__version__
> transformers.__version__
> ```
{: .comment}

# Configure fine-tuning

## Check and configure available resources

We select the appropriate device (CUDA-enabled GPU if available) for running PyTorch operations

```python
torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

Let's check the GPU usage and RAM:

```python
!nvidia-smi
```

We now set an environment variable that configures how PyTorch manages CUDA memory allocations

```python
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"
```

## Specify settings for quantization

Quantization is a technique used in machine learning and signal processing to reduce the precision of numerical values, typically to decrease memory usage and computational requirements. This process is particularly useful when working with large models as it allows them to be deployed on hardware with limited resources without significantly sacrificing performance.

Here, we use `BitsAndBytesConfig` to configure a 4-bit quantization. Using 4-bit precision reduces the memory footprint of the model, which is particularly useful for very large models that might not fit into GPU memory otherwise:

```python
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `load_in_4bit=True`
> 2. `bnb_4bit_use_double_quant=True`
> 3. `bnb_4bit_compute_dtype=torch.bfloat16`
>
> > <solution-title></solution-title>
> >
> > 1. `load_in_4bit=True`: Specifies that the model should be loaded with 4-bit quantization. Using 4-bit precision reduces the memory footprint of the model, which is particularly useful for very large models that might not fit into GPU memory otherwise.
> >
> > 2. `bnb_4bit_use_double_quant=True`: enables double quantization, which means that the quantization constants from the first quantization are quantized again. This further reduces the memory footprint, although it may introduce additional computational overhead.
> >
> > 3. `bnb_4bit_compute_dtype=torch.bfloat16`: sets the compute data type to bfloat16 (Brain Floating Point 16-bit format). Using bfloat16 can provide a good balance between computational efficiency and numerical stability, especially on hardware that supports this format, such as certain GPUs and TPUs.
> >
> {: .solution}
>
{: .question}

## Configure Accelerate

Now, we will configure the [Hugging Face Accelerate library](https://huggingface.co/docs/accelerate/en/index) to optimize the training process for large models using Fully Sharded Data Parallel (FSDP). This setup is crucial for efficiently utilizing GPU resources and enabling distributed training across multiple devices.

First, we need to configure the FSDP plugin, which will manage how model parameters and optimizer states are sharded across GPUs. This configuration helps in reducing memory usage and allows for the training of larger models.

```python
fsdp_plugin = FullyShardedDataParallelPlugin(
    state_dict_config=FullStateDictConfig(offload_to_cpu=True, rank0_only=False),
    optim_state_dict_config=FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=False),
)
```
> <question-title></question-title>
>
> What do the parameters?
> 
> 1. `state_dict_config=FullStateDictConfig(offload_to_cpu=True, rank0_only=False)`?
> 2. `optim_state_dict_config=FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=False)`?
>
> > <solution-title></solution-title>
> >
> > 1. `state_dict_config=FullStateDictConfig(offload_to_cpu=True, rank0_only=False)`
> >    - `FullStateDictConfig`: Configures how the model's state dictionary (parameters) is managed.
> >    - `offload_to_cpu=True`: Specifies that the model's parameters should be offloaded to CPU memory when not in use. This helps free up GPU memory, especially useful when working with large models.
> >    - `rank0_only=False`: Indicates that the state dictionary operations (like saving and loading) are not restricted to the rank 0 process. This allows all processes to participate in these operations, which can be beneficial for distributed training setups.
> >
> > 2. `optim_state_dict_config=FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=False)`
> >    - `FullOptimStateDictConfig`: Configures how the optimizer's state dictionary is managed.
> >    - `offload_to_cpu=True`: Similar to the model's state dictionary, this setting offloads the optimizer states to CPU memory when not in use, further reducing GPU memory usage.
> >    - `rank0_only=False`: Allows all processes to handle the optimizer state dictionary operations, ensuring that the optimizer states are managed efficiently across the distributed setup.
> >
> {: .solution}
>
{: .question}

Next, we initialize the Accelerator from the Hugging Face Accelerate library, integrating the FSDP plugin for seamless distributed training:

```python
accelerator = Accelerator(fsdp_plugin=fsdp_plugin)
```

By passing the FSDP plugin to the `Accelerator`, we enable sharded data parallelism, which efficiently manages model and optimizer states across multiple GPUs.

With this configuration, the `Accelerator` will handle the complexities of distributed training, allowing us to focus on developing and experimenting with our models. This setup is particularly beneficial when working with large-scale models and limited GPU resources, as it optimizes memory usage and enables faster training times.

## Configure LoRA for Parameter-Efficient Fine-Tuning

We will configure the LoRA (Low-Rank Adaptation) settings for parameter-efficient fine-tuning of a large language model. LoRA is a technique that allows us to fine-tune only a small number of additional parameters while keeping the original model weights frozen, making it highly efficient for adapting large models to specific tasks.

We use the `LoraConfig` class to define the settings for LoRA. This configuration specifies how the low-rank adaptations are applied to the model.

```python
peft_config = LoraConfig(
    r=16,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type="SEQ_CLS",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]
)
```

> <question-title></question-title>
>
> What do the parameters?
> 
> 1. `r=16`?
> 2. `lora_alpha=16`?
> 3. `lora_dropout=0.05`?
> 4. `bias="none"`?
> 5. `task_type="SEQ_CLS"`?
> 6. `target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]`?
>
> > <solution-title></solution-title>
> >
> > 1. `r=16`: This parameter specifies the rank of the low-rank matrices used in the adaptation. A higher rank allows the model to capture more complex patterns but also increases the number of trainable parameters.
> >
> > 2. `lora_alpha=16`: This scaling factor controls the magnitude of the updates applied by the low-rank matrices. It helps balance the influence of the adaptations relative to the original model weights.
> >
> > 3. `lora_dropout=0.05`: Dropout is applied to the low-rank matrices during training to prevent overfitting. A dropout rate of 0.05 means that 5% of the elements are randomly set to zero during each training step.
> >
> > 4. `bias="none"`: This setting specifies that no bias parameters are added to the low-rank adaptations. Other options include "all" to add biases to all layers or "lora_only" to add biases only to the LoRA layers.
> >
> > 5. `task_type="SEQ_CLS"`: This indicates that the model is being fine-tuned for a sequence classification task. Other task types might include "CAUSAL_LM" for causal language modeling or "SEQ_2_SEQ_LM" for sequence-to-sequence tasks.
> >
> > 6. `target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]`: This list specifies the modules within the model architecture to which the LoRA adaptations will be applied. These modules are typically the attention layers in transformer models:
> >    - `"q_proj"`: query projections
> >    - `"k_proj"`: key projections
> >    - `"v_proj"`: value projections
> >    - `"o_proj"`: output projections
> >    - `"gate_proj"`: gating projections in some architectures.
> >
> {: .solution}
>
{: .question}

By configuring LoRA in this way, we can efficiently adapt a large pretrained model to a specific task with minimal computational overhead, making it feasible to fine-tune on consumer-grade hardware. This approach is particularly useful for tasks like text classification, sentiment analysis, or any other application where we need to specialize a general-purpose language model.

## Configure Training Arguments

Let's now set up the training arguments using the `TrainingArguments` class from the Hugging Face Transformers library. These arguments define the training configuration, including hyperparameters and settings for saving and evaluating the model.

```python
training_args = transformers.TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=1e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    weight_decay=0.01,
    bf16=True,
    report_to="none",
    load_best_model_at_end = True,
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `output_dir="./results"`
> 2. `evaluation_strategy="epoch"`
> 3. `save_strategy="epoch"`
> 4. `learning_rate=1e-5`
> 5. `per_device_train_batch_size=16`
> 6. `per_device_eval_batch_size=16`
> 7. `num_train_epochs=5`
> 8. `weight_decay=0.01`
> 9. `bf16=True`
> 10. `report_to="none"`
> 11. `load_best_model_at_end=True`
> 
> > <solution-title></solution-title>
> >
> > 1. `output_dir="./results"`: Specifies the directory where the model predictions and checkpoints will be saved.
> >
> > 2. `evaluation_strategy="epoch"`: The model will be evaluated at the end of each epoch. This allows for monitoring the model's progress and adjusting the training process as needed.
> >
> > 3. `save_strategy="epoch"`: The model checkpoints will be saved at the end of each epoch.  This ensures that checkpoints are available for each complete pass through the dataset.
> >
> > 4. `learning_rate=1e-5`: Sets the initial learning rate for the optimizer. This rate determines how much the model's weights are updated during training.
> >
> > 5. `per_device_train_batch_size=16`: The number of samples per device (e.g., GPU) to load for training. 
> >
> > 6. `per_device_eval_batch_size=16`: The number of samples per device to load for evaluation.
> >
> > 7. `num_train_epochs=5`: The total number of training epochs. An epoch is one complete pass through the training dataset.
> >
> > 8. `weight_decay=0.01`: Applies L2 regularization to the model weights to prevent overfitting.
> >
> > 9. `bf16=True`: Enables mixed precision training using bfloat16, which can speed up training and reduce memory usage on compatible hardware.
> >
> > 10. `report_to="none"`: Disables reporting to external services like WandB or TensorBoard. If you want to track metrics, you can set this to "wandb", "tensorboard", etc.
> >
> > 11. `load_best_model_at_end=True`: Ensures that the best model based on evaluation metrics is loaded at the end of training.
> >
> {: .solution}
>
{: .question}

These settings provide a balanced configuration for training a model efficiently while ensuring that the best version of the model is saved and can be used for further evaluation or deployment. Adjust these parameters based on your specific use case and available computational resources.



# Prepare the tokenizer

We will now set up the tokenizer to convert DNA sequences into numerical tokens that the model can process. The tokenizer is a crucial component in preparing the data for model training and inference: it transforms raw text into a format that can be processed by machine learning models. 

We use the `AutoTokenizer` class from the Hugging Face Transformers library to load a pre-trained tokenizer. We specify the pre-trained model from which to load the tokenizer. This should match the model you plan to use for training or inference. This tokenizer will be configured to handle DNA sequences efficiently.

```python
tokenizer = transformers.AutoTokenizer.from_pretrained(
    model_name,
    model_max_length=200,
    padding_side="right",
    use_fast=True,
    trust_remote_code=True,
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `model_max_length=200`
> 2. `padding_side="right"`
> 3. `use_fast=True`
> 4. `trust_remote_code=True`
> 
> > <solution-title></solution-title>
> >
> > 1. `model_max_length=200`: Sets the maximum length of the tokenized sequences. Sequences longer than this will be truncated, and shorter ones will be padded.
> > 
> > 2. `padding_side="right"`: Specifies that padding should be added to the right side of the sequences. This ensures that all sequences in a batch have the same length.
> >
> > 3. `use_fast=True`: Enables the use of the fast tokenizer implementation, which is optimized for speed and is suitable for most use cases.
> >
> > 4. `trust_remote_code=True`: Allows the tokenizer to execute custom code from the model repository, which may be necessary for some models that require specific preprocessing steps.
> >
> {: .solution}
>
{: .question}

By configuring the tokenizer in this way, we ensure that our DNA sequences are properly tokenized and formatted for input into the model. This step is essential for preparing our data for efficient and effective model training and evaluation.

Let's now tailor the tokenizer to better suit our specific use case, ensuring that the model processes sequences accurately and efficiently. Special tokens play a crucial role in defining how sequences are processed and interpreted by the model. Here, we sets:
- the end-of-sequence (EOS) token, which indicates the end of a sequence. It is essential for tasks where the model needs to generate sequences or understand where a sequence ends.
- the padding (PAD) token, which is used to pad sequences to a uniform length within a batch. Padding ensures that all sequences in a batch have the same length, which is necessary for efficient processing during training and inference

```
tokenizer.eos_token = "[EOS]"
tokenizer.pad_token = "[PAD]"
```

# Prepare data

To finetune the model, we must provide a dataset to train the model. We will the data with the 1st transcription factor (`tf0`) in mouse from {% cite zhou2024dnabert2efficientfoundationmodel %}. The data is stored on [GitHub](https://github.com/raphaelmourad/Mistral-DNA).

## Get data

Let's get the data for from GitHub:

```python
!git clone https://github.com/raphaelmourad/Mistral-DNA.git
```

We now need to uncompress the labeled data:

```python
!tar -xf Mistral-DNA/data/GUE.tar.xz -C Mistral-DNA/data/
```

We change the current working directory to the `Mistral-DNA` folder.

```python
os.chdir("Mistral-DNA/")
print(os.getcwd())
```

Let's define experience and path to data variables

```python
expe = "tf/0"
data_path = f"data/GUE/{ expe }" 
```




## Prepare Datasets for Training and Validation

We now need to set up the datasets required for training and validating. Properly preparing these datasets is crucial for ensuring that the model finetunes effectively and generalizes well to new data.

We will use the files `data_path` folder we just defined:
- `train.csv` for training
- `dev.csv` for validation

> <question-title></question-title>
>
> How is the content of each file?
>
> 1. `train.csv`
> 2. `dev.csv`
> 
> > <solution-title></solution-title>
> >
> > The 2 files are CSV files with 2 columns (`sequence` and `label`) and different number of rows:
> > 1. `train.csv`: 32,379 rows. 
> > 2. `dev.csv`: 1,000 rows
> > 
> > Values in `label` are:
> > - `0`: The DNA sequence in `sequence` column does not bind to the 1st transcription factor.
> > - `1`: The DNA sequence in `sequence` column binds to the transcription factor.
> >
> {: .solution}
>
{: .question}


Before we proceed we import some classes and functions from `scriptPython/function.py`:

```python
### LOAD FUNCTIONS MODULE
import sys
sys.path.append("scriptPython/")
from functions import *
```

We use the `SupervisedDataset` class to load and prepare the datasets. This class handles the tokenization and formatting of the data, making it ready for model training and evaluation.

```python
train_dataset = SupervisedDataset(
    tokenizer=tokenizer,
    data_path=Path(data_path) / "train.csv",
    kmer=-1,
)
val_dataset = SupervisedDataset(
    tokenizer=tokenizer,
    data_path=Path(data_path) / "dev.csv",
    kmer=-1,
)
```

> <question-title></question-title>
>
> What does `kmer=-1`?
>
> > <solution-title></solution-title>
> >
> > This parameter is used to specify the length of k-mers (substrings of length k) to be considered in the dataset. A value of -1 typically means that no k-mer splitting is applied, and the sequences are processed as they are.
> >
> {: .solution}
>
{: .question}

## Configure Data Collation

A data collator ensures that sequences are properly padded and formatted, which is crucial for optimizing the training process.

We'll use the `DataCollatorForSupervisedDataset` class to handle the collation of tokenized data. This collator will manage padding and ensure that all sequences in a batch are of uniform length.

```python
data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer)
```

# Load and Configure the Model for Sequence Classification

Let's now load the pre-trained model, a model originally trained for large language modeling tasks, not specifically for classification. To adapt it for our binary classification task, we will add a new classification head on top of the existing architecture. This head will consist of a single neuron that connects to the output of the language model, enabling it to classify whether a DNA sequence binds to a transcription factor (label `1`) or not (label `0`).

This additional layer, or **classification head**, is a simple neural network layer that takes the high-level features extracted by the language model and maps them to our binary classification output. It learns to weigh these features appropriately to make accurate predictions for our specific task.

We use the `AutoModelForSequenceClassification` class from the Hugging Face Transformers library to load the pre-trained model and set it up for our specific classification task:

```python
model=transformers.AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2,
    output_hidden_states=False,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `num_labels=2`
> 2. `output_hidden_states=False`
> 3. `quantization_config=bnb_config`
> 4. `device_map="auto"`
> 5. `trust_remote_code=True`
> 
> > <solution-title></solution-title>
> >
> > 1. `num_labels=2`: Sets the number of output labels to 2, corresponding to the binary classification task (binding or not binding to transcription factors).
> >
> > 2. `output_hidden_states=False`: Indicates that the model should not output hidden states. This is typically set to False unless you need access to the intermediate representations for further analysis.
> > 
> > 3. `quantization_config=bnb_config`: Applies predefined quantization configuration to the model, which helps reduce memory usage and enables efficient training on consumer-grade hardware.
> > 
> > 4. `device_map="auto"`: Automatically determines the best device placement for the model's layers, optimizing for available hardware (e.g., GPUs). If it finds a GPU, it will use a GPU. If there's no GPU, it will not use the GPU
> > 
> > 5. `trust_remote_code=True`: Allows the model to execute custom code from the model repository, which may be necessary for certain architectures or preprocessing steps.
> >
> {: .solution}
>
{: .question}

To ensure that the model correctly handles padding tokens, we need to align the padding token configuration between the model and the tokenizer. This step is crucial for maintaining consistency during training and inference, especially when dealing with sequences of varying lengths:

```python
model.config.pad_token_id = tokenizer.pad_token_id
```

# Initialize the Trainer

We can now set up the `Trainer` to manage the training and evaluation process of our model. The `Trainer` class simplifies the training loop, handling many of the complexities involved in training deep learning models.

We first need to attach the LoRA adapter to the model:

```python
model.add_adapter(peft_config, adapter_name="lora_1")
```

Let's now set up the `Trainer`:

```python
trainer = transformers.Trainer(
    model=model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    data_collator=data_collator,
    callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]
)
```

> <question-title></question-title>
>
> What do the `callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]` parameter?
> 
> > <solution-title></solution-title>
> >
> > It adds an early stopping mechanism to the training process. This mechanism is designed to halt training when the model's performance on the validation set stops improving, helping to prevent overfitting and conserve computational resources.
> > 
> > How Early Stopping Works?
> > 
> > **Purpose**: The primary goal of early stopping is to capture the model parameters when the loss reaches its minimum value during training. This is crucial because, after a certain point, continued training may lead to overfitting, where the model starts to perform worse on unseen data.
> > 
> > **Patience Parameter**: The `early_stopping_patience=3` setting specifies that training should continue for three additional epochs after the model's performance on the validation set stops improving. This "patience" period helps mitigate the effects of noise in the training process. Noise can cause temporary fluctuations in the loss, making it seem like the model has reached a local minimum when further training might yield better results.
> >
> > **Process**: During training, the loss is monitored at each epoch. If the loss does not decrease for three consecutive epochs, training is stopped. However, if a better model with a lower loss is found within those three epochs, training continues. This approach ensures that the model has truly reached a robust local minimum, rather than being prematurely halted due to noise.
> >
> > By incorporating early stopping with a patience of three epochs, you balance the need to find an optimal model with the risk of overfitting, ultimately leading to more efficient and effective training outcomes.
> > 
> {: .solution}
>
{: .question}

Ffor distributed training, where multiple GPUs or nodes are used to accelerate the training process, it is essential to do:

```python
trainer.local_rank=training_args.local_rank
```

The `local_rank` parameter identifies the rank of the current process within its local node, enabling coordinated communication and synchronization between processes. This setup is crucial for managing tasks such as gradient synchronization and data partitioning, ensuring that each process operates on the correct portion of the model or dataset. By assigning the local rank from `training_args` to the `Trainer`, we facilitate efficient and scalable training, leveraging the full computational power of multi-GPU environments.

# Start the training

Let's start the training process for our model using the trainer.train() method:

```python
trainer.train()
```

After launching `trainer.train()`, we can notice that the training process is significantly faster compared to training a model from scratch seen in ["" tutorial]({% link topics/statistics/tutorials/genomic-llm-pretraining/tutorial.md %}). This efficiency is due to the use of a pre-trained model, which has already undergone extensive training on large datasets using powerful computational resources. For example, pre-training a model on even a small portion of the human genome can take dozens of hours, but fine-tuning this model on a specific task, such as classifying DNA sequences, is much quicker. Fine-tuning leverages the pre-trained model's foundational knowledge, allowing you to adapt it to new tasks with a smaller, labeled dataset. This approach not only saves time but also reduces the need for extensive computational power. By downloading a pre-trained model from platforms like Hugging Face and fine-tuning it on a local machine with a modest GPU, we can achieve high performance with minimal overhead, making advanced modeling techniques accessible for a wide range of applications.

# Evaluate Model Performance

After successfully training the model, the next essential step is to evaluate its performance on a test dataset. This evaluation process is crucial for understanding how well the model generalizes to new, unseen data and for assessing its readiness for real-world applications.

> <comment-title></comment-title>
>
> If finetuning is too long, you can stop the training.
>
{: .comment}

The test data is stored in `data_path/test.csv`, we prepare it as for training and validation data.

```python
test_dataset = SupervisedDataset(
    tokenizer=tokenizer,
    data_path=Path(data_path) / "test.csv",
    kmer=-1,
)
```

We then use the `trainer.evaluate()` method. This methods is designed to assess the model's performance on a specified dataset, typically the test dataset, which contains data that the model has not encountered during training.

```python
results = trainer.evaluate(eval_dataset=test_dataset)
```

The method computes various evaluation metrics, such as accuracy, precision, recall, and F1 score, depending on the task and the configuration specified in `compute_metrics`. These metrics provide a comprehensive view of the model's performance, highlighting its strengths and weaknesses.

The Trainer uses the `data_collator` to ensure that the test data is properly formatted and padded, maintaining consistency with the training process. This consistency is crucial for accurate evaluation.

The evaluation results are stored in the `results` variable, which contains the computed metrics. We can analyze these `results` to gain insights into the model's performance and make informed decisions about further improvements or deployment.

> <question-title></question-title>
>
> What is stored in `results`? How do you interpret this information?
> 
> > <solution-title></solution-title>
> >
> > `results` provides a comprehensive overview of the model's performance on the evaluation dataset with:
> > 1. **eval_loss (0.424961)**: This metric represents the loss value calculated on the evaluation dataset. Lower values indicate better model performance.
> >
> >    A loss of 0.425 suggests that the model is reasonably well-fitted to the data, though the specific interpretation depends on the context and the loss function used (e.g., cross-entropy for classification tasks).
> > 
> > 2. **eval_accuracy (0.804000)**: Accuracy measures the proportion of correctly predicted instances out of the total instances.
> > 
> >    An accuracy of 80.4% indicates that the model correctly predicted the class for 80.4% of the samples in the evaluation dataset.
> > 
> > 4. **eval_f1 (0.800838)**: The F1 score is the harmonic mean of precision and recall, providing a single metric that balances both concerns.
> >    
> >    An F1 score of 0.801 suggests a good balance between precision and recall, indicating that the model performs well in both identifying positive cases and minimizing false positives and negatives.
> > 
> > 4. **eval_matthews_correlation (0.628276)**: The Matthews Correlation Coefficient (MCC) is a measure of the quality of binary classifications, taking into account true and false positives and negatives.
> >    
> >    An MCC of 0.628 indicates a moderate to strong correlation between the predicted and actual classes, suggesting the model is performing better than random guessing.
> > 
> > 6. **eval_precision (0.824614)**: Precision is the ratio of correctly predicted positive observations to the total predicted positives.
> > 
> >    A precision of 82.5% means that out of all the instances predicted as positive, 82.5% were actually positive.
> > 
> > 6. **eval_recall (0.804000)**: Recall (or sensitivity) is the ratio of correctly predicted positive observations to all observations in the actual class.
> >    
> >    A recall of 80.4% indicates that the model correctly identified 80.4% of all actual positive cases.
> > 
> > 8. **eval_runtime (6.548800)**: The total time taken to evaluate the model on the dataset.
> >    
> >    A runtime of 6.55 seconds provides insight into the computational efficiency of the evaluation process.
> > 
> > 8. **eval_samples_per_second (152.699000)**: The number of samples processed per second during evaluation.
> > 
> >    Processing 152.7 samples per second indicates the efficiency of the evaluation pipeline.
> > 
> > 9. **eval_steps_per_second (9.620000)**: The number of evaluation steps completed per second.
> >     
> >    Completing 9.62 steps per second reflects the speed of the evaluation process.
> > 
> > 11. **epoch (3.000000)**: The number of training epochs completed before this evaluation.
> > 
> >    The evaluation was conducted after 3 epochs of training, providing context for the model's learning progress.
> > 
> {: .solution}
>
{: .question}


# Conclusion

In this tutorial, we explored the process of fine-tuning a large language model (LLM) for DNA sequence classification. By following the steps outlined, you have learned how to leverage pre-trained models to achieve efficient and effective classification of DNA sequences, specifically focusing on their binding affinity to transcription factors.

We began by configuring the fine-tuning process, ensuring that available computational resources were optimally utilized. This included specifying settings for quantization, configuring Accelerate for distributed training, and implementing LoRA for parameter-efficient fine-tuning. These steps were crucial for maximizing performance and minimizing computational overhead.

Next, we prepared the tokenizer and data, ensuring that DNA sequences were properly tokenized and formatted for model input. We created datasets for training and validation, and configured data collation to handle batch processing efficiently.

We then loaded and configured the model for sequence classification, adding a classification head to adapt the pre-trained model to our specific task. With the model and data prepared, we initialized the `Trainer`, which streamlined the training process by managing the training loop, evaluation, and checkpointing.

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python", "deep-learning-without-gai-with-python", "genomic-llm-pretraining"]


