# tutorials

topic: statistics
tutorial: flexynesis_unsupervised
title: Unsupervised Analysis of Bone Marrow Cells with Flexynesis
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: 3fe2913182438a398c46d00677b02dd5
last_modified: 2026-05-27T09:01:49.828447
zenodo_link: https://zenodo.org/records/16287482
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/flexynesis_unsupervised/tutorial.html

content:
Traditional dimensionality reduction techniques, while useful, often fail to capture the complex non-linear relationships present in high-dimensional data. Deep learning approaches, particularly Variational Autoencoders (VAEs), have emerged as powerful tools for unsupervised analysis of single-cell transcriptomic data ({% cite DBLP:journals/corr/ZhaoSE17b %}). VAEs combine the representational power of neural networks with probabilistic modeling, enabling them to learn meaningful latent representations while accounting for the inherent uncertainty in biological data.
The key advantage of VAEs lies in their ability to encode high-dimensional gene expression profiles into a lower-dimensional latent space that preserves the most informative biological variation. This latent representation can then be used for various downstream analyses, including clustering, trajectory inference, and data integration.

Flexynesis represents a state-of-the-art deep learning framework specifically designed for multi-modal data integration in biological research ({% cite Uyar2024 %}). What sets Flexynesis apart is its comprehensive suite of deep learning architectures, including supervised and unsupervised VAEs, that can handle various data integration scenarios while providing robust feature selection and hyperparameter optimization.

When an outcome variable is not available, or it is desired to do an unsupervised training, the `supervised_vae` model in flexynesis can be utilized. The supervised variational autoencoder class can be trained on the input dataset without a supervisor head. If the user passes no target variables, batch variables, or survival variables, then the class behaves as a plain variational autoencoder.

This training is inspired from the original flexynesis analysis notebook: [unsupervised_analysis_single_cell.ipynb](https://github.com/BIMSBbioinfo/flexynesis/blob/main/examples/tutorials/unsupervised_analysis_single_cell.ipynb).

Here, we demonstrate the capabilities of flexynesis on a Single-cell CITE-Seq dataset of Bone Marrow samples ({% cite Stuart2019 %}). The dataset was downloaded and processed using Seurat (v5.1.0) ({% cite Hao2021 %}). 5000 cells were randomly sampled for training and 5000 cells were sampled for testing.

> <warning-title>LICENSE</warning-title>
>
>Flexynesis is only available for NON-COMMERCIAL use. Permission is only granted for academic, research, and educational purposes. Before using, be sure to review, agree, and comply with the license.
>For commercial use, please review the flexynesis license on GitHub and contact the [copyright holders](https://github.com/BIMSBbioinfo/flexynesis)
{: .warning}

<!-- This is a comment. -->

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

In the first part of this tutorial we will upload processed CITE-seq data from bone marrow tissue.

All data are in tabular format and they include:

* ADT (Antibody-Derived Tags which indicates the quantification of cell surface proteins) data
* RNA expression data
* Clinical data includes some information about each cell like number of RNAs, genes, ... (In next steps we will use the clustering information "**celltype_l2**")

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    ```text
>    {{ page.zenodo_link }}/files/test-ADT_BMscRNAseq.tabular
>    {{ page.zenodo_link }}/files/test-clin_BMscRNAseq.tabular
>    {{ page.zenodo_link }}/files/test-RNA_BMscRNAseq.tabular
>    {{ page.zenodo_link }}/files/train-ADT_BMscRNAseq.tabular
>    {{ page.zenodo_link }}/files/train-clin_BMscRNAseq.tabular
>    {{ page.zenodo_link }}/files/train-RNA_BMscRNAseq.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each dataset a representative tag (RNA, ADT, clin)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Unsupervised Training of Flexynesis

> <hands-on-title> Train unsupervised model </hands-on-title>
>
> 1. {% tool [Flexynesis](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Type of Analysis"*: `Unsupervised Training`
>        - {% icon param-file %} *"Training clinical data"*: `train-clin_BMscRNAseq.tabular`
>        - {% icon param-file %} *"Test clinical data"*: `test-clin_BMscRNAseq.tabular`
>        - {% icon param-file %} *"Training omics data"*: `train-RNA_BMscRNAseq.tabular`
>        - {% icon param-file %} *"Test omics data"*: `test-RNA_BMscRNAseq.tabular`
>        - *"What type of assay is your input?"*: `RNA`
>        - In *"Multiple omics layers?"*:
>            - {% icon param-repeat %} *"Insert Multiple omics layers?"*
>                - {% icon param-file %} *"Training omics data"*: `train-ADT_BMscRNAseq.tabular`
>                - {% icon param-file %} *"Test omics data"*: `test-ADT_BMscRNAseq.tabular`
>                - *"What type of assay is your input?"*: `ADT`
>        - In *"Advanced Options"*:
>            - *"How many epochs to wait when no improvements in validation loss are observed."*: `5`
>            - *"Number of iterations for hyperparameter optimization."*: `1`
>
>    > <comment-title> Advanced options </comment-title>
>    >
>    > In this tutorial, for the sake of time, we are using **1** iteration for hyperparameter optimization. In a real-life analysis you might want to increase this number according to your dataset.
>    {: .comment}
>
{: .hands_on}
> <question-title></question-title>
>
> 1. What are the outputs from Flexynesis?
>
> > <solution-title></solution-title>
> >
> > 1. There are two tabular files for the latent space embeddings and two feature log files for each of the modalities.
> >
> {: .solution}
>
{: .question}

# Clustering and visualisation

Now, we extract the sample embeddings from the test dataset, cluster the cells using Louvain clustering, and visualize the clusters along with known cell type labels.

> <hands-on-title> Extract test embeddings </hands-on-title>
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input List"*: `results` (output of **Flexynesis** {% icon tool %})
>    - *"How should a dataset be selected?"*: `The first dataset`
>
{: .hands_on}
> <question-title></question-title>
>
> 1. What are other options to extract datasets from a collection?
>
> > <solution-title></solution-title>
> >
> > 1. It is also possible to use index (here index **0**) or data name (here **job.embeddings_test**) to extract the data. Please always check your collection before extraction.
> >
> {: .solution}
>
{: .question}

## Louvain clustering

> <hands-on-title> Cluster cells by Louvain method </hands-on-title>
>
> 1. {% tool [Flexynesis utils](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_utils/flexynesis_utils/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis utils"*: `Louvain Clustering`
>        - {% icon param-file %} *"Matrix"*: `job.embeddings_test` (output of **Extract dataset** {% icon tool %})
>        - {% icon param-file %} *"Predicted labels"*: `test-clin_BMscRNAseq.tabular` (Input dataset)
>        - *"Number of nearest neighbors to connect for each node"*: `15`
>
{: .hands_on}
> <question-title></question-title>
>
> 1. What is the output of this tool?
>
> > <solution-title></solution-title>
> >
> > 1. The output is the `test-clin_BMscRNAseq.tabular` file with a column added containing Louvain clustering values.
> >
> {: .solution}
>
{: .question}

## Get optimal clusters
Now we will use k-means clustering with a varying number of expected clusters and pick the best one based on silhouette scores.

> <hands-on-title> Get optimal clusters </hands-on-title>
>
> 1. {% tool [Flexynesis utils](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_utils/flexynesis_utils/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis utils"*: `Get Optimal Clusters`
>        - {% icon param-file %} *"Matrix"*: `job.embeddings_test` (output of **Extract dataset** {% icon tool %})
>        - {% icon param-file %} *"Predicted labels"*: `louvain_clustering` (output of **Flexynesis utils** {% icon tool %})
>        - *"Minimum number of clusters to try"*: `5`
>        - *"Maximum number of clusters to try"*: `15`
>
>    > <comment-title> Predicted labels </comment-title>
>    >
>    > Please make sure to use the output of Louvain clustering. We need those values in one table for next steps.
>    {: .comment}
>
> 2. Rename the output to `labels with optimal clusters`
{: .hands_on}

> <question-title></question-title>
>
> 1. What is the output of this tool?
>
> > <solution-title></solution-title>
> >
> > 1. Another column is added to the previous table for k-means clustering values.
> >
> {: .solution}
>
{: .question}

In the next step, we will calculate the concrdance between the known cell types and unsupervised cluster labels using AMI (Adjusted Mutual Information) and ARI (Adjusted Rand Index) indices.

## Compute AMI, ARI

**AMI** (Adjusted Mutual Information) and **ARI** (Adjusted Rand Index) are used to compare clustering results with ground truth labels. They measure concordance (agreement) between two clusterings.
AMI ranges from 0 (no agreement) to 1 (perfect match) and ARI ranges from -1 (complete disagreement) to 1 (perfect agreement).

> <hands-on-title> Louvain vs true labels </hands-on-title>
>
> 1. {% tool [Flexynesis utils](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_utils/flexynesis_utils/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis utils"*: `Compute AMI and ARI`
>        - {% icon param-file %} *"Predicted labels"*: `labels with optimal clusters` (output of **Flexynesis utils** {% icon tool %})
>        - *"Column name in the labels file to use for the true labels"*: `c10` (celltype_l2)
>        - *"Column name in the labels file to use for the predicted labels"*: `c12` (louvain_cluster)
>
{: .hands_on}

> <hands-on-title> k-means vs true labels </hands-on-title>
>
> 1. {% tool [Flexynesis utils](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_utils/flexynesis_utils/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis utils"*: `Compute AMI and ARI`
>        - {% icon param-file %} *"Predicted labels"*: `labels with optimal clusters` (output of **Flexynesis utils** {% icon tool %})
>        - *"Column name in the labels file to use for the true labels"*: `c10` (celltype_l2)
>        - *"Column name in the labels file to use for the predicted labels"*: `c13` (optimal_kmeans_cluster)
{: .hands_on}
> <question-title></question-title>
>
> 1. Which of the clusterings has better concordance with the known cell type? Louvain and k-means?
>
> > <solution-title></solution-title>
> >
> > 1. The Louvain has AMI = 0.66 and ARI = 0.49 and k-means has AMI = 0.55 and ARI = 0.43.
> > Louvain Clustering seems to yield better AMI/ARI scores. So, we use them to do more visualizations.
> >
> {: .solution}
>
{: .question}


## UMAP visualisation of true and Louvain lables

> <hands-on-title> Dimension reduction plot </hands-on-title>
>
> 1. {% tool [Flexynesis plot](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis plot"*: `Dimensionality reduction`
>        - {% icon param-file %} *"Embeddings"*: `job.embeddings_test` (output of **Extract dataset** {% icon tool %})
>        - {% icon param-file %} *"Predicted labels"*: `labels with optimal clusters` (output of **Flexynesis utils** {% icon tool %})
>        - *"Column in the labels file to use for coloring the points in the plot"*: `c10` (celltype_l2)
>        - *"Transformation method"*: `UMAP`
{: .hands_on}

> <hands-on-title> Dimension reduction plot </hands-on-title>
>
> 1. {% tool [Flexynesis plot](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis plot"*: `Dimensionality reduction`
>        - {% icon param-file %} *"Predicted labels"*: `labels with optimal clusters` (output of **Flexynesis utils** {% icon tool %})
>        - *"Column in the labels file to use for coloring the points in the plot"*: `c12` (louvain_cluster)
>        - *"Transformation method"*: `UMAP`
{: .hands_on}

> <question-title></question-title>
>
> 1. Compare these two UMAP plots, Is the unsupervised clustering close to the ground truth labels?
>
> > <solution-title></solution-title>
> >
> > 1. We can see that like true labels, each UMAP clusters have unique Louvain clusters assigned. This shows that this clustering based on the latent space is close to the ground truth. However, we still don't know which Louvain cluster, corresponds to which true label.
> >
> >![UMAP plot of test Embeddings colored by true labels](../../images/flexynesis_unsupervised/true_label_umap.jpeg "UMAP plot of test Embeddings colored by true labels")
> >![UMAP plot of test Embeddings colored by predicted labels](../../images/flexynesis_unsupervised/predicted_label_umap.jpeg "UMAP plot of test Embeddings colored by predicted labels")
> >
> {: .solution}
>
{: .question}

To see the real concordance between Louvain clusters and true values, we can observe a tabulation of the concordance between them. (Each row sums up to 1).

> <hands-on-title> Concordance plot </hands-on-title>
>
> 1. {% tool [Flexynesis plot](toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3) %} with the following parameters:
>    - *"I certify that I am not using this tool for commercial purposes."*: `Yes`
>    - *"Flexynesis plot"*: `Label concordance heatmap`
>        - {% icon param-file %} *"Predicted labels"*: `labels with optimal clusters` (output of **Flexynesis utils** {% icon tool %})
>        - *"Column in the labels file to use for true labels"*: `c10` (celltype_l2)
>        - *"Column in the labels file to use for predicted labels"*: `c12` (louvain_cluster)
{: .hands_on}

Now it is easier to see which Lovain cluster corresponds to which true value.

![Concordance plot of true values vs predicted values](../../images/flexynesis_unsupervised/concordance.jpeg "Concordance plot of true values vs predicted values")


# Conclusion

Here we demonstrated the power of Flexynesis for unsupervised analysis of multi-modal single-cell data. We explored how variational autoencoders can capture cellular heterogeneity without requiring labeled training data.

tools_json:
["Flexynesis", "toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis/flexynesis/0.2.20+galaxy3", "Extract dataset", "__EXTRACT_DATASET__", "Flexynesis utils", "toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_utils/flexynesis_utils/0.2.20+galaxy3", "Flexynesis plot", "toolshed.g2.bx.psu.edu/repos/bgruening/flexynesis_plot/flexynesis_plot/0.2.20+galaxy3"]


topic: statistics
tutorial: multimodal_learner
title: Gleam Multimodal Learner - Head and Neck cancer Recurrence Prediction with
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["Multimodal Learning", "GLEAM", "HANCOCK Dataset", "Recurrence Prediction"]
content_hash: 9a1dd1770220a02aeb6e3cd0c010cfe0
last_modified: 2026-05-27T09:01:49.841059
zenodo_link: https://zenodo.org/records/18807820
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/multimodal_learner/tutorial.html

content:
In this tutorial, we use the **HANCOCK** head-and-neck cancer cohort ({% cite Dorrich2025 %}) to build a **recurrence prediction** model with **GLEAM Multimodal Learner**.

The primary run in this tutorial uses a **tutorial-focused derivative** of HANCOCK in which CD3/CD8 images were converted from PNG to JPEG (quality 67) to reduce upload size and setup time. This makes the workflow more accessible while preserving the multimodal structure.

Multimodal Learner wraps AutoGluon Multimodal to train a **late-fusion** model. It automatically builds modality-specific encoders: tabular features with a tabular transformer, text fields with a pretrained language model, and images with a pretrained vision backbone. A fusion network then combines these modality embeddings into a single prediction.

You provide a single CSV/TSV with one row per patient containing the target label and modality references (text columns and image-path columns). Image files are uploaded as one or more ZIP archives; the paths in the table must match filenames inside the ZIP. We use the **published train/test split** and interpret results with ROC/PR curves, confusion matrices, and threshold-dependent metrics.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Dataset Overview and Composition

The HANCOCK dataset used in this tutorial is a multimodal precision oncology resource that integrates multiple data types to enable comprehensive cancer outcome prediction.

## Dataset Description

The HANCOCK (Head and Neck Cancer Outcome Cohort) dataset was published by {% cite Dorrich2025 %} and represents one of the first large-scale multimodal datasets specifically designed for head and neck cancer outcome prediction.

For this tutorial, we use a Zenodo derivative package built for practical training runs:

- **Image archive**: `CD3_CD8_images_3GB_jpeg.zip`
- **Metadata tables**: `HANCOCK_train_split_3GB_jpeg.csv` and `HANCOCK_test_split_3GB_jpeg.csv`
- **Image conversion**: PNG to JPEG with lossy compression (quality 67), while preserving original image dimensions
- **Archive size**: about 2.89 GB (3,037 images)
- **Matched records**: 653 samples, with an approximately 80/20 train/test split

This modified dataset is intended for tutorial and tool demonstration use. It is **not** a replacement for the original full-quality HANCOCK dataset for research-grade analysis or final benchmarking.

### Patient Cohort
- **Original HANCOCK cohort**: 763 head and neck cancer patients
- **Tutorial derivative used here**: 653 matched records with updated JPEG paths
- **Cancer sites**: Oral cavity, oropharynx, hypopharynx, larynx
- **Follow-up**: Comprehensive outcome tracking with recurrence and survival data
- **Data collection**: Multi-institutional cohort with standardized data collection protocols

### Data Modalities

#### 1. Tabular Data (Clinical and Pathological Features)
- **Clinical features**: Age, sex, smoking status, primary tumor site
- **Pathological features**: Tumor grade, stage, histological type
- **Blood biomarkers**: Complete blood count, liver function, kidney function
- **TMA cell density**: Quantitative cell counts from tissue microarray analysis

#### 2. Image Data (Tissue Microarray Cores)
- **Modality**: CD3 and CD8 immunohistochemistry staining
- **Resolution**: Original image dimensions are preserved
- **Content**: Tumor microenvironment immune cell infiltration
- **Format in this tutorial**: JPEG images of individual TMA cores

#### 3. Text Data (ICD Diagnostic Codes)
- **Source**: International Classification of Diseases (ICD-10) codes
- **Content**: Diagnostic codes representing comorbidities and medical history
- **Format**: Free-text ICD code descriptions

## Target Variable: 3-Year Recurrence

The primary prediction task is **3-year recurrence**, defined as:
- **Positive (recurrence = 1)**: Recurrence within 3 years (1,095 days) of diagnosis
- **Negative (recurrence = 0)**: No recurrence with adequate follow-up (>3 years) or currently living without recurrence

This binary classification task is clinically relevant as early recurrence (within 3 years) is associated with poor prognosis and may benefit from more aggressive treatment strategies.

## Data Splits

The tutorial derivative provides a predefined split:

- **Training set**: approximately 80% of samples
- **Test set**: approximately 20% of samples

## Expected table structure

Because Multimodal Learner requires a specific input structure—a single sample-level table that contains the label and all modality references, plus, when using images, a ZIP archive containing the referenced image files—you must preprocess the dataset to match this format before running the tool. For a HANCOCK-like setup, the training and test tables typically include:

- `target`: recurrence label (0 = no recurrence, 1 = recurrence/progression)
- `clinical covariates`: numeric and categorical clinical/pathological variables
- `icd_text`: free-text ICD descriptions
- `cd3_path` and `cd8_path`: image filenames (or relative paths) for CD3 and CD8 TMA images, which must match files inside the ZIP archive (`.jpg` paths in this tutorial dataset)

> <comment-title>Data preparation: shaping HANCOCK for the tool</comment-title>
>
> The raw data published by ({% cite Dorrich2025 %}) can be found here:
> [HANCOCK raw dataset](https://www.hancock.research.uni-erlangen.org/download)
>
> We preprocessed the raw data to create a tutorial-friendly derivative:
>
> 1) Normalize identifiers consistently (e.g., remove leading zeros; standardize missing values) before merging.
>
> 2) Join modality tables using a stable identifier (e.g., patient_id).
>
> 3) Construct recurrence label for each patient.
>
> 4) Keep ICD code text as free text for the text encoder.
>
> 5) Convert image files from PNG to JPEG (quality 67), keeping original dimensions.
>
> 6) Update metadata paths from `.png` to `.jpg` so table references match the ZIP archive.
>
> 7) Preserve a stable train/test partition and avoid leakage across patients.
>
> 8) Provide training and test tables in CSV format and all images in one **ZIP** archive.
>
> A Python script for preprocessing can be found at: [HANCOCK_preprocessing](https://github.com/goeckslab/gleam_use_cases/tree/main/multimodal_learner)
>
{:  .comment}

![Schema of the whole process of training model and test](./../../images/multimodal_learner/workflow.png "Overview of the process steps to obtain the model from the HANCOCK dataset."){: style="width: 60%; display: block; margin-left: auto; margin-right: auto;"}

# Using Multimodal Learner Tool

## How it works

Multimodal Learner trains **late-fusion prediction models** using **AutoGluon Multimodal**.

- **One row per sample:** a single CSV/TSV contains the target label plus tabular features and (optionally) text and image references for the same sample (e.g., patient).
- **Encoders per modality:**
  - tabular features are encoded by a transformer-based tabular backbone (handled automatically),
  - text columns are encoded by a user-selected pretrained language model,
  - image columns are encoded by a user-selected pretrained vision backbone.
- **Fusion:** modality embeddings are combined by a learned fusion network to produce the final prediction.
- **Outputs:** an interactive HTML report (ROC/PR curves, confusion matrix, calibration and threshold views), plus a YAML config record and JSON metrics.

## Prepare environment and get the data

> <hands-on-title>Environment and Data Upload</hands-on-title>
>
> 1. Create a new history for this tutorial. For example, name it *HANCOCK Multimodal Recurrence*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the dataset files from Zenodo:
>
>    ```
>    https://zenodo.org/records/18807820/files/HANCOCK_train_split_3GB_jpeg.csv
>    https://zenodo.org/records/18807820/files/HANCOCK_test_split_3GB_jpeg.csv
>    https://zenodo.org/records/18807820/files/CD3_CD8_images_3GB_jpeg.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the data formats are assigned correctly. If not, follow the Changing the datatype tip:
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

## Tool setup and run

After the train and test tables are uploaded, configure Multimodal Learner with the following parameters.

> <hands-on-title>Configure Multimodal Learner for HANCOCK</hands-on-title>
>
> 1. {% tool [Multimodal Learner](toolshed.g2.bx.psu.edu/repos/goeckslab/multimodal_learner/multimodal_learner/0.1.0) %} with the following parameters:
>    - {% icon param-file %} *Training dataset (CSV/TSV)*: `HANCOCK_train_split_3GB_jpeg.csv`
>    - {% icon param-select %} *Target / Label column*: `target`
>    - {% icon param-toggle %} *Provide separate test dataset?*: `Yes`
>    - {% icon param-file %} *Test dataset (CSV/TSV)*: `HANCOCK_test_split_3GB_jpeg.csv`
>    - {% icon param-select %} *Text backbone*: `google/electra-base-discriminator`
>    - {% icon param-toggle %} *Use image modality?*: `Yes`
>    - {% icon param-select %} *+ Insert Image archive*: `click`
>    - {% icon param-file %} *ZIP file containing images*: `CD3_CD8_images_3GB_jpeg.zip`
>    - {% icon param-select %} *Image backbone*: `caformer_b36.sail_in22k_ft_in1k`
>    - {% icon param-toggle %} *Drop rows with missing images?*: `No`
>    - {% icon param-select %} *Primary evaluation metric*: `ROC AUC`
>    - {% icon param-text %} *Random seed*: `42`
>    - {% icon param-toggle %} *Advanced: customize training settings?*: `Yes`
>    - {% icon param-text %} *Binary classification threshold*: `0.25`
>
> 2. Run the tool and review the HTML report.
>
{: .hands_on}

> <tip-title>Recommended Multimodal Learner configuration (and why)</tip-title>
>
> | Parameter | Rationale |
> |---|---|
> | Text backbone | ELECTRA base performs well on clinical free text |
> | Image backbone | CAFormer b36 is a strong baseline for CD3/CD8 pathology images |
> | ROC AUC metric | Robust for imbalanced binary outcomes |
> | 5-fold CV | Estimates stability across folds |
> | Threshold 0.25 | Matches the recurrence decision rule used in the use case |
>
{: .tip}

## Tool Output Files

After the run finishes, you should see these outputs in your history:

- **Multimodal Learner analysis report (HTML)**: Summary metrics, ROC/PR curves, and diagnostic plots
- **Multimodal Learner metric results (JSON)**: Machine-readable metrics for train/validation/test
- **Multimodal Learner training config (YAML)**: Full run settings for reproducibility

## Interpreting the Multimodal Learner Report

The HTML report is designed to help you answer two questions quickly: **how well does the model discriminate outcomes** and **what kinds of errors it makes at a chosen operating point**.

![Report tabs containing configuration, metrics and plots of the model](./../../images/multimodal_learner/report_tabs.png "Picture of the report tabs generated after running the multimodal learner tool.")

- **Performance summary (by split)**
  Review ROC–AUC and PR–AUC alongside threshold-dependent metrics (precision, recall, F1). For recurrence prediction, PR–AUC is often especially informative because the positive class can be relatively rare, and false negatives directly reduce recall.
- **Diagnostics (curves and confusion matrix)**
  Use ROC/PR curves to understand ranking performance, then move to the confusion matrix and class-wise metrics to see *where* errors occur. In the HANCOCK recurrence use case, a common pattern is **stronger performance for the negative class (no recurrence)** than the positive class, so the main concern is typically **missed recurrence cases (false negatives)**.
- **Calibration and threshold behavior**
  Calibration plots help you decide whether predicted probabilities can be interpreted as risk. Threshold views show how changing the cutoff trades off false negatives versus false positives. The configuration used was **0.25** decision threshold to make this tradeoff explicit (rather than relying on the default 0.5).
- **Configuration and reproducibility**
  Confirm that the report matches your intended setup: which modalities were used, chosen text and image backbones, missing-image handling, split strategy, random seed, and time budget.

# Tutorial Results (JPEG Derivative Dataset)

The following held-out test metrics are from the tutorial run using the JPEG-compressed image archive (`CD3_CD8_images_3GB_jpeg.zip`). The smaller archive greatly improves upload and execution practicality, with the expected tradeoff of reduced predictive performance compared with the full-resolution image setup.

| Metric | Value |
|---|---:|
| Accuracy | 0.72 |
| Precision | 0.36 |
| Recall (Sensitivity/TPR) | 0.15 |
| F1-Score | 0.21 |
| Specificity (TNR) | 0.91 |
| ROC-AUC | 0.63 |
| PR-AUC | 0.33 |
| LogLoss | 0.60 |
| MCC | 0.09 |

How to read this run:

- **The model still captures some recurrence signal** (ROC-AUC above 0.5), but discrimination is moderate.
- **Specificity remains high (0.91)**, while **recall is low (0.15)**, indicating under-detection of recurrence cases at the selected threshold.
- **PR-AUC and MCC are modest**, which is consistent with a harder positive-class detection setting under image compression.

## What to do next with this tutorial run

- Use threshold plots in the report to test operating-point tradeoffs (higher recall usually lowers precision/specificity).
- Keep this lightweight setup for learning the workflow and tool behavior.
- Use the high-quality setup (below) when you need stronger performance analysis or paper-level comparison.

## Conclusion

Using the HANCOCK recurrence task as an example, this tutorial demonstrates how Multimodal Learner integrates **tabular clinical variables**, **free-text ICD descriptions**, and **CD3/CD8 images** in a single late-fusion workflow. The tutorial-focused JPEG derivative provides a much smoother user experience for data upload and execution, but the resulting metrics are lower than the full-resolution setup. This contrast highlights a core practical lesson for multimodal learning in pathology workflows: **data quality, especially image quality, has direct impact on downstream performance and benchmark comparability**.

# High-Quality Image Run (Reference Section, ~45 GB ZIP)

The results that were previously shown in the main tutorial are retained here as a separate reference run using the larger, higher-quality image archive (approximately 45 GB ZIP). This setup is closer to research-grade benchmarking but requires substantially more storage, upload time, and compute.

Reference files for the higher-quality run:

- `https://zenodo.org/records/17933596/files/HANCOCK_train_split.csv`
- `https://zenodo.org/records/17933596/files/HANCOCK_test_split.csv`
- `https://zenodo.org/records/17727354/files/tma_cores_cd3_cd8_images.zip`

> <comment-title>Data preparation: shaping HANCOCK for the tool</comment-title>
>
> A Python script for preprocessing can be found at: [HANCOCK_preprocessing](https://github.com/goeckslab/gleam_use_cases/tree/main/multimodal_learner)
>
{:  .comment}

## Reference Results with Full-Quality Images

| Metric | Value |
|---|---:|
| Accuracy | 0.81 |
| Precision | 0.90 |
| Recall (Sensitivity/TPR) | 0.27 |
| F1-Score | 0.42 |
| Specificity (TNR) | 0.99 |
| ROC-AUC | 0.78 |
| PR-AUC | 0.67 |

![ROC-AUC curve generated by the test set](./../../images/multimodal_learner/test_roc_auc.png "ROC-AUC plot generated on the test set for the higher-quality image setup.")

## Comparison with the HANCOCK Benchmark Paper

The original HANCOCK study reported an ROC-AUC of **0.79** for recurrence prediction using engineered multimodal features ({% cite Dorrich2025 %}). With full-quality images and raw modalities in Multimodal Learner, the reference run reached ROC-AUC **0.78**, which is close to the published benchmark.

| Metric | HANCOCK paper (reference) | Multimodal Learner (full-quality image run) |
|---|---:|---:|
| ROC-AUC | 0.79 | 0.78 |

tools_json:
["Multimodal Learner", "toolshed.g2.bx.psu.edu/repos/goeckslab/multimodal_learner/multimodal_learner/0.1.0"]


topic: statistics
tutorial: genomic-llm-zeroshot-prediction
title: Predicting Mutation Impact with Zero-shot Learning using a pretrained DNA LLM
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml", "Large Language Model"]
content_hash: dcea0de77dc50acc7c3655387184b855
last_modified: 2026-05-27T09:01:49.830486
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/genomic-llm-zeroshot-prediction/tutorial.html

content:
Predicting the impact of mutations is a critical task in genomics, as it provides insights into how genetic variations influence biological functions and contribute to diseases. Traditional methods for assessing mutation impact often rely on extensive experimental data or computationally intensive simulations. However, with the advent of large language models (LLMs) and zero-shot learning, we can now predict mutation impacts more efficiently and effectively.

Zero-shot learning is a technique that allows a pre-trained model to make predictions on tasks it wasn't explicitly trained for, leveraging its existing knowledge. This approach is particularly valuable when labeled data is scarce or when rapid predictions are needed. By using a pre-trained DNA LLM, we can compute embeddings for both wild-type and mutated DNA sequences and compare them to quantify the impact of mutations.

This tutorial focuses on this innovative method, utilizing a [pre-trained DNA LLM available on Hugging Face](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-17M-hg38) to assess the impact of mutations. This approach opens new avenues for bioinformatics research, particularly in genomics and personalized medicine, by enabling researchers to gain insights into the functional impact of DNA mutations efficiently.

Building upon this foundation, our new tutorial focuses on predicting the impact of mutations using zero-shot learning with a pre-trained DNA LLM. Zero-shot learning allows us to utilize the pre-trained model directly, without additional training, to make predictions on new, unseen tasks. Specifically, we will use a pre-trained model available on Hugging Face, designed for DNA sequences, to assess the impact of mutations.

We will use [`Mistral-DNA-v1-17M-hg38`](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-1M-hg38), a mixed model that was pre-trained on the entire Human Genome. It contains approximately 17 million parameters and was trained using the Human Genome assembly GRCh38 on sequences of 10,000 bases (10K):

```python
model_name="RaphaelMourad/Mistral-DNA-v1-17M-hg38"
```

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare resources 

## Install dependencies

The first step is to install the required dependencies:

```python
!pip install Bio==1.7.1
!pip install transformers -U
```

## Import Python libraries

Let's now import them.

```python
import os

import matplotlib.pyplot as plt
import numpy as np
import scipy as sp
import torch
import tensorflow as tf
import gzip
from Bio import SeqIO
from transformers import (
  AutoConfig,
  AutoModelForCausalLM,
  AutoTokenizer,
  EarlyStoppingCallback,
  Trainer,
  TrainingArguments,
)
```

> <comment-title>Versions</comment-title>
>
> This tutorial has been tested with following versions:
> - `transformers` = 4.48.3
> 
> You can check the versions with:
>
> ```python
> transformers.__version__
> ```
{: .comment}

## Check and configure available resources

We select the appropriate device (CUDA-enabled GPU if available) for running PyTorch operations

```python
torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

Let's check the GPU usage and RAM:

```python
!nvidia-smi
```

Let's configure PyTorch and the CUDA environment -- software and hardware ecosystem provided by NVIDIA to enable parallel computing on GPU -- to optimize GPU memory usage and performance:

1. Enables CuDNN benchmarking in PyTorch:

    ```python
    torch.backends.cudnn.benchmark=True
    ```

2. Set an environment variable that configures how PyTorch manages CUDA memory allocations

    ```python
    os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"
    ```


# Tokenizing DNA Sequences

We now set up the tokenizer to convert raw DNA sequences into a format that the model can process, enabling it to understand and analyze the sequences effectively:

```python
tokenizer = transformers.AutoTokenizer.from_pretrained(
    model_name,
    use_fast=True,
    trust_remote_code=True,
)
```

> <question-title></question-title>
>
> What do the parameters?
> 
> 1. `use_fast=True`?
> 2. `trust_remote_code=True`?
>
> > <solution-title></solution-title>
> >
> > 1. `use_fast=True`: Enables the use of a fast tokenizer implementation, which is optimized for speed and efficiency. This is particularly useful when working with large datasets or when performance is a priority.
> > 2. `trust_remote_code=True`: Allows the tokenizer to execute custom code from the model repository. This may be necessary for certain architectures or preprocessing steps that require additional functionality.
> >
> {: .solution}
>
{: .question}

# Load and Configure the Pre-trained Model

We will now load the pre-trained DNA large language model (LLM) and configure it for our specific task of predicting the impact of DNA mutations. 

```python
model=transformers.AutoModelForCausalLM.from_pretrained(
    model_name,
)
```

We would like to ensure that the model correctly handles padding tokens, which are used to standardize the length of sequences within a batch:

```python
model.config.pad_token_id = tokenizer.pad_token_id
```

Aligning the padding token ID between the model and tokenizer is crucial for maintaining consistency during training and inference.

Let's look at the model:

```python
model
```

```
MixtralForCausalLM(
  (model): MixtralModel(
    (embed_tokens): Embedding(4096, 256)
    (layers): ModuleList(
      (0-7): 8 x MixtralDecoderLayer(
        (self_attn): MixtralAttention(
          (q_proj): Linear(in_features=256, out_features=256, bias=False)
          (k_proj): Linear(in_features=256, out_features=256, bias=False)
          (v_proj): Linear(in_features=256, out_features=256, bias=False)
          (o_proj): Linear(in_features=256, out_features=256, bias=False)
        )
        (block_sparse_moe): MixtralSparseMoeBlock(
          (gate): Linear(in_features=256, out_features=8, bias=False)
          (experts): ModuleList(
            (0-7): 8 x MixtralBlockSparseTop2MLP(
              (w1): Linear(in_features=256, out_features=256, bias=False)
              (w2): Linear(in_features=256, out_features=256, bias=False)
              (w3): Linear(in_features=256, out_features=256, bias=False)
              (act_fn): SiLU()
            )
          )
        )
        (input_layernorm): MixtralRMSNorm((256,), eps=1e-05)
        (post_attention_layernorm): MixtralRMSNorm((256,), eps=1e-05)
      )
    )
    (norm): MixtralRMSNorm((256,), eps=1e-05)
    (rotary_emb): MixtralRotaryEmbedding()
  )
  (lm_head): Linear(in_features=256, out_features=4096, bias=False)
)
```

> <question-title></question-title>
>
> What do the parameters?
> 
> 1. How does the vocabulary size of 4,096 relate to k-mers in DNA sequences?
> 2. What is the role of the embedding layer in `MixtralForCausalLM`?
> 3. How many layers does the `MixtralForCausalLM` model have, and what is their purpose?
> 4. What components make up the self-attention mechanism in MixtralAttention?
> 5. How does the Mixture of Experts (MoE) mechanism reduce computational load?
> 6. Why is layer normalization important in `MixtralForCausalLM`?
> 7. What advantage do rotary embeddings offer in understanding DNA sequences?
>
> > <solution-title></solution-title>
> >
> > 1. The vocabulary size corresponds to the number of unique "words" or k-mers (subsequences of DNA) the model can recognize, similar to using k-mers of size six, enhanced by byte-pair encoding for more nuanced patterns.
> > 2. The embedding layer converts DNA sequences into numerical vectors, enabling the model to process and analyze them.
> > 3. The model has 8 layers, each containing a `MixtralDecoderLayer`. These layers process the embedded input sequences through a series of transformations, including self-attention and mixture of experts, to capture complex patterns in the data.
> > 4. The self-attention mechanism includes query, key, value, and output projections, which weigh the importance of different tokens in the sequence.
> > 5. MoE activates only a subset of parameters during each forward pass, using a routing mechanism to direct sequences to specific experts.
> > 6. Layer normalization stabilizes and accelerates training by ensuring consistent scaling of inputs to the attention mechanism and subsequent layers.
> > 7. Rotary embeddings enhance the model's understanding of positional information, providing a more nuanced representation of sequence order compared to traditional methods.
> >
> {: .solution}
>
{: .question}

> <question-title></question-title> 
>
> For a DNA sequence "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC"
>
> 1. How can we get the hidden states?
> 2. How can we compute mean of the hidden states accross the sequence length dimension?
> 3. What is the shape of the output?
> 4. What does the mean of the hidden states accross the sequence length dimension represent?
>
> > <solution-title></solution-title>
> >
> > Let's start by defining the DNA sequence:
> > 
> > ```python
> > dna = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC"
> > ```
> > 
> > 1. To get the hidden state:
> >    1. Tokenize the DNA sequence using the tokenizer
> >      
> >        ```python
> >        tokenized_dna = tokenizer(dna, return_tensors = "pt")
> >        ```
> >
> >    2. Extract the tensor containing the token IDs from the tokenized output
> >      
> >        ```python
> >        inputs = tokenized_dna["input_ids"]
> >        ```
> >
> >    3. Pass the tokenized input through the model.
> >      
> >        ```python
> >        model_outputs = model(inputs)
> >        ```
> >
> >    4. Extract the hidden states from the model's output.
> >      
> >        ```python
> >        hidden_states = model_outputs[0].detach()
> >        ```
> >
> > 2. To compute mean of the hidden states accross the sequence length dimension:
> >    
> >    ```python
> >    embedding_mean = torch.mean(hidden_states[0], dim=0)
> >    ```
> >
> > 3. The shape is 4,096, the number of possible tokens
> > 
> > 4. It represents the average embedding of the DNA sequence. This fixed-size representation can be used for various downstream tasks, such as classification, clustering, or similarity comparisons.
> {: .solution}
>
{: .question}

# Compare the effect of mutations with or without amino acid modification

Let's look at the portion of the Cystic fibrosis transmembrane conductance regulator (CFTR) gene where a mutation responsible of the Cystic fibrosis appears.

In this portion, we can observe several cases:

Cases | Sequences
--- | ---
Wild-type without any mutation | ATTAAAGAAAATATCATCTTTGGTGTTTCCTAT
Mutation ATT to ATA | ATAAAAGAAAATATCATCTTTGGTGTTTCCTAT
Deletion of TCT codon | ATTAAAGAAAATATCA---TTGGTGTTTCCTAT

In the second case, the amino acid does not change (silent mutation). In the last case, an amino acid is removed. Let's look if the mutation and the deletion have the same distance to the wild-type when computing using the DNA embedding.

First, we define the sequences:

```python
dna_wt= "ATTAAAGAAAATATCATCTTTGGTGTTTCCTAT"
dna_mut="ATAAAAGAAAATATCATCTTTGGTGTTTCCTAT"
dna_del="ATTAAAGAAAATATCATTGGTGTTTCCTAT"
```

Let's compute the hidden states for all DNA sequences:

```python
tokenized_dna = tokenizer(
  [dna_wt, dna_mut, dna_del],
  return_tensors="pt",
  padding=True,
)
inputs_seqs = tokenized_dna["input_ids"]
model_outputs = model(inputs_seqs)
hidden_states = model_outputs[0].detach()
```

We now compute the maximum of the hidden states accross the sequence length dimension:

```python
embedding_max = torch.max(hidden_states, dim=1)[0]
```


To compare the effects of silent mutation and amino acid deletion, we will compute the distance between the wild-type embeddings and the mutation / deletion embedding using the L2 (Euclidean) distance

> <details-title>L2 (Euclidean) distance</details-title>
>
> The L2 distance, also known as the Euclidean distance, is a measure of the straight-line distance between two points in Euclidean space. It is commonly used to quantify the difference between two vectors, such as embeddings in machine learning.
>
> For two vectors $$a$$ and $$b$$ in an $$n$$-dimensional space, where $$a=[a_{1},a_{2},...,a_{n}]$$ and $$b=[b_{1},b_{2},...,b_{n}]$$, the L2 distance is calculated as:
>
> \\(L2 = \sqrt{\sum_{i} (a_{i}-b_{i})^{2}\\)
>
{: .details}
          
```python
wt_mut_L2 = torch.norm(embedding_max[0] - embedding_max[1])
print(wt_mut_L2)
wt_del_L2 = torch.norm(embedding_max[0] - embedding_max[2])
print(wt_del_L2)
```

```
tensor(27.4657)
tensor(145.7797)
```

> <question-title></question-title>
>
> 1. Is it the silent mutation or the amino acid deletion that have the lowest L2 distance to the wild-type?
> 2. How to interpret the result?
>
> > <solution-title></solution-title>
> >
> > 1. The silent mutation has a lower L2 distance to the wild-type compared to the mutation with amino acid deletion.
> > 2. A smaller L2 distance indicates that the sequences are more similar, while a larger distance suggests greater dissimilarity.
> {: .solution}
>
{: .question}


# Compare effects of SNPs in exons and introns

Single Nucleotide Polymorphisms (SNPs) are the most common type of genetic variation, involving a change in a single nucleotide within a DNA sequence. These variations can occur in different regions of the genome, including exons (the coding regions of genes) and introns (the non-coding regions within genes). Understanding the impact of SNPs in these regions is crucial for assessing their role in genetic disorders, phenotypic traits, and overall genome function.

We will now leverage the pre-trained DNA language model to compare the effects of SNPs in exons and introns. By utilizing embeddings generated from DNA sequences, we can quantify the impact of these variations and gain insights into their functional consequences.

## Load sequences

To begin our analysis, we need to load the sequence data, which includes sequences with Single Nucleotide Polymorphisms (SNPs) and their corresponding reference sequences (wild-type) without SNPs. These sequences are derived from both introns and exons and are stored in compressed FASTA format on [GitHub](https://github.com/raphaelmourad/Mistral-DNA/tree/main/data/SNP):

| Wild-type (without SNPs) | Mutated (with SNP)
--- | --- | ---
Intron | [`SNPintron_ref_201b.fasta.gz`](https://github.com/raphaelmourad/Mistral-DNA/raw/refs/head/master/data/SNP/SNPintron_ref_201b.fasta.gz)  |  [`SNPintron_alt_201b.fasta.gz`](https://github.com/raphaelmourad/Mistral-DNA/raw/refs/head/master/data/SNP/SNPintron_alt_201b.fasta.gz)
Exon | [`SNPexon_ref_201b.fasta.gz`](https://github.com/raphaelmourad/Mistral-DNA/raw/refs/head/master/data/SNP/SNPexon_ref_201b.fasta.gz)  |  [`SNPexon_alt_201b.fasta.gz`](https://github.com/raphaelmourad/Mistral-DNA/raw/refs/head/master/data/SNP/SNPexon_alt_201b.fasta.gz)

```python
baseurl = "https://github.com/raphaelmourad/Mistral-DNA/raw/refs/head/master/data/SNP/"
win=201
exon_wt_snp_fp = f"{ baseurl }/SNPexon_ref_{ win }b.fasta.gz"
exon_mut_snp_fp = f"{ baseurl }/SNPexon_alt_{ win }b.fasta.gz"
intron_wt_snp_fp = f"{ baseurl }/SNPintron_ref_{ win }b.fasta.gz"
intron_mut_snp_fp = f"{ baseurl }/SNPintron_alt_{ win }b.fasta.gz"
```

We need to get files and read the sequences from them:

```python
import requests
def downloadReadFastaFile(fasta_file):
  response = requests.get(fasta_file)
  # Check if the request was successful
  if response.status_code == 200:
      # Open a local file in binary write mode and save the content
      with open('file.gz', 'wb') as file:
          file.write(response.content)
      print("File downloaded successfully.")
  else:
      print(f"Failed to download file. HTTP Status code: {response.status_code}")
  # Read the file
  seql_list=[]
  with gzip.open("file.gz", "rt") as handle:
      for record in SeqIO.parse(handle, "fasta"):
          seqj=str(record.seq)
          seql_list.append(seqj)
  return seql_list

exon_wt_seqs = downloadReadFastaFile(exon_wt_snp_fp)
exon_mut_seqs = downloadReadFastaFile(exon_mut_snp_fp)
intron_wt_seqs = downloadReadFastaFile(intron_wt_snp_fp)
intron_mut_seqs = downloadReadFastaFile(intron_mut_snp_fp)
```

> <question-title></question-title>
>
> 1. How many sequences are in the data?
> 2. How long are the sequences?
>
> > <solution-title></solution-title>
> >
> > 1. ```python 
> > len( exon_wt_seqs ) , len(exon_mut_seqs) , len(intron_wt_seqs) , len( intron_mut_seqs ) 
> > ```
> >  `(10000, 10000, 10000, 10000)`
> > 2. the sequences are 201 nucleotides long.
> {: .solution}
>
{: .question}

We will keep only the 100 first sequences:

```python
kseq = 100
exon_wt_seqs = exon_wt_seqs[0:kseq]
exon_mut_seqs = exon_mut_seqs[0:kseq]
intron_wt_seqs = intron_wt_seqs[0:kseq]
intron_mut_seqs = intron_mut_seqs[0:kseq]
```

> <question-title></question-title>
>
> How many differences (SNPs) between reference and alternative sequences are there for first exon sequence?
>
> > <solution-title></solution-title>
> >
> > ```python
> > n_diff = 0
> > for i in range( len( exon_wt_seqs[0] ) ) :
> >     n_diff += exon_mut_seqs[0][i] == exon_wt_seqs[0][i]
> > print(n_diff)
> > ```
> > 
> {: .solution}
>
{: .question}

## Compute effect of SNPs

To compute the effect of SNPs, we need :

1. Compute the embdedding of DNA sequences

  ```python
  def computeEmbedding(seqs):
    tokenized_dna = tokenizer(
      seqs,
      return_tensors="pt",
      padding=True,
    )
    inputs_seqs = tokenized_dna["input_ids"]
    hidden_states = model(inputs_seqs)[0].detach().cpu()
    return torch.max(hidden_states, dim=1)[0]
  ```

2. Compute the L2 distance between reference (without SNP) and alternative (with SNP):

  ```python
  def computeMutationEffect(wt_seqs, mut_seqs):
    wt_embedding = computeEmbedding(wt_seqs)
    mut_embedding = computeEmbedding(mut_seqs)
    return torch.norm(mut_embedding - wt_embedding, dim=1)
  ```


```python
exon_SNP_distL2 = computeMutationEffect(exon_wt_seqs, exon_mut_seqs)
intron_SNP_distL2 = computeMutationEffect(intron_wt_seqs,intron_mut_seqs)
```

> <question-title></question-title>
>
> What are the dimensions of `exon_SNP_distL2` and `intron_SNP_distL2`?
>
> > <solution-title></solution-title>
> >
> > 10,000 each: one value per pair of wt/mutated sequence
> > 
> {: .solution}
>
{: .question}


## Compare SNPs effects between introns and exons

We can now quantify the impact of SNPs and determine if the differences are statistically significant.

To visualize the predicted effects of SNPs, we can use a boxplot to compare the L2 distances for SNPs in exons and introns. The L2 distance serves as a metric for the impact of mutations, with higher distances indicating a more significant effect.

Boxplot of predicted SNP effects

```python
SNPs_distL2 = {"exons": exon_SNP_distL2, "introns": intron_SNP_distL2}

fig, ax = plt.subplots()
ax.boxplot(SNPs_distL2.values())
ax.set_xticklabels(SNPs_distL2.keys())
```

> <question-title></question-title>
>
> ![Box plot comparing two categories: exonSNPs and intronSNPs. The plot displays the distribution of values for each category, with the median indicated by an orange line within each box. The boxes represent the interquartile range, while the whiskers extend to show the range of the data, excluding outliers, which are depicted as individual points above the whiskers. The exonSNPs category shows a higher median and more variability compared to the intronSNPs category.](./images/boxplot.png)
>
> What can we conclude from the boxplot?
>
> > <solution-title></solution-title>
> >
> > From the plot, we observe that the L2 distance is generally higher for SNPs in exons compared to those in introns. This suggests that SNPs in exons have a stronger predicted effect, which aligns with their role in coding regions of genes.
> {: .solution}
>
{: .question}

To determine if the observed differences in L2 distances are statistically significant, we can perform hypothesis tests to compare the distributions:

- T-test:

  ```python
  sp.stats.ttest_ind(exon_SNP_distL2, intron_SNP_distL2)
  ```

  > <question-title></question-title>
  >
  > What can we conclude from the t-test?
  >
  > > <solution-title></solution-title>
  > >
  > > The t-test yields a p-value of approximately $$7.56 \cdot 10^{−7}$$, indicating a statistically significant difference between the L2 distances of SNPs in exons and introns.
  > {: .solution}
  >
  {: .question}

- Wilcoxon rank-sum test:

  ```python
  sp.stats.wilcoxon(exon_SNP_distL2, intron_SNP_distL2)
  ```

  > <question-title></question-title>
  >
  > What can we conclude from the Wilcoxon test?
  >
  > > <solution-title></solution-title>
  > >
  > > The Wilcoxon rank-sum test also shows a significant p-value of approximately $$2.87 \cdot 10^{−6}$$, confirming that the distributions are significantly different.
  > {: .solution}
  >
  {: .question}

The analysis demonstrates that SNPs in exons have a more substantial impact on the sequence embeddings compared to SNPs in introns, as evidenced by higher L2 distances. This finding is statistically significant, highlighting the importance of exonic SNPs in potentially altering gene function and expression. By understanding these differences, researchers can gain insights into the functional consequences of genetic variations in different genomic regions.

# Conclusion

Throughout this tutorial, we explored a comprehensive workflow for analyzing DNA sequences and assessing the impact of mutations using a pre-trained DNA language model. We began by tokenizing DNA sequences, converting them into numerical representations that the model could process and analyze effectively. Following this, we loaded and configured a pre-trained model to handle these sequences, ensuring seamless integration with the tokenizer. We then delved into comparing the effects of mutations, both with and without amino acid modifications, showcasing the model's ability to discern subtle differences in sequence impacts. Furthermore, we focused on analyzing the effects of Single Nucleotide Polymorphisms (SNPs) in exons and introns. By loading the relevant sequences, computing the L2 distances between embeddings of wild-type and mutated sequences, and visualizing the results, we quantified and compared the impact of SNPs in these regions. Our analysis revealed that SNPs in exons generally have a more significant impact than those in introns, a finding supported by statistical tests. This tutorial underscores the power of pre-trained models in bioinformatics, offering valuable insights into the functional consequences of genetic variations and paving the way for further research in genomics.

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python", "deep-learning-without-gai-with-python", "genomic-llm-pretraining", "genomic-llm-finetuning"]


topic: statistics
tutorial: text-mining_simtext
title: Text-mining with the SimText toolset
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 1382ce363a8d2cc89fa78e593b629fc4
last_modified: 2026-05-27T09:01:49.846523
zenodo_link: https://doi.org/10.5281/zenodo.4638516
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/text-mining_simtext/tutorial.html

content:
Literature exploration in [PubMed](https://pubmed.ncbi.nlm.nih.gov/) on a large number of biomedical entities (e.g., genes, diseases, or experiments) can be time-consuming and challenging, especially when assessing associations between entities. Here, we use SimText, a toolset for literature research that allows you to collect text from PubMed for any given set of biomedical entities, extract associated terms, and analyze similarities among them and their key characteristics in an interactive tool.

This tutorial is based on a proof-of-concept example given in {% cite gramm2020 %}. We are going to analyze similarities among 95 genes based on their associated biomedical terms in the literature, and compare their pre-existing disorder categories to their grouping based on the literature.

The workflow combines 3 main steps, starting with the retrieval of PubMed data for each of the genes. We then use the PubMed data from each gene to extract related scientific terms that are all combined in one large binary matrix. Finally, we explore the generated data in an interactive tool that performs different unsupervised machine-learning algorithms to analyze the similarities/ the grouping among the genes based on their extracted terms from the literature.

![Tutorial overview](images/simtext_overview_tutorial.png "Schematic presentation of the workflow.")

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Input data

The input data is a simple table with the genes we want to analyze as well as their pre-existing grouping (the grouping is required later on to compare it to our text-based gene grouping). In order for the tools to recognize the column with the biomedical entities of interest, our 95 genes, the column name should start with "ID_", and for the grouping variable with "GROUPING_".

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the input file from [Zenodo](https://zenodo.org/records/4638516)
>
>    ```
>    https://zenodo.org/records/4638516/files/clingen_data
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

# Retrieval of PubMed data

In the first step we collect PubMed data for each of the genes. The genes are used as search queries to download a defined number of PMIDs, here up to 500, from PubMed. The PMIDs are saved in additional columns of our input data.

> <details-title>NCBI API key (optional)</details-title>
>
> To speed up the the download of PubMed data users can obtain an API key from the settings page of their NCBI account (to create an account, visit http://www.ncbi.nlm.nih.gov/account/) and add it to the Galaxy user-preferences (User → Preferences → Manage Information).
>
{: .details}

> <hands-on-title>Step 1: PubMed query tool</hands-on-title>
>
> 1. Run {% tool [PubMed query](toolshed.g2.bx.psu.edu/repos/iuc/pubmed_by_queries/pubmed_by_queries/0.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Input file with query terms"*: Input dataset
>    - *"Number of PMIDs (or abstracts) to save per ID"*: `500`
>
>    > <comment-title></comment-title>
>    >
>    > The tool is also able to save the abstracts as text instead of their PMIDs. This feature is used for another type of analysis (see {% cite gramm2020 %}), or can be used if the user wants to use the tool independent from a workflow to retrieve many abstract texts at once. For the next step of our example in this tutorial, only PMIDs are required.
>    {: .comment}
>
{: .hands_on}

# Extraction of biomedical terms from PubMed abstracts

Next, we extract the 100 most frequent 'Disease' and 'Gene' terms (PubTator annotations) from the PubMed data. All genes with their 100 associated terms are then combined in one large binary matrix. Each row represents a gene and each column one of the extracted terms. This matrix is later used to find similar genes, i.e. genes that have many common terms associated with them.

> <hands-on-title>Extraction of PubTator annotations</hands-on-title>
>
> 1. Run {% tool [PMIDs to PubTator](toolshed.g2.bx.psu.edu/repos/iuc/pmids_to_pubtator_matrix/pmids_to_pubtator_matrix/0.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Input file with PMID IDs"*: output of **PubMed query** {% icon tool %}
>    - *"categories"*: `Genes Diseases`
>    - *"Number of most frequent terms/IDs to extract."*: `100`
>
>    > <comment-title>PubTator</comment-title>
>    > PubTator annotates terms of the following categories: Gene, Disease, Mutation, Species and Chemical.
>    > In this example we chose to only extract gene and disease terms but you can also select other categories if you are interested in those.
>    >
>    {: .comment}
>
{: .hands_on}


# Exploration of data in interactive tool

After generating the large binary matrix, we can explore the similarities/ the grouping among the genes in the interactive SimText tool.
The following features are generated:

1. Word clouds for each gene
2. Dimension reduction and hierarchical clustering of the binary matrix
3. Calculation of the adjusted rand index (similarity between text-based grouping and the pre-existing disorder categories)
4. Table with terms and their frequency among the genes

> <hands-on-title>Explore data interactively</hands-on-title>
>
> 1. Run {% tool [interactive_tool_simtext_app](interactive_tool_simtext_app) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: initial input file with genes and pre-existing grouping
>    - {% icon param-file %} *"Matrix file"*: output of **PMIDs to PubTator** {% icon tool %}
>
> 2. Open interactive tool
>
>    {% snippet faqs/galaxy/interactive_tools_open.md tool="SimText app" %}
>
{: .hands_on}

![Wordcloud](images/simtextapp_wordcloud.gif "Left: For all genes a wordcloud with their associated terms cen be generated. Right: In the dimension reduction plot the similarities among the genes can be explored. The colouring corresponds to the pre-existing disorder categories.")
![Clustering](images/simtextapp_clustering.gif "Hierarchical clustering of the matrix. Different clustering methods can be selected and the adjusted rand index be calculated.")

# Conclusion

tools_json:
["PubMed query", "toolshed.g2.bx.psu.edu/repos/iuc/pubmed_by_queries/pubmed_by_queries/0.0.2", "PMIDs to PubTator", "toolshed.g2.bx.psu.edu/repos/iuc/pmids_to_pubtator_matrix/pmids_to_pubtator_matrix/0.0.2", "interactive_tool_simtext_app"]


topic: statistics
tutorial: fine_tuning_protTrans
title: Fine tune large protein model (ProtTrans) using HuggingFace
difficulty: intermediate
time_estimation: 1H
requirements_json: ["type: internal", "jupyterlab", "type: internal", "gpu_jupyter_lab"]
content_hash: 0acc72a2973feb63a34902681f77a4b9
last_modified: 2026-05-27T09:01:49.825367
zenodo_link: https://zenodo.org/records/10986248
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/fine_tuning_protTrans/tutorial.html

content:
The advent of [large language models](https://en.wikipedia.org/wiki/Large_language_model) has transformed the field of natural language processing, enabling machines to comprehend and generate human-like language with unprecedented accuracy. Pre-trained language models, such as [BERT](https://arxiv.org/abs/1810.04805), [RoBERTa](https://arxiv.org/abs/1907.11692), and their variants, have achieved state-of-the-art results on various tasks, from sentiment analysis and question answering to language translation and text classification. Moreover, the emergence of transformer-based models, such as Generative Pre-trained Transformer ([GPT](https://openai.com/index/gpt-2-1-5b-release/)) and its variants, has enabled the creation of highly advanced language models to generate coherent and context-specific text. The latest iteration of these models, [ChatGPT](https://openai.com/index/chatgpt/), has taken the concept of conversational AI to new heights, allowing users to engage in natural-sounding conversations with machines. However, despite their impressive capabilities, these models are imperfect, and their performance can be significantly improved through fine-tuning. Fine-tuning involves adapting the pre-trained model to a specific task or domain by adjusting its parameters to optimise its performance on a target dataset. This process allows the model to learn task-specific features and relationships that may not be captured by the pre-trained model alone, resulting in highly accurate and specialised language models that can be applied to a wide range of applications. In this tutorial, we will discuss and fine-tune large language model trained on protein sequences [ProtT5](https://github.com/agemagician/ProtTrans/tree/master/Fine-Tuning), exploring the benefits and challenges of this approach, as well as the various techniques and strategies such as low ranking adaptations (LoRA) that can be employed to fit large language models with billions of parameters on regular GPUs. [Protein large language models](https://ieeexplore.ieee.org/document/9477085) (LLMs) represent a significant advancement in Bioinformatics, leveraging the power of deep learning to understand and predict the behaviour of proteins at an unprecedented scale. These models, exemplified by the [ProtTrans](https://github.com/agemagician/ProtTrans) suite, are inspired by natural language processing (NLP) techniques, applying similar methodologies to biological sequences. ProtTrans models, including BERT and T5 adaptations, are trained on vast datasets of protein sequences from databases such as [UniProt](https://www.uniprot.org/) and [BFD](https://bfd.mmseqs.com/), storing millions of protein sequences and enabling them to capture the complex patterns and functions encoded within amino acid sequences. By interpreting these sequences much like languages, protein LLMs offer transformative potential in drug discovery, disease understanding, and synthetic biology, bridging the gap between computational predictions and experimental biology. In this tutorial, we will fine-tune the ProtT5 pre-trained model for [dephosphorylation](https://en.wikipedia.org/wiki/Dephosphorylation) site prediction, a binary classification task.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Fine tuning to predict dephosphorylation sites
Fine-tuning is a transfer learning method where a pre-trained model (e.g. LLM) is further trained on a new, usually smaller dataset to adapt it to a specific task. This process begins with a model trained on a large, general dataset. By leveraging the knowledge already captured in the pre-trained model's weights, fine-tuning allows for more targeted and efficient learning on the new dataset, requiring fewer computational resources and less training time than training a model from scratch. The fine-tuning process typically involves adjusting several hyperparameters, such as learning rate, as the pre-trained model's parameters are updated to fit the new data better. The method is particularly effective in scenarios where the new dataset is too small to train a robust model independently, as it benefits from the general patterns and features learned during the initial (pre) training phase. Fine-tuning keeps a balance between retaining the model's original capabilities and adapting to the specific nuances of the new task, leading to improved performance in a wide range of applications, from natural language processing to computer vision. The protT5 model used in this tutorial has been trained on [UniRef50](https://www.uniprot.org/help/uniref) protein database consisting of 45 million protein sequences. The model captures general features from the large training sequences and is available for further training tasks such as fine-tuning on [HuggingFace](https://huggingface.co/). 


### Dephosphorylation
Dephosphorylation is a biochemical process (post-translational modification) involving removing a phosphate group from an organic compound, typically mediated by phosphatase enzymes. This process regulates cellular functions, including signal transduction, metabolism, and protein activity. By removing phosphate groups from proteins, phosphatases counterbalance the actions of kinases, which add phosphate groups, thus maintaining the dynamic equilibrium of phosphorylation states within the cell. Dephosphorylation can activate or deactivate enzymes and receptors, alter protein-protein interactions, and influence the cellular localisation of proteins. This regulation is essential for many physiological processes, such as cell growth, differentiation, and apoptosis. Disruptions in dephosphorylation mechanisms are associated with numerous diseases, including cancer, diabetes, and neurodegenerative disorders, highlighting the importance of precise control over this process for maintaining cellular health and function. Labelled datasets specifying whether a protein sequence contains dephosphorylation sites are scarce. [Chaudhari et al.](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8264445/) has used deep learning techniques to classify dephosphorylation sites but on a small dataset consisting of around 1,000 protein sequences having dephosphorylation sites at Serine (S), Threonine (T) and Tyrosine (Y). We explore fine-tuning a pre-trained ProtT5 model on a dephosphorylation dataset to showcase an approach for classifying protein sequences. In the tutorial, only those sequences having Y site as dephosphorylated are used for fine-tuning to restrict the training time to a reasonable limit. In the following sections, the GPU-enabled JupyterLab tool in Galaxy is used to fine-tune the ProtT5 model to learn and predict dephosphorylation sites.


## JupyterLab in Galaxy Europe


### Open JupyterLab

> <hands-on-title>GPU-enabled Interactive Jupyter Notebook for Machine Learning</hands-on-title>
>
> - {% tool [GPU-enabled Interactive Jupyter Notebook for Machine Learning](interactive_tool_ml_jupyter_notebook) %}
>    - *"Do you already have a notebook?"*: `Start with a code repository`
>    - *"Online code repository (Git-based) URL"*: `https://github.com/anuprulez/fine-tune-protTrans-repository` 
>    - Click *"Run Tool"*
>
>    > <comment-title></comment-title>
>    > The above step automatically fetches the notebook and datasets from the provided GitHub URL and initiates a JupyterLab.
>    >  If you do not have access to this resource in Galaxy Europe, please apply for it at: [Access GPU-JupyterLab](http://usegalaxy.eu/gpu-request). It may take a day or two to receive access.
>    >
>    {: .comment}
{: .hands_on}


## Fine-tuning notebook
From the cloned repository, open the `fine-tune-protTrans-dephophorylation.ipynb` notebook. The notebook contains all the necessary scripts for processing protein sequences, creating and configuring protein large language models, training it on the protein sequences evaluating them on the test protein sequences and visualising results. Let's look at these key steps of fine-tuning.


### Install necessary Python packages
The protein large language model has been developed using Pytorch and the model weights are stored at HuggingFace. Therefore, packages such as Pytorch, Transformers, and SentencePiece must be installed in the notebook to recreate the model. Additional packages such as Scikit-learn, Pandas, Matplotlib and Seaborn are also required for data preprocessing, manipulation and visualisation of model training and test performances. 	All the necessary packages are installed in the notebook using `!pip install` command. Note: the installed packages have a lifespan equal to the notebook sessions. When a new session of JupyterLab is created, all the packages need to be installed again.

### Fetch and split data
After installing and importing all the necessary packages, protein sequences (available as a FASTA file) and their labels are read into the notebook. These sequences are further divided into training and validation sets. The training set is used for fine-tuning the protein large language model, and the validation set is used for model evaluation after each training epoch.

### Define configurations for LoRA with transformer (ProtT5) model
The protein large language model (ProtT5) used in this tutorial has over 1.2 billion parameters (1,209,193,474). Training such a large model on any commercial GPU with 15GB of memory is impossible. [LoRA](https://arxiv.org/abs/2106.09685), the low-ranking adaption technique, has been devised to make the fine-tuning process feasible on such GPUs. LoRA learns low-rank matrices and, when multiplied, takes the shape of a matrix of the original large language model. During fine-tuning, the weight matrices of the original large language model are kept frozen (not updated) while only these low-rank matrices are updated. Once fine-tuning is finished, these low-rank matrices are combined with the original frozen weight matrices to update the model. The low-rank matrices contain all the knowledge obtained by fine-tuning a small dataset. This approach helps retain the original knowledge of the model while adding the additional knowledge from the fine-tuning dataset. When LoRA is applied to the ProtT5 model, the trainable parameters become a little over 3 million (3,559,426), making it possible to fine-tune on a commercial GPU with at least around 10 GB of memory. The following figure compares [fine-tuning with and without LoRA](https://magazine.sebastianraschka.com/p/lora-and-dora-from-scratch). Fine-tuning without LoRA requires additional weight matrices to be the same size as the original model, which needs much more computational resources than LoRA, where much smaller weight matrices are learned.

![lora_non_lora](images/lora.png "Low ranking adaptations (LoRA).")

### Create ProtT5 model
The ProtT5 model (inspired by [T5](https://huggingface.co/docs/transformers/en/model_doc/t5)) has two significant components - [encoder and sequence classifier](https://github.com/agemagician/ProtTrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb).  Encoder learns a representation of protein sequences, and classifier is used for downstream classification of the learned representations of sequences. The self-attention technique is used to learn sequence representations by computing weights of highly interacting regions in sequences, thereby establishing long-range dependencies. Amino acids in protein sequences are represented in vector spaces in combination with positional embedding to maintain the order of amino acids in sequences.  

### Create a model training method and train
Once the model architecture is created, the weights of the pre-trained ProtT5 are downloaded from [HuggingFace](https://huggingface.co/Rostlab/ProstT5). HuggingFace provides an openly available repository of pre-trained weights of many LLM-like architectures such as ProtT5, [Llama](https://huggingface.co/meta-llama/Meta-Llama-3-8B), [BioGPT](https://huggingface.co/microsoft/BioGPT-Large) and so on. The download of the pre-trained weights is facilitated by a Python package, `Transformers`, which provides methods for downloading weight matrices and tokenisers. After downloading the model weights and tokeniser, the original model is modified by adding LoRA layers to have low-rank matrices and the original weights are frozen. This brings down the number of parameters of the original ProtT5 model from 1.2 billion to 3.5 million. Then, the LoRA updated model is trained for several epochs until the error rate stops decreasing which signifies training stabilisation. Next, the fine-tuned model is saved to a file where it can be reused for prediction.


### Analyse results
The saved trained model is recreated and used to predict the classes of protein sequences from the test set. Different metrics assess fine-tuning performance, such as Matthews’s correlation coefficient (MCC), specificity, sensitivity, accuracy, ROC-AUC and confusion matrix. The higher the value of MCC and closer to 1, the better the correlation between true and predicted classes. High specificity means few false positive results in the predictions, while high sensitivity refers to a few false negatives. Accuracy provides the fraction of correctly predicted sequences from the entire test set. The ROC-AUC metric specifies how well a classifier distinguishes between two classes in a binary classification problem. Its value varies between 0 and 1, where 1 means the perfect classifier and 0.5 means random guessing. The training history plot below shows the error of the fine-tuning process going down and stabilising around training iteration 20.

![training_history_protT5](images/training_history.png "Fine-tuning history of ProtT5 with protein sequences.")

The performance of the fine-tuned model is reasonable, showing ROC-AUC as 0.8 and an accuracy of 0.74, classifying 37 out of 50 sequences correctly. The confusion matrix further elaborates on the fine-tuning results, showing the classification performance of both classes (0 and 1).

![confusion_matrix](images/confusion_matrix.png "Confusion matrix of prediction on test sequences showing performance for both classes.")

## Conclusion
In the tutorial, we have discussed an approach to fine-tune a large language model trained on millions of protein sequences to classify dephosphorylation sites. Using low-ranking adaptation technique, it becomes possible to fine-tune a model having 1.2 billion trainable parameters by reducing it to contain just 3.5 million ones. The availability of the fine-tuning notebook provided with the tutorial and the GPU-JupyterLab infrastructure in Galaxy simplify the complex process of fine-tuning on different datasets. In addition to classification, it is also possible to extract embeddings/representations of entire protein sequences and individual amino acids in protein sequences.

tools_json:
["GPU-enabled Interactive Jupyter Notebook for Machine Learning", "interactive_tool_ml_jupyter_notebook"]

tags_json:
["interactive-tools", "machine-learning", "deep-learning", "jupyter-lab", "fine-tuning", "dephosphorylation-site-prediction"]


topic: statistics
tutorial: intro-to-ml-with-r
title: Introduction to Machine Learning using R
difficulty: intermediate
time_estimation: 3H
tools_json: ["RStudio", "interactive_tool_rstudio"]
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 3efea5f7dcc10aae8a1f773706f680f0
last_modified: 2026-05-15T22:28:17.026841
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/intro-to-ml-with-r/tutorial.html

content:
This is an Introduction to Machine Learning in R, in which you'll learn the basics of unsupervised learning for pattern recognition and supervised learning for prediction. At the end of this workshop, we hope that you will:
- appreciate the importance of performing exploratory data analysis (or EDA) before starting to model your data.
- understand the basics of unsupervised learning and know the examples of principal component analysis (PCA) and k-means clustering.
- understand the basics of supervised learning for prediction and the differences between classification and regression.
- understand modern machine learning techniques and principles, such as test train split, k-fold cross validation and regularization.
- be able to write code to implement the above techniques and methodologies using `R`, `caret` and `glmnet`.

We will not be focusing on the mathematical foundation for each of the methods and approaches we'll be discussing. There are many resources that can provide this context, but for the purposes of this workshop we believe that they are beyond the scope.

_**Note**_: All material here has been adapted from the course material for the Machine Learning course at SIB (22-23/07/2020) {% cite ZENODO.3958880 %}

## Machine Learning basic concepts

Machine Learning (ML) is a subset of Artificial Intelligence (AI) in the field of computer science that often uses statistical techniques to give computers the ability to “learn” (i.e., progressively improve performance on a specific task) with data, without being explicitly programmed.

Machine Learning is often closely related, if not used as an alternate term, to fields like Data Mining (the process of discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems), Pattern Recognition, Statistical Inference or Statistical Learning. All these areas often employ the same methods and perhaps the name changes based on the practitioner’s expertise or the application domain.


## Taxonomy of ML and examples of algorithms

The main ML tasks are typically classified into two broad categories, depending on whether there is "feedback" or a "teacher" available to the learning system or not.

- **Supervised Learning**: The system is presented with example inputs and their desired outputs provided by the “teacher” and the goal of the machine learning algorithm is to create a mapping from the inputs to the outputs. The mapping can be thought of as a function that if it is given as an input one of the training samples it should output the desired value.
- **Unsupervised Learning**: In the unsupervised learning case, the machine learning algorithm is not given any examples of desired output, and is left on its own to find structure in its input.

The main machine learning tasks are separated based on what the system tries to accomplish in the end:
- **Dimensionality Reduction**: simplifies inputs by mapping them into a lower-dimensional space. Topic modeling is a related problem, where a program is given a list of human language documents and is tasked with finding out which documents cover similar topics.
- **Clustering**: a set of inputs is to be divided into groups. Unlike in classification, the groups
are not known beforehand, making this typically an unsupervised task.
- **Classification**: inputs are divided into two or more classes, and the learner must produce a model that assigns unseen inputs to one or more (multi-label classification) of these classes. This is typically tackled in a supervised manner. Identification of patient vs cases is an example of classification, where the inputs are gene expression and/or clinical profiles and the classes are "patient" and "healthy".
- **Regression**: also a supervised problem, the outputs are continuous rather than discrete.
- **Association Rules learning** (or dependency modelling): Searches for relationships between inputs. For example, a supermarket might gather data on customer purchasing habits. Using association rule learning, the supermarket can determine which products are frequently bought together and use this information for marketing purposes. This is sometimes referred to as market basket analysis.

## Overview of Deep learning

Deep learning is a recent trend in machine learning that models highly non-linear representations of data. In the past years, deep learning has gained a tremendous momentum and prevalence for a variety of applications. Among these are image and speech recognition, driverless cars, natural language processing and many more. Interestingly, the majority of mathematical concepts for deep learning have been known for decades. However, it is only through several recent developments that the full potential of deep learning has been unleashed. The success of deep learning has led to a wide range of frameworks and libraries for various programming languages. Examples include `Caffee`, `Theano`, `Torch` and `TensorFlow`, amongst others.

The R programming language has gained considerable popularity among statisticians and data miners for its ease-of-use, as well as its sophisticated visualizations and analyses. With the advent of the deep learning era, the support for deep learning in R has grown ever since, with an increasing number of packages becoming available. This section presents an overview on deep learning in R as provided by the following packages: `MXNetR`, `darch`, `deepnet`, `H2O` and `deepr`. It's important noting that the underlying learning algorithms greatly vary from one package to another. As such, the following table shows a list of the available methods/architectures in each of the packages.

| Package | Available architectures of neural networks |
|---------|--------------------------------------------|
| MXNetR | Feed-forward neural network, convolutional neural network (CNN) |
| darch | Restricted Boltzmann machine, deep belief network |
| deepnet | Feed-forward neural network, restricted Boltzmann machine, deep belief network, stacked autoencoders |
| H2O | Feed-forward neural network, deep autoencoders |
| deepr | Simplify some functions from H2O and deepnet packages |


## Applications of ML in Bioinformatics

There are several biological domains where machine learning techniques are applied for knowledge extraction from data. The following figure (retrieved from {% cite Larra_aga_2006 %}) shows a scheme of the main biological problems where computational methods are being applied.

![A series of overlapping boxes showing intersections of different topics like text mining and proteomics and evolution and microarrays, with various topics listed in the intersections. Unfortunately the source image is too low resolution even for sighted users.](../../images/intro-to-ml-with-r/bioinformatics-ml.png "Classification of the topics where machine learning methods are applied (<a href="https://doi.org/10.1093/bib/bbk007">https://doi.org/10.1093/bib/bbk007</a>)")

> <tip-title>Examples of different Machine Learning / Data Mining techniques that can be applied to different NGS data analysis pipelines.</tip-title>
> An extensive list of examples of applications of Machine Learning in Bioinformatics can be found in the {% cite Larra_aga_2006 %}
{: .tip}

## How to choose the right Machine Learning technique?

Tip 4 in the "Ten quick tips for machine learning in computational biology" ({% cite Chicco_2017 %}) provides a nice overview of what one should keep in mind, when choosing the right Machine Learning technique in Bioinformatics.

> **Which algorithm should you choose to start? In short; The simplest one!**
>
> Once you understand what kind of biological problem you are trying to solve, and which method category can fit your situation, you then have to choose the machine learning algorithm with which to start your project. Even if it always advisable to use multiple techniques and compare their results, the decision on which one to start can be tricky.
>
> Many textbooks suggest to select a machine learning method by just taking into account the problem representation, while Pedro Domingos ("A few useful things to know about machine learning", Commun ACM. 2012; 55(10):78–87) suggests to take into account also the cost evaluation, and the performance optimization.
>
> This algorithm-selection step, which usually occurs at the beginning of a machine learning journey, can be dangerous for beginners. In fact, an inexperienced practitioner might end up choosing a complicated, inappropriate data mining method which might lead him/her to bad results, as well as to lose precious time and energy. Therefore, this is our tip for the algorithm selection: if undecided, start with the simplest algorithm (Hand DJ, "Classifier technology and the illusion of progress". Stat Sci. 2006; 21(1):1–14).
>
> By employing a simple algorithm, you will be able to keep everything under control, and better understand what is happening during the application of the method. In addition, a simple algorithm will provide better generalization skills, less chance of overfitting, easier training and faster learning properties than complex methods. As David J. Hand explained, complex models should be employed only if the dataset features provide some reasonable justification for their usage.
{: .quote cite="{% cite_url Chicco_2017 %}"}


# Exploratory Data Analysis (EDA) and Unsupervised Learning

Before diving in the tutorial, we need to open {% tool [RStudio](interactive_tool_rstudio) %}. If you do not know how or never interacted with RStudio, please follow the [dedicated tutorial]({% link topics/galaxy-interface/tutorials/rstudio/tutorial.md %}).

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

> <hands-on-title>Installing Required Packages</hands-on-title>
>
> 1. Run the following code to install required packages
>
>    ```R
>    ## To install needed CRAN packages:
>    install.packages("tidyverse")
>    install.packages("GGally")
>    install.packages("caret")
>    install.packages("gmodels")
>    install.packages("rpart")
>    install.packages("rpart.plot")
>    install.packages("dendextend")
>    install.packages("randomForest")
>    install.packages("mlr3")
>    install.packages("devtools")
>
>    ## To install needed Bioconductor packages:
>    if (!requireNamespace("BiocManager", quietly = TRUE))
>        install.packages("BiocManager")
>    BiocManager::install()
>    BiocManager::install(c("limma", "edgeR"))
>
>    # To install libraries from GitHub source
>    library(devtools)
>    install_github("vqv/ggbiplot")
>    ```
{: .hands_on}

## Loading and exploring data

The data that we will be using for this workshop are from the following sources:

- The [Breast Cancer Wisconsin (Diagnostic) Data Set](http://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+%28diagnostic%29) from the [UCI Machine Learning repository](http://archive.ics.uci.edu/ml/).
- RNA-Seq data from the study of tooth growth in mouse embryos from the [Gene Expression Omnibus ID:GSE76316](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE76316)

We will first load up the UCI dataset. The dataset itself does not contain column names, we've created a second file with only the column names, which we will use.
We will be using [tidyverse](https://www.tidyverse.org), a collection of R packages for Data Science.

> <hands-on-title>Load the UCI Dataset</hands-on-title>
>
> 1. Load the data
>
>    ```r
>    library(tidyverse) # working with data frames, plotting
>
>    breastCancerData <- read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data",
>                   col_names = FALSE)
>
>    breastCancerDataColNames <- read_csv("https://raw.githubusercontent.com/fpsom/2020-07-machine-learning-sib/master/data/wdbc.colnames.csv",
>                                         col_names = FALSE)
>
>    colnames(breastCancerData) <- breastCancerDataColNames$X1
>    ```
>
> 2. If all goes well, we can see that our dataset contains 569 observations across 32 variables. This is what the first 6 lines look like:
>
>    ```
>    # Check out head of dataframe
>    breastCancerData %>% head()
>
>    # A tibble: 6 x 32
>          ID Diagnosis Radius.Mean Texture.Mean Perimeter.Mean Area.Mean Smoothness.Mean
>       <dbl> <chr>           <dbl>        <dbl>          <dbl>     <dbl>           <dbl>
>    1 8.42e5 M                18.0         10.4          123.      1001           0.118
>    2 8.43e5 M                20.6         17.8          133.      1326           0.0847
>    3 8.43e7 M                19.7         21.2          130       1203           0.110
>    4 8.43e7 M                11.4         20.4           77.6      386.          0.142
>    5 8.44e7 M                20.3         14.3          135.      1297           0.100
>    6 8.44e5 M                12.4         15.7           82.6      477.          0.128
>    # ... with 25 more variables: Compactness.Mean <dbl>, Concavity.Mean <dbl>,
>    #   Concave.Points.Mean <dbl>, Symmetry.Mean <dbl>, Fractal.Dimension.Mean <dbl>,
>    #   Radius.SE <dbl>, Texture.SE <dbl>, Perimeter.SE <dbl>, Area.SE <dbl>,
>    #   Smoothness.SE <dbl>, Compactness.SE <dbl>, Concavity.SE <dbl>, Concave.Points.SE <dbl>,
>    #   Symmetry.SE <dbl>, Fractal.Dimension.SE <dbl>, Radius.Worst <dbl>, Texture.Worst <dbl>,
>    #   Perimeter.Worst <dbl>, Area.Worst <dbl>, Smoothness.Worst <dbl>,
>    #   Compactness.Worst <dbl>, Concavity.Worst <dbl>, Concave.Points.Worst <dbl>,
>    #   Symmetry.Worst <dbl>, Fractal.Dimension.Worst <dbl>
>    ```
>
> 3. We will also make our `Diagnosis` column a factor:
>
>    ```r
>    # Make Diagnosis a factor
>    breastCancerData$Diagnosis <- as.factor(breastCancerData$Diagnosis)
>    ```
>
>    > <question-title></question-title>
>    >
>    > What is a factor?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > TODO
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

## What is Exploratory Data Analysis (EDA) and why is it useful?

Before thinking about modeling, have a look at your data. There is no point in throwing a 10000 layer convolutional neural network (whatever that means) at your data before you even know what you're dealing with.

We will first remove the first column, which is the unique identifier of each row:

> <question-title></question-title>
>
> Why?
>
> > <solution-title></solution-title>
> >
> > TODO
> >
> {: .solution}
{: .question}

> <hands-on-title>Exploratory Data Analysis</hands-on-title>
> 1. Remove the first column
>
>    ```r
>    # Remove first column
>    breastCancerDataNoID <- breastCancerData[2:ncol(breastCancerData)]
>    ```
>
> 2. View the dataset. The output should like like this:
>
>    ```
>    # View head
>    breastCancerDataNoID %>% head()
>
>    # A tibble: 6 x 31
>      Diagnosis Radius.Mean Texture.Mean Perimeter.Mean Area.Mean Smoothness.Mean
>      <fct>           <dbl>        <dbl>          <dbl>     <dbl>           <dbl>
>    1 M                18.0         10.4          123.      1001           0.118
>    2 M                20.6         17.8          133.      1326           0.0847
>    3 M                19.7         21.2          130       1203           0.110
>    4 M                11.4         20.4           77.6      386.          0.142
>    5 M                20.3         14.3          135.      1297           0.100
>    6 M                12.4         15.7           82.6      477.          0.128
>    # ... with 25 more variables: Compactness.Mean <dbl>, Concavity.Mean <dbl>,
>    #   Concave.Points.Mean <dbl>, Symmetry.Mean <dbl>, Fractal.Dimension.Mean <dbl>,
>    #   Radius.SE <dbl>, Texture.SE <dbl>, Perimeter.SE <dbl>, Area.SE <dbl>,
>    #   Smoothness.SE <dbl>, Compactness.SE <dbl>, Concavity.SE <dbl>, Concave.Points.SE <dbl>,
>    #   Symmetry.SE <dbl>, Fractal.Dimension.SE <dbl>, Radius.Worst <dbl>, Texture.Worst <dbl>,
>    #   Perimeter.Worst <dbl>, Area.Worst <dbl>, Smoothness.Worst <dbl>,
>    #   Compactness.Worst <dbl>, Concavity.Worst <dbl>, Concave.Points.Worst <dbl>,
>    #   Symmetry.Worst <dbl>, Fractal.Dimension.Worst <dbl>
>    ```
>
> 3. We have many variables in this dataset. For the interest of time, we will focus only on the first five. Let's have a look at a plot:
>
>    ```r
>    library(GGally)
>
>    ggpairs(breastCancerDataNoID[1:5], aes(color=Diagnosis, alpha=0.4))
>    ```
>
>    ![ggpairs output of the first 5 variables](../../images/intro-to-ml-with-r/ggpairs5variables.png "ggpairs output of the first 5 variables")
>
> 4. Next, we need to center and scale the data.
>
>    Note that the features have widely varying centers and scales (means and standard deviations), so we'll want to center and scale them in some situations. We will use the `[caret](https://cran.r-project.org/web/packages/caret/vignettes/caret.html)` package for this, and specifically, the `preProcess` function.
>
>    The `preProcess` function can be used for many operations on predictors, including centering and scaling. The function `preProcess` estimates the required parameters for each operation and `predict.preProcess` is used to apply them to specific data sets. This function can also be interfaced when calling the `train` function.
>
>    ```r
>    library(caret)
>
>    # Center & scale data
>    ppv <- preProcess(breastCancerDataNoID, method = c("center", "scale"))
>    breastCancerDataNoID_tr <- predict(ppv, breastCancerDataNoID)
>    ```
>
> 5. Let's have a look on the impact of this process by viewing the summary of the first 5 variables before and after the process:
>
>    ```r
>    # Summarize first 5 columns of the original data
>    breastCancerDataNoID[1:5] %>% summary()
>    ```
>
>    It should look like:
>
>    ```
>    Diagnosis  Radius.Mean      Texture.Mean   Perimeter.Mean     Area.Mean
>    B:357     Min.   : 6.981   Min.   : 9.71   Min.   : 43.79   Min.   : 143.5
>    M:212     1st Qu.:11.700   1st Qu.:16.17   1st Qu.: 75.17   1st Qu.: 420.3
>              Median :13.370   Median :18.84   Median : 86.24   Median : 551.1
>              Mean   :14.127   Mean   :19.29   Mean   : 91.97   Mean   : 654.9
>              3rd Qu.:15.780   3rd Qu.:21.80   3rd Qu.:104.10   3rd Qu.: 782.7
>              Max.   :28.110   Max.   :39.28   Max.   :188.50   Max.   :2501.0
>    ```
>
> 6. Let's check the summary of the re-centered and scaled data
>
>    ```r
>    # Summarize first 5 columns of the re-centered and scaled data
>    breastCancerDataNoID_tr[1:5] %>% summary()
>    ```
>
>    It now should look like this:
>
>    ```
>    Diagnosis  Radius.Mean       Texture.Mean     Perimeter.Mean      Area.Mean
>    B:357     Min.   :-2.0279   Min.   :-2.2273   Min.   :-1.9828   Min.   :-1.4532
>    M:212     1st Qu.:-0.6888   1st Qu.:-0.7253   1st Qu.:-0.6913   1st Qu.:-0.6666
>              Median :-0.2149   Median :-0.1045   Median :-0.2358   Median :-0.2949
>              Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.0000
>              3rd Qu.: 0.4690   3rd Qu.: 0.5837   3rd Qu.: 0.4992   3rd Qu.: 0.3632
>              Max.   : 3.9678   Max.   : 4.6478   Max.   : 3.9726   Max.   : 5.2459
>    ```
>
>    As, we can observe here, all variables in our new data have a mean of 0 while maintaining the same distribution of the values. However, this also means that the absolute values do not correspond to the "real", original data - and is just a representation of them.
>
> 7. We can also check whether our plot has changed with the new data:
>
>    ```r
>    library(GGally)
>
>    ggpairs(breastCancerDataNoID_tr[1:5], aes(color=Diagnosis, alpha=0.4))
>    ```
>
>    ![ggpairs output of the first 5 variables of the recentered/rescaled data](../../images/intro-to-ml-with-r/ggpairs5variables_tr.png "ggpairs output of the first 5 variables of the recentered/rescaled data")
>
>    > <question-title></question-title>
>    >
>    > Do you see any differences?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > TODO
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

## Unsupervised Learning

### Dimensionality Reduction and PCA

**Machine learning** is the science and art of giving computers the ability to learn to make decisions from data without being explicitly programmed.

**Unsupervised learning**, in essence, is the machine learning task of uncovering hidden patterns and structures from **unlabeled data**. For example, a researcher might want to group their samples into distinct groups, based on their gene expression data without in advance what these categories maybe. This is known as **clustering**, one branch of unsupervised learning.

**Supervised learning** (which will be addressed later in depth), is the branch of machine learning that involves **predicting labels**, such as whether a tumor will be benign or malignant.

Another form of unsupervised learning, is dimensionality reduction; in the UCI dataset, for example, there are too many features to keep track of. What if we could reduce the number of features yet still keep much of the information?

Principal component analysis (PCA) is one of the most commonly used methods of dimensionality reduction, and extracts the features with the largest variance. What PCA essentially does is the following:
- The first step of PCA is to decorrelate your data and this corresponds to a linear transformation of the vector space your data lie in;
- The second step is the actual dimension reduction; what is really happening is that your decorrelation step (the first step above) transforms the features into new and uncorrelated features; this second step then chooses the features that contain most of the information about the data.


> <hands-on-title>Dimensionality Reduction & PCA</hands-on-title>
>
> 1. Let's have a look into the variables that we currently have, and apply PCA to them. As you can see, we will be using only the numerical variables (i.e. we will exclude the first two, `ID` and `Diagnosis`):
>    ```r
>    ppv_pca <- prcomp(breastCancerData[3:ncol(breastCancerData)], center = TRUE, scale. = TRUE)
>    ```
>
> 2. We can use the `summary()` function to get a summary of the PCA:
>
>    ```r
>    summary(ppv_pca)
>    ```
>
>    The resulting table, shows us the importance of each Principal Component; the standard deviation, the proportion of the variance that it captures, as well as the cumulative proportion of variance capture by the principal components.
>
>    ```
>    Importance of components:
>                              PC1    PC2     PC3     PC4     PC5     PC6     PC7     PC8    PC9
>    Standard deviation     3.6444 2.3857 1.67867 1.40735 1.28403 1.09880 0.82172 0.69037 0.6457
>    Proportion of Variance 0.4427 0.1897 0.09393 0.06602 0.05496 0.04025 0.02251 0.01589 0.0139
>    Cumulative Proportion  0.4427 0.6324 0.72636 0.79239 0.84734 0.88759 0.91010 0.92598 0.9399
>                              PC10   PC11    PC12    PC13    PC14    PC15    PC16    PC17
>    Standard deviation     0.59219 0.5421 0.51104 0.49128 0.39624 0.30681 0.28260 0.24372
>    Proportion of Variance 0.01169 0.0098 0.00871 0.00805 0.00523 0.00314 0.00266 0.00198
>    Cumulative Proportion  0.95157 0.9614 0.97007 0.97812 0.98335 0.98649 0.98915 0.99113
>                              PC18    PC19    PC20   PC21    PC22    PC23   PC24    PC25    PC26
>    Standard deviation     0.22939 0.22244 0.17652 0.1731 0.16565 0.15602 0.1344 0.12442 0.09043
>    Proportion of Variance 0.00175 0.00165 0.00104 0.0010 0.00091 0.00081 0.0006 0.00052 0.00027
>    Cumulative Proportion  0.99288 0.99453 0.99557 0.9966 0.99749 0.99830 0.9989 0.99942 0.99969
>                              PC27    PC28    PC29    PC30
>    Standard deviation     0.08307 0.03987 0.02736 0.01153
>    Proportion of Variance 0.00023 0.00005 0.00002 0.00000
>    Cumulative Proportion  0.99992 0.99997 1.00000 1.00000
>    ```
{: .hands_on}


Principal Components are the underlying structure in the data. They are the directions where there is the most variance, the directions where the data is most spread out. This means that we try to find the straight line that best spreads the data out when it is projected along it. This is the first principal component, the straight line that shows the most substantial variance in the data.

PCA is a type of linear transformation on a given data set that has values for a certain number of variables (coordinates) for a certain amount of spaces. In this way, you transform a set of `x` correlated variables over `y` samples to a set of `p` uncorrelated principal components over the same samples.

Where many variables correlate with one another, they will all contribute strongly to the same principal component. Where your initial variables are strongly correlated with one another, you will be able to approximate most of the complexity in your dataset with just a few principal components. As you add more principal components, you summarize more and more of the original dataset. Adding additional components makes your estimate of the total dataset more accurate, but also more unwieldy.

Every eigenvector has a corresponding eigenvalue. Simply put, an eigenvector is a direction, such as "vertical" or "45 degrees", while an eigenvalue is a number telling you how much variance there is in the data in that direction. The eigenvector with the highest eigenvalue is, therefore, the first principal component. The number of eigenvalues and eigenvectors that exits is equal to the number of dimensions the data set has. In our case, we had 30 variables (32 original, minus the first two), so we have produced 30 eigenvectors / PCs. And we can see that we can address more than 95% of the variance (0.95157) using only the first 10 PCs.

> <hands-on-title>Deeper look into PCA</hands-on-title>
>
> 1. We should also have a deeper look in our PCA object:
>
>    ```r
>    str(ppv_pca)
>    ```
>
>    The output should look like this:
>
>    ```
>    List of 5
>     $ sdev    : num [1:30] 3.64 2.39 1.68 1.41 1.28 ...
>     $ rotation: num [1:30, 1:30] -0.219 -0.104 -0.228 -0.221 -0.143 ...
>      ..- attr(*, "dimnames")=List of 2
>      .. ..$ : chr [1:30] "Radius.Mean" "Texture.Mean" "Perimeter.Mean" "Area.Mean" ...
>      .. ..$ : chr [1:30] "PC1" "PC2" "PC3" "PC4" ...
>     $ center  : Named num [1:30] 14.1273 19.2896 91.969 654.8891 0.0964 ...
>      ..- attr(*, "names")= chr [1:30] "Radius.Mean" "Texture.Mean" "Perimeter.Mean" "Area.Mean" ...
>     $ scale   : Named num [1:30] 3.524 4.301 24.299 351.9141 0.0141 ...
>      ..- attr(*, "names")= chr [1:30] "Radius.Mean" "Texture.Mean" "Perimeter.Mean" "Area.Mean" ...
>     $ x       : num [1:569, 1:30] -9.18 -2.39 -5.73 -7.12 -3.93 ...
>      ..- attr(*, "dimnames")=List of 2
>      .. ..$ : NULL
>      .. ..$ : chr [1:30] "PC1" "PC2" "PC3" "PC4" ...
>     - attr(*, "class")= chr "prcomp"
>    ```
>
> 2. The information listed captures the following:
>
>    1. The center point (`$center`), scaling (`$scale`) and the standard deviation(`$sdev`) of each original variable
>    2. The relationship (correlation or anticorrelation, etc) between the initial variables and the principal components (`$rotation`)
>    3. The values of each sample in terms of the principal components (`$x`)
>
>    Let's try to visualize the results we've got so far. We will be using the [`ggbiplot` library](https://github.com/vqv/ggbiplot) for this purpose.
>
>    ```r
>    ggbiplot(ppv_pca, choices=c(1, 2),
>             labels=rownames(breastCancerData),
>             ellipse=TRUE,
>             groups = breastCancerData$Diagnosis,
>             obs.scale = 1,
>             var.axes=TRUE, var.scale = 1) +
>      ggtitle("PCA of Breast Cancer Dataset")+
>      theme_minimal()+
>      theme(legend.position = "bottom")
>    ```
>
>    ![Visualization of the first two PCs on the UCI Breast Cancer dataset](../../images/intro-to-ml-with-r/pc12Visualization_Full.png "Visualization of the first two PCs on the UCI Breast Cancer dataset")
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Try changing the parameters of the plot. For example, check the `choices` and the `var.scale`. Is there an impact? What does this mean?
> 2. We have been using the entire table of data. What if we restrict our analysis on the `mean` values (i.e. columns 3-12)? Is there an impact?
>
> > <solution-title></solution-title>
> >
> > TODO
> >
> {: .solution}
{: .question}

### Clustering

One popular technique in unsupervised learning is clustering. As the name itself suggests, Clustering algorithms group a set of data points into subsets or clusters. The algorithms' goal is to create clusters that are coherent internally, but clearly different from each other externally. In other words, entities within a cluster should be as similar as possible and entities in one cluster should be as dissimilar as possible from entities in another.

Broadly speaking there are two ways of clustering data points based on the algorithmic structure and operation, namely agglomerative and divisive.

- **Agglomerative**: An agglomerative approach begins with each observation in a distinct (singleton) cluster, and successively merges clusters together until a stopping criterion is satisfied.
- **Divisive**: A divisive method begins with all patterns in a single cluster and performs splitting until a stopping criterion is met.

Essentially, this is the task of grouping your data points, based on something about them, such as closeness in space. Clustering is more of a tool to help you explore a dataset, and should not always be used as an automatic method to classify data. Hence, you may not always deploy a clustering algorithm for real-world production scenario. They are often too unreliable, and a single clustering alone will not be able to give you all the information you can extract from a dataset.

### K-Means

What we are going to do is group the tumor data points into two clusters using an algorithm called `k-means`, which aims to cluster the data in order to minimize the variances of the clusters. The basic idea behind k-means clustering consists of defining clusters so that the total intra-cluster variation (known as total within-cluster variation) is minimized. There are several k-means algorithms available. However, the standard algorithm defines the total within-cluster variation as the sum of squared distances Euclidean distances between items and the corresponding centroid:

> <hands-on-title>Let's cluster our data</hands-on-title>
>
> 1. Let's cluster our data points (ignoring their know classes) using k-means and then we'll compare the results to the actual labels that we know:
>
>    ```r
>    set.seed(1)
>    km.out <- kmeans(breastCancerData[3:ncol(breastCancerData)], centers=2, nstart=20)
>    ```
>
>    The `nstart` option attempts multiple initial configurations and reports on the best one within the kmeans function. Seeds allow us to create a starting point for randomly generated numbers, so that each time our code is run, the same answer is generated.
>    Also, note that k-means requires the number of clusters to be defined beforehand and given via the `centers` option.
>
> 2. Let's check now what the output contains:
>
>    ```r
>    str(km.out)
>    ```
>
>    The output will be:
>
>    ```
>    List of 9
>     $ cluster     : int [1:569] 2 2 2 1 2 1 2 1 1 1 ...
>     $ centers     : num [1:2, 1:30] 12.6 19.4 18.6 21.7 81.1 ...
>      ..- attr(*, "dimnames")=List of 2
>      .. ..$ : chr [1:2] "1" "2"
>      .. ..$ : chr [1:30] "Radius.Mean" "Texture.Mean" "Perimeter.Mean" "Area.Mean" ...
>     $ totss       : num 2.57e+08
>     $ withinss    : num [1:2] 28559677 49383423
>     $ tot.withinss: num 77943100
>     $ betweenss   : num 1.79e+08
>     $ size        : int [1:2] 438 131
>     $ iter        : int 1
>     $ ifault      : int 0
>     - attr(*, "class")= chr "kmeans"
>    ```
>
>    The information contained here is:
>    - `$cluster`: a vector of integers (from 1:k) indicating the cluster to which each point is allocated.
>    - `$centers`: a matrix of cluster centers.
>    - `$withinss`: vector of within-cluster sum of squares, one component per cluster.
>    - `$tot.withinss`: total within-cluster sum of squares (i.e. `sum(withinss)`).
>    - `$size`: the number of points in each cluster.
>
> 3. Let's have a look at the clusters, and we will do this in relationship to the principal components we identified earlier:
>
>    ```r
>    ggplot(as.data.frame(ppv_pca$x), aes(x=PC1, y=PC2, color=as.factor(km.out$cluster), shape = breastCancerData$Diagnosis)) +
>      geom_point( alpha = 0.6, size = 3) +
>      theme_minimal()+
>      theme(legend.position = "bottom") +
>      labs(title = "K-Means clusters against PCA", x = "PC1", y = "PC2", color = "Cluster", shape = "Diagnosis")
>    ```
>
>    ![Visualization of the k-means results against the first two PCs on the UCI Breast Cancer dataset](../../images/intro-to-ml-with-r/kmeans-pc12-Visualization.png "Visualization of the k-means results against the first two PCs on the UCI Breast Cancer dataset")
>
>    This is a rather complex plotting command that is based on the `ggplot` library. For an overview of how `ggplot` works, have a look at the [RNA Seq Counts to Viz in R]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-viz-in-r/tutorial.md %}) tutorial.
>
> 4. Now that we have a cluster for each tumor (clusters 1 and 2), we can check how well they coincide with the labels that we know. To do this we will use a cool method called **cross-tabulation**: a cross-tab is a table that allows you to read off how many data points in clusters 1 and 2 were actually benign or malignant respectively.
>
>    ```r
>    # Cross-tab of clustering & known labels
>    library(gmodels)
>    CrossTable(breastCancerData$Diagnosis, km.out$cluster)
>    ```
>
>    The output should look like this:
>
>    ```
>    Cell Contents
>    |-------------------------|
>    |                       N |
>    | Chi-square contribution |
>    |           N / Row Total |
>    |           N / Col Total |
>    |         N / Table Total |
>    |-------------------------|
>
>
>    Total Observations in Table:  569
>
>
>                               | km.out$cluster
>    breastCancerData$Diagnosis |         1 |         2 | Row Total |
>    ---------------------------|-----------|-----------|-----------|
>                             B |       356 |         1 |       357 |
>                               |    23.988 |    80.204 |           |
>                               |     0.997 |     0.003 |     0.627 |
>                               |     0.813 |     0.008 |           |
>                               |     0.626 |     0.002 |           |
>    ---------------------------|-----------|-----------|-----------|
>                             M |        82 |       130 |       212 |
>                               |    40.395 |   135.060 |           |
>                               |     0.387 |     0.613 |     0.373 |
>                               |     0.187 |     0.992 |           |
>                               |     0.144 |     0.228 |           |
>    ---------------------------|-----------|-----------|-----------|
>                  Column Total |       438 |       131 |       569 |
>                               |     0.770 |     0.230 |           |
>    ---------------------------|-----------|-----------|-----------|
>    ```
>
>    _Question: **How well did the clustering work?**_
>
{: .hands_on}

#### Optimal k

One technique to choose the best `k` is called the **elbow method**. This method uses within-group homogeneity or within-group heterogeneity to evaluate the variability. In other words, you are interested in the percentage of the variance explained by each cluster. You can expect the variability to increase with the number of clusters, alternatively, heterogeneity decreases. Our challenge is to find the `k` that is beyond the diminishing returns. Adding a new cluster does not improve the variability in the data because very few information is left to explain.

> <hands-on-title>Finding the optimal k</hands-on-title>
>
> 1. First of all, let's create a function that computes the total within clusters sum of squares:
>
>    ```r
>    kmean_withinss <- function(k) {
>      cluster <- kmeans(breastCancerData[3:ncol(breastCancerData)], k)
>      return (cluster$tot.withinss)
>    }
>    ```
>
> 2. We can try for a single `k` (e.g. 2), and see the value:
>
>    ```r
>    kmean_withinss(2)
>    ```
>
>    ```
>    [1] 77943100
>    ```
>
> 3. However, we need to test this `n` times. We will use the `sapply()` function to run the algorithm over a range of `k`. This technique is faster than creating a loop and store the value each time.
>
>    ```r
>    # Set maximum cluster
>    max_k <-20
>    # Run algorithm over a range of k
>    wss <- sapply(2:max_k, kmean_withinss)
>    ```
>
> 4. Finally, let's save the results into a data frame, so that we can work with it:
>
>    ```r
>    # Create a data frame to plot the graph
>    elbow <-data.frame(2:max_k, wss)
>    ```
>
> 5. Now that we have the data, we can plot them and try to identify the "elbow" point:
>
>    ```r
>    # Plot the graph with gglop
>    ggplot(elbow, aes(x = X2.max_k, y = wss)) +
>      geom_point() +
>      geom_line() +
>      scale_x_continuous(breaks = seq(1, 20, by = 1))
>    ```
>
>    ![Elbow plot for multiple values of k](../../images/intro-to-ml-with-r/elbow-plot-kmeans.png "Elbow plot for multiple values of k")
>
>    > <question-title></question-title>
>    >
>    > What is the optimal `k` value?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > From the graph, you can see the optimal `k` is around 10, where the curve is starting to have a diminishing return.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Try re-running the clustering step with the new k. Is there a significant difference?
> 2. Try to think of alternative metrics that could be used as a "distance" measure, instead of the default "Euclidean". Do you think there might be an optimal for our case?
>
> > <solution-title></solution-title>
> >
> > TODO
> >
> {: .solution}
{: .question}

### Hierarchical clustering

k-means clustering requires us to specify the number of clusters, and determining the optimal number of clusters is often not trivial. Hierarchical clustering is an alternative approach which builds a hierarchy from the bottom-up, and doesn’t require us to specify the number of clusters beforehand but requires extra steps to extract final clusters.
The algorithm works as follows:

- Put each data point in its own cluster.
- Identify the closest two clusters and combine them into one cluster.
- Repeat the above step till all the data points are in a single cluster.

Once this is done, it is usually represented by a dendrogram like structure. There are a few ways to determine how close two clusters are:

1. **Complete linkage clustering**: Find the maximum possible distance between points belonging to two different clusters.
2. **Single linkage clustering**: Find the minimum possible distance between points belonging to two different clusters.
3. **Mean linkage clustering**: Find all possible pairwise distances for points belonging to two different clusters and then calculate the average.
4. **Centroid linkage clustering**: Find the centroid of each cluster and calculate the distance between centroids of two clusters.


> <hands-on-title>k-means Clustering</hands-on-title>
> 1. We will be applying Hierarchical clustering to our dataset, and see what the result might be. Remember that our dataset has some columns with nominal (categorical) values (columns `ID` and `Diagnosis`), so we will need to make sure we only use the columns with numerical values. There are no missing values in this dataset that we need to clean before clustering. But the scales of the features are different and we need to normalize it.
>
>    ```r
>    breastCancerDataScaled <- as.data.frame(scale(breastCancerData[3:ncol(breastCancerData)]))
>    summary(breastCancerDataScaled)
>    ```
>
> 2. We can now proceed with creating the distance matrix:
>
>    ```r
>    dist_mat <- dist(breastCancerDataScaled, method = 'euclidean')
>    ```
>
>    There are several options for `method`: `euclidean`, `maximum`, `manhattan`, `canberra`, `binary` or `minkowski`.
>
> 3. The next step is to actually perform the hierarchical clustering, which means that at this point we should decide which linkage method we want to use. We can try all kinds of linkage methods and later decide on which one performed better. Here we will proceed with `average` linkage method (i.e. UPGMA); other methods include `ward.D`, `ward.D2`, `single`, `complete`, `mcquitty` (= WPGMA), `median` (= WPGMC) and `centroid` (= UPGMC).
>
>    ```r
>    hclust_avg <- hclust(dist_mat, method = 'average')
>
>    plot(hclust_avg)
>    ```
>
>    ![Hierarchical clustering (attempt 1)](../../images/intro-to-ml-with-r/hclust-fig1.png "Hierarchical clustering (attempt 1)")
>
>    Notice how the dendrogram is built and every data point finally merges into a single cluster with the height(distance) shown on the y-axis.
>
> 4. Next, we can cut the dendrogram in order to create the desired number of clusters. In our case, we might want to check whether our two groups (`M` and `B`) can be identified as sub-trees of our clustering - so we'll set `k = 2` and then plot the result.
>
>    ```r
>    cut_avg <- cutree(hclust_avg, k = 2)
>
>    plot(hclust_avg, labels = breastCancerData$ID, hang = -1, cex = 0.2,
>         main = "Cluster dendrogram (k = 2)", xlab = "Breast Cancer ID", ylab = "Height")
>    # k: Cut the dendrogram such that exactly k clusters are produced
>    # border: Vector with border colors for the rectangles. Coild also be a number vector 1:2
>    # which: A vector selecting the clusters around which a rectangle should be drawn (numbered from left to right)
>    rect.hclust(hclust_avg , k = 2, border = c("red","green"), which = c(1, 2))
>    # Draw a line at the height that the cut takes place
>    abline(h = 18, col = 'red', lwd=3, lty=2)
>    ```
>    ![Hierarchical clustering (attempt 2)](../../images/intro-to-ml-with-r/hclust-fig2.png "Hierarchical clustering (attempt 2)")
>
> 5. Now we can see the two clusters enclosed in two different colored boxes. We can also use the `color_branches()` function from the `dendextend` library to visualize our tree with different colored branches.
>
>    ```r
>    library(dendextend)
>    avg_dend_obj <- as.dendrogram(hclust_avg)
>    # We can use either k (number of clusters), or clusters (and specify the cluster type)
>    avg_col_dend <- color_branches(avg_dend_obj, k = 2, groupLabels=TRUE)
>    plot(avg_col_dend, main = "Cluster dendrogram with color per cluster (k = 2)", xlab = "Breast Cancer ID", ylab = "Height")
>    ```
>
>    ![Hierarchical clustering (attempt 3)](../../images/intro-to-ml-with-r/hclust-fig3.png "Hierarchical clustering (attempt 3)")
>
> 6. We can change the way branches are colored, to reflect the `Diagnosis` value:
>
>    ```r
>    avg_col_dend <- color_branches(avg_dend_obj, clusters = breastCancerData$Diagnosis)
>    plot(avg_col_dend, main = "Cluster dendrogram with Diagnosis color", xlab = "Breast Cancer ID", ylab = "Height")
>    ```
>
>    ![Hierarchical clustering (attempt 4)](../../images/intro-to-ml-with-r/hclust-fig4.png "Hierarchical clustering (attempt 4)")
>
> 7. TODO? Step Title
>
>    ```r
>    ggplot(as.data.frame(ppv_pca$x), aes(x=PC1, y=PC2, color=as.factor(cut_avg), shape = breastCancerData$Diagnosis)) +
>      geom_point( alpha = 0.6, size = 3) +
>      theme_minimal()+
>      theme(legend.position = "bottom") +
>      labs(title = "Hierarchical clustering (cut at k=2) against PCA", x = "PC1", y = "PC2", color = "Cluster", shape = "Diagnosis")
>    ```
>
>    ![Visualization of the Hierarchical clustering (cut at k=2) results against the first two PCs on the UCI Breast Cancer dataset](../../images/intro-to-ml-with-r/hclust-pc12-Visualization.png "Visualization of the Hierarchical clustering (cut at k=2) results against the first two PCs on the UCI Breast Cancer dataset")
>
{: .hands_on}


> <question-title></question-title>
>
> 1. The hierarchical clustering performed so far, only used two methods: `euclidean` and `average`. Try experimenting with different methods. Do the final results improve?
> 2. Obviously the cut-off selection (k=2) was not optimal. Try using different cut-offs to ensure that the final clustering could provide some context to the original question.
>
> > <solution-title></solution-title>
> >
> > TODO
> >
> {: .solution}
{: .question}

# Supervised Learning

Supervised learning is the branch of Machine Learning (ML) that involves predicting labels, such as 'Survived' or 'Not'. Such models learn from labelled data, which is data that includes whether a passenger survived (called "model training"), and then predict on unlabeled data.

These are generally called train and test sets because
- You want to build a model that learns patterns in the training set, and
- You then use the model to make predictions on the test set.

We can then calculate the percentage that you got correct: this is known as the accuracy of your model.

## How To Start with Supervised Learning

As you might already know, a good way to approach supervised learning is the following:
- Perform an Exploratory Data Analysis (EDA) on your data set;
- Build a quick and dirty model, or a baseline model, which can serve as a comparison against later models that you will build;
- Iterate this process. You will do more EDA and build another model;
- Engineer features: take the features that you already have and combine them or extract more information from them to eventually come to the last point, which is
- Get a model that performs better.

A common practice in all supervised learning is the construction and use of the **train- and test- datasets**. This process takes all of the input randomly splits into the two datasets (training and test); the ratio of the split is usually up to the researcher, and can be anything: 80/20, 70/30, 60/40...

## Supervised Learning I: classification

There are various classifiers available:

- **Decision Trees** – These are organized in the form of sets of questions and answers in the tree structure.
- **Naive Bayes Classifiers** – A probabilistic machine learning model that is used for classification.
- **K-NN Classifiers** – Based on the similarity measures like distance, it classifies new cases.
- **Support Vector Machines** – It is a non-probabilistic binary classifier that builds a model to classify a case into one of the two categories. They rely on a `kernel` function that essentially projects the data points to higher-dimensional space; depending on this new space, there can be both linear and non-linear SVMs.

### Decision trees

It is a type of supervised learning algorithm. We use it for classification problems. It works for both types of input and output variables. In this technique, we split the population into two or more homogeneous sets. Moreover, it is based on the most significant splitter/differentiator in input variables.

The Decision Tree is a powerful non-linear classifier. A Decision Tree makes use of a tree-like structure to generate relationship among the various features and potential outcomes. It makes use of branching decisions as its core structure.

There are two types of decision trees:
- **Categorical (classification)** Variable Decision Tree: Decision Tree which has a categorical target variable.
- **Continuous (Regression)** Variable Decision Tree: Decision Tree has a continuous target variable.

Regression trees are used when the dependent variable is continuous while classification trees are used when the dependent variable is categorical. In continuous, a value obtained is a mean response of observation. In classification, a value obtained by a terminal node is a mode of observations.


> <hands-on-title>Decision Trees</hands-on-title>
> 1. Here, we will use the `rpart` and the `rpart.plot` package in order to produce and visualize a decision tree. First of all, we'll create the train and test datasets using a 70/30 ratio and a fixed seed so that we can reproduce the results.
>
>    ```r
>    # split into training and test subsets
>    set.seed(1000)
>    ind <- sample(2, nrow(breastCancerData), replace=TRUE, prob=c(0.7, 0.3))
>    breastCancerData.train <- breastCancerDataNoID[ind==1,]
>    breastCancerData.test <- breastCancerDataNoID[ind==2,]
>    ```
>
> 2. Now, we will load the library and create our model. We would like to create a model that predicts the `Diagnosis` based on the mean of the radius and the area, as well as the SE of the texture. For ths reason we'll use the notation of `myFormula <- Diagnosis ~ Radius.Mean + Area.Mean + Texture.SE`. If we wanted to create a prediction model based on all variables, we will have used `myFormula <- Diagnosis ~ .` instead. Finally, `minsplit` stands for the the minimum number of instances in a node so that it is split.
>
>    ```r
>    library(rpart)
>    library(rpart.plot)
>    myFormula <- Diagnosis ~ Radius.Mean + Area.Mean + Texture.SE
>
>    breastCancerData.model <- rpart(myFormula,
>                                    method = "class",
>                                    data = breastCancerData.train,
>                                    minsplit = 10,
>                                    minbucket = 1,
>                                    maxdepth = 3,
>                                    cp = -1)
>
>    print(breastCancerData.model$cptable)
>    rpart.plot(breastCancerData.model)
>    ```
>
> 3. We see the following output and a figu


topic: statistics
tutorial: age-prediction-with-ml
title: Age prediction using machine learning
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: dba15224824c97ed7f176f3b38b02371
last_modified: 2026-05-27T09:01:49.813519
zenodo_link: https://zenodo.org/record/2545213
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/age-prediction-with-ml/tutorial.html

content:
[Machine Learning](https://en.wikipedia.org/wiki/Machine_learning) is used to create predictive models by learning features from datasets. In the studies performed by [Jason G. Fleischer et al. 2018](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1599-6#Sec9) and [Jana Naue et al. 2017](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub), biomarkers are examined to predict the chronological age of humans by analysing the RNA-seq gene expression levels and DNA methylation pattern respectively. Different machine learning algorithms are used in these studies to select specific biomarkers to make age prediction. The RNA-seq gene expression ([FPKM](https://www.ebi.ac.uk/training/online/glossary/fpkm)) dataset is generated using fibroblast cell lines of humans. The skin fibroblasts cells keep damage that happens with age. Epigenomic and phenotypic changes which are age-dependent are also contained in these cells. Within each individual, [DNA methylation](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3174260/) changes with age. This knowledge is used to select useful biomarkers from DNA methylation dataset. The [CpGs sites](https://en.wikipedia.org/wiki/CpG_site) with the highest correlation to age are selected as the biomarkers/features. In both these studies, specific biomarkers are analysed by machine learning algorithms to create an age prediction model.

The datasets from these studies contain features (present as columns). The last column in both the datasets refers to `age` which is used as labels/targets. Since the targets are real numbers, the machine learning task becomes [regression](https://en.wikipedia.org/wiki/Regression_analysis). Using these features and targets, a model is learned using machine learning (regression) which learns a mapping between these features and targets. Using machine learning tools in Galaxy, we can achieve comparable prediction scores to those achieved by published analyses. In this tutorial, we will apply a couple of ([scikit-learn](https://scikit-learn.org/stable/)) machine learning tools to [RNA-seq](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1599-6#Sec9) and [DNA methylation](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub) datasets to predict the chronological age of humans. This tutorial is divided into two parts - one with an RNA-seq dataset and another with a DNA methylation dataset.

> <comment-title></comment-title>
> In cross-validation, a dataset is divided into `k` equal parts (`k` should be at least `2`). One part is used as the validation set to evaluate the performance of any machine learning algorithm and the remaining parts are used for learning/training.
{: .comment}

#### R2 (coefficient of determination)
In both the parts, learning on datasets is done using cross-validation and the [R2](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html) scoring metric is used to evaluate the performance of the trained model. The closer it is to 1.0, the better it is. If it is negative, then the trained model is not good. To infer how its values exhibit model performance, we can compare the figures [1](#figure-1) and [2](#figure-2). In both the plots, the true and predicted targets are plotted in a scatter plot. For a good model, most of the points should lie along the `x = y` line as the true and predicted targets are close to each other. In figure [1](#figure-1), we can see that the points are scattered and do not show any pattern. Therefore, the R2 score is `-0.06`. But, figure [2](#figure-2) shows a better pattern as most of the points lie along the line and the R2 score is almost `1.0`. For RNA-seq dataset, we will compute the cross-validated R2 score using the training set and for DNA methylation dataset, we will compute the R2 score for the test set.

![model_bad](../../images/age-prediction-with-ml/model_bad.png "This shows an example of a bad model as most of the points are scattered. For a model to be good, most of the points should lie along the x = y  line. It means that the predicted target on the y (vertical) axis is close to the true targets on the x (horizontal) axis.")

![model_good](../../images/age-prediction-with-ml/model_good.png "This shows an example of a good model as most of the points lie along the x = y line. Because most of the points lie along the x = y line, R2 score is 0.99 very close to the best score of 1.0")


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Analyze RNA-seq dataset

The RNA-seq dataset is collected from fibroblast cell lines belonging to 133 healthy patients with age ranging from 1 to 94 years. The skin fibroblasts cells dataset is conducive for age prediction related studies for multiple reasons - these skin fibroblasts hold age-related damage, contain phenotypic, epigenomic and transcriptomic changes which are age-dependent and it is easy to collect this dataset using non-invasive techniques. Using this dataset, we perform an [exhaustive search](https://en.wikipedia.org/wiki/Hyperparameter_optimization) (also known as grid search) for finding the best features and then apply [ElasticNet regressor](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html#sklearn.linear_model.ElasticNet) with 5-fold cross-validation. The R2 score achieved is comparable to the predictions found in the study performed by [Jason G. Fleischer et al. 2018](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1599-6#Sec9).

> <details-title>5-fold cross-validation</details-title>
>
> It is a model validation technique which estimates the performance of a predictive model on an unseen data. A dataset is divided into `5` folds and these folds are categorised into training and validation sets. The idea of cross-validation is shown in figure [3](#figure-3). The complete dataset is divided into `5` equal parts. 80% of the dataset is used for training and the remaining 20% is used for validating the performance of training. This is done for `5` folds/iterations, each time the validation set (20% of the dataset) is different. In all five folds, the complete dataset is used for training and validation. The final validation performance is averaged over `5` folds.
>
> ![5fold_cv](../../images/age-prediction-with-ml/5fold_cv.png "The image shows how the 5-fold cross-validation works. The complete dataset is divided into 5 equal parts/folds. 4 parts (80%) of the data (training set shown in yellow) are used for training the model and the remaining one part is used for evaluating (validation set shown in blue) the trained model. This is repeated for 5 times till every part/fold is used as the validation set. The accuracies computed for different validation folds are averaged to give 5-fold cross-validation accuracy.")
>
{: .details}

## Get dataset

We proceed to the analysis by uploading the RNA-seq dataset. The dataset has `133` rows corresponding to human patients and over `27,000` columns specifying genes. The last column `age` contains the chronological age. For the validation set, this `age` column is predicted and the R2 metric is computed.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/record/2545213/files/training_data_normal.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the dataset to `training_data_normal`.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

> <comment-title></comment-title>
> In the `training_data_normal` dataset, you might have noticed that it showed `134` rows instead of `133` rows. The first row contains the header information i.e. the description of each column.
{: .comment}

## Create data processing pipeline

We can see that this RNA-seq dataset is high-dimensional. There are over `27,000` columns/features. Generally, not all the features in the dataset are useful for prediction. We need only those features which increase the predictive ability of the model. To filter these features, we perform feature selection and retain only those which are useful. To do that, we use [SelectKBest](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest) module. This approach involves extracting those features which are most correlated to the target (`age` in our dataset). [F-regression](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression) is used for the extraction of features. Moreover, we are not sure of how many of these features we will need. To find the right number of features, we do a hyperparameter search (to find the best combination of values of different parameters). It works by setting many different numbers of features to determine to number which gives the highest accuracy. To wrap this feature selector with a regressor, we will use the **Pipeline builder** tool. This tool creates a sequential flow of algorithms to execute on datasets. Since the hyperparameters will be tuned, we choose to output the parameters for searchCV. The tool does not take any dataset as input. Rather, the outputs will be used as inputs to the **Hyperparameter search** tool (explained in the following step). We will use ElasticNet as a regressor which creates an age prediction model. It is a linear regressor with `l1` (also called lasso) and `l2` (also called ridge) as regularisers. Regularisation is a technique used in machine learning to prevent overfitting. Overfitting happens when a machine learning algorithm starts memorising the dataset it is trained upon, rather than learning general features. The consequence of overfitting is that the accuracy on the training set is good, but results on the unseen set (test set) are poor, which happens because the algorithm has not learned general features from the dataset. To prevent overfitting, regularisers like `l1` and `l2` are used. `L1` is a linear term added to the error function of a machine learning algorithm and `l2` adds a squared term to the error function. [More details about `l1` and `l2` can found here](https://www.kaggle.com/residentmario/l1-norms-versus-l2-norms).

> <hands-on-title>Create pipeline</hands-on-title>
>
> 1. **Pipeline builder** {% icon tool %} with the following parameters:
>    - In *"1: Pre-processing step"*:
>        - *"Choose the type of transformation"*: `Feature Selection`
>            - *"Select a feature selection algorithm"*: `SelectKBest - Select features according to the k highest scores`
>                - *"Select a score function"*: `f_regression - Univariate linear regression tests`
>    - In *"Final Estimator"*:
>        - *"Choose the module that contains target estimator"*: `sklearn.linear_model`
>            - *"Choose estimator class"*: `ElasticNet`
>            - *"Type in parameter settings if different from default"*: `random_state=42`
>    - In *"Output parameters for searchCV?"*: `Yes`
>
{: .hands_on}

## Optimise hyperparameters

In any machine learning algorithm, there are many parameters (hyperparameters). We are not sure which values of these parameters will give an optimal prediction. The default values given for these parameters may not be optimal for different datasets. To find the best combination of the values of different parameters for a dataset, [hyperparameter optimisation](https://en.wikipedia.org/wiki/Hyperparameter_optimization) is performed. There are different techniques to optimise the hyperparameters of any algorithm given a dataset:
- [Grid search](https://scikit-learn.org/0.16/modules/generated/sklearn.grid_search.GridSearchCV.html)
- [Random search](https://scikit-learn.org/0.16/modules/generated/sklearn.grid_search.RandomizedSearchCV.html#sklearn.grid_search.RandomizedSearchCV)

For our analyses, we will use the grid search approach. It is an exhaustive search which tries out all the combinations of different hyperparameters and ranks these combinations based on a scoring metric. In the random search, the values of a parameter are selected randomly from a given range and the best one is found. Grid search works well for parameters taking categorical as well as numerical values but for the random search, it becomes difficult for parameters which take categorical values.

In the **Pipeline builder** tool, we added two steps - preprocessing (feature selection) and an estimator (regressor). There are different hyperparameters for these two steps and their best combination needs to be found. We will perform grid search to estimate the best values for these parameters: **k** (number of features), **normalize** (subtract the mean and divide by the l2-norm of the dataset) and **alpha** (a constant which is multiplied by the regularisation term). For each parameter, we need to specify a set of values to choose from:

- **k**: [5880, 5890, 5895, 5900]

    These values of `k` are chosen to get the best accuracy. We can choose any number (integers) between `1` and `27,000` (maximum number of features in the dataset). We will use only these values (shown above) for `k` as the accuracy remains the best around these numbers. However, it may vary for a different RNA-seq dataset. That's the reason why we perform hyperparameter search to find the best values of parameters for any dataset.

- **normalize**: [True, False]

    The default value of `normalize` is `False`. We will check both, `True` and `False`.

- **alpha**: [0.00001, 0.0001, 0.001]

    The parameter `alpha` takes a positive real number and its default value is `1.0`.

For these three parameters, we have 24 different combinations (4 x 2 x 3) of values and we will verify the performance of each combination. The parameter **k** is used for feature selection and parameters **normalize** and **alpha** are used for the regressor. There are many more hyperparameters of the [ElasticNet](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html#sklearn.linear_model.ElasticNet) regressor which are explained in the official documentation of scikit-learn. However, the combination of the above three parameters already gives a comparable accuracy published in the study [Jason G. Fleischer et al. 2018](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1599-6#Sec9). Therefore, we will stick to these parameters.

> <comment-title></comment-title>
> It is advisable to tune all the parameters of a machine learning algorithm for a dataset if no prior information is available about the subset of parameters which works best for the dataset.
{: .comment}

> <comment-title></comment-title>
> These parameters have the same description and values in the second part of the tutorial where we will again use the **Hyperparameter search** tool.
{: .comment}

### Search for the best values of hyperparameters

We will use the **Hyperparameter search** tool to find the best values for each hyperparameter. These values will lead us to create the best model based on the search space chosen for each hyperparameter.

> <hands-on-title>Hyperparameter search</hands-on-title>
>
> 1. **Hyperparameter search** {% icon tool %} with the following parameters:
>    - *"Select a model selection search scheme"*: `GridSearchCV - Exhaustive search over specified parameter values for an estimator `
>        - {% icon param-files %} *"Choose the dataset containing pipeline/estimator object"*: `zipped` file (one of the outputs of **Pipeline builder** {% icon tool %})
>        - In *"Search parameters Builder"*:
>             - {% icon param-files %} *"Choose the dataset containing parameter names"*: `tabular` file (the other output of **Pipeline builder** {% icon tool %})
>             - In *"Parameter settings for search"*:
>                 - {% icon param-repeat %} *"1: Parameter settings for search"*
>                    - *"Choose a parameter name (with current value)"*: `selectkbest__k: 10`
>                    - *"Search list"*: `[5880, 5890, 5895, 5900]`
>                - {% icon param-repeat %} *"2: Parameter settings for search"*
>                    - *"Choose a parameter name (with current value)"*: `elasticnet__normalize: False`
>                    - *"Search list"*: `[True, False]`
>                - {% icon param-repeat %} *"3: Parameter settings for search"*
>                    - *"Choose a parameter name (with current value)"*: `elasticnet__alpha: 1.0`
>                    - *"Search list"*: `[0.00001, 0.0001, 0.001]`
>        - In *"Advanced Options for SearchCV"*:
>            - *"Select the primary metric (scoring)"*: `Regression -- 'r2'`
>
>               A scoring metric can be set. In this tutorial, we use `Regression -- 'r2'`
>
>            - *"Select the cv splitter"*: `KFold`
>
>               There are different ways to split the dataset into training and validation sets. In our tutorial, we will use `KFold` which splits the dataset into `K` consecutive parts. It is used for cross-validation. It is set to `5` using another parameter `n_splits`.
>
>                - *"n_splits"*: `5`
>                - *"Whether to shuffle data before splitting"*: `Yes`
>                - *"Random seed number"*: `3111696`
>
>                   It is set to an integer and used to retain the randomness/accuracy when *"Whether to shuffle data before splitting"* is `Yes` across successive experiments.
>
>            - *"Raise fit error"*: `No`
>
>               While setting different values for a parameter during hyperparameter search, it can happen that wrong values are set which may generate exceptions. To avoid stopping the execution of a regressor, it is set to `No` which means even if a wrong parameter value is encountered, the regressor does not stop running and skips that value.
>
>    - *"Select input type"*: `tabular data`
>        - {% icon param-files %} *"Training samples dataset"*: `training_data_normal` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>            - *"Type header name(s)"*: `age`
>        - {% icon param-files %} *"Dataset containing class labels or target values"*: `training_data_normal` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `Select columns by column header name(s)`
>            - *"Type header name(s)"*: `age`
>    - *"Whether to hold a portion of samples for test exclusively?"*: `Nope`
>
{: .hands_on}

The tool returns two outputs, one of which is a table with numerical results. Please inspect it carefully: the `rank_test_score` column shows the ranking of different combinations based on the values in the `mean_test_score` column.

> <question-title></question-title>
>
> 1. What is the best `mean_test_score` value estimated by the **Hyperparameter search** tool?
> 2. Which combination of parameters gives the best result?
> 3. How many possible combinations of parameters the **Hyperparameter search** tool estimated?
>
> > <solution-title></solution-title>
> >
> > 1. 0.73 (it is close to the best R2 score (0.81) achieved by a customised ensemble algorithm explained by [Jason G. Fleischer et al. 2018](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1599-6#Sec9))
> > 2. alpha: 0.001, normalize: True, k: 5880
> > 3. 24 (it is equal to the number of rows in the tabular output of **Hyperparameter search** {% icon tool %})
> >
> {: .solution}
>
{: .question}

## Create parallel coordinates plot

We will visualize the tabular output of **Hyperparameter search** tool from the previous step using the **Parallel coordinates plot of tabular data** tool. There are multiple columns in the tabular output, but we will focus on only a few of them.

> <hands-on-title>Create parallel coordinates plot</hands-on-title>
>
> 1. **Parallel coordinates plot** {% icon tool %} with the following parameters:
>    - {% icon param-files %} *"Select data file"*: `tabular` file (output of **Hyperparameter search** {% icon tool %})
>    - *"Select the columns for dimensions"*: `c4, c5, c6`
>    - *"Select a column containing the values for coloring"*: `c3`
>
{: .hands_on}

The output plot has the following legend: the colour-coding is based on the `mean_test_score` (`c3`) column. You can follow the line leading to the score along every column with the parameters' settings. The columns `c4, c5` and `c6` are the parameters we chose and `c3` is the accuracy column present in the `tabular` output of **Hyperparameter search** tool.

![data](../../images/age-prediction-with-ml/parallel_coor_plot.png "The visualization of the hyperparameter optimisation tool output. We optimised the values of 3 hyperparameters (alpha, normalize and k). These can be seen as the columns (first three from left to right) in the plot. The rightmost column contains the accuracy values (mean_test_score).")

> <question-title></question-title>
>
> What are the worst performing combinations of hyperparameters (name four)?
>
> > <solution-title></solution-title>
> >
> > Worst four: 
> >    - alpha: 0.00001, normalize: False, k: 5880
> >    - alpha: 0.00001, normalize: False, k: 5890
> >    - alpha: 0.00001, normalize: False, k: 5895
> >    - alpha: 0.00001, normalize: False, k: 5900
> >
> {: .solution}
>
{: .question}

## Compare results with original paper

![rna_seq_age_paper](../../images/age-prediction-with-ml/rna_seq_age.png "The plot shows accuracy obtained in the paper from Jason G. Fleischer et al. 2018 (Predicting age from the transcriptome of human dermal fibroblasts) using ElasticNet regressor. The R2 scores achieved in the tutorial and mentioned in the paper using the same regressor are comparable.")

## Summary

Figure [4](#figure-4) shows that we achieved an R2 score of `0.73` (last column) with 5-fold cross-validation on the training set. In the study [Jason G. Fleischer et al. 2018](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1599-6#Sec9) as well, a similar R2 score is mentioned for linear regressors (Linear regression and ElasticNet). Moreover, the study also included a customised ensemble regressor which achieved better performance (`R2 = 0.81`). However, our analysis showcases the use of machine learning tools in Galaxy to reproduce the results published in the paper.

# Analyze DNA methylation dataset

In the second part of the analysis, we will use the [DNA methylation dataset](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub) to predict chronological age. One important reason to choose this dataset for an age prediction task is that DNA methylation changes with age and this change occurs at specific CpG sites in humans. Whole blood samples are collected from humans with their ages falling in the range 18-69 and the best age-correlated CpG sites in the genome are chosen as features. The dataset is divided into two parts - training and test sets. The training set is used to train a regressor and the test set is used to evaluate the performance of the trained model using the R2 scoring metric. 5-fold cross-validation is used for training.

## Get train and test datasets

We proceed with the analysis by uploading new datasets. You might want to create a new history first. The training set contains `208` rows corresponding to humans and `13` features (age-correlated CpG sites in DNA methylation dataset). The last column is `age`. The test set contains `104` rows and the same number of features as the training set. The `age` column in the test set is predicted after training on the training set. Another dataset `test_rows_labels` contains the true age values of the test set which is used to compute R2 scores between true and predicted age.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo](https://zenodo.org/record/2545213#.XEWTJ9-YVa0)
>
>    ```
>    https://zenodo.org/record/2545213/files/train_rows.csv
>    https://zenodo.org/record/2545213/files/test_rows_labels.csv
>    https://zenodo.org/record/2545213/files/test_rows.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets as `train_rows`, `test_rows_labels` and `test_rows` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype of all the three datasets is `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

The `train_rows` contains a column `Age` which is the label or target. We will evaluate our model on `test_rows` and compare the predicted age with the true age in `test_rows_labels`
{: .comment}

## Create data processing pipeline

We will create a pipeline with **Pipeline builder** tool, but this time, we just specify the regressor. [Jana Naue et al. 2017](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub) has used [Random Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor) as the regressor and we can conclude from this study that the ensemble-based regressor works well on this DNA methylation dataset. Therefore, we will use [Gradient boosting](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor) which is an ensemble-based regressor because it uses multiple decision tree regressors internally and predicts by taking the collective performances of the predictions (by multiple decision trees). It has a good predictive power and is robust to outliers. It creates an ensemble of weak learners (decision trees) and iteratively minimises error. One disadvantage, which comes from its basic principle of boosting, is that it cannot be parallelised. The **Pipeline builder** tool will wrap this regressor and return a zipped file and a tabular file containing all tunable hyperparameters.

> <hands-on-title>Create pipeline</hands-on-title>
>
> 1. **Pipeline builder** {% icon tool %} with the following parameters:
>    - In *"Final Estimator"*:
>        - *"Choose the module that contains target estimator"*: `sklearn.ensemble`
>            - *"Choose estimator class"*: `GradientBoostingRegressor`
>            - *"Type in parameter settings if different from default"*: `random_state=42`
>    - In *"Output parameters for searchCV?"*: `Yes`
> 
>
{: .hands_on}

> <comment-title></comment-title>
> The [*ensemble*](https://scikit-learn.org/stable/modules/ensemble.html#ensemble) method uses multiple learning models internally for better predictions.
{: .comment}

## Optimise hyperparameters

For this analysis, we will also use the **Hyperparameter search** tool to estimate the best values of parameters for the given dataset.
We use only one parameter `n_estimators` of the `Gradient boosting` regressor for this task. This parameter specifies the number of boosting stages the learning process has to go through. The default value of `n_estimators` for this regressor is `100`. However, we are not sure if this gives the best accuracy. Therefore, it is important to set this parameter to different values to find the optimal one. We choose some values which are less than `100` and a few more than `100`. The hyperparameter search will look for the optimal number of estimators and gives the best-trained model as one of the outputs. This model is used in the next step to predict age in the test dataset.

### Search for the best values of hyperparameters

We will use the **Hyperparameter search** tool to find the best values for each hyperparameter. These values will lead us to create the best model based on the search space chosen for each hyperparameter.

> <hands-on-title>Hyperparameter search</hands-on-title>
>
> 1. **Hyperparameter search** {% icon tool %} with the following parameters:
>    - *"Select a model selection search scheme"*: `GridSearchCV - Exhaustive search over specified parameter values for an estimator `
>        - {% icon param-files %} *"Choose the dataset containing pipeline/estimator object"*: `zipped` file (one of the outputs of **Pipeline builder** {% icon tool %})
>        - In *"Search parameters Builder"*:
>             - {% icon param-files %} *"Choose the dataset containing parameter names"*: `tabular` file (the other output of **Pipeline builder** {% icon tool %})
>             - In *"Parameter settings for search"*:
>                 - {% icon param-repeat %} *"1: Parameter settings for search"*
>                    - *"Choose a parameter name (with current value)"*: `n_estimators: 100`
>                    - *"Search list"*: `[25, 50, 75, 100, 200]`
>        - In *"Advanced Options for SearchCV"*:
>            - *"Select the primary metric (scoring)"*: `Regression -- 'r2'`
>
>               A scoring metric can be set. In this tutorial, we use `Regression -- 'r2'`
>
>            - *"Select the cv splitter"*: `KFold`
>
>               There are different ways to split the dataset into training and validation sets. In our tutorial, we will use `KFold` which splits the dataset into `K` consecutive parts. It is used for cross-validation. It is set to `5` using another parameter `n_splits`.
>
>                - *"n_splits"*: `5`
>                - *"Whether to shuffle data before splitting"*: `Yes`
>                - *"Random seed number"*: `3111696`
>
>                   It is set to an integer and used to retain the randomness/accuracy when *"Whether to shuffle data before splitting"* is `Yes` across successive experiments.
>
>            - *"Raise fit error"*: `No`
>
>               While setting different values for a parameter during hyperparameter search, it can happen that wrong values are set which may generate exceptions. To avoid stopping the execution of a regressor, it is set to `No` which means even if a wrong parameter value is encountered, the regressor does not stop running and skips that value.
>
>    - *"Select input type"*: `tabular data`
>        - {% icon param-files %} *"Training samples dataset"*: `train_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>            - *"Type header name(s)"*: `Age`
>        - {% icon param-files %} *"Dataset containing class labels or target values"*: `train_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `Select columns by column header name(s)`
>            - *"Type header name(s)"*: `Age`
>    - *"Whether to hold a portion of samples for test exclusively?"*: `Nope`
>
{: .hands_on}

> <question-title></question-title>
>
> What is the optimal number of estimators for the given dataset?
>
> Hint: Please look at the `mean_test_score` column in the tabular result from the **Hyperparameter search** tool.
>
> > <solution-title></solution-title>
> >
> > 75. (Even though the default value of the number of estimators for Gradient boosting regressor is `100`, `75` gives the best accuracy. That's why it is important to perform hyperparameter search to tune these parameters for any dataset). 50 estimators also give almost the same accuracy.
> >
> {: .solution}
>
{: .question}

## Predict age

Using the **Hyperparameter search** tool, we found the best model based on the training data. Now, we will predict age in the test dataset using this model in order to see if the model has learned important features which can be generalised on a new dataset. The test dataset (`test_rows`) contains the same number of features, but does not contain the `age` column. This is predicted using the trained model.

> <hands-on-title>Predict age</hands-on-title>
>
> 1. **Ensemble methods for classification and regression** {% icon tool %} with the following parameters:
>    - *"Select a Classification Task"*: `Load a model and predict`
>        - {% icon param-files %} *"Models"*: `zipped` file (output of **Hyperparameter search** {% icon tool %})
>        - {% icon param-files %} *"Data (tabular)"*: `test_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>
{: .hands_on}

## Create plots

In the previous step, we generated predictions for the test dataset. We have one more dataset (`test_rows_labels`) which contains the true age values of the test set. Using the true and predicted values of age in the test set, we will verify the performance by analysing the plots.

> <hands-on-title>Create regression plots</hands-on-title>
>
> 1. **Plot actual vs predicted curves and residual plots of tabular data** {% icon tool %} with the following parameters:
>    - {% icon param-files %} *"Select input data file"*: `test_rows_labels` tabular file
>    - {% icon param-files %} *"Select predicted data file"*: `tabular` file (output of **Ensemble methods for classification and regression** {% icon tool %})
>
{: .hands_on}

The tool outputs three HTML files which contain the interactive plots.

> <question-title></question-title>
>
> Inspect the plots. What can you say about the predictions?
>
> > <solution-title></solution-title>
> >
> > Figures [6](#figure-6) and [8](#figure-8) show that the prediction is good because the predicted age lies close to the true age.
> >
> {: .solution}
{: .question}

![Scatter plot](../../images/age-prediction-with-ml/scatter_plot.png "Scatter plot for true and predicted age for test set. It is evident from the plot that most of the points lie along the x = y line, which means that true and predicted ages are close to each other. The root mean squared error in predicting age is 3.76 years and R2 score (0.94) is close to the best score of 1.0.")

We can see in the scatter plot (figure [6](#figure-6)) that most of the points lie along the x=y curve. It means that the true and predicted ages are close to each other. The root mean square error (`RMSE`) is `3.76` and the R2 score is `0.94`.

![Residuals](../../images/age-prediction-with-ml/residual_plot.png "The plot shows the residuals (predicted age - true) age against the predicted age. For a good learning/training, this plot should not show any distinct pattern and the points should be symmetrically distributed along the y = 0 line.")

The [residual plot](http://docs.statwing.com/interpreting-residual-plots-to-improve-your-regression/) shown in figure [7](#figure-7) is generated to see if there is any visible pattern between residual (predicted age - true age) and predicted age. For a good model, there should not be any visible pattern with the plotted points.

![True vs predicted age](../../images/age-prediction-with-ml/true_vs_predicted_plot.png "The plot shows the true vs predicted age for all the samples in the test set. We can see that the predicted values are close to the true values.")

## Compare results with original paper

![dnam_image_paper](../../images/age-prediction-with-ml/dnam_age_paper.png "The plot is from Jana Naue et al. 2017 (Chronological age prediction based on DNA methylation) and shows a scatter plot with the predicted age on the vertical axis and the true age on the horizontal axis. The plot is comparable to the scatter plot shown in figure 6. The RMSE score is also comparable. The paper used Random forest as the regressor and we used Gradient boosting as the regressor.")


## Summary

Figure [6](#figure-6) shows that we achieved an R2 score of `0.94` and root mean square score of `3.76` for the test set using the Gradient boosting regressor. The study of [Jana Naue et al. 2017](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub) also mentions a similar root mean square score (`3.93`), derived using the random forest regressor. The root mean square score shows the difference in the true and predicted age of humans. The R2 score (`0.94`) is close to the best achievable score of `1.0` which shows that the trained model is good. Overall, the second part of the analysis also shows that using the machine learning tools in Galaxy, we can achieve state-of-the-art predictions mentioned in the recent scientific studies.

# Conclusion


In our tutorial, we were able to use machine learning tools in Galaxy to reproduce the results reported in these scientific studies - [Jason G. Fleischer et al. 2018](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-018-1599-6#Sec9) and [Jana Naue et al. 2017](https://www.sciencedirect.com/science/article/pii/S1872497317301643?via%3Dihub). We also learned how to work with high-dimensional datasets, perform a hyperparameter search and cross-validation. Further, we can reuse the trained models to make predictions on a new dataset provided that this new dataset has the same features. There are numerous other machine learning algorithms available in Galaxy which can also be tried out on these datasets to verify whether the accuracy can be improved.


topic: statistics
tutorial: aberrant_pi3k_pathway_analysis
title: PAPAA PI3K_OG: PanCancer Aberrant Pathway Activity Analysis
difficulty: intermediate
time_estimation: 1H30M
requirements_json: []
tags_json: ["Machine learning", "Pan-cancer", "cancer biomarkers", "oncogenes and tumor suppressor genes"]
content_hash: a33895a435b255731391e28208736a48
last_modified: 2026-05-15T22:28:17.004695
zenodo_link: https://zenodo.org/record/4306639
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/aberrant_pi3k_pathway_analysis/tutorial.html

content:
Signaling pathways are among the most commonly altered across different tumor types. Many tumors possess at least one driver alteration and nearly half of such alterations are potentially targeted by currently available drugs. A recent study in TCGA tumors has identified patterns of somatic variations and mechanisms in 10 canonical pathways
({% cite SanchezVega2018 %}). One-third of these tumors possess multiple alterations and have potentially complex phenotypes. Identifying a transcriptomic signature in these tumors would enable personalized therapeutic design strategies. A plethora of evidence suggests complex diseases, like cancer, can be the result of multiple genetic aberrations in biological networks or pathways rather than variation in a single gene. Often starter mutations occur in a key component network that ultimately leads to multi-gene dysregulation causing hallmark cancer phenotypes ({% cite Hanahan2000 %}). Many of these phenotypes are the result of disrupted transcriptional programs that affect the clinical progression and therapeutic responsiveness. Recent progress in exploring these transcriptomic changes in cancer pathogenesis provided useful clues in precision medicine ({% cite Bradner2017 %}).

The RTK/RAS/PI3K molecular genetic axis controls critical cellular functions and is commonly altered in various cancers ({% cite Fruman2014 %}). Perturbations across this axis can lead to deficiencies in cell-cycle, survival, metabolism, motility and genome stability, triggering hallmark phenotypes of cancer. The constitutive activation and presence of phosphatidylinositol-3,4,5-trisphosphate (PIP3) trigger membrane-bound onco-signalosomes. This presents significant challenges for treatment, as PI3K cascade can be activated in several ways ({% cite Zhao2006 %}).

Cancer driver genes comprising both oncogenes (OG) and Tumor suppressor genes (TSG) share common phenotypical outcome. However, they often have divergent molecular mechanisms  that drive the outcome. We are interested in capturing mutational specific differential transcriptional outcome among OG and TSG. In Figure 1, genes in red are oncogenes (have activating or copy gain) and genes in blue are tumor suppressor genes (have inactivating or copy loss).

In this tutorial we plan to measure aberrant PI3K pathway activity in TCGA dataset using RNA-Seq information and mutational and copy number information of following frequently altered genes. We named this tutorial as PanCancer Aberrant Pathway Activity Analysis (PAPAA).

![pi3k_pathway](../../images/aberrant_pi3k_pathway_analysis/pi3k_pathway.png "Genes used in this study from ERK/RAS/PI3K pathway. Red text indicates Oncogenes (OG), blue text indicates Tumor suppressors genes (TSG).")

> <agenda-title></agenda-title>
>
> Let's learn how to install this tutorial using Docker with galaxy, and how to download the datasets from Zenodo :
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pre-installed tutorial tools, datasets and workflows from the docker image

An efficient way to install and run the tutorial using papaa tools is available on docker based galaxy instance that has pre-installed papaa tool-suite as **papaa** under tools section. Additionally this local galaxy instance comes with datasets and workflow for generating PI3K_OG classifier. Instructions to run the docker image is below.

> <hands-on-title>Tutorial for galaxy docker container installation and running the workflow:</hands-on-title>
> 1. Pulling the docker image from docker hub: Open a terminal and type the following command:
>       ```docker pull nvk747/galaxy_papaa:0.1.9```
> 2. For running the docker container use the following command :
>       ```docker run -p 8888:80 --detach nvk747/galaxy_papaa:0.1.9```
> 3. Open the following [*http://localhost:8888*](http://localhost:8888) in your web browser.
> 4. Use *'admin'* as username and *'password'* as password to login as **admin** into the docker galaxy instance.
> 5. The Workflow **'papaa@0.1.9_PI3K_OG_model_tutorial'** from workflow section to run PI3K_OG tutorial is pre-installed.
> 6. Instructions to get datasets from shared library is given in Hands-on: Data upload.
{: .hands_on}

# Get data for analysis from classifier to pharmacological correlations

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [PAPAA-Zenodo](https://zenodo.org/record/4306639) or from the docker galaxy `Shared Data -> Data Libraries -> (GTN - Material -> {{ page.topic_name }} -> {{ page.title }} )`
>
>    ```
>    https://zenodo.org/records/4306639/files/CCLE_DepMap_18Q1_maf_20180207.txt.gz
>    https://zenodo.org/records/4306639/files/CCLE_MUT_CNA_AMP_DEL_binary_Revealer.tsv.gz
>    https://zenodo.org/records/4306639/files/ccle_rnaseq_genes_rpkm_20180929_mod.tsv.gz
>    https://zenodo.org/records/4306639/files/compounds_of_interest.txt
>    https://zenodo.org/records/4306639/files/copy_number_gain_status.tsv.gz
>    https://zenodo.org/records/4306639/files/copy_number_loss_status.tsv.gz
>    https://zenodo.org/records/4306639/files/cosmic_cancer_classification.tsv
>    https://zenodo.org/records/4306639/files/gdsc1_ccle_pharm_fitted_dose_data.txt.gz
>    https://zenodo.org/records/4306639/files/gdsc2_ccle_pharm_fitted_dose_data.txt.gz
>    https://zenodo.org/records/4306639/files/GDSC_CCLE_common_mut_cnv_binary.tsv.gz
>    https://zenodo.org/records/4306639/files/GDSC_EXP_CCLE_converted_name.tsv.gz
>    https://zenodo.org/records/4306639/files/GSE69822_pi3k_sign.txt
>    https://zenodo.org/records/4306639/files/GSE69822_pi3k_trans.csv
>    https://zenodo.org/records/4306639/files/path_rtk_ras_pi3k_genes.txt
>    https://zenodo.org/records/4306639/files/mc3.v0.2.8.PUBLIC.maf.gz
>    https://zenodo.org/records/4306639/files/mutation_burden_freeze.tsv
>    https://zenodo.org/records/4306639/files/pancan_mutation_freeze.tsv.gz
>    https://zenodo.org/records/4306639/files/pancan_rnaseq_freeze.tsv.gz
>    https://zenodo.org/records/4306639/files/sample_freeze.tsv
>    https://zenodo.org/records/4306639/files/tcga_dictionary.tsv
>    ```
{: .hands_on}

> <hands-on-title>Some additional files that you can use for alternative analysis</hands-on-title>
> 1. These files are provided to optionally enable you to perform an additional analysis, using the same methodology
>    described here, but are not required for the main tutorial.
> 2. Import the files from [PAPAA-Zenodo](https://zenodo.org/record/4306639) or from the docker galaxy `Shared Data -> Data Libraries -> (GTN - Material -> {{ page.topic_name }} -> {{ page.title }} )`
>
>    ```
>    https://zenodo.org/records/4306639/files/GSE94937_kras_sign.txt
>    https://zenodo.org/records/4306639/files/GSE94937_rpkm_kras.csv
>    https://zenodo.org/records/4306639/files/path_cell_cycle_genes.txt
>    https://zenodo.org/records/4306639/files/path_myc_genes.txt
>    https://zenodo.org/records/4306639/files/path_ras_genes.txt
>    https://zenodo.org/records/4306639/files/path_wnt_genes.txt
>    ```
{: .hands_on}

> <comment-title>Note:</comment-title>
> Some of the imported files may still contain a **.gz** extension but only uncompressed files are used in our analysis. So kindly ignore the extension!!
{: .comment}

> <details-title>Datasets descriptions</details-title>
>
> -    **pancan_rnaseq_freeze.tsv:** Publicly available gene expression data for the TCGA Pan-cancer dataset. This file has gene-expression data for ~20,000 genes (columns) in ~10,000 samples (rows).
> -    **pancan_mutation_freeze.tsv:** Publicly available Mutational information for TCGA Pan-cancer dataset. This file has mutational data for all genes (columns) as binary valued (0/1) in all samples (rows).
> -    **mutation_burden_freeze.tsv:** Publicly available Mutational information for TCGA Pan-cancer dataset. This file has mutational burden information for all samples(rows).
> -    **sample_freeze.tsv:** The file lists the frozen samples as determined by TCGA PanCancer Atlas consortium along with raw RNA-Seq and mutation data. These were previously determined and included for all downstream analysis All other datasets were processed and subset according to the frozen samples.
> -    **cosmic_cancer_classification.tsv:** Compendium of OG and TSG used for the analysis. This file has list of cancer genes(rows) from [cosmic database](https://cancer.sanger.ac.uk/cosmic) classified as Oncogene or tumor suppressor (columns).
> -    **CCLE_DepMap_18Q1_maf_20180207.txt:** Publicly available Mutational data for CCLE cell lines from Broad Institute Cancer Cell Line Encyclopedia [CCLE](https://portals.broadinstitute.org/ccle)/[DepMap Portal](https://depmap.org/portal/). Variant classification along with nucleotide and protein level changes are provided in the columns for genes(rows).
> -    **ccle_rnaseq_genes_rpkm_20180929_mod.tsv:** Publicly available Expression data for 1,019 cell lines (RPKM) from Broad Institute Cancer Cell Line Encyclopedia (CCLE) / DepMap Portal. This file has gene-expression data for genes(rows) in various cell lines (columns).
> -    **CCLE_MUT_CNA_AMP_DEL_binary_Revealer.tsv:** Publicly available merged Mutational and copy number alterations that include gene amplifications and deletions for the CCLE cell lines. This file has mutational/copy number variation data for all cancer genes (rows) as binary valued (0/1) in all CCLE cell lines (columns).
> -    **GDSC_cell_lines_EXP_CCLE_names.tsv:** Publicly available RMA normalized expression data for Genomics of Drug Sensitivity in Cancer [GDSC](https://www.cancerrxgene.org/) cell-lines. This data was subset to 382 cell lines that are common among CCLE and GDSC. This file has gene-expression data for genes(rows) in various cell lines (columns).
> -    **GDSC_CCLE_common_mut_cnv_binary.tsv:** A subset of merged Mutational and copy number alterations that include gene amplifications and deletions for common cell lines between GDSC and CCLE.
> -    **gdsc1_ccle_pharm_fitted_dose_data.txt:** Pharmacological data for GDSC-1 cell lines. This data was subset to 379 cell lines that are common among CCLE and GDSC. This file has pharmacological data like IC50, Z-scores, drug information, concentrations used, AUC(columns) of 304 tested compounds in various cell-lines(rows).
> -    **gdsc2_ccle_pharm_fitted_dose_data.txt:** Pharmacological data for GDSC-2 cell lines. This data was subset to 347 cell lines that are common among CCLE and GDSC. This file has pharmacological data like IC50, Z-scores, drug information, concentrations used, AUC(columns) of 170 tested compounds in various cell-lines(rows).
> -    **compounds_of_interest.txt:** This file contains the compounds of interest for generation of pharmacological correlations with classifier scores. List of inhibitor compounds against EGFR-signaling, ERK-MAPK-signaling, Other-kinases, PI3K/MTOR-signaling, and RTK-signaling pathways.
> -    **tcga_dictonary.tsv:** List of cancer types used in the analysis.
> -    **GSE69822_pi3k_sign.txt:** File with values assigned for tumor [1] or normal [-1] for external data samples deposited in Gene Expression Omnibus database accession:[GSE69822](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE69822).
> -    **GSE69822_pi3k_trans.csv:** Variant stabilized transformed values for the RNA expression levels in the external samples from [GSE69822](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE69822).
> -    **path_rtk_ras_pi3k_genes.txt:** List of genes belong to RTK,RAS,PI3K used in our study.
> -    **path_myc_genes.txt:** List of genes belong to Myc signaling pathway [Sanchez Vega et.al 2018](https://www.cell.com/action/showPdf?pii=S0092-8674%2818%2930359-3).
> -    **path_ras_genes.txt:** List of genes belong to Ras signaling Pathway [Sanchez Vega et.al 2018](https://www.cell.com/action/showPdf?pii=S0092-8674%2818%2930359-3).
> -    **path_cell_cycle_genes.txt:** List of genes belong to cell cycle pathway [Sanchez Vega et.al 2018](https://www.cell.com/action/showPdf?pii=S0092-8674%2818%2930359-3).
> -    **path_wnt_genes.txt:** List of genes belong to wnt signaling pathway [Sanchez Vega et.al 2018](https://www.cell.com/action/showPdf?pii=S0092-8674%2818%2930359-3).
> -    **GSE94937_rpkm_kras.csv:** RNA expression levels in the external samples from [GSE94937](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE94937).
> -    **GSE94937_kras_sign.txt:** File with values assigned for tumor [1] or normal [-1] for external data samples deposited in Gene Expression Omnibus database accession: [GSE94937](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE94937).
{:.details}

# PanCancer aberrant pathway activity analysis (PAPAA)

[Machine Learning](https://en.wikipedia.org/wiki/Machine_learning) algorithms  learn from various features in datasets and generate predictive models. Extracting transcriptional patterns and learning insights of tumor physiology from gene expression data is a developing research area. Transcriptional profiling was used to identify differentially expressed genes and pathways associated with drug resistance in breast cancer ({% cite Men2017 %}). Hyperactive oncogenic pathway  signatures can be useful in predicting sensitivity to therapeutic agents ({% cite Bild2005 %}). Machine learning-based modeling provides a systematic manner to leverage these multi-omic data to predict phenotype or stratify tumors based on gene expression and pathway variations. We extended a previously developed elastic net penalized logistic regression classification modeling approach to derive transcription signature or pathway alterations to measure aberrant PI3K activity in the PanCancer data ({% cite Way2018 %}). This method integrates bulk RNA Sequencing (RNA-Seq), copy number and mutation data from [PanCanAtlas](https://gdc.cancer.gov/about-data/publications/pancanatlas).

TCGA Pancancer has uniformly processed multi-omic data including RNA-Seq, copy number and mutational data. It covers 33 different cancer types and having information from over 10000 samples. We used publicly available RNA-Seq, mutation and CNV data sets from TCGA. Description and processing details of these data sets are listed at this site: [PanCancer aberrant pathway activity analysis](https://github.com/nvk747/papaa.git).

## Machine learning methodology
Logistic regression is a kind of machine learning approach where statistical analysis is used to predict the outcome of a dependent variable based on observed changes in other variables. e.g. Changes in gene expression are directly connected to alterations/mutations in genes. We used above approach to predict mutational status given the gene expression. Optimizing to the above prediction of mutational status with gene expression variable, we used elastic net penalty with gradient descent algorithm is used to find the optimal cost function by going over a number of iterations. The objective of the classifier is to determine the probability a given sample (*i*) has a aberrant gene event given the sample’s RNA-Seq measurements (*Xi*). In order to achieve the objective, the classifier learns a vector of coefficients or gene-specific weights (*w*) that optimize the following penalized logistic function.

![Equations for probability measurement ](../../images/aberrant_pi3k_pathway_analysis/equation.png "Equation for prediction of mutational status(Yi) from expression data X(i) for each sample. Mutational status can be estimated by Multiplying Xi with gene specific weights (W). The negative log likelihood (L) is used for calculating minimum weights for each sample")

Where *alpha* and *l* are regularization and elastic net mixing hyperparameters respectively. Their optimal values are found during 5 fold cross validation by using scikit-learn [girdsearchCV](https://scikit-learn.org/0.17/modules/generated/sklearn.grid_search.GridSearchCV.html) approach.

## Sample Processing step:

- **x-matrix:**: Gene-expression data comprises of expression levels for ~20,000 genes/sample and ~10,000 samples. Top 8,000 highly variable genes per sample with in each disease were measured by median absolute deviation (MAD) and considered for analysis.

- **y-matrix:**: Copy number and mutational data as binary valued (0/1) datasets for all samples. This matrix is subset to given pathway target genes and cancer types.

We then randomly held out 10% of the samples to create a test set and rest 90% for training.  The testing set is used as the validation to evaluate the performance of any machine learning algorithm and the remaining parts are used for learning/training. The training set is balanced for different cancer-types and PI3K status.

Predicting probabilities of an observation belonging to each class in a classification problem is more flexible rather than predicting classes directly. This method has an added advantage to trade-off errors made by the model as it depends on interpretation of probabilities using different thresholds. Two diagnostic tools that help in the interpretation of probabilistic forecast for binary (two-class) classification predictive modeling problems are Receiver Operating Characteristic (ROC) Curves and Precision-Recall curves. Each model generated is evaluated and performance metrics are measured using Area Under the Receiver Operating Characteristic (AUROC) and Area Under the Precision-Recall (AUPR).

As elastic net penalty with stochastic decent gradient approach induces sparsity in the number of features used in classification, and the best/top features (genes) are likely to represent transcriptional signature of given disease or aberrant activity of the mutated genes in a pathway.

Each feature (gene) is given a rank and score (negative or positive) depending on its contribution to classification. The positive scored genes are likely to be up-regulated in activated pathway samples and negatively scored genes are likely to be downstream targets of altered pathways.

In this tutorial, we made series of steps to generate classification models and used those models for predicting pharmacological response or identifying potential biomarkers that may be helpful for treatment of various cancers. Here, we generate models using [ERBB2](https://www.genecards.org/cgi-bin/carddisp.pl?gene=ERBB2), [KRAS](https://www.genecards.org/cgi-bin/carddisp.pl?gene=KRAS), [PIK3CA](https://www.genecards.org/cgi-bin/carddisp.pl?gene=PIK3CA), and [AKT1](https://www.genecards.org/cgi-bin/carddisp.pl?gene=AKT1) oncogenes from the ERK/RAS/PI3K signaling axis pathway.  **Have fun!**

## PAPAA: PanCancer classifier
This first step is designed to generate model with ERK/RAS/PI3K signaling axis pathway genes (path_genes): ERBB2, KRAS, PIK3CA ,AKT11 genes and various the Cancer Genome Atlas (TCGA) cancer types/diseases: BLCA, BRCA, CESC, COAD, ESCA, LUAD, LUSC, OV, PRAD, READ, STAD, UCEC, and UCS (ref: tcga_dictionary.tsv). This step takes gene expression information (pancan_rnaseq_freeze.tsv), mutational information (pancan_mutation_freeze.tsv), load of mutations in each samples (mutation_burden_freeze.tsv), threshold passed sample information (sample_freeze.tsv) and copy number information (copy_number_loss_status.tsv & copy_number_gain_status.tsv) and generates PI3K_OG model. AUROC and AUPR metrics are calculated for training,test,CV and random shuffled sets. As an additional option, the generated model was used to evaluate alternative genes (PTEN, PIK3R1, and STK11) and alternative diseases (BRCA, COAD, ESCA, HNSC, LGG, LUAD, LUSC, PRAD, READ, GBM, UCEC, and UCS) performance.

![Prediction metrics for PI3K_OG model](../../images/aberrant_pi3k_pathway_analysis/pi3k_OG_auroc_aupr.png "Prediction metrics of the PI3K_OG classifier- AUROC and AUPR curve for training, test, CV, and random shuffled sets.")
![PI3K_OG Classifier Coefficients](../../images/aberrant_pi3k_pathway_analysis/PI3K_OG_classifier_coef.png "PI3K_OG classifier coefficients represent the genes that impact the classification")

> <hands-on-title>Generating model from ERBB2,PIK3CA,KRAS,AKT1 genes with specific disease types</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer classifier](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_classifier/pancancer_classifier/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"Filename of features to use in model"*: `pancan_rnaseq_freeze.tsv` (Input dataset).
>    - {% icon param-file %} *"Filename mutations"*: `pancan_mutation_freeze.tsv` (Input dataset).
>    - {% icon param-file %} *"Filename of mutation burden"*: `mutation_burden_freeze.tsv` (Input dataset).
>    - {% icon param-file %} *"Filename of sample"*: `sample_freeze.tsv` (Input dataset) Input TCGA processed sample information data. See dataset descriptions for details.
>    - {% icon param-text %} *"Comma separated string of HUGO gene symbols"*: `ERBB2,PIK3CA,KRAS,AKT1` 
>    - {% icon param-text %} *"Comma sep string of TCGA disease acronyms. If no arguments are passed, filtering will default to options given in --filter_count and --filter_prop."*: `BLCA,BRCA,CESC,COAD,ESCA,LUAD,LUSC,OV,PRAD,READ,STAD,UCEC,UCS`
>    - {% icon param-select %} *"Supplement Y matrix with copy number events"*: `Yes` 
>    - {% icon param-file %} *"File with Copy number loss"*: `copy_number_loss_status.tsv` (Input dataset) 
>    - {% icon param-file %} *"File with Copy number gain"*: `copy_number_gain_status.tsv` (Input dataset) 
>    - {% icon param-file %} *"File with cancer gene classification table"*: `cosmic_cancer_classification.tsv` (Input dataset) 
>    - {% icon param-text %} *"alternative genes to test performance"*: `PTEN,PIK3R1,STK11` 
>    - {% icon param-text %} *"The alternative diseases to test performance"*: `BRCA,COAD,ESCA,HNSC,LGG,LUAD,LUSC,PRAD,READ,GBM,UCEC,UCS`
>    - *"Decision to drop input genes from X matrix"*: `Yes`
>    - *"Remove hypermutated samples"*: `Yes`
>    - *"Keep intermediate ROC values for plotting"*: `Yes`
>    - *"Shuffle the input gene exprs matrix alongside"*: `Yes` 
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `pancancer_classifier.py` inputs</details-title>
>    - \--genes: Comma separated string of HUGO symbols for target genes or targenes_list.csv file
>    - \--diseases: Comma separated string of disease types/TCGA acronyms for classifier
>    - \--folds: Number of cross validation folds
>    - \--drop: Drop the input genes from the X matrix
>    - \--seed: Value specifies the initial value of the random number seed
>    - \--copy_number: Optional flag to supplement copy number to define Y
>    - \--filter_count: Integer number for minimum count of mutations to include per disease
>    - \--filter_prop: Floating value for minimum proportion of mutated samples per disease
>    - \--num_features: Integer number for genes to include in classifier
>    - \--alphas: Comma separated string of alphas to test in pipeline
>    - \--l1_ratios: Comma separated string of l1 parameters to test
>    - \--alt_genes: Comma separated string of alternative genes to test
>    - \--alt_diseases: Comma separated string of alternative diseases to test
>    - \--alt_filter_count: Integer number for minimum count of mutations to include per alt_diseases
>    - \--alt_filter_prop: Floating value for minimum proportion of mutated samples alt_disease
>    - \--alt_folder: String for location to save the classifier figures
>    - \--remove_hyper: Store_true: remove hypermutated samples
>    - \--keep_intermediate: Store_true: keep intermediate roc curve items
>    - \--x_matrix: File name of features gene expression matrix to use
>    - \--classifier_folder: String of the location to store classifier data
{: .details}

> <comment-title>Outputs</comment-title>
>    - Log file for script run and additional information
>    - A combined disease model (Pan-model) and individual disease models are generated.
>    - An overall model statistics are provided in the form of classifier summary: classifier_summary.txt
>    - Logistic regression with elastic net penalization outputs sparse classifiers with 150-250 genes whose scores or weights are important for classification tasks: classifier_coefficients.tsv . *Lets examine this file for the top scored genes. These genes are potential predictive biomarkers and can be helpful for precision medicine approaches*.
>    - A tabular output that includes each gene mutation and copy number counts per each targene and proportions in various diseases is listed: *summary_counts.csv*
>    - Heatmap for all the AUROC values at different alphas and L1 rations: cv_heatmap.pdf
>    - Disease classifier figures: list of 2 files [disease_pr and disease_auroc]
>    - This tool computes predictions, accuracy measurements and measures AUROC and AUPR curves for the combined disease model (Pan-model): *all_disease_pr.pdf ; pancan_roc_results.tsv ; all_disease_roc.pdf ; disease_aupr.pdf ; disease_auroc.pdf*
>    - Additionally, the output also includes above performance and prediction metrics when the pan-model applied to alternative genes and cancer types: *alt_gene_alt_disease_summary.tsv ; alt_summary_counts.csv ; alt_gene_alt_disease_aupr_bar.pdf ; alt_gene_alt_disease_auroc_bar.pdf* Examine these files to analyze our PI3K_OG model on other alternate diseases.
{: .comment}

## PanCancer within disease analysis
This step is designed to generate individual pan-within models for each individual disease. It takes the same inputs as first step and generates similar output for each individual disease.

> <hands-on-title>Generating models for individual diseases listed for ERBB2,PIK3CA,KRAS,AKT1</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer within disease analysis](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_within_disease_analysis/pancancer_within_disease_analysis/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"Filename of features to use in model"*: `pancan_rnaseq_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename mutations"*: `pancan_mutation_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename of mutation burden"*: `mutation_burden_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename of sample"*: `sample_freeze.tsv` (Input dataset)
>    - *"Comma separated string of HUGO gene symbols"*: `ERBB2,PIK3CA,KRAS,AKT1`
>    - *"Comma sep string of TCGA disease acronyms. If no arguments are passed, filtering will default to options given in --filter_count and --filter_prop."*: `BLCA,BRCA,CESC,COAD,ESCA,LUAD,LUSC,OV,PRAD,READ,STAD,UCEC,UCS`
>    - {% icon param-file %} *"File with Copy number loss"*: `copy_number_loss_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with Copy number gain"*: `copy_number_gain_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with cancer gene classification table"*: `cosmic_cancer_classification.tsv` (Input dataset)
>    - *"Remove hypermutated samples"*: `Yes`
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `within_disease_analysis.py` inputs</details-title>
>    - \--genes: Comma separated string of HUGO symbols for target genes or targenes_list.csv file
>    - \--diseases: Comma separated string of disease types/TCGA acronyms for classifier
>    - \--alphas: Comma separated string of alphas to test in pipeline
>    - \--l1_ratios: Comma separated string of l1 parameters to test
>    - \--remove_hyper: Remove hypermutated samples
>    - \--alt_folder: String of location to classification folder extending to individual diseases
>    - \--x_matrix: File name of features gene expression matrix to use in the model
>    - \--filename_mut: Filename of sample/gene mutations to use in model
>    - \--filename_mut_burden: Filename of sample mutation burden to use in model
>    - \--filename_sample: Filename of patient/samples to use in model
>    - \--filename_copy_loss: Filename of copy number loss
>    - \--filename_copy_gain: Filename of copy number gain
>    - \--filename_cancer_gene_classification: Filename of cancer gene classification table
{: .details}

> <comment-title>Outputs</comment-title>
> This tool creates individual classifiers from each of the diseases specified to generate the full or pan model. Each of the classifiers has a similar output as specified for the pan-cancer classifier. The within models are trained using the same combination of targenes but considering only one disease for training and testing.
>    - Log file for script run and additional information
>    - list of classifier_summary.txt for each disease
>    - list of classifier_coefficients.tsv for each disease
>    - list of pancan_roc_results.tsv for each disease
>    - list of summary_counts.csv for each disease
>    - Within disease figures:  List of 5 files [all_disease_pr, all_disease_roc, cv_heatmap, disease_pr, disease_roc] for individual diseases
>    - Disease classifier figures: list of 2 files [disease_pr disease_roc] for each disease
{: .comment}

## PanCancer compare within models
we next do a performance comparison between the ERBB2,PIK3CA,KRAS,AKT1 pan model and individual models.
![Compare Pan model with with-in disease model](../../images/aberrant_pi3k_pathway_analysis/within.png "Cross-validation performance characteristic metric AUROC for the pan-cancer model compared to individual models trained on each cancer type")

> <hands-on-title>compare the ERBB2_PIK3CA_KRAS_AKT1 pan model with individual disease models</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer compare within models](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_compare_within_models/pancancer_compare_within_models/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"pancancer classifier summary"*: `classifier_summary.txt` (output of **PAPAA: PanCancer classifier** {% icon tool %})
>    - {% icon param-file %} *"pancancer classifier coefficients"*: `classifier_coefficients.tsv` (output of **PAPAA: Pancancer classifier** {% icon tool %})
>    - {% icon param-file %} *"pan_within classifier summary"*: `classifier_summary` (output of **PAPAA: PanCancer within disease analysis** {% icon tool %})(Note: Select all pan-within classifier summaries as dataset collection instead of multiple datasets)
>    - {% icon param-file %} *"pan_within classifier coefficients"*: `classifier_coefficients` (output of **PAPAA: Pancancer within disease analysis** {% icon tool %}) (Note: Select all pan-within classifier coefficients as dataset collection instead of multiple datasets)
>    - *"Would you want to compare given model with alt gene model?"*: `do not do alt gene`
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `compare_within_models.R` inputs</details-title>
>    - \--pancan_model: location of pan and pan-within-disease models classifier_summary.txt/s and classifier_coefficients.tsv/s dataset collections.
>    - \--alt_model: location of alt and alt-within-disease models classifier_summary.txt/s and classifier_coefficients.tsv/s dataset collections.
{: .details}

> <comment-title>Outputs</comment-title>
>    - Comparison plots for Pan and Pan_within models ("auroc_comparison.pdf" and "aupr_comparison.pdf")
>    - Comparison plots for altgene, alt_within, Pan_alt models ("alt_gene_auroc_comparison.pdf" and "alt_gene_aupr_comparison.pdf")
{: .comment}

## PanCancer apply weights
In this step we would like to predict 'y' status (mutational status) using 'x' matrix (gene expression). We subset the x matrix to top variable genes (determined by Median Absolute Deviation), scaling the expression and adding covariate information. A logit transformation will be applied to output probabilities and classifier decisions.

> <hands-on-title>Apply weights for ERBB2_PIK3CA_KRAS_AKT1 model</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer apply weights](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_apply_weights/pancancer_apply_weights/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"Filename of features to use in model"*: `pancan_rnaseq_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename mutations"*: `pancan_mutation_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename of mutation burden"*: `mutation_burden_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename of sample"*: `sample_freeze.tsv` (Input dataset)
>    - *"Supplement Y matrix with copy number events"*: `Yes`
>    - {% icon param-file %} *"File with Copy number loss"*: `copy_number_loss_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with Copy number gain"*: `copy_number_gain_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with cancer gene classification table"*: `cosmic_cancer_classification.tsv` (Input dataset)
>    - {% icon param-file %} *"pancancer classifier summary"*: `classifier_summary` (output of **PAPAA: PanCancer classifier** {% icon tool %})
>    - {% icon param-file %} *"pancancer classifier coefficients"*: `classifier_coefficients` (output of **PAPAA: PanCancer classifier** {% icon tool %})
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `apply_weights.py` inputs</details-title>
>    - \--classifier_summary: String of the location of classifier_summary.txt file
>    - \--copy_number: Supplement Y matrix with copy number events
>    - \--x_matrix: Filename of gene expression features to use in model
>    - \--filename_mut: Filename of sample/gene mutations to use in model
>    - \--filename_mut_burden: Filename of sample mutation burden to use in model
>    - \--filename_sample: Filename of patient/samples to use in model
>    - \--filename_copy_loss: Filename of copy number loss
>    - \--filename_copy_gain: Filename of copy number gain
>    - \--filename_cancer_gene_classification: Filename of cancer gene classification table
{: .details}

> <comment-title>Outputs</comment-title>
>    - Apply a logit transform on expression values (y = 1/(1+e^(-wX))) to output mutational probabilities. Generates "classifier_decisions.tsv" file which has scores/probabilities and other covariate information.  The scores/probabilities will be used for gene ranking and variant specific classification.
>    - Log file for script run and additional information
{: .comment}

## PanCancer visualize decisions
In this step we generate visualization plots using classifier decision function for samples in each individual disease, total decisions for all samples in all diseases, and hypermutated samples in all diseases.
![PI3K_OG_total_decisions_plot](../../images/aberrant_pi3k_pathway_analysis/PI3K_OG_total_decisions.png "Probability density plot for total decisions for all samples with the given classifier decision function")

> <hands-on-title>Visualize decisions for ERBB2_PIK3CA_KRAS_AKT1 model</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer visualize decisions](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_visualize_decisions/pancancer_visualize_decisions/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"pancancer decisions"*: `classifier_decisions` (output of **PAPAA: PanCancer apply weights** {% icon tool %})
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `visualize_decisions.py` inputs</details-title>
>    - \--classifier_decisions: String of the folder location of classifier_decisions.tsv
>    - \--custom: Comma separated list of columns to plot
>    (optional: True)
{: .details}

> <comment-title>Outputs</comment-title>
>    - Log file for script run and additional information
>    - Visualize decision function for all samples ("total_decisions.pdf")
>    - Plot disease type specific decision functions ("decision_plot_{}.pdf")
>    - Visualize decision function for hyper mutated tumors ("hyper_mutated.pdf")
{: .comment}

## PanCancer map mutation class
In this step we combined variant level information for each mutation combining with classifier predictions.

> <hands-on-title>map mutation class for ERBB2_PIK3CA_KRAS_AKT1 model</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer map mutation class](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_map_mutation_class/pancancer_map_mutation_class/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"pancancer decisions"*: `classifier_decisions.tsv` (output of **PAPAA: PanCancer apply weights** {% icon tool %})
>    - {% icon param-file %} *"string of the genes to extract or gene list file"*: `path_rtk_ras_pi3k_genes.txt` (Input dataset)
>    - *"Supplement Y matrix with copy number events"*: `Yes`
>    - {% icon param-file %} *"File with Copy number loss"*: `copy_number_loss_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with Copy number gain"*: `copy_number_gain_status.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename of raw mut MAF"*: `mc3.v0.2.8.PUBLIC.maf` (Input dataset)
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `map_mutation_class.py` inputs</details-title>
>    - \--classifier_decisions: String of the location of folder containing classifier_decisions.tsv
>    - \--path_genes: Comma separated string of HUGO symbols for all genes in the pathway or Pathway genes list file
>    - \--filename_copy_loss: Filename of copy number loss
>    - \--filename_copy_gain: Filename of copy number gain
>    - \--filename_raw_mut: Filename of raw mut MAF
{: .details}

> <comment-title>Outputs</comment-title>
>    - Merge per sample classifier scores with mutation types present in each sample and generate "mutation_classification_scores.tsv" file
>    - Log file for script run and additional information
{: .comment}

## PanCancer alternative genes pathwaymapper
In this step we combine classifier weights, copy number information, recalculate metrics for positive samples, visualize distribution for AUROC and AUPR for all genes and metrics for each gene.

> <hands-on-title>alternative genes pathway mapper for ERBB2_PIK3CA_KRAS_AKT1 model</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer alternative genes pathwaymapper](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_alternative_genes_pathwaymapper/pancancer_alternative_genes_pathwaymapper/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"pancancer decisions"*: `classifier_decisions.tsv` (output of **PAPAA: PanCancer apply weights** {% icon tool %})
>    - *"Comma separated string of HUGO gene symbols"*: `ERBB2,PIK3CA,KRAS,AKT1`
>    - {% icon param-file %} *"string of the genes to extract or gene list file"*: `path_rtk_ras_pi3k_genes.txt` (Input dataset)
>    - {% icon param-file %} *"Filename mutations"*: `pancan_mutation_freeze.tsv ` (Input dataset)
>    - {% icon param-file %} *"Filename of sample"*: `sample_freeze.tsv` (Input dataset)
>    - *"Supplement Y matrix with copy number events"*: `Yes`
>    - {% icon param-file %} *"File with Copy number loss"*: `copy_number_loss_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with Copy number gain"*: `copy_number_gain_status.tsv` (Input dataset)
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `targene_alternative_genes_pathwaymapper.py` inputs</details-title>
>    - \--genes: Comma separated string of HUGO symbols for target genes or targenes_list.csv file
>    - \--path_genes: Comma separated string of HUGO symbols for all genes in the target pathway or path_genes.csv file
>    - \--classifier_decisions: String of the location of classifier scores/alt_folder
>    - \--filename_mut: Filename of sample/gene mutations to use in model
>    - \--filename_mut_burden: Filename of sample mutation burden to use in model
>    - \--filename_sample: Filename of patient/samples to use in model
>    - \--filename_copy_loss: Filename of copy number loss
>    - \--filename_copy_gain: Filename of copy number gain
{: .details}

> <comment-title>Outputs</comment-title>
>    - Calculate and display pathway metrics ("pathway_metrics_pathwaymapper.txt")
>    - Visualize Distribution of AUROC and AUPRC for all genes and Get Metrics for All Genes ("all_gene_metric_ranks.tsv")
>    - Log file for script run and additional information
{: .comment}

## PanCancer pathway count heatmaps
This step generates combined heatmap from mutation and copy number information and summarizes mutation, copy and total counts per sample for all the genes in target pathway.
![PI3K_OG_combined_heatmap](../../images/aberrant_pi3k_pathway_analysis/PI3K_OG_combined_heatmap.png "Combined count heatmap (mutation and copynumber) for all genes in the target pathway with in each cancer type")

> <hands-on-title>Heatmaps for ERBB2_PIK3CA_KRAS_AKT1 model</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer pathway count heatmaps](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_pathway_count_heatmaps/pancancer_pathway_count_heatmaps/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"pancancer decisions"*: `classifier_decisions.tsv` (output of **PAPAA: PanCancer apply weights** {% icon tool %})
>    - {% icon param-file %} *"pancancer metrics pathwaymapper"*: `pathway_metrics_pathwaymapper.txt` (output of **PAPAA: PanCancer alternative genes pathwaymapper** {% icon tool %})
>    - {% icon param-file %} *"pancancer gene metric ranks"*: `all_gene_metric_ranks.tsv` (output of **PAPAA: PanCancer alternative genes pathwaymapper** {% icon tool %})
>    - *"Comma separated string of HUGO gene symbols"*: `ERBB2,PIK3CA,KRAS,AKT1`
>    - {% icon param-file %} *"String of the pathway genes to extract"*: `path_rtk_ras_pi3k_genes.txt` (Input dataset)
>    - {% icon param-file %} *"Filename of features to use in model"*: `pancan_rnaseq_freeze.tsv.gz` (Input dataset)
>    - {% icon param-file %} *"Filename mutations"*: `pancan_mutation_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename of mutation burden"*: `mutation_burden_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename of sample"*: `sample_freeze.tsv` (Input dataset)
>    - {% icon param-file %} *"File with Copy number loss"*: `copy_number_loss_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with Copy number gain"*: `copy_number_gain_status.tsv` (Input dataset)
>    - {% icon param-file %} *"File with cancer gene classification table"*: `cosmic_cancer_classification.tsv` (Input dataset)
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `targene_pathway_count_heatmaps.py` inputs</details-title>
>    - \--genes: Comma separated string of HUGO symbols for target genes or targenes_list.csv file
>    - \--path_genes: Comma separated string of HUGO symbols for all genes in the target pathway or path_genes.csv file
>    - \--classifier_decisions: String of the location of classifier scores/alt_folder
>    - \--x_matrix: Filename of features to use in model
>    - \--filename_mut: Filename of sample/gene mutations to use in model
>    - \--filename_mut_burden: Filename of sample mutation burden to use in model
>    - \--filename_sample: Filename of patient/samples to use in model
>    - \--filename_copy_loss: Filename of copy number loss
>    - \--filename_copy_gain: Filename of copy number gain
>    - \--filename_cancer_gene_classification: Filename of cancer gene classification table
{: .details}

> <comment-title>Outputs</comment-title>
>    - Mutational count heatmap for all genes in the target pathway with in each cancer type (cancer_type_mutation_heatmap.pdf).
>    - Copy number count heatmap for all genes in the target pathway with in each cancer type (cancer_type_copy_number_heatmap.pdf).
>    - Combined count heatmap for all genes in the target pathway with in each cancer type (cancer_type_combined_total_heatmap.pdf).
>    - Calculates Mutations and copy number percentages of the genes in the and generates "pathway_mapper_percentages.txt" file.
>    - Summarizes mutation, copy, and total counts per sample by targene pathway and generates "path_events_per_sample.tsv" file
{: .comment}

## PanCancer targene summary figures
This step generates plots summarizing various analysis, including heatmaps for distribution of aberrant events across tumors, distribution of predictions at variant levels.

> <hands-on-title>Summary figures for ERBB2_PIK3CA_KRAS_AKT1 model</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer targene summary figures](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_targene_summary_figures/pancancer_targene_summary_figures/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"Classifier data"*: `classifier_summary.txt` (output of **PAPAA: PanCancer classifier** {% icon tool %})
>    - {% icon param-file %} *"pancancer classifier coefficients"*: `classifier_coefficients.tsv` (output of **PAPAA: PanCancer classifier** {% icon tool %})
>    - *"option to set seed"*: `123`
>    - {% icon param-file %} *"summary counts"*: `summary_counts.csv` (output of **PAPAA: PanCancer classifier** {% icon tool %})
>    - {% icon param-file %} *"mutation classification scores"*: `mutation_classification_scores.tsv` (output of **PAPAA: PanCancer map mutation class** {% icon tool %})
>    - {% icon param-file %} *"path events per sample"*: `path_events_per_sample.tsv` (output of **PAPAA: PanCancer pathway count heatmaps** {% icon tool %})
>    - {% icon param-file %} *"pancancer gene metric ranks"*: `all_gene_metric_ranks.tsv` (output of **PAPAA: PanCancer alternative genes pathwaymapper** {% icon tool %})
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activity Analysis `targene_summary_figures.R` inputs</details-title>
>    - \--classifier_summary: String of the location of classifier data
>    - \--seed: Value specifies the initial value of the random number seed
{: .details}

> <comment-title>Outputs</comment-title>
>    - Heatmap for mutation and gain proportions for the genes used in model across all the TCGA cancer types (all_targene_heatmap.pdf")
>    - Heatmap for total mutation and total gain proportions for the genes used in model across all the TCGA cancer types (targene_heatmap.pdf")
>    - Gene weights/Coefficients contributing to the model (targene_coef_plot.pdf)
>    - Plot distributions of predictions according to variant classification for OG and TSG ("variant_gain_fill_map.pdf" and "variant_loss_fill_map.pdf")
>    - Targene Summary Counts Distribution ("path_events_per_sample.tsv")
>    - Targene pathway events counts ("targene_pathway_events_counts.pdf")
>    - Performance Metrics Distribution across pathway members ("aupr_distribution.pdf" and "auroc_distribution.pdf")
>    - T-Test for AUPR between targene pathway genes and Other genes ("targene_pathway_variant_AUPR_ttest.txt")
>    - Log file for script run and additional information
>    - Extracting sample classifier scores for nucleotide level alterations in each sample and generate "nucleotide_mutation_scores.tsv" file
>    - Extracting sample classifier scores for amino-acid level alterations in each sample and generate "amino_acid_mutation_scores.tsv" file
{: .comment}

## PanCancer targene cell line predictions
In this step we use our classifier information and predict mutational status for various cell lines in CCLE and GDSC data sources.
![CCLE_targene_celline_classification](../../images/aberrant_pi3k_pathway_analysis/ccle_classification.png "PI3K_OG classifiers applied to CCLE cell lines. Cell lines that harbor mutations in ERBB2, KRAS, PIK3CA, AKT1 are indicated by yellow boxes with red dots and wild types with the green boxes and blue dots.")
![GDSC_targene_celline_classification](../../images/aberrant_pi3k_pathway_analysis/gdsc_classification.png "PI3K_OG classifiers applied to CCLE 390 standard cell lines between CCLE and GDSC cell lines that harbor mutations in ERBB2, KRAS, PIK3CA, AKT1 are indicated by yellow boxes with red dots and wild types with the green boxes and blue dots. ")

> <hands-on-title>Analysis of CCLE and GDSC cell-lines using ERBB2_PIK3CA_KRAS_AKT1 model</hands-on-title>
>
> 1. {% tool [PAPAA: PanCancer targene cell line predictions](toolshed.g2.bx.psu.edu/repos/vijay/pancancer_targene_cell_line_predictions/pancancer_targene_cell_line_predictions/0.1.9) %} with the following parameters:
>    - {% icon param-file %} *"Classifier data"*: `classifier_summary.txt` (output of **PAPAA: PanCancer classifier** {% icon tool %})
>    - {% icon param-file %} *"pancancer classifier coefficients"*: `classifier_coefficients.tsv` (output of **PAPAA: PanCancer classifier** {% icon tool %})
>    - {% icon param-file %} *"nucleotide mutation scores"*: `nucleotide_acid_mutation_scores.tsv` (output of **PAPAA: PanCancer targene summary figures** {% icon tool %})
>    - {% icon param-file %} *"amino acid mutation scores"*: `amino_acid_mutation_scores.tsv` (output of **PAPAA: PanCancer targene summary figures** {% icon tool %})
>    - *"Comma separated string of HUGO targene symbols"*: `ERBB2_MUT,PIK3CA_MUT,KRAS_MUT,AKT1_MUT`
>    - {% icon param-file %} *"string of the genes to extract or gene list file"*: `path_rtk_ras_pi3k_genes.txt` (Input dataset)
>    - {% icon param-file %} *"Filename ccle rnaseq data"*: `ccle_rnaseq_genes_rpkm_20180929_mod.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename ccle mutational data"*: `CCLE_MUT_CNA_AMP_DEL_binary_Revealer.gct` (Input dataset)
>    - {% icon param-file %} *"Filename ccle variant data"*: `CCLE_DepMap_18Q1_maf_20180207.txt` (Input dataset)
>    - {% icon param-file %} *"Filename gdsc rnaseq data"*: `GDSC_EXP_CCLE_converted_name.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename gdsc mutational data"*: `GDSC_CCLE_common_mut_cnv_binary.tsv` (Input dataset)
>    - {% icon param-file %} *"Filename for gdsc1 pharmacological data file"*: `gdsc1_ccle_pharm_fitted_dose_data.txt` (Input dataset)
>    - {% icon param-file %} *"Filename for gdsc2 pharmacological data file"*: `gdsc2_ccle_pharm_fitted_dose_data.txt` (Input dataset)
{: .hands_on}

> <details-title>PanCancer Aberrant Pathway Activ

tools_json:
["PAPAA: PanCancer classifier", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_classifier/pancancer_classifier/0.1.9", "PAPAA: PanCancer within disease analysis", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_within_disease_analysis/pancancer_within_disease_analysis/0.1.9", "PAPAA: PanCancer compare within models", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_compare_within_models/pancancer_compare_within_models/0.1.9", "PAPAA: PanCancer apply weights", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_apply_weights/pancancer_apply_weights/0.1.9", "PAPAA: PanCancer visualize decisions", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_visualize_decisions/pancancer_visualize_decisions/0.1.9", "PAPAA: PanCancer map mutation class", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_map_mutation_class/pancancer_map_mutation_class/0.1.9", "PAPAA: PanCancer alternative genes pathwaymapper", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_alternative_genes_pathwaymapper/pancancer_alternative_genes_pathwaymapper/0.1.9", "PAPAA: PanCancer pathway count heatmaps", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_pathway_count_heatmaps/pancancer_pathway_count_heatmaps/0.1.9", "PAPAA: PanCancer targene summary figures", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_targene_summary_figures/pancancer_targene_summary_figures/0.1.9", "PAPAA: PanCancer targene cell line predictions", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_targene_cell_line_predictions/pancancer_targene_cell_line_predictions/0.1.9", "PAPAA: PanCancer external sample status prediction", "toolshed.g2.bx.psu.edu/repos/vijay/pancancer_external_sample_status_prediction/pancancer_external_sample_status_prediction/0.1.9"]


topic: statistics
tutorial: genomic-llm-sequence-generation
title: Generating Artificial Yeast DNA Sequences using a DNA LLM
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml", "Large Language Model"]
content_hash: 001ec6ae6b6251f50a226cbd1d3e3673
last_modified: 2026-05-27T09:01:49.830173
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/genomic-llm-sequence-generation/tutorial.html

content:
Generating synthetic DNA sequences using pre-trained language models  bridges the fields of synthetic biology and artificial intelligence, enabling the creation of novel DNA sequences that closely mimic natural genomes. By leveraging the power of advanced language models, we can generate biologically relevant sequences that have the potential to revolutionize genetic engineering, drug discovery, and our understanding of genomic function.

Throughout this tutorial, we will learn how to set up a computational environment tailored for DNA sequence generation, configure pre-trained language models to produce synthetic sequences, and analyze the results to assess their biological significance. Here the aim is to generate DNA sequences similar to yeast, more specifically to *Saccharomyces cerevisiae*.

For this tutorial, we'll use a [pre-trained language model](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-138M-yeast) which was trained on 1,011 *Saccharomyces cerevisiae* isolates from {% cite peter2018genome %}. This model has 138 million parameters and uses a mixture of experts architecture, making it efficient and powerful for generating DNA sequences.

```python
model_name = "RaphaelMourad/Mistral-DNA-v1-138M-yeast"
```

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare resources 


## Install dependencies

The first step is to install the required dependencies:

```python
!pip install Bio==1.7.1
!pip install orfipy
!pip install sklearn
!pip install transformers -U
!pip install torch==2.5.0
```

## Import Python libraries

Let's now import them.


```python
import os

import itertools
import matplotlib.pyplot as plt
import numpy as np
import orfipy_core
import pandas as pd
import torch
from Bio import (SeqIO, Seq)
from collections import defaultdict
from sklearn.decomposition import PCA
from transformers import pipeline
from typing import List, Tuple
```

> <comment-title>Versions</comment-title>
>
> This tutorial has been tested with following versions:
> - `numpy` > 1.26.4
> - `transformers` > 4.47.1
> 
> You can check the versions with:
>
> ```python
> np.__version__
> transformers.__version__
> ```
{: .comment}

## Check and configure available resources

Let's check the GPU usage and RAM:

```python
!nvidia-smi
```

Let's configure PyTorch and the CUDA environment -- software and hardware ecosystem provided by NVIDIA to enable parallel computing on GPU -- to optimize GPU memory usage and performance:

1. Enables CuDNN benchmarking in PyTorch:

    ```python
    torch.backends.cudnn.benchmark=True
    ```

2. Set an environment variable that configures how PyTorch manages CUDA memory allocations

    ```python
    os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"
    ```

# Generate Synthetic DNA Sequences

Using the pre-trained language model which was trained on 1,011 *Saccharomyces cerevisiae* isolates, we would like to generate 100 synthetic yeast DNA sequences.

## Build the Sequence Generator

First, we need to set up the sequence generation pipeline. This pipeline will enable us to generate synthetic DNA sequences that mimic natural genomic sequences. By leveraging the power of language models, we can create novel DNA sequences for various applications in synthetic biology.

We use the `pipeline` function from the `transformers` library which simplifies the process of setting up a sequence generation pipeline. It abstracts away the complexities of model loading and configuration, allowing us to focus on generating sequences.

```python
generator = pipeline("text-generation", model=model_name)
```

> <question-title></question-title>
>
> 1. What do the parameters `"text-generation"`?
> 2. What is a pipeline?
> 3. What are the different types of pipelines?
>
> > <solution-title></solution-title>
> >
> > 1. It specifies that we are creating a pipeline for text generation, which is suitable for generating DNA sequences.
> >
> > 2. Pipelines are high-level abstractions that simplify the process of applying models to various NLP tasks
> >
> > 3. Besides text-generation, there are several other types of pipelines designed for different applications. Here are some of the most commonly used pipelines:
> >     - `feature-extraction`: Extracts features (embeddings) from text using a model. Useful for tasks that require text representation, such as clustering or similarity measurement.
> >     - `sentiment-analysis`: Determines the sentiment of a given text, classifying it as positive, negative, or neutral.
> >     - `text-classification`: Classifies text into predefined categories or labels, useful for tasks like spam detection, topic classification, etc.
> >     - `token-classification`: Assigns a label to each token in the input text, commonly used for Named Entity Recognition (NER) and Part-Of-Speech (POS) tagging.
> >     - `question-answering`: Answers questions based on a given context or passage of text. Useful for building Q&A systems.
> >     - `fill-mask`: Predicts the missing word(s) in a sentence with a masked token, often used with models like BERT.
> >     - `summarization`: Generates a concise summary of a longer text, useful for creating abstracts or condensing information.
> >     - `translation`: Translates text from one language to another, supporting various language pairs.
> >     - `conversational`: Engages in a conversation by generating responses based on input prompts, useful for building chatbots.
> >     - `zero-shot-classification`: Classifies text into categories that were not seen during training, allowing for flexible and dynamic classification tasks.
> >     - `table-question-answering`: Answers questions based on structured data, such as tables, combining NLP with data querying capabilities.
> >
> >     Each of these pipelines is designed to handle specific NLP tasks efficiently, leveraging pre-trained models to provide accurate and fast results. You can initialize these pipelines using the pipeline function from the transformers library, specifying the task you want to perform.
> >
> {: .solution}
>
{: .question}

## Generate Synthetic DNA Sequences

Once the pipeline is set up, we can generate synthetic DNA sequences by calling the generator with appropriate parameters. We first need to specify the parameters for sequence generation:

- Maximum length of the generated sequence in term of tokens (i.e. k-mers of 3 to 7 bases)

    ```python
    max_length = 30 
    ```

- Number of sequences to generate

    ```python
    num_sequences = 100
    ```

- The temperature: 

    It controls the randomness of the generated sequences. A higher temperature results in more varied outputs, while a lower temperature produces more deterministic sequences.

    ```python
    temperature = 0.1
    ```

Let's now generate the sequences:

```python
synthetic_dna = generator(
  "",
  max_length=max_length,
  do_sample=True,
  top_k=50,
  temperature=temperature,
  repetition_penalty=1.2,
  num_return_sequences=num_sequences,
  eos_token_id=0,
)
```

> <question-title></question-title>
>
> What do the parameters?
>
> 1. `""`
> 2. `do_sample=True`
> 3. `top_k=50`
> 4. `repetition_penalty=1.2`
> 5. `eos_token_id=-0`
>
> > <solution-title></solution-title>
> >
> > 1. Starting prompt (empty for unconditional generation)
> >
> >
> > 2. `do_sample=True`: When set to `True`, the model uses sampling to generate sequences, which introduces randomness and variability in the output.
> > 3. `top_k=50`: Limits the generated tokens to the top k most probable tokens. This helps in controlling the diversity of the output while ensuring biological relevance.
> > 4. `repetition_penalty=1.2`: Penalizes the repetition of the same token in the generated sequence. A value greater than 1.0 discourages the model from repeating tokens, promoting diversity.
> > 5. `eos_token_id=0`: Specifies the end-of-sequence token ID, which signals the model to stop generating further tokens. This is useful for controlling the termination of sequence generation
> >
> {: .solution}
>
{: .question}

Let's extract the sequences:

```python
artificial_sequences=[]
for seq in synthetic_dna:
    artificial_sequences.append(seq["generated_text"].replace(" ", ""))
```

> <question-title></question-title>
>
> What is the length of the 5 first sequences?
>
> > <solution-title></solution-title>
> >
> > ```python
> > for i in range(5):
> >   print(len(artificial_sequences[i]))
> > ```
> >
> {: .solution}
>
{: .question}

> <details-title>Generate random sequences from a sequence seed</details-title> 
> 
> To generate artificial DNA sequence starting by a defined substring, e.g. "TATA", we replace `""` in `generator` function by the defined substring:
> 
> ```python
> synthetic_dna = generator(
>     "TATA",
>     max_length=max_length,
>     do_sample=True,
>     top_k=50,
>     temperature=0.4,
>     repetition_penalty=1.2,
>     num_return_sequences=100,
>     eos_token_id=0,
> )
> artificial_sequences = [seq["generated_text"].replace(" ", "") for seq in synthetic_dna]
> artificial_sequences[0:5]
> ```
>
{: .details}


## Compare with Real Yeast Genome

We would like to compare the generated sequences to real sequences from *Saccharomyces cerevisiae*.

Let's download the yeast genome assembly:

```python
!wget http://hgdownload.soe.ucsc.edu/goldenPath/sacCer3/bigZips/sacCer3.fa.gz
!gunzip sacCer3.fa.gz
```

We would like now to extract 1,000 random sequences of a length 100 bases from the downloaded genome:

```python
def extract_random_sequences(
    genome_file: str,
    seq_length: int = 100,
    num_seqs: int = 100,
) -> List[str]:
    """
    Extracts random sequences from a genome FASTA file.

    Parameters:
    - genome_file (str): Path to the genome FASTA file.
    - seq_length (int): Length of each sequence to extract.
    - num_seqs (int): Number of sequences to extract.

    Returns:
    - List[str]: A list of extracted sequences.
    """
    try:
        # Load genome sequences from the FASTA file
        genome = [str(record.seq) for record in SeqIO.parse(genome_file, "fasta")]

        # Join all chromosomes or scaffolds into one large sequence
        genome_seq = "".join(genome)
        genome_size = len(genome_seq)

        if genome_size < seq_length:
            raise ValueError("Sequence length is larger than the genome size.")

        # List to store extracted sequences
        extracted_seqs = []

        # Extract random sequences
        for _ in range(num_seqs):
            start_pos = random.randint(0, genome_size - seq_length)  # Random start position
            sequence = genome_seq[start_pos:start_pos + seq_length]  # Extract sequence
            extracted_seqs.append(sequence)

        return extracted_seqs

    except Exception as e:
        print(f"An error occurred: {e}")
        return []

genome_file = "sacCer3.fa"  # Path to the yeast genome FASTA file
real_sequences = extract_random_sequences(genome_file, seq_length=100, num_seqs=1000)
```

To compare synthetic and real DNA sequences, we utilize k-mer counts as a method to numerically describe and analyze DNA sequences. K-mers are short, overlapping subsequences of a fixed length $$k$$ within a DNA sequence. We can think of k-mers as "words" within the DNA sequence. Just as paragraphs in text share common words, similar DNA sequences will share common k-mers. By counting the occurrences of these k-mers, we can transform DNA sequences into numerical vectors, allowing us to compare them quantitatively. **Sequences that are very similar are expected to have similar k-mer counts**, much like how similar texts share common vocabulary. By comparing the k-mer counts of synthetic and real DNA sequences, we can assess their similarity. If two sequences share many k-mers, it indicates that they are likely to be similar in composition and structure.

This approach leverages the power of k-mer analysis to provide insights into the similarity between synthetic and real DNA sequences, aiding in the validation and evaluation of synthetic biology techniques.

```python
def generate_all_kmers(k: int) -> List[str]:
    """
    Generate all possible k-mers of a given length.

    Parameters:
    - k (int): Length of each k-mer.

    Returns:
    - List[str]: A list of all possible k-mers.
    """
    return ["".join(p) for p in itertools.product("ACGT", repeat=k)]

def kmer_counts_matrix(sequences: List[str], k: int = 6) -> Tuple[np.ndarray, List[str]]:
    """
    Compute k-mer counts for a list of sequences and return a count matrix.

    Parameters:
    - sequences (List[str]): List of DNA sequences.
    - k (int): Length of each k-mer. Default is 6.

    Returns:
    - Tuple[np.ndarray, List[str]]: A tuple containing the k-mer count matrix and the list of all possible k-mers.
    """
    all_kmers = generate_all_kmers(k)
    kmer_index = {kmer: idx for idx, kmer in enumerate(all_kmers)}  # Map k-mers to column indices
    matrix = np.zeros((len(sequences), len(all_kmers)), dtype=int)  # Initialize count matrix

    for seq_idx, seq in enumerate(sequences):
        if len(seq) < k:
            raise ValueError(f"Sequence length is less than {k} for sequence {seq_idx}: {seq}")

        kmer_dict = defaultdict(int)
        for i in range(len(seq) - k + 1):
            kmer = seq[i:i+k]
            kmer_dict[kmer] += 1

        # Fill in the matrix for this sequence
        for kmer, count in kmer_dict.items():
            if kmer in kmer_index:
                matrix[seq_idx, kmer_index[kmer]] = count

    return matrix, all_kmers
```

Let's now count 4-mers for artificial and real sequences.

```python
k = 4
artificial_seq_kmer_counts, kmers = kmer_counts_matrix(artificial_sequences, k=k)
real_seq_kmer_counts, kmers = kmer_counts_matrix(real_sequences, k=k)
```

> <question-title></question-title>
>
> 1. What is the format of `artificial_seq_kmer_counts` and `real_seq_kmer_counts`?
> 2. What are the dimensions?
>
> > <solution-title></solution-title>
> >
> > 1. `artificial_seq_kmer_counts` and `real_seq_kmer_counts` are 2-matrices.
> > 2. 100 x 256 ($$256 = 4^{4}$$)
> >
> {: .solution}
>
{: .question}

To visualize and compare real and generated DNA sequences in a continuous space, we can utilize Principal Component Analysis (PCA). PCA is a powerful dimensionality reduction technique that transforms high-dimensional data into a lower-dimensional space while preserving as much variability as possible. This allows us to visualize the similarities and differences between the sequences effectively.

```python
def plot_pca_projection(set1: np.ndarray, set2: np.ndarray, title: str = "PCA Projection") -> None:
    """
    Fit PCA on set1 and project both set1 and set2 into the PCA space, then plot the results.

    Parameters:
    - set1 (np.ndarray): K-mer count matrix for the first set of sequences (used to fit PCA).
    - set2 (np.ndarray): K-mer count matrix for the second set of sequences (projected into PCA space).
    - title (str): Title of the plot.

    Returns:
    - None
    """
    # Fit PCA on set1 only
    pca = PCA(n_components=2)
    pca.fit(set1)

    # Project both set1 and set2 into the PCA space
    pca_set1 = pca.transform(set1)
    pca_set2 = pca.transform(set2)

    # Plot the first two principal components
    plt.figure(figsize=(8, 6))
    plt.scatter(
        pca_set1[:, 0],
        pca_set1[:, 1],
        color="blue",
        label="Real sequences",
        alpha=0.7,
    )
    plt.scatter(
        pca_set2[:, 0],
        pca_set2[:, 1],
        color="red",
        label="Artificial sequences",
        alpha=0.7,
    )

    # Add labels and legend
    plt.xlabel("Principal Component 1")
    plt.ylabel("Principal Component 2")
    plt.title(title)
    plt.legend()

    # Show the plot
    plt.show()


plot_pca_projection(real_seq_kmer_counts, artificial_seq_kmer_counts)
```
    
![Scatter plot titled 'PCA Projection of Set1 and Set2 (PCA built from Set1)' displaying the first two principal components. Blue dots represent 'Real sequences' and red dots represent 'Artificial sequences.' The x-axis is labeled 'Principal Component 1' and the y-axis is labeled 'Principal Component 2.' The plot shows a dense cluster of points near the origin with some outliers, indicating the distribution and overlap between real and artificial sequences in the PCA space.](./images/pca_k_4_t_0_1.png)

> <hands-on-title></hands-on-title> 
>
> For different temperature values ($$0.001$$, $$0.01$$, $$0.1$$, $$0.5$$, $$1.0$$, $$1.5$$):
> 1. Generate synthetic DNA sequences
> 2. Observe the 5 first generated sequences
> 3. Generate the PCA plots
> 4. Compute the variance of k-mers counts between artificial sequences 
>
> > <solution-title></solution-title>
> > 
> > ```python
> > 
> > def generate_synthetic_dna(
> >     model_name: str,
> >     max_length: int,
> >     num_sequences: int,
> >     temp: float,
> >     top_k: int = 50,
> >     repetition_penalty: float = 1.2,
> > ) -> List[str]:
> >     """
> >     Generate synthetic DNA sequences using a pre-trained language model.
> >     
> >     Parameters:
> >     - model_name (str): Name or path of the pre-trained model.
> >     - max_length (int): Maximum length of each generated sequence.
> >     - num_sequences (int): Number of sequences to generate.
> >     - temp (float): Temperature value to control sequence variability.
> >     - top_k (int): Number of highest probability vocabulary tokens to consider. Default is 50.
> >     - repetition_penalty (float): Penalty for repeating the same token. Default is 1.2.
> >     
> >     Returns:
> >     - List[str]: List of generated DNA sequences.
> >     """
> >     # Initialize the sequence generation pipeline
> >     generator = pipeline("text-generation", model=model_name)
> >     # Generate synthetic DNA sequences
> >     synthetic_dna = generator(
> >        "",
> >        max_length=max_length,
> >        do_sample=True,
> >        top_k=top_k,
> >        temperature=temp,
> >        repetition_penalty=repetition_penalty,
> >        num_return_sequences=num_sequences,
> >        eos_token_id=0,
> >     )
> >     # Extract and clean the generated sequences
> >     artificial_sequences = [seq["generated_text"].replace(" ", "") for seq in synthetic_dna]
> >     return artificial_sequences
> >
> >
> > temperatures = [0.001, 0.01, 0.1, 0.5, 1.0, 1.5]
> > for temp in temperatures:
> >     art_sequences = generate_synthetic_dna(model_name, max_length,  num_sequences, temp)
> >     art_seq_kmer_counts, kmers = kmer_counts_matrix(art_sequences, k=k)
> >     plot_pca_projection(real_seq_kmer_counts, art_seq_kmer_counts)
> >     var = np.mean(np.var(art_seq_kmer_counts, axis=0))
> >     print(f"Variance for temperature of { temp }: { var }")
> > ```
> {: .solution}
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How similar do the 5 first generated DNA sequences look like given the temperature?
> 2. What are the different values for the variance?
> 3. What can we conclude?
>
> > <solution-title></solution-title>
> >
> > 1. Lower the temperature, more similar the sequences.
> > 2. The variance for the different temperatures are:
> >    - Variance $$=0.039 $$ for temperature $$= 0.001$$
> >    - Variance $$=0.219 $$ for temperature $$= 0.01$$
> >    - Variance $$=0.367 $$ for temperature $$= 0.1$$
> >    - Variance $$=0.404 $$ for temperature $$= 0.5$$
> >    - Variance $$=0.376 $$ for temperature $$= 1$$
> >    - Variance $$=0.377 $$ for temperature $$= 1.5$$
> > 3. The higher the temperature, higher the variance, higher the variability.
> >
> {: .solution}
>
{: .question}

## Checking for Novelty in Generated DNA Sequences Using BLAST

After generating synthetic DNA sequences, it's crucial to verify whether these sequences are truly novel or if they closely resemble existing sequences in biological databases. One effective method to accomplish this is by performing a BLAST (Basic Local Alignment Search Tool) search ({% cite altschul1990basic %}). BLAST allows us to compare our generated sequences against a comprehensive database of known DNA sequences, helping us determine their uniqueness and potential biological relevance.

Let's start with a synthetic DNA sequence generated with a low temperature setting. Low temperature settings reduce variability, making the generated sequences more similar to real sequences. This step helps ensure that the sequence is biologically plausible and likely to have counterparts in existing databases.

> <hands-on-title></hands-on-title> 
>
> 1. Generate the synthetic DNA sequences with a temperature value of $$0.01$$
> 2. Get the first sequence
> 3. Make a [Standard Nucleotide BLAST search](https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastn&PAGE_TYPE=BlastSearch&LINK_LOC=blasthome) of the first sequence
>
> > <solution-title></solution-title>
> > 
> > ```python
> > 
> > art_sequences = generate_synthetic_dna(model_name, max_length,  num_sequences, 0.01)
> > art_sequences[0]
> > ```
> {: .solution}
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many significant similarities have been found?
> 2. Have we generated new sequence?
>
> > <solution-title></solution-title>
> >
> > 1. The BLAST search do not return any significant match
> > 2. When the BLAST search returns no significant matches, it suggests that the synthetic sequence is novel and does not closely resemble any known sequences. This outcome is particularly interesting because it indicates that the sequence generation process has produced something entirely new.
> >
> {: .solution}
>
{: .question}

When generating sequences with low temperature, we observed limited variability among the sequences. This lack of variability can be useful for ensuring consistency but may limit the exploration of novel sequence spaces. To introduce more variability and potentially discover even more novel sequences, we can increase the temperature setting during sequence generation. Higher temperatures introduce greater randomness, leading to more diverse and potentially unique sequences.

> <hands-on-title></hands-on-title> 
>
> 1. Generate the synthetic DNA sequences with a temperature value of $$1$$
> 2. Get the first sequence
> 3. Make a [Standard Nucleotide BLAST search](https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastn&PAGE_TYPE=BlastSearch&LINK_LOC=blasthome) of the first sequence
>
> > <solution-title></solution-title>
> > 
> > ```python
> > 
> > art_sequences = generate_synthetic_dna(model_name, max_length,  num_sequences, 1)
> > art_sequences[0]
> > ```
> {: .solution}
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many significant similarities have been found?
> 2. Have we generated new sequence?
>
> > <solution-title></solution-title>
> >
> > 1. The BLAST search do not return any significant match
> > 2. When the BLAST search returns no significant matches, it suggests that the synthetic sequence is novel and does not closely resemble any known sequences. This outcome is particularly interesting because it indicates that the sequence generation process has produced something entirely new.
> >
> {: .solution}
>
{: .question}

Even if the generated sequences are novel, novelty alone does not guarantee functionality, so further analysis may be necessary to assess the potential applications of these sequences.

# Exploring Synthetic Gene Generation

Having successfully generated and analyzed synthetic DNA sequences, the next intriguing question is: Can we extend this approach to generate entire synthetic genes? This endeavor requires producing longer sequences that mimic the complexity and functionality of natural genes. The mean length of yeast genes is several hundred bases. To effectively mimic natural genes, our synthetic sequences must match this length while maintaining biological plausibility. Generating longer sequences is not merely about increasing length; it's about ensuring that these sequences contain the necessary genetic elements, such as promoters, coding regions, and regulatory sequences, that are essential for gene function.

To start, we generate 10 sequences with `max_length = 400`:

```python
synthetic_dna = generate_synthetic_dna(model_name, 400, 10, 1)
```

> <question-title></question-title>
>
> How long are the first 5 sequences?
>
> > <solution-title></solution-title>
> >
> > ```python
> > [len(s) for s in synthetic_dna[:10] ]
> > ```
> >
> {: .solution}
>
{: .question}

## Detecting Open Reading Frames (ORFs)

To determine if the generated sequences could contain genes, we examine them for the presence of key genetic elements, such as Open Reading Frames (ORFs).

An ORF is a portion of a DNA sequence that begins with a start codon (e.g., ATG) and ends with a stop codon (e.g., TAA, TAG, TGA). Identifying ORFs is essential for determining the potential functionality of synthetic DNA sequences, as ORFs can indicate whether a sequence could encode a protein.

To detect ORFs in sequences, we use `orfipy_core`:

```python
def detect_orfs(
    sequence: str,
    minlen: int = 100,
    maxlen: int = 1000,
) -> List[Tuple[int, int, str, str]]:
    """
    Detect ORFs in a given DNA sequence.

    Parameters:
    - sequence (str): The DNA sequence to analyze.
    - minlen (int): Minimum length of ORFs to detect. Default is 100.
    - maxlen (int): Maximum length of ORFs to detect. Default is 1000.

    Returns:
    - List[Tuple[int, int, str, str]]: List of ORFs with start, stop positions, strand, and description.
    """
    orfs = []
    for start, stop, strand, description in orfipy_core.orfs(sequence, minlen=minlen, maxlen=maxlen):
        orfs.append((start, stop, strand, description))
        print(f"Start: {start}, Stop: {stop}, Strand: {strand}, Description: {description}")
    return orfs
```

Let's extract the ORFs for the first DNA sequence:

```python
seq = artificial_sequences[0]
orfs = detect_orfs(seq)
```

> <question-title></question-title>
>
> How many ORFs have been detected for the first DNA sequence?
>
> > <solution-title></solution-title>
> >
> > `len(orfs)`
> >
> {: .solution}
>
{: .question}

## Extracting and Translating Detected Open Reading Frames (ORFs)

After detecting Open Reading Frames (ORFs) in our synthetic DNA sequences, the next step is to extract these ORFs and translate them into protein sequences. This process allows us to explore the potential protein products encoded by our synthetic genes, providing insights into their possible functions and biological roles.

To extract and translate the ORFs, we need to:

1. Extract the ORF Sequence: Using the start and stop positions identified by the `detect_orfs` function, we can extract the corresponding DNA sequence for each ORF. This sequence represents the potential protein-coding region within the synthetic DNA.
2. Translate the ORF: We can convert the DNA sequence of the ORF into its corresponding amino acid sequence. This step reveals the potential protein product encoded by the ORF.

```python
def extract_and_translate_orf(
    sequence: str,
    orfs: List[Tuple[int, int, str, str]],
    orf_index: int = 0,
) -> str:
    """
    Extract an ORF from a DNA sequence and translate it into a protein sequence.

    Parameters:
    - sequence (str): The DNA sequence containing the ORF.
    - orfs (List[Tuple[int, int, str, str]]): List of detected ORFs with start, stop positions, strand, and description.
    - orf_index (int): Index of the ORF to extract and translate. Default is 0.

    Returns:
    - str: The translated protein sequence.
    """
    # Extract the ORF sequence using the start and stop positions
    start, stop, strand, description = orfs[orf_index]
    orf_sequence = sequence[start:stop]

    # Translate the ORF sequence into a protein sequence
    protein_sequence = str(Seq.Seq(orf_sequence).translate())

    return protein_sequence
```

For the first detected ORFs:

```python
protein_seq = extract_and_translate_orf(seq, orfs)
print(f"Translated Protein Sequence: {protein_seq}")
```

After translating a synthetic DNA sequence into a protein sequence, the next crucial step is to analyze this sequence to uncover its potential biological function. This involves identifying known protein domains or motifs within the sequence that may indicate specific functions, such as binding sites or active regions. Verifying the biological plausibility of the translated sequence is essential, which can be achieved by comparing it to known protein sequences in databases like UniProt or GenBank using tools like Diamond and InterProScan. Additionally, examining the conservation of the sequence across different species can provide insights into its functional importance. If the sequence shows potential for biological function, it can serve as a starting point for further research, including structural modeling to predict its three-dimensional structure, functional assays to test its activity, and evolutionary studies to understand its origins and adaptations. This comprehensive analysis not only enhances our understanding of the generated sequences but also opens new avenues for scientific exploration and application in synthetic biology and genetic engineering.

# Conclusion
Throughout this tutorial, we have explored the process of generating synthetic DNA sequences using a DNA Language Model (LLM), focusing on creating sequences that mimic the complexity and functionality of natural yeast genomes. We began by building a sequence generator, leveraging pre-trained models to produce synthetic DNA sequences with controlled variability. 

We compared of these synthetic sequences to real yeast genomes, utilizing k-mer counts and Principal Component Analysis (PCA) to visualize and assess their similarities and differences. This comparative analysis provided valuable insights into how well our generated sequences aligned with natural counterparts, highlighting both the strengths and areas for improvement in our approach.

To ensure the novelty of our generated sequences, we conducted BLAST searches, confirming that many of our synthetic sequences were indeed unique and did not closely match existing sequences in biological databases. This step was crucial in validating the potential of our model to produce truly innovative DNA sequences.

Expanding our exploration, we delved into the realm of synthetic gene generation. By detecting Open Reading Frames (ORFs) within our synthetic sequences, we identified potential protein-coding regions that could be translated into amino acid sequences. This process allowed us to analyze the biological relevance and potential functions of the proteins encoded by our synthetic genes, opening avenues for further research and application.

In conclusion, this tutorial has demonstrated the power and potential of DNA LLMs in generating synthetic DNA sequences and exploring synthetic gene generation. By combining advanced computational techniques with biological insights, we have shown how to create novel sequences that not only mimic natural genomes but also offer new possibilities for synthetic biology and genetic engineering. As we continue to refine these methods, the future holds exciting prospects for innovation and discovery in the field of genomics.

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python", "deep-learning-without-gai-with-python", "genomic-llm-pretraining", "genomic-llm-finetuning", "genomic-llm-zeroshot-prediction"]


topic: statistics
tutorial: FNN
title: Deep Learning (Part 1) - Feedforward neural networks (FNN)
url: https://training.galaxyproject.org/training-material/topics/statistics/tutorials/FNN/tutorial.html
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: 2a22ca431cb5befbde05489bcf21e856
last_modified: 2026-05-27T09:01:49.813045
zenodo_link: https://zenodo.org/record/4660497
workflows_json: []
content:
Artificial neural networks are a machine learning discipline roughly inspired by how neurons in a
human brain work. In the past decade, there has been a huge resurgence of neural networks thanks
to the vast availability of data and enormous increases in computing capacity (Successfully
training complex neural networks in some domains requires lots of data and compute capacity). There
are various types of neural networks (Feedforward, recurrent, etc.). In this tutorial, we discuss
feedforward neural networks (FNN), which have been successfully applied to pattern classification,
clustering, regression, association, optimization, control, and forecasting ({% cite JainEtAl %}).
We will discuss biological neurons that inspired artificial neural networks, review activation
functions, classification/regression problems solved by neural networks, and the backpropagation
learning algorithm. Finally, we construct a FNN to solve a regression problem using car purchase
price prediction dataset.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Inspiration for artificial neural networks

A neuron is a special biological cell with information processing ability ({% cite JainEtAl %}).
Figure 1 shows a biological neuron. It has a cell body and two outreaching tree-like branches:
axon and dendrites. A neuron receives signals from other neurons through its dendrites, and
transmits signals generated by its cell body to other neurons via its axon. A synapse is a place
of contact between two neurons, an axon strand of one neuron and a dendrite strand of another
neuron. A synapse can either enhance or inhibit the signal that passes through it. Learning occurs
by changing the effectiveness of synapse. If the signals received exceeds a threshold, the neuron
*fires*, i.e., it transmits a signal to other neurons. If not, it will not fire.

![Sketch of a biological neuron and its components](../../images/FNN_bio_neuron.png "A biological neuron ({% cite JainEtAl %})")

Cerebral cortex, the outer most layer of the brain, is a sheet of neurons about 2 to 3 mm thick,
with a surface area of about 2,200 $$ cm^{2} $$. Cerebral cortex has about $$ 10^{11} $$ neurons.
Each neuron is connected to $$ 10^{3} to 10^{4} $$ neurons. Hence, a human brain has around
 $$ 10^{14} to 10^{15}$$ connections. Neurons communicate by a short train of signals, usually
milliseconds in duration. The frequency in which the signals are transmitted can be up to several
hundred Hertz, which is millions of times slower than an electronic circuit. However, complex tasks,
such as face recognition are made within a few hundred milliseconds. This implies that computation
involved cannot take more than 100 serial steps, i.e., brain runs parallel programs that are about
100 steps long for such complex perceptual tasks. The amount of information sent from one neuron to
another is also very small. This implies that critical information is not transmitted directly, but
is captured by the interconnections. What enables slow computing elements in the brain to perform
complex tasks so quickly is the distributed computation and representation nature of the brain ({% cite JainEtAl %}).

# Perceptron

Perceptron ({% cite Rosenblatt %}) is the oldest neural network still in use today. It’s a form of a
feedforward neural network, in which the connections between the nodes do not form a loop. It accepts
multiple inputs, each input is multiplied by a weight, and the products are added up. The weights
simulate the role of synapse in biological neurons (to enhance or inhibit a signal). A *bias* value
is then added to the result before it is passed to an *activation function*. An activation function
simulates the neuron firing or not. For example, in a *binary step* activation function, if the sum of
weighted inputs and bias is greater than zero, the neuron output is 1 (it fires). Else, the neuron
output is 0 (it does not fire). Bias allows us to shift the activation function.

$$ f(x) =
\left\{
	\begin{array}{ll}
		1  & \mbox{if } \boldsymbol{x} \cdot \boldsymbol{w} + b \geq 0 \\
		0  & \mbox{otherwise }
	\end{array}
\right.
$$

Figure 2 shows a Perceptron, a single layer FNN, where the input is 3 dimensional (input layer has
3 nodes), and output is 1 dimensional (output layer has 1 node).

![Neurons forming the input and output layers of a single layer feedforward neural network](../../images/FFNN_no_hidden.png "A perceptron")

In supervised learning, we are given a set of input-output pairs, called the *training
set*. Given the training set, the learning algorithm (iteratively) adjusts the model
parameters (weights and biases), so that the model can accurately map inputs to outputs.
The learning algorithm for Perceptron is very simple and reduces the weights (via a
small learning rate multiplier) if the predicted output is more than the expected output
and increases them otherwise ({% cite Rosenblatt %}).

Minsky and Papert showed that a single layer FNN cannot solve problems in which the data is not
linearly separable, such as the XOR problem ({% cite Newell780 %}). Adding one (or more) hidden
layers to FNN enables it to solve problems in which data is non-linearly separable. Per Universal
Approximation Theorem, a FNN with one hidden layer can represent any function ({% cite Cybenko1989 %}),
although in practice training such a model is very difficult (if not impossible), hence, we usually
add multiple hidden layers to solve complex problems.

![Neurons forming the input, output, and hidden layers of a multi-layer feedforward neural network](../../images/FFNN.png "Feedforward neural network with a hidden layer. Biases to hidden/output layer neurons are omitted for clarity")

The problem with multi-layer FNN was lack of a learning algorithm, as the Perceptron's learning
algorithm could not be extended to multi-layer FNN. This along with Minsky and Papert highlighting
the limitations of Perceptron resulted in sudden drop in interest in neural networks (referred to
as *AI winter*). In the 80's the backpropagation algorithm was proposed ({% cite Rumelhart1986 %}),
which enabled learning in multi-layer FNN and resulted in a renewed interest in the field.

In a multi-layer neural network, we have an input layer, an output layer, and one or more hidden layers
(between input and output layers). The input layer has as many neurons as the dimension of the input
data. The number of neurons in the output layer depends on the type of the problem the neural network
is trying to solve (See Supervised learning section below). The more hidden layers that we have (and
the more neurons we have in each hidden layer), our neural network can estimate more complex functions.
However, this comes at the cost of increased training time (due to increased number of parameters) and
increased likelihood of *overfitting*. Overfitting happens when a model captures the details of the
training data, performs well on the training data, but is unable to perform well on data not used in
the training. The neural network, hence, cannot *generalize* to unseen data. There are regularization
techniques that can prevent that ({% cite KukackaEtAl  %}) but they are outside the scope of this tutorial.

# Activation functions

There are many activation functions besides the step function used in Perceptron ({% cite nwankpaEtAl %}). Figure 4 shows
some of the more common activation functions.

![Table showing the formula, graph, derivative, and range of common activation functions](../../images/FNN_activation_functions.png "Common activation functions (Source: https://en.wikipedia.org/wiki/Activation_function)")

Linear activation function is used in the output layer of a network when we have a regression problem. It does
not make sense to use it in all layers, as such multi-layer network can be reduced to a single layer network.
Also, networks with linear activation functions cannot model non-linear relationships between input and output.

Binary step activation function is used in Perceptron. It cannot be used in multi-layers networks as they use
back propagation learning algorithm, which changes network weights/biases based on the derivative
of the activation function, which is zero. Hence, there would no weights/biases updates in back propagation.

Sigmoid activation function can be used both at the output layer and hidden layers of a multilayer network. They
allow the network to model non-linear relationships between input and output. The problem with Sigmoid activation
function is that the derivative values away from the origin are very small and quickly approach zero. In a multi
layer network, in order to calculate weight updates in layers closer to the input layer, we use the chain rule
which requires multiplying multiple Sigmoid derivative values (formula given in Backpropagation learning algorithm
section below). Multiplying multiple small numbers results in a *very* small number, meaning that the weight updates
will be minimal and the learning algorithm will be very slow. This is known as the *vanishing gradient* problem. In
networks with many hidden layers (so called *deep networks*), we generally avoid Sigmoid and use ReLU activation
function.

Hyperbolic tangent (or tanh), similar to Sigmoid function, is a soft step function. But its range is between -1
and 1 (instead of 0 and 1). One benefit of tanh over Sigmoid is that its derivative values are larger, so it
suffers less from the vanishing gradient problem.

Finally, ReLU (Rectified Linear Unit) is an activation function popular is deep neural networks. Since it
does not suffer from vanishing gradient problem, it is preferred to Sigmoid or tanh. Sigmoid or tanh can
still be used in the output layer of deep networks.

# Supervised learning

In supervised learning a *training set* is defined as
$$ {(\boldsymbol{x^{(1)}}, \boldsymbol{y^{(1)}}), ((\boldsymbol{x^{(2)}}, \boldsymbol{y^{(2)}}), ..., ((\boldsymbol{x^{(m)}}, \boldsymbol{y^{(m)}})} $$
and each pair $$ (\boldsymbol{x^{(i)}}, \boldsymbol{y^{(i)}}) $$ is called a *training example*.
*m* is the number of training examples and $$ \boldsymbol{x^{(i)}} $$ is called *feature vector*
or *input vector*.  Each element of the vector is called a *feature*. Each $$ \boldsymbol{x^{(i)}} $$
corresponds to a label $$ \boldsymbol{y^{(i)}} $$. We assume there is an unknown function
$$ \boldsymbol{y} = f(\boldsymbol{x})$$ that maps the feature vectors to labels. The goal of
supervised learning is to use the training set to learn or estimate *f*. We call this estimated
function $$ \hat{f}(\boldsymbol{x}) $$. We want $$ \hat{f}(\boldsymbol{x}) $$ to be close to
$$ f(\boldsymbol{x})$$ not only for training set, but for training examples not in training
set ({% cite Bagheri %}).

When the label is a numerical variable, we call the problem a *regression* problem, and when it's a categorical variable,
we call it a *classification* problem. In classification problems, the label can be represented by the set
$$ \boldsymbol{y^{i}} \in {1,2,...,c}$$, where each number is a class label and *c* is the number of classes.
If *c*=2, the class labels are mutually exclusive, we call it a *binary classification* problem. If *c* > 2, and
the labels are mutually exclusive, we call *multiclass classification* problem. If labels are *not* mutually exclusive,
we call it a *multilabel classification* problem ({% cite Bagheri %}).

We use a method called *one-hot encoding* to convert binary and multiclass classification class label numbers into
binary values. We convert the scalar label *y* into a vector $$  \boldsymbol{y} $$ which has *c* elements. When y is
equal to k, the k-th element of $$ \boldsymbol{y} $$ is one and all other elements are zero. When labels are *not*
mutually exclusive, we use another method called *multi-hot encoding*. Suppose we are doing a multilabel image
classification, where an image can have a dog, panda, or cat in it. We represent the label by a vector of 3, and if
dog and cat are present in the image, first and third element of the vector are one and the second element is zero ({% cite Bagheri %}).

![Three images illustrating binary, multiclass, and multilabel classifications and their label representation](../../images/FNN_output_encoding.png "Examples of binary, multiclass, and multilabel classifications and their label representation ({% cite Bagheri %})")

Figure 5 shows examples of binary, multiclass, and multilabel classification problems and their associated one-hot
encoded or multi-hot encoded labels. The output layer of a neural network for binary classification usually has a
single neuron with Sigmoid activation function. If the neuron's output is greater than 0.5, we assume the output is 1,
and otherwise, we assume the output is 0. For multilabel classification problems, the output layer of the neural network
usually has as many neurons as the number of classes and the neurons use Sigmoid activation function. Again, we use a
threshold of 0.5 to determine whether the output of each neuron is 1 or 0. For multiclass classification problems, the
output layer usually has as many neurons as the number of classes. However, instead of Sigmoid,  we use a *Softmax*
activation function, which takes the input to all the neurons in the output layer, and creates a probability distribution,
so, the sum of outputs of all output layer neurons adds up to 1. The neuron with the highest probability denotes the predicted
label.

# Loss/Cost function

During training, for each training example in the training set $$ (\boldsymbol{x^{(i)}}, \boldsymbol{y^{(i)}}) $$, we
present the feature vector $$ \boldsymbol{x^{(i)}} $$ to the neural network, and compare the network's predicted output
$$ \boldsymbol{\hat{y}} $$ with the corresponding label $$ \boldsymbol{y^{(1)}} $$. We need to define a **loss function**
to objectively measure how much the network's predicted output is different than the expected output (the corresponding
label). We use the **cross entropy** loss function for classification problems, and *quadratic* loss function for
regression problems.

For multiclass classification problems, the cross entropy is calculated as below

$$ \mathcal{L}(\boldsymbol{\hat{y}^{(j)}}, \boldsymbol{y^{(j)}}) = - \sum_{i=1}^{c} \boldsymbol{y_{i}^{(j)}}ln(\boldsymbol{\hat{y}_{i}^{(j)}}) $$

You can find the cross entropy formula for binary and multilabel classifications in {% cite Bagheri %}. They are just special
cases of multiclass cross entropy and are not give here for the sake of brevity.

The loss function is calculated for each training example in the training set. The average of the calculated loss functions
for all training examples in the training set is the **cost function**. For multiclass classification problems, the cost function
is calculated as below (again refer to {% cite Bagheri %} for binary classification and multilabel classification formulas).

$$ J(\boldsymbol{W}, \boldsymbol{b}) = - \frac{1}{m} \sum_{j=1}^{m} \sum_{i=1}^{c} \boldsymbol{y_{i}^{(j)}}ln(\boldsymbol{\hat{y}_{i}^{(j)}}) $$

For regression problems, the quadratic loss function is calculated as below:

$$ \mathcal{L}(\boldsymbol{\hat{y}^{(j)}}, \boldsymbol{y^{(j)}}) = \frac{1}{2} \| \boldsymbol{y^{(j)}} - \boldsymbol{\hat{y}^{(j)}} \| ^ 2$$

Similarly, the *quadratic* cost function (or *Mean Squared Error (MSE)*) is the average of the calculated loss functions
for all training examples in the training set.

$$ J(\boldsymbol{W}, \boldsymbol{b}) = \frac{1}{2m} \sum_{j=1}^{m} \| \boldsymbol{y^{(j)}} - \boldsymbol{\hat{y}^{(j)}} \| ^ 2 $$

# Backpropagation Learning algorithm

The **backpropagation** algorithm {% cite Rumelhart1986 %} is a gradient descent technique. Gradient descent aims to find
a local minimum of a function by iteratively moving in the opposite direction of the gradient (i.e., the slope) of the
function at the current point. The goal of a learning in neural networks is to minimize the cost function given the
training set. The cost function is a function of network weights and biases of all the neurons in all the layers.
Backpropagation iteratively computes the gradient of cost function relative to each weight and bias, then updates
the weights and biases in the opposite direction of the gradient, to find a local minimum.

In order to specify the formula for backpropagation, we need to define the error of the $$i^{th}$$ neuron in $$l^{th}$$
layer of a network for the $$j^{th}$$ training example as follows (where $$ z_{i}^{[l](j)} $$ is the weighted some of
input to the neuron, and $$ \mathcal{L} $$ is the loss function):

$$ \delta_{i}^{[l](j)} = \frac{\partial \mathcal{L}(\boldsymbol{\hat{y}^{(j)}}, \boldsymbol{y^{(j)}})}{\partial z_{i}^{[l](j)}}$$

Backpropagation formulas are expressed in terms of the error defined above. Full derivation of the formulas below is outside
the scope of this tutorial (Repeated use of chain rule is needed. Please refer to the excellent article by {% cite Bagheri %}
for details). Note that in formulas below L denotes the output layer, g the activation function, $$ \nabla $$ the gradient,
$$ W^{[l]^{T}} $$ layer l weights transposed, $$ b_{i}^{l} $$ bias of neuron i at layer l, $$ w_{ik}^{l} $$ weight to neuron
i at layer l from neuron k from layer l-1, and $$ a_{k}^{[l-1](j)} $$ activation of neuron k at layer l-1 for training example j.


$$ \boldsymbol{\delta}^{[L](j)} = \nabla_{\boldsymbol{\hat{y}^{(j)}}}\mathcal{L} \odot (g^{[L]})^{'} (\boldsymbol{z}^{[L](j)}) = \boldsymbol{\hat{y}^{(j)}} - \boldsymbol{y^{(j)}} $$

$$ \boldsymbol{\delta}^{[l](j)} = W^{[l+1]^{T}} \boldsymbol{\delta}^{[l+1](j)}  \odot (g^{[l]})^{'} (\boldsymbol{z}^{[l](j)}) $$

$$ \frac{\partial L}{\partial b_{i}^{[l]}} = \boldsymbol{\delta}_{i}^{[l](j)} $$

$$ \frac{\partial L}{\partial w_{ik}^{[l]}} = \boldsymbol{\delta}_{i}^{[l](j)} a_{k}^{[l-1](j)} $$

As you can see, we can calculate the error at the output layer for sample *j* using the first equation. Afterwards, we can calculate
the error in the layer right before the output layer for sample *j* using the second equation. The second equation is recursive,
meaning that we can calculate the error in any layer, given the error values for the next layer. This backward calculation of the
errors is the reason this algorithm is called *backpropagation*.

After the error values for all the layers are calculated for sample *j*, we use the third and fourth equations to calculate the gradient
of loss function relative to biases and weights for sample *j*. We can repeat these steps for all the samples, average the gradients of
the loss function relative to biases and weights and use the average value to update the biases and weights. This is called *batch
gradient descent*. If we have too many samples, such calculations will take a long time. An alternative is to update the biases/weights
based on the gradient of each sample. This is called *stochastic gradient descent*. While this is much faster than batch gradient descent,
the gradient calculated based on a single sample is not a good estimate of the gradient calculated in the batch version of the algorithm.
A middle ground solution is to calculate the gradient of a *batch*, and update the biases and weights based on the average of the gradients
in the batch. This is called *mini-batch gradient descent* and is preferred to the other two variations of the algorithm.

Also, note that in the second equation which is recursive, we have a term that is the derivative of the activation function for that
layer. The recursive nature of this equation means, calculating the error values in the layer prior to the output layer requires 1
multiplication by the derivative value; calculating the error values in two (or more) layers before the output layer requires 2 (or more)
multiplication by the derivative values. If these derivative values are small, as could be the case for the Sigmoid function, the product
of multiple small values will result in a *very* small value (e.g., 0.001). Since these error values decide the updates for biases and weights, this
means the update to biases and weights in layers closer to the input layer will be very small, slowing the learning algorithm to a halt.
This phenomenon is known as the *vanishing gradient* problem and is the reason Sigmoid function cannot be used in very deep networks (And
why ReLU is so popular in deep networks).

# Get Data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://zenodo.org/record/4660497) or from the shared data library
>
>    ```
>    https://zenodo.org/record/4660497/files/X_test.tsv
>    https://zenodo.org/record/4660497/files/X_train.tsv
>    https://zenodo.org/record/4660497/files/y_test.tsv
>    https://zenodo.org/record/4660497/files/y_train.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 3. Rename the datasets as `X_test`, `X_train`, `y_test`, and `y_train` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype of all the four datasets is `tabular`. If not, change the dataset's datatype to tabular.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

# Solve a simple regression problem using car purchase price prediction dataset via FNN in Galaxy

In this section, we define a FNN ({% cite Python %}) and train it using car purchase price prediction dataset ({% cite Grogan %}). Given 5 attributes
about an individual (age, gender, average miles driven per day, personal debt, and monthly income), and the money they spent on purchasing
a car, the goal is to learn a model such that given an individual's attributes, we can accurately predict how much money they are will spend
purchasing a car. We then evaluate the trained FNN on the test dataset and plot various graphs to assess the model's performance. Our training
dataset has 723 training examples, and our test dataset has 242 test examples. Input features have been scaled to be in 0 to 1 range.

## Create a deep learning model architecture

> <hands-on-title>Model config</hands-on-title>
>
> - {% tool [Create a deep learning model architecture](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/1.0.10.0) %}
>    - *"Select keras model type"*: `sequential`
>    - *"input_shape"*: `(5,)`
>    - In *"LAYER"*:
>        - {% icon param-repeat %} *"1: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `12`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"2: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `8`
>                - *"Activation function"*: `relu`
>        - {% icon param-repeat %} *"3: LAYER"*:
>            - *"Choose the type of layer"*: `Core -- Dense`
>                - *"units"*": `1`
>                - *"Activation function"*: `linear`
>    - Click *"Run Tool"*
{: .hands_on}

Input has 5 attributes: age, gender, average miles driven per day, personal debt, and monthly income. Our neural network has 3 layers. All
three layers are fully connected. The last layer has a single neuron with a linear activation function, used in regression problems. Prior
layers use ReLU activation function. The model config can be downloaded as a JSON file.

## Create a deep learning model

> <hands-on-title>Model builder (Optimizer, loss function, and fit parameters)</hands-on-title>
>
> - {% tool [Create deep learning model](toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/1.0.10.0) %}
>    - *"Choose a building mode"*: `Build a training model`
>    - *"Select the dataset containing model configuration"*: Select the *Keras Model Config* from the previous step.
>    - *"Do classification or regression?"*: `KerasGRegressor`
>    - In *"Compile Parameters"*:
>        - *"Select a loss function"*: `mse / MSE / mean_squared_error`
>        - *"Select an optimizer"*: `Adam - Adam optimizer `
>        - *"Select metrics"*: `mse / MSE / mean_squared_error`
>    - In *"Fit Parameters"*:
>        - *"epochs"*: `150`
>        - *"batch_size"*: `50`
>    - Click *"Run Tool"*
{: .hands_on}

A loss function measures how different the predicted output is from the expected output. For regression problems, we use
*Mean Squared Error (MSE)* loss function, which averages the square of the difference between predicted and actual values for
the batch. Epochs is the number of times the whole training data is used to train the model. If we update network weights/biases
after all the training data is fed to the network, the training will be slow (as we have 723 training examples in our dataset).
To speed up the training, we present only a subset of the training examples to the network, after which we update the weights/biases.
batch_size decides the size of this subset (which we set to 50). The model builder can be downloaded as a zip file.

## Deep learning training and evaluation

> <hands-on-title>Training the model</hands-on-title>
>
> - {% tool [Deep learning training and evaluation](toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.10.0) %}
>    - *"Select a scheme"*: `Train and Validate`
>    - *"Choose the dataset containing pipeline/estimator object"*: Select the *Keras Model Builder* from the previous step.
>    - *"Select input type:"*: `tabular data`
>        - *"Training samples dataset"*: Select `X_train` dataset
>        - *"Does the dataset contain header:"*: `Yes`
>        - *"Choose how to select data by column:"*: `All columns`
>        - *"Dataset containing class labels or target values"*: Select `y_train` dataset
>        - *"Does the dataset contain header:"*: `Yes`
>        - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
>
{: .hands_on}

The training step generates 2 datasets. 1) accuracy of the trained model, 2) the trained model, in h5mlm format. These files are needed for prediction in the next step.

## Model Prediction

> <hands-on-title>Testing the model</hands-on-title>
>
> - {% tool [Model Prediction](toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.10.0) %}
>    - *"Choose the dataset containing pipeline/estimator object"* : Select the trained model from the previous step.
>    - *"Select invocation method"*: `predict`
>    - *"Select input data type for prediction"*: `tabular data`
>    - *"Training samples dataset"*: Select `X_test` dataset
>    - *"Does the dataset contain header:"*: `Yes`
>    - *"Choose how to select data by column:"*: `All columns`
>    - Click *"Run Tool"*
>
{: .hands_on}

The prediction step generates 1 dataset. It's a file that has the predicted car purchase price for every row in the test dataset.

## Plot actual vs predicted curves and residual plots

> <hands-on-title>Check and visualize the predictions</hands-on-title>
>
> - {% tool [Plot actual vs predicted curves and residual plots](toolshed.g2.bx.psu.edu/repos/bgruening/plotly_regression_performance_plots/plotly_regression_performance_plots/0.1) %}
>    - *"Select input data file"*: `y_test`
>    - *"Select predicted data file"*": Select `Model Prediction` from the previous step
>    - Click *"Run Tool"*
>
{: .hands_on}

This step generates 3 graphs. The first graph (Figure 6) plots true vs predicted values. The closer the points are to each other,
the better our model's performance at predicting. The second graph (Figure 7) is a scatter plot of true vs predicted values. If
our model predicts all values correctly, we would get a diagonal line (going from bottom left to upper right). The more the
predicted vs true points are off this diagonal line, the worse our model's performance at predicting. The R2 (coefficient of
determination) score for our model is 0.87 (out of the best possible score of 1.0). The RMSE (root mean squared error) is 0.11.
The best value for RMSE is obviously 0 for perfect prediction. The third graph (Figure 8) plots residual (predicted - true) vs
predicted values. The better our model's predictions, the closer the points to y=0 line.

![True vs predicted values plot](../../images/FNN_true_predicted_plot.png "True vs predicted values plot")

![Scatterplot of true vs predicted values plot](../../images/FNN_scatter_plot.png "Scatterplot of true vs predicted values plot")

![Residual vs predicted values plot](../../images/FNN_residual_plot.png "Residual vs predicted values plot")

# Conclusion


In this tutorial, we discussed the inspiration behind the neural networks, and explained Perceptron, one of the earliest neural
networks still in use today. We then discussed different activation functions, what supervised learning is, what are loss/cost functions,
and how backpropagation minimizes the cost function. Finally, we implemented a FNN in Galaxy to solve a regression problem on car purchase price prediction data.

tools_json:
["Create a deep learning model architecture", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_config/keras_model_config/1.0.10.0", "Create deep learning model", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_model_builder/keras_model_builder/1.0.10.0", "Deep learning training and evaluation", "toolshed.g2.bx.psu.edu/repos/bgruening/keras_train_and_eval/keras_train_and_eval/1.0.10.0", "Model Prediction", "toolshed.g2.bx.psu.edu/repos/bgruening/model_prediction/model_prediction/1.0.10.0", "Plot actual vs predicted curves and residual plots", "toolshed.g2.bx.psu.edu/repos/bgruening/plotly_regression_performance_plots/plotly_regression_performance_plots/0.1"]


topic: statistics
tutorial: classification_machinelearning
title: Classification in Machine Learning
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: 0ea2d75dc8146c7ee8a0866c50a96e58
last_modified: 2026-05-27T09:01:49.815751
zenodo_link: https://zenodo.org/record/3738729
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/classification_machinelearning/tutorial.html

content:
In this tutorial you will learn how to apply Galaxy tools to solve [classification](https://en.wikipedia.org/wiki/Statistical_classification) problems. First, we will introduce classification briefly, and then examine logistic regression, which is an example of a linear classifier. Next, we will discuss the nearest neighbor classifier, which is a simple but nonlinear classifier. Then advanced classifiers, such as support vector machines, random forest and ensemble classifiers will be introduced and applied. Furthermore, we will show how to visualize the results in each step.

Finally, we will discuss how to train the classifiers by finding the values of their parameters that minimize a cost function. We will work through a real problem in the field of cheminformatics to learn how the classifiers and learning algorithms work.

Classification is a [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning) method in machine learning and the algorithm which is used for this learning task is called a classifier. In this tutorial we will build a classifier which can predict whether a chemical substance is biodegradable or not. Substances which degrade quickly are preferable to those which degrade slowly, as they do not accumulate and pose a risk to the environment. Therefore, it is useful to be able to predict easily in advance whether a substance is biodegradable prior to production and usage in consumer products.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Classification

Classification is the process of assigning every object from a collection to exactly one class from a known set of classes by learning a "decision boundary" in a dataset. This dataset is called a training dataset and contains multiple samples, together with a desired class for each sample. The training dataset contains "features" as columns and a mapping between these features and the class label is learned for each sample.

The performance of mapping is evaluated using a test dataset, which is separate from the training dataset. The test dataset contains only the feature columns, but not the class column. The class column is predicted using the mapping learned on the training dataset. An example of a classification task is assigning a patient (the object) to a group of healthy or ill (the classes) people on the basis of his or her medical record. In this tutorial, we will use a classifier to train a model using a training dataset, predict the targets for test dataset and visualize the results using plots.

![classification](images/classification.png "Classification of samples belonging to different classes.")

In figure [1](#figure-1), the line is a boundary which separates a class from another class (for example from tumor to no tumor). The task of a classifier is to learn this boundary, which can be used to classify or categorize an unseen/new sample. The line is the decision boundary; there are different ways to learn it, which correspond to different classification algorithms. If the dataset is linearly separable, linear classifiers can produce good classification results. However, when the dataset is complex and requires non-linear decision boundaries, more powerful classifiers like `support vector machine` or `ensemble` based classifiers may prove to be beneficial.

The data classification process includes two steps:
1. Building the classifier or model: This step is the learning step, in which the classification algorithms build the classifier. The classifier is built from the training set made up of database samples and their associated class labels. Each sample that constitutes the training set is referred to as a class.

2. Applying the classifier to a classification task: In this step, the classifier is used for classification. Here the test data is used to estimate the accuracy of classification rules. The classification rules can be applied to the new data samples if the accuracy is considered acceptable.


# Quantitative structure - activity relationship biodegradation

The classification problem we will study in this tutorial is related to biodegradation. Chemical substances which decay slowly will accumulate over time, which poses a threat to the environment. Therefore, it is useful to be able to predict in advance whether a substance will break down quickly or not.

Quantitative structure-activity relationship (QSAR) and quantitative structure-property relationship (QSPR) models attempt to predict the activity or property of chemicals based on their chemical structure. To achieve this, a database of compounds is collected for which the property of interest is known. For each compound, molecular descriptors are collected which describe the structure (for example: molecular weight, number of nitrogen atoms, number of carbon-carbon double bonds). Using these descriptors, a model is constructed which is capable of predicting the property of interest for a new, unknown molecule. In this tutorial we will use a database assembled from experimental data of the Japanese Ministry of International Trade and Industry to create a classification model for biodegradation. We then will be able to use this model to classify new molecules into one of two classes: biodegradable or non-biodegradable.

As a benchmark, we will use the [dataset](https://pubs.acs.org/doi/10.1021/ci4000213) assembled by Mansouri et al. using data from the National Institute of Technology and Evaluation of Japan. This database contains 1055 molecules, together with precalculated molecular descriptors.

In this tutorial, we will apply a couple of [scikit-learn](https://scikit-learn.org/stable/) machine learning tools to the dataset provided by Mansouri et al. to predict whether a molecule is biodegradable or not.
In the following part, we will perform classification on the biodegradability dataset using a linear classifier and then will create some plots to analyze the results.

## Get train and test datasets

We have two datasets available; the training dataset contains 837 molecules, while the test dataset contains 218 molecules.

Let's begin by uploading the necessary datasets.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/record/3738729/files/train_rows.csv
>    https://zenodo.org/record/3738729/files/test_rows_labels.csv
>    https://zenodo.org/record/3738729/files/test_rows.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets as `train_rows`, `test_rows_labels` and `test_rows` respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

The `train_rows` contains a column `Class` which is the class label or target. We will evaluate our model on `test_rows` and compare the predicted class with the true class value in `test_rows_labels`
{: .comment}

> <details-title>Preparing the data for classification</details-title>
>
> Preparing the data involves these following major tasks:
> 1. Data cleaning: involves removing noise and treatment of missing values. The noise is removed by applying noise filtering techniques and the problem of missing values is solved by replacing a missing value with different techniques, for example substitution, mean imputation and regression imputation.
> 2. Relevance analysis: the database may also have attributes which are irrelevant for classification. Correlation analysis is used to know whether any two given attributes are related - e.g. one of the features and the target variable.
> 3. Normalization: the data is transformed using normalization. Normalization involves scaling all values for q given attribute in order to make them fall within a small specified range. Normalization is used when in the learning step, neural networks or the methods involving measurements are used.
>
{: .details}

# Learn the logistic regression classifier

As the first step, to learn the mapping between several features and the classes, we will apply the linear classifier. It learns features from the training dataset and maps all the rows to their respective class. The process of mapping gives a trained model. [Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression) is named for the function used at the core of the method, the logistic function, and it is an instance of supervised classification in which we know the correct label of the class for each sample and the algorithm estimate of the true class. We want to learn parameters (weight and bias for the line) that make the estimated class for each training observation as close as possible to the true class label. This requires two components; the first is a metric for how close the current class label is to the true label. Rather than measure similarity, we usually talk about the opposite of this, the distance between the classifier output and the desired output, and we call this distance, the loss function or the cost function.

The second thing we need is an optimization algorithm for iteratively updating the weights so as to minimize this loss function. The standard algorithm for this is gradient descent. So, the dataset is divided into two parts - training and test sets. The training set is used to train a classifier and the test set is used to evaluate the performance of the trained model.

> <hands-on-title>Train logistic regression classifier</hands-on-title>
>
> 1. {% tool [Generalized linear models for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_generalized_linear/sklearn_generalized_linear/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Select a linear method"*: `Logistic Regression`
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Class`
>             - {% icon param-file %} *"Dataset containing class labels"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - {% icon param-text %} *"Select target column(s)"*: `Class`
> 2. Rename the generated file to `LogisticRegression_model`
{: .hands_on}

> <question-title></question-title>
>
> What is learned by the logistic regression model?
>
> > <solution-title></solution-title>
> >
> > In the logistic regressoion model, the coefficients of the logistic regression algorithm have be estimated from our training data. This is done using [maximum-likelihood estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation).
> > 
> {: .solution}
>
{: .question}

## Predict class using test dataset

After learning on the training dataset, we should evaluate the performance on the test dataset to know whether the learning algorithm learned a good classifier from the training dataset or not. This classifier is used to predict a new sample and a similar accuracy is expected.

Now, we will predict the class in the test dataset using this classifier in order to see if it has learned important features which can be generalized on a new dataset. The test dataset (`test_rows`) contains the same number of features but does not contain the `Class` column. This is predicted using the trained classifier.


> <hands-on-title>Predict class using the logistic regression classifier</hands-on-title>
>
> 1. {% tool [Generalized linear models for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_generalized_linear/sklearn_generalized_linear/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `LogisticRegression_model`
>       - {% icon param-file %} *"Data (tabular)"*: `test_rows`
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `LogisticRegression_result`
{: .hands_on}

## Visualize the logistic regression classification results

We will evaluate the classification by comparing the predicted with the expected classes. In the previous step, we classified the test dataset (`LogisticRegression_result`). We have one more dataset (`test_rows_labels`) which contains the true class label of the test set. Using the true and predicted class labels in the test set, we will verify the performance by analyzing the plots. As you can see, `LogisticRegression_result` has no header, so first we should remove the header from `test_rows_labels` to compare. 

> <hands-on-title>Remove the header</hands-on-title>
>
> 1. **Remove beginning** {% icon tool %} with the following parameters:
>       - {% icon param-file %} *"Remove first"*: `1`
>       - {% icon param-file %} *"from"*: `test_rows_labels`
> 2. Rename the generated file to `test_rows_labels_noheader`
{: .hands_on}


Now we visualize and analyze the classification using the "Plot confusion matrix, precision, recall and ROC and AUC curves" tool.

> <hands-on-title>Check and visualize the classification</hands-on-title>
>  {% tool [Plot confusion matrix, precision, recall and ROC and AUC curves](toolshed.g2.bx.psu.edu/repos/bgruening/plotly_ml_performance_plots/plotly_ml_performance_plots/0.4) %}:
>    - {% icon param-file %} *"Select input data file"*: `test_rows_labels_noheader`
>    - {% icon param-file %} *"Select predicted data file"*: `LogisticRegression_result`
>    - {% icon param-file %} *"Select trained model"*: `LogisticRegression_model`
{: .hands_on}

The visualization tool creates the following plots:

1. [Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix): The confusion matrix summarizes the classification performance of a classifier with respect to the test data. It is a two-dimensional matrix; the horizontal axis (x-axis) shows the predicted labels and the vertical axis (y-axis) shows the true labels. Each rectangular box shows a count of samples falling into the four output combinations (true class, predicted class) - (1, 0), (1, 1), (0, 1) and (0, 0). In Figure 2, the confusion matrix of the predictions is a colour-coded heatmap. For a good prediction, the diagonal running from top-left to bottom-right should contain a smaller number of samples, because it shows the counts of incorrectly predicted samples. Hovering over each box in Galaxy shows the true and predicted class labels and the count of samples.

    ![confusion_matrix](images/confusion_matrix_linear.png "Confusion matrix for the logistic regression classifier. ")

2. [Precision, recall and F1 score](https://en.wikipedia.org/wiki/Precision_and_recall): Precision, recall and F1 score. These scores determine the robustness of classification. It is important to analyze the plot for any classification task to verify the accuracy across different classes which provides more information about the balanced or imbalanced accuracy across multiple classes present in the dataset.

    ![prf1_scores](images/precision_recall_linear.png "Precision, recall and F1 score for the logistic regression classifier.")

3. [Receiver operator characteristics (ROC) and area under ROC (AUC)](https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5): Receiver operator characteristics (ROC) and area under ROC (AUC). The ROC curve is shown in blue. For a good prediction, it should be more towards the top-left of this plot, which results in a high AUC value. For a bad prediction, it is close to the orange line (y = x), resulting in a low AUC value (closer to 0.5). An AUC value of exactly 0.5 means the prediction is doing no better than a random number generator at predicting the classes.

    ![roc_scores](images/roc_linear.png "Receiver operator characteristics (ROC) and area under ROC (AUC) for the logistic regression classifier.")

These plots are important to visualize the quality of the classifier and the true and predicted classes.


> <question-title></question-title>
>
> Inspect the plots. What can you say about the classification?
>
> > <solution-title></solution-title>
> >
> > Figures 2,3 and 4 show that the classification is acceptable, but as you will see in the next steps, the results can be improved. 
> >
> {: .solution}
{: .question}

# K-Nearest Neighbor (KNN)

At the second step, we will use k-nearest neighbor classifier. In the [k-nearest neighbor](https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) classifier, a sample is classified by a majority vote of its neighbors.  The sample is assigned to the class which is most common among its k nearest neighbors.  k is a positive integer and typically it is small. For example, if k = 1, then the sample is simply assigned to the class of that single nearest neighbor. Surprisingly, when the number of data points is large, this classifier is not that bad. Choosing the best value of k is very important. If k is too small, the classifier will be sensitive to noise points and if k is too large, the neighborhood may include points from other classes and causes errors. To select the k that is right for your data, we recommend running the KNN algorithm several times with different values of k and choosing the k that reduces the number of errors the most.

> <question-title></question-title>
>
> What are advantages and disadvantages about this model?
>
> > <solution-title></solution-title>
> > Advantages:
> > - It is a very simple algorithm to understand and interpret.
> >
> > - It is very useful for nonlinear data because there is no assumption of linearity in this algorithm.
> >
> > - It is a versatile algorithm, as we can use it for classification as well as regression.
> >
> > - It has relatively high accuracy, but there are much better supervised learning models than KNN.
> >
> > - It works very well in low dimensions for complex decision surfaces.
> >
> > Disadvantages:
> >
> > - Classification is slow, because it stores all the training data.
> >
> > - High memory storage required as compared to other supervised learning algorithms.
> >
> > - Prediction is slow in case of big training samples.
> >
> > - It is very sensitive to the scale of data as well as irrelevant features.
> >
> > - It suffers a lot from the curse of dimensionality.
> >
> {: .solution}
{: .question}

> <hands-on-title>Train k-nearest neighbor classifier</hands-on-title>
>
> {% tool [Nearest Neighbors Classification](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_nn_classifier/sklearn_nn_classifier/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Classifier type"*: `Nearest Neighbors`
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Class`
>             - {% icon param-file %} *"Dataset containing class labels"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - {% icon param-text %} *"Select target column(s)"*: `Class`
>             - {% icon param-select %} *"Neighbor selection method"*: `k-nearest neighbors`
> 2. Rename the generated file to `NearestNeighbors_model`
{: .hands_on}

> <question-title></question-title>
>
> What is the value of k (number of neighbors) for the model?
>
> > <solution-title></solution-title>
> > As you can see in the Advanced Options, the default value for the number of neighbors is 5, and we used the default value. You can set this parameter based on your problem and data. 
> >
> {: .solution}
{: .question}

Now, we should evaluate the performance on the test dataset to find out whether the KNN classifier is a good model from the training dataset or not. 

> <hands-on-title>Predict class using the k-nearest neighbor classifier</hands-on-title>
>
> {% tool [Nearest Neighbors Classification](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_nn_classifier/sklearn_nn_classifier/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `NearestNeighbors_model`
>       - {% icon param-file %} *"Data (tabular)"*: `test_rows`
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `NearestNeighbors_result`
{: .hands_on}


Now we visualize and analyze the classification. As you can see, `NearestNeighbors_result` has a header, so use `test_rows_labels` to compare. 

> <hands-on-title>Check and visualize the classification</hands-on-title>
>  {% tool [Plot confusion matrix, precision, recall and ROC and AUC curves](toolshed.g2.bx.psu.edu/repos/bgruening/plotly_ml_performance_plots/plotly_ml_performance_plots/0.4) %}:
>    - {% icon param-file %} *"Select input data file"*: `test_rows_labels`
>    - {% icon param-file %} *"Select predicted data file"*: `NearestNeighbors_result`
>    - {% icon param-file %} *"Select trained model"*: `NearestNeighbors_model`
{: .hands_on}

The visualization tool creates diagrams for the Confusion matrix, Precision, recall and F1 score, Receiver operator characteristics (ROC) and area under ROC (AUC) as follows:

![confusion_matrix](images/confusion_matrix_NN.png "Confusion matrix for the k-nearest neighbor classifier.")

![prf1_scores](images/precision_recall_NN.png "Precision, recall and F1 score for the k-nearest neighbor classifier.")

![roc_scores](images/roc_NN.png "Receiver operator characteristics (ROC) and area under ROC (AUC) for the k-nearest neighbor classifier.")


# Support Vector Machines (SVM)

[Support Vector Machines](https://en.wikipedia.org/wiki/Support-vector_machine) (SVMs) have been extensively researched in the machine learning community for the last decade and actively applied to applications in various domains such as bioinformatics. SVM is a generalization of a classifier called maximal margin classifier and is introduced as a binary classifier intended to separate two classes when obtaining the optimal hyperplane and decision boundary. SVMs are based on the assumption that the input data can be linearly separable in a geometric space. The maximal margin classifier is simple, but it cannot be applied to the majority of datasets, since the classes must be separated by a linear boundary and this is often not the case when working with real world data. That is why the support vector classifier was introduced as an extension of the maximal margin classifier, which can be applied in a broader range of cases.

To solve this problem, SVM uses kernel functions to map the input to a high dimension feature space, i.e hyperplane, where a linear decision boundary is constructed in such a manner that the boundary maximises the margin between two classes. The kernel approach is simply an efficient computational approach for accommodating a non-linear boundary between classes.

Without going into technical details, a kernel is a function that quantifies the similarity of two observations. Two special properties of SVMs are that SVMs achieve (1) high generalization by maximizing the margin and (2) support an efficient learning of nonlinear functions by
kernel trick. In the next step, we will build a SVM classifier with our data. 

> <hands-on-title>Train a SVM classifier</hands-on-title>
>
>  {% tool [Support vector machines (SVMs)](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_svm_classifier/sklearn_svm_classifier/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Select a linear method"*: `Linear Support Vector Classification`
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Class`
>             - {% icon param-file %} *"Dataset containing class labels"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - {% icon param-text %} *"Select target column(s)"*: `Class`
> 2. Rename the generated file to `SVM_model`
{: .hands_on}

> <question-title></question-title>
>
> What is learned by the support vector machines?
>
> > <solution-title></solution-title>
> >
> > The coefficients of the line with the maximal margin in the kernel space is learned in the training phase.
> > 
> {: .solution}
>
{: .question}


Now we will evaluate the performance of the SVM classifier:

> <hands-on-title>Predict class SVM classifier</hands-on-title>
>
>  {% tool [Support vector machines (SVMs)](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_svm_classifier/sklearn_svm_classifier/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `SVM_model`
>       - {% icon param-file %} *"Data (tabular)"*: `test_rows`
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `SVM_result`
{: .hands_on}


Now let's visualize the results:

> <hands-on-title>Check and visualize the classification</hands-on-title>
>  {% tool [Plot confusion matrix, precision, recall and ROC and AUC curves](toolshed.g2.bx.psu.edu/repos/bgruening/plotly_ml_performance_plots/plotly_ml_performance_plots/0.4) %}:
>    - {% icon param-file %} *"Select input data file"*: `test_rows_labels`
>    - {% icon param-file %} *"Select predicted data file"*: `SVM_result`
>    - {% icon param-file %} *"Select trained model"*: `SVM_model`
{: .hands_on}

The visualization tool creates the following ROC plot:

![roc_scores](images/roc_svm.png "Receiver operator characteristics (ROC) and area under ROC (AUC) for the SVM classifier.")


# Random Forest

[Random forest](https://en.wikipedia.org/wiki/Random_forest) is an ensemble of decision trees, and usually trained with the “bagging” method. The [Ensemble](https://scikit-learn.org/stable/modules/ensemble.html#ensemble) method uses multiple learning models internally for better predictions and the general idea of the bagging method is that a combination of learning models increases the overall result. It uses multiple decision tree regressors internally and predicts by taking the collective performances of the predictions by multiple decision trees. It has a good predictive power and is robust to outliers. It creates an ensemble of weak learners (decision trees) and iteratively minimizes error.

One big advantage of random forest is that it can be used for both classification and regression problems. The main idea behind the random forest is adding additional randomness to the model, while growing the trees and instead of searching for the most important feature while splitting a node, it searches for the best feature among a random subset of features. This results in a better model because of wide diversity. Generally, the more trees in the forest, the more robust the model. Therefore, when using the random forest classifier, a larger number of trees in the forest gives higher accuracy results. Similarly there are two stages in the random forest algorithm: one is random forest creation, the other is to make a prediction from the random forest classifier created in the first stage.

> <hands-on-title>Train random forest</hands-on-title>
>
> 1. {% tool [Ensemble methods for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_ensemble/sklearn_ensemble/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Train a model`
>       - *"Select an ensemble method"*: `Random forest classifier`
>          - *"Select input type"*: `tabular data`
>             - {% icon param-file %} *"Training samples dataset"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>                - {% icon param-text %} *"Type header name(s)"*: `Class`
>             - {% icon param-file %} *"Dataset containing class labels"*: `train_rows`
>             - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>             - {% icon param-select %} *"Choose how to select data by column"*: `Select columns by column header name(s)`
>                - {% icon param-text %} *"Select target column(s)"*: `Class`
> 2. Rename the generated file to `RandomForest_model`
{: .hands_on}

> <question-title></question-title>
>
> What are the advantages of random forest classifier compared with KNN and SVM?
>
> > <solution-title></solution-title>
> > 1. It is resistant to overfitting as it produces results based on multiple decision trees.
> > 2. It scales well with large datasets.
> > 3. The random forest algorithm can be used for feature engineering, which means identifying the most important features out of the available features from the training dataset.
> {: .solution}
>
{: .question}


After learning on the training dataset, we should evaluate the performance on the test dataset.

> <hands-on-title>Predict targets using the random forest</hands-on-title>
>
> 1. {% tool [Ensemble methods for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_ensemble/sklearn_ensemble/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>       - {% icon param-file %} *"Models"*: `RandomForest_model`
>       - {% icon param-file %} *"Data (tabular)"*: `test_rows`
>       - {% icon param-check %} *"Does the dataset contain header"*: `Yes`
>       - {% icon param-select %} *"Select the type of prediction"*: `Predict class labels`
> 2. Rename the generated file to `RandomForest_result`
{: .hands_on}

The visualization tool creates the following ROC plot:

![roc_scores](images/roc_rf.png "Receiver operator characteristics (ROC) and area under ROC (AUC) for the random forest classifier.")


> <question-title></question-title>
>
> Inspect the plots. What can you say about the classification?
>
> > <solution-title></solution-title>
> >
> > Figures show that we achieved an AUC score of `1.0`  for the test set using random forest. It means the prediction is very good, in fact it has no error at all. Unfortunately, this is not usually the case when dealing with chemical data.
> {: .solution}
{: .question}


# Create data processing pipeline

At the last step, we will create a bagging classifier by using  the **Pipeline builder** tool. Bagging or Bootstrap Aggregating is a widely used ensemble learning algorithm in machine learning. The bagging algorithm creates multiple models from randomly taken subsets of the training dataset and then aggregates learners to build overall stronger classifiers that combine the predictions to produce a final prediction. The **Pipeline builder** tool builds the classifier and returns a `h5mlm` file. This file also contains a list of all the different hyperparameters of the preprocessors and estimators which will be used in the **Hyperparameter search** tool to populate the list of hyperparameters with their respective (default) values.

> <hands-on-title>Create pipeline</hands-on-title>
>
> {% tool [Pipeline builder](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_build_pipeline/sklearn_build_pipeline/1.0.11.0) %}:
>    - In *"Final Estimator"*:
>        - *"Choose the module that contains target estimator"*: `sklearn.ensemble`
>            - *"Choose estimator class"*: `BaggingClassifier`
> 
>      We choose `Final Estimator` as we have only the estimator and no preprocessor and need the parameters of only the estimator.
>
{: .hands_on}


## Search for the best values of hyperparameters

After extracting the parameter names from the **Pipeline builder** file, we will use the **Hyperparameter search** tool to find the best values for each hyperparameter. These values will lead us to create the best model based on the search space chosen for each hyperparameter. We use only one parameter `n_estimators` of `BaggingClassifier` for this task. This parameter specifies the number of bagging stages the learning process has to go through. The default value of `n_estimators` for this regressor is `10`. However, we are not sure if this gives the best accuracy. Therefore, it is important to set this parameter to different values to find the optimal one. We choose a value which is less than `10` and a few which are more than `10`. The hyperparameter search will look for the optimal number of estimators and gives the best-trained model as one of the outputs. This model is used in the next step to classify the test dataset.

> <hands-on-title>Hyperparameter search</hands-on-title>
>
> {% tool [Hyperparameter search](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_searchcv/sklearn_searchcv/1.0.11.0) %}:
>    - *"Select a model selection search scheme"*: `GridSearchCV - Exhaustive search over specified parameter values for an estimator `
>        - {% icon param-files %} *"Choose the dataset containing pipeline/estimator object"*: `h5mlm` file (output of **Pipeline builder** {% icon tool %})
>        - {% icon param-files %} *"Is the estimator a deep learning model?"*: `NO` {% icon tool %})
>        - In *"Search parameters Builder"*:
>             - In *"Parameter settings for search"*:
>                 - {% icon param-repeat %} *"1: Parameter settings for search"*
>                    - *"Choose a parameter name (with current value)"*: `n_estimators: 10`
>                    - *"Search list"*: `[5,10,20,50]`
>    - *"Select input type"*: `tabular data`
>        - {% icon param-files %} *"Training samples dataset"*: `train_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns BUT by column header name(s)`
>            - *"Type header name(s)"*: `Class`
>        - {% icon param-files %} *"Dataset containing class labels or target values"*: `train_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `Select columns by column header name(s)`
>            - *"Type header name(s)"*: `Class`
>    - *"Whether to hold a portion of samples for test exclusively?"*: `Nope`
>    - *"Save best estimator?"*: `Fitted best estimator or Detailed cv_results_from nested CV`
>
{: .hands_on}

> <question-title></question-title>
>
> What is the optimal number of estimators for the given dataset?
>
> Hint: Please look at the `mean_test_score` column in the tabular result from the **Hyperparameter search** tool.
>
> > <solution-title></solution-title>
> >
> > 20 - even though the default value of the number of estimators for Bagging Classifier is `10`, `20` gives the best accuracy. That's why it is important to perform hyperparameter search to tune these parameters for any dataset. 
> >
> {: .solution}
>
{: .question}

Using the **Hyperparameter search** tool, we found the best model, based on the training data. Now, we will predict age in the test dataset using this model.

> <hands-on-title>Predict age</hands-on-title>
>
> 1. {% tool [Ensemble methods for classification and regression](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_ensemble/sklearn_ensemble/1.0.11.0) %}:
>    - *"Select a Classification Task"*: `Load a model and predict`
>        - {% icon param-files %} *"Models"*: `h5mlm` file (output of **Hyperparameter search** {% icon tool %})
>        - {% icon param-files %} *"Data (tabular)"*: `test_rows` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>
{: .hands_on}


Now we will verify the performance by creating and inspecting the plots:


![confusion_matrix](images/confusion_matrix_bagging.png "Confusion matrix for the bagging classifier.")

![prf1_scores](images/precision_recall_bagging.png "Precision, recall and F1 score for the bagging classifier.")

![roc_scores](images/roc_bagging.png "Residual plot between residual (predicted - true) and predicted targets. The plot shows a random pattern of points.")


Figure 13 shows that we again achieved an AUC value of `1.00`, which shows that our model is highly effective at predicting whether or not a molecule is biodegradable.


# Conclusion
By following these steps, we learned how to build classifiers and visualize the classification results using Galaxy's machine learning and plotting tools. The features of the training dataset are mapped to the classes. This mapping is used to make predictions on an unseen (test) dataset. The quality of classifiers is visualized using a plotting tool.

There are multiple other classification algorithms, a few are simpler to use (with fewer parameters) and some are more powerful, which can be tried out on this dataset and on other datasets as well. Different datasets can also be analyzed using these classifiers. The classifiers have many parameters which can be altered while performing the analyses to see if they affect the classification accuracy. It may be beneficial to perform a hyperparameter search to tune these parameters for different datasets. In addition, we learned the relevance of machine algorithms for QSAR analyses and constructed a model which successfully predicted an important chemical property - the biodegradability of a substance.

tools_json:
["Generalized linear models for classification and regression", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_generalized_linear/sklearn_generalized_linear/1.0.11.0", "Plot confusion matrix, precision, recall and ROC and AUC curves", "toolshed.g2.bx.psu.edu/repos/bgruening/plotly_ml_performance_plots/plotly_ml_performance_plots/0.4", "Nearest Neighbors Classification", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_nn_classifier/sklearn_nn_classifier/1.0.11.0", "Support vector machines (SVMs)", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_svm_classifier/sklearn_svm_classifier/1.0.11.0", "Ensemble methods for classification and regression", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_ensemble/sklearn_ensemble/1.0.11.0", "Pipeline builder", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_build_pipeline/sklearn_build_pipeline/1.0.11.0", "Hyperparameter search", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_searchcv/sklearn_searchcv/1.0.11.0"]


topic: statistics
tutorial: iwtomics
title: Interval-Wise Testing for omics data
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 418f0b64a19e4e5e6022d239aa84d030
last_modified: 2026-05-15T22:28:17.030026
zenodo_link: https://doi.org/10.5281/zenodo.5589610
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/iwtomics/tutorial.html

content:
IWTomics ({% cite Cremona2018 %}) implements the Interval-Wise Testing (IWT; {% cite Pini2017 %}) for omics data. This
inferential procedure tests for differences in "Omics" data between two groups
of genomic regions, and does not require fixing location and scale at the
outset.

In this tutorial we use real data about endogenous retroviruses (ERVs) in mouse.
In particular, we compare recombination hotspots in the flanking regions of fixed ETn versus control regions.
The dataset contains two region datasets "ETn fixed", "Control" and one feature "Recombination hotspots content". The region dataset "ETn fixed" contains 1296 genomic regions of 64 kb surrounding fixed ETns elements (32-kb flanking sequences upstream and 32-kb flanking sequences downstream of each element). The region dataset "Control" contains 1142 regions of 64 kb without elements, used as control in the test. The regions are aligned around their center (i.e. around the ETn integration sites).

Recombination hotspots measurements are associated to each "ETn fixed" and "Control" region. In particular, this feature is measured in 1-kb windows, so that each region is associated to a recombination hotspots curve made of 64 values. The measurement used is the feature content, i.e. the fraction of the 1-kb window that is covered by recombination hotspots

The data we use in this tutorial is available at [Zenodo](https://doi.org/10.5281/zenodo.5589610).

> <agenda-title></agenda-title>
>
> IWTomics is composed of three steps (corresponding to three tools):
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Step 1: Loading and pre-processing

The first tool (IWTomics Load Smooth and Plot) imports a collection of genomic region datasets, and associates to each region multiple genomic feature measurements. It allows to align the regions in multiple ways (center, left, right or scale alignment), to smooth the feature curves (possibly filling gaps in the measurements) and to create a graphical representation of the feature measurements in each region datasets (aligned curves or pointwise quantile curves).

> <hands-on-title>Get the data</hands-on-title>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import datasets:
>    - Fixed ETn (`ETn_fixed.bed`) regions
>    - Control (`Control.bed`) regions
>    - Recombination Hotspot (`Recombination_hotspots.txt`) content
>
>    ```
>    https://zenodo.org/record/5589610/files/ETn_fixed.bed
>    https://zenodo.org/record/5589610/files/Control.bed
>    https://zenodo.org/record/5589610/files/Recombination_hotspots.txt
>    ```
> 3. Import header files for regions and features:
>    - `regions_header.tabular`
>    - `features_header.tabular`
>
>    ```
>    https://zenodo.org/record/5589610/files/regions_header.tabular
>    https://zenodo.org/record/5589610/files/features_header.tabular
>    ```
>
>    Please note that the file names in the first column of the header files must exactly match the dataset names in the Galaxy history.
{: .hands_on}

> <hands-on-title>Pre-process data and create pointwise boxplot</hands-on-title>
> **Load, Smooth and Plot** {% icon tool %}: Run **Load, Smooth and Plot** with:
>    * Select the two region datasets `ETn_fixed.bed` and `Control.bed`.
>    * Select the region header file `regions_header.tabular`.
>    * Select the feature file `Recombination_hotspot.txt`.
>    * Select the feature header file `features_header.tabular`.
>    * Start positions are 0-based in this example; all region alignments are equivalent in this case.
>    * You can choose to smooth the feature curves.
>    * Select the graphical representation type **Pointwise quantile curves (boxplot)**.
>
> ![Recombination hotspot pointwise boxplot](../../images/load_smooth_and_plot_output.png)
{: .hands_on}

# Step 2: Performing Interval-Wise Testing

The second tool (IWTomics Test and Plot) statistically evaluates differences in genomic features between groups of regions along the genome. In particular, it implements the Interval-Wise Testing for omics data, an extended version of the Interval-Wise Testing for functional data presented in {% cite Pini2017 %}.

It allows to perform multiple two sample permutation tests between pairs of region datasets, on several features. It returns the adjusted p-value curves for every test and all possible scales. Moreover, it creates a graphical representation of the Interval-Wise Testing results and a summary plot (optional) with p-values at the maximum scale. The tool IWTomics Plot with Threshold on Test Scale permits to select the scale to be used in the plots.

> <hands-on-title>Test for difference between ETn and Control regions</hands-on-title>
> **Test and Plot** {% icon tool %}: Run **Test and Plot** with:
>    * Select the `IWTomicsData Object`, `Region Dataset IDs`, and `Feature IDs` created during **Step 1**.
>    * Select pairs of genomic region datasets to be compared using two-sample Interval-Wise Testing. In this example we consider one two-sample test contrasting `ETn_fixed` (select it as **Region 1**) to `Control` (select it as **Region 2**).
>    * Select feature measurements to be tested. In this example, select `Recomb_hot` feature.
>    * You can choose the test statistic, the number of permutations, and the significance level of the test (alpha).
>    * Select the graphical representation type **Pointwise quantile curves (boxplot)**.
>
> ![Recombination hotspot IWT results](../../images/test_and_plot_output.png)
{: .hands_on}

# Step 3: Selecting test scale

The third tool (IWTomics Plot with Threshold on Test Scale) allows to select the scale for the Interval-Wise Testing results. In particular it returns the p-value curves for the different tests performed at the selected scale, and it creates a graphical representation of the Interval-Wise Testing results and a summary plot (optional) at the selected scale.

> <hands-on-title>Change scale for test results</hands-on-title>
> **Plot with Threshold on Test Scale** {% icon tool %}: Run **Plot with Threshold on Test Scale** with:
>    * Select the `IWTomicsData Object with Test Results`, `Test IDs`, and `Feature IDs` created during **Step 2**.
>    * Select the scale to be used in Interval-Wise Testing results and plot. Select the test identifier `ETn_fixed vs Control`, the feature identifier `Recomb_hot`, and the scale `8` (8-kb scale).
>    * Select the graphical representation type **Pointwise quantile curves (boxplot)**.
>
> ![Recombination hotspot IWT results scale 8](../../images/plot_with_threshold_on_test_scale_output.png)
{: .hands_on}

> <details-title>Additional resources</details-title>
>
> Read more about **IWTomics** [in the documentation (PDF)](https://bioconductor.org/packages/release/bioc/vignettes/IWTomics/inst/doc/IWTomics.pdf).
{:.details}


topic: statistics
tutorial: genomic-llm-pretraining
title: Pretraining a Large Language Model (LLM) from Scratch on DNA Sequences
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml", "Large Language Model"]
content_hash: 96ed7fff1a61dd5e98bd3fe11e535422
last_modified: 2026-05-27T09:01:49.829819
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/genomic-llm-pretraining/tutorial.html

content:
**Generative Artificial Intelligence** (AI) represents a cutting-edge domain within machine learning, focused on creating new, synthetic yet realistic data. This includes generating text, images, music, and even biological sequences. At the heart of many generative AI applications are **Large Language Models** (LLMs), which have revolutionized natural language processing and beyond.

LLMs are **sophisticated neural networks** trained on vast amounts of text data to understand, generate, and interact with human language. Their architecture, often based on **Transformers**, allows them to capture complex patterns and context within data, making them powerful tools for various applications, from chatbots to creative writing and scientific discovery. 

> <details-title> Transformers </details-title>
> Transformers are a type of neural network model designed to handle sequential data, such as text, by using self-attention mechanisms to weigh the importance of input elements relative to each other, enabling the model to understand and generate coherent and contextually relevant outputs.
> 
{: .details}

In this tutorial, we will explore the intersection of generative AI and genomics by **pretraining an LLM from scratch on DNA sequences**. This process will equip the model with a foundational understanding of the "grammar" of DNA, enabling it to generate and analyze genetic data with remarkable accuracy.

[Mistral AI](https://mistral.ai/), French artificial intelligence (AI) startup, recently launched large language models (LLMs) showing performances superior to Llama2. In particular, Mixtral-8x7B implements:

- **Grouped-Query Attention**: Efficiently computes attention by grouping queries, reducing computational load and memory usage.
- **Sliding-Window Attention**: Focuses on a fixed-size window of tokens, sliding over the sequence to manage long texts efficiently.
- **Byte-fallback BPE Tokenizer**: Tokenizes text into subword units, falling back to byte-level tokenization for unknown words, ensuring robust handling of diverse text inputs.

These techniques collectively enhance the performance and efficiency of large language models, enabling them to process and generate text more effectively.

In this tutorial, we will use a simplified Mistral model architecture with fewer layers and hidden units to reduce computational requirements. The model will be trained to predict the next base in the sequence. For instance, for a sequence like `ATTTGTTGGT`, the model will be trained to predict the suffix `TTGGT` given the prefix `ATTTG`. This process is called **causal language modeling**.

To pretrain the model, we will use a file containing 100,000 non-overlapping DNA sequences of 200 bases, corresponding to around 1% of the human genome (hg38 assembly). This involves training the model to predict the end of a DNA sequence. 

By the end of this tutorial, we will obtain a Mistral-DNA model with an internal representation of DNA sequence grammar. This pretrained model can then be used for various applications, such as fine-tuning for classification tasks or predicting mutational effects.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prepare resources

To pretrain the model, let's open a Notebook or a Python script.

## Install dependencies

The first step is to install the required dependencies:

```python
!pip install accelerate
!pip install datasets==3.0.1
!pip install transformers
!pip install torch
!pip install flash-attn
```

> <question-title></question-title>
>
> What are the required dependencies doing?
>
> > <solution-title></solution-title>
> >
> > - `accelerate`: A library by [Hugging Face](https://huggingface.co/) -- a platform that provides tools and resources for building, training, and deploying machine learning models -- designed to simplify the process of training and deploying machine learning models across different hardware environments. It provides tools to optimize performance on GPUs, TPUs, and other accelerators, making it easier to scale models efficiently.
> > 
> > - `datasets`: A library by Hugging Face for managing and processing datasets. It provides tools to load, manipulate, and share datasets in a standardized format, making it easier to work with machine learning data.
> > 
> > - `numpy`: A fundamental package for scientific computing in Python.
> > 
> > - `torch`: Also known as PyTorch, it is an open-source machine learning library developed by Facebook's AI Research lab. It provides a flexible platform for building and training neural networks, with a focus on tensor computations and automatic differentiation.
> > 
> > - `transformers`: A library by Hugging Face that provides implementations of state-of-the-art transformer models for natural language processing (NLP). It includes pre-trained models and tools for fine-tuning, making it easier to apply transformers to various NLP tasks.
> >
> > - `flash-attn`: Implementation of FlashAttention, a Fast and Memory-Efficient Exact Attention with IO-Awareness
> > These libraries are widely used in the machine learning and data science communities for their efficiency, flexibility, and extensive functionality.
> {: .solution}
>
{: .question}

## Import Python libraries

Let's now import them.

```python
import os

import accelerate
import flash_attn
import torch
import transformers
from datasets import load_dataset
from transformers import (
    AutoConfig,
    AutoModelForCausalLM,
    AutoTokenizer,
    DataCollatorForLanguageModeling,
    EarlyStoppingCallback,
    Trainer,
    TrainingArguments,
)
```

> <details-title>Loaded functions and classes from datasets and transformers libraries</details-title>
>
> - `datasets`:
>   - `load_dataset`: function to load datasets from the Hugging Face Hub or local files. 
> - `transformers`:
>   - `AutoConfig`: Automatically loads the configuration for a pre-trained model. It defines the architecture and hyperparameters of the model.
>   - `AutoModelForCausalLM`: Loads a pre-trained causal language model for tasks like text generation, where the model predicts the next token in a sequence.
>   - `AutoTokenizer`: Loads the tokenizer associated with a pre-trained model. It converts text into tokens that the model can process.
>   - `DataCollatorForLanguageModeling`: A data collator specifically designed for language modeling tasks. It prepares batches of data for training by handling padding and masking.
>   - `EarlyStoppingCallback`: A callback used during training to stop the process early if the model's performance on the validation set stops > improving, saving time and resources.
>   - `Trainer`: A high-level API for training and evaluating transformer > models. It simplifies the training loop and handles tasks like gradient accumulation and evaluation.
>   - `TrainingArguments`: A class to define the training configuration, including hyperparameters like learning rate, batch size, and number > of epochs. It is used to configure the `Trainer`.
> 
> These components work together to streamline the process of training and fine-tuning transformer models for various NLP tasks.
> 
{: .details}

> <comment-title>Versions</comment-title>
>
> This tutorial has been tested with following versions:
> - `accelerate` > 0.32.1
> - `flash_attn` > 2.6.0.post1 and 2.7.0.post2
> - `transformers` > 4.47.1
> 
> You can check the versions with:
>
> ```python
> accelerate.__version__
> flash_attn.__version__
> transformers.__version__
> ```
{: .comment}

## Check and configure available resources

To pretrain the model, we need to specific resources:
- **Graphics Processing Unit** (GPU): a specialized processor designed to handle complex graphical computations, often used for rendering images, videos, and accelerating machine learning tasks
- **Video Random Access Memory** (VRAM): dedicated memory used by a GPU to store and process graphical data, enabling smooth rendering of images and videos

Let's check the resources:

```python
!nvidia-smi
```

The command `nvidia-smi` (NVIDIA System Management Interface) is used to monitor and manage NVIDIA GPU devices. It provides information about the GPU's utilization, memory usage, temperature, and running processes. This tool is essential for developers and researchers to track the performance and health of GPUs, especially when running computationally intensive tasks like machine learning training. 

> <question-title></question-title>
>
> How do you interpret the following output?
>
> ```
> Tue Mar 25 13:49:35 2025       
> +-----------------------------------------------------------------------------> ------------+
> | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA > Version: 12.4     |
> |-----------------------------------------+------------------------> +----------------------+
> | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile > Uncorr. ECC |
> | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | > GPU-Util  Compute M. |
> |                                         |                        |          >      MIG M. |
> |=========================================+========================> +======================|
> |   0  Tesla T4                       Off |   00000000:00:04.0 > Off |                    0 |
> | N/A   40C    P8              9W /   70W |       2MiB /  15360MiB |      > 0%      Default |
> |                                         |                        |          >         N/A |
> +-----------------------------------------+------------------------> +----------------------+
>                                                                               >            
> +-----------------------------------------------------------------------------> ------------+
> | > Processes:                                                                    >           |
> |  GPU   GI   CI        PID   Type   Process > name                              GPU Memory |
> |        ID   > ID                                                               Usage      |
> |> ==============================================================================> ===========|
> |  No running processes > found                                                             |
> +-----------------------------------------------------------------------------> ------------+
> ```
>
> > <solution-title></solution-title>
> >
> > - `Driver Version`: The version of the NVIDIA driver installed on the system (`550.54.15`).
> > - `CUDA Version`: The version of CUDA installed, which is a parallel computing platform and API model created by NVIDIA (`12.4`).
> > - `GPU Name`: The model of the GPU, in this case, a `Tesla T4`.
> > - `Persistence-M`: Indicates whether Persistence Mode is enabled (`Off` in this case), which can improve performance for certain applications.
> > - `Bus-Id`: The PCI bus ID of the GPU (`00000000:00:04.0`).
> > - `Fan`: The speed of the GPU fan (`N/A` means not available or not reporting).
> > - `Temp`: The current temperature of the GPU (`40°C`).
> > - `Perf`: The performance state of the GPU (P8 indicates a low-power state).
> > - `Pwr:Usage/Cap`: The current power usage (9W) and the power cap (70W).
> > - `Memory-Usage`: The amount of GPU memory currently in use (2MiB) out of the total available (15360MiB).
> > - `GPU-Util`: The percentage of GPU utilization (0% indicates the GPU is idle).
> > - `Compute M.`: The compute mode of the GPU (Default).
> > - `Processes`: Lists any processes currently using the GPU. In this case, there are no running processes.
> >
> {: .solution}
>
{: .question}

Let's configure PyTorch and the CUDA environment -- software and hardware ecosystem provided by NVIDIA to enable parallel computing on GPU -- to optimize GPU memory usage and performance:

1. Enables CuDNN benchmarking in PyTorch:

    ```python
    torch.backends.cudnn.benchmark=True
    ```

    > <question-title></question-title>
    >
    > 1. What is CuDNN?
    > 2. Why enabling benchmarking?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. CuDNN is a GPU-accelerated library for deep neural networks.
    > > 2. Enabling benchmarking allows CuDNN to select the fastest algorithms for the specific GPU and input size. This can improve the performance of the model, especially for fixed-size inputs.
    > >
    > {: .solution}
    >
    {: .question}

2. Set an environment variable that configures how PyTorch manages CUDA memory allocations

    ```python
    os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"
    ```

    > <question-title></question-title>
    >
    > What is this command doing?
    >
    > > <solution-title></solution-title>
    > >
    > > It sets the maximum split size for memory allocations to 32 megabytes. This can help reduce memory fragmentation and improve memory utilization, which is particularly useful when working with large models or limited GPU memory.
    > >
    > {: .solution}
    >
    {: .question}

# Prepare the model

## Load the model 

Let's load now the model, `Mistral-DNA`. The Mixtral model ([Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)) -- [a pretrained generative Sparse Mixture of Experts outperforming Llama 2 70B](https://mistral.ai/news/mixtral-of-experts) -- was modified to significantly reduce the number of parameters mostly by removing layers, such that it could be trained on a GPU such as an RTX3090.

We will get the model from GitHub:

```python
!git clone https://github.com/raphaelmourad/Mistral-DNA.git
```

Let's check if we have the model now:

```python
!ls
```

We should get two folders: `Mistral-DNA` and `sample_data`. Let's change the current working directory to `Mistral-DNA/`:

```python
os.chdir("Mistral-DNA/")
```

## Choose the LLM architecture

Let's look at the original archicture of `Mixtral-8x7B-v0.1` which is stored in the `data/models/Mixtral-8x7B-v0.1` folder ([GitHub](https://github.com/raphaelmourad/Mistral-DNA/tree/main/data/models/Mixtral-8x7B-v0.1)). 

> <question-title></question-title>
>
> 1. Which file is essential for the configuring the language model?
> 2. What are the key parameters of the simplified architecture used here?
>
> > <solution-title></solution-title>
> >
> > 1. The [`config.json` file](https://github.com/raphaelmourad/Mistral-DNA/blob/main/data/models/Mixtral-8x7B-v0.1/config.json) is essential for configuring the language model as a Mistral model. It specifies the architecture for causal language modeling (`MixtralForCausalLM`) and details the size of the neural network components. The original Mistral model has a larger hidden size, but it is reduced here to make pre-training feasible. 
> > 2. The key parameters are:
> >    - **Intermediate Size** (`intermediate_size`): Size of the intermediate (or hidden) layers within the model. It determines the number of neurons in these layers, influencing the model's capacity to capture complex patterns in the data. A larger intermediate size can capture more nuanced details but also requires more computational resources. Set to 256, which is relatively small compared to the original model.
> >    - **Number of Attention Heads** (`num_attention_heads`): Number of attention heads in the multi-head attention mechanism. Each head allows the model to focus on different parts of the input sequence simultaneously, capturing diverse aspects of the data. More attention heads can provide a richer representation but also increase computational complexity. Reduced to 8 for efficiency.
> >    - **Number of Experts per token** (`num_experts_per_tok`): Specific to models that use a Mixture of Experts (MoE) architecture. It indicates the number of expert networks that are activated for each token in the input sequence. Experts are specialized sub-networks that handle different parts of the data, improving efficiency and performance, especially for large models. Set to 1 expert per token.
> >    - **Number of Local Experts** (`num_local_experts`): Number of local experts available in the model. Local experts are a subset of the total experts and are used to process specific parts of the input data. This localization can help in managing computational resources more effectively, especially when dealing with large-scale data. Set to 64.
> >    - **Vocabulary Size** (`vocab_size`): Specifically designed for DNA sequences, with a size of $$ 4,096 = 4^6$$, as DNA consists of four possible letters (A, T, C, and G) and the words are  6-mers (sequences of six nucleotides). By modeling DNA using 6-mers, we capture meaningful patterns within the genetic sequence, enabling the model to understand and generate DNA data effectively.
> >
> {: .solution}
>
{: .question}

Let's load the configuration of the pre-trained model:

```python
config = AutoConfig.from_pretrained("data/models/Mixtral-8x7B-v0.1")
```

By loading the configuration, we can inspect or modify the model's architecture without loading the actual model weights. Let's now initialize a causal language model from the loaded configuration object, with a specific attention implementation:

```python
model = AutoModelForCausalLM.from_config(config, attn_implementation="eager")
```

> <question-title></question-title>
>
> What does `attn_implementation="eager"` do?
>
> > <solution-title></solution-title>
> >
> > `attn_implementation="eager"` specifies the attention implementation to use. Setting it to "eager" means that the attention mechanism will be executed eagerly, which can be useful for debugging or when working with dynamic computation graphs. Eager execution runs operations immediately as they are called in Python, rather than adding them to a graph for later execution.
> >
> {: .solution}
>
{: .question}

How does the model look like?

```python
model
```

```
MixtralForCausalLM(
  (model): MixtralModel(
    (embed_tokens): Embedding(4096, 256)
    (layers): ModuleList(
      (0-7): 8 x MixtralDecoderLayer(
        (self_attn): MixtralAttention(
          (q_proj): Linear(in_features=256, out_features=256, bias=False)
          (k_proj): Linear(in_features=256, out_features=256, bias=False)
          (v_proj): Linear(in_features=256, out_features=256, bias=False)
          (o_proj): Linear(in_features=256, out_features=256, bias=False)
          (rotary_emb): MixtralRotaryEmbedding()
        )
        (block_sparse_moe): MixtralSparseMoeBlock(
          (gate): Linear(in_features=256, out_features=64, bias=False)
          (experts): ModuleList(
            (0-63): 64 x MixtralBlockSparseTop2MLP(
              (w1): Linear(in_features=256, out_features=256, bias=False)
              (w2): Linear(in_features=256, out_features=256, bias=False)
              (w3): Linear(in_features=256, out_features=256, bias=False)
              (act_fn): SiLU()
            )
          )
        )
        (input_layernorm): MixtralRMSNorm((256,), eps=1e-05)
        (post_attention_layernorm): MixtralRMSNorm((256,), eps=1e-05)
      )
    )
    (norm): MixtralRMSNorm((256,), eps=1e-05)
  )
  (lm_head): Linear(in_features=256, out_features=4096, bias=False)
)
```

As expected, the model is a `MixtralForCausalLM` model with several key components:

1. **Embedding Layer (`embed_tokens`)**: Converts input DNA sequences into dense vectors of fixed size. It maps each of the 4,096 ($$4^{6}$$) possible DNA tokens (representing 6-mers) to a 256-dimensional vector space. This embedding layer is crucial for transforming discrete DNA sequences into a format suitable for neural network processing.

2. **Decoder Layers (`layers`)**: Consists of eight `MixtralDecoderLayer` modules, each containing several sub-components:
    - **Self-Attention Mechanism (`self_attn`)**

      > <question-title></question-title> 
      >
      > 1. What are the components?
      > 2. How is the purpose?
      >
      > > <solution-title></solution-title>
      > >
      > > 1. The components are linear projections (`q_proj`, `k_proj`,`v_proj`, `o_proj`) for queries, keys, values, and outputs, along witha rotary embedding (`rotary_emb`) to incorporate positiona linformation.
      > > 2. This allows the model to weigh the importance of differenttokens in the sequence relative to each other, capturing dependenciesand context.
      > >
      > {: .solution}
      >
      {: .question}

    - **Sparse Mixture of Experts (`block_sparse_moe`)**:

      > <question-title></question-title> 
      >
      > 1. What are the components?
      > 2. How is the purpose?
      >
      > > <solution-title></solution-title>
      > >
      > > 1. The components are gating mechanism (`gate`) and list of 64 expert networks (`experts`), each with multiple linear layers (`w1`, `w2`, `w3`) and an activation function (`act_fn`).
      > > 2. This efficiently processes input data by activating only a subset of expert networks, reducing computational load while maintaining model capacity.
      > >
      > {: .solution}
      >
      {: .question}

    - **Layer Normalization (`input_layernorm`, `post_attention_layernorm`)**: Stabilizes and accelerates the training process by normalizing the inputs and outputs of the attention mechanism.

3. **Final Layer Normalization (`norm`)**: Applies normalization to the output of the final decoder layer, ensuring stable and consistent outputs.

4. **Language Model Head (`lm_head`)**: Projects the 256-dimensional output of the final decoder layer back into the 4,096-dimensional vocabulary space of DNA tokens. This linear layer (`Linear`) maps the hidden states to the original token space, enabling the model to predict the next DNA token accurately.

This architecture ensures that the model can capture complex patterns in DNA sequences while maintaining computational efficiency, making it suitable for tasks like DNA sequence generation and analysis. The model's design culminates in the output of 4,096 tokens, aligning with the input dimension. This consistency is crucial for accurately predicting the next token in a given DNA sequence, ensuring that the model's predictions are coherent and reliable.


> <question-title></question-title>
>
> How many parameters are in this model?
>
> > <solution-title></solution-title>
> >
> > 
> > ```python
> > pytorch_total_params = sum(p.numel() for p in model.parameters())
> > print(f"Model size: {pytorch_total_params/1000**2:.1f}M parameters")
> > ```
> > 
> > There are 105 millions parameters. It is a big model.
> >
> {: .solution}
>
{: .question}


# Prepare the tokenizer

A tokenizer is a crucial component in natural language processing (NLP) that transforms raw text into a format that can be processed by machine learning models. In this section, we will load and configure the **Byte-Pair Encoding (BPE) letter tokenizer**. The BPE tokenizer efficiently handles rare and unknown words by breaking them down into frequent subword units, ensuring that the model can generalize better to unseen data. This process involves initializing the tokenizer with a predefined vocabulary and settings, enabling it to convert text into a format suitable for neural network processing. By doing so, we prepare the tokenizer to effectively manage DNA sequences, facilitating accurate and reliable model predictions.

Let's loads a pre-trained tokenizer from the Hugging Face Model Hub. The tokenizer is associated with the model `DNABERT-2-117M`, which is designed for processing DNA sequences.

```python
tokenizer = AutoTokenizer.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True)
```

> <question-title></question-title>
>
> What does the above command do?
>
> > <solution-title></solution-title>
> >
> > - `AutoTokenizer.from_pretrained` automatically identifies and loads the appropriate tokenizer for the specified model. There are 1876 sequences.
> > - `trust_remote_code=True` allows the loading of custom tokenizers that may include remote code execution. It is necessary when the tokenizer requires additional custom code to function correctly.
> >
> {: .solution}
>
{: .question}

Let's look at the created tokenizer now:

```python
print(tokenizer)
```

```
PreTrainedTokenizerFast(name_or_path='zhihan1996/DNABERT-2-117M',vocab_size=4096, model_max_length=1000000000000000019884624838656,is_fast=True, padding_side='right', truncation_side='right',special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': [PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'},clean_up_tokenization_spaces=False, added_tokens_decoder={
	0: AddedToken("[UNK]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	1: AddedToken("[CLS]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	2: AddedToken("[SEP]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	3: AddedToken("[PAD]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	4: AddedToken("[MASK]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)
```

The `PreTrainedTokenizerFast` is a fast and efficient tokenizer used to process text data for the `DNABERT-2-117M` model. Here's a breakdown of its configuration:

- `name_or_path='zhihan1996/DNABERT-2-117M'`: Specifies the name or path of the pre-trained tokenizer, indicating that it is associated with the `DNABERT-2-117M` model, which is designed for processing DNA sequences.

- `vocab_size=4096`: Defines the size of the tokenizer's vocabulary. 

    > <question-title></question-title>
    >
    > Why is the size of the tokenizer's vocabulary set to 4,096?
    >
    > > <solution-title></solution-title>
    > >
    > > It corresponds to the number of unique tokens (6-mers) that the model can recognize in DNA sequences.
    > >
    > {: .solution}
    >
    {: .question}

- `special_tokens`: Defines a set of special tokens used by the tokenizer:
  - `unk_token: '[UNK]'` - Represents unknown or out-of-vocabulary tokens.
  - `sep_token: '[SEP]'` - Used to separate segments within a sequence.
  - `pad_token: '[PAD]'` - Used for padding sequences to a uniform length.
  - `cls_token: '[CLS]'` - Typically used as the first token in a sequence to represent the classification token.
  - `mask_token: '[MASK]'` - Used in masked language modeling to hide tokens that the model must predict.

> <question-title></question-title>
>
> What do the other configuration parameters mean?
>
> 1. `model_max_length=1000000000000000019884624838656`
> 2. `is_fast=True`
> 3. `padding_side='right'`
> 4. `truncation_side='right'`
> 5. `clean_up_tokenization_spaces=False`
> 6. `added_tokens_decoder` 
>
> > <solution-title></solution-title>
> >
> > 1. `model_max_length=1000000000000000019884624838656`: Represents the maximum length of sequences that the model can handle. 
> > 
> >     This extremely large value suggests that the model is designed to process very long sequences, although in practice, the actual limit will be constrained by available computational resources.
> > 
> > 2. `is_fast=True`: Indicates that this tokenizer is optimized for speed, leveraging Rust-based implementations to accelerate tokenization processes.
> > 3. `padding_side='right'`: Configures the tokenizer to pad sequences on the right side, ensuring that all sequences in a batch have the same length by adding padding tokens to the end of shorter sequences.
> > 4. `truncation_side='right'`: Specifies that sequences will be truncated from the right side if they exceed the maximum length, preserving the beginning of the sequence.
> > 5. `clean_up_tokenization_spaces=False`: Indicates that the tokenizer will not remove spaces after tokenization, preserving the original spacing in the text.
> > 6. `added_tokens_decoder`: Maps token IDs to their corresponding `AddedToken` objects, which include metadata such as whether the token is a special token and how it should be processed (e.g., stripping whitespace).
> >
> {: .solution}
>
{: .question}

This configuration ensures that the tokenizer is tailored to efficiently process DNA sequences, handling both the tokenization and padding/truncation of sequences in a manner that aligns with the model's requirements.


By default, tokenizers may pad sequences on the right side (`padding_side='right'`). Let's set the padding direction for the tokenizer.

```python
tokenizer.padding_side  = "left"
```

When tokenizing a batch of sequences, shorter sequences will be padded with special tokens on the left to match the length of the longest sequence in the batch. This can be useful for ensuring consistent input sizes, especially in models that expect fixed-size inputs.

Let's look at how some DNA sequences are encoded by the tokenizer. We start with a simple sequence "ATT":

```python
encoding = tokenizer("ATT", padding="longest", return_tensors="pt")
print(encoding)
```

The code tokenizes the DNA sequence "ATT", pads it to the longest sequence in the batch (`padding="longest"`), and returns the result as PyTorch tensors (`return_tensors="pt"`). 

```
{'input_ids': tensor([[   1, 2061,    2]]), 'token_type_ids': tensor([[0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1]])}
```

Here's a breakdown of each output component:

- `input_ids`: A tensor containing the token IDs for the sequence. Each number corresponds to a specific token in the tokenizer's vocabulary. In this case, `[1, 2061, 2]` represents the tokens for the sequence:
  - `1`: the beginning of the sentence (`[CLS]`)
  - `2061`: the sentence itself (`ATT`)
  - `2`: the end of the sentence, a separator between sentence (`[SEP]`).

- `token_type_ids`: A tensor indicating the type of each token, often used in models that process multiple segments (e.g., question-answering). Here, all tokens are of type `0`, suggesting a single segment.

- `attention_mask`: A tensor that specifies which tokens should be attended to by the model (`1` for real tokens, `0` for padding). In this case, all tokens are valid, so the mask is `[1, 1, 1]`.

This encoded format is ready for input into a transformer model, ensuring that the sequence is correctly processed and understood by the model.

> <question-title></question-title>
>
> What is the encoding for "ATTGTGGGTCCCCGTAGATGATAGGGGCCCCCC"? Specify that the tokenized sequence should have a maximum length of 5 tokens and ensure that the sequence is padded to the specified `max_length` of 5 tokens.
>
> > <solution-title></solution-title>
> >
> > - To specify that the tokenized sequence should have a maximum length of 5 tokens, you need to put `max_length=5` -- if the sequence is longer, it will be truncated --
> > - To ensure that the sequence is padded to the specified `max_length` of 5 tokens, you need to add `padding='max_length'` -- if the sequence is shorter, padding tokens will be added
> >
> > ```python
> > encoding = tokenizer("ATTGTGGGTCCCCGTAGATGATAGGGGCCCCCC", max_length=5, padding='max_length', truncation=True, return_tensors="pt")
> > print(encoding)
> > ```
> >
> > ```
> > {'input_ids': tensor([[   1, 2061,  281,  485,    2]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])}
> > ```
> > In this case, `[1, 2061, 281, 485, 2]` represents the tokens for the sequence, likely including special tokens like [CLS] and [SEP]. As before, all tokens are of type `0`, suggesting a single segment, and are valid, so the mask is `[1, 1, 1, 1, 1]`.
> {: .solution}
>
{: .question}


# Prepare data

We will now prepare the data. 

## Load data

First we load the data. We will not use here the whole human genome because it comprises too many sequences. Instead, we use a small subset of the data, which is less than 1% of the sequences from the human genome.

> <comment-title>Pre-trained model on the whole human genome</comment-title>
>
> A compact DNA model with approximately 1 million parameters that has been trained on the entire human genome can be found on [Hugging Face](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-1M-hg38)
{: .comment}

We use the `load_dataset` function from the `datasets` library. This function is commonly used for loading data for Hugging Face Transformers.

```python
dataset_text = load_dataset("csv", data_files="data/genome_sequences/hg38/sequences_hg38_200b_verysmall.csv.gz")
```

> <question-title></question-title>
>
> 1. How is `dataset_text` structured?
> 2. What are the first 5 train dataset in the data?
> 3. How long are the sequences?
>
> > <solution-title></solution-title>
> >
> > 1. `dataset_text` is a `DatasetDict` with a `train` `Dataset` containing 1 feature (`'text'`) of 99,999 rows (obtained with `dataset_text`)
> > 2. To get the 5 train dataset in the data:
> > 
> >    ```python
> >    dataset_text['train']['text'][0:5]
> >    ```
> >    
> >    ```
> >    ```
> >
> > 2. The sequences are 200 base pair long: 
> >
> >    ```python
> >    len(dataset_text['train']['text'][0])
> >    ```
> >
> >    ```
> >    200
> >    ```
> {: .solution}
>
{: .question}

## Tokenize data

Let's tokenize the data. First, we create a function that tokenizes a text using the BPE letter tokenizer:

```python
def tokenize_function(examples):
    return tokenizer(examples['text'], padding="longest", truncation=True, return_tensors="pt")
```

> <question-title></question-title> 
>
> What do the following parameters?
> 
> 1. `padding="longest"`
> 2. `truncation=True`
> 3. `return_tensors="pt"`
>
> > <solution-title></solution-title>
> >
> > 1. `padding="longest"` ensures that all sequences in the batch are padded to the length of the longest sequence, adding padding tokens as needed.
> > 2. `truncation=True` specifies that sequences exceeding the model's maximum length will be truncated to fit.
> > 3. `return_tensors="pt"` indicates that the output should be in the form of PyTorch tensors, suitable for use with PyTorch-based models.
> >
> {: .solution}
>
{: .question}

We can now apply this function to the load dataset:

```python
dataset = dataset_text.map(tokenize_function, batched=True)
```

It is quite fast for the almsot 100,000 sequence of length 200 bp.

> <question-title></question-title> 
>
> 1. How is `dataset` structured?
> 2. What is in the first tokenized sequence of `train` `Dataset`?
>
> > <solution-title></solution-title>
> > 
> > 1. `dataset` is
> >    ```
> >    DatasetDict({
> >        train: Dataset({
> >            features: ['text', 'input_ids', 'token_type_ids', 'attention_mask'],
> >            num_rows: 99999
> >        })
> >    })
> >    ```
> >    
> >    `dataset` is a `DatasetDict` with 1 `train` `Dataset` made of 99,999 rows and 4 features:
> >    - `text`: The original text data before tokenization.
> >    - `input_ids`: The tokenized input data, represented as numerical IDs.
> >    - `token_type_ids`: Indicates the type of each token, useful for models that handle multiple segments.
> >    - `attention_mask`: Specifies which tokens should be attended to by the model (`1` for real tokens, `0` for padding).
> >
> > 2. The first tokenized sequence of `train` `Dataset` (`dataset["train"][1]`) is a dictionary with:
> >    - `text`: 200 base pair sequence
> >    - `input_ids`: list of 49 numerical values, the token IDs.
> >    - `token_type_ids`: list 49 `0`
> >    - `attention_mask`: list of 7 `0` (padding) and 42 `1` (real tokens)
> > 
> {: .solution}
>
{: .question}

## Split data

We will now split data between training and validation sets randomly. This is a crucial step in machine learning to ensure the model can generalize to unseen data.

For that, 80% of the entire data will be used for the training set and the remaining 20% will go into the validation set. We first compute the size of training and validation sets:

```python
train_size = int(0.8 * len(dataset["train"]))
val_size = len(dataset["train"]) - train_size
```

> <question-title></question-title> 
>
> How big are training and validation sets?
>
> > <solution-title></solution-title>
> > 
> > Training set has 79,999 sequences and the validation set 20,000.
> >
> {: .solution}
>
{: .question}

To perform the actual splitting of the training dataset into two subsets, we use the `torch.utils.data.random_split` function from the PyTorch library that randomly splits a dataset into subsets.

```python
train_set, val_set = torch.utils.data.random_split(dataset["train"], [train_size, val_size])
```

## Data Collation

The `DataCollatorForLanguageModeling` is a utility class, designed to prepare and format batches of data for language modeling tasks. It handles the dynamic padding and masking of input sequences, ensuring that each batch fed into the model is correctly formatted and optimized for training.

```python
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
```

> <question-title></question-title> 
>
> What are the different parameters?
>
> > <solution-title></solution-title>
> > 
> > - `tokenizer=tokenizer` specifies the tokenizer to be used for processing the input data. The tokenizer converts raw text into numerical tokens that the model can understand.
> > - `mlm=False`: Indicates that the data collator is set up for causal language modeling (CLM) rather than masked language modeling (MLM).
> >
> {: .solution}
>
{: .question}

This will:

1. Automatically pads sequences within a batch to ensure they are of equal length, which is necessary for efficient batch processing in neural networks.
2. Generates attention masks that indicate which tokens should be attended to by the model, ignoring padding tokens.
3. Collates individual examples into batches, handling the necessary formatting and ensuring compatibility with the model's input requirements.

The `DataCollatorForLanguageModeling` is typically used in conjunction with a `Trainer` from the Hugging Face library. It simplifies the data preparation process, allowing you to focus on model training and evaluation without worrying about the intricacies of batch formatting.

# Train the model

## Define parameters for pretraining

We are now going to defines the hyperparameters and configurations for training the language model using the Hugging Face `transformers`.

Before, we specify the batch size for training and evaluation. A batch size of 32 means that 32 samples will be processed before the model updates its weights. This size is chosen to balance computational efficiency and memory usage.

```python
batchsize=32
training_args = TrainingArguments(
  output_dir="./results/models",
  eval_strategy="epoch",
  save_strategy="epoch",
  num_train_epochs=50,
  per_device_train_batch_size=batchsize,
  per_device_eval_batch_size=batchsize,
  learning_rate=5e-4,
  weight_decay=0.01,
  logging_dir="./logs",
  load_best_model_at_end=True,
  bf16=True,
  gradient_accumulation_steps=50,
  report_to="none",
)
```

- `output_dir="./results/models"`: directory where the training outputs, including model checkpoints and results, will be saved.
- `eval_strategy="epoch"` indicates that the model's performance will be evaluated at the end of each epoch, a complete pass through the entire training dataset. This allows for monitoring the model's progress and adjusting the training process as needed.
- `save_strategy="epoch"` specifies that the model will be saved at the end of each epoch. This ensures that checkpoints are available for each complete pass through the dataset.
- `num_train_epochs=50` sets the total number of training epochs to 50. This means the model will iterate over the entire dataset 50 times, allowing it to learn and optimize over multiple passes.
- `per_device_train_batch_size=batchsize` and `per_device_eval_batch_size=batchsize` set the batch size for training and evaluation on each device (e.g., GPU) to 32. This ensures consistency in batch processing across different stages of training and evaluation.
- `learning_rate=5e-4` defines the learning rate for the optimizer, set to $$ 5 \times 10^{-4} $$. This rate controls the step size during gradient descent and is a common choice for pre-training models.
- `weight_decay=0.01` applies L2 regularization to the model weights with a standard decay rate of 0.01. This helps prevent overfitting by penalizing large weights.
- `logging_dir="./logs"` specifies the directory where training logs will be stored, allowing for monitoring and analysis of the training process.
- `load_best_model_at_end=True` ensures that the best model, based on the lowest evaluation loss, is loaded at the end of training. This helps in selecting the model with the best performance across all epochs. During gradient descent, the model will be optimized, and at some point, the loss will start to increase again. We want to pick the model with the lowest loss, not when it starts increasing. So, "load best model at the end" means selecting the model with the best loss across all epochs.
- `fp16=True` enables mixed-precision training using 16-bit floating-point numbers. This reduces memory usage and can speed up training on compatible hardware.
- `gradient_accumulation_steps=50` accumulates gradients over 50 steps before performing a backward pass. This effectively increases the batch size without requiring additional memory, helping to stabilize training.
- `report_to="none"` disables [Weights & Biases (WandB)](https://wandb.ai/), a popular platform used for experiment tracking, dataset versioning, and model management in machine learning

  > <comment-title>Why Disable WandB?</comment-title> 
  >
  > Disabling WandB is often done in specific scenarios:
  > - Avoiding Unwanted Logging: If we do not intend to use WandB for tracking our experiments or if we want to avoid potential conflicts with other logging mechanisms, we would disable it.
  > - Reducing Overhead: WandB logging can introduce some overhead,   particularly when dealing with large datasets or complex models. Disabling it can slightly improve performance if tracking is not essential.
  > - Testing/Debugging: During testing or debugging, we might prefer to have more control over logging or we might want to avoid cluttering our WandB workspace with intermediate results.
  >
  {: .comment}

> <question-title></question-title> 
>
> What is stored in `training_args`: the parameters to the model, the parameter of the LLM or the parameters of the trainer function?
>
> > <solution-title></solution-title>
> > 
> > The parameters of the trainer function
> >
> {: .solution}
>
{: .question}


## Pretrain the model

Here is the most important part: the pre-training process. For this, we will use a `Trainer` function. This function takes as input the model that we built previously, which has an architecture but no initialized weights.

```python
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_set,
    eval_dataset=val_set,
    callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]
)
```

The Trainer function also takes:
- `args`: the training arguments we configured earlier
- `data_collator`: the data collator function feeding the tokenized data sequences to the model.
- `train_dataset`: the training set, i.e. the data used for computing the gradients
- `eval_dataset`: the validation set, i.e. the data used to assess the prediction accuracy at each epoch. It's important to use a validation set that is independent of the training set to ensure unbiased evaluation.
- `callbacks`: `EarlyStoppingCallback` with a patience of three is used to monitor the training process.

  During training, we minimize the loss at each step. However, at some point, the loss may start to increase again. We want to capture the model parameters when the loss reaches its minimum. By using a patience of three, we aim to mitigate the effects of noise during training. Noise can cause fluctuations in the loss, making it seem like we've reached a local minimum when a better one might be found with further training.

  With a patience of three, even if we find a good minimum, we wait for three more epochs to ensure that the loss does not improve further. If the loss does not decrease for three consecutive epochs, we stop training. However, if a better model with a lower loss is found within those three epochs, training continues. This approach helps in finding a more robust local minimum by reducing the impact of noise in the training data.

Let's launch the training with `trainer.train()` method

```python
trainer.train()
```

Here, the trainer is set to run for 50 epochs. After the initiation, we get an estimation of the time it takes per epoch to get an idea of the total training duration. Let's run it for a bit to see how long it takes.

With this small model and dataset, the estimated time to run 50 epochs is 20 hours -- this value changes depending on the infrastructure --.

> <question-title></question-title> 
>
> Will the model be trained to 50 epochs?
>
> > <solution-title></solution-title>
> > 
> > Setting the number of epochs to 50 doesn't mean the model will train for all 50 epochs. It's likely to stop earlier
> >
> {: .solution}
>
{: .question}

The 50 epochs serve as a maximum limit. The model will stop training earlier if it reaches the minimum loss and then starts to increase again, thanks to the early stopping callback. This means the model might only require half the epochs, perhaps 25 epochs or 10 hours, to achieve optimal performance.

> <comment-title>Don't train until the end</comment-title>
>
> The idea here is not to train the model until completion, as it would take too much time.
>
{: .comment}

Let's stop the actual training and cheat a bit by loading a previously [trained Mistral model](https://huggingface.co/RaphaelMourad/Mistral-DNA-v1-1M-hg38):

```python
model = AutoModelForCausalLM.from_pretrained("RaphaelMourad/Mistral-DNA-v1-17M-hg38")
```

This is a mixed model that was pre-trained on the entire Human Genome. It contains approximately 17 million parameters and was trained using the Human Genome assembly GRCh38. Unlike models pre-trained on sequences of 200 bases, this model was pre-trained on sequences of 10,000 bases (10K). The advantage of this model is its ability to process larger DNA contexts or sequences. This capability allows it to capture more extensive patterns and dependencies within the genomic data.

> <question-title></question-title> 
>
> By looking at the output of:
>
> 
> ```python
> model
> ```
>
> 1. How many transformer layers does this model have?
> 2. Is it similar to previous model?
>
> > <solution-title></solution-title>
> > 
> > 1. 8 transformer layers
> > 2. Yes
> >
> {: .solution}
>
{: .question}


# Compute the embedding of a DNA sequence 

With this kind of model something, we can convert the DNA sequence to a vector.

Let's:
1. Take a DNA sequence
2. Tokenizes the DNA sequence using the tokenizer created before
3. Extracts the tensor containing the token IDs from the tokenized output
4. Passes the tokenized input through the model.
5. Extracts the hidden states from the model's output.

```python
dna = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC"
tokenized_dna = tokenizer(dna, return_tensors = 'pt')
inputs = tokenized_dna["input_ids"]
model_outputs = model(inputs)
hidden_states = model_outputs[0]
```

The generated hidden states are the internal representations of the input sequence at different layers of the model. Here we look at the hidden neurons of the last layer. They capture contextual information about the sequence and provide a richer representation of the sequence compared to the raw nucleotide string, capturing contextual information that can be used for tasks such as sequence similarity analysis, functional prediction, variant impact analysis, and more.

> <question-title></question-title> 
>
> What is the shape of `hidden_states`?
>
> > <solution-title></solution-title>
> > 
> > `[1, 17, 4096]`:
> > - `1`: number of sequences, here 1 DNA sequence
> > - `17`: number of words, here the DNA sequence has been converted to 17 words larger that 1
> > - `4096`: size of the vocabulary, the number of possible tokens
> >
> {: .solution}
>
{: .question}

We would like now to calculate the mean of the hidden states across a specific dimension, here the first layer of the model (`hidden_states[0]`):

```python
embedding_mean = torch.mean(hidden_states[0], dim=0)
```

`dim=0` indicates that the mean is calculated across the sequence length dimension. This effectively averages the hidden states for each token position in the sequence, resulting in a single vector that represents the entire sequence.

> <question-title></question-title> 
>
> 1. What is the shape of `embedding_mean`?
> 2. Which type of data is in `embedding_mean`?
>
> > <solution-title></solution-title>
> > 
> > 1. `4096`, the number of possible tokens.
> > 2. `embedding_mean` is a vector of numerical values.
> {: .solution}
>
{: .question}

`emb

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python", "deep-learning-without-gai-with-python"]


topic: statistics
tutorial: deep-learning-without-gai-with-python
title: Deep Learning (without Generative Artificial Intelligence) using Python
difficulty: intermediate
time_estimation: 3H
tools_json: []
tags_json: ["elixir", "ai-ml"]
content_hash: d41d8cd98f00b204e9800998ecf8427e
last_modified: 2026-05-27T09:01:49.822246
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/deep-learning-without-gai-with-python/tutorial.html

requirements_json:
["python-basics", "python-warmup-stat-ml", "intro-to-ml-with-python", "neural-networks-with-python"]


topic: statistics
tutorial: image_learner
title: GLEAM Image Learner - Validating Skin Lesion Classification on HAM10000
difficulty: intermediate
time_estimation: 1h
tools_json: ["Image Learner", "toolshed.g2.bx.psu.edu/repos/goeckslab/image_learner/image_learner/0.1.5"]
requirements_json: []
content_hash: c848deeb3f747cb018177aabb0436cc5
last_modified: 2026-05-27T09:01:49.831254
zenodo_link: https://zenodo.org/records/17114688
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/image_learner/tutorial.html

content:
In this tutorial, we will use the HAM10000 ("Human Against Machine with 10,000 training images") dataset to develop a deep learning classifier for dermoscopic skin lesion classification. The goal is to accurately classify seven types of pigmented skin lesions using the GLEAM Image Learner tool.

To achieve this, we will follow three essential steps: (i) upload the HAM10000 images and metadata to Galaxy, (ii) set up and run the Image Learner tool to train a deep learning model, and (iii) evaluate the model's predictive performance by analyzing key performance metrics such as accuracy, ROC-AUC, and confusion matrices.

![Workflow overview for HAM10000 classification in GLEAM Image Learner](../../images/skin_tutorial/image_learner_workflow_diagram.png "Workflow overview for HAM10000 classification in GLEAM Image Learner")

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
>
> The [HAM10000 dataset](https://zenodo.org/records/17114688) is a preprocessed subset of the original HAM10000 collection, following the methodology described by {% cite Shetty2022 %}. The dataset covers seven types of pigmented skin lesions:
> 1. Melanoma (mel)
> 2. Melanocytic nevus (nv)
> 3. Basal cell carcinoma (bcc)
> 4. Actinic keratosis (akiec)
> 5. Benign keratosis (bkl)
> 6. Dermatofibroma (df)
> 7. Vascular lesion (vasc)
>
> To address class imbalance in the original dataset, we applied preprocessing steps.
>
{:  .comment}

# Dataset Preprocessing and Composition

The dataset used in this tutorial has been preprocessed following the methodology from {% cite Shetty2022 %} to create a balanced training set suitable for deep learning.

## Preprocessing Steps

Starting from the original HAM10000 dataset (10,015 images with severe class imbalance), we applied the following preprocessing:

### Step 1: Image Selection
- Selected 100 images per class from the original dataset
- Ensured balanced representation across all 7 lesion types

### Step 2: Image Resizing
- Resized all images to 96×96 pixels
- Standardized format as PNG for consistent processing

### Step 3: Data Augmentation
- Applied horizontal flip augmentation to each image
- Generated 200 images per class (100 original + 100 flipped)
- **Total dataset: 1,400 images** (200 × 7 classes)

This preprocessing addresses the severe class imbalance in the original HAM10000 dataset where melanocytic nevi represented 67% of images while dermatofibroma represented only 1.1%.

## Balanced Dataset Composition

The preprocessed dataset provides balanced representation:

| Lesion Type | Images | Percentage |
|---|---|---|
| Melanocytic nevus (nv) | 200 | 14.3% |
| Melanoma (mel) | 200 | 14.3% |
| Basal cell carcinoma (bcc) | 200 | 14.3% |
| Actinic keratosis (akiec) | 200 | 14.3% |
| Benign keratosis (bkl) | 200 | 14.3% |
| Dermatofibroma (df) | 200 | 14.3% |
| Vascular lesion (vasc) | 200 | 14.3% |
| **Total** | **1,400** | **100%** |

This balanced dataset allows the Image Learner model to learn effectively from all lesion types without bias toward the majority class.

## Metadata Columns (HAM10000 CSV)

The metadata CSV now includes additional fields while keeping the same number of samples and the same flip augmentation strategy. Each row corresponds to one image file.

| Column | Description |
|---|---|
| `lesion_id` | Lesion identifier used to group original and augmented images from the same lesion. |
| `image_id` | Image identifier from the source dataset (shared by original and flipped versions). |
| `dx` | Diagnosis label (target class). |
| `dx_type` | Diagnosis confirmation method (for example, `histo`). |
| `age` | Patient age in years. |
| `sex` | Patient sex (`male`/`female`/`unknown`). |
| `localization` | Anatomical site of the lesion. |
| `image_path` | Image filename within the image ZIP. |

> <tip-title>Why Horizontal Flip Augmentation?</tip-title>
>
> Following the preprocessing pipeline described by {% cite Shetty2022 %}, **horizontal flip augmentation** is applied during dataset preparation. Horizontal flips:
>
> - Improve robustness to lesion orientation and acquisition variability
> - Increase effective training diversity without collecting additional images
> - Help reduce sensitivity to class- and pose-specific patterns
> - Preserve diagnostically relevant structures while introducing harmless variation
>
> In {% cite Shetty2022 %}, this preprocessing strategy (including horizontal flips) is associated with improved HAM10000 skin-lesion classification performance (reported accuracy: 95.18%).
>
> ![Example of horizontal flip augmentation applied to a skin lesion image.](../../images/skin_tutorial/horizontal_flip_augmentation.png "Example of horizontal flip augmentation. Adapted from {% cite Shetty2022 %}.")
>
{: .tip}

# Using Image Learner Tool

## Prepare environment and get the data

> <comment-title>Dataset Preprocessing</comment-title>
>
> The dataset available on Zenodo has been preprocessed following {% cite Shetty2022 %} methodology.
>
{:  .comment}

> <hands-on-title> Environment and Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial. If you are not inspired, you can name it *HAM10000 Image Classification*.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the dataset files from Zenodo
>
>    ```
>    https://zenodo.org/records/18394055/files/selected_HAM10000_img_metadata_aug.csv
>    https://zenodo.org/records/18394055/files/skin_image.zip
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    > <tip-title>Data Type</tip-title>
>    > - For the `.zip` file, set the datatype to `zip`
>    > - For the `.csv` file, leave as `Auto-Detect` (it will be recognized as tabular)
>    >
>    {: .tip}
>
> 3. Check that the data formats are assigned correctly:
>    - The `.zip` file should have type `zip`
>    - The `.csv` file should have type `tabular`
>
>    If they are not, follow the Changing the datatype tip:
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Add tags to the datasets for better organization:
>    - Add tag `HAM10000_images` to the skin_image.zip file
>    - Add tag `HAM10000_metadata` to the image_metadata_new.csv file
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Model Configuration in GLEAM Image Learner

## Tool setup and run

After uploading the dataset, configure the Image Learner parameters as follows. These settings are based on best practices for dermoscopic image classification and have been optimized for the HAM10000 dataset.

> <hands-on-title>Configure Image Learner for HAM10000</hands-on-title>
>
> 1. {% tool [Image Learner](toolshed.g2.bx.psu.edu/repos/goeckslab/image_learner/image_learner/0.1.5) %} with the following parameters:
>    - {% icon param-file %} *The metadata csv containing image_path column, label column*: `image_metadata_new.csv`
>    - {% icon param-file %} *Image zip*: `skin_image.zip`
>    - {% icon param-select %} *Task Type*: `Multi-class Classification`
>    - {% icon param-select %} *Overwrite label and/or image column names?*: `Yes`
>    - {% icon param-text %} *Target/label column name*: `c3: dx`
>    - {% icon param-text %} *Image column name*: `c8: image_path`
>    - {% icon param-select %} *Select a model for your experiment*: `CAFormer S18 384`
>    - {% icon param-select %} *Image Augmentation*: `Select all`
>    - {% icon param-select %} *Customize Default Settings*: `Yes`
>    - {% icon param-text %} *Epochs*: `30`
>    - {% icon param-text %} *Early Stop*: `30`
>
> 2. Run training and review the generated evaluation report.
>
{: .hands_on}

These settings are based on best practices for image classification and have been optimized for the HAM10000 dataset.

> <tip-title>Recommended Image Learner configuration (and why)</tip-title>
>
> | Parameter | Value | Rationale |
> |---|---|---|
> | Task Type | Classification | Multi-class image classification task |
> | Label column | `dx` | Target diagnosis label |
> | Image column | `image_path` | Image filename in the ZIP archive |
> | Model Name | caformer_s18_384 | Efficient transformer-based model ([CAFormer S18 384](https://github.com/sail-sg/metaformer/blob/main/metaformer_baselines.py)) |
> | Epochs | 30 | Sufficient for convergence without overfitting |
> | Early Stop | 30 | Stop when validation metrics stall to avoid overfitting |
> | Fine Tune | True | Leverage pre-trained features for better performance |
> | Use Pretrained | True | Transfer learning from ImageNet-trained weights |
> | Learning Rate | 0.001 | Conservative learning rate for fine-tuning |
> | Random Seed | 42 | Reproducible results across runs |
> | Data Split | 70/10/20 | Standard split for training/validation/test (automatically applied when no split column exists in metadata CSV) |
> | Data Augmentation | Horizontal and Vertical Flip; Rotate; Blur; Brightness; Contrast | Improve generalization |
{: .tip}

> <tip-title>Data Split Configuration</tip-title>
>
> The Image Learner tool automatically applies a **stratified 70/10/20 train/validation/test split** by default when no split column is present in the metadata CSV file. Per-class counts can differ by a few samples due to rounding. This ensures balanced representation of all classes across the three datasets. The stratified split maintains the same class distribution in each split, which is particularly important for imbalanced datasets. If you want to use a custom split, you can add a `split` column to your metadata CSV with values 0 (train), 1 (validation), or 2 (test).
>
{: .tip}

![GLEAM Image Learner tool interface.](../../images/skin_tutorial/tool_configuration_interface.png "GLEAM Image Learner tool interface")

## Tool Output Files

After training and testing your model, you should see several new files in your history list:

- **Image Learner Trained Model (`ludwig_model`)**: A reusable model bundle that includes the model configuration JSONs and model weights.

- **Image Learner Model Report (HTML)**: An interactive report that summarizes configuration, metrics, and plots.

- **Image Learner Predictions/Stats/Plots (collection)**: A list collection containing:
  - `predictions.csv` with model predictions and confidence scores
  - JSON files (for example `training_statistics.json`, `test_statistics.json`, `description.json`) with experiment metadata and metrics
  - PNG plots from `visualizations/train` and `visualizations/test`, plus feature importance example images
  - `feature_importance_examples.zip` bundling the feature importance examples

For this tutorial, we will focus on the Image Learner Model Report and the performance metrics.

# Image Learner Model Report

The Image Learner HTML report provides a comprehensive and interactive overview of the trained model's performance. It is organized into three tabs that separate configuration, training/validation diagnostics, and test results.

## Config and Overall Performance Summary

This tab combines dataset composition, overall metrics, and configuration details:

- **Dataset Overview**: Sample counts per class and split (train/validation/test). For regression tasks, only split counts are shown.
- **Model Performance Summary**: A sortable table of metrics across train, validation, and test splits.
- **Training Configuration**: Model architecture, image size, augmentation, split strategy, optimizer, learning rate, epochs, early stopping, and random seed.
- **Metrics Help**: A "Help" button that opens a glossary explaining each metric.

![Model and training summary interface in GLEAM Image Learner.](../../images/skin_tutorial/training_config.png "Model and training summary interface")

## Training and Validation Results

This tab focuses on optimization dynamics and validation diagnostics:

- **Train/Validation Performance Summary**: Side-by-side metrics for train vs. validation.
- **Learning Curves**: Loss/accuracy/F1/ROC-AUC (as applicable) and overfitting gap plots across epochs.
- **Validation Diagnostics**: Prediction confidence distributions and, for binary tasks, a threshold selection plot.

## Test Results

The test tab provides final evaluation plots and metrics:

- **Test Performance Summary**: Test-only metrics table.
- **Classification Diagnostics**: Confusion matrix, ROC/PR curves, and per-class metric plots.
- **Prediction Confidence**: Test-set confidence distributions.
- **Grad-CAM Heatmaps**: Visual explanations for convolutional backbones when available.

![Test Performance Summary - Accuracy and Loss Progression](../../images/skin_tutorial/test_metrics.png "Test Performance Summary")

These weighted metrics indicate balanced performance across classes under the explicitly balanced split. The report also includes ROC-AUC and Cohen's Kappa for additional discrimination and agreement context.

### Per-class Metrics

The report summarizes performance **for each lesion class** using a heatmap of key classification metrics. Rows correspond to classes (e.g., `akiec`, `bcc`, `bkl`, `df`, `mel`, `nv`, `vasc`) and columns correspond to evaluation metrics. Darker cells indicate stronger performance (values closer to 1.0).

- **Precision**: of the images predicted as a class, how many are correct (higher = fewer false positives).
- **Recall**: of the true images of a class, how many were found (higher = fewer false negatives).
- **F1 score**: balance of precision and recall.
- **Accuracy**: class-wise correctness under the one-vs-rest view reported by the tool.
- **Matthews correlation coefficient (MCC)**: correlation-style score robust to class imbalance (higher is better).
- **Specificity**: how well the model avoids labeling other classes as this class (higher = fewer false positives).

Use this view to quickly spot classes that are consistently strong across metrics (darker row) versus classes where performance lags in specific dimensions (lighter cells), guiding targeted follow-ups (e.g., more data, label review, or augmentation).

![Per-class metrics heatmap (precision, recall, F1, accuracy, MCC, specificity) by lesion class](../../images/skin_tutorial/per_class_metrics.png "Per-class metrics for each lesion class")

### Confusion Matrix

The confusion matrix provides a detailed breakdown of correct and incorrect predictions for each class, highlighting which lesion types are most frequently confused.

![Confusion matrix showing classification results for all lesion classes](../../images/skin_tutorial/confusion_matrix.png "Confusion matrix of model predictions")

> <tip-title>Interpreting the Confusion Matrix</tip-title>
>
> - **Diagonal elements**: Correct predictions (True Positives and True Negatives)
> - **Off-diagonal elements**: Misclassifications (False Positives and False Negatives)
> - **High values on diagonal**: Good overall classification performance
> - **Pattern analysis**: Can identify which classes are confused with each other
>
{: .tip}

# Comparison with Shetty et al. (2022)

To contextualize our results, we compare against the CNN results reported by Shetty et al. (2022) on HAM10000 {% cite Shetty2022 %}.

| Metric | Shetty et al., 2022 (CNN) | Image Learner (this tutorial) |
|---|---:|---:|
| Accuracy | 0.94 (94%) | 0.87 (87%) |
| Weighted Precision | 0.88 (88%) | 0.87 (87%) |
| Weighted Recall | 0.85 (85%) | 0.87 (87%) |
| Weighted F1-Score | 0.86 (86%) | 0.87 (87%) |

# Data Leakage-Aware Experiment (Sample ID)

We repeat the same experiment but add a feature selection to prevent leakage across splits:

- {% icon param-select %} *Sample ID column (optional)*: `c1: lesion_id`

![Data leakage-aware configuration](../../images/skin_tutorial/training_config_2.png "Data leakage-aware configuration")

## Data Leakage-Aware Results

When we keep original and flipped images from the same lesion in the same split, the metrics drop:

| Metric | Image Learner (leakage-aware split) |
|---|---:|
| Accuracy | 0.63 |
| Weighted Precision | 0.63 |
| Weighted Recall | 0.63 |
| Weighted F1-Score | 0.63 |

This reduction reflects a more realistic evaluation because the model no longer sees near-duplicate images across training and test splits.

![Data leakage-aware metric results](../../images/skin_tutorial/test_metrics_2.png "Data leakage-aware metric results")

# Key takeaways
- **Image Learner shows slightly lower accuracy** (0.88 vs. 0.94) but **higher weighted precision/recall/F1** (0.90 vs. 0.88/0.85/0.86).
- The balanced split comparison aligns with the published benchmark and shows strong weighted metrics under the same evaluation style.
- The leakage-aware split provides a more conservative, realistic estimate by keeping original and flipped images from the same lesion together.
- Image Learner provides publication-ready metrics and visualizations with full reproducibility through Galaxy.

# Tutorial takeaways
- The Image Learner comparison is competitive with the published CNN benchmark on the balanced HAM10000 subset.
- Leakage-aware splitting prevents inflated performance and is essential when augmentations create near-duplicate images.
- The tool makes it easy to enforce leakage-aware splits while keeping diagnostics transparent and reproducible.

# Conclusion

In this tutorial, we used the Galaxy Image Learner tool to build and evaluate a dermoscopic lesion classifier on the HAM10000 dataset with a balanced split and a CaFormer backbone:

- Upload the images and metadata.
- Configure and train the model.
- Review test metrics and diagnostic plots, and compare results to the Shetty et al. benchmark.

The model achieved ~88% accuracy with balanced weighted precision/recall/F1 of ~0.90 under the balanced split, and ~0.66 across metrics under the leakage-aware split. These steps generalize to other biomedical image-classification tasks while highlighting the importance of leakage-aware evaluation.

tags_json:
["HAM10000 Dataset", "Image Classification", "Deep Learning", "Image Learner", "Skin Lesion Classification"]


topic: statistics
tutorial: machinelearning
title: Basics of machine learning
difficulty: intermediate
time_estimation: 30M
requirements_json: []
tags_json: []
content_hash: 8328347bc9344ed6820b769399709cbf
last_modified: 2026-05-27T09:01:49.840555
zenodo_link: https://zenodo.org/record/1468039
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/statistics/tutorials/machinelearning/tutorial.html

content:
[Machine learning](https://en.wikipedia.org/wiki/Machine_learning) uses techniques from statistics, mathematics and computer science to make computer programs learn from data. It is one of the most popular fields of computer science and finds applications in multiple streams of data analysis such as [classification](https://en.wikipedia.org/wiki/Statistical_classification), [regression](https://en.wikipedia.org/wiki/Regression_analysis), [clustering](https://en.wikipedia.org/wiki/Cluster_analysis), [dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction), [density estimation](https://en.wikipedia.org/wiki/Density_estimation) and many more. Some real-life applications are spam filtering, medical diagnosis, autonomous driving, recommendation systems, facial recognition, stock prices prediction and many more. The following image shows a basic flow of any machine learning task. Data is provided by a user to a machine learning algorithm for analysis.

![data](images/ml_basics.png "Flow of a machine learning task.")

There are multiple ways in which machine learning can be used to perform data analysis. They depend on the nature of data and the kind of data analysis. The following image shows the most popular ones. In [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning) techniques, the categories of data records are known beforehand. But in [unsupervised learning](https://en.wikipedia.org/wiki/Unsupervised_learning), the categories of data records are not known.

![data](images/variants_ml.png "Different types of machine learning.")

In general, machine learning can be used in multiple real-life tasks by applying its variants as depicted in the following image.

![data](images/usage_ml.png "Real-life usage of machine learning.")

The following image shows how a classification task is performed. The complete data is divided into training and test sets. The training set is used by a classifier to learn features. It results in a trained model and its robustness (of learning) is evaluated using the test set (unseen by the classifier during the training).

![data](images/prediction.png "Supervised learning.")

This tutorial shows how to use a machine learning module implemented as a Galaxy tool. The data used in this tutorial is available at [Zenodo](https://zenodo.org/record/1468039).

> <agenda-title></agenda-title>
>
> Performing a machine learning task (classification) using a tool involves the following steps:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data upload

The datasets required for this tutorial contain 9 features of breast cells which include the thickness of clump, cell-size, cell-shape and so on ([more information](https://github.com/EpistasisLab/pmlb/tree/master/datasets/breast_w)). In addition to these features, the training dataset contains one more column as `target`. It has a binary value (0 or 1) for each row. `0` indicates no breast cancer and `1` indicates breast cancer. The test dataset does not contain the `target` column.


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following datasets and choose the type of data as `tabular`.
>
>    ```
>    https://zenodo.org/record/1401230/files/breast-w_train.tsv
>    https://zenodo.org/record/1401230/files/breast-w_test.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename datasets to `breast-w_train` and `breast-w_test`.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. The datasets should look like these:
>
>
>    ![data](images/train_data.png "Training data (breast-w_train) with targets (9 features and one target).")
>
>
>    ![data](images/test_data.png "Test data (breast-w_test) (9 features and no target).")
{: .hands_on}


# Train a classifier
In this step, we will use the [SVM (support vector machine)](https://scikit-learn.org/stable/modules/svm.html#svm-classification) classifier for training on the `breast-w_train` dataset. The classifier learns a mapping between each row and its category. SVM is a memory efficient classifier which needs only those data points which lie on the decision boundaries among different classes to predict a class for a new sample. The rest of the data points can be thrown away. We will use the `LinearSVC` variant of SVM which is faster. Other variants `SVC` and `NuSVC` have high running time for large datasets. The last column of the training dataset contains a category/class for each row. The classifier learns a mapping between data row and its category which is called a trained model. The trained model is used to predict the categories of the unseen data.

> <hands-on-title>Train a classifier</hands-on-title>
>
> {% tool [Support vector machines (SVMs) for classification](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_svm_classifier/sklearn_svm_classifier/1.0.11.2) %} with the following parameters to train:
>    - *"Select a Classification Task"*: `Train a model`
>        - *"Classifier type"*: `Linear Support Vector Classification`
>        - *"Select input type"*: `tabular data`
>        - {% icon param-file %} *"Training samples dataset"*: `breast-w_train` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `All columns EXCLUDING some by column header name(s)`
>        - *"Type header name(s)"*: `target`
>        - {% icon param-file %} *"Dataset containing class labels or target values"*: `breast-w_train` tabular file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Choose how to select data by column"*: `Select columns by column header name(s)`
>        - *"Type header name(s):"*: `target`
>
{: .hands_on}


# Predict using a trained model
The previous step produced a trained model (a zip archive) which we will now use to predict classes for the test data (`breast-w_test`).

> <hands-on-title>Predict using a trained model</hands-on-title>
>
> {% tool [Support vector machines (SVMs) for classification](toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_svm_classifier/sklearn_svm_classifier/1.0.11.2) %} with the following parameters:
>    - *"Select a Classification Task"*: `Load a model and predict`
>        - {% icon param-file %} *"Models"*: the `h5mlm` dataset produced by **Support vector machines (SVMs) for classification** {% icon tool %}
>        - {% icon param-file %} *"Data (tabular)"*: `breast-w_test` file
>        - *"Does the dataset contain header"*: `Yes`
>        - *"Select the type of prediction"*: `Predict class labels`
>
{: .hands_on}


# See predictions
The last column of the predicted dataset shows the category of each row. A row either got `0` (no breast cancer) or `1` (breast cancer) as its predicted category.

> <hands-on-title>See the predicted column</hands-on-title>
> 1. Click on the {% icon galaxy-eye %} (eye) icon of the dataset created by the previous step.
> 2. The last column of the table shows the predicted category (`target`) for each row.
>
{: .hands_on}


> <details-title>Additional ML resources</details-title>
>
> Read more about [machine learning using scikit-learn](http://scikit-learn.org/stable/).
{:.details}

tools_json:
["Support vector machines (SVMs) for classification", "toolshed.g2.bx.psu.edu/repos/bgruening/sklearn_svm_classifier/sklearn_svm_classifier/1.0.11.2"]


