# tutorials

topic: proteomics
tutorial: protein-id-oms
title: Peptide and Protein ID using OpenMS tools
difficulty: advanced
time_estimation: 45m
requirements_json: []
tags_json: ["[DDA, HeLa]"]
content_hash: b9d3e73e13eb01eb7fb8e7537333c650
last_modified: 2026-05-27T09:01:49.461781
zenodo_link: https://zenodo.org/record/546301
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/protein-id-oms/tutorial.html

content:
Identifying the proteins contained in a sample is an important step in any proteomic experiment. However, in most settings, proteins are digested to peptides prior to LC-MS/MS analysis. In this so-called "bottom-up" procedure, only peptide masses are measured. Therefore, protein identification cannot be performed directly from raw data, but is a multi-step process:

1. Raw data preparations
2. Peptide-to-Spectrum matching
3. Peptide inference
4. Protein inference

A plethora of different software solutions exists for each step. In this tutorial, we will use ***msconvert*** {% icon tool %}  for raw data conversion and tools from the [OpenMS software suite](https://openms.de) for all other steps. We will use one peptide search engine at first and later on show how to expand the workflow for using multiple search engines. Protein inference will be performed with the Fido algorithm ([Serang et al, JPR, (2010)](https://www.ncbi.nlm.nih.gov/pubmed/20712337)).
This tutorial covers peptide and protein **identification** only, but you may use the output of this tutorial for the [tutorial on protein quantitation]({{site.baseurl}}/topics/proteomics/tutorials/protein-quant-sil/tutorial.html).

For an alternative protein ID workflow using the [Compomics](https://compomics.com/) tools [SearchGUI](https://compomics.github.io/projects/searchgui.html) and [PeptideShaker](https://compomics.github.io/projects/peptide-shaker.html), please consult [this tutorial]({{site.baseurl}}/topics/proteomics/tutorials/protein-id-sg-ps/tutorial.html).
The latter tutorial does not allow to continue with the tutorial on protein quantitation.

# Input data


As an example dataset, we will use an LC-MS/MS analysis of HeLa cell lysate published
in [Vaudel et al., 2014, Proteomics](https://www.ncbi.nlm.nih.gov/pubmed/24678044). Detailed information
about the dataset can be found on [PRIDE](https://www.ebi.ac.uk/pride/archive/projects/PXD000674).
For step 2 we will use a validated human Uniprot FASTA database with appended decoys.

If you already completed the tutorial on [Database Handling]({{site.baseurl}}/topics/proteomics/tutorials/database-handling/tutorial.html)
you can use the constructed database including decoys.
You can find a prepared database, as well as the input LC-MS/MS data in different file formats on [Zenodo](https://zenodo.org/record/796184).

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preparing Raw Data

Raw data conversion is the first step of any proteomic data analysis. The most common converter is MSConvert from the [ProteoWizard software suite](http://proteowizard.sourceforge.net/), the format to convert to is mzML.
Due to licensing reasons, MSConvert runs only on windows systems and will not work on most Galaxy servers.

Depending on your machine settings, raw data will be generated either in profile mode or centroid mode. For most peptide search engines, the MS2 data have to be converted to centroid mode, a process called "peak picking" or "centroiding".
Machine vendors offer algorithms to extract peaks from profile raw data. Those are integrated in ***msconvert*** {% icon tool %} and can be run in parallel to the mzML conversion.
However, the OpenMS tool ***PeakPickerHiRes*** {% icon tool %} is reported to generate slightly better results ([Lange et al., 2006, Pac Symp Biocomput](https://www.ncbi.nlm.nih.gov/pubmed/17094243)) and is therefore recommended for quantitative studies ([Vaudel et al., 2010, Proteomics](https://www.ncbi.nlm.nih.gov/pubmed/19953549)).
If your data were generated on a low resolution mass spectrometer, use **PeakPickerWavelet** {% icon tool %} instead.

> <hands-on-title>File Conversion and Peak Picking</hands-on-title>
>
> We provide the [input data](https://zenodo.org/record/796184) in the original `raw` format and also already converted to `mzML`. If **msconvert** {% icon tool %} does not run on your Galaxy instance, please download the preconverted `mzML` as an input and continue with step 5 of the following hands-on training.
>
> 1. Create a new history for this Peptide and Protein ID exercise.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Load one of the example datasets into your history from Zenodo
>
>    ```
>    https://zenodo.org/record/892005/files/qExactive01819.raw
>    https://zenodo.org/record/892005/files/qExactive01819_profile.mzml
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the dataset to something meaningful
>
> 4. Run {% tool [msconvert](toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.19052.1) %} on the training `raw` file to convert to the `mzML` format
>    - {% icon param-file %} *"Input unrefined MS data"*: imported `raw` file
>    - *"Do you agree to the vendor licenses"*: set to `Yes`
>    - *"Output Type"*: set to `mzML`
>
> 5. Run {% tool [PeakPickerHiRes](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_peakpickerhires/PeakPickerHiRes/2.3.0) %} on the resulting file
>    - {% icon param-file %} *"input profile data file": output of **msconvert** or `mzML` file
>    - In *"param_algorithm_ms_levels"*
>      - {% icon param-repeat %} Click on *"Insert param_algorithm_ms_levels"*
>      - In *"1: param_algorithm_ms_levels"*
>         - *"List of MS levels for which the peak picking is applied"*: `2`
>
>           Peak picking will only be performed on MS2 level.
>
>   > <comment-title>Local Use of MSConvert</comment-title>
>   > The vendor libraries used by MSConvert are only licensed for Windows systems and are therefore rarely implemented in Galaxy instances. If **msconvert** {% icon tool %} is not available in your Galaxy instance, please install the software on a Windows computer and run the conversion locally. You can find a [detailed description of the necessary steps on this site](https://compomics.com/bioinformatics-for-proteomics/identification/) ("Peak List Generation"). Afterwards, upload the resulting mzML file to your Galaxy history.
>   {: .comment}
>
{: .hands_on}

> <comment-title>MS2 peak picking during data acquisition</comment-title>
> MS2 peaks are often acquired in centroided mode in first place. The profile data are converted to centroided mode already during data acquisition, resulting in MS2-centroided `raw` files. If your MS2 data are already centroided, simply omit the peak picking step.
{: .comment}

# Peptide Identification

MS/MS experiments identify peptides by isolating them and subsequently colliding them with a gas for fragmentation. This method generates a spectrum of peptide fragment masses for each isolated peptide - an MS2 spectrum.
To find out the peptide sequences, the MS2 spectrum is compared to a theoretical spectrum generated from a protein database. This step is called peptide-to-spectrum (also: spectrum-to-sequence) matching. Accordingly, a peptide that is successfully matched to a sequence is termed PSM (Peptide-Spectrum-Match). There can be multiple PSMs per peptide, if the peptide was fragmented several times.

Different peptide search engines have been developed to fulfill the matching procedure. Here, we will use the search engine [X!Tandem](https://www.ncbi.nlm.nih.gov/pubmed/14976030). OpenMS provides "adapters" (wrappers) for several other peptide search engines, like MSGF+ or OMSSA. You may replace the XTandemAdapter by another search engine of your choice.

> <hands-on-title>Peptide Identification</hands-on-title>
>
> 1. Copy the prepared protein database (Human database including cRAP contaminants and decoys) from the tutorial [Database Handling]({% link topics/proteomics/tutorials/database-handling/tutorial.md %}) into your current history by using the multiple history view
>
>   > <comment-title>You did not run the Database Handling first?</comment-title>
>   > You can upload the ready-made database from Zenodo
>   >
>   > ```
>   > https://zenodo.org/record/892005/files/Human_database_including_decoys_%28cRAP_added%29.fasta
>   > ```
>   {: .comment}
>
> 2. Run {% tool [XTandemAdapter](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_xtandemadapter/XTandemAdapter/2.6+galaxy0) %} with:
>   - {% icon param-file %} *"Input file containing MS2 spectra"*: MS2-centroided mzML file
>   - {% icon param-file %} *"FASTA file"*: FASTA protein database
>   - *"Fragment mass error"*: `10`
>   - *"Fragment monoisotopic mass error units"*: `ppm`
>   - *"Optional Outputs"* select `out (Output file containing search results)`
>
> 3. Run {% tool [FileInfo](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_fileinfo/FileInfo/2.6+galaxy0) %}
>    - {% icon param-file %} *"input file"*: **XTandem** output
{: .hands_on}

> <comment-title>Settings for labelled data</comment-title>
> Several common quantitation methods are based on labels e.g. SILAC, TMT, iTRAQ, Dimethyl. Those labels must be specified in the search engine as a (variable) modification. See also [Peptide and Protein Quantification via Stable Isotope Labelling]({% link topics/proteomics/tutorials/protein-quant-sil/tutorial.md %}).
{: .comment}

> <comment-title>Advanced Search Engine Parameters</comment-title>
> The OpenMS adapters do not always allow to set every option of the underlying search engine. If an option is missing, you may also run the search engine locally or by using a Galaxy wrapper. Afterwards, convert the search engine output to the OpenMS format `idXML` by running **IDFileConverter** {% icon tool %}.
>
> The search engine X!Tandem features some more advanced options than the ones reflected in the **XTandemAdapter** {% icon tool %}. If you need those advanced options, the **XTandemAdapter** {% icon tool %} allows for the optional input of a classic X!Tandem parameter file. Upload your parameter file to the history and use it as an input in the field `Default X!Tandem configuration file`. You may also set the option *"ignore_adapter_param"* to `Yes` to overwrite all options set by the GUI.
{: .comment}

# Peptide FDR filtering

The next step of peptide identification is to decide which PSMs will be used for protein inference. Measured MS2 spectra never perfectly fit the theoretical spectra. Therefore, peptide search engines calculate a score which indicates how well the measured MS2 spectrum was fitting the theoretical spectrum. How do we decide which PSMs are likely true and which are false?

In proteomics, this decision is typically done by calculating false discovery rates (FDRs). Remember that the database we were using for peptide-to-spectrum matching consisted not only of true proteins, but also the same number of "fake entries", the so-called decoys. Those decoys can now be used to estimate the number of false identifications in the list of PSMs.
The calculation is based on a simple assumption: for every decoy peptide identified with a given score, we expect one false positive with at least the same score.
The false discovery rate is therefore defined as the number of false discoveries (decoy hits) divided by the number of false and correct discoveries (both target and decoy hits) at a given score threshold.

To calculate FDRs, we first have to annotate the identified peptides to determine which of them are decoys. This is done with the tool **PeptideIndexer** {% icon tool %}. This tool allows the annotation of protein names and calculates the protein coverage based on the identified peptides. Then we calculate peptide posterior error probabilities (PEPs), because they are needed for the protein inference algorithm Fido, which is used by OpenMS. We will then filter PSMs for 1 % FDR with **FalseDiscoveryRate** {% icon tool %}. and set the score back to PEP with **IDScoreSwitcher** {% icon tool %}.

> <hands-on-title>Peptide FDR filtering</hands-on-title>
>
> 1. Run {% tool [PeptideIndexer](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_peptideindexer/PeptideIndexer/2.6+galaxy0) %} with
>    - {% icon param-file %} *"Input idXML file containing the identifications"*: output of **XTandemAdapter**
>    - {% icon param-file %} *"Input sequence database in FASTA format"*: FASTA protein database
>    - *"If set, the protein sequences are stored as well"*: `Yes`
>    - *"If set, the protein description is stored as well"*: `Yes`
>    - In the *"enzyme"* section set *"Specificity of the enzyme"*: `none`
>
> 2. Run {% tool [IDPosteriorErrorProbability](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_idposteriorerrorprobability/IDPosteriorErrorProbability/2.6+galaxy) %} with
>    - {% icon param-file %} *"input file"*: **PeptideIndexer** output
>    - *"If set scores will be calculated as '1 - ErrorProbabilities' and can be interpreted as probabilities for correct identifications"*: `Yes`
>
> 3. Run {% tool [FalseDiscoveryRate](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_falsediscoveryrate/FalseDiscoveryRate/2.6+galaxy0) %} with
>    - {% icon param-file %} *"Identifications from searching a target-decoy database"*: output of **IDPosteriorErrorProbability**
>    - *"Perform FDR calculation on protein level"*: `false`
>    - In the *"FDR control"* section select *"Filter PSMs based on q-value"*: `0.01`
>    - In the "Parameter section for the FDR calculation algorithm"* section select *"If 'true' decoy peptides will be written to output file, too"*: `Yes`
>
> 4. Run {% tool [IDScoreSwitcher](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_idscoreswitcher/IDScoreSwitcher/2.6+galaxy0) %} with
>    - {% icon param-file %} *"Input file"*: output of **FalseDiscoveryRate**
>    - *"Name of the meta value to use as the new score"*: `Posterior Probability_score`
>    - *"Orientation of the new score"*: `higher_better`
>
> 5. Run {% tool [FileInfo](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_fileinfo/FileInfo/2.6+galaxy0) %} to get basic information about the identified peptides
>    - {% icon param-file %} *"input file"*: **IDScoreSwitcher** output
>
{: .hands_on}

> <question-title></question-title>
> 1. How many peptides were identified?
> 2. How many peptides with oxidized methionine were identified?
>
> > <solution-title></solution-title>
> > 1. You should have identified 3378 non-redundant peptide hits.
> > 2. 804 peptides contain an oxidized methionine.
> > Numbers may slightly vary depending on the versions of the tools and the used FASTA file.
> {: .solution }
{: .question}


# Protein Inference
In bottom-up proteomics, it is necessary to combine the identified peptides to proteins. This is not a trivial task, as proteins are redundant to some degree. Thus, not every peptide can be assigned to only one protein.
The OpenMS suite implemented the [Fido](https://www.ncbi.nlm.nih.gov/pubmed/20712337) algorithm for protein inference. Fido uses a Bayesian probabilistic model to group and score proteins based on peptide-spectrum matches. Afterwards, we keep only proteins with 1% FDR.

> <hands-on-title>Protein inference</hands-on-title>
>
> 1. Run {% tool [FidoAdapter](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_fidoadapter/FidoAdapter/2.6+galaxy0) %}
>    - {% icon param-file %} *"Input: identification results"*: output of **IDScoreSwitcher**
>    - *"Post-process Fido output with greedy resolution of shared peptides based on the protein probabilities"*: `Yes`
>
> 2. Run {% tool [FalseDiscoveryRate](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_falsediscoveryrate/FalseDiscoveryRate/2.6+galaxy0) %}
>    - {% icon param-file %} *"Identifications from searching a target-decoy database"*: output of **FidoAdapter**
>    - *"Perform FDR calculation on PSM level"*: `false`
>    - In the *"FDR control"* section select *"Filter proteins based on q-value"*: `0.01`
>
> 3. Run {% tool [FileInfo](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_fileinfo/FileInfo/2.6+galaxy0) %} to get basic information about the identified proteins
>    - {% icon param-file %} *"input file"*: output of **FalseDiscoveryRate**
{: .hands_on}

> <comment-title>"Greedy" Group Resolution</comment-title>
> Protein groups are reported, when an identified peptide maps to multiple proteins in the used database [Nesvizhskii and Aebersold (2005)](https://www.ncbi.nlm.nih.gov/pubmed/16009968). Some peptides may map to different protein groups and can therefore not be used for protein quantitation. The option `-greedy_group_resolution` solves this problem by assigning peptides only to the one most probable protein group, thus enabling to quantify proteins based not only on unique, but also on shared peptides. This usually leads to a much higher number of quantified proteins. However it will introduce noise in the FCs when a peptide was indeed shared by different proteins and the quantity of this peptide was a weighted sum of contributions. The greedy group resolution is similar to Occam's razor.
{: .comment}

> <question-title></question-title>
> How many proteins were finally identified?
>
> > <solution-title></solution-title>
> > You should have identified 1252 proteins.
> > Numbers may slightly vary depending on the versions of the tools and the used FASTA file.
> {: .solution }
{: .question}

# Analysis of Contaminants

The FASTA database used for the peptide to spectrum matching contained some entries that were not expected to stem from the HeLa cell lysate, but are common contaminations in LC-MS/MS samples. The main reason to add those is to avoid false assignment of contaminant spectra to other proteins.
It also enables you to check for contaminations in your samples.

**CAVE:** When analyzing human samples, many proteins that are common contaminants may also stem from the sample. Therefore, human contaminants do not have to be excluded from further analysis, but you should keep in mind that the source of these proteins is unclear.

> <hands-on-title>Analysis of Contaminants</hands-on-title>
>
> 1. Run {% tool [TextExporter](toolshed.g2.bx.psu.edu/repos/galaxyp/openms_textexporter/TextExporter/2.6+galaxy0) %} to convert the idXML output to a human-readable tabular file.
>    - {% icon param-file %} *"Input file"*: **FalseDiscoveryRate** output
>
> 2. Run {% tool [Select lines that match an expression](Grep1) %}
>    - {% icon param-file %} *"Select lines from"*: **TextExporter**
>    - *"that"*: `Matching`
>    - *"the pattern"*: `CONTAMINANT`
>
> 2. Run {% tool [Select lines that match an expression](Grep1) %} to remove all non human proteins (e.g. bovine)
>    - {% icon param-file %} *"Select lines from"*: **TextExporter**
>    - *"that"*: `Matching`
>    - *"the pattern"*: `HUMAN`
>
{: .hands_on}

> <question-title></question-title>
> 1. Which contaminants did you identify? Where do these contaminations likely come from?
> 2. What other sources of contaminants exist?
> 3. How many false positives do we expect in our list?
>
> > <solution-title></solution-title>
> > 1.  TRY1_BOVIN is bovine trypsin. It was used to degrade the proteins to peptides. ALBU_BOVIN is bovine serum albumin. It is added to cell culture medium in high amounts. Also, eight human proteins are listed, these are commonly introduced during sample preparation. As we were analyzing a human sample, it is not neccessary to remove these proteins, as they may as well originate from the HeLa cells.
> > 2.  Contaminants often stem from the experimenter, these are typically keratins or other high-abundant human proteins. Basically any protein present in the room of the mass spectrometer might get into the ion source, if it is airborne. As an example, sheep keratins are sometimes found in proteomic samples, stemming from clothing made of sheep wool.
> > 3.  As we were allowing for a false discovery rate of 1 %, we would expect 13 (1255/0.01) false positive proteins in our list.
> {: .solution }
{: .question}


# Using multiple search engines

It is generally recommended to use more than one peptide search engine and use the combined results for peptide inference ([Shteynberg et al., 2013, Mol. Cell. Proteomics](https://www.ncbi.nlm.nih.gov/pubmed/23720762)).
By comparing results of multiple search engines, you may improve the *sensitivity* (when accepting peptides that were found by only one of the engines), the *specificity* (when accepting only peptides that were found by all of the search engines) or *both* (when using n>2 search engines and accept peptides found by a fraction of the (e.g. n-1) search engines).

Here, we will use the OpenMS tool [ConsensusID](https://abibuilder.informatik.uni-tuebingen.de/archive/openms/Documentation/release/latest/html/TOPP_ConsensusID.html) to combine the search engine results.

> <hands-on-title>Multiple search engines</hands-on-title>
>
> 1. Run **MSGFPlusAdapter** {% icon tool %} with
>    - {% icon param-file %} *"Input file"*: the MS2-centroided mzML
>    - {% icon param-file %} *"Protein sequence database*": the FASTA protein database
>    - *"Precursor monoisotopic mass tolerance"*: `10.0`
>    - *"Instrument that generated the data"*: `Q_Exactive`
>    - In *"param_fixed_modifications"*
>      - {% icon param-repeat %} Click on *"Insert param_fixed_modifications"*
>      - In *"1: param_fixed_modifications"*
>         - *"Fixed modifications"*: `Carbamidomethyl (C)`
>    - In *"param_variable_modifications"*
>      - {% icon param-repeat %} Click on *"Insert param_variable_modifications"*
>      - In *"1: param_variable_modifications"*
>         - *"Variable modifications"*: `Oxidation (M)`
>
> 2. Run **FileInfo** {% icon tool %} to get basic information about the identified proteins
>    - {% icon param-file %} *"input file"*: **MSGFPlusAdapter** output
>
> 3. Run **IDPosteriorErrorProbability** {% icon tool %} with
>    - {% icon param-file %} *"input file"*: **MSGFPlusAdapter** output
>    - *"If set scores will be calculated as '1 - ErrorProbabilities' and can be interpreted as probabilities for correct identifications"*: `No`
>
> 4. Run **IDMerger** {% icon tool %}
>    - *"Reduce collectons"*: `reduce collections by aggregating single files of multiple collections`
>      - In *"inputs"*
>        - In *"1: Repeat"*
>          - {% icon param-file %} *"Input files separated by blanks"*: output of **IDScoreSwitcher** based on **XTandemAdapter**
>        - In *"2: Repeat"*
>          - {% icon param-file %} *"Input files separated by blanks"*: output of **IDScoreSwitcher** based on **MSGFPlusAdapter**
>
> 5. Run **ConsensusID** {% icon tool %}
>    - {% icon param-file %} *"input file"*: **IDMerger** output
>
> 6. Run **PeptideIndexer** {% icon tool %} with
>    - {% icon param-file %} *"Input idXML file containing the identifications"*: **ConsensusID** output
>    - {% icon param-file %} *"Input sequence database in FASTA format"*: FASTA protein database
>    - *"Specificity of the enzyme"*: `none`
>
> 7. Run **FalseDiscoveryRate** {% icon tool %} with
>    - {% icon param-file %} *"Identifications from searching a target-decoy database"*: output of **PeptideIndexer**
>    - *"Perform FDR calculation on protein level"*: `false`
>    - *"Filter PSMs based on q-value"*: `0.01`
>    - *"If 'true' decoy peptides will be written to output file, too"*: `Yes`
>
> 8. Run **IDScoreSwitcher** {% icon tool %} with
>    - {% icon param-file %} *"Input file"*: **FalseDiscoveryRate** output
>    - *"Name of the meta value to use as the new score"*: `Posterior Error Probability_score`
>    - *"Orientation of the new score"*: `lower_better`
>    - *"Name to use as the type of the new score"*: `Posterior Error Probability`
>
> 9. Run **FileInfo** {% icon tool %} to get basic information about the identified peptides
>    - {% icon param-file %} *"input file"*: **IDScoreSwitcher** output
>
> 10. Proceed with the protein inference as described [above](#protein-inference)
>
{: .hands_on}

> <question-title></question-title>
> 1. How many PSMs could be matched with XTandem and MSGFPlus alone? How many peptides were identified?
> 2. How many PSMs could be matched after combining the results with ConsensusID? How many peptides were identified?
>
> > <solution-title></solution-title>
> > 1.  After FDR-filtering, XTandem matched 3,552 PSMs (2,616 unique peptides) and MSGFPlus matched 4,292 PSMs (2,991 peptides).
> > 2.  Combining the results with ConsensusID leads to matching of 4,299 PSMs (3,041 unique peptides).
> {: .solution }
{: .question}


# Premade Workflow

- [A premade workflow for this tutorial]({% link topics/proteomics/tutorials/protein-id-oms/workflows/workflow.ga %}).
- [A premade workflow using the search engines XTandem and MSGF+]({% link topics/proteomics/tutorials/protein-id-oms/workflows/workflow_two-search-engines.ga %}).

# Further Reading

- [Protein inference](https://www.ncbi.nlm.nih.gov/pubmed/16009968)
- [Fido publication](https://www.ncbi.nlm.nih.gov/pubmed/20712337)
- [Evaluation of protein inference algorithms](https://www.ncbi.nlm.nih.gov/pubmed/27498275)

tools_json:
["msconvert", "toolshed.g2.bx.psu.edu/repos/galaxyp/msconvert/msconvert/3.0.19052.1", "PeakPickerHiRes", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_peakpickerhires/PeakPickerHiRes/2.3.0", "XTandemAdapter", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_xtandemadapter/XTandemAdapter/2.6+galaxy0", "FileInfo", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_fileinfo/FileInfo/2.6+galaxy0", "PeptideIndexer", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_peptideindexer/PeptideIndexer/2.6+galaxy0", "IDPosteriorErrorProbability", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_idposteriorerrorprobability/IDPosteriorErrorProbability/2.6+galaxy", "FalseDiscoveryRate", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_falsediscoveryrate/FalseDiscoveryRate/2.6+galaxy0", "IDScoreSwitcher", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_idscoreswitcher/IDScoreSwitcher/2.6+galaxy0", "FidoAdapter", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_fidoadapter/FidoAdapter/2.6+galaxy0", "TextExporter", "toolshed.g2.bx.psu.edu/repos/galaxyp/openms_textexporter/TextExporter/2.6+galaxy0", "Select lines that match an expression", "Grep1"]


topic: proteomics
tutorial: metaproteomics
title: Metaproteomics tutorial
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: ["[microgalaxy]"]
content_hash: 4bbdb822fc0c751e55fa922bbf881c98
last_modified: 2026-05-27T09:01:49.456428
zenodo_link: https://doi.org/10.5281/zenodo.839701
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/metaproteomics/tutorial.html

content:
In this metaproteomics tutorial we will identify expressed proteins from a complex bacterial community sample.
For this MS/MS data will be matched to peptide sequences provided through a FASTA file.

Metaproteomics is the large-scale characterization of the entire protein complement of environmental microbiota
at a given point in time. It has the potential to unravel the mechanistic details of microbial interactions with
the host / environment by analyzing the functional dynamics of the microbiome.

In this tutorial, we will analyze a sample of sea water that was collected in August of 2013 from the Bering
Strait chlorophyll maximum layer (7m depth, 65° 43.44″ N, 168° 57.42″ W). The data were originally published in {% cite May_2016 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pretreatments

## Data upload

There are three ways to upload your data.

*   Upload/Import the files from your computer
*   Using a direct link
*   Import from the data library if your instance provides the files

In this tutorial, we will get the data from Zenodo: [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.839701.svg)](https://doi.org/10.5281/zenodo.839701).

> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and name it something meaningful (e.g. *Metaproteomics tutorial*)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the three MGF MS/MS files and the FASTA sequence file from Zenodo.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>    ```
>    https://zenodo.org/record/839701/files/2016_Jan_12_QE2_45.mgf
>    https://zenodo.org/record/839701/files/2016_Jan_12_QE2_46.mgf
>    https://zenodo.org/record/839701/files/2016_Jan_12_QE2_47.mgf
>    https://zenodo.org/record/839701/files/FASTA_Bering_Strait_Trimmed_metapeptides_cRAP.fasta
>    https://zenodo.org/record/839701/files/Gene_Ontology_Terms.tabular
>    ```
>
>    As default, Galaxy takes the link as name.
>
>    > <comment-title></comment-title>
>    > - Rename the datasets to a more descriptive name
>    {: .comment}
>
> 3. Build a **Dataset list** for the three MGF files
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
{: .hands_on}

We have a choice to run all these steps using a single workflow, then discuss each step and the results in more detail.

> <hands-on-title>Pretreatments</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/metaproteomics/workflows/workflow.ga" title="Pretreatments" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"1: SixGill generated protein fasta file"*: `FASTA_Bering_Strait_Trimmed_metapeptides_cRAP.fasta`
>    - {% icon param-file %} *"2: Dataset collection of Bering Strait MGF files"*: `Dataset collection of bering MGF`
>    - {% icon param-file %} *"3: GeneOntology terms (selected)"*: `Gene_Ontology_terms.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

# Analysis

## Match peptide sequences

The search database labelled `FASTA_Bering_Strait_Trimmed_metapeptides_cRAP.FASTA` is the input database that
will be used to match MS/MS to peptide sequences via a sequence database search. It is a small excerpt of the original database, which was constructed based on a metagenomic screening of the sea water samples (see {% cite May_2016 %}). The full original database [is available online](https://noble.gs.washington.edu/proj/metapeptide/data/metapeptides_BSt.fasta). The contaminant database (cRAP) was merged with the original database.

For this, the sequence database-searching program called [SearchGUI](https://compomics.github.io/projects/searchgui.html) will be used.
The created dataset collection of the three *MGF files* in the history is used as the MS/MS input.

### SearchGUI

> <hands-on-title>SearchGUI</hands-on-title>
>
> 1. {% tool [Search GUI](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/search_gui/3.3.10.1) %} with the following parameters:
>    - **Protein Database**: `FASTA_Bering_Strait_Trimmed_metapeptides_cRAP.FASTA`(or however you named the `FASTA` file)
>    - **Input Peak lists (mgf)**: `MGF files` dataset collection.
>
>    > <tip-title>Select dataset collections as input</tip-title>
>    >
>    > * Click the **Dataset collection** icon on the left of the input field:
>    >
>    >      ![Dataset collection button](../../images/dataset_button.png)
>    > * Select the appropriate dataset collection from the list
>    {: .tip}
>
>    Section **Search Engine Options**:
>
>    - **Search Engines**: `X!Tandem`
>
>    > <comment-title></comment-title>
>    >
>    > The section **Search Engine Options** contains a selection of sequence database searching
>    > algorithms that are available in SearchGUI. Any combination of these programs can be used for
>    > generating PSMs from MS/MS data. For the purpose of this tutorial, **X!Tandem** we will be used.
>    {: .comment}
>
>    Section **Precursor Options**:
>
>    - **Fragment Tolerance Units**: `Daltons`
>    - **Fragment Tolerance**: `0.2`- this is high resolution MS/MS data
>    - **Maximum Charge**: `6`
>
>    Section **Protein Modification Options**:
>
>    - **Fixed Modifications**: `Carbamidomethylation of C`
>    - **Variable modifications**: `Oxidation of M`
>
>    > <tip-title>Search for options</tip-title>
>    >
>    > * For selection lists, typing the first few letters in the window will filter the available options.
>    {: .tip}
>
>    Section **Advanced Options**:
>    - **X!Tandem Options**: `Advanced`
>    - **X!Tandem: Quick Acetyl**: `No`
>    - **X!Tandem: Quick Pyrolidone**: `No`
>    - **X!Tandem: Protein stP Bias**: `No`
>    - **X!Tandem: Maximum Valid Expectation Value**: `100`
>
>    - leave everything else as default
>
> 2. Click **Run Tool**.
>
{: .hands_on}

Once the database search is completed, the SearchGUI tool will output a file (called a
SearchGUI archive file) that will serve as an input for the next section, PeptideShaker.

> <comment-title></comment-title>
> Note that sequence databases used for metaproteomics are usually much larger than the excerpt used in this tutorial. When using large databases, the peptide identification step can take much more time for computation. In metaproteomics, choosing the optimal database is a crucial step of your workflow, for further reading see [Timmins-Schiffman et al (2017)](https://www.ncbi.nlm.nih.gov/pubmed/27824341).
>
> To learn more about database construction in general, like integrating contaminant databases or using a decoy strategy for FDR searching, please consult our tutorial on [Database Handling]({{site.baseurl}}/topics/proteomics/tutorials/database-handling/tutorial.html).
>
{: .comment}

### PeptideShaker

[PeptideShaker](https://compomics.github.io/projects/peptide-shaker.html) is a post-processing software tool that
processes data from the SearchGUI software tool. It serves to organize the Peptide-Spectral
Matches (PSMs) generated from SearchGUI processing and is contained in the SearchGUI archive.
It provides an assessment of confidence of the data, inferring proteins identified from the
matched peptide sequences and generates outputs that can be visualized by users to interpret
results. PeptideShaker has been wrapped in Galaxy to work in combination with SearchGUI
outputs.

> <comment-title></comment-title>
> There are a number of choices for different data files that can be generated using
> PeptideShaker. A compressed file can be made containing all information needed to view the
> results in the standalone PeptideShaker viewer. A `mzidentML` file can be created that contains
> all peptide sequence matching information and can be utilized by compatible downstream
> software. Other outputs are focused on the inferred proteins identified from the PSMs, as well
> as phosphorylation reports, relevant if a phosphoproteomics experiment has been undertaken.
> More detailed information on peptide inference using SearchGUI and PeptideShaker can be found in our tutorial on [Peptide and Protein ID]({{site.baseurl}}/topics/proteomics/tutorials/protein-id-sg-ps/tutorial.html).
{: .comment}

> <hands-on-title>PeptideShaker</hands-on-title>
>
> 1. {% tool [Peptide Shaker](toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/peptide_shaker/1.16.36.3) %} with the following parameters:
>   - **Compressed SearchGUI results**: The SearchGUI archive file
>   - **Specify Advanced PeptideShaker Processing Options**: `Default Processing Options`
>   - **Specify Advanced Filtering Options**: `Advanced Filtering Options`
>   - **Maximum Precursor Error Type**: `Daltons`
>   - **Specify Contact Information for mzIdendML**: You can leave the default dummy options for now, but feel free to enter custom contact information.
>   - **Include the protein sequences in mzIdentML**: `No`
>   - **Output options**: Select the `PSM Report` (Peptide-Spectral Match) and the `Certificate of Analysis`
>
>       > <comment-title></comment-title>
>       >
>       > The **Certificate of Analysis** provides details on all the parameters
>       > used by both SearchGUI and PeptideShaker in the analysis. This can be downloaded from the
>       > Galaxy instance to your local computer in a text file if desired.
>       {: .comment}
>
> 2. Click **Run Tool** and inspect the resulting files after they turned green with the **View data** icon:
>     ![View data button](../../images/view_data_icon.png)
>
{: .hands_on}


A number of new items will appear in your history, each corresponding to the outputs selected
in the PeptideShaker parameters. Most relevant for this tutorial is the PSM report:

![Display of the PSM report tabular file](../../images/psm_report.png "The PSM report")

Scrolling towards left will show the sequence for the PSM that matched to these
metapeptide entries. Column 3 is the sequence matched for each PSM entry. Every identified PSM is a
new row in the tabular output.

In the following steps of this tutorial, selected portions of this output will be extracted and used for
analysis of the taxonomic make-up of the sample as well as the biochemical functions
represented by the proteins identified.

## Taxonomy analysis

In the previous section, the genome sequencing and mass spectrometry data from
processing of biological samples was used to identify peptides present in those samples.
Now those peptides are used as evidence to infer which organisms are represented in the sample,
and what biological functions those peptides and associated proteins suggest are occurring.

The UniProt organization collects and annotates all known proteins for organisms. A UniProt
entry includes the protein amino acid sequence, the NCBI taxonomy, and any annotations
about structure and function of the protein. The UniPept web resource developed
by Ghent University will be used to match the sample peptides to proteins. UniPept indexes all Uniprot
proteins and provides a fast matching algorithm for peptides.

> <comment-title>Unipept</comment-title>
>
> Users can access UniPept via a [web page](https://unipept.ugent.be) and paste peptide
> sequences into the search form to retrieve protein information. But we'll use the Galaxy
> *Unipept* tool to automate the process. The *Unipept* tool sends the peptide list to the
> UniPept REST API service, then transforms the results into datasets that can be further analyzed
> or operated on within Galaxy.
{: .comment}

### Recieving the list of peptides: Query Tabular

In order to use *Unipept*, a list containing the peptide sequences has to be generated.
The tool **Query Tabular** can load tabular data (the PSM report in this case) into a SQLite data base.
As a tabular file is being read, line filters may be applied and an SQL query can be performed.

> <hands-on-title>Query Tabular</hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.0.0) %} with the following parameters:
>
>    - **Database Table**: Click on `+ Insert Database Table`:
>    - **Tabular Dataset for Table**: The PSM report
>
>    Section **Filter Dataset Input**:
>
>    - **Filter Tabular Input Lines**: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `by regex expression matching`
>        - **regex pattern**: `^\d`
>        - **action for regex match**: `include line on pattern match`
>
>    Section **Table Options**:
>
>    - **Specify Name for Table**: `psm`
>    - **Specify Column Names (comma-separated list)**: `id,,sequence,,,,,,,,,,,,,,,,,,,,confidence,validation`
>
>        > <comment-title></comment-title>
>        >
>        > By default, table columns will be named: c1,c2,c3,...,cn (column names for a table must be unique).
>        > You can override the default names by entering a comma separated list of names, e.g. `,name1,,,name2`
>        > would rename the second and fifth columns.
>        >
>        > Check your input file to find the settings which best fits your needs.
>        {: .comment}
>
>    - **Only load the columns you have named into database**: `Yes`
>
>    - **Save the sqlite database in your history**: `Yes`
>
>        > <comment-title>Querying SQLite Databases</comment-title>
>        >
>        > * **Query Tabular** can also use an existing SQLite database. Activating `Save the sqlite database in your history`
>        > will store the created database in the history, allowing to reuse it directly.
>        >
>        {: .comment}
>
>    - **SQL Query to generate tabular output**:
>
>          SELECT distinct sequence
>
>          FROM psm
>
>          WHERE confidence >= 95
>
>          ORDER BY sequence
>
>    > <question-title></question-title>
>    >
>    > The SQL query might look confusing at first, but having a closer look should clarify a lot.
>    >
>    > 1. What does `FROM psm` mean?
>    > 2. What need to be changed if we only want peptides with a confidence higher then 98%?
>    >
>    > > <solution-title></solution-title>
>    > > 1. We want to read from table "psm". We defined the name before in the "Specify Name for Table" option.
>    > > 2. We need to change the value in line 3: "WHERE validation IS NOT 'Confident' AND confidence >= 98"
>    > {: .solution }
>    {: .question}
>
>    - **include query result column headers**: `No`
>
> 2. Click **Run Tool** and inspect the query results file after it turned green. If everything went well, it should look similiar:
>
>     ![Query Tabular output showing the peptides](../../images/query_tabular_1.png "Query Tabular output")
>
{: .hands_on}

While we can proceed with this list of peptides, let's practice using the created SQLite database for further queries.
We might not only be interested in all the distinct peptides, but also on how many PSMs a single peptide had.
Therefore we can search the database for the peptides and count the occurrence without configuring the tables and columns again:

> <hands-on-title>SQLite to tabular</hands-on-title>
>
> 1. {% tool [SQLite to tabular](toolshed.g2.bx.psu.edu/repos/iuc/sqlite_to_tabular/sqlite_to_tabular/2.0.0) %} with the following parameters:
>
>    - **SQL Query**:
>
>          SELECT sequence as "peptide", count(id) as "PSMs"
>
>          FROM psm
>
>          WHERE confidence >= 95
>
>          GROUP BY sequence
>
>          ORDER BY sequence
>
>
> 2. Click **Run Tool**. The resulting file should have two columns, one with the distinct peptides, the other with the count number of PSMs.
>
{: .hands_on}


### Retrieve taxonomy for peptides: Unipept

The generated list of peptides can now be used to search via *Unipept*.
We do a taxonomy analysis using the UniPept pept2lca function to return the taxonomic lowest common ancestor for each peptide:

> <hands-on-title>Unipept</hands-on-title>
>
> 1. {% tool [Unipept](toolshed.g2.bx.psu.edu/repos/galaxyp/unipept/unipept/6.2.4+galaxy1) %} with the following parameters:
>
>    - **Unipept application**: `pept2lca: lowest common ancestor`
>    - **Peptides input format**: `tabular`
>    - **Tabular Input Containing Peptide column**: The query results file.
>    - **Select column with peptides**: `Column 1`
>    - **Choose outputs**: Select `tabular` and `JSON taxonomy tree`
>
> 2. Click **Run Tool**. The history should grow by two files. View each to see the difference.
>
>       > <comment-title></comment-title>
>       >
>       > The JSON (JavaScript Object Notation) file contains the same information as the tabular file but is not comfortably human readable.
>       > Instead, we can use it to use JavaScript libraries to visualize this data.
>       {: .comment}
>
> 3. Visualize the data:
>
>    - Click on the JSON output file from the *Unipept* tool to expand it. Click on the **Visualize** button and select **Unipept Tree viewer**:
>
>       ![Visualize button](../../images/visualize_button.png)
>
>    - A new window should appear with a visualization of the taxonomy tree of your data. Use the mouse wheel to scroll in and out and click on nodes to expand or collapse them:
>
>       ![Unipept Tree viewer visual output](../../images/unipept_tree_viewer.png "Interactive visualization from the Unipept Tree viever plugin")
>
{: .hands_on}

## Genus taxonomy level summary

The tabular *Unipept* output lists the taxonomy assignments for each peptide. To create a meaningful summary, the **Query Tabular** tool is
once again used, aggregating the number of peptides and PSMs for each genus level taxonomy assignment:

> <hands-on-title>Query Tabular</hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.0.0) %} with the following parameters:
>
>    - **Database Table**: Click on `+ Insert Database Table`
>    - **Tabular Dataset for Table**: The PSM report
>
>    Section **Filter Dataset Input**:
>
>    - **Filter Tabular Input Lines**: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `by regex expression matching`
>        - **regex pattern**: `^\d`
>        - **action for regex match**: `include line on pattern match`
>
>    Section **Table Options**:
>
>    - **Specify Name for Table**: `psm`
>    - **Specify Column Names (comma-separated list)**: `,,sequence,,,,,,,,,,,,,,,,,,,,confidence,validation`
>
>    - **Only load the columns you have named into database**: `Yes`
>
> 2. Repeat this step to have a second **Database Table**:
>
>    - **Database Table**: Click on `+ Insert Database Table`
>    - **Tabular Dataset for Table**: The **Unipept** `tabular`/`tsv` output
>
>    Section **Filter Dataset Input**:
>
>    - **Filter Tabular Input Lines**: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `by regex expression matching`
>        - **regex pattern**: `#peptide`
>        - **action for regex match**: `exclude line on pattern match`
>
>    Section **Table Options**:
>
>    - **Specify Name for Table**: `lca`
>    - **Specify Column Names (comma-separated list)**: `peptide,,,,,,,,,,,,,,,,,,,,,genus`
>
>    - **Only load the columns you have named into database**: `Yes`
>
>    - **Save the sqlite database in your history**: `Yes`
>
>    - **SQL Query to generate tabular output**:
>
>          SELECT lca.genus,count(psm.sequence) as "PSMs",count(distinct psm.sequence) as "DISTINCT PEPTIDES"
>
>          FROM psm LEFT JOIN lca ON psm.sequence = lca.peptide
>
>          WHERE confidence >= 95
>
>          GROUP BY lca.genus
>
>          ORDER BY PSMs desc, 'DISTINCT PEPTIDES' desc
>
>
> 2. Click **Run Tool** and inspect the query results file after it turned green:
>
>     ![Query Tabular output showing gene, PSMs and distinct peptides](../../images/metaproteomics_summary.png "Query Tabular output")
>
{: .hands_on}

## Functional Analysis

Recent advances in microbiome research indicate that functional characterization via metaproteomics analysis has the potential to accurately
measure the microbial response to perturbations. In particular, metaproteomics enables the estimation of the function of the microbial
community based on expressed microbial proteome.

In the following chapter, a functional analysis will be performed using the **UniPept** application `pept2prot` in order to match the list of peptides with the correlated Gene Ontology terms.
This allows to get an insight of the **biological process**, the **molecular function** and the **cellular component** related to the sample data.

> <comment-title>Gene Ontology (GO) Consortium</comment-title>
>
> The [Gene Ontology Consortium](http://www.geneontology.org/) provides with its Ontology a framework for the model of biology.
> The GO defines concepts/classes used to describe gene function, and relationships between these concepts. It classifies functions along three aspects:
>
>
> - **molecular function**
>
>   - molecular activities of gene products
>
> - **cellular component**
>
>   - where gene products are active
>
> - **biological process**
>
>   - pathways and larger processes made up of the activities of multiple gene products.
>
> [more information](http://geneontology.org/page/ontology-documentation)
>
{: .comment}

### Data upload

For this tutorial, a tabular file containing the relevant GO terms has been created. It contains the GO aspect, the ID and the name.
It is available at Zenodo: [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.839701.svg)](https://doi.org/10.5281/zenodo.839701).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the file `Gene_Ontology_Terms.tabular` from Zenodo.
>
>    > <tip-title>Setting file metadata on upload</tip-title>
>    >
>    > In the upload window of Galaxy you can set the filetype and related genome of the file you're uploading in the corresponding columns beforehand.
>    > This might be handy if the automatic detection of the filetype didn't work out perfectly or if you want to avoid setting the genome later on, especially for multiple files.
>    >
>    {: .tip}
>
>    As default, Galaxy takes the link as name.
>
>    > <comment-title></comment-title>
>    > - Rename the datasets to a more descriptive name, e.g. `Gene Ontology Terms`
>    {: .comment}
>
>
{: .hands_on}

> <details-title>Creating your own Gene Ontology list</details-title>
>
> The latest Gene Ontology can be downloaded [the GO website](http://geneontology.org/page/download-ontology) as a text file in the `OBO` format.
> `OBO` files are human-readable (in addition to machine-readable) and can be opened in any text editor. They contain more information than just the name and aspect.
>
> In order to receive a file like we use in the tutorial for your own analysis, different tools are available to extract information from `OBO` files,
> one of them being ONTO-PERL ({% cite Antezana_2008 %}).
> An example file with all GO terms from 08.07.2017 named `Gene_Ontology_Terms_full_07.08.2017.tabular` can be found on the [Zenodo repository](https://doi.org/10.5281/zenodo.839701) of this tutorial as well.
> You could also upload the Gene Ontology Terms by copying this link on to the Upload Data - Paste/Fetch data `https://zenodo.org/record/839701/files/Gene_Ontology_Terms_full_07.08.2017.tabular`
>
{: .details}

### Retrieve GO IDs for peptides: Unipept

The **UniPept** application `pept2prot` can be used to return the list of proteins containing each peptide.
The option `retrieve extra information` option is set to `yes` so that we retrieve Gene Ontology assignments (`go_references`)
for each protein.

> <hands-on-title>Unipept</hands-on-title>
>
> 1. {% tool [Unipept](toolshed.g2.bx.psu.edu/repos/galaxyp/unipept/unipept/6.2.4+galaxy1) %} with the following parameters:
>
>    - **Unipept application**: `pept2prot: UniProt entries containing a given tryptic peptide`
>    - **retrieve extra information**: `Yes`
>    - **Peptides input format**: `tabular`
>    - **Tabular Input Containing Peptide column**: The first query results file.
>    - **Select column with peptides**: `Column 1`
>    - **Choose outputs**: Select `tabular`
>
> 2. Click **Run Tool**.
>
> 3. inspect the result:
>
>    - The output should be a tabular file containing a column labeled `go_references`. This is what we're looking for.
>
{: .hands_on}


### Combine all information to quantify the GO results

As a final step we will use **Query Tabular** in a more sophisticated way to combine all information to quantify the GO analysis. The three used file and the extracted information are:

- **Gene Ontology Terms**:
    - `go_id` to match with **Normalized UniPept output**
    - The GO `aspect` to group the results in three separate files
    - The GO `description` to annotate the results
- **Normalized UniPept output**:
    - `peptide` to match with **PSM Report** and to count distinct peptides per GO term
    - `go_reference` to match with **Gene Ontology Terms**
- **PSM Report**:
    - `sequence` to match with **Normalized UniPept output**
    - `id` to count distinct PSM's per GO term

> <hands-on-title>Query Tabular</hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.0.0) %} with the following parameters:
>
>    - **Database Table**: Click on `+ Insert Database Table`
>    - **Tabular Dataset for Table**: The `Gene Ontology Terms` file
>
>    Section **Filter Dataset Input**:
>
>    - **Filter Tabular Input Lines**: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `skip leading lines`
>        - **Skip lines**: `1`
>
>    Section **Table Options**:
>
>    - **Specify Name for Table**: `go`
>    - **Specify Column Names (comma-separated list)**: `aspect,go_id,description`
>    - **Table Index**: Click on `+ Insert Table Index`:
>        - **This is a unique index**: `No`
>        - **Index on Columns**: `aspect,go_id`
>
>
> 2. Repeat this step to have a second **Database Table**:
>
>    - **Database Table**: Click on `+ Insert Database Table`
>    - **Tabular Dataset for Table**: The **Unipept** `tabluar`/`tsv` output
>
>    Section **Filter Dataset Input**:
>
>    - **Filter Tabular Input Lines**: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `skip leading lines`
>        - **Skip lines**: `1`
>    - Add another Filter: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `prepend a line number column`
>
>    Section **Table Options**:
>
>    - **Specify Name for Table**: `bering_prot`
>    - **Specify Column Names (comma-separated list)**: `id,peptide,uniprot_id,taxon_id,taxon_name,ec_references,go_references,refseq_ids,refseq_protein_ids,insdc_ids,insdc_protein_ids`
>    - **Table Index**: Click on `+ Insert Table Index`:
>        - **This is a unique index**: `No`
>        - **Index on Columns**: `id,peptide`
>
> 3. Repeat this step to have another **Database Table**:
>
>    - **Database Table**: Click on `+ Insert Database Table`
>    - **Tabular Dataset for Table**: The same **Unipept** `tabluar`/`tsv` output
>
>    Section **Filter Dataset Input**:
>
>    - **Filter Tabular Input Lines**: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `skip leading lines`
>        - **Skip lines**: `1`
>    - Add another Filter: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `prepend a line number column`
>    - Add another Filter: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `select columns`
>        - **enter column numbers to keep**: `1,7`
>    - Add another Filter: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `normalize list columns, replicates row for each item in list`
>        - **enter column numbers to normalize**: `2`
>        - **List item delimiter in column**: ` ` (a single blank character)
>
>    > <comment-title></comment-title>
>    > - The UniPept result file can contain multiple GO IDs in a single row. In order to create a normalized table of this data, these rows will be split so each record contains only one GO ID.
>    {: .comment}
>
>    Section **Table Options**:
>
>    - **Specify Name for Table**: `bering_prot_go`
>    - **Specify Column Names (comma-separated list)**: `id,go_reference`
>    - **Table Index**: Click on `+ Insert Table Index`:
>        - **This is a unique index**: `No`
>        - **Index on Columns**: `go_reference,id`
>
> 4. Repeat this step to have another **Database Table**:
>
>    - **Database Table**: Click on `+ Insert Database Table`
>    - **Tabular Dataset for Table**: The `PSM Report`
>
>    Section **Filter Dataset Input**:
>
>    - **Filter Tabular Input Lines**: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `by regex expression matching`
>        - **regex pattern**: `^\d`
>        - **action for regex match**: `include line on pattern match`
>    - Add another Filter: Click on `+ Insert Filter Tabular Input Lines`:
>    - **Filter By**: Select `select columns`
>        - **enter column numbers to keep**: `1,3,23,24`
>
>    Section **Table Options**:
>
>    - **Specify Name for Table**: `bering_psms`
>    - **Specify Column Names (comma-separated list)**: `id,sequence,confidence,validation`
>    - **Only load the columns you have named into database**: `Yes`
>    - **Table Index**: Click on `+ Insert Table Index`:
>        - **This is a unique index**: `No`
>        - **Index on Columns**: `sequence,id`
>
>    - **Save the sqlite database in your history**: `Yes`
>
>    - **SQL Query to generate tabular output**:
>
>          SELECT sequence as "peptide", count(id) as "PSMs"
>
>          FROM bering_psms
>
>          WHERE confidence >= 95
>
>          GROUP BY sequence
>
>          ORDER BY sequence
>
>
> 5. Click **Run Tool**.
>
{: .hands_on}

With this we have combined all the data into a single database which we can now query to extract the desired information with **SQLite to tabular**:

> <hands-on-title>SQLite to tabular</hands-on-title>
>
> 1. {% tool [SQLite to tabular](toolshed.g2.bx.psu.edu/repos/iuc/sqlite_to_tabular/sqlite_to_tabular/2.0.0) %} with the following parameters:
>
>    - **SQLite Database**: The created SQLite database from the former step
>    - **SQL Query**:
>
>          SELECT go.description,
>
>          count(distinct bering_psms.sequence) as "bering_peptides", count(distinct bering_psms.id) as "bering_psms"
>
>          FROM go JOIN bering_prot_go ON go.go_id = bering_prot_go.go_reference JOIN bering_prot on bering_prot_go.id = bering_prot.id JOIN
>
>          bering_psms ON bering_prot.peptide = bering_psms.sequence
>
>          WHERE go.aspect = 'molecular_function'
>
>          GROUP BY go.description
>
>          ORDER BY  bering_peptides desc,bering_psms desc
>
> 2. Click **Run Tool**.
> 3. Repeat these steps two times by replacing `molecular_function` in the fifth row of the SQL query by `biological_process` and `cellular_component`.
>
{: .hands_on}

With these three output files the functional analysis of this tutorial is finished. Each record contains the name of a GO term, the amount of peptides related to it and the amount of PSMs for these peptides.

> <comment-title>References</comment-title>
>
> - [Dataset](https://www.ncbi.nlm.nih.gov/pubmed/27824341) and [SixGill software](https://www.ncbi.nlm.nih.gov/pubmed/27396978)
>
> - [Galaxy workflows for metaproteomics](https://www.ncbi.nlm.nih.gov/pubmed/26058579)
>
> - [Metaproteomics community effort](https://z.umn.edu/gcc2017mporal)
>
> - [Unipept](https://www.ncbi.nlm.nih.gov/pubmed/28552653)
>
> - [Metaproteomics video](http://z.umn.edu/mpvideo2018)
{: .comment}

tools_json:
["Search GUI", "toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/search_gui/3.3.10.1", "Peptide Shaker", "toolshed.g2.bx.psu.edu/repos/galaxyp/peptideshaker/peptide_shaker/1.16.36.3", "Query Tabular", "toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.0.0", "SQLite to tabular", "toolshed.g2.bx.psu.edu/repos/iuc/sqlite_to_tabular/sqlite_to_tabular/2.0.0", "Unipept", "toolshed.g2.bx.psu.edu/repos/galaxyp/unipept/unipept/6.2.4+galaxy1"]


topic: proteomics
tutorial: neoantigen-variant-annotation
title: Neoantigen 4: Variant Annotation
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-free]"]
content_hash: 2ee3d65f2b64f3fa83b991361c9c526c
last_modified: 2026-05-27T09:01:49.460091
zenodo_link: https://zenodo.org/records/14375693
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/neoantigen-variant-annotation/tutorial.html

content:
Neoantigens are tumor-specific antigens that arise from somatic mutations in cancer cells. These mutations result in the generation of abnormal peptides that can be presented by the immune system’s Major Histocompatibility Complex (MHC) molecules. Neoantigens are gaining significant attention in cancer immunotherapy, as they hold the potential to be used in personalized vaccines and therapies aimed at stimulating the immune system to target and destroy tumor cells.

The process of identifying neoantigens begins with the analysis of genomic data to detect somatic mutations, followed by the prediction of the corresponding peptide sequences that can bind to MHC molecules. This process involves a combination of bioinformatics tools and techniques, including variant calling, peptide prediction, and MHC binding affinity analysis. Successful identification of neoantigens can pave the way for personalized treatment strategies that improve patient outcomes in cancer therapy.

This tutorial focuses on the Neoantigen Annotation pipeline, which is designed to assist researchers and clinicians in annotating and predicting neoantigens from high-throughput genomic data. By the end of this tutorial, you will be equipped with the skills to extract meaningful insights from genomic datasets, predict immunogenic peptides, and understand how these peptides interact with the immune system. These insights are essential for advancing the field of personalized immunotherapy and improving cancer treatment outcomes.

![Variant-annotation-overview-workflow]({% link topics/proteomics/images/neoantigen/PepPointer_Characterization_1.PNG %})

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Neoantigen Variant Annotation

This tutorial introduces the process of annotating neoantigens using the PepPointer tool. The aim is to identify potential neoantigens by predicting peptide sequences derived from somatic mutations in cancer genomes, followed by their binding affinity to MHC molecules. The workflow integrates several key bioinformatics steps, including variant analysis, peptide prediction, and immune system interaction analysis. Below is an overview of each major step involved in the tutorial:It is divided into two parts: A, which is the variant annotation, and B, which focuses on the database generation for IEDB.

- **A: Variant Annotation**
   1. **Get Data** The first step in the process is to gather the necessary data for analysis, which typically includes genomic data in VCF (Variant Call Format) or other formats containing information about somatic mutations. The data is then uploaded into the analysis environment. A well-organized dataset is crucial for smooth and efficient analysis, so it’s important to ensure proper file structure and metadata tagging before beginning.
   2. **Mutation to Peptide Mapping** The next step is to map these mutations to peptide sequences. This involves creating a list of peptides that contain the mutated residue, representing potential neoantigens. Peptide prediction tools are used to generate the corresponding peptide sequences from the mutated positions.
   3. **Annotation and Filtering** At this stage, the predicted peptides are annotated with relevant biological and immunological information, such as their predicted MHC class, binding affinity, and potential for being recognized by T-cells. Filtering is performed to retain only the most promising candidates, based on binding affinity thresholds and relevance to the tumor type being studied.
   4. **Mapping Peptide sequences with PepPointer** PepPointer is used to map the peptide sequences to their corresponding genomic coordinates. This tool helps align peptide sequences (often derived from proteomic data) to the genomic context, providing useful insights into where these peptides are located in the genome. It allows researchers to determine which genomic regions are associated with the peptides of interest, facilitating the study of their potential functional roles.
   5. **Visualization and Interpretation** The final step involves visualizing the results of the annotation and filtering steps. Various bioinformatics tools can be used to present the data in a way that is easy to interpret, such as visualizing peptide binding affinity scores or generating summary plots that highlight the most immunogenic neoantigens. This step helps in drawing meaningful conclusions about the potential of the identified peptides for cancer immunotherapy.

- **B: Database Generation for IEDB**
   1. **Generating FASTA for MHC binding tool** In this step, we prepare the peptide sequences in FASTA format to be used with an MHC binding prediction tool. MHC (Major Histocompatibility Complex) binding tools are often used in immunology research to predict which peptides can bind to specific MHC molecules and present them to T-cells.


# Variant annotation

![Variant-annotation-overview-workflow]({% link topics/proteomics/images/neoantigen/PepPointer_Characterization_2.PNG %})

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/14375693/files/Novel_Peptides_from_PepQuery.tabular
>    https://zenodo.org/records/14375693/files/Fragpipe-Peptide-Report.tabular
>    https://zenodo.org/records/14375693/files/GffCompare_Annotated_GTF_to_BED.bed
>    https://zenodo.org/records/14375693/files/Homo_sapiens.GRCh38_canon.106.gtf
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to ...
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}



## Mutation to Peptide Mapping with Query Tabular

In this step, we will use the Query Tabular tool to perform a SQL query on a tabular dataset. This tool allows you to filter and extract data from a dataset by applying SQL commands, making it ideal for working with structured data such as CSV or TSV files. You will use it to join tables, select relevant columns, and generate a new output based on the criteria specified in the query. This step helps in refining and organizing the data for further analysis.
This step extracts information about novel peptides from FragPipe, which primarily includes protein details and the location of the peptide on the protein.

> <hands-on-title> Query Tabular </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `Novel_Peptides_from_PepQuery.tabular` (Input dataset)
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `Fragpipe-Peptide-Report.tabular` (Input dataset)
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT t1.c1,t2.c14,t2.c4,t2.c5
> FROM t1
> INNER JOIN t2
> ON t1.c1 = t2.c1
> ```
>    - *"include query result column headers"*: `No`
>
{: .hands_on}



## Annotation and Filtering

Because the annotation structures differ between **assembly-derived peptides** and **SAV (single-amino-acid–variant) peptides**, we apply separate text-formatting procedures for each. We first format the assembly-derived peptide annotations, followed by the SAV annotations, and will be processed separately with **PepPointer**.

### Extracting assembly-derived peptides

The **Select lines that match an expression** tool is used to extract only the rows corresponding to StringTie-assembled transcripts from a mixed text file. StringTie typically labels transcript or gene identifiers with the prefix `STRG`. By filtering for lines containing `STRG`, we keep only the relevant transcript entries and discard header or unrelated lines. This cleaned file is then used for downstream integration with genomic and variant annotations.

> <hands-on-title> Select StringTie transcript lines </hands-on-title>
>
> 1. {% tool [Select lines that match an expression](toolshed.g2.bx.psu.edu/repos/iuc/select/select/1.0.4) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"* (input): `Peptide_to_Protein_Annotation (Query Tabular output)`
>    - *"that"*: `Matching`
>    - *"the pattern"*:
>      ```
>      (STRG)
>      ```
>
>    This will retain only lines that contain the `STRG` label.
>    
{: .hands_on}

> <question-title></question-title>
>
> 1. Why do we filter for lines containing `STRG` in the StringTie output?  
> 2. What types of lines are removed by this filtering step?
>
> > <solution-title></solution-title>
> >
> > 1. The `STRG` prefix identifies StringTie-assembled genes/transcripts. Filtering for `STRG` ensures that only these biologically relevant entries are carried forward for coordinate annotation.  
> > 2. Header lines, comments, or any non-StringTie records that do not contain `STRG` are removed. This avoids clutter and prevents unrelated entries from interfering with downstream joins and comparisons.  
> >
> {: .solution}
>
{: .question}

### Converting delimiters - Assembly-derived peptides
In this step, we will use the Convert tool to modify characters in a dataset. Specifically, the tool will be used to convert all instances of pipe characters (|) to another format. The tool helps clean and standardize the data for subsequent processing or analysis. This is often a necessary step when preparing data for tools that require specific formats or when performing tasks such as parsing or file importation.


> <hands-on-title> Convert </hands-on-title>
>
> 1. {% tool [Convert - delimiters to TAB](Convert characters1) %} with the following parameters:
>    - *"Convert all"*: `Pipes`
>    - {% icon param-file %} *"in Dataset"*: `output` (output of **Select** {% icon tool %})
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the purpose of using the "Convert" tool in this step?
>
> > <solution-title></solution-title>
> >
> > 1. The "Convert" tool is used to replace or remove specific characters in a dataset, such as pipe characters (`|`). This ensures that the data conforms to the required format for subsequent analysis or tools. Pipe characters are often used as delimiters in tabular data. Replacing them might be necessary if the data needs to be formatted for a different tool or if the pipe character interferes with parsing in downstream steps.
> >
> {: .solution}
>
{: .question}

### Removing Colons and converting into a different tabular
In this step, we will use the Convert tool to remove colons from the dataset. This conversion is useful when you need to standardize the data or prepare it for downstream processing, where colons may interfere with the analysis or formatting. By selecting the "Convert all" option for colons, all occurrences of colons in the dataset will be removed, creating a clean tabular format for further analysis.

> <hands-on-title> Convert </hands-on-title>
>
> 1. {% tool [Convert - delimiters to TAB](Convert characters1) %} with the following parameters:
>    - *"Convert all"*: `Colons`
>    - {% icon param-file %} *"in Dataset"*: `out_file1` (output of **Convert** {% icon tool %})
>
>
{: .hands_on}

### Editing certain "u" characters
In this step, we will use the Column Regex Find And Replace tool to find and replace specific patterns in the data columns. Regular expressions (regex) allow for pattern matching, making it easier to identify and modify data based on specific patterns. In this case, the tool will search for occurrences of the pattern u_ and replace them with u: in column c3 of the dataset. This is commonly done to standardize data formatting or clean data for further analysis. In this step, any protein sequences with a "u_" will be replaced with "u".

> <hands-on-title> Column Regex Find And Replace </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `out_file1` (output of **Convert** {% icon tool %})
>    - *"using column"*: `3`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `u_`
>            - *"Replacement"*: `u:`
>
>
>
{: .hands_on}

### Removing Colons and converting into a different tabular

> <hands-on-title> Convert </hands-on-title>
>
> 1. {% tool [Convert - delimiters to TAB](Convert characters1) %} with the following parameters:
>    - *"Convert all"*: `Colons`
>    - {% icon param-file %} *"in Dataset"*: `out_file1` (output of **Column Regex and Replace** {% icon tool %})
>
>
{: .hands_on}


### Extracting bed file information using Query Tabular - Assembly-derived peptides
In this step, we will use the Query Tabular tool to extract specific information from a dataset, such as a BED file containing genomic regions, and match it with novel peptides. This allows for identifying the relevant genomic and peptide information by querying data from two sources and combining them through an SQL query. By using an INNER JOIN operation, we can merge data from two tables based on shared columns, and retrieve the necessary information. This query extracts specific columns from both the BED file (such as genomic coordinates) and the novel peptide dataset (such as peptide sequences or identifiers), enabling the identification of peptides that correspond to specific genomic regions. These are the columns that will be extracted:

- Chrom: Chromosome name (e.g., chr1).
- Start: Starting position of the feature (zero-based index).
- End: Ending position of the feature (one-based index).
- Name: Identifier for the feature (e.g., gene name).
- Score: Numeric score representing feature strength or relevance.
- Strand: Strand orientation (+ or -).
- ThickStart and ThickEnd: Define the start and end of the transcribed or relevant part of the feature, often the coding region.

> <hands-on-title> Extract BED file for Assembly-derived peptides </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `out_file1` (output of **Convert** {% icon tool %})
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `GffCompare_Annotated_GTF_to_BED.bed` (Input dataset)
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT t1.c1,t1.c8,t1.c9,t1.c11,t1.c12,t2.c5,t2.c6,t1.c7
> FROM t1
> INNER JOIN t2
> ON t1.c3 = t2.c4
> ```
>    - *"include query result column headers"*: `No`
>
> 2. Rename file to "assembly_formatted_table".
> 
{: .hands_on}


### Convert proteomic coordinates to genomic coordinates - Assembly-derived peptides
To convert proteomic coordinates to genomic coordinates, it is essential to account for the relationship between the protein sequence and its corresponding gene or genomic region. In this workflow, the proteomic coordinates have already been extracted at the amino acid level. Since each amino acid in the protein sequence corresponds to a triplet of nucleotides (a codon) in the mRNA, we need to multiply the proteomic coordinate by 3 to obtain the genomic coordinate. This conversion will give us the position of each amino acid within the genomic sequence. The resulting genomic coordinates are stored in a separate column for easy reference. Once this step is completed, we can extract and organize the information in the correct order for further analysis or mapping to the genomic reference.


> <hands-on-title> Text manipulation for Assembly-derived peptides </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `assembly_formatted_table` (output of **Query Tabular - Assembly** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
>      ```
>      SELECT t1.*,
>      (t1.c4 - 1) * 3 AS c2_multiplied,
>      t1.c5 * 3 AS c3_multiplied
>      FROM t1
>      ```
>    - *"include query result column headers"*: `No`
>
>
{: .hands_on}


### Annotating the genomic coordinates - Assembly-derived peptides

The Query Tabular step in this workflow is used to extract and calculate genomic coordinates based on the proteomic data. The SQL query within the tool defines two calculations for genomic coordinates, start and stop, based on the strand information of the data. For each row in the input dataset (t1), if the strand (t1.c7) is "-" (negative), the genomic coordinates are calculated by subtracting the position from the given end (t1.c3 - t1.c9 for start, and t1.c3 - t1.c10 for stop). If the strand is "+" (positive), the genomic coordinates are calculated by adding the respective positions (t1.c2 + t1.c9 for start, and t1.c2 + t1.c10 for stop). These calculated coordinates are then returned in the query results, where they will be included as new columns (start and stop). This step is essential for transforming the proteomic information into genomic positions for further analysis.

> <hands-on-title> Table processing for Assembly-derived peptides </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `output` (output of **Text manipulation - Assembly** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
>      ```
>      SELECT t1.*,
>      CASE
>      WHEN t1.c7 = '-' THEN t1.c3 - t1.c9
>      WHEN t1.c7 = '+' THEN t1.c2 + t1.c9
>      END AS start,
>      CASE
>      WHEN t1.c7 = '-' THEN t1.c3 - t1.c10
>      WHEN t1.c7 = '+' THEN t1.c2 + t1.c10
>      END AS stop FROM t1
>      ```
>    - *"include query result column headers"*: `No`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why do I need to differentiate between the positive and negative strands when calculating genomic coordinates?
> 2. What is the significance of the t1.c9 and t1.c10 columns in the SQL query?
>
> > <solution-title></solution-title>
> >
> > 1. The positive and negative strands represent the two directions in which DNA is read, and they affect how genomic coordinates are calculated. For the negative strand, the coordinates are subtracted, whereas for the positive strand, they are added. This ensures the correct mapping of the protein sequence to the genome.
> > 2. The `t1.c9` and `t1.c10` columns represent specific offsets or lengths within the dataset, which are used to adjust the calculated genomic start and stop coordinates. These offsets could correspond to peptide lengths or specific features of the protein that need to be accounted for in the conversion to genomic coordinates.
> >
> {: .solution}
>
{: .question}


### Generating BED file for PepPointer - Assembly-derived peptides
This step is necessary to extract and reorganize relevant genomic information from the dataset. By querying specific columns such as chromosome (chromosome), start (chromStart), end (chromEnd), and strand (strand), we are preparing the data for further analysis. These values are essential for mapping proteomic or peptide data to the genomic coordinates, ensuring accurate alignment and interpretation of the sequence in the context of its genomic location. Additionally, renaming columns enhances clarity and standardizes the format, making it easier to work with the data in subsequent steps.

> <hands-on-title> BED file for Assembly-derived peptides </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `output` (output of **Table processing - Assembly** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT
> c8 AS `chromosome`,
> c11  AS `chromStart`,
> c12 AS `chromEnd`,
> c1 AS `name`,
> c6 AS `score`,
> c7 AS `strand`
> FROM t1
> ```
>    - *"include query result column headers"*: `No`
>
> 2. Change data type to ".bed".
> 
{: .hands_on}


> <question-title></question-title>
>
> 1. What is the purpose of renaming columns like chromosome, chromStart, and chromEnd in the SQL query?
> 2. Why is the strand column important in this query, and how does it affect genomic coordinate interpretation?
>
> > <solution-title></solution-title>
> >
> > 1. Renaming the columns helps in providing more meaningful names for the data, making it easier to understand the genomic features such as the chromosome and start/end coordinates. These renamed columns align with common genomic nomenclature, improving readability and analysis.
> > 2. The strand column indicates whether the sequence is on the positive or negative strand of the DNA. This is important for correctly interpreting the orientation of the sequence and calculating its genomic coordinates. A positive strand means the coordinates will be calculated from the start, while a negative strand requires reverse calculations to adjust the coordinates properly.
> >
> {: .solution}
>
{: .question}



## Mapping Assembly-derived Peptide sequences with PepPointer

PepPointer is a tool designed to map peptide sequences to their respective genomic locations using data such as GTF and BED files. In this workflow, PepPointer takes the GTF file, which contains gene annotations and genomic coordinates, and combines it with a BED file, which provides chromosomal coordinates of the peptides. By doing this, PepPointer can identify the exact genomic locations of the peptides, linking them to genes and exons. This step is crucial for accurately correlating proteomic data to genomic sequences, enabling a better understanding of the genomic context in which the peptides are found.

> <hands-on-title> PepPointer for Assembly-derived peptides </hands-on-title>
>
> 1. {% tool [PepPointer](toolshed.g2.bx.psu.edu/repos/galaxyp/pep_pointer/pep_pointer/0.1.3+galaxy1) %} with the following parameters:
>    - *"Choose the source of the GTF file"*: `From history`
>        - {% icon param-file %} *"GTF file with the genome of interest"*: `Homo_sapiens.GRCh38_canon.106.gtf` (Input dataset)
>    - {% icon param-file %} *"BED file with chromosomal coordinates of peptide"*: `output` (from **Query Tabular - BED file for Assembly**)
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How does the GTF file help in identifying the genomic location of peptides?
> 2. What happens if the coordinates in the BED file don't align with the genomic features in the GTF file?
>
> > <solution-title></solution-title>
> >
> > 1. The GTF file contains detailed annotations of the genome, including information about genes, exons, and other genomic features. By providing the chromosomal coordinates of these features, the GTF file allows PepPointer to match peptide sequences to the corresponding genes and exons. This ensures that the peptides are mapped accurately within the genome, linking them to the correct genomic context.
> > 2. If the coordinates in the BED file don't align with the genomic features in the GTF file, PepPointer may not be able to accurately map the peptides to their corresponding genes or exons. This misalignment could result in incorrect annotations or a failure to identify the proper genomic location of the peptides. It is important to ensure that both files are from the same genome assembly to avoid such discrepancies.
> >
> {: .solution}
>
{: .question}


## Visualization and Interpretation of Novel Assembly-derived peptides with annotation

In this step, we are using Query Tabular to extract and format relevant information from the results produced by PepPointer. The SQL query in this tool is designed to structure the data into a more readable format, providing key details such as the peptide ID, chromosome location, start and end positions, strand orientation, and any annotations. Additionally, it generates genome coordinates in a format suitable for viewing in genome browsers like IGV and UCSC Genome Browser. This formatting step helps to visualize and interpret the data more effectively by linking the peptides to their genomic context.

> <hands-on-title> Query Tabular </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `classified` (output of **PepPointer - Assembly** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT
> c4 AS Peptide,
> c1 AS Chromosome,
> c2 AS Start,
> c3 AS End,
> c6 AS Strand,
> c7 AS Annotation,
> c1||':'||c2||'-'||c3 AS IGV_Genome_Coordinate,
> 'https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position='||c1||'%3A'||c2||'-'||c3 AS UCSC_Genome_Browser
> FROM  t1
> ```
>    - *"include query result column headers"*: `Yes`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why is it important to include both the "IGV_Genome_Coordinate" and "UCSC_Genome_Browser" columns?
> 2. What is the significance of the "Annotation" column, and what type of data might it contain?
>
> > <solution-title></solution-title>
> >
> > 1. The IGV_Genome_Coordinate column provides a formatted coordinate that is directly usable in genome viewers like IGV, helping users visualize the peptide's genomic location. The UCSC_Genome_Browser column provides a clickable URL to launch UCSC Genome Browser with the coordinates, offering another convenient method for visualizing the peptide's location in a widely used genome browser. Including both makes it easier for users to explore the data in different genomic tools.
> > 2. The Annotation column provides additional context about the genomic feature associated with the peptide, such as whether the peptide is part of a gene, exon, or other regulatory element. This column can help to interpret the biological relevance of the peptide in the genomic region and is important for understanding the functional role of the peptide within the genome. The data might include annotations like "gene", "exon", "intron", or other functional genomic elements.
> >
> {: .solution}
>
{: .question}


## Extracting SAV-derived peptides

The **Select lines that match an expression** tool is used to extract only the rows corresponding to SAV-assembled transcripts from a mixed text file.

> <hands-on-title> Select SAV transcript lines </hands-on-title>
>
> 1. {% tool [Select lines that match an expression](toolshed.g2.bx.psu.edu/repos/iuc/select/select/1.0.4) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"* (input): `Peptide_to_Protein_Annotation (Query Tabular output)`
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*:
>      ```
>      (STRG)
>      ```
>
>    This will retain only lines that do not contain the `STRG` label.
>    
{: .hands_on}


### Converting delimiters
In this step, we will use the Convert tool to modify characters in a dataset. Specifically, the tool will be used to convert all instances of pipe characters (|) to another format. The tool helps clean and standardize the data for subsequent processing or analysis. This is often a necessary step when preparing data for tools that require specific formats or when performing tasks such as parsing or file importation.


> <hands-on-title> Convert </hands-on-title>
>
> 1. {% tool [Convert - delimiters to TAB](Convert characters1) %} with the following parameters:
>    - *"Convert all"*: `Pipes`
>    - {% icon param-file %} *"in Dataset"*: `output` (output of **Select** {% icon tool %})
>
>
{: .hands_on}

### Extracting Relevant Columns for SAV Peptides

The **Cut Columns** tool is used to extract only the fields required for downstream annotation of SAV-derived peptides. Since the structure of SAV output differs from assembly-derived peptide annotations, this step isolates the key identifiers and coordinate fields that will later be used for peptide mapping. By selecting only the columns necessary for variant-driven peptide interpretation, we ensure a clean and consistent table for integration with tools such as PeptidePointer.

> <hands-on-title> Extracting Columns for SAV </hands-on-title>
>
> 1. {% tool [Cut columns from a table](toolshed.g2.bx.psu.edu/repos/iuc/cut/cut/1.0.2) %} with the following parameters:
>    - *“Cut columns”*:  
>      ```
>      c1,c3,c9,c10
>      ```
>    - *“Delimited by”*: `Tab`
>    - {% icon param-file %} *“From”*: (output of **Convert (SAV)** {% icon tool %})
>
> 2. Execute the tool to produce a reduced SAV-specific annotation table containing only the essential fields for downstream processing.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why do we extract only selected columns from the SAV output?  
> 2. How will this SAV table be used later in the workflow?
>
> > <solution-title></solution-title>
> >
> > 1. Extracting only the required fields reduces unnecessary complexity, removes irrelevant columns, and ensures consistent formatting.  
> >
> > 2. The SAV table will later be used in downstream tools such as PeptidePointer, enabling consistent mapping of peptides to genomic features.  
> >
> {: .solution}
>
{: .question}

### Editing SAV Annotation Columns Using Column Regex Find and Replace

The **Column Regex Find and Replace** tool is used in this step to clean and standardize SAV (Single Amino-acid Variant) annotation fields. SAV entries often begin with prefixes such as `SAV_` or `INDEL_`, which must be reformatted to a consistent delimiter-based structure (`SAV|`, `INDEL|`) to be compatible with downstream peptide-annotation tools such as PepPointer.

> <hands-on-title> Editing SAV Annotations </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/bgruening/column_regex_find_replace/column_regex_find_and_replace/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *“Select cells from”* (tabular): (output of **Cut (SAV)** {% icon tool %})
>    - *“using column”*: `2`
>
>    **Regex Rule 1 — Convert INDEL notation**
>    - *Find Regex*: ```INDEL_```
>    - *Replacement*: ```INDEL|```
>
>    **Regex Rule 2 — Convert SNV notation**
>    - *Find Regex*: ```SNV_```
>    - *Replacement*:```SNV|```
>
{: .hands_on}


> <hands-on-title> Convert </hands-on-title>
>
> 1. {% tool [Convert - delimiters to TAB](Convert characters1) %} with the following parameters:
>    - *"Convert all"*: `Pipes`
>    - {% icon param-file %} *"in Dataset"*: `output` (output of **Column Regex Find And Replace (SAV)** {% icon tool %})
>
>
{: .hands_on}


> <hands-on-title> Query Tabular </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `Genomic-Variant-Protein-BED` (output of **Concatenate from Non-Reference workflow** {% icon tool %})
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: (output of **Convert - delimiters to TAB (SAV)** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT t1.*, t2.*
> FROM t1
> JOIN t2
> ON t1.c1 = t2.c3
> ```
>    - *"include query result column headers"*: `No`
>
>
{: .hands_on}

> <hands-on-title> Extracting Columns for SAV BED </hands-on-title>
>
> 1. {% tool [Cut columns from a table](toolshed.g2.bx.psu.edu/repos/iuc/cut/cut/1.0.2) %} with the following parameters:
>    - *“Cut columns”*:  
>      ```
>      c8,c3,c4,c11,c12,c9,c5,c2
>      ```
>    - *“Delimited by”*: `Tab`
>    - {% icon param-file %} *“From”*: (output of **Query Tabular (SAV)** {% icon tool %})
>
> 2. Execute the tool to produce a reduced SAV-specific annotation table containing only the essential fields for downstream processing.
> 
{: .hands_on}

> <hands-on-title> Editing SAV Annotations </hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/bgruening/column_regex_find_replace/column_regex_find_and_replace/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *“Select cells from”* (tabular): (output of **Cut columns (SAV)** {% icon tool %})
>    - *“using column”*: `6`
>
>    **Regex Rule 1 — Convert INDEL notation**
>    - *Find Regex*: ```INDEL```
>    - *Replacement*: ``` ```
>
>    **Regex Rule 2 — Convert SNV notation**
>    - *Find Regex*: ```SNV```
>    - *Replacement*:``` ```
>
> 2. Rename file to "sav_formatted_table".
>
> 
{: .hands_on}

Because the annotation formats differ between **assembly-derived peptides** and **SAV-derived peptides**, we must process them separately. We first perform text formatting on the assembly peptide annotations, followed by formatting of the SAV annotations, and finally process both sets for separate peptide annotation using **PepPointer**.


### Convert proteomic coordinates to genomic coordinates - SAV-derived peptides
To convert proteomic coordinates to genomic coordinates, it is essential to account for the relationship between the protein sequence and its corresponding gene or genomic region. In this workflow, the proteomic coordinates have already been extracted at the amino acid level. Since each amino acid in the protein sequence corresponds to a triplet of nucleotides (a codon) in the mRNA, we need to multiply the proteomic coordinate by 3 to obtain the genomic coordinate. This conversion will give us the position of each amino acid within the genomic sequence. The resulting genomic coordinates are stored in a separate column for easy reference. Once this step is completed, we can extract and organize the information in the correct order for further analysis or mapping to the genomic reference.


> <hands-on-title> Text manipulation - SAV-derived peptides </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `sav_formatted_table` (output of **Column Regex Find And Replace - SAV** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT t1.*,
> (t1.c4 - 1) * 3 AS c2_multiplied,
> t1.c5 * 3 AS c3_multiplied
> FROM t1
> ```
>    - *"include query result column headers"*: `No`
>
>
{: .hands_on}


### Annotating the genomic coordinates - SAV-derived peptides

The Query Tabular step in this workflow is used to extract and calculate genomic coordinates based on the proteomic data. The SQL query within the tool defines two calculations for genomic coordinates, start and stop, based on the strand information of the data. For each row in the input dataset (t1), if the strand (t1.c7) is "-" (negative), the genomic coordinates are calculated by subtracting the position from the given end (t1.c3 - t1.c9 for start, and t1.c3 - t1.c10 for stop). If the strand is "+" (positive), the genomic coordinates are calculated by adding the respective positions (t1.c2 + t1.c9 for start, and t1.c2 + t1.c10 for stop). These calculated coordinates are then returned in the query results, where they will be included as new columns (start and stop). This step is essential for transforming the proteomic information into genomic positions for further analysis.

> <hands-on-title> Text processing - SAV </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `output` (output of **Text manipulation - SAV** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT t1.*,
> CASE
> WHEN t1.c7 = '-' THEN t1.c3 - t1.c9
> WHEN t1.c7 = '+' THEN t1.c2 + t1.c9
> END AS start,
> CASE
> WHEN t1.c7 = '-' THEN t1.c3 - t1.c10
> WHEN t1.c7 = '+' THEN t1.c2 + t1.c10
> END AS stop FROM t1
> ```
>    - *"include query result column headers"*: `No`
>
>
{: .hands_on}



### Generating BED file for PepPointer - SAV-derived peptides
This step is necessary to extract and reorganize relevant genomic information from the dataset. By querying specific columns such as chromosome (chromosome), start (chromStart), end (chromEnd), and strand (strand), we are preparing the data for further analysis. These values are essential for mapping proteomic or peptide data to the genomic coordinates, ensuring accurate alignment and interpretation of the sequence in the context of its genomic location. Additionally, renaming columns enhances clarity and standardizes the format, making it easier to work with the data in subsequent steps.

> <hands-on-title> BED file for SAV-derived peptides </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `output` (output of **Text processing - SAV** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT
> c8 AS `chromosome`,
> c11  AS `chromStart`,
> c12 AS `chromEnd`,
> c1 AS `name`,
> c6 AS `score`,
> c7 AS `strand`
> FROM t1
> ```
>    - *"include query result column headers"*: `No`
>
> 2. Change data type to ".bed".
> 
{: .hands_on}


## Mapping SAV-derived Peptide sequences with PepPointer

PepPointer is a tool designed to map peptide sequences to their respective genomic locations using data such as GTF and BED files. In this workflow, PepPointer takes the GTF file, which contains gene annotations and genomic coordinates, and combines it with a BED file, which provides chromosomal coordinates of the peptides. By doing this, PepPointer can identify the exact genomic locations of the peptides, linking them to genes and exons. This step is crucial for accurately correlating proteomic data to genomic sequences, enabling a better understanding of the genomic context in which the peptides are found.

> <hands-on-title> PepPointer for SAV-derived peptides </hands-on-title>
>
> 1. {% tool [PepPointer](toolshed.g2.bx.psu.edu/repos/galaxyp/pep_pointer/pep_pointer/0.1.3+galaxy1) %} with the following parameters:
>    - *"Choose the source of the GTF file"*: `From history`
>        - {% icon param-file %} *"GTF file with the genome of interest"*: `Homo_sapiens.GRCh38_canon.106.gtf` (Input dataset)
>    - {% icon param-file %} *"BED file with chromosomal coordinates of peptide"*: `output` (from **Query Tabular - BED file for Assembly**)
>
>
{: .hands_on}


## Visualization and Interpretation of Novel SAV-derived peptides with annotation

In this step, we are using Query Tabular to extract and format relevant information from the results produced by PepPointer. The SQL query in this tool is designed to structure the data into a more readable format, providing key details such as the peptide ID, chromosome location, start and end positions, strand orientation, and any annotations. Additionally, it generates genome coordinates in a format suitable for viewing in genome browsers like IGV and UCSC Genome Browser. This formatting step helps to visualize and interpret the data more effectively by linking the peptides to their genomic context.

> <hands-on-title> Query Tabular </hands-on-title>
>
> 1. {% tool [Query Tabular](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2) %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `classified` (output of **PepPointer - SAV** {% icon tool %})
>    - *"SQL Query to generate tabular output"*:
> ```
> SELECT
> c4 AS Peptide,
> c1 AS Chromosome,
> c2 AS Start,
> c3 AS End,
> c6 AS Strand,
> c7 AS Annotation,
> c1||':'||c2||'-'||c3 AS IGV_Genome_Coordinate,
> 'https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position='||c1||'%3A'||c2||'-'||c3 AS UCSC_Genome_Browser
> FROM  t1
> ```
>    - *"include query result column headers"*: `Yes`
>
>
{: .hands_on}


# B: Database for IEDB
![Variant-annotation-overview-workflow]({% link topics/proteomics/images/neoantigen/PepPointer_Characterization_3.PNG %})

## Generating FASTA for MHC binding tool
This output is an input for the next workflow (HLA Binding Novel Peptides).

> <hands-on-title> Tabular-to-FASTA </hands-on-title>
>
> 1. {% tool [Tabular-to-FASTA](toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fasta/tab2fasta/1.1.1) %} with the following parameters:
>    - {% icon param-file %} *"Tab-delimited file"*: `output` (output of **Query Tabular** {% icon tool %})
>    - *"Title column(s)"*: `2`
>    - *"Sequence column"*: `1`
>
>
{: .hands_on}


# Conclusion

In this tutorial, we have covered the process of annotating neoantigen peptides using the PepPointer tool in a bioinformatics pipeline. The main focus was to guide you through the various steps involved in the analysis, including querying tabular datasets, converting data formats, and using the PepPointer tool to classify peptides based on genomic information.

By the end of this tutorial, you should now be able to:
- Understand the steps involved in preparing and processing peptide data for neoantigen prediction.
- Use tools like Query Tabular, Convert, and PepPointer to manipulate and classify peptide sequences.
- Recognize the importance of annotating neoantigens for immunotherapy research and how bioinformatics tools can facilitate this process.

The key takeaway from this tutorial is the ability to manipulate complex genomic data and generate insights into potential therapeutic targets using bioinformatics techniques. You should now be better equipped to handle similar datasets and apply these techniques to other types of genomic and proteomic research.

# Rerunning on your own data

To rerun this entire analysis at once, you can use our workflow. Below we show how to do this:


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>    - [Neoantigen Variant Annotation](https://tinyurl.com/ipepgen-pep-annot-wf)
>
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *"Genomic-Variant-BED-file"*: `Genomic-Variant-BED-file.tabular`
>    - {% icon param-file %} *"Novel Peptides from PepQuery"*: `Novel_Peptides_from_PepQuery.tabular`
>    - {% icon param-file %} *"Fragpipe Peptide Report"*: `Fragpipe-Peptide-Report.tabular`
>    - {% icon param-file %} *"Annotated-GffCompared-GTFtoBED"*: `GffCompare_Annotated_GTF_to_BED.bed`
>    - {% icon param-file %} *"Human Reference Genome Annotation"*: `Homo_sapiens.GRCh38_canon.106.gtf`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> 
> 
> 
>
> <comment-title>DISCLAIMER</comment-title>
>
>  - If any step in this workflow fails, please ensure that the input files have been correctly generated and formatted by the preceding tools. Workflow failures often result from improperly called or incomplete input data rather than errors in the workflow itself. Users are responsible for verifying their input before troubleshooting workflow issues.
>
> {: .comment}
>
>
>
{: .hands_on}

# Are you feeling adventurous? ✨

## One-Click Neoantigen Workflow

This new [One-Click Neoantigen Workflow](https://usegalaxy.eu/u/galaxyp/w/ipepgen-one-click-workflow) brings together all key modules of the neoantigen discovery process into a single, streamlined analysis within Galaxy. Instead of launching each tutorial separately, users can now execute the entire end-to-end pipeline—from database creation to HLA binding prediction—with just one click —without ever leaving Galaxy.

# Disclaimer

Please note that all the software tools used in this workflow are subject to version updates and changes. As a result, the parameters, functionalities, and outcomes may differ with each new version. We recommend that users verify the specific versions of software tools used to ensure the reproducibility and accuracy of results.

tools_json:
["Query Tabular", "toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.3.2", "Select lines that match an expression", "toolshed.g2.bx.psu.edu/repos/iuc/select/select/1.0.4", "Convert - delimiters to TAB", "Convert characters1", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3", "PepPointer", "toolshed.g2.bx.psu.edu/repos/galaxyp/pep_pointer/pep_pointer/0.1.3+galaxy1", "Cut columns from a table", "toolshed.g2.bx.psu.edu/repos/iuc/cut/cut/1.0.2", "toolshed.g2.bx.psu.edu/repos/bgruening/column_regex_find_replace/column_regex_find_and_replace/1.0.3", "Tabular-to-FASTA", "toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fasta/tab2fasta/1.1.1"]


topic: proteomics
tutorial: clinical-mp-4-quantitation
title: Clinical Metaproteomics 4: Quantitation
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: c177e1b90366ad012fb84648be68fcb3
last_modified: 2026-05-27T09:01:49.454010
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/clinical-mp-4-quantitation/tutorial.html

content:
The next step of the clinical metaproteomics workflow is the quantification workflow. Running a quantification workflow in proteomics is essential for several critical purposes. It allows researchers to measure and compare the abundance of proteins or peptides in biological samples, offering valuable insights into biomarker discovery, comparative analysis, and differential expression studies. Quantitative proteomics helps reveal the functional roles of proteins, the stoichiometry of protein complexes, and the effects of drugs on protein expression in pharmacological studies. Additionally, it serves as a quality control measure, validating initial protein identifications, and providing data normalization for increased accuracy. Quantitative data are indispensable for hypothesis testing, systems biology, and their clinical relevance in areas such as disease diagnosis, prognosis, and therapeutic decision-making. In summary, the quantitation workflow in proteomics is a cornerstone for deciphering the complexities of protein expression and regulation, facilitating a wide array of biological and clinical applications.

In this current workflow, we perform Quantification using the MaxQuant tool and the output will be interpreted in our next module.

![Quantitation workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-quantification.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/Experimental-Design_Discovery_MaxQuant.tabular
>    https://zenodo.org/records/10105821/files/Quantitation_Database_for_MaxQuant.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to input files.
> 6. Create a dataset of the RAW files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
{: .hands_on}


# Import Workflow

> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-4-quantitation/workflows/WF4_Quantitation_Workflow.ga" title="Quantitation Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Quantitation_Database-For-MaxQuant * "*: `Quantitation_Database_for_MaxQuant.fasta`
>    - {% icon param-file %} *" Experimental-Design Discovery MaxQuant"*: `Experimental-Design_Discovery_MaxQuant.tabular`
>    - {% icon param-file %} *" Input Raw-files"*: `RAW dataset collection`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}


# Peptide quantification

In the [Discovery Module](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md), we used MaxQuant to identify peptides for verification. Now, we will again use MaxQuant to further quantify the PepQuery-verified peptides, both microbial and human. More information about quantitation using MaxQuant is available, including [Label-free data analysis](https://gxy.io/GTN:T00218) and [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

The outputs we are most interested in consist of the `MaxQuant Evidence file`, `MaxQuant Protein Groups`, and `MaxQuant Peptides`. The `MaxQuant Peptides` file will allow us to group them to generate a list of quantified microbial peptides.

> <hands-on-title> Quantify verified peptides (from PepQuery2) </hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy4) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-file %} *"FASTA files"*: `Quantitation Database for MaxQuant` (Input dataset)
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template (if needed). For detailed                           instructions see the help text."*: `output` (Input dataset)
>        - *"minimum peptide length"*: `8`
>        - *"Match between runs"*: `Yes`
>        - *"Maximum peptide length for unspecific searches"*: `50`
>    - In *"Protein quantification"*:
>        - *"Use only unmodified peptides"*: `Yes`
>            - *"Modifications used in protein quantification"*: `Oxidation (M)`
>        - In *"LFQ Options"*:
>            - *"iBAQ (calculates absolute protein abundances by normalizing to copy number and not protein mass)"*: `No`
>    - In *"Parameter Group"*:
>        - {% icon param-repeat %} *"Insert Parameter Group"*
>            - {% icon param-collection %} *"Infiles"*: `output` (Input dataset collection)
>            - *"fixed modifications"*: `Carbamidomethyl (C)`
>            - *"variable modifications"*: `Oxidation (M)`
>            - *"enzyme"*: `Trypsin/P`
>            - *"Quantitation Methods"*: `reporter ion MS2`
>                - *"isobaric labeling"*: `TMT11plex`
>                - *"Filter by PIF"*: `Yes`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Peptides` `Evidence` `MSMS`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why can we switch back to using RAW files for MaxQuant, instead of using MGF files?
>
> > <solution-title></solution-title>
> >
> > 1. MaxQuant prefers RAW format compared to MGF as it has more information compared to MGF.
> >
> {: .solution}
>
{: .question}
> <question-title></question-title>
>
> 1. Previously, we used MaxQuant in the Discovery workflow. Why are we using MaxQuant again, instead of Search GUI/PeptideShaker?
>
> > <solution-title></solution-title>
> >
> > 1. We are using MaxQuant for quantification purposes only. SearchGUI Peptide Shaker doesn't have the capability to perform quantification of peptides or proteins.
> >
> {: .solution}
>
{: .question}

## Using Text Manipulation Tools to Manage MaxQuant Outputs

> <hands-on-title> Select microbial protein groups from MaxQuant with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `proteinGroups` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 2. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `peptides` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
>
> 4. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
{: .hands_on}

## Generating a list of quantified proteins and peptides

> <hands-on-title> Group quantified proteins </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}
> <hands-on-title> Group quantified peptides </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


# Conclusion

In summary, the implementation of a quantitation workflow using MaxQuant represents a significant advancement in quantitative proteomic research. This approach enables precise measurement of protein and peptide abundances, enhancing our ability to unravel the complexities of biological systems. This workflow is instrumental in biomarker discovery, comparative analysis, and understanding differential protein expression by offering detailed insights into quantitative changes across different experimental conditions. Its capacity to generate accurate data supports a wide spectrum of applications, including disease research, drug development, and systems biology investigations. Furthermore, the MaxQuant-based quantitation workflow ensures data quality, enabling reliable and reproducible results. It serves as a vital step for quality control, allowing researchers to draw meaningful conclusions from proteomic experiments confidently.

tools_json:
["MaxQuant", "toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy4", "Select", "Grep1", "Cut", "Cut1", "Group", "Grouping1"]


topic: proteomics
tutorial: maxquant-msstats-tmt
title: MaxQuant and MSstats for the analysis of TMT data
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[DDA, TMT]"]
content_hash: f6727cefbba49c4cf813712a764aeca1
last_modified: 2026-05-15T22:28:14.455801
zenodo_link: https://zenodo.org/record/5195800
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/maxquant-msstats-tmt/tutorial.html

content:
In this training we will cover the full analysis workflow from tandem mass tag (TMT) labeled samples using MaxQuant in conjunction with MSstatsTMT. 

The training dataset consists of a dataset from {% cite Baumert_2020 %} which investigates the effects of a recently discovered histone methyltransferase, lysine methyl transferase 9 (KMT9α) in A549 cells. High KMT9α expression in lung cancer correlates with poor patient survival. Therefore,  KMT9α is a very interesting research target in light of the fact that lung cancer survival rates could not be improved over the last 15 years. The example dataset consists of 4 cell culture samples from KMT9α knock-down cells and 4 cell culture samples from control (Ctrl) cells. In order to increase the proteome coverage the TMT experiment was pre-fractionated using offline high pH reverse phase chromatography resulting in 12 MS runs.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

The raw data is available via the PRIDE repository under the ID: [PXD014145](https://www.ebi.ac.uk/pride/archive/projects/PXD014145). The MaxQuant experimental design template file, MSstatsTMT annotation file and FASTA file for this training are deposited at Zenodo(https://zenodo.org/record/5195800). It is of course possible to use another FASTA file with human proteome sequences, but to ensure that the results are compatible, we recommend using the provided FASTA file. MaxQuant not only adds known contaminants to the FASTA file, but also generates the “decoy” hits for false discovery rate estimation itself, therefore the FASTA file is not allowed to have decoy entries. To learn more about FASTA files, have a look at [Protein FASTA Database Handling tutorial]({% link topics/proteomics/tutorials/database-handling/tutorial.md %}).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a meaningful name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the FASTA database, experimental design template, and annotation file [Zenodo](https://zenodo.org/record/5195800)
>
>    ```
>    https://zenodo.org/record/5195800/files/protein_database.fasta
>    https://zenodo.org/record/5195800/files/MaxQuant%20experimental%20design%20template.tabular
>    https://zenodo.org/record/5195800/files/MSstatsTMT%20annotation%20file.tabular
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Once the files are green, rename the fasta file into 'protein database', the experimental design template into 'experimental design template', and the annotation file into 'annotation file'.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. To run MaxQuant, import the raw data from [PRIDE](https://www.ebi.ac.uk/pride/archive/projects/PXD014145) as a 'Collection'.
>    ```
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA380.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA381.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA382.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA383.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA384.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA385.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA386.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA387.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA388.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA389.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA390.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA391.raw
>    ```
>    
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true collection_type="List" collection_name="raw_files" format="thermo.raw" %}
{: .hands_on}


# MaxQuant analysis

We start the MaxQuant run with TMT parameters for MS2 based reporter quantitation. A quality control report will be generated with the [PTXQC functionality](https://pubs.acs.org/doi/10.1021/acs.jproteome.5b00780) that is directly implemented in the MaxQuant Galaxy tool. For more information about MaxQuant and its parameter please have a look at the [Label-free data analysis using MaxQuant tutorial]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-label-free/tutorial.html). To continue with statistical analysis in MSstatsTMT, the Protein Groups and the Evidence files are needed from MaxQuant. The run time of MaxQuant depends on the number and size of the input files and on the chosen parameters. The run of the training datasets will take a while, but the training can be directly continued with the MaxQuant result files from Zenodo. 


> <hands-on-title>MaxQuant analysis</hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy2) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-collection %} *"FASTA files"*: `protein database`
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template"*: `experimental design template` 
>    - In *"Parameter Group"*:
>        - {% icon param-collection %} *"Infiles"*: `MFA380.raw` `MFA381.raw` `MFA382.raw` `MFA383.raw` `MFA384.raw` `MFA385.raw` `MFA386.raw` `MFA387.raw` `MFA388.raw` `MFA389.raw` `MFA390.raw` `MFA391.raw`
>        -  *"variable Modifications"*: `Oxidation (M)` `Acetyl (Protein N-term)`
>        -  *"enzyme"*: `Trypsin/P`
>        -  *"Quantitation Methods"*: `reporter ion MS2`
>            - *"isobaric labeling"*: `TMT11plex`
>            - *"Filter by PIF"*: `Yes`
>    - *"Generate PTXQC (proteomics quality control pipeline) report? (experimental setting)"*: `Yes`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Evidence` `MaxQuant and PTXQC log`
>
>    > <tip-title>Continue with results from Zenodo</tip-title>
>    >
>    > Because the MaxQuant run takes quite a while, we recommend to download the MaxQuant results from Zenodo and continue with the tutorial
>    > 1. Import the files from [Zenodo](https://zenodo.org/record/5195800)
>    >
>    >  ```
>    > https://zenodo.org/record/5195800/files/MaxQuant%20Result%20evidence.tabular
>    > https://zenodo.org/record/5195800/files/MaxQuant%20Result%20protein%20groups.tabular
>    > ```
>    {: .tip}
{: .hands_on}

> <question-title></question-title>
>
> 1. How many protein groups were found with MaxQuant?
>
> > <solution-title></solution-title>
> >
> > 1. 6911 (numbers might slightly vary with different MaxQuant versions); number of lines minus header line in the MaxQuant protein groups output
> >
> {: .solution}
>
{: .question}



> <tip-title>PIF parameter for TMT data</tip-title>
> Co-isolation of multiple peptides can lead to inaccurate TMT signals, as not only one peptide is fragmented but multiple different peptides are fragmented and the TMT signals cannot be distinguished anymore. MaxQuant can “check” the MS1 scans to predict, whether multiple different peptides were cofragmented. Precursor Intensity Filtering (PIF) allows you to implement a cutoff how high the purity of fragmentation was (1 = no coisolation, 0.75 = default).
{: .tip}

> <tip-title>Preparing an MaxQuant experimental design template for your own analysis</tip-title>
> For this tutorial all files needed for the analysis are provided. Nevertheless here are some guidelines for the generation of the experimental design template for MaxQuant for your own data analyses.
> For thermo.raw files, “.raw” is considered a part of the filename. Therefore, file names should be 'Filename1.raw', 'Filename2.raw' etc.. Please pay special attention to your experimental design, since MaxQuant will summarize peptide and protein intensities differently depending on this input. In brief, MaxQuant will summarize the protein intensities within each specified experiment. Often, TMT experiments are not measured in one MS run, but in multiple runs (either as so called fractions, or as replicates). Over the last years, high pH prefractionation has become the standard for fractionating TMT samples. Alternatively, SCX or HILIC fractionations can be employed. Although you fractionated your samples prior to measurement you still want MaxQuant to group all samples which belong to “one” TMT experiment. Since the fractions originate from one sample the experimental design template allows to map multiple measurements to one experiment (= Sample). Additionally, it is possible to have a more complex experimental setup including multiple TMT experiments, possibly containing multiple fractions per experiment. All MS files which are labeled with the same “Experiment” name are summarized in the proteinGroups and peptides output of MaxQuant.
{: .tip}

# MSstatsTMT analysis

The protein groups and evidence files of MaxQuant can directly be used as an input for MSstatsTMT. MSstatsTMT automatically removes all proteins that are labelled as contaminants (‘+’ sign in the column ‘potential contaminant’ of both MaxQuant outputs). In case you want to prevent this step either run MaxQuant with "Include Contaminants: No" or remove the '+' sign from the MaxQuant files as described in the [label-free MSstats training]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-msstats-dda-lfq/tutorial.html). Additionally, MSstatsTMT will combine all quantitations according to the annotation file, which you have to provide (e.g all fractions of the same mixture are combined).
For this dataset, the standard parameters of MSstatsTMT fit quite nicely, as this is a straightforward two group comparison. If you have more than two groups, you may consider using the comparison matrix option of MSstatsTMT (for more information see [label-free MSstats training]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-msstats-dda-lfq/tutorial.html)).


> <hands-on-title>MSstatsTMT analysis</hands-on-title>
>
> 1. {% tool [MSstatsTMT](toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy0) %} with the following parameters:
>    - *"Input Source"*: `MaxQuant`
>        - {% icon param-file %} *"evidence.txt - feature-level data"*: `evidence` (output of **MaxQuant** {% icon tool %})
>        - {% icon param-file %} *"proteinGroups.txt"*: `proteinGroups` (output of **MaxQuant** {% icon tool %})
>        - {% icon param-file %} *"annotation.txt"*: `annotation file`
>    - In *"Plot Output Options"*:
>        - *"Select protein IDs to draw plots"*: `Option for QC plot: "allonly" will generate one QC plot with all proteins`
>    - *"Compare Groups"*: `Yes`
>        - In *"Comparison Plot Options"*:
>            - *"Display protein names in Volcano Plot."*: `No`
>    - *"Select Outputs"*: `MSstatsTMT summarization log` `Protein Abundance` `QC Plot`
{: .hands_on}

> <tip-title>Waiting for MSstatsTMT result</tip-title>
> This training consists of a real proteomics experiments. Therefore, calculation times take longer than in usual Galaxy tutorials. You may want to come back later to the training or use the time to prepare your own data for the analysis with MaxQuant and MSstatsTMT with the tips provided in this training (yellow boxes).
{: .tip}

> <question-title></question-title>
>
> 1. How many differentially regulated proteins were detected in the MSstatsTMT group comparison? 
>
> > <solution-title></solution-title>
> >
> > 1. 7139 (numbers might slightly vary with different MSstatsTMT versions); number of lines minus header line in the MSstatsTMT group comparison output
> >
> {: .solution}
>
{: .question}

> <tip-title>Preparing an MSstatsTMT annotation file for your own analysis</tip-title>
> 
> For this tutorial all files needed for the analysis are provided. Nevertheless here are some guidelines for the generation of annotation file for MSstatsTMT for your own data analyses.
> As MSstatsTMT has to perform statistical analyses, you have to provide the following basic information: Which quantitation channel in every sample corresponds to which original biological sample and which biological condition (e.g WT or knock-down) does this sample belong to? To fully automate the process of choosing an appropriate statistical model for analysis, MSstatsTMT needs an annotation matrix that is extremely powerful because it can deal with any experimental setup. 
> 
> The annotation file contains the following columns:
> - **Run** corresponds to your MS file names. Please note that MaxQuant in Galaxy will add a “.thermo” to your raw file name as it appears in your history. Since we first analyse the data with MaxQuant, MSstatsTMT needs this additional extension for this annotation file. The correct run names can be found after MaxQuant is finished in the evidence file in the column ‘Raw file’.
> - **Fraction** corresponds to the aforementioned optional pre-fractionation fraction of your samples (e.g high pH RP, or SCX)
> - **TechRepMixture** corresponds to the n-th time this fraction was measured. If you measured the first Fraction of your first sample twice, Fraction would stay the same, but the TechRepMixture would have to differ.
> - **Channel** corresponds to the TMT channel which was used in your Experiments.We recommend to always include all channels in your searches and your annotations, even when these channels are empty. So if you only used 9 of 11 channels in your TMT experiments, add the last 2 channels anyway and just label them as “Empty” in the Condition column. In Galaxy the channel names need to be named ‘channel.X’, with X being the number of the channel.
> - **Condition** represents the biological condition, which the observation belongs to (e.g Ctrl). Normalization Channel are labeled as “Norm”. Empty Channels as “Empty”
> - **Mixture** corresponds to the “TMT experiment” mentioned above.
> - **BioReplicate** corresponds to the biological specimen this observation derives from.
> On first glance this might sound overwhelming, but used correctly, this annotation matrix can describe very complex experimental setups in a definitive way.
> 
> An example: 
>
> Consider you perform a study on the difference between primary liver tumors and liver tumor metastases. It makes a very big difference whether all your samples derive from different patients, or whether you were able to obtain some primary tumors and metastases samples from the same patient. The samples obtained from the same patient are expected to be related to each other (they are not independent observations and this must be considered during statistical testing).
> A short illustration: 
> You measured 2 different samples of one patient (metastasis and primary tumor sample).This patient would be considered patient 1 and you would label the primary tumor with the first channel of a TMT-11 plex kit (126) and you would label the metastasis with the second channel (127N). After labeling you combine both samples in one Mixture and measure them in one MS run.
>
> Your annotation file for this case would look like this:
>
> | Run                | Fraction | TechRepMixture | Channel | Condition | Mixture | BioReplicate |
> |--------------------|----------|----------------|---------|-----------|---------|--------------|
> | File1.thermo.raw   | 1        | 1              |channel.1|primary    |mixture1 |1             |
> | File1.thermo.raw   | 1        | 1              |channel.2|metastasis |mixture1 |1             |
>
> 
> Let us assume you now have 10 primary tumors and 10 metastases (from 20 different patients). Since you do not have 20plex capabilities, you have to divide your samples into two experiments / mixtures. For simplicity, after labeling and combining all channels belonging to the same experiments, the mixtures are directly measured ( 1 fraction per Mixture). To avoid batch effects due to allocation you use randomization. In order to be able to normalize all quantitations you use a reference sample, which you add to each experiment (this is referred to as “Normalization Channel”). The reference sample is ideally a mixture of all your samples. 
>
> The annotation matrix would now look like this:
>
> | Run                | Fraction | TechRepMixture | Channel  | Condition | Mixture | BioReplicate |
> |--------------------|----------|----------------|----------|-----------|---------|--------------|
> | File1.thermo.raw   | 1        | 1              |channel.1 |primary    |mixture1 |1             |
> | File1.thermo.raw   | 1        | 1              |channel.2 |metastasis |mixture1 |2             |
> | File1.thermo.raw   | 1        | 1              |channel.3 |primary    |mixture1 |3             |
> | File1.thermo.raw   | 1        | 1              |channel.4 |metastasis |mixture1 |4             |
> | File1.thermo.raw   | 1        | 1              |channel.5 |primary    |mixture1 |5             |
> | File1.thermo.raw   | 1        | 1              |channel.6 |metastasis |mixture1 |6             |
> | File1.thermo.raw   | 1        | 1              |channel.7 |primary    |mixture1 |7             |
> | File1.thermo.raw   | 1        | 1              |channel.8 |metastasis |mixture1 |8             |
> | File1.thermo.raw   | 1        | 1              |channel.9 |primary    |mixture1 |9             |
> | File1.thermo.raw   | 1        | 1              |channel.10|metastasis |mixture1 |10            |
> | File1.thermo.raw   | 1        | 1              |channel.11|Norm       |mixture1 |100           |
> | File2.thermo.raw   | 1        | 1              |channel.1 |primary    |mixture2 |1             |
> | File2.thermo.raw   | 1        | 1              |channel.2 |metastasis |mixture2 |2             |
> | File2.thermo.raw   | 1        | 1              |channel.3 |primary    |mixture2 |3             |
> | File2.thermo.raw   | 1        | 1              |channel.4 |metastasis |mixture2 |4             |
> | File2.thermo.raw   | 1        | 1              |channel.5 |primary    |mixture2 |5             |
> | File2.thermo.raw   | 1        | 1              |channel.6 |metastasis |mixture2 |6             |
> | File2.thermo.raw   | 1        | 1              |channel.7 |primary    |mixture2 |7             |
> | File2.thermo.raw   | 1        | 1              |channel.8 |metastasis |mixture2 |8             |
> | File2.thermo.raw   | 1        | 1              |channel.9 |primary    |mixture2 |9             |
> | File2.thermo.raw   | 1        | 1              |channel.10|metastasis |mixture2 |10            |
> | File2.thermo.raw   | 1        | 1              |channel.11|Norm       |mixture2 |100           |
>
>
> Normalization Channels are always labeled as “Norm” in Condition and get an arbitrary BioReplicate number. Since we want the Normalization channel to be a reference, it always has to be the same sample and therefore also should have the same BioReplicate number.
> If you have empty channels, label them as “Empty” in the Condition column and also give them an arbitrary BioReplicate number.
> 
> Exercise: 
>
> How would the annotation file look like for the following experiment:
> 27 samples (12 primary, 12 metastases samples from different patients + 3 reference samples) over 3 TMT experiments / mixtures. 
> You use Channels 1-8 in Exp1, 2-9 in Exp2 and 3-11 in Exp3. Your first used channel is always taken for the reference channel (Exp1: Channel1 = Norm, Exp2: Channel2 = Norm etc). 
> You fractionate all 3 experiments / mixtures into 3 fractions each and measure every MS sample twice (resulting in 18 measurements / raw files total). 
>
> The solution can be found on [Zenodo](https://zenodo.org/record/5195800/files/annotation_exercise.txt).
{: .tip}

For more information on MSstats parameters and ideas on how to follow up with your MSstatsTMT results visit the [label-free MSstats training]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-msstats-dda-lfq/tutorial.html). 

# Follow up on MSstatsTMT results

We obtain several output files from MSstatsTMT. MSstats log file contains the MSstats report with warnings and information about the analysis steps. 
The QC report allows to visualize protein abundance of conditions for all proteins (first page) or if enabled for each individual protein (following pages). 

![QC report](../../images/maxquant-msstats-tmt/QC_plot.png "QC report for all samples, each boxplot summarizes the protein abundances of one sample")

The MSstatsTMT Group Comparison Plot is a very convenient pdf, which you can search for your favorite proteins of interest to check whether these are dysregulated between the conditions. The red error bar gives you a confidence interval of the fold change.

![Group comparison plot](../../images/maxquant-msstats-tmt/example_protein.png "Group comparison plot for the protein A5YKK6 (CCR4-NOT transcription complex subunit 1)")

The volcano plot plots the negative log10 of the adjusted P Value derived from statistical testing over the log2 fold change between the conditions you compared. A value of 3 on the y axis corresponds to an adjusted P Value of 0.001, whereas a value of 1 on the x axis corresponds to a ratio of 2 meaning that the protein is twice as high in condition 1 than condition 2..
Please bear in mind that TMT quantitation suffers from ratio compression. There are ways to calculate more accurate fold changes from measured data (or alternatively you can use MS3 quantitation on some machines). However, even with ratio compression, MS2 TMT is an excellent quantitation strategy to detect differentially regulated proteins. Just bear in mind that the reported fold chances of proteins are almost always smaller than the real fold changes between your conditions.

![Volcano plot](../../images/maxquant-msstats-tmt/volcano_plot.png "Volcano plot QC report for all samples, each boxplot summarizes the protein abundances of one sample")

tools_json:
["MaxQuant", "toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy2", "MSstatsTMT", "toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy0"]


topic: proteomics
tutorial: clinical-mp-1-database-generation
title: Clinical Metaproteomics 1: Database-Generation
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 113e6a7484bc258c821fcfc28ea5ed66
last_modified: 2026-05-27T09:01:49.453643
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/clinical-mp-1-database-generation/tutorial.html

content:
Metaproteomics is the large-scale characterization of the entire complement of proteins expressed by microbiota. However, metaproteomics analysis of clinical samples is challenged by the presence of abundant human (host) proteins which hampers the confident detection of lower abundant microbial proteins {% cite Batut2018 %} ; {% cite Jagtap2015 %} .

To address this, we used tandem mass spectrometry (MS/MS) and bioinformatics tools on the Galaxy platform to develop a metaproteomics workflow to characterize the metaproteomes of clinical samples. This clinical metaproteomics workflow holds potential for general clinical applications such as potential secondary infections during COVID-19 infection, microbiome changes during cystic fibrosis as well as broad research questions regarding host-microbe interactions.

![Clinical Metaproteomics workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-overview.JPG %})


The first workflow for the clinical metaproteomics data analysis is the Database generation workflow. The Galaxy-P team has developed a workflow wherein a large database is generated by downloading protein sequences of known disease-causing microorganisms and then generating a compact database from the comprehensive database using the Metanovo tool.


![Database Generation Workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-database-generation.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Upload

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/HUMAN_SwissProt_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/Species_UniProt_FASTA.fasta
>    https://zenodo.org/records/10105821/files/Contaminants_(cRAP)_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.mgf
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
>
> 5. Optional-Add to each database a tag corresponding to the file name.
> 6. Create a dataset collection of the 4 MGF datasets.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-1-database-generation/workflows/WF1_Database_Generation_Workflow.ga" title="Database Generation" %}
>
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Input Dataset collection"*: `MGF dataset collection`
>    - {% icon param-file %} *" Species_tabular"*: `Species_tabular.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

# Step-by-step analysis

## Download Protein Sequences using taxon names
First, we want to generate a large comprehensive protein sequence database using the UniProt XML Downloader to extract sequences for species of interest. To do so, you will need a tabular file that contains a list of species.

For this tutorial, a literature survey was conducted to obtain 118 taxonomic species of organisms that are commonly associated with the female reproductive tract {% cite AfiuniZadeh2018 %}. This species list was used to generate a protein sequence FASTA database was generated using the UniProt XML Downloader tool within the Galaxy framework. In this tutorial, the Species FASTA database (~3.38 million sequences) has already been provided as input. However, if you have your own list of species of interest as a tabular file (`Your_Species_tabular.tabular`), steps to generate a FASTA file from a tabular file are included:


## Download Species Protein Sequences using UniProt XML downloader with **UniProt**

> <hands-on-title> UniProt XML downloader</hands-on-title>
> 1. {% tool [UniProt](toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.3.0) %} with the following parameters:
>    - *"Select"*: `Your_Species_tabular.tabular`
>        - {% icon param-file %} *"Dataset (tab separated) with Taxon ID/Name column"*: `output` (Input dataset)
>        - *"Column with Taxon ID/name"*: `c1`
>    - *"UniProt output format"*: `fasta`
> 2. Rename the output as Species_UniProt_FASTA.fasta
>
>    > <comment-title> UniProt description </comment-title>
>    >
>    > This tool will help download the protein fasta sequences by inputting the taxon names.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Can we use a higher taxonomy clade than species for the UniProt XML downloader?
> 2. Why are we using the tools separately? Can we run it all together?
> 3. Can we select multiple files together?
> 4. How many FASTA files can be merged at once, i.e. is there a limit on the number/size of files?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Yes, the UniProt XML downloader can also be used for generating a database from Genus, Family, Order, or any other higher taxonomy clade.
> > 2. The tools are run separately to reduce the load on the server and tool. If you have a limited number of taxon names, then you can run it all together.
> > 3. Yes, that certainly can be done. We used one input file at a time to maintain the order of sequences in the database.
> > 4. There is no limit.
> >
> {: .solution}
>
{: .question}


## Merging databases to obtain a large comprehensive database for MetaNovo
Once generated, the Species UniProt database (~3.38 million sequences) will be merged with the Human SwissProt database (reviewed only; ~20.4K sequences) and contaminant (cRAP) sequences database (116 sequences) and filtered to generate the large comprehensive database (~2.59 million sequences). The large comprehensive database will be used to generate a compact database using MetaNovo, which is much more manageable.

> <hands-on-title> Download contaminants with **Protein Database Downloader** </hands-on-title>
>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `cRAP (contaminants)`
> 2. Rename as "Protein Database Contaminants (cRAP)"
>
{: .hands_on}

> <hands-on-title> Human SwissProt (reviewed) database</hands-on-title>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `UniProtKB(reviewed only)`
>        - In *"Taxonomy"*: `Homo sapiens (Human)`
>        - In *"reviewed"*: `UniProtKB/Swiss-Prot (reviewed only)`
>	 - In  *"Proteome Set"*: `Reference Proteome Set`
>	 - In  *"Include isoform data"*: `False`
> 2. Rename as "Protein Database Human SwissProt".
{: .hands_on}

> <question-title></question-title>
>
> 1. How often is the Protein Database Downloader updated?
>
>
> > <solution-title></solution-title>
> >
> >
> > 1. It is updated every 3 months.
> >
> {: .solution}
>
{: .question}


> <hands-on-title> FASTA Merge Files and Filter Unique Sequences </hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Species_UniProt_FASTA ` (output of **UniProt XML downloader** {% icon tool %})
>
> 2. Rename out as "Human UniProt Microbial Proteins cRAP for MetaNovo".
>
{: .hands_on}

# Reducing Database size

## Metanovo tool generates a compact database from your comprehensive database with **MetaNovo**
Next, the large comprehensive database of ~2.59 million sequences can be reduced using the MetaNovo tool to generate a more manageable database that contains identified proteins.

The compact MetaNovo-generated database (~1.9K sequences) will be merged with Human SwissProt (reviewed only) and contaminants (cRAP) databases to generate the reduced database (~21.2k protein sequences) that will be used for peptide identification (see [Discovery Module tutorial](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md)).

> <hands-on-title> MetaNovo </hands-on-title>
>
> 1. {% tool [MetaNovo](toolshed.g2.bx.psu.edu/repos/galaxyp/metanovo/metanovo/1.9.4+galaxy4) %} with the following parameters:
>    - *"MGF Input Type"*: `Collection`
>        - {% icon param-collection %} *"MGF Collection"*: `output` (Input dataset collection)
>    - {% icon param-file %} *"FASTA File"*: `output` (output of **FASTA Merge Files and Filter Unique Sequences** {% icon tool %})
>    - In *"Spectrum Matching Parameters"*:
>        - *"Fragment ion mass tolerance"*: `0.01`
>        - *"Enzyme"*: `Trypsin (no P rule)`
>        - *"Fixed modifications as comma separated list"*: `Carbamidomethylation of C` `TMT 10-plex of K` `TMT 10-plex of peptide N-term`
>        - *"Variable modifications as comma separated list"*: `Oxidation of M`
>        - *"Maximal charge to search for"*: `5`
>    - In *"Import Filters"*:
>        - *"The maximal peptide length to consider when importing identification files"*: `50`
> 2. Rename as "MetaNovo Compact Database".
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are we reducing the size of the database?
> 2. Why is this running TMT10 plex modification when the data is 11-plex?
> 3. Regarding MetaNovo Spectrum Matching parameters, what are the most “important” parameters? Meaning, that if a user wants to reduce or increase the sensitivity/number of output sequences, what should they change?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Reducing the size of the database improves search speed, FDR, and sensitivity.
> > 2. There is no option for 11-plex modifications in Metanovo, hence we use the TMT-10plex.
> > 3. The most important parameters are the tolerance (MS1 and MS2) and any modifications introduced during the processing of the data.
> >
> {: .solution}
>
{: .question}

## Merging databases to obtain reduced MetaNovo database for peptide discovery with **FASTA Merge Files and Filter Unique Sequences**

> <hands-on-title> FASTA Merge Files and Filter Unique Sequences</hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `MetaNovo Compact Database` (output of **MetaNovo** {% icon tool %})
>
>
{: .hands_on}

# Conclusion

The first step for the Clinical Metaproteomics study is database generation. As we didn’t have a reference database or information from 16srRNA-seq data, we generated a fasta database doing a literature survey, however, if 16S rRNA data is present, the taxon identified can be used for a customized database generation. As the size of the comprehensive database is generally  too large, we used the Metanovo tool to reduce the size of the database. This reduced database will be then used for clinical metaproteomics discovery workflow.

tools_json:
["UniProt", "toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.3.0", "Protein Database Downloader", "toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4", "FASTA Merge Files and Filter Unique Sequences", "toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0", "MetaNovo", "toolshed.g2.bx.psu.edu/repos/galaxyp/metanovo/metanovo/1.9.4+galaxy4"]


topic: single-cell
tutorial: EBI-retrieval
title: Importing files from public atlases
difficulty: intermediate
time_estimation: 15m
requirements_json: []
tags_json: []
content_hash: 4cffa72a6dd46a9ca047e37fc3313019
last_modified: 2026-05-27T09:01:49.736762
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/EBI-retrieval/tutorial.html

content:
anndata-scanpy-or-seurat-object-1
    date: 2024-01-10
  - label: "EBI SCXA retrieval tool (UseGalaxy.eu)-ARCHIVED"
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/ebi-scxa---anndata-scanpy-or-seurat-object
    date: 2024-12-10
  - label: "HCA Downloader (UseGalaxy.eu)"
    history: https://singlecell.usegalaxy.eu/u/j.jakiela/h/hca-downloader---example-files-1
    date: 2024-09-08
  - label: "HCA Downloader (UseGalaxy.eu)-ARCHIVED"
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/hca-downloader---example-files
    date: 2024-12-10

input_histories:
  - label: "UseGalaxy.eu"
    history: https://singlecell.usegalaxy.eu/u/j.jakiela/h/input-history-ebi-scxa---anndata-scanpy-or-seurat-object
  - label: "UseGalaxy.eu-ARCHIVED"
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/input-history-ebi-scxa---anndata-scanpy-or-seurat-object
    date: 2024-12-10


requirements:
  -
    type: "internal"
    topic_name: single-cell
    tutorials:
      - scrna-case_alevin
      - scrna-case_alevin-combine-datasets

follow_up_training:
  -
    type: "internal"
    topic_name: single-cell
    tutorials:
        - scrna-case_basic-pipeline

recordings:
  - speakers:
    - hexhowells
    date: '2024-09-12'
    length: 18M54S
    youtube_id: "Yx7m7Uh4tVg"

tags:
  - data import
  - data management

---

Public single cell datasets seem to accumulate by the second. Well annotated, quality datasets are slightly trickier to find, which is why projects like the [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home) (SCXA) exist - to curate datasets for public use. Here, we will guide you through transforming data imported from the SCXA repository into the input file required for the [Filter, Plot, Explore tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) and we will also show how to use the public atlases for your own research.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Getting data from the Single Cell Expression Atlas

Galaxy has a specific tool for importing data from the SCXA ({% cite Moreno2020.04.08.032698 %}), which combines all the preprocessing steps shown in [the corresponding tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}) into one! For this tutorial, the dataset can be seen [at the EBI](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/downloads) with experiment ID of `E-MTAB-6945`.

> <details-title> How to access other experiments? </details-title>
>
> You can search datasets according to various criteria either using search box in **Home** tab or choosing kingdom, experiment collection, technology type (and others) in **Browse experiments** tab. When you find the experiment you are interested in, just click on it and the experiment ID will be displayed in the website URL, as shown below.
>
> ![Arrow pointing to the website URL where you can find experiment ID.](../../images/scrna-data/exp_id.jpg "Where to find experiment ID on the EBI Single Cell Expression Atlas website.")
>
> Once you know the experiment ID, you can use EBI SCXA Data Retrieval tool in Galaxy!
>
{: .details}


> <hands-on-title>Retrieving data from Single Cell Expression Atlas</hands-on-title>
>
> 1. {% tool [EBI SCXA Data Retrieval](toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2) %} with the following parameters:
>      - *"SC-Atlas experiment accession"*: `E-MTAB-6945`
>      - *"Choose the type of matrix to download"*: `Raw filtered counts`
>
{: .hands_on}

It's important to note that this matrix is processed somewhat through the SCXA pipeline, which is quite similar to the pre-processing that has been shown in this case study tutorial series. The resultant datasets contain any and all metadata provided by the SCXA pipeline as well as the metadata contributed by the original authors (for instance, more cell or gene annotations). So while the AnnData object generated at the end of this tutorial will be similar to that generated using the [Alevin workflows]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}) on the original FASTQ files, some of the metadata will be slightly different. Relevant results and interpretation will not change, however!

## Examine the imported files

> <question-title></question-title>
>
> 1. What format has this tool imported?
>
> > <tip-title></tip-title>
> >
> > Selecting the title of each resultant dataset will expand the dataset in the Galaxy history.
> {: .tip}
> > <solution-title></solution-title>
> >
> > Matrix Market Format! We can tell this because our first file helpfully says `MatrixMarket` in the first line.
> > ![Green box containing first output, the matrix.mtx file. Columns are labelled 14458, 5218, and 5308559](../../tutorials/EBI-retrieval/images/matrix-output.png "Matrix Market Output")
> >
> > This {% icon param-file %} `matrix.mtx` file, in Matrix Market format, contains a column referring to each gene (column 1), to each cell (column 2), and the expression values themselves in the final column. To be useful, then, we need to know which genes and cells the numbers are referring to. That's why this format comes with two more files.
> >
> > ![Green box containing second output, the genes.tsv file. The first column contains EnsemblIDs such as ENSMUSG######, while the second column contains gene names. There are 14,457 lines.](../../tutorials/EBI-retrieval/images/genes-output.png "Genes Output")
> >
> > The {% icon param-file %} `genes.tsv` file lists each EnsemblID and its gene name. The lines (14,457) corresponds with the 14458 in the Matrix file...but the 14458 contains a header, so that's why it has one more than the genes file!
> >
> > ![Green box containing third output, the barcodes.tsv file. The file consists of 5,217 lines and a single column containing the cell barcode, variations of ERR2704656-AAAACACTCTGA.](../../tutorials/EBI-retrieval/images/cells-output.png "Cells Output")
> >
> > The {% icon param-file %} `barcodes.tsv` file lists each barcode. The lines (5,217) again correspond with the 5,218 lines in the Matrix file...which adds in the header again!
> >
> > ![Green box containing fourth output, the exp_design.tsv file. The file consists of 5,218 lines and numerous columns starting with 'Assay' and 'Sample Characteristic'.](../../tutorials/EBI-retrieval/images/cell-metadata.png "Experimental Design")
> >
> > Finally, and helpfully, the tool also includes cell metadata where the `Assay` column corresponds with the barcodes in the {% icon param-file %} `barcodes.tsv` file. While this is not a required file to create an AnnData object from the three matrix market files, it is extremely necessary for actually interpreting the data. Imagine not knowing which barcodes came from which sample!
> >
> {: .solution}
{: .question}


## Metadata manipulation

At this point you might want to do some modifications in the files before downstream analysis. That can include re-formating the cell metadata or changing the names of the column headers, it all depends on your dataset and how you want to perfrom your analysis. It's also fine to transform those files straight away. Here, we will show an extended version of metadata manipulation which allows us to create an input file consistent with [the next tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) workflow.

Before creating an AnnData object, we need to make a small modification in experimental design table. The dataset contains information about the 7 experimental samples (N701 – N707). However, in the {% icon param-file %} `exp_design.tsv` dataset, which contains the cell metadata, these samples are just numbered from 1 to 7.

> <details-title>How did we know this?</details-title>
>
> You can preview this column in the the {% icon param-file %} `exp_design.tsv` dataset by selecting the {% icon galaxy-eye %} in the {% icon galaxy-history %} Galaxy history. If you scroll to the right, and move to the column `Sample Characteristic[individual]`, you will find the batch information. Don't worry, we're about to rename and reformat this whole dataset to more useful titles. Make a note of the number of that column - number 12 - as we will need it to change the batch number to a batch name shortly.
>
{: .details}

The plotting tool that we will use later will fail if the entries are integers and not categorical values, so we will change `1` to `N01` and so on.

> <hands-on-title> Change batch numbers into names </hands-on-title>
>
> 1. Change the datatype of {% icon param-file %} `EBI SCXA Data Retrieval on E-MTAB-6945 exp_design.tsv` to `tabular`:
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 2. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `EBI SCXA Data Retrieval on E-MTAB-6945 exp_design.tsv`
>    - *"using column"*: `Column: 12`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `1`
>            - *"Replacement"*: `N01`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `2`
>            - *"Replacement"*: `N02`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `3`
>            - *"Replacement"*: `N03`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `4`
>            - *"Replacement"*: `N04`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `5`
>            - *"Replacement"*: `N05`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `6`
>            - *"Replacement"*: `N06`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `7`
>            - *"Replacement"*: `N07`
>
{: .hands_on}

While we're renaming things, let's also fix our titles.

> <hands-on-title> Change cell metadata titles </hands-on-title>
>
> 1. {% tool [Replace parts of text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: output from **Column Regex and Replace** {% icon tool %}
>    - In *"Find and Replace"*:
>        - *"Find pattern"*: `"Sample Characteristic[genotype]"`
>        - *"Replace with"*: `genotype`
>    - {% icon param-repeat %} *"Insert Find and Replace"*
>        - *"Find pattern"*: `"Sample Characteristic[individual]"`
>        - *"Replace with"*: `batch`
>    - {% icon param-repeat %} *"Insert Find and Replace"*
>        - *"Find pattern"*: `"Sample Characteristic[sex]"`
>        - *"Replace with"*: `sex`
>    - {% icon param-repeat %} *"Insert Find and Replace"*
>        - *"Find pattern"*: `"Sample Characteristic[cell type]"`
>        - *"Replace with"*: `cell_type`
> 2. Rename {% icon galaxy-pencil %} output `Cell metadata`
{: .hands_on}

## Check mitochondrial gene name format

We might like to flag mitochondrial genes. They can be identified quite easily since - depending on the species and formatting convention - their names often start with `mt`. Since tools for flagging mitochondrial genes are often case-sensitive, it might be a good idea to check the formatting of the mitochondrial genes in our dataset.

> <hands-on-title> Check the format of mitochondrial genes names </hands-on-title>
>
> 1. {% tool [Search in textfiles](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %}  *"Select lines from"*: `EBI SCXA Data Retrieval on E-MTAB-6945 genes.tsv (Raw filtered counts)`
>    - *"that"*: `Match`
>    - *"Regular Expression"*: `mt`
>    - *"Match type"*: `case insensitive`
>    - *"Output"*: `Highlighted HTML (for easier viewing)`
> 3. Rename {% icon galaxy-pencil %} output `Mito genes check`
>
{: .hands_on}

If you click on that dataset, you will see all the genes containing `mt` in their name. We can now clearly see that mitochondrial genes in our dataset start with `mt-`. Keep that in mind, we might use it in a moment!

Now we can create our single cell object!

{% include _includes/cyoa-choices.html option1="Scanpy" option2="Seurat" default="Scanpy"
       text="You can choose whether you want to create an AnnData object for Scanpy Analysis or an RDS object for Seurat Analysis. Galaxy has more resources for Scanpy analysis, but sometimes Seurat might have what you want. The two packages are constantly trying to outdo the other! It often depends on what is more 'standard' in your work environment!" %}

 <div class="Scanpy" markdown="1">


## Creating the AnnData object

We will do several modifications within the AnnData object so that you can follow [the next tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}).

> <hands-on-title> Create the AnnData Object </hands-on-title>
>
> 1. {% tool [Scanpy Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.9.3+galaxy0) %}
> 2. Make sure you are using version **1.8.1+galaxy9** of the tool (change by clicking on {% icon tool-versions %} Versions button):
>   ![List of available tool versions shown when clicking on the 'Versions' button on the top of the page.](../../images/scrna-casestudy/version.png "How to change the version of the tool")
>
> 3. Set the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `EBI SCXA Data Retrieval on E-MTAB-6945 matrix.mtx (Raw filtered counts)`
>    - *"Gene table"*:  `EBI SCXA Data Retrieval on E-MTAB-6945 genes.tsv (Raw filtered counts)`
>    - *"Barcode/cell table"*: `EBI SCXA Data Retrieval on E-MTAB-6945 barcodes.tsv (Raw filtered counts)`
>    - *"Cell metadata table"*: `Cell metadata`
> 4. Rename {% icon galaxy-pencil %} output `AnnData object`
>
{: .hands_on}

## AnnData manipulation

We will now change the header of the column containing gene names from `gene_symbols` to `Symbol`. This edit is only needed to make our AnnData object compatible with this tutorial's workflow. We will also flag the mitochondrial genes.

And the good news is that we can do both those steps using only one tool!

> <hands-on-title> Modify AnnData object </hands-on-title>
>
> 1. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0) %}
> 2. Make sure you are using version **1.8.1+galaxy92** of the tool (change by clicking on {% icon tool-versions %} Versions button)
> 3. Set the following parameters:
>    - {% icon param-file %} In *"Input object in hdf5 AnnData format"*: `AnnData object`
>    - In *"Change field names in AnnData var"*:
>        - {% icon param-repeat %} *"Insert Change field names in AnnData var"*
>            - *"Original name"*: `gene_symbols`
>            - *"New name"*: `Symbol`
>    - *"Gene symbols field in AnnData"*: `Symbol`
>    - In *"Flag genes that start with these names"*:
>        - {% icon param-repeat %} *"Insert Flag genes that start with these names"*
>            - *"Starts with"*: `mt-`
>            - *"Var name"*: `mito`
> 4. Rename {% icon galaxy-pencil %} output `Mito-counted AnnData for downstream analysis`
>
{: .hands_on}

And that's all! What's even more exciting about the {% icon tool %} **AnnData Operations** tool is that it automatically calculates a bunch of metrics, such as `log1p_mean_counts`, `log1p_total_counts`, `mean_counts`, `n_cells`, `n_cells_by_counts`, `n_counts`, `pct_dropout_by_counts`, and `total_counts`. Amazing, isn't it?

## Conclusion
Now you can use this object as input for the [Filter, Plot, Explore tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) and its associated workflow!

Even though this tutorial was designed specifically to modify the AnnData object to be compatible with the subsequent tutorial, it also shows useful tools that you can use for your own, independent data analysis. You can find the [workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/workflow-constructed-from-history-ebi-workflow) and the [answer key history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/ebi-scxa---anndata-scanpy-or-seurat-object-1). However, if you want to use the workflow from this tutorial, you have to keep in mind that different datasets may have different column names. So you have to check them first, and only then you can modify them.

</div>

<div class="Seurat" markdown="1">

## Creating the Seurat Object

> <hands-on-title> Create the Seurat Object </hands-on-title>
>
> 1. {% tool [Seurat Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/4.0.4+galaxy0) %}
> 2. Set the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `EBI SCXA Data Retrieval on E-MTAB-6945 matrix.mtx (Raw filtered counts)`
>    - *"Gene table"*:  `EBI SCXA Data Retrieval on E-MTAB-6945 genes.tsv (Raw filtered counts)`
>    - *"Barcode/cell table"*: `EBI SCXA Data Retrieval on E-MTAB-6945 barcodes.tsv (Raw filtered counts)`
>    - *"Cell metadata"*: `Cell metadata`
> 3. Rename {% icon galaxy-pencil %} output `Seurat object`
>
> You can also choose if you want to create Seurat object, Loom or Single Cell Experiment by selecting your option in *"Choose the format of the output"*.
>
{: .hands_on}

## Conclusion
And you're there! You now have a usable Seurat object for analysis with Seurat tools in your history! {% icon congratulations %} Congrats!

</div>


<!---
HCA doesn't work well for other datasets for now...
https://github.com/galaxyproject/training-material/issues/4567
-->

# Human Cell Atlas Matrix Downloader

Another public atlas that you can use to access the datasets is [Human Cell Atlas data portal](https://data.humancellatlas.org/). We will show you the tool in Galaxy which allows to retrieve expression matrices and metadata for any public experiment available in that repository.

To use it, simply set the project title, project label or project UUID, which can be found at the [HCA data browser](https://data.humancellatlas.org/explore/projects), and select the desired matrix format (Matrix Market or Loom).

![Image showing project UUID as a final fragment of link address, project title (self-explanatory) and project label as an entry in the box on the right side of the page.](../../images/scrna-data/HCA.jpg "Where to find project title, project label and project UUID")

For projects that have more than one organism, one needs to be specified. Otherwise, there is no need to set the species.

Let's use the suggested example of the project *Single cell transcriptome analysis of human pancreas*. If you check this project in HCA, you'll find out that it's actually its label. But it should work well if you enter the title or UUID!

> <hands-on-title>Create AnnData object</hands-on-title>
>
> {% tool [Human Cell Atlas Matrix Downloader](toolshed.g2.bx.psu.edu/repos/ebi-gxa/hca_matrix_downloader/hca_matrix_downloader/v0.0.4+galaxy0) %} with the following parameters:
>    - *"Human Cell Atlas project name/label/UUID"*: `Single cell transcriptome analysis of human pancreas`
>    - *"Choose the format of matrix to download"*:  `Matrix Market`
{: .hands_on}

> <warning-title>Errors that you might encounter</warning-title>
>  If your dataset turns red, there might be several reasons for that, for example:
>    - "There are too many connected users" - please be patient and re-run the step later, as it is advised
>    - "Project identifier was not found in the database" - double check the spelling, try entering project title, project label or project UUID.
>
{: .warning}

> <details-title>What will be the output?</details-title>
>
> When "Matrix Market" is seleted, outputs are in 10X-compatible Matrix Market format:
> - **Matrix (txt)**: Contains the expression values for genes (rows) and cells (columns) in raw counts. This text file is formatted as a Matrix Market file, and as such it is accompanied by separate files for the gene identifiers and the cells identifiers.
> - **Genes (tsv)**: Identifiers (column repeated) for the genes present in the matrix of expression, in the same order as the matrix rows.
> - **Barcodes (tsv)**: Identifiers for the cells of the data matrix. The file is ordered to match the columns of the matrix.
> - **Experiment Design file (tsv)**: Contains metadata for the different cells of the experiment.
>
> When "Loom" is selected, output is a single Loom HDF5 file:
> - **Loom (h5)**: Contains expression values for genes (rows) and cells (columns) in raw counts, cell metadata table and gene metadata table, in a [single HDF5 file](http://linnarssonlab.org/loompy/format/index.html).
>
{: .details}

If you chose **Loom** format and you need to convert your file to other datatype, you can use {% tool [SCEasy](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy1) %} (more details in the next section). If you chose **Matrix Market** format, you can then transform the output to AnnData or Seurat, as shown in the EBI SCXA example above. Below, you will find an example of transforming the output to AnnData object.


> <hands-on-title>Create AnnData object</hands-on-title>
>
> {% tool [Scanpy Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.8.1+galaxy9) %} with the following parameters:
>  Make sure you are using version **1.8.1+galaxy9** of the tool (change by clicking on {% icon tool-versions %} Versions button)
>    - *"Expression matrix in sparse matrix format (.mtx)"*: `Human Cell Atlas Matrix Downloader on matrix.mtx`
>    - *"Gene table"*:  `Human Cell Atlas Matrix Downloader on genes.tsv`
>    - *"Barcode/cell table"*: `Human Cell Atlas Matrix Downloader on barcodes.tsv`
>    - *"Cell metadata table"*: `Human Cell Atlas Matrix Downloader on exp_design.tsv`
{: .hands_on}

> <tip-title>Flagging genes by using AnnData Operations</tip-title>
>
> After you create AnnData file, you can additionally use the {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92) %} tool (note the version **1.8.1+galaxy92**) before downstream analysis. It's quite a useful tool since not only does it flag mitochondrial genes, but also automatically calculates a bunch of metrics, such as `log1p_mean_counts`, `log1p_total_counts`, `mean_counts`, `n_cells`, `n_cells_by_counts`, `n_counts`, `pct_dropout_by_counts`, and `total_counts`.
>
> When you use it to flag mitochondrial genes, here are some formatting tips:
> - Remember to check the name of the column with gene symbols
> - This tool is case sensitive
> - No parentheses needed when typing in the values
> - Including a dash is important to identify mitochondrial genes (eg. **MT-**)
{: .tip}

You can have a look at the [answer history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/hca-downloader---example-files-1) of performing those three quick steps.

tools_json:
["EBI SCXA Data Retrieval", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3", "Replace parts of text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.3+galaxy1", "Search in textfiles", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.3+galaxy1", "Scanpy Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.9.3+galaxy0", "AnnData Operations", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0", "Seurat Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/4.0.4+galaxy0", "Human Cell Atlas Matrix Downloader", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/hca_matrix_downloader/hca_matrix_downloader/v0.0.4+galaxy0", "SCEasy", "toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy1", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.8.1+galaxy9", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92"]


topic: single-cell
tutorial: scatac-standard-processing-snapatac2
title: Single-cell ATAC-seq standard processing with SnapATAC2
difficulty: intermediate
time_estimation: 4H
requirements_json: []
tags_json: ["10x", "epigenetics", "single-cell"]
content_hash: 33d2504abc035943978abbd5c2135e36
last_modified: 2026-05-27T09:01:49.742401
zenodo_link: https://zenodo.org/records/12707159
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.html

content:
{scATAC-seq} analysis is a method to decipher the chromatin states of the analyzed cells. In general, genes are only expressed in accessible (i.e. "open") chromatin and not in closed chromatin.
By analyzing which genomic sites have an _open_ chromatin state, cell-type specific patterns of gene accessibility can be determined.
{scATAC-seq} is particularly useful for analyzing tissue containing different cell populations, such as {PBMCs}.

In this tutorial we will analyze {scATAC-seq} data using the tool suites [SnapATAC2](https://kzhang.org/SnapATAC2/version/2.5/index.html) ({% cite Zhang2024 %}) and [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({%cite Wolf2018%}).
With both of these tool suites we will perform preprocessing, clustering and identification of {scATAC-seq} datasets from [10x Genomics](https://www.10xgenomics.com/products/single-cell-atac).
The analysis will be performed using a dataset of {PBMCs} containing ~4,620 single nuclei.

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/tutorial_mode.md %}


<!-- This is a comment. -->

> <comment-title></comment-title>
>
> This tutorial is significantly based on ["Standard pipeline" tutorial from SnapATAC2](https://kzhang.org/SnapATAC2/version/2.5/tutorials/pbmc.html). It can be seen as the {scATAC-seq} counterpart to the scRNA-seq tutorial [Clustering 3K PBMCs with Scanpy]( {% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %} ).
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# {scATAC-seq} with 10X Genomics
ATAC-seq utilizes a hyperactive Tn5 transposase ({% cite Kia2017 %}) to ligate adaptors to genome fragments, created by the transposase. Performing ATAC-seq on individual cells used to be an expensive and time-consuming labour.
The 10X Chromium NextGEM system made {scATAC-seq} a cost-effective method for gaining high-resolution data with a simple protocol.
After the transposition of nuclei in bulk, individual nuclei are put into Gel beads in Emulsion (GEM), containing unique 10x cell barcodes and sequencing adaptors for Illumina sequencing.
![Library Preparation]({% link topics/single-cell/images/scatac-standard-snapatac2/tenx_libprep_scatac.png %} "An overview of the 10X single-nuclei ATAC-seq library preparation")


# Data

The 5k {PBMCs} dataset for this tutorial is available for free from [10X Genomics](https://www.10xgenomics.com/datasets/5-k-peripheral-blood-mononuclear-cells-pbm-cs-from-a-healthy-donor-next-gem-v-1-1-1-1-standard-2-0-0). The blood samples were collected from a healthy donor and were prepared following the Chromium Next GEM scATAC-seq protocol. After sequencing on Illumina NovaSeq, the reads were processed by the **Cell Ranger ATAC 2.0.0** pipeline from 10X to generate a [*Fragments File*](https://support.10xgenomics.com/single-cell-atac/software/pipelines/latest/output/fragments).

> <details-title>Fragments File </details-title>
>
>   The Fragments File is a tabular file in a [BED](https://genome.ucsc.edu/FAQ/FAQformat.html#format1)-like format, containing information about the position of the fragments on the chromosome and their corresponding 10x cell barcodes.
>
{: .details}
SnapATAC2 requires 3 input files for the standard pathway of processing:
- `fragments_file.tsv`: A tabular file containing the chromosome positions of the fragments and their corresponding 10x cell barcodes.
- `chrom_sizes.txt`: A tabular file of the number of bases of each chromosome of the reference genome
- `gene_annotation.gtf.gz`: A tabular file listing genomic features of the reference genome (GENCODE GRCh38)

> <details-title>Chromosome sizes </details-title>
>
> - A chromosome sizes file can be generated using the tool {% tool [Compute sequence length](toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.3) %}.
> - The reference genome can either be selected from cached genomes or uploaded to the galaxy history.
>
{: .details}

> <comment-title></comment-title>
> - This tutorial starts with a `fragment` file.
> - SnapATAC2 also accepts mapped reads in a `BAM` file.
> - To learn how to get a `fragment` file or `BAM` file from raw `.FASTQ`-reads, please check out the tutorial ["Pre-processing of 10X Single-Cell ATAC-seq Datasets"]( {% link topics/single-cell/tutorials/scatac-preprocessing-tenx/tutorial.md %} )
> - If you would like to start the analysis with a `BAM` file, you can expand the details section ["Details: Creating a fragment file"]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#creating-a-fragment-file).
{: .comment}



## Get Data
> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the `fragments_file.tsv`, `chrom_sizes.txt` and `gene_annotation.gtf.gz` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>
>    ```
>    {{ page.zenodo_link }}/files/atac_pbmc_5k_nextgem_fragments.tsv.gz
>    {{ page.zenodo_link }}/files/chrom_sizes.txt
>    {{ page.zenodo_link }}/files/gencode.v46.annotation.gtf.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
>   - {% icon galaxy-pencil %} **Rename** the file `atac_pbmc_5k_nextgem_fragments.tsv` to `fragments_file.tsv`
>   - {% icon galaxy-pencil %} **Rename** the file `gencode.v46.annotation.gtf.gz` to `gene_annotation.gtf.gz`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Inspect `chrom_sizes` and `fragments_file`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many chromosomes are in `chrom_sizes`?
> 2. In which column are the cell barcodes stored in the `fragments_file`?
>
> > <solution-title></solution-title>
> >
> > 1. There are 25 chromosomes. The 22 autosomes (Chr. 1-22), both sex chromosomes (Chr. X and Y) and the small circular mitochondrial chromosome (Chr. M).
> > 2. The cell barcodes are unique 16 bp oligos, located in column 4.
> >
> {: .solution}
>
{: .question}

## Creating a fragment file
> <details-title>Creating a fragment file</details-title>
>  > <hands-on-title>fragment file</hands-on-title>
>  > 1. Import the file  `BAM_500-PBMC` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>  > ```
>  > {{ page.zenodo_link }}/files/BAM_500-PBMC.bam
>  > ```
>  >   - This dataset contains mapped reads in the `BAM` format.
>  >   - It was generated by following the tutorial ["Pre-processing of 10X Single-Cell ATAC-seq Datasets"]( {% link topics/single-cell/tutorials/scatac-preprocessing-tenx/tutorial.md %} ) until the output of {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.18) %}
>  > 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>  >    - *"Method used for preprocessing"*: `Convert a BAM file to a fragment file, using 'pp.make_fragment_file'`
>  >        - {% icon param-file %} *"File name of the BAM file"*: `BAM_500-PBMC` (Input dataset)
>  >        - {% icon param-toggle %} *"Indicate whether the BAM file contain paired-end reads"*: `Yes`
>  >        - *"How to extract barcodes from BAM records?"*: `From read names using regular expressions`
>  >          - *"Extract barcodes from read names of BAM records using regular expressions"*: `(................):`
>  >
>  >    > <comment-title></comment-title>
>  >    > - Not every regular expression type is supported.
>  >    > - This expression selects 16 characters if they are followed by a colon. Only the cell barcodes of the `BAM` file will match.
>  >    {: .comment}
>  >
>  > 3. Rename the generated file to `Fragments 500 PBMC`
>  > 4. Now you can continue with either the `fragments_file` from earlier or the new file `Fragments 500 PBMC`.
>  >    - {% icon galaxy-info %} The tool `pp.make_fragment_file` {% icon tool %} has implemented additional {QC} measures. These filter out larger fragments, which will be noticeable in the log-scale fragment size distribution.
>  >    - {% icon galaxy-info %} Please note that `Fragments 500 PBMC` only contains 500 {PBMCs} and thus the clustering will produce different outputs compared to the outputs generated by `fragments_file` (with 5k PBMC).
>  {: .hands_on}
{: .details}

# Preprocessing

Preprocessing of the scATAC-seq data contained in the `fragment` file with SnapATAC2 begins with importing the files and computing basic {QC} metrics.

SnapATAC2 compresses and stores the fragments into an `AnnData` object.

## AnnData
The [`AnnData`](https://anndata.readthedocs.io/en/latest/) format was initially developed for the [`Scanpy`](https://scanpy.readthedocs.io/en/stable/index.html) package and is now a widely accepted data format to
store annotated data matrices in a space-efficient manner.

![Anndata format]({% link topics/single-cell/images/scatac-standard-snapatac2/anndata_schema.svg %} "<code>AnnData</code> format stores a count matrix <code>X</code> together with annotations of
observations (i.e. cells) <code>obs</code>, variables (i.e. genes) <code>var</code> and unstructured annotations <code>uns</code>.")


## Import files to SnapATAC2

> <hands-on-title> Create an AnnData object </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Import data fragment files and compute basic QC metrics, using 'pp.import_data'`
>        - {% icon param-file %} *"Fragment file, optionally compressed with gzip or zstd"*: `fragments_file.tsv` (Input dataset)
>        - {% icon param-file %} *"A tabular file containing chromosome names and sizes"*: `chrom_sizes.txt` (Input dataset)
>        - {% icon param-toggle %} *"Whether the fragment file has been sorted by cell barcodes"*: `No`
>
>    > <details-title>Sorted by barcodes</details-title>
>    > - This tool requires the fragment file to be sorted according to cell barcodes.
>    > - If **pp.make_fragment_file** {% icon tool %} was used to generate the fragment file, this has automatically been done.
>    >   - Otherwise, the setting *"sorted by cell barcodes"* should remain `No`.
>    {: .details}
>
> 2. Rename the generated file to `Anndata 5k PBMC`
>
> 3. Check that the format is `h5ad`
{: .hands_on}

Because the `AnnData` format is an extension of the HDF5 format, i.e. a binary format, an `AnnData` object can not be inspected directly in Galaxy by clicking on the {% icon galaxy-eye %} (**View data**) icon.
Instead, we need to use a dedicated tool from the **AnnData** suite.

> <hands-on-title>Inspect an AnnData object</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC`
>    - *"What to inspect?"*: `General information about the object`
>
> 2. {% icon galaxy-eye %} Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 14232 × 0
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito'
>    >  uns: 'reference_sequences'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. How many observations are there? What do they represent?
>    > 2. How many variables are there?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 14232 observations, representing the cells.
>    > > 2. There are 0 variables, representing genomic regions. This is because genome-wide 500-bp bins are only added after initial filtering.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    > <tip-title>Faster Method for General Information</tip-title>
>    > * Many toolsets producing outputs in *AnnData* formats in Galaxy, provide the general information by default:
>    >    * Click on the name of the dataset in the history to expand it.
>    >    * General Anndata information will be given in the expanded box:
>    >
>    >      e.g.
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    14232 × 0
>    >      ```
>    >    * {% icon details %} This feature isn't the most reliable and might display:
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    1 × 1
>    >      ```
>    > * In such cases and for more specific queries, {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} is required.
>    {: .tip}
>
{: .hands_on}


## Calculate and visualize {QC} metrics

> <hands-on-title> Fragment-size distribution </hands-on-title>
>
> 1. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot fragment size distribution, using 'pl.frag_size_distr'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC` (output of **pp.import_data** {% icon tool %})
> 2. {% icon galaxy-eye %} Inspect the `.png` output
> 3. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot fragment size distribution, using 'pl.frag_size_distr'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC` (output of **pp.import_data** {% icon tool %})
>        - *"Change the y-axis (fragment counts) to log scale"*: `Yes`
> 4. {% icon galaxy-eye %} Inspect the `.png` output
>
>
>  ![fragment_size_distribution]({% link topics/single-cell/images/scatac-standard-snapatac2/pl.frag_size.png %})
>
>  ![log_fragment_size_distribution]({% link topics/single-cell/images/scatac-standard-snapatac2/log_pl.frag_size.png %})
> > <question-title></question-title>
> >
> > 1. What distinct features do the plots have? And what do they represent?
> > 2. Which fragments are generally from open chromatin?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 3 peaks are clearly visible (at <100-bp, ~200-bp and ~400-bp). The smallest fragments are from nucleosome-free regions, while the larger peaks of 200- and 400-bp contain mono- and di-nucleosome fragments, respectively.
> > > 2. The small fragments (<100-bp) are from open chromatin reads, since the Tn5 transposase could easily access the loosely packed DNA ({% cite Yan2020 %}).
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}


The {TSSe} is another important {QC} metric. Nucleosome-free fragments are expected to be enriched at {TSS}. TSSe shows increased fragmentation of chromatin around the TSS. This suggests open and accessible nucleosome-free chromatin.

{TSSe} is used as a QC metric, since an increased enrichment around TSS regions suggests that the experiment has captured biological meaningful genomic features.
TSSe scores of individual cells can be calculated using SnapATAC2's *metrics.tsse* function.

> <hands-on-title> Calculate and Plot TSSe </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute the TSS enrichment score (TSSe) for each cell, using 'metrics.tsse'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC` (output of **pp.import_data** {% icon tool %})
>        - {% icon param-file %} *"GTF/GFF file containing the gene annotation"*: `gene_annotation.gtf.gz` (Input dataset)
>
> 2. Rename the generated file to `Anndata 5k PBMC TSSe` or add the tag {% icon galaxy-tags %} `TSSe` to the dataset:
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>
> 3. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the TSS enrichment vs. number of fragments density figure, using 'pl.tsse'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC TSSe` (output of **metrics.tsse** {% icon tool %})
> 4. {% icon galaxy-eye %} Inspect the `.png` output
>
>
> ![TSSe plot against number of unique fragments]({% link topics/single-cell/images/scatac-standard-snapatac2/pl.tsse.png %})
> High-quality cells can be identified in the plot of {TSSe} scores against a number of unique fragments for each cell.
>
> > <question-title></question-title>
> >
> > 1. Where are high-quality cells located in the plot?
> > 2. Based on this plot, how should the filter be set?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. The cells in the upper right are high-quality cells, enriched for {TSS}. Fragments in the lower left represent low-quality cells or empty droplets and should be filtered out.
> > > 2. Setting the minimum number of counts at 5,000 and the minimum TSS enrichment to 10.0 is an adequate filter.
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

## Filtering
Based on the {TSSe} plot the cells can be filtered by TSSe and fragment counts.

> <hands-on-title> Filter cells </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC TSSe` (output of **metrics.tsse** {% icon tool %})
>        - *"Minimum number of counts required for a cell to pass filtering"*: `5000`
>        - *"Minimum TSS enrichemnt score required for a cell to pass filtering"*: `10.0`
>        - *"Maximum number of counts required for a cell to pass filtering"*: `100000`
>
> 2. Rename the generated file to `Anndata 5k PBMC TSSe filtered` or add the tag {% icon galaxy-tags %} `filtered` to the dataset
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4564 × 0
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse'
>    >  uns: 'reference_sequences'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. How have the observations changed, compared to the first `Anndata 5k PBMC` AnnData file?
>    > 2. What does this tell us about the quality of the data?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are only 4564 observations, compared to the initial 14232 observations.
>    > >
>    > >    And the `obs: 'tsse'` has been added (but already during **metrics.tsse**{% icon tool %})
>    > > 2. The empty droplets and low-quality cells have been filtered out, leaving us with 4564 high-quality cells.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}

## Feature selection
Currently, our AnnData matrix does not contain any variables. The variables will be added in the following step with the function *pp.add_tile_matrix*.
This creates a cell-by-bin matrix containing insertion counts across genome-wide 500-bp bins.

After creating the variables, the most accessible features are selected.
> <hands-on-title> Select features </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Generate cell by bin count matrix, using 'pp.add_tile_matrix'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC TSSe filtered` (output of **pp.filter_cells** {% icon tool %})
>        - *"The size of consecutive genomic regions used to record the counts"*: `500`
>        - *"The strategy to compute feature counts"*: `"insertion": based on the number of insertions that overlap with a region of interest`
>
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Perform feature selection, using 'pp.select_features'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata tile_matrix` (output of **pp.add_tile_matrix** {% icon tool %})
>        - *"Number of features to keep"*: `250000`
>
>    > <comment-title> Select features </comment-title>
>    >
>    > - Including more features improves resolution and can reveal finer details, but it may also introduce noise.
>    >    - To optimize results, experiment with the `n_features` parameter to find the most appropriate value for your dataset.
>    >
>    >  > <details-title> Different number of features </details-title>
>    >  > - To demonstrate the differences when selecting features, the following UMAP plots are the outputs from processing with a number of features between 1,000 and 500,000.
>    >  > - Fewer features result in fewer, but larger clusters. Selecting a lot of features will output more granular clusters and the compute time will increase.
>    >  > ![Different number of features UMAP]({% link topics/single-cell/images/scatac-standard-snapatac2/number_features.png %}"UMAP plots with different selected features")
>    >  >
>    >  {: .details}
>    > - At this step you can provide a blacklist or whitelist to specifically select relevant features.
>    >    - For example the [**ENCODE Blacklist**](https://github.com/Boyle-Lab/Blacklist) ({% cite Amemiya2019 %}) can be applied here.
>    {: .comment}
>
> 3. Rename the generated file to `Anndata 5k PBMC select_features` or add the tag {% icon galaxy-tags %} `select_features` to the dataset
> 4. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4564 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse'
>    >  var: 'count', 'selected'
>    >  uns: 'reference_sequences'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. How did `n_vars` change compared to `Anndata 5k PBMC TSSe filtered`
>    > 2. Where are the selected features stored in the count matrix?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 6,062,095 variables, compared to 0 in `TSSe filtered`.
>    > > 2. Selected features are stored in `var: 'selected'`.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Doublet removal

Doublets are removed by calling a customized [**scrublet**](https://github.com/AllonKleinLab/scrublet) ({% cite Wolock2019 %}) algorithm. *pp.scrublet* will identify potential doublets and the function *pp.filter_doublets* removes them.

> <details-title>Doublets in Single-cell datasets</details-title>
>
> - During single-cell sequencing, multiple nuclei can be encapsulated into the same 10x gel bead. The resulting multiplets (>97% of which are doublets) produce sequences from both cells.
> - Doublets can confound the results by appearing as "new" clusters or artifactual intermediary cell states.
>    - These problematic doublets are called **neotypic** doublets, since they appear as "new" cell types.
> - **Scrublet** (Single-cell Remover of Doublets) is an algorithm which can detect neotypic doublets that produce false results.
>    - The algorithm first simulates doublets by combining random pairs of observed cell features.
>    - The observed features of the "cells" are then compared to the simulated doublets and scored on their doublet probability.
>    - SnapATAC2's *pp.filter_doublets* then removes all cells with a doublet probability >50%.
>
> ![Doublet removal with scrublet]({% link topics/single-cell/images/scatac-standard-snapatac2/doublets-and-scrublet.png %} "Scrublet simulates expected doublets and produces doublet scores for each cell. ({% cite Wolock2019 %})")
>
{: .details}

> <hands-on-title> Scrublet </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute probability of being a doublet using the scrublet algorithm, using 'pp.scrublet'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC select_features` (output of **pp.select_features** {% icon tool %})
>
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Remove doublets according to the doublet probability or doublet score, using 'pp.filter_doublets'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata scrublet` (output of **pp.scrublet** {% icon tool %})
>        - *"Threshold for doublet probability"*: `0.5`
> 3. Rename the generated file to `Anndata 5k PBMC filter_doublets` or add the tag {% icon galaxy-tags %} `filter_doublets` to the dataset
> 4. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score'
>    >  var: 'count', 'selected'
>    >  uns: 'doublet_rate', 'reference_sequences', 'scrublet_sim_doublet_score'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. What was removed by **pp.filter_doublets**?
>    > 2. Where are the new annotations stored?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Cell doublets were removed. `n_obs` was reduced from 4564 to 4430 cells.
>    > > 2. The outputs of **pp.scrublet** are stored in observations `obs: 'doublet_probability', 'doublet_score'` and in unstructured annotations `uns: 'scrublet_sim_doublet_score'`. The output of **pp.filter_doublets** is stored in `uns: 'doublet_rate'`.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>
{: .hands_on}


# Dimension reduction

Dimension reduction is a very important step during the analysis of single cell data. During this, the complex multi-dimensional data is projected into lower-dimensional space, while the lower-dimensional embedding of the complex data retains as much information as possible. Dimension reduction enables batch correction, data visualization and quicker downstream analysis since the data is more simplified and the memory usage is reduced ({% cite Zhang2024%}).


> <details-title>Dimension reduction with SnapATAC2</details-title>
>
> - Dimension reduction algorithms can be either linear or non-linear.
> - Linear methods are generally computationally efficient and well scalable.
>
>   A popular linear dimension reduction algorithm is:
>     - **PCA** (Principle Component Analysis), implemented in **Scanpy** (please check out our [Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial for an explanation).
> - Nonlinear methods however are well suited for multimodal and complex datasets.
>     - in contrast to linear methods, which often preserve global structures, non-linear methods have a locality-preserving character.
>     - This makes non-linear methods relatively insensitive to outliers and noise while emphasizing natural clusters in the data ({% cite Belkin2003%})
>     - As such, they are implemented in many algorithms to visualize the data in 2 dimensions (f.ex. **UMAP** embedding).
> - The nonlinear dimension reduction algorithm, through *matrix-free spectral embedding*, used in **SnapATAC2** is a very fast and memory efficient non-linear algorithm ({% cite Zhang2024%}).
>     - **Spectral embedding** utilizes an iterative algorithm to calculate the **spectrum** (*eigenvalues* and *eigenvectors*) of a matrix without computing the matrix itself.
> - For a simple introduction into *spectral embedding* and how it compares to *PCA*, please check out the blog post ["On Laplacian Eigenmaps for Dimensionality Reduction"](https://juanitorduz.github.io/laplacian_eigenmaps_dim_red/) by Juan Orduz.
>
{: .details}

## Spectral embedding
The dimension reduction, produced by the algorithm *tl.spectral*, is required for later steps, such as plotting and clustering.

> <hands-on-title> Spectral embedding </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Perform dimension reduction using Laplacian Eigenmap, using 'tl.spectral'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC filter_doublets` (output of **pp.filter_doublets** {% icon tool %})
>        - *"Distance metric"*: `cosine`
>
>    > <comment-title> Distance metric </comment-title>
>    >
>    > - The fast and well scalable *"matrix-free spectral embedding"* algorithm depends on the distance metric: `cosine`
>    {: .comment}
>
> 2. Rename the generated file to `Anndata 5k PBMC spectral` or add the tag {% icon galaxy-tags %} `spectral` to the dataset
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score'
>    >  var: 'count', 'selected'
>    >  uns: 'doublet_rate', 'reference_sequences', 'scrublet_sim_doublet_score', 'spectral_eigenvalue'
>    >  obsm: 'fragment_paired', 'X_spectral'
>    > ```
>    >
>    > Where are the new annotations stored?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The outputs of **tl.spectral** are stored in unstructured annotations `uns: 'spectral_eigenvalue'` and as multidimensional observations `obsm: 'X_spectral'`.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}
## UMAP embedding
With the already reduced dimensionality of the data stored in `X_spectral`, the cells can be further embedded (i.e. transformed into lower dimensions) with {UMAP}.
**UMAP** projects the cells and their relationship to each other into 2-dimensional space, which can be easily visualized ({% cite McInnes2018%}).

> <hands-on-title> UMAP embedding </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute Umap, using 'tl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC spectral` (output of **tl.spectral** {% icon tool %})
>        - *"Use the indicated representation in '.obsm'"*: `X_spectral`
>
> 2. Rename the generated file to `Anndata 5k PBMC UMAP` or add the tag  {% icon galaxy-tags %} `UMAP` to the dataset
{: .hands_on}

# Clustering
During clustering, cells that share similar accessibility profiles are organized into clusters. **SnapATAC2** utilizes graph-based community clustering with the *Leiden* algorithm ({% cite Traag2019%}).
This method takes the k-nearest neighbor (KNN) graph as input data and produces well-connected communities.


## Community clustering

> <hands-on-title> Clustering analysis </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute a neighborhood graph of observations, using 'pp.knn'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC UMAP` (output of **tl.umap** {% icon tool %})
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Cluster cells into subgroups, using 'tl.leiden'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata knn` (output of **pp.knn** {% icon tool %})
>        - *"Whether to use the Constant Potts Model (CPM) or modularity"*: `modularity`
>
>    > <comment-title> CPM or modularity </comment-title>
>    > - make sure you selected `modularity`
>    > - the clusters produced by `CPM` are not represented well in the UMAP projections
>    >
>    {: .comment}
> 2. Rename the generated file to `Anndata 5k PBMC leiden` or add the tag {% icon galaxy-tags %} `leiden` to the dataset
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4564 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden'
>    >  var: 'count', 'selected'
>    >  uns: 'doublet_rate', 'reference_sequences', 'scrublet_sim_doublet_score', 'spectral_eigenvalue'
>    >  obsm: 'X_spectral', 'X_umap', 'fragment_paired'
>    >  obsp: 'distances'
>    > ```
>    >
>    > Where are the **leiden** clusters stored in the AnnData?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The clusters are stored in `obs: 'leiden'`
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Plotting of clusters
Now that we have produced **UMAP** embeddings of our cells and have organized the cells into **leiden** clusters, we can now visualize this information with *pl.umap*.

> <hands-on-title> Plotting the clusters </hands-on-title>
>
> 1. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding, using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC leiden` (output of **tl.leiden** {% icon tool %})
>        - *"Color"*: `leiden`
>        - *"Height of the plot"*: `500`
> 4. {% icon galaxy-eye %} Inspect the `.png` output
>
>  ![umap_leiden_clustering]({% link topics/single-cell/images/scatac-standard-snapatac2/pl.umap.png %})
>
> > <question-title></question-title>
> >
> > 1. How many leiden clusters were discovered?
> > 2. What does the distance of clusters to each other tell us about their chromatin states?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 13 leiden clusters.
> > > 2. Clusters in close proximity (f.ex. clusters 0 and 5) share a similar chromatin accessibility profile (and very likely also a similar cell type), compared to a cluster further away (f.ex. cluster 9).
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

# Cell cluster annotation

After clustering the cells, they must be annotated. This categorizes the clusters into known cell types. **Manual Cell Annotation** requires known marker genes and varying expression profiles of the marker genes among clusters.

Luckily, the marker genes for {PBMCs} are known ({% cite Sun2019 %}).
Marker genes for other single cell datasets can also be found in databases such as [PanglaoDB](https://panglaodb.se/markers.html) ({% cite Franzn2019 %}). Using the known marker genes we can now annotate our clusters manually.

## Gene matrix
Since our data currently doesn't contain gene information, we have to create a cell-by-gene activity matrix using the function *pp.make_gene_matrix*.

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Generate cell by gene activity matrix, using 'pp.make_gene_matrix'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC leiden` (output of **tl.leiden** {% icon tool %})
>        - {% icon param-file %} *"GTF/GFF file containing the gene annotation"*: `gene_annotation.gtf.gz` (Input dataset)
> 2. Rename the generated file to `Anndata 5k PBMC gene_matrix` or add the tag {% icon galaxy-tags %} `gene_matrix` to the dataset
>    > <tip-title> Gene matrix </tip-title>
>    >
>    > - Please note that *pp.make_gene_matrix* removes all annotations except those stored in `obs`.
>    > - Therefore it might be necessary to remove propagating tags {% icon galaxy-tags %} (tags starting with `#`) from `Anndata 5k PBMC gene_matrix`.
>    >    - Tags can be removed by expanding the dataset with a tag and clicking {% icon galaxy-cross %} next to the tag.
>    {: .tip}
>
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 60606
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden'
>    > ```
>    >
>    > What does `n_vars` represent in `Anndata 5k PBMC gene_matrix` and what did it represent in `Anndata 5k PBMC leiden`?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - The variables now represent accessible genes. There are 60606 accessible genes in our samples. In `Anndata 5k PBMC leiden` and all earlier AnnData the variables represented the 6062095 fixed-sized genomic bins.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Imputation with Scanpy and MAGIC
Similar to scRNA-seq data, the cell-by-gene-activity matrix is very sparse. Additionally, high gene variance between cells, due to technical confounders, could impact the downstream analysis.
In scRNA-seq, filtering and normalization are therefore required to produce a high-quality gene matrix.

Since the *cell-by-gene-activity* matrix resembles the *cell-by-gene-expression* matrix of scRNA-seq, we can use the tools of the [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({%cite Wolf2018%}) tool suite to continue with our data.

> <details-title>Imputation with MAGIC</details-title>
>
> - The count matrices of single-cell data are sparse and noisy.
> - Confounding issues, such as "dropout" effects, where some mRNA or DNA-segments are not detected although they are present in the cell, also result in some cells missing important cell-type defining features.
>    - These problems can obscure the data, as only the strongest gene-gene relationships are still detectable.
> - The *Markov Affinity-based Graph Imputation of Cells* (MAGIC) algorithm ({%cite vanDijk2018%}) tries to solve these issues by filling in missing data from some cells with transcript information from similar cells.
>    - The algorithm calculates the likely gene expression of a single cell, based on similar cells and fills in the missing data to produce the expected expression.
>      - *MAGIC* achieves this by building a graph from the data and using data diffusion to smooth out the noise.
>
> ![Imputation with the MAGIC algorithm]({% link topics/single-cell/images/scatac-standard-snapatac2/magic_method.png %} "MAGIC restores noisy and sparse single-cell data using diffusion geometry ({%cite vanDijk2018%})")
>
{: .details}

> <hands-on-title> Filter and normalize </hands-on-title>
>
> 1. {% tool [Filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC gene_matrix` (output of **pp.make_gene_matrix** {% icon tool %})
>    - *"Method used for filtering"*: `Filter genes based on number of cells or counts, using 'pp.filter_genes'`
>        - {% icon param-select %} *"Filter"*: `Minimum number of cells expressed`
>            - *"Minimum number of cells expressed required for a gene to pass filtering"*: `5`
>
> 2. {% tool [Normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata filter_genes` (output of **Filter** {% icon tool %})
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>    - {% icon param-toggle %} *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>
> 3. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata normalize` (output of **Normalize** {% icon tool %})
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
>
> 4. {% tool [Normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata log1p` (output of **log1p** {% icon tool %})
>    - *"Method used for normalization"*: `Denoising using Markov Affinity-based Graph Imputation of Cells (MAGIC) API 'external.pp.magic'`
>        - *"Denoised genes to return"*: `All genes`
>        - *"Which solver to use"*: `"approximate", is faster that performs imputation in the PCA space and then projects back to the gene space`
>
>    > <comment-title> </comment-title>
>     >   - Choosing the setting `Which solver to use: 'exact'` will result in a output file with better resolution.
>     >   - This is not necessary for our purposes, since the compute time also increases with this setting.
>     >
>     {: .comment}
>
> 5. Rename the generated file to `Anndata 5k PBMC magic` or add the tag {% icon galaxy-tags %} `magic` to the dataset
>
> 6. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 55106
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden', 'n_genes', 'n_counts'
>    >  var: 'n_cells', 'n_counts'
>    >  uns: 'log1p'
>    > ```
>    >
>    > 1. How did `n_vars` change, compared to `Anndata 5k PBMC gene_matrix`?
>    > 2. Which data was 'lost', compared to `Anndata 5k PBMC leiden`, and must be added to the file in order to produce **UMAP** plots?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The number of genes was reduced from 60606 to 55106 by the filtering. Additional annotations were added, such as: `obs: 'n_genes', 'n_counts'`, `var: 'n_cells', 'n_counts'` and `uns: 'log1p'`.
>    > > 2. The UMAP embeddings `obsm: 'X_umap'` are missing and should be added to the Anndata in the next step. Without `X_umap` it won't be possible to visualize the plots.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}

## Copy-over embeddings

> <hands-on-title> Copy UMAP embedding </hands-on-title>
>
> 1. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in hdf5 AnnData format"*: `Anndata 5k PBMC magic` (output of **external.pp.magic** {% icon tool %})
>    - *"Copy embeddings (such as UMAP, tSNE)"*: `Yes`
>       - *"Keys from embeddings to copy"*: `X_umap`
>       - {% icon param-file %} *"IAnnData objects with embeddings to copy"*: `Anndata 5k PBMC leiden`
>
>    > <comment-title> Annotations to copy </comment-title>
>    >
>    > - This tutorial only focuses on producing an **UMAP** plot with marker-genes.
>    > - If further analysis, with tools requiring more annotations, is intended, these can be added in a similar way as shown above.
>    >     - f.ex. *Peak and Motif Analysis* with {% tool [Snapatac2 peaks and motif](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_peaks_and_motif/snapatac2_peaks_and_motif/2.6.4+galaxy1) %} requires annotations from `uns`.
>    > - It is also possible to leave the input *"Keys from embeddings to copy"* empty, to copy all annotations of a given category such as `obsm`.
>    {: .comment}
>
> 5. Rename the generated file to `Anndata 5k PBMC magic UMAP` or add the tag {% icon galaxy-tags %} `UMAP` to the dataset
>
> 6. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output, to check if `obsm` contains `X_umap`
>
>   ```
>   AnnData object with n_obs × n_vars = 4430 × 55106
>    obs: obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden', 'n_genes', 'n_counts'
>    var: 'n_cells', 'n_counts'
>    uns: 'log1p'
>    obsm: 'X_umap'
>   ```
>
{: .hands_on}

## Visualize gene activity of marker genes
The gene activity of selected marker genes can now be visualized with Scanpy.

> <hands-on-title> Plot marker genes </hands-on-title>
>
> 1. {% tool [Plot with Scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `output_h5ad` (output of **AnnData Operations** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `leiden, CD3D, CD8A, CD4, MS4A1, NKG7, CD14, FCER1A`
>        - {% icon param-toggle %} *"Show edges?"*: `No`
>        - In *"Plot attributes"*
>           - *"Number of panels per row"*: `2`
>
>  ![umap_leiden_marker_gene_clustering]({% link topics/single-cell/images/scatac-standard-snapatac2/umap_leiden_marker-genes.png %})
>
> > <question-title></question-title>
> >
> > 1. Are the marker genes selectively expressed in the clusters?
> > 2. Which marker genes have overlapping expression profiles? And what could that imply?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. Some marker genes, such as `MS4A1` or `CD8A`, are only expressed in a few clusters (clusters 6+11 and clusters 1+8, respectively).
> > > 2. The marker gene `CD3D` is expressed in multiple clusters (1, 2, 4, 7 and 8). Overlapping expression profiles imply similar cell types since similar cell types have similar marker genes upregulated. In this case, `CD3D` expression classifies the cells in these clusters as T-cells.
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

## Manual cluster annotation
Comparison of marker gene expression in our clusters with a table of canonical marker genes ({% cite Sun2019 %}), enables us to annotate the clusters manually.

Cell type | Marker genes
--- | ---
CD8+ T cells | CD3D+, CD8A+, CD4-
CD4+ T cells | CD3D+, CD8A-, CD4+
B cells | CD3D-, MS4A1+
Natural killer (NK) cells | CD3D-, NKG7+
Monocytes | CD3D-, CD14+
Dendritic cells | CD3D-, FCER1A+

These canonical marker genes can match the clusters to known cell types:

Cluster | Cell type
--- | ---
0 | Monocytes
1 | CD8+ T cells
2 | CD4+ T cells
3 | Monocytes
4 | CD4+ T cells
5 | Monocytes
6 | B cells
7 | CD4+ T cells
8 | CD8+ T cells
9 | NK cells
10 | Monocytes
11 | B cells
12 | Dendritic cells

> <comment-title></comment-title>
> Note that some clusters contain subtypes (f.ex. the annotated B cell clusters contain both naive and memory B cells). The cell-type annotation can be refined by choosing more specific marker genes.
{: .comment}

To manually annotate the *Leiden* clusters, we will need to perform multiple steps:

1. **Inspect** the key-indexed observations of `Anndata 5k PBMC gene_matrix magic UMAP`
2. **Cut** the *Leiden* annotations out of the table
3. **Upload** a *replace file* containing the new cell type annotations for the *Leiden* clusters
4. **Replace** the values of the cluster annotation with cell type annotation
5. **Add** the cell type annotation to the AnnData
6. **Plot** the annotated cell types

> <hands-on-title> Manual annotation </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC gene_matrix magic UMAP`
>    - *"What to inspect?"*: `Key-indexed observations annotation`
> 2. Rename the generated file to `5k PBMC observations`
> 3. {% icon galaxy-eye %} Inspect the generated file
>
>    > <question-title></question-title>
>    > In which column is the **Leiden** annotation located?
>    > > <solution-title></solution-title>
>    > > The **Leiden** annotation is in column 8.
>    > >
>    > > Column 1 | Column 2 | Column 3 | Column 4 | Column 5 | Column 6 | Column 7 | Column 8 | Column 9 | Column 10
>    > > --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
>    > > "" | n_fragment | frac_dup | frac_mito | tsse | doublet_probability | doublet_score | leiden | n_genes | n_counts
>    > > AAACGAAAGACGTCAG-1 | 22070 | 0.5219425551271499 | 0.0 | 30.43315066436454 | 0.004634433324822066 | 0.009276437847866418 | 8 | 52303 | 16521.599844068267
>    > > AAACGAAAGATTGACA-1 | 10500 | 0.5345125681606597 | 0.0 | 29.10551296093465 | 0.004668403569267374 | 0.001088139281828074 | 1 | 54501 | 15020.42495602328
>    > > AAACGAAAGGGTCCCT-1 | 19201 | 0.5101785714285714 | 0.0 | 19.90011850347046 | 0.004634433324822066 | 0.009276437847866418 | 5 | 54212 | 16294.751533305309
>    > > AAACGAACAATTGTGC-1 | 13242 | 0.487399837417257 | 0.0 | 29.060913705583758 | 0.004660125753854076 | 0.0022172949002217295 | 7 | 53530 | 15456.629863655084
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>
> 4. {% tool [Cut columns](toolshed.g2.bx.psu.edu/repos/devteam/cut_columns/Cut1/1.0.2) %} with the following parameters:
>    - {% icon param-select %} *"

tools_json:
["Compute sequence length", "toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.3", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.18", "SnapATAC2 Preprocessing", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0", "SnapATAC2 Plotting", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1", "SnapATAC2 Clustering", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1", "Filter", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.9.6+galaxy3", "Normalize", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.9.6+galaxy3", "Inspect and manipulate", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.9.6+galaxy3", "AnnData Operations", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0", "Snapatac2 peaks and motif", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_peaks_and_motif/snapatac2_peaks_and_motif/2.6.4+galaxy1", "Plot with Scanpy", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.9.6+galaxy3", "Cut columns", "toolshed.g2.bx.psu.edu/repos/devteam/cut_columns/Cut1/1.0.2", "Replace column", "toolshed.g2.bx.psu.edu/repos/bgruening/replace_column_by_key_value_file/replace_column_with_key_value_file/0.2", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.3+galaxy0", "SnapATAC2 Peaks and Motif"]


topic: single-cell
tutorial: scrna-scanpy-pbmc3k
title: Clustering 3K PBMCs with Scanpy
difficulty: intermediate
time_estimation: 8H
requirements_json: ["type: internal", "scrna-preprocessing", "scrna-preprocessing-tenx"]
tags_json: ["10x"]
content_hash: 0cf493b71d3b0e96e3dc7cc04809b4da
last_modified: 2026-05-27T09:01:49.748821
zenodo_link: https://zenodo.org/record/3581213
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.html

content:
Single-cell RNA-seq analysis is a rapidly evolving field at the forefront of transcriptomic research, used in high-throughput developmental studies and rare transcript studies to examine cell heterogeneity within a populations of cells. The cellular resolution and genome wide scope make it possible to draw new conclusions that are not otherwise possible with bulk RNA-seq.

In this tutorial, we will investigate clustering of single-cell data from 10x Genomics, including preprocessing, clustering and the identification of cell types via known marker genes, using [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({% cite wolf2018scanpy %}). It will be illustrated using a dataset of Peripheral Blood Mononuclear Cells (PBMC), containing 2,700 single cells.


> <comment-title></comment-title>
>
> This tutorial is significantly based on ["Clustering 3K PBMCs" tutorial from Scanpy](https://scanpy-tutorials.readthedocs.io/en/latest/pbmc3k.html#Clustering-3K-PBMCs), ["Seurat - Guided Clustering Tutorial"](https://satijalab.org/seurat/v3.1/pbmc3k_tutorial.html) and ["Orchestrating Single-Cell Analysis with Bioconductor"](https://bioconductor.org/books/release/OSCA/) {% cite amezquita2019orchestrating %}.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data

For this tutorial, we analyze a dataset of Peripheral Blood Mononuclear Cells (PBMC) extracted from a healthy donor, freely available from 10X Genomics. The dataset contains 2,700 single cells sequenced using Illumina NextSeq 500. The raw sequences have been processed by the [**cellranger**](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) pipeline from 10X to extract a unique molecular identifier (UMI) count matrix, in a similar way to that as explained in ["Pre-processing of 10X Single-Cell RNA Datasets" tutorial]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}).

In this matrix, the values represent the number for each feature (i.e. gene; row) that are detected in each cell (column). Such matrices can be quite large, where here there are 2,700 columns with 32,738 lines, with mostly zero values, i.e. an extremely sparse matrix. To optimize the storage of such a table and the information about the genes and cells, **cellranger** creates 3 files:

- `genes.tsv`: a tabular file with information about the 32,738 genes in 2 columns (Ensembl gene id and the gene symbol)
- `barcodes.tsv`: a tabular file with the barcode for each of the 2,700 cells
- `matrix.mtx`: a condensed version of the count matrix

    The count matrix is therefore represented by its non-zero values. Each non-zero value is indicated by its line number (1st column), its column number (2nd column) and its value (3rd column). The first row gives indication about the number of lines, column and non-zero values. More information on the Matrix Market Exchange (mtx) format can be found [in this documentation](https://math.nist.gov/MatrixMarket/formats.html)

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the `genes.tsv`, `barcodes.tsv` and `matrix.mtx` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/genes.tsv
>    {{ page.zenodo_link }}/files/barcodes.tsv
>    {{ page.zenodo_link }}/files/matrix.mtx
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Inspect the `matrix` file
{: .hands_on}

> <question-title></question-title>
>
> ```
> 32738	2700	2286884
> 32709	1	4
> 32707	1	1
> 32706	1	10
> 32704	1	1
> ```
>
> 1. How many non-zero values are in the matrix?
> 2. How many counts are found for the 32,706th gene in the 1st cell?
>
> > <solution-title></solution-title>
> >
> > 1. There are 2,286,884 (2.6%) non-zero values for the 88,392,600 possible counts of the 32,738 genes (rows) and 2,700 cells (columns).
> > 2. 10 counts are found for the 32,706th row and 1st column.
> >
> {: .solution}
>
{: .question}

The representation of the matrix with 3 files is convenient to share the data but not to process them. Single-cell analysis packages have attempted to solve the problem of storage and analysis by inventing their own standard, which has led to the proliferation of many different "standards" in the scRNA-seq package ecosystem.

## AnnData

The most common format, called [`AnnData`](https://anndata.readthedocs.io/en/stable/), stores the matrix as well as gene and cell annotations in a concise, compressed and extremely readable manner:

![Anndata format]({% link topics/single-cell/images/scrna-pre-processing/tenx_anndata.svg %} "<code>AnnData</code> format stores a count matrix <code>X</code> together with annotations of observations (i.e. cells) <code>obs</code>, variables (i.e. genes) <code>var</code> and unstructured annotations <code>uns</code>.")

This format is used by [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({% cite wolf2018scanpy %}), the tool suite for analyzing single-cell gene expression data that we will use in this tutorial. So we need first to import the matrix and annotations of genes and cells into an `AnnData` object.

> <hands-on-title>Transform matrix and all into AnnData object</hands-on-title>
>
> 1. {% tool [Import Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.10.9+galaxy0) %} with the following parameters:
>    - *"Format for the annotated data matrix"*: `Matrix Market (mtx), from Cell ranger or not`
>        - {% icon param-file %} *"Matrix"*: `matrix.mtx`
>        - *"Use 10x Genomics formatted mtx"*: `Output from Cell Ranger v2 or earlier versions`
>            - {% icon param-file %} *"2-column genes file with gene id and gene name"*: `genes.tsv`
>            - {% icon param-file %} *"Barcodes (whitelist) file with one barcode per line"*: `barcodes.tsv`
>            - *"Variables index"*: `gene_symbols`
>            - *"Make the variable index unique by appending '-1', '-2'?"*: `Yes`
>
> 2. Rename the generated file to `Input 3k PBMC`
>
> 3. Check that the format is `h5ad`
{: .hands_on}

Because the `AnnData` format is an extension of the HDF5 format, i.e. a binary format, an `AnnData` object can not be inspected directly in Galaxy by clicking on the {% icon galaxy-eye %} (**View data**) icon. Instead we need to use a dedicated tool from the **AnnData** suite.

> <hands-on-title>Inspect an AnnData object</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `General information about the object`
>
> 2. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 2700 × 32738
>    >     var: 'gene_ids'
>    > ```
>    >
>    > 1. How many observations are there? What do they represent?
>    > 2. How many variables are there? What do they represent?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 2,700 observations, representing the cells.
>    > > 2. There are 32,738 variables, representing the genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    > <comment-title>Faster Method for General Information</comment-title>
>    >
>    > * The **ScanPy** toolset in Galaxy produces *AnnData* formats as
>    >   output, where general information is provided for each dataset:
>    >    * Click on the name of the dataset in the history to expand it.
>    >    * General Anndata information would be given in the expanded box:
>    >
>    >      e.g.
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    2700 x 32738
>    >      ```
>    > * For more specific queries, {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} is required.
>    {: .comment}
>
>    > <warning-title>Large compute resource consumption! Please run the next step responsibly.</warning-title>
>    > Extracting the complete matrix requires large compute resources and long time to finish. Use this step on matrices with less than 10k cells. To inspect
>    > a subset of matrix contents on huge matrices, please change *“What to inspect?”*: `Random chunk of defined size`
>    {: .warning}
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > 	MIR1302-10	FAM138A	OR4F5	RP11-34P13.7	RP11-34P13.8 ...
>    > AAACATACAACCAC-1	0.0	0.0	0.0	0.0
>    > AAACATTGAGCTAC-1	0.0	0.0	0.0	0.0
>    > AAACATTGATCAGC-1	0.0	0.0	0.0	0.0
>    > AAACCGTGCTTCCG-1	0.0	0.0	0.0	0.0
>    > AAACCGTGTATGCG-1	0.0	0.0	0.0	0.0
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a table with 2,700 lines (observations or cells) and 32,738 columns (variables or genes): the count matrix for each of the 32,738 genes and 2,700 cells. The 1st row contains the gene symbol as annotation of the columns and the 1st column the barcodes of the cells as annotation of the rows.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
> 6. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > ""
>    > AAACATACAACCAC-1
>    > AAACATTGAGCTAC-1
>    > AAACATTGATCAGC-1
>    > AAACCGTGCTTCCG-1
>    > AAACCGTGTATGCG-1
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a tabular with annotations of the observations. i.e. the cells. Here there are only the barcodes as annotation, so only one column, being also the index for the count matrix.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 7. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 8. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > 	gene_ids
>    > MIR1302-10	ENSG00000243485
>    > FAM138A	ENSG00000237613
>    > OR4F5	ENSG00000186092
>    > RP11-34P13.7	ENSG00000238009
>    > RP11-34P13.8	ENSG00000239945
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a tabular with annotations of the variables, i.e. the genes. Here the files has 2 columns: the gene symbols (also the index in the count matrix), `gene_ids` or the Ensemble gene ids.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

# Preprocessing

The first step when we get our counts is to prepare them before we perform the clustering. This includes ({% cite amezquita2019orchestrating %}):

1. **Selection and filtration of cells and genes based on quality metrics**

    Low-quality cells would interfere with downstream analyses. These cells may have been damaged during processing and may not have been fully captured by the sequencing protocol.

2. **Data normalization and scaling**

    It will eliminate cell-specific biases (e.g., in capture efficiency), allowing us to perform explicit comparisons across cells afterwards. Some transformations, typically log, are also applied to adjust for the mean-variance relationship.

3. **Selection of features**

    This selection picks a subset of interesting features for downstream analysis, by modelling the variance across cells for each gene and retaining genes that are highly variable. This step is done to reduce computational overhead and noise from uninteresting genes.

## Quality control

Genes that appear in less than a few cells can be considered noise and thus removed.

> <hands-on-title>Remove genes found in less than 3 cells</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"Method used for filtering"*: `Filter genes based on number of cells or counts, using 'pp.filter_genes'`
>        - *"Filter"*: `Minimum number of cells expressed`
>            - *"Minimum number of cells expressed required for a gene to pass filtering"*: `3`
>
> 2. Rename the generated file `3k PBMC`
> 3. Inspect the dataset.
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2700 × 13714
>    > [var]
>    > -    gene_ids
>    > -    n_cells
>    > ```
>    >
>    > How many genes have been removed because they are expressed in less than 3 expressed cells?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are now 13,714 genes. So 19,024 (32,738 - 13,714) genes have been removed.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

Low-quality cells may be due to a variety of sources such as cell damage during dissociation or failure in library preparation (e.g., inefficient reverse transcription or PCR amplification, {% cite ilicic2016classification %}). Usually these low-quality cells have low total counts, few expressed genes and high mitochondrial or spike-in proportions. Low-quality cells are problematic for downstream analyses and may contribute to misleading results.

> <details-title>Impact of low-quality cells on the downstream analyses</details-title>
>
> #### Formation of their own distinct cluster(s)
>
> Increased mitochondrial proportions or enrichment for nuclear RNAs after cell damage are the most obvious causes for this phenomenon. Low-quality cells in different cell types can, in the worst case, cluster together (because of similarities in the damage-induced expression profiles) leading to artificial intermediate states or trajectories between subpopulations that should be otherwise distinct. This phenomenon makes the results difficult to interpret.
>
> #### Distortion of population heterogeneity during variance estimation or principal components analysis
>
> With low-quality cells, the main differences captured by the first few principal components will be based on quality rather than biology, reducing then the effectiveness of dimensionality reduction. Similarly, differences between low- and high-quality cells will create genes with the largest variances. For example, in low-quality cells with very low counts, the scaling normalization increases the variance of genes that happen to have a non-zero count in those cells.
>
> #### Misidentification of upregulated genes
>
> Due to aggressive scaling to normalize for small cell sizes, low-quality cells shows genes that appear to be strongly "upregulated". For example, contaminating transcripts may be present in all cells with a low but constant levels. With the increased scaling normalization in low-quality cells, the small counts for these transcripts may become large normalized expression values, i.e. an apparent upregulation compared to other cells.
>
{: .details}

To mitigate these problems, we need to remove these low-quality cells at the start of the analysis. Several common QC metrics can be used to identify these cells based on their expression profile:

- **Cell size**, i.e. the total sum of counts across all genes for each cells

    When cells are very degraded or absent from the library preparation, the number of reads sequenced from that library will be very low. Cells with small sizes are then considered to be of low quality: the RNA has been lost during the library preparation (cell lysis, inefficient cDNA capture and amplification, etc.)

- **Number of expressed genes**, i.e. the number of genes with non-zero counts for each cells

    A low number of expressed genes may be a result of poor-quality cells (e.g. dying, degraded, damaged, etc.), followed by high PCR amplification of the remaining RNA. The diverse transcript population has not been successfully captured so the cell is considered of low quality. Cell doublets or multiplets may also exhibit an aberrantly high gene count.

- **Proportion of reads mapped to genes in the mitochondrial genome**

    High concentrations of mitochondrial genes is often a result of damaged cells ({% cite islam2014quantitative %}, {% cite ilicic2016classification %}) where the endogenous RNA escapes or degrades. As mitochondria has its own cell membranes, it is often the last DNA/RNA in damaged cells to degrade and hence occurs in high quantities during sequencing.

We will make the key assumption that these metrics are independent of the biological state of each cell. Technical factors rather than biological processes are presumed to drive poor values (e.g., low cell sizes, high mitochondrial proportions). The subsequent removal of impacted cells do not misrepresent the biology in downstream analyses.

To identify low-quality cells, the simplest approach is to apply thresholds on the QC metrics. This strategy, while simple, is required to determine appropriate thresholds that will change given the experimental protocol and the biological system.

### Computation of QC metrics

To estimate the thresholds, we first need to look at the distribution of QC metrics for our dataset.

The first 2 QC metrics (cell size and number of expressed genes) can be easily estimated from the count table. For the third metric (proportion of reads mapped to genes in the mitochondrial genome), we need the information about which genes are mitochondrial or not.

> <hands-on-title>Extract gene annotation</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 2. Rename the generated file `Gene annotation`
{: .hands_on}

In the gene annotation, we currently have the gene symbol the Ensembl gene ids and the number of cells in which the genes are expressed, but have no information on whether a gene is mitochondrial. This can be extracted from the gene symbol as all mitochondrial genes have a name starting with `MT-` and added to the `var` of the `AnnData` object using **Manipulate AnnData** tool.

A new annotation for `var` should be new column named `mito` on the 1st line and then `True` for mitochondrial gene or `False` for non-mitochondrial genes.

> <hands-on-title>Add mitochondrial gene annotation</hands-on-title>
>
> 1. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC`
>    - *"Function to manipulate the object"*: `Flag genes start with a pattern`
>    -  Click on *"Insert Flag genes that start with these names"*
>      - *"Text that you expect the genes to be flagged to start with?"*: `MT-`
>      - *"Name of the column in var.names where this boolean flag is stored"*: `mito`
>
> 2. Rename the generated file `3k PBMC with mito annotation`
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 4. Inspect the generated file and check if the mitochondrial annotation has been added
>
>    > <question-title></question-title>
>    >
>    > 1. How can I annotate genes like ribosomal protein genes or non-protein coding genes that might interfere with my downstream analysis?
>    > 2. What care should be taken while annotating such genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. To annotate genes such as ribosomal protein genes or non-protein coding genes, follow these steps:
>    > >
>    > >    - {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} tool as above with
>    > >        - *"Function to manipulate the object"*: `Flag genes start with a pattern`
>    > >        -  Click on *"Insert Flag genes that start with these names"*
>    > >          - *"Text that you expect the genes to be flagged to start with?"*: `RP` for ribosomal proteins or
>    > >          - *"Name of the column in var.names where this boolean flag is stored"*: `rpgene`
>    > >        -  Click on *"Insert Flag genes that start with these names"*
>    > >          - *"Text that you expect the genes to be flagged to start with?"*: `LINC` for long non-protein-coding RNAs
>    > >          - *"Name of the column in var.names where this boolean flag is stored"*: `lincrna`
>    > >
>    > >    - {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    > >        - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation`
>    > >        - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>    > >    - Inspect your AnnData to confirm the new gene annotations are correctly added.
>    > > 2. Here we are annotating genes starting with a pattern. Long Non-Coding RNAs often start with "LINC" followed by a number, while Ribosomal Protein Genes typically start with "RP". Be aware that some non-ribosomal genes also start with "RP" and are associated with conditions like Retinitis Pigmentosa. If these genes are relevant, annotate 'RP' genes carefully. Maybe use more precise naming like `RPL` and `RPS`. In general, make sure that your name/pattern does not annotate any genes that you did not want annotate.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

We can now compute QC metrics on the `AnnData` object.

> <hands-on-title>Compute QC metrics</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation`
>    - *"Method used for inspecting"*: `Calculate quality control metrics, using 'pp.calculate_qc_metrics'`
>      - *"Name of kind of values in X"*: `counts`
>      - *"The kind of thing the variables are"*: `genes`
>      - *"Keys for boolean columns of `.var` which identify variables you could want to control for"*: `mito`
>
> 2. Rename the generated file `3k PBMC with mito annotation and qc metrics`
>
> 3. Inspect the dataset
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2700 × 13714
>    > [obs]
>    > -    n_genes_by_counts
>    > -    log1p_n_genes_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > -    pct_counts_in_top_50_genes
>    > -    pct_counts_in_top_100_genes
>    > -    pct_counts_in_top_200_genes
>    > -    pct_counts_in_top_500_genes
>    > -    total_counts_mito
>    > -    log1p_total_counts_mito
>    > -    pct_counts_mito
>    > [var]
>    > -    gene_ids
>    > -    n_cells
>    > -    mito
>    > -    n_cells_by_counts
>    > -    mean_counts
>    > -    log1p_mean_counts
>    > -    pct_dropout_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > ```
>    >
>    > Which QC metrics have been computed?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The tool computed several QC metrics at both cell and gene levels. These metrics are added to the annotation tables.
>    > >
>    > > At gene levels (stored in `var`):
>    > > - `total_counts`: sum of counts for a gene
>    > > - `mean_counts`: mean expression for a gene over all cells
>    > > - `n_cells_by_counts`: number of cells with non-zero counts for a gene
>    > > - `pct_dropout_by_counts`: percentage of cells this gene does not appear in
>    > >
>    > > At cell levels (stored in `obs`):
>    > > - `total_counts`: total number of counts for a cell
>    > > - `total_counts_mito`: total number of counts for the mitochondrial genes in a cell
>    > > - `n_genes_by_counts`: number of genes with non-zero counts
>    > > - `pct_counts_mito`: proportion of total counts for a cell which are from mitochondrial genes
>    > > - `pct_counts_in_top_50_genes`: cumulative percentage of counts for 50 most expressed genes in a cell
>    > > - `pct_counts_in_top_100_genes`: cumulative percentage of counts for 100 most expressed genes in a cell
>    > > - `pct_counts_in_top_200_genes`: cumulative percentage of counts for 200 most expressed genes in a cell
>    > > - `pct_counts_in_top_500_genes`: cumulative percentage of counts for 500 most expressed genes in a cell
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

We would like to visualize 3 of the more informative QC metrics:

- the cell size, i.e. `total_counts`
- the number of expressed genes, i.e. `n_genes_by_counts`
- the proportion of reads mapped to mitochondrial genes, i.e. `pct_counts_mito`

> <hands-on-title>Visualize QC metrics</hands-on-title>
>
> 1. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>      - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>        - *"Keys for accessing variables"*: `n_genes_by_counts, total_counts, pct_counts_mito`
>      - In *"Violin plot attributes"*:
>        - *"Display keys in multiple panels"*: `Yes`
>
> 2. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ![QC violin plot]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/qc_violin_plot.png %})
>    >
>    > How do the distributions of the 3 QC metrics look?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > For the cell size, i.e. `total_counts`, most of the values are between 1,000 reads and 4,000 reads, with some extremely high values skewing the distribution.
>    > >
>    > > The numbers of expressed genes, i.e. `n_genes_by_counts`, are mostly between 500 genes and 1,200 genes, with also some extremely high values skewing the distribution.
>    > >
>    > > The distribution of the proportions of reads mapped to mitochondrial genes, i.e. `pct_counts_mito`, is even more narrow with some cells having no counts from mitochondrial genes but also having some really extreme values (above 5%).
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>      - *"Plotting tool that computed coordinates"*: `Using coordinates`
>        - *"x coordinate"*: `total_counts`
>        - *"y coordinate"*: `n_genes_by_counts`
>        - *"Use the layers attribute?"*: `No`
>
> 4. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ![QC total_counts n_genes_by_counts]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/qc_scatter_total_counts_n_genes_by_counts.png %})
>    >
>    > Is there any relationship between the cell size and the number of expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > On the plot, we can see a strong correlation between the total number of counts for a cell and the number of genes with positive counts.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `n_genes_by_counts`
>            - *"y coordinate"*: `pct_counts_mito`
>            - *"Use the layers attribute?"*: `No`
>
> 6. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ![QC total_counts pct_counts_mito]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/qc_scatter_n_genes_by_counts_pct_counts_mito.png %})
>    >
>    > 1. Is there any relationship between the number of expressed genes and the proportion of reads mapped to mitochondrial genes?
>    > 2. What could be a good threshold to filter for cells with high concentrations of mitochondrial genes?
>    > 3. What could be good thresholds to filter for cells based on the number of expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Cells with a high proportion of mitochondrial genes are not also cells that have many expressed genes. There is no visible correlation.
>    > > 2. The cells with a percentage of mitochondrial counts above 5% have also few genes. So 5% may be a good threshold.
>    > > 3. As discussed before, a low number of expressed genes may be a sign of poor-quality cells. Any cells with less than 200 genes is definitely out of the main distribution so this could be a good threshold to use. High gene count may also be a sign for cell multiplets. We will choose here a threshold of 2,500 genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

### Filtering of low-quality cells

As explained before, we would like now to filter for low-quality cells based on the 3 previous metrics (cell size, number of expressed genes, and proportion of reads mapped to mitochondrial genes). As the cell size is highly correlated with the number expressed genes, we can focus only on the number expressed genes and the proportion of reads mapped to mitochondrial genes.

Based on the previous plot, we would like to remove cells that have:

- a number of expressed genes below 200 or above 2,500
- a percentage of reads mapped to mitochondrial genes above 5%

> <hands-on-title>Remove low-quality cells</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - *"Filter"*: `Minimum number of genes expressed`
>            - *"Minimum number of genes expressed required for a cell to pass filtering"*: `200`
>
> 2. Inspect the dataset
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2700 × 13714
>    > ```
>    >
>    > How many cells have been removed because they have less than 200 expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are still 2,700 cells. So no cells have been removed because they have less than 200 expressed genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 3. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy filter** {% icon tool %}
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - *"Filter"*: `Maximum number of genes expressed`
>            - *"Maximum number of genes expressed required for a cell to pass filtering"*: `2500`
>
> 4. Inspect the dataset
>    >
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2695 × 13714
>    > ```
>    >
>    > How many cells have been removed because they have more than 2,500 expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are now 2,695 cells. So 5 cells have been removed because they have more than 2,500 expressed genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy filter** {% icon tool %}
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `pct_counts_mito`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `5.0`
>
> 6. Rename the generated file `3k PBMC after QC filtering`
>
> 7. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC after QC filtering`
>    - *"What to inspect?"*: `General information about the object`
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 2638 × 13714
>    > ```
>    >
>    > How many cells have been removed because they have more than 5% of reads mapped to mitochondrial genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are now 2,638 cells. So 57 cells have been removed because they have more than 5% of reads mapped to mitochondrial genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

## Normalization and scaling

In scRNA-seq, we can observe systematic differences in sequencing coverage between cells ({% cite stegle2015computational %}), because of technical differences in cDNA capture or PCR amplification efficiency across cells, attributable to the difficulty of achieving consistent library preparation with minimal starting material. After removing low-quality cells, normalization of the counts removes these differences to avoid that they interfere with comparisons of the expression profiles between cells. Any observed heterogeneity or differential expression within the cell population after normalization are then driven by biology and not technical biases.

Scaling normalization is the simplest and most commonly-used class of normalization strategies. All counts for each cell are divided by a cell-specific scaling factor, often called a "size factor". The assumption behind this process is that any technical biases tend to affect genes in a similar manner. The relative bias for each cell is represented by the size factor for that cell. Dividing the cell counts by its size factor should then remove the bias.

The simplest strategy for scaling normalization is the cell size normalization, i.e. similar total sum of counts across all genes for each cell. The "cell size factor" for each cell is computed directly for its cell size and transformed such that the mean size factor across all cells is equal to 1. The normalized expression values are then kept on the same scale as the original counts.

Here we would to normalize our count table such that each cell have 10,000 reads.

> <hands-on-title>Normalize for cell size</hands-on-title>
>
> 1. {% tool [Scanpy normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC after QC filtering`
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>      - *"Target sum"*: `10000.0`
>      - *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>      - *"Name of the field in 'adata.obs' where the normalization factor is stored"*: `norm`
>
{: .hands_on}

The normalized counts should be log-transformed afterwards to adjust for the mean-variance relationship.

With log-transformation, the differences in the log-values represent log-fold changes in expression. Log-transformation focuses on promoting contributions from genes with strong relative differences (e.g. a gene that is expressed at an average count of 50 in cell type A and 10 in cell type B rather than a gene that is expressed at an average count of 1100 in A and 1000 in B).

> <hands-on-title>Log-transform the counts</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy normalize** {% icon tool %}
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
{: .hands_on}

We will freeze the current state of the AnnData object, i.e. the logarithmized raw gene expression, in the a `raw` attribute. This information will be used later in differential testing and visualizations of gene expression.

> <hands-on-title>Freeze the state of the AnnData object</hands-on-title>
>
> 1. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy Inspect and manipulate** {% icon tool %}
>    - *"Function to manipulate the object"*: `Freeze the current state into the 'raw' attribute`
>
> 2. Rename the generated output `3k PBMC after QC filtering and normalization`
{: .hands_on}

## Selection of features

Data from scRNA-Seq is often used in exploratory analyses to characterize heterogeneity across cells. With clustering and dimensionality reduction, cells are compared based on their gene expression profiles. The choice of genes to use may have a major impact on the behaviour of the clustering and the dimensionality reduction. We need then to remove genes with random noise while keeping genes containing useful information about the biology of the system. This reduces the data size but still highlights any interesting biological signal without the noise that obscures that structure.

Selecting the most variable genes based on their expression across the cells (i.e. genes highly expressed in some cells, and lowly expressed in others) is the simplest approach for feature selection. With this approach, we assume that increased variation in some genes compared to other genes are genuine biological differences and not technical noise or a baseline level of "uninteresting" biological variation.

To quantify the per-gene variation, the simplest approach consists of computing the variance of the log-normalized expression values for each gene across all cells in the population. With this approach, the feature selection is based on the same log-values as the ones used in clustering, ensuring then that the quantitative definition of heterogeneity is consistent throughout the entire analysis.

Calculation of the per-gene variance is simple but feature selection requires modelling of the mean-variance relationship, as done in the Seurat procedure ({% cite stuart2019comprehensive %}) we will use.

Once the per-gene variation has been quantified, we need to select the subset of highly variable genes that we will use in downstream analyses. A large subset reduces the risk of discarding any interesting biological signal, but increases the noise from irrelevant genes on the signal. The optimal trade-off is difficult to determine but there are several common strategies routinely used. You can read the [Chapter 8 from "Orchestrating Single-Cell Analysis with Bioconductor"](https://osca.bioconductor.org/feature-selection.html#hvg-selection) for a nice presentation of the different strategies. Here, we will define the set of highly variable genes as those which, after normalization, have a normalized dispersion amount higher than 0.5.

> <hands-on-title>Identify the highly variable genes</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC after QC filtering and normalization`
>    - *"Method used for filtering"*: `Annotate (and filter) highly variable genes, using 'pp.highly_variable_genes'`
>      - *"Flavor for computing normalized dispersion"*: `seurat`
>        - *"Minimal mean cutoff"*: `0.0125`
>        - *"Maximal mean cutoff"*: `3`
>        - *"Minimal normalized dispersion cutoff"*: `0.5`
>      - *"Inplace subset to highly-variable genes?"*: `No`
>
> 2. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of the last **Scanpy filter** {% icon tool %}
>    - *"Method used for plotting"*: `Preprocessing: Plot dispersions versus means for genes, using 'pl.highly_variable_genes'`
>
{: .hands_on}

![Highly variable genes]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/feature_selection_highly_variable_genes.png %})

Both highly variable genes and other genes are still in the `AnnData` object. We can now actually keep only the highly variable genes.

> <hands-on-title>Keep the highly variable genes</hands-on-title>
>
> 1. Inspect the output of the last **Scanpy filter** {% icon tool %}
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2638 × 13714
>    > [obs]
>    > -    n_genes_by_counts
>    > -    log1p_n_genes_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > -    pct_counts_in_top_50_genes
>    > -    pct_counts_in_top_100_genes
>    > -    pct_counts_in_top_200_genes
>    > -    pct_counts_in_top_500_genes
>    > -    total_counts_mito
>    > -    log1p_total_counts_mito
>    > -    pct_counts_mito
>    > -    n_genes
>    > -    norm
>    > [var]
>    > -    gene_ids
>    > -    n_genes
>    > -    mito
>    > -    n_cells_by_counts
>    > -    mean_counts
>    > -    log1p_mean_counts
>    > -    pct_dropout_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > -    highly_variable
>    > -    means
>    > -    dispersions
>    > -    dispersions_norm
>    > ```
>    >
>    > 1. How many genes are in the `AnnData` object?
>    > 2. Where is the stored the information about the genes and if they are highly variable or not?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are now 13,714 genes, as before.
>    > > 2. Extra annotations have been added to `var`, whose a boolean annotation `highly_variable` for highly variable genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 2. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of the last **Filter** {% icon tool %}
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>      - *"What to filter?"*: `Variables (var)`
>        - *"Type of filtering?"*: `By key (column) values`
>          - *"Key to filter"*: `highly_variable`
>          - *"Type of value to filter"*: `Boolean`
>          - *"Value to keep"*: `Yes`
>
> 3. Rename the generated output `3k PBMC with only HVG`
>
> 4. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with only HVG`
>    - *"What to inspect?"*: `General information about the object`
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 2638 × 1838
>    > ```
>    >
>    > How many genes have been removed?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > Only 1,838 (over the 13,714) genes are kept.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

## Scaling the data

Prior to any downstream analysis like dimensional reduction, we need to apply a linear transformation or scaling to:

1. Regress out unwanted sources of variation in the total counts per cell and the percentage of mitochondrial genes expressed.
2. Scale data to unit variance and zero mean, i.e. the variance across cells is 1 and the mean expression is 0, in order to give equal weight in downstream analyses and ensure that highly-expressed genes do not dominate.

> <hands-on-title>Scale the data</hands-on-title>
>
> 1. {% tool [Scanpy remove confounders](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_remove_confounders/scanpy_remove_confounders/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with only HVG`
>    - *"Method used for plotting"*: `Regress out unwanted sources of variation, using 'pp.regress_out'`
>      - *"Keys for observation annotation on which to regress on"*: `total_counts, pct_counts_mito`
>
> 2. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %}
>     with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy remove confounders** {% icon tool %}
>    - *"Method used for inspecting"*: `Scale data to unit variance and zero mean, using 'pp.scale'`
>      - *"Zero center?"*: `Yes`
>      - *"Maximum value"*: `10.0`
>
>        It clips values exceeding a standard deviation of 10.
>
> 3. Rename the generated output `3k PBMC with only HVG, after scaling`
{: .hands_on}

# Dimensionality reduction

We aim in scRNA-seq to compare cells based on their expressions across genes, e.g. to identify similar transcriptomic profiles. Each gene represents then a dimension of the data.

With a dataset of 2 genes, we could make a 2-dimensional plot where each point is a cell and each axis is the expression of one gene. For datasets with thousands of genes, the concept is the same: each cell's expression profile defines its location in the high-dimensional expression space.

Expressions of different genes are correlated if they are affected by the same biological process. The separate information for these individual genes do not need to be stored, but can instead be compressed into a single dimension, e.g. an "eigengene". Dimensionality reduction aims then to reduce the number of separate dimensions in the data and then:

- reduces the computational work in downstream analyses to only a few dimensions
- reduces noise by averaging across multiple genes to obtain a more precise representation of the patterns in the data
- enables effective plotting of the data.

## Principal Component Analysis

Principal Component Analysis (PCA) is a dimensionality reduction technique consisting in the identification of axes in high-dimensional space that capture the largest amount of variation. This simple, highly effective strategy is widely used in data science.

In PCA, the first axis (or Principal Component (PC)) is chosen such that it captures the greatest variance across cells. The next PC should be orthogonal to the first and capture the greatest remaining amount of variation, and so on.

By applying PCA to scRNA-Seq, we assume that multiple genes are affected by the same biological processes in a coordinated way and random technical or biological noise affects each gene independently. As more variation can be captured by considering the correlated behaviour of many genes, the top PCs are likely to represent the biological signal and the noise are concentrated into the later PCs. The dominant factors of heterogeneity are then captured by the top PCs. Restricting downstream analyses to the top PCs will then reduce the dimensionality of the data whilst focusing on the biological signal and removing the noise.

Here we perform the PCA on the log-normalized expression values and compute the first 50 PCs.

> <hands-on-title>Perform the PCA</hands-on-title>
>
> 1. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy0) %}
>    with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with only HVG, after scaling`
>    - *"Method used for plotting"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'pp.pca'`
>      - *"Number of principal components to compute"*: `50`
>      - *"Type of PCA?"*: `Full PCA`
>        - *"Compute

tools_json:
["Import Anndata", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.10.9+galaxy0", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0", "Scanpy filter", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0", "Manipulate Anndata", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0", "Scanpy Inspect and manipulate", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1", "Scanpy plot", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0", "Scanpy normalize", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.10.2+galaxy0", "Scanpy remove confounders", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_remove_confounders/scanpy_remove_confounders/1.10.2+galaxy0", "Scanpy cluster, embed", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy0", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.11.5+galaxy0", "CellTypist", "toolshed.g2.bx.psu.edu/repos/iuc/celltypist/celltypist/1.7.1+galaxy0"]


topic: single-cell
tutorial: scrna-plant
title: Analysis of plant scRNA-Seq Data with Scanpy
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["plants", "paper-replication"]
content_hash: 1076c92d61ef3198101dfe26c0da63b1
last_modified: 2026-05-15T22:28:16.810338
zenodo_link: https://zenodo.org/record/4597857
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-plant/tutorial.html

content:
Single cell RNA-seq analysis is a cornerstone of developmental research and provides a great level of detail in understanding the underlying dynamic processes within tissues. In the context of plants, this highlights some of the key differentiation pathways that root cells undergo.

![The various cell subpopulations that are expected within this tutorial]({% link topics/single-cell/images/scrna-plant/scrna-plant1.jpg %})


This tutorial replicates the paper *"Spatiotemporal Developmental Trajectories in the Arabidopsis Root Revealed Using High-Throughput Single-Cell RNA Sequencing"* ({% cite denyer2019spatiotemporal %}), where the major plant cell types are recovered in the data as well as distinguishing between QC and meristematic cells. The original paper used the Seurat analysis suite ({% cite satija2015spatial %}), but here we will use the ScanPy analysis suite ({% cite wolf2018scanpy %}) integrated within the single-cell resources in Galaxy ({% cite tekman2020single %}).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title></comment-title>
>
> Please familiarise yourself with the ["Clustering 3K PBMCs with ScanPy"]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial first, as much of the process is the same, and the accompanying slide deck better explains some of the methods and concepts better.
>
{: .comment}

# Data

The *Arabidopsis* root cells come from two biological replicates which were isolated and profiles using droplet-based scRNA-seq (please see: ["Pre-processing of 10X Single-Cell RNA Datasets"]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %})); a "short root" mutant (labelled: `shr`) and a wild-type (labelled: `wt`):

* GSE123818\_Root\_single\_cell\_**shr**\_datamatrix.fixednames.transposed.csv.gz
* GSE123818\_Root\_single\_cell\_**wt**\_datamatrix.fixednames.transposed.csv.gz

As explained in the Zenodo link, the datasets have been modified to use more common gene names (hence the "fixednames" suffix) and the datasets have been transposed to better suit the AnnData format (hence the "transposed" suffix). The datasets are CSV tabular files which have been compressed using gunzip. We will now import these for analysis.


## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the two datasets from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/GSE123818_Root_single_cell_shr_datamatrix.fixednames.transposed.csv.gz
>    {{ page.zenodo_link }}/files/GSE123818_Root_single_cell_wt_datamatrix.fixednames.transposed.csv.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Apply the tags `#shr` and `#wt` to each of the datasets, with respect to the initial labels. These tags will be inherited for each task that the dataset undergoes.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

> <question-title></question-title>
>
> We can peek at the `#shr` file, and determine the dimensionality and naming scheme of the data. The rows and the columns depict different variables.
>
> 1. Which are the cell barcodes? Rows or Columns?
> 1. Which are the gene names? Rows or Columns?
> 1. How many cells in the dataset?
> 1. How many genes in the dataset?
>
> > <solution-title></solution-title>
> >
> > 1. Rows, notice the long A-C-T-G based names.
> > 1. Columns, "ATXG" are the gene prefixes commonly used by the Arabidopsis genomes.
> > 1. There are 1 099 lines and therefore 1 099 cells
> > 1. If we scroll the preview window all the way to the right we see 27 630 as the last number. Given that the first column are the cell names, we must subtract by 1 to get 27 629 cells.
> >
> {: .solution}
>
{: .question}

If the above feels like a convoluted way to get the dimensionality, that's because we haven't imported the data into the right format. For this we need to import both datasets into a single AnnData object (see the [AnnData section in the previous tutorial]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}#anndata)).

## CSV to AnnData

> <hands-on-title>Converting the Data</hands-on-title>
>
> 1. {% tool [Import Anndata and loom](toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.7.5+galaxy0) %} with the following parameters:
>    - *"hd5 format to be created"*: `Anndata file`
>        - *"Format for the annotated data matrix"*: `Tabular, CSV, TSV`
>            - {% icon param-file %} *"Annotated data matrix"*: Multi-select both `SHR` and `WT` datasets
>
>    > <comment-title>Checking Dimensionality</comment-title>
>    >
>    > We can now inspect the dimensionality of the dataset by "peeking" at the dataset in the history and observing the general information, simply by clicking on the name of the dataset.
>    >
>    > The `#shr` dataset should show:
>    >    ```
>    >    [n_obs x n_vars]
>    >        1099 x 27629
>    >    ```
>    >
>    > and the `#wt` should show:
>    >    ```
>    >    [n_obs x n_vars]
>    >        4727 x 27629
>    >    ```
>    >
>    > where *obs* refers to our observed cells, and *vars* references the genes/variables.
>    >
>    {: .comment}
>
{: .hands_on}

Currently we have two seperate datasets, but we can merge them into one single AnnData object seperated by batch identifiers "shr" and "wt". To do this, we simply manipulate one of the datasets and concatenate the other onto it.

## Merge Batches and Relabel

> <hands-on-title>Merging Data</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `SHR dataset` (output of **Import Anndata and loom** {% icon tool %})
>    - *"Function to manipulate the object"*: `Concatenate along the observations axis`
>        - {% icon param-file %} *"Annotated data matrix to add"*: `WT dataset` (output of **Import Anndata and loom** {% icon tool %})
>
>    > <tip-title>Scheduling</tip-title>
>    >
>    > You can queue on the next task without the previous one being complete yet, just ensure that the input dataset for the next task is at least orange and not grey.
>    {: .tip}
>
>    > <comment-title></comment-title>
>    >
>    > At this point, the AnnData dataset batch information has a value of `0` for the "SHR" cells and `1` for the "WT" cells. This is fine if you wish to remember these two values for the remainder of the analysis since the step below is optional, however for plotting purposes it can just be easier to relabel the batch information with the action text labels `shr` and `wt`.
>    {: .comment}
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Rename categories of annotation`
>        - *"Key for observations or variables annotation"*: `batch`
>        - *"Comma-separated list of new categories"*: `shr, wt`
>
>    > <warning-title>Order matters!</warning-title>
>    >
>    > The order in which the datasets are concatenated can affect the accuracy of the new labelling. If the `#wt` dataset was first selected and then the `#shr` dataset concatenated onto it, then the comma-seperated list of new categories above should be inverted.
>    {: .warning}
>
{: .hands_on}

# Quality Control

We can confirm that our datasets have been combined into a single object by peeking at the dataset in the history and confirming two things:

1. The number of observations is equal to the total of the two initial AnnData datasets.
2. The file size of the new AnnData dataset is approximately equal to the total of the two initial AnnData datasets.

> |     Batch | Number of Cells | File Size |
> |----------:|:----------------|----------:|
> |       shr | 1099            |  117.2 MB |
> |        wt | 4727            |  499.7 MB |
> |----------:|:----------------|----------:|
> | **Total** | 5826            |  615.6 MB |
{: .matrix}

A happy coincidence here is that both datasets already had the exact same variables, so that the merging of the two datasets still yields the same number of genes.

## Filtering the Matrix

> <hands-on-title>Generating some metrics</hands-on-title>
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for inspecting"*: `Calculate quality control metrics, using 'pp.calculate_qc_metrics'`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>      - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>        - *"Keys for accessing variables"*: `n_genes_by_counts, total_counts`
>      - *"The key of the observation grouping to consider"*: `batch`
>      - In *"Violin plot attributes"*:
>        - *"Add a stripplot on top of the violin plot"*: `Yes`
>          - *"Add a jitter to the stripplot"*: `Yes`
>            - *"Size of the jitter points"*: `0.4`
>        - *"Display keys in multiple panels"*: `Yes`
>
{: .hands_on}

This should result in the following violin plot, where we see the distribution of the number of genes per cell (`n_genes_by_counts`) and the library size per cell, where the library size is the total number of mRNA in a cell, regardless of which gene it came from (`total_counts`).

!["Violin Plots per Batch"]({% link topics/single-cell/images/scrna-plant/violin1.png %})

Each dot is a cell, and the x-axis has two groups showing the absolute values of for each batch (`shr` and `wt`). The y-values correspond to the actual value of each cell, but the x-vales within each group are just random jitter to help see the cells more clearly. The outline of the violin plot shows us the density of each batch, where we can see that most of the cells in the `shr` group have an average number of features just below 2000 genes, compared to `wt` which has an average number of features just below 4000 genes. If we compare the library size, however, we notice that the outlines are very similar and that the average library sizes across batches are comparable.

### Filter

Most scRNA-seq datasets need to set a minimum threshold of detectability in order to ensure that the cells are of high enough quality to be used for analysis. Conversely, sometimes an upper threshold also should be set to counter the case where a cell is a cell-doublet (in the sense that two cells were captured and sequenced accidentally as a single cell).

The above violin plots suggest quite a few outliers, mostly on the upper-end of the density plots, suggesting a few cell-doublets in the data. The effect of these is not something we cannot easily regress out, and so we must filter them out instead. By looking at the plots, we can see that a good maximum library size here would be 100 000 transcripts, and a good maximum feature size would be 10 000 transcripts.

For this analysis, we will set a minimum threshold of detectability that each cell should count transcripts from at least 200 genes, and that each gene should be expressed in at least 5 cells. These lower-bound thresholds are not so easy to derive from such QC plots, but depend mostly on the type of data you are using. If the number of features and the library sizes were one order of magnitude higher, one might consider also scaling the lower-bound thresholds by one order of magnitude.

> <hands-on-title>Filtering</hands-on-title>
>
> 1. {% tool [Filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - *"Filter"*: `Minimum number of genes expressed`
>            - *"Minimum number of genes expressed required for a cell to pass filtering"*: `100`
>
> 1. {% tool [Filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Filter** {% icon tool %})
>    - *"Method used for filtering"*: `Filter genes based on number of cells or counts, using 'pp.filter_genes'`
>        - *"Filter"*: `Minimum number of cells expressed`
>            - *"Minimum number of cells expressed required for a gene to pass filtering"*: `2`
>
>    We now set the upper limits to the analysis.
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Filter** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `n_genes_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `12000`
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `total_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `120000`
>
{: .hands_on}

If we inspect the resulting dataset, we see that the number of cells remaining is `5826` and the number of genes is `18913`, which is still a good number of cells and genes. It is good practice to save the raw data in case we need to use it data, so we "freeze the raw data" into a slot for later safe keeping.

> <hands-on-title>Saving the original matrix</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Filter** {% icon tool %})
>    - *"Function to manipulate the object"*: `Freeze the current state into the 'raw' attribute`
>
{: .hands_on}


## Confounder Removal

Now we have a set of cells we are reasonably confident contain desirable biological variation (please view the  following segment from [*"An introduction to scRNA-seq data analysis"*]({% link videos/watch.md %}?v=single-cell/tutorials/scrna-intro/slides&t=11:28) as a reference for the types of wanted and unwanted variation). We wish to now regress out the library size variation, as it is not indicative of cell type. To do this, we will first ensure that all cells are of the same comparable library size by applying individual size factors to each cell.

We will also apply a log transformation to the cells, as this compresses the variation into a less extreme space, making it easier to see the relative differences between cells.

After normalising and regressing out unwanted factors, we will then scale the data to have unit variance with a zero mean, so that the mean expression of each gene is not a factor in the analysis and only the variation is, ensuring that the later clustering is driven only by the relative variation of the genes, and not neccesarily how expressive those genes are.

> <tip-title>Highly Variable Genes?</tip-title>
>
> Those of you who are familiar with the [ScanPy Tutorial]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) might wonder why we have not reduced the number of genes by performing a highly variable gene selection.
> The answer is simply that it did not help with this particular dataset, and that by removing the least variable genes in the analysis, it did help us replicate the analysis in the paper. Try it for yourself as an intermediate step (after this analysis) and see!
{: .tip}

> <hands-on-title>Normalize and Scale</hands-on-title>
>
> 1. {% tool [Normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>        - *"Target sum"*: `10000.0`
>        - *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Normalize** {% icon tool %})
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
>
> 1. {% tool [Remove confounders](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_remove_confounders/scanpy_remove_confounders/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used for plotting"*: `Regress out unwanted sources of variation, using 'pp.regress_out'`
>        - *"Keys for observation annotation on which to regress on"*: `total_counts`
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Remove confounders** {% icon tool %})
>    - *"Method used for inspecting"*: `Scale data to unit variance and zero mean, using 'pp.scale'`
>        - *"Maximum value"*: `10.0`
>
{: .hands_on}


## Dimensionality Reduction

Dimensionality reduction is the art of reducing a high dimensional dataset into a low dimensional "embedding" that humans can actually see (i.e. 2 or 3 dimensions), ideally such that the relationships or distances between data points are preserved in this embedding. In the context of single-cell datasets, this essentially means compressing > 10 000 genes into just 2 X/Y variables.


> <tip-title>What is Dimensionality Reduction?</tip-title>
>
> You can learn more about dimensionality reduction by consulting the following segment from [*"An introduction to scRNA-seq data analysis"*]({% link videos/watch.md %}?v=single-cell/tutorials/scrna-intro/slides&t=13:46).
>
{: .tip}

This is usually a two step process:

1. A Principal Component Analysis (PCA) is used to perform an optimized "rotation" of the ~ 20 000 "unit" gene axes that we have, in order to better fit the data. These new axes or "principal components" are then *linear* combinations of the original unit axes, with an associated score of how variable each new axis is. By sorting these principal components by most variable to least variable, we can select the top N components and discard the rest, leaving us with most of the variation still in the data. Usually we select the top 40 principal components, which from 20 000 is a *huge* reduction in the dimensionality of the dataset.
1. We then perform a more complex kind of dimensionality reduction, one that does not assume any linearity in the axes. For scRNA-seq, this is either tSNE or UMAP, with UMAP being the main choice due to how flexible it is in incorporating new data. Though UMAP is capable of working on extremely high dimensional datasets, it is often limited by time and space constraints (read: the computer does not respond in a reasonable timeframe, or it crashes), and so therefore it is quite normal to feed UMAP the output of the PCA as input. This will then be a dimensionality reduction from 50 to 2. With these final 2 dimensions, we can plot the data.

> <hands-on-title>PCA and UMAP</hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'tl.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>            - *"SVD solver to use"*: `ARPACK wrapper in SciPy`
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for inspecting"*: `Compute a neighborhood graph of observations, using 'pp.neighbors'`
>        - *"The size of local neighborhood (in terms of number of neighboring data points) used for manifold approximation"*: `10`
>        - *"Number of PCs to use"*: `40`
>
>    > <comment-title></comment-title>
>    >
>    > UMAP relies on a connected graph of cells to operate. Please view the following segment from [*"An introduction to scRNA-seq data analysis"*]({% link videos/watch.md %}?v=single-cell/tutorials/scrna-intro/slides&t=13:40) for more information on how this process works.
>    {: .comment}
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used"*: `Embed the neighborhood graph using UMAP, using 'tl.umap'`
>
{: .hands_on}

With our data now sufficiently "flat" and ready for human consumption, we can now do a naive plot of both the PCA and the UMAP and show cell assignments coloured by batch.

> <hands-on-title>Plot the PCA and UMAP by Batch</hands-on-title>
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in PCA coordinates, using 'pl.pca`
>      - *"Keys for annotations of observations/cells or variables/genes"*: `batch`
>      - In *"Plot attributes"*
>        - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>      - *"Keys for annotations of observations/cells or variables/genes"*: `batch`
>      - *"Show edges?"*: `No`
>      - In *"Plot attributes"*
>        - *"Legend font size"*: `14`
>        - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
{: .hands_on}

| PCA                                | UMAP                                 |
|:----------------------------------:|:------------------------------------:|
| ![PCA]({% link topics/single-cell/images/scrna-plant/pca_batch.png %}) | ![UMAP]({% link topics/single-cell/images/scrna-plant/umap_batch.png %})|


From this, we can see that there is a reasonable overlap in our batches shown both in the PCA and UMAP embeddings. This is good because it shows that though there is *some* batch effect (i.e. cells from one batch appear to cluster on a different side of the plot than the other) it is not significant enough for there not to be some commonality between the batches.

There are batch correction algorithms for cases where one batch clusters completely seperate from the other, but this is not necessary here.

# Finding Cell Types

## Clustering

Let us cluster the cells and see what cell types we can discover in the plots. The cell types that we wish to recover are those shown in the original paper, as highlighted by the gene markers in the DotPlot below.

| UMAP                                   | DotPlot                                    |
|:--------------------------------------:|:------------------------------------------:|
| ![PCA]({% link topics/single-cell/images/scrna-plant/clusters_expected.png %}) | ![UMAP]({% link topics/single-cell/images/scrna-plant/dotplot_expected.png %})|


> <hands-on-title>Generate and Plots Clusters</hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used"*: `Cluster cells into subgroups, using 'tl.leiden'`
>        - *"Coarseness of the clusterin"*: `0.35`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `leiden, batch`
>        - *"Show edges?"*: `No`
>        - In *"Plot attributes"*:
>            - *"Location of legend"*: `on data`
>            - *"Legend font size"*: `14`
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
>    > <comment-title></comment-title>
>    >
>    > We print the legend on the data because it's easier to see where the cluster labels apply.
>    {: .comment}
>
{: .hands_on}

![UMAP Leiden Clusters]({% link topics/single-cell/images/scrna-plant/clusters_found.png %})

Here we have recovered 13 clusters but we don't yet know what types they describe. If we have a list of genes that we know are indicative of a certain cell type (i.e. are marker genes) then we can use this to assign labels to our clusters.

Let us here try to recreate the DotPlot from the paper using the clusters we have discovered.

## DotPlot and Validating Cell Types

> <hands-on-title>Dot Plot</hands-on-title>
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Generic: Makes a dot plot of the expression values, using 'pl.dotplot'`
>        - *"Variables to plot (columns of the heatmaps)"*: `Subset of variables in 'adata.var_names'`
>            - *"List of variables to plot"*: `AT1G26680,COBL2,AT3G55180,NCED2,AT5G22550,ATL63,RGF8,TEL1,DUF9,AGL42,PLT1,PLT2,PER3,PER9,PBL15,TBL30,AT2G48130,AT3G22620,AED3,T3P18-7,MBK21-8,NPF6-4,CYP71B34,IAMT1,AT1G31950,F14G6-22,GL2,ANL2,PMEI4,MUD21-4,EXPA7,MES15,ADF8,COBL9,TPR14,BHLH144,MYB83,SCPL35,MYB46,COBL4,PIN7,PIN3   `
>        - *"The key of the observation grouping to consider"*: `leiden`
>        - *"Number of categories"*: `9`
>        - *"Use 'raw' attribute of input if present"*: `Yes`
>        - In *"Group of variables to highlight"*:
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `0`
>                - *"End"*: `5`
>                - *"Label"*: `Columella`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `6`
>                - *"End"*: `9`
>                - *"Label"*: `QC`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `10`
>                - *"End"*: `11`
>                - *"Label"*: `NC`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `12`
>                - *"End"*: `17`
>                - *"Label"*: `Endodermis`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `18`
>                - *"End"*: `23`
>                - *"Label"*: `Cortex`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `24`
>                - *"End"*: `29`
>                - *"Label"*: `Atrichoblast`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `30`
>                - *"End"*: `34`
>                - *"Label"*: `Trichoblast`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `35`
>                - *"End"*: `39`
>                - *"Label"*: `Xylem`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `40`
>                - *"End"*: `41`
>                - *"Label"*: `VC`
>        - *"Custom figure size"*: `No: the figure width is set based on the number of variable names and the height is set to 10.`
>
{: .hands_on}

By running the above we end up with the following DotPlot:

![Dotplot In Analyssis]({% link topics/single-cell/images/scrna-plant/dotplot_found.png %})

Notice how we have the Columella, QC and NC sharing the same cluster 11, that the Endodermis is localised in cluster 8, the Cortex in cluster 9, the Trichoblast cells in cluster 3, the Xylem showing strong expression in cluster 12, VC in clusters 4 and 6, and the Atrichoblasts have a mixed expression in a range of clusters. These characteristics are the same as the Dotplot in the original paper.

We can use this Dotplot as a guide to relabel our clusters and give more meaningful annotations to the plots.

## Relabel clusters

> <hands-on-title>Cluster Re-labelling</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Function to manipulate the object"*: `Rename categories of annotation`
>        - *"Key for observations or variables annotation"*: `leiden`
>        - *"Comma-separated list of new categories"*: `0, 1, 2, trichoblasts, 4, 5, 6, atrichoblasts, endodermis, cortex, 10, columella+QC+NC, xylem`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `leiden, batch`
>        - *"Show edges?"*: `No`
>        - In *"Plot attributes"*:
>            - *"Location of legend"*: `on data`
>            - *"Legend font size"*: `14`
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
{: .hands_on}

![Relabelled clusters]({% link topics/single-cell/images/scrna-plant/clusters_found_relabelled.png %})


If we look at the clusters now, and compare them to the original image in the paper, we can infer that the meristematic cells are likely to be derived from cluster 1, which shares soft clustering with the trichoblasts suggesting a trajectory pathway which could be explored.


# Conclusions

In this tutorial, we have recapitulated the same clustering analysis in the "Spatiotemporal Developmental Trajectories in the Arabidopsis Root Revealed Using High-Throughput Single-Cell RNA Sequencing" {% cite denyer2019spatiotemporal %} paper, and validated them by comparing DotPlots for specific genes that were used as markers in that paper.

From this point, we can perform a lineage analysis to infer a differentiation pathway between the clusters. For ScanPy there is the PAGA option, however, this does not work so well with the current dataset, so it is encouraged that users use the original Seurat trajectory suite that was given in the paper, or to experiment with Monocle.

Both libraries are available within the RStudio and Jupyter Notebook libraries in the interactive Galaxy envronments which can be found in the *"Miscellaneous Tools"* section under the *"Interactive Tools"* subheading. An excellent follow-up tutorial to perform a trajectory analysis in Galaxy using Jupyter notebooks would be the [*"Trajectory Analysis using Python (Jupyter Notebook) in Galaxy"*]({% link topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.md %}).

This entire tutorial can be invoked from the scRNA Plant Workflow shown below:

![scRNA Plant]({% link topics/single-cell/images/scrna-plant/scrna-plant-workflow.png %})

The input parameters take two datasets as input, and 5 parameters using the defaults shown in this tutorial: min cells, min genes, max features, max library size, and Leiden clustering resolution. Play with these options to see how robust the analysis really is!

tools_json:
["Import Anndata and loom", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.7.5+galaxy0", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0", "Inspect and manipulate", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0", "Plot with scanpy", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0", "Filter", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.7.1+galaxy0", "Normalize", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.7.1+galaxy0", "Remove confounders", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_remove_confounders/scanpy_remove_confounders/1.7.1+galaxy0", "Cluster, infer trajectories and embed", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0"]


topic: single-cell
tutorial: scrna-case_monocle3-rstudio
title: Inferring single cell trajectories with Monocle3 (R)
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 0b91a62c2b04b2e99a2d3af95e319e0c
last_modified: 2026-05-27T09:01:49.746967
zenodo_link: https://zenodo.org/record/7455590
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_monocle3-rstudio/tutorial.html

content:
{% snippet topics/single-cell/faqs/notebook_warning.md %}

## Setting up the environment and file upload
Once the installation is done, we should load the needed packages into our notebook. Navigate back to your `notebook`. If you are using our prepopulated notebook, you can follow the tutorial from there. Otherwise, input the following into your fresh notebook.

```r
install.packages("Rcpp")    # needed for reduce_dimension to avoid AnnoyAngular error; library(Rcpp) might work as well depending on the version
```

```r
library(monocle3)
library(biomaRt)
library(magrittr)     # needed for %>%
library(viridisLite)
```

```r
install.packages("anndata")
```

Alright, the libraries are here - so now let's get our AnnData file.

## Upload, view and modify the files

We can download the file into Jupyter notebook also using a Zendo link. The first argument is the download link and the second one is the name under which the downloaded file is saved:
```r
download.file('https://zenodo.org/record/7455590/files/AnnData_filtered.h5ad', 'AnnData.h5ad')
```
Then we have to read in this h5ad file:
```r
ann <- anndata::read_h5ad('AnnData.h5ad')
```
Now we store all the information we need in AnnData object. However, Monocle uses *cell_data_set class* to hold expression data, which requires three input files: `expression_matrix`, `cell_metadata` and `gene_metadata`. Therefore, we have to somehow 'transfer' that information from our AnnData object to Monocle's cell_data_set (cds). AnnData stores a data matrix `X` together with annotations of observations `obs` and variables `var`, so we can extract those parameters and use them for further analysis.

```r
expression_matrix <- ann$X
cell_metadata <- ann$obs
gene_metadata <- ann$var
```

> <tip-title>Uploading files from your computer is also an option!</tip-title>
>
> If you already have files containing the expression matrix, genes and cells metadata, you can upload them to JupyLab and generate a cds file from them instead. For example, if you first downloaded the files from Galaxy, your files will have the `.tabular` extension. In this case, we will use the `read.delim()` function to read them in. In this function, the first argument is the file path - in our case, the files are in the same folder as the notebook, so the file path is the same as the file name. You can always check that by right-clicking on the file and choosing `Copy path`. The second argument, `row.names=1` takes the column number of the data file from which to take the row names.
> ```r
> # read in the files
> cell_metadata <- read.delim('cells.tabular', row.names=1)
> gene_metadata <- read.delim('genes.tabular', row.names=1)
> expression_matrix <- read.delim('expression.tabular', row.names=1)
> ```
>
> > <question-title></question-title>
> >
> > Why should we set `row.names=1`?
> >
> > > <solution-title></solution-title>
> > >
> > > This allows us to ensure that the expression value matrix has the same number of columns as the `cell_metadata` has rows and the same number of rows as the `gene_metadata` has rows. Importantly, the row names of the `cell_metadata` object should match the column names of the expression matrix and the row names of the `gene_metadata` object should match the row names of the expression matrix.
> > >
> > {: .solution}
> {: .question}
>
{: .tip}


><tip-title>Reading in the files for RStudio users</tip-title>
>
> If you are working in RStudio Cloud, you have to click on ‘Upload’ button in the right bottom window toolbar and choose already downloaded data files to upload. You should now see all three files in this window. You might want to rename the files to make their names shorter.
>
> ![Screenshot of Files tab in RStudio, highlighting 'Upload' and 'Rename' buttons and listing three uploaded and renamed files: 'cell_metadata', 'gene_metadata', 'expression_matrix'.](../../images/scrna-casestudy-monocle/r_files_tab.png "The view of the Files tab with uploaded files and highlighted relevant buttons.")
>
> If you are using RStudio Galaxy tool, you can get data directly from your history by running:
> ```
># get the files from Galaxy history
>file.copy(gx_get(2), "cell_metadata")
>file.copy(gx_get(3), "gene_metadata")
>file.copy(gx_get(4), "expression_matrix")
>```
>The number in the brackets corresponds to the dataset number in your history, so make sure you put the right number for the corresponding file. We can specify the name of the fetched files in the quotation marks, as shown above. All three files should appear in the Files tab window.
>
> If you are using RStudio locally, then you don’t have to bother about uploading the files – just run `file.choose()` and choose the corresponding file which you want to get path to:
>```
># get the file path
>cells_path <- file.choose()
>genes_path <- file.choose()
>expression_path <- file.choose()
>```
> You should now see the new variables in the Environment tab window.
>
{: .tip}


According to the [Monocle3 documentation](https://cole-trapnell-lab.github.io/monocle3/docs/starting/), the `expression_matrix` should have genes as rows and cells as columns. Let's check if that's the case here.
```r
head(expression_matrix,c(5, 5))           # preview the content of the file by calling its the first 5 rows by 5 columns
```
We can see that in our matrix rows are cells and genes are columns, so we have to transpose the matrix simply using function `t()`. But before doing so, we will change its type from dataframe to matrix - this is Monocle's requirement to generate cell_data_set afterwards.
```r
expression_matrix <- as.matrix(expression_matrix)   # change the type to matrix
expression_matrix <- t(expression_matrix)           # transpose the matrix
```
Another condition we have to satisfy is that one of the columns of the `gene_metadata` should be named "gene_short_name", which represents the gene symbol for each gene. Some functions won't work without that. Do we have such a column? Let's check.
```r
head(gene_metadata)             # preview the top ten rows of the file to check the name of the column containing gene symbols
```

The second column indeed contains gene symbols, but is called "Symbol" instead of "gene_short_name". That can be easily changed by a simple assignment, as long as we know the number of the column that we want to modify. In our case the gene symbols are stored in column 2. We can access the column names by `colnames()`.
```r
colnames(gene_metadata)[2] <- 'gene_short_name'     # change the column name
colnames(gene_metadata)                             # see the changes
```

## Generating CDS object
Now let’s store our files in one object – the `cell_data_set`. This is the main class used by Monocle to hold single cell expression data. The class is derived from the Bioconductor SingleCellExperiment class. Similar to Python's AnnData, the cell_data_set stores a data matrix together with annotations of observations and variables. There are three ways of creating CDS object in monocle:
-	Using ```new_cell_data_set() ``` function with three data frames as arguments (not their paths!): expression matrix (can also be a sparseMatrix), cell metadata and gene metadata
-	Using ```load_cellranger_data() ``` function providing the path to the folder containing 10X Genomics Cell Ranger output files. This function takes an argument `umi_cutoff` that determines how many reads a cell must have to be included
-	Using ```load_mm_data() ``` function providing the paths to matrix file and two metadata files (features and cell information).

In this tutorial we will use the first option:
```r
# create CDS object
cds <- new_cell_data_set(expression_matrix, cell_metadata, gene_metadata)
```
We are now ready to process our data!

> <details-title>Format conversion</details-title>
>
>  Since Monocle’s CDS object is analogous to Python's AnnData, why don’t we use some kind of conversion between those two formats? There is indeed a package called `sceasy` that helps easy conversion of different single-cell data formats to each other. However, when we tested this conversion on our dataset and then used Monocle to plot the expression of genes, the plots were not correct – the expression was shown to be identical throughout the sample. For comparison, Seurat did well when plotting gene expression of the same converted object! Although conversion functions are very handy, you have to be aware that their output might be interpreted differently by certain packages. Therefore, to make sure that the analysis is reliable, we decided to generate CDS object directly using Monocle’s function.
> ![Comparison between plots of gene expression generated by Monocle and Seurat using the CDS object that was converted from AnnData using SCEasy tool. Gene expression in Monocle plots is identical throughout the sample, while the expression of the same genes plotted by Seurat is noticeable only in specific clusters.](../../images/scrna-casestudy-monocle/monocle_seurat.png "Comparison between plots of gene expression generated by Monocle and Seurat using the CDS object that was converted from AnnData using SCEasy tool.")
>
{: .details}

# Additional step: adding genes symbols based on their IDs
> <warning-title>Additional step</warning-title>
>  This step is not necessary for the dataset we are working on but some users might find it helpful when analysing their own data.
>
{: .warning}

If you remember the very first tutorial in the case study, we started with gene IDs and added gene symbols based on the Ensembl GTF file.  
But what if we didn’t have the genes symbols in our CDS object and wanted to add them now? Of course - it's possible! We will also base this annotation on Ensembl - the genome database – with the use of the library BioMart. We will use the same archive as in the Alevin tutorial (Genome assembly GRCm38) to get the gene names. Please note that the updated version (GRCm39) is available, but some of the gene IDs are not in that EnsEMBL database. The code below is written in a way that it will work for the updated dataset too, but will produce ‘NA’ where the corresponding gene name couldn’t be found.
```r
cds_extra <- cds                                        # assign our CDS to a new object for the demonstration purpose
head(rownames(fData(cds_extra)))                        # preview of the gene IDs as rownames
```
```r
# get relevant gene names
library("biomaRt")                                      # load the BioMart library
ensembl.ids <- rownames(fData(cds_extra))               # fData() allows to access cds rowData table
mart <- useEnsembl(biomart = "ENSEMBL_MART_ENSEMBL")    # connect to a specified BioMart database and dataset hosted by Ensembl
ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl", host='https://nov2020.archive.ensembl.org') 	

# The line above connects to a specified BioMart database and dataset within this database.
# In our case we choose the mus musculus database and to get the desired Genome assembly GRCm38,
# we specify the host with this archive. If you want to use the most recent version of the dataset, just run:
# ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl")
```
```r
genes <- getBM(attributes=c('ensembl_gene_id','external_gene_name'),
               filters = 'ensembl_gene_id',
               values = ensembl.ids,
               mart = ensembl_m)

# The line above retrieves the specified attributes from the connected BioMart database;
# 'ensembl_gene_id' are genes IDs,
# 'external_gene_name' are the genes symbols that we want to get for our values stored in ‘ensembl.ids’.
```
```r
# see the resulting data
head(genes)                          
```
```r
# replace IDs for gene names
gene_names <- ensembl.ids	 
count = 1 	 
for (geneID in gene_names)
{
 index <- which(genes==geneID)    # finds an index of geneID in the genes object created by getBM()
 if (length(index)==0)            # condition in case if there is no corresponding gene name in the chosen dataset
  {
    gene_names[count] <- 'NA'
  }
  else
  {
    gene_names[count] <- genes$external_gene_name[index] 	# replaces gene ID by the corresponding gene name based on the found geneID’s index
  }
 count = count + 1                # increased count so that every element in gene_names is replaced
}
```
```r
# store the gene names in our CDS object in a new column gene_short_name_extra
fData(cds_extra)$gene_short_name_extra <- gene_names
```
```r
# see the changes
fData(cds_extra)                    
```

If you are working on your own data and it’s not mouse data, you can check available datasets for other species and just use relevant dataset in `useMart()` function.
```r
listDatasets(mart)                # available datasets
```

> <warning-title>Ensembl connection problems</warning-title>
> Sometimes you may encounter some connection issues with Ensembl. To improve performance Ensembl provides several mirrors of their site distributed around the globe. When you use the default settings for useEnsembl() your queries will be directed to your closest mirror geographically. In theory this should give you the best performance, however this is not always the case in practice. For example, if the nearest mirror is experiencing many queries from other users it may perform poorly for you. In such cases, the other mirrors should be chosen automatically.
>
{: .warning}

# Monocle workflow
Do you remember the Monocle workflow introduced in the previous tutorial? Here is a recap:
![Monocle workflow: scRNA-seq dataset, pre-process data (normalise, remove batch effects), non-linear dimensionality reduction (t-SNE, UMAP), cluster cells, compare clusters (identify top markers, targeted contrasts), trajectory analysis](../../images/scrna-casestudy-monocle/monocle3_new_workflow.png "Workflow provided by Monocle3 documentation")

## Pre-processing
Let’s start with normalisation and pre-processing that can be performed using the function `preprocess_cds()`. The argument `num_dim` is the number of principal components that will be used. You can check that you're using enough PCs to capture most of the variation in gene expression across all the cells in the data set. Note that “PCA” is the default method of pre-processing in Monocle3, so although we can specify this in our function, we don’t have to.

Note that this step can take awhile - around 5 or so minutes due to the high number of PCs calculated. Feel free to use a `num_dim` value around 100 to accelerate preprocessing and compare the results.

```r
# PCA pre-processing with 210 principal components
cds_preprocessing <- preprocess_cds(cds, method = "PCA", num_dim = 210)
```
```r
 # plot the variation in gene expression vs PCA components
plot_pc_variance_explained(cds_preprocessing)                              		                            
```

![Plot of variation in gene expression vs PCA components, decreasing exponentially.](../../images/scrna-casestudy-monocle/pca_plot.jpg " Plot of variation in gene expression vs PCA components.")

The plot shows that actually using more than ~100 PCs captures only a small amount of additional variation. However, if we look at how the cells are plotted on 2D graph when using different values of PCs, it is easier to visualise how the `num_dim` actually affects the output. We will use the value of 210, which, compared to the results from the previous tutorial, makes the most sense for our dataset.

![Six plots showing only the shaded shape of how the cells are clustered depending on the num_dim argument. The general trend is maintained though.](../../images/scrna-casestudy-monocle/num_dim.jpg "The "shape" of the plot showing how the cells are clustered depending on the 'num_dim' argument.")

## Batch correction and Dimensionality reduction
Our dataset actually comprises data from 7 samples, so there is a risk that batch effects will impact analysis. Batch effects are systematic differences in the transcriptome of cells measured in different experimental batches. However, we can use Monocle to deal with that!
First, let’s check how our dataset looks in terms of batch effects. We can do that by colouring the cells by batch.  This information is stored in our CDS object from the `cell_metadata` file. Before asking Monocle to plot anything, let’s check the exact column name of the batch information column.
```r
colnames(colData(cds_preprocessing)) 	          # check column names
```
In our case it’s indeed ‘batch’, but your data might have another name (eg. "plate", etc.), so make sure you put the correct argument value.

In order to plot our cells on a 2D graph, we need to reduce the numbers of dimensions. The [previous tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}) introduced the methods of dimensionality reduction in Monocle. You can replicate what we did in Galaxy to compare the output of dimensionality reduction using different methods, simply by changing the `reduction_method` argument. Options currently supported by Monocle are "UMAP", "tSNE", "PCA", "LSI", and "Aligned". However, as for now, let’s just recall that UMAP gave the best results, so we will use UMAP here as well.
```r
# reduce dimension first
cds_preprocessing_UMAP <- reduce_dimension(cds_preprocessing, reduction_method = "UMAP", preprocess_method = "PCA")
```
```r
# then produce a plot to check for batch effects
plot_cells(cds_preprocessing_UMAP, color_cells_by="batch", label_cell_groups=FALSE)
```

We can see that upper and lower right branches mostly consist of N705 and N706, so indeed batch correction might be helpful. Let’s run this.
```r
# perform batch correction
cds_batch <- align_cds(cds_preprocessing_UMAP, preprocess_method = "PCA", alignment_group = "batch")
```

To see the changes, we have to run UMAP again, but this time on the aligned dataset. We will specify that `preprocess_method ` as "Aligned" and not “PCA". Monocle would use the "Aligned" argument automatically if no `preprocess_method` was specified.
```r
# dimensionality reduction after alignment
cds_red_dim <- reduce_dimension(cds_batch, preprocess_method = "Aligned", reduction_method = "UMAP")  
```
```r
# see the batch correction effect on a plot
plot_cells(cds_red_dim, color_cells_by="batch", label_cell_groups=FALSE)		
```

![Left image showing dataset before batch correction: upper and lower right branches mostly consist of N705 and N706. Right image showing the dataset after batch correction: the cells from all the samples are evenly spread throughout the whole dataset.](../../images/scrna-casestudy-monocle/batch_correction.png "Comparison of the dataset before and after batch correction.")

> <question-title></question-title>
>
> Does your plot look the same as the one in the Figure?
>
> > <solution-title></solution-title>
> >
> > Your plot might be slightly different to the one shown in the Figure but this is fine, as long as you see analogical patterns. Some libraries that you're using might have been updated, giving non-identical output. However, the principle behind the analysis is still the same, so you can peacefully follow the tutorial. Just keep your eyes open and... THINK!
> >
> {: .solution}
>
{: .question}

Do you see this? It’s amazing! Batch correction did a great job here! Now the dataset is nicely aligned, and the cells from all the samples are evenly spread throughout the whole dataset. It is worth mentioning that removing batch effects was done using mutual nearest neighbor alignment, a technique introduced by John Marioni's lab ({% cite Haghverdi_2018 %}) and supported by Aaron Lun's package [batchelor](https://bioconductor.org/packages/release/bioc/html/batchelor.html). Also, due to the machine learning elements of the code of this technique - as well as the fact that packages are updated regularly - your plots may not look identical to the ones pictured here. Nevertheless, the interpretation should be the same - the batch corrected plot should show better batch distribution than the uncorrected one.

Now we can move to the next step and perform dimensionality reduction.


> <tip-title>Plotting: labels vs legend</tip-title>
>
>  When creating graphs, we sometimes use labels and sometimes just a legend. You can choose whichever you think makes the data clear and readable.
>
> If you want to use a legend, then specify an argument `label_cell_groups=FALSE` in the function `plot_cells()`.
>
> The labels are set automatically, but if you want to change their size (default labels are tiny), use the argument `group_label_size`.
{: .tip}


## Clustering: clusters
We want to get some information about cell types, don’t we? In order to do so, we have to cluster our cells first.
Monocle uses a technique called "community detection" ({% cite Traag_2019 %}) to group cells. This approach was introduced by {% cite Levine_2015 %} as part of the phenoGraph algorithm.
Monocle also divides the cells into larger, more well separated groups called partitions, using a statistical test from {% cite Wolf_2019 %}, introduced as part of their [PAGA](https://github.com/theislab/paga) algorithm.

> <details-title>Clusters vs partitions</details-title>
>
> Clusters are particularly useful while trying to assign cells to a certain type, because they are based on the similarity in gene expression. The relationships between different clusters are analysed to identify possible trajectories.
>
> Partitions, meanwhile, are larger groups of cells that usually contain several clusters. Trajectory inference is performed only within one partition, so it is essential that all the cells that we want to analyse in pseudotime belong to the same partition.
>
{: .details}

Therefore, let’s perform clustering and visualise the resulting clusters.
```r
# clustering
cds_auto_cluster <- cluster_cells(cds_red_dim, reduction_method = "UMAP")
```
```r
# see the clusters
plot_cells(cds_auto_cluster, reduction_method = "UMAP", color_cells_by = 'cluster', group_label_size = 5)     
```

When using standard igraph louvain clustering, the value of resolution parameter is by default set to NULL, which means that it is determined automatically. Although the resulting clusters are OK, it would be nice to get some more granularity to identify cell types more specifically. The higher the resolution value, the more clusters we get. We will set the resolution value to 0.0002, but you are very welcome to try different values to see the changes.

```r
# clustering with changed resolution value
cds_clustered <- cluster_cells(cds_red_dim, reduction_method = "UMAP", resolution = 0.0002)
```
```r
# see the new clusters
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'cluster', group_label_size = 5) 	
```

![Left image showing 6 clusters formed using automatic standard igraph louvain clustering. Right image showing the dataset with clusters formed using resolution argument set to 0.0002: now there are 7 clusters, as one automatically formed cluster could be divided into two smaller distinct clusters.](../../images/scrna-casestudy-monocle/clusters_compare.png "Comparison of the clusters formed using standard igraph louvain clustering and using resolution argument set to 0.0002.")



## Clustering: partitions
OK, what about partitions? They were also created during the clustering step and it’s important to check them before calculating the trajectory because it is performed only within one partition. It is essential that all the cells that we want to analyse in pseudotime belong to the same partition.
```r
# see the partitions
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```

While your plot might be slightly different due to package updates, we can see that there are 3 partitions identified in `cds_clustered` object. Ideally, we would like to combine partitions 1 and 2 to draw a trajectory through all those cells (we can ignore cells in outlier partition). Sometimes using the default values might result in multiple partitions while you only need one. Then you would have to change the q-value cutoff in `partition_qval`. The default is 0.05 and by increasing this value you can increase the span of partitions, meaning that you would get fewer partitions. When trying different values of q-value, you also have to check if the clusters didn't change. It's all about finding a balance between the value of `resolution` and `partition_qval` so that both clusters and partitions are satisfactory enough for downstream analysis. Let's try that on our dataset.

```r
# changing the partition q-value
cds_clustered <- cluster_cells(cds_red_dim, reduction_method = "UMAP", resolution = 0.0002, partition_qval = 1)
```
```r
# see the partitions with the changed q-value
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```
```r
# check if clusters didn't change
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'cluster', label_cell_groups=FALSE)
```

> <question-title></question-title>
>
> Have the clusters change after changing the partition q-value?
>
> > <solution-title></solution-title>
> >
> > It might be the case that after changing partition q-value, you will notice that additional clusters appeared. In that situation, you might either play around the `resolution` and `partition_qval` values, go forward with the current clustering (adjusting the parameters accordingly), or check the other method of assigning cells to one partition given below.
> >
> {: .solution}
>
{: .question}

Now we have cells of interest in one partition, we still have reasonable clusters, so now we can learn the trajectory. However, in some cases even this method might not be enough. Then, there is a last resort… assigning cells to a partition manually.

## Additional step: assigning cells to one partition
> <warning-title>Additional step</warning-title>
>  This step is not necessary for the dataset we are working on but some users might find it helpful when analysing their own data.
>
{: .warning}

Let’s assume we have 4 partitions that cannot be extended to one big partition using `partition_qval` (it might sound unreasonable but it does happen!) and we are desperate to have all the cells in one partition to draw a trajectory through all of them. We can simulate the initial dataset by setting `partition_qval` value to 0.0001.
```r
# simulate the dataset
cds_partitions_extra <- cluster_cells(cds_red_dim, reduction_method = "UMAP", partition_qval = 0.0001)		
```
```r
# see the simulated partitions
plot_cells(cds_partitions_extra, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```
```r
# store ‘1’ the number of times equal to the number of cells
big_partition <- c(rep(1,length(cds_partitions_extra@colData@rownames))) 	
```
```r
# take the barcode of each cell and assign ‘1’ to each of them (now the ‘ones’ are named)
names(big_partition) <- cds_partitions_extra@colData@rownames 	
```
```r
# convert from numeric to factor
big_partition <- as.factor(big_partition)
```
```r
# assign the created barcodes-partition list to the location where information about partitions is stored in CDS object
cds_partitions_extra@clusters$UMAP$partitions <- big_partition 	
```
```r
# see the new partition assignment
plot_cells(cds_partitions_extra, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```

![Left image showing the dataset divided into 4 partitions. Right image showing all cells assigned to one partition.](../../images/scrna-casestudy-monocle/partitions_compare.png "The result of the manual assignment of all the cells to one partition.")


> <details-title>@ and $ operators</details-title>
>
> As you saw above, we used `@` and `$` operators to navigate in CDS object. What is the difference between them? `$` operator is used to access one variable/column, while `@` extracts the contents of a slot in an object with a formal (S4) class structure. If you use `View()` function on our CDS object, you will see the elements of type "S4" that you can access using `@` operator. The suggestion list of elements that can be accessed should also pop up as you type the name of the object followed by the corresponding operator.
>
{: .details}

# Assigning cell types
There are two main approaches to assigning cell types to clusters that we’ve just identified – supervised and unsupervised, both based on gene expression in each cluster.

## Supervised approach
The supervised approach relies on prior knowledge of which cell types to expect. We can simply check the expression of marker genes specific to the expected cell types. Let’s then check the markers mentioned in the original paper {% cite Bacon2018 %}.

| Marker | Cell type |
|--------------------|
| Il2ra    | Double negative (early T-cell)    |
| Cd8b1, Cd8a, Cd4    | Double positive (middle T-cell)|
| Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell)|
| Itm2a    | Mature T-cell |
| Aif1    | Macrophages    |
| Hba-a1    | RBC    |

To plot the expression of all those genes across our dataset in one go, we will pass a vector with the names of those markers into a parameter `genes` in the `plot_cells` function.
```r
# expression of marker genes across the sample
plot_cells(cds_clustered, genes=c('Il2ra','Cd8b1','Cd8a','Cd4','Itm2a','Aif1','Hba-a1'), reduction_method = "UMAP")
```

![7 graphs, each showing expression of one of the genes in the corresponding clusters: 'Il2ra' – cluster 4; 'Cd8b1' and 'Cd8a' – clusters 1,6,2; 'Cd4' – high expression in cluster 5; 'Itm2a' – cluster 3; 'Aif1' – barely anything, no specific cluster assigned; 'Hba-a1' - throughout the entire sample in low numbers.](../../images/scrna-casestudy-monocle/genes.png "Expression of the reference marker genes across analysed sample.")


> <question-title>Genes, cell types and clusters</question-title>
> Based on the gene expression graph that we just generated, the table above and your knowledge from the previous tutorials, how would you assign the clusters?
>
> > <solution-title></solution-title>
> >
> > Keep in mind that these results refer to our numbered clusters, while yours might be slightly different.
> > - `Il2ra` (DN): mostly expressed in cluster 4
> > - `Cd8b1, Cd8a` (DP middle): expressed in clusters 1, 6, and highly in cluster 2
> > - `Cd4` (DP late): average expression in clusters 1, 6, 2 and high expression in cluster 5
> > - `Itm2a` (T-mat): expressed in cluster 3
> > - `Aif1` (macrophages): barely anything here, minimal expression spread across the sample with some more cells in cluster 4 and 3 – not enough to form a distinct cluster though). In theory, we shouldn’t have any macrophages in our sample. If you remember from the previous tutorials, we actually filtered out macrophages from the sample during the processing step, because we worked on annotated data. When analysing unannotated data, we could only assign macrophages and then filter them out, provided that Monocle clusters them into a separate group. As you can see, it’s not the case here, so we will just carry on with the analysis, interpreting this as a contamination.
> > - `Hba-a1` (RBC): appears throughout the entire sample in low numbers suggesting some background contamination of red blood cell debris in the cell samples during library generation, but also shows higher expression in a distinct tiny bit of cluster 3, at the border between clusters 1 and 5. However, it’s too small to be clustered into a separate group and filtered out in this case.
> > If you remember, this gene was found to be expressed in the previous Scanpy tutorial also in low numbers across the sample, and in the other Monocle tutorial (using Galaxy tools and annotated data) algorithms allowed us to gather the cells expressing that gene into a distinct group. Our result now sits somewhere in between.
> > ![In Scanpy graph the marker gene appears throughout the entire sample in low numbers, in Monocle in Galaxy cells expressing hemoglobin gene were grouped into a small branch of DP-M4, allowing to group those cells. Monocle in RStudio graph is somewhere in between showing mostly low expression across the sample, but also having a tiny bit of grouped cells, less distinct than in Galaxy though.](../../images/scrna-casestudy-monocle/hb_all.png "Hemoglobin across clusters - comparison between Scanpy, Monocle using Galaxy tools and Monocle run in RStudio.")
> >
> {: .solution}
{: .question}

Having identified which cluster corresponds to a specific cell type, we can finally run some code to add the annotation to our CDS object. First, we will create a new column called `cell_type`  in `colData()` - this is where the information about the cells is stored (eg. batch, genotype, sex, etc) - and initialize it with the values of clusters. Then, we will get the `dplyr` package which will be used for cluster annotation.

```r
# just to keep the objects tidy and not overwrite them so that you can come back to any point of the analysis
cds_annotated <- cds_clustered 	
```
```r
# create a new column ‘cell_type’ and initialise it with clusters values
colData(cds_annotated)$cell_type <- as.character(clusters(cds_annotated)) 	
```
```r
# annotate clusters
colData(cds_annotated)$cell_type <- dplyr::recode(colData(cds_annotated)$cell_type,
                                                       '1'='DP-M1',   # double positive – middle T-cell (1st cluster)
                                                       '2'='DP-M2',	  # double positive – middle T-cell (2nd cluster)
                                                       '3'='T-mat',	  # mature T-cell
                                                       '4'='DN',		  # double negative – early T-cell
                                                       '5'='DP-L',	  # double positive – late middle T-cell
                                                       '6'='DP-M3',	  # double positive – middle T-cell (3rd cluster)
                                                       '7'='Unknown') # no info for now, so call it ‘Unknown’
                                                       '8'='Unknown') # no info for now, so call it ‘Unknown’
```
```r
# check the annotation
plot_cells(cds_annotated, color_cells_by="cell_type", label_cell_groups=FALSE)    
```

![A plot showing the identified clusters, now each coloured by the assigned cell type.](../../images/scrna-casestudy-monocle/annotated.png "Our annotated dataset.")

## Unsupervised approach
But what if we don’t have any reference that we can use to assign our clusters? In that case, we will turn to the mentioned unsupervised approach - we will check what are the specifically expressed genes for each cluster. Then we can identify the cell types by looking up what cell types contain those genes. That’s a more tedious process, but sometimes can lead to exciting and unexpected results.
We will use Monocle’s function `top_markers()` and store the information about specifically expressed genes for each cluster in the data frame `marker_test`.
```r
# find top marker genes in each cluster
marker_test <- top_markers(cds_clustered, group_cells_by="cluster", reduction_method = "UMAP", reference_cells=1000, cores=8)
```
You can group the cells by any categorical variable in `colData(cds_clustered)`. The parameter `reference_cells` is used to accelerate the marker significance test at some cost in sensitivity. It works by randomly selecting a specified number of cells and performing the marker significance test against a chosen set of cells. If your dataset is not massively big, you might skip this parameter as it wouldn’t help much.

> <question-title></question-title>
>
> What are the variables stored in `marker_test` data frame and what do they mean?
>
> > <solution-title></solution-title>
> >
> > Those are 10 variables that you can check easily either using colnames(marker_test) or `View(marker_test)` which will also display all the corresponding values.
> >- `gene_id` - Ensembl gene ID
> >- `gene_short_name` - short name of the gene corresponding to its ID
> >- `cell_group` - a group to which the cell belongs, specified in the `group_cells_by` argument
> >- `marker_score` - numeric vector of marker scores as the fraction expressing scaled by the specificity. The value ranges from 0 to 1
> >- `mean_expression` -  numeric vector of mean normalized expression of the gene in the cell group
> >- `fraction_expressing` - numeric vector of fraction of cells expressing the gene within the cell group
> >- `specificity` - numeric vector of a measure of how specific the gene's expression is to the cell group based on the Jensen-Shannon divergence. The value ranges from 0 to 1.
> >- `pseudo_R2` - numeric vector of pseudo R-squared values, a measure of how well the gene expression model fits the categorical data relative to the null model. The value ranges from 0 to 1.
> >- `marker_test_p_value` - numeric vector of likelihood ratio p-values; p-value is an area in the tail of a distribution that tells you the odds of a result happening by chance
> >- `marker_test_q_value` -  numeric vector of likelihood ratio q-values; q-value is a p-value that has been adjusted for the False Discovery Rate (FDR)
> >
> {: .solution}
>
{: .question}

We can now use data in `marker_test` to rank the cells based on one of the specificity metrics and take the top gene(s) for each cluster. We will filter the expressing cells that constitute more than 10% of the cell group and we will take one gene in each cluster with the highest `pseudo_R2` value (you can of course modify this value and choose more genes to be selected).

```r
# filter the ‘marker_test’ data frame
top_specific_markers <- marker_test %>%
                            dplyr::filter(fraction_expressing >= 0.10) %>%       # set the fraction of expressing cells
                            dplyr::group_by(cell_group) %>%                      # set a group to which the cells belong
                            dplyr::top_n(1, pseudo_R2)                           # set the number of top genes and the variable from 'marker_test' to rank by
```
```r
# store the names of the marker genes
# you can also use IDs, the conversion to gene names should happen automatically when plotting
top_marker_names <- unique(top_specific_markers %>% dplyr::pull(gene_short_name))
```

Now we have all elements to plot the expression and fraction of cells that express found markers in each group.

```r
plot_genes_by_group(cds_clustered,                    # our CDS object
                    top_marker_names,                 # a list of gene names to show in the plot
                    group_cells_by="cluster",         # how to group cells when labeling
                    ordering_type="maximal_on_diag")  # how to order the genes / groups on the dot plot
```

> <tip-title>Unexpectedly too many genes on y-axis?</tip-title>
>
>  If you notice that on your dot plot any cluster has more genes than you specified (here we set one gene per cluster `top_n(1, pseudo_R2)`), go back to the code and create a new cell, right after assigning `top_specific_markers` object and just before `top_marker_names`. In the new cell, paste and execute the following:
>  
> ```r
> top_specific_markers <- top_specific_markers %>%
>                            dplyr::distinct(pseudo_R2, .keep_all=TRUE)     # select only one row if there are multiple rows with the same value in 'pseudo_R2' column
> ```
> Then you can execute `top_marker_names` again and see how the plot looks like now. Better, right?
> This problem may arise when several genes in one cluster have the same values of specific variable from 'marker_test' (in our case we chose `pseudo_R2`). It might likely happen in small and quite unsignificant clusters.
{: .tip}


![A dot plot showing the expression of genes and fraction of cells that express found markers in each group. On the diagonal there are two genes corresponding to each cluster with the highest pseudo_R2 score in their group.](../../images/scrna-casestudy-monocle/gene_group.png "A plot of the expression and fraction of cells that express found markers in each group. Here, the two top genes from each cluster were plotted on diagonal beased on the highest pseudo_R2 score. Shown is the result of changing the value from 1 to 2 in "top_n(1, pseudo_R2)" in the code above.")

Look at this – we have identified some more marker genes specific to each cluster! However, sometimes it happens that the found genes are not as specific as one would expect, and they appear across the whole sample. Therefore, it is a good idea to plot all those marker genes and check how they appear in the bigger picture.

```r
# plot the identified genes to check their expression
plot_cells(cds_clustered, genes=c('Pcdhgc4','Pcdhga5','Gm5559','Gm10359','Ccr9','Cd8b1','Plac8', 'Il2ra', 'Cd52', 'Tmsb10', 'Mki67', 'Hmgb2', 'Pclaf', 'Npm1'), reduction_method = "UMAP")
```

![Plots showing the expression of fourteen found marker genes across the whole sample. Those genes are 'Pcdhgc4','Pcdhga5','Gm5559','Gm10359','Ccr9','Cd8b1','Plac8', 'Il2ra', 'Cd52', 'Tmsb10', 'Mki67', 'Hmgb2', 'Pclaf', 'Npm1'.](../../images/scrna-casestudy-monocle/markers_plot.png "Plots of the expression of marker genes across the sample.")

Further steps from now would include reviewing literature and checking what cell types correspond to the genes expressed in each cluster. Then you can annotate your clusters in the same way as shown above. Once you have your clusters annotated, Monocle can generate a file of marker genes for the identified cell types. This file can be then used with [Garnett](https://cole-trapnell-lab.github.io/garnett/), a software toolkit for automatically annotating cells. We will not go through this in the current tutorial, but we will generate the file of marker genes.

```r
# use ‘top_markers()’ again, now grouping cells by the assigned cell type
assigned_marker_test <- top_markers(cds_annotated,
                                             group_cells_by="cell_type",
                                             reference_cells=1000,
                                             cores=8)
```
```r
# filter these markers according to how stringent you want to be
garnett_markers <- assigned_marker_test %>%
                        dplyr::filter(marker_test_q_value < 0.05 & specificity >= 0.25) %>%
                        dplyr::group_by(cell_group) %>%
                        dplyr::top_n(5, marker_score)
```
```r
# exclude genes that are good markers for more than one cell type:
garnett_markers_filtered <- garnett_markers %>%
                        dplyr::group_by(gene_short_name) %>%
                        dplyr::filter(dplyr::n() == 1)
```
```r
# generate a file of marker genes
generate_garnett_marker_file(garnett_markers_filtered, file="./marker_file.txt")
```

A new file should appear in the ‘Files’ window. If you click on it, you will see the cell types and their corresponding marker genes, satisfying your chosen conditions.

> <tip-title>Marker genes file</tip-title>
>
>  Note that you can use the above block of code to generate file with the marker genes for unannotated CDS object to help you identify and check specifically expressed genes – you’d only have to change `group_cells_by` parameter from "cell_type" to “cluster”.
{: .tip}


> <tip-title>Only for users of local RStudio</tip-title>
>
> If you are working in RStudio locally, you might want to try a great function `choose_cells()`, which allows you to make a subset of the CDS object containing only cells of interest to investigate certain clusters more in-depth. It only works in interactive mode, so can be only used locally - when you call it, then a pop-up window will appear where you can choose cells to subset.
>```r
># create a CDS subset
>cds_subset <- choose_cells(cds_clustered)
>```
>Now the chosen cluster is stored as a separate CDS object and you can analyse it independently, using the methods described above.
>
{: .tip}

# Trajectory inference
It’s time to perform trajectory analysis! First, let’s learn the trajectory graph. With an argument `use_partition` we can specify if we want to learn a disjoint graph (value TRUE - default) in each partition, or to learn a single graph (value FALSE) across all partitions. The thing is, we can visualise the cells in pseudotime only if they belong to one partition. This is why it is important to make sure that all the cells that you want to analyse in pseudotime belong to one partition.

In one of the previous sections we've already prepared the partitions for trajectory inference - we assigned all the cells of interest into one partition by changing the q-value. As a result, we got two partitions – one containing only the cells that we labeled as ‘Unknown’ and another including all other cells. That is a perfect assignment – we’d like to focus only on maturation of T-cells, so we don’t want to connect those two partitions (therefore we specify `use_partition=TRUE` to get a disjoint graph) and we will work downstream only on the graph going through annotated T-cells.

```r
# learn trajectory graph
cds_trajectory <- learn_graph(cds_annotated, use_partition=TRUE)
```
```r
# visualise the learned trajectory
plot_cells(cds_trajectory,
           color_cells_by = "cell_type",
           label_cell_groups=FALSE,
           label_groups_by_cluster=FALSE,
           label_leaves=FALSE,
           label_branch_points=FALSE)
```

![Learned trajectory graph connecting all the annotated clusters except the ‘Unknown’ group of cells.](../../images/scrna-casestudy-monocle/learned_trajectory_r.png "Learned trajectory graph.")

We have to tell Monocle where to start ordering the cells, ie. when we expect the analysed biological process to begin. Thanks to our biological knowledge, we know that the beginning of the trajectory should be at DN cluster.
There are a couple of ways to specify the root cells:

## Option 1: Root nodes
Here, you will use `root_pr_nodes` argument in `order_cells()` function.

To find the names of the principal points, you have to plot the learned trajectory again, specifying `label_principal_points = TRUE`
```r
# specifying root cells: `root_pr_nodes` argument - check the principal points
plot_cells(cds_trajectory,
          color_cells_by = "cell_type",
          label_cell_groups=FALSE,
          label_groups_by_cluster=FALSE,
          label_leaves=FALSE,
          label_branch_points=FALSE,
          label_principal_points = TRUE,       # set this to TRUE
          graph_label_size=3)
```
You can see now the principal points and their labels in the form `Y_number`. Pick the principal point in the cluster that you expect to be the beginning of the trajectory and type its name in the `root_pr_nodes` argument when calling `order_cells()` function.

```r
# specifying root cells: `root_pr_nodes` argument - use the relevant principal point
cds_order_1 <- order_cells(cds_trajectory, root_pr_nodes='Y_14')
```

There is also a helper function to identify the root principal points based on the annotated cell types. This function uses `pr_graph_cell_proj_closest_vertex` which is just a matrix with a single column that stores for each cell, the ID of the principal graph node it's closest to.

```r
# a helper function to identify the root principal points
get_correct_root_state <- function(cds, cell_phenotype, root_type){
      cell_ids <- which(pData(cds)[, cell_phenotype] == root_type)

      closest_vertex <- cds@principal_graph_aux[["UMAP"]]$pr_graph_cell_proj_closest_vertex
      closest_vertex <- as.matrix(closest_vertex[colnames(cds), ])
      root_pr_nodes <- igraph::V(principal_graph(cds)[["UMAP"]])$name[as.numeric(names
                                                                    (which.max(table(closest_vertex[cell_ids,]))))]       
      root_pr_nodes
      }
```
```r
# call the function to automatically find the node in the principal graph where our DN cells reside
DN_node_id = get_correct_root_state(cds_trajectory, cell_phenotype = 'cell_type', "DN")

DN_node_id      # check the node found
```
```r
# order cells using the helper function output
cds_order_1_helper <- order_cells(cds_trajectory, root_pr_nodes = DN_node_id)
```

## Option 2: Root cells
Here, you will use `root_cells` argument in `order_cells()` function.

Specify a vector of starting cell IDs. You can provide only one cell as well as all cells of a given type.

```r
# find the names of all cells belonging to a certain type, identified as a beginning of a trajectory
starting_cell_type <- 'DN'
index <- which(cds_trajectory@colData$cell_type == starting_cell_type)
DN_cells <- colnames(cds_trajectory)[index]
```
```r
# alternatively, if you work on unannotated data, you can use the number of the cluster that should be used as the beginning of the trajectory and pass it in the ‘root_cells’ argument
starting_cluster <- colnames(cds_trajectory[,clusters(cds_trajectory) == 4])
```
```r
# order cells
cds_order_2 <- order_cells(cds_trajectory, root_cells = DN_cells)
```

> <tip-title>Only for users of local RStudio</tip-title>
>
> If you are working in RStudio locally, you can use `order_cells()` function in the interactive mode. The pop-up window should appear and then you can simply select root nodes in the cluster with the cell type identified as the beginning of the trajectory.
>
> ```r
># specifying root cells: pop up window
>cds_order_1 <- order_cells(cds_trajectory)
>```
>
{: .tip}

# Plotting in pseudotime

You can use any `cds_order` object for the downstream analysis. Let’s pick one and assign it to an object with a shorter and more general name.
```r
cds_order <- cds_order_1_helper
```

The function `order_cells()` calculates pseudotime for each cell, so we can now visualise the T-cell maturation process in pseudotime! Additionally, we can access it and store in our CDS object f


topic: single-cell
tutorial: scatac-batch-correction-snapatac2
title: Multi-sample batch correction with Harmony and SnapATAC2
difficulty: intermediate
time_estimation: 4H
requirements_json: ["type: internal", "scatac-preprocessing-tenx", "scatac-standard-processing-snapatac2"]
tags_json: ["10x", "epigenetics"]
content_hash: 2c28a2890cf0e158dce8f6c192c47fc7
last_modified: 2026-05-27T09:01:49.740387
zenodo_link: https://zenodo.org/records/12683310
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scatac-batch-correction-snapatac2/tutorial.html

content:
<!-- This is a comment. -->
Performing experiments in replicates is a cornerstone of modern biological science. However, when integrating data from multiple single-cell sequencing experiments, technical confounders might impact the results.
To reduce technical confounders, such as different experimenters, experimental protocols, sequencing lanes or sequencing technologies, batch correction is often beneficial.

In this tutorial, we will perform batch correction on five datasets of {scATAC-seq} data with the three algorithms *Harmony* ({% cite Korsunsky2019 %}), *Scanorama* ({% cite Hie2019 %}) and the *mutual nearest neighbor-based* ({% cite Haghverdi2018%}) algorithm *MNC-correct* ({% cite Zhang2024 %}). The {scATAC-seq} analysis will be performed with the tool suite [**SnapATAC2**](https://kzhang.org/SnapATAC2/version/2.5/index.html) ({% cite Zhang2024 %}).

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/tutorial_mode.md %}

> <comment-title></comment-title>
>
> This tutorial is significantly based on the ["Multi-sample Pipeline" tutorial](https://kzhang.org/SnapATAC2/version/2.5/tutorials/integration.html) from **SnapATAC2**.
>
> The data analysis is performed with the same tools shown in the tutorial [Single-cell ATAC-seq standard processing with SnapATAC2]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %} ).
> That tutorial also explains the steps of the ATAC-seq analysis with SnapATAC2 in more detail.
> We recommend completing that tutorial before continuing with batch correction using SnapATAC2.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data

In this tutorial we will analyze colon samples from multiple donors, provided by the [SnapATAC2 documentation](https://kzhang.org/SnapATAC2/version/2.5/tutorials/integration.html). The `chrom_sizes` file and the `gene_annotation` file are identical to the previous tutorial [Single-cell ATAC-seq standard processing with SnapATAC2]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %} ).
The five `colon_multisample` files have been generated by the **Cell Ranger ATAC 2.0.0** pipeline from 10X to generate a [*Fragments File*](https://support.10xgenomics.com/single-cell-atac/software/pipelines/latest/output/fragments).


> <details-title>Chromosome sizes </details-title>
>
> - A chromosome sizes file can be generated using the tool {% tool [Compute sequence length](toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.3) %}.
> - The reference genome can either be selected from cached genomes or uploaded to the galaxy history.
>
{: .details}

First, we will import the datasets into Galaxy. Next, we will organize all `colon_multisample` datasets into a collection. Using collections helps streamline the workflow, making the subsequent analysis steps more efficient.

{% snippet faqs/galaxy/histories_datasets_vs_collections.md %}


## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library
>
>
>    ```
>    {{ page.zenodo_link }}/files/colon_multisample_01.gz
>    {{ page.zenodo_link }}/files/colon_multisample_02.gz
>    {{ page.zenodo_link }}/files/colon_multisample_03.gz
>    {{ page.zenodo_link }}/files/colon_multisample_04.gz
>    {{ page.zenodo_link }}/files/colon_multisample_05.gz
>    {{ page.zenodo_link }}/files/chrom_sizes.txt
>    {{ page.zenodo_link }}/files/gencode.v46.annotation.gtf.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <warning-title>Large file sizes!</warning-title>
>    > - The `colon_multisample` datasets are quite large. The entire tutorial requires approximately 50 GB of storage.
>    > - To reduce storage you can change the {% icon galaxy-gear %} **Upload Configuration** to *Defer dataset resolution* for the upload of the `colon_multisample` datasets.
>    > - This will delete the datasets after the first analysis step and helps in reducing storage.
>    > - You can also permanently delete datasets, which are no longer required.
>    >
>    >   {% snippet faqs/galaxy/datasets_deleting.md %}
>    >
>    {: .warning}
>
> 3. Rename the datasets
>    - {% icon galaxy-pencil %} **Rename** the file `gencode.v46.annotation.gtf.gz` to `gene_annotation.gtf.gz`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatypes of the `colon_multisample` files are set to `bed`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bed" %}
>
> 5. Create a dataset collection with all `colon_multisample` datasets and rename the collection to `Colon Multisample Fragments`.
>
>    {% snippet faqs/galaxy/collections_build_list.md name="Colon Multisample Fragments" %}
>
{: .hands_on}

# SnapATAC2 preprocessing and filtering

With our data imported and the collection assembled, we can now begin the {scATAC-seq} data preprocessing with SnapATAC2.

The first step is to import the datasets into an AnnData object using the *pp.import_data* tool. Next, the `{TSSe}` will be calculated, serving as a `{QC}` metric to selectively filter droplets containing high-quality cells.

> <details-title>AnnData format </details-title>
>
> - The [**AnnData**](https://anndata.readthedocs.io/en/latest/) format was initially developed for the [**Scanpy**](https://scanpy.readthedocs.io/en/stable/index.html) package and is now a widely accepted data format to store annotated data matrices in a space-efficient manner.
>
> ![Anndata format]({% link topics/single-cell/images/scatac-standard-snapatac2/anndata_schema.svg %} "AnnData format stores a count matrix `X` together with annotations of observations (i.e., cells) `obs`, variables (i.e., genes) `var`, and unstructured annotations `uns`.")
>
{: .details}

> <hands-on-title> Preprocessing and QC </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Import data fragment files and compute basic QC metrics, using 'pp.import_data'`
>        - {% icon param-collection %} *"Fragment file, optionally compressed with gzip or zstd"*: `Colon Multisample Fragments` (Input dataset collection)
>        - {% icon param-file %} *"A tabular file containing chromosome names and sizes"*: `chrom_sizes.txt` (Input dataset)
>        - *"Number of unique fragments threshold used to filter cells"*: `1000`
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute the TSS enrichment score (TSSe) for each cell, using 'metrics.tsse'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas` (dataset collection output of **pp.import_data** {% icon tool %})
>        - {% icon param-file %} *"GTF/GFF file containing the gene annotation"*: `gene_annotation` (Input dataset)
> 3. {% icon galaxy-pencil %} Rename the generated collection to `Colon Multisample AnnDatas TSSe`.
>
> 4. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the TSS enrichment vs. number of fragments density figure, using 'pl.tsse'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas TSSe` (output of **metrics.tsse** {% icon tool %})
> 5. {% icon galaxy-eye %} Inspect a few exemplary `.png` outputs of the collection
>
> ![TSSe plots against number of unique fragments]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/tsse-colon01-colon02.png %}"Examplary plots of TSSe from <code>colon_multisample_01</code> and <code>colon_multisample_02</code>")
>
> High-quality cells can be identified in the plot of {TSSe} scores against a number of unique fragments for each cell.
>
> > <question-title></question-title>
> >
> > 1. Where are high-quality cells typically located on a {TSSe} plot?
> > 2. Based on these plots, how should the filtering threshold be set?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. The cells in the upper right are high-quality cells, enriched for {TSS}. Fragments in the lower left represent low-quality cells or empty droplets and should be filtered out.
> > > 2. Setting the minimum {TSSe} to 7.0 will filter out the lowest quality droplets without loosing too much data.
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

## Filtering the count matrices

The {TSSe} threshold balances data retention and quality control. In our dataset, the {TSSe} distributions indicate substantial differences in sample quality between batches. To retain as much biological data as possible, we will use a broader filter (e.g., minimum {TSSe} = 7.0).

Standard {TSSe} Thresholds
- **Strict (TSSe ≥ 10–15)**: High-confidence cells.
- **Moderate (TSSe ≥ 7–10)**: Balances data retention and quality but may include lower-quality cells.
- **Broad (TSSe ≥ 5–7)**: Maximizes retention but increases the risk of including lower-quality data.

Choosing the Right Cutoff
- If sample quality varies, a **moderate threshold (≥7.0)** helps retain more data.
- If high-quality data is required, **stricter filtering (≥10.0)** is recommended.
- Inspect {TSSe} distributions to determine the most suitable threshold for your dataset.

> <hands-on-title> Filtering </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas TSSe` (output of **metrics.tsse** {% icon tool %})
>        - *"Minimum TSS enrichemnt score required for a cell to pass filtering"*: `7.0`
>
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Generate cell by bin count matrix, using 'pp.add_tile_matrix'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas filtered` (output of **pp.filter_cells** {% icon tool %})
>        - *"The size of consecutive genomic regions used to record the counts"*: `5000`
>
> 3. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Perform feature selection, using 'pp.select_features'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas tile_matrix` (output of **pp.add_tile_matrix** {% icon tool %})
>        - *"Number of features to keep"*: `50000`
>
>    > <details-title> Bin size and features </details-title>
>    >
>    > - *pp.add_tile_matrix* divides the genome into a specified number of bins, depending on the bin size (f.ex. 5000bp). For each bin, the ATAC-seq reads of individual cells are checked to determine if a read is located in the bin. This is counted as a feature and stored under `n_vars` in the AnnData object.
>    >    - Increasing the bin size greatly reduces compute time at the cost of some biological data.
>    >    - For this reason, the bin size 5000bp has been selected for the colon datasets. In the [previous tutorial]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#feature-selection), the lower bin size of 500bp was chosen, since fewer cells were analyzed.
>    > - *pp.select_features* uses the previously identified features to select the most accessible features for further analysis.
>    >    - The parameter *"Number of features to keep"* determines the upper limit of features which can be selected.
>    >    - Similarly to the *bin_size*, the *Number of Features to keep* can also impact downstream clustering. This was demonstrated in the [previous tutorial]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#feature-selection ).
>    {: .details}
>
> 4. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute probability of being a doublet using the scrublet algorithm, using 'pp.scrublet'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas features` (output of **pp.select_features** {% icon tool %})
>
> 5. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Remove doublets according to the doublet probability or doublet score, using 'pp.filter_doublets'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Colon Multisample AnnDatas scrublet` (output of **pp.scrublet** {% icon tool %})
>
{: .hands_on}

# Concatenate the Collection
Before we can continue with the analysis and batch correction, we need to extract the datasets out of the collection and merge them into a single AnnData object.

## Extracting datasets from a collection
This is achieved in two steps:
1. The first dataset is extracted from the collection using *Extract dataset*
2. Afterwards, the first dataset will be removed from the collection. This is done by extracting element identifiers and filtering the collection with the name of the first dataset.

> <comment-title>  </comment-title>
> - It is also possible to manually remove a dataset from a collection.
> - However, manually removing datasets can not be implemented in Galaxy workflows. In contrast the automatic method, shown here, can be represented in a workflow and is therefore much better scalable.
{: .comment}

> <hands-on-title> Extract datasets </hands-on-title>
>
> 1. {% tool [Extract element from collection](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-collection %} *"Input List"*: `Colon Multisample AnnDatas filtered_doublets` (output of **pp.filter_doublets** {% icon tool %})
>    - *"How should a dataset be selected?"*: `The first dataset`
> 2. {% tool [Extract element identifiers](toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Colon Multisample AnnDatas filtered_doublets` (output of **pp.filter_doublets** {% icon tool %})
>
> 3. {% tool [Select first](Show beginning1) %} with the following parameters:
>    - *"Select first"*: `1`
>    - {% icon param-file %} *"from"*: `Element identifiers` (output of **Extract element identifiers** {% icon tool %})
> 4. {% tool [Filter collection](__FILTER_FROM_FILE__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: `Colon Multisample AnnDatas filtered_doublets` (output of **pp.filter_doublets** {% icon tool %})
>    - *"How should the elements to remove be determined?"*: `Remove if identifiers are PRESENT in file`
>        - {% icon param-file %} *"Filter out identifiers present in"*: `select_first` (output of **Select first** {% icon tool %})
> 5. {% icon galaxy-pencil %} Rename the filtered collection to `Colon Multisample 02-05`
>
{: .hands_on}

## Concatenating all AnnData files

> <hands-on-title> Concatenate </hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `colon_multisample_01` (output of **Extract dataset** {% icon tool %})
>    - *"Function to manipulate the object"*: `Concatenate along the observations axis`
>        - {% icon param-collection %} *"Annotated data matrix to add"*: `Colon Multisample 02-05` (output of **Filter collection** {% icon tool %})
>        - *"Join method"*: `Intersection of variables`
>        - *"Key to add the batch annotation to obs"*: `batch`
>
>    > <details-title>Issues with concatenation</details-title>
>    > - The runtime of this operation depends on the dataset size. For `colon_multisample` datasets, concatenation may take up to 1 hour.
>    > - For larger datasets, the tool's allocated memory may be insufficient, causing the operation to fail. In that case, you will see the following error message:
>    >
>    >   ```
>    >                Fatal error: Exit code 137 ()
>    >   ```
>    > - In this case, report the issue so that administrators can increase the memory limit, allowing the job to complete successfully.
>    >
>    >   {% snippet faqs/galaxy/analysis_troubleshooting_reporting.md %}
>    >
>    {: .details}
>
> 2. {% icon galaxy-pencil %} Rename the AnnData output to `Multisample AnnData`
> 3. {% icon galaxy-eye %} Inspect the general information of `Multisample AnnData`
>
>    > <tip-title>Inspecting AnnData objects</tip-title>
>    > * Many toolsets producing outputs in *AnnData* formats in Galaxy, provide the general information by default:
>    >    * Click on the name of the dataset in the history to expand it. The general Anndata information will be given in the expanded box.
>    >    * Alternatively, expand the dataset and click on {% icon details %}*Dataset Details*. Scroll to Job Information and inspect the Tool Standard Output.
>    > * If a tool does not provide general AnnData information or a more specific query is needed, you can use {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %}.
>    {: .tip}
>
>    ```
>    AnnData object with n_obs × n_vars = 34372 × 606219
>     obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'batch'
>     var: 'count-0', 'selected-0', 'count-1', 'selected-1', 'count-2', 'selected-2', 'count-3', 'selected-3', 'count-4', 'selected-4'
>     obsm: 'fragment_paired'
>    ```
>
> > <question-title></question-title>
> >
> > 1. How many colon cells are present in this AnnData object?
> > 2. What does the 'batch' annotation indicate?
> > 3. What do the 'count-' and 'selected-' annotations represent?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 34,372 cells.
> > > 2. The 'batch' annotation labels cells according to their sample number (0–4). This annotation will later be used by batch correction algorithms to generate clusters from multiple samples.
> > > 3. 'count' and 'selected' are variable annotations representing detected and selected features. The sample number (0–4) indicates the dataset that generated these annotations.
> > >
> > {: .solution}
>  {: .question}
>
{: .hands_on}

# Dimension Reduction
Now that all samples have been concatenated into a single AnnData object, the most accessible features of the combined count matrix must be selected. The previously chosen features were sample-specific and cannot be used for dimensionality reduction. Therefore, the most accessible features will be reselected, followed by dimensionality reduction using *matrix-free spectral embedding*.


> <hands-on-title> Spectral Embedding </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Perform feature selection, using 'pp.select_features'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData` (output of **Manipulate AnnData** {% icon tool %})
>        - *"Number of features to keep"*: `50000`
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and clustering"*: `Perform dimensionality reduction using Laplacian Eigenmap ('tl.spectral')`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData features` (output of **pp.select_features** {% icon tool %})
>        - *"Distance metric"*: `cosine`
>
> 3. {% icon galaxy-pencil %} Rename the output AnnData object to `Multisample AnnData spectral`
>
{: .hands_on}


## Control Without Batch Correction
Batch effects can be visualized using a `{UMAP}` projection, where samples are colored by their batch annotation (`obs: 'batch'`).
> <hands-on-title> UMAP Projection Without Batch Correction </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute UMAP using 'tl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData spectral` (output of **tl.spectral** {% icon tool %})
>
> 2. {% icon galaxy-pencil %} Rename the output AnnData object to `Multisample AnnData UMAP`.
>
> 3. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData UMAP` (output of **tl.umap** {% icon tool %})
>        - *"Color"*: `batch`
>        - *"Height of the plot"*: `500`
>
> 4. {% icon galaxy-pencil %} Rename the generated image to `spectral-UMAP-No Batch Correction`.
>
> 5. {% icon galaxy-eye %} Inspect the `.png` output.
>
> ![UMAP plot before batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/batch-umap-no_harmony-spectral.png %})
>
> > <question-title></question-title>
> >
> > 1. How do batch effects appear in this plot?
> > 2. What would a plot without batch effects look like?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. Batch effects appear as distinct clusters where samples are unevenly distributed. For example, in the upper right corner, each batch forms a separate group. These differences are likely technical artifacts rather than true biological variation.
> > > 2. A plot without batch effects would show cells from all samples evenly mixed, with no distinct clusters corresponding to individual batches.
> > >
> > {: .solution}
>  {: .question}
>
{: .hands_on}

# Batch correction
After confirming the presence of batch effects, they should be removed using correction algorithms. **SnapATAC2** provides three batch correction methods: *Harmony*, *MNC-correct*, and *Scanorama*. To determine the most suitable algorithm for your dataset, compare the outputs of each method.

> <details-title> Batch correction algorithms </details-title>
> - Batch correction algorithms adjust the cell-by-feature count matrix to account for batch-specific differences between samples.
> - Batch effects can arise from various technical sources, such as sequencing lanes, plates, protocols, and handling. Additionally, biological factors, including tissue types, species, and inter-individual variation, can also contribute to batch effects {% cite Luecken2021 %}.
> - Many batch correction algorithms have been developed, with most scATAC-seq methods adapted from scRNA-seq batch removal techniques.
> - *Harmony* {% cite Korsunsky2019 %} is a principal component analysis (PCA)-based method that leverages lower-dimensional data to assign cells to new clusters. It prioritizes multi-sample clustering to integrate datasets by computing linear correction factors for each batch and cluster, iteratively adjusting cell positions until optimal batch correction is achieved.
>
>   ![Graphical abstract of Harmony batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/harmony-graphical-abstract.png %})
>
> - *Scanorama* {% cite Hie2019 %} performs panorama stiching, to find and merge overlapping cell types.
>
>   ![Graphical abstract of Scanorama batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/scanorama-graphical-abstract.png %})
>
> - *MNC-correct* {% cite Zhang2024 %} is a modified version of the *mutual nearest neighbor* algorithm {% cite Haghverdi2018 %}. It calculates centroids for batch-specific clusters and identifies pairs of mutual nearest centroids (MNC) across batches. Correction vectors then align the batches in the same plane. Additionally, *MNC-correct* can be run iteratively to refine corrections for optimal alignment.
>
>   ![Graphical abstract of MNC-correct batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/mnn-graphical-abstract.png %})
>
{: .details}

We will first use *Harmony* to correct batch effects, followed by testing the other methods.
> <hands-on-title> Batch correction and visualization</hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Use harmonypy to integrate different experiments,using 'pp.harmony'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData UMAP` (output of **tl.umap** {% icon tool %})
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute Umap, using 'tl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony` (output of **pp.harmony** {% icon tool %})
>        - *"Use the indicated representation in `.obsm`"*: `X_spectral_harmony`
>        - *"`adata.obs` key under which t add cluster labels"*: `umap_harmony`
>
> 2. {% icon galaxy-pencil %} Rename the AnnData output to `Multisample AnnData harmony UMAP`
>
>    > <comment-title> Key for Cluster Labels </comment-title>
>    > - Adding the new *UMAP embeddings* under the key `umap_harmony` preserves the non-batch-corrected embeddings in the AnnData object.
>    >   - Alternatively, leaving this parameter empty will overwrite the existing embeddings.
>    {: .comment}
>
> 4. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding, using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony UMAP` (output of **tl.umap** {% icon tool %})
>        - *"Color"*: `batch`
>        - *"Use the indicated representation in .obsm"*: `X_umap_harmony`
>        - *"Height of the plot"*: `500`
>
> 5. {% icon galaxy-pencil %} Rename the generated image to `spectral-UMAP-harmony`
>
> 6. {% icon galaxy-eye %} Inspect the `.png` output
>
>  ![UMAP plot of Batch correction with Harmony]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/batch-umap-harmony-spectral.png %})
>
> > <question-title></question-title>
> >
> > 1. How did *Harmony* affect the appearance of this plot?
> > 2. Are there areas in the plot where batch effects persist?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. *Harmony* has successfully removed most batch effects. Notably, the center-left groups have merged into a single larger group containing all batches.
> > > 2. Yes, some batch effects remain, particularly in the upper-right corner, where batch-specific colors do not fully overlap. However, this could also be due to certain samples having fewer cells, leading to underrepresentation in clusters of rare cell types. Therefore, it is unclear whether *Harmony* has completely eliminated all batch effects.
> > >
> > {: .solution}
>  {: .question}
>
>
{: .hands_on}

> <details-title>Batch Correction with Scanorama and MNC-correct</details-title>
> - Other batch correction methods can be applied using {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %}, including *pp.mnc_correct* and *pp.scanorama_integrate*.
>   - For *MNC-correct*, the number of iterations can be adjusted.
> - To determine the optimal batch correction method, it is recommended to test different algorithms and parameter settings.
> - Example outputs from *Scanorama* and *MNC-correct* are shown below:
>
>  ![Batch correction UMAP plots of MNC-correct with different settings and Scanorama]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/mnc-correct-scanorama-umap-spectral.png %} "UMAP plots of batch correction with different methods. (a) Batch correction with MNC-correct and default settings. (b) Batch correction with Scanorama. (c) Batch correction with MNC-correct and 30 iterations.")
>
> > <question-title></question-title>
> >
> > - Compare these plots with the output of *Harmony*. Which algorithm is best-suited for the colon datasets?
> >
> > > <solution-title></solution-title>
> > >
> > > - *Harmony* achieved the best batch integration, as it produced the fewest single-batch groups. *Scanorama* and *MNC-correct* (with default settings) did not integrate the batches as effectively. However, *MNC-correct* with 30 iterations significantly reduced batch effects and could also be used for further analysis.
> > >
> > {: .solution}
>  {: .question}
>
{: .details}


# Clustering of the batch corrected samples
The analysis can now continue using the same methods outlined in the [standard pathway]({% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}). The batch-corrected embeddings are clustered and visualized using the *Leiden* algorithm.

> <hands-on-title> Leiden clustering and visualization </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute a neighborhood graph of observations, using 'pp.knn'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony UMAP` (output of **tl.umap** {% icon tool %})
>        - *"The key for the matrix"*: `X_spectral_harmony`
>
>    > <tip-title>Matrix Keys</tip-title>
>    >
>    > - Each batch correction algorithm stores its corrected matrix under a specific key:
>    >   - *Harmony*: `X_spectral_harmony`
>    >   - *MNC-correct*: `X_spectral_mnn`
>    >   - *Scanorama*: `X_spectral_scanorama`
>    > - These keys are stored in the AnnData object under `'obsm'`.
>    >
>    {: .tip}
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Cluster cells into subgroups, using 'tl.leiden'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony knn` (output of **pp.knn** {% icon tool %})
>        - *"Whether to use the Constant Potts Model (CPM) or modularity"*: `modularity`
>
> 3. {% icon galaxy-pencil %} Rename the AnnData output to `Multisample AnnData harmony leiden`
>
> 4. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding, using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **tl.leiden** {% icon tool %})
>        - *"Color"*: `leiden`
>        - *"Use the indicated representation in .obsm"*: `X_umap_harmony`
>        - *"Height of the plot"*: `500`
>
> 5. {% icon galaxy-pencil %} Rename the generated image to `spectral-UMAP-harmony-leiden`
>
> 6. {% icon galaxy-eye %} Inspect the `.png` output
>
>  ![UMAP plot of Batch-corrected leiden clusters]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/batch-umap-harmony-leiden.png %})
>
{: .hands_on}

After integrating the datasets and clustering the cells, the scATAC-seq analysis can proceed with downstream analysis. This includes [cell cluster annotation]({% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#cell-cluster-annotation) and differential peak analysis.
> <comment-title></comment-title>
>
> The **SnapATAC2** tools for differential peak analysis are available in Galaxy, but no GTN training materials are currently provided. Until a tutorial is available, you can refer to the **SnapATAC2** documentation for a [tutorial on differential peak analysis](https://kzhang.org/SnapATAC2/version/2.6/tutorials/diff.html).
>
> The tools are available in Galaxy under {% tool [SnapATAC2 Peaks and Motif](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_peaks_and_motif/snapatac2_peaks_and_motif/2.6.4+galaxy1) %}.
>
{: .comment}


# Conclusion
{% icon congratulations %} Well done, you’ve made it to the end! You might want to consult your results with this [control history](https://usegalaxy.eu/u/timonschlegel/h/multisample-batch-correction-with-harmony-and-snapatac2), or check out the [full workflow](https://usegalaxy.eu/u/timonschlegel/w/multisample-batch-correction-with-snapatac2-and-harmony) for this tutorial.

In this tutorial, we integrated five {scATAC-seq} colon samples using a scalable Galaxy workflow. We compared different batch integration algorithms to identify the most suitable method for our data. Finally, we clustered the cells to prepare the data for downstream analysis.

![SnapATAC2 batch correction pipeline]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/Batch-correction-pipeline-overview.png %})

tools_json:
["Compute sequence length", "toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.3", "SnapATAC2 Preprocessing", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1", "SnapATAC2 Plotting", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1", "Extract element from collection", "__EXTRACT_DATASET__", "Extract element identifiers", "toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2", "Select first", "Show beginning1", "Filter collection", "__FILTER_FROM_FILE__", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.3+galaxy0", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0", "SnapATAC2 Clustering", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1", "SnapATAC2 Peaks and Motif", "toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_peaks_and_motif/snapatac2_peaks_and_motif/2.6.4+galaxy1"]


