# tutorials

topic: proteomics
tutorial: clinical-mp-4-quantitation
title: Clinical Metaproteomics 4: Quantitation
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: c177e1b90366ad012fb84648be68fcb3
last_modified: 2026-02-26T21:12:17.465149
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/clinical-mp-4-quantitation/tutorial.html

content:
The next step of the clinical metaproteomics workflow is the quantification workflow. Running a quantification workflow in proteomics is essential for several critical purposes. It allows researchers to measure and compare the abundance of proteins or peptides in biological samples, offering valuable insights into biomarker discovery, comparative analysis, and differential expression studies. Quantitative proteomics helps reveal the functional roles of proteins, the stoichiometry of protein complexes, and the effects of drugs on protein expression in pharmacological studies. Additionally, it serves as a quality control measure, validating initial protein identifications, and providing data normalization for increased accuracy. Quantitative data are indispensable for hypothesis testing, systems biology, and their clinical relevance in areas such as disease diagnosis, prognosis, and therapeutic decision-making. In summary, the quantitation workflow in proteomics is a cornerstone for deciphering the complexities of protein expression and regulation, facilitating a wide array of biological and clinical applications.

In this current workflow, we perform Quantification using the MaxQuant tool and the output will be interpreted in our next module.

![Quantitation workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-quantification.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.raw
>    https://zenodo.org/records/10105821/files/Experimental-Design_Discovery_MaxQuant.tabular
>    https://zenodo.org/records/10105821/files/Quantitation_Database_for_MaxQuant.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to input files.
> 6. Create a dataset of the RAW files.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
{: .hands_on}


# Import Workflow

> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-4-quantitation/workflows/WF4_Quantitation_Workflow.ga" title="Quantitation Workflow" %}
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Quantitation_Database-For-MaxQuant * "*: `Quantitation_Database_for_MaxQuant.fasta`
>    - {% icon param-file %} *" Experimental-Design Discovery MaxQuant"*: `Experimental-Design_Discovery_MaxQuant.tabular`
>    - {% icon param-file %} *" Input Raw-files"*: `RAW dataset collection`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}


# Peptide quantification

In the [Discovery Module](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md), we used MaxQuant to identify peptides for verification. Now, we will again use MaxQuant to further quantify the PepQuery-verified peptides, both microbial and human. More information about quantitation using MaxQuant is available, including [Label-free data analysis](https://gxy.io/GTN:T00218) and [MaxQuant and MSstats for the analysis of TMT data](https://gxy.io/GTN:T00220).

The outputs we are most interested in consist of the `MaxQuant Evidence file`, `MaxQuant Protein Groups`, and `MaxQuant Peptides`. The `MaxQuant Peptides` file will allow us to group them to generate a list of quantified microbial peptides.

> <hands-on-title> Quantify verified peptides (from PepQuery2) </hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy4) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-file %} *"FASTA files"*: `Quantitation Database for MaxQuant` (Input dataset)
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template (if needed). For detailed                           instructions see the help text."*: `output` (Input dataset)
>        - *"minimum peptide length"*: `8`
>        - *"Match between runs"*: `Yes`
>        - *"Maximum peptide length for unspecific searches"*: `50`
>    - In *"Protein quantification"*:
>        - *"Use only unmodified peptides"*: `Yes`
>            - *"Modifications used in protein quantification"*: `Oxidation (M)`
>        - In *"LFQ Options"*:
>            - *"iBAQ (calculates absolute protein abundances by normalizing to copy number and not protein mass)"*: `No`
>    - In *"Parameter Group"*:
>        - {% icon param-repeat %} *"Insert Parameter Group"*
>            - {% icon param-collection %} *"Infiles"*: `output` (Input dataset collection)
>            - *"fixed modifications"*: `Carbamidomethyl (C)`
>            - *"variable modifications"*: `Oxidation (M)`
>            - *"enzyme"*: `Trypsin/P`
>            - *"Quantitation Methods"*: `reporter ion MS2`
>                - *"isobaric labeling"*: `TMT11plex`
>                - *"Filter by PIF"*: `Yes`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Peptides` `Evidence` `MSMS`
>
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Why can we switch back to using RAW files for MaxQuant, instead of using MGF files?
>
> > <solution-title></solution-title>
> >
> > 1. MaxQuant prefers RAW format compared to MGF as it has more information compared to MGF.
> >
> {: .solution}
>
{: .question}
> <question-title></question-title>
>
> 1. Previously, we used MaxQuant in the Discovery workflow. Why are we using MaxQuant again, instead of Search GUI/PeptideShaker?
>
> > <solution-title></solution-title>
> >
> > 1. We are using MaxQuant for quantification purposes only. SearchGUI Peptide Shaker doesn't have the capability to perform quantification of peptides or proteins.
> >
> {: .solution}
>
{: .question}

## Using Text Manipulation Tools to Manage MaxQuant Outputs

> <hands-on-title> Select microbial protein groups from MaxQuant with Select </hands-on-title>
>
> 1. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `proteinGroups` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 2. {% tool [Select](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `peptides` (output of **MaxQuant** {% icon tool %})
>    - *"that"*: `NOT Matching`
>    - *"the pattern"*: `(_HUMAN)|(_REVERSED)|(REV_)|(CON)|(con)`
>
>
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
>
> 4. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `out_file1` (output of **Select** {% icon tool %})
>
>
{: .hands_on}

## Generating a list of quantified proteins and peptides

> <hands-on-title> Group quantified proteins </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}
> <hands-on-title> Group quantified peptides </hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Group by column"*: `c1`
>
>
{: .hands_on}


# Conclusion

In summary, the implementation of a quantitation workflow using MaxQuant represents a significant advancement in quantitative proteomic research. This approach enables precise measurement of protein and peptide abundances, enhancing our ability to unravel the complexities of biological systems. This workflow is instrumental in biomarker discovery, comparative analysis, and understanding differential protein expression by offering detailed insights into quantitative changes across different experimental conditions. Its capacity to generate accurate data supports a wide spectrum of applications, including disease research, drug development, and systems biology investigations. Furthermore, the MaxQuant-based quantitation workflow ensures data quality, enabling reliable and reproducible results. It serves as a vital step for quality control, allowing researchers to draw meaningful conclusions from proteomic experiments confidently.


topic: proteomics
tutorial: maxquant-msstats-tmt
title: MaxQuant and MSstats for the analysis of TMT data
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["[DDA, TMT]"]
content_hash: f6727cefbba49c4cf813712a764aeca1
last_modified: 2026-02-26T21:12:17.493284
zenodo_link: https://zenodo.org/record/5195800
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/maxquant-msstats-tmt/tutorial.html

content:
In this training we will cover the full analysis workflow from tandem mass tag (TMT) labeled samples using MaxQuant in conjunction with MSstatsTMT. 

The training dataset consists of a dataset from {% cite Baumert_2020 %} which investigates the effects of a recently discovered histone methyltransferase, lysine methyl transferase 9 (KMT9α) in A549 cells. High KMT9α expression in lung cancer correlates with poor patient survival. Therefore,  KMT9α is a very interesting research target in light of the fact that lung cancer survival rates could not be improved over the last 15 years. The example dataset consists of 4 cell culture samples from KMT9α knock-down cells and 4 cell culture samples from control (Ctrl) cells. In order to increase the proteome coverage the TMT experiment was pre-fractionated using offline high pH reverse phase chromatography resulting in 12 MS runs.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

The raw data is available via the PRIDE repository under the ID: [PXD014145](https://www.ebi.ac.uk/pride/archive/projects/PXD014145). The MaxQuant experimental design template file, MSstatsTMT annotation file and FASTA file for this training are deposited at Zenodo(https://zenodo.org/record/5195800). It is of course possible to use another FASTA file with human proteome sequences, but to ensure that the results are compatible, we recommend using the provided FASTA file. MaxQuant not only adds known contaminants to the FASTA file, but also generates the “decoy” hits for false discovery rate estimation itself, therefore the FASTA file is not allowed to have decoy entries. To learn more about FASTA files, have a look at [Protein FASTA Database Handling tutorial]({% link topics/proteomics/tutorials/database-handling/tutorial.md %}).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and give it a meaningful name
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the FASTA database, experimental design template, and annotation file [Zenodo](https://zenodo.org/record/5195800)
>
>    ```
>    https://zenodo.org/record/5195800/files/protein_database.fasta
>    https://zenodo.org/record/5195800/files/MaxQuant%20experimental%20design%20template.tabular
>    https://zenodo.org/record/5195800/files/MSstatsTMT%20annotation%20file.tabular
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Once the files are green, rename the fasta file into 'protein database', the experimental design template into 'experimental design template', and the annotation file into 'annotation file'.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. To run MaxQuant, import the raw data from [PRIDE](https://www.ebi.ac.uk/pride/archive/projects/PXD014145) as a 'Collection'.
>    ```
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA380.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA381.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA382.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA383.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA384.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA385.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA386.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA387.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA388.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA389.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA390.raw
>    ftp://ftp.pride.ebi.ac.uk/pride-archive/2020/05/PXD014145/MFA391.raw
>    ```
>    
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true collection_type="List" collection_name="raw_files" format="thermo.raw" %}
{: .hands_on}


# MaxQuant analysis

We start the MaxQuant run with TMT parameters for MS2 based reporter quantitation. A quality control report will be generated with the [PTXQC functionality](https://pubs.acs.org/doi/10.1021/acs.jproteome.5b00780) that is directly implemented in the MaxQuant Galaxy tool. For more information about MaxQuant and its parameter please have a look at the [Label-free data analysis using MaxQuant tutorial]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-label-free/tutorial.html). To continue with statistical analysis in MSstatsTMT, the Protein Groups and the Evidence files are needed from MaxQuant. The run time of MaxQuant depends on the number and size of the input files and on the chosen parameters. The run of the training datasets will take a while, but the training can be directly continued with the MaxQuant result files from Zenodo. 


> <hands-on-title>MaxQuant analysis</hands-on-title>
>
> 1. {% tool [MaxQuant](toolshed.g2.bx.psu.edu/repos/galaxyp/maxquant/maxquant/1.6.17.0+galaxy2) %} with the following parameters:
>    - In *"Input Options"*:
>        - {% icon param-collection %} *"FASTA files"*: `protein database`
>    - In *"Search Options"*:
>        - {% icon param-file %} *"Specify an experimental design template"*: `experimental design template` 
>    - In *"Parameter Group"*:
>        - {% icon param-collection %} *"Infiles"*: `MFA380.raw` `MFA381.raw` `MFA382.raw` `MFA383.raw` `MFA384.raw` `MFA385.raw` `MFA386.raw` `MFA387.raw` `MFA388.raw` `MFA389.raw` `MFA390.raw` `MFA391.raw`
>        -  *"variable Modifications"*: `Oxidation (M)` `Acetyl (Protein N-term)`
>        -  *"enzyme"*: `Trypsin/P`
>        -  *"Quantitation Methods"*: `reporter ion MS2`
>            - *"isobaric labeling"*: `TMT11plex`
>            - *"Filter by PIF"*: `Yes`
>    - *"Generate PTXQC (proteomics quality control pipeline) report? (experimental setting)"*: `Yes`
>    - In *"Output Options"*:
>        - *"Select the desired outputs."*: `Protein Groups` `mqpar.xml` `Evidence` `MaxQuant and PTXQC log`
>
>    > <tip-title>Continue with results from Zenodo</tip-title>
>    >
>    > Because the MaxQuant run takes quite a while, we recommend to download the MaxQuant results from Zenodo and continue with the tutorial
>    > 1. Import the files from [Zenodo](https://zenodo.org/record/5195800)
>    >
>    >  ```
>    > https://zenodo.org/record/5195800/files/MaxQuant%20Result%20evidence.tabular
>    > https://zenodo.org/record/5195800/files/MaxQuant%20Result%20protein%20groups.tabular
>    > ```
>    {: .tip}
{: .hands_on}

> <question-title></question-title>
>
> 1. How many protein groups were found with MaxQuant?
>
> > <solution-title></solution-title>
> >
> > 1. 6911 (numbers might slightly vary with different MaxQuant versions); number of lines minus header line in the MaxQuant protein groups output
> >
> {: .solution}
>
{: .question}



> <tip-title>PIF parameter for TMT data</tip-title>
> Co-isolation of multiple peptides can lead to inaccurate TMT signals, as not only one peptide is fragmented but multiple different peptides are fragmented and the TMT signals cannot be distinguished anymore. MaxQuant can “check” the MS1 scans to predict, whether multiple different peptides were cofragmented. Precursor Intensity Filtering (PIF) allows you to implement a cutoff how high the purity of fragmentation was (1 = no coisolation, 0.75 = default).
{: .tip}

> <tip-title>Preparing an MaxQuant experimental design template for your own analysis</tip-title>
> For this tutorial all files needed for the analysis are provided. Nevertheless here are some guidelines for the generation of the experimental design template for MaxQuant for your own data analyses.
> For thermo.raw files, “.raw” is considered a part of the filename. Therefore, file names should be 'Filename1.raw', 'Filename2.raw' etc.. Please pay special attention to your experimental design, since MaxQuant will summarize peptide and protein intensities differently depending on this input. In brief, MaxQuant will summarize the protein intensities within each specified experiment. Often, TMT experiments are not measured in one MS run, but in multiple runs (either as so called fractions, or as replicates). Over the last years, high pH prefractionation has become the standard for fractionating TMT samples. Alternatively, SCX or HILIC fractionations can be employed. Although you fractionated your samples prior to measurement you still want MaxQuant to group all samples which belong to “one” TMT experiment. Since the fractions originate from one sample the experimental design template allows to map multiple measurements to one experiment (= Sample). Additionally, it is possible to have a more complex experimental setup including multiple TMT experiments, possibly containing multiple fractions per experiment. All MS files which are labeled with the same “Experiment” name are summarized in the proteinGroups and peptides output of MaxQuant.
{: .tip}

# MSstatsTMT analysis

The protein groups and evidence files of MaxQuant can directly be used as an input for MSstatsTMT. MSstatsTMT automatically removes all proteins that are labelled as contaminants (‘+’ sign in the column ‘potential contaminant’ of both MaxQuant outputs). In case you want to prevent this step either run MaxQuant with "Include Contaminants: No" or remove the '+' sign from the MaxQuant files as described in the [label-free MSstats training]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-msstats-dda-lfq/tutorial.html). Additionally, MSstatsTMT will combine all quantitations according to the annotation file, which you have to provide (e.g all fractions of the same mixture are combined).
For this dataset, the standard parameters of MSstatsTMT fit quite nicely, as this is a straightforward two group comparison. If you have more than two groups, you may consider using the comparison matrix option of MSstatsTMT (for more information see [label-free MSstats training]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-msstats-dda-lfq/tutorial.html)).


> <hands-on-title>MSstatsTMT analysis</hands-on-title>
>
> 1. {% tool [MSstatsTMT](toolshed.g2.bx.psu.edu/repos/galaxyp/msstatstmt/msstatstmt/2.0.0+galaxy0) %} with the following parameters:
>    - *"Input Source"*: `MaxQuant`
>        - {% icon param-file %} *"evidence.txt - feature-level data"*: `evidence` (output of **MaxQuant** {% icon tool %})
>        - {% icon param-file %} *"proteinGroups.txt"*: `proteinGroups` (output of **MaxQuant** {% icon tool %})
>        - {% icon param-file %} *"annotation.txt"*: `annotation file`
>    - In *"Plot Output Options"*:
>        - *"Select protein IDs to draw plots"*: `Option for QC plot: "allonly" will generate one QC plot with all proteins`
>    - *"Compare Groups"*: `Yes`
>        - In *"Comparison Plot Options"*:
>            - *"Display protein names in Volcano Plot."*: `No`
>    - *"Select Outputs"*: `MSstatsTMT summarization log` `Protein Abundance` `QC Plot`
{: .hands_on}

> <tip-title>Waiting for MSstatsTMT result</tip-title>
> This training consists of a real proteomics experiments. Therefore, calculation times take longer than in usual Galaxy tutorials. You may want to come back later to the training or use the time to prepare your own data for the analysis with MaxQuant and MSstatsTMT with the tips provided in this training (yellow boxes).
{: .tip}

> <question-title></question-title>
>
> 1. How many differentially regulated proteins were detected in the MSstatsTMT group comparison? 
>
> > <solution-title></solution-title>
> >
> > 1. 7139 (numbers might slightly vary with different MSstatsTMT versions); number of lines minus header line in the MSstatsTMT group comparison output
> >
> {: .solution}
>
{: .question}

> <tip-title>Preparing an MSstatsTMT annotation file for your own analysis</tip-title>
> 
> For this tutorial all files needed for the analysis are provided. Nevertheless here are some guidelines for the generation of annotation file for MSstatsTMT for your own data analyses.
> As MSstatsTMT has to perform statistical analyses, you have to provide the following basic information: Which quantitation channel in every sample corresponds to which original biological sample and which biological condition (e.g WT or knock-down) does this sample belong to? To fully automate the process of choosing an appropriate statistical model for analysis, MSstatsTMT needs an annotation matrix that is extremely powerful because it can deal with any experimental setup. 
> 
> The annotation file contains the following columns:
> - **Run** corresponds to your MS file names. Please note that MaxQuant in Galaxy will add a “.thermo” to your raw file name as it appears in your history. Since we first analyse the data with MaxQuant, MSstatsTMT needs this additional extension for this annotation file. The correct run names can be found after MaxQuant is finished in the evidence file in the column ‘Raw file’.
> - **Fraction** corresponds to the aforementioned optional pre-fractionation fraction of your samples (e.g high pH RP, or SCX)
> - **TechRepMixture** corresponds to the n-th time this fraction was measured. If you measured the first Fraction of your first sample twice, Fraction would stay the same, but the TechRepMixture would have to differ.
> - **Channel** corresponds to the TMT channel which was used in your Experiments.We recommend to always include all channels in your searches and your annotations, even when these channels are empty. So if you only used 9 of 11 channels in your TMT experiments, add the last 2 channels anyway and just label them as “Empty” in the Condition column. In Galaxy the channel names need to be named ‘channel.X’, with X being the number of the channel.
> - **Condition** represents the biological condition, which the observation belongs to (e.g Ctrl). Normalization Channel are labeled as “Norm”. Empty Channels as “Empty”
> - **Mixture** corresponds to the “TMT experiment” mentioned above.
> - **BioReplicate** corresponds to the biological specimen this observation derives from.
> On first glance this might sound overwhelming, but used correctly, this annotation matrix can describe very complex experimental setups in a definitive way.
> 
> An example: 
>
> Consider you perform a study on the difference between primary liver tumors and liver tumor metastases. It makes a very big difference whether all your samples derive from different patients, or whether you were able to obtain some primary tumors and metastases samples from the same patient. The samples obtained from the same patient are expected to be related to each other (they are not independent observations and this must be considered during statistical testing).
> A short illustration: 
> You measured 2 different samples of one patient (metastasis and primary tumor sample).This patient would be considered patient 1 and you would label the primary tumor with the first channel of a TMT-11 plex kit (126) and you would label the metastasis with the second channel (127N). After labeling you combine both samples in one Mixture and measure them in one MS run.
>
> Your annotation file for this case would look like this:
>
> | Run                | Fraction | TechRepMixture | Channel | Condition | Mixture | BioReplicate |
> |--------------------|----------|----------------|---------|-----------|---------|--------------|
> | File1.thermo.raw   | 1        | 1              |channel.1|primary    |mixture1 |1             |
> | File1.thermo.raw   | 1        | 1              |channel.2|metastasis |mixture1 |1             |
>
> 
> Let us assume you now have 10 primary tumors and 10 metastases (from 20 different patients). Since you do not have 20plex capabilities, you have to divide your samples into two experiments / mixtures. For simplicity, after labeling and combining all channels belonging to the same experiments, the mixtures are directly measured ( 1 fraction per Mixture). To avoid batch effects due to allocation you use randomization. In order to be able to normalize all quantitations you use a reference sample, which you add to each experiment (this is referred to as “Normalization Channel”). The reference sample is ideally a mixture of all your samples. 
>
> The annotation matrix would now look like this:
>
> | Run                | Fraction | TechRepMixture | Channel  | Condition | Mixture | BioReplicate |
> |--------------------|----------|----------------|----------|-----------|---------|--------------|
> | File1.thermo.raw   | 1        | 1              |channel.1 |primary    |mixture1 |1             |
> | File1.thermo.raw   | 1        | 1              |channel.2 |metastasis |mixture1 |2             |
> | File1.thermo.raw   | 1        | 1              |channel.3 |primary    |mixture1 |3             |
> | File1.thermo.raw   | 1        | 1              |channel.4 |metastasis |mixture1 |4             |
> | File1.thermo.raw   | 1        | 1              |channel.5 |primary    |mixture1 |5             |
> | File1.thermo.raw   | 1        | 1              |channel.6 |metastasis |mixture1 |6             |
> | File1.thermo.raw   | 1        | 1              |channel.7 |primary    |mixture1 |7             |
> | File1.thermo.raw   | 1        | 1              |channel.8 |metastasis |mixture1 |8             |
> | File1.thermo.raw   | 1        | 1              |channel.9 |primary    |mixture1 |9             |
> | File1.thermo.raw   | 1        | 1              |channel.10|metastasis |mixture1 |10            |
> | File1.thermo.raw   | 1        | 1              |channel.11|Norm       |mixture1 |100           |
> | File2.thermo.raw   | 1        | 1              |channel.1 |primary    |mixture2 |1             |
> | File2.thermo.raw   | 1        | 1              |channel.2 |metastasis |mixture2 |2             |
> | File2.thermo.raw   | 1        | 1              |channel.3 |primary    |mixture2 |3             |
> | File2.thermo.raw   | 1        | 1              |channel.4 |metastasis |mixture2 |4             |
> | File2.thermo.raw   | 1        | 1              |channel.5 |primary    |mixture2 |5             |
> | File2.thermo.raw   | 1        | 1              |channel.6 |metastasis |mixture2 |6             |
> | File2.thermo.raw   | 1        | 1              |channel.7 |primary    |mixture2 |7             |
> | File2.thermo.raw   | 1        | 1              |channel.8 |metastasis |mixture2 |8             |
> | File2.thermo.raw   | 1        | 1              |channel.9 |primary    |mixture2 |9             |
> | File2.thermo.raw   | 1        | 1              |channel.10|metastasis |mixture2 |10            |
> | File2.thermo.raw   | 1        | 1              |channel.11|Norm       |mixture2 |100           |
>
>
> Normalization Channels are always labeled as “Norm” in Condition and get an arbitrary BioReplicate number. Since we want the Normalization channel to be a reference, it always has to be the same sample and therefore also should have the same BioReplicate number.
> If you have empty channels, label them as “Empty” in the Condition column and also give them an arbitrary BioReplicate number.
> 
> Exercise: 
>
> How would the annotation file look like for the following experiment:
> 27 samples (12 primary, 12 metastases samples from different patients + 3 reference samples) over 3 TMT experiments / mixtures. 
> You use Channels 1-8 in Exp1, 2-9 in Exp2 and 3-11 in Exp3. Your first used channel is always taken for the reference channel (Exp1: Channel1 = Norm, Exp2: Channel2 = Norm etc). 
> You fractionate all 3 experiments / mixtures into 3 fractions each and measure every MS sample twice (resulting in 18 measurements / raw files total). 
>
> The solution can be found on [Zenodo](https://zenodo.org/record/5195800/files/annotation_exercise.txt).
{: .tip}

For more information on MSstats parameters and ideas on how to follow up with your MSstatsTMT results visit the [label-free MSstats training]({{site.baseurl}}/topics/proteomics/tutorials/maxquant-msstats-dda-lfq/tutorial.html). 

# Follow up on MSstatsTMT results

We obtain several output files from MSstatsTMT. MSstats log file contains the MSstats report with warnings and information about the analysis steps. 
The QC report allows to visualize protein abundance of conditions for all proteins (first page) or if enabled for each individual protein (following pages). 

![QC report](../../images/maxquant-msstats-tmt/QC_plot.png "QC report for all samples, each boxplot summarizes the protein abundances of one sample")

The MSstatsTMT Group Comparison Plot is a very convenient pdf, which you can search for your favorite proteins of interest to check whether these are dysregulated between the conditions. The red error bar gives you a confidence interval of the fold change.

![Group comparison plot](../../images/maxquant-msstats-tmt/example_protein.png "Group comparison plot for the protein A5YKK6 (CCR4-NOT transcription complex subunit 1)")

The volcano plot plots the negative log10 of the adjusted P Value derived from statistical testing over the log2 fold change between the conditions you compared. A value of 3 on the y axis corresponds to an adjusted P Value of 0.001, whereas a value of 1 on the x axis corresponds to a ratio of 2 meaning that the protein is twice as high in condition 1 than condition 2..
Please bear in mind that TMT quantitation suffers from ratio compression. There are ways to calculate more accurate fold changes from measured data (or alternatively you can use MS3 quantitation on some machines). However, even with ratio compression, MS2 TMT is an excellent quantitation strategy to detect differentially regulated proteins. Just bear in mind that the reported fold chances of proteins are almost always smaller than the real fold changes between your conditions.

![Volcano plot](../../images/maxquant-msstats-tmt/volcano_plot.png "Volcano plot QC report for all samples, each boxplot summarizes the protein abundances of one sample")


topic: proteomics
tutorial: clinical-mp-1-database-generation
title: Clinical Metaproteomics 1: Database-Generation
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["[label-TMT11]"]
content_hash: 113e6a7484bc258c821fcfc28ea5ed66
last_modified: 2026-02-26T21:12:17.462483
zenodo_link: https://doi.org/10.5281/zenodo.10105821
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/proteomics/tutorials/clinical-mp-1-database-generation/tutorial.html

content:
Metaproteomics is the large-scale characterization of the entire complement of proteins expressed by microbiota. However, metaproteomics analysis of clinical samples is challenged by the presence of abundant human (host) proteins which hampers the confident detection of lower abundant microbial proteins {% cite Batut2018 %} ; {% cite Jagtap2015 %} .

To address this, we used tandem mass spectrometry (MS/MS) and bioinformatics tools on the Galaxy platform to develop a metaproteomics workflow to characterize the metaproteomes of clinical samples. This clinical metaproteomics workflow holds potential for general clinical applications such as potential secondary infections during COVID-19 infection, microbiome changes during cystic fibrosis as well as broad research questions regarding host-microbe interactions.

![Clinical Metaproteomics workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-overview.JPG %})


The first workflow for the clinical metaproteomics data analysis is the Database generation workflow. The Galaxy-P team has developed a workflow wherein a large database is generated by downloading protein sequences of known disease-causing microorganisms and then generating a compact database from the comprehensive database using the Metanovo tool.


![Database Generation Workflow]({% link topics/proteomics/images/clinical-mp/clinical-mp-database-generation.JPG %})



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data Upload

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/records/10105821/files/HUMAN_SwissProt_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/Species_UniProt_FASTA.fasta
>    https://zenodo.org/records/10105821/files/Contaminants_(cRAP)_Protein_Database.fasta
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F10_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F11_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F13_9Aug19_Rage_Rep-19-06-08.mgf
>    https://zenodo.org/records/10105821/files/PTRC_Skubitz_Plex2_F15_9Aug19_Rage_Rep-19-06-08.mgf
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
>
> 5. Optional-Add to each database a tag corresponding to the file name.
> 6. Create a dataset collection of the 4 MGF datasets.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Import Workflow


> <hands-on-title>Running the Workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy:
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/proteomics/tutorials/clinical-mp-1-database-generation/workflows/WF1_Database_Generation_Workflow.ga" title="Database Generation" %}
>
>
> 2. Run **Workflow** {% icon workflow %} using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - {% icon param-file %} *" Input Dataset collection"*: `MGF dataset collection`
>    - {% icon param-file %} *" Species_tabular"*: `Species_tabular.tabular`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

# Step-by-step analysis

## Download Protein Sequences using taxon names
First, we want to generate a large comprehensive protein sequence database using the UniProt XML Downloader to extract sequences for species of interest. To do so, you will need a tabular file that contains a list of species.

For this tutorial, a literature survey was conducted to obtain 118 taxonomic species of organisms that are commonly associated with the female reproductive tract {% cite AfiuniZadeh2018 %}. This species list was used to generate a protein sequence FASTA database was generated using the UniProt XML Downloader tool within the Galaxy framework. In this tutorial, the Species FASTA database (~3.38 million sequences) has already been provided as input. However, if you have your own list of species of interest as a tabular file (`Your_Species_tabular.tabular`), steps to generate a FASTA file from a tabular file are included:


## Download Species Protein Sequences using UniProt XML downloader with **UniProt**

> <hands-on-title> UniProt XML downloader</hands-on-title>
> 1. {% tool [UniProt](toolshed.g2.bx.psu.edu/repos/galaxyp/uniprotxml_downloader/uniprotxml_downloader/2.3.0) %} with the following parameters:
>    - *"Select"*: `Your_Species_tabular.tabular`
>        - {% icon param-file %} *"Dataset (tab separated) with Taxon ID/Name column"*: `output` (Input dataset)
>        - *"Column with Taxon ID/name"*: `c1`
>    - *"UniProt output format"*: `fasta`
> 2. Rename the output as Species_UniProt_FASTA.fasta
>
>    > <comment-title> UniProt description </comment-title>
>    >
>    > This tool will help download the protein fasta sequences by inputting the taxon names.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Can we use a higher taxonomy clade than species for the UniProt XML downloader?
> 2. Why are we using the tools separately? Can we run it all together?
> 3. Can we select multiple files together?
> 4. How many FASTA files can be merged at once, i.e. is there a limit on the number/size of files?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Yes, the UniProt XML downloader can also be used for generating a database from Genus, Family, Order, or any other higher taxonomy clade.
> > 2. The tools are run separately to reduce the load on the server and tool. If you have a limited number of taxon names, then you can run it all together.
> > 3. Yes, that certainly can be done. We used one input file at a time to maintain the order of sequences in the database.
> > 4. There is no limit.
> >
> {: .solution}
>
{: .question}


## Merging databases to obtain a large comprehensive database for MetaNovo
Once generated, the Species UniProt database (~3.38 million sequences) will be merged with the Human SwissProt database (reviewed only; ~20.4K sequences) and contaminant (cRAP) sequences database (116 sequences) and filtered to generate the large comprehensive database (~2.59 million sequences). The large comprehensive database will be used to generate a compact database using MetaNovo, which is much more manageable.

> <hands-on-title> Download contaminants with **Protein Database Downloader** </hands-on-title>
>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `cRAP (contaminants)`
> 2. Rename as "Protein Database Contaminants (cRAP)"
>
{: .hands_on}

> <hands-on-title> Human SwissProt (reviewed) database</hands-on-title>
> 1. {% tool [Protein Database Downloader](toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder/dbbuilder/0.3.4) %} with the following parameters:
>    - *"Download from?"*: `UniProtKB(reviewed only)`
>        - In *"Taxonomy"*: `Homo sapiens (Human)`
>        - In *"reviewed"*: `UniProtKB/Swiss-Prot (reviewed only)`
>	 - In  *"Proteome Set"*: `Reference Proteome Set`
>	 - In  *"Include isoform data"*: `False`
> 2. Rename as "Protein Database Human SwissProt".
{: .hands_on}

> <question-title></question-title>
>
> 1. How often is the Protein Database Downloader updated?
>
>
> > <solution-title></solution-title>
> >
> >
> > 1. It is updated every 3 months.
> >
> {: .solution}
>
{: .question}


> <hands-on-title> FASTA Merge Files and Filter Unique Sequences </hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Species_UniProt_FASTA ` (output of **UniProt XML downloader** {% icon tool %})
>
> 2. Rename out as "Human UniProt Microbial Proteins cRAP for MetaNovo".
>
{: .hands_on}

# Reducing Database size

## Metanovo tool generates a compact database from your comprehensive database with **MetaNovo**
Next, the large comprehensive database of ~2.59 million sequences can be reduced using the MetaNovo tool to generate a more manageable database that contains identified proteins.

The compact MetaNovo-generated database (~1.9K sequences) will be merged with Human SwissProt (reviewed only) and contaminants (cRAP) databases to generate the reduced database (~21.2k protein sequences) that will be used for peptide identification (see [Discovery Module tutorial](https://github.com/subinamehta/training-material/blob/main/topics/proteomics/tutorials/clinical-mp-discovery/tutorial.md)).

> <hands-on-title> MetaNovo </hands-on-title>
>
> 1. {% tool [MetaNovo](toolshed.g2.bx.psu.edu/repos/galaxyp/metanovo/metanovo/1.9.4+galaxy4) %} with the following parameters:
>    - *"MGF Input Type"*: `Collection`
>        - {% icon param-collection %} *"MGF Collection"*: `output` (Input dataset collection)
>    - {% icon param-file %} *"FASTA File"*: `output` (output of **FASTA Merge Files and Filter Unique Sequences** {% icon tool %})
>    - In *"Spectrum Matching Parameters"*:
>        - *"Fragment ion mass tolerance"*: `0.01`
>        - *"Enzyme"*: `Trypsin (no P rule)`
>        - *"Fixed modifications as comma separated list"*: `Carbamidomethylation of C` `TMT 10-plex of K` `TMT 10-plex of peptide N-term`
>        - *"Variable modifications as comma separated list"*: `Oxidation of M`
>        - *"Maximal charge to search for"*: `5`
>    - In *"Import Filters"*:
>        - *"The maximal peptide length to consider when importing identification files"*: `50`
> 2. Rename as "MetaNovo Compact Database".
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are we reducing the size of the database?
> 2. Why is this running TMT10 plex modification when the data is 11-plex?
> 3. Regarding MetaNovo Spectrum Matching parameters, what are the most “important” parameters? Meaning, that if a user wants to reduce or increase the sensitivity/number of output sequences, what should they change?
>
> > <solution-title></solution-title>
> >
> >
> > 1. Reducing the size of the database improves search speed, FDR, and sensitivity.
> > 2. There is no option for 11-plex modifications in Metanovo, hence we use the TMT-10plex.
> > 3. The most important parameters are the tolerance (MS1 and MS2) and any modifications introduced during the processing of the data.
> >
> {: .solution}
>
{: .question}

## Merging databases to obtain reduced MetaNovo database for peptide discovery with **FASTA Merge Files and Filter Unique Sequences**

> <hands-on-title> FASTA Merge Files and Filter Unique Sequences</hands-on-title>
>
> 1. {% tool [FASTA Merge Files and Filter Unique Sequences](toolshed.g2.bx.psu.edu/repos/galaxyp/fasta_merge_files_and_filter_unique_sequences/fasta_merge_files_and_filter_unique_sequences/1.2.0) %} with the following parameters:
>    - *"Run in batch mode?"*: `Merge individual FASTAs (output collection if input is collection)`
>        - In *"Input FASTA File(s)"*:
>            - {% icon param-repeat %} *"Insert Input FASTA File(s)"*
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Human SwissProt` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `Protein Database Contaminants (cRAP)` (output of **Protein Database Downloader** {% icon tool %})
>                - {% icon param-file %} *"FASTA File"*: `MetaNovo Compact Database` (output of **MetaNovo** {% icon tool %})
>
>
{: .hands_on}

# Conclusion

The first step for the Clinical Metaproteomics study is database generation. As we didn’t have a reference database or information from 16srRNA-seq data, we generated a fasta database doing a literature survey, however, if 16S rRNA data is present, the taxon identified can be used for a customized database generation. As the size of the comprehensive database is generally  too large, we used the Metanovo tool to reduce the size of the database. This reduced database will be then used for clinical metaproteomics discovery workflow.


topic: single-cell
tutorial: EBI-retrieval
title: Importing files from public atlases
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 4cffa72a6dd46a9ca047e37fc3313019
last_modified: 2026-02-26T21:12:17.894013
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/EBI-retrieval/tutorial.html

content:
anndata-scanpy-or-seurat-object-1
    date: 2024-01-10
  - label: "EBI SCXA retrieval tool (UseGalaxy.eu)-ARCHIVED"
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/ebi-scxa---anndata-scanpy-or-seurat-object
    date: 2024-12-10
  - label: "HCA Downloader (UseGalaxy.eu)"
    history: https://singlecell.usegalaxy.eu/u/j.jakiela/h/hca-downloader---example-files-1
    date: 2024-09-08
  - label: "HCA Downloader (UseGalaxy.eu)-ARCHIVED"
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/hca-downloader---example-files
    date: 2024-12-10

input_histories:
  - label: "UseGalaxy.eu"
    history: https://singlecell.usegalaxy.eu/u/j.jakiela/h/input-history-ebi-scxa---anndata-scanpy-or-seurat-object
  - label: "UseGalaxy.eu-ARCHIVED"
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/input-history-ebi-scxa---anndata-scanpy-or-seurat-object
    date: 2024-12-10


requirements:
  -
    type: "internal"
    topic_name: single-cell
    tutorials:
      - scrna-case_alevin
      - scrna-case_alevin-combine-datasets

follow_up_training:
  -
    type: "internal"
    topic_name: single-cell
    tutorials:
        - scrna-case_basic-pipeline

recordings:
  - speakers:
    - hexhowells
    date: '2024-09-12'
    length: 18M54S
    youtube_id: "Yx7m7Uh4tVg"

tags:
  - data import
  - data management

---

Public single cell datasets seem to accumulate by the second. Well annotated, quality datasets are slightly trickier to find, which is why projects like the [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home) (SCXA) exist - to curate datasets for public use. Here, we will guide you through transforming data imported from the SCXA repository into the input file required for the [Filter, Plot, Explore tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) and we will also show how to use the public atlases for your own research.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Getting data from the Single Cell Expression Atlas

Galaxy has a specific tool for importing data from the SCXA ({% cite Moreno2020.04.08.032698 %}), which combines all the preprocessing steps shown in [the corresponding tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}) into one! For this tutorial, the dataset can be seen [at the EBI](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/downloads) with experiment ID of `E-MTAB-6945`.

> <details-title> How to access other experiments? </details-title>
>
> You can search datasets according to various criteria either using search box in **Home** tab or choosing kingdom, experiment collection, technology type (and others) in **Browse experiments** tab. When you find the experiment you are interested in, just click on it and the experiment ID will be displayed in the website URL, as shown below.
>
> ![Arrow pointing to the website URL where you can find experiment ID.](../../images/scrna-data/exp_id.jpg "Where to find experiment ID on the EBI Single Cell Expression Atlas website.")
>
> Once you know the experiment ID, you can use EBI SCXA Data Retrieval tool in Galaxy!
>
{: .details}


> <hands-on-title>Retrieving data from Single Cell Expression Atlas</hands-on-title>
>
> 1. {% tool [EBI SCXA Data Retrieval](toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2) %} with the following parameters:
>      - *"SC-Atlas experiment accession"*: `E-MTAB-6945`
>      - *"Choose the type of matrix to download"*: `Raw filtered counts`
>
{: .hands_on}

It's important to note that this matrix is processed somewhat through the SCXA pipeline, which is quite similar to the pre-processing that has been shown in this case study tutorial series. The resultant datasets contain any and all metadata provided by the SCXA pipeline as well as the metadata contributed by the original authors (for instance, more cell or gene annotations). So while the AnnData object generated at the end of this tutorial will be similar to that generated using the [Alevin workflows]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}) on the original FASTQ files, some of the metadata will be slightly different. Relevant results and interpretation will not change, however!

## Examine the imported files

> <question-title></question-title>
>
> 1. What format has this tool imported?
>
> > <tip-title></tip-title>
> >
> > Selecting the title of each resultant dataset will expand the dataset in the Galaxy history.
> {: .tip}
> > <solution-title></solution-title>
> >
> > Matrix Market Format! We can tell this because our first file helpfully says `MatrixMarket` in the first line.
> > ![Green box containing first output, the matrix.mtx file. Columns are labelled 14458, 5218, and 5308559](../../tutorials/EBI-retrieval/images/matrix-output.png "Matrix Market Output")
> >
> > This {% icon param-file %} `matrix.mtx` file, in Matrix Market format, contains a column referring to each gene (column 1), to each cell (column 2), and the expression values themselves in the final column. To be useful, then, we need to know which genes and cells the numbers are referring to. That's why this format comes with two more files.
> >
> > ![Green box containing second output, the genes.tsv file. The first column contains EnsemblIDs such as ENSMUSG######, while the second column contains gene names. There are 14,457 lines.](../../tutorials/EBI-retrieval/images/genes-output.png "Genes Output")
> >
> > The {% icon param-file %} `genes.tsv` file lists each EnsemblID and its gene name. The lines (14,457) corresponds with the 14458 in the Matrix file...but the 14458 contains a header, so that's why it has one more than the genes file!
> >
> > ![Green box containing third output, the barcodes.tsv file. The file consists of 5,217 lines and a single column containing the cell barcode, variations of ERR2704656-AAAACACTCTGA.](../../tutorials/EBI-retrieval/images/cells-output.png "Cells Output")
> >
> > The {% icon param-file %} `barcodes.tsv` file lists each barcode. The lines (5,217) again correspond with the 5,218 lines in the Matrix file...which adds in the header again!
> >
> > ![Green box containing fourth output, the exp_design.tsv file. The file consists of 5,218 lines and numerous columns starting with 'Assay' and 'Sample Characteristic'.](../../tutorials/EBI-retrieval/images/cell-metadata.png "Experimental Design")
> >
> > Finally, and helpfully, the tool also includes cell metadata where the `Assay` column corresponds with the barcodes in the {% icon param-file %} `barcodes.tsv` file. While this is not a required file to create an AnnData object from the three matrix market files, it is extremely necessary for actually interpreting the data. Imagine not knowing which barcodes came from which sample!
> >
> {: .solution}
{: .question}


## Metadata manipulation

At this point you might want to do some modifications in the files before downstream analysis. That can include re-formating the cell metadata or changing the names of the column headers, it all depends on your dataset and how you want to perfrom your analysis. It's also fine to transform those files straight away. Here, we will show an extended version of metadata manipulation which allows us to create an input file consistent with [the next tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) workflow.

Before creating an AnnData object, we need to make a small modification in experimental design table. The dataset contains information about the 7 experimental samples (N701 – N707). However, in the {% icon param-file %} `exp_design.tsv` dataset, which contains the cell metadata, these samples are just numbered from 1 to 7.

> <details-title>How did we know this?</details-title>
>
> You can preview this column in the the {% icon param-file %} `exp_design.tsv` dataset by selecting the {% icon galaxy-eye %} in the {% icon galaxy-history %} Galaxy history. If you scroll to the right, and move to the column `Sample Characteristic[individual]`, you will find the batch information. Don't worry, we're about to rename and reformat this whole dataset to more useful titles. Make a note of the number of that column - number 12 - as we will need it to change the batch number to a batch name shortly.
>
{: .details}

The plotting tool that we will use later will fail if the entries are integers and not categorical values, so we will change `1` to `N01` and so on.

> <hands-on-title> Change batch numbers into names </hands-on-title>
>
> 1. Change the datatype of {% icon param-file %} `EBI SCXA Data Retrieval on E-MTAB-6945 exp_design.tsv` to `tabular`:
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 2. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.3) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `EBI SCXA Data Retrieval on E-MTAB-6945 exp_design.tsv`
>    - *"using column"*: `Column: 12`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `1`
>            - *"Replacement"*: `N01`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `2`
>            - *"Replacement"*: `N02`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `3`
>            - *"Replacement"*: `N03`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `4`
>            - *"Replacement"*: `N04`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `5`
>            - *"Replacement"*: `N05`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `6`
>            - *"Replacement"*: `N06`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `7`
>            - *"Replacement"*: `N07`
>
{: .hands_on}

While we're renaming things, let's also fix our titles.

> <hands-on-title> Change cell metadata titles </hands-on-title>
>
> 1. {% tool [Replace parts of text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: output from **Column Regex and Replace** {% icon tool %}
>    - In *"Find and Replace"*:
>        - *"Find pattern"*: `"Sample Characteristic[genotype]"`
>        - *"Replace with"*: `genotype`
>    - {% icon param-repeat %} *"Insert Find and Replace"*
>        - *"Find pattern"*: `"Sample Characteristic[individual]"`
>        - *"Replace with"*: `batch`
>    - {% icon param-repeat %} *"Insert Find and Replace"*
>        - *"Find pattern"*: `"Sample Characteristic[sex]"`
>        - *"Replace with"*: `sex`
>    - {% icon param-repeat %} *"Insert Find and Replace"*
>        - *"Find pattern"*: `"Sample Characteristic[cell type]"`
>        - *"Replace with"*: `cell_type`
> 2. Rename {% icon galaxy-pencil %} output `Cell metadata`
{: .hands_on}

## Check mitochondrial gene name format

We might like to flag mitochondrial genes. They can be identified quite easily since - depending on the species and formatting convention - their names often start with `mt`. Since tools for flagging mitochondrial genes are often case-sensitive, it might be a good idea to check the formatting of the mitochondrial genes in our dataset.

> <hands-on-title> Check the format of mitochondrial genes names </hands-on-title>
>
> 1. {% tool [Search in textfiles](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %}  *"Select lines from"*: `EBI SCXA Data Retrieval on E-MTAB-6945 genes.tsv (Raw filtered counts)`
>    - *"that"*: `Match`
>    - *"Regular Expression"*: `mt`
>    - *"Match type"*: `case insensitive`
>    - *"Output"*: `Highlighted HTML (for easier viewing)`
> 3. Rename {% icon galaxy-pencil %} output `Mito genes check`
>
{: .hands_on}

If you click on that dataset, you will see all the genes containing `mt` in their name. We can now clearly see that mitochondrial genes in our dataset start with `mt-`. Keep that in mind, we might use it in a moment!

Now we can create our single cell object!

{% include _includes/cyoa-choices.html option1="Scanpy" option2="Seurat" default="Scanpy"
       text="You can choose whether you want to create an AnnData object for Scanpy Analysis or an RDS object for Seurat Analysis. Galaxy has more resources for Scanpy analysis, but sometimes Seurat might have what you want. The two packages are constantly trying to outdo the other! It often depends on what is more 'standard' in your work environment!" %}

 <div class="Scanpy" markdown="1">


## Creating the AnnData object

We will do several modifications within the AnnData object so that you can follow [the next tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}).

> <hands-on-title> Create the AnnData Object </hands-on-title>
>
> 1. {% tool [Scanpy Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.9.3+galaxy0) %}
> 2. Make sure you are using version **1.8.1+galaxy9** of the tool (change by clicking on {% icon tool-versions %} Versions button):
>   ![List of available tool versions shown when clicking on the 'Versions' button on the top of the page.](../../images/scrna-casestudy/version.png "How to change the version of the tool")
>
> 3. Set the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `EBI SCXA Data Retrieval on E-MTAB-6945 matrix.mtx (Raw filtered counts)`
>    - *"Gene table"*:  `EBI SCXA Data Retrieval on E-MTAB-6945 genes.tsv (Raw filtered counts)`
>    - *"Barcode/cell table"*: `EBI SCXA Data Retrieval on E-MTAB-6945 barcodes.tsv (Raw filtered counts)`
>    - *"Cell metadata table"*: `Cell metadata`
> 4. Rename {% icon galaxy-pencil %} output `AnnData object`
>
{: .hands_on}

## AnnData manipulation

We will now change the header of the column containing gene names from `gene_symbols` to `Symbol`. This edit is only needed to make our AnnData object compatible with this tutorial's workflow. We will also flag the mitochondrial genes.

And the good news is that we can do both those steps using only one tool!

> <hands-on-title> Modify AnnData object </hands-on-title>
>
> 1. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0) %}
> 2. Make sure you are using version **1.8.1+galaxy92** of the tool (change by clicking on {% icon tool-versions %} Versions button)
> 3. Set the following parameters:
>    - {% icon param-file %} In *"Input object in hdf5 AnnData format"*: `AnnData object`
>    - In *"Change field names in AnnData var"*:
>        - {% icon param-repeat %} *"Insert Change field names in AnnData var"*
>            - *"Original name"*: `gene_symbols`
>            - *"New name"*: `Symbol`
>    - *"Gene symbols field in AnnData"*: `Symbol`
>    - In *"Flag genes that start with these names"*:
>        - {% icon param-repeat %} *"Insert Flag genes that start with these names"*
>            - *"Starts with"*: `mt-`
>            - *"Var name"*: `mito`
> 4. Rename {% icon galaxy-pencil %} output `Mito-counted AnnData for downstream analysis`
>
{: .hands_on}

And that's all! What's even more exciting about the {% icon tool %} **AnnData Operations** tool is that it automatically calculates a bunch of metrics, such as `log1p_mean_counts`, `log1p_total_counts`, `mean_counts`, `n_cells`, `n_cells_by_counts`, `n_counts`, `pct_dropout_by_counts`, and `total_counts`. Amazing, isn't it?

## Conclusion
Now you can use this object as input for the [Filter, Plot, Explore tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) and its associated workflow!

Even though this tutorial was designed specifically to modify the AnnData object to be compatible with the subsequent tutorial, it also shows useful tools that you can use for your own, independent data analysis. You can find the [workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/workflow-constructed-from-history-ebi-workflow) and the [answer key history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/ebi-scxa---anndata-scanpy-or-seurat-object-1). However, if you want to use the workflow from this tutorial, you have to keep in mind that different datasets may have different column names. So you have to check them first, and only then you can modify them.

</div>

<div class="Seurat" markdown="1">

## Creating the Seurat Object

> <hands-on-title> Create the Seurat Object </hands-on-title>
>
> 1. {% tool [Seurat Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/4.0.4+galaxy0) %}
> 2. Set the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `EBI SCXA Data Retrieval on E-MTAB-6945 matrix.mtx (Raw filtered counts)`
>    - *"Gene table"*:  `EBI SCXA Data Retrieval on E-MTAB-6945 genes.tsv (Raw filtered counts)`
>    - *"Barcode/cell table"*: `EBI SCXA Data Retrieval on E-MTAB-6945 barcodes.tsv (Raw filtered counts)`
>    - *"Cell metadata"*: `Cell metadata`
> 3. Rename {% icon galaxy-pencil %} output `Seurat object`
>
> You can also choose if you want to create Seurat object, Loom or Single Cell Experiment by selecting your option in *"Choose the format of the output"*.
>
{: .hands_on}

## Conclusion
And you're there! You now have a usable Seurat object for analysis with Seurat tools in your history! {% icon congratulations %} Congrats!

</div>


<!---
HCA doesn't work well for other datasets for now...
https://github.com/galaxyproject/training-material/issues/4567
-->

# Human Cell Atlas Matrix Downloader

Another public atlas that you can use to access the datasets is [Human Cell Atlas data portal](https://data.humancellatlas.org/). We will show you the tool in Galaxy which allows to retrieve expression matrices and metadata for any public experiment available in that repository.

To use it, simply set the project title, project label or project UUID, which can be found at the [HCA data browser](https://data.humancellatlas.org/explore/projects), and select the desired matrix format (Matrix Market or Loom).

![Image showing project UUID as a final fragment of link address, project title (self-explanatory) and project label as an entry in the box on the right side of the page.](../../images/scrna-data/HCA.jpg "Where to find project title, project label and project UUID")

For projects that have more than one organism, one needs to be specified. Otherwise, there is no need to set the species.

Let's use the suggested example of the project *Single cell transcriptome analysis of human pancreas*. If you check this project in HCA, you'll find out that it's actually its label. But it should work well if you enter the title or UUID!

> <hands-on-title>Create AnnData object</hands-on-title>
>
> {% tool [Human Cell Atlas Matrix Downloader](toolshed.g2.bx.psu.edu/repos/ebi-gxa/hca_matrix_downloader/hca_matrix_downloader/v0.0.4+galaxy0) %} with the following parameters:
>    - *"Human Cell Atlas project name/label/UUID"*: `Single cell transcriptome analysis of human pancreas`
>    - *"Choose the format of matrix to download"*:  `Matrix Market`
{: .hands_on}

> <warning-title>Errors that you might encounter</warning-title>
>  If your dataset turns red, there might be several reasons for that, for example:
>    - "There are too many connected users" - please be patient and re-run the step later, as it is advised
>    - "Project identifier was not found in the database" - double check the spelling, try entering project title, project label or project UUID.
>
{: .warning}

> <details-title>What will be the output?</details-title>
>
> When "Matrix Market" is seleted, outputs are in 10X-compatible Matrix Market format:
> - **Matrix (txt)**: Contains the expression values for genes (rows) and cells (columns) in raw counts. This text file is formatted as a Matrix Market file, and as such it is accompanied by separate files for the gene identifiers and the cells identifiers.
> - **Genes (tsv)**: Identifiers (column repeated) for the genes present in the matrix of expression, in the same order as the matrix rows.
> - **Barcodes (tsv)**: Identifiers for the cells of the data matrix. The file is ordered to match the columns of the matrix.
> - **Experiment Design file (tsv)**: Contains metadata for the different cells of the experiment.
>
> When "Loom" is selected, output is a single Loom HDF5 file:
> - **Loom (h5)**: Contains expression values for genes (rows) and cells (columns) in raw counts, cell metadata table and gene metadata table, in a [single HDF5 file](http://linnarssonlab.org/loompy/format/index.html).
>
{: .details}

If you chose **Loom** format and you need to convert your file to other datatype, you can use {% tool [SCEasy](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy1) %} (more details in the next section). If you chose **Matrix Market** format, you can then transform the output to AnnData or Seurat, as shown in the EBI SCXA example above. Below, you will find an example of transforming the output to AnnData object.


> <hands-on-title>Create AnnData object</hands-on-title>
>
> {% tool [Scanpy Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.8.1+galaxy9) %} with the following parameters:
>  Make sure you are using version **1.8.1+galaxy9** of the tool (change by clicking on {% icon tool-versions %} Versions button)
>    - *"Expression matrix in sparse matrix format (.mtx)"*: `Human Cell Atlas Matrix Downloader on matrix.mtx`
>    - *"Gene table"*:  `Human Cell Atlas Matrix Downloader on genes.tsv`
>    - *"Barcode/cell table"*: `Human Cell Atlas Matrix Downloader on barcodes.tsv`
>    - *"Cell metadata table"*: `Human Cell Atlas Matrix Downloader on exp_design.tsv`
{: .hands_on}

> <tip-title>Flagging genes by using AnnData Operations</tip-title>
>
> After you create AnnData file, you can additionally use the {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92) %} tool (note the version **1.8.1+galaxy92**) before downstream analysis. It's quite a useful tool since not only does it flag mitochondrial genes, but also automatically calculates a bunch of metrics, such as `log1p_mean_counts`, `log1p_total_counts`, `mean_counts`, `n_cells`, `n_cells_by_counts`, `n_counts`, `pct_dropout_by_counts`, and `total_counts`.
>
> When you use it to flag mitochondrial genes, here are some formatting tips:
> - Remember to check the name of the column with gene symbols
> - This tool is case sensitive
> - No parentheses needed when typing in the values
> - Including a dash is important to identify mitochondrial genes (eg. **MT-**)
{: .tip}

You can have a look at the [answer history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/hca-downloader---example-files-1) of performing those three quick steps.


topic: single-cell
tutorial: scatac-standard-processing-snapatac2
title: Single-cell ATAC-seq standard processing with SnapATAC2
difficulty: intermediate
time_estimation: 4H
tools_json: []
requirements_json: []
tags_json: ["10x", "epigenetics", "single-cell"]
content_hash: 33d2504abc035943978abbd5c2135e36
last_modified: 2026-02-26T21:12:17.931695
zenodo_link: https://zenodo.org/records/12707159
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.html

content:
{scATAC-seq} analysis is a method to decipher the chromatin states of the analyzed cells. In general, genes are only expressed in accessible (i.e. "open") chromatin and not in closed chromatin.
By analyzing which genomic sites have an _open_ chromatin state, cell-type specific patterns of gene accessibility can be determined.
{scATAC-seq} is particularly useful for analyzing tissue containing different cell populations, such as {PBMCs}.

In this tutorial we will analyze {scATAC-seq} data using the tool suites [SnapATAC2](https://kzhang.org/SnapATAC2/version/2.5/index.html) ({% cite Zhang2024 %}) and [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({%cite Wolf2018%}).
With both of these tool suites we will perform preprocessing, clustering and identification of {scATAC-seq} datasets from [10x Genomics](https://www.10xgenomics.com/products/single-cell-atac).
The analysis will be performed using a dataset of {PBMCs} containing ~4,620 single nuclei.

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/tutorial_mode.md %}


<!-- This is a comment. -->

> <comment-title></comment-title>
>
> This tutorial is significantly based on ["Standard pipeline" tutorial from SnapATAC2](https://kzhang.org/SnapATAC2/version/2.5/tutorials/pbmc.html). It can be seen as the {scATAC-seq} counterpart to the scRNA-seq tutorial [Clustering 3K PBMCs with Scanpy]( {% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %} ).
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# {scATAC-seq} with 10X Genomics
ATAC-seq utilizes a hyperactive Tn5 transposase ({% cite Kia2017 %}) to ligate adaptors to genome fragments, created by the transposase. Performing ATAC-seq on individual cells used to be an expensive and time-consuming labour.
The 10X Chromium NextGEM system made {scATAC-seq} a cost-effective method for gaining high-resolution data with a simple protocol.
After the transposition of nuclei in bulk, individual nuclei are put into Gel beads in Emulsion (GEM), containing unique 10x cell barcodes and sequencing adaptors for Illumina sequencing.
![Library Preparation]({% link topics/single-cell/images/scatac-standard-snapatac2/tenx_libprep_scatac.png %} "An overview of the 10X single-nuclei ATAC-seq library preparation")


# Data

The 5k {PBMCs} dataset for this tutorial is available for free from [10X Genomics](https://www.10xgenomics.com/datasets/5-k-peripheral-blood-mononuclear-cells-pbm-cs-from-a-healthy-donor-next-gem-v-1-1-1-1-standard-2-0-0). The blood samples were collected from a healthy donor and were prepared following the Chromium Next GEM scATAC-seq protocol. After sequencing on Illumina NovaSeq, the reads were processed by the **Cell Ranger ATAC 2.0.0** pipeline from 10X to generate a [*Fragments File*](https://support.10xgenomics.com/single-cell-atac/software/pipelines/latest/output/fragments).

> <details-title>Fragments File </details-title>
>
>   The Fragments File is a tabular file in a [BED](https://genome.ucsc.edu/FAQ/FAQformat.html#format1)-like format, containing information about the position of the fragments on the chromosome and their corresponding 10x cell barcodes.
>
{: .details}
SnapATAC2 requires 3 input files for the standard pathway of processing:
- `fragments_file.tsv`: A tabular file containing the chromosome positions of the fragments and their corresponding 10x cell barcodes.
- `chrom_sizes.txt`: A tabular file of the number of bases of each chromosome of the reference genome
- `gene_annotation.gtf.gz`: A tabular file listing genomic features of the reference genome (GENCODE GRCh38)

> <details-title>Chromosome sizes </details-title>
>
> - A chromosome sizes file can be generated using the tool {% tool [Compute sequence length](toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.3) %}.
> - The reference genome can either be selected from cached genomes or uploaded to the galaxy history.
>
{: .details}

> <comment-title></comment-title>
> - This tutorial starts with a `fragment` file.
> - SnapATAC2 also accepts mapped reads in a `BAM` file.
> - To learn how to get a `fragment` file or `BAM` file from raw `.FASTQ`-reads, please check out the tutorial ["Pre-processing of 10X Single-Cell ATAC-seq Datasets"]( {% link topics/single-cell/tutorials/scatac-preprocessing-tenx/tutorial.md %} )
> - If you would like to start the analysis with a `BAM` file, you can expand the details section ["Details: Creating a fragment file"]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#creating-a-fragment-file).
{: .comment}



## Get Data
> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the `fragments_file.tsv`, `chrom_sizes.txt` and `gene_annotation.gtf.gz` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>
>    ```
>    {{ page.zenodo_link }}/files/atac_pbmc_5k_nextgem_fragments.tsv.gz
>    {{ page.zenodo_link }}/files/chrom_sizes.txt
>    {{ page.zenodo_link }}/files/gencode.v46.annotation.gtf.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
>   - {% icon galaxy-pencil %} **Rename** the file `atac_pbmc_5k_nextgem_fragments.tsv` to `fragments_file.tsv`
>   - {% icon galaxy-pencil %} **Rename** the file `gencode.v46.annotation.gtf.gz` to `gene_annotation.gtf.gz`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Inspect `chrom_sizes` and `fragments_file`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many chromosomes are in `chrom_sizes`?
> 2. In which column are the cell barcodes stored in the `fragments_file`?
>
> > <solution-title></solution-title>
> >
> > 1. There are 25 chromosomes. The 22 autosomes (Chr. 1-22), both sex chromosomes (Chr. X and Y) and the small circular mitochondrial chromosome (Chr. M).
> > 2. The cell barcodes are unique 16 bp oligos, located in column 4.
> >
> {: .solution}
>
{: .question}

## Creating a fragment file
> <details-title>Creating a fragment file</details-title>
>  > <hands-on-title>fragment file</hands-on-title>
>  > 1. Import the file  `BAM_500-PBMC` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>  > ```
>  > {{ page.zenodo_link }}/files/BAM_500-PBMC.bam
>  > ```
>  >   - This dataset contains mapped reads in the `BAM` format.
>  >   - It was generated by following the tutorial ["Pre-processing of 10X Single-Cell ATAC-seq Datasets"]( {% link topics/single-cell/tutorials/scatac-preprocessing-tenx/tutorial.md %} ) until the output of {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.18) %}
>  > 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>  >    - *"Method used for preprocessing"*: `Convert a BAM file to a fragment file, using 'pp.make_fragment_file'`
>  >        - {% icon param-file %} *"File name of the BAM file"*: `BAM_500-PBMC` (Input dataset)
>  >        - {% icon param-toggle %} *"Indicate whether the BAM file contain paired-end reads"*: `Yes`
>  >        - *"How to extract barcodes from BAM records?"*: `From read names using regular expressions`
>  >          - *"Extract barcodes from read names of BAM records using regular expressions"*: `(................):`
>  >
>  >    > <comment-title></comment-title>
>  >    > - Not every regular expression type is supported.
>  >    > - This expression selects 16 characters if they are followed by a colon. Only the cell barcodes of the `BAM` file will match.
>  >    {: .comment}
>  >
>  > 3. Rename the generated file to `Fragments 500 PBMC`
>  > 4. Now you can continue with either the `fragments_file` from earlier or the new file `Fragments 500 PBMC`.
>  >    - {% icon galaxy-info %} The tool `pp.make_fragment_file` {% icon tool %} has implemented additional {QC} measures. These filter out larger fragments, which will be noticeable in the log-scale fragment size distribution.
>  >    - {% icon galaxy-info %} Please note that `Fragments 500 PBMC` only contains 500 {PBMCs} and thus the clustering will produce different outputs compared to the outputs generated by `fragments_file` (with 5k PBMC).
>  {: .hands_on}
{: .details}

# Preprocessing

Preprocessing of the scATAC-seq data contained in the `fragment` file with SnapATAC2 begins with importing the files and computing basic {QC} metrics.

SnapATAC2 compresses and stores the fragments into an `AnnData` object.

## AnnData
The [`AnnData`](https://anndata.readthedocs.io/en/latest/) format was initially developed for the [`Scanpy`](https://scanpy.readthedocs.io/en/stable/index.html) package and is now a widely accepted data format to
store annotated data matrices in a space-efficient manner.

![Anndata format]({% link topics/single-cell/images/scatac-standard-snapatac2/anndata_schema.svg %} "<code>AnnData</code> format stores a count matrix <code>X</code> together with annotations of
observations (i.e. cells) <code>obs</code>, variables (i.e. genes) <code>var</code> and unstructured annotations <code>uns</code>.")


## Import files to SnapATAC2

> <hands-on-title> Create an AnnData object </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Import data fragment files and compute basic QC metrics, using 'pp.import_data'`
>        - {% icon param-file %} *"Fragment file, optionally compressed with gzip or zstd"*: `fragments_file.tsv` (Input dataset)
>        - {% icon param-file %} *"A tabular file containing chromosome names and sizes"*: `chrom_sizes.txt` (Input dataset)
>        - {% icon param-toggle %} *"Whether the fragment file has been sorted by cell barcodes"*: `No`
>
>    > <details-title>Sorted by barcodes</details-title>
>    > - This tool requires the fragment file to be sorted according to cell barcodes.
>    > - If **pp.make_fragment_file** {% icon tool %} was used to generate the fragment file, this has automatically been done.
>    >   - Otherwise, the setting *"sorted by cell barcodes"* should remain `No`.
>    {: .details}
>
> 2. Rename the generated file to `Anndata 5k PBMC`
>
> 3. Check that the format is `h5ad`
{: .hands_on}

Because the `AnnData` format is an extension of the HDF5 format, i.e. a binary format, an `AnnData` object can not be inspected directly in Galaxy by clicking on the {% icon galaxy-eye %} (**View data**) icon.
Instead, we need to use a dedicated tool from the **AnnData** suite.

> <hands-on-title>Inspect an AnnData object</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC`
>    - *"What to inspect?"*: `General information about the object`
>
> 2. {% icon galaxy-eye %} Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 14232 × 0
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito'
>    >  uns: 'reference_sequences'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. How many observations are there? What do they represent?
>    > 2. How many variables are there?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 14232 observations, representing the cells.
>    > > 2. There are 0 variables, representing genomic regions. This is because genome-wide 500-bp bins are only added after initial filtering.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    > <tip-title>Faster Method for General Information</tip-title>
>    > * Many toolsets producing outputs in *AnnData* formats in Galaxy, provide the general information by default:
>    >    * Click on the name of the dataset in the history to expand it.
>    >    * General Anndata information will be given in the expanded box:
>    >
>    >      e.g.
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    14232 × 0
>    >      ```
>    >    * {% icon details %} This feature isn't the most reliable and might display:
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    1 × 1
>    >      ```
>    > * In such cases and for more specific queries, {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} is required.
>    {: .tip}
>
{: .hands_on}


## Calculate and visualize {QC} metrics

> <hands-on-title> Fragment-size distribution </hands-on-title>
>
> 1. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot fragment size distribution, using 'pl.frag_size_distr'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC` (output of **pp.import_data** {% icon tool %})
> 2. {% icon galaxy-eye %} Inspect the `.png` output
> 3. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot fragment size distribution, using 'pl.frag_size_distr'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC` (output of **pp.import_data** {% icon tool %})
>        - *"Change the y-axis (fragment counts) to log scale"*: `Yes`
> 4. {% icon galaxy-eye %} Inspect the `.png` output
>
>
>  ![fragment_size_distribution]({% link topics/single-cell/images/scatac-standard-snapatac2/pl.frag_size.png %})
>
>  ![log_fragment_size_distribution]({% link topics/single-cell/images/scatac-standard-snapatac2/log_pl.frag_size.png %})
> > <question-title></question-title>
> >
> > 1. What distinct features do the plots have? And what do they represent?
> > 2. Which fragments are generally from open chromatin?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 3 peaks are clearly visible (at <100-bp, ~200-bp and ~400-bp). The smallest fragments are from nucleosome-free regions, while the larger peaks of 200- and 400-bp contain mono- and di-nucleosome fragments, respectively.
> > > 2. The small fragments (<100-bp) are from open chromatin reads, since the Tn5 transposase could easily access the loosely packed DNA ({% cite Yan2020 %}).
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}


The {TSSe} is another important {QC} metric. Nucleosome-free fragments are expected to be enriched at {TSS}. TSSe shows increased fragmentation of chromatin around the TSS. This suggests open and accessible nucleosome-free chromatin.

{TSSe} is used as a QC metric, since an increased enrichment around TSS regions suggests that the experiment has captured biological meaningful genomic features.
TSSe scores of individual cells can be calculated using SnapATAC2's *metrics.tsse* function.

> <hands-on-title> Calculate and Plot TSSe </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute the TSS enrichment score (TSSe) for each cell, using 'metrics.tsse'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC` (output of **pp.import_data** {% icon tool %})
>        - {% icon param-file %} *"GTF/GFF file containing the gene annotation"*: `gene_annotation.gtf.gz` (Input dataset)
>
> 2. Rename the generated file to `Anndata 5k PBMC TSSe` or add the tag {% icon galaxy-tags %} `TSSe` to the dataset:
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>
> 3. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the TSS enrichment vs. number of fragments density figure, using 'pl.tsse'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC TSSe` (output of **metrics.tsse** {% icon tool %})
> 4. {% icon galaxy-eye %} Inspect the `.png` output
>
>
> ![TSSe plot against number of unique fragments]({% link topics/single-cell/images/scatac-standard-snapatac2/pl.tsse.png %})
> High-quality cells can be identified in the plot of {TSSe} scores against a number of unique fragments for each cell.
>
> > <question-title></question-title>
> >
> > 1. Where are high-quality cells located in the plot?
> > 2. Based on this plot, how should the filter be set?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. The cells in the upper right are high-quality cells, enriched for {TSS}. Fragments in the lower left represent low-quality cells or empty droplets and should be filtered out.
> > > 2. Setting the minimum number of counts at 5,000 and the minimum TSS enrichment to 10.0 is an adequate filter.
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

## Filtering
Based on the {TSSe} plot the cells can be filtered by TSSe and fragment counts.

> <hands-on-title> Filter cells </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC TSSe` (output of **metrics.tsse** {% icon tool %})
>        - *"Minimum number of counts required for a cell to pass filtering"*: `5000`
>        - *"Minimum TSS enrichemnt score required for a cell to pass filtering"*: `10.0`
>        - *"Maximum number of counts required for a cell to pass filtering"*: `100000`
>
> 2. Rename the generated file to `Anndata 5k PBMC TSSe filtered` or add the tag {% icon galaxy-tags %} `filtered` to the dataset
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4564 × 0
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse'
>    >  uns: 'reference_sequences'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. How have the observations changed, compared to the first `Anndata 5k PBMC` AnnData file?
>    > 2. What does this tell us about the quality of the data?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are only 4564 observations, compared to the initial 14232 observations.
>    > >
>    > >    And the `obs: 'tsse'` has been added (but already during **metrics.tsse**{% icon tool %})
>    > > 2. The empty droplets and low-quality cells have been filtered out, leaving us with 4564 high-quality cells.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}

## Feature selection
Currently, our AnnData matrix does not contain any variables. The variables will be added in the following step with the function *pp.add_tile_matrix*.
This creates a cell-by-bin matrix containing insertion counts across genome-wide 500-bp bins.

After creating the variables, the most accessible features are selected.
> <hands-on-title> Select features </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Generate cell by bin count matrix, using 'pp.add_tile_matrix'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC TSSe filtered` (output of **pp.filter_cells** {% icon tool %})
>        - *"The size of consecutive genomic regions used to record the counts"*: `500`
>        - *"The strategy to compute feature counts"*: `"insertion": based on the number of insertions that overlap with a region of interest`
>
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Perform feature selection, using 'pp.select_features'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata tile_matrix` (output of **pp.add_tile_matrix** {% icon tool %})
>        - *"Number of features to keep"*: `250000`
>
>    > <comment-title> Select features </comment-title>
>    >
>    > - Including more features improves resolution and can reveal finer details, but it may also introduce noise.
>    >    - To optimize results, experiment with the `n_features` parameter to find the most appropriate value for your dataset.
>    >
>    >  > <details-title> Different number of features </details-title>
>    >  > - To demonstrate the differences when selecting features, the following UMAP plots are the outputs from processing with a number of features between 1,000 and 500,000.
>    >  > - Fewer features result in fewer, but larger clusters. Selecting a lot of features will output more granular clusters and the compute time will increase.
>    >  > ![Different number of features UMAP]({% link topics/single-cell/images/scatac-standard-snapatac2/number_features.png %}"UMAP plots with different selected features")
>    >  >
>    >  {: .details}
>    > - At this step you can provide a blacklist or whitelist to specifically select relevant features.
>    >    - For example the [**ENCODE Blacklist**](https://github.com/Boyle-Lab/Blacklist) ({% cite Amemiya2019 %}) can be applied here.
>    {: .comment}
>
> 3. Rename the generated file to `Anndata 5k PBMC select_features` or add the tag {% icon galaxy-tags %} `select_features` to the dataset
> 4. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4564 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse'
>    >  var: 'count', 'selected'
>    >  uns: 'reference_sequences'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. How did `n_vars` change compared to `Anndata 5k PBMC TSSe filtered`
>    > 2. Where are the selected features stored in the count matrix?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 6,062,095 variables, compared to 0 in `TSSe filtered`.
>    > > 2. Selected features are stored in `var: 'selected'`.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Doublet removal

Doublets are removed by calling a customized [**scrublet**](https://github.com/AllonKleinLab/scrublet) ({% cite Wolock2019 %}) algorithm. *pp.scrublet* will identify potential doublets and the function *pp.filter_doublets* removes them.

> <details-title>Doublets in Single-cell datasets</details-title>
>
> - During single-cell sequencing, multiple nuclei can be encapsulated into the same 10x gel bead. The resulting multiplets (>97% of which are doublets) produce sequences from both cells.
> - Doublets can confound the results by appearing as "new" clusters or artifactual intermediary cell states.
>    - These problematic doublets are called **neotypic** doublets, since they appear as "new" cell types.
> - **Scrublet** (Single-cell Remover of Doublets) is an algorithm which can detect neotypic doublets that produce false results.
>    - The algorithm first simulates doublets by combining random pairs of observed cell features.
>    - The observed features of the "cells" are then compared to the simulated doublets and scored on their doublet probability.
>    - SnapATAC2's *pp.filter_doublets* then removes all cells with a doublet probability >50%.
>
> ![Doublet removal with scrublet]({% link topics/single-cell/images/scatac-standard-snapatac2/doublets-and-scrublet.png %} "Scrublet simulates expected doublets and produces doublet scores for each cell. ({% cite Wolock2019 %})")
>
{: .details}

> <hands-on-title> Scrublet </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute probability of being a doublet using the scrublet algorithm, using 'pp.scrublet'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC select_features` (output of **pp.select_features** {% icon tool %})
>
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Remove doublets according to the doublet probability or doublet score, using 'pp.filter_doublets'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata scrublet` (output of **pp.scrublet** {% icon tool %})
>        - *"Threshold for doublet probability"*: `0.5`
> 3. Rename the generated file to `Anndata 5k PBMC filter_doublets` or add the tag {% icon galaxy-tags %} `filter_doublets` to the dataset
> 4. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score'
>    >  var: 'count', 'selected'
>    >  uns: 'doublet_rate', 'reference_sequences', 'scrublet_sim_doublet_score'
>    >  obsm: 'fragment_paired'
>    > ```
>    >
>    > 1. What was removed by **pp.filter_doublets**?
>    > 2. Where are the new annotations stored?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Cell doublets were removed. `n_obs` was reduced from 4564 to 4430 cells.
>    > > 2. The outputs of **pp.scrublet** are stored in observations `obs: 'doublet_probability', 'doublet_score'` and in unstructured annotations `uns: 'scrublet_sim_doublet_score'`. The output of **pp.filter_doublets** is stored in `uns: 'doublet_rate'`.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>
{: .hands_on}


# Dimension reduction

Dimension reduction is a very important step during the analysis of single cell data. During this, the complex multi-dimensional data is projected into lower-dimensional space, while the lower-dimensional embedding of the complex data retains as much information as possible. Dimension reduction enables batch correction, data visualization and quicker downstream analysis since the data is more simplified and the memory usage is reduced ({% cite Zhang2024%}).


> <details-title>Dimension reduction with SnapATAC2</details-title>
>
> - Dimension reduction algorithms can be either linear or non-linear.
> - Linear methods are generally computationally efficient and well scalable.
>
>   A popular linear dimension reduction algorithm is:
>     - **PCA** (Principle Component Analysis), implemented in **Scanpy** (please check out our [Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial for an explanation).
> - Nonlinear methods however are well suited for multimodal and complex datasets.
>     - in contrast to linear methods, which often preserve global structures, non-linear methods have a locality-preserving character.
>     - This makes non-linear methods relatively insensitive to outliers and noise while emphasizing natural clusters in the data ({% cite Belkin2003%})
>     - As such, they are implemented in many algorithms to visualize the data in 2 dimensions (f.ex. **UMAP** embedding).
> - The nonlinear dimension reduction algorithm, through *matrix-free spectral embedding*, used in **SnapATAC2** is a very fast and memory efficient non-linear algorithm ({% cite Zhang2024%}).
>     - **Spectral embedding** utilizes an iterative algorithm to calculate the **spectrum** (*eigenvalues* and *eigenvectors*) of a matrix without computing the matrix itself.
> - For a simple introduction into *spectral embedding* and how it compares to *PCA*, please check out the blog post ["On Laplacian Eigenmaps for Dimensionality Reduction"](https://juanitorduz.github.io/laplacian_eigenmaps_dim_red/) by Juan Orduz.
>
{: .details}

## Spectral embedding
The dimension reduction, produced by the algorithm *tl.spectral*, is required for later steps, such as plotting and clustering.

> <hands-on-title> Spectral embedding </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Perform dimension reduction using Laplacian Eigenmap, using 'tl.spectral'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC filter_doublets` (output of **pp.filter_doublets** {% icon tool %})
>        - *"Distance metric"*: `cosine`
>
>    > <comment-title> Distance metric </comment-title>
>    >
>    > - The fast and well scalable *"matrix-free spectral embedding"* algorithm depends on the distance metric: `cosine`
>    {: .comment}
>
> 2. Rename the generated file to `Anndata 5k PBMC spectral` or add the tag {% icon galaxy-tags %} `spectral` to the dataset
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score'
>    >  var: 'count', 'selected'
>    >  uns: 'doublet_rate', 'reference_sequences', 'scrublet_sim_doublet_score', 'spectral_eigenvalue'
>    >  obsm: 'fragment_paired', 'X_spectral'
>    > ```
>    >
>    > Where are the new annotations stored?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The outputs of **tl.spectral** are stored in unstructured annotations `uns: 'spectral_eigenvalue'` and as multidimensional observations `obsm: 'X_spectral'`.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}
## UMAP embedding
With the already reduced dimensionality of the data stored in `X_spectral`, the cells can be further embedded (i.e. transformed into lower dimensions) with {UMAP}.
**UMAP** projects the cells and their relationship to each other into 2-dimensional space, which can be easily visualized ({% cite McInnes2018%}).

> <hands-on-title> UMAP embedding </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute Umap, using 'tl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC spectral` (output of **tl.spectral** {% icon tool %})
>        - *"Use the indicated representation in '.obsm'"*: `X_spectral`
>
> 2. Rename the generated file to `Anndata 5k PBMC UMAP` or add the tag  {% icon galaxy-tags %} `UMAP` to the dataset
{: .hands_on}

# Clustering
During clustering, cells that share similar accessibility profiles are organized into clusters. **SnapATAC2** utilizes graph-based community clustering with the *Leiden* algorithm ({% cite Traag2019%}).
This method takes the k-nearest neighbor (KNN) graph as input data and produces well-connected communities.


## Community clustering

> <hands-on-title> Clustering analysis </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute a neighborhood graph of observations, using 'pp.knn'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC UMAP` (output of **tl.umap** {% icon tool %})
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Cluster cells into subgroups, using 'tl.leiden'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata knn` (output of **pp.knn** {% icon tool %})
>        - *"Whether to use the Constant Potts Model (CPM) or modularity"*: `modularity`
>
>    > <comment-title> CPM or modularity </comment-title>
>    > - make sure you selected `modularity`
>    > - the clusters produced by `CPM` are not represented well in the UMAP projections
>    >
>    {: .comment}
> 2. Rename the generated file to `Anndata 5k PBMC leiden` or add the tag {% icon galaxy-tags %} `leiden` to the dataset
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4564 × 6062095
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden'
>    >  var: 'count', 'selected'
>    >  uns: 'doublet_rate', 'reference_sequences', 'scrublet_sim_doublet_score', 'spectral_eigenvalue'
>    >  obsm: 'X_spectral', 'X_umap', 'fragment_paired'
>    >  obsp: 'distances'
>    > ```
>    >
>    > Where are the **leiden** clusters stored in the AnnData?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The clusters are stored in `obs: 'leiden'`
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Plotting of clusters
Now that we have produced **UMAP** embeddings of our cells and have organized the cells into **leiden** clusters, we can now visualize this information with *pl.umap*.

> <hands-on-title> Plotting the clusters </hands-on-title>
>
> 1. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding, using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC leiden` (output of **tl.leiden** {% icon tool %})
>        - *"Color"*: `leiden`
>        - *"Height of the plot"*: `500`
> 4. {% icon galaxy-eye %} Inspect the `.png` output
>
>  ![umap_leiden_clustering]({% link topics/single-cell/images/scatac-standard-snapatac2/pl.umap.png %})
>
> > <question-title></question-title>
> >
> > 1. How many leiden clusters were discovered?
> > 2. What does the distance of clusters to each other tell us about their chromatin states?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 13 leiden clusters.
> > > 2. Clusters in close proximity (f.ex. clusters 0 and 5) share a similar chromatin accessibility profile (and very likely also a similar cell type), compared to a cluster further away (f.ex. cluster 9).
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

# Cell cluster annotation

After clustering the cells, they must be annotated. This categorizes the clusters into known cell types. **Manual Cell Annotation** requires known marker genes and varying expression profiles of the marker genes among clusters.

Luckily, the marker genes for {PBMCs} are known ({% cite Sun2019 %}).
Marker genes for other single cell datasets can also be found in databases such as [PanglaoDB](https://panglaodb.se/markers.html) ({% cite Franzn2019 %}). Using the known marker genes we can now annotate our clusters manually.

## Gene matrix
Since our data currently doesn't contain gene information, we have to create a cell-by-gene activity matrix using the function *pp.make_gene_matrix*.

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Generate cell by gene activity matrix, using 'pp.make_gene_matrix'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC leiden` (output of **tl.leiden** {% icon tool %})
>        - {% icon param-file %} *"GTF/GFF file containing the gene annotation"*: `gene_annotation.gtf.gz` (Input dataset)
> 2. Rename the generated file to `Anndata 5k PBMC gene_matrix` or add the tag {% icon galaxy-tags %} `gene_matrix` to the dataset
>    > <tip-title> Gene matrix </tip-title>
>    >
>    > - Please note that *pp.make_gene_matrix* removes all annotations except those stored in `obs`.
>    > - Therefore it might be necessary to remove propagating tags {% icon galaxy-tags %} (tags starting with `#`) from `Anndata 5k PBMC gene_matrix`.
>    >    - Tags can be removed by expanding the dataset with a tag and clicking {% icon galaxy-cross %} next to the tag.
>    {: .tip}
>
> 3. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 60606
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden'
>    > ```
>    >
>    > What does `n_vars` represent in `Anndata 5k PBMC gene_matrix` and what did it represent in `Anndata 5k PBMC leiden`?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - The variables now represent accessible genes. There are 60606 accessible genes in our samples. In `Anndata 5k PBMC leiden` and all earlier AnnData the variables represented the 6062095 fixed-sized genomic bins.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}


## Imputation with Scanpy and MAGIC
Similar to scRNA-seq data, the cell-by-gene-activity matrix is very sparse. Additionally, high gene variance between cells, due to technical confounders, could impact the downstream analysis.
In scRNA-seq, filtering and normalization are therefore required to produce a high-quality gene matrix.

Since the *cell-by-gene-activity* matrix resembles the *cell-by-gene-expression* matrix of scRNA-seq, we can use the tools of the [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({%cite Wolf2018%}) tool suite to continue with our data.

> <details-title>Imputation with MAGIC</details-title>
>
> - The count matrices of single-cell data are sparse and noisy.
> - Confounding issues, such as "dropout" effects, where some mRNA or DNA-segments are not detected although they are present in the cell, also result in some cells missing important cell-type defining features.
>    - These problems can obscure the data, as only the strongest gene-gene relationships are still detectable.
> - The *Markov Affinity-based Graph Imputation of Cells* (MAGIC) algorithm ({%cite vanDijk2018%}) tries to solve these issues by filling in missing data from some cells with transcript information from similar cells.
>    - The algorithm calculates the likely gene expression of a single cell, based on similar cells and fills in the missing data to produce the expected expression.
>      - *MAGIC* achieves this by building a graph from the data and using data diffusion to smooth out the noise.
>
> ![Imputation with the MAGIC algorithm]({% link topics/single-cell/images/scatac-standard-snapatac2/magic_method.png %} "MAGIC restores noisy and sparse single-cell data using diffusion geometry ({%cite vanDijk2018%})")
>
{: .details}

> <hands-on-title> Filter and normalize </hands-on-title>
>
> 1. {% tool [Filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC gene_matrix` (output of **pp.make_gene_matrix** {% icon tool %})
>    - *"Method used for filtering"*: `Filter genes based on number of cells or counts, using 'pp.filter_genes'`
>        - {% icon param-select %} *"Filter"*: `Minimum number of cells expressed`
>            - *"Minimum number of cells expressed required for a gene to pass filtering"*: `5`
>
> 2. {% tool [Normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata filter_genes` (output of **Filter** {% icon tool %})
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>    - {% icon param-toggle %} *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>
> 3. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata normalize` (output of **Normalize** {% icon tool %})
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
>
> 4. {% tool [Normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata log1p` (output of **log1p** {% icon tool %})
>    - *"Method used for normalization"*: `Denoising using Markov Affinity-based Graph Imputation of Cells (MAGIC) API 'external.pp.magic'`
>        - *"Denoised genes to return"*: `All genes`
>        - *"Which solver to use"*: `"approximate", is faster that performs imputation in the PCA space and then projects back to the gene space`
>
>    > <comment-title> </comment-title>
>     >   - Choosing the setting `Which solver to use: 'exact'` will result in a output file with better resolution.
>     >   - This is not necessary for our purposes, since the compute time also increases with this setting.
>     >
>     {: .comment}
>
> 5. Rename the generated file to `Anndata 5k PBMC magic` or add the tag {% icon galaxy-tags %} `magic` to the dataset
>
> 6. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output
>
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 4430 × 55106
>    >  obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden', 'n_genes', 'n_counts'
>    >  var: 'n_cells', 'n_counts'
>    >  uns: 'log1p'
>    > ```
>    >
>    > 1. How did `n_vars` change, compared to `Anndata 5k PBMC gene_matrix`?
>    > 2. Which data was 'lost', compared to `Anndata 5k PBMC leiden`, and must be added to the file in order to produce **UMAP** plots?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The number of genes was reduced from 60606 to 55106 by the filtering. Additional annotations were added, such as: `obs: 'n_genes', 'n_counts'`, `var: 'n_cells', 'n_counts'` and `uns: 'log1p'`.
>    > > 2. The UMAP embeddings `obsm: 'X_umap'` are missing and should be added to the Anndata in the next step. Without `X_umap` it won't be possible to visualize the plots.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
{: .hands_on}

## Copy-over embeddings

> <hands-on-title> Copy UMAP embedding </hands-on-title>
>
> 1. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in hdf5 AnnData format"*: `Anndata 5k PBMC magic` (output of **external.pp.magic** {% icon tool %})
>    - *"Copy embeddings (such as UMAP, tSNE)"*: `Yes`
>       - *"Keys from embeddings to copy"*: `X_umap`
>       - {% icon param-file %} *"IAnnData objects with embeddings to copy"*: `Anndata 5k PBMC leiden`
>
>    > <comment-title> Annotations to copy </comment-title>
>    >
>    > - This tutorial only focuses on producing an **UMAP** plot with marker-genes.
>    > - If further analysis, with tools requiring more annotations, is intended, these can be added in a similar way as shown above.
>    >     - f.ex. *Peak and Motif Analysis* with {% tool [Snapatac2 peaks and motif](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_peaks_and_motif/snapatac2_peaks_and_motif/2.6.4+galaxy1) %} requires annotations from `uns`.
>    > - It is also possible to leave the input *"Keys from embeddings to copy"* empty, to copy all annotations of a given category such as `obsm`.
>    {: .comment}
>
> 5. Rename the generated file to `Anndata 5k PBMC magic UMAP` or add the tag {% icon galaxy-tags %} `UMAP` to the dataset
>
> 6. {% icon galaxy-eye %} Inspect the general information of the `.h5ad` output, to check if `obsm` contains `X_umap`
>
>   ```
>   AnnData object with n_obs × n_vars = 4430 × 55106
>    obs: obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'leiden', 'n_genes', 'n_counts'
>    var: 'n_cells', 'n_counts'
>    uns: 'log1p'
>    obsm: 'X_umap'
>   ```
>
{: .hands_on}

## Visualize gene activity of marker genes
The gene activity of selected marker genes can now be visualized with Scanpy.

> <hands-on-title> Plot marker genes </hands-on-title>
>
> 1. {% tool [Plot with Scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.9.6+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `output_h5ad` (output of **AnnData Operations** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `leiden, CD3D, CD8A, CD4, MS4A1, NKG7, CD14, FCER1A`
>        - {% icon param-toggle %} *"Show edges?"*: `No`
>        - In *"Plot attributes"*
>           - *"Number of panels per row"*: `2`
>
>  ![umap_leiden_marker_gene_clustering]({% link topics/single-cell/images/scatac-standard-snapatac2/umap_leiden_marker-genes.png %})
>
> > <question-title></question-title>
> >
> > 1. Are the marker genes selectively expressed in the clusters?
> > 2. Which marker genes have overlapping expression profiles? And what could that imply?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. Some marker genes, such as `MS4A1` or `CD8A`, are only expressed in a few clusters (clusters 6+11 and clusters 1+8, respectively).
> > > 2. The marker gene `CD3D` is expressed in multiple clusters (1, 2, 4, 7 and 8). Overlapping expression profiles imply similar cell types since similar cell types have similar marker genes upregulated. In this case, `CD3D` expression classifies the cells in these clusters as T-cells.
> > >
> > {: .solution}
> >
> {: .question}
{: .hands_on}

## Manual cluster annotation
Comparison of marker gene expression in our clusters with a table of canonical marker genes ({% cite Sun2019 %}), enables us to annotate the clusters manually.

Cell type | Marker genes
--- | ---
CD8+ T cells | CD3D+, CD8A+, CD4-
CD4+ T cells | CD3D+, CD8A-, CD4+
B cells | CD3D-, MS4A1+
Natural killer (NK) cells | CD3D-, NKG7+
Monocytes | CD3D-, CD14+
Dendritic cells | CD3D-, FCER1A+

These canonical marker genes can match the clusters to known cell types:

Cluster | Cell type
--- | ---
0 | Monocytes
1 | CD8+ T cells
2 | CD4+ T cells
3 | Monocytes
4 | CD4+ T cells
5 | Monocytes
6 | B cells
7 | CD4+ T cells
8 | CD8+ T cells
9 | NK cells
10 | Monocytes
11 | B cells
12 | Dendritic cells

> <comment-title></comment-title>
> Note that some clusters contain subtypes (f.ex. the annotated B cell clusters contain both naive and memory B cells). The cell-type annotation can be refined by choosing more specific marker genes.
{: .comment}

To manually annotate the *Leiden* clusters, we will need to perform multiple steps:

1. **Inspect** the key-indexed observations of `Anndata 5k PBMC gene_matrix magic UMAP`
2. **Cut** the *Leiden* annotations out of the table
3. **Upload** a *replace file* containing the new cell type annotations for the *Leiden* clusters
4. **Replace** the values of the cluster annotation with cell type annotation
5. **Add** the cell type annotation to the AnnData
6. **Plot** the annotated cell types

> <hands-on-title> Manual annotation </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Anndata 5k PBMC gene_matrix magic UMAP`
>    - *"What to inspect?"*: `Key-indexed observations annotation`
> 2. Rename the generated file to `5k PBMC observations`
> 3. {% icon galaxy-eye %} Inspect the generated file
>
>    > <question-title></question-title>
>    > In which column is the **Leiden** annotation located?
>    > > <solution-title></solution-title>
>    > > The **Leiden** annotation is in column 8.
>    > >
>    > > Column 1 | Column 2 | Column 3 | Column 4 | Column 5 | Column 6 | Column 7 | Column 8 | Column 9 | Column 10
>    > > --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
>    > > "" | n_fragment | frac_dup | frac_mito | tsse | doublet_probability | doublet_score | leiden | n_genes | n_counts
>    > > AAACGAAAGACGTCAG-1 | 22070 | 0.5219425551271499 | 0.0 | 30.43315066436454 | 0.004634433324822066 | 0.009276437847866418 | 8 | 52303 | 16521.599844068267
>    > > AAACGAAAGATTGACA-1 | 10500 | 0.5345125681606597 | 0.0 | 29.10551296093465 | 0.004668403569267374 | 0.001088139281828074 | 1 | 54501 | 15020.42495602328
>    > > AAACGAAAGGGTCCCT-1 | 19201 | 0.5101785714285714 | 0.0 | 19.90011850347046 | 0.004634433324822066 | 0.009276437847866418 | 5 | 54212 | 16294.751533305309
>    > > AAACGAACAATTGTGC-1 | 13242 | 0.487399837417257 | 0.0 | 29.060913705583758 | 0.004660125753854076 | 0.0022172949002217295 | 7 | 53530 | 15456.629863655084
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>
> 4. {% tool [Cut columns](toolshed.g2.bx.psu.edu/repos/devteam/cut_columns/Cut1/1.0.2) %} with the following parameters:
>    - {% icon param-select %} *"


topic: single-cell
tutorial: scrna-scanpy-pbmc3k
title: Clustering 3K PBMCs with Scanpy
difficulty: intermediate
time_estimation: 8H
tools_json: []
requirements_json: ["type: internal", "scrna-preprocessing", "scrna-preprocessing-tenx"]
tags_json: ["10x"]
content_hash: 106a91e8420a1d6c57a5e274520ec137
last_modified: 2026-02-26T21:12:17.992217
zenodo_link: https://zenodo.org/record/3581213
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.html

content:
Single-cell RNA-seq analysis is a rapidly evolving field at the forefront of transcriptomic research, used in high-throughput developmental studies and rare transcript studies to examine cell heterogeneity within a populations of cells. The cellular resolution and genome wide scope make it possible to draw new conclusions that are not otherwise possible with bulk RNA-seq.

In this tutorial, we will investigate clustering of single-cell data from 10x Genomics, including preprocessing, clustering and the identification of cell types via known marker genes, using [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({% cite wolf2018scanpy %}). It will be illustrated using a dataset of Peripheral Blood Mononuclear Cells (PBMC), containing 2,700 single cells.


> <comment-title></comment-title>
>
> This tutorial is significantly based on ["Clustering 3K PBMCs" tutorial from Scanpy](https://scanpy-tutorials.readthedocs.io/en/latest/pbmc3k.html#Clustering-3K-PBMCs), ["Seurat - Guided Clustering Tutorial"](https://satijalab.org/seurat/v3.1/pbmc3k_tutorial.html) and ["Orchestrating Single-Cell Analysis with Bioconductor"](https://bioconductor.org/books/release/OSCA/) {% cite amezquita2019orchestrating %}.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data

For this tutorial, we analyze a dataset of Peripheral Blood Mononuclear Cells (PBMC) extracted from a healthy donor, freely available from 10X Genomics. The dataset contains 2,700 single cells sequenced using Illumina NextSeq 500. The raw sequences have been processed by the [**cellranger**](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) pipeline from 10X to extract a unique molecular identifier (UMI) count matrix, in a similar way to that as explained in ["Pre-processing of 10X Single-Cell RNA Datasets" tutorial]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}).

In this matrix, the values represent the number for each feature (i.e. gene; row) that are detected in each cell (column). Such matrices can be quite large, where here there are 2,700 columns with 32,738 lines, with mostly zero values, i.e. an extremely sparse matrix. To optimize the storage of such a table and the information about the genes and cells, **cellranger** creates 3 files:

- `genes.tsv`: a tabular file with information about the 32,738 genes in 2 columns (Ensembl gene id and the gene symbol)
- `barcodes.tsv`: a tabular file with the barcode for each of the 2,700 cells
- `matrix.mtx`: a condensed version of the count matrix

    The count matrix is therefore represented by its non-zero values. Each non-zero value is indicated by its line number (1st column), its column number (2nd column) and its value (3rd column). The first row gives indication about the number of lines, column and non-zero values. More information on the Matrix Market Exchange (mtx) format can be found [in this documentation](https://math.nist.gov/MatrixMarket/formats.html)

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the `genes.tsv`, `barcodes.tsv` and `matrix.mtx` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/genes.tsv
>    {{ page.zenodo_link }}/files/barcodes.tsv
>    {{ page.zenodo_link }}/files/matrix.mtx
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Inspect the `matrix` file
{: .hands_on}

> <question-title></question-title>
>
> ```
> 32738	2700	2286884
> 32709	1	4
> 32707	1	1
> 32706	1	10
> 32704	1	1
> ```
>
> 1. How many non-zero values are in the matrix?
> 2. How many counts are found for the 32,706th gene in the 1st cell?
>
> > <solution-title></solution-title>
> >
> > 1. There are 2,286,884 (2.6%) non-zero values for the 88,392,600 possible counts of the 32,738 genes (rows) and 2,700 cells (columns).
> > 2. 10 counts are found for the 32,706th row and 1st column.
> >
> {: .solution}
>
{: .question}

The representation of the matrix with 3 files is convenient to share the data but not to process them. Single-cell analysis packages have attempted to solve the problem of storage and analysis by inventing their own standard, which has led to the proliferation of many different "standards" in the scRNA-seq package ecosystem.

## AnnData

The most common format, called [`AnnData`](https://anndata.readthedocs.io/en/stable/), stores the matrix as well as gene and cell annotations in a concise, compressed and extremely readable manner:

![Anndata format]({% link topics/single-cell/images/scrna-pre-processing/tenx_anndata.svg %} "<code>AnnData</code> format stores a count matrix <code>X</code> together with annotations of observations (i.e. cells) <code>obs</code>, variables (i.e. genes) <code>var</code> and unstructured annotations <code>uns</code>.")

This format is used by [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({% cite wolf2018scanpy %}), the tool suite for analyzing single-cell gene expression data that we will use in this tutorial. So we need first to import the matrix and annotations of genes and cells into an `AnnData` object.

> <hands-on-title>Transform matrix and all into AnnData object</hands-on-title>
>
> 1. {% tool [Import Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.10.9+galaxy0) %} with the following parameters:
>    - *"Format for the annotated data matrix"*: `Matrix Market (mtx), from Cell ranger or not`
>        - {% icon param-file %} *"Matrix"*: `matrix.mtx`
>        - *"Use 10x Genomics formatted mtx"*: `Output from Cell Ranger v2 or earlier versions`
>            - {% icon param-file %} *"2-column genes file with gene id and gene name"*: `genes.tsv`
>            - {% icon param-file %} *"Barcodes (whitelist) file with one barcode per line"*: `barcodes.tsv`
>            - *"Variables index"*: `gene_symbols`
>            - *"Make the variable index unique by appending '-1', '-2'?"*: `Yes`
>
> 2. Rename the generated file to `Input 3k PBMC`
>
> 3. Check that the format is `h5ad`
{: .hands_on}

Because the `AnnData` format is an extension of the HDF5 format, i.e. a binary format, an `AnnData` object can not be inspected directly in Galaxy by clicking on the {% icon galaxy-eye %} (**View data**) icon. Instead we need to use a dedicated tool from the **AnnData** suite.

> <hands-on-title>Inspect an AnnData object</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `General information about the object`
>
> 2. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 2700 × 32738
>    >     var: 'gene_ids'
>    > ```
>    >
>    > 1. How many observations are there? What do they represent?
>    > 2. How many variables are there? What do they represent?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 2,700 observations, representing the cells.
>    > > 2. There are 32,738 variables, representing the genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    > <comment-title>Faster Method for General Information</comment-title>
>    >
>    > * The **ScanPy** toolset in Galaxy produces *AnnData* formats as
>    >   output, where general information is provided for each dataset:
>    >    * Click on the name of the dataset in the history to expand it.
>    >    * General Anndata information would be given in the expanded box:
>    >
>    >      e.g.
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    2700 x 32738
>    >      ```
>    > * For more specific queries, {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} is required.
>    {: .comment}
>
>    > <warning-title>Large compute resource consumption! Please run the next step responsibly.</warning-title>
>    > Extracting the complete matrix requires large compute resources and long time to finish. Use this step on matrices with less than 10k cells. To inspect
>    > a subset of matrix contents on huge matrices, please change *“What to inspect?”*: `Random chunk of defined size`
>    {: .warning}
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > 	MIR1302-10	FAM138A	OR4F5	RP11-34P13.7	RP11-34P13.8 ...
>    > AAACATACAACCAC-1	0.0	0.0	0.0	0.0
>    > AAACATTGAGCTAC-1	0.0	0.0	0.0	0.0
>    > AAACATTGATCAGC-1	0.0	0.0	0.0	0.0
>    > AAACCGTGCTTCCG-1	0.0	0.0	0.0	0.0
>    > AAACCGTGTATGCG-1	0.0	0.0	0.0	0.0
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a table with 2,700 lines (observations or cells) and 32,738 columns (variables or genes): the count matrix for each of the 32,738 genes and 2,700 cells. The 1st row contains the gene symbol as annotation of the columns and the 1st column the barcodes of the cells as annotation of the rows.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
> 6. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > ""
>    > AAACATACAACCAC-1
>    > AAACATTGAGCTAC-1
>    > AAACATTGATCAGC-1
>    > AAACCGTGCTTCCG-1
>    > AAACCGTGTATGCG-1
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a tabular with annotations of the observations. i.e. the cells. Here there are only the barcodes as annotation, so only one column, being also the index for the count matrix.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 7. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 8. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > 	gene_ids
>    > MIR1302-10	ENSG00000243485
>    > FAM138A	ENSG00000237613
>    > OR4F5	ENSG00000186092
>    > RP11-34P13.7	ENSG00000238009
>    > RP11-34P13.8	ENSG00000239945
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a tabular with annotations of the variables, i.e. the genes. Here the files has 2 columns: the gene symbols (also the index in the count matrix), `gene_ids` or the Ensemble gene ids.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

# Preprocessing

The first step when we get our counts is to prepare them before we perform the clustering. This includes ({% cite amezquita2019orchestrating %}):

1. **Selection and filtration of cells and genes based on quality metrics**

    Low-quality cells would interfere with downstream analyses. These cells may have been damaged during processing and may not have been fully captured by the sequencing protocol.

2. **Data normalization and scaling**

    It will eliminate cell-specific biases (e.g., in capture efficiency), allowing us to perform explicit comparisons across cells afterwards. Some transformations, typically log, are also applied to adjust for the mean-variance relationship.

3. **Selection of features**

    This selection picks a subset of interesting features for downstream analysis, by modelling the variance across cells for each gene and retaining genes that are highly variable. This step is done to reduce computational overhead and noise from uninteresting genes.

## Quality control

Genes that appear in less than a few cells can be considered noise and thus removed.

> <hands-on-title>Remove genes found in less than 3 cells</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Input 3k PBMC`
>    - *"Method used for filtering"*: `Filter genes based on number of cells or counts, using 'pp.filter_genes'`
>        - *"Filter"*: `Minimum number of cells expressed`
>            - *"Minimum number of cells expressed required for a gene to pass filtering"*: `3`
>
> 2. Rename the generated file `3k PBMC`
> 3. Inspect the dataset.
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2700 × 13714
>    > [var]
>    > -    gene_ids
>    > -    n_cells
>    > ```
>    >
>    > How many genes have been removed because they are expressed in less than 3 expressed cells?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are now 13,714 genes. So 19,024 (32,738 - 13,714) genes have been removed.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

Low-quality cells may be due to a variety of sources such as cell damage during dissociation or failure in library preparation (e.g., inefficient reverse transcription or PCR amplification, {% cite ilicic2016classification %}). Usually these low-quality cells have low total counts, few expressed genes and high mitochondrial or spike-in proportions. Low-quality cells are problematic for downstream analyses and may contribute to misleading results.

> <details-title>Impact of low-quality cells on the downstream analyses</details-title>
>
> #### Formation of their own distinct cluster(s)
>
> Increased mitochondrial proportions or enrichment for nuclear RNAs after cell damage are the most obvious causes for this phenomenon. Low-quality cells in different cell types can, in the worst case, cluster together (because of similarities in the damage-induced expression profiles) leading to artificial intermediate states or trajectories between subpopulations that should be otherwise distinct. This phenomenon makes the results difficult to interpret.
>
> #### Distortion of population heterogeneity during variance estimation or principal components analysis
>
> With low-quality cells, the main differences captured by the first few principal components will be based on quality rather than biology, reducing then the effectiveness of dimensionality reduction. Similarly, differences between low- and high-quality cells will create genes with the largest variances. For example, in low-quality cells with very low counts, the scaling normalization increases the variance of genes that happen to have a non-zero count in those cells.
>
> #### Misidentification of upregulated genes
>
> Due to aggressive scaling to normalize for small cell sizes, low-quality cells shows genes that appear to be strongly "upregulated". For example, contaminating transcripts may be present in all cells with a low but constant levels. With the increased scaling normalization in low-quality cells, the small counts for these transcripts may become large normalized expression values, i.e. an apparent upregulation compared to other cells.
>
{: .details}

To mitigate these problems, we need to remove these low-quality cells at the start of the analysis. Several common QC metrics can be used to identify these cells based on their expression profile:

- **Cell size**, i.e. the total sum of counts across all genes for each cells

    When cells are very degraded or absent from the library preparation, the number of reads sequenced from that library will be very low. Cells with small sizes are then considered to be of low quality: the RNA has been lost during the library preparation (cell lysis, inefficient cDNA capture and amplification, etc.)

- **Number of expressed genes**, i.e. the number of genes with non-zero counts for each cells

    A low number of expressed genes may be a result of poor-quality cells (e.g. dying, degraded, damaged, etc.), followed by high PCR amplification of the remaining RNA. The diverse transcript population has not been successfully captured so the cell is considered of low quality. Cell doublets or multiplets may also exhibit an aberrantly high gene count.

- **Proportion of reads mapped to genes in the mitochondrial genome**

    High concentrations of mitochondrial genes is often a result of damaged cells ({% cite islam2014quantitative %}, {% cite ilicic2016classification %}) where the endogenous RNA escapes or degrades. As mitochondria has its own cell membranes, it is often the last DNA/RNA in damaged cells to degrade and hence occurs in high quantities during sequencing.

We will make the key assumption that these metrics are independent of the biological state of each cell. Technical factors rather than biological processes are presumed to drive poor values (e.g., low cell sizes, high mitochondrial proportions). The subsequent removal of impacted cells do not misrepresent the biology in downstream analyses.

To identify low-quality cells, the simplest approach is to apply thresholds on the QC metrics. This strategy, while simple, is required to determine appropriate thresholds that will change given the experimental protocol and the biological system.

### Computation of QC metrics

To estimate the thresholds, we first need to look at the distribution of QC metrics for our dataset.

The first 2 QC metrics (cell size and number of expressed genes) can be easily estimated from the count table. For the third metric (proportion of reads mapped to genes in the mitochondrial genome), we need the information about which genes are mitochondrial or not.

> <hands-on-title>Extract gene annotation</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 2. Rename the generated file `Gene annotation`
{: .hands_on}

In the gene annotation, we currently have the gene symbol the Ensembl gene ids and the number of cells in which the genes are expressed, but have no information on whether a gene is mitochondrial. This can be extracted from the gene symbol as all mitochondrial genes have a name starting with `MT-` and added to the `var` of the `AnnData` object using **Manipulate AnnData** tool.

A new annotation for `var` should be new column named `mito` on the 1st line and then `True` for mitochondrial gene or `False` for non-mitochondrial genes.

> <hands-on-title>Add mitochondrial gene annotation</hands-on-title>
>
> 1. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC`
>    - *"Function to manipulate the object"*: `Flag genes start with a pattern`
>    -  Click on *"Insert Flag genes that start with these names"*
>      - *"Text that you expect the genes to be flagged to start with?"*: `MT-`
>      - *"Name of the column in var.names where this boolean flag is stored"*: `mito`
>
> 2. Rename the generated file `3k PBMC with mito annotation`
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 4. Inspect the generated file and check if the mitochondrial annotation has been added
>
>    > <question-title></question-title>
>    >
>    > 1. How can I annotate genes like ribosomal protein genes or non-protein coding genes that might interfere with my downstream analysis?
>    > 2. What care should be taken while annotating such genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. To annotate genes such as ribosomal protein genes or non-protein coding genes, follow these steps:
>    > >
>    > >    - {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} tool as above with
>    > >        - *"Function to manipulate the object"*: `Flag genes start with a pattern`
>    > >        -  Click on *"Insert Flag genes that start with these names"*
>    > >          - *"Text that you expect the genes to be flagged to start with?"*: `RP` for ribosomal proteins or
>    > >          - *"Name of the column in var.names where this boolean flag is stored"*: `rpgene`
>    > >        -  Click on *"Insert Flag genes that start with these names"*
>    > >          - *"Text that you expect the genes to be flagged to start with?"*: `LINC` for long non-protein-coding RNAs
>    > >          - *"Name of the column in var.names where this boolean flag is stored"*: `lincrna`
>    > >
>    > >    - {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    > >        - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation`
>    > >        - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>    > >    - Inspect your AnnData to confirm the new gene annotations are correctly added.
>    > > 2. Here we are annotating genes starting with a pattern. Long Non-Coding RNAs often start with "LINC" followed by a number, while Ribosomal Protein Genes typically start with "RP". Be aware that some non-ribosomal genes also start with "RP" and are associated with conditions like Retinitis Pigmentosa. If these genes are relevant, annotate 'RP' genes carefully. Maybe use more precise naming like `RPL` and `RPS`. In general, make sure that your name/pattern does not annotate any genes that you did not want annotate.
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

We can now compute QC metrics on the `AnnData` object.

> <hands-on-title>Compute QC metrics</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation`
>    - *"Method used for inspecting"*: `Calculate quality control metrics, using 'pp.calculate_qc_metrics'`
>      - *"Name of kind of values in X"*: `counts`
>      - *"The kind of thing the variables are"*: `genes`
>      - *"Keys for boolean columns of `.var` which identify variables you could want to control for"*: `mito`
>
> 2. Rename the generated file `3k PBMC with mito annotation and qc metrics`
>
> 3. Inspect the dataset
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2700 × 13714
>    > [obs]
>    > -    n_genes_by_counts
>    > -    log1p_n_genes_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > -    pct_counts_in_top_50_genes
>    > -    pct_counts_in_top_100_genes
>    > -    pct_counts_in_top_200_genes
>    > -    pct_counts_in_top_500_genes
>    > -    total_counts_mito
>    > -    log1p_total_counts_mito
>    > -    pct_counts_mito
>    > [var]
>    > -    gene_ids
>    > -    n_cells
>    > -    mito
>    > -    n_cells_by_counts
>    > -    mean_counts
>    > -    log1p_mean_counts
>    > -    pct_dropout_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > ```
>    >
>    > Which QC metrics have been computed?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The tool computed several QC metrics at both cell and gene levels. These metrics are added to the annotation tables.
>    > >
>    > > At gene levels (stored in `var`):
>    > > - `total_counts`: sum of counts for a gene
>    > > - `mean_counts`: mean expression for a gene over all cells
>    > > - `n_cells_by_counts`: number of cells with non-zero counts for a gene
>    > > - `pct_dropout_by_counts`: percentage of cells this gene does not appear in
>    > >
>    > > At cell levels (stored in `obs`):
>    > > - `total_counts`: total number of counts for a cell
>    > > - `total_counts_mito`: total number of counts for the mitochondrial genes in a cell
>    > > - `n_genes_by_counts`: number of genes with non-zero counts
>    > > - `pct_counts_mito`: proportion of total counts for a cell which are from mitochondrial genes
>    > > - `pct_counts_in_top_50_genes`: cumulative percentage of counts for 50 most expressed genes in a cell
>    > > - `pct_counts_in_top_100_genes`: cumulative percentage of counts for 100 most expressed genes in a cell
>    > > - `pct_counts_in_top_200_genes`: cumulative percentage of counts for 200 most expressed genes in a cell
>    > > - `pct_counts_in_top_500_genes`: cumulative percentage of counts for 500 most expressed genes in a cell
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

We would like to visualize 3 of the more informative QC metrics:

- the cell size, i.e. `total_counts`
- the number of expressed genes, i.e. `n_genes_by_counts`
- the proportion of reads mapped to mitochondrial genes, i.e. `pct_counts_mito`

> <hands-on-title>Visualize QC metrics</hands-on-title>
>
> 1. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>      - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>        - *"Keys for accessing variables"*: `n_genes_by_counts, total_counts, pct_counts_mito`
>      - In *"Violin plot attributes"*:
>        - *"Display keys in multiple panels"*: `Yes`
>
> 2. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ![QC violin plot]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/qc_violin_plot.png %})
>    >
>    > How do the distributions of the 3 QC metrics look?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > For the cell size, i.e. `total_counts`, most of the values are between 1,000 reads and 4,000 reads, with some extremely high values skewing the distribution.
>    > >
>    > > The numbers of expressed genes, i.e. `n_genes_by_counts`, are mostly between 500 genes and 1,200 genes, with also some extremely high values skewing the distribution.
>    > >
>    > > The distribution of the proportions of reads mapped to mitochondrial genes, i.e. `pct_counts_mito`, is even more narrow with some cells having no counts from mitochondrial genes but also having some really extreme values (above 5%).
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>      - *"Plotting tool that computed coordinates"*: `Using coordinates`
>        - *"x coordinate"*: `total_counts`
>        - *"y coordinate"*: `n_genes_by_counts`
>        - *"Use the layers attribute?"*: `No`
>
> 4. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ![QC total_counts n_genes_by_counts]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/qc_scatter_total_counts_n_genes_by_counts.png %})
>    >
>    > Is there any relationship between the cell size and the number of expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > On the plot, we can see a strong correlation between the total number of counts for a cell and the number of genes with positive counts.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `n_genes_by_counts`
>            - *"y coordinate"*: `pct_counts_mito`
>            - *"Use the layers attribute?"*: `No`
>
> 6. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ![QC total_counts pct_counts_mito]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/qc_scatter_n_genes_by_counts_pct_counts_mito.png %})
>    >
>    > 1. Is there any relationship between the number of expressed genes and the proportion of reads mapped to mitochondrial genes?
>    > 2. What could be a good threshold to filter for cells with high concentrations of mitochondrial genes?
>    > 3. What could be good thresholds to filter for cells based on the number of expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Cells with a high proportion of mitochondrial genes are not also cells that have many expressed genes. There is no visible correlation.
>    > > 2. The cells with a percentage of mitochondrial counts above 5% have also few genes. So 5% may be a good threshold.
>    > > 3. As discussed before, a low number of expressed genes may be a sign of poor-quality cells. Any cells with less than 200 genes is definitely out of the main distribution so this could be a good threshold to use. High gene count may also be a sign for cell multiplets. We will choose here a threshold of 2,500 genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

### Filtering of low-quality cells

As explained before, we would like now to filter for low-quality cells based on the 3 previous metrics (cell size, number of expressed genes, and proportion of reads mapped to mitochondrial genes). As the cell size is highly correlated with the number expressed genes, we can focus only on the number expressed genes and the proportion of reads mapped to mitochondrial genes.

Based on the previous plot, we would like to remove cells that have:

- a number of expressed genes below 200 or above 2,500
- a percentage of reads mapped to mitochondrial genes above 5%

> <hands-on-title>Remove low-quality cells</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with mito annotation and qc metrics`
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - *"Filter"*: `Minimum number of genes expressed`
>            - *"Minimum number of genes expressed required for a cell to pass filtering"*: `200`
>
> 2. Inspect the dataset
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2700 × 13714
>    > ```
>    >
>    > How many cells have been removed because they have less than 200 expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are still 2,700 cells. So no cells have been removed because they have less than 200 expressed genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 3. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy filter** {% icon tool %}
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - *"Filter"*: `Maximum number of genes expressed`
>            - *"Maximum number of genes expressed required for a cell to pass filtering"*: `2500`
>
> 4. Inspect the dataset
>    >
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2695 × 13714
>    > ```
>    >
>    > How many cells have been removed because they have more than 2,500 expressed genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are now 2,695 cells. So 5 cells have been removed because they have more than 2,500 expressed genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy filter** {% icon tool %}
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `pct_counts_mito`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `5.0`
>
> 6. Rename the generated file `3k PBMC after QC filtering`
>
> 7. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC after QC filtering`
>    - *"What to inspect?"*: `General information about the object`
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 2638 × 13714
>    > ```
>    >
>    > How many cells have been removed because they have more than 5% of reads mapped to mitochondrial genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > There are now 2,638 cells. So 57 cells have been removed because they have more than 5% of reads mapped to mitochondrial genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

## Normalization and scaling

In scRNA-seq, we can observe systematic differences in sequencing coverage between cells ({% cite stegle2015computational %}), because of technical differences in cDNA capture or PCR amplification efficiency across cells, attributable to the difficulty of achieving consistent library preparation with minimal starting material. After removing low-quality cells, normalization of the counts removes these differences to avoid that they interfere with comparisons of the expression profiles between cells. Any observed heterogeneity or differential expression within the cell population after normalization are then driven by biology and not technical biases.

Scaling normalization is the simplest and most commonly-used class of normalization strategies. All counts for each cell are divided by a cell-specific scaling factor, often called a "size factor". The assumption behind this process is that any technical biases tend to affect genes in a similar manner. The relative bias for each cell is represented by the size factor for that cell. Dividing the cell counts by its size factor should then remove the bias.

The simplest strategy for scaling normalization is the cell size normalization, i.e. similar total sum of counts across all genes for each cell. The "cell size factor" for each cell is computed directly for its cell size and transformed such that the mean size factor across all cells is equal to 1. The normalized expression values are then kept on the same scale as the original counts.

Here we would to normalize our count table such that each cell have 10,000 reads.

> <hands-on-title>Normalize for cell size</hands-on-title>
>
> 1. {% tool [Scanpy normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC after QC filtering`
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>      - *"Target sum"*: `10000.0`
>      - *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>      - *"Name of the field in 'adata.obs' where the normalization factor is stored"*: `norm`
>
{: .hands_on}

The normalized counts should be log-transformed afterwards to adjust for the mean-variance relationship.

With log-transformation, the differences in the log-values represent log-fold changes in expression. Log-transformation focuses on promoting contributions from genes with strong relative differences (e.g. a gene that is expressed at an average count of 50 in cell type A and 10 in cell type B rather than a gene that is expressed at an average count of 1100 in A and 1000 in B).

> <hands-on-title>Log-transform the counts</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy normalize** {% icon tool %}
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
{: .hands_on}

We will freeze the current state of the AnnData object, i.e. the logarithmized raw gene expression, in the a `raw` attribute. This information will be used later in differential testing and visualizations of gene expression.

> <hands-on-title>Freeze the state of the AnnData object</hands-on-title>
>
> 1. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy Inspect and manipulate** {% icon tool %}
>    - *"Function to manipulate the object"*: `Freeze the current state into the 'raw' attribute`
>
> 2. Rename the generated output `3k PBMC after QC filtering and normalization`
{: .hands_on}

## Selection of features

Data from scRNA-Seq is often used in exploratory analyses to characterize heterogeneity across cells. With clustering and dimensionality reduction, cells are compared based on their gene expression profiles. The choice of genes to use may have a major impact on the behaviour of the clustering and the dimensionality reduction. We need then to remove genes with random noise while keeping genes containing useful information about the biology of the system. This reduces the data size but still highlights any interesting biological signal without the noise that obscures that structure.

Selecting the most variable genes based on their expression across the cells (i.e. genes highly expressed in some cells, and lowly expressed in others) is the simplest approach for feature selection. With this approach, we assume that increased variation in some genes compared to other genes are genuine biological differences and not technical noise or a baseline level of "uninteresting" biological variation.

To quantify the per-gene variation, the simplest approach consists of computing the variance of the log-normalized expression values for each gene across all cells in the population. With this approach, the feature selection is based on the same log-values as the ones used in clustering, ensuring then that the quantitative definition of heterogeneity is consistent throughout the entire analysis.

Calculation of the per-gene variance is simple but feature selection requires modelling of the mean-variance relationship, as done in the Seurat procedure ({% cite stuart2019comprehensive %}) we will use.

Once the per-gene variation has been quantified, we need to select the subset of highly variable genes that we will use in downstream analyses. A large subset reduces the risk of discarding any interesting biological signal, but increases the noise from irrelevant genes on the signal. The optimal trade-off is difficult to determine but there are several common strategies routinely used. You can read the [Chapter 8 from "Orchestrating Single-Cell Analysis with Bioconductor"](https://osca.bioconductor.org/feature-selection.html#hvg-selection) for a nice presentation of the different strategies. Here, we will define the set of highly variable genes as those which, after normalization, have a normalized dispersion amount higher than 0.5.

> <hands-on-title>Identify the highly variable genes</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC after QC filtering and normalization`
>    - *"Method used for filtering"*: `Annotate (and filter) highly variable genes, using 'pp.highly_variable_genes'`
>      - *"Flavor for computing normalized dispersion"*: `seurat`
>        - *"Minimal mean cutoff"*: `0.0125`
>        - *"Maximal mean cutoff"*: `3`
>        - *"Minimal normalized dispersion cutoff"*: `0.5`
>      - *"Inplace subset to highly-variable genes?"*: `No`
>
> 2. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of the last **Scanpy filter** {% icon tool %}
>    - *"Method used for plotting"*: `Preprocessing: Plot dispersions versus means for genes, using 'pl.highly_variable_genes'`
>
{: .hands_on}

![Highly variable genes]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/feature_selection_highly_variable_genes.png %})

Both highly variable genes and other genes are still in the `AnnData` object. We can now actually keep only the highly variable genes.

> <hands-on-title>Keep the highly variable genes</hands-on-title>
>
> 1. Inspect the output of the last **Scanpy filter** {% icon tool %}
>
>    > <question-title></question-title>
>    >
>    > ```
>    > [n_obs × n_vars]
>    > -    2638 × 13714
>    > [obs]
>    > -    n_genes_by_counts
>    > -    log1p_n_genes_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > -    pct_counts_in_top_50_genes
>    > -    pct_counts_in_top_100_genes
>    > -    pct_counts_in_top_200_genes
>    > -    pct_counts_in_top_500_genes
>    > -    total_counts_mito
>    > -    log1p_total_counts_mito
>    > -    pct_counts_mito
>    > -    n_genes
>    > -    norm
>    > [var]
>    > -    gene_ids
>    > -    n_genes
>    > -    mito
>    > -    n_cells_by_counts
>    > -    mean_counts
>    > -    log1p_mean_counts
>    > -    pct_dropout_by_counts
>    > -    total_counts
>    > -    log1p_total_counts
>    > -    highly_variable
>    > -    means
>    > -    dispersions
>    > -    dispersions_norm
>    > ```
>    >
>    > 1. How many genes are in the `AnnData` object?
>    > 2. Where is the stored the information about the genes and if they are highly variable or not?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are now 13,714 genes, as before.
>    > > 2. Extra annotations have been added to `var`, whose a boolean annotation `highly_variable` for highly variable genes.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 2. {% tool [Manipulate Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of the last **Filter** {% icon tool %}
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>      - *"What to filter?"*: `Variables (var)`
>        - *"Type of filtering?"*: `By key (column) values`
>          - *"Key to filter"*: `highly_variable`
>          - *"Type of value to filter"*: `Boolean`
>          - *"Value to keep"*: `Yes`
>
> 3. Rename the generated output `3k PBMC with only HVG`
>
> 4. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with only HVG`
>    - *"What to inspect?"*: `General information about the object`
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 2638 × 1838
>    > ```
>    >
>    > How many genes have been removed?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > Only 1,838 (over the 13,714) genes are kept.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

## Scaling the data

Prior to any downstream analysis like dimensional reduction, we need to apply a linear transformation or scaling to:

1. Regress out unwanted sources of variation in the total counts per cell and the percentage of mitochondrial genes expressed.
2. Scale data to unit variance and zero mean, i.e. the variance across cells is 1 and the mean expression is 0, in order to give equal weight in downstream analyses and ensure that highly-expressed genes do not dominate.

> <hands-on-title>Scale the data</hands-on-title>
>
> 1. {% tool [Scanpy remove confounders](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_remove_confounders/scanpy_remove_confounders/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with only HVG`
>    - *"Method used for plotting"*: `Regress out unwanted sources of variation, using 'pp.regress_out'`
>      - *"Keys for observation annotation on which to regress on"*: `total_counts, pct_counts_mito`
>
> 2. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %}
>     with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Scanpy remove confounders** {% icon tool %}
>    - *"Method used for inspecting"*: `Scale data to unit variance and zero mean, using 'pp.scale'`
>      - *"Zero center?"*: `Yes`
>      - *"Maximum value"*: `10.0`
>
>        It clips values exceeding a standard deviation of 10.
>
> 3. Rename the generated output `3k PBMC with only HVG, after scaling`
{: .hands_on}

# Dimensionality reduction

We aim in scRNA-seq to compare cells based on their expressions across genes, e.g. to identify similar transcriptomic profiles. Each gene represents then a dimension of the data.

With a dataset of 2 genes, we could make a 2-dimensional plot where each point is a cell and each axis is the expression of one gene. For datasets with thousands of genes, the concept is the same: each cell's expression profile defines its location in the high-dimensional expression space.

Expressions of different genes are correlated if they are affected by the same biological process. The separate information for these individual genes do not need to be stored, but can instead be compressed into a single dimension, e.g. an "eigengene". Dimensionality reduction aims then to reduce the number of separate dimensions in the data and then:

- reduces the computational work in downstream analyses to only a few dimensions
- reduces noise by averaging across multiple genes to obtain a more precise representation of the patterns in the data
- enables effective plotting of the data.

## Principal Component Analysis

Principal Component Analysis (PCA) is a dimensionality reduction technique consisting in the identification of axes in high-dimensional space that capture the largest amount of variation. This simple, highly effective strategy is widely used in data science.

In PCA, the first axis (or Principal Component (PC)) is chosen such that it captures the greatest variance across cells. The next PC should be orthogonal to the first and capture the greatest remaining amount of variation, and so on.

By applying PCA to scRNA-Seq, we assume that multiple genes are affected by the same biological processes in a coordinated way and random technical or biological noise affects each gene independently. As more variation can be captured by considering the correlated behaviour of many genes, the top PCs are likely to represent the biological signal and the noise are concentrated into the later PCs. The dominant factors of heterogeneity are then captured by the top PCs. Restricting downstream analyses to the top PCs will then reduce the dimensionality of the data whilst focusing on the biological signal and removing the noise.

Here we perform the PCA on the log-normalized expression values and compute the first 50 PCs.

> <hands-on-title>Perform the PCA</hands-on-title>
>
> 1. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy0) %}
>    with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `3k PBMC with only HVG, after scaling`
>    - *"Method used for plotting"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'pp.pca'`
>      - *"Number of principal components to compute"*: `50`
>      - *"Type of PCA?"*: `Full PCA`
>        - *"Compute


topic: single-cell
tutorial: scrna-plant
title: Analysis of plant scRNA-Seq Data with Scanpy
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["plants", "paper-replication"]
content_hash: 1076c92d61ef3198101dfe26c0da63b1
last_modified: 2026-02-26T21:12:17.987258
zenodo_link: https://zenodo.org/record/4597857
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-plant/tutorial.html

content:
Single cell RNA-seq analysis is a cornerstone of developmental research and provides a great level of detail in understanding the underlying dynamic processes within tissues. In the context of plants, this highlights some of the key differentiation pathways that root cells undergo.

![The various cell subpopulations that are expected within this tutorial]({% link topics/single-cell/images/scrna-plant/scrna-plant1.jpg %})


This tutorial replicates the paper *"Spatiotemporal Developmental Trajectories in the Arabidopsis Root Revealed Using High-Throughput Single-Cell RNA Sequencing"* ({% cite denyer2019spatiotemporal %}), where the major plant cell types are recovered in the data as well as distinguishing between QC and meristematic cells. The original paper used the Seurat analysis suite ({% cite satija2015spatial %}), but here we will use the ScanPy analysis suite ({% cite wolf2018scanpy %}) integrated within the single-cell resources in Galaxy ({% cite tekman2020single %}).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title></comment-title>
>
> Please familiarise yourself with the ["Clustering 3K PBMCs with ScanPy"]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial first, as much of the process is the same, and the accompanying slide deck better explains some of the methods and concepts better.
>
{: .comment}

# Data

The *Arabidopsis* root cells come from two biological replicates which were isolated and profiles using droplet-based scRNA-seq (please see: ["Pre-processing of 10X Single-Cell RNA Datasets"]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %})); a "short root" mutant (labelled: `shr`) and a wild-type (labelled: `wt`):

* GSE123818\_Root\_single\_cell\_**shr**\_datamatrix.fixednames.transposed.csv.gz
* GSE123818\_Root\_single\_cell\_**wt**\_datamatrix.fixednames.transposed.csv.gz

As explained in the Zenodo link, the datasets have been modified to use more common gene names (hence the "fixednames" suffix) and the datasets have been transposed to better suit the AnnData format (hence the "transposed" suffix). The datasets are CSV tabular files which have been compressed using gunzip. We will now import these for analysis.


## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the two datasets from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/GSE123818_Root_single_cell_shr_datamatrix.fixednames.transposed.csv.gz
>    {{ page.zenodo_link }}/files/GSE123818_Root_single_cell_wt_datamatrix.fixednames.transposed.csv.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Apply the tags `#shr` and `#wt` to each of the datasets, with respect to the initial labels. These tags will be inherited for each task that the dataset undergoes.
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

> <question-title></question-title>
>
> We can peek at the `#shr` file, and determine the dimensionality and naming scheme of the data. The rows and the columns depict different variables.
>
> 1. Which are the cell barcodes? Rows or Columns?
> 1. Which are the gene names? Rows or Columns?
> 1. How many cells in the dataset?
> 1. How many genes in the dataset?
>
> > <solution-title></solution-title>
> >
> > 1. Rows, notice the long A-C-T-G based names.
> > 1. Columns, "ATXG" are the gene prefixes commonly used by the Arabidopsis genomes.
> > 1. There are 1 099 lines and therefore 1 099 cells
> > 1. If we scroll the preview window all the way to the right we see 27 630 as the last number. Given that the first column are the cell names, we must subtract by 1 to get 27 629 cells.
> >
> {: .solution}
>
{: .question}

If the above feels like a convoluted way to get the dimensionality, that's because we haven't imported the data into the right format. For this we need to import both datasets into a single AnnData object (see the [AnnData section in the previous tutorial]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}#anndata)).

## CSV to AnnData

> <hands-on-title>Converting the Data</hands-on-title>
>
> 1. {% tool [Import Anndata and loom](toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.7.5+galaxy0) %} with the following parameters:
>    - *"hd5 format to be created"*: `Anndata file`
>        - *"Format for the annotated data matrix"*: `Tabular, CSV, TSV`
>            - {% icon param-file %} *"Annotated data matrix"*: Multi-select both `SHR` and `WT` datasets
>
>    > <comment-title>Checking Dimensionality</comment-title>
>    >
>    > We can now inspect the dimensionality of the dataset by "peeking" at the dataset in the history and observing the general information, simply by clicking on the name of the dataset.
>    >
>    > The `#shr` dataset should show:
>    >    ```
>    >    [n_obs x n_vars]
>    >        1099 x 27629
>    >    ```
>    >
>    > and the `#wt` should show:
>    >    ```
>    >    [n_obs x n_vars]
>    >        4727 x 27629
>    >    ```
>    >
>    > where *obs* refers to our observed cells, and *vars* references the genes/variables.
>    >
>    {: .comment}
>
{: .hands_on}

Currently we have two seperate datasets, but we can merge them into one single AnnData object seperated by batch identifiers "shr" and "wt". To do this, we simply manipulate one of the datasets and concatenate the other onto it.

## Merge Batches and Relabel

> <hands-on-title>Merging Data</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `SHR dataset` (output of **Import Anndata and loom** {% icon tool %})
>    - *"Function to manipulate the object"*: `Concatenate along the observations axis`
>        - {% icon param-file %} *"Annotated data matrix to add"*: `WT dataset` (output of **Import Anndata and loom** {% icon tool %})
>
>    > <tip-title>Scheduling</tip-title>
>    >
>    > You can queue on the next task without the previous one being complete yet, just ensure that the input dataset for the next task is at least orange and not grey.
>    {: .tip}
>
>    > <comment-title></comment-title>
>    >
>    > At this point, the AnnData dataset batch information has a value of `0` for the "SHR" cells and `1` for the "WT" cells. This is fine if you wish to remember these two values for the remainder of the analysis since the step below is optional, however for plotting purposes it can just be easier to relabel the batch information with the action text labels `shr` and `wt`.
>    {: .comment}
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Rename categories of annotation`
>        - *"Key for observations or variables annotation"*: `batch`
>        - *"Comma-separated list of new categories"*: `shr, wt`
>
>    > <warning-title>Order matters!</warning-title>
>    >
>    > The order in which the datasets are concatenated can affect the accuracy of the new labelling. If the `#wt` dataset was first selected and then the `#shr` dataset concatenated onto it, then the comma-seperated list of new categories above should be inverted.
>    {: .warning}
>
{: .hands_on}

# Quality Control

We can confirm that our datasets have been combined into a single object by peeking at the dataset in the history and confirming two things:

1. The number of observations is equal to the total of the two initial AnnData datasets.
2. The file size of the new AnnData dataset is approximately equal to the total of the two initial AnnData datasets.

> |     Batch | Number of Cells | File Size |
> |----------:|:----------------|----------:|
> |       shr | 1099            |  117.2 MB |
> |        wt | 4727            |  499.7 MB |
> |----------:|:----------------|----------:|
> | **Total** | 5826            |  615.6 MB |
{: .matrix}

A happy coincidence here is that both datasets already had the exact same variables, so that the merging of the two datasets still yields the same number of genes.

## Filtering the Matrix

> <hands-on-title>Generating some metrics</hands-on-title>
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for inspecting"*: `Calculate quality control metrics, using 'pp.calculate_qc_metrics'`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>      - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>        - *"Keys for accessing variables"*: `n_genes_by_counts, total_counts`
>      - *"The key of the observation grouping to consider"*: `batch`
>      - In *"Violin plot attributes"*:
>        - *"Add a stripplot on top of the violin plot"*: `Yes`
>          - *"Add a jitter to the stripplot"*: `Yes`
>            - *"Size of the jitter points"*: `0.4`
>        - *"Display keys in multiple panels"*: `Yes`
>
{: .hands_on}

This should result in the following violin plot, where we see the distribution of the number of genes per cell (`n_genes_by_counts`) and the library size per cell, where the library size is the total number of mRNA in a cell, regardless of which gene it came from (`total_counts`).

!["Violin Plots per Batch"]({% link topics/single-cell/images/scrna-plant/violin1.png %})

Each dot is a cell, and the x-axis has two groups showing the absolute values of for each batch (`shr` and `wt`). The y-values correspond to the actual value of each cell, but the x-vales within each group are just random jitter to help see the cells more clearly. The outline of the violin plot shows us the density of each batch, where we can see that most of the cells in the `shr` group have an average number of features just below 2000 genes, compared to `wt` which has an average number of features just below 4000 genes. If we compare the library size, however, we notice that the outlines are very similar and that the average library sizes across batches are comparable.

### Filter

Most scRNA-seq datasets need to set a minimum threshold of detectability in order to ensure that the cells are of high enough quality to be used for analysis. Conversely, sometimes an upper threshold also should be set to counter the case where a cell is a cell-doublet (in the sense that two cells were captured and sequenced accidentally as a single cell).

The above violin plots suggest quite a few outliers, mostly on the upper-end of the density plots, suggesting a few cell-doublets in the data. The effect of these is not something we cannot easily regress out, and so we must filter them out instead. By looking at the plots, we can see that a good maximum library size here would be 100 000 transcripts, and a good maximum feature size would be 10 000 transcripts.

For this analysis, we will set a minimum threshold of detectability that each cell should count transcripts from at least 200 genes, and that each gene should be expressed in at least 5 cells. These lower-bound thresholds are not so easy to derive from such QC plots, but depend mostly on the type of data you are using. If the number of features and the library sizes were one order of magnitude higher, one might consider also scaling the lower-bound thresholds by one order of magnitude.

> <hands-on-title>Filtering</hands-on-title>
>
> 1. {% tool [Filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - *"Filter"*: `Minimum number of genes expressed`
>            - *"Minimum number of genes expressed required for a cell to pass filtering"*: `100`
>
> 1. {% tool [Filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Filter** {% icon tool %})
>    - *"Method used for filtering"*: `Filter genes based on number of cells or counts, using 'pp.filter_genes'`
>        - *"Filter"*: `Minimum number of cells expressed`
>            - *"Minimum number of cells expressed required for a gene to pass filtering"*: `2`
>
>    We now set the upper limits to the analysis.
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Filter** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `n_genes_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `12000`
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `total_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `120000`
>
{: .hands_on}

If we inspect the resulting dataset, we see that the number of cells remaining is `5826` and the number of genes is `18913`, which is still a good number of cells and genes. It is good practice to save the raw data in case we need to use it data, so we "freeze the raw data" into a slot for later safe keeping.

> <hands-on-title>Saving the original matrix</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Filter** {% icon tool %})
>    - *"Function to manipulate the object"*: `Freeze the current state into the 'raw' attribute`
>
{: .hands_on}


## Confounder Removal

Now we have a set of cells we are reasonably confident contain desirable biological variation (please view the  following segment from [*"An introduction to scRNA-seq data analysis"*]({% link videos/watch.md %}?v=single-cell/tutorials/scrna-intro/slides&t=11:28) as a reference for the types of wanted and unwanted variation). We wish to now regress out the library size variation, as it is not indicative of cell type. To do this, we will first ensure that all cells are of the same comparable library size by applying individual size factors to each cell.

We will also apply a log transformation to the cells, as this compresses the variation into a less extreme space, making it easier to see the relative differences between cells.

After normalising and regressing out unwanted factors, we will then scale the data to have unit variance with a zero mean, so that the mean expression of each gene is not a factor in the analysis and only the variation is, ensuring that the later clustering is driven only by the relative variation of the genes, and not neccesarily how expressive those genes are.

> <tip-title>Highly Variable Genes?</tip-title>
>
> Those of you who are familiar with the [ScanPy Tutorial]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) might wonder why we have not reduced the number of genes by performing a highly variable gene selection.
> The answer is simply that it did not help with this particular dataset, and that by removing the least variable genes in the analysis, it did help us replicate the analysis in the paper. Try it for yourself as an intermediate step (after this analysis) and see!
{: .tip}

> <hands-on-title>Normalize and Scale</hands-on-title>
>
> 1. {% tool [Normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>        - *"Target sum"*: `10000.0`
>        - *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Normalize** {% icon tool %})
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
>
> 1. {% tool [Remove confounders](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_remove_confounders/scanpy_remove_confounders/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used for plotting"*: `Regress out unwanted sources of variation, using 'pp.regress_out'`
>        - *"Keys for observation annotation on which to regress on"*: `total_counts`
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Remove confounders** {% icon tool %})
>    - *"Method used for inspecting"*: `Scale data to unit variance and zero mean, using 'pp.scale'`
>        - *"Maximum value"*: `10.0`
>
{: .hands_on}


## Dimensionality Reduction

Dimensionality reduction is the art of reducing a high dimensional dataset into a low dimensional "embedding" that humans can actually see (i.e. 2 or 3 dimensions), ideally such that the relationships or distances between data points are preserved in this embedding. In the context of single-cell datasets, this essentially means compressing > 10 000 genes into just 2 X/Y variables.


> <tip-title>What is Dimensionality Reduction?</tip-title>
>
> You can learn more about dimensionality reduction by consulting the following segment from [*"An introduction to scRNA-seq data analysis"*]({% link videos/watch.md %}?v=single-cell/tutorials/scrna-intro/slides&t=13:46).
>
{: .tip}

This is usually a two step process:

1. A Principal Component Analysis (PCA) is used to perform an optimized "rotation" of the ~ 20 000 "unit" gene axes that we have, in order to better fit the data. These new axes or "principal components" are then *linear* combinations of the original unit axes, with an associated score of how variable each new axis is. By sorting these principal components by most variable to least variable, we can select the top N components and discard the rest, leaving us with most of the variation still in the data. Usually we select the top 40 principal components, which from 20 000 is a *huge* reduction in the dimensionality of the dataset.
1. We then perform a more complex kind of dimensionality reduction, one that does not assume any linearity in the axes. For scRNA-seq, this is either tSNE or UMAP, with UMAP being the main choice due to how flexible it is in incorporating new data. Though UMAP is capable of working on extremely high dimensional datasets, it is often limited by time and space constraints (read: the computer does not respond in a reasonable timeframe, or it crashes), and so therefore it is quite normal to feed UMAP the output of the PCA as input. This will then be a dimensionality reduction from 50 to 2. With these final 2 dimensions, we can plot the data.

> <hands-on-title>PCA and UMAP</hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'tl.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>            - *"SVD solver to use"*: `ARPACK wrapper in SciPy`
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for inspecting"*: `Compute a neighborhood graph of observations, using 'pp.neighbors'`
>        - *"The size of local neighborhood (in terms of number of neighboring data points) used for manifold approximation"*: `10`
>        - *"Number of PCs to use"*: `40`
>
>    > <comment-title></comment-title>
>    >
>    > UMAP relies on a connected graph of cells to operate. Please view the following segment from [*"An introduction to scRNA-seq data analysis"*]({% link videos/watch.md %}?v=single-cell/tutorials/scrna-intro/slides&t=13:40) for more information on how this process works.
>    {: .comment}
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Method used"*: `Embed the neighborhood graph using UMAP, using 'tl.umap'`
>
{: .hands_on}

With our data now sufficiently "flat" and ready for human consumption, we can now do a naive plot of both the PCA and the UMAP and show cell assignments coloured by batch.

> <hands-on-title>Plot the PCA and UMAP by Batch</hands-on-title>
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in PCA coordinates, using 'pl.pca`
>      - *"Keys for annotations of observations/cells or variables/genes"*: `batch`
>      - In *"Plot attributes"*
>        - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>      - *"Keys for annotations of observations/cells or variables/genes"*: `batch`
>      - *"Show edges?"*: `No`
>      - In *"Plot attributes"*
>        - *"Legend font size"*: `14`
>        - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
{: .hands_on}

| PCA                                | UMAP                                 |
|:----------------------------------:|:------------------------------------:|
| ![PCA]({% link topics/single-cell/images/scrna-plant/pca_batch.png %}) | ![UMAP]({% link topics/single-cell/images/scrna-plant/umap_batch.png %})|


From this, we can see that there is a reasonable overlap in our batches shown both in the PCA and UMAP embeddings. This is good because it shows that though there is *some* batch effect (i.e. cells from one batch appear to cluster on a different side of the plot than the other) it is not significant enough for there not to be some commonality between the batches.

There are batch correction algorithms for cases where one batch clusters completely seperate from the other, but this is not necessary here.

# Finding Cell Types

## Clustering

Let us cluster the cells and see what cell types we can discover in the plots. The cell types that we wish to recover are those shown in the original paper, as highlighted by the gene markers in the DotPlot below.

| UMAP                                   | DotPlot                                    |
|:--------------------------------------:|:------------------------------------------:|
| ![PCA]({% link topics/single-cell/images/scrna-plant/clusters_expected.png %}) | ![UMAP]({% link topics/single-cell/images/scrna-plant/dotplot_expected.png %})|


> <hands-on-title>Generate and Plots Clusters</hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used"*: `Cluster cells into subgroups, using 'tl.leiden'`
>        - *"Coarseness of the clusterin"*: `0.35`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `leiden, batch`
>        - *"Show edges?"*: `No`
>        - In *"Plot attributes"*:
>            - *"Location of legend"*: `on data`
>            - *"Legend font size"*: `14`
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
>    > <comment-title></comment-title>
>    >
>    > We print the legend on the data because it's easier to see where the cluster labels apply.
>    {: .comment}
>
{: .hands_on}

![UMAP Leiden Clusters]({% link topics/single-cell/images/scrna-plant/clusters_found.png %})

Here we have recovered 13 clusters but we don't yet know what types they describe. If we have a list of genes that we know are indicative of a certain cell type (i.e. are marker genes) then we can use this to assign labels to our clusters.

Let us here try to recreate the DotPlot from the paper using the clusters we have discovered.

## DotPlot and Validating Cell Types

> <hands-on-title>Dot Plot</hands-on-title>
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `Generic: Makes a dot plot of the expression values, using 'pl.dotplot'`
>        - *"Variables to plot (columns of the heatmaps)"*: `Subset of variables in 'adata.var_names'`
>            - *"List of variables to plot"*: `AT1G26680,COBL2,AT3G55180,NCED2,AT5G22550,ATL63,RGF8,TEL1,DUF9,AGL42,PLT1,PLT2,PER3,PER9,PBL15,TBL30,AT2G48130,AT3G22620,AED3,T3P18-7,MBK21-8,NPF6-4,CYP71B34,IAMT1,AT1G31950,F14G6-22,GL2,ANL2,PMEI4,MUD21-4,EXPA7,MES15,ADF8,COBL9,TPR14,BHLH144,MYB83,SCPL35,MYB46,COBL4,PIN7,PIN3   `
>        - *"The key of the observation grouping to consider"*: `leiden`
>        - *"Number of categories"*: `9`
>        - *"Use 'raw' attribute of input if present"*: `Yes`
>        - In *"Group of variables to highlight"*:
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `0`
>                - *"End"*: `5`
>                - *"Label"*: `Columella`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `6`
>                - *"End"*: `9`
>                - *"Label"*: `QC`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `10`
>                - *"End"*: `11`
>                - *"Label"*: `NC`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `12`
>                - *"End"*: `17`
>                - *"Label"*: `Endodermis`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `18`
>                - *"End"*: `23`
>                - *"Label"*: `Cortex`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `24`
>                - *"End"*: `29`
>                - *"Label"*: `Atrichoblast`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `30`
>                - *"End"*: `34`
>                - *"Label"*: `Trichoblast`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `35`
>                - *"End"*: `39`
>                - *"Label"*: `Xylem`
>            - {% icon param-repeat %} *"Insert Group of variables to highlight"*
>                - *"Start"*: `40`
>                - *"End"*: `41`
>                - *"Label"*: `VC`
>        - *"Custom figure size"*: `No: the figure width is set based on the number of variable names and the height is set to 10.`
>
{: .hands_on}

By running the above we end up with the following DotPlot:

![Dotplot In Analyssis]({% link topics/single-cell/images/scrna-plant/dotplot_found.png %})

Notice how we have the Columella, QC and NC sharing the same cluster 11, that the Endodermis is localised in cluster 8, the Cortex in cluster 9, the Trichoblast cells in cluster 3, the Xylem showing strong expression in cluster 12, VC in clusters 4 and 6, and the Atrichoblasts have a mixed expression in a range of clusters. These characteristics are the same as the Dotplot in the original paper.

We can use this Dotplot as a guide to relabel our clusters and give more meaningful annotations to the plots.

## Relabel clusters

> <hands-on-title>Cluster Re-labelling</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Function to manipulate the object"*: `Rename categories of annotation`
>        - *"Key for observations or variables annotation"*: `leiden`
>        - *"Comma-separated list of new categories"*: `0, 1, 2, trichoblasts, 4, 5, 6, atrichoblasts, endodermis, cortex, 10, columella+QC+NC, xylem`
>
> 1. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for plotting"*: `Embeddings: Scatter plot in UMAP basis, using 'pl.umap'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `leiden, batch`
>        - *"Show edges?"*: `No`
>        - In *"Plot attributes"*:
>            - *"Location of legend"*: `on data`
>            - *"Legend font size"*: `14`
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
{: .hands_on}

![Relabelled clusters]({% link topics/single-cell/images/scrna-plant/clusters_found_relabelled.png %})


If we look at the clusters now, and compare them to the original image in the paper, we can infer that the meristematic cells are likely to be derived from cluster 1, which shares soft clustering with the trichoblasts suggesting a trajectory pathway which could be explored.


# Conclusions

In this tutorial, we have recapitulated the same clustering analysis in the "Spatiotemporal Developmental Trajectories in the Arabidopsis Root Revealed Using High-Throughput Single-Cell RNA Sequencing" {% cite denyer2019spatiotemporal %} paper, and validated them by comparing DotPlots for specific genes that were used as markers in that paper.

From this point, we can perform a lineage analysis to infer a differentiation pathway between the clusters. For ScanPy there is the PAGA option, however, this does not work so well with the current dataset, so it is encouraged that users use the original Seurat trajectory suite that was given in the paper, or to experiment with Monocle.

Both libraries are available within the RStudio and Jupyter Notebook libraries in the interactive Galaxy envronments which can be found in the *"Miscellaneous Tools"* section under the *"Interactive Tools"* subheading. An excellent follow-up tutorial to perform a trajectory analysis in Galaxy using Jupyter notebooks would be the [*"Trajectory Analysis using Python (Jupyter Notebook) in Galaxy"*]({% link topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.md %}).

This entire tutorial can be invoked from the scRNA Plant Workflow shown below:

![scRNA Plant]({% link topics/single-cell/images/scrna-plant/scrna-plant-workflow.png %})

The input parameters take two datasets as input, and 5 parameters using the defaults shown in this tutorial: min cells, min genes, max features, max library size, and Leiden clustering resolution. Play with these options to see how robust the analysis really is!


topic: single-cell
tutorial: scrna-case_monocle3-rstudio
title: Inferring single cell trajectories with Monocle3 (R)
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 0b91a62c2b04b2e99a2d3af95e319e0c
last_modified: 2026-02-26T21:12:17.965889
zenodo_link: https://zenodo.org/record/7455590
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_monocle3-rstudio/tutorial.html

content:
{% snippet topics/single-cell/faqs/notebook_warning.md %}

## Setting up the environment and file upload
Once the installation is done, we should load the needed packages into our notebook. Navigate back to your `notebook`. If you are using our prepopulated notebook, you can follow the tutorial from there. Otherwise, input the following into your fresh notebook.

```r
install.packages("Rcpp")    # needed for reduce_dimension to avoid AnnoyAngular error; library(Rcpp) might work as well depending on the version
```

```r
library(monocle3)
library(biomaRt)
library(magrittr)     # needed for %>%
library(viridisLite)
```

```r
install.packages("anndata")
```

Alright, the libraries are here - so now let's get our AnnData file.

## Upload, view and modify the files

We can download the file into Jupyter notebook also using a Zendo link. The first argument is the download link and the second one is the name under which the downloaded file is saved:
```r
download.file('https://zenodo.org/record/7455590/files/AnnData_filtered.h5ad', 'AnnData.h5ad')
```
Then we have to read in this h5ad file:
```r
ann <- anndata::read_h5ad('AnnData.h5ad')
```
Now we store all the information we need in AnnData object. However, Monocle uses *cell_data_set class* to hold expression data, which requires three input files: `expression_matrix`, `cell_metadata` and `gene_metadata`. Therefore, we have to somehow 'transfer' that information from our AnnData object to Monocle's cell_data_set (cds). AnnData stores a data matrix `X` together with annotations of observations `obs` and variables `var`, so we can extract those parameters and use them for further analysis.

```r
expression_matrix <- ann$X
cell_metadata <- ann$obs
gene_metadata <- ann$var
```

> <tip-title>Uploading files from your computer is also an option!</tip-title>
>
> If you already have files containing the expression matrix, genes and cells metadata, you can upload them to JupyLab and generate a cds file from them instead. For example, if you first downloaded the files from Galaxy, your files will have the `.tabular` extension. In this case, we will use the `read.delim()` function to read them in. In this function, the first argument is the file path - in our case, the files are in the same folder as the notebook, so the file path is the same as the file name. You can always check that by right-clicking on the file and choosing `Copy path`. The second argument, `row.names=1` takes the column number of the data file from which to take the row names.
> ```r
> # read in the files
> cell_metadata <- read.delim('cells.tabular', row.names=1)
> gene_metadata <- read.delim('genes.tabular', row.names=1)
> expression_matrix <- read.delim('expression.tabular', row.names=1)
> ```
>
> > <question-title></question-title>
> >
> > Why should we set `row.names=1`?
> >
> > > <solution-title></solution-title>
> > >
> > > This allows us to ensure that the expression value matrix has the same number of columns as the `cell_metadata` has rows and the same number of rows as the `gene_metadata` has rows. Importantly, the row names of the `cell_metadata` object should match the column names of the expression matrix and the row names of the `gene_metadata` object should match the row names of the expression matrix.
> > >
> > {: .solution}
> {: .question}
>
{: .tip}


><tip-title>Reading in the files for RStudio users</tip-title>
>
> If you are working in RStudio Cloud, you have to click on ‘Upload’ button in the right bottom window toolbar and choose already downloaded data files to upload. You should now see all three files in this window. You might want to rename the files to make their names shorter.
>
> ![Screenshot of Files tab in RStudio, highlighting 'Upload' and 'Rename' buttons and listing three uploaded and renamed files: 'cell_metadata', 'gene_metadata', 'expression_matrix'.](../../images/scrna-casestudy-monocle/r_files_tab.png "The view of the Files tab with uploaded files and highlighted relevant buttons.")
>
> If you are using RStudio Galaxy tool, you can get data directly from your history by running:
> ```
># get the files from Galaxy history
>file.copy(gx_get(2), "cell_metadata")
>file.copy(gx_get(3), "gene_metadata")
>file.copy(gx_get(4), "expression_matrix")
>```
>The number in the brackets corresponds to the dataset number in your history, so make sure you put the right number for the corresponding file. We can specify the name of the fetched files in the quotation marks, as shown above. All three files should appear in the Files tab window.
>
> If you are using RStudio locally, then you don’t have to bother about uploading the files – just run `file.choose()` and choose the corresponding file which you want to get path to:
>```
># get the file path
>cells_path <- file.choose()
>genes_path <- file.choose()
>expression_path <- file.choose()
>```
> You should now see the new variables in the Environment tab window.
>
{: .tip}


According to the [Monocle3 documentation](https://cole-trapnell-lab.github.io/monocle3/docs/starting/), the `expression_matrix` should have genes as rows and cells as columns. Let's check if that's the case here.
```r
head(expression_matrix,c(5, 5))           # preview the content of the file by calling its the first 5 rows by 5 columns
```
We can see that in our matrix rows are cells and genes are columns, so we have to transpose the matrix simply using function `t()`. But before doing so, we will change its type from dataframe to matrix - this is Monocle's requirement to generate cell_data_set afterwards.
```r
expression_matrix <- as.matrix(expression_matrix)   # change the type to matrix
expression_matrix <- t(expression_matrix)           # transpose the matrix
```
Another condition we have to satisfy is that one of the columns of the `gene_metadata` should be named "gene_short_name", which represents the gene symbol for each gene. Some functions won't work without that. Do we have such a column? Let's check.
```r
head(gene_metadata)             # preview the top ten rows of the file to check the name of the column containing gene symbols
```

The second column indeed contains gene symbols, but is called "Symbol" instead of "gene_short_name". That can be easily changed by a simple assignment, as long as we know the number of the column that we want to modify. In our case the gene symbols are stored in column 2. We can access the column names by `colnames()`.
```r
colnames(gene_metadata)[2] <- 'gene_short_name'     # change the column name
colnames(gene_metadata)                             # see the changes
```

## Generating CDS object
Now let’s store our files in one object – the `cell_data_set`. This is the main class used by Monocle to hold single cell expression data. The class is derived from the Bioconductor SingleCellExperiment class. Similar to Python's AnnData, the cell_data_set stores a data matrix together with annotations of observations and variables. There are three ways of creating CDS object in monocle:
-	Using ```new_cell_data_set() ``` function with three data frames as arguments (not their paths!): expression matrix (can also be a sparseMatrix), cell metadata and gene metadata
-	Using ```load_cellranger_data() ``` function providing the path to the folder containing 10X Genomics Cell Ranger output files. This function takes an argument `umi_cutoff` that determines how many reads a cell must have to be included
-	Using ```load_mm_data() ``` function providing the paths to matrix file and two metadata files (features and cell information).

In this tutorial we will use the first option:
```r
# create CDS object
cds <- new_cell_data_set(expression_matrix, cell_metadata, gene_metadata)
```
We are now ready to process our data!

> <details-title>Format conversion</details-title>
>
>  Since Monocle’s CDS object is analogous to Python's AnnData, why don’t we use some kind of conversion between those two formats? There is indeed a package called `sceasy` that helps easy conversion of different single-cell data formats to each other. However, when we tested this conversion on our dataset and then used Monocle to plot the expression of genes, the plots were not correct – the expression was shown to be identical throughout the sample. For comparison, Seurat did well when plotting gene expression of the same converted object! Although conversion functions are very handy, you have to be aware that their output might be interpreted differently by certain packages. Therefore, to make sure that the analysis is reliable, we decided to generate CDS object directly using Monocle’s function.
> ![Comparison between plots of gene expression generated by Monocle and Seurat using the CDS object that was converted from AnnData using SCEasy tool. Gene expression in Monocle plots is identical throughout the sample, while the expression of the same genes plotted by Seurat is noticeable only in specific clusters.](../../images/scrna-casestudy-monocle/monocle_seurat.png "Comparison between plots of gene expression generated by Monocle and Seurat using the CDS object that was converted from AnnData using SCEasy tool.")
>
{: .details}

# Additional step: adding genes symbols based on their IDs
> <warning-title>Additional step</warning-title>
>  This step is not necessary for the dataset we are working on but some users might find it helpful when analysing their own data.
>
{: .warning}

If you remember the very first tutorial in the case study, we started with gene IDs and added gene symbols based on the Ensembl GTF file.  
But what if we didn’t have the genes symbols in our CDS object and wanted to add them now? Of course - it's possible! We will also base this annotation on Ensembl - the genome database – with the use of the library BioMart. We will use the same archive as in the Alevin tutorial (Genome assembly GRCm38) to get the gene names. Please note that the updated version (GRCm39) is available, but some of the gene IDs are not in that EnsEMBL database. The code below is written in a way that it will work for the updated dataset too, but will produce ‘NA’ where the corresponding gene name couldn’t be found.
```r
cds_extra <- cds                                        # assign our CDS to a new object for the demonstration purpose
head(rownames(fData(cds_extra)))                        # preview of the gene IDs as rownames
```
```r
# get relevant gene names
library("biomaRt")                                      # load the BioMart library
ensembl.ids <- rownames(fData(cds_extra))               # fData() allows to access cds rowData table
mart <- useEnsembl(biomart = "ENSEMBL_MART_ENSEMBL")    # connect to a specified BioMart database and dataset hosted by Ensembl
ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl", host='https://nov2020.archive.ensembl.org') 	

# The line above connects to a specified BioMart database and dataset within this database.
# In our case we choose the mus musculus database and to get the desired Genome assembly GRCm38,
# we specify the host with this archive. If you want to use the most recent version of the dataset, just run:
# ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl")
```
```r
genes <- getBM(attributes=c('ensembl_gene_id','external_gene_name'),
               filters = 'ensembl_gene_id',
               values = ensembl.ids,
               mart = ensembl_m)

# The line above retrieves the specified attributes from the connected BioMart database;
# 'ensembl_gene_id' are genes IDs,
# 'external_gene_name' are the genes symbols that we want to get for our values stored in ‘ensembl.ids’.
```
```r
# see the resulting data
head(genes)                          
```
```r
# replace IDs for gene names
gene_names <- ensembl.ids	 
count = 1 	 
for (geneID in gene_names)
{
 index <- which(genes==geneID)    # finds an index of geneID in the genes object created by getBM()
 if (length(index)==0)            # condition in case if there is no corresponding gene name in the chosen dataset
  {
    gene_names[count] <- 'NA'
  }
  else
  {
    gene_names[count] <- genes$external_gene_name[index] 	# replaces gene ID by the corresponding gene name based on the found geneID’s index
  }
 count = count + 1                # increased count so that every element in gene_names is replaced
}
```
```r
# store the gene names in our CDS object in a new column gene_short_name_extra
fData(cds_extra)$gene_short_name_extra <- gene_names
```
```r
# see the changes
fData(cds_extra)                    
```

If you are working on your own data and it’s not mouse data, you can check available datasets for other species and just use relevant dataset in `useMart()` function.
```r
listDatasets(mart)                # available datasets
```

> <warning-title>Ensembl connection problems</warning-title>
> Sometimes you may encounter some connection issues with Ensembl. To improve performance Ensembl provides several mirrors of their site distributed around the globe. When you use the default settings for useEnsembl() your queries will be directed to your closest mirror geographically. In theory this should give you the best performance, however this is not always the case in practice. For example, if the nearest mirror is experiencing many queries from other users it may perform poorly for you. In such cases, the other mirrors should be chosen automatically.
>
{: .warning}

# Monocle workflow
Do you remember the Monocle workflow introduced in the previous tutorial? Here is a recap:
![Monocle workflow: scRNA-seq dataset, pre-process data (normalise, remove batch effects), non-linear dimensionality reduction (t-SNE, UMAP), cluster cells, compare clusters (identify top markers, targeted contrasts), trajectory analysis](../../images/scrna-casestudy-monocle/monocle3_new_workflow.png "Workflow provided by Monocle3 documentation")

## Pre-processing
Let’s start with normalisation and pre-processing that can be performed using the function `preprocess_cds()`. The argument `num_dim` is the number of principal components that will be used. You can check that you're using enough PCs to capture most of the variation in gene expression across all the cells in the data set. Note that “PCA” is the default method of pre-processing in Monocle3, so although we can specify this in our function, we don’t have to.

Note that this step can take awhile - around 5 or so minutes due to the high number of PCs calculated. Feel free to use a `num_dim` value around 100 to accelerate preprocessing and compare the results.

```r
# PCA pre-processing with 210 principal components
cds_preprocessing <- preprocess_cds(cds, method = "PCA", num_dim = 210)
```
```r
 # plot the variation in gene expression vs PCA components
plot_pc_variance_explained(cds_preprocessing)                              		                            
```

![Plot of variation in gene expression vs PCA components, decreasing exponentially.](../../images/scrna-casestudy-monocle/pca_plot.jpg " Plot of variation in gene expression vs PCA components.")

The plot shows that actually using more than ~100 PCs captures only a small amount of additional variation. However, if we look at how the cells are plotted on 2D graph when using different values of PCs, it is easier to visualise how the `num_dim` actually affects the output. We will use the value of 210, which, compared to the results from the previous tutorial, makes the most sense for our dataset.

![Six plots showing only the shaded shape of how the cells are clustered depending on the num_dim argument. The general trend is maintained though.](../../images/scrna-casestudy-monocle/num_dim.jpg "The "shape" of the plot showing how the cells are clustered depending on the 'num_dim' argument.")

## Batch correction and Dimensionality reduction
Our dataset actually comprises data from 7 samples, so there is a risk that batch effects will impact analysis. Batch effects are systematic differences in the transcriptome of cells measured in different experimental batches. However, we can use Monocle to deal with that!
First, let’s check how our dataset looks in terms of batch effects. We can do that by colouring the cells by batch.  This information is stored in our CDS object from the `cell_metadata` file. Before asking Monocle to plot anything, let’s check the exact column name of the batch information column.
```r
colnames(colData(cds_preprocessing)) 	          # check column names
```
In our case it’s indeed ‘batch’, but your data might have another name (eg. "plate", etc.), so make sure you put the correct argument value.

In order to plot our cells on a 2D graph, we need to reduce the numbers of dimensions. The [previous tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}) introduced the methods of dimensionality reduction in Monocle. You can replicate what we did in Galaxy to compare the output of dimensionality reduction using different methods, simply by changing the `reduction_method` argument. Options currently supported by Monocle are "UMAP", "tSNE", "PCA", "LSI", and "Aligned". However, as for now, let’s just recall that UMAP gave the best results, so we will use UMAP here as well.
```r
# reduce dimension first
cds_preprocessing_UMAP <- reduce_dimension(cds_preprocessing, reduction_method = "UMAP", preprocess_method = "PCA")
```
```r
# then produce a plot to check for batch effects
plot_cells(cds_preprocessing_UMAP, color_cells_by="batch", label_cell_groups=FALSE)
```

We can see that upper and lower right branches mostly consist of N705 and N706, so indeed batch correction might be helpful. Let’s run this.
```r
# perform batch correction
cds_batch <- align_cds(cds_preprocessing_UMAP, preprocess_method = "PCA", alignment_group = "batch")
```

To see the changes, we have to run UMAP again, but this time on the aligned dataset. We will specify that `preprocess_method ` as "Aligned" and not “PCA". Monocle would use the "Aligned" argument automatically if no `preprocess_method` was specified.
```r
# dimensionality reduction after alignment
cds_red_dim <- reduce_dimension(cds_batch, preprocess_method = "Aligned", reduction_method = "UMAP")  
```
```r
# see the batch correction effect on a plot
plot_cells(cds_red_dim, color_cells_by="batch", label_cell_groups=FALSE)		
```

![Left image showing dataset before batch correction: upper and lower right branches mostly consist of N705 and N706. Right image showing the dataset after batch correction: the cells from all the samples are evenly spread throughout the whole dataset.](../../images/scrna-casestudy-monocle/batch_correction.png "Comparison of the dataset before and after batch correction.")

> <question-title></question-title>
>
> Does your plot look the same as the one in the Figure?
>
> > <solution-title></solution-title>
> >
> > Your plot might be slightly different to the one shown in the Figure but this is fine, as long as you see analogical patterns. Some libraries that you're using might have been updated, giving non-identical output. However, the principle behind the analysis is still the same, so you can peacefully follow the tutorial. Just keep your eyes open and... THINK!
> >
> {: .solution}
>
{: .question}

Do you see this? It’s amazing! Batch correction did a great job here! Now the dataset is nicely aligned, and the cells from all the samples are evenly spread throughout the whole dataset. It is worth mentioning that removing batch effects was done using mutual nearest neighbor alignment, a technique introduced by John Marioni's lab ({% cite Haghverdi_2018 %}) and supported by Aaron Lun's package [batchelor](https://bioconductor.org/packages/release/bioc/html/batchelor.html). Also, due to the machine learning elements of the code of this technique - as well as the fact that packages are updated regularly - your plots may not look identical to the ones pictured here. Nevertheless, the interpretation should be the same - the batch corrected plot should show better batch distribution than the uncorrected one.

Now we can move to the next step and perform dimensionality reduction.


> <tip-title>Plotting: labels vs legend</tip-title>
>
>  When creating graphs, we sometimes use labels and sometimes just a legend. You can choose whichever you think makes the data clear and readable.
>
> If you want to use a legend, then specify an argument `label_cell_groups=FALSE` in the function `plot_cells()`.
>
> The labels are set automatically, but if you want to change their size (default labels are tiny), use the argument `group_label_size`.
{: .tip}


## Clustering: clusters
We want to get some information about cell types, don’t we? In order to do so, we have to cluster our cells first.
Monocle uses a technique called "community detection" ({% cite Traag_2019 %}) to group cells. This approach was introduced by {% cite Levine_2015 %} as part of the phenoGraph algorithm.
Monocle also divides the cells into larger, more well separated groups called partitions, using a statistical test from {% cite Wolf_2019 %}, introduced as part of their [PAGA](https://github.com/theislab/paga) algorithm.

> <details-title>Clusters vs partitions</details-title>
>
> Clusters are particularly useful while trying to assign cells to a certain type, because they are based on the similarity in gene expression. The relationships between different clusters are analysed to identify possible trajectories.
>
> Partitions, meanwhile, are larger groups of cells that usually contain several clusters. Trajectory inference is performed only within one partition, so it is essential that all the cells that we want to analyse in pseudotime belong to the same partition.
>
{: .details}

Therefore, let’s perform clustering and visualise the resulting clusters.
```r
# clustering
cds_auto_cluster <- cluster_cells(cds_red_dim, reduction_method = "UMAP")
```
```r
# see the clusters
plot_cells(cds_auto_cluster, reduction_method = "UMAP", color_cells_by = 'cluster', group_label_size = 5)     
```

When using standard igraph louvain clustering, the value of resolution parameter is by default set to NULL, which means that it is determined automatically. Although the resulting clusters are OK, it would be nice to get some more granularity to identify cell types more specifically. The higher the resolution value, the more clusters we get. We will set the resolution value to 0.0002, but you are very welcome to try different values to see the changes.

```r
# clustering with changed resolution value
cds_clustered <- cluster_cells(cds_red_dim, reduction_method = "UMAP", resolution = 0.0002)
```
```r
# see the new clusters
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'cluster', group_label_size = 5) 	
```

![Left image showing 6 clusters formed using automatic standard igraph louvain clustering. Right image showing the dataset with clusters formed using resolution argument set to 0.0002: now there are 7 clusters, as one automatically formed cluster could be divided into two smaller distinct clusters.](../../images/scrna-casestudy-monocle/clusters_compare.png "Comparison of the clusters formed using standard igraph louvain clustering and using resolution argument set to 0.0002.")



## Clustering: partitions
OK, what about partitions? They were also created during the clustering step and it’s important to check them before calculating the trajectory because it is performed only within one partition. It is essential that all the cells that we want to analyse in pseudotime belong to the same partition.
```r
# see the partitions
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```

While your plot might be slightly different due to package updates, we can see that there are 3 partitions identified in `cds_clustered` object. Ideally, we would like to combine partitions 1 and 2 to draw a trajectory through all those cells (we can ignore cells in outlier partition). Sometimes using the default values might result in multiple partitions while you only need one. Then you would have to change the q-value cutoff in `partition_qval`. The default is 0.05 and by increasing this value you can increase the span of partitions, meaning that you would get fewer partitions. When trying different values of q-value, you also have to check if the clusters didn't change. It's all about finding a balance between the value of `resolution` and `partition_qval` so that both clusters and partitions are satisfactory enough for downstream analysis. Let's try that on our dataset.

```r
# changing the partition q-value
cds_clustered <- cluster_cells(cds_red_dim, reduction_method = "UMAP", resolution = 0.0002, partition_qval = 1)
```
```r
# see the partitions with the changed q-value
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```
```r
# check if clusters didn't change
plot_cells(cds_clustered, reduction_method = "UMAP", color_cells_by = 'cluster', label_cell_groups=FALSE)
```

> <question-title></question-title>
>
> Have the clusters change after changing the partition q-value?
>
> > <solution-title></solution-title>
> >
> > It might be the case that after changing partition q-value, you will notice that additional clusters appeared. In that situation, you might either play around the `resolution` and `partition_qval` values, go forward with the current clustering (adjusting the parameters accordingly), or check the other method of assigning cells to one partition given below.
> >
> {: .solution}
>
{: .question}

Now we have cells of interest in one partition, we still have reasonable clusters, so now we can learn the trajectory. However, in some cases even this method might not be enough. Then, there is a last resort… assigning cells to a partition manually.

## Additional step: assigning cells to one partition
> <warning-title>Additional step</warning-title>
>  This step is not necessary for the dataset we are working on but some users might find it helpful when analysing their own data.
>
{: .warning}

Let’s assume we have 4 partitions that cannot be extended to one big partition using `partition_qval` (it might sound unreasonable but it does happen!) and we are desperate to have all the cells in one partition to draw a trajectory through all of them. We can simulate the initial dataset by setting `partition_qval` value to 0.0001.
```r
# simulate the dataset
cds_partitions_extra <- cluster_cells(cds_red_dim, reduction_method = "UMAP", partition_qval = 0.0001)		
```
```r
# see the simulated partitions
plot_cells(cds_partitions_extra, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```
```r
# store ‘1’ the number of times equal to the number of cells
big_partition <- c(rep(1,length(cds_partitions_extra@colData@rownames))) 	
```
```r
# take the barcode of each cell and assign ‘1’ to each of them (now the ‘ones’ are named)
names(big_partition) <- cds_partitions_extra@colData@rownames 	
```
```r
# convert from numeric to factor
big_partition <- as.factor(big_partition)
```
```r
# assign the created barcodes-partition list to the location where information about partitions is stored in CDS object
cds_partitions_extra@clusters$UMAP$partitions <- big_partition 	
```
```r
# see the new partition assignment
plot_cells(cds_partitions_extra, reduction_method = "UMAP", color_cells_by = 'partition', label_cell_groups=FALSE)
```

![Left image showing the dataset divided into 4 partitions. Right image showing all cells assigned to one partition.](../../images/scrna-casestudy-monocle/partitions_compare.png "The result of the manual assignment of all the cells to one partition.")


> <details-title>@ and $ operators</details-title>
>
> As you saw above, we used `@` and `$` operators to navigate in CDS object. What is the difference between them? `$` operator is used to access one variable/column, while `@` extracts the contents of a slot in an object with a formal (S4) class structure. If you use `View()` function on our CDS object, you will see the elements of type "S4" that you can access using `@` operator. The suggestion list of elements that can be accessed should also pop up as you type the name of the object followed by the corresponding operator.
>
{: .details}

# Assigning cell types
There are two main approaches to assigning cell types to clusters that we’ve just identified – supervised and unsupervised, both based on gene expression in each cluster.

## Supervised approach
The supervised approach relies on prior knowledge of which cell types to expect. We can simply check the expression of marker genes specific to the expected cell types. Let’s then check the markers mentioned in the original paper {% cite Bacon2018 %}.

| Marker | Cell type |
|--------------------|
| Il2ra    | Double negative (early T-cell)    |
| Cd8b1, Cd8a, Cd4    | Double positive (middle T-cell)|
| Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell)|
| Itm2a    | Mature T-cell |
| Aif1    | Macrophages    |
| Hba-a1    | RBC    |

To plot the expression of all those genes across our dataset in one go, we will pass a vector with the names of those markers into a parameter `genes` in the `plot_cells` function.
```r
# expression of marker genes across the sample
plot_cells(cds_clustered, genes=c('Il2ra','Cd8b1','Cd8a','Cd4','Itm2a','Aif1','Hba-a1'), reduction_method = "UMAP")
```

![7 graphs, each showing expression of one of the genes in the corresponding clusters: 'Il2ra' – cluster 4; 'Cd8b1' and 'Cd8a' – clusters 1,6,2; 'Cd4' – high expression in cluster 5; 'Itm2a' – cluster 3; 'Aif1' – barely anything, no specific cluster assigned; 'Hba-a1' - throughout the entire sample in low numbers.](../../images/scrna-casestudy-monocle/genes.png "Expression of the reference marker genes across analysed sample.")


> <question-title>Genes, cell types and clusters</question-title>
> Based on the gene expression graph that we just generated, the table above and your knowledge from the previous tutorials, how would you assign the clusters?
>
> > <solution-title></solution-title>
> >
> > Keep in mind that these results refer to our numbered clusters, while yours might be slightly different.
> > - `Il2ra` (DN): mostly expressed in cluster 4
> > - `Cd8b1, Cd8a` (DP middle): expressed in clusters 1, 6, and highly in cluster 2
> > - `Cd4` (DP late): average expression in clusters 1, 6, 2 and high expression in cluster 5
> > - `Itm2a` (T-mat): expressed in cluster 3
> > - `Aif1` (macrophages): barely anything here, minimal expression spread across the sample with some more cells in cluster 4 and 3 – not enough to form a distinct cluster though). In theory, we shouldn’t have any macrophages in our sample. If you remember from the previous tutorials, we actually filtered out macrophages from the sample during the processing step, because we worked on annotated data. When analysing unannotated data, we could only assign macrophages and then filter them out, provided that Monocle clusters them into a separate group. As you can see, it’s not the case here, so we will just carry on with the analysis, interpreting this as a contamination.
> > - `Hba-a1` (RBC): appears throughout the entire sample in low numbers suggesting some background contamination of red blood cell debris in the cell samples during library generation, but also shows higher expression in a distinct tiny bit of cluster 3, at the border between clusters 1 and 5. However, it’s too small to be clustered into a separate group and filtered out in this case.
> > If you remember, this gene was found to be expressed in the previous Scanpy tutorial also in low numbers across the sample, and in the other Monocle tutorial (using Galaxy tools and annotated data) algorithms allowed us to gather the cells expressing that gene into a distinct group. Our result now sits somewhere in between.
> > ![In Scanpy graph the marker gene appears throughout the entire sample in low numbers, in Monocle in Galaxy cells expressing hemoglobin gene were grouped into a small branch of DP-M4, allowing to group those cells. Monocle in RStudio graph is somewhere in between showing mostly low expression across the sample, but also having a tiny bit of grouped cells, less distinct than in Galaxy though.](../../images/scrna-casestudy-monocle/hb_all.png "Hemoglobin across clusters - comparison between Scanpy, Monocle using Galaxy tools and Monocle run in RStudio.")
> >
> {: .solution}
{: .question}

Having identified which cluster corresponds to a specific cell type, we can finally run some code to add the annotation to our CDS object. First, we will create a new column called `cell_type`  in `colData()` - this is where the information about the cells is stored (eg. batch, genotype, sex, etc) - and initialize it with the values of clusters. Then, we will get the `dplyr` package which will be used for cluster annotation.

```r
# just to keep the objects tidy and not overwrite them so that you can come back to any point of the analysis
cds_annotated <- cds_clustered 	
```
```r
# create a new column ‘cell_type’ and initialise it with clusters values
colData(cds_annotated)$cell_type <- as.character(clusters(cds_annotated)) 	
```
```r
# annotate clusters
colData(cds_annotated)$cell_type <- dplyr::recode(colData(cds_annotated)$cell_type,
                                                       '1'='DP-M1',   # double positive – middle T-cell (1st cluster)
                                                       '2'='DP-M2',	  # double positive – middle T-cell (2nd cluster)
                                                       '3'='T-mat',	  # mature T-cell
                                                       '4'='DN',		  # double negative – early T-cell
                                                       '5'='DP-L',	  # double positive – late middle T-cell
                                                       '6'='DP-M3',	  # double positive – middle T-cell (3rd cluster)
                                                       '7'='Unknown') # no info for now, so call it ‘Unknown’
                                                       '8'='Unknown') # no info for now, so call it ‘Unknown’
```
```r
# check the annotation
plot_cells(cds_annotated, color_cells_by="cell_type", label_cell_groups=FALSE)    
```

![A plot showing the identified clusters, now each coloured by the assigned cell type.](../../images/scrna-casestudy-monocle/annotated.png "Our annotated dataset.")

## Unsupervised approach
But what if we don’t have any reference that we can use to assign our clusters? In that case, we will turn to the mentioned unsupervised approach - we will check what are the specifically expressed genes for each cluster. Then we can identify the cell types by looking up what cell types contain those genes. That’s a more tedious process, but sometimes can lead to exciting and unexpected results.
We will use Monocle’s function `top_markers()` and store the information about specifically expressed genes for each cluster in the data frame `marker_test`.
```r
# find top marker genes in each cluster
marker_test <- top_markers(cds_clustered, group_cells_by="cluster", reduction_method = "UMAP", reference_cells=1000, cores=8)
```
You can group the cells by any categorical variable in `colData(cds_clustered)`. The parameter `reference_cells` is used to accelerate the marker significance test at some cost in sensitivity. It works by randomly selecting a specified number of cells and performing the marker significance test against a chosen set of cells. If your dataset is not massively big, you might skip this parameter as it wouldn’t help much.

> <question-title></question-title>
>
> What are the variables stored in `marker_test` data frame and what do they mean?
>
> > <solution-title></solution-title>
> >
> > Those are 10 variables that you can check easily either using colnames(marker_test) or `View(marker_test)` which will also display all the corresponding values.
> >- `gene_id` - Ensembl gene ID
> >- `gene_short_name` - short name of the gene corresponding to its ID
> >- `cell_group` - a group to which the cell belongs, specified in the `group_cells_by` argument
> >- `marker_score` - numeric vector of marker scores as the fraction expressing scaled by the specificity. The value ranges from 0 to 1
> >- `mean_expression` -  numeric vector of mean normalized expression of the gene in the cell group
> >- `fraction_expressing` - numeric vector of fraction of cells expressing the gene within the cell group
> >- `specificity` - numeric vector of a measure of how specific the gene's expression is to the cell group based on the Jensen-Shannon divergence. The value ranges from 0 to 1.
> >- `pseudo_R2` - numeric vector of pseudo R-squared values, a measure of how well the gene expression model fits the categorical data relative to the null model. The value ranges from 0 to 1.
> >- `marker_test_p_value` - numeric vector of likelihood ratio p-values; p-value is an area in the tail of a distribution that tells you the odds of a result happening by chance
> >- `marker_test_q_value` -  numeric vector of likelihood ratio q-values; q-value is a p-value that has been adjusted for the False Discovery Rate (FDR)
> >
> {: .solution}
>
{: .question}

We can now use data in `marker_test` to rank the cells based on one of the specificity metrics and take the top gene(s) for each cluster. We will filter the expressing cells that constitute more than 10% of the cell group and we will take one gene in each cluster with the highest `pseudo_R2` value (you can of course modify this value and choose more genes to be selected).

```r
# filter the ‘marker_test’ data frame
top_specific_markers <- marker_test %>%
                            dplyr::filter(fraction_expressing >= 0.10) %>%       # set the fraction of expressing cells
                            dplyr::group_by(cell_group) %>%                      # set a group to which the cells belong
                            dplyr::top_n(1, pseudo_R2)                           # set the number of top genes and the variable from 'marker_test' to rank by
```
```r
# store the names of the marker genes
# you can also use IDs, the conversion to gene names should happen automatically when plotting
top_marker_names <- unique(top_specific_markers %>% dplyr::pull(gene_short_name))
```

Now we have all elements to plot the expression and fraction of cells that express found markers in each group.

```r
plot_genes_by_group(cds_clustered,                    # our CDS object
                    top_marker_names,                 # a list of gene names to show in the plot
                    group_cells_by="cluster",         # how to group cells when labeling
                    ordering_type="maximal_on_diag")  # how to order the genes / groups on the dot plot
```

> <tip-title>Unexpectedly too many genes on y-axis?</tip-title>
>
>  If you notice that on your dot plot any cluster has more genes than you specified (here we set one gene per cluster `top_n(1, pseudo_R2)`), go back to the code and create a new cell, right after assigning `top_specific_markers` object and just before `top_marker_names`. In the new cell, paste and execute the following:
>  
> ```r
> top_specific_markers <- top_specific_markers %>%
>                            dplyr::distinct(pseudo_R2, .keep_all=TRUE)     # select only one row if there are multiple rows with the same value in 'pseudo_R2' column
> ```
> Then you can execute `top_marker_names` again and see how the plot looks like now. Better, right?
> This problem may arise when several genes in one cluster have the same values of specific variable from 'marker_test' (in our case we chose `pseudo_R2`). It might likely happen in small and quite unsignificant clusters.
{: .tip}


![A dot plot showing the expression of genes and fraction of cells that express found markers in each group. On the diagonal there are two genes corresponding to each cluster with the highest pseudo_R2 score in their group.](../../images/scrna-casestudy-monocle/gene_group.png "A plot of the expression and fraction of cells that express found markers in each group. Here, the two top genes from each cluster were plotted on diagonal beased on the highest pseudo_R2 score. Shown is the result of changing the value from 1 to 2 in "top_n(1, pseudo_R2)" in the code above.")

Look at this – we have identified some more marker genes specific to each cluster! However, sometimes it happens that the found genes are not as specific as one would expect, and they appear across the whole sample. Therefore, it is a good idea to plot all those marker genes and check how they appear in the bigger picture.

```r
# plot the identified genes to check their expression
plot_cells(cds_clustered, genes=c('Pcdhgc4','Pcdhga5','Gm5559','Gm10359','Ccr9','Cd8b1','Plac8', 'Il2ra', 'Cd52', 'Tmsb10', 'Mki67', 'Hmgb2', 'Pclaf', 'Npm1'), reduction_method = "UMAP")
```

![Plots showing the expression of fourteen found marker genes across the whole sample. Those genes are 'Pcdhgc4','Pcdhga5','Gm5559','Gm10359','Ccr9','Cd8b1','Plac8', 'Il2ra', 'Cd52', 'Tmsb10', 'Mki67', 'Hmgb2', 'Pclaf', 'Npm1'.](../../images/scrna-casestudy-monocle/markers_plot.png "Plots of the expression of marker genes across the sample.")

Further steps from now would include reviewing literature and checking what cell types correspond to the genes expressed in each cluster. Then you can annotate your clusters in the same way as shown above. Once you have your clusters annotated, Monocle can generate a file of marker genes for the identified cell types. This file can be then used with [Garnett](https://cole-trapnell-lab.github.io/garnett/), a software toolkit for automatically annotating cells. We will not go through this in the current tutorial, but we will generate the file of marker genes.

```r
# use ‘top_markers()’ again, now grouping cells by the assigned cell type
assigned_marker_test <- top_markers(cds_annotated,
                                             group_cells_by="cell_type",
                                             reference_cells=1000,
                                             cores=8)
```
```r
# filter these markers according to how stringent you want to be
garnett_markers <- assigned_marker_test %>%
                        dplyr::filter(marker_test_q_value < 0.05 & specificity >= 0.25) %>%
                        dplyr::group_by(cell_group) %>%
                        dplyr::top_n(5, marker_score)
```
```r
# exclude genes that are good markers for more than one cell type:
garnett_markers_filtered <- garnett_markers %>%
                        dplyr::group_by(gene_short_name) %>%
                        dplyr::filter(dplyr::n() == 1)
```
```r
# generate a file of marker genes
generate_garnett_marker_file(garnett_markers_filtered, file="./marker_file.txt")
```

A new file should appear in the ‘Files’ window. If you click on it, you will see the cell types and their corresponding marker genes, satisfying your chosen conditions.

> <tip-title>Marker genes file</tip-title>
>
>  Note that you can use the above block of code to generate file with the marker genes for unannotated CDS object to help you identify and check specifically expressed genes – you’d only have to change `group_cells_by` parameter from "cell_type" to “cluster”.
{: .tip}


> <tip-title>Only for users of local RStudio</tip-title>
>
> If you are working in RStudio locally, you might want to try a great function `choose_cells()`, which allows you to make a subset of the CDS object containing only cells of interest to investigate certain clusters more in-depth. It only works in interactive mode, so can be only used locally - when you call it, then a pop-up window will appear where you can choose cells to subset.
>```r
># create a CDS subset
>cds_subset <- choose_cells(cds_clustered)
>```
>Now the chosen cluster is stored as a separate CDS object and you can analyse it independently, using the methods described above.
>
{: .tip}

# Trajectory inference
It’s time to perform trajectory analysis! First, let’s learn the trajectory graph. With an argument `use_partition` we can specify if we want to learn a disjoint graph (value TRUE - default) in each partition, or to learn a single graph (value FALSE) across all partitions. The thing is, we can visualise the cells in pseudotime only if they belong to one partition. This is why it is important to make sure that all the cells that you want to analyse in pseudotime belong to one partition.

In one of the previous sections we've already prepared the partitions for trajectory inference - we assigned all the cells of interest into one partition by changing the q-value. As a result, we got two partitions – one containing only the cells that we labeled as ‘Unknown’ and another including all other cells. That is a perfect assignment – we’d like to focus only on maturation of T-cells, so we don’t want to connect those two partitions (therefore we specify `use_partition=TRUE` to get a disjoint graph) and we will work downstream only on the graph going through annotated T-cells.

```r
# learn trajectory graph
cds_trajectory <- learn_graph(cds_annotated, use_partition=TRUE)
```
```r
# visualise the learned trajectory
plot_cells(cds_trajectory,
           color_cells_by = "cell_type",
           label_cell_groups=FALSE,
           label_groups_by_cluster=FALSE,
           label_leaves=FALSE,
           label_branch_points=FALSE)
```

![Learned trajectory graph connecting all the annotated clusters except the ‘Unknown’ group of cells.](../../images/scrna-casestudy-monocle/learned_trajectory_r.png "Learned trajectory graph.")

We have to tell Monocle where to start ordering the cells, ie. when we expect the analysed biological process to begin. Thanks to our biological knowledge, we know that the beginning of the trajectory should be at DN cluster.
There are a couple of ways to specify the root cells:

## Option 1: Root nodes
Here, you will use `root_pr_nodes` argument in `order_cells()` function.

To find the names of the principal points, you have to plot the learned trajectory again, specifying `label_principal_points = TRUE`
```r
# specifying root cells: `root_pr_nodes` argument - check the principal points
plot_cells(cds_trajectory,
          color_cells_by = "cell_type",
          label_cell_groups=FALSE,
          label_groups_by_cluster=FALSE,
          label_leaves=FALSE,
          label_branch_points=FALSE,
          label_principal_points = TRUE,       # set this to TRUE
          graph_label_size=3)
```
You can see now the principal points and their labels in the form `Y_number`. Pick the principal point in the cluster that you expect to be the beginning of the trajectory and type its name in the `root_pr_nodes` argument when calling `order_cells()` function.

```r
# specifying root cells: `root_pr_nodes` argument - use the relevant principal point
cds_order_1 <- order_cells(cds_trajectory, root_pr_nodes='Y_14')
```

There is also a helper function to identify the root principal points based on the annotated cell types. This function uses `pr_graph_cell_proj_closest_vertex` which is just a matrix with a single column that stores for each cell, the ID of the principal graph node it's closest to.

```r
# a helper function to identify the root principal points
get_correct_root_state <- function(cds, cell_phenotype, root_type){
      cell_ids <- which(pData(cds)[, cell_phenotype] == root_type)

      closest_vertex <- cds@principal_graph_aux[["UMAP"]]$pr_graph_cell_proj_closest_vertex
      closest_vertex <- as.matrix(closest_vertex[colnames(cds), ])
      root_pr_nodes <- igraph::V(principal_graph(cds)[["UMAP"]])$name[as.numeric(names
                                                                    (which.max(table(closest_vertex[cell_ids,]))))]       
      root_pr_nodes
      }
```
```r
# call the function to automatically find the node in the principal graph where our DN cells reside
DN_node_id = get_correct_root_state(cds_trajectory, cell_phenotype = 'cell_type', "DN")

DN_node_id      # check the node found
```
```r
# order cells using the helper function output
cds_order_1_helper <- order_cells(cds_trajectory, root_pr_nodes = DN_node_id)
```

## Option 2: Root cells
Here, you will use `root_cells` argument in `order_cells()` function.

Specify a vector of starting cell IDs. You can provide only one cell as well as all cells of a given type.

```r
# find the names of all cells belonging to a certain type, identified as a beginning of a trajectory
starting_cell_type <- 'DN'
index <- which(cds_trajectory@colData$cell_type == starting_cell_type)
DN_cells <- colnames(cds_trajectory)[index]
```
```r
# alternatively, if you work on unannotated data, you can use the number of the cluster that should be used as the beginning of the trajectory and pass it in the ‘root_cells’ argument
starting_cluster <- colnames(cds_trajectory[,clusters(cds_trajectory) == 4])
```
```r
# order cells
cds_order_2 <- order_cells(cds_trajectory, root_cells = DN_cells)
```

> <tip-title>Only for users of local RStudio</tip-title>
>
> If you are working in RStudio locally, you can use `order_cells()` function in the interactive mode. The pop-up window should appear and then you can simply select root nodes in the cluster with the cell type identified as the beginning of the trajectory.
>
> ```r
># specifying root cells: pop up window
>cds_order_1 <- order_cells(cds_trajectory)
>```
>
{: .tip}

# Plotting in pseudotime

You can use any `cds_order` object for the downstream analysis. Let’s pick one and assign it to an object with a shorter and more general name.
```r
cds_order <- cds_order_1_helper
```

The function `order_cells()` calculates pseudotime for each cell, so we can now visualise the T-cell maturation process in pseudotime! Additionally, we can access it and store in our CDS object f


topic: single-cell
tutorial: scatac-batch-correction-snapatac2
title: Multi-sample batch correction with Harmony and SnapATAC2
difficulty: intermediate
time_estimation: 4H
tools_json: []
requirements_json: ["type: internal", "scatac-preprocessing-tenx", "scatac-standard-processing-snapatac2"]
tags_json: ["10x", "epigenetics"]
content_hash: 2c28a2890cf0e158dce8f6c192c47fc7
last_modified: 2026-02-26T21:12:17.929626
zenodo_link: https://zenodo.org/records/12683310
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scatac-batch-correction-snapatac2/tutorial.html

content:
<!-- This is a comment. -->
Performing experiments in replicates is a cornerstone of modern biological science. However, when integrating data from multiple single-cell sequencing experiments, technical confounders might impact the results.
To reduce technical confounders, such as different experimenters, experimental protocols, sequencing lanes or sequencing technologies, batch correction is often beneficial.

In this tutorial, we will perform batch correction on five datasets of {scATAC-seq} data with the three algorithms *Harmony* ({% cite Korsunsky2019 %}), *Scanorama* ({% cite Hie2019 %}) and the *mutual nearest neighbor-based* ({% cite Haghverdi2018%}) algorithm *MNC-correct* ({% cite Zhang2024 %}). The {scATAC-seq} analysis will be performed with the tool suite [**SnapATAC2**](https://kzhang.org/SnapATAC2/version/2.5/index.html) ({% cite Zhang2024 %}).

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/tutorial_mode.md %}

> <comment-title></comment-title>
>
> This tutorial is significantly based on the ["Multi-sample Pipeline" tutorial](https://kzhang.org/SnapATAC2/version/2.5/tutorials/integration.html) from **SnapATAC2**.
>
> The data analysis is performed with the same tools shown in the tutorial [Single-cell ATAC-seq standard processing with SnapATAC2]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %} ).
> That tutorial also explains the steps of the ATAC-seq analysis with SnapATAC2 in more detail.
> We recommend completing that tutorial before continuing with batch correction using SnapATAC2.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data

In this tutorial we will analyze colon samples from multiple donors, provided by the [SnapATAC2 documentation](https://kzhang.org/SnapATAC2/version/2.5/tutorials/integration.html). The `chrom_sizes` file and the `gene_annotation` file are identical to the previous tutorial [Single-cell ATAC-seq standard processing with SnapATAC2]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %} ).
The five `colon_multisample` files have been generated by the **Cell Ranger ATAC 2.0.0** pipeline from 10X to generate a [*Fragments File*](https://support.10xgenomics.com/single-cell-atac/software/pipelines/latest/output/fragments).


> <details-title>Chromosome sizes </details-title>
>
> - A chromosome sizes file can be generated using the tool {% tool [Compute sequence length](toolshed.g2.bx.psu.edu/repos/devteam/fasta_compute_length/fasta_compute_length/1.0.3) %}.
> - The reference genome can either be selected from cached genomes or uploaded to the galaxy history.
>
{: .details}

First, we will import the datasets into Galaxy. Next, we will organize all `colon_multisample` datasets into a collection. Using collections helps streamline the workflow, making the subsequent analysis steps more efficient.

{% snippet faqs/galaxy/histories_datasets_vs_collections.md %}


## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library
>
>
>    ```
>    {{ page.zenodo_link }}/files/colon_multisample_01.gz
>    {{ page.zenodo_link }}/files/colon_multisample_02.gz
>    {{ page.zenodo_link }}/files/colon_multisample_03.gz
>    {{ page.zenodo_link }}/files/colon_multisample_04.gz
>    {{ page.zenodo_link }}/files/colon_multisample_05.gz
>    {{ page.zenodo_link }}/files/chrom_sizes.txt
>    {{ page.zenodo_link }}/files/gencode.v46.annotation.gtf.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <warning-title>Large file sizes!</warning-title>
>    > - The `colon_multisample` datasets are quite large. The entire tutorial requires approximately 50 GB of storage.
>    > - To reduce storage you can change the {% icon galaxy-gear %} **Upload Configuration** to *Defer dataset resolution* for the upload of the `colon_multisample` datasets.
>    > - This will delete the datasets after the first analysis step and helps in reducing storage.
>    > - You can also permanently delete datasets, which are no longer required.
>    >
>    >   {% snippet faqs/galaxy/datasets_deleting.md %}
>    >
>    {: .warning}
>
> 3. Rename the datasets
>    - {% icon galaxy-pencil %} **Rename** the file `gencode.v46.annotation.gtf.gz` to `gene_annotation.gtf.gz`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatypes of the `colon_multisample` files are set to `bed`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bed" %}
>
> 5. Create a dataset collection with all `colon_multisample` datasets and rename the collection to `Colon Multisample Fragments`.
>
>    {% snippet faqs/galaxy/collections_build_list.md name="Colon Multisample Fragments" %}
>
{: .hands_on}

# SnapATAC2 preprocessing and filtering

With our data imported and the collection assembled, we can now begin the {scATAC-seq} data preprocessing with SnapATAC2.

The first step is to import the datasets into an AnnData object using the *pp.import_data* tool. Next, the `{TSSe}` will be calculated, serving as a `{QC}` metric to selectively filter droplets containing high-quality cells.

> <details-title>AnnData format </details-title>
>
> - The [**AnnData**](https://anndata.readthedocs.io/en/latest/) format was initially developed for the [**Scanpy**](https://scanpy.readthedocs.io/en/stable/index.html) package and is now a widely accepted data format to store annotated data matrices in a space-efficient manner.
>
> ![Anndata format]({% link topics/single-cell/images/scatac-standard-snapatac2/anndata_schema.svg %} "AnnData format stores a count matrix `X` together with annotations of observations (i.e., cells) `obs`, variables (i.e., genes) `var`, and unstructured annotations `uns`.")
>
{: .details}

> <hands-on-title> Preprocessing and QC </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Import data fragment files and compute basic QC metrics, using 'pp.import_data'`
>        - {% icon param-collection %} *"Fragment file, optionally compressed with gzip or zstd"*: `Colon Multisample Fragments` (Input dataset collection)
>        - {% icon param-file %} *"A tabular file containing chromosome names and sizes"*: `chrom_sizes.txt` (Input dataset)
>        - *"Number of unique fragments threshold used to filter cells"*: `1000`
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute the TSS enrichment score (TSSe) for each cell, using 'metrics.tsse'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas` (dataset collection output of **pp.import_data** {% icon tool %})
>        - {% icon param-file %} *"GTF/GFF file containing the gene annotation"*: `gene_annotation` (Input dataset)
> 3. {% icon galaxy-pencil %} Rename the generated collection to `Colon Multisample AnnDatas TSSe`.
>
> 4. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the TSS enrichment vs. number of fragments density figure, using 'pl.tsse'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas TSSe` (output of **metrics.tsse** {% icon tool %})
> 5. {% icon galaxy-eye %} Inspect a few exemplary `.png` outputs of the collection
>
> ![TSSe plots against number of unique fragments]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/tsse-colon01-colon02.png %}"Examplary plots of TSSe from <code>colon_multisample_01</code> and <code>colon_multisample_02</code>")
>
> High-quality cells can be identified in the plot of {TSSe} scores against a number of unique fragments for each cell.
>
> > <question-title></question-title>
> >
> > 1. Where are high-quality cells typically located on a {TSSe} plot?
> > 2. Based on these plots, how should the filtering threshold be set?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. The cells in the upper right are high-quality cells, enriched for {TSS}. Fragments in the lower left represent low-quality cells or empty droplets and should be filtered out.
> > > 2. Setting the minimum {TSSe} to 7.0 will filter out the lowest quality droplets without loosing too much data.
> > >
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

## Filtering the count matrices

The {TSSe} threshold balances data retention and quality control. In our dataset, the {TSSe} distributions indicate substantial differences in sample quality between batches. To retain as much biological data as possible, we will use a broader filter (e.g., minimum {TSSe} = 7.0).

Standard {TSSe} Thresholds
- **Strict (TSSe ≥ 10–15)**: High-confidence cells.
- **Moderate (TSSe ≥ 7–10)**: Balances data retention and quality but may include lower-quality cells.
- **Broad (TSSe ≥ 5–7)**: Maximizes retention but increases the risk of including lower-quality data.

Choosing the Right Cutoff
- If sample quality varies, a **moderate threshold (≥7.0)** helps retain more data.
- If high-quality data is required, **stricter filtering (≥10.0)** is recommended.
- Inspect {TSSe} distributions to determine the most suitable threshold for your dataset.

> <hands-on-title> Filtering </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Filter cell outliers based on counts and numbers of genes expressed, using 'pp.filter_cells'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas TSSe` (output of **metrics.tsse** {% icon tool %})
>        - *"Minimum TSS enrichemnt score required for a cell to pass filtering"*: `7.0`
>
> 2. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Generate cell by bin count matrix, using 'pp.add_tile_matrix'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas filtered` (output of **pp.filter_cells** {% icon tool %})
>        - *"The size of consecutive genomic regions used to record the counts"*: `5000`
>
> 3. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Perform feature selection, using 'pp.select_features'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas tile_matrix` (output of **pp.add_tile_matrix** {% icon tool %})
>        - *"Number of features to keep"*: `50000`
>
>    > <details-title> Bin size and features </details-title>
>    >
>    > - *pp.add_tile_matrix* divides the genome into a specified number of bins, depending on the bin size (f.ex. 5000bp). For each bin, the ATAC-seq reads of individual cells are checked to determine if a read is located in the bin. This is counted as a feature and stored under `n_vars` in the AnnData object.
>    >    - Increasing the bin size greatly reduces compute time at the cost of some biological data.
>    >    - For this reason, the bin size 5000bp has been selected for the colon datasets. In the [previous tutorial]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#feature-selection), the lower bin size of 500bp was chosen, since fewer cells were analyzed.
>    > - *pp.select_features* uses the previously identified features to select the most accessible features for further analysis.
>    >    - The parameter *"Number of features to keep"* determines the upper limit of features which can be selected.
>    >    - Similarly to the *bin_size*, the *Number of Features to keep* can also impact downstream clustering. This was demonstrated in the [previous tutorial]( {% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#feature-selection ).
>    {: .details}
>
> 4. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Compute probability of being a doublet using the scrublet algorithm, using 'pp.scrublet'`
>        - {% icon param-collection %} *"Annotated data matrix"*: `Colon Multisample AnnDatas features` (output of **pp.select_features** {% icon tool %})
>
> 5. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Remove doublets according to the doublet probability or doublet score, using 'pp.filter_doublets'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Colon Multisample AnnDatas scrublet` (output of **pp.scrublet** {% icon tool %})
>
{: .hands_on}

# Concatenate the Collection
Before we can continue with the analysis and batch correction, we need to extract the datasets out of the collection and merge them into a single AnnData object.

## Extracting datasets from a collection
This is achieved in two steps:
1. The first dataset is extracted from the collection using *Extract dataset*
2. Afterwards, the first dataset will be removed from the collection. This is done by extracting element identifiers and filtering the collection with the name of the first dataset.

> <comment-title>  </comment-title>
> - It is also possible to manually remove a dataset from a collection.
> - However, manually removing datasets can not be implemented in Galaxy workflows. In contrast the automatic method, shown here, can be represented in a workflow and is therefore much better scalable.
{: .comment}

> <hands-on-title> Extract datasets </hands-on-title>
>
> 1. {% tool [Extract element from collection](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-collection %} *"Input List"*: `Colon Multisample AnnDatas filtered_doublets` (output of **pp.filter_doublets** {% icon tool %})
>    - *"How should a dataset be selected?"*: `The first dataset`
> 2. {% tool [Extract element identifiers](toolshed.g2.bx.psu.edu/repos/iuc/collection_element_identifiers/collection_element_identifiers/0.0.2) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Colon Multisample AnnDatas filtered_doublets` (output of **pp.filter_doublets** {% icon tool %})
>
> 3. {% tool [Select first](Show beginning1) %} with the following parameters:
>    - *"Select first"*: `1`
>    - {% icon param-file %} *"from"*: `Element identifiers` (output of **Extract element identifiers** {% icon tool %})
> 4. {% tool [Filter collection](__FILTER_FROM_FILE__) %} with the following parameters:
>    - {% icon param-collection %} *"Input Collection"*: `Colon Multisample AnnDatas filtered_doublets` (output of **pp.filter_doublets** {% icon tool %})
>    - *"How should the elements to remove be determined?"*: `Remove if identifiers are PRESENT in file`
>        - {% icon param-file %} *"Filter out identifiers present in"*: `select_first` (output of **Select first** {% icon tool %})
> 5. {% icon galaxy-pencil %} Rename the filtered collection to `Colon Multisample 02-05`
>
{: .hands_on}

## Concatenating all AnnData files

> <hands-on-title> Concatenate </hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `colon_multisample_01` (output of **Extract dataset** {% icon tool %})
>    - *"Function to manipulate the object"*: `Concatenate along the observations axis`
>        - {% icon param-collection %} *"Annotated data matrix to add"*: `Colon Multisample 02-05` (output of **Filter collection** {% icon tool %})
>        - *"Join method"*: `Intersection of variables`
>        - *"Key to add the batch annotation to obs"*: `batch`
>
>    > <details-title>Issues with concatenation</details-title>
>    > - The runtime of this operation depends on the dataset size. For `colon_multisample` datasets, concatenation may take up to 1 hour.
>    > - For larger datasets, the tool's allocated memory may be insufficient, causing the operation to fail. In that case, you will see the following error message:
>    >
>    >   ```
>    >                Fatal error: Exit code 137 ()
>    >   ```
>    > - In this case, report the issue so that administrators can increase the memory limit, allowing the job to complete successfully.
>    >
>    >   {% snippet faqs/galaxy/analysis_troubleshooting_reporting.md %}
>    >
>    {: .details}
>
> 2. {% icon galaxy-pencil %} Rename the AnnData output to `Multisample AnnData`
> 3. {% icon galaxy-eye %} Inspect the general information of `Multisample AnnData`
>
>    > <tip-title>Inspecting AnnData objects</tip-title>
>    > * Many toolsets producing outputs in *AnnData* formats in Galaxy, provide the general information by default:
>    >    * Click on the name of the dataset in the history to expand it. The general Anndata information will be given in the expanded box.
>    >    * Alternatively, expand the dataset and click on {% icon details %}*Dataset Details*. Scroll to Job Information and inspect the Tool Standard Output.
>    > * If a tool does not provide general AnnData information or a more specific query is needed, you can use {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %}.
>    {: .tip}
>
>    ```
>    AnnData object with n_obs × n_vars = 34372 × 606219
>     obs: 'n_fragment', 'frac_dup', 'frac_mito', 'tsse', 'doublet_probability', 'doublet_score', 'batch'
>     var: 'count-0', 'selected-0', 'count-1', 'selected-1', 'count-2', 'selected-2', 'count-3', 'selected-3', 'count-4', 'selected-4'
>     obsm: 'fragment_paired'
>    ```
>
> > <question-title></question-title>
> >
> > 1. How many colon cells are present in this AnnData object?
> > 2. What does the 'batch' annotation indicate?
> > 3. What do the 'count-' and 'selected-' annotations represent?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 34,372 cells.
> > > 2. The 'batch' annotation labels cells according to their sample number (0–4). This annotation will later be used by batch correction algorithms to generate clusters from multiple samples.
> > > 3. 'count' and 'selected' are variable annotations representing detected and selected features. The sample number (0–4) indicates the dataset that generated these annotations.
> > >
> > {: .solution}
>  {: .question}
>
{: .hands_on}

# Dimension Reduction
Now that all samples have been concatenated into a single AnnData object, the most accessible features of the combined count matrix must be selected. The previously chosen features were sample-specific and cannot be used for dimensionality reduction. Therefore, the most accessible features will be reselected, followed by dimensionality reduction using *matrix-free spectral embedding*.


> <hands-on-title> Spectral Embedding </hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Perform feature selection, using 'pp.select_features'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData` (output of **Manipulate AnnData** {% icon tool %})
>        - *"Number of features to keep"*: `50000`
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and clustering"*: `Perform dimensionality reduction using Laplacian Eigenmap ('tl.spectral')`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData features` (output of **pp.select_features** {% icon tool %})
>        - *"Distance metric"*: `cosine`
>
> 3. {% icon galaxy-pencil %} Rename the output AnnData object to `Multisample AnnData spectral`
>
{: .hands_on}


## Control Without Batch Correction
Batch effects can be visualized using a `{UMAP}` projection, where samples are colored by their batch annotation (`obs: 'batch'`).
> <hands-on-title> UMAP Projection Without Batch Correction </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute UMAP using 'tl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData spectral` (output of **tl.spectral** {% icon tool %})
>
> 2. {% icon galaxy-pencil %} Rename the output AnnData object to `Multisample AnnData UMAP`.
>
> 3. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData UMAP` (output of **tl.umap** {% icon tool %})
>        - *"Color"*: `batch`
>        - *"Height of the plot"*: `500`
>
> 4. {% icon galaxy-pencil %} Rename the generated image to `spectral-UMAP-No Batch Correction`.
>
> 5. {% icon galaxy-eye %} Inspect the `.png` output.
>
> ![UMAP plot before batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/batch-umap-no_harmony-spectral.png %})
>
> > <question-title></question-title>
> >
> > 1. How do batch effects appear in this plot?
> > 2. What would a plot without batch effects look like?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. Batch effects appear as distinct clusters where samples are unevenly distributed. For example, in the upper right corner, each batch forms a separate group. These differences are likely technical artifacts rather than true biological variation.
> > > 2. A plot without batch effects would show cells from all samples evenly mixed, with no distinct clusters corresponding to individual batches.
> > >
> > {: .solution}
>  {: .question}
>
{: .hands_on}

# Batch correction
After confirming the presence of batch effects, they should be removed using correction algorithms. **SnapATAC2** provides three batch correction methods: *Harmony*, *MNC-correct*, and *Scanorama*. To determine the most suitable algorithm for your dataset, compare the outputs of each method.

> <details-title> Batch correction algorithms </details-title>
> - Batch correction algorithms adjust the cell-by-feature count matrix to account for batch-specific differences between samples.
> - Batch effects can arise from various technical sources, such as sequencing lanes, plates, protocols, and handling. Additionally, biological factors, including tissue types, species, and inter-individual variation, can also contribute to batch effects {% cite Luecken2021 %}.
> - Many batch correction algorithms have been developed, with most scATAC-seq methods adapted from scRNA-seq batch removal techniques.
> - *Harmony* {% cite Korsunsky2019 %} is a principal component analysis (PCA)-based method that leverages lower-dimensional data to assign cells to new clusters. It prioritizes multi-sample clustering to integrate datasets by computing linear correction factors for each batch and cluster, iteratively adjusting cell positions until optimal batch correction is achieved.
>
>   ![Graphical abstract of Harmony batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/harmony-graphical-abstract.png %})
>
> - *Scanorama* {% cite Hie2019 %} performs panorama stiching, to find and merge overlapping cell types.
>
>   ![Graphical abstract of Scanorama batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/scanorama-graphical-abstract.png %})
>
> - *MNC-correct* {% cite Zhang2024 %} is a modified version of the *mutual nearest neighbor* algorithm {% cite Haghverdi2018 %}. It calculates centroids for batch-specific clusters and identifies pairs of mutual nearest centroids (MNC) across batches. Correction vectors then align the batches in the same plane. Additionally, *MNC-correct* can be run iteratively to refine corrections for optimal alignment.
>
>   ![Graphical abstract of MNC-correct batch correction]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/mnn-graphical-abstract.png %})
>
{: .details}

We will first use *Harmony* to correct batch effects, followed by testing the other methods.
> <hands-on-title> Batch correction and visualization</hands-on-title>
>
> 1. {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for preprocessing"*: `Use harmonypy to integrate different experiments,using 'pp.harmony'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData UMAP` (output of **tl.umap** {% icon tool %})
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute Umap, using 'tl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony` (output of **pp.harmony** {% icon tool %})
>        - *"Use the indicated representation in `.obsm`"*: `X_spectral_harmony`
>        - *"`adata.obs` key under which t add cluster labels"*: `umap_harmony`
>
> 2. {% icon galaxy-pencil %} Rename the AnnData output to `Multisample AnnData harmony UMAP`
>
>    > <comment-title> Key for Cluster Labels </comment-title>
>    > - Adding the new *UMAP embeddings* under the key `umap_harmony` preserves the non-batch-corrected embeddings in the AnnData object.
>    >   - Alternatively, leaving this parameter empty will overwrite the existing embeddings.
>    {: .comment}
>
> 4. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding, using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony UMAP` (output of **tl.umap** {% icon tool %})
>        - *"Color"*: `batch`
>        - *"Use the indicated representation in .obsm"*: `X_umap_harmony`
>        - *"Height of the plot"*: `500`
>
> 5. {% icon galaxy-pencil %} Rename the generated image to `spectral-UMAP-harmony`
>
> 6. {% icon galaxy-eye %} Inspect the `.png` output
>
>  ![UMAP plot of Batch correction with Harmony]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/batch-umap-harmony-spectral.png %})
>
> > <question-title></question-title>
> >
> > 1. How did *Harmony* affect the appearance of this plot?
> > 2. Are there areas in the plot where batch effects persist?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. *Harmony* has successfully removed most batch effects. Notably, the center-left groups have merged into a single larger group containing all batches.
> > > 2. Yes, some batch effects remain, particularly in the upper-right corner, where batch-specific colors do not fully overlap. However, this could also be due to certain samples having fewer cells, leading to underrepresentation in clusters of rare cell types. Therefore, it is unclear whether *Harmony* has completely eliminated all batch effects.
> > >
> > {: .solution}
>  {: .question}
>
>
{: .hands_on}

> <details-title>Batch Correction with Scanorama and MNC-correct</details-title>
> - Other batch correction methods can be applied using {% tool [SnapATAC2 Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_preprocessing/snapatac2_preprocessing/2.6.4+galaxy1) %}, including *pp.mnc_correct* and *pp.scanorama_integrate*.
>   - For *MNC-correct*, the number of iterations can be adjusted.
> - To determine the optimal batch correction method, it is recommended to test different algorithms and parameter settings.
> - Example outputs from *Scanorama* and *MNC-correct* are shown below:
>
>  ![Batch correction UMAP plots of MNC-correct with different settings and Scanorama]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/mnc-correct-scanorama-umap-spectral.png %} "UMAP plots of batch correction with different methods. (a) Batch correction with MNC-correct and default settings. (b) Batch correction with Scanorama. (c) Batch correction with MNC-correct and 30 iterations.")
>
> > <question-title></question-title>
> >
> > - Compare these plots with the output of *Harmony*. Which algorithm is best-suited for the colon datasets?
> >
> > > <solution-title></solution-title>
> > >
> > > - *Harmony* achieved the best batch integration, as it produced the fewest single-batch groups. *Scanorama* and *MNC-correct* (with default settings) did not integrate the batches as effectively. However, *MNC-correct* with 30 iterations significantly reduced batch effects and could also be used for further analysis.
> > >
> > {: .solution}
>  {: .question}
>
{: .details}


# Clustering of the batch corrected samples
The analysis can now continue using the same methods outlined in the [standard pathway]({% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}). The batch-corrected embeddings are clustered and visualized using the *Leiden* algorithm.

> <hands-on-title> Leiden clustering and visualization </hands-on-title>
>
> 1. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Compute a neighborhood graph of observations, using 'pp.knn'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony UMAP` (output of **tl.umap** {% icon tool %})
>        - *"The key for the matrix"*: `X_spectral_harmony`
>
>    > <tip-title>Matrix Keys</tip-title>
>    >
>    > - Each batch correction algorithm stores its corrected matrix under a specific key:
>    >   - *Harmony*: `X_spectral_harmony`
>    >   - *MNC-correct*: `X_spectral_mnn`
>    >   - *Scanorama*: `X_spectral_scanorama`
>    > - These keys are stored in the AnnData object under `'obsm'`.
>    >
>    {: .tip}
>
> 2. {% tool [SnapATAC2 Clustering](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_clustering/snapatac2_clustering/2.6.4+galaxy1) %} with the following parameters:
>    - *"Dimension reduction and Clustering"*: `Cluster cells into subgroups, using 'tl.leiden'`
>        - {% icon param-file %} *"Annotated data matrix"*: `Multisample AnnData harmony knn` (output of **pp.knn** {% icon tool %})
>        - *"Whether to use the Constant Potts Model (CPM) or modularity"*: `modularity`
>
> 3. {% icon galaxy-pencil %} Rename the AnnData output to `Multisample AnnData harmony leiden`
>
> 4. {% tool [SnapATAC2 Plotting](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_plotting/snapatac2_plotting/2.6.4+galaxy1) %} with the following parameters:
>    - *"Method used for plotting"*: `Plot the UMAP embedding, using 'pl.umap'`
>        - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **tl.leiden** {% icon tool %})
>        - *"Color"*: `leiden`
>        - *"Use the indicated representation in .obsm"*: `X_umap_harmony`
>        - *"Height of the plot"*: `500`
>
> 5. {% icon galaxy-pencil %} Rename the generated image to `spectral-UMAP-harmony-leiden`
>
> 6. {% icon galaxy-eye %} Inspect the `.png` output
>
>  ![UMAP plot of Batch-corrected leiden clusters]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/batch-umap-harmony-leiden.png %})
>
{: .hands_on}

After integrating the datasets and clustering the cells, the scATAC-seq analysis can proceed with downstream analysis. This includes [cell cluster annotation]({% link topics/single-cell/tutorials/scatac-standard-processing-snapatac2/tutorial.md %}#cell-cluster-annotation) and differential peak analysis.
> <comment-title></comment-title>
>
> The **SnapATAC2** tools for differential peak analysis are available in Galaxy, but no GTN training materials are currently provided. Until a tutorial is available, you can refer to the **SnapATAC2** documentation for a [tutorial on differential peak analysis](https://kzhang.org/SnapATAC2/version/2.6/tutorials/diff.html).
>
> The tools are available in Galaxy under {% tool [SnapATAC2 Peaks and Motif](toolshed.g2.bx.psu.edu/repos/iuc/snapatac2_peaks_and_motif/snapatac2_peaks_and_motif/2.6.4+galaxy1) %}.
>
{: .comment}


# Conclusion
{% icon congratulations %} Well done, you’ve made it to the end! You might want to consult your results with this [control history](https://usegalaxy.eu/u/timonschlegel/h/multisample-batch-correction-with-harmony-and-snapatac2), or check out the [full workflow](https://usegalaxy.eu/u/timonschlegel/w/multisample-batch-correction-with-snapatac2-and-harmony) for this tutorial.

In this tutorial, we integrated five {scATAC-seq} colon samples using a scalable Galaxy workflow. We compared different batch integration algorithms to identify the most suitable method for our data. Finally, we clustered the cells to prepare the data for downstream analysis.

![SnapATAC2 batch correction pipeline]({% link topics/single-cell/images/scatac-batch-correction-snapatac2/Batch-correction-pipeline-overview.png %})


topic: single-cell
tutorial: GO-enrichment
title: GO Enrichment Analysis on Single-Cell RNA-Seq Data
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["single cell", "GO enrichment"]
content_hash: 95e63b93df9d4805e9fa26ab3c6df33d
last_modified: 2026-02-26T21:12:17.895105
zenodo_link: https://zenodo.org/records/13461890
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/GO-enrichment/tutorial.html

content:
In the tutorial [Filter, plot and explore single-cell RNA-seq data with Scanpy]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}), we took an important step in our single-cell RNA sequencing analysis by identifying marker genes for each of the clusters in our dataset. These marker genes are crucial, as they help us distinguish between different cell types and states, giving us a clearer picture of the cellular diversity within our samples.
However, simply identifying marker genes is just the beginning. To truly understand what makes each cluster unique, we need to dive deeper into the biological functions these genes are involved in. This is where Gene Ontology (GO) enrichment analysis comes into play.
We will perform GO enrichment analysis as a type of over-representation analysis (ORA). ORA is a statistical method that determines whether genes from pre-defined sets (e.g. genes belonging to a specific GO term) are expressed more than would be expected in a subset of your data. The most commonly used statistical tests are Fischer's exact test and hypergeometric test, more details about them are explained in the [tutorial slides]({% link topics/single-cell/tutorials/GO-enrichment/slides.html %}).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data description

In this tutorial will use the following datasets:

## [A] Marker Genes:

We'll start with two input datasets of marker genes (Study sets):
  * **Marker genes per cell cluster:** This dataset lists the genes that are differentially enriched in each cell cluster.
  * **Marker genes per condition (wt and ko):** This dataset lists the genes that are differentially enriched between the wild-type (wt) and knockout (ko) conditions.


*Note:* Marker genes were obtained using Scanpy FindMarkers tool. The top 50 marker genes were included in the downstream GO enrichment analysis. Scanpy FindMarkers tool selects the marker genes based on their log2 fold change and p-values. Focusing on the top-ranked genes helps to filter out less relevant genes, thereby addressing the concern of high false positives that can arise from traditional methods.


## [B] GO Enrichment Files:

We'll also use three additional files for GO enrichment analysis.
  * **Gene Ontology file:** This file contains information about Gene Ontology terms.
  * **GO Annotations file:** This file maps genes to their corresponding GO terms.
  * **Population set file:** This file provides a list of all genes involved in the experiment and is used as a background gene set for the analysis.

*Note:* There are several online databases available for downloading GO and GO Annotations files, including the Gene Ontology website, Ensembl, and the UCSC Genome Browser.

>  <comment-title>Concept behind GO Enrichment Analysis</comment-title>
>
>  The goal of GO enrichment analysis is to interpret the biological significance of long lists of **marker genes** by summarizing these genes into a shorter list of enriched **GO terms**. The analysis works by comparing each GO term between your list of marker genes and a **background gene set**. Statistical tests are then used to calculate a p-value that indicates whether a particular GO term is significantly enriched in the marker gene list compared to the background.
{: .comment}

# Get data

You can access the data for this tutorial in multiple ways:

1. **Importing from a history** - You can import [this history](https://usegalaxy.eu/u/mennagamal/h/go-enrichment-analysis-on-scrna-seq-data)

   {% snippet faqs/galaxy/histories_import.md %}

2. **Uploading from Zenodo** (see below)

> <hands-on-title> Data Upload from Zenodo </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Galaxy3-[GO].obo
>    {{ page.zenodo_link }}/files/Galaxy2-[GO_annotations_Mus_musculus].tabular
>    {{ page.zenodo_link }}/files/Galaxy5-[Markers_-_clusters].tabular
>    {{ page.zenodo_link }}/files/Galaxy4-[Background_gene_set].tabular
>    {{ page.zenodo_link }}/files/Galaxy1-[Markers_-_genotype_].tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

## Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}


# Data processing

To perform GO enrichment analysis on each cell cluster individually, we need to separate our "Markers_clusters Dataset" into seven files, one for each cluster and the "Markers_genotype Dataset" into 2 files, one for each condition. We'll use the "Split file" tool for this step.

> <hands-on-title>File splitting</hands-on-title>
>
> 1. {% tool [Split file](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.4) %} with the following parameters:
>    - {% icon param-file %} *"File to select"*: `Markers_cluster` (Input dataset)
>    - *"on column"*: `c1`
>    - *"Include the header in all splitted files?"*: `Yes`
>
>
>    > <comment-title> Input Dataset </comment-title>
>    >
>    > As we have two datasets, one with the marker genes for all the seven clusters and one with the marker genes for the knockout (KO) and wild-type (WT) conditions. Make sure to repeat the analysis twice for the two different datasets. Alternatively, you can run this [workflow](https://usegalaxy.eu/u/mennagamal/w/unnamed-workflow) for parallel analysis of the datasets, under Marker genes choose the second icon to select multiple datasets as shown in the below image.
>    {: .comment}
> ![Multiple input datasets workflow](../../images/workflow.png "Multiple input datasets workflow")
>
{: .hands_on}

Next, we need to isolate the Ensembl gene IDs column from each file. We'll use the "Cut Columns" tool to achieve this.

> <hands-on-title> Extract Ensembl IDs </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c4`
>    - {% icon param-file %} *"From"*: `split_output` (output of **Split file** {% icon tool %})
>
>
>    > <comment-title> The gene format to use </comment-title>
>    >
>    > In this example we extract column 4 because it contains the Ensembl gene IDs on which the subsequent steps are ideally working. While there are other gene formats like gene symbols, Entrez gene IDs, and more, make sure to check the specific format accepted by the tool you are using. There are also tools available to convert between different gene formats if needed.
>    {: .comment}
>
{: .hands_on}


# GO Analysis using **GOEnrichment** tool

Now we will perform the GO Enrichment analysis on the list of ensembl gene IDs.

> <hands-on-title> GOEnrichment </hands-on-title>
>
> 1. {% tool [GOEnrichment](toolshed.g2.bx.psu.edu/repos/iuc/goenrichment/goenrichment/2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO` (Input dataset)
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus` (Input dataset)
>    - {% icon param-file %} *"Study Set File"*: `out_file1` (output of **Cut** {% icon tool %})
>    - {% icon param-file %} *"Population Set File (Optional)"*: `Background gene set` (Input dataset)
>
>
>    > <comment-title> Population Set File Selection for GO Enrichment </comment-title>
>    >
>    > When choosing a background for GO enrichment analysis (Population Set File), it's important to consider the context of your data. While using a broad background (like all genes in the organism) is common, it might be more informative to limit the background to genes expressed in the specific tissue or cell type being profiled. In this tutorial we used only genes involved in the experiment before selecting the marker genes.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Take a look at the enriched terms for the different clusters, Can you find any GO terms that are specific to cluster 7?
> 2. Can we perform manual annotation of cluster 7 based on GO enrichment results?
>
> > <solution-title></solution-title>
> >
> > 1. Cluster 7 is enriched for terms like "regulation of cell death", "T cell-mediated cytotoxicity", and "peptidase activator activity involved in the apoptotic process".
> > 2. By looking at the most enriched functions and using our biological knowledge, we can figure out the cell types for many clusters. For example, since the data comes from thymus tissue, we already have an idea of the cell types we might find. The enriched terms in cluster 7 confirm that the cell type is macrophages, which support thymocyte maturation by cleaning up dead cells and debris.
> >
> {: .solution}
>
{: .question}

# GO Analysis using **gProfiler GOSt** tool

The gProfiler GOSt (Gene Ontology Sequential Testing) is another popular tool used to perform gene ontology (GO) enrichment analysis. In addition to providing enrichment results for the standard GO categories of Biological Process (BP), Cellular Component (CC), and Molecular Function (MF), the tool also analyzes enrichment across several other functional annotation databases, including KEGG Pathways, Reactome Pathways, WikiPathways and TF Targets. It also gives a plot to better visualize the results.

> <hands-on-title> gProfiler GOSt </hands-on-title>
>
> 1. {% tool [gProfiler GOSt](toolshed.g2.bx.psu.edu/repos/iuc/gprofiler_gost/gprofiler_gost/0.1.7+galaxy11) %} with the following parameters:
>    - {% icon param-file %} *"Input is whitespace-separated list of genes, proteins, probes, term IDs or chromosomal regions."*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Organism"*: `Common organisms`
>        - *"Common organisms"*: `Mus musculus (Mouse)`
>    - In *"Tool settings"*:
>        - *"Export plot"*: `Yes`
>
>
>    > <comment-title> Picking the right species matter </comment-title>
>    >
>    > The species you select should match the species your genes come from. If you choose the wrong species, the tool might use incorrect information, leading to inaccurate results. For example, human genes behave differently from mouse genes, so selecting the correct species ensures the analysis is relevant to your data.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Can you find enriched GO terms that are inline with the [published study](https://www.frontiersin.org/journals/immunology/articles/10.3389/fimmu.2018.02523/full) findings in KO results file?
> 2. What might be happening to the stem cells in the KO mice compared to the WT mice?
>
> > <solution-title></solution-title>
> >
> > 1. In the KO g:GOSt result file, enrichment for the GO term "Negative regulation of stem cell differentiation" is found.
> > 2. This suggests that the KO condition is causing a delay in the differentiation of stem cells into mature T cells in the thymus which is inline with the study findings.
> >
> {: .solution}
>
{: .question}

# Conclusion

In this tutorial, we have performed GO enrichment analysis on the differentially expressed genes between 2 different conditions and between different cell types. This analysis provided valuable insights into the biological processes, molecular functions, and cellular components associated with the gene sets, enhancing our understanding of the underlying mechanisms involved in the studied conditions.


topic: single-cell
tutorial: pseudobulk-analysis
title: Pseudobulk Analysis with Decoupler and EdgeR
difficulty: intermediate
time_estimation: 4H
tools_json: []
requirements_json: ["type: internal", "scrna-scanpy-pbmc3k"]
tags_json: ["transcriptomics", "pseudobulk"]
content_hash: b19226b904c1c85290f414bcd1d3edca
last_modified: 2026-02-26T21:12:17.927898
zenodo_link: https://zenodo.org/records/15275834
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/pseudobulk-analysis/tutorial.html

content:
Pseudobulk analysis is a powerful technique that bridges the gap between single-cell and bulk RNA-seq data. It involves aggregating gene expression data from groups of cells within the same biological replicate, such as a mouse or patient, typically based on clustering or cell type annotations ({% cite Murphy2022 %}).

A key advantage of this approach in differential expression (DE) analysis is that it avoids treating individual cells as independent samples, which can underestimate variance and lead to inflated significance or overly optimistic p-values ({% cite Squair2021 %}). This occurs because cells from the same biological replicate are inherently more similar to each other than cells from different samples. By grouping data into pseudobulk samples, the analysis aligns with the experimental design, as in bulk RNA-seq, leading to more reliable and robust statistical results ({% cite Murphy2022 %}).

Beyond enhancing statistical validity, pseudobulk analysis enables the identification of cell-type-specific gene expression and functional changes across biological conditions. It balances the detailed resolution of single-cell data with the statistical power of bulk RNA-seq, providing insights into the functional transcriptomic landscape relevant to biological questions. Overall, for DE analysis in multi-sample single-cell experiments, pseudobulk approaches demonstrate superior performance compared to single-cell-specific DE methods ({% cite Squair2021 %}).

In this tutorial, we will guide you through a pseudobulk analysis workflow using the **Decoupler** ({% cite Badia-iMompel2022 %}) and **edgeR** ({% cite Liu2015 %}) tools available in Galaxy. These tools facilitate functional and differential expression analysis, and their output can be integrated with other Galaxy tools to visualize results, such as creating Volcano Plots, which we will also cover in this tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get the Data!

## Overview of the Data

The dataset used in this tutorial is an `AnnData` object containing a subset of single-cell RNA-seq data, specifically including non-classical monocytes (NCMs) and plasmacytoid dendritic cells (pDCs) from bone marrow and peripheral blood samples. This subset was selected to illustrate differential gene expression between two immune cell types across two biologically distinct tissues. A portion of the data corresponding to bone marrow was derived from a published study {% cite rettkowski2025modulation %}.

In this tutorial, we will focus on these two cell populations and explore how their expression profiles differ between bone marrow and blood using pseudobulk analysis. The dataset includes three bone marrow samples and seven blood samples.

Pseudobulk analysis is an advanced approach in single-cell data analysis. It involves aggregating single-cell expression data by group (e.g., by cell type and condition) to perform bulk-style analyses while preserving single-cell resolution through metadata.

For this tutorial, we assume familiarity with standard single-cell data formats, such as `AnnData` or `Seurat` objects, and prior experience with single-cell workflows, particularly clustering and cell type annotation. We will use the `decoupler` tool to perform the pseudobulk aggregation and downstream differential analysis {% cite decoupler-pseudobulk %}.

If you're new to these concepts, we recommend exploring the following tutorials before performing pseudobulk analysis:

- [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}): Learn how to perform clustering and cell type annotation using Galaxy's Scanpy and Anndata tool suites.
- [Clustering 3K PBMCs with Seurat]({% link topics/single-cell/tutorials/scrna-seurat-pbmc3k/tutorial.md %}): Explore an alternative approach to clustering and annotation using Seurat within Galaxy.
- [Combining single-cell datasets after pre-processing]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md %}): Learn how to merge multiple single-cell datasets into one AnnData object and enrich it with experimental metadata.

The data object, which you will import from Zenodo into Galaxy via the provided link, has been preprocessed, analysed, and annotated. It includes the following key observation metadata:
- **annotated**: The assigned cell type.
- **condition**: The experimental condition or sample group.
- **batch**: The sample identifier.
- **tissue**: Indicates the tissue of origin (bone marrow or peripheral blood).

## Data Upload

> <hands-on-title>Upload your data</hands-on-title>
>
> 1. Create a new history for this tutorial and name it "Pseudobulk DE Analysis with edgeR"
> 2. Import the AnnData file from [Zenodo]({{page.zenodo_link}}):
>
>    ```
>    {{ page.zenodo_link }}/files/ncm_pdcs_subset.h5ad
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the dataset: "**AnnData for Pseudobulk**."
> 4. Ensure that the datatype is correct. It should be an AnnData object (`h5ad`).
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

Before we dive into the analysis, lets inspect our `AnnData file` with tools in galaxy to get familiar with the structure of the data.

> <hands-on-title> Inspect AnnData Object </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} tool with the following parameters:
>     - {% icon param-file %} **Annotated data matrix**: `AnnData for Pseudobulk` (Input dataset obtained > from Zenodo)`
>     - *"What to inspect?"*: `General information about the object`
>       
>    > <question-title></question-title>
>    > 1. How many cells and genes are there in the `AnnData`? 
>    > 2. What other information can be retrive by the `Inspect AnnData` tool? 
>    >
>    > > <solution-title></solution-title>
>    > > 1.  
>    > > ```
>    > > [n_obs x n_vars]
>    > > 9118 × 18926
>    > > ```
>    > > There are 9,118 observations, representing cells, and 18,926 variables, representing genes expressed in total.
>    > > 2. When selecting **General Inspection**, you can view the labels for all entries in `[obs]`, `[var]`, `[obsm]`, `[varm]`, and `[uns]`. This includes the labels described above within the observation field, such as `annotated`, `condition`, `batch`, and `tissue`, which will be needed as inputs for the pseudobulk tool.
>    > > To view more specific details in the AnnData object, select a different parameter under *"What to inspect?"*.
>    > {: .solution}
>    > 
>    {: .question}
>
{: .hands_on}

# Generation of the Pseudobulk Count Matrix with Decoupler

In this step, our goal is to perform a "bioinformatic cell sorting" based on the annotated clusters of the single-cell data.

To start a pseudobulk analysis, ensure that the `AnnData` object contains all the necessary metadata for ***pseudobulking***. This includes key annotations such as `annotated` or `cell_type`, `condition`, `disease`, and `batch`. These labels may vary depending on how the dataset was originally annotated.

Most importantly, the `AnnData` object must include a layer containing the raw gene expression counts. This layer is essential for accurate aggregation and downstream differential expression analysis. Raw counts are allows to generate accurate pseudobulk aggregates. Since single-cell data is typically normalized after annotation, it’s important to preserve the raw counts in the AnnData object before normalization steps if you would like to perform a pseudobulk analysis later on. These raw counts are directly used by tools like **Decoupler** to generate the pseudobulk count matrix. Note that normalized count matrices should not be used with **Decoupler**, even if the tool appears to process them successfully.

> <tip-title> Missing Raw Counts? </tip-title>
>
> If your AnnData object lacks raw counts, you can use the **[AnnData Operations](https://toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0)** tool on the **[Single Cell Galaxy instance](https://singlecell.usegalaxy.eu)**. This tool allows you to copy the `.X` matrix into a new layer that you may assign a new label, for e.g., `counts` or `raw_counts`, making it available as a parameter for Decoupler.
>
> Importantly, ensure that the copying of this matrix as a raw counts layer is done carefully and correctly. To verify that your AnnData object contains the necessary raw counts layer, you can use the **[Inspect AnnData Object](https://toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0)** tool. This tool helps confirm the presence of raw counts and other essential metadata in your AnnData object before proceeding with pseudobulk analysis.
>
{: .tip}

> <hands-on-title> Decoupler Pseudobulk </hands-on-title>
>
> 1. {% tool [Decoupler pseudo-bulk](toolshed.g2.bx.psu.edu/repos/ebi-gxa/decoupler_pseudobulk/decoupler_pseudobulk/1.4.0+galaxy8) %} tool with the following parameters:
>     - {% icon param-file %} **Input AnnData file**: `AnnData for Pseudobulk` (Input dataset obtained > from Zenodo)
>     - *"Produce a list of genes to filter out per contrast?"*: `No`
>     - *"Obs Fields to Merge"*: *(Leave empty if not applicable)*
>     - *"Groupby column"*: `annotated` (Column containing cell type annotations)
>     - *"Sample Key column"*: `batch` (Column containing individual sample identifiers)
>     - *"Layer"*: `raw_counts` (Layer containing raw gene expression counts)
>     - *"Factor Fields"*: `tissue` (Column in `adata.obs` specifying experimental factors. For edgeR, the first field should be the main contrast field, followed by  covariates.)
>     - *"Use Raw"*: `No`
>     - *"Produce AnnData with Pseudo-bulk"*: *(Optional, if yes, it will generate an h5ad output file with pseudobulks)*
>     - *"Minimum Cells:*: `10`
>     - *"Produce plots:*: `Yes`
>     - *"Minimum Counts"*: `10`
>     - *"Minimum Total Counts"*: `1000`
>     - *"Minimum Counts Per Gene Per Contrast Field"*: `20` (Genes with fewer counts in specific contrasts are flagged in a separate file but not excluded from the results.)
>     - *"Enable Filtering by Expression"*: `Yes`
>     - *"Plot Samples Figsize"*: `13 13`
>     - *"Plot Filtering Figsize"*: `13 13`
>
>    > <comment-title> Performing DEG within Clusters </comment-title>
>    >
>    > **Important!** The count matrix retrieved from this tool includes all of our samples `batch` aggregated by `annotated`, which can be identified by the column headers. If you want to perform comparisons of conditions within clusters of each individual cell type, you will need to subset the relevant columns of the matrix and use them as your new count matrix. We will demonstrate this in the last section of this tutorial with detailed hands-on steps.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many outputs does the Decoupler tool generate?
> 2. What do the outputs represent? What is the interpretation of the plots?
> 3. Which output(s) will we use in edgeR for differential expression analysis?
>
> > <solution-title></solution-title>
> >
> > 1. The Decoupler tool generates multiple outputs, including:
> >    - **Pseudobulk Count Matrix** (tabular file)
> >    - **Samples Metadata (Factor file)** (tabular file)
> >    - **Genes Metadata** (tabular file)
> >    - **Pseudobulk Plot** (PNG format)
> >    - **Filter by Expression Plot** (PNG format)
> >    - **Genes to Ignore by Contrast Field** (tabular file)
> >    - **Pseudobulk AnnData file** (If chosen, h5ad file)
> >
> > 2. The output files contain the following:
> >    - **Pseudobulk Count Matrix:** Contains the raw count aggregates for each pseudobulk sample.
> >    - **Samples Metadata (Factor File):** Provides metadata annotated for each sample, including factors or annotations added to the AnnData object.
> >    - **Genes Metadata:** Includes gene-related information such as gene symbols, Ensembl IDs, dispersion values, etc.
> >    - **Genes to Ignore:** Lists of genes that could be excluded or should be carefully considered for specific contrasts.
> >      This file contains a contrast field and the corresponding genes written as gene symbols.
> >    - **Pseudobulk Plot:** A visual representation of the pseudobulk data.
> >      ![Pseudobulk Plot Example](../../images/pseudobulk-analysis/Decoupler_Pseudobulk_Plot.png)
> >    - **Filter by Expression Plot:** Illustrates the expression filtering applied to the data.
> >      ![Filter by Expression Plot Example](../../images/pseudobulk-analysis/Decoupler_Filter_by_Expression_Plot.png)
> >    - **Pseudobulk AnnData file:** An AnnData file that contains the aggregated pseudobulks.
> >
> > 3. The **pseudobulk count matrix** is the primary input required for analysis using **edgeR**, a tool designed for differential expression analysis.
> >    The **Samples Metadata** is another file that will serve as an input for the edgeR tool.
> >
> {: .solution}
>
{: .question}

## Sanitation Steps - Part 1

The next steps will help you refine your data for easier handling. We will use standard galaxy tools, like: [Replace Text](https://toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1), [Remove columns](https://toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) and [Text reformatting](https://toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1).

> <hands-on-title>Replace Text and Remove Columns</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `count_matrix` (output of **Decoupler pseudo-bulk** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `[ --+*^]+`
>            - *"Replace with:"*: `_`
>
> 2. {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file"*: `genes_metadata` (output of **Decoupler pseudo-bulk** {% icon tool %})
>    - In *"Select Columns"*:
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `start`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `end`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `width`
>
> 3. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `samples_metadata` (output of **Decoupler pseudo-bulk** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `[ --+*^]+`
>            - *"Replace with:"*: `_`
>
> 4. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `outfile` (output of **Replace Text: Sample Metadata Step** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `c2`
>            - *"Find pattern"*: `^([0-9])(.+)`
>            - *"Replace with"*: `GG_\\1\\2`
>
> In the previous steps, the following modifications were made to the files:
>
> 1. **Replace Text**: Replaces special characters (`[ --+*^]+`) with underscores (`_`) in the `count_matrix` file. Rename output to **Count matrix**.
> 2. **Remove Columns**: Removes the `start`, `end`, and `width` columns from the `genes_metadata` file. Rename output to **Genes Metadata**.
> 3. **Replace Text**: Replaces special characters (`[ --+*^]+`) with underscores (`_`) in the `samples_metadata` file.
> 4. **Replace Text in Column**: Modifies values in column `c2` of `outfile from step 3` by prefixing `GG_` to numbers at the beginning of the string. Rename output to **Samples Metadata**
>
>    {% snippet faqs/galaxy/analysis_regular_expressions.md %}
>
{: .hands_on}


## Generating the Contrast File

This file will be used as the contrast input file in the edgeR tool.

> <hands-on-title> Creating a Contrast File for edgeR </hands-on-title>
>
> 1. {% tool [Text Reformatting](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Samples Metadata` (output generated from the **Replace Text** {% icon tool %} step).
>    - *"AWK Program"*:
>      ```BEGIN { print "header" }
>         NR > 1 { if (!seen[$2]++) words[++count]=$2 }
>         END { for (i=1; i<=count; i++)
>             for (j=i+1; j<=count; j++)
>               print words[i]"-"words[j] }
>      ```
>
> Rename to **Contrast File** 
>
>    > <comment-title> Explanation of the AWK Program </comment-title>
>    >
>    > This AWK script performs the following:
>    > - Initializes the `header` variable for the output file.
>    > - Processes the input file (`NR > 1` skips the header line).
>    > - Tracks unique elements in column 2 (`seen[$2]++`) to avoid duplicates.
>    > - Creates pairs of unique elements and calculates their difference, outputting the contrast for downstream edgeR analysis.
>    > - **Important!** When using the *"Text reformatting with awk"* tool, make sure the code is written all in one line, without any line breaks. Otherwise, it won’t work as expected.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How does your contrast file look, and what contrast is generated by the tool?
>
> > <solution-title></solution-title>
> >
> > 1. The contrast file is a simple tab-delimited text file. It contains:
> >    - A **header** in the first row that labels the column.
> >    - The contrast, such as `bonemarrow-pbmcs`, written in the second row.
> >
> > When working with your own data, the contrast file will look different, as it will reflect the specific contrasts in your dataset but the **header** should be the same
> >
> {: .solution}
>
{: .question}


# Differential Expression Analysis (DE) with edgeR

The tool **edgeR** is commonly used for differential expression (DE) analysis of pseudobulk aggregates in single-cell RNA-seq studies ({% cite Chen2024 %}). It calculates gene dispersions based on the total counts for each gene and adjusts these estimates using an approach called empirical Bayes. This method improves reliability by pooling information across all genes in the dataset to stabilize dispersion estimates, particularly for genes with low counts or limited data. By "borrowing strength" from the behavior of other genes, empirical Bayes reduces the risk of overestimating or underestimating variability, ensuring more robust and accurate results ({% cite Chen2024 %}). Differential expression is then evaluated using an exact test for the negative binomial distribution, which is similar to Fisher’s exact test but modified to handle the variability typically seen in RNA-seq data ({% cite Chen2024 %}).

In addition to exact tests, edgeR employs generalized linear models (GLMs), which allow for the analysis of more complex experimental designs. GLMs model gene counts as a function of experimental conditions (e.g., treatment groups or time points) and estimate how these conditions influence gene expression. Likelihood-based methods, such as quasi-likelihood (QL) approaches, are central to this framework. Standard likelihood methods evaluate the fit of the model to the data, while quasi-likelihood methods add an extra layer by explicitly accounting for biological and technical variability, improving the reliability of DE results. These models allow edgeR to identify subtle expression differences while controlling for overdispersion in the data ({% cite Chen2016 %}).

Several plots can be generated to assist in understanding the data and the results of the analysis, including Multidimensional scaling (MDS), Biological Coefficient of Variation plot (BCV), Quasi-Likelihood (QL), and Mean-Difference plot o MA (MD) plots. These visualizations provide insights into sample relationships, variability, and differential expression, and will be explained further in the tutorial. With these concepts in mind, let's now perform our DE analysis using our edgeR tool in Galaxy!

> <hands-on-title> Run a DGE Analysis with edgeR </hands-on-title>
>
> 1. {% tool [edgeR](toolshed.g2.bx.psu.edu/repos/iuc/edger/edger/3.36.0+galaxy5) %} with the following parameters:
>    - *"Count Files or Matrix?"*: `Single Count Matrix`
>        - {% icon param-file %} *"Count Matrix"*: `Count Matrix` (output of **Replace Text: Count Matrix** {% icon tool %})
>        - *"Input factor information from file?"*: `Yes`
>            - {% icon param-file %} *"Factor File"*: `Samples Metadata` (output of **Replace Text: Creating Factor File** {% icon tool %})
>    - *"Use Gene Annotations?"*: `Yes`
>        - {% icon param-file %} *"Gene Annotations"*: `Genes Metadata` (output of **Remove Columns: Gene Metadata** {% icon tool %})
>    - *"Formula for linear model"*: `~ 0 + tissue`
>        *(Customize this formula based on your data's contrast names and factor file. Ensure the formula matches EdgeR's syntax and uses only elements from the factor file.~ 0 + factor_A` or `~ 0 + factor_A + factor_B:factor_C)*
>    - *"Input contrasts manually or through a file?"*: `file`
>        - {% icon param-file %} *"Contrasts File"*: `Contrast File` (output of **Text Reformatting: Creating a Contrast File for edgeR** {% icon tool %})
>    - In *"Filter Low Counts"*:
>        - *"Filter lowly expressed genes?"*: `No`
>
>    > <comment-title> edgeR Tool Overview </comment-title>
>    >
>    > Use this tool to perform DGE analysis using edgeR. It is highly customizable, allowing you to input your count matrix, factor information, and gene annotations prepared during earlier steps. The contrast file should be generated from the previous step, **Text Reformatting: Creating a Contrast File for edgeR**. The model formula should be defined based on your dataset and must include factors that were already specified in the contrast file.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What are the output(s) of the edgeR tool?
> 2. How can we interpret our output result file?
>
> > <solution-title></solution-title>
> >
> > 1. **Output files**:
> >    - **edgeR Tables**: This file includes columns with information such as:
> >    - **Gene Symbol**: The identifier for each gene.
> >    - **Log Fold Change (logFC)**: Represents the direction and magnitude of differential expression. Positive values indicate upregulation, and negative values indicate downregulation.
> >    - **Raw p-value (PValue)**: The unadjusted statistical significance value for each gene, indicating the likelihood of observing the result under the null hypothesis.
> >    - **False Discovery Rate (FDR)**: The adjusted p-value using the Benjamini-Hochberg correction method to account for multiple testing and control the expected proportion of false positives.
> >    - **(Optional)**: If additional outputs, such as "normalized counts," are selected in the edgeR tool, the table will also include these counts.
> >    - **edgeR Report**: This is an HTML file containing various visualizations and summaries of the analysis, such as:
> >    - **MDS Plot (Multidimensional Scaling)**: Visualizes the relationships between samples based on gene expression profiles.
> >    - **BCV Plot (Biological Coefficient of Variation)**: Displays the estimated dispersion for each gene.
> >    - **QL Plot (Quasi-Likelihood)**: Shows the quasi-likelihood dispersions for the dataset.
> >    - **MD Plot (Mean-Difference Plot)**: Compares the mean expression of genes versus their log fold change.
> >
> > 2. **Interpreting the Results**:
> >    - The table contains the results of the differential expression analysis.
> >    - The **first column** typically lists the gene symbols from the dataset.
> >    - The **logFC** indicates the direction and magnitude of differential expression:
> >    - **Upregulated genes**: Genes with higher expression in the first group of the contrast (e.g., 'bone marrow' group in a "bonemarrow-pbmcs" contrast).
> >    - **Downregulated genes**: Genes with lower expression in the first group of the contrast (e.g., 'pbmcs' group in a "bonemarrow-pbmcs" contrast).
> >    - The **raw p-value** represents the statistical significance of the result for each gene before adjustment for multiple comparisons. Lower values indicate stronger evidence against the null hypothesis.
> >    - The **FDR** is the adjusted p-value, calculated using the Benjamini-Hochberg method, which helps control for false positives when testing many genes. Genes with an FDR below a threshold (e.g., 0.05) are considered statistically significant.
> >
> > **Plot Interpretations**:
> >   - **MDS Plot**: Displays relationships between samples based on gene expression profiles. Samples that cluster closely are more similar in their expression. Use this to identify whether samples separate by biological condition or to detect potential batch effects.  ![MDS Plot](../../images/pseudobulk-analysis/mdsplot_tissue.png)
> >   - **BCV Plot**: Shows the dispersion for each gene, with higher values indicating greater variability. This is useful for assessing how variability is modeled in the dataset. ![BCV Plot](../../images/pseudobulk-analysis/bcvplot.png)
> >   - **QL Plot**: Highlights the quasi-likelihood dispersions, which represent variability modeled during statistical testing. Proper dispersion modeling ensures robust differential expression analysis. ![QL Plot](../../images/pseudobulk-analysis/qlplot.png)
> >   - **MD Plot**: Visualizes the mean expression levels against log fold change for each gene. Genes far from the center indicate stronger differential expression, with points above or below the horizontal line showing upregulated or downregulated genes, respectively.  ![MD Plot](../../images/pseudobulk-analysis/mdplot_bonemarrow-pbmcs.png)
> >
> {: .solution}
>
{: .question}

## Sanitation Steps - Part 2

After performing the differential expression analysis with edgeR, we will clean the data to prepare it for visualization. This involves extracting dataset from a collection, removing unnecessary columns, standardizing text, and splitting the file if needed. We will use the {% tool [Extract dataset](__EXTRACT_DATASET__) %}, {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %}.

> <hands-on-title></hands-on-title>
>
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input list"*: `Tables from edgeR data` (output from the **edgeR** tool {% icon tool %})
>
>    **Extract dataset** will allow us extract the table output from **edgeR** which was retrived as a collection.  
>
> 2. {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file"*: `outTables` (output of **edgeR** {% icon tool %})
>    - In *"Select Columns"*:
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `X` (In other datasets, the header of the _GeneID_ columns may look different)
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `logFC`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `PValue`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `FDR`
>    - *"Keep named columns"*: `Yes`
>    - *"Rename output file to"*: `edgeR_DEG_summary`
>
>    **Remove columns** to filter out unnecessary columns from the **edgeR** output and retain only the essential ones for analysis: Gene ID (`id`), Log Fold Change (`logFC`), P-value (`PValue`), and False Discovery Rate (`FDR`).
>
{: .hands_on}

# Volcano Plot

In this step, we will use the sanitized output from the previous steps to generate a Volcano Plot, which visualizes the relationship between statistical significance (P-value) and fold change (LogFC) for differentially expressed genes (DEGs). The input file for the Volcano Plot must include four essential columns: _FDR (adjusted P-value), P-value (raw), Log Fold Change, and Gene Symbols (Labels)._ As long as these columns are present, the Volcano Plot can be generated successfully.

> <hands-on-title> Create a Volcano Plot of the DEG </hands-on-title>
>
> 1. {% tool [Volcano Plot](toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.6) %} with the following parameters:
>    - {% icon param-file %} *"Specify an input file"*: `edgeR_DEG_summary` (tabular file output from **Remove Columns** {% icon tool %})
>    - *"File has header?"*: `Yes`
>    - *"FDR (adjusted P value) column number"*: `c4`
>    - *"P value (raw) column number"*: `c3`
>    - *"Log Fold Change column number"*: `c2`
>    - *"Labels column number"*: `c1`
>    - *"LogFC threshold to colour"*: `0.58`
>    - *"Points to label"*: `Significant`
>        - *"Only label top most significant"*: `40`
>    - In *"Plot Options"*:
>        - *"Plot title"*: `Differential Expression Volcano Plot`
>
>    > <tip-title> What does the Volcano Plot show? </tip-title>
>    >
>    > The Volcano Plot highlights genes with statistical significance (low P-values) and large fold changes. Genes meeting the significance thresholds are typically colored and labeled for easier identification.
>    {: .tip}
>
{: .hands_on}

## What are the results of the Volcano Plot?

Let's take a moment to interpret the Volcano Plot:
![Volcano Plot](../../images/pseudobulk-analysis/VolcanoPlot.png)

> <question-title></question-title>
>
> 1. What is the significance of genes located at the extremes of the plot (e.g., high LogFC and low P-value)?
> 2. How many genes meet the significance threshold in this analysis?
> 3. What do the differentially expressed genes reveal about immune cells in bone marrow versus peripheral blood?
> 
> > <solution-title></solution-title>
> >
> > 1. Genes at the extremes of the volcano plot are highly significant and show strong differential expression. Investigating their biological roles may provide valuable insights into the underlying mechanisms.
> > 2. This analysis identified a substantial number of differentially expressed genes (DEGs), highlighting distinct expression profiles between immune cells from bone marrow and peripheral blood. Specifically, 188 genes were upregulated in bone marrow and 136 in PBMCs.
> > 3. These findings reveal tissue-specific immune cell behaviour, pointing to transcriptional specialization between bone marrow and peripheral blood compartments.
> >
> {: .solution}
>
{: .question}

# Subsetting Samples from the Original AnnData Object

In our initial analysis, we observed transcriptional differences between bone marrow and peripheral blood samples. However, since the dataset includes more than one immune cell type (pDCs and NCMs), it's not immediately clear which population is primarily driving these differences.

To gain further insight, it's helpful to perform pseudobulk analysis on each cell type separately. By subsetting the data by cluster and repeating the differential analysis (as we did with the full dataset at the start of this tutorial), we can better assess the individual contribution of each cell type.

In the following step, we’ll demonstrate how to subset the AnnData object by cell type before reapplying the pseudobulk workflow. In the hands-on example, we will filter for pDCs. To extract NCMs instead, you can follow the exact same steps, but set the *"Value"* parameter to `Non_Classical_Monocyte`, which corresponds to the annotation of that cell type.

If you would like to extract all annotated clusters at once, for example to analyse each of them independently, refer to the tip box below titled **“Split AnnData object by cluster or other observation key into a collection”**.

## Extracting observations of interest as AnnData object

> <hands-on-title>Use Manipulate AnnData Tools to extract observations</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - *"Annotated data matrix"*: `AnnData for Pseudobulk` (your preprocessed, analysed, and annotated AnnData object)
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>    - *"What to filter?"*: `Observations (obs)` (select this to filter cells)
>    - *"Type of filtering?"*: `By key (column) values`
>    - *"Key to filter"*: `annotated` (the column that contains the cell type annotations)
>    - *"Type of value to filter"*: `Text`
>    - *"Filter"*: `equal to`
>    - *"Value"*: `pDC` (the cluster name for the cell type you want to extract)
>
>    > <comment-title>Pre-extracted clusters also available</comment-title>
>    >
>    > In addition to performing the filtering yourself, the `pDCs` and `Monocytes_NC` clusters have also been pre-extracted and are available for direct download.  
>    > This allows you to jump ahead and run the downstream workflows independently for each cluster, as described later in this tutorial.  
>    >
>    > Import the pre-filtered datasets from Zenodo:
>    >
>    > ```
>    > {{ page.zenodo_link }}/files/Monocytes_NC_subset.h5ad
>    > {{ page.zenodo_link }}/files/pDCs_subset.h5ad
>    > ```
>    >
>    {: .comment}
>
{: .hands_on}

After using the {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} to extract the cell type of interest, return to the **Pseudobulk with Decoupler** step at the beginning of this tutorial. You can now repeat the same steps using this smaller AnnData object, which contains only the selected cell type (e.g., pDCs). The resulting analysis will reveal differential gene expression between conditions (e.g., COVID-19 vs. healthy) for that specific cell type only.

> <question-title></question-title>
>
> 1. What data is included in the new pseudobulk count matrix for the pDCs only? How is this matrix structured, and what do the column labels represent?
> 2. How many samples are included in the pDCs AnnData? How many of these are from bone marrow and how many from peripheral blood?
> 3. After performing pseudobulk analysis on pDCs and NCMs independently, how do the volcano plots compare? Do they reveal differentially expressed genes between tissues? If so, which cell type appears to drive the tissue-specific differences?
>
> > <solution-title></solution-title>
> >
> > 1. The new count matrix includes the original 1,373 rows, each representing a gene, with gene identifiers listed in the first column. The remaining columns correspond to individual samples, such as *bmacute_pDC* and *pbmcsacuteone_pDC*.
> > 2. This dataset includes a total of eleven samples: three from bone marrow and eight from peripheral blood.
> > 3. The analysis of pDCs and NCMs reveals distinct patterns of differentially expressed genes (DEGs) between tissues. Based on the results, NCMs show stronger transcriptional differences and appear to be the main contributors to the tissue-specific expression patterns observed in the full dataset.
> > ![Volcano Plot pDCs](../../images/pseudobulk-analysis/Volcano_plot_pDCs.png)  
> > ![Volcano Plot NCMs](../../images/pseudobulk-analysis/Volcano_plot_NCMs.png)
> >
> {: .solution}
>
{: .question}


> <tip-title>Split AnnData object by cluster or other observation key into a collection</tip-title>
>
> You can split an `AnnData` object into multiple objects **at once** based on the values of a given `obs` key using the: {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %}.
>
> This tool automatically creates a collection containing one `AnnData` object for each unique value in the selected observation key.
>
> For example, to split the dataset by cluster annotation, you can use the key `louvain`, or in the case of this tutorial, `annotated`. Each resulting element in the collection will correspond to a different cluster or cell type, which can then be analysed independently.
>
{: .tip}

## Recommendations
1. **Data Preprocessing:**
   - Ensure raw counts are preserved in your AnnData object before pseudobulk analysis.
   - Verify the presence of important metadata fields to ensure meaningful grouping and contrasts.
2. **Focus on Subsets:**
   - Analyse subsets of cell types or clusters for more granular insights. For example, pseudobulk analysis focused on specific immune cell populations can reveal subtle but important expression changes.
3. **Quality Control:**
   - Always inspect the outputs of Decoupler (e.g., pseudobulk plots and filtering summaries) to confirm data integrity and address outliers.
   - Use edgeR’s diagnostic plots (e.g., MDS, BCV) to check for batch effects or unexpected variability.
4. **Follow-Up Analysis:**
   - Consider functional analysis (e.g., GO enrichment, pathway analysis) for the differentially expressed genes to link expression changes to biological processes.
   - Use visualization tools, such as heatmaps or boxplots, to further explore expression trends in key genes.
5. **Future Applications:**
   - Pseudobulk analysis can be extended to multi-condition experiments, time-course data, or other cell-type-specific investigations.


topic: single-cell
tutorial: alevin-commandline
title: Generating a single cell matrix using Alevin and combining datasets (bash + R)
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["10x", "paper-replication", "MIGHTS"]
content_hash: 0a91dcf1345c2490cace457509526ad2
last_modified: 2026-02-26T21:12:17.895865
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/alevin-commandline/tutorial.html

content:
# Setting up the environment

Alevin is a tool integrated with the [Salmon software](https://salmon.readthedocs.io/en/latest/salmon.html), so first we need to get Salmon. You can install Salmon using conda, but in this tutorial we will show an alternative method - downloading the pre-compiled binaries from the [releases page](https://github.com/COMBINE-lab/salmon/releases). Note that binaries are usually compiled for specific CPU architectures, such as the 64-bit (x86_64) machine release referenced below .

```bash
wget -nv https://github.com/COMBINE-lab/salmon/releases/download/v1.10.0/salmon-1.10.0_linux_x86_64.tar.gz
```

Once you've downloaded a specific binary (here we're using version 1.10.0), just extract it like so:

```bash
tar -xvzf salmon-1.10.0_linux_x86_64.tar.gz
```

> <details-title>Conda installation</details-title>
>
> As mentioned, installing salmon using conda is also an option, and you can do it using the following command in the terminal:
> ```
> conda install -c conda-forge -c bioconda salmon
> ```
>
> However, for this tutorial, it would be easier and quicker to use the downloaded pre-compiled binaries, as shown above.
>
{: .details}

We're going to use Alevin for demonstration purposes, but we do not endorse one method over another.

# Get Data

We continue working on the same example data - a very small subset of the reads in a mouse dataset of fetal growth restriction {% cite Bacon2018 %} (see the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)). For the purposes of this tutorial, the datasets have been subsampled to only 50k reads (around 1% of the original files). Those are two fastq files - one with transcripts and the another one with cell barcodes. You can download the files by running the code below:

```bash
wget -nv https://zenodo.org/records/10116786/files/transcript_701.fastq
wget -nv https://zenodo.org/records/10116786/files/barcodes_701.fastq
```

 > <question-title></question-title>
>
> How to differentiate between the two files if they are just called 'Read 1' and 'Read 2'?
>
> > <solution-title></solution-title>
> >
> > The file which contains the cell barcodes and UMI is significantly shorter (indeed, 20 bp!) compared to the other file containing longer, transcript read. For ease, we will use explicit file names.
> >
> {: .solution}
>
{: .question}


Additionally, to map your reads, you will need a transcriptome to align against (a FASTA) as well as the gene information for each transcript (a gtf) file. These files are included in the data import step below. You can also download these for your species of interest [from Ensembl](https://www.ensembl.org/info/data/ftp/index.html).

```bash
wget -c https://zenodo.org/record/4574153/files/Mus_musculus.GRCm38.100.gtf.gff -O GRCm38_gtf.gff
wget -c https://zenodo.org/record/4574153/files/Mus_musculus.GRCm38.cdna.all.fa.fasta -O GRCm38_cdna.fasta
```

Why do we need FASTA and GTF files?
To generate gene-level quantifications based on transcriptome quantification, Alevin and similar tools require a conversion between transcript and gene identifiers. We can derive a transcript-gene conversion from the gene annotations available in genome resources such as Ensembl. The transcripts in such a list need to match the ones we will use later to build a binary transcriptome index. If you were using spike-ins, you'd need to add these to the transcriptome and the transcript-gene mapping.

We will use the murine reference annotation as retrieved from Ensembl (*GRCm38* or *mm10*) in GTF format. This annotation contains gene, exon, transcript and all sorts of other information on the sequences. We will use these to generate the transcript-gene mapping by passing that information to a tool that extracts just the transcript identifiers we need.

Finally, we will download some files for later so that you don't have to switch kernels again - that's an Alevin output folder for another sample!

```bash
wget https://zenodo.org/records/10116786/files/alevin_output_702.zip
unzip alevin_output_702.zip
```


# Generate a transcript to gene map and filtered FASTA

We will now install the package `atlas-gene-annotation-manipulation` to help us generate the files required for Salmon and Alevin.

```bash
conda install -y -c conda-forge -c bioconda atlas-gene-annotation-manipulation
```

After the installation is completed, you can execute the code cell below and while it's running, I'll explain all the parameters we set here.

```bash
gtf2featureAnnotation.R -g GRCm38_gtf.gff -c GRCm38_cdna.fasta -d "transcript_id" -t "transcript" -f "transcript_id" -o map -l "transcript_id,gene_id" -r -e filtered_fasta
```

In essence, [gtf2featureAnnotation.R script](https://github.com/ebi-gene-expression-group/atlas-gene-annotation-manipulation) takes a GTF annotation file and creates a table of annotation by feature, optionally filtering a cDNA file supplied at the same time. Therefore the first parameter `-g` stands for "gtf-file" and requires a path to a valid GTF file. Then `-c` takes a cDNA file for extracting meta info and/or filtering - that's our FASTA! Where --parse-cdnas (that's our `-c`) is specified, we need to specify, using `-d`, which field should be used to compare to identfiers from the FASTA. We set that to "transcript_id" - feel free to inspect the GTF file to explore other attributes. We pass the same value in `-f`, meaning first-field, ie. the name of the field to place first in output table. To specify which other fields to retain in the output table, we provide comma-separated list of those fields, and since we're only interested in transcript to gene map, we put those two names ("transcript_id,gene_id") into `-l`. `-t` stands for the feature type to use, and in our case we're using "transcript". Guess what `-o` is! Indeed, that's the output annotation table - here we specify the file path of our transcript to gene map. We will also have another output denoted by `-e` and that's the path to a filtered FASTA. Finally, we also put `-r` which is there only to suppress header on output. Summarising, output will be a an annotation table, and a FASTA-format cDNAs file with unannotated transcripts removed.

Why filtered FASTA?
Sometimes it's important that there are no transcripts in a FASTA-format transcriptome that cannot be matched to a transcript/gene mapping. Salmon, for example,  used to produce errors when this mismatch was present. We can synchronise the cDNA file by removing mismatches as we have done above.


# Generate a transcriptome index

We will use Salmon in mapping-based mode, so first we have to build a Salmon index for our transcriptome. We will run the Salmon indexer as so:

```bash
salmon-latest_linux_x86_64/bin/salmon index -t filtered_fasta -i salmon_index -k 31
```

Where `-t` stands for our filtered FASTA file, and `-i` is the output of the mapping-based index. To build it, the function is using an auxiliary k-mer hash over k-mers of length 31. While the mapping algorithms will make use of arbitrarily long matches between the query and reference, the k size selected here will act as the minimum acceptable length for a valid match. Thus, a smaller value of k may slightly improve sensitivity. We find that a k of 31 seems to work well for reads of 75bp or longer, but you might consider a smaller k if you plan to deal with shorter reads. Also, a shorter value of k may improve sensitivity even more when using selective alignment (enabled via the –validateMappings flag). So, if you are seeing a smaller mapping rate than you might expect, consider building the index with a slightly smaller k.


> <details-title>What is the index?</details-title>
>
> To be able to search a transcriptome quickly, Salmon needs to convert the text (FASTA) format sequences into something it can search quickly, called an 'index'. The index is in a binary rather than human-readable format, but allows fast lookup by Alevin. Because the types of biological and technical sequences we need to include in the index can vary between experiments, and because we often want to use the most up-to-date reference sequences from Ensembl or NCBI, we can end up re-making the indices quite often.
>
{: .details}


# Use Alevin

Time to use Alevin now! Alevin works under the same indexing scheme (as Salmon) for the reference, and consumes the set of FASTA/Q files(s) containing the Cellular Barcode(CB) + Unique Molecule identifier (UMI) in one read file and the read sequence in the other. Given just the transcriptome and the raw read files, Alevin generates a cell-by-gene count matrix (in a fraction of the time compared to other tools).


> <details-title>How does Alevin work in detail?</details-title>
>
> Alevin works in two phases. In the first phase it quickly parses the read file containing the CB and UMI information to generate the frequency distribution of all the observed CBs, and creates a lightweight data-structure for fast-look up and correction of the CB. In the second round, Alevin utilizes the read-sequences contained in the files to map the reads to the transcriptome, identify potential PCR/sequencing errors in the UMIs, and performs hybrid de-duplication while accounting for UMI collisions. Finally, a post-abundance estimation CB whitelisting procedure is done and a cell-by-gene count matrix is generated.
>
{: .details}

Alevin can be run using the following command:

```bash
salmon-latest_linux_x86_64/bin/salmon alevin -l ISR -1 barcodes_701.fastq -2 transcript_701.fastq --dropseq  -i salmon_index -p 10 -o alevin_output --tgMap map --freqThreshold 3 --keepCBFraction 1 --dumpFeatures
```

All the required input parameters are described in [the documentation](https://salmon.readthedocs.io/en/latest/alevin.html), but for the ease of use, they are presented below as well:

> <details-title>Alevin input parameters</details-title>
> - `-l`: library type (same as Salmon), we recommend using ISR for both Drop-seq and 10x-v2 chemistry.
>
> - `-1`: CB+UMI file(s), alevin requires the path to the FASTQ file containing CB+UMI raw sequences to be given under this command line flag. Alevin also supports parsing of data from multiple files as long as the order is the same as in -2 flag. That's our barcodes_701.fastq file.
>
> - `-2`: Read-sequence file(s), alevin requires the path to the FASTQ file containing raw read-sequences to be given under this command line flag. Alevin also supports parsing of data from multiple files as long as the order is the same as in -1 flag. That's our transcript_701.fastq file.
>
> - `--dropseq` / `--chromium` / `--chromiumV3`: the protocol, this flag tells the type of single-cell protocol of the input sequencing-library. This is a study using the Drop-seq chemistry, so we specify that in the flag.
>
> - `-i`: index, file containing the salmon index of the reference transcriptome, as generated by salmon index command.
>
> - `-p`: number of threads, the number of threads which can be used by alevin to perform the quantification, by default alevin utilizes all the available threads in the system, although we recommend using ~10 threads which in our testing gave the best memory-time trade-off.
>
> - `-o`: output, path to folder where the output gene-count matrix (along with other meta-data) would be dumped. We simply call it alevin_output
>
> - `--tgMap`: transcript to gene map file, a tsv (tab-separated) file — with no header, containing two columns mapping of each transcript present in the reference to the corresponding gene (the first column is a transcript and the second is the corresponding gene). In our case, that's map_code generated by using gtf2featureAnnotation.R function.
>
> - `--freqThreshold` - minimum frequency for a barcode to be considered. We've chosen 3 as this will only remove cell barcodes with a frequency of less than 3, a low bar to pass but useful way of avoiding processing a bunch of almost certainly empty barcodes.
>
> - `--keepCBFraction` - fraction of cellular barcodes to keep. We're using 1 to quantify all!
>
> - `--dumpFeatures` - if activated, alevin dumps all the features used by the CB classification and their counts at each cell level. It’s generally used in pair with other command line flags.
>
{: .details}

We have also added some additional parameters (`--freqThreshold`, `--keepCBFraction`) and their values are derived from the [Alevin Galaxy tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}) after QC to stop Alevin from applying its own thresholds. However, if you're not sure what value to pick, you can simply allow Alevin to make its own calls on what constitutes empty droplets.


This tool will take a while to run. Alevin produces many file outputs, not all of which we'll use. You can refer to the [Alevin documentation](https://salmon.readthedocs.io/en/latest/alevin.html) if you're curious what they all are, you can look through all the different files to find information such as the mapping rate, but we'll just pass the whole output folder directory for downstream analysis.


> <question-title></question-title>
>
> 1. Can you find the information what was the mapping rate?
> 2. How many transcripts did Alevin find?
>
> > <solution-title></solution-title>
> >
> > 1. As mentioned above, in *alevin_output* folder there will be many different files, including the log files. To check the mapping rate, go to *alevin_output* -> *logs* and open *salmon_quant* file. There you will find not only information about mapping rate, but also many more, calculated at salmon indexing step.
> > 2. Alevin log can be found in *alevin_output* -> *alevin* and the file name is also *alevin*. You can find many details about the alevin process there, including the number of transcripts found.
> >
> {: .solution}
>
{: .question}

> <warning-title>Process stopping</warning-title>
>
> The command above will display the log of the process and will say "Analyzed X cells (Y% of all)". For some reason, running Alevin may sometimes cause problems in Jupyter Notebook and this process will stop and not go to completion. This is the reason why we use hugely subsampled dataset here - bigger ones couldn't be fully analysed (they worked fine locally though). The dataset used in this tutorial shouldn't make any issues when you're using Jupyter notebook through galaxy.eu, however might not work properly on galaxy.org. If you're accessing Jupyter notebook via galaxy.eu and alevin process stopped, just restart the kernel (in the upper left corner: *Kernel* -> *Restart kernel...*) and that should help. However, if it doesn't, here are the alternatives:
> - We prepared a Jupyter notebook with the same workflow as shown here, compatible with Google Collab. So if you have troubles with running the above step, you can use the [notebook provided](https://colab.research.google.com/drive/1nRidlpiU-_qUG1zLgpc8mJZewr_EQL0N?usp=sharing), upload it to your [Google Collab](https://colab.research.google.com/) and run the code there, following the tutorial on GTN.
> - You can download the output folder that you would get as a result of running the code above. Simply run the code below.
>
{: .warning}

```bash
# get the output folder from Alevin in case yours crashes
wget https://zenodo.org/records/13732502/files/alevin_output.zip
unzip alevin_output.zip
```

<!---
check if we can get alevinQC to work - and paste the info from the first Alevin tutorial?
-->

# Alevin output to SummarizedExperiment

Let's change gear a little bit. We've done the work in bash, and now we're switching to R to complete the processing. To do so, you have to change Kernel to R (either click on `Kernel` -> `Change Kernel...` in the upper left corner of your JupyterLab or click on the displayed current kernel in the upper right corner and change it).
![Figure showing the JupyterLab interface with an arrow pointing to the left corner, showing the option 'Kernel' -> 'Change Kernel...' and another arrow pointing to the right corner, showing the icon of the current kernel. The pop-up window asks which kernel should be chosen instead.](../../images/scrna-pre-processing/switch_kernel.jpg "Two ways of switching kernel.")


Now, when we are in the R environment, let's install the packages required for this part - `tximeta` and `DropletUtils`:

```bash
system("conda install -y bioconductor-tximeta bioconductor-dropletutils", intern=TRUE)
```
Double check that the installation went fine:
```bash
require("tximeta")
require("DropletUtils")
```

> <details-title>"There is no package called ‘tximeta’ or ‘DropletUtils’"?</details-title>
> If you encounter this error message while loading the library, you can try to install them with BiocManager, as shown below. Just replace "package_name" with either `tximeta` or `DropletUtils`. Beware, it can take up to 25 minutes to run!
> ```bash
> # run the code below only if you cannot load tximeta library (or using notebook version 1.0.1)
> if (!require("BiocManager", quietly = TRUE))
>     install.packages("BiocManager")
>
> BiocManager::install("package_name")
> library("package_name")
> ```
{: .details}


The [tximeta package](https://bioconductor.org/packages/devel/bioc/vignettes/tximeta/inst/doc/tximeta.html) created by {% cite Love2020 %} is used for import of transcript-level quantification data into R/Bioconductor and requires that the entire output of alevin is present and unmodified.

In the *alevin_output* -> *alevin* folder you can find the following files:
- *quants_mat.gz*- Compressed count matrix
- *quants_mat_rows.txt*- Row Index (CB-ids) of the matrix.
- *quants_mat_cols.txt* - Column Header (Gene-ids) of the matrix.
- *quants_tier_mat.gz* – Tier categorization of the matrix.

We will only focus on *quants_mat.gz* though. First, let's specify the path to that file:

```bash
path <- 'alevin_output/alevin/quants_mat.gz'
```

We will specify the following arguments when running *tximeta*:
- 'coldata' a data.frame with at least two columns:
  - files - character, paths of quantification files
  - names - character, sample names
- 'type' - what quantifier was used (can be 'salomon', 'alevin', etc.)

With that we can create a dataframe and pass it to tximeta to create SummarizedExperiment object.

```bash
coldata <- data.frame(files = path, names="sample701")
alevin_se <- tximeta(coldata, type = "alevin")
```

Inspect the created object:
```bash
alevin_se
```

As you can see, *rowData names* and *colData names* are still empty. Before we add some metadata,  we will first identify barcodes that correspond to non-empty droplets.

# Identify barcodes that correspond to non-empty droplets

Some sub-populations of small cells may not be distinguished from empty droplets based purely on counts by barcode. Some libraries produce multiple ‘knees’ (see the [Alevin Galaxy tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}#interpreting-the-qc-plots) for multiple sub-populations. The `emptyDrops` method ({% cite Lun2019 %}) has become a popular way of dealing with this. `emptyDrops` still retains barcodes with very high counts, but also adds in barcodes that can be statistically distinguished from the ambient profiles, even if total counts are similar.

EmptyDrops takes multiple arguments that you can read about in the [documentation](https://rdrr.io/github/MarioniLab/DropletUtils/man/emptyDrops.html). However, in this case, we will only specify the following arguments:

> <details-title>emptyDrops input parameters</details-title>
>
> - `m` -	A numeric matrix-like object - usually a dgTMatrix or dgCMatrix - containing droplet data prior to any filtering or cell calling. Columns represent barcoded droplets, rows represent genes.
> - `lower` - A numeric scalar specifying the lower bound on the total UMI count, at or below which all barcodes are assumed to correspond to empty droplets.
> - `niters` - An integer scalar specifying the number of iterations to use for the Monte Carlo p-value calculations.
> - `retain` - A numeric scalar specifying the threshold for the total UMI count above which all barcodes are assumed to contain cells.
>
{: .details}

Let's then extract the matrix from our `alevin_se` object. It's stored in *assays* -> *counts*.

```bash
matrix_alevin <- assays(alevin_se)$counts
```

And now run emptyDrops:
```bash
# Identify likely cell-containing droplets
out <- emptyDrops(matrix_alevin, lower = 100, niters = 1000, retain = 20)
out
```

We also correct for multiple testing by controlling the false discovery rate (FDR) using the Benjamini-Hochberg (BH) method ({% cite Benjamini1995 %}). Putative cells are defined as those barcodes that have significantly poor fits to the ambient model at a specified FDR threshold. Here, we will use an FDR threshold of 0.01. This means that the expected proportion of empty droplets in the set of retained barcodes is no greater than 1%, which we consider to be acceptably low for downstream analyses.

```bash
is.cell <- out$FDR <= 0.01
sum(is.cell, na.rm=TRUE)                              # check how many cells left after filtering
```

We got rid of the background droplets containing no cells, so now we will filter the matrix that we passed on to emptyDrops, so that it corresponds to the remaining cells.

```bash
emptied_matrix <- matrix_alevin[,which(is.cell),drop=FALSE]          # filter the matrix
```

From here, we can move on to adding metadata and we'll return to `emptied_matrix` soon.

# Adding cell metadata

The cells barcodes are stored in *colnames*. Let's exctract them into a separate object:
```bash
barcode <- colnames(alevin_se)
```

Now, we can simply add those barcodes into *colData names* where we will keep the cell metadata. To do this, we will create a column called `barcode` in *colData* and pass the stored values into there.

```bash
colData(alevin_se)$barcode <- barcode
colData(alevin_se)
```

That's only cell barcodes for now! However, after running *emptyDrops*, we generated lots of cell information that is currently stored in `out` object (Total, LogProb, PValue, Limited, FDR). Let's add those values to cell metadata! Since we already have *barcodes* in there, we will simply bind the emptyDrops output to the existing dataframe:

```bash
colData(alevin_se) <- cbind(colData(alevin_se),out)
colData(alevin_se)
```

As you can see, the new columns were appended successfully and now the dataframe has 6 columns.

If you have a look at the Experimental Design from that study, you might notice that there is actually more information about the cells. The most important for us would be batch, genotype and sex, summarised in the small table below.

| Index | Batch | Genotype | Sex |
|------ |--------------------|
|--:|--:|:--|:-:|
| N701 | 0    | wildtype    | male    |
| N702 | 1    | knockout   | male    |
| N703 | 2    | knockout   | female    |
| N704 | 3    | wildtype    | male    |
| N705 | 4    | wildtype    | male    |
| N706 | 5    | wildtype    | male    |
| N707 | 6    | knockout    | male    |

We are currently analysing sample N701, so let's add its information from the table.

## Batch

We will label batch as an integer - "0" for sample N701, "1" for N702 etc. The way to do it is creating a list with zeros of the length corresponding to the number of cells that we have in our SummarizedExperiment object, like so:

```bash
batch <- rep("0", length(colnames(alevin_se)))
```

And now create a batch slot in the *colData names* and append the `batch` list in the same way as we did with barcodes:

```bash
colData(alevin_se)$batch <- batch
colData(alevin_se)
```

A new column appeared, full of zeros - as expected!

## Genotype

It's all the same for genotype, but instead creating a list with zeros, we'll create a list with string "wildtype" and append it into genotype slot:

```bash
genotype <- rep("wildtype", length(colnames(alevin_se)))
colData(alevin_se)$genotype <- genotype
```

## Sex

You already know what to do, right? A list with string "male" and then adding it into a new slot!
```bash
sex <- rep("male", length(colnames(alevin_se)))
colData(alevin_se)$sex <- sex
```

Check if all looks fine:
```bash
colData(alevin_se)
```

3 new columns appeared with the information that we've just added - perfect! You can add any information you need in this way, as long as it's the same for all the cells from one sample.


# Adding gene metadata

The genes IDs are stored in *rownames*. Let's exctract them into a separate object:

```bash
gene_ID <- rownames(alevin_se)
```

Analogically, we will add those genes IDs into *rowData names* which stores gene metadata. To do this, we will create a column called `gene_ID` in *rowData* and pass the stored values into there.

```bash
rowData(alevin_se)$gene_ID <- gene_ID
```

## Adding genes symbols based on their IDs

Since gene symbols are much more informative than only gene IDs, we will add them to our metadata. We will base this annotation on Ensembl - the genome database – with the use of the library BioMart. We will use the archive Genome assembly GRCm38 to get the gene names. Please note that the updated version (GRCm39) is available, but some of the gene IDs are not in that EnsEMBL database. The code below is written in a way that it will work for the updated dataset too, but will produce ‘NA’ where the corresponding gene name couldn’t be found.

```bash
# get relevant gene names
library("biomaRt")                                      # load the BioMart library
ensembl.ids <- gene_ID
mart <- useEnsembl(biomart = "ENSEMBL_MART_ENSEMBL")    # connect to a specified BioMart database and dataset hosted by Ensembl
ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl")

# The line above connects to a specified BioMart database and dataset within this database.
# In our case we choose the mus musculus database and to get the desired Genome assembly GRCm38,
# we specify the host with this archive. If you want to use the most recent version of the dataset, just run:
# ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl")
```

> <warning-title>Ensembl connection problems</warning-title>
> Sometimes you may encounter some connection issues with Ensembl. To improve performance Ensembl provides several mirrors of their site distributed around the globe. When you use the default settings for useEnsembl() your queries will be directed to your closest mirror geographically. In theory this should give you the best performance, however this is not always the case in practice. For example, if the nearest mirror is experiencing many queries from other users it may perform poorly for you. In such cases, the other mirrors should be chosen automatically.
>
{: .warning}

```bash
genes <- getBM(attributes=c('ensembl_gene_id','external_gene_name'),
               filters = 'ensembl_gene_id',
               values = ensembl.ids,
               mart = ensembl_m)

# The line above retrieves the specified attributes from the connected BioMart database;
# 'ensembl_gene_id' are genes IDs,
# 'external_gene_name' are the genes symbols that we want to get for our values stored in ‘ensembl.ids’.
```
```bash
# see the resulting data
head(genes)
```
```bash
# replace IDs for gene names
gene_names <- ensembl.ids
count = 1
for (geneID in gene_names)
{
 index <- which(genes==geneID)    # finds an index of geneID in the genes object created by getBM()
 if (length(index)==0)            # condition in case if there is no corresponding gene name in the chosen dataset
  {
    gene_names[count] <- 'NA'
  }
  else
  {
    gene_names[count] <- genes$external_gene_name[index] 	# replaces gene ID by the corresponding gene name based on the found geneID’s index
  }
 count = count + 1                # increased count so that every element in gene_names is replaced
}
```
```bash
# add the gene names into rowData in a new column gene_name
rowData(alevin_se)$gene_name <- gene_names
```
```bash
# see the changes
rowData(alevin_se)
```

If you are working on your own data and it’s not mouse data, you can check available datasets for other species and just use relevant dataset in `useMart()` function.

```bash
listDatasets(mart)                # available datasets
```


## Flag mitochondrial genes

We can also flag mitochondrial genes. Usually those are the genes whose name starts with 'mt-' or 'MT-'. Therefore, we will store those characters in `mito_genes_names` and then use *grepl()* to find those genes stored in *gene_name* slot.

```bash
mito_genes_names <- '^mt-|^MT-'                                    # how mitochondrial gene names can start
mito <- grepl(mito_genes_names, rowData(alevin_se)$gene_name)      # find mito genes
mito                                                               # see the resulting boolean list
```

Now we can add another slot in *rowData* and call it *mito* that will keep boolean values (true/false) to indicate which genes are mitochondrial.
```bash
rowData(alevin_se)$mito <- mito
rowData(alevin_se)
```


# Subsetting the object

Let's go back to the `emptied_matrix` object. Do you remember how many cells were left after filtering? We can check that by looking at the matrix' dimensions:

```bash
dim(matrix_alevin)                                                  # check the dimension of the unfiltered matrix
dim(emptied_matrix)                                                 # check the dimension of the filtered matrix
```

We've gone from 3608 to 35 cells. We've filtered the matrix, but not our SummarizedExperiment. We can subset `alevin_se` based on the cells that were left after filtering. We will store them in a separate list, as we did with the barcodes:

```bash
retained_cells <- colnames(emptied_matrix)
retained_cells
```

And now we can subset our SummarizedExperiment based on the barcodes that are in the `retained_cells` list:

```bash
alevin_subset <- alevin_se[, colData(alevin_se)$barcode %in% retained_cells]
alevin_701 <- alevin_subset
alevin_701
```

And that's our subset, ready for downstream analysis!


# More datasets

We've done the analysis for one sample. But there are 7 samples in this experiment and it would be very handy to have all the information in one place. Therefore, you would need to repeat all the steps for the subsequent samples (that's when you'll appreciate wrapped tools and automation in Galaxy workflows!). To make your life easier, we will show you how to combine the datasets on smaller scale. Also, to save you some time, we've already run Alevin on sample 702 (also subsampled to 50k reads). Let's quickly repeat the steps we performed in R to complete the analysis of sample 702 in the same way as we did with 701.

But first, we have to save the results of our hard work on sample 701!

## Saving sample 701 data

Saving files is quite straightforward. Just specify which object you want to save and how you want the file to be named. Don't forget the extension!

```bash
save(alevin_701, file = "alevin_701.rdata")
```

You will see the new file in the panel on the left.


## Analysis of sample 702

Normally, at this point you would switch kernel to bash to run alevin, and then back to R to complete the analysis of another sample. Here, we are providing you with the alevin output for the next sample, which you have already downloaded and unzipped at the beginning of the tutorial when we were using bash.

> <warning-title>Switching kernels & losing variables</warning-title>
>
> Be aware that every time when you switch kernel, you will lose variables you store in the objects that you've created, unless you save them. Therefore, if you want to switch from R to bash, make sure you save your R objects! You can then load them anytime.
>
{: .warning}

Above we described all the steps done in R and explained what each bit of code does. Below all those steps are in one block of code, so read carefully and make sure you understand everything!

```bash
## load libraries again ##
library(tximeta)
library(DropletUtils)
library(biomaRt)

## running tximeta ##
path2 <- 'alevin_output_702/alevin/quants_mat.gz'                                                    # path to alevin output for N702
alevin2 <- tximeta(coldata = data.frame(files = path2, names = "sample702"), type = "alevin")        # create SummarizedExperiment from Alevin output

## running emptyDrops ##
matrix_alevin2 <- assays(alevin2)$counts                                                             # extract matrix from SummarizedExperiment
out2 <- emptyDrops(matrix_alevin2, lower = 100, niters = 1000, retain = 20)                          # apply emptyDrops
is.cell2 <- out2$FDR <= 0.01                                                                         # apply FDR threshold
emptied_matrix2 <- matrix_alevin2[,which(is.cell2),drop=FALSE]                                       # subset the matrix to the cell-containing droplets

## adding cell metadata ##
barcode2 <- colnames(alevin2)
colData(alevin2)$barcode <- barcode2                                             # add barcodes

colData(alevin2) <- cbind(colData(alevin2),out2)                                 # add emptyDrops info

batch2 <- rep("1", length(colnames(alevin2)))
colData(alevin2)$batch <- batch2                                                # add batch info

genotype2 <- rep("wildtype", length(colnames(alevin2)))
colData(alevin2)$genotype <- genotype2                                          # add genotype info

sex2 <- rep("male", length(colnames(alevin2)))
colData(alevin2)$sex <- sex2                                                    # add sex info

## adding gene metadata ##
gene_ID2 <- rownames(alevin2)
rowData(alevin2)$gene_ID <- gene_ID2

# get relevant gene names
ensembl.ids2 <- gene_ID2
mart <- useEnsembl(biomart = "ENSEMBL_MART_ENSEMBL")    # connect to a specified BioMart database and dataset hosted by Ensembl
ensembl_m2 = useMart("ensembl", dataset="mmusculus_gene_ensembl")

genes2 <- getBM(attributes=c('ensembl_gene_id','external_gene_name'),
               filters = 'ensembl_gene_id',
               values = ensembl.ids2,
               mart = ensembl_m2)

# replace IDs for gene names
gene_names2 <- ensembl.ids2
count = 1
for (geneID in gene_names2)
{
 index <- which(genes2==geneID)    # finds an index of geneID in the genes object created by getBM()
 if (length(index)==0)            # condition in case if there is no corresponding gene name in the chosen dataset
  {
    gene_names2[count] <- 'NA'
  }
  else
  {
    gene_names2[count] <- genes2$external_gene_name[index] 	# replaces gene ID by the corresponding gene name based on the found geneID’s index
  }
 count = count + 1                # increased count so that every element in gene_names is replaced
}

rowData(alevin2)$gene_name <- gene_names2                              # add gene names to gene metadata

mito_genes_names <- '^mt-|^MT-'                                        # how mitochondrial gene names can start
mito2 <- grepl(mito_genes_names, rowData(alevin2)$gene_name)           # find mito genes
rowData(alevin2)$mito <- mito2                                         # add mitochondrial information to gene metadata

## create a subset of filtered object ##
retained_cells2 <- colnames(emptied_matrix2)
alevin_subset2 <- alevin2[, colData(alevin2)$barcode %in% retained_cells2]

alevin_702 <- alevin_subset2
alevin_702
```

Alright, another sample pre-processed!


# Combining datasets

Pre-processed sample 702 is there, but keep in mind that if you switched kernels in the meantime, you still need to load sample 701 that was saved before switching kernels. It's equally easy as saving the object:

```bash
load("alevin_701.rdata")
```

Check if it was loaded ok:
```bash
alevin_701
```

Now we can combine those two objects into one using one simple command:

```bash
alevin_combined <- cbind(alevin_701, alevin_702)
alevin_combined
```

If you have more samples, just append them in the same way. We won't process another sample here, but pretending that we have third sample, we would combine it like this:

```bash
alevin_subset3 <- alevin_702                        # copy dataset for demonstration purposes
alevin_combined_demo <- cbind(alevin_combined, alevin_subset3)
alevin_combined_demo
```

You get the point, right? It's important though that the rowData names and colData names are the same in each sample.


# Saving and exporting the files

It is generally more common to use SingleCellExperiment format rather than SummarizedExperiment. The conversion is quick and easy, and goes like this:

```bash
library(SingleCellExperiment)                                 # might need to load this library
alevin_sce <- as(alevin_combined, "SingleCellExperiment")
alevin_sce
```
As you can see, all the embeddings have been successfully transfered during this conversion and believe me, SCE object will be more useful for you!

You've already learned how to save and load objects in Jupyter notebook, let's then save the SCE file:

```bash
save(alevin_sce, file = "alevin_sce.rdata")
```

The last thing that might be useful is exporting the files into your Galaxy history. To do it... guess what! Yes - **switching kernels again**! But this time we choose **Python 3** kernel and run the following command:

```bash
# that's Python now!
put("alevin_sce.rdata")
```
A new dataset should appear in your Galaxy history now.

# Conclusion

Well done! In this tutorial we have:
- examined raw read data, annotations and necessary input files for quantification
- created an index in Salmon and run Alevin
- identified barcodes that correspond to non-empty droplets
- added gene and cell metadata
- applied the necessary conversion to pass these data to downstream processes.

As you might now appreciate, some tasks are much quicker and easier when run in the code, but the reproducibility and automation of Galaxy workflows is a huge advantage that helps in dealing with many samples more quickly and efficiently.


topic: single-cell
tutorial: scrna-seurat-pbmc3k
title: Clustering 3K PBMCs with Seurat
difficulty: introductory
tools_json: []
requirements_json: []
tags_json: []
content_hash: 4c906fb9e72008bf7f83da8b678cff61
last_modified: 2026-02-26T21:12:17.996247
zenodo_link: https://zenodo.org/record/3581213
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-seurat-pbmc3k/tutorial.html

content:
separate-preprocessing---answer-key
  date: 2023-10-28
- label: UseGalaxy.eu - Using SCTransform
  history: https://usegalaxy.eu/u/marisa_jl/h/clustering-3k-pbmcs-with-seurat---sctransform---answer-ke
  date: 2023-10-28
- label: UseGalaxy.org.au - Using Separate Preprocessing Tools
  history: https://usegalaxy.org.au/u/videmp/h/gtn-clustering-3k-pbmcs-with-seurat-separate-preprocessing-may-2025
  date: 2025-05-02
- label: UseGalaxy.org.au - Using SCTransform
  history: https://usegalaxy.org.au/u/videmp/h/gtn-cluster-3k-pbmcs-with-seurat-sctransform-may-2025
  date: 2025-05-02
questions:
- How can we identify cell types in single cell RNA-Seq data?
- What are the steps for clustering single cell data with Seurat?
objectives:
- Explain the steps involved in clustering single cell data
- Evaluate the quality of single cell data and filter out low quality cells
- Prepare single cell data for analysis with Seurat
- Perform clustering with Seurat
- Be ready to apply the Seurat pipeline to new datasets
time_estimation: 8H
key_points:
- Seurat is a commonly-used pipeline for single cell data analysis
- Clustering makes single cell datasets easier for us to understand
- Different tools and parameters should be considered when analysing different datasets
requirements:
- type: internal
  topic_name: single-cell
  tutorials:
  - scrna-preprocessing
  - scrna-preprocessing-tenx
follow_up_training:
- type: internal
  topic_name: single-cell
  tutorials:
  - EBI-retrieval
tags:
- 10x
contributions:
  authorship:
  - MarisaJL
  editing:
  - pavanvidem
  - shiltemann
recordings:
- youtube_id: -tIOSBXeXPs
  length: 2H11M
  galaxy_version: 24.2.4.dev0
  date: '2025-04-15'
  speakers:
  - MarisaJL
  captioners:
  - MarisaJL
  bot-timestamp: 1744754154


---


Single cell RNA-seq analysis enables us to explore differences in gene expression between cells. It can reveal the heterogenity within cell populations and help us to identify cell types that could play roles in development, disease, or other processes. Single cell omics is a relatively young field, but there are a few commonly-used analysis pipelines that you will often see in the literature. In this tutorial, we will use one of these pipelines, Seurat, to cluster single cell data from a 10X Genomics experiment ({% cite seurat2023v5 %}). You can follow the same analysis using the Scanpy pipeline in the [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial.

Clustering is typically the first type of analysis we will perform on a single cell dataset. It groups together cells that are expressing similar genes, which makes the data easier to understand and often helps us to identify specific cell types.

> <comment-title></comment-title>
>
> This tutorial is based on the [Seurat - Guided Clustering Tutorial](https://satijalab.org/seurat/articles/pbmc3k_tutorial). The SCTransform sections also draw from the [Using sctransform in Seurat](https://satijalab.org/seurat/articles/sctransform_vignette.html) tutorial.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Important tips for easier analysis

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

# Data

For this tutorial, we will analyze a dataset of Peripheral Blood Mononuclear Cells (PBMC) extracted from a healthy donor, which is freely available from 10X Genomics. The dataset contains 2700 single cells sequenced using Illumina NextSeq 500. The raw sequences have been processed by the [**cellranger**](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) pipeline from 10X to extract a unique molecular identifier (UMI) count matrix, in a similar way to that explained in the [Pre-processing of 10X Single-Cell RNA Datasets]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}) tutorial.

In this matrix, the values represent the number of each feature (i.e. gene; row) detected in each cell (column). Single cell matrices can be quite large: here there are 2700 columns with 32,738 rows, so for each of our 2700 cells we know how many times we found RNAs matching each of the 32,738 genes. Since most of these genes weren't detected in most of the cells, the matrix is largely filled with zeros, i.e. it is an extremely sparse matrix. To optimize the storage of such a table and the information about the genes and cells, **cellranger** creates 3 files:

- `genes.tsv`: a tabular file with information about the 32,738 genes in 2 columns (Ensembl gene id and the gene symbol)
- `barcodes.tsv`: a tabular file with the barcode for each of the 2700 cells
- `matrix.mtx`: a condensed version of the count matrix (including the non-zero values only)

    The count matrix is represented by its non-zero values - we don't need to store all of those zeroes as long as we know where our non-zero values are in the matrix. Each non-zero value is represented by its line number (1st column), its column number (2nd column) and its value (3rd column). The first row gives the total number of rows (genes), columns (cells) and non-zero values. More information on the Matrix Market Exchange (mtx) format can be found [in this documentation](https://math.nist.gov/MatrixMarket/formats.html)

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the `genes.tsv`, `barcodes.tsv` and `matrix.mtx` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/genes.tsv
>    {{ page.zenodo_link }}/files/barcodes.tsv
>    {{ page.zenodo_link }}/files/matrix.mtx
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets as `genes`, `barcodes`, and `matrix` if necessary
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check the datatypes are correct - the `genes` and `barcodes` files should be tsv or tabular while the `matrix` should be an mtx file
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 5. Inspect the `matrix` file by clicking on the {% icon galaxy-eye %} icon
>
>    {% snippet faqs/galaxy/histories_dataset_item.md %}
>
{: .hands_on}

The beginning of the file should look like this:

> <question-title></question-title>
>
> ```
> 32738	2700	2286884
> 32709	1	4
> 32707	1	1
> 32706	1	10
> 32704	1	1
> ```
>
> 1. How many non-zero values are in the matrix?
> 2. How many counts were found for the 32,706th gene in the 1st cell?
>
> > <solution-title></solution-title>
> >
> > 1. The first row tells us there are 2,286,884 non-zero values for the 32,738 genes (rows) and 2,700 cells (columns) - so only 2.6% of the 88,392,600 potential values we could have in this matrix are non-zero. Getting rid of all those zeros has made the matrix much more compact.
> > 2. 10 counts were found for the 32,706th row (gene) and 1st column (cell), so we collected 10 RNAs that the first cell had produced from this particular gene.
> >
> {: .solution}
>
{: .question}

Representing the matrix with these three files is convenient for sharing the data, but not for processing them. Different single cell analysis packages have attempted to solve the problem of storage and analysis by inventing their own formats, which has led to the proliferation of many different 'standards' in the scRNA-seq package ecosystem.

## SeuratObject

In order to analyse the data using Seurat, we will first need to create a SeuratObject. A SeuratObject can store all our data, including the gene names, cell barcodes and matrix in a single RDS file. Since Seurat was written in the R programming language, SeuratObjects are saved as RData or RDS files. An RDS file is simply a type of RData file that contains a single object.

SeuratObjects can also hold any metadata we might have about our cells or genes, as well as the information we will produce during our analysis, such as our dimensional reductions and cell clusters. A SeuratObject can hold data in multiple layers (also known as slots in earlier version of Seurat). We can have layers of raw counts, normalised data, and scaled data. We could also store multiple assays or types of data in a single object, although in this tutorial we will only be using one assay for RNA-seq data. The SeuratObject is designed to be self-contained so that we don't have to work with lots of different files, even if we have different versions or types of data.

Creating a SeuratObject in R would require two steps - first, we would need to read in our data, in this case using the `Read10X` function, then secondly we would turn it into a SeuratObject using the `CreateSeuratObject` function. On Galaxy, we can perform both steps with a single tool. The `CreateSeuratObject` function also generates some QC metrics and performs basic filtering of the data.

> <hands-on-title>Create a SeuratObject</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Create Seurat Object`
>        - *"Select format of input"*: `matrix market (for e.g. 10x data)`
>            - {% icon param-file %} *"Counts matrix with features as rows, cells as columns (.mtx)"*: `matrix.mtx`
>            - {% icon param-file %} *"List of gene names (for rows)"*: `genes.tsv`
>            - {% icon param-file %} *"List of cell barcodes (for columns)"*: `barcodes.tsv`
>        - *"Include features detected in at least this many cells"*: `3`
>        - *"Include cells where at least this many features are detected"*: `200`
>        - *"Calculate percentage of mito genes in each cell"*: `No`
>
> 2. Rename the generated file to `Input 3k PBMC`
>
> 3. Check that the format is `rds`
{: .hands_on}

We can't look at the RDS file directly as it is designed for computers to read, rather than humans, but the Seurat tools will now be able to interact with the data. We can also use the {% tool Seurat Data Management %} tool to look inside our new SeuratObject.

> <hands-on-title>Inspect the `Input 3k PBMC` SeuratObject</hands-on-title>
>
> 1. {% tool [Seurat Data Management](toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0) %} with the following parameters:
>    - *"Method used"*: `Inspect Seurat Object`
>        - *"Display information about"*: `General`
>
> 2. Click the {% icon galaxy-eye %} on the output in your history to see some information about your SeuratObject
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many cells and genes are in the `Input 3k PBMC` SeuratObject we just created?
> 2. Is this the same as the number of cells and genes we started out with?
>
> > <solution-title></solution-title>
> > 1. If we click on the {% icon galaxy-eye %} of the output from {% tool Seurat Data Management %} we can see some general information about our SeuratObject. We can see that it contains information about 2700 samples (cells) and 13,714 features (genes).
> > 2. When we looked at the `matrix.mtx` file previously, we saw that it contained information on the same 2700 cells, but that there were 32,738 genes with non-zero values in the matrix. That means we are missing 19,024 genes - but don't worry, this is supposed to happen! Remember that we set the `Include features detected in at least this many cells` parameter to `3` - this got rid of any genes that were found in fewer than three cells. We assume that these genes are just technical noise (e.g. misreads, reads mapped to the wrong gene) rather than real biological features. We also set the `Include cells where at least this many features are detected` to `200` - this would have removed any cells that had fewer than 200 genes found in them. In this case, we didn't remove any cells as all 2700 of them must have had at least 200 genes. If we did have cells with fewer than 200 genes, we would want to get rid of them they are probably low quality cells or cell fragments. We would usually expect to find more features in an undamaged cell that was captured and sequenced properly.
> >
> {: .solution}
{: .question}

> <comment-title></comment-title>
> If you would like to see the impact of the initial filtering performed by {% tool Seurat Create %}, then you can rerun this step with these filters removed (or changed) and then perform the QC visualisation steps described in the next section on the unfiltered dataset. The Seurat pipeline usually includes some filtering during object creation, which shouldn't cause any problems as long as we use reasonable thresholds that only filter out the lowest quality cells and genes.  We can always come back and create a new object with different values if we suspect they weren't right for our dataset once we make our QC plots or if we have problems later on in the analysis.
{: .comment}

# Preprocessing

The standard Seurat pipeline begins with preprocessing, which includes identifying and filtering out low-quality cells, normalising and scaling the data, and selecting highly variable features.

## Quality Control

Quality control is an essential step in preparing single cell data for analysis. The results from a single cell experiment can vary in quality due to cell damage during dissociation, capturing more than one cell in a well or droplet, failures in library preparation such as inefficient reverse transcription (copying of captured RNAs to cDNA) or PCR amplification (making more copies of the cDNA so we have enough material for sequencing) or other technical problems. Low quality cells can cause issues for some analysis tools and may contribute to misleading results.

> <details-title>Impact of low-quality cells on the downstream analyses</details-title>
>
> #### Formation of their own distinct cluster(s)
>
> The most obvious cause of this problem is that damaged cells can appear more similar to each other because they have increased proportions of mitochondrial genes or are enriched for certain damage-induced genes. We could end up with a cluster made up of different cell types that share similar RNAs because they've been damaged. We might think these damaged cells represent an intermediate state between other cell types or end up misinterpreting our data in other ways.
>
> #### Distortion of population heterogeneity during variance estimation or principal components analysis
>
> When we cluster single cell data, we're looking for the biggest differences between groups of cells. If we have lots of low-quality cells, then the biggest differences we'll see might be these differences in quality, rather than something more biologically interesting. Differences in quality can have a big impact because low-quality cells often have low total RNA counts. When we perform scaling and normalisation on these cells during preprocessing, this can make the variances for the genes they do express much bigger than for other cells. When we select the most variable genes in the dataset, we'll end up picking the ones expressed by low-quality cells. We'll use these in our PCA analysis, likely ending up with top principal components based on cell quality rather than biology, making it harder to detect the differences we're actually interested in.
>
> #### Misidentification of upregulated genes
>
> Another problem that arises when we apply our preprocessing steps to small, low quality cells is that the genes we detect in them can appear to be strongly upregulated. Since we didn't detect many other genes in these cells, even a small difference in the number of transcripts detected can end up becoming much larger after normalisation. For example, contaminating transcripts may be present in all cells at low but constant levels. With the increased scaling and normalization in low-quality cells, the small counts for these transcripts may become large normalized expression values, so we might think we've found a population of cells where these genes are upregulated.
>
{: .details}

In order to mitigate these problems, we need to remove low-quality cells at the start of the analysis.

The `CreateSeuratObject` function we used in the {% tool Seurat Create %} tool automatically calculates some QC metrics and allowed us to filter out the lowest quality cells and features immediately. We will now check the data to see if it requires any further filtering.

### Computation of QC metrics

We already have two useful QC metrics that were calculated when we created our SeuratObject:

- **nCount_RNA** the total sum of RNAs that were found in each cell.
    The total number of counts is related to cell size, but it can also be an indication of quality. If `nCount_RNA` is very high we might be looking at results from a doublet or multiplet - two or more cells that were isolated together during the experiment. If `nCount_RNA` is very low, then it is likely that we lost a lot of the RNA due to cell lysis (breakage) or inefficient cDNA capture and amplification - or perhaps we only captured a fragment of a cell.
- **nFeature_RNA** the number of unique genes that were detected in each cell.
    We would expect to see some variation in the variety of genes expressed by cells of different sizes, types, and conditions, but if this number is unusually high or low then it could be a sign of poor quality. High `nFeature_RNA` could be another sign of a doublet or multiplet - we might have captured cells of different types and processed them together. Low `nFeature_RNA` could be due to loss of RNA if `nCount_RNA` is also low, or a sign that we have failed to capture the diversity of the transcript population, perhaps due to technical problems in our experiment.

One other metric that is often used to assess cell quality is the proportion of reads that came from the mitochondrial genome. The proportion is often higher in low quality, damaged or dying cells. Mitochondrial RNAs, which are protected inside the mitochondrial membranes, can be the last RNAs to be degraded or lost from a damaged cell, so we can end up with higher proportions of them in a low quality cell. We could have calculated the proportion of mitochondrial genes while creating our SeuratObject, but we will calculate it separately here to see how it is done - and how we could do the same for other types of genes.

We can identify mitochondrial genes from their gene symbols. Human genes that are encoded in the mitochondrial DNA (rather than in the cell nucleus) have names beginning with 'MT-'. Different naming patterns are used in other species. The {% tool Seurat Create %} tool can identify genes based on a specific naming pattern and then calculate the proportion of reads in each cell that came from these genes. We could use this tool to calculate the proportions of other genes with shared naming patterns too, such as ribosomal genes.

> <hands-on-title>Calculate the Proportion of Mitochondrial Reads</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Add QC Metrics`
>        - {% icon param-file %} *"Input file with the Seurat object"*: `Input 3k PBMC` (output of **Seurat Create** {% icon tool %})
>        - *"Calculate percentage of reads based on"*: `Pattern in gene names`
>        - *"Pattern/regex to match in gene names"*: `^MT-`
>
> 2. Rename the output as `Mitochondrial Annotations`
{: .hands_on}

When we use these QC metrics to remove low quality cells, we are assuming they are independent of the biological state of each cell, which isn't always the case in reality. For example, some cell types with higher energy demands may have higher proportions of mitochondrial reads because they have more mitochondria. We need to think carefully when deciding which metrics to use for filtering, but for now we will assume that these differences in our dataset are being driven by technical factors (e.g. cell damage, loss of RNAs, problems with capture and sequencing) rather than biological processes (e.g. cell types or conditions). We can therefore remove cells with poor values without misrepresenting the biology in downstream analyses.

### Filtering of low-quality cells

The simplest approch for identifing low-quality cells is to apply thresholds on the QC metrics. We assume that any cells beyond these thresholds are low quality cells and that this is due to those technical factors. Although this is a simple strategy, we will need to decide where to set our thresholds, and this will depend on the experimental protocol and biological system - there are no standard thresholds that we can use for every dataset.

We can visualise the QC metrics to help us decide where to set our thresholds for filtering out low quality cells. We want to get rid of cells that have unusually high or low numbers of genes or unique features, as well as cells that have higher proportions of mitochondrial genes.

> <hands-on-title>Visualise QC Metrics</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Violin Plot with 'VlnPlot'`
>        - *"Features to plot"*: `nFeature_RNA,nCount_RNA,percent.mt`
>        - In *"Plot Formatting Options"*:
>            - *"Number of columns to display"*: `3`
>
> 2. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `percent.mt`
>
> 3. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `nFeature_RNA`
>
{: .hands_on}

> <tip-title></tip-title>
> If you are re-running the same step again or just want to change a couple of parameters for the same tool, then you can click on a previous dataset to expand it in the history and then click the {% icon dataset-rerun button %}. The tool you used to create that dataset will open in the main panel, with the settings and inputs you used before. Make any changes and run it again.
{: .tip}

![Three violin plots each showing most points grouped together in the lower part of the plot with some outliers at higher values](../../images/scrna-seurat-pbmc3k/seurat_violin_QC_before.png "Violin Plots showing the unique features (nFeature_RNA), total counts (nCount_RNA) and the proportion of reads coming from mitochondial genes (percent.mt) for all cells")

![Scatter plot A shows clear positive correlation with the number of unique features increasing as total counts increases. A small number of cells have very high values of both. Scatter plot B does not show a relationship between the proportion of mitochondrial genes and total counts. A small number of cells with low total counts have high values of percent.mt](../../images/scrna-seurat-pbmc3k/seurat_QC_scatter_before.png "Scatter plots showing the relationships between the total counts (nCount_RNA) and A. the number of unique features (nFeature_RNA) and B. the proportion of mitochondrial reads (percent.mt)")

> <question-title></question-title>
>
> 1. What do the violin plots tell us about the cell sizes and quality in our dataset?
> 2. What do the scatter plots in tell us about the relationship between cell size and the other QC metrics?
>
> > <solution-title></solution-title>
> > 1. The violin plots give us an overview of the cell sizes (nCount_RNA), number of unique genes (nFeature_RNA), and proportion of mitochondrial reads for our cells. We can see that most of the cells are grouped together near the bottom of each plot, but there are some outliers that have unusually high or low values of each metric. Cells with higher values of nCount_RNA are likely to be larger as they contained more RNA. Cells with higher values of nFeature_RNA had RNAs produced from lots of different genes, while those with low nFeature_RNA had RNAs produced from a smaller range of genes. Cells with high percent.mt had lots of reads from mitochondrial genes. Although there will be some natural variation in cell size, the range of genes being expressed, and mitochondrial content, we suspect that extreme values of these three metrics reflect low quality cells.
> > 2. The scatter plots show us how these QC metrics relate to each other. As expected, the cells with higher total RNA counts also tend to have higher numbers of unique features, but some cells have particularly high values of both - these could be doublets. The proportion of mitochondrial genes doesn't increase with cell size (total counts) in the same way. We can see a small number of cells that have very high proportions of mitochondrial reads despite having low total reads - these are likely to be damaged cells that have lost a lot of their non-mitochondrial RNAs.
> >
> {: .solution}
{: .question}

We can now set our QC thresholds based on these plots. Unfortunately, there are no standard thresholds that can be applied to every dataset, so we need to look at our data and make this decision for ourselves.

We've already filtered out the cells with the lowest total counts when we created the SeuratObject, so we'll focus on filtering the number of unique features and the proportion of mitochondrial reads. We saw on the scatter plots that the cells with high values for these two metrics also had the highest nCount_RNA values, so we'll actually be getting rid of the cells with the highest total counts too.

> <comment-title></comment-title>
> If you've used Scanpy or followed the [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial then you might be wondering why we aren't visualising and filtering our genes as well as our cells. Seurat enables us to set the minimum number of cells that genes must be present in when we create the SeuratObject, but the pipeline doesn't usually include any further quality checks for genes. If we did need to do further filtering, we could do this in R by calculating an appropriate QC metric and subsetting the data.
{: .comment}

> <question-title></question-title>
>
> 1. What threshold would you set for nFeature_RNA?
> 2. What threshold would you set for percent.mt?
>
> > <solution-title></solution-title>
> > 1. A threshold of 2500 seems sensible for this dataset. The violin plot for nFeature_RNA shows that most of our cells should be under this threshold, so we won't lose too much of our data. The scatter plot shows that the cells above this threshold also had unusually high values of nCount_RNA, which suggest they could include some doublets.
> > 2. A threshold of 5% should work for this dataset. As before, the violin plot shows that the majority of our cells are below this threshold, but in this case the cells above it had low total RNA counts, which suggests these could be damaged cells that had lost a lot of their other RNAs. Although there are few standards to guide us in single cell analysis, you will see the same 5% threshold for mitochondrial content used in many studies. It often works well, but it can filter out energetic cell types such as muscle cells, so we shouldn't apply this threshold without considering whether it works for our dataset. Some studies don't filter on mitochondrial reads at all!
> >
> {: .solution}
{: .question}

We're setting QC thresholds based on our visual inspections of the data, but we could take different approaches too. We could calculate the interquartile range for our QC metrics and use this to set a threshold (e.g. maximum of 1.5 times the IQ range above the median). We could also try running the analysis with different thresholds to see how it affects the results - this could help us to see if we're excluding any biological variation, for example if we're filtering out a specific high-energy cell type with our mitochondrial threshold.

> <hands-on-title>Filter Out Low Quality Cells</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Filter cells by QC metrics`
>        - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>        - *"Minimum nFeature_RNA"*: `200`
>        - *"Maximum nFeature_RNA"*: `2500`
>        - *"Maximum percent.mt"*: `5.0`
>        - *"Filter by a different metric"*: `No`
>
> 2. Rename the output as `Filtered Dataset`
>
{: .hands_on}

If we produce the same plots again, we can see what has changed in our data.

> <hands-on-title>Re-Visualise QC Metrics</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Violin Plot with 'VlnPlot'`
>        - *"Features to plot"*: `nFeature_RNA,nCount_RNA,percent.mt`
>        - In *"Plot Formatting Options"*:
>            - *"Number of columns to display"*: `3`
>
> 2. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `percent.mt`
>
> 3. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `nFeature_RNA`
>
{: .hands_on}

![Three violin plots each showing most points grouped together and no extreme outliers](../../images/scrna-seurat-pbmc3k/seurat_violin_after.png "Violin Plots showing the unique features (nFeature_RNA), total counts (nCount_RNA) and the proportion of reads coming from mitochondial genes for all cells after filtering")

![Scatter plot A shows clear positive correlation with the number of unique features increasing as total counts increases. No extreme outliers. Scatter plot B does not show a relationship between the proportion of mitochondrial genes and total counts. No extreme outliers.](../../images/scrna-seurat-pbmc3k/seurat_QC_scatter_after.png "Scatter plots showing the relationships between the total counts (nCount_RNA) and A. the number of unique features (nFeature_RNA) and B. the proportion of mitochondrial reads (percent.mt) after filtering")

> <question-title></question-title>
>
> 1. Have we eliminated the low-quality cells from our data?
>
> > <solution-title></solution-title>
> > 1. We can see in both the violin and scatter plots that the outliers have been eliminated from our dataset - we've removed the cells with high mitochondrial counts and/or unusually high numbers of detected genes. We hope that this means we've got rid of the poor quality cells without losing any real biological variation, but we can't really be sure. Sometimes it is necessary to come back and re-think our filters - for example if you get to the end of the analysis and realise that you're missing a cell type that should be present in your data!
> >
> {: .solution}
{: .question}

## Further Preprocessing

Now that we're happy with the cells left in our SeuratObject, we can begin preparing the data for analysis. Seurat provides two routes for preprocessing, one with separate tools for each step and another that combines all these preprocessing steps into a single tool, `SCTransform`.

> <comment-title></comment-title>
>
> `SCTransform` makes preprocessing quicker and easier because it combines several steps from the standard Seurat pipeline into one. `SCTransform` normalises and scales the data while also selecting highly variable features. It also takes a different approach to preprocessing that can be more effective at removing technical effects from the data.
>
> Once you are familiar with the separate steps, you might find it more effective and convenient to use `SCTransform`. The main difference to be aware of when using `SCTransform` is that the results will be stored as a new assay called `SCT` in the SeuratObject. You will need to use this assay in the following analysis steps, instead of the original `RNA` assay. After running `SCTransform`, `SCT` will be set as the default assay for the SeuratObject so the tools should automatically use the correct one. The methods and default settings used in `SCTransform` also differ from the standard pipeline, so you should expect to see some differences in your results compared to the other route.
>
{: .comment}

The usual preprocessing steps for single cell data are normalisation, selection of the most variable features, and scaling the data. The same steps are performed in slightly different ways by the separate preprocessing functions and SCTransform.

- **Normalisation**
    Normalisation deals with variations in the number of reads we counted for each cell that were caused by systemic technical differences. We might have captured more RNAs or produced more copies of them during the PCR amplification step in some cells. If we don't remove these differences then they could obscure the biological differences we're trying to explore. We could end up with clusters of cells based on how many RNAs we captured or copied rather than on their cell type.
    The default normalisation method for the separate `NormalizeData` tool in Seurat is `LogNormalize`, which simply divides the number of counts for each gene by the total counts for the cell, multiplies this by a scale factor (10,000 is the default), and then log-transforms the results. Essentially, we're working out how many counts we would have for each gene if we had produced 10,000 reads from each cell, instead of getting different numbers of reads from different cells. The log-transformation then increases the impact that genes that showed stronger relative differences in expression between cells will have on our analysis (e.g. a gene that is expressed at an average count of 50 in cell type A and 10 in cell type B should have a bigger impact than a gene that is expressed at an average count of 1100 in A and 1000 in B). The normalised data will be stored in the `data` layer of the SeuratObject. The unnormalised data will still be available in the `counts` layer.
    Although this form of normalisation is widely-used, it does assume that each cell originally had the same number of RNA molecules, which isn't the case when our samples contain cells of different types and sizes. SCTransform takes a different approach to normalisation that doesn't make the same assumption about the cells {% cite SCTransform2022 %}. Instead, it creates a model of the UMI counts that enables it to regress out variation in sequencing depth (nFeature_RNA) and pools information from genes with similar abundances to adjust the variances. The corrected counts will be stored in the `counts` layer of the new SCT assay. The log1p(counts) will be stored in the `data` layer and the pearson residuals will be stored in the `scale.data` layer.
- **Selection of highly variable features**
    Rather than use the entire dataset in every stage of the analysis, we can focus on the genes that provide the most information - the highly variable genes that showed the biggest differences in expression between cells. We assume that these bigger differences reflect genuine biological differences, while the smaller differences in expression seen in other genes is down to chance or technical noise. Focusing on the most variable genes can make biological differences clearer in single cell analysis.
    Feature selection can have a big impact on our analysis as we will run downstream analyses (e.g. PCA) on these genes. We will be comparing cells based on the expression of these variable genes.
    Seurat's separate `FindVariableFeatures` tool selects the 2000 most variable genes by default, while `SCTransform` selects the top 3000 genes. We can select more variable genes when we use SCTransform because this method is better at removing technical effects from the data, so the additional variable features are more likely to represent biological variation.
  We may need to change this setting for some datasets to ensure we're selecting the most useful genes without including too many others.
- **Scaling**
    Scaling is a linear transformation that we apply to prepare our data for dimensional reduction.
    The default scaling method for the separate `ScaleData` tool in Seurat shifts the expression of each gene so that the mean expression across all the cells is 0. It also scales the expression of each gene so that the variance across all cells is 1. Scaling ensures that highly expressed genes don't dominate the analysis too much - we're interested in differences in expression between cells, not in genes that are always highly expressed in all the cells. The results will be stored in the `scale.data` layer.
    SCTransform doesn't scale data in the same way - although it centers the data by default, it won't scale the data unless you select this option. Instead, SCTransform usually stores the pearson residuals in the `scale.data` layer, which don't need to be scaled to the same variance.
    By default, both approaches only center/scale the highly variable genes that we'll use for dimensional reduction, but it is possible to scale more genes if required. If you choose to use the separate preprocessing tool in this tutorial, then we'll use the ScaleData tool to scale all the genes - this means that we can compare or plot the expression of any of the genes, as they will all be scaled in the same way. We won't perform scaling with SCTransform, as we're following the [original SCT tutorial](https://satijalab.org/seurat/articles/sctransform_vignette.html), but we will center all genes.
- **Regression**
  The Seurat pipeline can include another step during preprocessing of our single cell data. We can regress out (or remove) the impact of unwanted sources of variation. We could use this technique to remove the effects of the cell cycle or the differences associated with the proportion of mitochondrial genes. The goal is to reduce differences that are related to factors we are not interested in as this can help the differences we are interested in (like those between cell types or experimental groups) stand out more.
    It is possible to use the `ScaleData` function to regress out unwanted variation, but the creators of Seurat recommend using `SCTransform` for preprocessing if you want to do any regression. `SCTransform` automatically regresses out variation associated with sequencing depth (unique counts or nFeature_RNA) and can also regress out other variables. If you choose to use SCTransform in this tutorial, then you'll regress out the variation associated with the proportion of mitochondrial content, just like in [Seurat's original version of this tutorial](https://satijalab.org/seurat/articles/sctransform_vignette.html)

{% include _includes/cyoa-choices.html option1="Separate Preprocessing Steps" option2="SCTransform" default="Separate Preprocessing Steps" text="You can perform each preprocessing step separately, which might give you a better understanding of the different elements involved in preprocessing, or run them all at once using SCTransform." %}

<div class="Separate-Preprocessing-Steps" markdown="1">
><hands-on-title>Separate Preprocessing Steps</hands-on-title>
>
> 1. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Normalize with 'NormalizeData'`
>        - *"Method for normalization"*: `LogNormalize`
>
> 2. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `rds_out` (output of previous **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Identify highly variable genes with 'FindVariableFeatures'`
>        - *"Method to select variable features"*: `vst`
>        - *"Output list of most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>
> 3. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `rds_out` (output of previous **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Scale and regress with 'ScaleData'`
>        - *"Regress out a variable"*: `No`
>        - *"Features to scale"*: `All Features`
>
>    We'll choose to scale `All Features` here, instead of just the highly variable genes.
>
> 4. Rename the output as `Preprocessed Data`
>
{: .hands_on}
</div>

<div class="SCTransform" markdown="1">
><hands-on-title>SCTransform</hands-on-title>
>
> 1. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Complete all preprocessing with "SCTransform"`
>        - *"Genes to calculate residual features for"*: `all genes`
>            - *"How to set variable features"*: `set number of variable features`
>        - *"Output list of most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>        - *"Variable(s) to regress out"*: `percent.mt`
>
> 2. Rename the output as `Preprocessed Data`
>
>    If we inspect this dataset, we can see that a new assay called `SCT` has been created and is set as the default assay. Click on the {% icon galaxy-eye %}- of the following output to check this.
>
> 3. {% tool [Seurat Data Management](toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Preprocessed Data` (output of **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Inspect Seurat Object`
>        - *"Display information about"*: `General`
>
{: .hands_on}
</div>

### Visualise Highly Variable Genes

><hands-on-title>Visualise Results</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Preprocessed Data` (output of **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Plot Variable Genes with 'VariableFeaturePlot'`
>        - *"Label the top most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>
{: .hands_on}

<div class="Separate-Preprocessing-Steps" markdown="1">

![Most genes are grouped together at the bottom of the plot with low standardized variance. The selected variable genes are highlighted in red and labelled by name. They have higher standardized variances although most are still quite close to the non variable genes. The labels read PPBP, S100A9, LYZ, IGLL5, GNLY, FTL, PF4, FTH1, GNG11, and FCER1A.](../../images/scrna-seurat-pbmc3k/seurat_variable_genes.png "Plot showing the standardized variances of variable and non-variable genes. The top 10 most variable genes are labelled.")

</div>

<div class="SCTransform" markdown="1">

![Most genes are closely grouped together, right at the bottom of the plot with low standardized variance. The selected variable genes are highlighted in red and labelled by name. They are higher up on the plot, showing they have higher standardized variances although most are still quite close to the non variable genes. They are S100A9, GNLY, LYZ, S100A8, NKG7, FTL, GZMB, IGLL5, FTH1, and CCL5.](../../images/scrna-seurat-pbmc3k/seurat_variable_genes_SCT.png "Plot showing the standardized variances of variable and non-variable genes. The top 10 most variable genes are labelled.")

</div>

> <question-title></question-title>
> 1. What are the top 10 most variable genes in this dataset?
> 2. Why are we interested in these genes?
>
> > <solution-title></solution-title>
> >
> > 1. We can check the list of the top variable genes in our history by clicking on the {% icon galaxy-eye %} or see them on our variable features plot.
> >    <div class="Separate-Preprocessing-Steps" markdown="1">
> >
> >    |    |         |
> >    |----|---------|
> >    | 1  | PPBP    |
> >    | 2  | LYZ     |
> >    | 3  | S100A9  |
> >    | 4  | IGLL5   |
> >    | 5  | GNLY    |
> >    | 6  | FTL     |
> >    | 7  | PF4     |
> >    | 8  | FTH1    |
> >    | 9  | GNG11   |
> >    | 10 | S100A8  |
> >    {: .matrix}
> >
> >    </div>
> >
> >    <div class="SCTransform" markdown="1">
> >
> >    |    |          |
> >    |----|----------|
> >    | 1  | S100A9   |
> >    | 2  | GNLY     |
> >    | 3  | LYZ      |
> >    | 4  | S100A8   |
> >    | 5  | NKG7     |
> >    | 6  | FTL      |
> >    | 7  | GZMB     |
> >    | 8  | IGLL5    |
> >    | 9  | FTH1     |
> >    | 10 | CCL5     |
> >    {: .matrix}
> >
> >    </div>
> >
> >    > <comment-title></comment-title>
> >    > The list of highly variable genes you end up with will depend on which preprocessing route you chose. The two preprocessing routes use different methods to select features, so they won't always end up with the same genes, although there are likely to be some similarities. `SCTransform` also returns 3000 variable genes by default, rather than the 2000 selected by `FindVariableFeatures`. We can select more features with `SCTransform` because its normalisation method is better at removing technical effects from the data, so we believe that these additional genes reflect subtler biological variations rather than technical differences.
> >    {: .comment}
> >
> > 2. Single cell datasets contain a lot of information, including expression data for thousands of different genes. Some of these genes don't tell us much about the data, for example they might be housekeeping genes that are expressed at similar levels in most of our cells. We want to find the genes that can tell us most about the differences between our cells, so we want to identify the genes whose expression varies most across the dataset - focusing on these highly variable features should help us to uncover the biological differences we're looking for.
> >
> {: .solution}
{: .question}

# Dimensionality Reduction

Dimensional reduction is a key step in single cell analysis that simplifies our big, complex datasets enough to enable us to perform and understand further analyses.

In single cell analysis, we're comparing cells based on their patterns of gene expression - we're looking for cells with similar transcriptomic profiles. Each gene represents a dimension of the data. If we only found two genes being expressed in our cells, we could represent them on a 2-dimensional plot. The two axes of the plot would represent the expression levels of the two genes. Each cell would become a point on the plot, positioned according to its expression of these genes. We might spot one group of cells with high levels of both genes and another group that expresses large amounts of one gene and not much of the other.

For datasets with thousands of genes, the concept is essentially the same: each cell's expression profile defines its location in the high-dimensional expression space. Each gene is still a dimension or axis on the plot and each cell can be positioned along these axes based on their expression. It's just much harder to imagine a plot with thousands of different axes - that's why we need dimensional reduction!

We don't have to think of the expression of each of our thousands of genes individually. The expression of some genes can be correlated if they are affected by the same biological process - cells that express a lot of one gene might also express higher levels of other genes that have similar functions. Instead of keeping the information from each of these genes as separate axes or dimensions, we can compress them into a single dimension (e.g. an 'eigengene'). We can do the same for all of the correlated groups of genes in our data, going from thousands of individual genes to perhaps 10, 20, or 50 grouped dimensions.

 Dimensionality re


