# tutorials

topic: single-cell
tutorial: GO-enrichment
title: GO Enrichment Analysis on Single-Cell RNA-Seq Data
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["single cell", "GO enrichment"]
content_hash: 95e63b93df9d4805e9fa26ab3c6df33d
last_modified: 2026-05-27T09:01:49.737385
zenodo_link: https://zenodo.org/records/13461890
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/GO-enrichment/tutorial.html

content:
In the tutorial [Filter, plot and explore single-cell RNA-seq data with Scanpy]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}), we took an important step in our single-cell RNA sequencing analysis by identifying marker genes for each of the clusters in our dataset. These marker genes are crucial, as they help us distinguish between different cell types and states, giving us a clearer picture of the cellular diversity within our samples.
However, simply identifying marker genes is just the beginning. To truly understand what makes each cluster unique, we need to dive deeper into the biological functions these genes are involved in. This is where Gene Ontology (GO) enrichment analysis comes into play.
We will perform GO enrichment analysis as a type of over-representation analysis (ORA). ORA is a statistical method that determines whether genes from pre-defined sets (e.g. genes belonging to a specific GO term) are expressed more than would be expected in a subset of your data. The most commonly used statistical tests are Fischer's exact test and hypergeometric test, more details about them are explained in the [tutorial slides]({% link topics/single-cell/tutorials/GO-enrichment/slides.html %}).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data description

In this tutorial will use the following datasets:

## [A] Marker Genes:

We'll start with two input datasets of marker genes (Study sets):
  * **Marker genes per cell cluster:** This dataset lists the genes that are differentially enriched in each cell cluster.
  * **Marker genes per condition (wt and ko):** This dataset lists the genes that are differentially enriched between the wild-type (wt) and knockout (ko) conditions.


*Note:* Marker genes were obtained using Scanpy FindMarkers tool. The top 50 marker genes were included in the downstream GO enrichment analysis. Scanpy FindMarkers tool selects the marker genes based on their log2 fold change and p-values. Focusing on the top-ranked genes helps to filter out less relevant genes, thereby addressing the concern of high false positives that can arise from traditional methods.


## [B] GO Enrichment Files:

We'll also use three additional files for GO enrichment analysis.
  * **Gene Ontology file:** This file contains information about Gene Ontology terms.
  * **GO Annotations file:** This file maps genes to their corresponding GO terms.
  * **Population set file:** This file provides a list of all genes involved in the experiment and is used as a background gene set for the analysis.

*Note:* There are several online databases available for downloading GO and GO Annotations files, including the Gene Ontology website, Ensembl, and the UCSC Genome Browser.

>  <comment-title>Concept behind GO Enrichment Analysis</comment-title>
>
>  The goal of GO enrichment analysis is to interpret the biological significance of long lists of **marker genes** by summarizing these genes into a shorter list of enriched **GO terms**. The analysis works by comparing each GO term between your list of marker genes and a **background gene set**. Statistical tests are then used to calculate a p-value that indicates whether a particular GO term is significantly enriched in the marker gene list compared to the background.
{: .comment}

# Get data

You can access the data for this tutorial in multiple ways:

1. **Importing from a history** - You can import [this history](https://usegalaxy.eu/u/mennagamal/h/go-enrichment-analysis-on-scrna-seq-data)

   {% snippet faqs/galaxy/histories_import.md %}

2. **Uploading from Zenodo** (see below)

> <hands-on-title> Data Upload from Zenodo </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Galaxy3-[GO].obo
>    {{ page.zenodo_link }}/files/Galaxy2-[GO_annotations_Mus_musculus].tabular
>    {{ page.zenodo_link }}/files/Galaxy5-[Markers_-_clusters].tabular
>    {{ page.zenodo_link }}/files/Galaxy4-[Background_gene_set].tabular
>    {{ page.zenodo_link }}/files/Galaxy1-[Markers_-_genotype_].tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

## Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}


# Data processing

To perform GO enrichment analysis on each cell cluster individually, we need to separate our "Markers_clusters Dataset" into seven files, one for each cluster and the "Markers_genotype Dataset" into 2 files, one for each condition. We'll use the "Split file" tool for this step.

> <hands-on-title>File splitting</hands-on-title>
>
> 1. {% tool [Split file](toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.4) %} with the following parameters:
>    - {% icon param-file %} *"File to select"*: `Markers_cluster` (Input dataset)
>    - *"on column"*: `c1`
>    - *"Include the header in all splitted files?"*: `Yes`
>
>
>    > <comment-title> Input Dataset </comment-title>
>    >
>    > As we have two datasets, one with the marker genes for all the seven clusters and one with the marker genes for the knockout (KO) and wild-type (WT) conditions. Make sure to repeat the analysis twice for the two different datasets. Alternatively, you can run this [workflow](https://usegalaxy.eu/u/mennagamal/w/unnamed-workflow) for parallel analysis of the datasets, under Marker genes choose the second icon to select multiple datasets as shown in the below image.
>    {: .comment}
> ![Multiple input datasets workflow](../../images/workflow.png "Multiple input datasets workflow")
>
{: .hands_on}

Next, we need to isolate the Ensembl gene IDs column from each file. We'll use the "Cut Columns" tool to achieve this.

> <hands-on-title> Extract Ensembl IDs </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c4`
>    - {% icon param-file %} *"From"*: `split_output` (output of **Split file** {% icon tool %})
>
>
>    > <comment-title> The gene format to use </comment-title>
>    >
>    > In this example we extract column 4 because it contains the Ensembl gene IDs on which the subsequent steps are ideally working. While there are other gene formats like gene symbols, Entrez gene IDs, and more, make sure to check the specific format accepted by the tool you are using. There are also tools available to convert between different gene formats if needed.
>    {: .comment}
>
{: .hands_on}


# GO Analysis using **GOEnrichment** tool

Now we will perform the GO Enrichment analysis on the list of ensembl gene IDs.

> <hands-on-title> GOEnrichment </hands-on-title>
>
> 1. {% tool [GOEnrichment](toolshed.g2.bx.psu.edu/repos/iuc/goenrichment/goenrichment/2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO` (Input dataset)
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus` (Input dataset)
>    - {% icon param-file %} *"Study Set File"*: `out_file1` (output of **Cut** {% icon tool %})
>    - {% icon param-file %} *"Population Set File (Optional)"*: `Background gene set` (Input dataset)
>
>
>    > <comment-title> Population Set File Selection for GO Enrichment </comment-title>
>    >
>    > When choosing a background for GO enrichment analysis (Population Set File), it's important to consider the context of your data. While using a broad background (like all genes in the organism) is common, it might be more informative to limit the background to genes expressed in the specific tissue or cell type being profiled. In this tutorial we used only genes involved in the experiment before selecting the marker genes.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Take a look at the enriched terms for the different clusters, Can you find any GO terms that are specific to cluster 7?
> 2. Can we perform manual annotation of cluster 7 based on GO enrichment results?
>
> > <solution-title></solution-title>
> >
> > 1. Cluster 7 is enriched for terms like "regulation of cell death", "T cell-mediated cytotoxicity", and "peptidase activator activity involved in the apoptotic process".
> > 2. By looking at the most enriched functions and using our biological knowledge, we can figure out the cell types for many clusters. For example, since the data comes from thymus tissue, we already have an idea of the cell types we might find. The enriched terms in cluster 7 confirm that the cell type is macrophages, which support thymocyte maturation by cleaning up dead cells and debris.
> >
> {: .solution}
>
{: .question}

# GO Analysis using **gProfiler GOSt** tool

The gProfiler GOSt (Gene Ontology Sequential Testing) is another popular tool used to perform gene ontology (GO) enrichment analysis. In addition to providing enrichment results for the standard GO categories of Biological Process (BP), Cellular Component (CC), and Molecular Function (MF), the tool also analyzes enrichment across several other functional annotation databases, including KEGG Pathways, Reactome Pathways, WikiPathways and TF Targets. It also gives a plot to better visualize the results.

> <hands-on-title> gProfiler GOSt </hands-on-title>
>
> 1. {% tool [gProfiler GOSt](toolshed.g2.bx.psu.edu/repos/iuc/gprofiler_gost/gprofiler_gost/0.1.7+galaxy11) %} with the following parameters:
>    - {% icon param-file %} *"Input is whitespace-separated list of genes, proteins, probes, term IDs or chromosomal regions."*: `out_file1` (output of **Cut** {% icon tool %})
>    - *"Organism"*: `Common organisms`
>        - *"Common organisms"*: `Mus musculus (Mouse)`
>    - In *"Tool settings"*:
>        - *"Export plot"*: `Yes`
>
>
>    > <comment-title> Picking the right species matter </comment-title>
>    >
>    > The species you select should match the species your genes come from. If you choose the wrong species, the tool might use incorrect information, leading to inaccurate results. For example, human genes behave differently from mouse genes, so selecting the correct species ensures the analysis is relevant to your data.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. Can you find enriched GO terms that are inline with the [published study](https://www.frontiersin.org/journals/immunology/articles/10.3389/fimmu.2018.02523/full) findings in KO results file?
> 2. What might be happening to the stem cells in the KO mice compared to the WT mice?
>
> > <solution-title></solution-title>
> >
> > 1. In the KO g:GOSt result file, enrichment for the GO term "Negative regulation of stem cell differentiation" is found.
> > 2. This suggests that the KO condition is causing a delay in the differentiation of stem cells into mature T cells in the thymus which is inline with the study findings.
> >
> {: .solution}
>
{: .question}

# Conclusion

In this tutorial, we have performed GO enrichment analysis on the differentially expressed genes between 2 different conditions and between different cell types. This analysis provided valuable insights into the biological processes, molecular functions, and cellular components associated with the gene sets, enhancing our understanding of the underlying mechanisms involved in the studied conditions.

tools_json:
["Split file", "toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.4", "Cut", "Cut1", "GOEnrichment", "toolshed.g2.bx.psu.edu/repos/iuc/goenrichment/goenrichment/2.0.1", "gProfiler GOSt", "toolshed.g2.bx.psu.edu/repos/iuc/gprofiler_gost/gprofiler_gost/0.1.7+galaxy11"]


topic: single-cell
tutorial: pseudobulk-analysis
title: Pseudobulk Analysis with Decoupler and EdgeR
difficulty: intermediate
time_estimation: 4H
requirements_json: ["type: internal", "scrna-scanpy-pbmc3k"]
tags_json: ["transcriptomics", "pseudobulk"]
content_hash: b19226b904c1c85290f414bcd1d3edca
last_modified: 2026-05-27T09:01:49.738964
zenodo_link: https://zenodo.org/records/15275834
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/pseudobulk-analysis/tutorial.html

content:
Pseudobulk analysis is a powerful technique that bridges the gap between single-cell and bulk RNA-seq data. It involves aggregating gene expression data from groups of cells within the same biological replicate, such as a mouse or patient, typically based on clustering or cell type annotations ({% cite Murphy2022 %}).

A key advantage of this approach in differential expression (DE) analysis is that it avoids treating individual cells as independent samples, which can underestimate variance and lead to inflated significance or overly optimistic p-values ({% cite Squair2021 %}). This occurs because cells from the same biological replicate are inherently more similar to each other than cells from different samples. By grouping data into pseudobulk samples, the analysis aligns with the experimental design, as in bulk RNA-seq, leading to more reliable and robust statistical results ({% cite Murphy2022 %}).

Beyond enhancing statistical validity, pseudobulk analysis enables the identification of cell-type-specific gene expression and functional changes across biological conditions. It balances the detailed resolution of single-cell data with the statistical power of bulk RNA-seq, providing insights into the functional transcriptomic landscape relevant to biological questions. Overall, for DE analysis in multi-sample single-cell experiments, pseudobulk approaches demonstrate superior performance compared to single-cell-specific DE methods ({% cite Squair2021 %}).

In this tutorial, we will guide you through a pseudobulk analysis workflow using the **Decoupler** ({% cite Badia-iMompel2022 %}) and **edgeR** ({% cite Liu2015 %}) tools available in Galaxy. These tools facilitate functional and differential expression analysis, and their output can be integrated with other Galaxy tools to visualize results, such as creating Volcano Plots, which we will also cover in this tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get the Data!

## Overview of the Data

The dataset used in this tutorial is an `AnnData` object containing a subset of single-cell RNA-seq data, specifically including non-classical monocytes (NCMs) and plasmacytoid dendritic cells (pDCs) from bone marrow and peripheral blood samples. This subset was selected to illustrate differential gene expression between two immune cell types across two biologically distinct tissues. A portion of the data corresponding to bone marrow was derived from a published study {% cite rettkowski2025modulation %}.

In this tutorial, we will focus on these two cell populations and explore how their expression profiles differ between bone marrow and blood using pseudobulk analysis. The dataset includes three bone marrow samples and seven blood samples.

Pseudobulk analysis is an advanced approach in single-cell data analysis. It involves aggregating single-cell expression data by group (e.g., by cell type and condition) to perform bulk-style analyses while preserving single-cell resolution through metadata.

For this tutorial, we assume familiarity with standard single-cell data formats, such as `AnnData` or `Seurat` objects, and prior experience with single-cell workflows, particularly clustering and cell type annotation. We will use the `decoupler` tool to perform the pseudobulk aggregation and downstream differential analysis {% cite decoupler-pseudobulk %}.

If you're new to these concepts, we recommend exploring the following tutorials before performing pseudobulk analysis:

- [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}): Learn how to perform clustering and cell type annotation using Galaxy's Scanpy and Anndata tool suites.
- [Clustering 3K PBMCs with Seurat]({% link topics/single-cell/tutorials/scrna-seurat-pbmc3k/tutorial.md %}): Explore an alternative approach to clustering and annotation using Seurat within Galaxy.
- [Combining single-cell datasets after pre-processing]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md %}): Learn how to merge multiple single-cell datasets into one AnnData object and enrich it with experimental metadata.

The data object, which you will import from Zenodo into Galaxy via the provided link, has been preprocessed, analysed, and annotated. It includes the following key observation metadata:
- **annotated**: The assigned cell type.
- **condition**: The experimental condition or sample group.
- **batch**: The sample identifier.
- **tissue**: Indicates the tissue of origin (bone marrow or peripheral blood).

## Data Upload

> <hands-on-title>Upload your data</hands-on-title>
>
> 1. Create a new history for this tutorial and name it "Pseudobulk DE Analysis with edgeR"
> 2. Import the AnnData file from [Zenodo]({{page.zenodo_link}}):
>
>    ```
>    {{ page.zenodo_link }}/files/ncm_pdcs_subset.h5ad
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the dataset: "**AnnData for Pseudobulk**."
> 4. Ensure that the datatype is correct. It should be an AnnData object (`h5ad`).
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

Before we dive into the analysis, lets inspect our `AnnData file` with tools in galaxy to get familiar with the structure of the data.

> <hands-on-title> Inspect AnnData Object </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} tool with the following parameters:
>     - {% icon param-file %} **Annotated data matrix**: `AnnData for Pseudobulk` (Input dataset obtained > from Zenodo)`
>     - *"What to inspect?"*: `General information about the object`
>       
>    > <question-title></question-title>
>    > 1. How many cells and genes are there in the `AnnData`? 
>    > 2. What other information can be retrive by the `Inspect AnnData` tool? 
>    >
>    > > <solution-title></solution-title>
>    > > 1.  
>    > > ```
>    > > [n_obs x n_vars]
>    > > 9118 × 18926
>    > > ```
>    > > There are 9,118 observations, representing cells, and 18,926 variables, representing genes expressed in total.
>    > > 2. When selecting **General Inspection**, you can view the labels for all entries in `[obs]`, `[var]`, `[obsm]`, `[varm]`, and `[uns]`. This includes the labels described above within the observation field, such as `annotated`, `condition`, `batch`, and `tissue`, which will be needed as inputs for the pseudobulk tool.
>    > > To view more specific details in the AnnData object, select a different parameter under *"What to inspect?"*.
>    > {: .solution}
>    > 
>    {: .question}
>
{: .hands_on}

# Generation of the Pseudobulk Count Matrix with Decoupler

In this step, our goal is to perform a "bioinformatic cell sorting" based on the annotated clusters of the single-cell data.

To start a pseudobulk analysis, ensure that the `AnnData` object contains all the necessary metadata for ***pseudobulking***. This includes key annotations such as `annotated` or `cell_type`, `condition`, `disease`, and `batch`. These labels may vary depending on how the dataset was originally annotated.

Most importantly, the `AnnData` object must include a layer containing the raw gene expression counts. This layer is essential for accurate aggregation and downstream differential expression analysis. Raw counts are allows to generate accurate pseudobulk aggregates. Since single-cell data is typically normalized after annotation, it’s important to preserve the raw counts in the AnnData object before normalization steps if you would like to perform a pseudobulk analysis later on. These raw counts are directly used by tools like **Decoupler** to generate the pseudobulk count matrix. Note that normalized count matrices should not be used with **Decoupler**, even if the tool appears to process them successfully.

> <tip-title> Missing Raw Counts? </tip-title>
>
> If your AnnData object lacks raw counts, you can use the **[AnnData Operations](https://toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0)** tool on the **[Single Cell Galaxy instance](https://singlecell.usegalaxy.eu)**. This tool allows you to copy the `.X` matrix into a new layer that you may assign a new label, for e.g., `counts` or `raw_counts`, making it available as a parameter for Decoupler.
>
> Importantly, ensure that the copying of this matrix as a raw counts layer is done carefully and correctly. To verify that your AnnData object contains the necessary raw counts layer, you can use the **[Inspect AnnData Object](https://toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0)** tool. This tool helps confirm the presence of raw counts and other essential metadata in your AnnData object before proceeding with pseudobulk analysis.
>
{: .tip}

> <hands-on-title> Decoupler Pseudobulk </hands-on-title>
>
> 1. {% tool [Decoupler pseudo-bulk](toolshed.g2.bx.psu.edu/repos/ebi-gxa/decoupler_pseudobulk/decoupler_pseudobulk/1.4.0+galaxy8) %} tool with the following parameters:
>     - {% icon param-file %} **Input AnnData file**: `AnnData for Pseudobulk` (Input dataset obtained > from Zenodo)
>     - *"Produce a list of genes to filter out per contrast?"*: `No`
>     - *"Obs Fields to Merge"*: *(Leave empty if not applicable)*
>     - *"Groupby column"*: `annotated` (Column containing cell type annotations)
>     - *"Sample Key column"*: `batch` (Column containing individual sample identifiers)
>     - *"Layer"*: `raw_counts` (Layer containing raw gene expression counts)
>     - *"Factor Fields"*: `tissue` (Column in `adata.obs` specifying experimental factors. For edgeR, the first field should be the main contrast field, followed by  covariates.)
>     - *"Use Raw"*: `No`
>     - *"Produce AnnData with Pseudo-bulk"*: *(Optional, if yes, it will generate an h5ad output file with pseudobulks)*
>     - *"Minimum Cells:*: `10`
>     - *"Produce plots:*: `Yes`
>     - *"Minimum Counts"*: `10`
>     - *"Minimum Total Counts"*: `1000`
>     - *"Minimum Counts Per Gene Per Contrast Field"*: `20` (Genes with fewer counts in specific contrasts are flagged in a separate file but not excluded from the results.)
>     - *"Enable Filtering by Expression"*: `Yes`
>     - *"Plot Samples Figsize"*: `13 13`
>     - *"Plot Filtering Figsize"*: `13 13`
>
>    > <comment-title> Performing DEG within Clusters </comment-title>
>    >
>    > **Important!** The count matrix retrieved from this tool includes all of our samples `batch` aggregated by `annotated`, which can be identified by the column headers. If you want to perform comparisons of conditions within clusters of each individual cell type, you will need to subset the relevant columns of the matrix and use them as your new count matrix. We will demonstrate this in the last section of this tutorial with detailed hands-on steps.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many outputs does the Decoupler tool generate?
> 2. What do the outputs represent? What is the interpretation of the plots?
> 3. Which output(s) will we use in edgeR for differential expression analysis?
>
> > <solution-title></solution-title>
> >
> > 1. The Decoupler tool generates multiple outputs, including:
> >    - **Pseudobulk Count Matrix** (tabular file)
> >    - **Samples Metadata (Factor file)** (tabular file)
> >    - **Genes Metadata** (tabular file)
> >    - **Pseudobulk Plot** (PNG format)
> >    - **Filter by Expression Plot** (PNG format)
> >    - **Genes to Ignore by Contrast Field** (tabular file)
> >    - **Pseudobulk AnnData file** (If chosen, h5ad file)
> >
> > 2. The output files contain the following:
> >    - **Pseudobulk Count Matrix:** Contains the raw count aggregates for each pseudobulk sample.
> >    - **Samples Metadata (Factor File):** Provides metadata annotated for each sample, including factors or annotations added to the AnnData object.
> >    - **Genes Metadata:** Includes gene-related information such as gene symbols, Ensembl IDs, dispersion values, etc.
> >    - **Genes to Ignore:** Lists of genes that could be excluded or should be carefully considered for specific contrasts.
> >      This file contains a contrast field and the corresponding genes written as gene symbols.
> >    - **Pseudobulk Plot:** A visual representation of the pseudobulk data.
> >      ![Pseudobulk Plot Example](../../images/pseudobulk-analysis/Decoupler_Pseudobulk_Plot.png)
> >    - **Filter by Expression Plot:** Illustrates the expression filtering applied to the data.
> >      ![Filter by Expression Plot Example](../../images/pseudobulk-analysis/Decoupler_Filter_by_Expression_Plot.png)
> >    - **Pseudobulk AnnData file:** An AnnData file that contains the aggregated pseudobulks.
> >
> > 3. The **pseudobulk count matrix** is the primary input required for analysis using **edgeR**, a tool designed for differential expression analysis.
> >    The **Samples Metadata** is another file that will serve as an input for the edgeR tool.
> >
> {: .solution}
>
{: .question}

## Sanitation Steps - Part 1

The next steps will help you refine your data for easier handling. We will use standard galaxy tools, like: [Replace Text](https://toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1), [Remove columns](https://toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) and [Text reformatting](https://toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1).

> <hands-on-title>Replace Text and Remove Columns</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `count_matrix` (output of **Decoupler pseudo-bulk** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `[ --+*^]+`
>            - *"Replace with:"*: `_`
>
> 2. {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file"*: `genes_metadata` (output of **Decoupler pseudo-bulk** {% icon tool %})
>    - In *"Select Columns"*:
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `start`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `end`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `width`
>
> 3. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `samples_metadata` (output of **Decoupler pseudo-bulk** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `[ --+*^]+`
>            - *"Replace with:"*: `_`
>
> 4. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `outfile` (output of **Replace Text: Sample Metadata Step** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"in column"*: `c2`
>            - *"Find pattern"*: `^([0-9])(.+)`
>            - *"Replace with"*: `GG_\\1\\2`
>
> In the previous steps, the following modifications were made to the files:
>
> 1. **Replace Text**: Replaces special characters (`[ --+*^]+`) with underscores (`_`) in the `count_matrix` file. Rename output to **Count matrix**.
> 2. **Remove Columns**: Removes the `start`, `end`, and `width` columns from the `genes_metadata` file. Rename output to **Genes Metadata**.
> 3. **Replace Text**: Replaces special characters (`[ --+*^]+`) with underscores (`_`) in the `samples_metadata` file.
> 4. **Replace Text in Column**: Modifies values in column `c2` of `outfile from step 3` by prefixing `GG_` to numbers at the beginning of the string. Rename output to **Samples Metadata**
>
>    {% snippet faqs/galaxy/analysis_regular_expressions.md %}
>
{: .hands_on}


## Generating the Contrast File

This file will be used as the contrast input file in the edgeR tool.

> <hands-on-title> Creating a Contrast File for edgeR </hands-on-title>
>
> 1. {% tool [Text Reformatting](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Samples Metadata` (output generated from the **Replace Text** {% icon tool %} step).
>    - *"AWK Program"*:
>      ```BEGIN { print "header" }
>         NR > 1 { if (!seen[$2]++) words[++count]=$2 }
>         END { for (i=1; i<=count; i++)
>             for (j=i+1; j<=count; j++)
>               print words[i]"-"words[j] }
>      ```
>
> Rename to **Contrast File** 
>
>    > <comment-title> Explanation of the AWK Program </comment-title>
>    >
>    > This AWK script performs the following:
>    > - Initializes the `header` variable for the output file.
>    > - Processes the input file (`NR > 1` skips the header line).
>    > - Tracks unique elements in column 2 (`seen[$2]++`) to avoid duplicates.
>    > - Creates pairs of unique elements and calculates their difference, outputting the contrast for downstream edgeR analysis.
>    > - **Important!** When using the *"Text reformatting with awk"* tool, make sure the code is written all in one line, without any line breaks. Otherwise, it won’t work as expected.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How does your contrast file look, and what contrast is generated by the tool?
>
> > <solution-title></solution-title>
> >
> > 1. The contrast file is a simple tab-delimited text file. It contains:
> >    - A **header** in the first row that labels the column.
> >    - The contrast, such as `bonemarrow-pbmcs`, written in the second row.
> >
> > When working with your own data, the contrast file will look different, as it will reflect the specific contrasts in your dataset but the **header** should be the same
> >
> {: .solution}
>
{: .question}


# Differential Expression Analysis (DE) with edgeR

The tool **edgeR** is commonly used for differential expression (DE) analysis of pseudobulk aggregates in single-cell RNA-seq studies ({% cite Chen2024 %}). It calculates gene dispersions based on the total counts for each gene and adjusts these estimates using an approach called empirical Bayes. This method improves reliability by pooling information across all genes in the dataset to stabilize dispersion estimates, particularly for genes with low counts or limited data. By "borrowing strength" from the behavior of other genes, empirical Bayes reduces the risk of overestimating or underestimating variability, ensuring more robust and accurate results ({% cite Chen2024 %}). Differential expression is then evaluated using an exact test for the negative binomial distribution, which is similar to Fisher’s exact test but modified to handle the variability typically seen in RNA-seq data ({% cite Chen2024 %}).

In addition to exact tests, edgeR employs generalized linear models (GLMs), which allow for the analysis of more complex experimental designs. GLMs model gene counts as a function of experimental conditions (e.g., treatment groups or time points) and estimate how these conditions influence gene expression. Likelihood-based methods, such as quasi-likelihood (QL) approaches, are central to this framework. Standard likelihood methods evaluate the fit of the model to the data, while quasi-likelihood methods add an extra layer by explicitly accounting for biological and technical variability, improving the reliability of DE results. These models allow edgeR to identify subtle expression differences while controlling for overdispersion in the data ({% cite Chen2016 %}).

Several plots can be generated to assist in understanding the data and the results of the analysis, including Multidimensional scaling (MDS), Biological Coefficient of Variation plot (BCV), Quasi-Likelihood (QL), and Mean-Difference plot o MA (MD) plots. These visualizations provide insights into sample relationships, variability, and differential expression, and will be explained further in the tutorial. With these concepts in mind, let's now perform our DE analysis using our edgeR tool in Galaxy!

> <hands-on-title> Run a DGE Analysis with edgeR </hands-on-title>
>
> 1. {% tool [edgeR](toolshed.g2.bx.psu.edu/repos/iuc/edger/edger/3.36.0+galaxy5) %} with the following parameters:
>    - *"Count Files or Matrix?"*: `Single Count Matrix`
>        - {% icon param-file %} *"Count Matrix"*: `Count Matrix` (output of **Replace Text: Count Matrix** {% icon tool %})
>        - *"Input factor information from file?"*: `Yes`
>            - {% icon param-file %} *"Factor File"*: `Samples Metadata` (output of **Replace Text: Creating Factor File** {% icon tool %})
>    - *"Use Gene Annotations?"*: `Yes`
>        - {% icon param-file %} *"Gene Annotations"*: `Genes Metadata` (output of **Remove Columns: Gene Metadata** {% icon tool %})
>    - *"Formula for linear model"*: `~ 0 + tissue`
>        *(Customize this formula based on your data's contrast names and factor file. Ensure the formula matches EdgeR's syntax and uses only elements from the factor file.~ 0 + factor_A` or `~ 0 + factor_A + factor_B:factor_C)*
>    - *"Input contrasts manually or through a file?"*: `file`
>        - {% icon param-file %} *"Contrasts File"*: `Contrast File` (output of **Text Reformatting: Creating a Contrast File for edgeR** {% icon tool %})
>    - In *"Filter Low Counts"*:
>        - *"Filter lowly expressed genes?"*: `No`
>
>    > <comment-title> edgeR Tool Overview </comment-title>
>    >
>    > Use this tool to perform DGE analysis using edgeR. It is highly customizable, allowing you to input your count matrix, factor information, and gene annotations prepared during earlier steps. The contrast file should be generated from the previous step, **Text Reformatting: Creating a Contrast File for edgeR**. The model formula should be defined based on your dataset and must include factors that were already specified in the contrast file.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What are the output(s) of the edgeR tool?
> 2. How can we interpret our output result file?
>
> > <solution-title></solution-title>
> >
> > 1. **Output files**:
> >    - **edgeR Tables**: This file includes columns with information such as:
> >    - **Gene Symbol**: The identifier for each gene.
> >    - **Log Fold Change (logFC)**: Represents the direction and magnitude of differential expression. Positive values indicate upregulation, and negative values indicate downregulation.
> >    - **Raw p-value (PValue)**: The unadjusted statistical significance value for each gene, indicating the likelihood of observing the result under the null hypothesis.
> >    - **False Discovery Rate (FDR)**: The adjusted p-value using the Benjamini-Hochberg correction method to account for multiple testing and control the expected proportion of false positives.
> >    - **(Optional)**: If additional outputs, such as "normalized counts," are selected in the edgeR tool, the table will also include these counts.
> >    - **edgeR Report**: This is an HTML file containing various visualizations and summaries of the analysis, such as:
> >    - **MDS Plot (Multidimensional Scaling)**: Visualizes the relationships between samples based on gene expression profiles.
> >    - **BCV Plot (Biological Coefficient of Variation)**: Displays the estimated dispersion for each gene.
> >    - **QL Plot (Quasi-Likelihood)**: Shows the quasi-likelihood dispersions for the dataset.
> >    - **MD Plot (Mean-Difference Plot)**: Compares the mean expression of genes versus their log fold change.
> >
> > 2. **Interpreting the Results**:
> >    - The table contains the results of the differential expression analysis.
> >    - The **first column** typically lists the gene symbols from the dataset.
> >    - The **logFC** indicates the direction and magnitude of differential expression:
> >    - **Upregulated genes**: Genes with higher expression in the first group of the contrast (e.g., 'bone marrow' group in a "bonemarrow-pbmcs" contrast).
> >    - **Downregulated genes**: Genes with lower expression in the first group of the contrast (e.g., 'pbmcs' group in a "bonemarrow-pbmcs" contrast).
> >    - The **raw p-value** represents the statistical significance of the result for each gene before adjustment for multiple comparisons. Lower values indicate stronger evidence against the null hypothesis.
> >    - The **FDR** is the adjusted p-value, calculated using the Benjamini-Hochberg method, which helps control for false positives when testing many genes. Genes with an FDR below a threshold (e.g., 0.05) are considered statistically significant.
> >
> > **Plot Interpretations**:
> >   - **MDS Plot**: Displays relationships between samples based on gene expression profiles. Samples that cluster closely are more similar in their expression. Use this to identify whether samples separate by biological condition or to detect potential batch effects.  ![MDS Plot](../../images/pseudobulk-analysis/mdsplot_tissue.png)
> >   - **BCV Plot**: Shows the dispersion for each gene, with higher values indicating greater variability. This is useful for assessing how variability is modeled in the dataset. ![BCV Plot](../../images/pseudobulk-analysis/bcvplot.png)
> >   - **QL Plot**: Highlights the quasi-likelihood dispersions, which represent variability modeled during statistical testing. Proper dispersion modeling ensures robust differential expression analysis. ![QL Plot](../../images/pseudobulk-analysis/qlplot.png)
> >   - **MD Plot**: Visualizes the mean expression levels against log fold change for each gene. Genes far from the center indicate stronger differential expression, with points above or below the horizontal line showing upregulated or downregulated genes, respectively.  ![MD Plot](../../images/pseudobulk-analysis/mdplot_bonemarrow-pbmcs.png)
> >
> {: .solution}
>
{: .question}

## Sanitation Steps - Part 2

After performing the differential expression analysis with edgeR, we will clean the data to prepare it for visualization. This involves extracting dataset from a collection, removing unnecessary columns, standardizing text, and splitting the file if needed. We will use the {% tool [Extract dataset](__EXTRACT_DATASET__) %}, {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %}.

> <hands-on-title></hands-on-title>
>
>
> 1. {% tool [Extract dataset](__EXTRACT_DATASET__) %} with the following parameters:
>    - {% icon param-file %} *"Input list"*: `Tables from edgeR data` (output from the **edgeR** tool {% icon tool %})
>
>    **Extract dataset** will allow us extract the table output from **edgeR** which was retrived as a collection.  
>
> 2. {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file"*: `outTables` (output of **edgeR** {% icon tool %})
>    - In *"Select Columns"*:
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `X` (In other datasets, the header of the _GeneID_ columns may look different)
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `logFC`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `PValue`
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `FDR`
>    - *"Keep named columns"*: `Yes`
>    - *"Rename output file to"*: `edgeR_DEG_summary`
>
>    **Remove columns** to filter out unnecessary columns from the **edgeR** output and retain only the essential ones for analysis: Gene ID (`id`), Log Fold Change (`logFC`), P-value (`PValue`), and False Discovery Rate (`FDR`).
>
{: .hands_on}

# Volcano Plot

In this step, we will use the sanitized output from the previous steps to generate a Volcano Plot, which visualizes the relationship between statistical significance (P-value) and fold change (LogFC) for differentially expressed genes (DEGs). The input file for the Volcano Plot must include four essential columns: _FDR (adjusted P-value), P-value (raw), Log Fold Change, and Gene Symbols (Labels)._ As long as these columns are present, the Volcano Plot can be generated successfully.

> <hands-on-title> Create a Volcano Plot of the DEG </hands-on-title>
>
> 1. {% tool [Volcano Plot](toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.6) %} with the following parameters:
>    - {% icon param-file %} *"Specify an input file"*: `edgeR_DEG_summary` (tabular file output from **Remove Columns** {% icon tool %})
>    - *"File has header?"*: `Yes`
>    - *"FDR (adjusted P value) column number"*: `c4`
>    - *"P value (raw) column number"*: `c3`
>    - *"Log Fold Change column number"*: `c2`
>    - *"Labels column number"*: `c1`
>    - *"LogFC threshold to colour"*: `0.58`
>    - *"Points to label"*: `Significant`
>        - *"Only label top most significant"*: `40`
>    - In *"Plot Options"*:
>        - *"Plot title"*: `Differential Expression Volcano Plot`
>
>    > <tip-title> What does the Volcano Plot show? </tip-title>
>    >
>    > The Volcano Plot highlights genes with statistical significance (low P-values) and large fold changes. Genes meeting the significance thresholds are typically colored and labeled for easier identification.
>    {: .tip}
>
{: .hands_on}

## What are the results of the Volcano Plot?

Let's take a moment to interpret the Volcano Plot:
![Volcano Plot](../../images/pseudobulk-analysis/VolcanoPlot.png)

> <question-title></question-title>
>
> 1. What is the significance of genes located at the extremes of the plot (e.g., high LogFC and low P-value)?
> 2. How many genes meet the significance threshold in this analysis?
> 3. What do the differentially expressed genes reveal about immune cells in bone marrow versus peripheral blood?
> 
> > <solution-title></solution-title>
> >
> > 1. Genes at the extremes of the volcano plot are highly significant and show strong differential expression. Investigating their biological roles may provide valuable insights into the underlying mechanisms.
> > 2. This analysis identified a substantial number of differentially expressed genes (DEGs), highlighting distinct expression profiles between immune cells from bone marrow and peripheral blood. Specifically, 188 genes were upregulated in bone marrow and 136 in PBMCs.
> > 3. These findings reveal tissue-specific immune cell behaviour, pointing to transcriptional specialization between bone marrow and peripheral blood compartments.
> >
> {: .solution}
>
{: .question}

# Subsetting Samples from the Original AnnData Object

In our initial analysis, we observed transcriptional differences between bone marrow and peripheral blood samples. However, since the dataset includes more than one immune cell type (pDCs and NCMs), it's not immediately clear which population is primarily driving these differences.

To gain further insight, it's helpful to perform pseudobulk analysis on each cell type separately. By subsetting the data by cluster and repeating the differential analysis (as we did with the full dataset at the start of this tutorial), we can better assess the individual contribution of each cell type.

In the following step, we’ll demonstrate how to subset the AnnData object by cell type before reapplying the pseudobulk workflow. In the hands-on example, we will filter for pDCs. To extract NCMs instead, you can follow the exact same steps, but set the *"Value"* parameter to `Non_Classical_Monocyte`, which corresponds to the annotation of that cell type.

If you would like to extract all annotated clusters at once, for example to analyse each of them independently, refer to the tip box below titled **“Split AnnData object by cluster or other observation key into a collection”**.

## Extracting observations of interest as AnnData object

> <hands-on-title>Use Manipulate AnnData Tools to extract observations</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - *"Annotated data matrix"*: `AnnData for Pseudobulk` (your preprocessed, analysed, and annotated AnnData object)
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>    - *"What to filter?"*: `Observations (obs)` (select this to filter cells)
>    - *"Type of filtering?"*: `By key (column) values`
>    - *"Key to filter"*: `annotated` (the column that contains the cell type annotations)
>    - *"Type of value to filter"*: `Text`
>    - *"Filter"*: `equal to`
>    - *"Value"*: `pDC` (the cluster name for the cell type you want to extract)
>
>    > <comment-title>Pre-extracted clusters also available</comment-title>
>    >
>    > In addition to performing the filtering yourself, the `pDCs` and `Monocytes_NC` clusters have also been pre-extracted and are available for direct download.  
>    > This allows you to jump ahead and run the downstream workflows independently for each cluster, as described later in this tutorial.  
>    >
>    > Import the pre-filtered datasets from Zenodo:
>    >
>    > ```
>    > {{ page.zenodo_link }}/files/Monocytes_NC_subset.h5ad
>    > {{ page.zenodo_link }}/files/pDCs_subset.h5ad
>    > ```
>    >
>    {: .comment}
>
{: .hands_on}

After using the {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} to extract the cell type of interest, return to the **Pseudobulk with Decoupler** step at the beginning of this tutorial. You can now repeat the same steps using this smaller AnnData object, which contains only the selected cell type (e.g., pDCs). The resulting analysis will reveal differential gene expression between conditions (e.g., COVID-19 vs. healthy) for that specific cell type only.

> <question-title></question-title>
>
> 1. What data is included in the new pseudobulk count matrix for the pDCs only? How is this matrix structured, and what do the column labels represent?
> 2. How many samples are included in the pDCs AnnData? How many of these are from bone marrow and how many from peripheral blood?
> 3. After performing pseudobulk analysis on pDCs and NCMs independently, how do the volcano plots compare? Do they reveal differentially expressed genes between tissues? If so, which cell type appears to drive the tissue-specific differences?
>
> > <solution-title></solution-title>
> >
> > 1. The new count matrix includes the original 1,373 rows, each representing a gene, with gene identifiers listed in the first column. The remaining columns correspond to individual samples, such as *bmacute_pDC* and *pbmcsacuteone_pDC*.
> > 2. This dataset includes a total of eleven samples: three from bone marrow and eight from peripheral blood.
> > 3. The analysis of pDCs and NCMs reveals distinct patterns of differentially expressed genes (DEGs) between tissues. Based on the results, NCMs show stronger transcriptional differences and appear to be the main contributors to the tissue-specific expression patterns observed in the full dataset.
> > ![Volcano Plot pDCs](../../images/pseudobulk-analysis/Volcano_plot_pDCs.png)  
> > ![Volcano Plot NCMs](../../images/pseudobulk-analysis/Volcano_plot_NCMs.png)
> >
> {: .solution}
>
{: .question}


> <tip-title>Split AnnData object by cluster or other observation key into a collection</tip-title>
>
> You can split an `AnnData` object into multiple objects **at once** based on the values of a given `obs` key using the: {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %}.
>
> This tool automatically creates a collection containing one `AnnData` object for each unique value in the selected observation key.
>
> For example, to split the dataset by cluster annotation, you can use the key `louvain`, or in the case of this tutorial, `annotated`. Each resulting element in the collection will correspond to a different cluster or cell type, which can then be analysed independently.
>
{: .tip}

## Recommendations
1. **Data Preprocessing:**
   - Ensure raw counts are preserved in your AnnData object before pseudobulk analysis.
   - Verify the presence of important metadata fields to ensure meaningful grouping and contrasts.
2. **Focus on Subsets:**
   - Analyse subsets of cell types or clusters for more granular insights. For example, pseudobulk analysis focused on specific immune cell populations can reveal subtle but important expression changes.
3. **Quality Control:**
   - Always inspect the outputs of Decoupler (e.g., pseudobulk plots and filtering summaries) to confirm data integrity and address outliers.
   - Use edgeR’s diagnostic plots (e.g., MDS, BCV) to check for batch effects or unexpected variability.
4. **Follow-Up Analysis:**
   - Consider functional analysis (e.g., GO enrichment, pathway analysis) for the differentially expressed genes to link expression changes to biological processes.
   - Use visualization tools, such as heatmaps or boxplots, to further explore expression trends in key genes.
5. **Future Applications:**
   - Pseudobulk analysis can be extended to multi-condition experiments, time-course data, or other cell-type-specific investigations.

tools_json:
["Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1", "Decoupler pseudo-bulk", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/decoupler_pseudobulk/decoupler_pseudobulk/1.4.0+galaxy8", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.3+galaxy1", "Remove columns", "toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1", "Text Reformatting", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_awk_tool/9.3+galaxy1", "edgeR", "toolshed.g2.bx.psu.edu/repos/iuc/edger/edger/3.36.0+galaxy5", "Extract dataset", "__EXTRACT_DATASET__", "Volcano Plot", "toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.6", "Scanpy filter", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1"]


topic: single-cell
tutorial: alevin-commandline
title: Generating a single cell matrix using Alevin and combining datasets (bash + R)
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["10x", "paper-replication", "MIGHTS"]
content_hash: 0a91dcf1345c2490cace457509526ad2
last_modified: 2026-05-27T09:01:49.737730
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/alevin-commandline/tutorial.html

content:
# Setting up the environment

Alevin is a tool integrated with the [Salmon software](https://salmon.readthedocs.io/en/latest/salmon.html), so first we need to get Salmon. You can install Salmon using conda, but in this tutorial we will show an alternative method - downloading the pre-compiled binaries from the [releases page](https://github.com/COMBINE-lab/salmon/releases). Note that binaries are usually compiled for specific CPU architectures, such as the 64-bit (x86_64) machine release referenced below .

```bash
wget -nv https://github.com/COMBINE-lab/salmon/releases/download/v1.10.0/salmon-1.10.0_linux_x86_64.tar.gz
```

Once you've downloaded a specific binary (here we're using version 1.10.0), just extract it like so:

```bash
tar -xvzf salmon-1.10.0_linux_x86_64.tar.gz
```

> <details-title>Conda installation</details-title>
>
> As mentioned, installing salmon using conda is also an option, and you can do it using the following command in the terminal:
> ```
> conda install -c conda-forge -c bioconda salmon
> ```
>
> However, for this tutorial, it would be easier and quicker to use the downloaded pre-compiled binaries, as shown above.
>
{: .details}

We're going to use Alevin for demonstration purposes, but we do not endorse one method over another.

# Get Data

We continue working on the same example data - a very small subset of the reads in a mouse dataset of fetal growth restriction {% cite Bacon2018 %} (see the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)). For the purposes of this tutorial, the datasets have been subsampled to only 50k reads (around 1% of the original files). Those are two fastq files - one with transcripts and the another one with cell barcodes. You can download the files by running the code below:

```bash
wget -nv https://zenodo.org/records/10116786/files/transcript_701.fastq
wget -nv https://zenodo.org/records/10116786/files/barcodes_701.fastq
```

 > <question-title></question-title>
>
> How to differentiate between the two files if they are just called 'Read 1' and 'Read 2'?
>
> > <solution-title></solution-title>
> >
> > The file which contains the cell barcodes and UMI is significantly shorter (indeed, 20 bp!) compared to the other file containing longer, transcript read. For ease, we will use explicit file names.
> >
> {: .solution}
>
{: .question}


Additionally, to map your reads, you will need a transcriptome to align against (a FASTA) as well as the gene information for each transcript (a gtf) file. These files are included in the data import step below. You can also download these for your species of interest [from Ensembl](https://www.ensembl.org/info/data/ftp/index.html).

```bash
wget -c https://zenodo.org/record/4574153/files/Mus_musculus.GRCm38.100.gtf.gff -O GRCm38_gtf.gff
wget -c https://zenodo.org/record/4574153/files/Mus_musculus.GRCm38.cdna.all.fa.fasta -O GRCm38_cdna.fasta
```

Why do we need FASTA and GTF files?
To generate gene-level quantifications based on transcriptome quantification, Alevin and similar tools require a conversion between transcript and gene identifiers. We can derive a transcript-gene conversion from the gene annotations available in genome resources such as Ensembl. The transcripts in such a list need to match the ones we will use later to build a binary transcriptome index. If you were using spike-ins, you'd need to add these to the transcriptome and the transcript-gene mapping.

We will use the murine reference annotation as retrieved from Ensembl (*GRCm38* or *mm10*) in GTF format. This annotation contains gene, exon, transcript and all sorts of other information on the sequences. We will use these to generate the transcript-gene mapping by passing that information to a tool that extracts just the transcript identifiers we need.

Finally, we will download some files for later so that you don't have to switch kernels again - that's an Alevin output folder for another sample!

```bash
wget https://zenodo.org/records/10116786/files/alevin_output_702.zip
unzip alevin_output_702.zip
```


# Generate a transcript to gene map and filtered FASTA

We will now install the package `atlas-gene-annotation-manipulation` to help us generate the files required for Salmon and Alevin.

```bash
conda install -y -c conda-forge -c bioconda atlas-gene-annotation-manipulation
```

After the installation is completed, you can execute the code cell below and while it's running, I'll explain all the parameters we set here.

```bash
gtf2featureAnnotation.R -g GRCm38_gtf.gff -c GRCm38_cdna.fasta -d "transcript_id" -t "transcript" -f "transcript_id" -o map -l "transcript_id,gene_id" -r -e filtered_fasta
```

In essence, [gtf2featureAnnotation.R script](https://github.com/ebi-gene-expression-group/atlas-gene-annotation-manipulation) takes a GTF annotation file and creates a table of annotation by feature, optionally filtering a cDNA file supplied at the same time. Therefore the first parameter `-g` stands for "gtf-file" and requires a path to a valid GTF file. Then `-c` takes a cDNA file for extracting meta info and/or filtering - that's our FASTA! Where --parse-cdnas (that's our `-c`) is specified, we need to specify, using `-d`, which field should be used to compare to identfiers from the FASTA. We set that to "transcript_id" - feel free to inspect the GTF file to explore other attributes. We pass the same value in `-f`, meaning first-field, ie. the name of the field to place first in output table. To specify which other fields to retain in the output table, we provide comma-separated list of those fields, and since we're only interested in transcript to gene map, we put those two names ("transcript_id,gene_id") into `-l`. `-t` stands for the feature type to use, and in our case we're using "transcript". Guess what `-o` is! Indeed, that's the output annotation table - here we specify the file path of our transcript to gene map. We will also have another output denoted by `-e` and that's the path to a filtered FASTA. Finally, we also put `-r` which is there only to suppress header on output. Summarising, output will be a an annotation table, and a FASTA-format cDNAs file with unannotated transcripts removed.

Why filtered FASTA?
Sometimes it's important that there are no transcripts in a FASTA-format transcriptome that cannot be matched to a transcript/gene mapping. Salmon, for example,  used to produce errors when this mismatch was present. We can synchronise the cDNA file by removing mismatches as we have done above.


# Generate a transcriptome index

We will use Salmon in mapping-based mode, so first we have to build a Salmon index for our transcriptome. We will run the Salmon indexer as so:

```bash
salmon-latest_linux_x86_64/bin/salmon index -t filtered_fasta -i salmon_index -k 31
```

Where `-t` stands for our filtered FASTA file, and `-i` is the output of the mapping-based index. To build it, the function is using an auxiliary k-mer hash over k-mers of length 31. While the mapping algorithms will make use of arbitrarily long matches between the query and reference, the k size selected here will act as the minimum acceptable length for a valid match. Thus, a smaller value of k may slightly improve sensitivity. We find that a k of 31 seems to work well for reads of 75bp or longer, but you might consider a smaller k if you plan to deal with shorter reads. Also, a shorter value of k may improve sensitivity even more when using selective alignment (enabled via the –validateMappings flag). So, if you are seeing a smaller mapping rate than you might expect, consider building the index with a slightly smaller k.


> <details-title>What is the index?</details-title>
>
> To be able to search a transcriptome quickly, Salmon needs to convert the text (FASTA) format sequences into something it can search quickly, called an 'index'. The index is in a binary rather than human-readable format, but allows fast lookup by Alevin. Because the types of biological and technical sequences we need to include in the index can vary between experiments, and because we often want to use the most up-to-date reference sequences from Ensembl or NCBI, we can end up re-making the indices quite often.
>
{: .details}


# Use Alevin

Time to use Alevin now! Alevin works under the same indexing scheme (as Salmon) for the reference, and consumes the set of FASTA/Q files(s) containing the Cellular Barcode(CB) + Unique Molecule identifier (UMI) in one read file and the read sequence in the other. Given just the transcriptome and the raw read files, Alevin generates a cell-by-gene count matrix (in a fraction of the time compared to other tools).


> <details-title>How does Alevin work in detail?</details-title>
>
> Alevin works in two phases. In the first phase it quickly parses the read file containing the CB and UMI information to generate the frequency distribution of all the observed CBs, and creates a lightweight data-structure for fast-look up and correction of the CB. In the second round, Alevin utilizes the read-sequences contained in the files to map the reads to the transcriptome, identify potential PCR/sequencing errors in the UMIs, and performs hybrid de-duplication while accounting for UMI collisions. Finally, a post-abundance estimation CB whitelisting procedure is done and a cell-by-gene count matrix is generated.
>
{: .details}

Alevin can be run using the following command:

```bash
salmon-latest_linux_x86_64/bin/salmon alevin -l ISR -1 barcodes_701.fastq -2 transcript_701.fastq --dropseq  -i salmon_index -p 10 -o alevin_output --tgMap map --freqThreshold 3 --keepCBFraction 1 --dumpFeatures
```

All the required input parameters are described in [the documentation](https://salmon.readthedocs.io/en/latest/alevin.html), but for the ease of use, they are presented below as well:

> <details-title>Alevin input parameters</details-title>
> - `-l`: library type (same as Salmon), we recommend using ISR for both Drop-seq and 10x-v2 chemistry.
>
> - `-1`: CB+UMI file(s), alevin requires the path to the FASTQ file containing CB+UMI raw sequences to be given under this command line flag. Alevin also supports parsing of data from multiple files as long as the order is the same as in -2 flag. That's our barcodes_701.fastq file.
>
> - `-2`: Read-sequence file(s), alevin requires the path to the FASTQ file containing raw read-sequences to be given under this command line flag. Alevin also supports parsing of data from multiple files as long as the order is the same as in -1 flag. That's our transcript_701.fastq file.
>
> - `--dropseq` / `--chromium` / `--chromiumV3`: the protocol, this flag tells the type of single-cell protocol of the input sequencing-library. This is a study using the Drop-seq chemistry, so we specify that in the flag.
>
> - `-i`: index, file containing the salmon index of the reference transcriptome, as generated by salmon index command.
>
> - `-p`: number of threads, the number of threads which can be used by alevin to perform the quantification, by default alevin utilizes all the available threads in the system, although we recommend using ~10 threads which in our testing gave the best memory-time trade-off.
>
> - `-o`: output, path to folder where the output gene-count matrix (along with other meta-data) would be dumped. We simply call it alevin_output
>
> - `--tgMap`: transcript to gene map file, a tsv (tab-separated) file — with no header, containing two columns mapping of each transcript present in the reference to the corresponding gene (the first column is a transcript and the second is the corresponding gene). In our case, that's map_code generated by using gtf2featureAnnotation.R function.
>
> - `--freqThreshold` - minimum frequency for a barcode to be considered. We've chosen 3 as this will only remove cell barcodes with a frequency of less than 3, a low bar to pass but useful way of avoiding processing a bunch of almost certainly empty barcodes.
>
> - `--keepCBFraction` - fraction of cellular barcodes to keep. We're using 1 to quantify all!
>
> - `--dumpFeatures` - if activated, alevin dumps all the features used by the CB classification and their counts at each cell level. It’s generally used in pair with other command line flags.
>
{: .details}

We have also added some additional parameters (`--freqThreshold`, `--keepCBFraction`) and their values are derived from the [Alevin Galaxy tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}) after QC to stop Alevin from applying its own thresholds. However, if you're not sure what value to pick, you can simply allow Alevin to make its own calls on what constitutes empty droplets.


This tool will take a while to run. Alevin produces many file outputs, not all of which we'll use. You can refer to the [Alevin documentation](https://salmon.readthedocs.io/en/latest/alevin.html) if you're curious what they all are, you can look through all the different files to find information such as the mapping rate, but we'll just pass the whole output folder directory for downstream analysis.


> <question-title></question-title>
>
> 1. Can you find the information what was the mapping rate?
> 2. How many transcripts did Alevin find?
>
> > <solution-title></solution-title>
> >
> > 1. As mentioned above, in *alevin_output* folder there will be many different files, including the log files. To check the mapping rate, go to *alevin_output* -> *logs* and open *salmon_quant* file. There you will find not only information about mapping rate, but also many more, calculated at salmon indexing step.
> > 2. Alevin log can be found in *alevin_output* -> *alevin* and the file name is also *alevin*. You can find many details about the alevin process there, including the number of transcripts found.
> >
> {: .solution}
>
{: .question}

> <warning-title>Process stopping</warning-title>
>
> The command above will display the log of the process and will say "Analyzed X cells (Y% of all)". For some reason, running Alevin may sometimes cause problems in Jupyter Notebook and this process will stop and not go to completion. This is the reason why we use hugely subsampled dataset here - bigger ones couldn't be fully analysed (they worked fine locally though). The dataset used in this tutorial shouldn't make any issues when you're using Jupyter notebook through galaxy.eu, however might not work properly on galaxy.org. If you're accessing Jupyter notebook via galaxy.eu and alevin process stopped, just restart the kernel (in the upper left corner: *Kernel* -> *Restart kernel...*) and that should help. However, if it doesn't, here are the alternatives:
> - We prepared a Jupyter notebook with the same workflow as shown here, compatible with Google Collab. So if you have troubles with running the above step, you can use the [notebook provided](https://colab.research.google.com/drive/1nRidlpiU-_qUG1zLgpc8mJZewr_EQL0N?usp=sharing), upload it to your [Google Collab](https://colab.research.google.com/) and run the code there, following the tutorial on GTN.
> - You can download the output folder that you would get as a result of running the code above. Simply run the code below.
>
{: .warning}

```bash
# get the output folder from Alevin in case yours crashes
wget https://zenodo.org/records/13732502/files/alevin_output.zip
unzip alevin_output.zip
```

<!---
check if we can get alevinQC to work - and paste the info from the first Alevin tutorial?
-->

# Alevin output to SummarizedExperiment

Let's change gear a little bit. We've done the work in bash, and now we're switching to R to complete the processing. To do so, you have to change Kernel to R (either click on `Kernel` -> `Change Kernel...` in the upper left corner of your JupyterLab or click on the displayed current kernel in the upper right corner and change it).
![Figure showing the JupyterLab interface with an arrow pointing to the left corner, showing the option 'Kernel' -> 'Change Kernel...' and another arrow pointing to the right corner, showing the icon of the current kernel. The pop-up window asks which kernel should be chosen instead.](../../images/scrna-pre-processing/switch_kernel.jpg "Two ways of switching kernel.")


Now, when we are in the R environment, let's install the packages required for this part - `tximeta` and `DropletUtils`:

```bash
system("conda install -y bioconductor-tximeta bioconductor-dropletutils", intern=TRUE)
```
Double check that the installation went fine:
```bash
require("tximeta")
require("DropletUtils")
```

> <details-title>"There is no package called ‘tximeta’ or ‘DropletUtils’"?</details-title>
> If you encounter this error message while loading the library, you can try to install them with BiocManager, as shown below. Just replace "package_name" with either `tximeta` or `DropletUtils`. Beware, it can take up to 25 minutes to run!
> ```bash
> # run the code below only if you cannot load tximeta library (or using notebook version 1.0.1)
> if (!require("BiocManager", quietly = TRUE))
>     install.packages("BiocManager")
>
> BiocManager::install("package_name")
> library("package_name")
> ```
{: .details}


The [tximeta package](https://bioconductor.org/packages/devel/bioc/vignettes/tximeta/inst/doc/tximeta.html) created by {% cite Love2020 %} is used for import of transcript-level quantification data into R/Bioconductor and requires that the entire output of alevin is present and unmodified.

In the *alevin_output* -> *alevin* folder you can find the following files:
- *quants_mat.gz*- Compressed count matrix
- *quants_mat_rows.txt*- Row Index (CB-ids) of the matrix.
- *quants_mat_cols.txt* - Column Header (Gene-ids) of the matrix.
- *quants_tier_mat.gz* – Tier categorization of the matrix.

We will only focus on *quants_mat.gz* though. First, let's specify the path to that file:

```bash
path <- 'alevin_output/alevin/quants_mat.gz'
```

We will specify the following arguments when running *tximeta*:
- 'coldata' a data.frame with at least two columns:
  - files - character, paths of quantification files
  - names - character, sample names
- 'type' - what quantifier was used (can be 'salomon', 'alevin', etc.)

With that we can create a dataframe and pass it to tximeta to create SummarizedExperiment object.

```bash
coldata <- data.frame(files = path, names="sample701")
alevin_se <- tximeta(coldata, type = "alevin")
```

Inspect the created object:
```bash
alevin_se
```

As you can see, *rowData names* and *colData names* are still empty. Before we add some metadata,  we will first identify barcodes that correspond to non-empty droplets.

# Identify barcodes that correspond to non-empty droplets

Some sub-populations of small cells may not be distinguished from empty droplets based purely on counts by barcode. Some libraries produce multiple ‘knees’ (see the [Alevin Galaxy tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}#interpreting-the-qc-plots) for multiple sub-populations. The `emptyDrops` method ({% cite Lun2019 %}) has become a popular way of dealing with this. `emptyDrops` still retains barcodes with very high counts, but also adds in barcodes that can be statistically distinguished from the ambient profiles, even if total counts are similar.

EmptyDrops takes multiple arguments that you can read about in the [documentation](https://rdrr.io/github/MarioniLab/DropletUtils/man/emptyDrops.html). However, in this case, we will only specify the following arguments:

> <details-title>emptyDrops input parameters</details-title>
>
> - `m` -	A numeric matrix-like object - usually a dgTMatrix or dgCMatrix - containing droplet data prior to any filtering or cell calling. Columns represent barcoded droplets, rows represent genes.
> - `lower` - A numeric scalar specifying the lower bound on the total UMI count, at or below which all barcodes are assumed to correspond to empty droplets.
> - `niters` - An integer scalar specifying the number of iterations to use for the Monte Carlo p-value calculations.
> - `retain` - A numeric scalar specifying the threshold for the total UMI count above which all barcodes are assumed to contain cells.
>
{: .details}

Let's then extract the matrix from our `alevin_se` object. It's stored in *assays* -> *counts*.

```bash
matrix_alevin <- assays(alevin_se)$counts
```

And now run emptyDrops:
```bash
# Identify likely cell-containing droplets
out <- emptyDrops(matrix_alevin, lower = 100, niters = 1000, retain = 20)
out
```

We also correct for multiple testing by controlling the false discovery rate (FDR) using the Benjamini-Hochberg (BH) method ({% cite Benjamini1995 %}). Putative cells are defined as those barcodes that have significantly poor fits to the ambient model at a specified FDR threshold. Here, we will use an FDR threshold of 0.01. This means that the expected proportion of empty droplets in the set of retained barcodes is no greater than 1%, which we consider to be acceptably low for downstream analyses.

```bash
is.cell <- out$FDR <= 0.01
sum(is.cell, na.rm=TRUE)                              # check how many cells left after filtering
```

We got rid of the background droplets containing no cells, so now we will filter the matrix that we passed on to emptyDrops, so that it corresponds to the remaining cells.

```bash
emptied_matrix <- matrix_alevin[,which(is.cell),drop=FALSE]          # filter the matrix
```

From here, we can move on to adding metadata and we'll return to `emptied_matrix` soon.

# Adding cell metadata

The cells barcodes are stored in *colnames*. Let's exctract them into a separate object:
```bash
barcode <- colnames(alevin_se)
```

Now, we can simply add those barcodes into *colData names* where we will keep the cell metadata. To do this, we will create a column called `barcode` in *colData* and pass the stored values into there.

```bash
colData(alevin_se)$barcode <- barcode
colData(alevin_se)
```

That's only cell barcodes for now! However, after running *emptyDrops*, we generated lots of cell information that is currently stored in `out` object (Total, LogProb, PValue, Limited, FDR). Let's add those values to cell metadata! Since we already have *barcodes* in there, we will simply bind the emptyDrops output to the existing dataframe:

```bash
colData(alevin_se) <- cbind(colData(alevin_se),out)
colData(alevin_se)
```

As you can see, the new columns were appended successfully and now the dataframe has 6 columns.

If you have a look at the Experimental Design from that study, you might notice that there is actually more information about the cells. The most important for us would be batch, genotype and sex, summarised in the small table below.

| Index | Batch | Genotype | Sex |
|------ |--------------------|
|--:|--:|:--|:-:|
| N701 | 0    | wildtype    | male    |
| N702 | 1    | knockout   | male    |
| N703 | 2    | knockout   | female    |
| N704 | 3    | wildtype    | male    |
| N705 | 4    | wildtype    | male    |
| N706 | 5    | wildtype    | male    |
| N707 | 6    | knockout    | male    |

We are currently analysing sample N701, so let's add its information from the table.

## Batch

We will label batch as an integer - "0" for sample N701, "1" for N702 etc. The way to do it is creating a list with zeros of the length corresponding to the number of cells that we have in our SummarizedExperiment object, like so:

```bash
batch <- rep("0", length(colnames(alevin_se)))
```

And now create a batch slot in the *colData names* and append the `batch` list in the same way as we did with barcodes:

```bash
colData(alevin_se)$batch <- batch
colData(alevin_se)
```

A new column appeared, full of zeros - as expected!

## Genotype

It's all the same for genotype, but instead creating a list with zeros, we'll create a list with string "wildtype" and append it into genotype slot:

```bash
genotype <- rep("wildtype", length(colnames(alevin_se)))
colData(alevin_se)$genotype <- genotype
```

## Sex

You already know what to do, right? A list with string "male" and then adding it into a new slot!
```bash
sex <- rep("male", length(colnames(alevin_se)))
colData(alevin_se)$sex <- sex
```

Check if all looks fine:
```bash
colData(alevin_se)
```

3 new columns appeared with the information that we've just added - perfect! You can add any information you need in this way, as long as it's the same for all the cells from one sample.


# Adding gene metadata

The genes IDs are stored in *rownames*. Let's exctract them into a separate object:

```bash
gene_ID <- rownames(alevin_se)
```

Analogically, we will add those genes IDs into *rowData names* which stores gene metadata. To do this, we will create a column called `gene_ID` in *rowData* and pass the stored values into there.

```bash
rowData(alevin_se)$gene_ID <- gene_ID
```

## Adding genes symbols based on their IDs

Since gene symbols are much more informative than only gene IDs, we will add them to our metadata. We will base this annotation on Ensembl - the genome database – with the use of the library BioMart. We will use the archive Genome assembly GRCm38 to get the gene names. Please note that the updated version (GRCm39) is available, but some of the gene IDs are not in that EnsEMBL database. The code below is written in a way that it will work for the updated dataset too, but will produce ‘NA’ where the corresponding gene name couldn’t be found.

```bash
# get relevant gene names
library("biomaRt")                                      # load the BioMart library
ensembl.ids <- gene_ID
mart <- useEnsembl(biomart = "ENSEMBL_MART_ENSEMBL")    # connect to a specified BioMart database and dataset hosted by Ensembl
ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl")

# The line above connects to a specified BioMart database and dataset within this database.
# In our case we choose the mus musculus database and to get the desired Genome assembly GRCm38,
# we specify the host with this archive. If you want to use the most recent version of the dataset, just run:
# ensembl_m = useMart("ensembl", dataset="mmusculus_gene_ensembl")
```

> <warning-title>Ensembl connection problems</warning-title>
> Sometimes you may encounter some connection issues with Ensembl. To improve performance Ensembl provides several mirrors of their site distributed around the globe. When you use the default settings for useEnsembl() your queries will be directed to your closest mirror geographically. In theory this should give you the best performance, however this is not always the case in practice. For example, if the nearest mirror is experiencing many queries from other users it may perform poorly for you. In such cases, the other mirrors should be chosen automatically.
>
{: .warning}

```bash
genes <- getBM(attributes=c('ensembl_gene_id','external_gene_name'),
               filters = 'ensembl_gene_id',
               values = ensembl.ids,
               mart = ensembl_m)

# The line above retrieves the specified attributes from the connected BioMart database;
# 'ensembl_gene_id' are genes IDs,
# 'external_gene_name' are the genes symbols that we want to get for our values stored in ‘ensembl.ids’.
```
```bash
# see the resulting data
head(genes)
```
```bash
# replace IDs for gene names
gene_names <- ensembl.ids
count = 1
for (geneID in gene_names)
{
 index <- which(genes==geneID)    # finds an index of geneID in the genes object created by getBM()
 if (length(index)==0)            # condition in case if there is no corresponding gene name in the chosen dataset
  {
    gene_names[count] <- 'NA'
  }
  else
  {
    gene_names[count] <- genes$external_gene_name[index] 	# replaces gene ID by the corresponding gene name based on the found geneID’s index
  }
 count = count + 1                # increased count so that every element in gene_names is replaced
}
```
```bash
# add the gene names into rowData in a new column gene_name
rowData(alevin_se)$gene_name <- gene_names
```
```bash
# see the changes
rowData(alevin_se)
```

If you are working on your own data and it’s not mouse data, you can check available datasets for other species and just use relevant dataset in `useMart()` function.

```bash
listDatasets(mart)                # available datasets
```


## Flag mitochondrial genes

We can also flag mitochondrial genes. Usually those are the genes whose name starts with 'mt-' or 'MT-'. Therefore, we will store those characters in `mito_genes_names` and then use *grepl()* to find those genes stored in *gene_name* slot.

```bash
mito_genes_names <- '^mt-|^MT-'                                    # how mitochondrial gene names can start
mito <- grepl(mito_genes_names, rowData(alevin_se)$gene_name)      # find mito genes
mito                                                               # see the resulting boolean list
```

Now we can add another slot in *rowData* and call it *mito* that will keep boolean values (true/false) to indicate which genes are mitochondrial.
```bash
rowData(alevin_se)$mito <- mito
rowData(alevin_se)
```


# Subsetting the object

Let's go back to the `emptied_matrix` object. Do you remember how many cells were left after filtering? We can check that by looking at the matrix' dimensions:

```bash
dim(matrix_alevin)                                                  # check the dimension of the unfiltered matrix
dim(emptied_matrix)                                                 # check the dimension of the filtered matrix
```

We've gone from 3608 to 35 cells. We've filtered the matrix, but not our SummarizedExperiment. We can subset `alevin_se` based on the cells that were left after filtering. We will store them in a separate list, as we did with the barcodes:

```bash
retained_cells <- colnames(emptied_matrix)
retained_cells
```

And now we can subset our SummarizedExperiment based on the barcodes that are in the `retained_cells` list:

```bash
alevin_subset <- alevin_se[, colData(alevin_se)$barcode %in% retained_cells]
alevin_701 <- alevin_subset
alevin_701
```

And that's our subset, ready for downstream analysis!


# More datasets

We've done the analysis for one sample. But there are 7 samples in this experiment and it would be very handy to have all the information in one place. Therefore, you would need to repeat all the steps for the subsequent samples (that's when you'll appreciate wrapped tools and automation in Galaxy workflows!). To make your life easier, we will show you how to combine the datasets on smaller scale. Also, to save you some time, we've already run Alevin on sample 702 (also subsampled to 50k reads). Let's quickly repeat the steps we performed in R to complete the analysis of sample 702 in the same way as we did with 701.

But first, we have to save the results of our hard work on sample 701!

## Saving sample 701 data

Saving files is quite straightforward. Just specify which object you want to save and how you want the file to be named. Don't forget the extension!

```bash
save(alevin_701, file = "alevin_701.rdata")
```

You will see the new file in the panel on the left.


## Analysis of sample 702

Normally, at this point you would switch kernel to bash to run alevin, and then back to R to complete the analysis of another sample. Here, we are providing you with the alevin output for the next sample, which you have already downloaded and unzipped at the beginning of the tutorial when we were using bash.

> <warning-title>Switching kernels & losing variables</warning-title>
>
> Be aware that every time when you switch kernel, you will lose variables you store in the objects that you've created, unless you save them. Therefore, if you want to switch from R to bash, make sure you save your R objects! You can then load them anytime.
>
{: .warning}

Above we described all the steps done in R and explained what each bit of code does. Below all those steps are in one block of code, so read carefully and make sure you understand everything!

```bash
## load libraries again ##
library(tximeta)
library(DropletUtils)
library(biomaRt)

## running tximeta ##
path2 <- 'alevin_output_702/alevin/quants_mat.gz'                                                    # path to alevin output for N702
alevin2 <- tximeta(coldata = data.frame(files = path2, names = "sample702"), type = "alevin")        # create SummarizedExperiment from Alevin output

## running emptyDrops ##
matrix_alevin2 <- assays(alevin2)$counts                                                             # extract matrix from SummarizedExperiment
out2 <- emptyDrops(matrix_alevin2, lower = 100, niters = 1000, retain = 20)                          # apply emptyDrops
is.cell2 <- out2$FDR <= 0.01                                                                         # apply FDR threshold
emptied_matrix2 <- matrix_alevin2[,which(is.cell2),drop=FALSE]                                       # subset the matrix to the cell-containing droplets

## adding cell metadata ##
barcode2 <- colnames(alevin2)
colData(alevin2)$barcode <- barcode2                                             # add barcodes

colData(alevin2) <- cbind(colData(alevin2),out2)                                 # add emptyDrops info

batch2 <- rep("1", length(colnames(alevin2)))
colData(alevin2)$batch <- batch2                                                # add batch info

genotype2 <- rep("wildtype", length(colnames(alevin2)))
colData(alevin2)$genotype <- genotype2                                          # add genotype info

sex2 <- rep("male", length(colnames(alevin2)))
colData(alevin2)$sex <- sex2                                                    # add sex info

## adding gene metadata ##
gene_ID2 <- rownames(alevin2)
rowData(alevin2)$gene_ID <- gene_ID2

# get relevant gene names
ensembl.ids2 <- gene_ID2
mart <- useEnsembl(biomart = "ENSEMBL_MART_ENSEMBL")    # connect to a specified BioMart database and dataset hosted by Ensembl
ensembl_m2 = useMart("ensembl", dataset="mmusculus_gene_ensembl")

genes2 <- getBM(attributes=c('ensembl_gene_id','external_gene_name'),
               filters = 'ensembl_gene_id',
               values = ensembl.ids2,
               mart = ensembl_m2)

# replace IDs for gene names
gene_names2 <- ensembl.ids2
count = 1
for (geneID in gene_names2)
{
 index <- which(genes2==geneID)    # finds an index of geneID in the genes object created by getBM()
 if (length(index)==0)            # condition in case if there is no corresponding gene name in the chosen dataset
  {
    gene_names2[count] <- 'NA'
  }
  else
  {
    gene_names2[count] <- genes2$external_gene_name[index] 	# replaces gene ID by the corresponding gene name based on the found geneID’s index
  }
 count = count + 1                # increased count so that every element in gene_names is replaced
}

rowData(alevin2)$gene_name <- gene_names2                              # add gene names to gene metadata

mito_genes_names <- '^mt-|^MT-'                                        # how mitochondrial gene names can start
mito2 <- grepl(mito_genes_names, rowData(alevin2)$gene_name)           # find mito genes
rowData(alevin2)$mito <- mito2                                         # add mitochondrial information to gene metadata

## create a subset of filtered object ##
retained_cells2 <- colnames(emptied_matrix2)
alevin_subset2 <- alevin2[, colData(alevin2)$barcode %in% retained_cells2]

alevin_702 <- alevin_subset2
alevin_702
```

Alright, another sample pre-processed!


# Combining datasets

Pre-processed sample 702 is there, but keep in mind that if you switched kernels in the meantime, you still need to load sample 701 that was saved before switching kernels. It's equally easy as saving the object:

```bash
load("alevin_701.rdata")
```

Check if it was loaded ok:
```bash
alevin_701
```

Now we can combine those two objects into one using one simple command:

```bash
alevin_combined <- cbind(alevin_701, alevin_702)
alevin_combined
```

If you have more samples, just append them in the same way. We won't process another sample here, but pretending that we have third sample, we would combine it like this:

```bash
alevin_subset3 <- alevin_702                        # copy dataset for demonstration purposes
alevin_combined_demo <- cbind(alevin_combined, alevin_subset3)
alevin_combined_demo
```

You get the point, right? It's important though that the rowData names and colData names are the same in each sample.


# Saving and exporting the files

It is generally more common to use SingleCellExperiment format rather than SummarizedExperiment. The conversion is quick and easy, and goes like this:

```bash
library(SingleCellExperiment)                                 # might need to load this library
alevin_sce <- as(alevin_combined, "SingleCellExperiment")
alevin_sce
```
As you can see, all the embeddings have been successfully transfered during this conversion and believe me, SCE object will be more useful for you!

You've already learned how to save and load objects in Jupyter notebook, let's then save the SCE file:

```bash
save(alevin_sce, file = "alevin_sce.rdata")
```

The last thing that might be useful is exporting the files into your Galaxy history. To do it... guess what! Yes - **switching kernels again**! But this time we choose **Python 3** kernel and run the following command:

```bash
# that's Python now!
put("alevin_sce.rdata")
```
A new dataset should appear in your Galaxy history now.

# Conclusion

Well done! In this tutorial we have:
- examined raw read data, annotations and necessary input files for quantification
- created an index in Salmon and run Alevin
- identified barcodes that correspond to non-empty droplets
- added gene and cell metadata
- applied the necessary conversion to pass these data to downstream processes.

As you might now appreciate, some tasks are much quicker and easier when run in the code, but the reproducibility and automation of Galaxy workflows is a huge advantage that helps in dealing with many samples more quickly and efficiently.


topic: single-cell
tutorial: scrna-seurat-pbmc3k
title: Clustering 3K PBMCs with Seurat
difficulty: introductory
requirements_json: []
tags_json: []
content_hash: 4c906fb9e72008bf7f83da8b678cff61
last_modified: 2026-05-27T09:01:49.751020
zenodo_link: https://zenodo.org/record/3581213
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-seurat-pbmc3k/tutorial.html

content:
separate-preprocessing---answer-key
  date: 2023-10-28
- label: UseGalaxy.eu - Using SCTransform
  history: https://usegalaxy.eu/u/marisa_jl/h/clustering-3k-pbmcs-with-seurat---sctransform---answer-ke
  date: 2023-10-28
- label: UseGalaxy.org.au - Using Separate Preprocessing Tools
  history: https://usegalaxy.org.au/u/videmp/h/gtn-clustering-3k-pbmcs-with-seurat-separate-preprocessing-may-2025
  date: 2025-05-02
- label: UseGalaxy.org.au - Using SCTransform
  history: https://usegalaxy.org.au/u/videmp/h/gtn-cluster-3k-pbmcs-with-seurat-sctransform-may-2025
  date: 2025-05-02
questions:
- How can we identify cell types in single cell RNA-Seq data?
- What are the steps for clustering single cell data with Seurat?
objectives:
- Explain the steps involved in clustering single cell data
- Evaluate the quality of single cell data and filter out low quality cells
- Prepare single cell data for analysis with Seurat
- Perform clustering with Seurat
- Be ready to apply the Seurat pipeline to new datasets
time_estimation: 8H
key_points:
- Seurat is a commonly-used pipeline for single cell data analysis
- Clustering makes single cell datasets easier for us to understand
- Different tools and parameters should be considered when analysing different datasets
requirements:
- type: internal
  topic_name: single-cell
  tutorials:
  - scrna-preprocessing
  - scrna-preprocessing-tenx
follow_up_training:
- type: internal
  topic_name: single-cell
  tutorials:
  - EBI-retrieval
tags:
- 10x
contributions:
  authorship:
  - MarisaJL
  editing:
  - pavanvidem
  - shiltemann
recordings:
- youtube_id: -tIOSBXeXPs
  length: 2H11M
  galaxy_version: 24.2.4.dev0
  date: '2025-04-15'
  speakers:
  - MarisaJL
  captioners:
  - MarisaJL
  bot-timestamp: 1744754154


---


Single cell RNA-seq analysis enables us to explore differences in gene expression between cells. It can reveal the heterogenity within cell populations and help us to identify cell types that could play roles in development, disease, or other processes. Single cell omics is a relatively young field, but there are a few commonly-used analysis pipelines that you will often see in the literature. In this tutorial, we will use one of these pipelines, Seurat, to cluster single cell data from a 10X Genomics experiment ({% cite seurat2023v5 %}). You can follow the same analysis using the Scanpy pipeline in the [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial.

Clustering is typically the first type of analysis we will perform on a single cell dataset. It groups together cells that are expressing similar genes, which makes the data easier to understand and often helps us to identify specific cell types.

> <comment-title></comment-title>
>
> This tutorial is based on the [Seurat - Guided Clustering Tutorial](https://satijalab.org/seurat/articles/pbmc3k_tutorial). The SCTransform sections also draw from the [Using sctransform in Seurat](https://satijalab.org/seurat/articles/sctransform_vignette.html) tutorial.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Important tips for easier analysis

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

# Data

For this tutorial, we will analyze a dataset of Peripheral Blood Mononuclear Cells (PBMC) extracted from a healthy donor, which is freely available from 10X Genomics. The dataset contains 2700 single cells sequenced using Illumina NextSeq 500. The raw sequences have been processed by the [**cellranger**](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) pipeline from 10X to extract a unique molecular identifier (UMI) count matrix, in a similar way to that explained in the [Pre-processing of 10X Single-Cell RNA Datasets]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}) tutorial.

In this matrix, the values represent the number of each feature (i.e. gene; row) detected in each cell (column). Single cell matrices can be quite large: here there are 2700 columns with 32,738 rows, so for each of our 2700 cells we know how many times we found RNAs matching each of the 32,738 genes. Since most of these genes weren't detected in most of the cells, the matrix is largely filled with zeros, i.e. it is an extremely sparse matrix. To optimize the storage of such a table and the information about the genes and cells, **cellranger** creates 3 files:

- `genes.tsv`: a tabular file with information about the 32,738 genes in 2 columns (Ensembl gene id and the gene symbol)
- `barcodes.tsv`: a tabular file with the barcode for each of the 2700 cells
- `matrix.mtx`: a condensed version of the count matrix (including the non-zero values only)

    The count matrix is represented by its non-zero values - we don't need to store all of those zeroes as long as we know where our non-zero values are in the matrix. Each non-zero value is represented by its line number (1st column), its column number (2nd column) and its value (3rd column). The first row gives the total number of rows (genes), columns (cells) and non-zero values. More information on the Matrix Market Exchange (mtx) format can be found [in this documentation](https://math.nist.gov/MatrixMarket/formats.html)

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the `genes.tsv`, `barcodes.tsv` and `matrix.mtx` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/genes.tsv
>    {{ page.zenodo_link }}/files/barcodes.tsv
>    {{ page.zenodo_link }}/files/matrix.mtx
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets as `genes`, `barcodes`, and `matrix` if necessary
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check the datatypes are correct - the `genes` and `barcodes` files should be tsv or tabular while the `matrix` should be an mtx file
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 5. Inspect the `matrix` file by clicking on the {% icon galaxy-eye %} icon
>
>    {% snippet faqs/galaxy/histories_dataset_item.md %}
>
{: .hands_on}

The beginning of the file should look like this:

> <question-title></question-title>
>
> ```
> 32738	2700	2286884
> 32709	1	4
> 32707	1	1
> 32706	1	10
> 32704	1	1
> ```
>
> 1. How many non-zero values are in the matrix?
> 2. How many counts were found for the 32,706th gene in the 1st cell?
>
> > <solution-title></solution-title>
> >
> > 1. The first row tells us there are 2,286,884 non-zero values for the 32,738 genes (rows) and 2,700 cells (columns) - so only 2.6% of the 88,392,600 potential values we could have in this matrix are non-zero. Getting rid of all those zeros has made the matrix much more compact.
> > 2. 10 counts were found for the 32,706th row (gene) and 1st column (cell), so we collected 10 RNAs that the first cell had produced from this particular gene.
> >
> {: .solution}
>
{: .question}

Representing the matrix with these three files is convenient for sharing the data, but not for processing them. Different single cell analysis packages have attempted to solve the problem of storage and analysis by inventing their own formats, which has led to the proliferation of many different 'standards' in the scRNA-seq package ecosystem.

## SeuratObject

In order to analyse the data using Seurat, we will first need to create a SeuratObject. A SeuratObject can store all our data, including the gene names, cell barcodes and matrix in a single RDS file. Since Seurat was written in the R programming language, SeuratObjects are saved as RData or RDS files. An RDS file is simply a type of RData file that contains a single object.

SeuratObjects can also hold any metadata we might have about our cells or genes, as well as the information we will produce during our analysis, such as our dimensional reductions and cell clusters. A SeuratObject can hold data in multiple layers (also known as slots in earlier version of Seurat). We can have layers of raw counts, normalised data, and scaled data. We could also store multiple assays or types of data in a single object, although in this tutorial we will only be using one assay for RNA-seq data. The SeuratObject is designed to be self-contained so that we don't have to work with lots of different files, even if we have different versions or types of data.

Creating a SeuratObject in R would require two steps - first, we would need to read in our data, in this case using the `Read10X` function, then secondly we would turn it into a SeuratObject using the `CreateSeuratObject` function. On Galaxy, we can perform both steps with a single tool. The `CreateSeuratObject` function also generates some QC metrics and performs basic filtering of the data.

> <hands-on-title>Create a SeuratObject</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Create Seurat Object`
>        - *"Select format of input"*: `matrix market (for e.g. 10x data)`
>            - {% icon param-file %} *"Counts matrix with features as rows, cells as columns (.mtx)"*: `matrix.mtx`
>            - {% icon param-file %} *"List of gene names (for rows)"*: `genes.tsv`
>            - {% icon param-file %} *"List of cell barcodes (for columns)"*: `barcodes.tsv`
>        - *"Include features detected in at least this many cells"*: `3`
>        - *"Include cells where at least this many features are detected"*: `200`
>        - *"Calculate percentage of mito genes in each cell"*: `No`
>
> 2. Rename the generated file to `Input 3k PBMC`
>
> 3. Check that the format is `rds`
{: .hands_on}

We can't look at the RDS file directly as it is designed for computers to read, rather than humans, but the Seurat tools will now be able to interact with the data. We can also use the {% tool Seurat Data Management %} tool to look inside our new SeuratObject.

> <hands-on-title>Inspect the `Input 3k PBMC` SeuratObject</hands-on-title>
>
> 1. {% tool [Seurat Data Management](toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0) %} with the following parameters:
>    - *"Method used"*: `Inspect Seurat Object`
>        - *"Display information about"*: `General`
>
> 2. Click the {% icon galaxy-eye %} on the output in your history to see some information about your SeuratObject
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many cells and genes are in the `Input 3k PBMC` SeuratObject we just created?
> 2. Is this the same as the number of cells and genes we started out with?
>
> > <solution-title></solution-title>
> > 1. If we click on the {% icon galaxy-eye %} of the output from {% tool Seurat Data Management %} we can see some general information about our SeuratObject. We can see that it contains information about 2700 samples (cells) and 13,714 features (genes).
> > 2. When we looked at the `matrix.mtx` file previously, we saw that it contained information on the same 2700 cells, but that there were 32,738 genes with non-zero values in the matrix. That means we are missing 19,024 genes - but don't worry, this is supposed to happen! Remember that we set the `Include features detected in at least this many cells` parameter to `3` - this got rid of any genes that were found in fewer than three cells. We assume that these genes are just technical noise (e.g. misreads, reads mapped to the wrong gene) rather than real biological features. We also set the `Include cells where at least this many features are detected` to `200` - this would have removed any cells that had fewer than 200 genes found in them. In this case, we didn't remove any cells as all 2700 of them must have had at least 200 genes. If we did have cells with fewer than 200 genes, we would want to get rid of them they are probably low quality cells or cell fragments. We would usually expect to find more features in an undamaged cell that was captured and sequenced properly.
> >
> {: .solution}
{: .question}

> <comment-title></comment-title>
> If you would like to see the impact of the initial filtering performed by {% tool Seurat Create %}, then you can rerun this step with these filters removed (or changed) and then perform the QC visualisation steps described in the next section on the unfiltered dataset. The Seurat pipeline usually includes some filtering during object creation, which shouldn't cause any problems as long as we use reasonable thresholds that only filter out the lowest quality cells and genes.  We can always come back and create a new object with different values if we suspect they weren't right for our dataset once we make our QC plots or if we have problems later on in the analysis.
{: .comment}

# Preprocessing

The standard Seurat pipeline begins with preprocessing, which includes identifying and filtering out low-quality cells, normalising and scaling the data, and selecting highly variable features.

## Quality Control

Quality control is an essential step in preparing single cell data for analysis. The results from a single cell experiment can vary in quality due to cell damage during dissociation, capturing more than one cell in a well or droplet, failures in library preparation such as inefficient reverse transcription (copying of captured RNAs to cDNA) or PCR amplification (making more copies of the cDNA so we have enough material for sequencing) or other technical problems. Low quality cells can cause issues for some analysis tools and may contribute to misleading results.

> <details-title>Impact of low-quality cells on the downstream analyses</details-title>
>
> #### Formation of their own distinct cluster(s)
>
> The most obvious cause of this problem is that damaged cells can appear more similar to each other because they have increased proportions of mitochondrial genes or are enriched for certain damage-induced genes. We could end up with a cluster made up of different cell types that share similar RNAs because they've been damaged. We might think these damaged cells represent an intermediate state between other cell types or end up misinterpreting our data in other ways.
>
> #### Distortion of population heterogeneity during variance estimation or principal components analysis
>
> When we cluster single cell data, we're looking for the biggest differences between groups of cells. If we have lots of low-quality cells, then the biggest differences we'll see might be these differences in quality, rather than something more biologically interesting. Differences in quality can have a big impact because low-quality cells often have low total RNA counts. When we perform scaling and normalisation on these cells during preprocessing, this can make the variances for the genes they do express much bigger than for other cells. When we select the most variable genes in the dataset, we'll end up picking the ones expressed by low-quality cells. We'll use these in our PCA analysis, likely ending up with top principal components based on cell quality rather than biology, making it harder to detect the differences we're actually interested in.
>
> #### Misidentification of upregulated genes
>
> Another problem that arises when we apply our preprocessing steps to small, low quality cells is that the genes we detect in them can appear to be strongly upregulated. Since we didn't detect many other genes in these cells, even a small difference in the number of transcripts detected can end up becoming much larger after normalisation. For example, contaminating transcripts may be present in all cells at low but constant levels. With the increased scaling and normalization in low-quality cells, the small counts for these transcripts may become large normalized expression values, so we might think we've found a population of cells where these genes are upregulated.
>
{: .details}

In order to mitigate these problems, we need to remove low-quality cells at the start of the analysis.

The `CreateSeuratObject` function we used in the {% tool Seurat Create %} tool automatically calculates some QC metrics and allowed us to filter out the lowest quality cells and features immediately. We will now check the data to see if it requires any further filtering.

### Computation of QC metrics

We already have two useful QC metrics that were calculated when we created our SeuratObject:

- **nCount_RNA** the total sum of RNAs that were found in each cell.
    The total number of counts is related to cell size, but it can also be an indication of quality. If `nCount_RNA` is very high we might be looking at results from a doublet or multiplet - two or more cells that were isolated together during the experiment. If `nCount_RNA` is very low, then it is likely that we lost a lot of the RNA due to cell lysis (breakage) or inefficient cDNA capture and amplification - or perhaps we only captured a fragment of a cell.
- **nFeature_RNA** the number of unique genes that were detected in each cell.
    We would expect to see some variation in the variety of genes expressed by cells of different sizes, types, and conditions, but if this number is unusually high or low then it could be a sign of poor quality. High `nFeature_RNA` could be another sign of a doublet or multiplet - we might have captured cells of different types and processed them together. Low `nFeature_RNA` could be due to loss of RNA if `nCount_RNA` is also low, or a sign that we have failed to capture the diversity of the transcript population, perhaps due to technical problems in our experiment.

One other metric that is often used to assess cell quality is the proportion of reads that came from the mitochondrial genome. The proportion is often higher in low quality, damaged or dying cells. Mitochondrial RNAs, which are protected inside the mitochondrial membranes, can be the last RNAs to be degraded or lost from a damaged cell, so we can end up with higher proportions of them in a low quality cell. We could have calculated the proportion of mitochondrial genes while creating our SeuratObject, but we will calculate it separately here to see how it is done - and how we could do the same for other types of genes.

We can identify mitochondrial genes from their gene symbols. Human genes that are encoded in the mitochondrial DNA (rather than in the cell nucleus) have names beginning with 'MT-'. Different naming patterns are used in other species. The {% tool Seurat Create %} tool can identify genes based on a specific naming pattern and then calculate the proportion of reads in each cell that came from these genes. We could use this tool to calculate the proportions of other genes with shared naming patterns too, such as ribosomal genes.

> <hands-on-title>Calculate the Proportion of Mitochondrial Reads</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Add QC Metrics`
>        - {% icon param-file %} *"Input file with the Seurat object"*: `Input 3k PBMC` (output of **Seurat Create** {% icon tool %})
>        - *"Calculate percentage of reads based on"*: `Pattern in gene names`
>        - *"Pattern/regex to match in gene names"*: `^MT-`
>
> 2. Rename the output as `Mitochondrial Annotations`
{: .hands_on}

When we use these QC metrics to remove low quality cells, we are assuming they are independent of the biological state of each cell, which isn't always the case in reality. For example, some cell types with higher energy demands may have higher proportions of mitochondrial reads because they have more mitochondria. We need to think carefully when deciding which metrics to use for filtering, but for now we will assume that these differences in our dataset are being driven by technical factors (e.g. cell damage, loss of RNAs, problems with capture and sequencing) rather than biological processes (e.g. cell types or conditions). We can therefore remove cells with poor values without misrepresenting the biology in downstream analyses.

### Filtering of low-quality cells

The simplest approch for identifing low-quality cells is to apply thresholds on the QC metrics. We assume that any cells beyond these thresholds are low quality cells and that this is due to those technical factors. Although this is a simple strategy, we will need to decide where to set our thresholds, and this will depend on the experimental protocol and biological system - there are no standard thresholds that we can use for every dataset.

We can visualise the QC metrics to help us decide where to set our thresholds for filtering out low quality cells. We want to get rid of cells that have unusually high or low numbers of genes or unique features, as well as cells that have higher proportions of mitochondrial genes.

> <hands-on-title>Visualise QC Metrics</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Violin Plot with 'VlnPlot'`
>        - *"Features to plot"*: `nFeature_RNA,nCount_RNA,percent.mt`
>        - In *"Plot Formatting Options"*:
>            - *"Number of columns to display"*: `3`
>
> 2. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `percent.mt`
>
> 3. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `nFeature_RNA`
>
{: .hands_on}

> <tip-title></tip-title>
> If you are re-running the same step again or just want to change a couple of parameters for the same tool, then you can click on a previous dataset to expand it in the history and then click the {% icon dataset-rerun button %}. The tool you used to create that dataset will open in the main panel, with the settings and inputs you used before. Make any changes and run it again.
{: .tip}

![Three violin plots each showing most points grouped together in the lower part of the plot with some outliers at higher values](../../images/scrna-seurat-pbmc3k/seurat_violin_QC_before.png "Violin Plots showing the unique features (nFeature_RNA), total counts (nCount_RNA) and the proportion of reads coming from mitochondial genes (percent.mt) for all cells")

![Scatter plot A shows clear positive correlation with the number of unique features increasing as total counts increases. A small number of cells have very high values of both. Scatter plot B does not show a relationship between the proportion of mitochondrial genes and total counts. A small number of cells with low total counts have high values of percent.mt](../../images/scrna-seurat-pbmc3k/seurat_QC_scatter_before.png "Scatter plots showing the relationships between the total counts (nCount_RNA) and A. the number of unique features (nFeature_RNA) and B. the proportion of mitochondrial reads (percent.mt)")

> <question-title></question-title>
>
> 1. What do the violin plots tell us about the cell sizes and quality in our dataset?
> 2. What do the scatter plots in tell us about the relationship between cell size and the other QC metrics?
>
> > <solution-title></solution-title>
> > 1. The violin plots give us an overview of the cell sizes (nCount_RNA), number of unique genes (nFeature_RNA), and proportion of mitochondrial reads for our cells. We can see that most of the cells are grouped together near the bottom of each plot, but there are some outliers that have unusually high or low values of each metric. Cells with higher values of nCount_RNA are likely to be larger as they contained more RNA. Cells with higher values of nFeature_RNA had RNAs produced from lots of different genes, while those with low nFeature_RNA had RNAs produced from a smaller range of genes. Cells with high percent.mt had lots of reads from mitochondrial genes. Although there will be some natural variation in cell size, the range of genes being expressed, and mitochondrial content, we suspect that extreme values of these three metrics reflect low quality cells.
> > 2. The scatter plots show us how these QC metrics relate to each other. As expected, the cells with higher total RNA counts also tend to have higher numbers of unique features, but some cells have particularly high values of both - these could be doublets. The proportion of mitochondrial genes doesn't increase with cell size (total counts) in the same way. We can see a small number of cells that have very high proportions of mitochondrial reads despite having low total reads - these are likely to be damaged cells that have lost a lot of their non-mitochondrial RNAs.
> >
> {: .solution}
{: .question}

We can now set our QC thresholds based on these plots. Unfortunately, there are no standard thresholds that can be applied to every dataset, so we need to look at our data and make this decision for ourselves.

We've already filtered out the cells with the lowest total counts when we created the SeuratObject, so we'll focus on filtering the number of unique features and the proportion of mitochondrial reads. We saw on the scatter plots that the cells with high values for these two metrics also had the highest nCount_RNA values, so we'll actually be getting rid of the cells with the highest total counts too.

> <comment-title></comment-title>
> If you've used Scanpy or followed the [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial then you might be wondering why we aren't visualising and filtering our genes as well as our cells. Seurat enables us to set the minimum number of cells that genes must be present in when we create the SeuratObject, but the pipeline doesn't usually include any further quality checks for genes. If we did need to do further filtering, we could do this in R by calculating an appropriate QC metric and subsetting the data.
{: .comment}

> <question-title></question-title>
>
> 1. What threshold would you set for nFeature_RNA?
> 2. What threshold would you set for percent.mt?
>
> > <solution-title></solution-title>
> > 1. A threshold of 2500 seems sensible for this dataset. The violin plot for nFeature_RNA shows that most of our cells should be under this threshold, so we won't lose too much of our data. The scatter plot shows that the cells above this threshold also had unusually high values of nCount_RNA, which suggest they could include some doublets.
> > 2. A threshold of 5% should work for this dataset. As before, the violin plot shows that the majority of our cells are below this threshold, but in this case the cells above it had low total RNA counts, which suggests these could be damaged cells that had lost a lot of their other RNAs. Although there are few standards to guide us in single cell analysis, you will see the same 5% threshold for mitochondrial content used in many studies. It often works well, but it can filter out energetic cell types such as muscle cells, so we shouldn't apply this threshold without considering whether it works for our dataset. Some studies don't filter on mitochondrial reads at all!
> >
> {: .solution}
{: .question}

We're setting QC thresholds based on our visual inspections of the data, but we could take different approaches too. We could calculate the interquartile range for our QC metrics and use this to set a threshold (e.g. maximum of 1.5 times the IQ range above the median). We could also try running the analysis with different thresholds to see how it affects the results - this could help us to see if we're excluding any biological variation, for example if we're filtering out a specific high-energy cell type with our mitochondrial threshold.

> <hands-on-title>Filter Out Low Quality Cells</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Filter cells by QC metrics`
>        - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>        - *"Minimum nFeature_RNA"*: `200`
>        - *"Maximum nFeature_RNA"*: `2500`
>        - *"Maximum percent.mt"*: `5.0`
>        - *"Filter by a different metric"*: `No`
>
> 2. Rename the output as `Filtered Dataset`
>
{: .hands_on}

If we produce the same plots again, we can see what has changed in our data.

> <hands-on-title>Re-Visualise QC Metrics</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Violin Plot with 'VlnPlot'`
>        - *"Features to plot"*: `nFeature_RNA,nCount_RNA,percent.mt`
>        - In *"Plot Formatting Options"*:
>            - *"Number of columns to display"*: `3`
>
> 2. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `percent.mt`
>
> 3. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `nFeature_RNA`
>
{: .hands_on}

![Three violin plots each showing most points grouped together and no extreme outliers](../../images/scrna-seurat-pbmc3k/seurat_violin_after.png "Violin Plots showing the unique features (nFeature_RNA), total counts (nCount_RNA) and the proportion of reads coming from mitochondial genes for all cells after filtering")

![Scatter plot A shows clear positive correlation with the number of unique features increasing as total counts increases. No extreme outliers. Scatter plot B does not show a relationship between the proportion of mitochondrial genes and total counts. No extreme outliers.](../../images/scrna-seurat-pbmc3k/seurat_QC_scatter_after.png "Scatter plots showing the relationships between the total counts (nCount_RNA) and A. the number of unique features (nFeature_RNA) and B. the proportion of mitochondrial reads (percent.mt) after filtering")

> <question-title></question-title>
>
> 1. Have we eliminated the low-quality cells from our data?
>
> > <solution-title></solution-title>
> > 1. We can see in both the violin and scatter plots that the outliers have been eliminated from our dataset - we've removed the cells with high mitochondrial counts and/or unusually high numbers of detected genes. We hope that this means we've got rid of the poor quality cells without losing any real biological variation, but we can't really be sure. Sometimes it is necessary to come back and re-think our filters - for example if you get to the end of the analysis and realise that you're missing a cell type that should be present in your data!
> >
> {: .solution}
{: .question}

## Further Preprocessing

Now that we're happy with the cells left in our SeuratObject, we can begin preparing the data for analysis. Seurat provides two routes for preprocessing, one with separate tools for each step and another that combines all these preprocessing steps into a single tool, `SCTransform`.

> <comment-title></comment-title>
>
> `SCTransform` makes preprocessing quicker and easier because it combines several steps from the standard Seurat pipeline into one. `SCTransform` normalises and scales the data while also selecting highly variable features. It also takes a different approach to preprocessing that can be more effective at removing technical effects from the data.
>
> Once you are familiar with the separate steps, you might find it more effective and convenient to use `SCTransform`. The main difference to be aware of when using `SCTransform` is that the results will be stored as a new assay called `SCT` in the SeuratObject. You will need to use this assay in the following analysis steps, instead of the original `RNA` assay. After running `SCTransform`, `SCT` will be set as the default assay for the SeuratObject so the tools should automatically use the correct one. The methods and default settings used in `SCTransform` also differ from the standard pipeline, so you should expect to see some differences in your results compared to the other route.
>
{: .comment}

The usual preprocessing steps for single cell data are normalisation, selection of the most variable features, and scaling the data. The same steps are performed in slightly different ways by the separate preprocessing functions and SCTransform.

- **Normalisation**
    Normalisation deals with variations in the number of reads we counted for each cell that were caused by systemic technical differences. We might have captured more RNAs or produced more copies of them during the PCR amplification step in some cells. If we don't remove these differences then they could obscure the biological differences we're trying to explore. We could end up with clusters of cells based on how many RNAs we captured or copied rather than on their cell type.
    The default normalisation method for the separate `NormalizeData` tool in Seurat is `LogNormalize`, which simply divides the number of counts for each gene by the total counts for the cell, multiplies this by a scale factor (10,000 is the default), and then log-transforms the results. Essentially, we're working out how many counts we would have for each gene if we had produced 10,000 reads from each cell, instead of getting different numbers of reads from different cells. The log-transformation then increases the impact that genes that showed stronger relative differences in expression between cells will have on our analysis (e.g. a gene that is expressed at an average count of 50 in cell type A and 10 in cell type B should have a bigger impact than a gene that is expressed at an average count of 1100 in A and 1000 in B). The normalised data will be stored in the `data` layer of the SeuratObject. The unnormalised data will still be available in the `counts` layer.
    Although this form of normalisation is widely-used, it does assume that each cell originally had the same number of RNA molecules, which isn't the case when our samples contain cells of different types and sizes. SCTransform takes a different approach to normalisation that doesn't make the same assumption about the cells {% cite SCTransform2022 %}. Instead, it creates a model of the UMI counts that enables it to regress out variation in sequencing depth (nFeature_RNA) and pools information from genes with similar abundances to adjust the variances. The corrected counts will be stored in the `counts` layer of the new SCT assay. The log1p(counts) will be stored in the `data` layer and the pearson residuals will be stored in the `scale.data` layer.
- **Selection of highly variable features**
    Rather than use the entire dataset in every stage of the analysis, we can focus on the genes that provide the most information - the highly variable genes that showed the biggest differences in expression between cells. We assume that these bigger differences reflect genuine biological differences, while the smaller differences in expression seen in other genes is down to chance or technical noise. Focusing on the most variable genes can make biological differences clearer in single cell analysis.
    Feature selection can have a big impact on our analysis as we will run downstream analyses (e.g. PCA) on these genes. We will be comparing cells based on the expression of these variable genes.
    Seurat's separate `FindVariableFeatures` tool selects the 2000 most variable genes by default, while `SCTransform` selects the top 3000 genes. We can select more variable genes when we use SCTransform because this method is better at removing technical effects from the data, so the additional variable features are more likely to represent biological variation.
  We may need to change this setting for some datasets to ensure we're selecting the most useful genes without including too many others.
- **Scaling**
    Scaling is a linear transformation that we apply to prepare our data for dimensional reduction.
    The default scaling method for the separate `ScaleData` tool in Seurat shifts the expression of each gene so that the mean expression across all the cells is 0. It also scales the expression of each gene so that the variance across all cells is 1. Scaling ensures that highly expressed genes don't dominate the analysis too much - we're interested in differences in expression between cells, not in genes that are always highly expressed in all the cells. The results will be stored in the `scale.data` layer.
    SCTransform doesn't scale data in the same way - although it centers the data by default, it won't scale the data unless you select this option. Instead, SCTransform usually stores the pearson residuals in the `scale.data` layer, which don't need to be scaled to the same variance.
    By default, both approaches only center/scale the highly variable genes that we'll use for dimensional reduction, but it is possible to scale more genes if required. If you choose to use the separate preprocessing tool in this tutorial, then we'll use the ScaleData tool to scale all the genes - this means that we can compare or plot the expression of any of the genes, as they will all be scaled in the same way. We won't perform scaling with SCTransform, as we're following the [original SCT tutorial](https://satijalab.org/seurat/articles/sctransform_vignette.html), but we will center all genes.
- **Regression**
  The Seurat pipeline can include another step during preprocessing of our single cell data. We can regress out (or remove) the impact of unwanted sources of variation. We could use this technique to remove the effects of the cell cycle or the differences associated with the proportion of mitochondrial genes. The goal is to reduce differences that are related to factors we are not interested in as this can help the differences we are interested in (like those between cell types or experimental groups) stand out more.
    It is possible to use the `ScaleData` function to regress out unwanted variation, but the creators of Seurat recommend using `SCTransform` for preprocessing if you want to do any regression. `SCTransform` automatically regresses out variation associated with sequencing depth (unique counts or nFeature_RNA) and can also regress out other variables. If you choose to use SCTransform in this tutorial, then you'll regress out the variation associated with the proportion of mitochondrial content, just like in [Seurat's original version of this tutorial](https://satijalab.org/seurat/articles/sctransform_vignette.html)

{% include _includes/cyoa-choices.html option1="Separate Preprocessing Steps" option2="SCTransform" default="Separate Preprocessing Steps" text="You can perform each preprocessing step separately, which might give you a better understanding of the different elements involved in preprocessing, or run them all at once using SCTransform." %}

<div class="Separate-Preprocessing-Steps" markdown="1">
><hands-on-title>Separate Preprocessing Steps</hands-on-title>
>
> 1. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Normalize with 'NormalizeData'`
>        - *"Method for normalization"*: `LogNormalize`
>
> 2. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `rds_out` (output of previous **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Identify highly variable genes with 'FindVariableFeatures'`
>        - *"Method to select variable features"*: `vst`
>        - *"Output list of most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>
> 3. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `rds_out` (output of previous **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Scale and regress with 'ScaleData'`
>        - *"Regress out a variable"*: `No`
>        - *"Features to scale"*: `All Features`
>
>    We'll choose to scale `All Features` here, instead of just the highly variable genes.
>
> 4. Rename the output as `Preprocessed Data`
>
{: .hands_on}
</div>

<div class="SCTransform" markdown="1">
><hands-on-title>SCTransform</hands-on-title>
>
> 1. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Complete all preprocessing with "SCTransform"`
>        - *"Genes to calculate residual features for"*: `all genes`
>            - *"How to set variable features"*: `set number of variable features`
>        - *"Output list of most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>        - *"Variable(s) to regress out"*: `percent.mt`
>
> 2. Rename the output as `Preprocessed Data`
>
>    If we inspect this dataset, we can see that a new assay called `SCT` has been created and is set as the default assay. Click on the {% icon galaxy-eye %}- of the following output to check this.
>
> 3. {% tool [Seurat Data Management](toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Preprocessed Data` (output of **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Inspect Seurat Object`
>        - *"Display information about"*: `General`
>
{: .hands_on}
</div>

### Visualise Highly Variable Genes

><hands-on-title>Visualise Results</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Preprocessed Data` (output of **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Plot Variable Genes with 'VariableFeaturePlot'`
>        - *"Label the top most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>
{: .hands_on}

<div class="Separate-Preprocessing-Steps" markdown="1">

![Most genes are grouped together at the bottom of the plot with low standardized variance. The selected variable genes are highlighted in red and labelled by name. They have higher standardized variances although most are still quite close to the non variable genes. The labels read PPBP, S100A9, LYZ, IGLL5, GNLY, FTL, PF4, FTH1, GNG11, and FCER1A.](../../images/scrna-seurat-pbmc3k/seurat_variable_genes.png "Plot showing the standardized variances of variable and non-variable genes. The top 10 most variable genes are labelled.")

</div>

<div class="SCTransform" markdown="1">

![Most genes are closely grouped together, right at the bottom of the plot with low standardized variance. The selected variable genes are highlighted in red and labelled by name. They are higher up on the plot, showing they have higher standardized variances although most are still quite close to the non variable genes. They are S100A9, GNLY, LYZ, S100A8, NKG7, FTL, GZMB, IGLL5, FTH1, and CCL5.](../../images/scrna-seurat-pbmc3k/seurat_variable_genes_SCT.png "Plot showing the standardized variances of variable and non-variable genes. The top 10 most variable genes are labelled.")

</div>

> <question-title></question-title>
> 1. What are the top 10 most variable genes in this dataset?
> 2. Why are we interested in these genes?
>
> > <solution-title></solution-title>
> >
> > 1. We can check the list of the top variable genes in our history by clicking on the {% icon galaxy-eye %} or see them on our variable features plot.
> >    <div class="Separate-Preprocessing-Steps" markdown="1">
> >
> >    |    |         |
> >    |----|---------|
> >    | 1  | PPBP    |
> >    | 2  | LYZ     |
> >    | 3  | S100A9  |
> >    | 4  | IGLL5   |
> >    | 5  | GNLY    |
> >    | 6  | FTL     |
> >    | 7  | PF4     |
> >    | 8  | FTH1    |
> >    | 9  | GNG11   |
> >    | 10 | S100A8  |
> >    {: .matrix}
> >
> >    </div>
> >
> >    <div class="SCTransform" markdown="1">
> >
> >    |    |          |
> >    |----|----------|
> >    | 1  | S100A9   |
> >    | 2  | GNLY     |
> >    | 3  | LYZ      |
> >    | 4  | S100A8   |
> >    | 5  | NKG7     |
> >    | 6  | FTL      |
> >    | 7  | GZMB     |
> >    | 8  | IGLL5    |
> >    | 9  | FTH1     |
> >    | 10 | CCL5     |
> >    {: .matrix}
> >
> >    </div>
> >
> >    > <comment-title></comment-title>
> >    > The list of highly variable genes you end up with will depend on which preprocessing route you chose. The two preprocessing routes use different methods to select features, so they won't always end up with the same genes, although there are likely to be some similarities. `SCTransform` also returns 3000 variable genes by default, rather than the 2000 selected by `FindVariableFeatures`. We can select more features with `SCTransform` because its normalisation method is better at removing technical effects from the data, so we believe that these additional genes reflect subtler biological variations rather than technical differences.
> >    {: .comment}
> >
> > 2. Single cell datasets contain a lot of information, including expression data for thousands of different genes. Some of these genes don't tell us much about the data, for example they might be housekeeping genes that are expressed at similar levels in most of our cells. We want to find the genes that can tell us most about the differences between our cells, so we want to identify the genes whose expression varies most across the dataset - focusing on these highly variable features should help us to uncover the biological differences we're looking for.
> >
> {: .solution}
{: .question}

# Dimensionality Reduction

Dimensional reduction is a key step in single cell analysis that simplifies our big, complex datasets enough to enable us to perform and understand further analyses.

In single cell analysis, we're comparing cells based on their patterns of gene expression - we're looking for cells with similar transcriptomic profiles. Each gene represents a dimension of the data. If we only found two genes being expressed in our cells, we could represent them on a 2-dimensional plot. The two axes of the plot would represent the expression levels of the two genes. Each cell would become a point on the plot, positioned according to its expression of these genes. We might spot one group of cells with high levels of both genes and another group that expresses large amounts of one gene and not much of the other.

For datasets with thousands of genes, the concept is essentially the same: each cell's expression profile defines its location in the high-dimensional expression space. Each gene is still a dimension or axis on the plot and each cell can be positioned along these axes based on their expression. It's just much harder to imagine a plot with thousands of different axes - that's why we need dimensional reduction!

We don't have to think of the expression of each of our thousands of genes individually. The expression of some genes can be correlated if they are affected by the same biological process - cells that express a lot of one gene might also express higher levels of other genes that have similar functions. Instead of keeping the information from each of these genes as separate axes or dimensions, we can compress them into a single dimension (e.g. an 'eigengene'). We can do the same for all of the correlated groups of genes in our data, going from thousands of individual genes to perhaps 10, 20, or 50 grouped dimensions.

 Dimensionality re

tools_json:
["Seurat Create", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1", "Seurat Data Management", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0", "Seurat Visualize", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0", "Seurat Preprocessing", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0", "Seurat Run Dimensional Reduction", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_reduce_dimension/seurat_reduce_dimension/5.0+galaxy0", "Seurat Find Clusters", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_clustering/seurat_clustering/5.0+galaxy0", "Filter", "Filter1", "Cut", "Cut1"]


topic: single-cell
tutorial: scatac-preprocessing-tenx
title: Pre-processing of 10X Single-Cell ATAC-seq Datasets
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["10x", "epigenetics"]
content_hash: 66e7a51b5fdfd9df3842a2567616b051
last_modified: 2026-05-27T09:01:49.741340
zenodo_link: https://zenodo.org/record/7855968
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scatac-preprocessing-tenx/tutorial.html

content:
Similar to bulk ATAC-Seq, single-cell ATAC-Seq (scATAC-seq) leverages the hyperactive Tn5 Transposase to profile open chromatin regions but at single-cell resolution. Thus helps in understanding cell type-specific chromatin accessibility from a heterogeneous cell population.

## Era of 10x Genomics

10x genomics has provided not only a cost-effective high-throughput solution to understanding sample heterogeneity at the individual cell level but has defined the standards of the field that many downstream analysis packages are now scrambling to accommodate. The gain in resolution reduces the granularity and noise issues that plagued the field of single-cell omics not long ago, where now individual clusters are much easier to decipher due to the added stability added by this gain in information.

## Library Preparation

![Library Preparation]({% link topics/single-cell/images/scatac-pre-processing/tenx_libprep_scatac.png %} "An overview of the 10x single-nuclei ATAC-seq library preparation")

The 10X barcoded gel beads consist of a pool of barcodes which are used to separately index each cell. The individual gel barcodes are delivered to each cell via flow cytometry, where each cell is fed single-file along a liquid tube and tagged with a 10X gel bead. The cells are then isolated from one another within thousands of nanoliter droplets, where each droplet is described by a unique 10x barcode that all reads in that droplet are associated with. The oil is then removed and all (now barcoded) DNA reads are pooled together to be sequenced.

More information can be found in the [reagent kit documentation](https://www.10xgenomics.com/support/single-cell-atac/documentation/steps/library-prep/chromium-single-cell-atac-reagent-kits-user-guide-v-2-chemistry).


## 10x Chemistries

There are two main reagent kits used during the library preparation. Below we can see the layout of the primers used in both chemistries. We can ignore most of these as they are not relevant, namely: the P5 and P7 Illumina primers are used in the Illumina [bridge amplification](https://en.wikipedia.org/wiki/Illumina_dye_sequencing#Bridge_amplification) process; the Sample Index is an 8bp primer which is related to the Chromium system that balances nucleotide bias and ensures that there is no sample overlap during the multiplexed sequencing. The primer of interest to us is the Cell Barcode (CB). Unlike scRNA-seq, Unique Molecular Identifiers (UMIs) are not used in scATAC-seq.

![chem]({% link topics/single-cell/images/scatac-pre-processing/tenx_primers_scatac.svg %} "10x scATAC-Seq sequencing library")

There are two major chemistries (v1 and v2) used in 10x scATAC-seq library construction. Technically, v2 chemistry is more sensitive and has better data yield. But the final sequencing library is the same. Hence, same the data analysis applies to both chemistries. Both chemistries generate the 4 FASTQ files containing:
* 8bp sample index sequence (I1)
* 50bp forward read (R1)
* 16bp barcode sequence (R2)
* 49bp reverse read (R3)

The major differences to 10x scRNA-seq sequencing libraries are: (a) barcode sequences get a separate FASTQ file and (b) there are no UMIs. Note that as opposed to the traditional FASTQ naming, R3 contains reverse mate-pair and R2 contains barcode sequence.


# Analysis Strategy

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

The tutorial is structured into four parts. The first part deals with the preprocessing of FASTQ files, quality control and mapping. Then in the second part, we will identify the open chromatin regions. In the third part, we will create a count matrix using the cell barcode and open chromatin regions information. In the final part, we will filter out empty cells and less informative peaks to produce a high-quality count matrix.

Here we will use a published data set from 10x genomics consisting of thousand peripheral blood mononuclear cells (PBMCs) extracted from a healthy donor. Analyzing these full datasets requires several hours of computation time. Hence, we subsampled the data so that the tutorial can be finished in a reasonable amount of time with sensible results. In the sub-sampled data, we have the reads that were mostly generated from chromosome 21. The original data has FASTQ files from two sequencing lanes *L001* and *L002*. We merged the sequences from both lanes. We also ignored the sample index FASTQ file as it is not useful for this analysis. Finally, we are left with three FASTQ files: **R1** (forward reads), **R2** (barcodes) and **R3** (reverse reads).

  * atac\_pbmc\_1k\_nextgem\_S1\_**R1**\_001\_chr21.fastq.gz
  * atac\_pbmc\_1k\_nextgem\_S1\_**R2**\_001\_chr21.fastq.gz
  * atac\_pbmc\_1k\_nextgem\_S1\_**R3**\_001\_chr21.fastq.gz

These files are provided in the [Zenodo](https://zenodo.org/record/7855968) data repository. To analyze the full datasets, get the original FASTQ files from [10x genomics website](https://www.10xgenomics.com/resources/datasets/1-k-peripheral-blood-mononuclear-cells-pbm-cs-from-a-healthy-donor-next-gem-v-1-1-1-1-standard-2-0-0)

# FASTQ processing and mapping

## Data upload and organization

We import all three FASTQ files into a history.

> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and rename it (e.g. scATAC-seq 10X preprocessing tutorial)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 1. Import the sub-sampled FASTQ data from [`Zenodo`](https://zenodo.org/record/7855968) or from the data library (ask your instructor)
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    ```
>    https://zenodo.org/record/7855968/files/atac_pbmc_1k_nextgem_S1_R1_001_chr21.fastq.gz
>    https://zenodo.org/record/7855968/files/atac_pbmc_1k_nextgem_S1_R2_001_chr21.fastq.gz
>    https://zenodo.org/record/7855968/files/atac_pbmc_1k_nextgem_S1_R3_001_chr21.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}
>
> <question-title></question-title>
> Inspect all the uploaded FASTQ files by clicking on the {% icon galaxy-eye %} symbol.
> 1. Which FASTQ file contains the barcode sequence?
> 1. What is the length of the barcode?
>
> > <solution-title></solution-title>
> > 1. In this case, R2 contains the cell barcodes
> > 1. 16bp
> >
> {: .solution}
{: .question}


## Attach cell barcodes to FASTQ headers

First, we will attach the barcodes from the barcodes FASTQ file to the ids of forward and reverse reads that carry the association between reads and barcodes throughout the analysis. We will use this information to demultiplex the cells at the end while creating the count matrix. We will be left with only two FASTQ files of forward and reverse reads after this step.

> <comment-title></comment-title>
>
> {% tool [Sinto barcode](toolshed.g2.bx.psu.edu/repos/iuc/sinto_barcode/sinto_barcode/0.9.0+galaxy1) %}  consumes the barcode sequence from the `atac_pbmc_1k_nextgem_S1_R2_001_chr21.fastq.gz` file and uses `:` delimiter to prepend it to both the forward and reverse read ids. The tool expects the same number of reads in all three FASTQ files and should be present in the same order.
{: .comment}

> <hands-on-title></hands-on-title>
>
> {% tool [Sinto barcode](toolshed.g2.bx.psu.edu/repos/iuc/sinto_barcode/sinto_barcode/0.9.0+galaxy1) %}  with the following parameters:
>    - *"FASTQ file containing cell barcode sequences"*: `atac_pbmc_1k_nextgem_S1_R2_001_chr21.fastq.gz`
>    - *"Single or Paired-end data"*: `Paired`
>    - *"Forward reads FASTQ file"*: `atac_pbmc_1k_nextgem_S1_R1_001_chr21.fastq.gz`
>    - *"Reverse reads FASTQ file"*: `atac_pbmc_1k_nextgem_S1_R3_001_chr21.fastq.gz`
>    - *"Number of bases to extract from barcode-containing FASTQ"*: `16`
>
>    > <comment-title>Sinto barcode output</comment-title>
>    >
>    > After a successful run, the barcode sequences from the barcode FASTQ file are prepended to the read names. We can inspect the output FASTQ files by clicking on the {% icon galaxy-eye %} symbol of the *barcoded read 1* file.
>    {: .comment}
>
{: .hands_on}


Now let's proceed with the next steps QC, mapping and peak calling as we do with the bulk ATAC-Seq data. For a thorough description of bulk ATAC-Seq analysis, please follow the bulk [ATAC-Seq data analysis tutorial]({% link topics/epigenetics/tutorials/atac-seq/tutorial.md %}). Here we use a slightly different workflow. Here we map reads using **BWA-MEM** instead of **Bowtie2**, then create ATAC fragments file that also stores the cell barcode information and finally use **MACS2** for peak calling. Note that the workflow from bulk ATAC-seq tutorial should also result in similar set of peaks.

## FASTQ Quality control
First things first. Let's do a basic FASTQ quality control using **FastQC**

> <hands-on-title></hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters:
>       - *"Raw read data from your current history"*: Use {% icon param-files %} **Multiple datasets** to choose both `barcoded read 1` and `barcoded read 2` (outputs of **Sinto barcode** {% icon tool %}).
> 2. Inspect the web page output of **FastQC** {% icon tool %} for the `barcoded read 1` sample.
>
> > <question-title></question-title>
> >
> > 1. How many reads are in the FASTQ?
> > 2. Which sections have a warning?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 685686 reads.
> > > 2. The 2 steps below have warnings:
> > >
> > >    1. **Per base sequence content**
> > >
> > >       It is well known that the Tn5 has a strong sequence bias at the insertion site. You can read more about it in {% cite Green2012 %}.
> > >
> > >    2. **Sequence Duplication Levels**
> > >
> > >       The read library quite often has PCR duplicates that are introduced
> > >       simply by the PCR itself. We will remove these duplicates later on.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

> <tip-title>Adapter trimming</tip-title>
>
> There is only a little amount of Nextera Transposase Sequence in the reads (see Adapter Content section of **FastQC** output). In this case, we skip the adapter trimming step and proceed to the mapping step. If your data has a significant amount of Nextera Transposase Sequences, then proceed to adapter trimming using either **cutadapt** or **Trim Galore!**. **Trim Galore** can automatically detect the adapter sequence. If you use **Cutadapt**, follow the bulk ATAC-Seq tutorial [Trimming Reads]({% link topics/epigenetics/tutorials/atac-seq/tutorial.md %}) step.
{: .tip}

## Mapping reads to a reference genome
Now we map the reads to a reference genome using {% tool [BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %}.

> <hands-on-title>Mapping</hands-on-title>
>
> {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} with the following parameters:
>    - *"Using reference genome"*: `hg38`
>    - In *"Single or Paired-end reads"*: `Paired`
>      - In *"Select first set of reads "*: `barcoded read 1` (output of **Sinto barcode** {% icon tool %})
>      - In *"Select second set of reads "*: `barcoded read 2` (output of **Sinto barcode** {% icon tool %})
>    - *"Select analysis mode"*: `1.Simple Illumina mode`
>    - *"BAM sorting mode"*: `Sort by chromosomal coordinates`
{: .hands_on}

# Peak calling
In scRNA-seq we always have the standard set of genes (usually downloaded from public databases) to quantify the expression levels. For scATAC-seq, there are no such reference open chromatin regions because the regions of chromatin accessibility are tissue dependent. Hence, we first have to detect open chromatin regions. There are several ways of doing this. The most common ways are the following:
* Find a published bulk ATAC-seq data that is the closest to your scATAC-seq data and use the regions from the bulk ATAC-seq data
* Chunk the genome into equal-sized bins and use these bins as reference locations for quantification
* Combine the data from all the cells of the scATAC-seq data together and detect open chromatin regions from the data. Later use these detected regions for quantification

In this tutorial, we opt for the 3rd option.

## Create scATAC-seq fragments file
An ATAC-seq fragment file is a BED file with Tn5 integration sites, the cell barcode associated with the fragment, and the frequency of the sequenced fragment. PCR duplicates are collapsed. Here we filter out the alignments with low mapping quality.

> <comment-title>on Tn5 insertion</comment-title>
>
> When Tn5 cuts an accessible chromatin locus it inserts adapters separated by 9bp ({% cite Kia2017 %}):
> ![Nextera_Library_Construction]({% link topics/epigenetics/images/atac-seq/NexteraLibraryConstruction.jpg %} "Nextera Library Construction")
>
> This means, to have the read start site reflecting the center of where Tn5 bound, the reads on the positive strand should be shifted 4 bp to the right and reads on the negative strands should be shifted 5 bp to the left as in [Buenrostro et al. 2013](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3959825). **Genrich** can apply these shifts when ATAC-seq mode is selected. In most cases, we do not have 9bp resolution so we don't take it into account but if you are interested in the footprint, this is important.
{: .comment}

> <hands-on-title>Create scATAC fragments</hands-on-title>
> 1. {% tool [Sinto fragments](toolshed.g2.bx.psu.edu/repos/iuc/sinto_fragments/sinto_fragments/0.9.0+galaxy1) %} with the following parameters:
>    - *"Input BAM file"*: `mapped reads in BAM format` (output of **Map with BWA-MEM** {% icon tool%})`
>    - *"Minimum MAPQ required to retain fragment"*: `30`
>    - *"Regular expression used to extract cell barcode from read name"*: `[^:]*` (matches all characters up to the first colon)
>    - *"Number of bases to shift Tn5 insertion position by on the forward strand"*: `4`
>    - *"Number of bases to shift Tn5 insertion position by on the reverse strand"*: `-5`
>    - *"Take cell barcode into account when collapsing duplicate fragments"*: `Yes`
>
> 1. {% tool [bedtools SortBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2) %} with the following parameters:
>    - *"Sort the following BED/bedGraph/GFF/VCF/EncodePeak file *"*: `fragments BED` (output of **Sinto fragments** {% icon tool%})`
>    - *"Sort by"*: `chromosome, then by start position (asc)`
> 1. Rename the dataset `sorted fragments`
{: .hands_on}

## Call Peaks

We have now finished the data preprocessing. Next, to find regions corresponding to potential open chromatin regions, we want to identify regions where reads have piled up (peaks) greater than the background read coverage. The tools which are currently used are [Genrich](https://github.com/jsh58/Genrich) and [MACS2](https://github.com/taoliu/MACS). MACS2 is more widely used.
At this step, two approaches exist:

- The first one is to select only pairs whose fragment length is below 100bp corresponding to nucleosome-free regions and to use a peak calling like you would do for a ChIP-seq, joining signal between mates. The disadvantage of this approach is that you can only use it if you have paired-end data and you will miss small open regions where only one Tn5 is bound.
- The second one chosen here is to use all reads to be more exhaustive. In this approach, it is very important to re-center the signal of each read on the 5' extremity (read start site) as this is where Tn5 cuts. Indeed, you want your peaks around the nucleosomes and not directly on the nucleosome:

![ATAC-Seq_reads_relative_to_nucleosomes]({% link topics/epigenetics/images/atac-seq/schemeWithLegend.jpg %} "Scheme of ATAC-Seq reads relative to nucleosomes")

If we only assess the coverage of the 5' extremity of the reads, the data would be too sparse and it would be impossible to call peaks. Thus, we will extend the start sites of the reads by 200bp (100bp in each direction) to assess coverage. We call peaks with MACS2. In order to get the coverage centered on the 5' extended 100bp on each side we will use `--shift -100` and `--extend 200`:
![MACS2_peak_centering_options]({% link topics/epigenetics/images/atac-seq/macs2Options.jpg %} "MACS2 options to get 100bp each side")

> <hands-on-title>Call peaks with MACS2</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6) %} with the following parameters:
>    - *"Are you pooling Treatment Files?"*: `No`
>    - *"ChIP-Seq Treatment File"*: `sorted fragments`
>    - *"Do you have a Control File?"*: `No`
>    - *"Format of Input Files"*: `Single-end BED`
>    - *"Effective genome size"*: `H. sapiens (2.7e9)`
>    - *"Build Model"*: `Do not build the shifting model (--nomodel)`
>        - *"Set extension size"*: `200`
>        - *"Set shift size"*: `-100`. It needs to be - half the extension size to be centered on the 5'.
>    - *"Additional Outputs"*:
>        - Check `Peaks as tabular file (compatible with MultiQC)`
>        - Check `Peak summits`
>        - Check `Scores in bedGraph files`
>    - In *"Advanced Options"*:
>        - *"Composite broad regions"*: `No broad regions`
>            - *"Use a more sophisticated signal processing approach to find subpeak summits in each enriched peak region"*: `Yes`
>        - *"How many duplicate tags at the exact same location are allowed?"*: `all`
>
>    > <comment-title>Why keeping all duplicates is important</comment-title>
>    >
>    > We previously removed duplicates using **Sinto fragment** {% icon tool %} using paired-end information. If two pairs had identical R1 but different R2, we knew it was not a PCR duplicate. Because we converted the BAM to BED we lost the pair information. If we keep the default (removing duplicates) one of the 2 identical R1 would be filtered out as duplicate.
>    {: .comment}
>    > <comment-title>BED / encode narrowPeak format</comment-title>
>    > If you are not familiar with BED format or encode narrowPeak format, see the [BED Format](https://genome.ucsc.edu/FAQ/FAQformat.html)
>    {: .comment}
>
{: .hands_on}

# Count matrix creation

Now we have ATAC fragments with cell barcode information as well as the peaks to create a scATAC count matrix. In the context of ATAC-seq, the peaks are also often called features. But the term *feature* is a generic term that denotes a *peak* in the case of scATAC-seq data and a *gene* in case of scRNA-seq data. So the count matrix we build is similar to that of scRNA-seq data with the exception that here we have open chromatin regions instead of genes. In this tutorial, both the terms *feature* and *peak* indicate an *open chromatin region*.

For count matrix creation, we will use **Build count matrix** from **EpiScanpy** tool suite. It takes fragments file and the peaks in either BED format or directly a narrowPeak file of **MACS2** output. Regardless of file type, the tool only considers the first 3 columns which contain chromosome id, start and end positions of the peak regions. Parameters like `--call-summits` from our MACS2 run can result in different peaks with the same peak boundaries. It is always a  good practice to remove duplicate peak regions so that our final count matrix will have unique features.


> <hands-on-title>Extract unique peak regions</hands-on-title>
>
> 1. {% tool [Unique occurrences of each record](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0) %} with the following parameters:
>    - *"File to scan for unique values"*: `narrow Peaks` (output of **MACS2** {% icon tool %})
>    - In *"Advanced Options"*: `Show Advanced Options`
>        - *"Column start"*: `Column: 1`
>        - *"Column end"*: `Column: 3`
>
> > <question-title></question-title>
> >
> > 1. How many peak regions were duplicated?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There were initially 1229 regions in the `narrow Peaks` file. Now there are 1046 regions after deduplication. Around 15% (184) of regions have the same peak boundaries.
> > >
> > {: .solution}
> >
>   {: .question}
{: .hands_on}

## AnnData

The most common format, called [`AnnData`](https://anndata.readthedocs.io/en/stable/), stores the matrix as well as gene and cell annotations in a concise, compressed and extremely readable manner:

![Anndata format]({% link topics/single-cell/images/scrna-pre-processing/tenx_anndata.png %} "<code>AnnData</code> format stores a count matrix <code>X</code> together with annotations of observations (i.e. cells) <code>obs</code>, variables (i.e. peaks) <code>var</code> and unstructured annotations <code>uns</code>.")

This format is used by [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({% cite wolf2018scanpy %}) and [EpiScanpy](https://colomemaria.github.io/episcanpy_doc/index.html) ({% cite danese2021episcanpy %}) tool suites for analyzing single-cell omics data. So we need first to import the matrix and annotations of peaks and cells (present in fragments BED file) into an `AnnData` object.

> <hands-on-title>Build count matrix with EpiScanpy</hands-on-title>
>
> 1. {% tool [Build count matrix with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_build_matrix/episcanpy_build_matrix/0.3.2+galaxy1) %} with the following parameters:
>    - *"ATAC fragments file"*: `sorted fragments`
>    - *"Features file"*: output of **Unique** tool {% icon tool %}
>    - *"Normalize peak sizes?"*: `Keep the peaks as they are`
>    - In *"Select the chromosomes of the species you are considering"*: `Custom list of chromosomes`
>        - *"Enter comma seperated list of chromosome ids (without chr prefix)"*: `21`
> 2. Rename the generated file to `PBMC 1k chr21 Anndata`
> 3. Check that the format is `h5ad`
>
> > <tip-title>Chromosome selection</tip-title>
> >
> > In this example data, our reads are from the *chr21*. Hence, we input `21` for chromosome ids. For human or mouse genomes, directly select from the dropdown list. Make sure that all the canonical chromosomes are present in the fragments file.
> >
> {: .tip}
>
{: .hands_on}

Because the `AnnData` format is an extension of the HDF5 format, i.e. a binary format, an `AnnData` object can not be inspected directly in Galaxy by clicking on the {% icon galaxy-eye %} (**View data**) icon. Instead, we need to use a dedicated tool from the **AnnData** suite.

> <hands-on-title>Inspect an AnnData object</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `General information about the object`
>
> 2. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 27388 × 1046
>    > ```
>    >
>    > 1. How many observations are there? What do they represent?
>    > 2. How many variables are there? What do they represent?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 27,388 observations, representing the cells.
>    > > 2. There are 1046 variables, representing the peaks.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    > <comment-title>Faster Method for General Information</comment-title>
>    >
>    > * To view general information of any *Anndata* file:
>    >    * Click on the name of the dataset in the history to expand it.
>    >    * General Anndata information would be given in the expanded box:
>    >
>    >      e.g.
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    27388 x 1046
>    >      ```
>    > * For more specific queries, {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} is required.
>    {: .comment}
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > 	chr21_10272276_10272566	chr21_10330743_10330987	chr21_10489276_10489476	chr21_13979698_13980574	chr21_14027127_14027558 ...
>    > AAAAAATACCCGAAGG	0.0	0.0	0.0	0.0
>    > AAAAACAAAAATATAC	0.0	0.0	0.0	0.0
>    > AAAAATAACAATGATA	0.0	0.0	0.0	0.0
>    > AAAACCGTGAGAGTCT	0.0	0.0	0.0	0.0
>    > AAAACCTCACCCAACA	0.0	0.0	0.0	0.0
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a table with 27,388 lines (observations or cells) and 1046 columns (variables or peaks): the count matrix for each of the 1046 peaks and 27,388 cells. The 1st row contains the peak location as an annotation of the columns and the 1st column the barcodes of the cells as an annotation of the rows.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
> 6. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AAAAAATACCCGAAGG
>    > AAAAACAAAAATATAC
>    > AAAAATAACAATGATA
>    > AAAACCGTGAGAGTCT
>    > AAAACCTCACCCAACA
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is tabular with annotations of the observations. i.e. the cells. Here there are only the barcodes as annotation, so only one column, being also the index for the count matrix.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 7. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 8. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > chr21_10272276_10272566
>    > chr21_10330743_10330987
>    > chr21_10489276_10489476
>    > chr21_13979698_13980574
>    > chr21_14027127_14027558
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is tabular with annotations of the variables, i.e. the peaks.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

# Producing a Quality Count Matrix
We just created a count matrix with all detected peaks and non-empty barcodes. In this section of the tutorial, we will filter the features and barcodes by a series of quality assessments to generate a high-quality count matrix. The final count matrix can be used for further downstream analysis like clustering, differential chromatin analysis and cell type annotation.

The current matrix contains peaks from chromosome 21 only and is too small to carry out this type of analysis. From this step onwards we will use the full count matrix generated from the original FASTQ files. Please import the following anndata file to start:

```
https://zenodo.org/record/7855968/files/atac_pbmc_1k_uniq_peaks.h5ad
```

> <question-title></question-title>
>
> 1. How many barcodes and features are there?
> 1. What do you observe in comparison to a scRNA-seq count matrix?
>
> > <solution-title></solution-title>
> >
> > 1. There are 441038 barcodes and 70120 features
> > 2. The initial number of barcodes in this raw count matrix is in the same range as the scRNA-seq data. But there is roughly double the number of features compared to a scRNA-seq dataset. The number of features varies from dataset to dataset. Based on the tools and parameters used for peak calling, the size of the initial feature set can grow up to several hundred of thousands.
> >
> {: .solution}
>
{: .question}

## Initial filtering to remove potential empty features and cells
First remove any potential empty features or barcodes. A non-empty cell should have a minimum number of non-empty features and a non-empty feature should be present in a minimum number of non-empty cells. Here we will different minimum thresholds for filtering cells and features.

> <hands-on-title>Initial filtering</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: `atac_pbmc_1k_uniq_peaks.h5ad`
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of features expressed, using 'pp.filter_cells'`
>    - *"Filter"*: `Minimum number of features expressed`
>    - *"Minimum features"*: `100`
>
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: Output of the previous (`pp.filter_cells`) step
>    - *"Method used for filtering"*: `Filter features based on counts and numbers of features expressed, using 'pp.filter_features'`
>    - *"Filter"*: `Minimum number of cells expressed`
>    - *"Minimum features"*: `1`
>
> > <question-title></question-title>
> >
> > How much data was filtered out?
> >
> > > <solution-title></solution-title>
> > >
> > > The resulting matrix has dimensions of 1815 x 67766, i.e., more than 99.5% of the cells and less than 4% of features were filtered out. This indicates the high sparsity of the count matrix.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

## Quality control and visualization
We will first plot the number of features per cell. This information is stored as `nb_features` layer of observations. As we are dealing with numbers with huge variances, it is sometimes hard to visualize them. Hence, we compute the *log* of `nb_features` first then plot the absolute and log values together.

> <hands-on-title>Violing plots of number of features</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: Output of previous (`pp.filter_features`) step
>    - *"Method used for filtering"*: `Compute log10 of nb_features`
> 2. Check if observations have an extra `log_nb_features` in addition to `nb_features` layer.
> 3. Rename the resulting dataset to `PBMC 1k after initial filtering`
> 3. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k after initial filtering`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>      - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>        - *"Keys for accessing variables"*: `nb_features, log_nb_features`
>      - In *"Violin plot attributes"*:
>        - *"Add a stripplot on top of the violin plot"*: `Yes`
>          - *"Add a jitter to the stripplot"*: `Yes`
>            - *"Size of the jitter points"*: `0.4`
>        - *"Display keys in multiple panels"*: `Yes`
>      - In *"Parameters for seaborn.violinplot"*:
>        - *"Color for all of the elements"*: `DarkCyan`
>
> > <question-title></question-title>
> >
> > What trends of the data do you observe in the plots?
> >
> > > <solution-title></solution-title>
> > > ![IntialViolin]({% link topics/single-cell/images/scatac-pre-processing/violin_after_initial_filtering.png %} "Violin plot of number of features per cell after initial filtering")
> > > Both plots indicate that there are two distinct sets of cells:
> > >    1. Cells with an acceptable number of open chromatin regions.
> > >    2. Cells with only a few detected peaks. These cells are uninformative and should be excluded.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

To determine decent filtering thresholds, we will further look at some histograms of cell-peak coverage trends. We will plot histograms of the number of open features per cell and feature commonness in cells.

> <hands-on-title>Coverage cells</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage cells: Histogram of the number of open features (in the case of ATAC-seq data) per cell, using 'pp.coverage_cells'`
>    - *"Binarized matrix?"*: `No`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `1000`
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage cells: Histogram of the number of open features (in the case of ATAC-seq data) per cell, using 'pp.coverage_cells'`
>    - *"Binarized matrix?"*: `No`
>    - *"Log transform?"*: `Yes`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `1000`
> 3. Now observe the plots
>
> > <question-title></question-title>
> >
> > 1. What information do you see in these plots?
> > 1. Do you think our minimum number of features threshold of 1000 indicated in the plot makes sense?
> >
> > > <solution-title></solution-title>
> > > ![CoverageCells]({% link topics/single-cell/images/scatac-pre-processing/coverage_cells.png %} "Histogram of the number of open chromatin regions per cell")
> > > ![CoverageCellsLog]({% link topics/single-cell/images/scatac-pre-processing/coverage_cells_log.png %} "Histogram of the number of open chromatin regions per cell in log scale")
> > > 1. The plots show the number of open chromatin regions per cell in 50 bins on the x-axis and the number of cells per bin on the y-axis. It is basically a histogram of the number of open chromatin regions per cell.
> > > 2. From the first plot with absolute counts, we see that with our threshold of 1000, we can separate between the cells with few detected peaks and cells with high chromatin accessibility.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

> <hands-on-title>Coverage features</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage features: Distribution of the feature commonness in cells, using 'pp.coverage_features'`
>    - *"Binarized matrix?"*: `No`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `5`
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage features: Distribution of the feature commonness in cells, using 'pp.coverage_features'`
>    - *"Binarized matrix?"*: `No`
>    - *"Log transform?"*: `Yes`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `5`
> 3. Now observe the plots
>
> > <question-title></question-title>
> >
> > 1. What information do you see in these plots?
> > 1. Do you think our minimum number of cells threshold of 5 indicated in the plot makes sense?
> >
> > > <solution-title></solution-title>
> > > ![CoverageFeatures]({% link topics/single-cell/images/scatac-pre-processing/coverage_features.png %} "Histogram of feature commonness in cells")
> > > ![CoverageFeaturesLog]({% link topics/single-cell/images/scatac-pre-processing/coverage_features_log.png %} "Histogram of feature commonness in cells in log scale")
> > > 1. The plots show a histogram of the number of cells sharing a feature. As we initially pooled the data from all the cells to detect the peaks, it is expected to see only a small number of cells have more than 10000 peaks in common.
> > > 2. The red vertical line of our 5 cells threshold is nearly at the left end of the histogram representing the majority of the features have at least 5 cells in common.
> > > From the log scale plot it is also clear that there is a sharp increase in the feature commonness from at least 10 cells (x-axis 1.0).
> > > So our threshold of 5 is a decent cutoff for filtering out the features. From the plots, only a very few non-informative features are left to be filtered out.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

## Final filtering to acquire quality cells and features

Based on the above QC plots, we will filter out all the cells with less than 1000 features and all the features that have less than 5 cells in common. Additionally, we will also filter out extreme outlier cells (which can be seen in the initial violin plot or coverage cells plot) with more than 12500 features.

> <hands-on-title>Final filtering based on QC plots</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of features expressed, using 'pp.filter_cells'`
>    - *"Filter"*: `Minimum number of features expressed`
>    - *"Minimum features"*: `1000`
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: Output of previous (`pp.filter_cells`) step
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of features expressed, using 'pp.filter_cells'`
>    - *"Filter"*: `Maximum number of features expressed`
>    - *"Minimum features"*: `12500`
>
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: Output of previous (`pp.filter_cells`) step
>    - *"Method used for filtering"*: `Filter features based on counts and numbers of features expressed, using 'pp.filter_features'`
>    - *"Filter"*: `Minimum number of cells expressed`
>    - *"Minimum features"*: `5`
>
> > <question-title></question-title>
> >
> > What are the dimensions of the count matrix after the final filtering?
> >
> > > <solution-title></solution-title>
> > >
> > > The resulting matrix has 1024 cells and 67719 peaks. The number of cells after filtering is close to what we expect in this dataset.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

# Conclusion


In this tutorial, we have learned how to map reads from 10X ATAC-seq data, identify open chromatin regions from aggregated cell data and generate a count matrix in *Anndata* format. We also performed quality control and filtered out empty and uninformative cells and features to generate a high-quality count matrix for downstream analysis.

Galaxy workflows for generating [Annadata from FASTQ files](workflows/scATAC-seq-FASTQ-to-Count-Matrix.ga) and [quality filtering of raw count matrix](workflows/scATAC-seq-Count-Matrix-Filtering.ga) are provided.scatac_tenx.ga).

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %}).

tools_json:
["Sinto barcode", "toolshed.g2.bx.psu.edu/repos/iuc/sinto_barcode/sinto_barcode/0.9.0+galaxy1", "FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2", "Map with BWA-MEM", "Sinto fragments", "toolshed.g2.bx.psu.edu/repos/iuc/sinto_fragments/sinto_fragments/0.9.0+galaxy1", "bedtools SortBED", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2", "MACS2 callpeak", "toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6", "Unique occurrences of each record", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0", "Build count matrix with EpiScanpy", "toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_build_matrix/episcanpy_build_matrix/0.3.2+galaxy1", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1", "scATAC-seq Preprocessing with EpiScanpy", "toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1", "Plot with scanpy", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0"]


topic: single-cell
tutorial: bulk-deconvolution-evaluate
title: Evaluating Reference Data for Bulk RNA Deconvolution
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["transcriptomics"]
content_hash: 93bd127fdb7cb1f8816a6714eb29d075
last_modified: 2026-05-27T09:01:49.737866
zenodo_link: https://zenodo.org/records/5719228
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-deconvolution-evaluate/tutorial.html

content:
There are various methods to estimate the proportions of cell types in bulk RNA data. Since the actual cell proportions of the data are unknown, how do we know if our tools are producing accurate results?

In this tutorial we will be using single-cell data with known cell-type proportions in order to create pseudo-bulk RNA data. We will then estimate the cell-type proportions of this pseudo-bulk data using the currently available deconvolution tools within Galaxy. Since we know the true proportion values, we will be able to measure and compare the accuracy of the tools' predictions.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get the single-cell data

First we need to create a new history in Galaxy and load in our single-cell data. We are going to use the single-cell dataset from a previous deconvolution tutorial found here: [https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music/tutorial.html]({% link topics/single-cell/tutorials/bulk-music/tutorial.md %}).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Evaluating Reference Data"*
> 2. Import the files from [Zenodo](https://zenodo.org/records/5719228) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    * Human pancreas single-cell RNA datasets (tag: `#scrna`)
>      ```
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.expression.tabular
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.phenotype.tabular
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Check the datatypes are `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add a `#metadata` tag to `EMTABesethealthy.phenotype.tabular` and a `#expression` tag to `EMTABesethealthy.expression.tabular`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

## Inspecting the single-cell data

Before continuing lets quickly inspect our single-cell data. We can find all of the cell types present in the data alongside their proportions by using the count tool to count the occurrence of each cell type category in the metadata file.

> <hands-on-title>Get cell counts</hands-on-title>
>
> 1. {% tool [Count](Count1) %} with the following parameters:
>    - {% icon param-file %} *"from dataset"*: `EMTABesethealthy.phenotype.tabular`
>    - *"Count occurrences of values in column(s)"*: `Column 5`
>    - *"Delimited by"*: `Tab`
>    - *"How should the results be sorted?"*: `With the most common value first`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Cell type counts`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

We can see from the output table below, there are various cell types present in the data. Note that many of the cell types have very low proportion values, this should be kept in mind later on as cell types that appear only a hand full of times (or even just once!) in the data may not be very useful and only add noise.

| Cell Type               | Count |
|-------------------------|-------|
| alpha                   | 443   |
| beta                    | 171   |
| ductal                  | 135   |
| acinar                  | 112   |
| gamma                   | 75    |
| delta                   | 59    |
| unclassified endocrine  | 29    |
| co-expression           | 26    |
| PSC                     | 23    |
| endothelial             | 13    |
| epsilon                 | 5     |
| mast                    | 4     |
| unclassified            | 1     |
| MHC class II            | 1     |

> <question-title>Inspecting the single-cell data</question-title>
>
> 1. How many cells are in the single-cell data?
> 2. How many cell types are present in the data?
>
> > <solution-title></solution-title>
> >
> > 1. Inspecting the general information of `EMTABesethealthy.expression.tabular` we can see that there are **1,097** cells in the data as there are 1,098 columns (we need to subtract 1 for the header).
> > 2. Looking at the output of the {% tool [Count](Count1) %} tool (or the above table), there are **14** distinct cell types in the data.
> >
> {: .solution}
>
{: .question}

# Process the single-cell data

In order to get a good understanding of the accuracy of our deconvolution tools, we are going to run our evaluations multiple times. This approach ensures that a single good or bad evaluation does not disproportionately represent the tool's overall performance.

However, instead of running all of our tools multiple times for each evaluation (which would be quite time consuming!), we will leverage "batch computation" in Galaxy. By storing our data in collections, any tools or workflows that use those collections will automatically run multiple times (once for each element in the collection). We will now perform some pre-processing of our data to get it into the right format.

## Transpose expression matrix

If we inspect the expression data file downloaded earlier, we can see that currently the rows represent genes and columns represent cells. However, this needs to be swapped for the later workflows. To fix this we will transpose the expression matrix.

> <hands-on-title>Transpose expression matrix</hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `EMTABesethealthy.expression.tabular`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Transposed expression matrix`
>
{: .hands_on}

## Generate batch mode collections

For this tutorial we will run the evaluations **20** times, this will both help improve the sample size and allow us to determine the consistency of the tools, whilst being small enough to run in a reasonable amount of time.

We will now duplicate our single-cell data 20 times and store it in a collection. This will be done for both the expression data and metadata files.

> <hands-on-title>Generate collections from data</hands-on-title>
>
> 1. {% tool [Duplicate file to collection](__DUPLICATE_FILE_TO_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset"*: `EMTABesethealthy.phenotype.tabular`
>    - *"Size of output collection"*: `20`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Metadata`
>
> 3. {% tool [Duplicate file to collection](__DUPLICATE_FILE_TO_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset"*: `Transposed expression matrix`
>    - *"Size of output collection"*: `20`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Expression data`
>
{: .hands_on}


# Create pseudo-bulk and actual cell proportions

We are now going to run our first workflow! This workflow will extract a subsample from the data containing 200 random cells. The workflow will then perform two things with this subsample:

1. Count the cell types and proportions of the data in order to be used as reference later against the predicted proportion values
2. Remove the cell types and convert the single-cell data into pseudo-bulk data to be later inputted into the deconvolution tools.

The above will be done twice to emulate multiple "subjects". Since the deconvolution tools will be expecting the bulk-RNA data to comprise of at least 2 subjects (each with their own bulk data). For this tutorial our subjects will simply be called **A** and **B**. However, in the real world these subjects could be different patients, tissue samples, diseased/healthy, etc.

> <comment-title>Different Results</comment-title>
> Note that since we are selecting 20 samples, each containing 200 randomly selected cells. The plots and results presented in this tutorial will differ from your own. There will be some similarities such as certain cells being in higher proportion to others but the exact values with differ!
{: .comment}

**Remember** since we have a collection of 20 inputs, the output of this workflow will be a collection of 20 elements, each corresponding to the input elements. Each output will have its own random selection of 200 cells.

> <hands-on-title>Run pseudo-bulk and actual proportions workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-1-create-data.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow pseudobulk and actual proportions** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Metadata"*: `Metadata`
>    - {% icon param-collection %} *"Expression Data"*: `Expression Data`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
> 3. Add a tag labelled `#A` to the first "Actual cell proportions" and "Pseudobulk" collections
>
> 4. Add a tag labelled `#B` to the second "Actual cell proportions" and "Pseudobulk" collections
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=c7936e9378155efe&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

The output of this workflow will be the psuedo-bulk and actual cell proportions for both samples A and B. If you inspect one of the elements in the `Actual Cell Proportions` collection, you should see a table similar to the following:

|                         | A_actual   |
|-------------------------|------------|
| acinar                  | 0.090000   |
| alpha                   | 0.415000   |
| beta                    | 0.170000   |
| co-expression           | 0.050000   |
| delta                   | 0.070000   |
| ductal                  | 0.105000   |
| endothelial             | 0.015000   |
| gamma                   | 0.050000   |
| mast                    | 0.010000   |
| unclassified endocrine  | 0.025000   |

Comparing the above table with the cell-type counts of the original single-cell data, does this look correct? Well the top 3 cell-types with the highest proportion in the single-cell data are: alpha, beta, ductal. Which aligns with the proportion values of the above data! There may be some variance due to the randomly selected cells. Also note that some of the lesser common cell types (like `MHC class II`) aren't present in the above table, again this is due to the 200 randomly selected cells for this specific sample and isn't of concern.

# Perform deconvolution on the pseudo-bulk data

Now that we have our pseudo-bulk data alongside the actual proportion values. Our next step is to run deconvolution to get predicted cell-type proportions! Currently, Galaxy contains two tools for performing deconvolution: **MuSiC** and **NNLS**. We will use both of these tools in this tutorial and compare their results together.

## Generate expression set objects

First we will need to use the single-cell data to build an expression set object, which will be used in the following workflow to perform deconvolution.

**Note: We are using the original imported data here, not the transposed data or collections.**

> <hands-on-title>Build the Expression Set object</hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy3) %} with the following parameters:
>    - *"Awk- actual header (A)"*: `BEGIN { print "A_actual\tcell_type" } { print $0 }`
>    - *"Awk - infer header (A)"*: `BEGIN { print "0\tA_infer\t0" } NR > 1 {print $0 }`
>    - {% icon param-file %} *"Assay Data"*: `EMTABesethealthy.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `EMTABesethealthy.phenotype.tabular` (Input dataset)
>    - *"Awk- actual header (B)"*: `BEGIN { print "B_actual\tcell_type" } { print $0 }`
>    - *"Awk - infer header (B)"*: `BEGIN { print "0\tB_infer\t0" } NR > 1 {print $0 }`
>
>    > <comment-title></comment-title>
>    >
>    > An ExpressionSet object has many data slots, the principle of which are the experiment data (*exprs*), the phenotype data (*pData*), as well metadata pertaining to experiment information and additional annotations (*fData*).
>    {: .comment}
>
{: .hands_on}

Similar to the expression data, this ExpressionSet object needs to be duplicated 20 times into a collection for later batch processing.

> <hands-on-title>Generate ESet collection</hands-on-title>
>
> 1. {% tool [Duplicate file to collection](__DUPLICATE_FILE_TO_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset"*: `RData ESet Object` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Size of output colection"*: `20`
>
> 2. **Rename** {% icon galaxy-pencil %} output `ESet Object`
>
{: .hands_on}

## Run the Workflow

The following workflow will take the two pseudo-bulk samples (A and B), as well as the original single-cell data as reference and output the deconvolution results for both samples and deconvolution methods. Thus producing 4 output collections. The pdf results of the deconvolution tools will also be outputted from the workflow but won't be needed for the tutorial.

> <hands-on-title>Run inferring cellular proportions workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-2-deconv.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow inferring cellular proportions** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Pseudobulk - A"*: `Pseudobulk (#A)`
>    - {% icon param-collection %} *"Pseudobulk - B"*: `Pseudobulk (#B)`
>    - {% icon param-collection %} *"ESet Reference scRNA-seq"*: `ESet Object`
>    - *"Cell Types Label from scRNA dataset"*: `cellType`
>    - *"Samples Identifier from scRNA dataset"*: `sampleID`
>    - *"Cell types to use from scRNA dataset"*:`alpha,beta,ductal,acinar,gamma,delta,unclassified endocrine,co-expression,PSC,endothelial,epsilon,mast,unclassified,MHC class II`
>    - {% icon param-collection %} *"Actual - B"*: `Actual cell proportions (#B)`
>    - {% icon param-collection %} *"Actual - A"*: `Actual cell proportions (#A)`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=cb978151b1686b71&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

# Visualise results

Now that we have our deconvolution results, the next step is to analyse the predictions and determine how accurate our tools are given our reference data. Since our pseudo-subjects **A** and **B** come from the same data, there isn't much point inspecting them both. So for the rest of the tutorial we will just focus our analysis on subject **A**.

In order to determine if our tools have produced accurate results, we will create various plots and compute different metrics to visualise and quantify the outputs of our tools.

## Pre-process the output results

Before visualising or inspecting the outputs of the deconvolution tools, we first need to perform some pre-processing. Up until now we have been working with collections in order to perform our evaluations multiple times in parallel. However, for analysing our data, collections will be a bit messy and are no longer needed. The following workflow will combine all the collections of the MuSiC and NNLS outputs into two tables:

1. A results table presenting the predicted and actual proportion values of each cell-type of each subsample
2. An error table showing the difference between the actual and predicted values. Which will be needed for a later plot.

> <hands-on-title>Run visualisation pre-processing workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-3-process.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow preprocess visualisations** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `A - Music Results`
>
> 3. Run **Workflow preprocess visualisations** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `B - NNLS Results`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=76d3408d0d22ad05&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

The following table shows a snippet of the `Results Table` for the MuSiC tool. A header has been added for better reading but has been omitted in the workflow output as it will interfere with the visualisation tools.

| Cell Type      | Actual Proportion   | Predicted Proportion   |
|----------------|---------------------|------------------------|
| acinar         | 0.090000            | 0.0814442584577275     |
| alpha          | 0.415000            | 0.427718807911522      |
| beta           | 0.170000            | 0.256954867012044      |
| co-expression  | 0.050000            | 0                      |
| delta          | 0.070000            | 0.0929840465107452     |
| ...            | ...                 | ...                    |

Already at first glance we can see some interesting results! Firstly we can see that the tool is able to make predictions close to the actual values such as with `acinar, alpha, delta`. We also see the tool failing to make any type of prediction for `co-expression` cells with a predicted proportion value of 0. This however isn't a compete surprise since `co-expression` cells are of small proportion in the bulk and reference data.

But this is only a small sample of the results. Lets create some visualisations to see the whole picture!

## Plot scatter plots of the results

The first type of visualisation we will do is a scatter plot. This plot will compare the actual and predicted proportion values for each cell across each subsample. We will also colour each point on the plot to indicate which cell type it belongs to. Let's do that now for both the MuSiC and NNLS results.

{% snippet faqs/galaxy/tools_rerun.md %}

> <hands-on-title>Plot the actual and inferred data</hands-on-title>
>
> 1. {% tool [Scatterplot with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `Results Table (Music)`
>    - *"Column to plot on x-axis"*: `2`
>    - *"Column to plot on y-axis"*: `3`
>    - *"Plot title"*: `Correlation between inferred and actual cell-type proportions`
>    - *"Label for x axis"*: `Actual proportions`
>    - *"Label for y axis"*: `Inferred proportions`
>    - In *"Advanced options"*:
>       - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>           - *"column differentiating the different groups"*: `1`
>           - *"Color schemes to differentiate your groups"*: `Paired - predefined color pallete (discrete, max=12 colors)`
>           - *"Reverse color scheme"*: `Default order of color scheme`
>    - In *"Output options"*:
>       - *"width of output"*: `6.0`
>       - *"height of output"*: `4.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Scatter plot - Music`
>
> 3. Add a `#plot` tag to `Scatter plot - Music`
>
> 4. {% tool [Scatterplot with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `Results Table (NNLS)`
>    - *"Column to plot on x-axis"*: `2`
>    - *"Column to plot on y-axis"*: `3`
>    - *"Plot title"*: `Correlation between inferred and actual cell-type proportions`
>    - *"Label for x axis"*: `Actual proportions`
>    - *"Label for y axis"*: `Inferred proportions`
>    - In *"Advanced options"*:
>       - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>           - *"column differentiating the different groups"*: `1`
>           - *"Color schemes to differentiate your groups"*: `Paired - predefined color pallete (discrete, max=12 colors)`
>           - *"Reverse color scheme"*: `Default order of color scheme`
>    - In *"Output options"*:
>       - *"width of output"*: `6.0`
>       - *"height of output"*: `4.0`
>
> 5. **Rename** {% icon galaxy-pencil %} output `Scatter plot - NNLS`
>
> 6. Add a `#plot` tag to `Scatter plot - NNLS`
>
{: .hands_on}

The output of this tool should produce two scatter plots that looks like the image below. Each point on the plot represents a cell-type for a specific subsample, so there should be 20 points of each colour (one for each subsample created earlier). Since we are comparing the actual and inferred proportions, the ideal scatter plot would have all of the points be at the `y=x` line. The further the deviations are from this ideal line, the less accurate the tool is. We can also use this plot to determine if the tool is under or over predicting proportion values for each cell-type, or if the tool is struggling to predict certain cell types.

![Scatter plot MuSiC](../../images/bulk-deconvolution-evaluate/scatterplot-music.png "Scatter plot of Music results")

> <question-title>Interpreting the Scatter Plots</question-title>
>
> 1. Which method has the most accurate results?
> 2. Which cell type has the biggest proportion in the dataset?
> 3. Do either of the tools struggle with any cell types?
>
> {% snippet faqs/galaxy/features_scratchbook.md datatype="tabular" %}
>
> > <solution-title></solution-title>
> >
> > ![Scatter plot comparison](../../images/bulk-deconvolution-evaluate/scatterplot-compare.png "Scatter plot comparison between Music and NNLS")
> >
> > 1. Comparing scatter plots, the MuSiC tool has the most accurate results since the points fall closer onto the x=y line
> > 2. Both scatter plots show `alpha` cells having the highest proportion by a large margin
> > 3. The MuSiC tool seems to handle all cell types well. However, NNLS appears to struggle predicting the proportions of beta cells, with many of the samples being predicted as having a proportion of 0
> >
> {: .solution}
>
{: .question}


## Plot violin plots of the errors

Next we will plot the distribution of errors between the predicted and actual cellular proportions for a select number of cell types. We could plot all cell types in the output, however too many will cause the visualisations to be messy and difficult to interpret.

We can use the cell-type counts we computed at the beginning of the tutorial to determine the best cell types to use. We will use the top 5 most abundant cell types in the single-cell data being: `alpha, beta, gamma, ductal, acinar`. Before plotting we will extract only these cell types from our table of errors.

{% snippet faqs/galaxy/tools_rerun.md %}

> <hands-on-title>Extract Cell Types</hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `Error Table (Music)`
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>       - *"Delimited by"*: `Tab`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>           - *"List of Fields"*: `Select the columns containing: alpha, beta, gamma, ductal, acinar`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Music Errors`
>
> 3. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `Error Table (NNLS)`
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>       - *"Delimited by"*: `Tab`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>           - *"List of Fields"*: `Select the columns containing: alpha, beta, gamma, ductal, acinar`
>
> 4. **Rename** {% icon galaxy-pencil %} output `NNLS Errors`
>
{: .hands_on}

Now we have our table of errors consisting of only the top 5 cell-types, we can plot the violin plots.

{% snippet faqs/galaxy/tools_rerun.md %}

> <hands-on-title>Plot violin plots</hands-on-title>
>
> 1. {% tool [Violin plot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_violin/ggplot2_violin/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `Music Errors`
>    - *"Plot title"*: `Error Distribution`
>    - *"Label for x axis"*: `Cell Type`
>    - *"Label for y axis"*: `Difference Error`
>    - In *"Advanced Options"*:
>       - *"Violin border options"*: `Purple`
>    - In *"Output Options"*:
>       - *"width of output"*: `3.0`
>       - *"height of output"*: `2.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Violin Plot - Music`
>
> 3. Add a `#plot` tag to `Violin Plot - Music`
>
> 4. {% tool [Violin plot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_violin/ggplot2_violin/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `NNLS Errors`
>    - *"Plot title"*: `Error Distribution`
>    - *"Label for x axis"*: `Cell Type`
>    - *"Label for y axis"*: `Difference Error`
>    - In *"Advanced Options"*:
>       - *"Violin border options"*: `Purple`
>    - In *"Output Options"*:
>       - *"width of output"*: `3.0`
>       - *"height of output"*: `2.0`
>
> 5. **Rename** {% icon galaxy-pencil %} output `Violin Plot - NNLS`
>
> 6. Add a `#plot` tag to `Violin Plot - NNLS`
>
{: .hands_on}

The output of this tool will be two violin plots that will look similar to the below image. Here we can see the distribution of errors for each cell type. Since we are using normal errors and not absolute or squared errors, we are also able to see whether the tool has under or over estimated the cell type. An ideal plot would have all the violin plots being short in height and close to 0 indicating that the estimated and actual values are close together (resulting in an error close to 0).

![Violin plot MuSiC](../../images/bulk-deconvolution-evaluate/violin-music.png "Violin plot of Music results")

> <question-title>Interpreting the Violin Plots</question-title>
>
> 1. Which method has the least errors?
> 2. Which method is the most balanced when to over and under estimating proportions?
> 3. Which is the most overestimated cell type in NNLS?
>
> {% snippet faqs/galaxy/features_scratchbook.md datatype="tabular" %}
>
> > <solution-title></solution-title>
> >
> > ![Scatter plot comparison](../../images/bulk-deconvolution-evaluate/violin-compare.png "Scatter plot comparison between Music and NNLS")
> >
> > 1. Comparing the two violin plots, MuSiC has the better error results, with more samples closer to zero. Inspecting the y-axis of the plots also show that the MuSiC errors span a smaller range compared to NNLS.
> > 2. MuSiC can be seen as having the most balanced results with the bulk of the estimates being around 0. Whereas the NNLS results show large amounts of both overestimation and underestimation of various cell types.
> > 3. From the NNLS violin plot it can be seen that ductal cells are greatly overestimated.
> >
> {: .solution}
>
{: .question}


# Compute accuracy metrics

Visualisations are a great tool for getting an intuitive overview of the data. However, some of the interpretations from visualisations can be subjective. Having quantitative results alongside visualisations can offer concrete and precise values about the data that can more easily be compared. We will use two different quantitative metrics in this tutorial; Pearson correlation and RMSE.

## Pearson Correlation

The Pearson correlation coefficient is a statistical value that represents the direction and correlation between two variables, the value of this metric ranges between -1 and 1, where:

- -1 = negative correlation
- 0 = no correlation
- 1 = positive correlation

The equation for calculating the Pearson correlation can be seen below, the workflow to compute this metric breaks down this formula into smaller steps.

![Pearson Correlation Equation](../../images/bulk-deconvolution-evaluate/pearson.png "Pearson Correlation Equation")

Where
- `x` = actual proportion values
- `x̄` = mean of actual proportion values
- `y` = predicted proportion values
- `ȳ` = mean of predicted proportion values

## Root Mean Squared Error (RMSE)

Root Mean Squared Error or RMSE is a common metric for measuring a tools prediction error. This metric calculates the average error between the predicted and actual values for each prediction then takes the mean and square root of the error to produce a final value. Lower RMSE values (close to 0) indicate accurate predictions similar to the actual value, as the value increases the accuracy score worsens.

The equation for calculating this metric is seen below, the implementation of this calculation is in the workflow alongside the Pearson correlation.

![Root Mean Squared Error Equation](../../images/bulk-deconvolution-evaluate/rmse.png "Root Mean Squared Error Equation")

Where
- `n` = number of samples
- `y` = actual proportion
- `ŷ` = predicted proportion


## Compute metrics

With a basic understanding of some useful metrics, we will now compute these to get quantitative values alongside our visualisation results. The following workflow needs to be run for both the MuSiC and NNLS results table.

> <hands-on-title>Run metrics workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-4-metrics.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow compute metrics** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `Results Table (Music)`
>
> 3. Run **Workflow compute metrics** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `Results Table (NNLS)`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=7caa58e36df6ad03&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

After running the workflow on both the MuSiC and NNLS results we should have the Pearson and RMSE metrics for both tools in various outputs. Below combines these metrics into a single summary table.

| Tool  | Pearson Correlation | RMSE  |
|-------|---------------------|-------|
| MuSiC | 0.982               | 0.022 |
| NNLS  | 0.778               | 0.678 |

From the table we can now see concrete values representing the error and correlation between the predictions and actual proportion values. We can see from the table that the MuSiC tool has a much better accuracy with a higher correlation score and lower error compared to NNLS.

The conclusions to draw from this analysis, is that our reference data is effective for use in deconvolution analysis since both tools were able to have high accuracy and low error scores. We also determined that (for at least this data) the MuSiC tool was the more effective/accurate tool and thus would likely be the more trustworthy when performing deconvolution with this single-cell reference data.

# Conclusion

{% icon congratulations %} Congratulations! You made it to the end of the tutorial!

In this tutorial we took some single-cell data with known cell-type proportions, subsampled the data, and converted them to pseudo-bulk data. We then used this pseudo-bulk data to perform deconvolution using the two tools available in Galaxy: MuSiC and NNLS. Using the known cell-type proportions we were able to analyse the predicted proportions to the ground truth in order to determine if the reference data can be used and which tool is the most effective. We used various visualisation and statistical techniques to analyse and quantify the tools accuracy, reliability, and error.

tools_json:
["Count", "Count1", "Transpose", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy1", "Duplicate file to collection", "__DUPLICATE_FILE_TO_COLLECTION__", "Construct Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy3", "Scatterplot with ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy2", "Violin plot w ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_violin/ggplot2_violin/3.4.0+galaxy1"]


topic: single-cell
tutorial: scrna-case_FilterPlotandExploreRStudio
title: Filter, plot, and explore single cell RNA-seq data with Seurat (R)
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 7cc76ac215e7e4818a2fb97f3bd30178
last_modified: 2026-05-27T09:01:49.744194
zenodo_link: https://zenodo.org/record/7053673
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_FilterPlotandExploreRStudio/tutorial.html

content:
{% snippet topics/single-cell/faqs/notebook_warning.md %}

# Setting your environment
First thing's first, we need to load the packages we will be using. In order to use any functions within a package, we must first call the library of that package. In your console (likely in the lower left corner of your RStudio window), run the following lines of code to do so:

```r
library(Matrix)
library(Seurat)
library(dplyr)
```

The packages are called--now let's get our data files moved from our Galaxy history and into the RStudio environment so that we can create a Seurat object.

# Upload, view and modify the files
Now that we've made it into RStudio and called the packages we'll use, let's begin moving datasets from Galaxy and into RStudio. Galaxy helps us with this by providing the gx_get() function, which will tell us the file paths for the datasets currently occupying our Galaxy histories.

So, for example, our matrix was the first piece of data to be imported. As such, we will ask for the file path of the first piece of data in our history: we will tell gx_get() the number of the dataset in our history that we are interested in, and the function will output the path we need to bring that dataset into R.

```r
gx_get(1)  #get the file path for matrix.mtx: #1 in Galaxy history
```

Now we have the file path! We can use the Matrix package, specifically the readMM() function, to read the counts matrix into our environment, using the file path above to let it know where to find the matrix.

```r
matrix.mtx<-readMM("/import/1")
```
Now we will do the same thing with the feature, barcode, and experimental design files.

> <warning-title>Don't skip ahead!</warning-title>
>  Don't try to skip ahead and simply fill in the position of the dataset within the readMM() function, reading in the files very often will not work without having used the gx_get() function--if it does, well, lucky you.
>
{: .warning}

We'll run the same command, this time replacing the data number in order to ask for the file path to data in an alternative slot in our Galaxy history. Then, we will use the read.delim() function in read in the list of genes, cells, and cell annotations provided to us  by the researchers:

```r
gx_get(2) #ask for the file path of the genes.tsv file
genes.tsv<-read.delim("/import/2", header = FALSE) #read in the genes file

gx_get(3) # ask for the file path of the barcodes.tsv file
barcodes.tsv<-read.delim("/import/3", header = FALSE) #read in the barcodes file

gx_get(4) #ask for the file path of the exp_design.tsv file
exp_design.tsv<-read.delim("/import/4") # read in the experimental design file
```
The format of the experimental design dataset contains cell barcodes as the first column of data with simply numbered (1-n) row names. In order for Seurat to properly use this dataset, we will need to make the cell barcodes the row names. This can be accomplished by running the following line of code:

```r
rownames(exp_design.tsv)<-exp_design.tsv$Assay
```

Now, in RStudio, we should have all of the data necessary to create a Seurat Object: the matrix, a file with feature (gene) names, a file with cell barcodes, and an optional, but highly useful, experimental design file containing sample (cell-level) metadata.

# Generating a Seurat object
Next we will add row and column names to our matrix. In the end, this will leave us with a matrix whose rows are genes, columns are cell barcodes, and each value in the matrix representing the expression value of a given gene in a given cell.

><tip-title>About Seurat Objects</tip-title>
>The order of the dimensions, such that genes are the first and cells are the second is a set characteristic of Seurat objects.
>
>You will recieve an error if you try to label the first dimension with cell barcodes or the second with genes. This is because the Dimname slots are like empty fill in the blanks: if the number of labels input doesn't match the number of blanks in that dimension (the number of labels don't match the number of cells/genes), Seurat will not accept the labels.
{: .tip}

```r
dimnames(matrix.mtx) <- list(genes.tsv$V2, barcodes.tsv$V1)
```

In a more typical Seurat pipeline, or on a local version of RStudio, this step would be replaced with a Read10x() step. Read10x() is Seurat's function to create a matrix and add in feature and barcode names simultaneously. However, due to the nature of how Galaxy histories and interactive environments communicate with one another, we'll use this manual method.

><tip-title>How to Use Read10X</tip-title>
>The only necessary parameter of Seurat's Read10X() fucntion is the file path to the directory (folder) containing the matrix.mtx, barcodes.tsv, and genes.tsv (sometimes also called features.tsv) files.
>
>So if the EBI SCXA Retrieval tool were to have output a data directory in the sixth history position, the following code would be able to import a labelled counts matrix called "labelled matrix" into our RStudio environment:
>
>```r
>gx_get(6) #get the file path
>labelled_matrix<-Read10X(dir = "/import/6")
>```
>
{: .tip}

Now we will create a Seurat object using our newly labelled counts matrix!

```r
srt<-CreateSeuratObject(counts = matrix.mtx)
```

You've created a Seurat object, congratulations!

# Adding Cell Level Metadata

Now that we have an object, we can add in the cell level metadata from our experimental design dataframe (table). This will be useful to us shortly as we begin to visualize the data!

```r
srt$Sex<-exp_design.tsv$Sample.Characteristic.sex.
srt$Organism<-exp_design.tsv$Sample.Characteristic.organism.
srt$Strain<-exp_design.tsv$Sample.Characteristic.strain.
srt$Developmental.Stage<-exp_design.tsv$Sample.Characteristic.developmental.stage.
srt$Age<-exp_design.tsv$Sample.Characteristic.age.
srt$Individual<-exp_design.tsv$Sample.Characteristic.individual.
srt$Disease<-exp_design.tsv$Sample.Characteristic.disease.
srt$Genotype<-exp_design.tsv$Sample.Characteristic.genotype.
srt$Organism.Part<-exp_design.tsv$Sample.Characteristic.organism.part.
srt$Cell.Type<-exp_design.tsv$Sample.Characteristic.cell.type.
srt$Factor.Value.Genotype<-exp_design.tsv$Factor.Value.genotype.
```
><tip-title>Syntax Lesson</tip-title>
>The code preceding the left pointing arrow will indicate where to put your metadata (the name of your new metadata column: object$new_metadata_columnname), and the code following the arrow will denote where to find that metadata information (metadata_table$columnname)
>
{: .tip}

Now that we have our almost fully annotated object, we will add one more metadata column: percent mitochondrial (perc.mt). This metadata column will denote what percentage of a cell's feature (gene) expression is mitochondrial--which will be useful to us shortly as we begin to filter cells.

```r
srt <- PercentageFeatureSet(srt, pattern = "^mt-", col.name = "perc.mt")
```

For the sake of this data set, and many others, the mitochondrial genes will all be marked with an "mt" as the prefix, so that is how we have asked Seurat's PercentageFeatureSet() function to search for mitochondrial genes in the line of code above. With that being said, when you are analyzing your own data, it is highly recommended that you identify how your data set has labelled mitochondrial genes to ensure that you are calculating the correct percentages.

> <warning-title>Careful</warning-title>
>  The "mt" prefix may not always include all mitochondrial genes depending on how your dataset has been labelled.
>
>Sometimes the labels may be done via a capital prefix ("^MT") and in some cases, mitochondrial genes must be identified via feature-level metadata like a GTF file.
{: .warning}


# QC Plots
Now that we have a complete Seurat object, we can begin the filtering process.

There will be a number of ‘cells’ that are low-quality or just empty droplets. There will also be genes which are mere sequencing artifacts or appear with such low frequency that statistical tools will fail to accurately analyse them.

This background noise of cells and genes not only makes it harder to distinguish real biological information from artifacts, but also makes the data computationally demanding to analyze.

We want to filter our cells to only include those containing informative biological information. Before doing so, we need to know what the data distribution looks like. There are a number of subjective decisions to make within scRNA-seq analysis, for instance we need to make informed decisions about where to set our thresholds (more on that soon!).

We’re going to plot our data a few different ways. Different bioinformaticians might prefer to see the data in different ways, and here we are only generating a few of the plots you can use. Ultimately you need to go with what makes the most sense to you.

So let's generate some QC plots. First off, let's check our dataset for batch effect:

```r
VlnPlot(srt, group.by = "Individual",
  features = "nCount_RNA",
  log = TRUE)
```
![Violin Plot split by Individual/Batch](../../images/scrna-SeuratRStudio/plot1.png "Violin Plot of counts split by Individual.")

This plot shows us the number of cells split by the individual (mouse) from which the cells came from. Now, depending on your experimental design, batch may be represented by something other than individual--like timepoint or even the wet lab researcher who isolated the cells.

Ideally, we'd like to see a relatively even distribution of counts for each individual (or batch) but if there isn’t, fear not, we can regress this variable out in a later step.

><tip-title>Syntax Lesson</tip-title>
>In order to accurately assess potential batch effects, use the "group.by" parameter to indicate the variable which differed across experiments.
>
{: .tip}

Now let's get an idea of how different variables, like the sex or genotype of the mice, might be represented across our dataset.

1. Sex?
    ```r
    VlnPlot(srt, group.by = "Sex",features = "nCount_RNA",log = TRUE)
    ```
    ![Violin Plot split by Sex](../../images/scrna-SeuratRStudio/plot2.png "Violin Plot of counts split by Sex.")

2. Genotype?
    ```r
    VlnPlot(srt, group.by = "Genotype", features = "nCount_RNA", log = TRUE)
    ```
    ![Violin Plot split by Genotype](../../images/scrna-SeuratRStudio/plot3.png "Violin Plot of counts split by Genotype--Mutant versus Control.")

# Finding Our Filtering Parameters
Now that we have a better understanding of what our data looks like, we can begin identifying those spurious reads and low quality cells to be removed. First, we'll plot the percent mito (perc.mt) against the cell count (nCount_RNA) to get an idea of what threshold we should set for nCount:

```r
plot(x = srt$nCount_RNA, y = srt$perc.mt, main = "UMI Counts x Percent Mito", xlab = "UMI_count", ylab = "percent mito")
```
![UMI x mito](../../images/scrna-SeuratRStudio/plot4.png "UMI counts x Percent mito.")
We are looking for cell counts with high mitochondrial percentages in their feature expression.

><comment-title>What Do High Mitochondrial Reads Mean?</comment-title>
>High mito expression will typically indicate stressed out cells (often due to the extraction, sorting, or sample prep protocols).
>
{: .comment}

These stressed out cells likely won't tell us much biologically, rather, they will contribute noise that we want to filter out of our dataset. With that being said, there is a level of metabolic activity that is expected and will be specific to your samples/tissue/organism--so it is worth looking into what that might look like when it comes time to analyze your own datasets. Asking the question "what is an expected proportion of mitochondrial expression in my sample?" can and likely will become a frustrating endeavor if you are analyzing a brand new kind of dataset. scRNA-seq analyses are inherently iterative for this exact reason, so try a threshold, label your folders well, and be prepared to return back to this question if things look strange down the line.

As a general eye-balling rule, I will look for the nCount value at which the perc.mito begind to rise. We can also zoom in on the x-axis to get a better idea of what threshold to set by adjusting the xlim parameter:

```r
plot(x = srt$nCount_RNA, y = srt$perc.mt, main = "UMI Counts x Percent Mito", xlab = "UMI_count", ylab = "percent mito", xlim = c(0,1750))
```
![UMI x mito zoomed in on X](../../images/scrna-SeuratRStudio/plot5.png "UMI counts x Percent mito-Zoomed in on X.")

><comment-title>Interpretations</comment-title>
>It looks like just before nCount_RNA = 1750, the perc.mito peaks above 2 percent--a conservative threshold.
>
{: .comment}

Now we can take a closer look at the y-axis to decide on a mito threshold to set. Once more, we want to get rid of as few cells as possible while still removing those with unexpectedly high mito percentages.

```r
plot(x = srt$nCount_RNA, y = srt$perc.mt, main = "UMI Counts x Percent Mito", xlab = "UMI_count", ylab = "percent mito", ylim = c(0,3))
```
![UMI x mito zoomed in Y](../../images/scrna-SeuratRStudio/plot6.png "UMI counts x Percent mito-Zoomed in on Y.")

><comment-title>Interpretations</comment-title>
>We can see a clear trend wherein cells that have around 3 percent mito counts or higher also have far fewer total counts. These cells are likely of low quality, will muddy our data, and were probably stressed or ruptured prior to their encapsulation within a droplet.
>
{: .comment}

Take a look at what proportion of cells those thresholds will include and disclude from our dataset:

```r
prop.table(table(srt@meta.data$nCount_RNA > 1750))
prop.table(table(srt@meta.data$perc.mt > 3))
```

If we are happy with those thresholds for cells and percent mito, we can look at the the gene count threshold next.

><comment-title>Otherwise</comment-title>
>If not, repeat the preceding steps to hone in on a threshold more suited to your needs.
>
{: .comment}

To set a threshold for gene count, let's plot the gene counts (nFeature_RNA) against the percent mito (perc.mt):

```r
plot(x = srt$nFeature_RNA, y = srt$perc.mt, main = "Gene Counts x Percent Mito", xlab = "gene_count", ylab = "percent mito")
```
![Gene x mito](../../images/scrna-SeuratRStudio/plot7.png "Gene counts x Percent mito.")

Once again, let's zoom in on the x-axis, but this time to get an idea of which nFeature_RNA threshold to set:

```r
plot(x = srt$nFeature_RNA, y = srt$perc.mt, main = "Gene Counts x Percent Mito", xlab = "gene_count", ylab = "percent mito", xlim = c(0,1275))
```
![Gene x mito--zoomed in](../../images/scrna-SeuratRStudio/plot8.png "Gene counts x Percent mito zoomed in.")

><comment-title>Interpretations</comment-title>
>You can see how cells with nFeature_RNA up to around, perhaps 575 genes, also have high perc.mt. The same can be said for cells with nFeature_RNA above 1275.
>
>We could also use the violin plots to come up with these thresholds, and thus also take batch into account. It’s good to look at the violins as well, because you don’t want to accidentally cut out an entire sample (i.e. N703 and N707 which both have cell counts on the lower side).
>
{: .comment}

Now let's take a look at what those nFeature_RNA thresholds will include and disclude from our data.

```r
prop.table(table(srt@meta.data$nFeature_RNA > 1275 | srt@meta.data$nFeature_RNA < 575))
```

# Applying our Thresholds
Once we are happy with our filtering thresholds, it’s time to apply them to our data!

><tip-title>Create a new object</tip-title>
> You will notice in the next line of code, we have indicated a new object name for this filtered (subset) data. This is good practice  so that you don't have to start all over in case you decide to change your filtering parameters (which you likely will) or if something goes awry.
>
{: .tip}

```r
subset_srt<-subset(srt, nCount_RNA > 1750 & nFeature_RNA > 1275 & perc.mt < 3 | nFeature_RNA < 600)
```

In this step we are also creating a new object (notice the new object name preceding the subset() function you just ran) so that we may compare back and forth between our unfiltered and filtered data set if we please.

Next, we want to filter out genes that no longer show any expression in the remaining high quality cells present in our dataset. In order to do so we will extract the filtered matrix from our filtered object.

```r
subset_matrix<-GetAssayData(subset_srt)
```

Since you’ve removed a whole heap of cells, and the captured genes are sporadic (i.e. a small percentage of the overall transcriptome per cell) this means there are a number of genes in your matrix that are not expressed in any of the cells left in the filtered matrix. Genes that do not appear in any cell, or even in only 1 or 2 cells, may break some analytical tools and will often not be biologically informative. So let’s remove them!

We can use the filtered matrix we extracted to create a new Seurat object, this time including the argument: min.cells = 3. This will remove any genes from our matrix that have less than 3 cells expressing them.

><tip-title>More Thresholds</tip-title>
> Note that 3 is not necessarily the best number, rather it is a fairly conservative threshold. You could go as high as 10 or more.
>
{: .tip}

```r
filtered_srt <- CreateSeuratObject(counts = subset_matrix, meta.data = subset_srt@meta.data, min.cells = 3)
```

Now that we have filtered out both noisy "cells" and genes from our dataset, let's clean up our environment. Remove objects that we no longer need to ensure that we stay organized and RStudio has enough memory capacity to perform downstream analyses. This likely will not be an issue while doing this tutorial, but in practice it will help things run smoothly.

```r
rm(subset_matrix, subset_srt)
```

# Processing
Even after filtering, we still have quite big data. This presents two concerns for downstream analyses:
 1. We already saw in our filtering plots that there are differences in how many transcripts and genes have been counted per cell. This technical variable could, and likely will, distract us from identifying true biological differences.

 2. We like to plot things on 2-dimensional (X/Y) plots. For instance, Gapdh could be on one axis, and Actin could be on another, and each cell could be plotted onto that 2D axis based on how many of each transcript they possess.

Although this plotting method would be _fine_, adding in a 3rd dimension (or, indeed, in our case, a dimension for each of the remaining thousands of genes), is a bit trickier.

So, our next steps will be to transform the large data object into something that is easy to analyse and easy to visualize.

We will run SCTransform, a combinatorial function by Seurat that normalizes the data, identifies variable features, and then scales the data. In their initial workflow, and in the Scanpy version of this tutorial, these steps are run individually. However, with the second version of SCTransform comes time efficiency and optimization for downstream analyses.

```r
filtered_srt<- SCTransform(filtered_srt, vars.to.regress = c("perc.mt", "nFeature_RNA", "nCount_RNA"), verbose = TRUE, return.only.var.genes = FALSE, seed.use = 1448145)
```
><comment-title>What is Normalization?</comment-title>
> Normalisation helps reduce the differences between gene and UMI counts by fitting total counts across cells in our data to be comparable to one another. SCTransform regularizes the gene expression profiles via a negative binomial regression while also controlling for overfitting of the data. This step can also be done using Seurat's NormalizeData() function, but would need to be followed by FindVariableFeatures() and ScaleData().
>
{: .comment}


We also have loads of genes, but not all of them vary in expression from cell to cell. For instance, housekeeping genes are defined as not changing much from cell to cell, so we could disclude these from our data to simplify our analyses.

The find variable features step within SCTransform (or Seurat's FindVariableFeatures() function) will flag genes that _do_ vary across cells, expediting future analyses and ensuring that we, and Seurat, don't waste time looking for meaningful differences where they don't exist.

Finally, SCTransform (or Seurat's ScaleData() function) will scale the data so that all genes have the same variance and a zero mean.

This is an important step to set up our data for further dimensionality reduction. It also helps negate sequencing depth differences between samples, since the gene levels across the cells become comparable.

><comment-title>Don't Worry!</comment-title>
> Note, that the differences from scaling etc. are not the values you have at the end - i.e. if your cell has average GAPDH levels, it will not appear as a ‘0’ when you calculate gene differences between clusters.
>
{: .comment}

Although we've made our expression values comparable to one another and our overall dataset less computationally demanding, we still have _way_ too many dimensions (n cells x n genes!).

Transcript changes are not usually singular--which is to say, genes function and exist in pathways and groups. It would be easier to analyse our data if we could group these differences. To address this we will run principal component analysis (PCA).

><comment-title>What is PCA?</comment-title>
>Principal components (PCs) are calculated from high dimensional data to find the most representative spread in the dataset. So in our highly variable gene dimensions, there will be one line (axis) that yields the most spread and variation across the cells. That will be our first principal component.
{: .comment}

We can calculate the first handful of principal components in our data to drastically reduce the number of dimensions:

><tip-title>Running Computationally Demanding Steps on Variable Features </tip-title>
>You'll notice that the RunPCA() function is run using the variable features from the previous step. This signficantly decreases the number of genes, and their expression changes, that must be grouped into principal components by this step.
{: .tip}

```r
filtered_srt <- RunPCA(filtered_srt, features = VariableFeatures(object = filtered_srt))
```

To visualize how our principal components (PCs) represent our data, let's create an elbow plot:

```r
ElbowPlot(filtered_srt, ndims = 50)
```
![PC Elbow Plot](../../images/scrna-SeuratRStudio/plot9.png "Elbow Plot: Varianvce Explained x PC.")

><comment-title>Interpretations</comment-title>
>We can see that there is really not much variation explained past the 9th PC. So we might save ourselves a great deal of time and muddied data by focusing on the top 10 PCs.
{: .comment}

You can also think about it like choosing a threshold of variance explained. Conservatively, 2.5 standard deviations are explained by about 10 of the PCs.

We’re still looking at around 10 dimensions at this point--likely not the easiest to visualize. To make our lives even easier, we must identify how similar a cell is to another cell, across every cell across each of these dimensions.

For this, we will use the k-nearest neighbor (kNN) graph, to identify which cells are close together and which are not.

The kNN graph plots connections between cells if their distance (when plotted in this 10 dimensional space) is amongst the k-th smallest distances from that cell to other cells. This will be crucial for identifying clusters, and is necessary for plotting a UMAP--which is what will ultimately allow us to visualize our data in 2 dimensions.

><comment-title>From UMAP developers:</comment-title>
>“Larger neighbor values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15 being a sensible default”.
{: .comment}

Let's now use the 10 PC threshold we chose from the Elbowplot and apply it to find neighbors:

```r
filtered_srt <- FindNeighbors(filtered_srt, dims = 1:10)
```

Now we can use the neighborhood graph to identify clusters of cells whose transcriptional profiles appear most similar to one another.

```r
filtered_srt <- FindClusters(filtered_srt, resolution = 0.5)
```

Unfortunately, identifying clusters is not as majestic as many biologists often think - the math doesn’t necessarily identify true clusters of cells. Every algorithm for identifying cell clusters falls short of a biologist knowing their data, knowing what cells should be where, and consequently proving it in the lab.

So, we’re going to make the best of it as a starting point and see what happens! We will define clusters from the kNN graph, based on how many connections cells have with one another. Roughly, this will depend on a resolution parameter for how granular you want to be.

><tip-title>On Clustering Resolution</tip-title>
>The resolution parameter available in the FindClusters() function allows for you, the bioinformatician, to dictate the granularity of the clusters.
>
>For example, a higher clustering resolution dictates increased granularity, and more stringent clusters. That is--cells must more closely resemble one another in order to be grouped into the same cluster than at a lower clustering resolution.
>
>In general, I find it easiest to think of a higher resolution producing more clusters and conversely, a lower resolution will produce less clusters. This parameter is a useful one that you will use often to help decipher how many true populations of cells are present in your data!
{: .tip}

Now that we have made note within our object of which cells cluster together, we can start to really visualize the data! Two major visualizations for this data type are tSNE and UMAP. We can calculate the coordinates for both prior to visualization. For tSNE, the parameter perplexity can be changed to best represent the data, while for UMAP the main change would be to change the kNN graph above itself, via the FindNeighbors() function.

><tip-title>On UMAP</tip-title>
>UMAP is the most recently developed, and most widely used dimensionality reduction for visualization of principal component data. It has been optimized since tSNE to better preserve global structure and is less computationally demanding.
>
{: .tip}

```r
filtered_srt <- RunUMAP(filtered_srt, dims = 1:10, seed.use = 1323)
```

# Let's Take a Look
Now that we have run dimensionality reduction on our dataset, it is ready for visualization. Let's take a look at what our cells look like in a UMAP projection:

```r
DimPlot(filtered_srt, reduction = "umap", label = TRUE, label.box = TRUE)+ NoLegend()
```
![DimPlot colored by 0.5 resolution cluster](../../images/scrna-SeuratRStudio/plot10.png "DimPlot colored by 0.5 resolution cluster.")
Good work! It looks like with a clustering resolution of 0.5, we are able to identify 7 clusters of cells in our data.

We can also look for expression of particular genes and see how those map to our UMAP projection. This is often useful in getting an initial understanding of which clusters might be representative of which cell types.

```r
FeaturePlot(filtered_srt, features = "Gapdh", order = TRUE)
```
![FeaturePlot: Gapdh](../../images/scrna-SeuratRStudio/plot11.png "FeaturePlot: Gapdh")

We plotted a housekeeping gene, Gapdh, so the broad expression we observe is expected.

><tip-title>Weird scale?</tip-title>
> If the scale of your data looks weird, it may be due to the DefaultAssay of your object. When we ran SCTransform, the function creates an entirely new assay within our Seurat object that includes scaled and normalized count values. This SCT Assay is what we want to visualize our expression values off of. So, if your scale is super broad, or goes negative in the case of integrated datasets, try running the following command before attempting to plot again:
>```r
>DefaultAssay(filtered_srt)<-"SCT"
>```
{: .tip}

In practice, it is helpful to plot known markers of cell types you expect to be in your dataset. This will give you a first look at how well your cells are clustered.

For example, we can plot early T-cell marker Il2ra and get an idea of which cells and/or clusters might resemble the early T-cells:

```r
FeaturePlot(filtered_srt, features = "Il2ra", order = TRUE)
```
![FeaturePlot: Aif1](../../images/scrna-SeuratRStudio/plot12.png "FeaturePlot: Il2ra")

It is a good idea, when analyzing your own data, to plot some markers of cell types you expect to be present. Later on we can also use these FeaturePlots to visualize manual annotation of clusters.

# Differential Expression Testing: Finding Markers
Because each cluster of cells was grouped based on similar transcriptomic profiles, each cluster will inherently differ from one another based on a set of "marker" genes.

Following an initial look at the DimPlots and FeaturePlots, we can take an even closer look at which genes are driving the clustering.

In order to do so we can run cluster level differential expression. First, we will need to set our object's active identity to be the clusters. This will ensure that when Seurat's differential expression function is run, the groupings of cells across which it will compare are the clusters.

><tip-title>What are Identities?</tip-title>
> Identities are, at their core, categorical metadata values. They are columns of cell-level metadata that somehow group the cells together. Examples of identities could be cluster number, cell type if/once known, genotype, etc.
>
{: .tip}

```r
Idents(filtered_srt)<- filtered_srt$seurat_clusters
```

><tip-title>Syntax Lesson</tip-title>
> There are often many different ways to get the same job done in R, but especially when manipulating Seurat objects. We could alternatively set the active identity of our object with the following line of code too:
>
>```r
>filtered_srt<-SetIdent(object = filtered_srt, value = "seurat_clusters")
>```
{: .tip}

Then, we'll run Seurat's FindAllMarkers function, which will compare each identity (cluster in this case) against every other identity within its class (all other clusters). This function of marker finding is particularly useful in identifying up, or down, regulated genes that drive differences in identity/cluster.

```r
cluster_markers<-FindAllMarkers(object = filtered_srt)
View(cluster_markers)
```

We'll use these marker lists later on to label our cell types.

We can also see which genes are differentially expressed across other variables in our metadata. For example, you can see which genes are up or down regulated across the different genotypes present in our dataset. To do so, let's first get a list of all the identity classes in our data. This information is kept in the metadata column, and any categorical variable will do. Here, let's pick genotype.

```r
metadata<-as.data.frame(filtered_srt@meta.data)
filtered_srt<-SetIdent(object = filtered_srt, value = "Genotype")
```

The "metadata" object now in your environment is a dataframe with column names representing the different identities you may choose to group your cells by when running differential expression or even when plotting Feature/DimPlots. The second line of code above will set the object's identity class to be the genotype from which the cell came from.

Now, let's see what transcripts differentiate our wildtype from our mutant cells. First, we can identify how many different genotypes are in our data:

```r
unique(filtered_srt$Genotype)
```

This output helpfully shows us what the genotypes are, and how they are labelled in our metadata. The small details, like capitalization, are important for referencing metadata information--our references must perfectly match the labelling in the object, otherwise they will not be recognized by the functions.

Now that we know how our wildtype and mutant cells have been labelled, we can use that information to directly compare the two. This time we will use a pairwise comparison method by using Seurat's FindMarkers() function (not to be confused with FindAllMarkers which has a comprehensive comparison approach):

```r
markers<-FindMarkers(object = filtered_srt, ident.1 = "wild type genotype", ident.2 = "Igf2-p0 heterozygous knockout", test.use = "wilcox")
```

The above function will find all of the differentially expressed genes between ident.1 (wildtype) and ident.2 (mutant) using the Wilcoxon test. The resulting output will show genes with positive fold changes (denoting a higher expression in the first identity--wildtype) and negative fold changes (denoting a higher expression value in the second identity--mutant).

><comment-title>On Finding Markers</comment-title>
> This same test of differential expression can be run using any identity class and any two identities within the same class. As this is a more fine tuned comparison than FindAllMarkers, it can be useful to uncover differences across specific samples.
{: .comment}

# Biological Interpretations
Now the fun bit! We can see where genes are expressed and begin considering and interpreting the biology of it. At this point, it’s really about what information you want to get from your data--the following is only the tip of the iceberg. However, a brief exploration is good, because it may help give you ideas going forward for your own data. Let's start interrogating!

Let's take another look at what our clusters look like:

```r
DimPlot(object = filtered_srt, reduction = "umap", label = TRUE, label.box = TRUE, group.by = "seurat_clusters") + NoLegend()
```
![DimPlot colored by 0.5 resolution cluster](../../images/scrna-SeuratRStudio/plot10.png "DimPlot colored by 0.5 resolution cluster.")

><comment-title>On Cluster Numbering</comment-title>
>Note that Seurat's cluster numbering is based on size alone, so clusters 0 and 1 are not necessarily related, they are just the clusters containing the most cells.
{: .comment}

It would be nice to know what these cells are. This analysis (googling all of the marker genes, both checking where the ones you know are and then going through marker tables we generated) is an adventurous task for any individual experiment, so we’re going to speed past that and nab the assessment from the original paper!

| Clusters | Markers                 | Cell Type                           |
|----------|-------------------------|-------------------------------------|
| 3        | Il2ra                   | Double negative (early T-cell)      |
| 1,2,5    | Cd8b1, Cd8a, Cd4        | Double positive (middle T-cell)     |
| 0        | Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell)|
| 4        | Itm2a                   | Mature T-cell                       |

Feel free to plot these markers onto our dataset to see where they fall. This is generally a useful method of discerning cell types and can be useful for initial annotations.

><tip-title>Plotting Markers</tip-title>
>To do so, simply use the same FeaturePlot() function we used above, but replace the feature parameter with your new marker of interest.
>```r
>FeaturePlot(object = filtered_srt, features = c("Il2ra", "Cd8b1", "Cd8a", "Cd4", "Itm2a"), order = T, ncol = 3)
>```
>![FeaturePlots of cell type markers](../../images/scrna-SeuratRStudio/plot21.png "FeaturePlots of our known cell type markers")
{: .tip}

We can then manually label the clusters in whatever way we please. [Dplyr](https://dplyr.tidyverse.org/reference/mutate.html)'s mutate() function allows us to incorporate conditional metadata. That is to say, we can ask the function to label cells based on the cluster in which they have been assigned:

```r
filtered_srt@meta.data<- mutate(filtered_srt@meta.data, celltype = case_when(
  seurat_clusters %in% c(3) ~ "Double negative (early T-cell)",
  seurat_clusters %in% c(1,2,5) ~ " Double positive (middle T-cell)",
  seurat_clusters %in% c(0) ~ "Double positive (late middle T-cell)",
  seurat_clusters %in% c(4) ~ "Mature T-cell"
))
```
Once we have labelled our clusters, we can visualize what our cell types actually look like:

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = "celltype")
```
![DimPlot colored by labelled celltype](../../images/scrna-SeuratRStudio/plot13.png "DimPlot colored by assigned cell type")

Now we can begin to feel a bit more oriented in exploring our data. The clusters are labelled with cell types, and our object has been processed enough such that we may now begin to answer some real biological questions! Now that we know what we’re dealing with, let’s examine the effect of our variable, real science!

## Keep Digging
Are there any differences in genotype? Or, a more biological question, is there an impact of growth restriction on T-cell development in the thymus? We can begin to answer this question visually by using the "split.by" parameter in Seurat's plot functions.

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = "celltype", split.by = "Genotype")
```
![DimPlot colored by labelled celltype split by genotype](../../images/scrna-SeuratRStudio/plot14.png "DimPlot colored by assigned cell typesplit by genotype")

We can see that there seems to be a decrease in cell counts across the cell types in the het mutant... INTERESTING! What next? We might look further at the transcripts present in both of those populations, and perhaps also look at the genotype marker table… So much to investigate! But before we set you off to explore to your heart’s delight, let’s also look at this a bit more technically.

# Technical Assessment
Is our analysis real? Is it correct? Well, we can assess that a little bit.

First thing's first, is there a batch effect?

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = "Individual")
```
![DimPlot colored by labelled celltype split by individual/batch](../../images/scrna-SeuratRStudio/plot15.png "DimPlot colored by assigned cell types split by individual/batch")

While some differences across batch are expected and nothing to be concerned about, the immature T-cells looks to be mainly comprised of Individual 3. There might be a bit of batch effect, so you could consider using batch correction on this dataset. However, if we focus our attention on the other cluster - mature T-cells - where there is batch mixing, we can still assess this biologically even without batch correction.

Additionally, we will also look at the confounding effect of sex:

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = c("Sex", "Individual", "Genotype"))
```
![DimPlot colored by Sex, Individual, and Genotype](../../images/scrna-SeuratRStudio/plot16.png "DimPlot colored by Sex, Individual, and Genotype")


We note that the one female knockout sample seems to be distributed in the same areas as the knockout samples at large, so luckily, this doesn’t seem to be a confounding factor and does not seem to prohibit us from learning more from this data. Ideally, this experiment would be re-run with either more female samples all around or swapping out this female from the male sample.

Are there any clusters or differences being driven by sequencing depth, a technical and random factor?

```r
FeaturePlot(object = filtered_srt, reduction = "umap", features = "nCount_SCT")
```
![FeaturePlot colored by counts](../../images/scrna-SeuratRStudio/plot17.png "FeaturePlot colored by counts")


There doesn't visually appear to be any differences in sequencing depth across the clusters, but let's check out some of those other variables we grouped by:

```r
FeaturePlot(object = filtered_srt, reduction = "umap", features = "nCount_SCT", split.by = "Individual")
```
![FeaturePlot colored by counts](../../images/scrna-SeuratRStudio/plot18.png "FeaturePlot colored by counts split by Individual")

There we go! This might explain the dramatic shift in early to middle T-Cell between wildtype and knockout cells--the leftmost early to middle T-cells simply have a higher sequencing depth represented by Individual 3 (UMIs/cell) than the ones on the right side. Well, that explains some of the sub-cluster that we’re seeing in that area (more specifically, this may account for the discernment between clusters 1, 2, and 5).

Luckily, and importantly, we don’t see the double negative or mature T-cells being similarly affected. So, although, this variable of sequencing depth, or moreso, Individual, might be something to regress out, it doesn’t seem to be impacting our dataset such that we cannot draw meaningful insights.


><tip-title>Overprocessing</tip-title>
>The less you can regress/modify your data, in general, the better--you want to stay as true as you can to the raw data, and only use math to correct your data when you really need to (and not to create insights where there are none!).
{: .tip}


Do you think we processed these samples well enough? We have seen in the previous images that these clusters are not very distinct, so we could consider stronger filtering. Let's take a look at gene expression of a gene we know should not be expressed in tCells as a sanity check:

```r
FeaturePlot(object = filtered_srt, reduction = "umap", features = "Hba-a1")
```
![FeaturePlot of Hemoglobin](../../images/scrna-SeuratRStudio/plot19.png "FeaturePlot of Hemoglobin")


Hemoglobin--a red blood cell marker that should NOT be found in T-cells--appears throughout the entire dataset in low numbers and as a likely marker of Cluster 6. This suggests that some background noise may have been introduced by the media the cells were in. We might consider in the wet lab trying to get a purer, more informative sample, with less background. Alternatively, in the dry lab, we can take advantage of techniques such as SoupX or others to remove this technical noise.

><tip-title>Removing Noise</tip-title>
>Adjusting the filtering settings (increasing minimum counts/cell, etc.) is often the place to start in the case of unexpected noise.
{: .tip}

Do you think the clustering is appropriate? i.e. are there single clusters that you think should be separate, and multiple clusters that could be combined?

```r
CellType_DimPlot<-DimPlot(object = filtered_srt, reduction = "umap", group.by = "celltype")
Cd4_FeaturePlot<-FeaturePlot(object = filtered_srt, reduction = "umap", features = "Cd4")
CellType_DimPlot | Cd4_FeaturePlot
```
![Double Positive differentiation?](../../images/scrna-SeuratRStudio/plot20.png "Double Positive differentiation?")

Important to note: just because a cluster doesn’t look like a cluster by eye, that is NOT enough to say it’s not a cluster! But looking at the biology here, we struggled to find marker genes to distinguish the double positive populations, which we know are also affected by depth of sequencing. That’s a reasonable argument that Clusters 1, 2, and 5 might not be all that different. Maybe we need more depth of sequencing across all those cells, or to compare these explicitly to each other (consider variations on FindMarkers!).

However, the late double positive cluster is both seemingly leaving the larger body of clusters and also has fewer knockout cells, so we might go and look at what those cells are expressing in the marker gene list. If we look at the mature T-cells further, we can see that their marker gene--Itm2a--is only expressed in half of the cluster. You might consider sub-clustering this to investigate further, either through changing the resolution or through analysing this cluster alone.

If we look at the differences between genotypes alone (the pseudo-bulk), we can see that many, if not most, of the genes in that list are actually ribosomal. This could be housekeeping background, it might be cell cycle related, it may be biological, or some combination of all three. You might consider investigating the cycling status of the cells, or even regressing this out (which is what the authors did).

Ultimately, there are quite a lot ways to analyse your scRNA-seq data, both within the confines of this tutorial (the many parameters that could be changed throughout) and outside of it (batch correction, sub-clustering, cell-cycle scoring, inferred trajectories, etc.) Most analyses will still yield the same general output, though: there are fewer knockout cells in the mature T-cell population, suggesting some sort of abberant development of T-cells in the Igf2-p0 hets.

Finally, we can export plots and objects from RStudio back into Galaxy. To do so, we'll use the gx_put() function provided to us by Galaxy. Let's save our Seurat object and the cell type labelled DimPlot!

```r
gx_put(filtered_srt)
gx_put(CellType_DimPlot)
```

The above functions will export your object and the plot into your Galaxy history from which you may download, share, and analyze!

Congratulations! You have interpreted your scRNA-seq data in several important ways!


topic: single-cell
tutorial: scrna-case_basic-pipeline
title: Filter, plot and explore single-cell RNA-seq data with Scanpy
difficulty: intermediate
requirements_json: []
tags_json: []
content_hash: 13b6ee4f0746885ac2f589608b262f50
last_modified: 2026-05-27T09:01:49.745791
zenodo_link: https://zenodo.org/record/15092468
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.html

content:
answer-key-2
    date: 2024-02-28

input_histories:
  - label: "UseGalaxy.eu"
    history:  https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/locked-filter-plot-and-explore-single-cell-rna-seq-data-input

questions:
- How can I assess my dataset quality?
- How do I pick thresholds and parameters in my analysis? What's a "reasonable" number, and will the world collapse if I pick the wrong one?
- How do I generate and annotate cell clusters?
objectives:
- Interpret quality control plots to inform parameter-decision making
- Repeat analysis from matrix to clustering
- Identify decision-making points
- Appraise data outputs and make informed decisions
- Explain why single cell analysis is an iterative (i.e. the first plots you generate are not final, but rather you go back and re-analyse your data repeatedly) process
time_estimation: 3H
key_points:
- Single-cell data is huge. To analyse single-cell data, you must reduce its many dimensions (cells x genes) via numerous methods
- Analysis is more subjective than we think. We need both prior, biological knowledge of the samples as well as many iterations of analysis, in order to derive novel, real biological insights

requirements:
-
    type: "internal"
    topic_name: single-cell
    tutorials:
        - scrna-case_alevin
        - scrna-case_alevin-combine-datasets
tags:
- paper-replication
- MIGHTS


contributions:
  authorship:
    - nomadscientist
  editing:
    - hexylena
    - wee-snufkin
    - nomadscientist
  testing:
    - wee-snufkin
  infrastructure:
    - pavanvidem
    - Nilchia


follow_up_training:
  -
    type: "internal"
    topic_name: single-cell
    tutorials:
        - scrna-case_JUPYTER-trajectories
        - scrna-case_monocle3-trajectories

recordings:
- captioners:
  - nomadscientist
  date: '2021-02-15'
  galaxy_version: '21.01'
  length: 30M
  youtube_id: M6iepSJh0EQ
  speakers:
  - nomadscientist

---



You've done all the work to make a single cell matrix, with mitochondrial genes flagge and buckets of cell metadata from all your variables of interest. Now it's time to fully process our data, to remove low quality cells, to reduce the many dimensions of the data that make it difficult to work with, and ultimately to try to define our clusters and to find our biological meaning and insights! There are many packages for analysing single cell data - Seurat {% cite Satija2015 %}, Scanpy {% cite Wolf2018 %}, Monocle {% cite Trapnell2014 %}, Scater {% cite McCarthy2017 %}, and so forth. We're working with Scanpy, although Galaxy has training using other packages, which you can explore on our {% icon level %} [Single-cell training topic]({% link topics/single-cell %}).

> <details-title>Where am I?</details-title>
>
> You are in one of the four tutorials associated with a Case Study, which replicates and expands on the analysis performed in a manuscript {% cite Bacon2018 %}.
>
> ![case study overview](../../images/scrna-casestudy/CS3-case_study_overview_3.png "Overview of the four parts of the case study, represented by boxes connected with noodles. There is a signpost specifying that you are currently in the first part.")
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

We've provided you with experimental data to analyse from a mouse dataset of fetal growth restriction {% cite Bacon2018 %}.

> <details-title>Fun facts about the dataset</details-title>
>
> We are particularly keen for learners to be able to go from raw FASTQ files all the way through analysis. We aren't handing you a curated dataset that we specially modified in order for this tutorial to work. Instead, this tutorial's input dataset is the full dataset generated from the {% icon level %} [previous tutorial in this case study]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md %}).
> - The only difference is that in that previous tutorial, we *downsampled* the datasets in so the tutorial would run faster. Our input data, however, now uses the *full dataset*, analysed exactly the same was as in the tutorial.
> - You can also find this dataset in the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/).
>
{: .details}

{% include _includes/cyoa-choices.html option1="Import History on EU server" option2="Zenodo" option3="SCXA" default="Import History on EU server"
       text="If you're on the EU server, (if your usegalaxy has an **EU** anywhere in the URL), then the quickest way to Get the Data for this tutorial is via importing a history. Otherwise, you can also import from Zenodo - it just might take a moment longer if you're in a live course and everyone is importing the same dataset at the same time! The SCXA is specifically for learners who are focusing on *Reusing public data*, so are not beginners." %}

<div class="Import-History-on-EU-server" markdown="1">

> <hands-on-title>Import History from EU server</hands-on-title>
>
> 1. Import the {% icon galaxy-history-input %} *Input history* by following the link below
>
>     {% for h in page.input_histories %}
>       [ {{h.label}} Input History]( {{h.history}} )
>     {% endfor %}
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
{: .hands_on}

</div>

<div class="Zenodo" markdown="1">

> <hands-on-title>Import from Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Batched_Object
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} the datasets `QC_Object`
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}

</div>

<div class="SCXA" markdown="1">

You can also pull the data from publicly available [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home).

> <hands-on-title>Import from the EBI Single Cell Expression Atlas</hands-on-title>
>
> 1. {% tool [EBI SCXA Data Retrieval](toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2) %} with the following parameters:
>    - **SC-Atlas experiment accession**: `E-MTAB-6945`
>
> 2. {% icon level %}**Follow tutorial to reformat dataset**: This [short tutorial]({% link topics/single-cell/tutorials/EBI-retrieval/tutorial.md %}) will show you how to use this tool and modify the output so that it's compatible with this tutorial and its workflow.
>
{: .hands_on}
</div>


# Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

{% snippet faqs/gtn/gtn_example_histories.md %}


# Quality control

You have generated an annotated AnnData object from your raw scRNA-seq fastq files. However, you have only completed a 'rough' filter of your dataset - there will still be a number of 'cells' that are actually just background from empty droplets or simply low-quality. There will also be genes that could be sequencing artifacts or that appear with such low frequency that statistical tools will fail to analyse them. This background garbage of both cells and genes not only makes it harder to distinguish real biological information from the noise, but also makes it computationally heavy to analyse. These spurious reads take a lot of computational power to analyse! First on our agenda is to filter this matrix to give us cleaner data to extract meaningful insight from, and to allow faster analysis.

## Calculate QC Metrics

To filter the object, we need to calculate some metrics for each cell and gene.

> <hands-on-title>Compute QC metrics</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Batched_Object`
>    - *"Method used for inspecting"*: `Calculate quality control metrics, using 'pp.calculate_qc_metrics'`
>      - *"Name of kind of values in X"*: `counts`
>      - *"The kind of thing the variables are"*: `genes`
>      - *"Keys for boolean columns of `.var` which identify variables you could want to control for"*: `mito`
>
> 2. Rename the generated file `QC_Object`
>
{: .hands_on}

## Inspect the AnnData Object

What has this tool calculated?

> <question-title></question-title>
>
> 1. What information is stored in your AnnData object? For example, the last tool to generate this object counted the mitochondrial associated genes in your matrix. Where is that data stored?
> 2. While you are figuring that out, how many genes and cells are in your object?
>
>    > <tip-title>Hint</tip-title>
>    > You want to use the same tool you used in the previous tutorial to examine your AnnData. Sometimes you can get the answers from *peeking* at your {% icon param-file %} AnnData object in your {% icon galaxy-history %} history, but sometimes it's not quite that simple!
>    >
>    >    > <hands-on-title>Inspecting AnnData Objects</hands-on-title>
>    >    >
>    >    > 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    >    >    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    >    >    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>    >    > 2. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    >    >    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    >    >    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>    >    {: .hands_on}
>    {: .tip}
>
> > <solution-title></solution-title>
> >
> > 1. If you examine your AnnData object, you'll find a number of different quality control metrics for:
> >  - *cells*, found in the {% icon param-file %} **Key-index observations annotation (obs)** output dataset
> >     - For example, you can find both discrete and log-based metrics for `n_genes` (how many genes are counted in a given cell), and `n_counts` (how many UMIs are counted in a given cell). This distinction between counts/UMIs or genes is because you might count multiple GAPDHs in a single cell. This would be 1 gene but multiple counts, therefore your `n_counts` should be higher than `n_genes` for an individual cell.
> >     - But what about the mitochondria?? You can also find `total_counts_mito`,  `log1p_total_counts_mito`, and `pct_counts_mito`, which has been calculated for each cell.
> >  - and *genes*, found in the {% icon param-file %} **Key-index observations variables/features (var)** output dataset.
> >     - For example, you can find `n_cells_by_counts` (number of cells that gene appears in).
> >
> > 2. There are `31670 cells` and `35734 genes` in the matrix.
> > - You can *peek* at your  {% icon param-file %} Anndata Object in your {% icon galaxy-history %} history by selecting it to reveal a drop-down window that has this same information in it.
> > - The matrix is `31670 x 35734`. This is `n_obs x n_vars`, or rather, `cells x genes`.
> >
> {: .solution}
>
{: .question}

## Generate QC Plots

We want to filter our cells, but first we need to know what our data looks like. There are a number of subjective choices to make within scRNA-seq analysis, for instance we now need to make our best informed decisions about where to set our thresholds (more on that soon!). We're going to plot our data a few different ways. Different bioinformaticians might prefer to see the data in different ways, and here we are only generating some of the myriad of plots you can use. Ultimately you need to go with what makes the most sense to you.

> <hands-on-title>Making QC visualisations - Violin Plots</hands-on-title>
>
> 1. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `sex`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_sex`
>
> 5. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `batch`
>
> 6. **Rename** {% icon galaxy-pencil %} output `Violin_log_batch`
>
{: .hands_on}

> <hands-on-title>Making QC visualisations - Scatterplots</hands-on-title>
>
> 1. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `log1p_total_counts`
>            - *"y coordinate"*: `pct_counts_mito`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Scatter_UMIxMito`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `log1p_n_genes_by_counts`
>            - *"y coordinate"*: `pct_counts_mito`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Scatter_GenesxMito`
>
> 5. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `log1p_n_genes_by_counts`
>            - *"y coordinate"*: `log1p_total_counts`
>            - *"Color by"*: `pct_counts_mito`
>
> 6. **Rename** {% icon galaxy-pencil %} output `Scatter_GenesxUMI`
>
{: .hands_on}

## Interpret the plots

That's a lot of information! Let's attack this in sections and see what questions these plots can help us answer.

{% snippet faqs/galaxy/features_scratchbook.md %}

> <question-title>Batch Variation</question-title>
>
> Are there differences in sequencing depth across the samples?
> 1. Which plot(s) addresses this?
> 2. How do you interpret it?
>
> > <solution-title></solution-title>
> >
> > 1. The plot `Violin_Log_Batch` will have what you're looking for!
> >     ![Violin_log_batch](../../images/scrna-casestudy/CS3-Violin_log_batch.png "Violin_Log_Batch (Raw)")
> > 2. Keeping in mind that this is a log scale - which means that small differences can mean large differences - the violin plots probably look pretty similar.
> >    - `N703` and `N707` might be a bit lower on genes and counts (or UMIs), but the differences aren't catastrophic.
> >    - The `pct_counts_mito` looks pretty similar across the batches, so this also looks good.
> >    - Nothing here would cause us to eliminate a sample from our analysis, but if you see a sample looking completely different from the rest, you would need to question why that is and consider eliminating it from your experiment!
> >
> {: .solution}
>
{: .question}

> <question-title>Biological Variables</question-title>
>
> Are there differences in sequencing depth across sex? Genotype?
> 1. Which plot(s) addresses this?
> 2. How do you interpret the `sex` differences?
> 3. How do you interpret the `genotype` differences?
>
> > <solution-title></solution-title>
> >
> > 1. Similar to above, the plots `violin - sex - log` and `violin - genotype - log` will have what you're looking for!
> >      ![Violin_Log_Sex](../../images/scrna-casestudy/CS3-Violin_log_sex.png "Violin_Log_Sex (Raw)")
> >      ![Violin_Log_Genotype](../../images/scrna-casestudy/CS3-Violin_log_genotype.png "Violin_Log_Genotype (Raw)")
> > 2. There isn’t a major difference in sequencing depth across sex, I would say - though you are welcome to disagree!
> >    - It is clear there are far fewer female cells, which makes sense given that only one sample was female. *Note - that was an unfortunate discovery made long after generating libraries. It's quite hard to identify the sex of a neonate in the lab! In practice, try hard to not let such a confounding factor into your data! You could consider re-running all the following analysis without that female sample, if you wish.*
> > 3. In `Violin - genotype - log`, however, we can see there is a difference. The `knockout` samples clearly have fewer genes and counts. From an experimental point of view, we can consider, does this make sense?
> >    - Would we biologically expect that those cells would be smaller or having fewer transcripts? Possibly, in this case, given that these cells were generated by growth restricted neonatal mice, and in which case we don’t need to worry about our good data, but rather keep this in mind when generating clusters, as we don’t want depth to define clusters, we want biology to!
> >    - On the other hand, it may be that those cells didn’t survive dissociation as well as the healthy ones (in which case we’d expect higher mitochondrial-associated genes, which we don’t see, so we can rule that out!).
> >    - Maybe we unluckily poorly prepared libraries for specifically those knockout samples. There are only three, so maybe those samples are under-sequenced.
> >    - So what do we do about all of this?
> >        - Ideally, we consider re-sequencing all the samples but with a higher concentration of the knockout samples in the library. Any bioinformatician will tell you that the best way to get clean data is in the lab, not the computer! Sadly, absolute best practice isn’t necessarily always a realistic option in the lab - for instance, that mouse line was long gone! - so sometimes, we have to make the best of it. There are options to try and address such discrepancy in sequencing depth. Thus, we’re going to take these samples forward and see if we can find biological insight despite the technical differences.
> >
> {: .solution}
>
{: .question}

Now that we've assessed the differences in our samples, we will look at the libraries overall to identify appropriate thresholds for our analysis.

> <question-title>Filter Thresholds: genes</question-title>
>
> What threshold should you set for `log1p_n_genes_by_counts`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. Any plot with `log1p_n_genes_by_counts` would do here, actually! Some people prefer scatterplots to violins.
> > ![Scatter_GenesxMito](../../images/scrna-casestudy/CS3-Scatter_GenesxMito.png "Scatter_GenesxMito (Raw)")
> >
> > 2. In `Scatter - mito x genes` you can see how cells with `log1p_n_genes_by_counts` up to around, perhaps, `5.7` (around 300 genes) often have high `pct_counts_mito`.
> >   - You can plot this as just `n_counts` and see this same trend at around 300 genes, but with this data the log format is clearer so that's how we're presenting it.
> >   - You could also use the violin plots to come up with the threshold, and thus also take batch into account. It's good to look at the violins as well, because you don't want to accidentally cut out an entire sample (i.e. N703 and N707).
> >   - Some bioinformaticians would recommend filtering each sample individually, but this is difficult in larger scale and in this case (you're welcome to give it a go! You'd have to filter separately and then concatenate), it won't make a notable difference in the final interpretation.
> >
> {: .solution}
{: .question}

> <question-title>Filter Thresholds: UMIs</question-title>
>
> What threshold should you set for `log1p_total_counts`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. As before, any plot with `log1p_total_counts` will do! Again, we'll use a scatterplot here, but you can use a violin plot if you wish!
> > ![Scatter_UMIxMito](../../images/scrna-casestudy/CS3-Scatter_UMIxMito.png "Scatter_UMIxMito (Raw)")
> >
> > 2. We can see that we will need to set a higher threshold (which makes sense, as you'd expect more UMI's per cell rather than unique genes!). Again, perhaps being a bit aggressive in our threshold, we might choose `6.3`, for instance (which amounts to around 500 counts/cell).
> >   - In an ideal world, you'll see a clear population of real cells separated from a clear population of debris. Many samples, like this one, are under-sequenced, and such separation would likely be seen after deeper sequencing!
> >
> {: .solution}
{: .question}

> <question-title>Filter Thresholds: mito</question-title>
>
> What threshold should you set for `pct_counts_mito`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. Any plot with `pct_counts_mito` would do here, however the scatterplots are likely the easiest to interpret. We'll use the same as last time.
> > ![Scatter_UMIxMito](../../images/scrna-casestudy/CS3-Scatter_UMIxMito.png "Scatter_UMIxMito (Raw)")
> >
> > 2. We can see a clear trend wherein cells that have around 5% mito counts or higher also have far fewer total counts. These cells are low quality, will muddy our data, and are likely stressed or ruptured prior to encapsulation in a droplet. While 5% is quite a common cut-off, this is quite messy data, so just for kicks we'll go more aggressive with a `4.5%`.
> >    - In general, you must adapt all cut-offs to your data - metabolically active cells might have higher mitochondrial RNA in general, and you don't want to lose a cell population because of a cut-off.
> >
> {: .solution}
{: .question}

## Apply the thresholds

It's now time to apply these thresholds to our data! First, a reminder of how many cells and genes are in your object: `31670 cells` and `35734 genes`. Let's see how that changes each time!

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **log1p_n_genes_by_counts** > `5.7`
>      - **log1p_total_counts** > `6.3`
>      - **pct_counts_mito** < `4.5%`
> - Everyone else: Choose your own thresholds and compare results!
{: .details}

### Genes/cell

> <hands-on-title>Filter cells by log1p_n_genes_by_counts</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_n_genes_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `greater than`
>                - *"Value"*: `5.7`
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy filter** {% icon tool %})
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_n_genes_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `20.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `Genes_Filtered_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Genes_Filtered_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype-Genes`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violin_Log_Genotype](../../images/scrna-casestudy/CS3-Violin_log_genotype.png "Raw")
> > ![Violinplot-filteronce](../../images/scrna-casestudy/CS3-Violin_log_genotype-Genes.png "1st filter - genes/cell")
> > 1. The only part that seems to change is the `log1p_n_genes_by_counts`.  You can see a flatter bottom to the violin plot - this is the lower threshold set. Ideally, this would create a beautiful violin plot because there would be a clear population of low-gene number cells. Sadly not the case here, but still a reasonable filter.
> > 2. If you *peek* at the AnnData object in your {% icon galaxy-history %}, you will find that you now have `17,104 cells x 35,734 genes`.
> >
> {: .solution}
>
{: .question}

### UMIs/cell

> <hands-on-title>Filter cells by log1p_total_counts</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Genes_Filtered_Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_total_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `greater than`
>                - *"Value"*: `6.3`
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy filter** {% icon tool %})
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_total_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `20.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `UMIs_Filtered_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `UMIs_Filtered_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype-UMIs`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violinplot-filteronce](../../images/scrna-casestudy/CS3-Violin_log_genotype-Genes.png "1st filter - genes/cell")
> > ![Violinplot-filtertwice](../../images/scrna-casestudy/CS3-Violin_log_genotype-UMIs.png "2nd filter - UMIs/cell")
> > 1. We will focus on the `log1p_total_counts` as that shows the biggest change. Similar to above, the bottom of the violin shape has flattered due to the threshold.
> > 2. You now have `8,677 cells x 35,734 genes` in the AnnData object.
> >
> {: .solution}
>
{: .question}

### % Mito/cell

> <hands-on-title>Filter cells by pct_counts_mito</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `UMIs_Filtered_Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `pct_counts_mito`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `4.5`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `Mito_Filtered_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Mito_Filtered_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype-Mito`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violinplot-filtertwice](../../images/scrna-casestudy/CS3-Violin_log_genotype-UMIs.png "2nd filter - UMIs/cell")
> > ![Violinplot-filterthrice](../../images/scrna-casestudy/CS3-Violin_log_genotype-Mito.png "3rd filter - % count Mito/cell")
> > 1. If we carefully check the axes, we can see that the `pct_counts_mito` has shrunk.
> > 2. Your object now has `8,604 cells x 35,734 genes`.
> >
> {: .solution}
>
{: .question}

> <details-title>See every step of filtering together</details-title>
> ![Violin_Log_Genotype](../../images/scrna-casestudy/CS3-Violin_log_genotype.png "Raw")
> ![Violinplot-filteronce](../../images/scrna-casestudy/CS3-Violin_log_genotype-Genes.png "1st filter - genes/cell")
> ![Violinplot-filtertwice](../../images/scrna-casestudy/CS3-Violin_log_genotype-UMIs.png "2nd filter - UMIs/cell")
> ![Violinplot-filterthrice](../../images/scrna-casestudy/CS3-Violin_log_genotype-Mito.png "3rd filter - % count Mito/cell")
>
{: .details}

Fantastic work! However, you've now removed a whole heap of cells, and since the captured genes are sporadic (i.e. a small percentage of the overall transcriptome per cell) this means there are a number of genes in your matrix that are currently not in any of the remaining cells. Genes that do not appear in any cell, or even in only 1 or 2 cells, will make some analytical tools break and overall will not be biologically informative. So let's remove them! Note that `3` is not necessarily the best number, rather it is a fairly conservative threshold. You could go as high as 10 or more.

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Variable: **n_cells**
> - Control > `3`
> - Everyone else: Choose your own thresholds and compare results! Note if you go less than 3 (or even remove this step entirely), future tools are likely to fail due to empty gene data.
{: .details}

### Cells/gene

> <hands-on-title>Filter genes</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Mito-filtered Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `n_cells_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `greater than`
>                - *"Value"*: `3.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `Cells_Filtered_Object`
>
{: .hands_on}

In practice, you'll likely choose your thresholds then set up all these filters to run without checking plots in between each one. But it's nice to see how they work!

We can summarise the results of our filtering:

|       | Cells | Genes |
|------ |--------------------|
| Raw | 31670    | 35734    |
| Filter genes/cell | 17104    | 35734    |
| Filter UMIs/cell | 8677    | 35734    |
| Filter mito/cell | 8604   | 35734    |
| Filter cells/gene | 8604    | 15950    |

{% icon congratulations %} Congratulations! You have filtered your object! Now it should be a lot faster to analyse and easier to interpret.

# Processing

So currently, you have a matrix that is 8604 cells by 15950 genes. This is still quite big data. We have two issues here - firstly, you already know there are differences in how many transcripts and genes have been counted per cell. This technical variable can obscure biological differences. Secondly, we like to plot things on x/y plots, so for instance *Gapdh* could be on one axis, and *Actin* can be on another, and you plot cells on that 2-dimensional axis based on how many of each transcript they possess. While that would be fine, adding in a 3rd dimension (or, indeed, in this case, 15950 more dimensions), is a bit trickier! So our next steps are to transform our big data object into something that is easy to analyse and easy to visualise.

> <hands-on-title>Normalisation</hands-on-title>
>
> 1. {% tool [Scanpy normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Cells_Filtered_Object`
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>        - *"Target sum"*: `10000.0`
>        - *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>        - *"Name of the field in 'adata.obs' where the normalization factor is stored"*: `norm`
>
> 2. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy normalize** {% icon tool %})
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
>
> 3. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy Inspect and manipulate** {% icon tool %})
>    - *"Function to manipulate the object"*: `Freeze the current state into the 'raw' attribute`
{: .hands_on}

Normalisation helps reduce the differences between gene and UMI counts by fitting total counts to 10,000 per cell. The subsequent log-transform (by log(count+1)) aligns the gene expression level better with a normal distribution. This is fairly standard to prepare for any future dimensionality reduction. Finally, we freeze this information in the 'raw' attribute before we further manipulate the values.

We next need to look at reducing our gene dimensions. We have loads of genes, but not all of them are different from cell to cell. For instance, housekeeping genes are defined as not changing much from cell to cell, so we could remove these from our data to simplify the dataset. We will flag genes that vary across the cells for future analysis.

> <hands-on-title>Find variable genes</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for filtering"*: `Annotate (and filter) highly variable genes, using 'pp.highly_variable_genes'`
>        - *"Choose the flavor for identifying highly variable genes"*: `Seurat`
>
{: .hands_on}

> <details-title>More details on the Highly Variable Genes</details-title>
>
> Would you like to know how *many* genes were flagged as **Highly variable genes**?
>
>
> > <hands-on-title>Find the number of variable genes</hands-on-title>
> >
> >
> > 1. Select {% icon galaxy-refresh %} **Run Job Again** on the `anndata_out` (output of **Scanpy filter** {% icon tool %}) in your {% icon galaxy-history %} history
> >
> > 2. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
> >   - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
> >   - *"Method used for filtering"*: `Annotate (and filter) highly variable genes, using 'pp.highly_variable_genes'`
> >       - *"Choose the flavor for identifying highly variable genes"*: `Seurat`
> >       - *"Inplace subset to highly-variable genes?"*: {% icon galaxy-toggle %} **Yes**
> >
> {: .hands_on}
>
>
> If you peek at the output, you will see that the number of *genes* in your AnnData object has drastically reduced to around `3216` - this dataset has *only* the highly variable genes! Some people prefer to only perform analysis on this dataset, however I have found that sometimes (for various reasons) important biological marker genes get excluded. For this reason, I personally will flag highly variable genes for use in the next analytical steps, however I keep all the genes in my AnnData object so that I can check for key ones in the future.
>
> - {% icon warning %} For this tutorial, you **must** keep all genes in your AnnData object. Therefore, delete the output that contains *only* the highly variable genes from your {% icon galaxy-history %} history now.
>
>
{: .details}

Next up, we're going to scale our data so that all genes have the same variance and a zero mean. This is important to set up our data for further dimensionality reduction. It also helps negate sequencing depth differences between samples, since the gene levels across the cells become comparable. Note, that the differences from scaling etc. are not the values you have at the end - i.e. if your cell has average GAPDH levels, it will not appear as a '0' when you calculate gene differences between clusters.

> <hands-on-title>Scale data</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy filter** {% icon tool %})
>    - *"Method used for inspecting"*: `Scale data to unit variance and zero mean, using 'pp.scale'`
>        - *"Maximum value"*: `10.0`
>
> 3. **Rename** {% icon galaxy-pencil %} output `Scaled_Object`
>
{: .hands_on}

{% icon congratulations %} Congratulations! You have processed your object!

> <details-title>More details on batch correction and removing unwanted variation</details-title>
>
> At this point, you might want to remove or regress out the effects of unwanted variation on our data. A common example of this is the cell cycle, which can affect which genes are expressed and how much material is present in our cells. If you’re interested in learning how to do this, then you can move over to the {% icon level %} [Removing the Effects of the Cell Cycle]({% link topics/single-cell/tutorials/scrna-case_cell-cycle/tutorial.md %}) tutorial now and return here to complete your analysis.
>
> - {% icon warning %} If you are in a *live course*, the time to do this *bonus tutorial* will not be factored into the schedule. Please instead return to this *after* your course is finished, or if you finish early!
>
{: .details}

# Preparing coordinates

We still have too many dimensions. Transcript changes are not usually singular - which is to say, genes were in pathways and in groups. It would be easier to analyse our data if we could more easily group these changes.

## Principal components
Principal components are calculated from highly dimensional data to find the most spread in the dataset. Given that our object has around `3216` highly variable genes, that's 3216 dimensions. There will, however, be one line/axis/dimension that yields the most spread and variation across the cells. That will be our first principal component. We can calculate the first `x` principal components in our data to drastically reduce the number of dimensions.

> <warning-title>Check the size of your AnnData object!</warning-title>
> Your AnnData object should have far more than 3216 genes in it (if you followed our settings and tool versions, you'll have a matrix around 8604 × 15950 (cells x genes). If you followed the *More details on the Highly Variable Genes above, you may have created an object with *only* the highly variable genes. Please delete this object and do not use it! Carry forward the object with around 8604 × 15950 (cells x genes)!
{: .warning}

> <hands-on-title>Calculate Principal Components</hands-on-title>
>
> 1. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Scaled_Object`
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'pp.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>            - *"Change to use different initial states for the optimization"*: `1`
>
> 2. **Rename** {% icon galaxy-pencil %} output `PCA_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PCA_Object`
>    - *"Method used for plotting"*: `PCA: Scatter plot in PCA coordinates, using 'pl.pca_variance_ratio'`
>        - *"Number of PCs to show"*: `50`
>
> 3. **Rename** {% icon galaxy-pencil %} plot output `PCA_Variance_Plot`
>
{: .hands_on}

Why 50 principal components you ask? Well, we're pretty confident 50 is an over-estimate. Examine `PCA Variance`.

![PCA-variance](../../images/scrna-casestudy/CS3-PCA_Variance_Plot.png "Variability explained per PC")

We can see that there is really not much variation explained past component 19. So we might save ourselves a great deal of time and muddied data by focusing on the top `20` PCs. (You could probably even go as low as 10!)

## Neighborhood graph

We're still looking at around 20 dimensions at this point in our analysis. We need to identify how similar a cell is to another cell, across every cell across these dimensions. For this, we will use the k-nearest neighbor (kNN) graph, to identify which cells are close together and which are not. The kNN graph plots connections between cells if their distance (when plotted in this 20 dimensional space!) is amongst the k-th smallest distances from that cell to other cells. This will be crucial for identifying clusters, and is necessary for plotting a UMAP. From [UMAP developers](https://github.com/lmcinnes/umap): "Larger neighbor values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15 being a sensible default".

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **Number of PCs to use** = `20`
>      - **Maximum number of neighbours used** = `15`
> - Everyone else: Use the PC variance plot to pick your own PC number, and choose your own neighbour maximum as well!b
{: .details}

> <hands-on-title>ComputeGraph</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PCA_Object`
>    - *"Method used for inspecting"*: `Compute a neighborhood graph of observations, using 'pp.neighbors'`
>        - *"Number of PCs to use"*: `20`
>        - *"Use the indicated representation"*: `X_pca`
>        - *"Numpy random seed"*: `1`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Neighbours_Object`
>
{: .hands_on}

## Dimensionality reduction for visualisation

Two major visualisations for this data are tSNE and UMAP. We must calculate the coordinates for both prior to visualisation. For tSNE, the parameter [**perplexity**](https://www.nature.com/articles/s41467-019-13056-x) can be changed to best represent the data, while for UMAP the main change would be to change the kNN graph above itself, by changing the **neighbours**.

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **Perplexity** = `30`
> - Everyone else: Choose your own perplexity, between 5 and 50!
{: .details}

> <hands-on-title>Calculating tSNE & UMAP</hands-on-title>
>
> 1. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Neighbours_Object`
>    - *"Method used"*: `t-distributed stochastic neighborhood embedding (tSNE), using 'tl.tsne'`
>        - *"Number of PCs to use"*: `20`
>        - *"Use the indicated representation"*: `X_pca`
>        - *"Random state"*: `1`
>
> 2. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy cluster, embed** {% icon tool %})
>    - *"Method used"*: `Embed the neighborhood graph using UMAP, using 'tl.umap'`
>        - *"Seed used by the random number generator"*: `1`
>
> 2. **Rename** {% icon galaxy-pencil %} output `UMAP_Object`
>
{: .hands_on}

{% icon congratulations %} Congratulations! You have prepared your object and created neighborhood coordinates. We can now use those to call some clusters!

# Cell clusters & gene markers

> <question-title></question-title>
>
> Let's take a step back here. What is it, exactly, that you are trying to get from your data? What do you want to visualise, and what information do you need from your data to gain insight?
>
> > <solution-title></solution-title>
> >
> > Really we need two things - firstly, we need to make sure our experiment was set up well. This is to say, our biological replicates should overlap and our variables should, ideally, show some difference. Secondly, we want insight - we want to know which cell types are in our data, which genes drive those cell types, and in this case, how they might be affected by our biological variable of growth restriction. How does this affect the developing cells, and what genes drive this? So let's add in information about cell

tools_json:
["EBI SCXA Data Retrieval", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2", "Scanpy Inspect and manipulate", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy2", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1", "Scanpy plot", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2", "Scanpy filter", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3", "Scanpy normalize", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.10.2+galaxy0", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1", "Scanpy cluster, embed", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2", "Interactive CELLxGENE VIP Environment", "interactive_tool_cellxgene_vip"]


topic: single-cell
tutorial: bulk-music-4-compare
title: Comparing inferred cell compositions using MuSiC deconvolution
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["transcriptomics"]
content_hash: 86ba0f8e1a27069712de333f7de4a509
last_modified: 2026-05-15T22:28:16.660167
zenodo_link: https://zenodo.org/record/7319925
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music-4-compare/tutorial.html

content:
The goal of this tutorial is to apply bulk RNA deconvolution techniques to a problem with multiple variables - in this case, a model of diabetes is compared with its healthy counterparts. All you need to compare inferred cell compositions are well-annotated, high quality reference scRNA-seq datasets, transformed into MuSiC-friendly Expression Set objects, and your bulk RNA-samples of choice (also transformed into MuSiC-friendly Expression Set objects). For more information on how MuSiC works, you can check out their github site [MuSiC](https://xuranw.github.io/MuSiC/articles/MuSiC.html) or published article ({% cite wang2019bulk %}).

> <comment-title>Research question</comment-title>
> - How does variable X impact the cell distributions in my samples?
> - Needs: scRNA-seq reference dataset; bulk RNA-seq samples of interest to compare
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data

In the standard MuSiC tutorial, we used human pancreas data. We will now use the same single cell reference dataset {%cite segerstolpe2016single %} with its 10 samples of 6 healthy subjects and 4 with Type-II diabetes (T2D), as well as the bulk RNA-samples from the same lab (3 healthy, 4 diseased). Both of these datasets were accessed from the public EMBL-EBI repositories and transformed into Expression Set objects in the previous two tutorials. For both the single cell reference and the bulk samples of interest, you have generated Expression Set objects with only T2D samples, only healthy samples, and a final everything-combined sample for the scRNA reference. We won't need the combined bulk RNA dataset. The plan is to analyse this data in three ways: using a combined reference (altogether); using only the healthy single cell reference (healthyscref); or using a healthy and combined reference separately (like4like), all to identify differences in cellular composition.

![Three colours of arrows connect bulk healthy & diseased data sets to a combined single cell (altogether); bulk healthy and single cell healthy & bulk diseased with single cell diseased (like4like); and bulk diseased and healthy with the single cell healthy reference (healthyscref).](../../images/bulk-music/comparison.png "Plan of analysis")

If you have followed the previous tutorials, you will have built your [single cell ESet object]({% link topics/single-cell/tutorials/bulk-music-2-preparescref/tutorial.md %}) and your [bulk ESet object]({% link topics/single-cell/tutorials/bulk-music-3-preparebulk/tutorial.md %}), then you can copy these into a new history now. Otherwise, follow the steps below to import the datasets you'll need.

{% snippet  faqs/galaxy/histories_copy_dataset.md %}

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Compare"*
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    * Human single cell RNA ESet objects (tag: `#singlecell`)
>
>      ```
>    {{ page.zenodo_link }}/files/ESet_object_sc_combined.rdata
>    {{ page.zenodo_link }}/files/ESet_object_sc_T2D.rdata
>    {{ page.zenodo_link }}/files/ESet_object_sc_healthy.rdata
>      ```
>
>    * Human bulk RNA ESet objects (tag: `#bulk`)
>      ```
>    {{ page.zenodo_link }}/files/ESet_object_bulk_healthy.rdata
>    {{ page.zenodo_link }}/files/ESet_object_bulk_T2D.rdata
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets as needed
>
> 4. Add to each file a tag corresponding to `#bulk` and `#scrna`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Infer cellular composition & compare

It's finally time!

## Altogether: Deconvolution with a combined sc reference

{% snippet faqs/galaxy/tutorial_mode.md %}

> <hands-on-title> Comparing: altogether </hands-on-title>
>
> 1. {% tool [MuSiC Compare](toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4) %} with the following parameters:
>    - In *"New scRNA Group"*:
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_combined.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:Normal`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_healthy.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:T2D`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_T2D.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>
> 2. To each of the outputs, add the `#altogether` tag.
>
{: .hands_on}

There are four sets of output files.
1. *Summarised Plots* <- This is the most interesting output, because it has the pretty pictures!
2. *Individual Heatmaps* <- This kind of does what standard (non-Comparing) MuSiC does for each sample, rather than combining them.
3. *Stats* <- This will be very handy if you want to make any statistical calculations, as it contains medians and quartiles
4. *Tables* <- This contains the cell proportions found within each sample as well as the number of reads.

### Summarised Plots

Examine {% icon galaxy-eye %} the output file *Summarised Plots (MuSiC)*. Now the first few pages are similar to the standard deconvolution tool, but now comparing across the factor of interest (disease). Among the myriad of visualisations available, our favourite is on page 5 - a comparison of inferred cell proportions across disease.

![Graph showing two rows for each cell type (gamma, ductal, delta, beta, alpha, and acinar cells) comparison normal or T2D proportions by either read or by sample.](../../images/bulk-music/plot_altogether.png "Altogether: Resulting Graph")

Here we can see that the bulk-RNA seq samples from the T2D patients contain markedly fewer beta cells as compared with their healthy counterparts. This makes sense, so that's good!

### Individual Heatmaps

Examine {% icon galaxy-eye %} the output file *Individual heatmaps (MuSiC)*. This shows the cell distribution across each of the individual samples, separated out by disease factor into two separate plots, but ultimately isn't particularly informative.

![Heatmap showing vertically four samples (Bulk) and horizontally each cell type (from alpha to ductal), with gradations of colour referring to the proportion of that cell type within each sample.](../../images/bulk-music/individual_heatmaps.png "Altogether: Individual heatmaps")

### Stats

If you select the *Stats* dataset, you'll find it contains four sets of data, `Bulk_disease: Read Props`, `Bulk_disease: Sample Props`, `Bulk_healthy: Read Props` and `Bulk_healthy: Sample Props`. Examine {% icon galaxy-eye %} the file `Bulk_disease: Sample Props`. This contains summary statistics (Min, quartiles, median, mean, etc.) for each phenotype. This could be quite helpful if you're trying to statistically identify differences across samples.

![Table with rows as cell types and columns as statistics: Min, 1st Qu., Median, Mean.](../../images/bulk-music/summary_statistics.png "Altogether: Stats")

### Tables

Finally, if you select the *Tables* dataset, you'll find it contains three sets of data, `Data Table`, `Matrix of Cell Type Read Counts`, and `Matrix of Cell Type Sample Proportions`.

Examine {% icon galaxy-eye %} the file `Data Table`. This contains the inferred proportions and reads associated with each sample and cell type, along with its important factor of interest (Disease). In this tutorial, we tend to use sample proportions rather than read count, but either works. The two other matrix files are just portions of this data table.

![Table with individual rows for each cell type within each sample, with columns of Cell, Factor, CT Prop in Sample and Number of Reads.](../../images/bulk-music/data_table.png "Altogether: Data table")

> <question-title></question-title>
>
> 1. Why does the data table contain 42 rows?
>
> > <solution-title></solution-title>
> >
> > 1. The data table contains a row for each cell type within each sample. Since there are 6 cell types and 7 samples, `6*7 = 42` rows.
> >
> {: .solution}
>
{: .question}

Hopefully, this has been illuminating! Now let's try two other ways of inferring from a reference and see if it makes a difference.

## Like4like: Deconvolution of healthy samples with a healthy reference and diseased samples with a diseased reference

> <hands-on-title> Like4like Inference </hands-on-title>
>
> 1. {% tool [MuSiC Compare](toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4) %} with the following parameters:
>    - In *"New scRNA Group"*:
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set:Normal`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_healthy.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:Normal`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_healthy.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set:T2D`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_T2D.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `bulk_set:T2D`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_T2D.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>
> 2. Add the `#like4like` tag to each of the outputs.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How have the cell inferences changed, now that we have changed the scRNA references used?
>
> > <solution-title></solution-title>
> >
> > ![Two graphs showing two rows for each cell type (gamma, ductal, delta, beta, alpha, and acinar cells) comparison normal or T2D proportions by either read or by sample, with the top graph labelled #altogether and the bottom labelled #like4like. Differences are more pronounced in the top #altogether graph.](../../images/bulk-music/compare_like.png "Altogether vs like4like")
> > 1. Overall, our interpretation here is that the differences are less pronounced. It's interesting to conjecture whether this is an artefact of analysis, or whether - possibly - the beta cells in the diseased samples are not only fewer, but also contain fewer beta-cell specific transcripts (and thereby inhibited beta cell function), thereby lowering the bar for the inference of a beta cell and leading to a higher proportion of interred B-cells.
> >
> {: .solution}
>
{: .question}

Let's try one more inference - this time, we'll use only healthy cells as a reference, to (theoretically) make a more consistent analysis across the two phenotypes.

## healthyscref: Deconvolution using only healthy cells as a reference

> <hands-on-title> Healthy sc reference only inference </hands-on-title>
>
> 1. {% tool [MuSiC Compare](toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4) %} with the following parameters:
>    - In *"New scRNA Group"*:
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set:Normal`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_healthy.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:Normal`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_healthy.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:T2D`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_T2D.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>
> 2. Add the `#healthyscref` tag to each of the outputs.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How have the cell inferences changed this time?
>
> > <solution-title></solution-title>
> >
> > ![Three graphs showing two rows for each cell type (gamma, ductal, delta, beta, alpha, and acinar cells) comparison normal or T2D proportions by either read or by sample, with the top graph labelled #altogether; the middle labelled #like4like; and the bottom labelled #healthyscref. Differences are most pronounced in the bottom #healthyscref graph.](../../images/bulk-music/compare_3.png "The impact of the single cell reference")
> >
> > 1. If using a like4like inference reduced the difference between the phenotype, aligning both phenotypes to the same (healthy) reference exacerbated them - there are even fewer beta cells in the output of this analysis.
> >
> {: .solution}
>
{: .question}

Overall, it's important to remember how the inference changes depending on the reference used - for example, a combined reference might have majority healthy samples or diseased samples, so that would impact the inferred cellular compositions.

# Conclusion

Congrats! You've made it to the end of this suite of deconvolution tutorials! You've learned how to find quality data for reference and for analysis, how to reformat it for deconvolution using MuSiC, and how to compare cellular inferences using multiple kinds of reference datasets. You can find the [workflow for this tutorial](https://usegalaxy.eu/u/wendi.bacon.training/w/music-deconvolution-compare-1) and an [example history](https://usegalaxy.eu/u/wendi.bacon.training/h/music-deconvolution-comparisons--final).

 We hope this helps you in your research!

![Workflow editor showing 5 inputs and 3 runs of the MuSiC Compare tool](../../images/bulk-music/compare_workflow.png "MuSiC Compare Tutorial Workflow")

This tutorial is part of the [https://singlecell.usegalaxy.eu](https://singlecell.usegalaxy.eu) portal ({% cite tekman2020single %}).

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["MuSiC Compare", "toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4"]


topic: single-cell
tutorial: scrna-case_cell-cycle
title: Removing the effects of the cell cycle
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: 2ac72844009e8f68f7f754d73b2cb74b
last_modified: 2026-05-27T09:01:49.746441
zenodo_link: https://zenodo.org/record/7311628/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_cell-cycle/tutorial.html

content:
Single-cell RNA sequencing can be sensitive to both biological and technical variation, which is why preparing your data carefully is an important part of the analysis. You want the results to reflect the interesting differences in expression between cells that relate to their type or state. Other sources of variation can conceal or confound this, making it harder for you to see what is going on.

One common biological confounder is the cell cycle ({% cite Luecken2019 %}). Cells express different genes during different parts of the cell cycle, depending on whether they are in their growing phase (G1), duplicating their DNA (the S or Synthesis phase), or dividing in two (G2/M or Mitosis phase). If these cell cycle genes are having a big impact on your data, then you could end up with separate clusters that actually represent cells of the same type that are just at different stages of the cycle.

In this tutorial, we will identify the genes whose expression is known to vary during the cell cycle so that we can use them to regress out (or remove) the effects of the cell cycle on the clustering.

> <comment-title>Other Scanpy and Seurat tutorials</comment-title>
>
> This tutorial is based on the Scanpy cell cycle regression tutorial ({% cite ScanpyCC %}), which was itself based on the Seurat vignette addressing the same issue ({% cite SeuratCC %}). However, we will be using a different dataset for this tutorial.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get Data

The data used in this tutorial is from a mouse dataset of fetal growth restriction ({% cite Bacon2018 %}). Cell cycle regression should be performed after the data has been filtered, normalised, and scaled. You can download the dataset below or import the history with the starting data.

> <comment-title></comment-title>
> - If you've been working through the Single-cell RNA-seq: Case Study then you can use your dataset from the [Filter, Plot and Explore Single-cell RNA-seq Data]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial here. Select the `Use_me_Scaled` dataset from your history. Rename that dataset as `Processed_Anndata`. You will still need to import the S and G2/M gene lists below through Zenodo.
> - At the end of this tutorial, you can return to the main tutorial to plot and explore your data with reduced effects from the cell cycle.
{: .comment}

In addition to the scRNA-seq dataset, we will also need lists of the genes that are known to be expressed at different points in the cell cycle. The lists used in this tutorial are part of the HBC tinyatlas and can be downloaded from Zenodo below {% cite kirchner_hbc_2018 %}. Between them, they include 97 genes that are expressed during the S and G2/M phases. The expression level of these cycle genes are - mostly - determined by the phases of the cell cycle. Make sure that the file type is tabular (not just the name of the file) - you can choose this when you download the files or change it after the files are in your history.

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
> Sometimes data upload can take a while, so a faster route is to import a history.
>
> 1. Import history from: [example input history](https://humancellatlas.usegalaxy.eu/u/marisa_jl/h/removing-the-effects-of-the-cell-cycle---input-history)
>
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the the history to your name of choice.
>
{: .hands_on}

> <hands-on-title>Option 2: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}files/Processed_AnnData.h5ad
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the dataset `Processed_Anndata`
>
>    {% snippet  faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}


> <hands-on-title>Option 2 continued: Data upload - Add to history </hands-on-title>
>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/sPhase.tabular
>    {{ page.zenodo_link }}/files/g2mPhase.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 2. Rename the datasets `sPhase` and `g2mPhase` respectively - be careful not to mix them up!
>
>    {% snippet  faqs/galaxy/datasets_rename.md %}
>
> 3. Check that the datatype for both is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

# Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

# Cell Cycle Scoring

The first step towards reducing the effects of the cell cycle on our dataset is cell cycle scoring. The cell cycle scoring algorithm will look at each cell in turn and calculate an S score based on the difference in the mean expression of the S Phase genes and a random sample of the same number of non-cell cycle genes from the same cell. It will do the same for the G2M genes in order to calculate the G2M score. The cells will then be assigned to the most likely phase: S, G2M, or G1, if neither G2M or S score highly. Three columns will be added to the AnnData dataset: `S_score`, `G2M_score` and `phase`.

<question-title></question-title>
>
> 1. Why don't we need a list of genes expressed in the G1 Phase?
>
> > <solution-title></solution-title>
> >
> > 1. Since we know which genes are expressed in the S and G2/M phases, we can classify cells that are expressing these genes into the S and G2/M phases respectively. Cells that aren't expressing either the S or G2/M genes must be in the other phase of the cell cycle, so we can classify them as in G1 phase.
> >
> {: .solution}
{: .question}

> <comment-title>When should we regress out the effects of the cell cycle?</comment-title>
>  Cell cycle regression can be particularly important if we are planning to do trajectory analysis down the line or if we have a dataset that is very strongly influenced by the cell cycle {% cite Luecken2019 %}. However, it isn't always appropriate to remove the effects of the cell cycle - sometimes they can be useful for distinguishing between dividing and non-dividing cell types. When you are analysing your own data, you might need to try it both ways to determine if the effects of the cell cycle are helpful or not. You could also check whether the cell cycle genes are among the top scoring genes expressed by your cell clusters to get an idea of how strong the effects are.
{: .comment}

> <hands-on-title>Score the cell cycle genes</hands-on-title>
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Processed_Anndata` (Input dataset)
>    - *"Method used for inspecting"*: `Score cell cycle genes, using 'tl.score_genes_cell_cycle'`
>        - *"Format for the list of genes associated with S phase"*: `File`
>            - {% icon param-file %} *"File with the list of genes associated with S phase"*: `sPhase` (Input dataset)
>        - *"Format for the list of genes associated with G2M phase"*: `File`
>            - {% icon param-file %} *"File with the list of genes associated with G2M phase"*: `g2mPhase` (Input dataset)
>
> 2. Rename the output `CellCycle_Annotated`
>
{: .hands_on}

# Cell Cycle Regression

The second step after scoring the cell cycle genes is to use these scores to regress out the effects of the cell cycle. Now that we know which phase each of our cells is in, we can work out how this is affecting gene expression in our cells. We can subtract these effects from our data so that they won't influence our cell clustering. You will need to type the `phase` variable in to the {% tool Scanpy RegressOut %} tool to enable it to regress out the cell cycle effects.

> <details-title>How does cell cycle regression work?</details-title>
>
>The {% tool Scanpy RegressOut %} tool will create a linear model of the relationship between gene expression and the cell cycle scores we assigned in the previous step. Basically, this model is a line that shows how gene expression changes as the S or G2M score changes. Each gene will have its own line, so for any S score or G2M score, we could look at the corresponding point on the line to see the expected expression level of that gene.
>
>{% tool Scanpy RegressOut %} will then regress out or remove this expected effect for the genes expressed by each cell, according to the cell's S and G2M scores. The expected effect is subtracted from the expression data, leaving behind the difference between the expected position on the line and the actual position of each data point. The data points won't sit exactly on the line because their expression levels aren't determined completely by the cell cycle - the linear model only tells us what we would expect based on the cell cycle scores alone.
>
> Understanding how the regression works should help you to see why we're not just deleting the cell cycle genes from the dataset. We are using these genes that are known to be expressed during different phases to calculate the cell cycle scores and determine the phase each cell is in. We then use this information to work out the effect of the cell cycle on all of the other genes expressed by the cells. Even if they are not cell cycle genes, their expression can still be affected by the cycle. Finally, we remove or regress out the effect of the cell cycle on all the genes, leaving behind the variation that we're interested in.
>
{: .details}


> <hands-on-title>Regress out the effects of the cell cycle</hands-on-title>
>
> 1. {% tool [Scanpy RegressOut](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_regress_variable/scanpy_regress_variable/1.8.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `CellCycle_Annotated` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Variables to regress out"*: `phase`
>
>
>  2. Rename the output `CellCycle_Regressed`
>
{: .hands_on}

# Plotting the Effects of Cell Cycle Regression

Your data is now ready for further analysis, so you can return to the [Filter, Plot and Explore Single-cell RNA-seq Data]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial and move on to the Preparing coordinates step there. Make sure that you use the `CellCycle_Regressed` dataset (you may want to rename it as `Use_me_Scaled` as that is the name used in the main tutorial). However, if you want to understand how cell cycle regression has affected your data then you can work through the following steps first, to visualise where the cell cycle genes are expressed - with or without regression.

In order to look at the cell cycle genes, we first need to label them in our AnnData dataset so that we can select them for plotting. To add a new annotation to the genes or variables, we need a column with entries for each gene, in the same order as in the dataset, and with a header at the top that will become the key for identifying these entries in the AnnData dataset. We want to end up with a column that reads `TRUE` for the 97 cell cycle genes and `FALSE` for all the other genes.

You might find it easier to create this new column using a spreadsheet and then upload it as a tabular dataset, but it is possible to complete all the steps on Galaxy.

## Prepare a table of cell cycle genes

If we're going to mark all the cell cycle genes, we'll need a single list of all 97 genes instead of the two separate lists for S Phase and G2/M Phase. We'll combine the two lists into a single column with 97 entries. We'll then add a second column that simply reads `TRUE` in every row, which we'll use later to mark these as cell cycle genes in the main dataset.

> <hands-on-title>Create a list of all cell cycle genes</hands-on-title>
>
> 1. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: `sPhase` (Input dataset)
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Select"*: `g2mPhase` (Input dataset)
>
>
> 2. {% tool [Add column](toolshed.g2.bx.psu.edu/repos/devteam/add_value/addValue/1.0.0) %} with the following parameters:
>    - *"Add this value"*: `TRUE`
>    - {% icon param-file %} *"to Dataset"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>
>
> 3. Rename the dataset `CC_Genes`
{: .hands_on}

## Create an ordered list of gene names

Next, we'll need a list of all the genes in our dataset, so that we can mark the ones that are in our cell cycle list. We'll also add a column of numbers as this will help us keep the gene names in order.

> <hands-on-title>Get the gene names from your dataset</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed` (Input dataset)
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
>
> 2. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy0) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `var` (output of **Inspect AnnData** {% icon tool %})
>        - *"Type of table operation"*: `Drop, keep or duplicate rows and columns`
>            - *"List of columns to select"*: `1`
>    - *"Output formatting options"*: `Unselect all`
>
>
> 3. {% tool [Add column](toolshed.g2.bx.psu.edu/repos/devteam/add_value/addValue/1.0.0) %} with the following parameters:
>    - {% icon param-file %} *"to Dataset"*: `table` (output of **Table Compute** {% icon tool %})
>    - *"Iterate?"*: `YES`
>
>
>    > <comment-title>Keeping the genes in order</comment-title>
>    >
>    > Adding these numbers will enable us to keep the genes in their original order. This is essential for adding the cell cycle gene annotation back into the AnnData dataset.
>    {: .comment}
>
>
> 4. Rename the output `Dataset_Genes`
{: .hands_on}


## Mark the cell cycle genes

We can now combine our table of cell cycle genes `CC_genes` with the table of gene names `Dataset_Genes`.

> <hands-on-title>Combine the two tables</hands-on-title>
>
> 1. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2) %} with the following parameters:
>    - {% icon param-file %} *"1st file"*: `Dataset_Genes` (output of **Add column** {% icon tool %})
>    - *"Column to use from 1st file"*: `c1`
>    - {% icon param-file %} *"2nd File"*: `CC_Genes` (output of **Add column** {% icon tool %})
>    - *"Column to use from 2nd file"*: `c1`
>    - *"Output lines appearing in"*: `All lines [-a 1 -a 2]`
>    - *"Value to put in unpaired (empty) fields"*: `FALSE`
>
>
>    > <comment-title>How the cell cycle genes are marked</comment-title>
>    >
>    > When we join the two tables, we'll ask for any empty fields to be filled in with `FALSE`. The cell cycle gene table has an extra column where they are all marked as `TRUE` - they will retain these entries when we join the tables, but since there are no entries for the rest of the genes in this column, their rows will be filled in as `FALSE`. This will enable us to pick out the cell cycle genes later.
>    {: .comment}
>
>
> 2. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: `output` (output of **Join** {% icon tool %})
>    - *"on column"*: `c2`
>    - *"everything in"*: `Ascending order`
>
>
>    > <comment-title>Putting the genes in order again</comment-title>
>    >
>    > Sorting the genes using the column of numbers we added earlier will put them back in their original order - make sure to sort them in ascending order, otherwise they'll end up the opposite way around.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What would happen if any of the cell cycle genes were not present in the dataset?
> 2. How would we remove these genes from the table?
>
> > <solution-title></solution-title>
> >
> > 1. Any cell cycle genes that weren't in the dataset would have an empty field in the numbered column, which would be filled in with `FALSE` when we created the table with the {% tool Join %} tool. These rows would appear at the top of the table after it was sorted.
> > 2. We should check the first rows of the table for any unnumbered genes and then cut these rows out in the next step.
> >
> {: .solution}
{: .question}

## Create the annotation column

We now have a table with all the gene names in the same order as the main dataset and a column indicating which ones are cell cycle genes. If we cut this column out of the table, then we can add it as a new annotation to the main dataset. We'll also need to add a column header, which will be used as the key for this annotation in the AnnData dataset.

> <hands-on-title>Create the cell cycle annotation column</hands-on-title>
>
> 1. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy0) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `out_file1` (output of **Sort** {% icon tool %})
>    - *"Input data has:"*: `Unselect all`
>        - *"Type of table operation"*: `Drop, keep or duplicate rows and columns`
>            - *"List of columns to select"*: `3`
>    - *"Output formatting options"*: `Unselect all`
>
>
>    > <comment-title>Removing rows for missing genes </comment-title>
>    >
>    > If there were any cell cycle genes that weren't present in the main dataset, we could remove them at this stage by excluding them from the List of rows to select. As before, if we were using a dataset of a different size, we would need to change this parameter to include all the rest of the rows.
>    {: .comment}
>
>
> 2. Create a new **tabular** file from the following
>
>     ```
>    CC_genes
>     ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" %}
>
>
> 3. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: `Pasted Entry` dataset
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Select"*: `table` (output of **Table Compute** {% icon tool %})
>
{: .hands_on}


## Add an annotation to the AnnData

We will need to add the annotation to both the annotated dataset `CellCycle_Annotated` and to the one that we created by regressing out the cell cycle genes `CellCycle_Regressed`. This will allow us to plot the cell cycle genes before and after regression. We can do this using the {% tool Manipulate Anndata %} tool and selecting the correct function from the dropdown menu.

> <hands-on-title>Add the new annotations</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Annotated` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Function to manipulate the object"*: `Add new annotation(s) for observations or variables`
>        - {% icon param-file %} *"Table with new annotations"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>
>
> 2. Rename the output `CellCycle_Annotated_CC`
>
>
> 3. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed` (output of **Scanpy RegressOut** {% icon tool %})
>    - *"Function to manipulate the object"*: `Add new annotation(s) for observations or variables`
>        - {% icon param-file %} *"Table with new annotations"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>
>
> 4. Rename the output `CellCycle_Regressed_CC`
>
{: .hands_on}

## Filter the cell cycle genes

To demonstrate the power of cell cycle regression, we're going to reduce our expression matrices to contain only the 97 cell cycle genes. This will force our dimension reduction and plotting to be based entirely on cell cycle genes. You wouldn't do this during analysis, but for proof of principle, let's go for it!

> <hands-on-title>Filter the AnnData datasets</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Annotated_CC` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `CC_genes`
>            - *"Type of value to filter"*: `Boolean`
>
>
> 2. Rename the output `CellCycle_Annotated_CC_Only`
>
>
> 3. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed_CC` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `CC_genes`
>            - *"Type of value to filter"*: `Boolean`
>
>
> 4. Rename the output `CellCycle_Regressed_CC_Only`
>
{: .hands_on}

## Plot the cell cycle genes before regression

Now that we have a dataset that only includes the cell cycle genes, we can visualise their effects in a PCA plot. We first calculate the PCA coordinates, which are a measure of how similar each pair of cells is in terms of the expression of the 97 cell cycle genes we've included in the filtered dataset. We will then visualise the cells on a PCA plot where the axes represent the principal components, which reflect the genes (or groups of genes) that had the biggest impact in these calculations.

You will learn more about plotting your data in the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial. For now, it is enough to know that each dot on the plot represents a cell and the closer two cells are together, the more similar they are.

> <hands-on-title>Create a PCA Plot of cell cycle genes </hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Annotated_CC_Only` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'tl.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>
>    > <comment-title>Plot all the genes </comment-title>
>    >
>    > Make sure that you de-select the option for the {% tool Cluster, infer trajectories and embed %} tool to use highly variable genes only - some of the cell cycle genes are also HVGs, but we want our plots to include the cell cycle genes that aren't HVGs too.
>    {: .comment}
>
> 2. {% tool [Plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `PCA: Plot PCA results, using 'pl.pca_overview'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `phase`
>        - In *"Plot attributes"*:
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Does the plot look as you expected?
>
> > <solution-title></solution-title>
> >
> > 1. The PCA plot shows that the three groups of cells are separated out according to what phase of the cell cycle they are in. This is what we would expect to see as we are only looking at the cell cycle genes, which by definition are expressed during particular phases.
> >
> > ![PCA plot showing three separate clusters of cells in the G1, S and G2M Phases](../../images/scrna-case_cell-cycle/CellCycle_PCA1.png "PCA Plot of Cell Cycle Genes before regression")
> >
> {: .solution}
{: .question}

## Plot the cell cycle genes after regression

We will now repeat the same steps to create a PCA plot of the filtered dataset after the effects of the cell cycle have been regressed out.

> <hands-on-title>Recreate the PCA plot of cell cycle genes after regression </hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed_CC_Only` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'tl.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>
>
> 2. {% tool [Plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `PCA: Plot PCA results, using 'pl.pca_overview'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `phase`
>        - In *"Plot attributes"*:
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Does the plot look as you expected?
>
> > <solution-title></solution-title>
> >
> > 1. The cells in different phases are now all mixed up together. This makes sense because we are only plotting the cell cycle genes, but the previously strong effects of the cell cycle on these genes have now been regressed out. There are still some differences between the cells (they don't all end up at the same point on the PCA chart) because the regression only removes the expected effects of the cell cycle, leaving behind any individual variation in the expression of the cell cycle genes.
> >
> > ![PCA plot showing one big cluster with the cells from G1, S and G2M Phases all mixed up together](../../images/scrna-case_cell-cycle/CellCycle_PCA2.png "PCA Plot of Cell Cycle Genes after regression")
> >
> {: .solution}
{: .question}

Comparing the before and after plots, we can clearly see that the effects of the cell cycle have been removed. Although you wouldn't usually need to filter out the cell cycle genes or create these plots when analysing your own data, hopefully you have found doing it now to be helpful for understanding the impact of cell cycle regression.

> <question-title></question-title>
>
> 1. What impact do you think the cell cycle regression will have when you analyse the whole dataset? What would happen if we plotted all of the genes from the main dataset?
>
> > <solution-title></solution-title>
> >
> > 1. The regression reduces the impact of the cell cycle on the data - this is why the cells are less separated by phase afterwards. When we analyse the whole `CellCycle_Regressed` dataset, with all of the genes, this could allow other differences in gene expression to become more apparent.
> >
> > We wouldn't expect to see such clear distinctions in PCA plots created using all of the genes (not just the cell cycle ones), even before the regression. Although the cell cycle genes can have a significant effect, these won't be as obvious when other genes are also being taken into account. However, we will still see a difference after we regress out the effects of the cell cycle - the cells in different phases will become more mixed up together. How much of a difference the regression makes will depend on how strong the effects of the cell cycle are in a particular dataset - you can see the effects on this dataset below. You can also replicate these plots after completing the rest of the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial by colouring your PCA plots by phase.
> >
> > ![PCA plot showing some separation between cells in the G1, S and G2M Phases before regression](../../images/scrna-case_cell-cycle/CellCycle_PCA3.png "PCA Plot using all genes before regression")
> >
> >
> > ![PCA plot showing that cells in the G1, S and G2M Phases are more mixed up with each other after the regression](../../images/scrna-case_cell-cycle/CellCycle_PCA4.png "PCA Plot using all genes after regression")
> >
> {: .solution}
{: .question}

# Conclusion

In this tutorial, you have annotated and scored the cell cycle genes and regressed out the effects of the cell cycle. You have also created PCA plots of the data before and after regression to visualise the effects.

You might want to check your results against this [example history](https://humancellatlas.usegalaxy.eu/u/marisa_jl/h/removing-the-effects-of-the-cell-cycle---answer-key).

You can now continue to analyse this data by returning to the Preparing coordinates step in the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial. If you use the `CellCycle_Regressed` dataset (which you may now want to rename as `Use_me_Scaled` since that is the name used in the main tutorial), you should notice some differences in your results compared to those shown there because the effects of the cell cycle have been regressed out.

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Inspect and manipulate", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0", "Scanpy RegressOut", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_regress_variable/scanpy_regress_variable/1.8.1+galaxy0", "Concatenate multiple datasets or collections", "cat1", "Add column", "toolshed.g2.bx.psu.edu/repos/devteam/add_value/addValue/1.0.0", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1", "Table Compute", "toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy0", "Join", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2", "Sort", "sort1", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1", "Cluster, infer trajectories and embed", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0", "Plot", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy1"]


topic: single-cell
tutorial: scrna-preprocessing-tenx
title: Pre-processing of 10X Single-Cell RNA Datasets
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["10x"]
content_hash: 98fe5bc006878259ee95c38b12cc3849
last_modified: 2026-05-27T09:01:49.748033
zenodo_link: https://zenodo.org/record/3457880
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.html

content:
Single-cell RNA-seq analysis is a rapidly evolving field at the forefront of transcriptomic research, used in high-throughput developmental studies and rare transcript studies to examine cell heterogeneity within a populations of cells.
The cellular resolution and genome wide scope make it possible to draw new conclusions that are not otherwise possible with bulk RNA-seq. The analysis requires a great deal of knowledge about statistics, wet-lab protocols, and some machine learning due to variability and sparseness of the data. The uncertainty from the low coverage and low cell numbers per sample that once were common setbacks in the field are overcome by 10x Genomics which provides high-throughput solutions which are quickly championing the field.

## Era of 10x Genomics

10x genomics has provided not only a cost-effective high-throughput solution to understanding sample heterogeneity at the individual cell level, but has defined the standards of the field that many downstream analysis packages are now scrambling to accommodate.

![clusters]({% link topics/single-cell/images/scrna-pre-processing/tenx_clusters_intro.png %} "From less than 1K to over 10K with 10x genomics: Analyses of two separate scRNA datasets using the (left) CelSEQ2 protocol, and the (right) 10x Chromium system.")

The gain in resolution reduces the granularity and noise issues that plagued the field of scRNA-seq not long ago, where now individual clusters are much easier to decipher due to the added stability added by this gain in information.

## Library Preparation

![Library Preparation]({% link topics/single-cell/images/scrna-pre-processing/tenx_libprep.png %} "An overview of the library preparation")

The 10X barcoded gel beads consist of a pool barcodes which are used to separately index each cell's transcriptome. The individual gel barcodes are delivered to each cell via flow-cytometry, where each cell is fed single-file along a liquid tube and tagged with a 10X gel bead. The cells are then isolated from one another within thousands of nanoliter droplets, where each droplet described by a unique 10x barcode that all reads in that droplet are associated with once they undergo reverse-transcription (RT) which reconstructs the mRNA into a cDNA counterpart. The oil is then removed and all (now barcoded) cDNA reads are pooled together to be sequenced.

Though there are approximately 3 million 10x gel barcodes used, the amount actually qualitatively profiled in a sample is ~10,000 due to majority of droplets (>90%) being empty in order to ensure that the remainder contains only one cell.

> <details-title>Whitelist Barcodes</details-title>
>
> There are actually two sets of barcodes for the different chemistries provided; one which has 737,000 barcodes, and one with ~3,7 million barcodes.
>
> Both are provided in the Zenodo link, but we will only work with the 3 million barcodes because this is what is provided with the chemistry version.
>
> If you are getting a low number of cells detected for your reads later on, it could be because the barcodes provided are wrong.
>
{: .details}



More information can be found in the [reagent kit documentation](https://support.10xgenomics.com/single-cell-gene-expression/library-prep/doc/user-guide-chromium-single-cell-3-reagent-kits-user-guide-v2-chemistry).

<!--TODO:
 * Discuss droplet-seq based workflows
 * What is Chromium, and do users need to know what it is?-->


# Analysis Strategy


The tutorial is structured into two parts:

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

![Overview of workflow]({% link topics/single-cell/images/scrna-pre-processing/tenx_workflow.png %} "An overview of the workflow")

The first part of this tutorial is essentially a one-click "fire and forget" solution to demultiplexing and quantifying scRNA-seq data, where much of the complexity required in this extremely crucial stage is simplified into a single step.

However, those who are more interested in learning the intricacies of how FASTQ files are transformed into a count matrix, please see the [Pre-processing of Single-Cell RNA Data]({% link topics/single-cell/tutorials/scrna-preprocessing/tutorial.md %}) tutorial.

10x Genomics has its own processing pipeline, [Cell Ranger](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) to process the scRNA-seq outputs it produces, but this process requires much configuration to run and is significantly slower than other mappers.

Since STARsolo is a drop-in solution to the *Cell Ranger* pipeline, the first part of the tutorial is a one-click solution where users are encouraged to launch their **RNA STARsolo** jobs and spend the time familiarising themselves with the pre-processing training materials mentioned above.


> <details-title>Benchmark of Cell Ranger to others</details-title>
>
> ![tenx_runtimes]({% link topics/single-cell/images/scrna-pre-processing/tenx_runtimes.jpg %} "Benchmark of different mapping software.")
>
> The image is from {% cite Melsted2019 %}. Notice the order of magnitude speed up that STARsolo and a few others display, for a variety of different datasets in comparison to Cell Ranger.
{: .details}

The second part of this tutorial also has a one-click solution to producing a matrix identical to that given by the *Cell Ranger* pipeline, but the more interesting aspects of the pipeline are explored in the *Introspective Method* part of the tutorial.


# Producing a Count Matrix from FASTQ

Here we will use the [1k PBMCs from a Healthy Donor (v3 chemistry)](https://support.10xgenomics.com/single-cell-gene-expression/datasets/3.0.0/pbmc_1k_v3) from 10x genomics, consisting of 1000 Peripheral blood mononuclear cells (PBMCs) extracted from a healthy donor, where PBMCs are primary cells with relatively small amounts of RNA (~1pg RNA/cell).

The source material consists of 6 FASTQ files split into two sequencing lanes *L001* and *L002*, each with three reads of **R1** (barcodes), **R2** (cDNA sequences), **I1** (illumina lane info):
  * pbmc\_1k\_v3\_S1\_*L001*\_**R1**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L001*\_**R2**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L001*\_**I1**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L002*\_**R1**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L002*\_**R2**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L002*\_**I1**\_001.fastq.gz

The *Cell Ranger* pipeline requires all three files to perform the demultiplexing and quantification, but **RNA STARsolo** does [not require](https://github.com/alexdobin/STAR/issues/640) the I1 lane file to perform the analysis. These source files are provided in the [Zenodo](https://zenodo.org/record/3457880) data repository, but they require approximately 2 hours to process. For this tutorial, we will use datasets sub-sampled from the source files to contain approximately 300 cells instead of 1000. Details of this sub-sampling process can be viewed at the [Zenodo link](https://zenodo.org/record/3457880/files/subsetting_data.txt).

## Data upload and organization

For the mapping, we require the sub-sampled source files, as well as a "whitelist" of (~3,7 million) known cell barcodes, [freely extracted](https://kb.10xgenomics.com/hc/en-us/articles/115004506263-What-is-a-barcode-whitelist) from the *Cell Ranger* pipeline. This whitelist file may be found within the Galaxy Data Library, but it is included here in the Zenodo record for convenience and also because the sequencing facility may not always provide this file.

The barcodes in the R1 FASTQ data are checked against these known cell barcodes in order assign a specific read to a specific known cell. The barcodes are designed in such a manner that there is virtually no chance that they will align to a place in the reference genome. In this tutorial we will be using hg19 (GRCh37) version of the human genome, and will therefore also need to use a hg19 GTF file to annotate our reads.


> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and rename it (e.g. scRNA-seq 10X dataset tutorial)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 1. Import the sub-sampled FASTQ data from [`Zenodo`](https://zenodo.org/record/3457880) or from the data library (ask your instructor)
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    ```
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L001_R1_001.fastq.gz
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L001_R2_001.fastq.gz
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L002_R1_001.fastq.gz
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L002_R2_001.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Import the Gene Annotations and Cell Barcodes from [`Zenodo`](https://zenodo.org/record/3457880) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3457880/files/Homo_sapiens.GRCh37.75.gtf
>    https://zenodo.org/record/3457880/files/3M-february-2018.txt.gz
>    ```
>
{: .hands_on}


## 10x Chemistries

There are two main reagent kits used during the library preparation, and the choice of one will influence the size of the sequences we work with. Below we can see the layout of the primers used in both chemistries. We can ignore most of these as they are not relevant, namely: the P5 and P7 illumina primers are used in the illumina [bridge amplification](https://en.wikipedia.org/wiki/Illumina_dye_sequencing#Bridge_amplification) process; the Sample Index is an 8bp primer which is related to the Chromium system that balances nucleotide bias and ensures that there is no sample overlap during the multiplexed sequencing; and the Poly(dT) VN primer used to capture RNA sequences with poly-A tails (i.e. mRNA).

![chem]({% link topics/single-cell/images/scrna-pre-processing/tenx_primers.svg %} "10x Chromiumv2 and Chromiumv3 Chemistries")

The primers of interest to us are the Cell Barcode (CB) and the Unique Molecular Identifiers (UMI) used in the Read 1 sequencing primer, as they describe to us how to demultiplex and deduplicate our reads. It is highly advised that the [Plates, Batches, and Barcodes]({% link topics/single-cell/tutorials/scrna-preprocessing/tutorial.md %}) slides are revisited to refresh your mind on these concepts.


| Chemistry | Read 2 | Read 1 (CB + UMI) | Insert (Read 2 + Read 1) |
|:---------:|:------:|:-----------------:|:------------------------:|
| v2        |  98    | 26 (16 + 10)      | 124 |
| v3        |  91    | 28 (16 + 12)      | 119 |

The table above gives a summary of the primers used in the image and the number of basepairs occupied by each.

> <details-title>Strandedness</details-title>
>
> Unstranded protocols do not distinguish between whether a fragment was sequenced from the forward or the reverse strand, which can lead to some ambiguity if the fragment overlaps two transcripts. In the image below, it is not clear whether the fragment is derived from GeneF or GeneR due to this overlap.
>
> ![strandedness]({% link topics/single-cell/images/scrna-pre-processing/tenx_strandedness.svg %} "Mapping fragments to overlapping transcripts is ambiguous with unstranded protocols.")
>
> Stranded protocols overcome this by fitting different 5' and 3' primers and adaptors, meaning that the orientation of the fragment is fixed during sequencing. For the Chromium v2 and v3 chemistries, the barcode information is purely within the R1 forward strand.
>
{: .details}



> <question-title></question-title>
>
> 1. What has stayed constant between the chemistry versions?
> 1. What advantage does this constant factor give?
> 1. What do UMIs do?
> 1. What advantage does the 2 extra bp in the v3 UMIs have over v2 UMIs?
> 1. What will be the strandedness of the generated library
>
> > <solution-title></solution-title>
> >
> > 1. The Cell Barcode (CB) has remained at 16bp for both chemistries.
> > 1. This has the advantage that the same set of barcodes can be used in both chemistries, which is important because barcodes are *very* hard to design.
> >    * They need to be designed in such a way to minimise accidentally aligning to the reference they were prepared to be used for.
> >    * Longer barcodes tend to be more unique, so this is a problem that is being solved as the barcodes increase in size, allowing for barcodes that can be used on more than one reference to be more common, as seen above.
> > 1. UMIs (or Unique Molecular identifiers) do not delineate cells as Cell Barcodes do, but instead serve as random 'salt' that tag molecules randomly and are used to mitigate amplification bias by deduplicating any two reads that map to the same position with the same UMI, where the chance of this happening will be astronomically small unless one read is a direct amplicon of the other.
> > 1. $$4^{10} = 1,048,576$$ unique molecules tagged, vs. $$4^{12} = 16,777,216$$ unique molecules tagged. The reality is much much smaller due to edit distances being used that would reduce both these numbers substantially (as seen in the [*Plates, Batches, and Barcodes*]({% link topics/single-cell/tutorials/scrna-plates-batches-barcodes/slides.html %}) slides), but the scale factor of 16 times more molecules ($$4^{12-10} = 16$$) can be uniquely tagged is true.
> > 1. Forward.
> {: .solution}
{: .question}

The differences in the chemistries is a slight change in the library size, where the v2 aims to capture *on average* 50,000 reads per cell, whereas the v3 aims to capture *at minimum* 20,000 reads per cell. This greatly reduces the lower-tail of the library size compared to the previous version.


### Determining what Chemistry our Data Contains

To perform the demultiplexing, we need to tell **RNA STARsolo** where to look in the R1 FASTQ to find the cell barcodes. We can do this by simply counting the number of basepairs in any read of the R1 files.

> <question-title></question-title>
> Peek at one of the R1 FASTQ files using the {% icon galaxy-eye %} symbol below the dataset name.
>
> 1. How many basepairs are there in any given read?
> 2. Which library preparation chemistry version was this read generated from?
>
> > <solution-title></solution-title>
> > 1. There are 28 basepairs.
> > 2. The v2 has 26 basepairs, but the v3 has 28 basepairs. Therefore the reads we have here use the Chromium v3 chemistry.
> {: .solution}
{: .question}



## Performing the Demultiplexing and Quantification

We will now proceed to demultiplex, map, and quantify both sets of reads using the correct chemistry discovered in the previous sub-section.

> <comment-title></comment-title>
>
> {% tool [RNA STARsolo](toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.10b+galaxy3) %} consumes a large amount of memory. During the Smörgåsbord training please use `Human (Homo Sapiens): hg19 chrX` as the reference genome if you follow this tutorial on [usegalaxy.org](https://usegalaxy.org). This performs the mapping only against chromosome X. The full output dataset is available at [zenodo](https://zenodo.org/record/3581213/files/matrix.mtx) and will be the starting point for the next tutorial.
{: .comment}

> <hands-on-title></hands-on-title>
>
> {% tool [RNA STARsolo](toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.11a+galaxy1) %}  with the following parameters:
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>        - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>            - *"Select reference genome"*: `Human (Homo Sapiens): hg19 Full` or `Human (Homo Sapiens) (b37): hg19`
>            - *"Gene model (gff3,gtf) file for splice junctions"*: `Homo_sapiens.GRCh37.75.gtf`
>            - *"Length of genomic sequence around annotated junctions"*: `100`
>    - *"Type of single-cell RNA-seq"*: `Drop-seq or 10X Chromium`
>        - *"Input Type"*: `Separate barcode and cDNA reads`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, Barcode reads"*: Multi-select `L001_R1_001` and `L002_R1_001` using the Ctrl key.
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, cDNA reads"*: Multi-select `L001_R2_001` and `L002_R2_001` using the Ctrl key.
>        - {% icon param-file %} *"RNA-Seq Cell Barcode Whitelist"*: `3M-february-2018.txt.gz`
>        - *"Configure Chemistry Options"*: `Chromium chemistry v3`
>        - *"UMI deduplication (collapsing) algorithm"*: `CellRanger2-4 algorithm`
>        - *"Type of UMI filtering"*: `Remove UMIs with N and homopolymers (similar to CellRanger 2.2.0)`
>        - *"Matching the Cell Barcodes to the WhiteList"*: `Multiple matches (CellRanger 2, 1MM_multi)`
>    - Under *"Advanced Settings"*:
>        - *"Strandedness of Library"*: `Read strand same as the original RNA molecule`
>        - *"Collect UMI counts for these genomic features"*: `Gene: Count reads matching the Gene Transcript`
>        - *"Cell filter type and parameters"*: `Do not filter`
>        - *"Field 3 in the Genes output"*: `Gene Expression`
>
>    > <comment-title></comment-title>
>    >
>    > The in-built Cell filtering is a relatively new feature that emulates the CellRanger pipeline. Here, we set the filtering options to not filter because we will use our own methods to better understand how this works. For your own future datasets, you may wish to enable the filtering parameter.
>    {: .comment}
>
{: .hands_on}

## Inspecting the Output Files

At this stage **RNA STARsolo** has output 6 files; a program log, a mapping quality file, a BAM file of alignments, and 3 count matrix files:
 1. Log
 1. Feature Statistic Summaries
 1. Alignments
 1. Matrix Gene Counts
 1. Barcodes
 1. Genes

The log and the summaries files give program logistics and metrics about the quality of the mapping. The BAM file contains the reads mapped to the reference genome. The matrix gene counts is the count matrix in *matrixmarket* format, accompanied by the list of genes and cell barcodes in separate files. These can be converted into *tabular* or *AnnData* formats using the {% icon tool %} **Import Anndata and loom** tool.


### Mapping Quality

Let us investigate the output log. This type of quality control is essential in any RNA-based analysis and it is strongly recommended that you familiarise yourself with the [Quality Control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) tutorial.


> <hands-on-title></hands-on-title>
>
> {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy0) %} with the following parameters:
>    - In *"Results"*:
>      - In *"1:Results"*:
>        - *"Which tool was used generate logs?"*: `STAR`
>        - In *"STAR output"*:
>          - In *"1:STAR output"*:
>            - *"Type of STAR output?"*: `Log`
>            - {% icon param-file %} *"STAR log output"*: `RNA STARsolo: log`
{: .hands_on}


> <question-title></question-title>
>
> What percentage of reads are uniquely mapped?
>
> > <solution-title></solution-title>
> > 87.5%
> > - This is good, and is expected of 10x datasets.
> >
> {: .solution}
{: .question}

### Quantification Quality

The above tool provides a nice visualisation of the output log, and is a fantastic way to combine multiple quality sources into one concise report.

However, sometimes it is often more informative to look directly at the source quality control file. For example, let us investigate the STARsolo Feature summaries file.

We can look at this directly by clicking on the {% icon galaxy-eye %} symbol of the *Feature Statistic Summaries* file.


> <comment-title>RNA STARsolo log output</comment-title>
> ```
> Barcodes:
>                                       noNoAdapter              0
>                                           noNoUMI              0
>                                            noNoCB              0
>                                           noNinCB              0
>                                          noNinUMI            358
>                                  noUMIhomopolymer            707
>                                       noNoWLmatch          50037
>                                       noTooManyMM              0
>                                noTooManyWLmatches              0
>                                   yesWLmatchExact        7530489
>                               yesOneWLmatchWithMM          19534
>                              yesMultWLmatchWithMM          84800
> Genes:
>                                        noUnmapped         278029
>                                       noNoFeature        3060392
>                                      MultiFeature         419262
>                       subMultiFeatureMultiGenomic         333163
>                                noTooManyWLmatches          28645
>                              noMMtoWLwithoutExact           7148
>                                        yesWLmatch        3841347
>                                yessubWLmatchExact        3829826
>                       yessubWLmatch_UniqueFeature        3841347
>                                   yesCellBarcodes           5200
>                                           yesUMIs        1689978
>
> ```
{: .comment}

The explanation of these parameters can be seen in the [RNA STAR Manual](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf) under the *STARsolo* section, with most of the information given in the details box below.

> <details-title>Explanation of parameters</details-title>
>
>  | Parameter | Explanation |
>  |-----------|-------------|
>  | noNinCB | number of reads with more than 2 Ns in cell barcode (CB) |
>  | noUMIhomopolymer | number of reads with homopolymer in CB |
>  | noTooManyWLmatches | (not used at the moment) |
>  | noNoWLmatch | number of reads with CBs that do not match whitelist even with one mismatch |
>
> All of the above reads are discarded from Solo output.
>
> The remaining reads are checked for any overlap with the features/genes:
>
>
>  | Parameter | Explanation |
>  |-----------|-------------|
>  | noUnmapped | number of reads unmapped to the genome |
>  | noNoFeature | number of reads that map to the genome but do not belong to a feature |
>  | MultiFeature | number of reads that belong to more than one feature |
>  | subMultiFeatureMultiGenomic | number of reads that belong to more than one feature and are also multimapping to the genome (this is a subset of the `nAmbigFeature`) |
>  | noTooManyWLmatches | number of reads with ambiguous CB (i.e. CB matches whitelist with one mismatch but with posterior probability 0.95) |
>  | noMMtoWLwithoutExact | number of reads with CB that matches a whitelist barcode with 1 mismatch, but this whitelist barcode does not get any other reads with exact matches of CB |
>
> These metrics can be grouped into more broad categories:
>
> * `noNinCB` + `noUMIhomopolymer` + `noNoWLmatch` + `noTooManyWLmatches` + `noMMtoWLwithoutExact` = number of reads with CBs that do not match whitelist.
> * `noUnmapped` + `MultiFeature` = number of reads without defined feature (gene).
> * `yessubWLmatch_UniqueFeature` = number of reads that are output as solo counts.
>
> The three categories above summed together should be equal to the total number of reads (which is also given in the MultiQC output).
>
{: .details}


The main information to gather at this stage is that the `yesCellBarcodes` tell us how many cells were detected in our sample, where we see that there are 5200 cells. Another metric to take into account is that the number of matches (`yessubWLmatch_UniqueFeature`) has the largest value, and that the number of reads that map to the genome but not to a feature/gene given in the GTF (`noNoFeature`) is not too large. The number of no features is also quite high when mapping the original (non-subsampled) 10x input datasets, so this appears to be the default expected behaviour.


# Producing a Quality Count Matrix

The matrix files produced by **RNA STARsolo** are in the *bundled* format, meaning that the information to create a tabular matrix of Genes vs Cells are separated into different files. These files are already 10x analysis datasets, compatible with any downstream single-cell RNA analysis pipeline, however the number of cells represented here are greatly over-represented, as they have not yet been filtered for high quality cells, and therefore the matrix represents *any* cells that were unambiguously detected in the sample.

If we were to construct a cell matrix using the data we have, we would have a large matrix of 60,000 Genes against 3 million Cells, of which most values would be zero, i.e. an *extremely* sparse matrix.

To get a high quality count matrix we must apply the **DropletUtils** tool, which will produce a filtered dataset that is more representative of the *Cell Ranger* pipeline.

## Cell Ranger Method

> <hands-on-title>Default Method</hands-on-title>
>
> {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %}  with the following parameters:
>    - *"Format for the input matrix"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - {% icon param-file %} *"Count Data"*: `Matrix Gene Counts` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Genes List"*: `Genes` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Barcodes List"*: `Barcodes` (output of **RNA STARsolo** {% icon tool %})
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `DefaultDrops`
>            - *"Expected Number of Cells"*: `3000`
>            - *"Upper Quantile"*: `0.99`
>            - *"Lower Proportion"*: `0.1`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>
> > <comment-title>Default Parameter</comment-title>
> >
> > The *"Expected Number of Cells"* parameter is the number of cells you expect to see in your sample, but does not correspond to how many cells you expect to see in a sub-sampled dataset like this one. The default is 3000 for *Cell Ranger*, which will yield ~300 cells here, but users are encouraged to experiment with this value when dealing with their own data, to recover the desired number of cells.
> >
> {: .comment}
{: .hands_on}


> <question-title></question-title>
>
> 1. How many cells were detected?
> 1. Does this agree with the STARsolo Feature Statistics output?
>
> > <solution-title></solution-title>
> >
> > 1. By clicking on the title of the output dataset in the history we can expand the box to see the output says that there are `272` cells in the output table.
> > 1. If we expand the {% icon galaxy-eye %} *RNA STARsolo Feature Statistic Summaries* Dataset and look at the `yesCellBarcodes` value, we see that *RNA STARsolo* detected `5200` cells. What this means is that *5200* barcodes were detected in total, but only *272* of them were above an acceptable threshold of quality, based on the default upper quantile and lower proportion parameters given in the tool. Later on, we will actually later visualise these thresholds ourselves by "ranking" the barcodes, to see the dividing line between high and low quality barcodes.
> {: .solution}
{: .question}

This will produce a count matrix in a human readable tabular format which can be used in downstream analysis tools, such as those used in the [*Downstream Single-cell RNA analysis with RaceID*]({% link topics/single-cell/tutorials/scrna-raceid/tutorial.md %}) tutorial.

> <details-title>File Formats in scRNA-seq</details-title>
>
> The tabular outputs here are selected mostly for transparency, since they are easy to inspect. However, the datasets produced with 10x data are very large and very sparse, meaning there is much data redundancy due to the repetition of zeroes everywhere in the data.
>
> [Many](https://bioconductor.org/packages/release/bioc/html/SingleCellExperiment.html) analysis [packages](https://www.rdocumentation.org/packages/RaceID/versions/0.1.3/topics/SCseq) have [attempted](https://github.com/satijalab/seurat/wiki/Seurat) to [solve](https://www.rdocumentation.org/packages/scater/versions/1.0.4/topics/SCESet) this problem by inventing their own standard, which has led to the proliferation of many different "standards" in the scRNA-seq package ecosystem, most of which require an R programming environment to inspect.
>
> As expected, the format that usually wins is the one which is most common in the field. In this case, the format *also* happens to be a very good format that stores data in a concise, compressed, and extremely readable manner:
>
> ![anddata]({% link topics/single-cell/images/scrna-pre-processing/tenx_anndata.svg %} "AnnData is an HDF5-based format, which stores gene and cell information in their own matrices, complementary to the main data matrix." )
>
> The *AnnData* format (`hda5`) is an extension of the [HDF5 format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format), which supports multidimensional datasets to be stored in a consistent and space-optimised way. This is the default output of *Cell Ranger* and so is also the default output of **RNA STARsolo**. The format is also now a widely accepted format in many downstream analysis suites.
{: .details}



## Introspective Method

The *DefaultDrops* method given in the previous sub-section is a good one-click solution to emulating the *Cell Ranger* process of producing a high quality count matrix.

However, the **DropletUtils** tool does provide other options for determining which cells are of "good" quality and which are not.

A useful diagnostic for droplet-based data is the barcode rank plot, which shows the (log-)total UMI count for each barcode on the y-axis and the (log-)rank on the x-axis. This is effectively a transposed empirical cumulative density plot with log-transformed axes. It is useful as it allows users to examine the distribution of total counts across barcodes, focusing on those with the largest counts.

> <hands-on-title>Rank Barcodes</hands-on-title>
>
> {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %}  with the following parameters:
>    - *"Format for the input matrix"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - {% icon param-file %} *"Count Data"*: `Matrix Gene Counts` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Genes List"*: `Genes` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Barcodes List"*: `Barcodes` (output of **RNA STARsolo** {% icon tool %})
>    - *"Operation"*: `Rank Barcodes`
>        - *"Lower Bound"*: `100`
>
{: .hands_on}

![knee]({% link topics/single-cell/images/scrna-pre-processing/tenx_knee.png %} "Barcode Ranks: The separating thresholds of high and low quality cells")

The knee and inflection points on the curve mark the transition between two components of the total count distribution. This is assumed to represent the difference between cells with high RNA content and empty droplets with little RNA, and gives us a rough idea of how many cells to expect in our sample.

> ### Question {% icon question %} Questions
>
> 1. How many presumably real cells can we expect to see in our sample based on the above plot?
> 1. What is the number of cells that we can expect to see with a cut-off detected at inflection line?
>
> > <solution-title></solution-title>
> > 1. We see the blue knee line cross the threshold of barcodes at just below than the 10000 Rank on the horizontal log scale, which is shown in the expanded view of our data as `"knee = 4861"`. This line intersects with the ranked cells near 100 on x-axis. We can expect 100-200 cells with high RNA content.
> > 1. This threshold is given by the inflection line, which is given at `"inflection = 260"`. The vertical drop in the ranked cells at the inflection line is between 100 and 10000 on the x-axis. The axis is log scaled it is close to 100. Hence, we can expect between 200 and 400 cells.
> {: .solution}
{: .question}

On large 10x datasets we can use these thresholds as metrics to utilise in our own custom filtering, which is once again provided by the **DropletUtils** tool. In this case, we will use a bit less stringent threshold of 200 (instead of 260 at inflection) and use a FDR threshold of 0.01 so that our detected cells contain only 1% of false positives.

> <hands-on-title>Custom Filtering</hands-on-title>
>
> {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %}  with the following parameters:
>    - *"Format for the input matrix"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - {% icon param-file %} *"Count Data"*: `Matrix Gene Counts` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Genes List"*: `Genes` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Barcodes List"*: `Barcodes` (output of **RNA STARsolo** {% icon tool %})
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `EmptyDrops`
>            - *"Lower-bound Threshold"*: `200`
>            - *"FDR Threshold"*: `0.01`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>
{: .hands_on}

![cells]({% link topics/single-cell/images/scrna-pre-processing/tenx_cells.png %} "Detected Cells (red)")


Here we recover 279 high quality cells instead of the 272 detected via the default method previously. On large datasets, this difference can help clean downstream clustering. For example, soft or less well-defined clusters are derived from too much noise in the data due to too many low quality cells being in the data during the clustering. Filtering these out during the pre-processing would produce much better separation, albeit at the cost of having less cells to cluster. This filter-cluster trade-off is discussed in more detail in the downstream analysis training materials.


# Conclusion


In this workflow we have learned to quickly perform mapping and quantification of scRNA-seq FASTQ data in a single step via **RNA STARsolo**, and have reproduced a *Cell Ranger* workflow using the **DropletUtils** suite, where we further explored the use of barcode rankings to determine better filtering thresholds to generate a high quality count matrix.

A full pipeline which produces both an AnnData and tabular file for inspection is provided [in this workflow](workflows/scRNA-seq-Preprocessing-TenX.ga)

Note that, since version *2.7.7a* of the tool, the entire *Cell Ranger* pipeline including the filtering can be performed natively within **RNA STARsolo**. As this is still a relatively new feature, we do not use it here in this tutorial, but eager users are encouraged to try it out.

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %}).

<!-- ![Recap of workflow]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow.svg %} "A recap of the entire workflow") -->

tools_json:
["RNA STARsolo", "toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.10b+galaxy3", "toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.11a+galaxy1", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy0", "DropletUtils", "toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2"]


