# tutorials

topic: single-cell
tutorial: scrna-seurat-pbmc3k
title: Clustering 3K PBMCs with Seurat
difficulty: introductory
requirements_json: []
tags_json: []
content_hash: 4c906fb9e72008bf7f83da8b678cff61
last_modified: 2026-05-27T09:01:49.751020
zenodo_link: https://zenodo.org/record/3581213
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-seurat-pbmc3k/tutorial.html

content:
separate-preprocessing---answer-key
  date: 2023-10-28
- label: UseGalaxy.eu - Using SCTransform
  history: https://usegalaxy.eu/u/marisa_jl/h/clustering-3k-pbmcs-with-seurat---sctransform---answer-ke
  date: 2023-10-28
- label: UseGalaxy.org.au - Using Separate Preprocessing Tools
  history: https://usegalaxy.org.au/u/videmp/h/gtn-clustering-3k-pbmcs-with-seurat-separate-preprocessing-may-2025
  date: 2025-05-02
- label: UseGalaxy.org.au - Using SCTransform
  history: https://usegalaxy.org.au/u/videmp/h/gtn-cluster-3k-pbmcs-with-seurat-sctransform-may-2025
  date: 2025-05-02
questions:
- How can we identify cell types in single cell RNA-Seq data?
- What are the steps for clustering single cell data with Seurat?
objectives:
- Explain the steps involved in clustering single cell data
- Evaluate the quality of single cell data and filter out low quality cells
- Prepare single cell data for analysis with Seurat
- Perform clustering with Seurat
- Be ready to apply the Seurat pipeline to new datasets
time_estimation: 8H
key_points:
- Seurat is a commonly-used pipeline for single cell data analysis
- Clustering makes single cell datasets easier for us to understand
- Different tools and parameters should be considered when analysing different datasets
requirements:
- type: internal
  topic_name: single-cell
  tutorials:
  - scrna-preprocessing
  - scrna-preprocessing-tenx
follow_up_training:
- type: internal
  topic_name: single-cell
  tutorials:
  - EBI-retrieval
tags:
- 10x
contributions:
  authorship:
  - MarisaJL
  editing:
  - pavanvidem
  - shiltemann
recordings:
- youtube_id: -tIOSBXeXPs
  length: 2H11M
  galaxy_version: 24.2.4.dev0
  date: '2025-04-15'
  speakers:
  - MarisaJL
  captioners:
  - MarisaJL
  bot-timestamp: 1744754154


---


Single cell RNA-seq analysis enables us to explore differences in gene expression between cells. It can reveal the heterogenity within cell populations and help us to identify cell types that could play roles in development, disease, or other processes. Single cell omics is a relatively young field, but there are a few commonly-used analysis pipelines that you will often see in the literature. In this tutorial, we will use one of these pipelines, Seurat, to cluster single cell data from a 10X Genomics experiment ({% cite seurat2023v5 %}). You can follow the same analysis using the Scanpy pipeline in the [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial.

Clustering is typically the first type of analysis we will perform on a single cell dataset. It groups together cells that are expressing similar genes, which makes the data easier to understand and often helps us to identify specific cell types.

> <comment-title></comment-title>
>
> This tutorial is based on the [Seurat - Guided Clustering Tutorial](https://satijalab.org/seurat/articles/pbmc3k_tutorial). The SCTransform sections also draw from the [Using sctransform in Seurat](https://satijalab.org/seurat/articles/sctransform_vignette.html) tutorial.
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Important tips for easier analysis

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

# Data

For this tutorial, we will analyze a dataset of Peripheral Blood Mononuclear Cells (PBMC) extracted from a healthy donor, which is freely available from 10X Genomics. The dataset contains 2700 single cells sequenced using Illumina NextSeq 500. The raw sequences have been processed by the [**cellranger**](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) pipeline from 10X to extract a unique molecular identifier (UMI) count matrix, in a similar way to that explained in the [Pre-processing of 10X Single-Cell RNA Datasets]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}) tutorial.

In this matrix, the values represent the number of each feature (i.e. gene; row) detected in each cell (column). Single cell matrices can be quite large: here there are 2700 columns with 32,738 rows, so for each of our 2700 cells we know how many times we found RNAs matching each of the 32,738 genes. Since most of these genes weren't detected in most of the cells, the matrix is largely filled with zeros, i.e. it is an extremely sparse matrix. To optimize the storage of such a table and the information about the genes and cells, **cellranger** creates 3 files:

- `genes.tsv`: a tabular file with information about the 32,738 genes in 2 columns (Ensembl gene id and the gene symbol)
- `barcodes.tsv`: a tabular file with the barcode for each of the 2700 cells
- `matrix.mtx`: a condensed version of the count matrix (including the non-zero values only)

    The count matrix is represented by its non-zero values - we don't need to store all of those zeroes as long as we know where our non-zero values are in the matrix. Each non-zero value is represented by its line number (1st column), its column number (2nd column) and its value (3rd column). The first row gives the total number of rows (genes), columns (cells) and non-zero values. More information on the Matrix Market Exchange (mtx) format can be found [in this documentation](https://math.nist.gov/MatrixMarket/formats.html)

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the `genes.tsv`, `barcodes.tsv` and `matrix.mtx` from [Zenodo]({{ page.zenodo_link }}) or from the shared data library
>
>    ```
>    {{ page.zenodo_link }}/files/genes.tsv
>    {{ page.zenodo_link }}/files/barcodes.tsv
>    {{ page.zenodo_link }}/files/matrix.mtx
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets as `genes`, `barcodes`, and `matrix` if necessary
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Check the datatypes are correct - the `genes` and `barcodes` files should be tsv or tabular while the `matrix` should be an mtx file
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 5. Inspect the `matrix` file by clicking on the {% icon galaxy-eye %} icon
>
>    {% snippet faqs/galaxy/histories_dataset_item.md %}
>
{: .hands_on}

The beginning of the file should look like this:

> <question-title></question-title>
>
> ```
> 32738	2700	2286884
> 32709	1	4
> 32707	1	1
> 32706	1	10
> 32704	1	1
> ```
>
> 1. How many non-zero values are in the matrix?
> 2. How many counts were found for the 32,706th gene in the 1st cell?
>
> > <solution-title></solution-title>
> >
> > 1. The first row tells us there are 2,286,884 non-zero values for the 32,738 genes (rows) and 2,700 cells (columns) - so only 2.6% of the 88,392,600 potential values we could have in this matrix are non-zero. Getting rid of all those zeros has made the matrix much more compact.
> > 2. 10 counts were found for the 32,706th row (gene) and 1st column (cell), so we collected 10 RNAs that the first cell had produced from this particular gene.
> >
> {: .solution}
>
{: .question}

Representing the matrix with these three files is convenient for sharing the data, but not for processing them. Different single cell analysis packages have attempted to solve the problem of storage and analysis by inventing their own formats, which has led to the proliferation of many different 'standards' in the scRNA-seq package ecosystem.

## SeuratObject

In order to analyse the data using Seurat, we will first need to create a SeuratObject. A SeuratObject can store all our data, including the gene names, cell barcodes and matrix in a single RDS file. Since Seurat was written in the R programming language, SeuratObjects are saved as RData or RDS files. An RDS file is simply a type of RData file that contains a single object.

SeuratObjects can also hold any metadata we might have about our cells or genes, as well as the information we will produce during our analysis, such as our dimensional reductions and cell clusters. A SeuratObject can hold data in multiple layers (also known as slots in earlier version of Seurat). We can have layers of raw counts, normalised data, and scaled data. We could also store multiple assays or types of data in a single object, although in this tutorial we will only be using one assay for RNA-seq data. The SeuratObject is designed to be self-contained so that we don't have to work with lots of different files, even if we have different versions or types of data.

Creating a SeuratObject in R would require two steps - first, we would need to read in our data, in this case using the `Read10X` function, then secondly we would turn it into a SeuratObject using the `CreateSeuratObject` function. On Galaxy, we can perform both steps with a single tool. The `CreateSeuratObject` function also generates some QC metrics and performs basic filtering of the data.

> <hands-on-title>Create a SeuratObject</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Create Seurat Object`
>        - *"Select format of input"*: `matrix market (for e.g. 10x data)`
>            - {% icon param-file %} *"Counts matrix with features as rows, cells as columns (.mtx)"*: `matrix.mtx`
>            - {% icon param-file %} *"List of gene names (for rows)"*: `genes.tsv`
>            - {% icon param-file %} *"List of cell barcodes (for columns)"*: `barcodes.tsv`
>        - *"Include features detected in at least this many cells"*: `3`
>        - *"Include cells where at least this many features are detected"*: `200`
>        - *"Calculate percentage of mito genes in each cell"*: `No`
>
> 2. Rename the generated file to `Input 3k PBMC`
>
> 3. Check that the format is `rds`
{: .hands_on}

We can't look at the RDS file directly as it is designed for computers to read, rather than humans, but the Seurat tools will now be able to interact with the data. We can also use the {% tool Seurat Data Management %} tool to look inside our new SeuratObject.

> <hands-on-title>Inspect the `Input 3k PBMC` SeuratObject</hands-on-title>
>
> 1. {% tool [Seurat Data Management](toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0) %} with the following parameters:
>    - *"Method used"*: `Inspect Seurat Object`
>        - *"Display information about"*: `General`
>
> 2. Click the {% icon galaxy-eye %} on the output in your history to see some information about your SeuratObject
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many cells and genes are in the `Input 3k PBMC` SeuratObject we just created?
> 2. Is this the same as the number of cells and genes we started out with?
>
> > <solution-title></solution-title>
> > 1. If we click on the {% icon galaxy-eye %} of the output from {% tool Seurat Data Management %} we can see some general information about our SeuratObject. We can see that it contains information about 2700 samples (cells) and 13,714 features (genes).
> > 2. When we looked at the `matrix.mtx` file previously, we saw that it contained information on the same 2700 cells, but that there were 32,738 genes with non-zero values in the matrix. That means we are missing 19,024 genes - but don't worry, this is supposed to happen! Remember that we set the `Include features detected in at least this many cells` parameter to `3` - this got rid of any genes that were found in fewer than three cells. We assume that these genes are just technical noise (e.g. misreads, reads mapped to the wrong gene) rather than real biological features. We also set the `Include cells where at least this many features are detected` to `200` - this would have removed any cells that had fewer than 200 genes found in them. In this case, we didn't remove any cells as all 2700 of them must have had at least 200 genes. If we did have cells with fewer than 200 genes, we would want to get rid of them they are probably low quality cells or cell fragments. We would usually expect to find more features in an undamaged cell that was captured and sequenced properly.
> >
> {: .solution}
{: .question}

> <comment-title></comment-title>
> If you would like to see the impact of the initial filtering performed by {% tool Seurat Create %}, then you can rerun this step with these filters removed (or changed) and then perform the QC visualisation steps described in the next section on the unfiltered dataset. The Seurat pipeline usually includes some filtering during object creation, which shouldn't cause any problems as long as we use reasonable thresholds that only filter out the lowest quality cells and genes.  We can always come back and create a new object with different values if we suspect they weren't right for our dataset once we make our QC plots or if we have problems later on in the analysis.
{: .comment}

# Preprocessing

The standard Seurat pipeline begins with preprocessing, which includes identifying and filtering out low-quality cells, normalising and scaling the data, and selecting highly variable features.

## Quality Control

Quality control is an essential step in preparing single cell data for analysis. The results from a single cell experiment can vary in quality due to cell damage during dissociation, capturing more than one cell in a well or droplet, failures in library preparation such as inefficient reverse transcription (copying of captured RNAs to cDNA) or PCR amplification (making more copies of the cDNA so we have enough material for sequencing) or other technical problems. Low quality cells can cause issues for some analysis tools and may contribute to misleading results.

> <details-title>Impact of low-quality cells on the downstream analyses</details-title>
>
> #### Formation of their own distinct cluster(s)
>
> The most obvious cause of this problem is that damaged cells can appear more similar to each other because they have increased proportions of mitochondrial genes or are enriched for certain damage-induced genes. We could end up with a cluster made up of different cell types that share similar RNAs because they've been damaged. We might think these damaged cells represent an intermediate state between other cell types or end up misinterpreting our data in other ways.
>
> #### Distortion of population heterogeneity during variance estimation or principal components analysis
>
> When we cluster single cell data, we're looking for the biggest differences between groups of cells. If we have lots of low-quality cells, then the biggest differences we'll see might be these differences in quality, rather than something more biologically interesting. Differences in quality can have a big impact because low-quality cells often have low total RNA counts. When we perform scaling and normalisation on these cells during preprocessing, this can make the variances for the genes they do express much bigger than for other cells. When we select the most variable genes in the dataset, we'll end up picking the ones expressed by low-quality cells. We'll use these in our PCA analysis, likely ending up with top principal components based on cell quality rather than biology, making it harder to detect the differences we're actually interested in.
>
> #### Misidentification of upregulated genes
>
> Another problem that arises when we apply our preprocessing steps to small, low quality cells is that the genes we detect in them can appear to be strongly upregulated. Since we didn't detect many other genes in these cells, even a small difference in the number of transcripts detected can end up becoming much larger after normalisation. For example, contaminating transcripts may be present in all cells at low but constant levels. With the increased scaling and normalization in low-quality cells, the small counts for these transcripts may become large normalized expression values, so we might think we've found a population of cells where these genes are upregulated.
>
{: .details}

In order to mitigate these problems, we need to remove low-quality cells at the start of the analysis.

The `CreateSeuratObject` function we used in the {% tool Seurat Create %} tool automatically calculates some QC metrics and allowed us to filter out the lowest quality cells and features immediately. We will now check the data to see if it requires any further filtering.

### Computation of QC metrics

We already have two useful QC metrics that were calculated when we created our SeuratObject:

- **nCount_RNA** the total sum of RNAs that were found in each cell.
    The total number of counts is related to cell size, but it can also be an indication of quality. If `nCount_RNA` is very high we might be looking at results from a doublet or multiplet - two or more cells that were isolated together during the experiment. If `nCount_RNA` is very low, then it is likely that we lost a lot of the RNA due to cell lysis (breakage) or inefficient cDNA capture and amplification - or perhaps we only captured a fragment of a cell.
- **nFeature_RNA** the number of unique genes that were detected in each cell.
    We would expect to see some variation in the variety of genes expressed by cells of different sizes, types, and conditions, but if this number is unusually high or low then it could be a sign of poor quality. High `nFeature_RNA` could be another sign of a doublet or multiplet - we might have captured cells of different types and processed them together. Low `nFeature_RNA` could be due to loss of RNA if `nCount_RNA` is also low, or a sign that we have failed to capture the diversity of the transcript population, perhaps due to technical problems in our experiment.

One other metric that is often used to assess cell quality is the proportion of reads that came from the mitochondrial genome. The proportion is often higher in low quality, damaged or dying cells. Mitochondrial RNAs, which are protected inside the mitochondrial membranes, can be the last RNAs to be degraded or lost from a damaged cell, so we can end up with higher proportions of them in a low quality cell. We could have calculated the proportion of mitochondrial genes while creating our SeuratObject, but we will calculate it separately here to see how it is done - and how we could do the same for other types of genes.

We can identify mitochondrial genes from their gene symbols. Human genes that are encoded in the mitochondrial DNA (rather than in the cell nucleus) have names beginning with 'MT-'. Different naming patterns are used in other species. The {% tool Seurat Create %} tool can identify genes based on a specific naming pattern and then calculate the proportion of reads in each cell that came from these genes. We could use this tool to calculate the proportions of other genes with shared naming patterns too, such as ribosomal genes.

> <hands-on-title>Calculate the Proportion of Mitochondrial Reads</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Add QC Metrics`
>        - {% icon param-file %} *"Input file with the Seurat object"*: `Input 3k PBMC` (output of **Seurat Create** {% icon tool %})
>        - *"Calculate percentage of reads based on"*: `Pattern in gene names`
>        - *"Pattern/regex to match in gene names"*: `^MT-`
>
> 2. Rename the output as `Mitochondrial Annotations`
{: .hands_on}

When we use these QC metrics to remove low quality cells, we are assuming they are independent of the biological state of each cell, which isn't always the case in reality. For example, some cell types with higher energy demands may have higher proportions of mitochondrial reads because they have more mitochondria. We need to think carefully when deciding which metrics to use for filtering, but for now we will assume that these differences in our dataset are being driven by technical factors (e.g. cell damage, loss of RNAs, problems with capture and sequencing) rather than biological processes (e.g. cell types or conditions). We can therefore remove cells with poor values without misrepresenting the biology in downstream analyses.

### Filtering of low-quality cells

The simplest approch for identifing low-quality cells is to apply thresholds on the QC metrics. We assume that any cells beyond these thresholds are low quality cells and that this is due to those technical factors. Although this is a simple strategy, we will need to decide where to set our thresholds, and this will depend on the experimental protocol and biological system - there are no standard thresholds that we can use for every dataset.

We can visualise the QC metrics to help us decide where to set our thresholds for filtering out low quality cells. We want to get rid of cells that have unusually high or low numbers of genes or unique features, as well as cells that have higher proportions of mitochondrial genes.

> <hands-on-title>Visualise QC Metrics</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Violin Plot with 'VlnPlot'`
>        - *"Features to plot"*: `nFeature_RNA,nCount_RNA,percent.mt`
>        - In *"Plot Formatting Options"*:
>            - *"Number of columns to display"*: `3`
>
> 2. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `percent.mt`
>
> 3. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `nFeature_RNA`
>
{: .hands_on}

> <tip-title></tip-title>
> If you are re-running the same step again or just want to change a couple of parameters for the same tool, then you can click on a previous dataset to expand it in the history and then click the {% icon dataset-rerun button %}. The tool you used to create that dataset will open in the main panel, with the settings and inputs you used before. Make any changes and run it again.
{: .tip}

![Three violin plots each showing most points grouped together in the lower part of the plot with some outliers at higher values](../../images/scrna-seurat-pbmc3k/seurat_violin_QC_before.png "Violin Plots showing the unique features (nFeature_RNA), total counts (nCount_RNA) and the proportion of reads coming from mitochondial genes (percent.mt) for all cells")

![Scatter plot A shows clear positive correlation with the number of unique features increasing as total counts increases. A small number of cells have very high values of both. Scatter plot B does not show a relationship between the proportion of mitochondrial genes and total counts. A small number of cells with low total counts have high values of percent.mt](../../images/scrna-seurat-pbmc3k/seurat_QC_scatter_before.png "Scatter plots showing the relationships between the total counts (nCount_RNA) and A. the number of unique features (nFeature_RNA) and B. the proportion of mitochondrial reads (percent.mt)")

> <question-title></question-title>
>
> 1. What do the violin plots tell us about the cell sizes and quality in our dataset?
> 2. What do the scatter plots in tell us about the relationship between cell size and the other QC metrics?
>
> > <solution-title></solution-title>
> > 1. The violin plots give us an overview of the cell sizes (nCount_RNA), number of unique genes (nFeature_RNA), and proportion of mitochondrial reads for our cells. We can see that most of the cells are grouped together near the bottom of each plot, but there are some outliers that have unusually high or low values of each metric. Cells with higher values of nCount_RNA are likely to be larger as they contained more RNA. Cells with higher values of nFeature_RNA had RNAs produced from lots of different genes, while those with low nFeature_RNA had RNAs produced from a smaller range of genes. Cells with high percent.mt had lots of reads from mitochondrial genes. Although there will be some natural variation in cell size, the range of genes being expressed, and mitochondrial content, we suspect that extreme values of these three metrics reflect low quality cells.
> > 2. The scatter plots show us how these QC metrics relate to each other. As expected, the cells with higher total RNA counts also tend to have higher numbers of unique features, but some cells have particularly high values of both - these could be doublets. The proportion of mitochondrial genes doesn't increase with cell size (total counts) in the same way. We can see a small number of cells that have very high proportions of mitochondrial reads despite having low total reads - these are likely to be damaged cells that have lost a lot of their non-mitochondrial RNAs.
> >
> {: .solution}
{: .question}

We can now set our QC thresholds based on these plots. Unfortunately, there are no standard thresholds that can be applied to every dataset, so we need to look at our data and make this decision for ourselves.

We've already filtered out the cells with the lowest total counts when we created the SeuratObject, so we'll focus on filtering the number of unique features and the proportion of mitochondrial reads. We saw on the scatter plots that the cells with high values for these two metrics also had the highest nCount_RNA values, so we'll actually be getting rid of the cells with the highest total counts too.

> <comment-title></comment-title>
> If you've used Scanpy or followed the [Clustering 3K PBMCs with Scanpy]({% link topics/single-cell/tutorials/scrna-scanpy-pbmc3k/tutorial.md %}) tutorial then you might be wondering why we aren't visualising and filtering our genes as well as our cells. Seurat enables us to set the minimum number of cells that genes must be present in when we create the SeuratObject, but the pipeline doesn't usually include any further quality checks for genes. If we did need to do further filtering, we could do this in R by calculating an appropriate QC metric and subsetting the data.
{: .comment}

> <question-title></question-title>
>
> 1. What threshold would you set for nFeature_RNA?
> 2. What threshold would you set for percent.mt?
>
> > <solution-title></solution-title>
> > 1. A threshold of 2500 seems sensible for this dataset. The violin plot for nFeature_RNA shows that most of our cells should be under this threshold, so we won't lose too much of our data. The scatter plot shows that the cells above this threshold also had unusually high values of nCount_RNA, which suggest they could include some doublets.
> > 2. A threshold of 5% should work for this dataset. As before, the violin plot shows that the majority of our cells are below this threshold, but in this case the cells above it had low total RNA counts, which suggests these could be damaged cells that had lost a lot of their other RNAs. Although there are few standards to guide us in single cell analysis, you will see the same 5% threshold for mitochondrial content used in many studies. It often works well, but it can filter out energetic cell types such as muscle cells, so we shouldn't apply this threshold without considering whether it works for our dataset. Some studies don't filter on mitochondrial reads at all!
> >
> {: .solution}
{: .question}

We're setting QC thresholds based on our visual inspections of the data, but we could take different approaches too. We could calculate the interquartile range for our QC metrics and use this to set a threshold (e.g. maximum of 1.5 times the IQ range above the median). We could also try running the analysis with different thresholds to see how it affects the results - this could help us to see if we're excluding any biological variation, for example if we're filtering out a specific high-energy cell type with our mitochondrial threshold.

> <hands-on-title>Filter Out Low Quality Cells</hands-on-title>
>
> 1. {% tool [Seurat Create](toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1) %} with the following parameters:
>    - *"Method used"*: `Filter cells by QC metrics`
>        - {% icon param-file %} *"Input file with the Seurat object"*: `Mitochondrial Annotations` (output of **Seurat Create** {% icon tool %})
>        - *"Minimum nFeature_RNA"*: `200`
>        - *"Maximum nFeature_RNA"*: `2500`
>        - *"Maximum percent.mt"*: `5.0`
>        - *"Filter by a different metric"*: `No`
>
> 2. Rename the output as `Filtered Dataset`
>
{: .hands_on}

If we produce the same plots again, we can see what has changed in our data.

> <hands-on-title>Re-Visualise QC Metrics</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Violin Plot with 'VlnPlot'`
>        - *"Features to plot"*: `nFeature_RNA,nCount_RNA,percent.mt`
>        - In *"Plot Formatting Options"*:
>            - *"Number of columns to display"*: `3`
>
> 2. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `percent.mt`
>
> 3. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Scatter Plot with 'FeatureScatter'`
>        - *"First feature to plot"*: `nCount_RNA`
>        - *"Second feature to plot"*: `nFeature_RNA`
>
{: .hands_on}

![Three violin plots each showing most points grouped together and no extreme outliers](../../images/scrna-seurat-pbmc3k/seurat_violin_after.png "Violin Plots showing the unique features (nFeature_RNA), total counts (nCount_RNA) and the proportion of reads coming from mitochondial genes for all cells after filtering")

![Scatter plot A shows clear positive correlation with the number of unique features increasing as total counts increases. No extreme outliers. Scatter plot B does not show a relationship between the proportion of mitochondrial genes and total counts. No extreme outliers.](../../images/scrna-seurat-pbmc3k/seurat_QC_scatter_after.png "Scatter plots showing the relationships between the total counts (nCount_RNA) and A. the number of unique features (nFeature_RNA) and B. the proportion of mitochondrial reads (percent.mt) after filtering")

> <question-title></question-title>
>
> 1. Have we eliminated the low-quality cells from our data?
>
> > <solution-title></solution-title>
> > 1. We can see in both the violin and scatter plots that the outliers have been eliminated from our dataset - we've removed the cells with high mitochondrial counts and/or unusually high numbers of detected genes. We hope that this means we've got rid of the poor quality cells without losing any real biological variation, but we can't really be sure. Sometimes it is necessary to come back and re-think our filters - for example if you get to the end of the analysis and realise that you're missing a cell type that should be present in your data!
> >
> {: .solution}
{: .question}

## Further Preprocessing

Now that we're happy with the cells left in our SeuratObject, we can begin preparing the data for analysis. Seurat provides two routes for preprocessing, one with separate tools for each step and another that combines all these preprocessing steps into a single tool, `SCTransform`.

> <comment-title></comment-title>
>
> `SCTransform` makes preprocessing quicker and easier because it combines several steps from the standard Seurat pipeline into one. `SCTransform` normalises and scales the data while also selecting highly variable features. It also takes a different approach to preprocessing that can be more effective at removing technical effects from the data.
>
> Once you are familiar with the separate steps, you might find it more effective and convenient to use `SCTransform`. The main difference to be aware of when using `SCTransform` is that the results will be stored as a new assay called `SCT` in the SeuratObject. You will need to use this assay in the following analysis steps, instead of the original `RNA` assay. After running `SCTransform`, `SCT` will be set as the default assay for the SeuratObject so the tools should automatically use the correct one. The methods and default settings used in `SCTransform` also differ from the standard pipeline, so you should expect to see some differences in your results compared to the other route.
>
{: .comment}

The usual preprocessing steps for single cell data are normalisation, selection of the most variable features, and scaling the data. The same steps are performed in slightly different ways by the separate preprocessing functions and SCTransform.

- **Normalisation**
    Normalisation deals with variations in the number of reads we counted for each cell that were caused by systemic technical differences. We might have captured more RNAs or produced more copies of them during the PCR amplification step in some cells. If we don't remove these differences then they could obscure the biological differences we're trying to explore. We could end up with clusters of cells based on how many RNAs we captured or copied rather than on their cell type.
    The default normalisation method for the separate `NormalizeData` tool in Seurat is `LogNormalize`, which simply divides the number of counts for each gene by the total counts for the cell, multiplies this by a scale factor (10,000 is the default), and then log-transforms the results. Essentially, we're working out how many counts we would have for each gene if we had produced 10,000 reads from each cell, instead of getting different numbers of reads from different cells. The log-transformation then increases the impact that genes that showed stronger relative differences in expression between cells will have on our analysis (e.g. a gene that is expressed at an average count of 50 in cell type A and 10 in cell type B should have a bigger impact than a gene that is expressed at an average count of 1100 in A and 1000 in B). The normalised data will be stored in the `data` layer of the SeuratObject. The unnormalised data will still be available in the `counts` layer.
    Although this form of normalisation is widely-used, it does assume that each cell originally had the same number of RNA molecules, which isn't the case when our samples contain cells of different types and sizes. SCTransform takes a different approach to normalisation that doesn't make the same assumption about the cells {% cite SCTransform2022 %}. Instead, it creates a model of the UMI counts that enables it to regress out variation in sequencing depth (nFeature_RNA) and pools information from genes with similar abundances to adjust the variances. The corrected counts will be stored in the `counts` layer of the new SCT assay. The log1p(counts) will be stored in the `data` layer and the pearson residuals will be stored in the `scale.data` layer.
- **Selection of highly variable features**
    Rather than use the entire dataset in every stage of the analysis, we can focus on the genes that provide the most information - the highly variable genes that showed the biggest differences in expression between cells. We assume that these bigger differences reflect genuine biological differences, while the smaller differences in expression seen in other genes is down to chance or technical noise. Focusing on the most variable genes can make biological differences clearer in single cell analysis.
    Feature selection can have a big impact on our analysis as we will run downstream analyses (e.g. PCA) on these genes. We will be comparing cells based on the expression of these variable genes.
    Seurat's separate `FindVariableFeatures` tool selects the 2000 most variable genes by default, while `SCTransform` selects the top 3000 genes. We can select more variable genes when we use SCTransform because this method is better at removing technical effects from the data, so the additional variable features are more likely to represent biological variation.
  We may need to change this setting for some datasets to ensure we're selecting the most useful genes without including too many others.
- **Scaling**
    Scaling is a linear transformation that we apply to prepare our data for dimensional reduction.
    The default scaling method for the separate `ScaleData` tool in Seurat shifts the expression of each gene so that the mean expression across all the cells is 0. It also scales the expression of each gene so that the variance across all cells is 1. Scaling ensures that highly expressed genes don't dominate the analysis too much - we're interested in differences in expression between cells, not in genes that are always highly expressed in all the cells. The results will be stored in the `scale.data` layer.
    SCTransform doesn't scale data in the same way - although it centers the data by default, it won't scale the data unless you select this option. Instead, SCTransform usually stores the pearson residuals in the `scale.data` layer, which don't need to be scaled to the same variance.
    By default, both approaches only center/scale the highly variable genes that we'll use for dimensional reduction, but it is possible to scale more genes if required. If you choose to use the separate preprocessing tool in this tutorial, then we'll use the ScaleData tool to scale all the genes - this means that we can compare or plot the expression of any of the genes, as they will all be scaled in the same way. We won't perform scaling with SCTransform, as we're following the [original SCT tutorial](https://satijalab.org/seurat/articles/sctransform_vignette.html), but we will center all genes.
- **Regression**
  The Seurat pipeline can include another step during preprocessing of our single cell data. We can regress out (or remove) the impact of unwanted sources of variation. We could use this technique to remove the effects of the cell cycle or the differences associated with the proportion of mitochondrial genes. The goal is to reduce differences that are related to factors we are not interested in as this can help the differences we are interested in (like those between cell types or experimental groups) stand out more.
    It is possible to use the `ScaleData` function to regress out unwanted variation, but the creators of Seurat recommend using `SCTransform` for preprocessing if you want to do any regression. `SCTransform` automatically regresses out variation associated with sequencing depth (unique counts or nFeature_RNA) and can also regress out other variables. If you choose to use SCTransform in this tutorial, then you'll regress out the variation associated with the proportion of mitochondrial content, just like in [Seurat's original version of this tutorial](https://satijalab.org/seurat/articles/sctransform_vignette.html)

{% include _includes/cyoa-choices.html option1="Separate Preprocessing Steps" option2="SCTransform" default="Separate Preprocessing Steps" text="You can perform each preprocessing step separately, which might give you a better understanding of the different elements involved in preprocessing, or run them all at once using SCTransform." %}

<div class="Separate-Preprocessing-Steps" markdown="1">
><hands-on-title>Separate Preprocessing Steps</hands-on-title>
>
> 1. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Normalize with 'NormalizeData'`
>        - *"Method for normalization"*: `LogNormalize`
>
> 2. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `rds_out` (output of previous **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Identify highly variable genes with 'FindVariableFeatures'`
>        - *"Method to select variable features"*: `vst`
>        - *"Output list of most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>
> 3. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `rds_out` (output of previous **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Scale and regress with 'ScaleData'`
>        - *"Regress out a variable"*: `No`
>        - *"Features to scale"*: `All Features`
>
>    We'll choose to scale `All Features` here, instead of just the highly variable genes.
>
> 4. Rename the output as `Preprocessed Data`
>
{: .hands_on}
</div>

<div class="SCTransform" markdown="1">
><hands-on-title>SCTransform</hands-on-title>
>
> 1. {% tool [Seurat Preprocessing](toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Filtered Dataset` (output of **Seurat Create** {% icon tool %})
>    - *"Method used"*: `Complete all preprocessing with "SCTransform"`
>        - *"Genes to calculate residual features for"*: `all genes`
>            - *"How to set variable features"*: `set number of variable features`
>        - *"Output list of most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>        - *"Variable(s) to regress out"*: `percent.mt`
>
> 2. Rename the output as `Preprocessed Data`
>
>    If we inspect this dataset, we can see that a new assay called `SCT` has been created and is set as the default assay. Click on the {% icon galaxy-eye %}- of the following output to check this.
>
> 3. {% tool [Seurat Data Management](toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Preprocessed Data` (output of **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Inspect Seurat Object`
>        - *"Display information about"*: `General`
>
{: .hands_on}
</div>

### Visualise Highly Variable Genes

><hands-on-title>Visualise Results</hands-on-title>
>
> 1. {% tool [Seurat Visualize](toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file with the Seurat object"*: `Preprocessed Data` (output of **Seurat Preprocessing** {% icon tool %})
>    - *"Method used"*: `Plot Variable Genes with 'VariableFeaturePlot'`
>        - *"Label the top most variable features"*: `Yes`
>        - *"Number to show"*: `10`
>
{: .hands_on}

<div class="Separate-Preprocessing-Steps" markdown="1">

![Most genes are grouped together at the bottom of the plot with low standardized variance. The selected variable genes are highlighted in red and labelled by name. They have higher standardized variances although most are still quite close to the non variable genes. The labels read PPBP, S100A9, LYZ, IGLL5, GNLY, FTL, PF4, FTH1, GNG11, and FCER1A.](../../images/scrna-seurat-pbmc3k/seurat_variable_genes.png "Plot showing the standardized variances of variable and non-variable genes. The top 10 most variable genes are labelled.")

</div>

<div class="SCTransform" markdown="1">

![Most genes are closely grouped together, right at the bottom of the plot with low standardized variance. The selected variable genes are highlighted in red and labelled by name. They are higher up on the plot, showing they have higher standardized variances although most are still quite close to the non variable genes. They are S100A9, GNLY, LYZ, S100A8, NKG7, FTL, GZMB, IGLL5, FTH1, and CCL5.](../../images/scrna-seurat-pbmc3k/seurat_variable_genes_SCT.png "Plot showing the standardized variances of variable and non-variable genes. The top 10 most variable genes are labelled.")

</div>

> <question-title></question-title>
> 1. What are the top 10 most variable genes in this dataset?
> 2. Why are we interested in these genes?
>
> > <solution-title></solution-title>
> >
> > 1. We can check the list of the top variable genes in our history by clicking on the {% icon galaxy-eye %} or see them on our variable features plot.
> >    <div class="Separate-Preprocessing-Steps" markdown="1">
> >
> >    |    |         |
> >    |----|---------|
> >    | 1  | PPBP    |
> >    | 2  | LYZ     |
> >    | 3  | S100A9  |
> >    | 4  | IGLL5   |
> >    | 5  | GNLY    |
> >    | 6  | FTL     |
> >    | 7  | PF4     |
> >    | 8  | FTH1    |
> >    | 9  | GNG11   |
> >    | 10 | S100A8  |
> >    {: .matrix}
> >
> >    </div>
> >
> >    <div class="SCTransform" markdown="1">
> >
> >    |    |          |
> >    |----|----------|
> >    | 1  | S100A9   |
> >    | 2  | GNLY     |
> >    | 3  | LYZ      |
> >    | 4  | S100A8   |
> >    | 5  | NKG7     |
> >    | 6  | FTL      |
> >    | 7  | GZMB     |
> >    | 8  | IGLL5    |
> >    | 9  | FTH1     |
> >    | 10 | CCL5     |
> >    {: .matrix}
> >
> >    </div>
> >
> >    > <comment-title></comment-title>
> >    > The list of highly variable genes you end up with will depend on which preprocessing route you chose. The two preprocessing routes use different methods to select features, so they won't always end up with the same genes, although there are likely to be some similarities. `SCTransform` also returns 3000 variable genes by default, rather than the 2000 selected by `FindVariableFeatures`. We can select more features with `SCTransform` because its normalisation method is better at removing technical effects from the data, so we believe that these additional genes reflect subtler biological variations rather than technical differences.
> >    {: .comment}
> >
> > 2. Single cell datasets contain a lot of information, including expression data for thousands of different genes. Some of these genes don't tell us much about the data, for example they might be housekeeping genes that are expressed at similar levels in most of our cells. We want to find the genes that can tell us most about the differences between our cells, so we want to identify the genes whose expression varies most across the dataset - focusing on these highly variable features should help us to uncover the biological differences we're looking for.
> >
> {: .solution}
{: .question}

# Dimensionality Reduction

Dimensional reduction is a key step in single cell analysis that simplifies our big, complex datasets enough to enable us to perform and understand further analyses.

In single cell analysis, we're comparing cells based on their patterns of gene expression - we're looking for cells with similar transcriptomic profiles. Each gene represents a dimension of the data. If we only found two genes being expressed in our cells, we could represent them on a 2-dimensional plot. The two axes of the plot would represent the expression levels of the two genes. Each cell would become a point on the plot, positioned according to its expression of these genes. We might spot one group of cells with high levels of both genes and another group that expresses large amounts of one gene and not much of the other.

For datasets with thousands of genes, the concept is essentially the same: each cell's expression profile defines its location in the high-dimensional expression space. Each gene is still a dimension or axis on the plot and each cell can be positioned along these axes based on their expression. It's just much harder to imagine a plot with thousands of different axes - that's why we need dimensional reduction!

We don't have to think of the expression of each of our thousands of genes individually. The expression of some genes can be correlated if they are affected by the same biological process - cells that express a lot of one gene might also express higher levels of other genes that have similar functions. Instead of keeping the information from each of these genes as separate axes or dimensions, we can compress them into a single dimension (e.g. an 'eigengene'). We can do the same for all of the correlated groups of genes in our data, going from thousands of individual genes to perhaps 10, 20, or 50 grouped dimensions.

 Dimensionality re

tools_json:
["Seurat Create", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_create/seurat_create/5.0+galaxy1", "Seurat Data Management", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_data/seurat_data/5.0+galaxy0", "Seurat Visualize", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_plot/seurat_plot/5.0+galaxy0", "Seurat Preprocessing", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_preprocessing/seurat_preprocessing/5.0+galaxy0", "Seurat Run Dimensional Reduction", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_reduce_dimension/seurat_reduce_dimension/5.0+galaxy0", "Seurat Find Clusters", "toolshed.g2.bx.psu.edu/repos/iuc/seurat_clustering/seurat_clustering/5.0+galaxy0", "Filter", "Filter1", "Cut", "Cut1"]


topic: single-cell
tutorial: scatac-preprocessing-tenx
title: Pre-processing of 10X Single-Cell ATAC-seq Datasets
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["10x", "epigenetics"]
content_hash: 66e7a51b5fdfd9df3842a2567616b051
last_modified: 2026-05-27T09:01:49.741340
zenodo_link: https://zenodo.org/record/7855968
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scatac-preprocessing-tenx/tutorial.html

content:
Similar to bulk ATAC-Seq, single-cell ATAC-Seq (scATAC-seq) leverages the hyperactive Tn5 Transposase to profile open chromatin regions but at single-cell resolution. Thus helps in understanding cell type-specific chromatin accessibility from a heterogeneous cell population.

## Era of 10x Genomics

10x genomics has provided not only a cost-effective high-throughput solution to understanding sample heterogeneity at the individual cell level but has defined the standards of the field that many downstream analysis packages are now scrambling to accommodate. The gain in resolution reduces the granularity and noise issues that plagued the field of single-cell omics not long ago, where now individual clusters are much easier to decipher due to the added stability added by this gain in information.

## Library Preparation

![Library Preparation]({% link topics/single-cell/images/scatac-pre-processing/tenx_libprep_scatac.png %} "An overview of the 10x single-nuclei ATAC-seq library preparation")

The 10X barcoded gel beads consist of a pool of barcodes which are used to separately index each cell. The individual gel barcodes are delivered to each cell via flow cytometry, where each cell is fed single-file along a liquid tube and tagged with a 10X gel bead. The cells are then isolated from one another within thousands of nanoliter droplets, where each droplet is described by a unique 10x barcode that all reads in that droplet are associated with. The oil is then removed and all (now barcoded) DNA reads are pooled together to be sequenced.

More information can be found in the [reagent kit documentation](https://www.10xgenomics.com/support/single-cell-atac/documentation/steps/library-prep/chromium-single-cell-atac-reagent-kits-user-guide-v-2-chemistry).


## 10x Chemistries

There are two main reagent kits used during the library preparation. Below we can see the layout of the primers used in both chemistries. We can ignore most of these as they are not relevant, namely: the P5 and P7 Illumina primers are used in the Illumina [bridge amplification](https://en.wikipedia.org/wiki/Illumina_dye_sequencing#Bridge_amplification) process; the Sample Index is an 8bp primer which is related to the Chromium system that balances nucleotide bias and ensures that there is no sample overlap during the multiplexed sequencing. The primer of interest to us is the Cell Barcode (CB). Unlike scRNA-seq, Unique Molecular Identifiers (UMIs) are not used in scATAC-seq.

![chem]({% link topics/single-cell/images/scatac-pre-processing/tenx_primers_scatac.svg %} "10x scATAC-Seq sequencing library")

There are two major chemistries (v1 and v2) used in 10x scATAC-seq library construction. Technically, v2 chemistry is more sensitive and has better data yield. But the final sequencing library is the same. Hence, same the data analysis applies to both chemistries. Both chemistries generate the 4 FASTQ files containing:
* 8bp sample index sequence (I1)
* 50bp forward read (R1)
* 16bp barcode sequence (R2)
* 49bp reverse read (R3)

The major differences to 10x scRNA-seq sequencing libraries are: (a) barcode sequences get a separate FASTQ file and (b) there are no UMIs. Note that as opposed to the traditional FASTQ naming, R3 contains reverse mate-pair and R2 contains barcode sequence.


# Analysis Strategy

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

The tutorial is structured into four parts. The first part deals with the preprocessing of FASTQ files, quality control and mapping. Then in the second part, we will identify the open chromatin regions. In the third part, we will create a count matrix using the cell barcode and open chromatin regions information. In the final part, we will filter out empty cells and less informative peaks to produce a high-quality count matrix.

Here we will use a published data set from 10x genomics consisting of thousand peripheral blood mononuclear cells (PBMCs) extracted from a healthy donor. Analyzing these full datasets requires several hours of computation time. Hence, we subsampled the data so that the tutorial can be finished in a reasonable amount of time with sensible results. In the sub-sampled data, we have the reads that were mostly generated from chromosome 21. The original data has FASTQ files from two sequencing lanes *L001* and *L002*. We merged the sequences from both lanes. We also ignored the sample index FASTQ file as it is not useful for this analysis. Finally, we are left with three FASTQ files: **R1** (forward reads), **R2** (barcodes) and **R3** (reverse reads).

  * atac\_pbmc\_1k\_nextgem\_S1\_**R1**\_001\_chr21.fastq.gz
  * atac\_pbmc\_1k\_nextgem\_S1\_**R2**\_001\_chr21.fastq.gz
  * atac\_pbmc\_1k\_nextgem\_S1\_**R3**\_001\_chr21.fastq.gz

These files are provided in the [Zenodo](https://zenodo.org/record/7855968) data repository. To analyze the full datasets, get the original FASTQ files from [10x genomics website](https://www.10xgenomics.com/resources/datasets/1-k-peripheral-blood-mononuclear-cells-pbm-cs-from-a-healthy-donor-next-gem-v-1-1-1-1-standard-2-0-0)

# FASTQ processing and mapping

## Data upload and organization

We import all three FASTQ files into a history.

> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and rename it (e.g. scATAC-seq 10X preprocessing tutorial)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 1. Import the sub-sampled FASTQ data from [`Zenodo`](https://zenodo.org/record/7855968) or from the data library (ask your instructor)
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    ```
>    https://zenodo.org/record/7855968/files/atac_pbmc_1k_nextgem_S1_R1_001_chr21.fastq.gz
>    https://zenodo.org/record/7855968/files/atac_pbmc_1k_nextgem_S1_R2_001_chr21.fastq.gz
>    https://zenodo.org/record/7855968/files/atac_pbmc_1k_nextgem_S1_R3_001_chr21.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}
>
> <question-title></question-title>
> Inspect all the uploaded FASTQ files by clicking on the {% icon galaxy-eye %} symbol.
> 1. Which FASTQ file contains the barcode sequence?
> 1. What is the length of the barcode?
>
> > <solution-title></solution-title>
> > 1. In this case, R2 contains the cell barcodes
> > 1. 16bp
> >
> {: .solution}
{: .question}


## Attach cell barcodes to FASTQ headers

First, we will attach the barcodes from the barcodes FASTQ file to the ids of forward and reverse reads that carry the association between reads and barcodes throughout the analysis. We will use this information to demultiplex the cells at the end while creating the count matrix. We will be left with only two FASTQ files of forward and reverse reads after this step.

> <comment-title></comment-title>
>
> {% tool [Sinto barcode](toolshed.g2.bx.psu.edu/repos/iuc/sinto_barcode/sinto_barcode/0.9.0+galaxy1) %}  consumes the barcode sequence from the `atac_pbmc_1k_nextgem_S1_R2_001_chr21.fastq.gz` file and uses `:` delimiter to prepend it to both the forward and reverse read ids. The tool expects the same number of reads in all three FASTQ files and should be present in the same order.
{: .comment}

> <hands-on-title></hands-on-title>
>
> {% tool [Sinto barcode](toolshed.g2.bx.psu.edu/repos/iuc/sinto_barcode/sinto_barcode/0.9.0+galaxy1) %}  with the following parameters:
>    - *"FASTQ file containing cell barcode sequences"*: `atac_pbmc_1k_nextgem_S1_R2_001_chr21.fastq.gz`
>    - *"Single or Paired-end data"*: `Paired`
>    - *"Forward reads FASTQ file"*: `atac_pbmc_1k_nextgem_S1_R1_001_chr21.fastq.gz`
>    - *"Reverse reads FASTQ file"*: `atac_pbmc_1k_nextgem_S1_R3_001_chr21.fastq.gz`
>    - *"Number of bases to extract from barcode-containing FASTQ"*: `16`
>
>    > <comment-title>Sinto barcode output</comment-title>
>    >
>    > After a successful run, the barcode sequences from the barcode FASTQ file are prepended to the read names. We can inspect the output FASTQ files by clicking on the {% icon galaxy-eye %} symbol of the *barcoded read 1* file.
>    {: .comment}
>
{: .hands_on}


Now let's proceed with the next steps QC, mapping and peak calling as we do with the bulk ATAC-Seq data. For a thorough description of bulk ATAC-Seq analysis, please follow the bulk [ATAC-Seq data analysis tutorial]({% link topics/epigenetics/tutorials/atac-seq/tutorial.md %}). Here we use a slightly different workflow. Here we map reads using **BWA-MEM** instead of **Bowtie2**, then create ATAC fragments file that also stores the cell barcode information and finally use **MACS2** for peak calling. Note that the workflow from bulk ATAC-seq tutorial should also result in similar set of peaks.

## FASTQ Quality control
First things first. Let's do a basic FASTQ quality control using **FastQC**

> <hands-on-title></hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0) %} with the following parameters:
>       - *"Raw read data from your current history"*: Use {% icon param-files %} **Multiple datasets** to choose both `barcoded read 1` and `barcoded read 2` (outputs of **Sinto barcode** {% icon tool %}).
> 2. Inspect the web page output of **FastQC** {% icon tool %} for the `barcoded read 1` sample.
>
> > <question-title></question-title>
> >
> > 1. How many reads are in the FASTQ?
> > 2. Which sections have a warning?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are 685686 reads.
> > > 2. The 2 steps below have warnings:
> > >
> > >    1. **Per base sequence content**
> > >
> > >       It is well known that the Tn5 has a strong sequence bias at the insertion site. You can read more about it in {% cite Green2012 %}.
> > >
> > >    2. **Sequence Duplication Levels**
> > >
> > >       The read library quite often has PCR duplicates that are introduced
> > >       simply by the PCR itself. We will remove these duplicates later on.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

> <tip-title>Adapter trimming</tip-title>
>
> There is only a little amount of Nextera Transposase Sequence in the reads (see Adapter Content section of **FastQC** output). In this case, we skip the adapter trimming step and proceed to the mapping step. If your data has a significant amount of Nextera Transposase Sequences, then proceed to adapter trimming using either **cutadapt** or **Trim Galore!**. **Trim Galore** can automatically detect the adapter sequence. If you use **Cutadapt**, follow the bulk ATAC-Seq tutorial [Trimming Reads]({% link topics/epigenetics/tutorials/atac-seq/tutorial.md %}) step.
{: .tip}

## Mapping reads to a reference genome
Now we map the reads to a reference genome using {% tool [BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %}.

> <hands-on-title>Mapping</hands-on-title>
>
> {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2) %} with the following parameters:
>    - *"Using reference genome"*: `hg38`
>    - In *"Single or Paired-end reads"*: `Paired`
>      - In *"Select first set of reads "*: `barcoded read 1` (output of **Sinto barcode** {% icon tool %})
>      - In *"Select second set of reads "*: `barcoded read 2` (output of **Sinto barcode** {% icon tool %})
>    - *"Select analysis mode"*: `1.Simple Illumina mode`
>    - *"BAM sorting mode"*: `Sort by chromosomal coordinates`
{: .hands_on}

# Peak calling
In scRNA-seq we always have the standard set of genes (usually downloaded from public databases) to quantify the expression levels. For scATAC-seq, there are no such reference open chromatin regions because the regions of chromatin accessibility are tissue dependent. Hence, we first have to detect open chromatin regions. There are several ways of doing this. The most common ways are the following:
* Find a published bulk ATAC-seq data that is the closest to your scATAC-seq data and use the regions from the bulk ATAC-seq data
* Chunk the genome into equal-sized bins and use these bins as reference locations for quantification
* Combine the data from all the cells of the scATAC-seq data together and detect open chromatin regions from the data. Later use these detected regions for quantification

In this tutorial, we opt for the 3rd option.

## Create scATAC-seq fragments file
An ATAC-seq fragment file is a BED file with Tn5 integration sites, the cell barcode associated with the fragment, and the frequency of the sequenced fragment. PCR duplicates are collapsed. Here we filter out the alignments with low mapping quality.

> <comment-title>on Tn5 insertion</comment-title>
>
> When Tn5 cuts an accessible chromatin locus it inserts adapters separated by 9bp ({% cite Kia2017 %}):
> ![Nextera_Library_Construction]({% link topics/epigenetics/images/atac-seq/NexteraLibraryConstruction.jpg %} "Nextera Library Construction")
>
> This means, to have the read start site reflecting the center of where Tn5 bound, the reads on the positive strand should be shifted 4 bp to the right and reads on the negative strands should be shifted 5 bp to the left as in [Buenrostro et al. 2013](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3959825). **Genrich** can apply these shifts when ATAC-seq mode is selected. In most cases, we do not have 9bp resolution so we don't take it into account but if you are interested in the footprint, this is important.
{: .comment}

> <hands-on-title>Create scATAC fragments</hands-on-title>
> 1. {% tool [Sinto fragments](toolshed.g2.bx.psu.edu/repos/iuc/sinto_fragments/sinto_fragments/0.9.0+galaxy1) %} with the following parameters:
>    - *"Input BAM file"*: `mapped reads in BAM format` (output of **Map with BWA-MEM** {% icon tool%})`
>    - *"Minimum MAPQ required to retain fragment"*: `30`
>    - *"Regular expression used to extract cell barcode from read name"*: `[^:]*` (matches all characters up to the first colon)
>    - *"Number of bases to shift Tn5 insertion position by on the forward strand"*: `4`
>    - *"Number of bases to shift Tn5 insertion position by on the reverse strand"*: `-5`
>    - *"Take cell barcode into account when collapsing duplicate fragments"*: `Yes`
>
> 1. {% tool [bedtools SortBED](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2) %} with the following parameters:
>    - *"Sort the following BED/bedGraph/GFF/VCF/EncodePeak file *"*: `fragments BED` (output of **Sinto fragments** {% icon tool%})`
>    - *"Sort by"*: `chromosome, then by start position (asc)`
> 1. Rename the dataset `sorted fragments`
{: .hands_on}

## Call Peaks

We have now finished the data preprocessing. Next, to find regions corresponding to potential open chromatin regions, we want to identify regions where reads have piled up (peaks) greater than the background read coverage. The tools which are currently used are [Genrich](https://github.com/jsh58/Genrich) and [MACS2](https://github.com/taoliu/MACS). MACS2 is more widely used.
At this step, two approaches exist:

- The first one is to select only pairs whose fragment length is below 100bp corresponding to nucleosome-free regions and to use a peak calling like you would do for a ChIP-seq, joining signal between mates. The disadvantage of this approach is that you can only use it if you have paired-end data and you will miss small open regions where only one Tn5 is bound.
- The second one chosen here is to use all reads to be more exhaustive. In this approach, it is very important to re-center the signal of each read on the 5' extremity (read start site) as this is where Tn5 cuts. Indeed, you want your peaks around the nucleosomes and not directly on the nucleosome:

![ATAC-Seq_reads_relative_to_nucleosomes]({% link topics/epigenetics/images/atac-seq/schemeWithLegend.jpg %} "Scheme of ATAC-Seq reads relative to nucleosomes")

If we only assess the coverage of the 5' extremity of the reads, the data would be too sparse and it would be impossible to call peaks. Thus, we will extend the start sites of the reads by 200bp (100bp in each direction) to assess coverage. We call peaks with MACS2. In order to get the coverage centered on the 5' extended 100bp on each side we will use `--shift -100` and `--extend 200`:
![MACS2_peak_centering_options]({% link topics/epigenetics/images/atac-seq/macs2Options.jpg %} "MACS2 options to get 100bp each side")

> <hands-on-title>Call peaks with MACS2</hands-on-title>
>
> 1. {% tool [MACS2 callpeak](toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6) %} with the following parameters:
>    - *"Are you pooling Treatment Files?"*: `No`
>    - *"ChIP-Seq Treatment File"*: `sorted fragments`
>    - *"Do you have a Control File?"*: `No`
>    - *"Format of Input Files"*: `Single-end BED`
>    - *"Effective genome size"*: `H. sapiens (2.7e9)`
>    - *"Build Model"*: `Do not build the shifting model (--nomodel)`
>        - *"Set extension size"*: `200`
>        - *"Set shift size"*: `-100`. It needs to be - half the extension size to be centered on the 5'.
>    - *"Additional Outputs"*:
>        - Check `Peaks as tabular file (compatible with MultiQC)`
>        - Check `Peak summits`
>        - Check `Scores in bedGraph files`
>    - In *"Advanced Options"*:
>        - *"Composite broad regions"*: `No broad regions`
>            - *"Use a more sophisticated signal processing approach to find subpeak summits in each enriched peak region"*: `Yes`
>        - *"How many duplicate tags at the exact same location are allowed?"*: `all`
>
>    > <comment-title>Why keeping all duplicates is important</comment-title>
>    >
>    > We previously removed duplicates using **Sinto fragment** {% icon tool %} using paired-end information. If two pairs had identical R1 but different R2, we knew it was not a PCR duplicate. Because we converted the BAM to BED we lost the pair information. If we keep the default (removing duplicates) one of the 2 identical R1 would be filtered out as duplicate.
>    {: .comment}
>    > <comment-title>BED / encode narrowPeak format</comment-title>
>    > If you are not familiar with BED format or encode narrowPeak format, see the [BED Format](https://genome.ucsc.edu/FAQ/FAQformat.html)
>    {: .comment}
>
{: .hands_on}

# Count matrix creation

Now we have ATAC fragments with cell barcode information as well as the peaks to create a scATAC count matrix. In the context of ATAC-seq, the peaks are also often called features. But the term *feature* is a generic term that denotes a *peak* in the case of scATAC-seq data and a *gene* in case of scRNA-seq data. So the count matrix we build is similar to that of scRNA-seq data with the exception that here we have open chromatin regions instead of genes. In this tutorial, both the terms *feature* and *peak* indicate an *open chromatin region*.

For count matrix creation, we will use **Build count matrix** from **EpiScanpy** tool suite. It takes fragments file and the peaks in either BED format or directly a narrowPeak file of **MACS2** output. Regardless of file type, the tool only considers the first 3 columns which contain chromosome id, start and end positions of the peak regions. Parameters like `--call-summits` from our MACS2 run can result in different peaks with the same peak boundaries. It is always a  good practice to remove duplicate peak regions so that our final count matrix will have unique features.


> <hands-on-title>Extract unique peak regions</hands-on-title>
>
> 1. {% tool [Unique occurrences of each record](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0) %} with the following parameters:
>    - *"File to scan for unique values"*: `narrow Peaks` (output of **MACS2** {% icon tool %})
>    - In *"Advanced Options"*: `Show Advanced Options`
>        - *"Column start"*: `Column: 1`
>        - *"Column end"*: `Column: 3`
>
> > <question-title></question-title>
> >
> > 1. How many peak regions were duplicated?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There were initially 1229 regions in the `narrow Peaks` file. Now there are 1046 regions after deduplication. Around 15% (184) of regions have the same peak boundaries.
> > >
> > {: .solution}
> >
>   {: .question}
{: .hands_on}

## AnnData

The most common format, called [`AnnData`](https://anndata.readthedocs.io/en/stable/), stores the matrix as well as gene and cell annotations in a concise, compressed and extremely readable manner:

![Anndata format]({% link topics/single-cell/images/scrna-pre-processing/tenx_anndata.png %} "<code>AnnData</code> format stores a count matrix <code>X</code> together with annotations of observations (i.e. cells) <code>obs</code>, variables (i.e. peaks) <code>var</code> and unstructured annotations <code>uns</code>.")

This format is used by [Scanpy](https://scanpy.readthedocs.io/en/stable/index.html) ({% cite wolf2018scanpy %}) and [EpiScanpy](https://colomemaria.github.io/episcanpy_doc/index.html) ({% cite danese2021episcanpy %}) tool suites for analyzing single-cell omics data. So we need first to import the matrix and annotations of peaks and cells (present in fragments BED file) into an `AnnData` object.

> <hands-on-title>Build count matrix with EpiScanpy</hands-on-title>
>
> 1. {% tool [Build count matrix with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_build_matrix/episcanpy_build_matrix/0.3.2+galaxy1) %} with the following parameters:
>    - *"ATAC fragments file"*: `sorted fragments`
>    - *"Features file"*: output of **Unique** tool {% icon tool %}
>    - *"Normalize peak sizes?"*: `Keep the peaks as they are`
>    - In *"Select the chromosomes of the species you are considering"*: `Custom list of chromosomes`
>        - *"Enter comma seperated list of chromosome ids (without chr prefix)"*: `21`
> 2. Rename the generated file to `PBMC 1k chr21 Anndata`
> 3. Check that the format is `h5ad`
>
> > <tip-title>Chromosome selection</tip-title>
> >
> > In this example data, our reads are from the *chr21*. Hence, we input `21` for chromosome ids. For human or mouse genomes, directly select from the dropdown list. Make sure that all the canonical chromosomes are present in the fragments file.
> >
> {: .tip}
>
{: .hands_on}

Because the `AnnData` format is an extension of the HDF5 format, i.e. a binary format, an `AnnData` object can not be inspected directly in Galaxy by clicking on the {% icon galaxy-eye %} (**View data**) icon. Instead, we need to use a dedicated tool from the **AnnData** suite.

> <hands-on-title>Inspect an AnnData object</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `General information about the object`
>
> 2. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AnnData object with n_obs × n_vars = 27388 × 1046
>    > ```
>    >
>    > 1. How many observations are there? What do they represent?
>    > 2. How many variables are there? What do they represent?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 27,388 observations, representing the cells.
>    > > 2. There are 1046 variables, representing the peaks.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
>    > <comment-title>Faster Method for General Information</comment-title>
>    >
>    > * To view general information of any *Anndata* file:
>    >    * Click on the name of the dataset in the history to expand it.
>    >    * General Anndata information would be given in the expanded box:
>    >
>    >      e.g.
>    >
>    >      ```
>    >      [n_obs x n_vars]
>    >      -    27388 x 1046
>    >      ```
>    > * For more specific queries, {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} is required.
>    {: .comment}
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > 	chr21_10272276_10272566	chr21_10330743_10330987	chr21_10489276_10489476	chr21_13979698_13980574	chr21_14027127_14027558 ...
>    > AAAAAATACCCGAAGG	0.0	0.0	0.0	0.0
>    > AAAAACAAAAATATAC	0.0	0.0	0.0	0.0
>    > AAAAATAACAATGATA	0.0	0.0	0.0	0.0
>    > AAAACCGTGAGAGTCT	0.0	0.0	0.0	0.0
>    > AAAACCTCACCCAACA	0.0	0.0	0.0	0.0
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is a table with 27,388 lines (observations or cells) and 1046 columns (variables or peaks): the count matrix for each of the 1046 peaks and 27,388 cells. The 1st row contains the peak location as an annotation of the columns and the 1st column the barcodes of the cells as an annotation of the rows.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 5. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
> 6. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > AAAAAATACCCGAAGG
>    > AAAAACAAAAATATAC
>    > AAAAATAACAATGATA
>    > AAAACCGTGAGAGTCT
>    > AAAACCTCACCCAACA
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is tabular with annotations of the observations. i.e. the cells. Here there are only the barcodes as annotation, so only one column, being also the index for the count matrix.
>    > >
>    > {: .solution}
>    >
>    {: .question}
>
> 7. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k chr21 Anndata`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 8. Inspect the generated file
>
>    > <question-title></question-title>
>    >
>    > ```
>    > chr21_10272276_10272566
>    > chr21_10330743_10330987
>    > chr21_10489276_10489476
>    > chr21_13979698_13980574
>    > chr21_14027127_14027558
>    > ```
>    >
>    > What is stored in the generated file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file is tabular with annotations of the variables, i.e. the peaks.
>    > >
>    > {: .solution}
>    >
>    {: .question}
{: .hands_on}

# Producing a Quality Count Matrix
We just created a count matrix with all detected peaks and non-empty barcodes. In this section of the tutorial, we will filter the features and barcodes by a series of quality assessments to generate a high-quality count matrix. The final count matrix can be used for further downstream analysis like clustering, differential chromatin analysis and cell type annotation.

The current matrix contains peaks from chromosome 21 only and is too small to carry out this type of analysis. From this step onwards we will use the full count matrix generated from the original FASTQ files. Please import the following anndata file to start:

```
https://zenodo.org/record/7855968/files/atac_pbmc_1k_uniq_peaks.h5ad
```

> <question-title></question-title>
>
> 1. How many barcodes and features are there?
> 1. What do you observe in comparison to a scRNA-seq count matrix?
>
> > <solution-title></solution-title>
> >
> > 1. There are 441038 barcodes and 70120 features
> > 2. The initial number of barcodes in this raw count matrix is in the same range as the scRNA-seq data. But there is roughly double the number of features compared to a scRNA-seq dataset. The number of features varies from dataset to dataset. Based on the tools and parameters used for peak calling, the size of the initial feature set can grow up to several hundred of thousands.
> >
> {: .solution}
>
{: .question}

## Initial filtering to remove potential empty features and cells
First remove any potential empty features or barcodes. A non-empty cell should have a minimum number of non-empty features and a non-empty feature should be present in a minimum number of non-empty cells. Here we will different minimum thresholds for filtering cells and features.

> <hands-on-title>Initial filtering</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: `atac_pbmc_1k_uniq_peaks.h5ad`
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of features expressed, using 'pp.filter_cells'`
>    - *"Filter"*: `Minimum number of features expressed`
>    - *"Minimum features"*: `100`
>
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: Output of the previous (`pp.filter_cells`) step
>    - *"Method used for filtering"*: `Filter features based on counts and numbers of features expressed, using 'pp.filter_features'`
>    - *"Filter"*: `Minimum number of cells expressed`
>    - *"Minimum features"*: `1`
>
> > <question-title></question-title>
> >
> > How much data was filtered out?
> >
> > > <solution-title></solution-title>
> > >
> > > The resulting matrix has dimensions of 1815 x 67766, i.e., more than 99.5% of the cells and less than 4% of features were filtered out. This indicates the high sparsity of the count matrix.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

## Quality control and visualization
We will first plot the number of features per cell. This information is stored as `nb_features` layer of observations. As we are dealing with numbers with huge variances, it is sometimes hard to visualize them. Hence, we compute the *log* of `nb_features` first then plot the absolute and log values together.

> <hands-on-title>Violing plots of number of features</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: Output of previous (`pp.filter_features`) step
>    - *"Method used for filtering"*: `Compute log10 of nb_features`
> 2. Check if observations have an extra `log_nb_features` in addition to `nb_features` layer.
> 3. Rename the resulting dataset to `PBMC 1k after initial filtering`
> 3. {% tool [Plot with scanpy](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PBMC 1k after initial filtering`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>      - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>        - *"Keys for accessing variables"*: `nb_features, log_nb_features`
>      - In *"Violin plot attributes"*:
>        - *"Add a stripplot on top of the violin plot"*: `Yes`
>          - *"Add a jitter to the stripplot"*: `Yes`
>            - *"Size of the jitter points"*: `0.4`
>        - *"Display keys in multiple panels"*: `Yes`
>      - In *"Parameters for seaborn.violinplot"*:
>        - *"Color for all of the elements"*: `DarkCyan`
>
> > <question-title></question-title>
> >
> > What trends of the data do you observe in the plots?
> >
> > > <solution-title></solution-title>
> > > ![IntialViolin]({% link topics/single-cell/images/scatac-pre-processing/violin_after_initial_filtering.png %} "Violin plot of number of features per cell after initial filtering")
> > > Both plots indicate that there are two distinct sets of cells:
> > >    1. Cells with an acceptable number of open chromatin regions.
> > >    2. Cells with only a few detected peaks. These cells are uninformative and should be excluded.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

To determine decent filtering thresholds, we will further look at some histograms of cell-peak coverage trends. We will plot histograms of the number of open features per cell and feature commonness in cells.

> <hands-on-title>Coverage cells</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage cells: Histogram of the number of open features (in the case of ATAC-seq data) per cell, using 'pp.coverage_cells'`
>    - *"Binarized matrix?"*: `No`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `1000`
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage cells: Histogram of the number of open features (in the case of ATAC-seq data) per cell, using 'pp.coverage_cells'`
>    - *"Binarized matrix?"*: `No`
>    - *"Log transform?"*: `Yes`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `1000`
> 3. Now observe the plots
>
> > <question-title></question-title>
> >
> > 1. What information do you see in these plots?
> > 1. Do you think our minimum number of features threshold of 1000 indicated in the plot makes sense?
> >
> > > <solution-title></solution-title>
> > > ![CoverageCells]({% link topics/single-cell/images/scatac-pre-processing/coverage_cells.png %} "Histogram of the number of open chromatin regions per cell")
> > > ![CoverageCellsLog]({% link topics/single-cell/images/scatac-pre-processing/coverage_cells_log.png %} "Histogram of the number of open chromatin regions per cell in log scale")
> > > 1. The plots show the number of open chromatin regions per cell in 50 bins on the x-axis and the number of cells per bin on the y-axis. It is basically a histogram of the number of open chromatin regions per cell.
> > > 2. From the first plot with absolute counts, we see that with our threshold of 1000, we can separate between the cells with few detected peaks and cells with high chromatin accessibility.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

> <hands-on-title>Coverage features</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage features: Distribution of the feature commonness in cells, using 'pp.coverage_features'`
>    - *"Binarized matrix?"*: `No`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `5`
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix""*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Coverage features: Distribution of the feature commonness in cells, using 'pp.coverage_features'`
>    - *"Binarized matrix?"*: `No`
>    - *"Log transform?"*: `Yes`
>    - *"Minimum number of cells or minimum number of features to be indicated in the plot"*: `5`
> 3. Now observe the plots
>
> > <question-title></question-title>
> >
> > 1. What information do you see in these plots?
> > 1. Do you think our minimum number of cells threshold of 5 indicated in the plot makes sense?
> >
> > > <solution-title></solution-title>
> > > ![CoverageFeatures]({% link topics/single-cell/images/scatac-pre-processing/coverage_features.png %} "Histogram of feature commonness in cells")
> > > ![CoverageFeaturesLog]({% link topics/single-cell/images/scatac-pre-processing/coverage_features_log.png %} "Histogram of feature commonness in cells in log scale")
> > > 1. The plots show a histogram of the number of cells sharing a feature. As we initially pooled the data from all the cells to detect the peaks, it is expected to see only a small number of cells have more than 10000 peaks in common.
> > > 2. The red vertical line of our 5 cells threshold is nearly at the left end of the histogram representing the majority of the features have at least 5 cells in common.
> > > From the log scale plot it is also clear that there is a sharp increase in the feature commonness from at least 10 cells (x-axis 1.0).
> > > So our threshold of 5 is a decent cutoff for filtering out the features. From the plots, only a very few non-informative features are left to be filtered out.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

## Final filtering to acquire quality cells and features

Based on the above QC plots, we will filter out all the cells with less than 1000 features and all the features that have less than 5 cells in common. Additionally, we will also filter out extreme outlier cells (which can be seen in the initial violin plot or coverage cells plot) with more than 12500 features.

> <hands-on-title>Final filtering based on QC plots</hands-on-title>
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: `PBMC 1k after initial filtering`
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of features expressed, using 'pp.filter_cells'`
>    - *"Filter"*: `Minimum number of features expressed`
>    - *"Minimum features"*: `1000`
>
> 1. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: Output of previous (`pp.filter_cells`) step
>    - *"Method used for filtering"*: `Filter cell outliers based on counts and numbers of features expressed, using 'pp.filter_cells'`
>    - *"Filter"*: `Maximum number of features expressed`
>    - *"Minimum features"*: `12500`
>
> 2. {% tool [scATAC-seq Preprocessing with EpiScanpy](toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1) %} with the following parameters:
>    - *"Annotated data matrix"*: Output of previous (`pp.filter_cells`) step
>    - *"Method used for filtering"*: `Filter features based on counts and numbers of features expressed, using 'pp.filter_features'`
>    - *"Filter"*: `Minimum number of cells expressed`
>    - *"Minimum features"*: `5`
>
> > <question-title></question-title>
> >
> > What are the dimensions of the count matrix after the final filtering?
> >
> > > <solution-title></solution-title>
> > >
> > > The resulting matrix has 1024 cells and 67719 peaks. The number of cells after filtering is close to what we expect in this dataset.
> > >
> > {: .solution}
> >
>    {: .question}
{: .hands_on}

# Conclusion


In this tutorial, we have learned how to map reads from 10X ATAC-seq data, identify open chromatin regions from aggregated cell data and generate a count matrix in *Anndata* format. We also performed quality control and filtered out empty and uninformative cells and features to generate a high-quality count matrix for downstream analysis.

Galaxy workflows for generating [Annadata from FASTQ files](workflows/scATAC-seq-FASTQ-to-Count-Matrix.ga) and [quality filtering of raw count matrix](workflows/scATAC-seq-Count-Matrix-Filtering.ga) are provided.scatac_tenx.ga).

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %}).

tools_json:
["Sinto barcode", "toolshed.g2.bx.psu.edu/repos/iuc/sinto_barcode/sinto_barcode/0.9.0+galaxy1", "FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.73+galaxy0", "BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.2", "Map with BWA-MEM", "Sinto fragments", "toolshed.g2.bx.psu.edu/repos/iuc/sinto_fragments/sinto_fragments/0.9.0+galaxy1", "bedtools SortBED", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_sortbed/2.30.0+galaxy2", "MACS2 callpeak", "toolshed.g2.bx.psu.edu/repos/iuc/macs2/macs2_callpeak/2.1.1.20160309.6", "Unique occurrences of each record", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0", "Build count matrix with EpiScanpy", "toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_build_matrix/episcanpy_build_matrix/0.3.2+galaxy1", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1", "scATAC-seq Preprocessing with EpiScanpy", "toolshed.g2.bx.psu.edu/repos/iuc/episcanpy_preprocess/episcanpy_preprocess/0.3.2+galaxy1", "Plot with scanpy", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy0"]


topic: single-cell
tutorial: bulk-deconvolution-evaluate
title: Evaluating Reference Data for Bulk RNA Deconvolution
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["transcriptomics"]
content_hash: 93bd127fdb7cb1f8816a6714eb29d075
last_modified: 2026-05-27T09:01:49.737866
zenodo_link: https://zenodo.org/records/5719228
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-deconvolution-evaluate/tutorial.html

content:
There are various methods to estimate the proportions of cell types in bulk RNA data. Since the actual cell proportions of the data are unknown, how do we know if our tools are producing accurate results?

In this tutorial we will be using single-cell data with known cell-type proportions in order to create pseudo-bulk RNA data. We will then estimate the cell-type proportions of this pseudo-bulk data using the currently available deconvolution tools within Galaxy. Since we know the true proportion values, we will be able to measure and compare the accuracy of the tools' predictions.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get the single-cell data

First we need to create a new history in Galaxy and load in our single-cell data. We are going to use the single-cell dataset from a previous deconvolution tutorial found here: [https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music/tutorial.html]({% link topics/single-cell/tutorials/bulk-music/tutorial.md %}).

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Evaluating Reference Data"*
> 2. Import the files from [Zenodo](https://zenodo.org/records/5719228) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    * Human pancreas single-cell RNA datasets (tag: `#scrna`)
>      ```
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.expression.tabular
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.phenotype.tabular
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 4. Check the datatypes are `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add a `#metadata` tag to `EMTABesethealthy.phenotype.tabular` and a `#expression` tag to `EMTABesethealthy.expression.tabular`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

## Inspecting the single-cell data

Before continuing lets quickly inspect our single-cell data. We can find all of the cell types present in the data alongside their proportions by using the count tool to count the occurrence of each cell type category in the metadata file.

> <hands-on-title>Get cell counts</hands-on-title>
>
> 1. {% tool [Count](Count1) %} with the following parameters:
>    - {% icon param-file %} *"from dataset"*: `EMTABesethealthy.phenotype.tabular`
>    - *"Count occurrences of values in column(s)"*: `Column 5`
>    - *"Delimited by"*: `Tab`
>    - *"How should the results be sorted?"*: `With the most common value first`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Cell type counts`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

We can see from the output table below, there are various cell types present in the data. Note that many of the cell types have very low proportion values, this should be kept in mind later on as cell types that appear only a hand full of times (or even just once!) in the data may not be very useful and only add noise.

| Cell Type               | Count |
|-------------------------|-------|
| alpha                   | 443   |
| beta                    | 171   |
| ductal                  | 135   |
| acinar                  | 112   |
| gamma                   | 75    |
| delta                   | 59    |
| unclassified endocrine  | 29    |
| co-expression           | 26    |
| PSC                     | 23    |
| endothelial             | 13    |
| epsilon                 | 5     |
| mast                    | 4     |
| unclassified            | 1     |
| MHC class II            | 1     |

> <question-title>Inspecting the single-cell data</question-title>
>
> 1. How many cells are in the single-cell data?
> 2. How many cell types are present in the data?
>
> > <solution-title></solution-title>
> >
> > 1. Inspecting the general information of `EMTABesethealthy.expression.tabular` we can see that there are **1,097** cells in the data as there are 1,098 columns (we need to subtract 1 for the header).
> > 2. Looking at the output of the {% tool [Count](Count1) %} tool (or the above table), there are **14** distinct cell types in the data.
> >
> {: .solution}
>
{: .question}

# Process the single-cell data

In order to get a good understanding of the accuracy of our deconvolution tools, we are going to run our evaluations multiple times. This approach ensures that a single good or bad evaluation does not disproportionately represent the tool's overall performance.

However, instead of running all of our tools multiple times for each evaluation (which would be quite time consuming!), we will leverage "batch computation" in Galaxy. By storing our data in collections, any tools or workflows that use those collections will automatically run multiple times (once for each element in the collection). We will now perform some pre-processing of our data to get it into the right format.

## Transpose expression matrix

If we inspect the expression data file downloaded earlier, we can see that currently the rows represent genes and columns represent cells. However, this needs to be swapped for the later workflows. To fix this we will transpose the expression matrix.

> <hands-on-title>Transpose expression matrix</hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `EMTABesethealthy.expression.tabular`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Transposed expression matrix`
>
{: .hands_on}

## Generate batch mode collections

For this tutorial we will run the evaluations **20** times, this will both help improve the sample size and allow us to determine the consistency of the tools, whilst being small enough to run in a reasonable amount of time.

We will now duplicate our single-cell data 20 times and store it in a collection. This will be done for both the expression data and metadata files.

> <hands-on-title>Generate collections from data</hands-on-title>
>
> 1. {% tool [Duplicate file to collection](__DUPLICATE_FILE_TO_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset"*: `EMTABesethealthy.phenotype.tabular`
>    - *"Size of output collection"*: `20`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Metadata`
>
> 3. {% tool [Duplicate file to collection](__DUPLICATE_FILE_TO_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset"*: `Transposed expression matrix`
>    - *"Size of output collection"*: `20`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Expression data`
>
{: .hands_on}


# Create pseudo-bulk and actual cell proportions

We are now going to run our first workflow! This workflow will extract a subsample from the data containing 200 random cells. The workflow will then perform two things with this subsample:

1. Count the cell types and proportions of the data in order to be used as reference later against the predicted proportion values
2. Remove the cell types and convert the single-cell data into pseudo-bulk data to be later inputted into the deconvolution tools.

The above will be done twice to emulate multiple "subjects". Since the deconvolution tools will be expecting the bulk-RNA data to comprise of at least 2 subjects (each with their own bulk data). For this tutorial our subjects will simply be called **A** and **B**. However, in the real world these subjects could be different patients, tissue samples, diseased/healthy, etc.

> <comment-title>Different Results</comment-title>
> Note that since we are selecting 20 samples, each containing 200 randomly selected cells. The plots and results presented in this tutorial will differ from your own. There will be some similarities such as certain cells being in higher proportion to others but the exact values with differ!
{: .comment}

**Remember** since we have a collection of 20 inputs, the output of this workflow will be a collection of 20 elements, each corresponding to the input elements. Each output will have its own random selection of 200 cells.

> <hands-on-title>Run pseudo-bulk and actual proportions workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-1-create-data.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow pseudobulk and actual proportions** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Metadata"*: `Metadata`
>    - {% icon param-collection %} *"Expression Data"*: `Expression Data`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
> 3. Add a tag labelled `#A` to the first "Actual cell proportions" and "Pseudobulk" collections
>
> 4. Add a tag labelled `#B` to the second "Actual cell proportions" and "Pseudobulk" collections
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=c7936e9378155efe&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

The output of this workflow will be the psuedo-bulk and actual cell proportions for both samples A and B. If you inspect one of the elements in the `Actual Cell Proportions` collection, you should see a table similar to the following:

|                         | A_actual   |
|-------------------------|------------|
| acinar                  | 0.090000   |
| alpha                   | 0.415000   |
| beta                    | 0.170000   |
| co-expression           | 0.050000   |
| delta                   | 0.070000   |
| ductal                  | 0.105000   |
| endothelial             | 0.015000   |
| gamma                   | 0.050000   |
| mast                    | 0.010000   |
| unclassified endocrine  | 0.025000   |

Comparing the above table with the cell-type counts of the original single-cell data, does this look correct? Well the top 3 cell-types with the highest proportion in the single-cell data are: alpha, beta, ductal. Which aligns with the proportion values of the above data! There may be some variance due to the randomly selected cells. Also note that some of the lesser common cell types (like `MHC class II`) aren't present in the above table, again this is due to the 200 randomly selected cells for this specific sample and isn't of concern.

# Perform deconvolution on the pseudo-bulk data

Now that we have our pseudo-bulk data alongside the actual proportion values. Our next step is to run deconvolution to get predicted cell-type proportions! Currently, Galaxy contains two tools for performing deconvolution: **MuSiC** and **NNLS**. We will use both of these tools in this tutorial and compare their results together.

## Generate expression set objects

First we will need to use the single-cell data to build an expression set object, which will be used in the following workflow to perform deconvolution.

**Note: We are using the original imported data here, not the transposed data or collections.**

> <hands-on-title>Build the Expression Set object</hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy3) %} with the following parameters:
>    - *"Awk- actual header (A)"*: `BEGIN { print "A_actual\tcell_type" } { print $0 }`
>    - *"Awk - infer header (A)"*: `BEGIN { print "0\tA_infer\t0" } NR > 1 {print $0 }`
>    - {% icon param-file %} *"Assay Data"*: `EMTABesethealthy.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `EMTABesethealthy.phenotype.tabular` (Input dataset)
>    - *"Awk- actual header (B)"*: `BEGIN { print "B_actual\tcell_type" } { print $0 }`
>    - *"Awk - infer header (B)"*: `BEGIN { print "0\tB_infer\t0" } NR > 1 {print $0 }`
>
>    > <comment-title></comment-title>
>    >
>    > An ExpressionSet object has many data slots, the principle of which are the experiment data (*exprs*), the phenotype data (*pData*), as well metadata pertaining to experiment information and additional annotations (*fData*).
>    {: .comment}
>
{: .hands_on}

Similar to the expression data, this ExpressionSet object needs to be duplicated 20 times into a collection for later batch processing.

> <hands-on-title>Generate ESet collection</hands-on-title>
>
> 1. {% tool [Duplicate file to collection](__DUPLICATE_FILE_TO_COLLECTION__) %} with the following parameters:
>    - {% icon param-file %} *"Input Dataset"*: `RData ESet Object` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Size of output colection"*: `20`
>
> 2. **Rename** {% icon galaxy-pencil %} output `ESet Object`
>
{: .hands_on}

## Run the Workflow

The following workflow will take the two pseudo-bulk samples (A and B), as well as the original single-cell data as reference and output the deconvolution results for both samples and deconvolution methods. Thus producing 4 output collections. The pdf results of the deconvolution tools will also be outputted from the workflow but won't be needed for the tutorial.

> <hands-on-title>Run inferring cellular proportions workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-2-deconv.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow inferring cellular proportions** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Pseudobulk - A"*: `Pseudobulk (#A)`
>    - {% icon param-collection %} *"Pseudobulk - B"*: `Pseudobulk (#B)`
>    - {% icon param-collection %} *"ESet Reference scRNA-seq"*: `ESet Object`
>    - *"Cell Types Label from scRNA dataset"*: `cellType`
>    - *"Samples Identifier from scRNA dataset"*: `sampleID`
>    - *"Cell types to use from scRNA dataset"*:`alpha,beta,ductal,acinar,gamma,delta,unclassified endocrine,co-expression,PSC,endothelial,epsilon,mast,unclassified,MHC class II`
>    - {% icon param-collection %} *"Actual - B"*: `Actual cell proportions (#B)`
>    - {% icon param-collection %} *"Actual - A"*: `Actual cell proportions (#A)`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=cb978151b1686b71&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

# Visualise results

Now that we have our deconvolution results, the next step is to analyse the predictions and determine how accurate our tools are given our reference data. Since our pseudo-subjects **A** and **B** come from the same data, there isn't much point inspecting them both. So for the rest of the tutorial we will just focus our analysis on subject **A**.

In order to determine if our tools have produced accurate results, we will create various plots and compute different metrics to visualise and quantify the outputs of our tools.

## Pre-process the output results

Before visualising or inspecting the outputs of the deconvolution tools, we first need to perform some pre-processing. Up until now we have been working with collections in order to perform our evaluations multiple times in parallel. However, for analysing our data, collections will be a bit messy and are no longer needed. The following workflow will combine all the collections of the MuSiC and NNLS outputs into two tables:

1. A results table presenting the predicted and actual proportion values of each cell-type of each subsample
2. An error table showing the difference between the actual and predicted values. Which will be needed for a later plot.

> <hands-on-title>Run visualisation pre-processing workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-3-process.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow preprocess visualisations** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `A - Music Results`
>
> 3. Run **Workflow preprocess visualisations** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `B - NNLS Results`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=76d3408d0d22ad05&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

The following table shows a snippet of the `Results Table` for the MuSiC tool. A header has been added for better reading but has been omitted in the workflow output as it will interfere with the visualisation tools.

| Cell Type      | Actual Proportion   | Predicted Proportion   |
|----------------|---------------------|------------------------|
| acinar         | 0.090000            | 0.0814442584577275     |
| alpha          | 0.415000            | 0.427718807911522      |
| beta           | 0.170000            | 0.256954867012044      |
| co-expression  | 0.050000            | 0                      |
| delta          | 0.070000            | 0.0929840465107452     |
| ...            | ...                 | ...                    |

Already at first glance we can see some interesting results! Firstly we can see that the tool is able to make predictions close to the actual values such as with `acinar, alpha, delta`. We also see the tool failing to make any type of prediction for `co-expression` cells with a predicted proportion value of 0. This however isn't a compete surprise since `co-expression` cells are of small proportion in the bulk and reference data.

But this is only a small sample of the results. Lets create some visualisations to see the whole picture!

## Plot scatter plots of the results

The first type of visualisation we will do is a scatter plot. This plot will compare the actual and predicted proportion values for each cell across each subsample. We will also colour each point on the plot to indicate which cell type it belongs to. Let's do that now for both the MuSiC and NNLS results.

{% snippet faqs/galaxy/tools_rerun.md %}

> <hands-on-title>Plot the actual and inferred data</hands-on-title>
>
> 1. {% tool [Scatterplot with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `Results Table (Music)`
>    - *"Column to plot on x-axis"*: `2`
>    - *"Column to plot on y-axis"*: `3`
>    - *"Plot title"*: `Correlation between inferred and actual cell-type proportions`
>    - *"Label for x axis"*: `Actual proportions`
>    - *"Label for y axis"*: `Inferred proportions`
>    - In *"Advanced options"*:
>       - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>           - *"column differentiating the different groups"*: `1`
>           - *"Color schemes to differentiate your groups"*: `Paired - predefined color pallete (discrete, max=12 colors)`
>           - *"Reverse color scheme"*: `Default order of color scheme`
>    - In *"Output options"*:
>       - *"width of output"*: `6.0`
>       - *"height of output"*: `4.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Scatter plot - Music`
>
> 3. Add a `#plot` tag to `Scatter plot - Music`
>
> 4. {% tool [Scatterplot with ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `Results Table (NNLS)`
>    - *"Column to plot on x-axis"*: `2`
>    - *"Column to plot on y-axis"*: `3`
>    - *"Plot title"*: `Correlation between inferred and actual cell-type proportions`
>    - *"Label for x axis"*: `Actual proportions`
>    - *"Label for y axis"*: `Inferred proportions`
>    - In *"Advanced options"*:
>       - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>           - *"column differentiating the different groups"*: `1`
>           - *"Color schemes to differentiate your groups"*: `Paired - predefined color pallete (discrete, max=12 colors)`
>           - *"Reverse color scheme"*: `Default order of color scheme`
>    - In *"Output options"*:
>       - *"width of output"*: `6.0`
>       - *"height of output"*: `4.0`
>
> 5. **Rename** {% icon galaxy-pencil %} output `Scatter plot - NNLS`
>
> 6. Add a `#plot` tag to `Scatter plot - NNLS`
>
{: .hands_on}

The output of this tool should produce two scatter plots that looks like the image below. Each point on the plot represents a cell-type for a specific subsample, so there should be 20 points of each colour (one for each subsample created earlier). Since we are comparing the actual and inferred proportions, the ideal scatter plot would have all of the points be at the `y=x` line. The further the deviations are from this ideal line, the less accurate the tool is. We can also use this plot to determine if the tool is under or over predicting proportion values for each cell-type, or if the tool is struggling to predict certain cell types.

![Scatter plot MuSiC](../../images/bulk-deconvolution-evaluate/scatterplot-music.png "Scatter plot of Music results")

> <question-title>Interpreting the Scatter Plots</question-title>
>
> 1. Which method has the most accurate results?
> 2. Which cell type has the biggest proportion in the dataset?
> 3. Do either of the tools struggle with any cell types?
>
> {% snippet faqs/galaxy/features_scratchbook.md datatype="tabular" %}
>
> > <solution-title></solution-title>
> >
> > ![Scatter plot comparison](../../images/bulk-deconvolution-evaluate/scatterplot-compare.png "Scatter plot comparison between Music and NNLS")
> >
> > 1. Comparing scatter plots, the MuSiC tool has the most accurate results since the points fall closer onto the x=y line
> > 2. Both scatter plots show `alpha` cells having the highest proportion by a large margin
> > 3. The MuSiC tool seems to handle all cell types well. However, NNLS appears to struggle predicting the proportions of beta cells, with many of the samples being predicted as having a proportion of 0
> >
> {: .solution}
>
{: .question}


## Plot violin plots of the errors

Next we will plot the distribution of errors between the predicted and actual cellular proportions for a select number of cell types. We could plot all cell types in the output, however too many will cause the visualisations to be messy and difficult to interpret.

We can use the cell-type counts we computed at the beginning of the tutorial to determine the best cell types to use. We will use the top 5 most abundant cell types in the single-cell data being: `alpha, beta, gamma, ductal, acinar`. Before plotting we will extract only these cell types from our table of errors.

{% snippet faqs/galaxy/tools_rerun.md %}

> <hands-on-title>Extract Cell Types</hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `Error Table (Music)`
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>       - *"Delimited by"*: `Tab`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>           - *"List of Fields"*: `Select the columns containing: alpha, beta, gamma, ductal, acinar`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Music Errors`
>
> 3. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `Error Table (NNLS)`
>    - *"Operation"*: `Keep`
>    - *"Cut by"*: `fields`
>       - *"Delimited by"*: `Tab`
>       - *"Is there a header for the data's columns ?"*: `Yes`
>           - *"List of Fields"*: `Select the columns containing: alpha, beta, gamma, ductal, acinar`
>
> 4. **Rename** {% icon galaxy-pencil %} output `NNLS Errors`
>
{: .hands_on}

Now we have our table of errors consisting of only the top 5 cell-types, we can plot the violin plots.

{% snippet faqs/galaxy/tools_rerun.md %}

> <hands-on-title>Plot violin plots</hands-on-title>
>
> 1. {% tool [Violin plot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_violin/ggplot2_violin/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `Music Errors`
>    - *"Plot title"*: `Error Distribution`
>    - *"Label for x axis"*: `Cell Type`
>    - *"Label for y axis"*: `Difference Error`
>    - In *"Advanced Options"*:
>       - *"Violin border options"*: `Purple`
>    - In *"Output Options"*:
>       - *"width of output"*: `3.0`
>       - *"height of output"*: `2.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Violin Plot - Music`
>
> 3. Add a `#plot` tag to `Violin Plot - Music`
>
> 4. {% tool [Violin plot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_violin/ggplot2_violin/3.4.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input in tabular format"*: `NNLS Errors`
>    - *"Plot title"*: `Error Distribution`
>    - *"Label for x axis"*: `Cell Type`
>    - *"Label for y axis"*: `Difference Error`
>    - In *"Advanced Options"*:
>       - *"Violin border options"*: `Purple`
>    - In *"Output Options"*:
>       - *"width of output"*: `3.0`
>       - *"height of output"*: `2.0`
>
> 5. **Rename** {% icon galaxy-pencil %} output `Violin Plot - NNLS`
>
> 6. Add a `#plot` tag to `Violin Plot - NNLS`
>
{: .hands_on}

The output of this tool will be two violin plots that will look similar to the below image. Here we can see the distribution of errors for each cell type. Since we are using normal errors and not absolute or squared errors, we are also able to see whether the tool has under or over estimated the cell type. An ideal plot would have all the violin plots being short in height and close to 0 indicating that the estimated and actual values are close together (resulting in an error close to 0).

![Violin plot MuSiC](../../images/bulk-deconvolution-evaluate/violin-music.png "Violin plot of Music results")

> <question-title>Interpreting the Violin Plots</question-title>
>
> 1. Which method has the least errors?
> 2. Which method is the most balanced when to over and under estimating proportions?
> 3. Which is the most overestimated cell type in NNLS?
>
> {% snippet faqs/galaxy/features_scratchbook.md datatype="tabular" %}
>
> > <solution-title></solution-title>
> >
> > ![Scatter plot comparison](../../images/bulk-deconvolution-evaluate/violin-compare.png "Scatter plot comparison between Music and NNLS")
> >
> > 1. Comparing the two violin plots, MuSiC has the better error results, with more samples closer to zero. Inspecting the y-axis of the plots also show that the MuSiC errors span a smaller range compared to NNLS.
> > 2. MuSiC can be seen as having the most balanced results with the bulk of the estimates being around 0. Whereas the NNLS results show large amounts of both overestimation and underestimation of various cell types.
> > 3. From the NNLS violin plot it can be seen that ductal cells are greatly overestimated.
> >
> {: .solution}
>
{: .question}


# Compute accuracy metrics

Visualisations are a great tool for getting an intuitive overview of the data. However, some of the interpretations from visualisations can be subjective. Having quantitative results alongside visualisations can offer concrete and precise values about the data that can more easily be compared. We will use two different quantitative metrics in this tutorial; Pearson correlation and RMSE.

## Pearson Correlation

The Pearson correlation coefficient is a statistical value that represents the direction and correlation between two variables, the value of this metric ranges between -1 and 1, where:

- -1 = negative correlation
- 0 = no correlation
- 1 = positive correlation

The equation for calculating the Pearson correlation can be seen below, the workflow to compute this metric breaks down this formula into smaller steps.

![Pearson Correlation Equation](../../images/bulk-deconvolution-evaluate/pearson.png "Pearson Correlation Equation")

Where
- `x` = actual proportion values
- `x̄` = mean of actual proportion values
- `y` = predicted proportion values
- `ȳ` = mean of predicted proportion values

## Root Mean Squared Error (RMSE)

Root Mean Squared Error or RMSE is a common metric for measuring a tools prediction error. This metric calculates the average error between the predicted and actual values for each prediction then takes the mean and square root of the error to produce a final value. Lower RMSE values (close to 0) indicate accurate predictions similar to the actual value, as the value increases the accuracy score worsens.

The equation for calculating this metric is seen below, the implementation of this calculation is in the workflow alongside the Pearson correlation.

![Root Mean Squared Error Equation](../../images/bulk-deconvolution-evaluate/rmse.png "Root Mean Squared Error Equation")

Where
- `n` = number of samples
- `y` = actual proportion
- `ŷ` = predicted proportion


## Compute metrics

With a basic understanding of some useful metrics, we will now compute these to get quantitative values alongside our visualisation results. The following workflow needs to be run for both the MuSiC and NNLS results table.

> <hands-on-title>Run metrics workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>    - Copy the URL (e.g. via right-click) of [this workflow]({% link topics/single-cell/tutorials/bulk-deconvolution-evaluate/workflows/deconv-eval-stage-4-metrics.ga %}) or download it to your computer.
>    - Import the workflow into Galaxy
>
> 2. Run **Workflow compute metrics** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `Results Table (Music)`
>
> 3. Run **Workflow compute metrics** {% icon workflow %} using the following parameters:
>    - {% icon param-collection %} *"Cell Proportions"*: `Results Table (NNLS)`
>
>    {% snippet faqs/galaxy/workflows_run.md %}
{: .hands_on}

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://usegalaxy.eu/published/workflow?id=7caa58e36df6ad03&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=-20&initialY=-20&zoom=0.5"></iframe>

After running the workflow on both the MuSiC and NNLS results we should have the Pearson and RMSE metrics for both tools in various outputs. Below combines these metrics into a single summary table.

| Tool  | Pearson Correlation | RMSE  |
|-------|---------------------|-------|
| MuSiC | 0.982               | 0.022 |
| NNLS  | 0.778               | 0.678 |

From the table we can now see concrete values representing the error and correlation between the predictions and actual proportion values. We can see from the table that the MuSiC tool has a much better accuracy with a higher correlation score and lower error compared to NNLS.

The conclusions to draw from this analysis, is that our reference data is effective for use in deconvolution analysis since both tools were able to have high accuracy and low error scores. We also determined that (for at least this data) the MuSiC tool was the more effective/accurate tool and thus would likely be the more trustworthy when performing deconvolution with this single-cell reference data.

# Conclusion

{% icon congratulations %} Congratulations! You made it to the end of the tutorial!

In this tutorial we took some single-cell data with known cell-type proportions, subsampled the data, and converted them to pseudo-bulk data. We then used this pseudo-bulk data to perform deconvolution using the two tools available in Galaxy: MuSiC and NNLS. Using the known cell-type proportions we were able to analyse the predicted proportions to the ground truth in order to determine if the reference data can be used and which tool is the most effective. We used various visualisation and statistical techniques to analyse and quantify the tools accuracy, reliability, and error.

tools_json:
["Count", "Count1", "Transpose", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy1", "Duplicate file to collection", "__DUPLICATE_FILE_TO_COLLECTION__", "Construct Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy3", "Scatterplot with ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.4.0+galaxy1", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/9.3+galaxy2", "Violin plot w ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_violin/ggplot2_violin/3.4.0+galaxy1"]


topic: single-cell
tutorial: scrna-case_FilterPlotandExploreRStudio
title: Filter, plot, and explore single cell RNA-seq data with Seurat (R)
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 7cc76ac215e7e4818a2fb97f3bd30178
last_modified: 2026-05-27T09:01:49.744194
zenodo_link: https://zenodo.org/record/7053673
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_FilterPlotandExploreRStudio/tutorial.html

content:
{% snippet topics/single-cell/faqs/notebook_warning.md %}

# Setting your environment
First thing's first, we need to load the packages we will be using. In order to use any functions within a package, we must first call the library of that package. In your console (likely in the lower left corner of your RStudio window), run the following lines of code to do so:

```r
library(Matrix)
library(Seurat)
library(dplyr)
```

The packages are called--now let's get our data files moved from our Galaxy history and into the RStudio environment so that we can create a Seurat object.

# Upload, view and modify the files
Now that we've made it into RStudio and called the packages we'll use, let's begin moving datasets from Galaxy and into RStudio. Galaxy helps us with this by providing the gx_get() function, which will tell us the file paths for the datasets currently occupying our Galaxy histories.

So, for example, our matrix was the first piece of data to be imported. As such, we will ask for the file path of the first piece of data in our history: we will tell gx_get() the number of the dataset in our history that we are interested in, and the function will output the path we need to bring that dataset into R.

```r
gx_get(1)  #get the file path for matrix.mtx: #1 in Galaxy history
```

Now we have the file path! We can use the Matrix package, specifically the readMM() function, to read the counts matrix into our environment, using the file path above to let it know where to find the matrix.

```r
matrix.mtx<-readMM("/import/1")
```
Now we will do the same thing with the feature, barcode, and experimental design files.

> <warning-title>Don't skip ahead!</warning-title>
>  Don't try to skip ahead and simply fill in the position of the dataset within the readMM() function, reading in the files very often will not work without having used the gx_get() function--if it does, well, lucky you.
>
{: .warning}

We'll run the same command, this time replacing the data number in order to ask for the file path to data in an alternative slot in our Galaxy history. Then, we will use the read.delim() function in read in the list of genes, cells, and cell annotations provided to us  by the researchers:

```r
gx_get(2) #ask for the file path of the genes.tsv file
genes.tsv<-read.delim("/import/2", header = FALSE) #read in the genes file

gx_get(3) # ask for the file path of the barcodes.tsv file
barcodes.tsv<-read.delim("/import/3", header = FALSE) #read in the barcodes file

gx_get(4) #ask for the file path of the exp_design.tsv file
exp_design.tsv<-read.delim("/import/4") # read in the experimental design file
```
The format of the experimental design dataset contains cell barcodes as the first column of data with simply numbered (1-n) row names. In order for Seurat to properly use this dataset, we will need to make the cell barcodes the row names. This can be accomplished by running the following line of code:

```r
rownames(exp_design.tsv)<-exp_design.tsv$Assay
```

Now, in RStudio, we should have all of the data necessary to create a Seurat Object: the matrix, a file with feature (gene) names, a file with cell barcodes, and an optional, but highly useful, experimental design file containing sample (cell-level) metadata.

# Generating a Seurat object
Next we will add row and column names to our matrix. In the end, this will leave us with a matrix whose rows are genes, columns are cell barcodes, and each value in the matrix representing the expression value of a given gene in a given cell.

><tip-title>About Seurat Objects</tip-title>
>The order of the dimensions, such that genes are the first and cells are the second is a set characteristic of Seurat objects.
>
>You will recieve an error if you try to label the first dimension with cell barcodes or the second with genes. This is because the Dimname slots are like empty fill in the blanks: if the number of labels input doesn't match the number of blanks in that dimension (the number of labels don't match the number of cells/genes), Seurat will not accept the labels.
{: .tip}

```r
dimnames(matrix.mtx) <- list(genes.tsv$V2, barcodes.tsv$V1)
```

In a more typical Seurat pipeline, or on a local version of RStudio, this step would be replaced with a Read10x() step. Read10x() is Seurat's function to create a matrix and add in feature and barcode names simultaneously. However, due to the nature of how Galaxy histories and interactive environments communicate with one another, we'll use this manual method.

><tip-title>How to Use Read10X</tip-title>
>The only necessary parameter of Seurat's Read10X() fucntion is the file path to the directory (folder) containing the matrix.mtx, barcodes.tsv, and genes.tsv (sometimes also called features.tsv) files.
>
>So if the EBI SCXA Retrieval tool were to have output a data directory in the sixth history position, the following code would be able to import a labelled counts matrix called "labelled matrix" into our RStudio environment:
>
>```r
>gx_get(6) #get the file path
>labelled_matrix<-Read10X(dir = "/import/6")
>```
>
{: .tip}

Now we will create a Seurat object using our newly labelled counts matrix!

```r
srt<-CreateSeuratObject(counts = matrix.mtx)
```

You've created a Seurat object, congratulations!

# Adding Cell Level Metadata

Now that we have an object, we can add in the cell level metadata from our experimental design dataframe (table). This will be useful to us shortly as we begin to visualize the data!

```r
srt$Sex<-exp_design.tsv$Sample.Characteristic.sex.
srt$Organism<-exp_design.tsv$Sample.Characteristic.organism.
srt$Strain<-exp_design.tsv$Sample.Characteristic.strain.
srt$Developmental.Stage<-exp_design.tsv$Sample.Characteristic.developmental.stage.
srt$Age<-exp_design.tsv$Sample.Characteristic.age.
srt$Individual<-exp_design.tsv$Sample.Characteristic.individual.
srt$Disease<-exp_design.tsv$Sample.Characteristic.disease.
srt$Genotype<-exp_design.tsv$Sample.Characteristic.genotype.
srt$Organism.Part<-exp_design.tsv$Sample.Characteristic.organism.part.
srt$Cell.Type<-exp_design.tsv$Sample.Characteristic.cell.type.
srt$Factor.Value.Genotype<-exp_design.tsv$Factor.Value.genotype.
```
><tip-title>Syntax Lesson</tip-title>
>The code preceding the left pointing arrow will indicate where to put your metadata (the name of your new metadata column: object$new_metadata_columnname), and the code following the arrow will denote where to find that metadata information (metadata_table$columnname)
>
{: .tip}

Now that we have our almost fully annotated object, we will add one more metadata column: percent mitochondrial (perc.mt). This metadata column will denote what percentage of a cell's feature (gene) expression is mitochondrial--which will be useful to us shortly as we begin to filter cells.

```r
srt <- PercentageFeatureSet(srt, pattern = "^mt-", col.name = "perc.mt")
```

For the sake of this data set, and many others, the mitochondrial genes will all be marked with an "mt" as the prefix, so that is how we have asked Seurat's PercentageFeatureSet() function to search for mitochondrial genes in the line of code above. With that being said, when you are analyzing your own data, it is highly recommended that you identify how your data set has labelled mitochondrial genes to ensure that you are calculating the correct percentages.

> <warning-title>Careful</warning-title>
>  The "mt" prefix may not always include all mitochondrial genes depending on how your dataset has been labelled.
>
>Sometimes the labels may be done via a capital prefix ("^MT") and in some cases, mitochondrial genes must be identified via feature-level metadata like a GTF file.
{: .warning}


# QC Plots
Now that we have a complete Seurat object, we can begin the filtering process.

There will be a number of ‘cells’ that are low-quality or just empty droplets. There will also be genes which are mere sequencing artifacts or appear with such low frequency that statistical tools will fail to accurately analyse them.

This background noise of cells and genes not only makes it harder to distinguish real biological information from artifacts, but also makes the data computationally demanding to analyze.

We want to filter our cells to only include those containing informative biological information. Before doing so, we need to know what the data distribution looks like. There are a number of subjective decisions to make within scRNA-seq analysis, for instance we need to make informed decisions about where to set our thresholds (more on that soon!).

We’re going to plot our data a few different ways. Different bioinformaticians might prefer to see the data in different ways, and here we are only generating a few of the plots you can use. Ultimately you need to go with what makes the most sense to you.

So let's generate some QC plots. First off, let's check our dataset for batch effect:

```r
VlnPlot(srt, group.by = "Individual",
  features = "nCount_RNA",
  log = TRUE)
```
![Violin Plot split by Individual/Batch](../../images/scrna-SeuratRStudio/plot1.png "Violin Plot of counts split by Individual.")

This plot shows us the number of cells split by the individual (mouse) from which the cells came from. Now, depending on your experimental design, batch may be represented by something other than individual--like timepoint or even the wet lab researcher who isolated the cells.

Ideally, we'd like to see a relatively even distribution of counts for each individual (or batch) but if there isn’t, fear not, we can regress this variable out in a later step.

><tip-title>Syntax Lesson</tip-title>
>In order to accurately assess potential batch effects, use the "group.by" parameter to indicate the variable which differed across experiments.
>
{: .tip}

Now let's get an idea of how different variables, like the sex or genotype of the mice, might be represented across our dataset.

1. Sex?
    ```r
    VlnPlot(srt, group.by = "Sex",features = "nCount_RNA",log = TRUE)
    ```
    ![Violin Plot split by Sex](../../images/scrna-SeuratRStudio/plot2.png "Violin Plot of counts split by Sex.")

2. Genotype?
    ```r
    VlnPlot(srt, group.by = "Genotype", features = "nCount_RNA", log = TRUE)
    ```
    ![Violin Plot split by Genotype](../../images/scrna-SeuratRStudio/plot3.png "Violin Plot of counts split by Genotype--Mutant versus Control.")

# Finding Our Filtering Parameters
Now that we have a better understanding of what our data looks like, we can begin identifying those spurious reads and low quality cells to be removed. First, we'll plot the percent mito (perc.mt) against the cell count (nCount_RNA) to get an idea of what threshold we should set for nCount:

```r
plot(x = srt$nCount_RNA, y = srt$perc.mt, main = "UMI Counts x Percent Mito", xlab = "UMI_count", ylab = "percent mito")
```
![UMI x mito](../../images/scrna-SeuratRStudio/plot4.png "UMI counts x Percent mito.")
We are looking for cell counts with high mitochondrial percentages in their feature expression.

><comment-title>What Do High Mitochondrial Reads Mean?</comment-title>
>High mito expression will typically indicate stressed out cells (often due to the extraction, sorting, or sample prep protocols).
>
{: .comment}

These stressed out cells likely won't tell us much biologically, rather, they will contribute noise that we want to filter out of our dataset. With that being said, there is a level of metabolic activity that is expected and will be specific to your samples/tissue/organism--so it is worth looking into what that might look like when it comes time to analyze your own datasets. Asking the question "what is an expected proportion of mitochondrial expression in my sample?" can and likely will become a frustrating endeavor if you are analyzing a brand new kind of dataset. scRNA-seq analyses are inherently iterative for this exact reason, so try a threshold, label your folders well, and be prepared to return back to this question if things look strange down the line.

As a general eye-balling rule, I will look for the nCount value at which the perc.mito begind to rise. We can also zoom in on the x-axis to get a better idea of what threshold to set by adjusting the xlim parameter:

```r
plot(x = srt$nCount_RNA, y = srt$perc.mt, main = "UMI Counts x Percent Mito", xlab = "UMI_count", ylab = "percent mito", xlim = c(0,1750))
```
![UMI x mito zoomed in on X](../../images/scrna-SeuratRStudio/plot5.png "UMI counts x Percent mito-Zoomed in on X.")

><comment-title>Interpretations</comment-title>
>It looks like just before nCount_RNA = 1750, the perc.mito peaks above 2 percent--a conservative threshold.
>
{: .comment}

Now we can take a closer look at the y-axis to decide on a mito threshold to set. Once more, we want to get rid of as few cells as possible while still removing those with unexpectedly high mito percentages.

```r
plot(x = srt$nCount_RNA, y = srt$perc.mt, main = "UMI Counts x Percent Mito", xlab = "UMI_count", ylab = "percent mito", ylim = c(0,3))
```
![UMI x mito zoomed in Y](../../images/scrna-SeuratRStudio/plot6.png "UMI counts x Percent mito-Zoomed in on Y.")

><comment-title>Interpretations</comment-title>
>We can see a clear trend wherein cells that have around 3 percent mito counts or higher also have far fewer total counts. These cells are likely of low quality, will muddy our data, and were probably stressed or ruptured prior to their encapsulation within a droplet.
>
{: .comment}

Take a look at what proportion of cells those thresholds will include and disclude from our dataset:

```r
prop.table(table(srt@meta.data$nCount_RNA > 1750))
prop.table(table(srt@meta.data$perc.mt > 3))
```

If we are happy with those thresholds for cells and percent mito, we can look at the the gene count threshold next.

><comment-title>Otherwise</comment-title>
>If not, repeat the preceding steps to hone in on a threshold more suited to your needs.
>
{: .comment}

To set a threshold for gene count, let's plot the gene counts (nFeature_RNA) against the percent mito (perc.mt):

```r
plot(x = srt$nFeature_RNA, y = srt$perc.mt, main = "Gene Counts x Percent Mito", xlab = "gene_count", ylab = "percent mito")
```
![Gene x mito](../../images/scrna-SeuratRStudio/plot7.png "Gene counts x Percent mito.")

Once again, let's zoom in on the x-axis, but this time to get an idea of which nFeature_RNA threshold to set:

```r
plot(x = srt$nFeature_RNA, y = srt$perc.mt, main = "Gene Counts x Percent Mito", xlab = "gene_count", ylab = "percent mito", xlim = c(0,1275))
```
![Gene x mito--zoomed in](../../images/scrna-SeuratRStudio/plot8.png "Gene counts x Percent mito zoomed in.")

><comment-title>Interpretations</comment-title>
>You can see how cells with nFeature_RNA up to around, perhaps 575 genes, also have high perc.mt. The same can be said for cells with nFeature_RNA above 1275.
>
>We could also use the violin plots to come up with these thresholds, and thus also take batch into account. It’s good to look at the violins as well, because you don’t want to accidentally cut out an entire sample (i.e. N703 and N707 which both have cell counts on the lower side).
>
{: .comment}

Now let's take a look at what those nFeature_RNA thresholds will include and disclude from our data.

```r
prop.table(table(srt@meta.data$nFeature_RNA > 1275 | srt@meta.data$nFeature_RNA < 575))
```

# Applying our Thresholds
Once we are happy with our filtering thresholds, it’s time to apply them to our data!

><tip-title>Create a new object</tip-title>
> You will notice in the next line of code, we have indicated a new object name for this filtered (subset) data. This is good practice  so that you don't have to start all over in case you decide to change your filtering parameters (which you likely will) or if something goes awry.
>
{: .tip}

```r
subset_srt<-subset(srt, nCount_RNA > 1750 & nFeature_RNA > 1275 & perc.mt < 3 | nFeature_RNA < 600)
```

In this step we are also creating a new object (notice the new object name preceding the subset() function you just ran) so that we may compare back and forth between our unfiltered and filtered data set if we please.

Next, we want to filter out genes that no longer show any expression in the remaining high quality cells present in our dataset. In order to do so we will extract the filtered matrix from our filtered object.

```r
subset_matrix<-GetAssayData(subset_srt)
```

Since you’ve removed a whole heap of cells, and the captured genes are sporadic (i.e. a small percentage of the overall transcriptome per cell) this means there are a number of genes in your matrix that are not expressed in any of the cells left in the filtered matrix. Genes that do not appear in any cell, or even in only 1 or 2 cells, may break some analytical tools and will often not be biologically informative. So let’s remove them!

We can use the filtered matrix we extracted to create a new Seurat object, this time including the argument: min.cells = 3. This will remove any genes from our matrix that have less than 3 cells expressing them.

><tip-title>More Thresholds</tip-title>
> Note that 3 is not necessarily the best number, rather it is a fairly conservative threshold. You could go as high as 10 or more.
>
{: .tip}

```r
filtered_srt <- CreateSeuratObject(counts = subset_matrix, meta.data = subset_srt@meta.data, min.cells = 3)
```

Now that we have filtered out both noisy "cells" and genes from our dataset, let's clean up our environment. Remove objects that we no longer need to ensure that we stay organized and RStudio has enough memory capacity to perform downstream analyses. This likely will not be an issue while doing this tutorial, but in practice it will help things run smoothly.

```r
rm(subset_matrix, subset_srt)
```

# Processing
Even after filtering, we still have quite big data. This presents two concerns for downstream analyses:
 1. We already saw in our filtering plots that there are differences in how many transcripts and genes have been counted per cell. This technical variable could, and likely will, distract us from identifying true biological differences.

 2. We like to plot things on 2-dimensional (X/Y) plots. For instance, Gapdh could be on one axis, and Actin could be on another, and each cell could be plotted onto that 2D axis based on how many of each transcript they possess.

Although this plotting method would be _fine_, adding in a 3rd dimension (or, indeed, in our case, a dimension for each of the remaining thousands of genes), is a bit trickier.

So, our next steps will be to transform the large data object into something that is easy to analyse and easy to visualize.

We will run SCTransform, a combinatorial function by Seurat that normalizes the data, identifies variable features, and then scales the data. In their initial workflow, and in the Scanpy version of this tutorial, these steps are run individually. However, with the second version of SCTransform comes time efficiency and optimization for downstream analyses.

```r
filtered_srt<- SCTransform(filtered_srt, vars.to.regress = c("perc.mt", "nFeature_RNA", "nCount_RNA"), verbose = TRUE, return.only.var.genes = FALSE, seed.use = 1448145)
```
><comment-title>What is Normalization?</comment-title>
> Normalisation helps reduce the differences between gene and UMI counts by fitting total counts across cells in our data to be comparable to one another. SCTransform regularizes the gene expression profiles via a negative binomial regression while also controlling for overfitting of the data. This step can also be done using Seurat's NormalizeData() function, but would need to be followed by FindVariableFeatures() and ScaleData().
>
{: .comment}


We also have loads of genes, but not all of them vary in expression from cell to cell. For instance, housekeeping genes are defined as not changing much from cell to cell, so we could disclude these from our data to simplify our analyses.

The find variable features step within SCTransform (or Seurat's FindVariableFeatures() function) will flag genes that _do_ vary across cells, expediting future analyses and ensuring that we, and Seurat, don't waste time looking for meaningful differences where they don't exist.

Finally, SCTransform (or Seurat's ScaleData() function) will scale the data so that all genes have the same variance and a zero mean.

This is an important step to set up our data for further dimensionality reduction. It also helps negate sequencing depth differences between samples, since the gene levels across the cells become comparable.

><comment-title>Don't Worry!</comment-title>
> Note, that the differences from scaling etc. are not the values you have at the end - i.e. if your cell has average GAPDH levels, it will not appear as a ‘0’ when you calculate gene differences between clusters.
>
{: .comment}

Although we've made our expression values comparable to one another and our overall dataset less computationally demanding, we still have _way_ too many dimensions (n cells x n genes!).

Transcript changes are not usually singular--which is to say, genes function and exist in pathways and groups. It would be easier to analyse our data if we could group these differences. To address this we will run principal component analysis (PCA).

><comment-title>What is PCA?</comment-title>
>Principal components (PCs) are calculated from high dimensional data to find the most representative spread in the dataset. So in our highly variable gene dimensions, there will be one line (axis) that yields the most spread and variation across the cells. That will be our first principal component.
{: .comment}

We can calculate the first handful of principal components in our data to drastically reduce the number of dimensions:

><tip-title>Running Computationally Demanding Steps on Variable Features </tip-title>
>You'll notice that the RunPCA() function is run using the variable features from the previous step. This signficantly decreases the number of genes, and their expression changes, that must be grouped into principal components by this step.
{: .tip}

```r
filtered_srt <- RunPCA(filtered_srt, features = VariableFeatures(object = filtered_srt))
```

To visualize how our principal components (PCs) represent our data, let's create an elbow plot:

```r
ElbowPlot(filtered_srt, ndims = 50)
```
![PC Elbow Plot](../../images/scrna-SeuratRStudio/plot9.png "Elbow Plot: Varianvce Explained x PC.")

><comment-title>Interpretations</comment-title>
>We can see that there is really not much variation explained past the 9th PC. So we might save ourselves a great deal of time and muddied data by focusing on the top 10 PCs.
{: .comment}

You can also think about it like choosing a threshold of variance explained. Conservatively, 2.5 standard deviations are explained by about 10 of the PCs.

We’re still looking at around 10 dimensions at this point--likely not the easiest to visualize. To make our lives even easier, we must identify how similar a cell is to another cell, across every cell across each of these dimensions.

For this, we will use the k-nearest neighbor (kNN) graph, to identify which cells are close together and which are not.

The kNN graph plots connections between cells if their distance (when plotted in this 10 dimensional space) is amongst the k-th smallest distances from that cell to other cells. This will be crucial for identifying clusters, and is necessary for plotting a UMAP--which is what will ultimately allow us to visualize our data in 2 dimensions.

><comment-title>From UMAP developers:</comment-title>
>“Larger neighbor values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15 being a sensible default”.
{: .comment}

Let's now use the 10 PC threshold we chose from the Elbowplot and apply it to find neighbors:

```r
filtered_srt <- FindNeighbors(filtered_srt, dims = 1:10)
```

Now we can use the neighborhood graph to identify clusters of cells whose transcriptional profiles appear most similar to one another.

```r
filtered_srt <- FindClusters(filtered_srt, resolution = 0.5)
```

Unfortunately, identifying clusters is not as majestic as many biologists often think - the math doesn’t necessarily identify true clusters of cells. Every algorithm for identifying cell clusters falls short of a biologist knowing their data, knowing what cells should be where, and consequently proving it in the lab.

So, we’re going to make the best of it as a starting point and see what happens! We will define clusters from the kNN graph, based on how many connections cells have with one another. Roughly, this will depend on a resolution parameter for how granular you want to be.

><tip-title>On Clustering Resolution</tip-title>
>The resolution parameter available in the FindClusters() function allows for you, the bioinformatician, to dictate the granularity of the clusters.
>
>For example, a higher clustering resolution dictates increased granularity, and more stringent clusters. That is--cells must more closely resemble one another in order to be grouped into the same cluster than at a lower clustering resolution.
>
>In general, I find it easiest to think of a higher resolution producing more clusters and conversely, a lower resolution will produce less clusters. This parameter is a useful one that you will use often to help decipher how many true populations of cells are present in your data!
{: .tip}

Now that we have made note within our object of which cells cluster together, we can start to really visualize the data! Two major visualizations for this data type are tSNE and UMAP. We can calculate the coordinates for both prior to visualization. For tSNE, the parameter perplexity can be changed to best represent the data, while for UMAP the main change would be to change the kNN graph above itself, via the FindNeighbors() function.

><tip-title>On UMAP</tip-title>
>UMAP is the most recently developed, and most widely used dimensionality reduction for visualization of principal component data. It has been optimized since tSNE to better preserve global structure and is less computationally demanding.
>
{: .tip}

```r
filtered_srt <- RunUMAP(filtered_srt, dims = 1:10, seed.use = 1323)
```

# Let's Take a Look
Now that we have run dimensionality reduction on our dataset, it is ready for visualization. Let's take a look at what our cells look like in a UMAP projection:

```r
DimPlot(filtered_srt, reduction = "umap", label = TRUE, label.box = TRUE)+ NoLegend()
```
![DimPlot colored by 0.5 resolution cluster](../../images/scrna-SeuratRStudio/plot10.png "DimPlot colored by 0.5 resolution cluster.")
Good work! It looks like with a clustering resolution of 0.5, we are able to identify 7 clusters of cells in our data.

We can also look for expression of particular genes and see how those map to our UMAP projection. This is often useful in getting an initial understanding of which clusters might be representative of which cell types.

```r
FeaturePlot(filtered_srt, features = "Gapdh", order = TRUE)
```
![FeaturePlot: Gapdh](../../images/scrna-SeuratRStudio/plot11.png "FeaturePlot: Gapdh")

We plotted a housekeeping gene, Gapdh, so the broad expression we observe is expected.

><tip-title>Weird scale?</tip-title>
> If the scale of your data looks weird, it may be due to the DefaultAssay of your object. When we ran SCTransform, the function creates an entirely new assay within our Seurat object that includes scaled and normalized count values. This SCT Assay is what we want to visualize our expression values off of. So, if your scale is super broad, or goes negative in the case of integrated datasets, try running the following command before attempting to plot again:
>```r
>DefaultAssay(filtered_srt)<-"SCT"
>```
{: .tip}

In practice, it is helpful to plot known markers of cell types you expect to be in your dataset. This will give you a first look at how well your cells are clustered.

For example, we can plot early T-cell marker Il2ra and get an idea of which cells and/or clusters might resemble the early T-cells:

```r
FeaturePlot(filtered_srt, features = "Il2ra", order = TRUE)
```
![FeaturePlot: Aif1](../../images/scrna-SeuratRStudio/plot12.png "FeaturePlot: Il2ra")

It is a good idea, when analyzing your own data, to plot some markers of cell types you expect to be present. Later on we can also use these FeaturePlots to visualize manual annotation of clusters.

# Differential Expression Testing: Finding Markers
Because each cluster of cells was grouped based on similar transcriptomic profiles, each cluster will inherently differ from one another based on a set of "marker" genes.

Following an initial look at the DimPlots and FeaturePlots, we can take an even closer look at which genes are driving the clustering.

In order to do so we can run cluster level differential expression. First, we will need to set our object's active identity to be the clusters. This will ensure that when Seurat's differential expression function is run, the groupings of cells across which it will compare are the clusters.

><tip-title>What are Identities?</tip-title>
> Identities are, at their core, categorical metadata values. They are columns of cell-level metadata that somehow group the cells together. Examples of identities could be cluster number, cell type if/once known, genotype, etc.
>
{: .tip}

```r
Idents(filtered_srt)<- filtered_srt$seurat_clusters
```

><tip-title>Syntax Lesson</tip-title>
> There are often many different ways to get the same job done in R, but especially when manipulating Seurat objects. We could alternatively set the active identity of our object with the following line of code too:
>
>```r
>filtered_srt<-SetIdent(object = filtered_srt, value = "seurat_clusters")
>```
{: .tip}

Then, we'll run Seurat's FindAllMarkers function, which will compare each identity (cluster in this case) against every other identity within its class (all other clusters). This function of marker finding is particularly useful in identifying up, or down, regulated genes that drive differences in identity/cluster.

```r
cluster_markers<-FindAllMarkers(object = filtered_srt)
View(cluster_markers)
```

We'll use these marker lists later on to label our cell types.

We can also see which genes are differentially expressed across other variables in our metadata. For example, you can see which genes are up or down regulated across the different genotypes present in our dataset. To do so, let's first get a list of all the identity classes in our data. This information is kept in the metadata column, and any categorical variable will do. Here, let's pick genotype.

```r
metadata<-as.data.frame(filtered_srt@meta.data)
filtered_srt<-SetIdent(object = filtered_srt, value = "Genotype")
```

The "metadata" object now in your environment is a dataframe with column names representing the different identities you may choose to group your cells by when running differential expression or even when plotting Feature/DimPlots. The second line of code above will set the object's identity class to be the genotype from which the cell came from.

Now, let's see what transcripts differentiate our wildtype from our mutant cells. First, we can identify how many different genotypes are in our data:

```r
unique(filtered_srt$Genotype)
```

This output helpfully shows us what the genotypes are, and how they are labelled in our metadata. The small details, like capitalization, are important for referencing metadata information--our references must perfectly match the labelling in the object, otherwise they will not be recognized by the functions.

Now that we know how our wildtype and mutant cells have been labelled, we can use that information to directly compare the two. This time we will use a pairwise comparison method by using Seurat's FindMarkers() function (not to be confused with FindAllMarkers which has a comprehensive comparison approach):

```r
markers<-FindMarkers(object = filtered_srt, ident.1 = "wild type genotype", ident.2 = "Igf2-p0 heterozygous knockout", test.use = "wilcox")
```

The above function will find all of the differentially expressed genes between ident.1 (wildtype) and ident.2 (mutant) using the Wilcoxon test. The resulting output will show genes with positive fold changes (denoting a higher expression in the first identity--wildtype) and negative fold changes (denoting a higher expression value in the second identity--mutant).

><comment-title>On Finding Markers</comment-title>
> This same test of differential expression can be run using any identity class and any two identities within the same class. As this is a more fine tuned comparison than FindAllMarkers, it can be useful to uncover differences across specific samples.
{: .comment}

# Biological Interpretations
Now the fun bit! We can see where genes are expressed and begin considering and interpreting the biology of it. At this point, it’s really about what information you want to get from your data--the following is only the tip of the iceberg. However, a brief exploration is good, because it may help give you ideas going forward for your own data. Let's start interrogating!

Let's take another look at what our clusters look like:

```r
DimPlot(object = filtered_srt, reduction = "umap", label = TRUE, label.box = TRUE, group.by = "seurat_clusters") + NoLegend()
```
![DimPlot colored by 0.5 resolution cluster](../../images/scrna-SeuratRStudio/plot10.png "DimPlot colored by 0.5 resolution cluster.")

><comment-title>On Cluster Numbering</comment-title>
>Note that Seurat's cluster numbering is based on size alone, so clusters 0 and 1 are not necessarily related, they are just the clusters containing the most cells.
{: .comment}

It would be nice to know what these cells are. This analysis (googling all of the marker genes, both checking where the ones you know are and then going through marker tables we generated) is an adventurous task for any individual experiment, so we’re going to speed past that and nab the assessment from the original paper!

| Clusters | Markers                 | Cell Type                           |
|----------|-------------------------|-------------------------------------|
| 3        | Il2ra                   | Double negative (early T-cell)      |
| 1,2,5    | Cd8b1, Cd8a, Cd4        | Double positive (middle T-cell)     |
| 0        | Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell)|
| 4        | Itm2a                   | Mature T-cell                       |

Feel free to plot these markers onto our dataset to see where they fall. This is generally a useful method of discerning cell types and can be useful for initial annotations.

><tip-title>Plotting Markers</tip-title>
>To do so, simply use the same FeaturePlot() function we used above, but replace the feature parameter with your new marker of interest.
>```r
>FeaturePlot(object = filtered_srt, features = c("Il2ra", "Cd8b1", "Cd8a", "Cd4", "Itm2a"), order = T, ncol = 3)
>```
>![FeaturePlots of cell type markers](../../images/scrna-SeuratRStudio/plot21.png "FeaturePlots of our known cell type markers")
{: .tip}

We can then manually label the clusters in whatever way we please. [Dplyr](https://dplyr.tidyverse.org/reference/mutate.html)'s mutate() function allows us to incorporate conditional metadata. That is to say, we can ask the function to label cells based on the cluster in which they have been assigned:

```r
filtered_srt@meta.data<- mutate(filtered_srt@meta.data, celltype = case_when(
  seurat_clusters %in% c(3) ~ "Double negative (early T-cell)",
  seurat_clusters %in% c(1,2,5) ~ " Double positive (middle T-cell)",
  seurat_clusters %in% c(0) ~ "Double positive (late middle T-cell)",
  seurat_clusters %in% c(4) ~ "Mature T-cell"
))
```
Once we have labelled our clusters, we can visualize what our cell types actually look like:

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = "celltype")
```
![DimPlot colored by labelled celltype](../../images/scrna-SeuratRStudio/plot13.png "DimPlot colored by assigned cell type")

Now we can begin to feel a bit more oriented in exploring our data. The clusters are labelled with cell types, and our object has been processed enough such that we may now begin to answer some real biological questions! Now that we know what we’re dealing with, let’s examine the effect of our variable, real science!

## Keep Digging
Are there any differences in genotype? Or, a more biological question, is there an impact of growth restriction on T-cell development in the thymus? We can begin to answer this question visually by using the "split.by" parameter in Seurat's plot functions.

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = "celltype", split.by = "Genotype")
```
![DimPlot colored by labelled celltype split by genotype](../../images/scrna-SeuratRStudio/plot14.png "DimPlot colored by assigned cell typesplit by genotype")

We can see that there seems to be a decrease in cell counts across the cell types in the het mutant... INTERESTING! What next? We might look further at the transcripts present in both of those populations, and perhaps also look at the genotype marker table… So much to investigate! But before we set you off to explore to your heart’s delight, let’s also look at this a bit more technically.

# Technical Assessment
Is our analysis real? Is it correct? Well, we can assess that a little bit.

First thing's first, is there a batch effect?

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = "Individual")
```
![DimPlot colored by labelled celltype split by individual/batch](../../images/scrna-SeuratRStudio/plot15.png "DimPlot colored by assigned cell types split by individual/batch")

While some differences across batch are expected and nothing to be concerned about, the immature T-cells looks to be mainly comprised of Individual 3. There might be a bit of batch effect, so you could consider using batch correction on this dataset. However, if we focus our attention on the other cluster - mature T-cells - where there is batch mixing, we can still assess this biologically even without batch correction.

Additionally, we will also look at the confounding effect of sex:

```r
DimPlot(object = filtered_srt, reduction = "umap", group.by = c("Sex", "Individual", "Genotype"))
```
![DimPlot colored by Sex, Individual, and Genotype](../../images/scrna-SeuratRStudio/plot16.png "DimPlot colored by Sex, Individual, and Genotype")


We note that the one female knockout sample seems to be distributed in the same areas as the knockout samples at large, so luckily, this doesn’t seem to be a confounding factor and does not seem to prohibit us from learning more from this data. Ideally, this experiment would be re-run with either more female samples all around or swapping out this female from the male sample.

Are there any clusters or differences being driven by sequencing depth, a technical and random factor?

```r
FeaturePlot(object = filtered_srt, reduction = "umap", features = "nCount_SCT")
```
![FeaturePlot colored by counts](../../images/scrna-SeuratRStudio/plot17.png "FeaturePlot colored by counts")


There doesn't visually appear to be any differences in sequencing depth across the clusters, but let's check out some of those other variables we grouped by:

```r
FeaturePlot(object = filtered_srt, reduction = "umap", features = "nCount_SCT", split.by = "Individual")
```
![FeaturePlot colored by counts](../../images/scrna-SeuratRStudio/plot18.png "FeaturePlot colored by counts split by Individual")

There we go! This might explain the dramatic shift in early to middle T-Cell between wildtype and knockout cells--the leftmost early to middle T-cells simply have a higher sequencing depth represented by Individual 3 (UMIs/cell) than the ones on the right side. Well, that explains some of the sub-cluster that we’re seeing in that area (more specifically, this may account for the discernment between clusters 1, 2, and 5).

Luckily, and importantly, we don’t see the double negative or mature T-cells being similarly affected. So, although, this variable of sequencing depth, or moreso, Individual, might be something to regress out, it doesn’t seem to be impacting our dataset such that we cannot draw meaningful insights.


><tip-title>Overprocessing</tip-title>
>The less you can regress/modify your data, in general, the better--you want to stay as true as you can to the raw data, and only use math to correct your data when you really need to (and not to create insights where there are none!).
{: .tip}


Do you think we processed these samples well enough? We have seen in the previous images that these clusters are not very distinct, so we could consider stronger filtering. Let's take a look at gene expression of a gene we know should not be expressed in tCells as a sanity check:

```r
FeaturePlot(object = filtered_srt, reduction = "umap", features = "Hba-a1")
```
![FeaturePlot of Hemoglobin](../../images/scrna-SeuratRStudio/plot19.png "FeaturePlot of Hemoglobin")


Hemoglobin--a red blood cell marker that should NOT be found in T-cells--appears throughout the entire dataset in low numbers and as a likely marker of Cluster 6. This suggests that some background noise may have been introduced by the media the cells were in. We might consider in the wet lab trying to get a purer, more informative sample, with less background. Alternatively, in the dry lab, we can take advantage of techniques such as SoupX or others to remove this technical noise.

><tip-title>Removing Noise</tip-title>
>Adjusting the filtering settings (increasing minimum counts/cell, etc.) is often the place to start in the case of unexpected noise.
{: .tip}

Do you think the clustering is appropriate? i.e. are there single clusters that you think should be separate, and multiple clusters that could be combined?

```r
CellType_DimPlot<-DimPlot(object = filtered_srt, reduction = "umap", group.by = "celltype")
Cd4_FeaturePlot<-FeaturePlot(object = filtered_srt, reduction = "umap", features = "Cd4")
CellType_DimPlot | Cd4_FeaturePlot
```
![Double Positive differentiation?](../../images/scrna-SeuratRStudio/plot20.png "Double Positive differentiation?")

Important to note: just because a cluster doesn’t look like a cluster by eye, that is NOT enough to say it’s not a cluster! But looking at the biology here, we struggled to find marker genes to distinguish the double positive populations, which we know are also affected by depth of sequencing. That’s a reasonable argument that Clusters 1, 2, and 5 might not be all that different. Maybe we need more depth of sequencing across all those cells, or to compare these explicitly to each other (consider variations on FindMarkers!).

However, the late double positive cluster is both seemingly leaving the larger body of clusters and also has fewer knockout cells, so we might go and look at what those cells are expressing in the marker gene list. If we look at the mature T-cells further, we can see that their marker gene--Itm2a--is only expressed in half of the cluster. You might consider sub-clustering this to investigate further, either through changing the resolution or through analysing this cluster alone.

If we look at the differences between genotypes alone (the pseudo-bulk), we can see that many, if not most, of the genes in that list are actually ribosomal. This could be housekeeping background, it might be cell cycle related, it may be biological, or some combination of all three. You might consider investigating the cycling status of the cells, or even regressing this out (which is what the authors did).

Ultimately, there are quite a lot ways to analyse your scRNA-seq data, both within the confines of this tutorial (the many parameters that could be changed throughout) and outside of it (batch correction, sub-clustering, cell-cycle scoring, inferred trajectories, etc.) Most analyses will still yield the same general output, though: there are fewer knockout cells in the mature T-cell population, suggesting some sort of abberant development of T-cells in the Igf2-p0 hets.

Finally, we can export plots and objects from RStudio back into Galaxy. To do so, we'll use the gx_put() function provided to us by Galaxy. Let's save our Seurat object and the cell type labelled DimPlot!

```r
gx_put(filtered_srt)
gx_put(CellType_DimPlot)
```

The above functions will export your object and the plot into your Galaxy history from which you may download, share, and analyze!

Congratulations! You have interpreted your scRNA-seq data in several important ways!


topic: single-cell
tutorial: scrna-case_basic-pipeline
title: Filter, plot and explore single-cell RNA-seq data with Scanpy
difficulty: intermediate
requirements_json: []
tags_json: []
content_hash: 13b6ee4f0746885ac2f589608b262f50
last_modified: 2026-05-27T09:01:49.745791
zenodo_link: https://zenodo.org/record/15092468
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.html

content:
answer-key-2
    date: 2024-02-28

input_histories:
  - label: "UseGalaxy.eu"
    history:  https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/locked-filter-plot-and-explore-single-cell-rna-seq-data-input

questions:
- How can I assess my dataset quality?
- How do I pick thresholds and parameters in my analysis? What's a "reasonable" number, and will the world collapse if I pick the wrong one?
- How do I generate and annotate cell clusters?
objectives:
- Interpret quality control plots to inform parameter-decision making
- Repeat analysis from matrix to clustering
- Identify decision-making points
- Appraise data outputs and make informed decisions
- Explain why single cell analysis is an iterative (i.e. the first plots you generate are not final, but rather you go back and re-analyse your data repeatedly) process
time_estimation: 3H
key_points:
- Single-cell data is huge. To analyse single-cell data, you must reduce its many dimensions (cells x genes) via numerous methods
- Analysis is more subjective than we think. We need both prior, biological knowledge of the samples as well as many iterations of analysis, in order to derive novel, real biological insights

requirements:
-
    type: "internal"
    topic_name: single-cell
    tutorials:
        - scrna-case_alevin
        - scrna-case_alevin-combine-datasets
tags:
- paper-replication
- MIGHTS


contributions:
  authorship:
    - nomadscientist
  editing:
    - hexylena
    - wee-snufkin
    - nomadscientist
  testing:
    - wee-snufkin
  infrastructure:
    - pavanvidem
    - Nilchia


follow_up_training:
  -
    type: "internal"
    topic_name: single-cell
    tutorials:
        - scrna-case_JUPYTER-trajectories
        - scrna-case_monocle3-trajectories

recordings:
- captioners:
  - nomadscientist
  date: '2021-02-15'
  galaxy_version: '21.01'
  length: 30M
  youtube_id: M6iepSJh0EQ
  speakers:
  - nomadscientist

---



You've done all the work to make a single cell matrix, with mitochondrial genes flagge and buckets of cell metadata from all your variables of interest. Now it's time to fully process our data, to remove low quality cells, to reduce the many dimensions of the data that make it difficult to work with, and ultimately to try to define our clusters and to find our biological meaning and insights! There are many packages for analysing single cell data - Seurat {% cite Satija2015 %}, Scanpy {% cite Wolf2018 %}, Monocle {% cite Trapnell2014 %}, Scater {% cite McCarthy2017 %}, and so forth. We're working with Scanpy, although Galaxy has training using other packages, which you can explore on our {% icon level %} [Single-cell training topic]({% link topics/single-cell %}).

> <details-title>Where am I?</details-title>
>
> You are in one of the four tutorials associated with a Case Study, which replicates and expands on the analysis performed in a manuscript {% cite Bacon2018 %}.
>
> ![case study overview](../../images/scrna-casestudy/CS3-case_study_overview_3.png "Overview of the four parts of the case study, represented by boxes connected with noodles. There is a signpost specifying that you are currently in the first part.")
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get data

We've provided you with experimental data to analyse from a mouse dataset of fetal growth restriction {% cite Bacon2018 %}.

> <details-title>Fun facts about the dataset</details-title>
>
> We are particularly keen for learners to be able to go from raw FASTQ files all the way through analysis. We aren't handing you a curated dataset that we specially modified in order for this tutorial to work. Instead, this tutorial's input dataset is the full dataset generated from the {% icon level %} [previous tutorial in this case study]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md %}).
> - The only difference is that in that previous tutorial, we *downsampled* the datasets in so the tutorial would run faster. Our input data, however, now uses the *full dataset*, analysed exactly the same was as in the tutorial.
> - You can also find this dataset in the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/).
>
{: .details}

{% include _includes/cyoa-choices.html option1="Import History on EU server" option2="Zenodo" option3="SCXA" default="Import History on EU server"
       text="If you're on the EU server, (if your usegalaxy has an **EU** anywhere in the URL), then the quickest way to Get the Data for this tutorial is via importing a history. Otherwise, you can also import from Zenodo - it just might take a moment longer if you're in a live course and everyone is importing the same dataset at the same time! The SCXA is specifically for learners who are focusing on *Reusing public data*, so are not beginners." %}

<div class="Import-History-on-EU-server" markdown="1">

> <hands-on-title>Import History from EU server</hands-on-title>
>
> 1. Import the {% icon galaxy-history-input %} *Input history* by following the link below
>
>     {% for h in page.input_histories %}
>       [ {{h.label}} Input History]( {{h.history}} )
>     {% endfor %}
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
{: .hands_on}

</div>

<div class="Zenodo" markdown="1">

> <hands-on-title>Import from Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Batched_Object
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} the datasets `QC_Object`
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}

</div>

<div class="SCXA" markdown="1">

You can also pull the data from publicly available [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home).

> <hands-on-title>Import from the EBI Single Cell Expression Atlas</hands-on-title>
>
> 1. {% tool [EBI SCXA Data Retrieval](toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2) %} with the following parameters:
>    - **SC-Atlas experiment accession**: `E-MTAB-6945`
>
> 2. {% icon level %}**Follow tutorial to reformat dataset**: This [short tutorial]({% link topics/single-cell/tutorials/EBI-retrieval/tutorial.md %}) will show you how to use this tool and modify the output so that it's compatible with this tutorial and its workflow.
>
{: .hands_on}
</div>


# Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

{% snippet faqs/gtn/gtn_example_histories.md %}


# Quality control

You have generated an annotated AnnData object from your raw scRNA-seq fastq files. However, you have only completed a 'rough' filter of your dataset - there will still be a number of 'cells' that are actually just background from empty droplets or simply low-quality. There will also be genes that could be sequencing artifacts or that appear with such low frequency that statistical tools will fail to analyse them. This background garbage of both cells and genes not only makes it harder to distinguish real biological information from the noise, but also makes it computationally heavy to analyse. These spurious reads take a lot of computational power to analyse! First on our agenda is to filter this matrix to give us cleaner data to extract meaningful insight from, and to allow faster analysis.

## Calculate QC Metrics

To filter the object, we need to calculate some metrics for each cell and gene.

> <hands-on-title>Compute QC metrics</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Batched_Object`
>    - *"Method used for inspecting"*: `Calculate quality control metrics, using 'pp.calculate_qc_metrics'`
>      - *"Name of kind of values in X"*: `counts`
>      - *"The kind of thing the variables are"*: `genes`
>      - *"Keys for boolean columns of `.var` which identify variables you could want to control for"*: `mito`
>
> 2. Rename the generated file `QC_Object`
>
{: .hands_on}

## Inspect the AnnData Object

What has this tool calculated?

> <question-title></question-title>
>
> 1. What information is stored in your AnnData object? For example, the last tool to generate this object counted the mitochondrial associated genes in your matrix. Where is that data stored?
> 2. While you are figuring that out, how many genes and cells are in your object?
>
>    > <tip-title>Hint</tip-title>
>    > You want to use the same tool you used in the previous tutorial to examine your AnnData. Sometimes you can get the answers from *peeking* at your {% icon param-file %} AnnData object in your {% icon galaxy-history %} history, but sometimes it's not quite that simple!
>    >
>    >    > <hands-on-title>Inspecting AnnData Objects</hands-on-title>
>    >    >
>    >    > 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    >    >    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    >    >    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>    >    > 2. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    >    >    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    >    >    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>    >    {: .hands_on}
>    {: .tip}
>
> > <solution-title></solution-title>
> >
> > 1. If you examine your AnnData object, you'll find a number of different quality control metrics for:
> >  - *cells*, found in the {% icon param-file %} **Key-index observations annotation (obs)** output dataset
> >     - For example, you can find both discrete and log-based metrics for `n_genes` (how many genes are counted in a given cell), and `n_counts` (how many UMIs are counted in a given cell). This distinction between counts/UMIs or genes is because you might count multiple GAPDHs in a single cell. This would be 1 gene but multiple counts, therefore your `n_counts` should be higher than `n_genes` for an individual cell.
> >     - But what about the mitochondria?? You can also find `total_counts_mito`,  `log1p_total_counts_mito`, and `pct_counts_mito`, which has been calculated for each cell.
> >  - and *genes*, found in the {% icon param-file %} **Key-index observations variables/features (var)** output dataset.
> >     - For example, you can find `n_cells_by_counts` (number of cells that gene appears in).
> >
> > 2. There are `31670 cells` and `35734 genes` in the matrix.
> > - You can *peek* at your  {% icon param-file %} Anndata Object in your {% icon galaxy-history %} history by selecting it to reveal a drop-down window that has this same information in it.
> > - The matrix is `31670 x 35734`. This is `n_obs x n_vars`, or rather, `cells x genes`.
> >
> {: .solution}
>
{: .question}

## Generate QC Plots

We want to filter our cells, but first we need to know what our data looks like. There are a number of subjective choices to make within scRNA-seq analysis, for instance we now need to make our best informed decisions about where to set our thresholds (more on that soon!). We're going to plot our data a few different ways. Different bioinformaticians might prefer to see the data in different ways, and here we are only generating some of the myriad of plots you can use. Ultimately you need to go with what makes the most sense to you.

> <hands-on-title>Making QC visualisations - Violin Plots</hands-on-title>
>
> 1. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `sex`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_sex`
>
> 5. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `batch`
>
> 6. **Rename** {% icon galaxy-pencil %} output `Violin_log_batch`
>
{: .hands_on}

> <hands-on-title>Making QC visualisations - Scatterplots</hands-on-title>
>
> 1. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `log1p_total_counts`
>            - *"y coordinate"*: `pct_counts_mito`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Scatter_UMIxMito`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `log1p_n_genes_by_counts`
>            - *"y coordinate"*: `pct_counts_mito`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Scatter_GenesxMito`
>
> 5. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for plotting"*: `Generic: Scatter plot along observations or variables axes, using 'pl.scatter'`
>        - *"Plotting tool that computed coordinates"*: `Using coordinates`
>            - *"x coordinate"*: `log1p_n_genes_by_counts`
>            - *"y coordinate"*: `log1p_total_counts`
>            - *"Color by"*: `pct_counts_mito`
>
> 6. **Rename** {% icon galaxy-pencil %} output `Scatter_GenesxUMI`
>
{: .hands_on}

## Interpret the plots

That's a lot of information! Let's attack this in sections and see what questions these plots can help us answer.

{% snippet faqs/galaxy/features_scratchbook.md %}

> <question-title>Batch Variation</question-title>
>
> Are there differences in sequencing depth across the samples?
> 1. Which plot(s) addresses this?
> 2. How do you interpret it?
>
> > <solution-title></solution-title>
> >
> > 1. The plot `Violin_Log_Batch` will have what you're looking for!
> >     ![Violin_log_batch](../../images/scrna-casestudy/CS3-Violin_log_batch.png "Violin_Log_Batch (Raw)")
> > 2. Keeping in mind that this is a log scale - which means that small differences can mean large differences - the violin plots probably look pretty similar.
> >    - `N703` and `N707` might be a bit lower on genes and counts (or UMIs), but the differences aren't catastrophic.
> >    - The `pct_counts_mito` looks pretty similar across the batches, so this also looks good.
> >    - Nothing here would cause us to eliminate a sample from our analysis, but if you see a sample looking completely different from the rest, you would need to question why that is and consider eliminating it from your experiment!
> >
> {: .solution}
>
{: .question}

> <question-title>Biological Variables</question-title>
>
> Are there differences in sequencing depth across sex? Genotype?
> 1. Which plot(s) addresses this?
> 2. How do you interpret the `sex` differences?
> 3. How do you interpret the `genotype` differences?
>
> > <solution-title></solution-title>
> >
> > 1. Similar to above, the plots `violin - sex - log` and `violin - genotype - log` will have what you're looking for!
> >      ![Violin_Log_Sex](../../images/scrna-casestudy/CS3-Violin_log_sex.png "Violin_Log_Sex (Raw)")
> >      ![Violin_Log_Genotype](../../images/scrna-casestudy/CS3-Violin_log_genotype.png "Violin_Log_Genotype (Raw)")
> > 2. There isn’t a major difference in sequencing depth across sex, I would say - though you are welcome to disagree!
> >    - It is clear there are far fewer female cells, which makes sense given that only one sample was female. *Note - that was an unfortunate discovery made long after generating libraries. It's quite hard to identify the sex of a neonate in the lab! In practice, try hard to not let such a confounding factor into your data! You could consider re-running all the following analysis without that female sample, if you wish.*
> > 3. In `Violin - genotype - log`, however, we can see there is a difference. The `knockout` samples clearly have fewer genes and counts. From an experimental point of view, we can consider, does this make sense?
> >    - Would we biologically expect that those cells would be smaller or having fewer transcripts? Possibly, in this case, given that these cells were generated by growth restricted neonatal mice, and in which case we don’t need to worry about our good data, but rather keep this in mind when generating clusters, as we don’t want depth to define clusters, we want biology to!
> >    - On the other hand, it may be that those cells didn’t survive dissociation as well as the healthy ones (in which case we’d expect higher mitochondrial-associated genes, which we don’t see, so we can rule that out!).
> >    - Maybe we unluckily poorly prepared libraries for specifically those knockout samples. There are only three, so maybe those samples are under-sequenced.
> >    - So what do we do about all of this?
> >        - Ideally, we consider re-sequencing all the samples but with a higher concentration of the knockout samples in the library. Any bioinformatician will tell you that the best way to get clean data is in the lab, not the computer! Sadly, absolute best practice isn’t necessarily always a realistic option in the lab - for instance, that mouse line was long gone! - so sometimes, we have to make the best of it. There are options to try and address such discrepancy in sequencing depth. Thus, we’re going to take these samples forward and see if we can find biological insight despite the technical differences.
> >
> {: .solution}
>
{: .question}

Now that we've assessed the differences in our samples, we will look at the libraries overall to identify appropriate thresholds for our analysis.

> <question-title>Filter Thresholds: genes</question-title>
>
> What threshold should you set for `log1p_n_genes_by_counts`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. Any plot with `log1p_n_genes_by_counts` would do here, actually! Some people prefer scatterplots to violins.
> > ![Scatter_GenesxMito](../../images/scrna-casestudy/CS3-Scatter_GenesxMito.png "Scatter_GenesxMito (Raw)")
> >
> > 2. In `Scatter - mito x genes` you can see how cells with `log1p_n_genes_by_counts` up to around, perhaps, `5.7` (around 300 genes) often have high `pct_counts_mito`.
> >   - You can plot this as just `n_counts` and see this same trend at around 300 genes, but with this data the log format is clearer so that's how we're presenting it.
> >   - You could also use the violin plots to come up with the threshold, and thus also take batch into account. It's good to look at the violins as well, because you don't want to accidentally cut out an entire sample (i.e. N703 and N707).
> >   - Some bioinformaticians would recommend filtering each sample individually, but this is difficult in larger scale and in this case (you're welcome to give it a go! You'd have to filter separately and then concatenate), it won't make a notable difference in the final interpretation.
> >
> {: .solution}
{: .question}

> <question-title>Filter Thresholds: UMIs</question-title>
>
> What threshold should you set for `log1p_total_counts`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. As before, any plot with `log1p_total_counts` will do! Again, we'll use a scatterplot here, but you can use a violin plot if you wish!
> > ![Scatter_UMIxMito](../../images/scrna-casestudy/CS3-Scatter_UMIxMito.png "Scatter_UMIxMito (Raw)")
> >
> > 2. We can see that we will need to set a higher threshold (which makes sense, as you'd expect more UMI's per cell rather than unique genes!). Again, perhaps being a bit aggressive in our threshold, we might choose `6.3`, for instance (which amounts to around 500 counts/cell).
> >   - In an ideal world, you'll see a clear population of real cells separated from a clear population of debris. Many samples, like this one, are under-sequenced, and such separation would likely be seen after deeper sequencing!
> >
> {: .solution}
{: .question}

> <question-title>Filter Thresholds: mito</question-title>
>
> What threshold should you set for `pct_counts_mito`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. Any plot with `pct_counts_mito` would do here, however the scatterplots are likely the easiest to interpret. We'll use the same as last time.
> > ![Scatter_UMIxMito](../../images/scrna-casestudy/CS3-Scatter_UMIxMito.png "Scatter_UMIxMito (Raw)")
> >
> > 2. We can see a clear trend wherein cells that have around 5% mito counts or higher also have far fewer total counts. These cells are low quality, will muddy our data, and are likely stressed or ruptured prior to encapsulation in a droplet. While 5% is quite a common cut-off, this is quite messy data, so just for kicks we'll go more aggressive with a `4.5%`.
> >    - In general, you must adapt all cut-offs to your data - metabolically active cells might have higher mitochondrial RNA in general, and you don't want to lose a cell population because of a cut-off.
> >
> {: .solution}
{: .question}

## Apply the thresholds

It's now time to apply these thresholds to our data! First, a reminder of how many cells and genes are in your object: `31670 cells` and `35734 genes`. Let's see how that changes each time!

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **log1p_n_genes_by_counts** > `5.7`
>      - **log1p_total_counts** > `6.3`
>      - **pct_counts_mito** < `4.5%`
> - Everyone else: Choose your own thresholds and compare results!
{: .details}

### Genes/cell

> <hands-on-title>Filter cells by log1p_n_genes_by_counts</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `QC_Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_n_genes_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `greater than`
>                - *"Value"*: `5.7`
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy filter** {% icon tool %})
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_n_genes_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `20.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `Genes_Filtered_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Genes_Filtered_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype-Genes`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violin_Log_Genotype](../../images/scrna-casestudy/CS3-Violin_log_genotype.png "Raw")
> > ![Violinplot-filteronce](../../images/scrna-casestudy/CS3-Violin_log_genotype-Genes.png "1st filter - genes/cell")
> > 1. The only part that seems to change is the `log1p_n_genes_by_counts`.  You can see a flatter bottom to the violin plot - this is the lower threshold set. Ideally, this would create a beautiful violin plot because there would be a clear population of low-gene number cells. Sadly not the case here, but still a reasonable filter.
> > 2. If you *peek* at the AnnData object in your {% icon galaxy-history %}, you will find that you now have `17,104 cells x 35,734 genes`.
> >
> {: .solution}
>
{: .question}

### UMIs/cell

> <hands-on-title>Filter cells by log1p_total_counts</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Genes_Filtered_Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_total_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `greater than`
>                - *"Value"*: `6.3`
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy filter** {% icon tool %})
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `log1p_total_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `20.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `UMIs_Filtered_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `UMIs_Filtered_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype-UMIs`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violinplot-filteronce](../../images/scrna-casestudy/CS3-Violin_log_genotype-Genes.png "1st filter - genes/cell")
> > ![Violinplot-filtertwice](../../images/scrna-casestudy/CS3-Violin_log_genotype-UMIs.png "2nd filter - UMIs/cell")
> > 1. We will focus on the `log1p_total_counts` as that shows the biggest change. Similar to above, the bottom of the violin shape has flattered due to the threshold.
> > 2. You now have `8,677 cells x 35,734 genes` in the AnnData object.
> >
> {: .solution}
>
{: .question}

### % Mito/cell

> <hands-on-title>Filter cells by pct_counts_mito</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `UMIs_Filtered_Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `pct_counts_mito`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `less than`
>                - *"Value"*: `4.5`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `Mito_Filtered_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Mito_Filtered_Object`
>    - *"Method used for plotting"*: `Generic: Violin plot, using 'pl.violin'`
>        - *"Keys for accessing variables"*: `Subset of variables in 'adata.var_names' or fields of '.obs'`
>            - *"Keys for accessing variables"*: `log1p_total_counts,log1p_n_genes_by_counts,pct_counts_mito`
>        - *"The key of the observation grouping to consider"*: `genotype`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Violin_log_genotype-Mito`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violinplot-filtertwice](../../images/scrna-casestudy/CS3-Violin_log_genotype-UMIs.png "2nd filter - UMIs/cell")
> > ![Violinplot-filterthrice](../../images/scrna-casestudy/CS3-Violin_log_genotype-Mito.png "3rd filter - % count Mito/cell")
> > 1. If we carefully check the axes, we can see that the `pct_counts_mito` has shrunk.
> > 2. Your object now has `8,604 cells x 35,734 genes`.
> >
> {: .solution}
>
{: .question}

> <details-title>See every step of filtering together</details-title>
> ![Violin_Log_Genotype](../../images/scrna-casestudy/CS3-Violin_log_genotype.png "Raw")
> ![Violinplot-filteronce](../../images/scrna-casestudy/CS3-Violin_log_genotype-Genes.png "1st filter - genes/cell")
> ![Violinplot-filtertwice](../../images/scrna-casestudy/CS3-Violin_log_genotype-UMIs.png "2nd filter - UMIs/cell")
> ![Violinplot-filterthrice](../../images/scrna-casestudy/CS3-Violin_log_genotype-Mito.png "3rd filter - % count Mito/cell")
>
{: .details}

Fantastic work! However, you've now removed a whole heap of cells, and since the captured genes are sporadic (i.e. a small percentage of the overall transcriptome per cell) this means there are a number of genes in your matrix that are currently not in any of the remaining cells. Genes that do not appear in any cell, or even in only 1 or 2 cells, will make some analytical tools break and overall will not be biologically informative. So let's remove them! Note that `3` is not necessarily the best number, rather it is a fairly conservative threshold. You could go as high as 10 or more.

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Variable: **n_cells**
> - Control > `3`
> - Everyone else: Choose your own thresholds and compare results! Note if you go less than 3 (or even remove this step entirely), future tools are likely to fail due to empty gene data.
{: .details}

### Cells/gene

> <hands-on-title>Filter genes</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Mito-filtered Object`
>    - *"Method used for filtering"*: `Filter on any column of observations or variables`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `n_cells_by_counts`
>            - *"Type of value to filter"*: `Number`
>                - *"Filter"*: `greater than`
>                - *"Value"*: `3.0`
>
> 2. **Rename** {% icon galaxy-pencil %} output as `Cells_Filtered_Object`
>
{: .hands_on}

In practice, you'll likely choose your thresholds then set up all these filters to run without checking plots in between each one. But it's nice to see how they work!

We can summarise the results of our filtering:

|       | Cells | Genes |
|------ |--------------------|
| Raw | 31670    | 35734    |
| Filter genes/cell | 17104    | 35734    |
| Filter UMIs/cell | 8677    | 35734    |
| Filter mito/cell | 8604   | 35734    |
| Filter cells/gene | 8604    | 15950    |

{% icon congratulations %} Congratulations! You have filtered your object! Now it should be a lot faster to analyse and easier to interpret.

# Processing

So currently, you have a matrix that is 8604 cells by 15950 genes. This is still quite big data. We have two issues here - firstly, you already know there are differences in how many transcripts and genes have been counted per cell. This technical variable can obscure biological differences. Secondly, we like to plot things on x/y plots, so for instance *Gapdh* could be on one axis, and *Actin* can be on another, and you plot cells on that 2-dimensional axis based on how many of each transcript they possess. While that would be fine, adding in a 3rd dimension (or, indeed, in this case, 15950 more dimensions), is a bit trickier! So our next steps are to transform our big data object into something that is easy to analyse and easy to visualise.

> <hands-on-title>Normalisation</hands-on-title>
>
> 1. {% tool [Scanpy normalize](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.10.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Cells_Filtered_Object`
>    - *"Method used for normalization"*: `Normalize counts per cell, using 'pp.normalize_total'`
>        - *"Target sum"*: `10000.0`
>        - *"Exclude (very) highly expressed genes for the computation of the normalization factor (size factor) for each cell"*: `No`
>        - *"Name of the field in 'adata.obs' where the normalization factor is stored"*: `norm`
>
> 2. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy normalize** {% icon tool %})
>    - *"Method used for inspecting"*: `Logarithmize the data matrix, using 'pp.log1p'`
>
> 3. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy Inspect and manipulate** {% icon tool %})
>    - *"Function to manipulate the object"*: `Freeze the current state into the 'raw' attribute`
{: .hands_on}

Normalisation helps reduce the differences between gene and UMI counts by fitting total counts to 10,000 per cell. The subsequent log-transform (by log(count+1)) aligns the gene expression level better with a normal distribution. This is fairly standard to prepare for any future dimensionality reduction. Finally, we freeze this information in the 'raw' attribute before we further manipulate the values.

We next need to look at reducing our gene dimensions. We have loads of genes, but not all of them are different from cell to cell. For instance, housekeeping genes are defined as not changing much from cell to cell, so we could remove these from our data to simplify the dataset. We will flag genes that vary across the cells for future analysis.

> <hands-on-title>Find variable genes</hands-on-title>
>
> 1. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used for filtering"*: `Annotate (and filter) highly variable genes, using 'pp.highly_variable_genes'`
>        - *"Choose the flavor for identifying highly variable genes"*: `Seurat`
>
{: .hands_on}

> <details-title>More details on the Highly Variable Genes</details-title>
>
> Would you like to know how *many* genes were flagged as **Highly variable genes**?
>
>
> > <hands-on-title>Find the number of variable genes</hands-on-title>
> >
> >
> > 1. Select {% icon galaxy-refresh %} **Run Job Again** on the `anndata_out` (output of **Scanpy filter** {% icon tool %}) in your {% icon galaxy-history %} history
> >
> > 2. {% tool [Scanpy filter](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3) %} with the following parameters:
> >   - {% icon param-file %} *"Annotated data matrix"*: `anndata` (output of **Manipulate AnnData** {% icon tool %})
> >   - *"Method used for filtering"*: `Annotate (and filter) highly variable genes, using 'pp.highly_variable_genes'`
> >       - *"Choose the flavor for identifying highly variable genes"*: `Seurat`
> >       - *"Inplace subset to highly-variable genes?"*: {% icon galaxy-toggle %} **Yes**
> >
> {: .hands_on}
>
>
> If you peek at the output, you will see that the number of *genes* in your AnnData object has drastically reduced to around `3216` - this dataset has *only* the highly variable genes! Some people prefer to only perform analysis on this dataset, however I have found that sometimes (for various reasons) important biological marker genes get excluded. For this reason, I personally will flag highly variable genes for use in the next analytical steps, however I keep all the genes in my AnnData object so that I can check for key ones in the future.
>
> - {% icon warning %} For this tutorial, you **must** keep all genes in your AnnData object. Therefore, delete the output that contains *only* the highly variable genes from your {% icon galaxy-history %} history now.
>
>
{: .details}

Next up, we're going to scale our data so that all genes have the same variance and a zero mean. This is important to set up our data for further dimensionality reduction. It also helps negate sequencing depth differences between samples, since the gene levels across the cells become comparable. Note, that the differences from scaling etc. are not the values you have at the end - i.e. if your cell has average GAPDH levels, it will not appear as a '0' when you calculate gene differences between clusters.

> <hands-on-title>Scale data</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy filter** {% icon tool %})
>    - *"Method used for inspecting"*: `Scale data to unit variance and zero mean, using 'pp.scale'`
>        - *"Maximum value"*: `10.0`
>
> 3. **Rename** {% icon galaxy-pencil %} output `Scaled_Object`
>
{: .hands_on}

{% icon congratulations %} Congratulations! You have processed your object!

> <details-title>More details on batch correction and removing unwanted variation</details-title>
>
> At this point, you might want to remove or regress out the effects of unwanted variation on our data. A common example of this is the cell cycle, which can affect which genes are expressed and how much material is present in our cells. If you’re interested in learning how to do this, then you can move over to the {% icon level %} [Removing the Effects of the Cell Cycle]({% link topics/single-cell/tutorials/scrna-case_cell-cycle/tutorial.md %}) tutorial now and return here to complete your analysis.
>
> - {% icon warning %} If you are in a *live course*, the time to do this *bonus tutorial* will not be factored into the schedule. Please instead return to this *after* your course is finished, or if you finish early!
>
{: .details}

# Preparing coordinates

We still have too many dimensions. Transcript changes are not usually singular - which is to say, genes were in pathways and in groups. It would be easier to analyse our data if we could more easily group these changes.

## Principal components
Principal components are calculated from highly dimensional data to find the most spread in the dataset. Given that our object has around `3216` highly variable genes, that's 3216 dimensions. There will, however, be one line/axis/dimension that yields the most spread and variation across the cells. That will be our first principal component. We can calculate the first `x` principal components in our data to drastically reduce the number of dimensions.

> <warning-title>Check the size of your AnnData object!</warning-title>
> Your AnnData object should have far more than 3216 genes in it (if you followed our settings and tool versions, you'll have a matrix around 8604 × 15950 (cells x genes). If you followed the *More details on the Highly Variable Genes above, you may have created an object with *only* the highly variable genes. Please delete this object and do not use it! Carry forward the object with around 8604 × 15950 (cells x genes)!
{: .warning}

> <hands-on-title>Calculate Principal Components</hands-on-title>
>
> 1. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Scaled_Object`
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'pp.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>            - *"Change to use different initial states for the optimization"*: `1`
>
> 2. **Rename** {% icon galaxy-pencil %} output `PCA_Object`
>
> 3. {% tool [Scanpy plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PCA_Object`
>    - *"Method used for plotting"*: `PCA: Scatter plot in PCA coordinates, using 'pl.pca_variance_ratio'`
>        - *"Number of PCs to show"*: `50`
>
> 3. **Rename** {% icon galaxy-pencil %} plot output `PCA_Variance_Plot`
>
{: .hands_on}

Why 50 principal components you ask? Well, we're pretty confident 50 is an over-estimate. Examine `PCA Variance`.

![PCA-variance](../../images/scrna-casestudy/CS3-PCA_Variance_Plot.png "Variability explained per PC")

We can see that there is really not much variation explained past component 19. So we might save ourselves a great deal of time and muddied data by focusing on the top `20` PCs. (You could probably even go as low as 10!)

## Neighborhood graph

We're still looking at around 20 dimensions at this point in our analysis. We need to identify how similar a cell is to another cell, across every cell across these dimensions. For this, we will use the k-nearest neighbor (kNN) graph, to identify which cells are close together and which are not. The kNN graph plots connections between cells if their distance (when plotted in this 20 dimensional space!) is amongst the k-th smallest distances from that cell to other cells. This will be crucial for identifying clusters, and is necessary for plotting a UMAP. From [UMAP developers](https://github.com/lmcinnes/umap): "Larger neighbor values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15 being a sensible default".

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **Number of PCs to use** = `20`
>      - **Maximum number of neighbours used** = `15`
> - Everyone else: Use the PC variance plot to pick your own PC number, and choose your own neighbour maximum as well!b
{: .details}

> <hands-on-title>ComputeGraph</hands-on-title>
>
> 1. {% tool [Scanpy Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `PCA_Object`
>    - *"Method used for inspecting"*: `Compute a neighborhood graph of observations, using 'pp.neighbors'`
>        - *"Number of PCs to use"*: `20`
>        - *"Use the indicated representation"*: `X_pca`
>        - *"Numpy random seed"*: `1`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Neighbours_Object`
>
{: .hands_on}

## Dimensionality reduction for visualisation

Two major visualisations for this data are tSNE and UMAP. We must calculate the coordinates for both prior to visualisation. For tSNE, the parameter [**perplexity**](https://www.nature.com/articles/s41467-019-13056-x) can be changed to best represent the data, while for UMAP the main change would be to change the kNN graph above itself, by changing the **neighbours**.

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **Perplexity** = `30`
> - Everyone else: Choose your own perplexity, between 5 and 50!
{: .details}

> <hands-on-title>Calculating tSNE & UMAP</hands-on-title>
>
> 1. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Neighbours_Object`
>    - *"Method used"*: `t-distributed stochastic neighborhood embedding (tSNE), using 'tl.tsne'`
>        - *"Number of PCs to use"*: `20`
>        - *"Use the indicated representation"*: `X_pca`
>        - *"Random state"*: `1`
>
> 2. {% tool [Scanpy cluster, embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Scanpy cluster, embed** {% icon tool %})
>    - *"Method used"*: `Embed the neighborhood graph using UMAP, using 'tl.umap'`
>        - *"Seed used by the random number generator"*: `1`
>
> 2. **Rename** {% icon galaxy-pencil %} output `UMAP_Object`
>
{: .hands_on}

{% icon congratulations %} Congratulations! You have prepared your object and created neighborhood coordinates. We can now use those to call some clusters!

# Cell clusters & gene markers

> <question-title></question-title>
>
> Let's take a step back here. What is it, exactly, that you are trying to get from your data? What do you want to visualise, and what information do you need from your data to gain insight?
>
> > <solution-title></solution-title>
> >
> > Really we need two things - firstly, we need to make sure our experiment was set up well. This is to say, our biological replicates should overlap and our variables should, ideally, show some difference. Secondly, we want insight - we want to know which cell types are in our data, which genes drive those cell types, and in this case, how they might be affected by our biological variable of growth restriction. How does this affect the developing cells, and what genes drive this? So let's add in information about cell

tools_json:
["EBI SCXA Data Retrieval", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2", "Scanpy Inspect and manipulate", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy2", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1", "Scanpy plot", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.10.2+galaxy2", "Scanpy filter", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_filter/scanpy_filter/1.10.2+galaxy3", "Scanpy normalize", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_normalize/scanpy_normalize/1.10.2+galaxy0", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.10.2+galaxy1", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1", "Scanpy cluster, embed", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.10.2+galaxy2", "Interactive CELLxGENE VIP Environment", "interactive_tool_cellxgene_vip"]


topic: single-cell
tutorial: bulk-music-4-compare
title: Comparing inferred cell compositions using MuSiC deconvolution
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["transcriptomics"]
content_hash: 86ba0f8e1a27069712de333f7de4a509
last_modified: 2026-05-15T22:28:16.660167
zenodo_link: https://zenodo.org/record/7319925
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music-4-compare/tutorial.html

content:
The goal of this tutorial is to apply bulk RNA deconvolution techniques to a problem with multiple variables - in this case, a model of diabetes is compared with its healthy counterparts. All you need to compare inferred cell compositions are well-annotated, high quality reference scRNA-seq datasets, transformed into MuSiC-friendly Expression Set objects, and your bulk RNA-samples of choice (also transformed into MuSiC-friendly Expression Set objects). For more information on how MuSiC works, you can check out their github site [MuSiC](https://xuranw.github.io/MuSiC/articles/MuSiC.html) or published article ({% cite wang2019bulk %}).

> <comment-title>Research question</comment-title>
> - How does variable X impact the cell distributions in my samples?
> - Needs: scRNA-seq reference dataset; bulk RNA-seq samples of interest to compare
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Data

In the standard MuSiC tutorial, we used human pancreas data. We will now use the same single cell reference dataset {%cite segerstolpe2016single %} with its 10 samples of 6 healthy subjects and 4 with Type-II diabetes (T2D), as well as the bulk RNA-samples from the same lab (3 healthy, 4 diseased). Both of these datasets were accessed from the public EMBL-EBI repositories and transformed into Expression Set objects in the previous two tutorials. For both the single cell reference and the bulk samples of interest, you have generated Expression Set objects with only T2D samples, only healthy samples, and a final everything-combined sample for the scRNA reference. We won't need the combined bulk RNA dataset. The plan is to analyse this data in three ways: using a combined reference (altogether); using only the healthy single cell reference (healthyscref); or using a healthy and combined reference separately (like4like), all to identify differences in cellular composition.

![Three colours of arrows connect bulk healthy & diseased data sets to a combined single cell (altogether); bulk healthy and single cell healthy & bulk diseased with single cell diseased (like4like); and bulk diseased and healthy with the single cell healthy reference (healthyscref).](../../images/bulk-music/comparison.png "Plan of analysis")

If you have followed the previous tutorials, you will have built your [single cell ESet object]({% link topics/single-cell/tutorials/bulk-music-2-preparescref/tutorial.md %}) and your [bulk ESet object]({% link topics/single-cell/tutorials/bulk-music-3-preparebulk/tutorial.md %}), then you can copy these into a new history now. Otherwise, follow the steps below to import the datasets you'll need.

{% snippet  faqs/galaxy/histories_copy_dataset.md %}

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Compare"*
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    * Human single cell RNA ESet objects (tag: `#singlecell`)
>
>      ```
>    {{ page.zenodo_link }}/files/ESet_object_sc_combined.rdata
>    {{ page.zenodo_link }}/files/ESet_object_sc_T2D.rdata
>    {{ page.zenodo_link }}/files/ESet_object_sc_healthy.rdata
>      ```
>
>    * Human bulk RNA ESet objects (tag: `#bulk`)
>      ```
>    {{ page.zenodo_link }}/files/ESet_object_bulk_healthy.rdata
>    {{ page.zenodo_link }}/files/ESet_object_bulk_T2D.rdata
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets as needed
>
> 4. Add to each file a tag corresponding to `#bulk` and `#scrna`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Infer cellular composition & compare

It's finally time!

## Altogether: Deconvolution with a combined sc reference

{% snippet faqs/galaxy/tutorial_mode.md %}

> <hands-on-title> Comparing: altogether </hands-on-title>
>
> 1. {% tool [MuSiC Compare](toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4) %} with the following parameters:
>    - In *"New scRNA Group"*:
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_combined.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:Normal`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_healthy.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:T2D`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_T2D.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>
> 2. To each of the outputs, add the `#altogether` tag.
>
{: .hands_on}

There are four sets of output files.
1. *Summarised Plots* <- This is the most interesting output, because it has the pretty pictures!
2. *Individual Heatmaps* <- This kind of does what standard (non-Comparing) MuSiC does for each sample, rather than combining them.
3. *Stats* <- This will be very handy if you want to make any statistical calculations, as it contains medians and quartiles
4. *Tables* <- This contains the cell proportions found within each sample as well as the number of reads.

### Summarised Plots

Examine {% icon galaxy-eye %} the output file *Summarised Plots (MuSiC)*. Now the first few pages are similar to the standard deconvolution tool, but now comparing across the factor of interest (disease). Among the myriad of visualisations available, our favourite is on page 5 - a comparison of inferred cell proportions across disease.

![Graph showing two rows for each cell type (gamma, ductal, delta, beta, alpha, and acinar cells) comparison normal or T2D proportions by either read or by sample.](../../images/bulk-music/plot_altogether.png "Altogether: Resulting Graph")

Here we can see that the bulk-RNA seq samples from the T2D patients contain markedly fewer beta cells as compared with their healthy counterparts. This makes sense, so that's good!

### Individual Heatmaps

Examine {% icon galaxy-eye %} the output file *Individual heatmaps (MuSiC)*. This shows the cell distribution across each of the individual samples, separated out by disease factor into two separate plots, but ultimately isn't particularly informative.

![Heatmap showing vertically four samples (Bulk) and horizontally each cell type (from alpha to ductal), with gradations of colour referring to the proportion of that cell type within each sample.](../../images/bulk-music/individual_heatmaps.png "Altogether: Individual heatmaps")

### Stats

If you select the *Stats* dataset, you'll find it contains four sets of data, `Bulk_disease: Read Props`, `Bulk_disease: Sample Props`, `Bulk_healthy: Read Props` and `Bulk_healthy: Sample Props`. Examine {% icon galaxy-eye %} the file `Bulk_disease: Sample Props`. This contains summary statistics (Min, quartiles, median, mean, etc.) for each phenotype. This could be quite helpful if you're trying to statistically identify differences across samples.

![Table with rows as cell types and columns as statistics: Min, 1st Qu., Median, Mean.](../../images/bulk-music/summary_statistics.png "Altogether: Stats")

### Tables

Finally, if you select the *Tables* dataset, you'll find it contains three sets of data, `Data Table`, `Matrix of Cell Type Read Counts`, and `Matrix of Cell Type Sample Proportions`.

Examine {% icon galaxy-eye %} the file `Data Table`. This contains the inferred proportions and reads associated with each sample and cell type, along with its important factor of interest (Disease). In this tutorial, we tend to use sample proportions rather than read count, but either works. The two other matrix files are just portions of this data table.

![Table with individual rows for each cell type within each sample, with columns of Cell, Factor, CT Prop in Sample and Number of Reads.](../../images/bulk-music/data_table.png "Altogether: Data table")

> <question-title></question-title>
>
> 1. Why does the data table contain 42 rows?
>
> > <solution-title></solution-title>
> >
> > 1. The data table contains a row for each cell type within each sample. Since there are 6 cell types and 7 samples, `6*7 = 42` rows.
> >
> {: .solution}
>
{: .question}

Hopefully, this has been illuminating! Now let's try two other ways of inferring from a reference and see if it makes a difference.

## Like4like: Deconvolution of healthy samples with a healthy reference and diseased samples with a diseased reference

> <hands-on-title> Like4like Inference </hands-on-title>
>
> 1. {% tool [MuSiC Compare](toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4) %} with the following parameters:
>    - In *"New scRNA Group"*:
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set:Normal`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_healthy.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:Normal`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_healthy.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set:T2D`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_T2D.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `bulk_set:T2D`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_T2D.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>
> 2. Add the `#like4like` tag to each of the outputs.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How have the cell inferences changed, now that we have changed the scRNA references used?
>
> > <solution-title></solution-title>
> >
> > ![Two graphs showing two rows for each cell type (gamma, ductal, delta, beta, alpha, and acinar cells) comparison normal or T2D proportions by either read or by sample, with the top graph labelled #altogether and the bottom labelled #like4like. Differences are more pronounced in the top #altogether graph.](../../images/bulk-music/compare_like.png "Altogether vs like4like")
> > 1. Overall, our interpretation here is that the differences are less pronounced. It's interesting to conjecture whether this is an artefact of analysis, or whether - possibly - the beta cells in the diseased samples are not only fewer, but also contain fewer beta-cell specific transcripts (and thereby inhibited beta cell function), thereby lowering the bar for the inference of a beta cell and leading to a higher proportion of interred B-cells.
> >
> {: .solution}
>
{: .question}

Let's try one more inference - this time, we'll use only healthy cells as a reference, to (theoretically) make a more consistent analysis across the two phenotypes.

## healthyscref: Deconvolution using only healthy cells as a reference

> <hands-on-title> Healthy sc reference only inference </hands-on-title>
>
> 1. {% tool [MuSiC Compare](toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4) %} with the following parameters:
>    - In *"New scRNA Group"*:
>        - {% icon param-repeat %} *"Insert New scRNA Group"*
>            - *"Name of scRNA Dataset"*: `scRNA_set:Normal`
>            - {% icon param-file %} *"scRNA Dataset"*: `ESet_object_sc_healthy.rdata` (Input dataset)
>            - In *"Advanced scRNA Parameters"*:
>                - *"Cell Types Label from scRNA dataset"*: `Inferred cell type - author labels`
>                - *"Samples Identifier from scRNA dataset"*: `Individual`
>                - *"Comma list of cell types to use from scRNA dataset"*: `alpha cell,beta cell,delta cell,gamma cell,acinar cell,ductal cell`
>            - In *"Bulk Datasets in scRNA Group"*:
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:Normal`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_healthy.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>                - {% icon param-repeat %} *"Insert Bulk Datasets in scRNA Group"*
>                    - *"Name of Bulk Dataset"*: `Bulk_set:T2D`
>                    - {% icon param-file %} *"Bulk RNA Dataset"*: `ESet_object_bulk_T2D.rdata` (Input dataset)
>                    - *"Factor Name"*: `Disease`
>
> 2. Add the `#healthyscref` tag to each of the outputs.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How have the cell inferences changed this time?
>
> > <solution-title></solution-title>
> >
> > ![Three graphs showing two rows for each cell type (gamma, ductal, delta, beta, alpha, and acinar cells) comparison normal or T2D proportions by either read or by sample, with the top graph labelled #altogether; the middle labelled #like4like; and the bottom labelled #healthyscref. Differences are most pronounced in the bottom #healthyscref graph.](../../images/bulk-music/compare_3.png "The impact of the single cell reference")
> >
> > 1. If using a like4like inference reduced the difference between the phenotype, aligning both phenotypes to the same (healthy) reference exacerbated them - there are even fewer beta cells in the output of this analysis.
> >
> {: .solution}
>
{: .question}

Overall, it's important to remember how the inference changes depending on the reference used - for example, a combined reference might have majority healthy samples or diseased samples, so that would impact the inferred cellular compositions.

# Conclusion

Congrats! You've made it to the end of this suite of deconvolution tutorials! You've learned how to find quality data for reference and for analysis, how to reformat it for deconvolution using MuSiC, and how to compare cellular inferences using multiple kinds of reference datasets. You can find the [workflow for this tutorial](https://usegalaxy.eu/u/wendi.bacon.training/w/music-deconvolution-compare-1) and an [example history](https://usegalaxy.eu/u/wendi.bacon.training/h/music-deconvolution-comparisons--final).

 We hope this helps you in your research!

![Workflow editor showing 5 inputs and 3 runs of the MuSiC Compare tool](../../images/bulk-music/compare_workflow.png "MuSiC Compare Tutorial Workflow")

This tutorial is part of the [https://singlecell.usegalaxy.eu](https://singlecell.usegalaxy.eu) portal ({% cite tekman2020single %}).

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["MuSiC Compare", "toolshed.g2.bx.psu.edu/repos/bgruening/music_compare/music_compare/0.1.1+galaxy4"]


topic: single-cell
tutorial: scrna-case_cell-cycle
title: Removing the effects of the cell cycle
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: 2ac72844009e8f68f7f754d73b2cb74b
last_modified: 2026-05-27T09:01:49.746441
zenodo_link: https://zenodo.org/record/7311628/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_cell-cycle/tutorial.html

content:
Single-cell RNA sequencing can be sensitive to both biological and technical variation, which is why preparing your data carefully is an important part of the analysis. You want the results to reflect the interesting differences in expression between cells that relate to their type or state. Other sources of variation can conceal or confound this, making it harder for you to see what is going on.

One common biological confounder is the cell cycle ({% cite Luecken2019 %}). Cells express different genes during different parts of the cell cycle, depending on whether they are in their growing phase (G1), duplicating their DNA (the S or Synthesis phase), or dividing in two (G2/M or Mitosis phase). If these cell cycle genes are having a big impact on your data, then you could end up with separate clusters that actually represent cells of the same type that are just at different stages of the cycle.

In this tutorial, we will identify the genes whose expression is known to vary during the cell cycle so that we can use them to regress out (or remove) the effects of the cell cycle on the clustering.

> <comment-title>Other Scanpy and Seurat tutorials</comment-title>
>
> This tutorial is based on the Scanpy cell cycle regression tutorial ({% cite ScanpyCC %}), which was itself based on the Seurat vignette addressing the same issue ({% cite SeuratCC %}). However, we will be using a different dataset for this tutorial.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Get Data

The data used in this tutorial is from a mouse dataset of fetal growth restriction ({% cite Bacon2018 %}). Cell cycle regression should be performed after the data has been filtered, normalised, and scaled. You can download the dataset below or import the history with the starting data.

> <comment-title></comment-title>
> - If you've been working through the Single-cell RNA-seq: Case Study then you can use your dataset from the [Filter, Plot and Explore Single-cell RNA-seq Data]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial here. Select the `Use_me_Scaled` dataset from your history. Rename that dataset as `Processed_Anndata`. You will still need to import the S and G2/M gene lists below through Zenodo.
> - At the end of this tutorial, you can return to the main tutorial to plot and explore your data with reduced effects from the cell cycle.
{: .comment}

In addition to the scRNA-seq dataset, we will also need lists of the genes that are known to be expressed at different points in the cell cycle. The lists used in this tutorial are part of the HBC tinyatlas and can be downloaded from Zenodo below {% cite kirchner_hbc_2018 %}. Between them, they include 97 genes that are expressed during the S and G2/M phases. The expression level of these cycle genes are - mostly - determined by the phases of the cell cycle. Make sure that the file type is tabular (not just the name of the file) - you can choose this when you download the files or change it after the files are in your history.

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
> Sometimes data upload can take a while, so a faster route is to import a history.
>
> 1. Import history from: [example input history](https://humancellatlas.usegalaxy.eu/u/marisa_jl/h/removing-the-effects-of-the-cell-cycle---input-history)
>
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the the history to your name of choice.
>
{: .hands_on}

> <hands-on-title>Option 2: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}files/Processed_AnnData.h5ad
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the dataset `Processed_Anndata`
>
>    {% snippet  faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}


> <hands-on-title>Option 2 continued: Data upload - Add to history </hands-on-title>
>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/sPhase.tabular
>    {{ page.zenodo_link }}/files/g2mPhase.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 2. Rename the datasets `sPhase` and `g2mPhase` respectively - be careful not to mix them up!
>
>    {% snippet  faqs/galaxy/datasets_rename.md %}
>
> 3. Check that the datatype for both is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

# Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

# Cell Cycle Scoring

The first step towards reducing the effects of the cell cycle on our dataset is cell cycle scoring. The cell cycle scoring algorithm will look at each cell in turn and calculate an S score based on the difference in the mean expression of the S Phase genes and a random sample of the same number of non-cell cycle genes from the same cell. It will do the same for the G2M genes in order to calculate the G2M score. The cells will then be assigned to the most likely phase: S, G2M, or G1, if neither G2M or S score highly. Three columns will be added to the AnnData dataset: `S_score`, `G2M_score` and `phase`.

<question-title></question-title>
>
> 1. Why don't we need a list of genes expressed in the G1 Phase?
>
> > <solution-title></solution-title>
> >
> > 1. Since we know which genes are expressed in the S and G2/M phases, we can classify cells that are expressing these genes into the S and G2/M phases respectively. Cells that aren't expressing either the S or G2/M genes must be in the other phase of the cell cycle, so we can classify them as in G1 phase.
> >
> {: .solution}
{: .question}

> <comment-title>When should we regress out the effects of the cell cycle?</comment-title>
>  Cell cycle regression can be particularly important if we are planning to do trajectory analysis down the line or if we have a dataset that is very strongly influenced by the cell cycle {% cite Luecken2019 %}. However, it isn't always appropriate to remove the effects of the cell cycle - sometimes they can be useful for distinguishing between dividing and non-dividing cell types. When you are analysing your own data, you might need to try it both ways to determine if the effects of the cell cycle are helpful or not. You could also check whether the cell cycle genes are among the top scoring genes expressed by your cell clusters to get an idea of how strong the effects are.
{: .comment}

> <hands-on-title>Score the cell cycle genes</hands-on-title>
>
> 1. {% tool [Inspect and manipulate](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Processed_Anndata` (Input dataset)
>    - *"Method used for inspecting"*: `Score cell cycle genes, using 'tl.score_genes_cell_cycle'`
>        - *"Format for the list of genes associated with S phase"*: `File`
>            - {% icon param-file %} *"File with the list of genes associated with S phase"*: `sPhase` (Input dataset)
>        - *"Format for the list of genes associated with G2M phase"*: `File`
>            - {% icon param-file %} *"File with the list of genes associated with G2M phase"*: `g2mPhase` (Input dataset)
>
> 2. Rename the output `CellCycle_Annotated`
>
{: .hands_on}

# Cell Cycle Regression

The second step after scoring the cell cycle genes is to use these scores to regress out the effects of the cell cycle. Now that we know which phase each of our cells is in, we can work out how this is affecting gene expression in our cells. We can subtract these effects from our data so that they won't influence our cell clustering. You will need to type the `phase` variable in to the {% tool Scanpy RegressOut %} tool to enable it to regress out the cell cycle effects.

> <details-title>How does cell cycle regression work?</details-title>
>
>The {% tool Scanpy RegressOut %} tool will create a linear model of the relationship between gene expression and the cell cycle scores we assigned in the previous step. Basically, this model is a line that shows how gene expression changes as the S or G2M score changes. Each gene will have its own line, so for any S score or G2M score, we could look at the corresponding point on the line to see the expected expression level of that gene.
>
>{% tool Scanpy RegressOut %} will then regress out or remove this expected effect for the genes expressed by each cell, according to the cell's S and G2M scores. The expected effect is subtracted from the expression data, leaving behind the difference between the expected position on the line and the actual position of each data point. The data points won't sit exactly on the line because their expression levels aren't determined completely by the cell cycle - the linear model only tells us what we would expect based on the cell cycle scores alone.
>
> Understanding how the regression works should help you to see why we're not just deleting the cell cycle genes from the dataset. We are using these genes that are known to be expressed during different phases to calculate the cell cycle scores and determine the phase each cell is in. We then use this information to work out the effect of the cell cycle on all of the other genes expressed by the cells. Even if they are not cell cycle genes, their expression can still be affected by the cycle. Finally, we remove or regress out the effect of the cell cycle on all the genes, leaving behind the variation that we're interested in.
>
{: .details}


> <hands-on-title>Regress out the effects of the cell cycle</hands-on-title>
>
> 1. {% tool [Scanpy RegressOut](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_regress_variable/scanpy_regress_variable/1.8.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `CellCycle_Annotated` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Variables to regress out"*: `phase`
>
>
>  2. Rename the output `CellCycle_Regressed`
>
{: .hands_on}

# Plotting the Effects of Cell Cycle Regression

Your data is now ready for further analysis, so you can return to the [Filter, Plot and Explore Single-cell RNA-seq Data]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial and move on to the Preparing coordinates step there. Make sure that you use the `CellCycle_Regressed` dataset (you may want to rename it as `Use_me_Scaled` as that is the name used in the main tutorial). However, if you want to understand how cell cycle regression has affected your data then you can work through the following steps first, to visualise where the cell cycle genes are expressed - with or without regression.

In order to look at the cell cycle genes, we first need to label them in our AnnData dataset so that we can select them for plotting. To add a new annotation to the genes or variables, we need a column with entries for each gene, in the same order as in the dataset, and with a header at the top that will become the key for identifying these entries in the AnnData dataset. We want to end up with a column that reads `TRUE` for the 97 cell cycle genes and `FALSE` for all the other genes.

You might find it easier to create this new column using a spreadsheet and then upload it as a tabular dataset, but it is possible to complete all the steps on Galaxy.

## Prepare a table of cell cycle genes

If we're going to mark all the cell cycle genes, we'll need a single list of all 97 genes instead of the two separate lists for S Phase and G2/M Phase. We'll combine the two lists into a single column with 97 entries. We'll then add a second column that simply reads `TRUE` in every row, which we'll use later to mark these as cell cycle genes in the main dataset.

> <hands-on-title>Create a list of all cell cycle genes</hands-on-title>
>
> 1. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: `sPhase` (Input dataset)
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Select"*: `g2mPhase` (Input dataset)
>
>
> 2. {% tool [Add column](toolshed.g2.bx.psu.edu/repos/devteam/add_value/addValue/1.0.0) %} with the following parameters:
>    - *"Add this value"*: `TRUE`
>    - {% icon param-file %} *"to Dataset"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>
>
> 3. Rename the dataset `CC_Genes`
{: .hands_on}

## Create an ordered list of gene names

Next, we'll need a list of all the genes in our dataset, so that we can mark the ones that are in our cell cycle list. We'll also add a column of numbers as this will help us keep the gene names in order.

> <hands-on-title>Get the gene names from your dataset</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed` (Input dataset)
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
>
> 2. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy0) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `var` (output of **Inspect AnnData** {% icon tool %})
>        - *"Type of table operation"*: `Drop, keep or duplicate rows and columns`
>            - *"List of columns to select"*: `1`
>    - *"Output formatting options"*: `Unselect all`
>
>
> 3. {% tool [Add column](toolshed.g2.bx.psu.edu/repos/devteam/add_value/addValue/1.0.0) %} with the following parameters:
>    - {% icon param-file %} *"to Dataset"*: `table` (output of **Table Compute** {% icon tool %})
>    - *"Iterate?"*: `YES`
>
>
>    > <comment-title>Keeping the genes in order</comment-title>
>    >
>    > Adding these numbers will enable us to keep the genes in their original order. This is essential for adding the cell cycle gene annotation back into the AnnData dataset.
>    {: .comment}
>
>
> 4. Rename the output `Dataset_Genes`
{: .hands_on}


## Mark the cell cycle genes

We can now combine our table of cell cycle genes `CC_genes` with the table of gene names `Dataset_Genes`.

> <hands-on-title>Combine the two tables</hands-on-title>
>
> 1. {% tool [Join](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2) %} with the following parameters:
>    - {% icon param-file %} *"1st file"*: `Dataset_Genes` (output of **Add column** {% icon tool %})
>    - *"Column to use from 1st file"*: `c1`
>    - {% icon param-file %} *"2nd File"*: `CC_Genes` (output of **Add column** {% icon tool %})
>    - *"Column to use from 2nd file"*: `c1`
>    - *"Output lines appearing in"*: `All lines [-a 1 -a 2]`
>    - *"Value to put in unpaired (empty) fields"*: `FALSE`
>
>
>    > <comment-title>How the cell cycle genes are marked</comment-title>
>    >
>    > When we join the two tables, we'll ask for any empty fields to be filled in with `FALSE`. The cell cycle gene table has an extra column where they are all marked as `TRUE` - they will retain these entries when we join the tables, but since there are no entries for the rest of the genes in this column, their rows will be filled in as `FALSE`. This will enable us to pick out the cell cycle genes later.
>    {: .comment}
>
>
> 2. {% tool [Sort](sort1) %} with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: `output` (output of **Join** {% icon tool %})
>    - *"on column"*: `c2`
>    - *"everything in"*: `Ascending order`
>
>
>    > <comment-title>Putting the genes in order again</comment-title>
>    >
>    > Sorting the genes using the column of numbers we added earlier will put them back in their original order - make sure to sort them in ascending order, otherwise they'll end up the opposite way around.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What would happen if any of the cell cycle genes were not present in the dataset?
> 2. How would we remove these genes from the table?
>
> > <solution-title></solution-title>
> >
> > 1. Any cell cycle genes that weren't in the dataset would have an empty field in the numbered column, which would be filled in with `FALSE` when we created the table with the {% tool Join %} tool. These rows would appear at the top of the table after it was sorted.
> > 2. We should check the first rows of the table for any unnumbered genes and then cut these rows out in the next step.
> >
> {: .solution}
{: .question}

## Create the annotation column

We now have a table with all the gene names in the same order as the main dataset and a column indicating which ones are cell cycle genes. If we cut this column out of the table, then we can add it as a new annotation to the main dataset. We'll also need to add a column header, which will be used as the key for this annotation in the AnnData dataset.

> <hands-on-title>Create the cell cycle annotation column</hands-on-title>
>
> 1. {% tool [Table Compute](toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy0) %} with the following parameters:
>    - *"Input Single or Multiple Tables"*: `Single Table`
>        - {% icon param-file %} *"Table"*: `out_file1` (output of **Sort** {% icon tool %})
>    - *"Input data has:"*: `Unselect all`
>        - *"Type of table operation"*: `Drop, keep or duplicate rows and columns`
>            - *"List of columns to select"*: `3`
>    - *"Output formatting options"*: `Unselect all`
>
>
>    > <comment-title>Removing rows for missing genes </comment-title>
>    >
>    > If there were any cell cycle genes that weren't present in the main dataset, we could remove them at this stage by excluding them from the List of rows to select. As before, if we were using a dataset of a different size, we would need to change this parameter to include all the rest of the rows.
>    {: .comment}
>
>
> 2. Create a new **tabular** file from the following
>
>     ```
>    CC_genes
>     ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md format="tabular" %}
>
>
> 3. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: `Pasted Entry` dataset
>    - In *"Dataset"*:
>        - {% icon param-repeat %} *"Insert Dataset"*
>            - {% icon param-file %} *"Select"*: `table` (output of **Table Compute** {% icon tool %})
>
{: .hands_on}


## Add an annotation to the AnnData

We will need to add the annotation to both the annotated dataset `CellCycle_Annotated` and to the one that we created by regressing out the cell cycle genes `CellCycle_Regressed`. This will allow us to plot the cell cycle genes before and after regression. We can do this using the {% tool Manipulate Anndata %} tool and selecting the correct function from the dropdown menu.

> <hands-on-title>Add the new annotations</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Annotated` (output of **Inspect and manipulate** {% icon tool %})
>    - *"Function to manipulate the object"*: `Add new annotation(s) for observations or variables`
>        - {% icon param-file %} *"Table with new annotations"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>
>
> 2. Rename the output `CellCycle_Annotated_CC`
>
>
> 3. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed` (output of **Scanpy RegressOut** {% icon tool %})
>    - *"Function to manipulate the object"*: `Add new annotation(s) for observations or variables`
>        - {% icon param-file %} *"Table with new annotations"*: `out_file1` (output of **Concatenate datasets** {% icon tool %})
>
>
> 4. Rename the output `CellCycle_Regressed_CC`
>
{: .hands_on}

## Filter the cell cycle genes

To demonstrate the power of cell cycle regression, we're going to reduce our expression matrices to contain only the 97 cell cycle genes. This will force our dimension reduction and plotting to be based entirely on cell cycle genes. You wouldn't do this during analysis, but for proof of principle, let's go for it!

> <hands-on-title>Filter the AnnData datasets</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Annotated_CC` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `CC_genes`
>            - *"Type of value to filter"*: `Boolean`
>
>
> 2. Rename the output `CellCycle_Annotated_CC_Only`
>
>
> 3. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed_CC` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `CC_genes`
>            - *"Type of value to filter"*: `Boolean`
>
>
> 4. Rename the output `CellCycle_Regressed_CC_Only`
>
{: .hands_on}

## Plot the cell cycle genes before regression

Now that we have a dataset that only includes the cell cycle genes, we can visualise their effects in a PCA plot. We first calculate the PCA coordinates, which are a measure of how similar each pair of cells is in terms of the expression of the 97 cell cycle genes we've included in the filtered dataset. We will then visualise the cells on a PCA plot where the axes represent the principal components, which reflect the genes (or groups of genes) that had the biggest impact in these calculations.

You will learn more about plotting your data in the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial. For now, it is enough to know that each dot on the plot represents a cell and the closer two cells are together, the more similar they are.

> <hands-on-title>Create a PCA Plot of cell cycle genes </hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Annotated_CC_Only` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'tl.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>
>    > <comment-title>Plot all the genes </comment-title>
>    >
>    > Make sure that you de-select the option for the {% tool Cluster, infer trajectories and embed %} tool to use highly variable genes only - some of the cell cycle genes are also HVGs, but we want our plots to include the cell cycle genes that aren't HVGs too.
>    {: .comment}
>
> 2. {% tool [Plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `PCA: Plot PCA results, using 'pl.pca_overview'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `phase`
>        - In *"Plot attributes"*:
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Does the plot look as you expected?
>
> > <solution-title></solution-title>
> >
> > 1. The PCA plot shows that the three groups of cells are separated out according to what phase of the cell cycle they are in. This is what we would expect to see as we are only looking at the cell cycle genes, which by definition are expressed during particular phases.
> >
> > ![PCA plot showing three separate clusters of cells in the G1, S and G2M Phases](../../images/scrna-case_cell-cycle/CellCycle_PCA1.png "PCA Plot of Cell Cycle Genes before regression")
> >
> {: .solution}
{: .question}

## Plot the cell cycle genes after regression

We will now repeat the same steps to create a PCA plot of the filtered dataset after the effects of the cell cycle have been regressed out.

> <hands-on-title>Recreate the PCA plot of cell cycle genes after regression </hands-on-title>
>
> 1. {% tool [Cluster, infer trajectories and embed](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `CellCycle_Regressed_CC_Only` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Method used"*: `Computes PCA (principal component analysis) coordinates, loadings and variance decomposition, using 'tl.pca'`
>        - *"Type of PCA?"*: `Full PCA`
>
>
> 2. {% tool [Plot](toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `anndata_out` (output of **Cluster, infer trajectories and embed** {% icon tool %})
>    - *"Method used for plotting"*: `PCA: Plot PCA results, using 'pl.pca_overview'`
>        - *"Keys for annotations of observations/cells or variables/genes"*: `phase`
>        - In *"Plot attributes"*:
>            - *"Colors to use for plotting categorical annotation groups"*: `rainbow (Miscellaneous)`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Does the plot look as you expected?
>
> > <solution-title></solution-title>
> >
> > 1. The cells in different phases are now all mixed up together. This makes sense because we are only plotting the cell cycle genes, but the previously strong effects of the cell cycle on these genes have now been regressed out. There are still some differences between the cells (they don't all end up at the same point on the PCA chart) because the regression only removes the expected effects of the cell cycle, leaving behind any individual variation in the expression of the cell cycle genes.
> >
> > ![PCA plot showing one big cluster with the cells from G1, S and G2M Phases all mixed up together](../../images/scrna-case_cell-cycle/CellCycle_PCA2.png "PCA Plot of Cell Cycle Genes after regression")
> >
> {: .solution}
{: .question}

Comparing the before and after plots, we can clearly see that the effects of the cell cycle have been removed. Although you wouldn't usually need to filter out the cell cycle genes or create these plots when analysing your own data, hopefully you have found doing it now to be helpful for understanding the impact of cell cycle regression.

> <question-title></question-title>
>
> 1. What impact do you think the cell cycle regression will have when you analyse the whole dataset? What would happen if we plotted all of the genes from the main dataset?
>
> > <solution-title></solution-title>
> >
> > 1. The regression reduces the impact of the cell cycle on the data - this is why the cells are less separated by phase afterwards. When we analyse the whole `CellCycle_Regressed` dataset, with all of the genes, this could allow other differences in gene expression to become more apparent.
> >
> > We wouldn't expect to see such clear distinctions in PCA plots created using all of the genes (not just the cell cycle ones), even before the regression. Although the cell cycle genes can have a significant effect, these won't be as obvious when other genes are also being taken into account. However, we will still see a difference after we regress out the effects of the cell cycle - the cells in different phases will become more mixed up together. How much of a difference the regression makes will depend on how strong the effects of the cell cycle are in a particular dataset - you can see the effects on this dataset below. You can also replicate these plots after completing the rest of the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial by colouring your PCA plots by phase.
> >
> > ![PCA plot showing some separation between cells in the G1, S and G2M Phases before regression](../../images/scrna-case_cell-cycle/CellCycle_PCA3.png "PCA Plot using all genes before regression")
> >
> >
> > ![PCA plot showing that cells in the G1, S and G2M Phases are more mixed up with each other after the regression](../../images/scrna-case_cell-cycle/CellCycle_PCA4.png "PCA Plot using all genes after regression")
> >
> {: .solution}
{: .question}

# Conclusion

In this tutorial, you have annotated and scored the cell cycle genes and regressed out the effects of the cell cycle. You have also created PCA plots of the data before and after regression to visualise the effects.

You might want to check your results against this [example history](https://humancellatlas.usegalaxy.eu/u/marisa_jl/h/removing-the-effects-of-the-cell-cycle---answer-key).

You can now continue to analyse this data by returning to the Preparing coordinates step in the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial. If you use the `CellCycle_Regressed` dataset (which you may now want to rename as `Use_me_Scaled` since that is the name used in the main tutorial), you should notice some differences in your results compared to those shown there because the effects of the cell cycle have been regressed out.

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Inspect and manipulate", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_inspect/scanpy_inspect/1.7.1+galaxy0", "Scanpy RegressOut", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_regress_variable/scanpy_regress_variable/1.8.1+galaxy0", "Concatenate multiple datasets or collections", "cat1", "Add column", "toolshed.g2.bx.psu.edu/repos/devteam/add_value/addValue/1.0.0", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1", "Table Compute", "toolshed.g2.bx.psu.edu/repos/iuc/table_compute/table_compute/1.2.4+galaxy0", "Join", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2", "Sort", "sort1", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1", "Cluster, infer trajectories and embed", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_cluster_reduce_dimension/scanpy_cluster_reduce_dimension/1.7.1+galaxy0", "Plot", "toolshed.g2.bx.psu.edu/repos/iuc/scanpy_plot/scanpy_plot/1.7.1+galaxy1"]


topic: single-cell
tutorial: scrna-preprocessing-tenx
title: Pre-processing of 10X Single-Cell RNA Datasets
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["10x"]
content_hash: 98fe5bc006878259ee95c38b12cc3849
last_modified: 2026-05-27T09:01:49.748033
zenodo_link: https://zenodo.org/record/3457880
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.html

content:
Single-cell RNA-seq analysis is a rapidly evolving field at the forefront of transcriptomic research, used in high-throughput developmental studies and rare transcript studies to examine cell heterogeneity within a populations of cells.
The cellular resolution and genome wide scope make it possible to draw new conclusions that are not otherwise possible with bulk RNA-seq. The analysis requires a great deal of knowledge about statistics, wet-lab protocols, and some machine learning due to variability and sparseness of the data. The uncertainty from the low coverage and low cell numbers per sample that once were common setbacks in the field are overcome by 10x Genomics which provides high-throughput solutions which are quickly championing the field.

## Era of 10x Genomics

10x genomics has provided not only a cost-effective high-throughput solution to understanding sample heterogeneity at the individual cell level, but has defined the standards of the field that many downstream analysis packages are now scrambling to accommodate.

![clusters]({% link topics/single-cell/images/scrna-pre-processing/tenx_clusters_intro.png %} "From less than 1K to over 10K with 10x genomics: Analyses of two separate scRNA datasets using the (left) CelSEQ2 protocol, and the (right) 10x Chromium system.")

The gain in resolution reduces the granularity and noise issues that plagued the field of scRNA-seq not long ago, where now individual clusters are much easier to decipher due to the added stability added by this gain in information.

## Library Preparation

![Library Preparation]({% link topics/single-cell/images/scrna-pre-processing/tenx_libprep.png %} "An overview of the library preparation")

The 10X barcoded gel beads consist of a pool barcodes which are used to separately index each cell's transcriptome. The individual gel barcodes are delivered to each cell via flow-cytometry, where each cell is fed single-file along a liquid tube and tagged with a 10X gel bead. The cells are then isolated from one another within thousands of nanoliter droplets, where each droplet described by a unique 10x barcode that all reads in that droplet are associated with once they undergo reverse-transcription (RT) which reconstructs the mRNA into a cDNA counterpart. The oil is then removed and all (now barcoded) cDNA reads are pooled together to be sequenced.

Though there are approximately 3 million 10x gel barcodes used, the amount actually qualitatively profiled in a sample is ~10,000 due to majority of droplets (>90%) being empty in order to ensure that the remainder contains only one cell.

> <details-title>Whitelist Barcodes</details-title>
>
> There are actually two sets of barcodes for the different chemistries provided; one which has 737,000 barcodes, and one with ~3,7 million barcodes.
>
> Both are provided in the Zenodo link, but we will only work with the 3 million barcodes because this is what is provided with the chemistry version.
>
> If you are getting a low number of cells detected for your reads later on, it could be because the barcodes provided are wrong.
>
{: .details}



More information can be found in the [reagent kit documentation](https://support.10xgenomics.com/single-cell-gene-expression/library-prep/doc/user-guide-chromium-single-cell-3-reagent-kits-user-guide-v2-chemistry).

<!--TODO:
 * Discuss droplet-seq based workflows
 * What is Chromium, and do users need to know what it is?-->


# Analysis Strategy


The tutorial is structured into two parts:

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

![Overview of workflow]({% link topics/single-cell/images/scrna-pre-processing/tenx_workflow.png %} "An overview of the workflow")

The first part of this tutorial is essentially a one-click "fire and forget" solution to demultiplexing and quantifying scRNA-seq data, where much of the complexity required in this extremely crucial stage is simplified into a single step.

However, those who are more interested in learning the intricacies of how FASTQ files are transformed into a count matrix, please see the [Pre-processing of Single-Cell RNA Data]({% link topics/single-cell/tutorials/scrna-preprocessing/tutorial.md %}) tutorial.

10x Genomics has its own processing pipeline, [Cell Ranger](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger) to process the scRNA-seq outputs it produces, but this process requires much configuration to run and is significantly slower than other mappers.

Since STARsolo is a drop-in solution to the *Cell Ranger* pipeline, the first part of the tutorial is a one-click solution where users are encouraged to launch their **RNA STARsolo** jobs and spend the time familiarising themselves with the pre-processing training materials mentioned above.


> <details-title>Benchmark of Cell Ranger to others</details-title>
>
> ![tenx_runtimes]({% link topics/single-cell/images/scrna-pre-processing/tenx_runtimes.jpg %} "Benchmark of different mapping software.")
>
> The image is from {% cite Melsted2019 %}. Notice the order of magnitude speed up that STARsolo and a few others display, for a variety of different datasets in comparison to Cell Ranger.
{: .details}

The second part of this tutorial also has a one-click solution to producing a matrix identical to that given by the *Cell Ranger* pipeline, but the more interesting aspects of the pipeline are explored in the *Introspective Method* part of the tutorial.


# Producing a Count Matrix from FASTQ

Here we will use the [1k PBMCs from a Healthy Donor (v3 chemistry)](https://support.10xgenomics.com/single-cell-gene-expression/datasets/3.0.0/pbmc_1k_v3) from 10x genomics, consisting of 1000 Peripheral blood mononuclear cells (PBMCs) extracted from a healthy donor, where PBMCs are primary cells with relatively small amounts of RNA (~1pg RNA/cell).

The source material consists of 6 FASTQ files split into two sequencing lanes *L001* and *L002*, each with three reads of **R1** (barcodes), **R2** (cDNA sequences), **I1** (illumina lane info):
  * pbmc\_1k\_v3\_S1\_*L001*\_**R1**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L001*\_**R2**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L001*\_**I1**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L002*\_**R1**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L002*\_**R2**\_001.fastq.gz
  * pbmc\_1k\_v3\_S1\_*L002*\_**I1**\_001.fastq.gz

The *Cell Ranger* pipeline requires all three files to perform the demultiplexing and quantification, but **RNA STARsolo** does [not require](https://github.com/alexdobin/STAR/issues/640) the I1 lane file to perform the analysis. These source files are provided in the [Zenodo](https://zenodo.org/record/3457880) data repository, but they require approximately 2 hours to process. For this tutorial, we will use datasets sub-sampled from the source files to contain approximately 300 cells instead of 1000. Details of this sub-sampling process can be viewed at the [Zenodo link](https://zenodo.org/record/3457880/files/subsetting_data.txt).

## Data upload and organization

For the mapping, we require the sub-sampled source files, as well as a "whitelist" of (~3,7 million) known cell barcodes, [freely extracted](https://kb.10xgenomics.com/hc/en-us/articles/115004506263-What-is-a-barcode-whitelist) from the *Cell Ranger* pipeline. This whitelist file may be found within the Galaxy Data Library, but it is included here in the Zenodo record for convenience and also because the sequencing facility may not always provide this file.

The barcodes in the R1 FASTQ data are checked against these known cell barcodes in order assign a specific read to a specific known cell. The barcodes are designed in such a manner that there is virtually no chance that they will align to a place in the reference genome. In this tutorial we will be using hg19 (GRCh37) version of the human genome, and will therefore also need to use a hg19 GTF file to annotate our reads.


> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and rename it (e.g. scRNA-seq 10X dataset tutorial)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 1. Import the sub-sampled FASTQ data from [`Zenodo`](https://zenodo.org/record/3457880) or from the data library (ask your instructor)
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    ```
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L001_R1_001.fastq.gz
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L001_R2_001.fastq.gz
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L002_R1_001.fastq.gz
>    https://zenodo.org/record/3457880/files/subset_pbmc_1k_v3_S1_L002_R2_001.fastq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Import the Gene Annotations and Cell Barcodes from [`Zenodo`](https://zenodo.org/record/3457880) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3457880/files/Homo_sapiens.GRCh37.75.gtf
>    https://zenodo.org/record/3457880/files/3M-february-2018.txt.gz
>    ```
>
{: .hands_on}


## 10x Chemistries

There are two main reagent kits used during the library preparation, and the choice of one will influence the size of the sequences we work with. Below we can see the layout of the primers used in both chemistries. We can ignore most of these as they are not relevant, namely: the P5 and P7 illumina primers are used in the illumina [bridge amplification](https://en.wikipedia.org/wiki/Illumina_dye_sequencing#Bridge_amplification) process; the Sample Index is an 8bp primer which is related to the Chromium system that balances nucleotide bias and ensures that there is no sample overlap during the multiplexed sequencing; and the Poly(dT) VN primer used to capture RNA sequences with poly-A tails (i.e. mRNA).

![chem]({% link topics/single-cell/images/scrna-pre-processing/tenx_primers.svg %} "10x Chromiumv2 and Chromiumv3 Chemistries")

The primers of interest to us are the Cell Barcode (CB) and the Unique Molecular Identifiers (UMI) used in the Read 1 sequencing primer, as they describe to us how to demultiplex and deduplicate our reads. It is highly advised that the [Plates, Batches, and Barcodes]({% link topics/single-cell/tutorials/scrna-preprocessing/tutorial.md %}) slides are revisited to refresh your mind on these concepts.


| Chemistry | Read 2 | Read 1 (CB + UMI) | Insert (Read 2 + Read 1) |
|:---------:|:------:|:-----------------:|:------------------------:|
| v2        |  98    | 26 (16 + 10)      | 124 |
| v3        |  91    | 28 (16 + 12)      | 119 |

The table above gives a summary of the primers used in the image and the number of basepairs occupied by each.

> <details-title>Strandedness</details-title>
>
> Unstranded protocols do not distinguish between whether a fragment was sequenced from the forward or the reverse strand, which can lead to some ambiguity if the fragment overlaps two transcripts. In the image below, it is not clear whether the fragment is derived from GeneF or GeneR due to this overlap.
>
> ![strandedness]({% link topics/single-cell/images/scrna-pre-processing/tenx_strandedness.svg %} "Mapping fragments to overlapping transcripts is ambiguous with unstranded protocols.")
>
> Stranded protocols overcome this by fitting different 5' and 3' primers and adaptors, meaning that the orientation of the fragment is fixed during sequencing. For the Chromium v2 and v3 chemistries, the barcode information is purely within the R1 forward strand.
>
{: .details}



> <question-title></question-title>
>
> 1. What has stayed constant between the chemistry versions?
> 1. What advantage does this constant factor give?
> 1. What do UMIs do?
> 1. What advantage does the 2 extra bp in the v3 UMIs have over v2 UMIs?
> 1. What will be the strandedness of the generated library
>
> > <solution-title></solution-title>
> >
> > 1. The Cell Barcode (CB) has remained at 16bp for both chemistries.
> > 1. This has the advantage that the same set of barcodes can be used in both chemistries, which is important because barcodes are *very* hard to design.
> >    * They need to be designed in such a way to minimise accidentally aligning to the reference they were prepared to be used for.
> >    * Longer barcodes tend to be more unique, so this is a problem that is being solved as the barcodes increase in size, allowing for barcodes that can be used on more than one reference to be more common, as seen above.
> > 1. UMIs (or Unique Molecular identifiers) do not delineate cells as Cell Barcodes do, but instead serve as random 'salt' that tag molecules randomly and are used to mitigate amplification bias by deduplicating any two reads that map to the same position with the same UMI, where the chance of this happening will be astronomically small unless one read is a direct amplicon of the other.
> > 1. $$4^{10} = 1,048,576$$ unique molecules tagged, vs. $$4^{12} = 16,777,216$$ unique molecules tagged. The reality is much much smaller due to edit distances being used that would reduce both these numbers substantially (as seen in the [*Plates, Batches, and Barcodes*]({% link topics/single-cell/tutorials/scrna-plates-batches-barcodes/slides.html %}) slides), but the scale factor of 16 times more molecules ($$4^{12-10} = 16$$) can be uniquely tagged is true.
> > 1. Forward.
> {: .solution}
{: .question}

The differences in the chemistries is a slight change in the library size, where the v2 aims to capture *on average* 50,000 reads per cell, whereas the v3 aims to capture *at minimum* 20,000 reads per cell. This greatly reduces the lower-tail of the library size compared to the previous version.


### Determining what Chemistry our Data Contains

To perform the demultiplexing, we need to tell **RNA STARsolo** where to look in the R1 FASTQ to find the cell barcodes. We can do this by simply counting the number of basepairs in any read of the R1 files.

> <question-title></question-title>
> Peek at one of the R1 FASTQ files using the {% icon galaxy-eye %} symbol below the dataset name.
>
> 1. How many basepairs are there in any given read?
> 2. Which library preparation chemistry version was this read generated from?
>
> > <solution-title></solution-title>
> > 1. There are 28 basepairs.
> > 2. The v2 has 26 basepairs, but the v3 has 28 basepairs. Therefore the reads we have here use the Chromium v3 chemistry.
> {: .solution}
{: .question}



## Performing the Demultiplexing and Quantification

We will now proceed to demultiplex, map, and quantify both sets of reads using the correct chemistry discovered in the previous sub-section.

> <comment-title></comment-title>
>
> {% tool [RNA STARsolo](toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.10b+galaxy3) %} consumes a large amount of memory. During the Smörgåsbord training please use `Human (Homo Sapiens): hg19 chrX` as the reference genome if you follow this tutorial on [usegalaxy.org](https://usegalaxy.org). This performs the mapping only against chromosome X. The full output dataset is available at [zenodo](https://zenodo.org/record/3581213/files/matrix.mtx) and will be the starting point for the next tutorial.
{: .comment}

> <hands-on-title></hands-on-title>
>
> {% tool [RNA STARsolo](toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.11a+galaxy1) %}  with the following parameters:
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>        - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>            - *"Select reference genome"*: `Human (Homo Sapiens): hg19 Full` or `Human (Homo Sapiens) (b37): hg19`
>            - *"Gene model (gff3,gtf) file for splice junctions"*: `Homo_sapiens.GRCh37.75.gtf`
>            - *"Length of genomic sequence around annotated junctions"*: `100`
>    - *"Type of single-cell RNA-seq"*: `Drop-seq or 10X Chromium`
>        - *"Input Type"*: `Separate barcode and cDNA reads`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, Barcode reads"*: Multi-select `L001_R1_001` and `L002_R1_001` using the Ctrl key.
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file, cDNA reads"*: Multi-select `L001_R2_001` and `L002_R2_001` using the Ctrl key.
>        - {% icon param-file %} *"RNA-Seq Cell Barcode Whitelist"*: `3M-february-2018.txt.gz`
>        - *"Configure Chemistry Options"*: `Chromium chemistry v3`
>        - *"UMI deduplication (collapsing) algorithm"*: `CellRanger2-4 algorithm`
>        - *"Type of UMI filtering"*: `Remove UMIs with N and homopolymers (similar to CellRanger 2.2.0)`
>        - *"Matching the Cell Barcodes to the WhiteList"*: `Multiple matches (CellRanger 2, 1MM_multi)`
>    - Under *"Advanced Settings"*:
>        - *"Strandedness of Library"*: `Read strand same as the original RNA molecule`
>        - *"Collect UMI counts for these genomic features"*: `Gene: Count reads matching the Gene Transcript`
>        - *"Cell filter type and parameters"*: `Do not filter`
>        - *"Field 3 in the Genes output"*: `Gene Expression`
>
>    > <comment-title></comment-title>
>    >
>    > The in-built Cell filtering is a relatively new feature that emulates the CellRanger pipeline. Here, we set the filtering options to not filter because we will use our own methods to better understand how this works. For your own future datasets, you may wish to enable the filtering parameter.
>    {: .comment}
>
{: .hands_on}

## Inspecting the Output Files

At this stage **RNA STARsolo** has output 6 files; a program log, a mapping quality file, a BAM file of alignments, and 3 count matrix files:
 1. Log
 1. Feature Statistic Summaries
 1. Alignments
 1. Matrix Gene Counts
 1. Barcodes
 1. Genes

The log and the summaries files give program logistics and metrics about the quality of the mapping. The BAM file contains the reads mapped to the reference genome. The matrix gene counts is the count matrix in *matrixmarket* format, accompanied by the list of genes and cell barcodes in separate files. These can be converted into *tabular* or *AnnData* formats using the {% icon tool %} **Import Anndata and loom** tool.


### Mapping Quality

Let us investigate the output log. This type of quality control is essential in any RNA-based analysis and it is strongly recommended that you familiarise yourself with the [Quality Control]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}) tutorial.


> <hands-on-title></hands-on-title>
>
> {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy0) %} with the following parameters:
>    - In *"Results"*:
>      - In *"1:Results"*:
>        - *"Which tool was used generate logs?"*: `STAR`
>        - In *"STAR output"*:
>          - In *"1:STAR output"*:
>            - *"Type of STAR output?"*: `Log`
>            - {% icon param-file %} *"STAR log output"*: `RNA STARsolo: log`
{: .hands_on}


> <question-title></question-title>
>
> What percentage of reads are uniquely mapped?
>
> > <solution-title></solution-title>
> > 87.5%
> > - This is good, and is expected of 10x datasets.
> >
> {: .solution}
{: .question}

### Quantification Quality

The above tool provides a nice visualisation of the output log, and is a fantastic way to combine multiple quality sources into one concise report.

However, sometimes it is often more informative to look directly at the source quality control file. For example, let us investigate the STARsolo Feature summaries file.

We can look at this directly by clicking on the {% icon galaxy-eye %} symbol of the *Feature Statistic Summaries* file.


> <comment-title>RNA STARsolo log output</comment-title>
> ```
> Barcodes:
>                                       noNoAdapter              0
>                                           noNoUMI              0
>                                            noNoCB              0
>                                           noNinCB              0
>                                          noNinUMI            358
>                                  noUMIhomopolymer            707
>                                       noNoWLmatch          50037
>                                       noTooManyMM              0
>                                noTooManyWLmatches              0
>                                   yesWLmatchExact        7530489
>                               yesOneWLmatchWithMM          19534
>                              yesMultWLmatchWithMM          84800
> Genes:
>                                        noUnmapped         278029
>                                       noNoFeature        3060392
>                                      MultiFeature         419262
>                       subMultiFeatureMultiGenomic         333163
>                                noTooManyWLmatches          28645
>                              noMMtoWLwithoutExact           7148
>                                        yesWLmatch        3841347
>                                yessubWLmatchExact        3829826
>                       yessubWLmatch_UniqueFeature        3841347
>                                   yesCellBarcodes           5200
>                                           yesUMIs        1689978
>
> ```
{: .comment}

The explanation of these parameters can be seen in the [RNA STAR Manual](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf) under the *STARsolo* section, with most of the information given in the details box below.

> <details-title>Explanation of parameters</details-title>
>
>  | Parameter | Explanation |
>  |-----------|-------------|
>  | noNinCB | number of reads with more than 2 Ns in cell barcode (CB) |
>  | noUMIhomopolymer | number of reads with homopolymer in CB |
>  | noTooManyWLmatches | (not used at the moment) |
>  | noNoWLmatch | number of reads with CBs that do not match whitelist even with one mismatch |
>
> All of the above reads are discarded from Solo output.
>
> The remaining reads are checked for any overlap with the features/genes:
>
>
>  | Parameter | Explanation |
>  |-----------|-------------|
>  | noUnmapped | number of reads unmapped to the genome |
>  | noNoFeature | number of reads that map to the genome but do not belong to a feature |
>  | MultiFeature | number of reads that belong to more than one feature |
>  | subMultiFeatureMultiGenomic | number of reads that belong to more than one feature and are also multimapping to the genome (this is a subset of the `nAmbigFeature`) |
>  | noTooManyWLmatches | number of reads with ambiguous CB (i.e. CB matches whitelist with one mismatch but with posterior probability 0.95) |
>  | noMMtoWLwithoutExact | number of reads with CB that matches a whitelist barcode with 1 mismatch, but this whitelist barcode does not get any other reads with exact matches of CB |
>
> These metrics can be grouped into more broad categories:
>
> * `noNinCB` + `noUMIhomopolymer` + `noNoWLmatch` + `noTooManyWLmatches` + `noMMtoWLwithoutExact` = number of reads with CBs that do not match whitelist.
> * `noUnmapped` + `MultiFeature` = number of reads without defined feature (gene).
> * `yessubWLmatch_UniqueFeature` = number of reads that are output as solo counts.
>
> The three categories above summed together should be equal to the total number of reads (which is also given in the MultiQC output).
>
{: .details}


The main information to gather at this stage is that the `yesCellBarcodes` tell us how many cells were detected in our sample, where we see that there are 5200 cells. Another metric to take into account is that the number of matches (`yessubWLmatch_UniqueFeature`) has the largest value, and that the number of reads that map to the genome but not to a feature/gene given in the GTF (`noNoFeature`) is not too large. The number of no features is also quite high when mapping the original (non-subsampled) 10x input datasets, so this appears to be the default expected behaviour.


# Producing a Quality Count Matrix

The matrix files produced by **RNA STARsolo** are in the *bundled* format, meaning that the information to create a tabular matrix of Genes vs Cells are separated into different files. These files are already 10x analysis datasets, compatible with any downstream single-cell RNA analysis pipeline, however the number of cells represented here are greatly over-represented, as they have not yet been filtered for high quality cells, and therefore the matrix represents *any* cells that were unambiguously detected in the sample.

If we were to construct a cell matrix using the data we have, we would have a large matrix of 60,000 Genes against 3 million Cells, of which most values would be zero, i.e. an *extremely* sparse matrix.

To get a high quality count matrix we must apply the **DropletUtils** tool, which will produce a filtered dataset that is more representative of the *Cell Ranger* pipeline.

## Cell Ranger Method

> <hands-on-title>Default Method</hands-on-title>
>
> {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %}  with the following parameters:
>    - *"Format for the input matrix"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - {% icon param-file %} *"Count Data"*: `Matrix Gene Counts` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Genes List"*: `Genes` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Barcodes List"*: `Barcodes` (output of **RNA STARsolo** {% icon tool %})
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `DefaultDrops`
>            - *"Expected Number of Cells"*: `3000`
>            - *"Upper Quantile"*: `0.99`
>            - *"Lower Proportion"*: `0.1`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>
> > <comment-title>Default Parameter</comment-title>
> >
> > The *"Expected Number of Cells"* parameter is the number of cells you expect to see in your sample, but does not correspond to how many cells you expect to see in a sub-sampled dataset like this one. The default is 3000 for *Cell Ranger*, which will yield ~300 cells here, but users are encouraged to experiment with this value when dealing with their own data, to recover the desired number of cells.
> >
> {: .comment}
{: .hands_on}


> <question-title></question-title>
>
> 1. How many cells were detected?
> 1. Does this agree with the STARsolo Feature Statistics output?
>
> > <solution-title></solution-title>
> >
> > 1. By clicking on the title of the output dataset in the history we can expand the box to see the output says that there are `272` cells in the output table.
> > 1. If we expand the {% icon galaxy-eye %} *RNA STARsolo Feature Statistic Summaries* Dataset and look at the `yesCellBarcodes` value, we see that *RNA STARsolo* detected `5200` cells. What this means is that *5200* barcodes were detected in total, but only *272* of them were above an acceptable threshold of quality, based on the default upper quantile and lower proportion parameters given in the tool. Later on, we will actually later visualise these thresholds ourselves by "ranking" the barcodes, to see the dividing line between high and low quality barcodes.
> {: .solution}
{: .question}

This will produce a count matrix in a human readable tabular format which can be used in downstream analysis tools, such as those used in the [*Downstream Single-cell RNA analysis with RaceID*]({% link topics/single-cell/tutorials/scrna-raceid/tutorial.md %}) tutorial.

> <details-title>File Formats in scRNA-seq</details-title>
>
> The tabular outputs here are selected mostly for transparency, since they are easy to inspect. However, the datasets produced with 10x data are very large and very sparse, meaning there is much data redundancy due to the repetition of zeroes everywhere in the data.
>
> [Many](https://bioconductor.org/packages/release/bioc/html/SingleCellExperiment.html) analysis [packages](https://www.rdocumentation.org/packages/RaceID/versions/0.1.3/topics/SCseq) have [attempted](https://github.com/satijalab/seurat/wiki/Seurat) to [solve](https://www.rdocumentation.org/packages/scater/versions/1.0.4/topics/SCESet) this problem by inventing their own standard, which has led to the proliferation of many different "standards" in the scRNA-seq package ecosystem, most of which require an R programming environment to inspect.
>
> As expected, the format that usually wins is the one which is most common in the field. In this case, the format *also* happens to be a very good format that stores data in a concise, compressed, and extremely readable manner:
>
> ![anddata]({% link topics/single-cell/images/scrna-pre-processing/tenx_anndata.svg %} "AnnData is an HDF5-based format, which stores gene and cell information in their own matrices, complementary to the main data matrix." )
>
> The *AnnData* format (`hda5`) is an extension of the [HDF5 format](https://en.wikipedia.org/wiki/Hierarchical_Data_Format), which supports multidimensional datasets to be stored in a consistent and space-optimised way. This is the default output of *Cell Ranger* and so is also the default output of **RNA STARsolo**. The format is also now a widely accepted format in many downstream analysis suites.
{: .details}



## Introspective Method

The *DefaultDrops* method given in the previous sub-section is a good one-click solution to emulating the *Cell Ranger* process of producing a high quality count matrix.

However, the **DropletUtils** tool does provide other options for determining which cells are of "good" quality and which are not.

A useful diagnostic for droplet-based data is the barcode rank plot, which shows the (log-)total UMI count for each barcode on the y-axis and the (log-)rank on the x-axis. This is effectively a transposed empirical cumulative density plot with log-transformed axes. It is useful as it allows users to examine the distribution of total counts across barcodes, focusing on those with the largest counts.

> <hands-on-title>Rank Barcodes</hands-on-title>
>
> {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %}  with the following parameters:
>    - *"Format for the input matrix"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - {% icon param-file %} *"Count Data"*: `Matrix Gene Counts` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Genes List"*: `Genes` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Barcodes List"*: `Barcodes` (output of **RNA STARsolo** {% icon tool %})
>    - *"Operation"*: `Rank Barcodes`
>        - *"Lower Bound"*: `100`
>
{: .hands_on}

![knee]({% link topics/single-cell/images/scrna-pre-processing/tenx_knee.png %} "Barcode Ranks: The separating thresholds of high and low quality cells")

The knee and inflection points on the curve mark the transition between two components of the total count distribution. This is assumed to represent the difference between cells with high RNA content and empty droplets with little RNA, and gives us a rough idea of how many cells to expect in our sample.

> ### Question {% icon question %} Questions
>
> 1. How many presumably real cells can we expect to see in our sample based on the above plot?
> 1. What is the number of cells that we can expect to see with a cut-off detected at inflection line?
>
> > <solution-title></solution-title>
> > 1. We see the blue knee line cross the threshold of barcodes at just below than the 10000 Rank on the horizontal log scale, which is shown in the expanded view of our data as `"knee = 4861"`. This line intersects with the ranked cells near 100 on x-axis. We can expect 100-200 cells with high RNA content.
> > 1. This threshold is given by the inflection line, which is given at `"inflection = 260"`. The vertical drop in the ranked cells at the inflection line is between 100 and 10000 on the x-axis. The axis is log scaled it is close to 100. Hence, we can expect between 200 and 400 cells.
> {: .solution}
{: .question}

On large 10x datasets we can use these thresholds as metrics to utilise in our own custom filtering, which is once again provided by the **DropletUtils** tool. In this case, we will use a bit less stringent threshold of 200 (instead of 260 at inflection) and use a FDR threshold of 0.01 so that our detected cells contain only 1% of false positives.

> <hands-on-title>Custom Filtering</hands-on-title>
>
> {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %}  with the following parameters:
>    - *"Format for the input matrix"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - {% icon param-file %} *"Count Data"*: `Matrix Gene Counts` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Genes List"*: `Genes` (output of **RNA STARsolo** {% icon tool %})
>        - {% icon param-file %} *"Barcodes List"*: `Barcodes` (output of **RNA STARsolo** {% icon tool %})
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `EmptyDrops`
>            - *"Lower-bound Threshold"*: `200`
>            - *"FDR Threshold"*: `0.01`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>
{: .hands_on}

![cells]({% link topics/single-cell/images/scrna-pre-processing/tenx_cells.png %} "Detected Cells (red)")


Here we recover 279 high quality cells instead of the 272 detected via the default method previously. On large datasets, this difference can help clean downstream clustering. For example, soft or less well-defined clusters are derived from too much noise in the data due to too many low quality cells being in the data during the clustering. Filtering these out during the pre-processing would produce much better separation, albeit at the cost of having less cells to cluster. This filter-cluster trade-off is discussed in more detail in the downstream analysis training materials.


# Conclusion


In this workflow we have learned to quickly perform mapping and quantification of scRNA-seq FASTQ data in a single step via **RNA STARsolo**, and have reproduced a *Cell Ranger* workflow using the **DropletUtils** suite, where we further explored the use of barcode rankings to determine better filtering thresholds to generate a high quality count matrix.

A full pipeline which produces both an AnnData and tabular file for inspection is provided [in this workflow](workflows/scRNA-seq-Preprocessing-TenX.ga)

Note that, since version *2.7.7a* of the tool, the entire *Cell Ranger* pipeline including the filtering can be performed natively within **RNA STARsolo**. As this is still a relatively new feature, we do not use it here in this tutorial, but eager users are encouraged to try it out.

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %}).

<!-- ![Recap of workflow]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow.svg %} "A recap of the entire workflow") -->

tools_json:
["RNA STARsolo", "toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.10b+galaxy3", "toolshed.g2.bx.psu.edu/repos/iuc/rna_starsolo/rna_starsolo/2.7.11a+galaxy1", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy0", "DropletUtils", "toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2"]


topic: single-cell
tutorial: scrna-case_FilterPlotandExplore_SeuratTools
title: Filter, plot, and explore single cell RNA-seq data with Seurat
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 252d0974a02ae9ddde5f52ed8bdc5442
last_modified: 2026-05-27T09:01:49.744384
zenodo_link: https://zenodo.org/record/7053673
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_FilterPlotandExplore_SeuratTools/tutorial.html

content:
You’ve previously done all the work to make a single cell matrix. Now it’s time to fully process our data using Seurat: remove low quality cells, reduce the many dimensions of data that make it difficult to work with, and ultimately try to define clusters and find some biological meaning and insights! There are many packages for analysing single cell data - Seurat ({% cite Satija2015 %}), Scanpy ({% cite Wolf2018 %}), Monocle ({% cite Trapnell2014 %}), Scater ({% cite McCarthy2017 %}), and many more. We’re working with Seurat because it is well updated, broadly used, and highly trusted within the field of bioinformatics.

> <comment-title></comment-title>
> This tutorial is significantly based on the Seurat documentation({% cite Satija2015 %}) as well as [Seurat's Guided Clustering Tutorial](https://satijalab.org/seurat/articles/pbmc3k_tutorial.html).
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

We’ll provided you with experimental data to analyse from a mouse dataset of fetal growth restriction ({% cite Bacon2018 %}). This is the full dataset generated from [this tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}).

# Get Data onto Galaxy and generate a Seurat object

## Get Data onto Galaxy
To start, let's get our dataset loaded into Galaxy.

{% include _includes/cyoa-choices.html option1="EBI Data Retrieval" option2="Importing from a history" option3="Uploading from Zenodo" default="EBI Data Retrieval" text="There are multiple ways in which to collect the data for this tutorial. I find it easiest to do so via the EBI Data Retrieval." %}

<div class='EBI-Data-Retrieval' markdown='1'>
> <hands-on-title>EBI Data Retrieval</hands-on-title>
>
> You may retrieve that files necessary to construct a Seurat Object in this way. Doing to will alleviate the necessity to convert AnnData (Python) objects into Seurat (R) objects:
>
> Run{% tool [EBI SCXA Data Retrieval](toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2) %} with the following parameters:
> - *"SC-Atlas experiment accession"*: `E-MTAB-6945`
> - *"Choose the type of matrix to download"*: `Raw filtered counts`
{: .hands_on}
</div>

<div class='Importing-from-a-history' markdown='1'>
> <hands-on-title>Importing from a history</hands-on-title>
> You can import [this history](https://singlecell.usegalaxy.eu/u/camila-goclowski/h/tool-based-seurat-fpe-input-data)
>
> {% snippet faqs/galaxy/histories_import.md %}
>
> This also alleviates the necessity to convert the AnnData object into a Seurat one, which is an additional step you must complete if you choose to use the next method.
{:.hands_on}
</div>

<div class='EBI-Data-Retrieval Importing-from-a-history' markdown='1'>

## Generating a Seurat object
You now should have imported the `matrix.mtx`, `genes.tsv`, `barcodes.tsv`, and `exp_design.tsv` files into your Galaxy history. In order for Seurat tools to work, we will have to convert the data into a format that Seurat recognizes. To do so, we will add row and column names to our matrix. In the end, this will leave us with a matrix whose rows are gene names, columns are cell barcodes, and each value in the matrix represent the expression value of a given gene in a given cell.

This can be accomplished via the Read10x step. **Read10x** tool implements Seurat's function to create a matrix and add in feature and barcode names simultaneously:

> <hands-on-title>Read10X</hands-on-title>
>
> Run{% tool [Seurat Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/4.0.4+galaxy0) %} with the following parameters:
> - *"Expression matrix in sparse matrix format (.mtx)"*: `EBI SCXA Data Retrieval on E-MTAB-6945 matrix.mtx (Raw filtered counts)`
> - *"Gene table"*: `EBI SCXA Data Retrieval on EMTAB-6945 genes.tsv (Raw filtered counts)`
> - *"Barcode/cell table"*: `EBI SCXA Data Retrieval on E-MTAB-6945 barcodes.tsv (Raw filtered counts)`
> - *"Cell Metadata"*: `EBI SCXA Data Retrieval on E-MTAB-6945 exp_design.tsv`
> - *"Minimum cells to include features"*: `5`
> - *"Choose the format of the output"*: `RDS with a Seurat object`
>
> **Rename** {% icon galaxy-pencil %} output `Initial Seurat Object`
{: .hands_on}

The output of this tool will result in a Seurat object with row/column names as described above.

</div>

<div class='Uploading-from-Zenodo' markdown='1'>

## Get Data onto Galaxy
> <hands-on-title>Uploading from Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Mito-counted_AnnData
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} the datasets `Mito-counted AnnData`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Mito-counted AnnData" %}
>
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}

## Generating a Seurat object
When you uploaded your data from Zenodo, it came in AnnData format, thus we will need to convert this to a Seurat Object. This can be easily accomplished using the Seurat FilterCells tool.
Simply run the tool without any actual filtering thresholds and with the following parameters:

> <hands-on-title>Filter Cells</hands-on-title>
>
> Run{% tool [Seurat FilterCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_filter_cells/seurat_filter_cells/4.0.4+galaxy0) %} with the following parameters:
> - *"Choose the format of the input"*: `AnnData`
> - *"AnnData file"*: `Mito-counted AnnData`
> - *"Choose the format of the output"*: `RDS with a Seurat object`
{: .hands_on}

</div>

You've created a Seurat object, congratulations!

# QC Plots
Now that we have a complete Seurat object, we can begin the filtering process.

There will  be a number of ‘cells’ that are actually just empty droplets or low-quality. There will also be genes that could be sequencing artifacts or that appear with such low frequency that statistical tools will fail to accurately analyse them.

This background noise of both cells and genes not only makes it harder to distinguish real biological information from artifacts, but also makes it computationally demanding to analyze.

We want to filter our cells, but first we need to know what our data looks like. There are a number of subjective decisions to make within scRNA-seq analysis, for instance we now need to make informed decisions about where to set our thresholds (more on that soon!).

We’re going to plot our data a few different ways. Different bioinformaticians might prefer to see the data in different ways, and here we are only generating a few of the plots you can use. Ultimately you need to go with what makes the most sense to you.

So let's generate some QC plots. First off, let's visualize the spread of our data:

> <hands-on-title>Visualize Counts</hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Initial Seurat Object` (output of **Seurat Read10x** {% icon tool %})
> - *"Plot_type_selector"*: `VlnPlot`
> - *"Features"*: `nCount_RNA`
> - *"Log"*: `Yes`
{: .hands_on}

![Violin Plot of Counts](../../images/scrna-case_FPE_SeuratTools/nCount_RNA_vln_plot.png "Violin Plot of counts.")

This plot will show us the spread of cells in our data containing a given number of counts (or transcripts) observed in a given cell. We can use this plot, and others like it in a moment, to help filter out the uninformative cells.

In a similar fashion we can visualize the spread of cells in our data expressing a given number of features (or genes):

> <hands-on-title>Visualize Features</hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Initial Seurat Object` (output of **Seurat Read10x** {% icon tool %})
> - *"Plot_type_selector"*: `VlnPlot`
> - *"Features"*: `nFeature_RNA`
> - *"Log"*: `Yes`
{: .hands_on}

![Violin Plot of Features](../../images/scrna-case_FPE_SeuratTools/nFeature_RNA_vln_plot.png "Violin Plot of features.")

Now, we could just pick filtering thresholds based on these plots, and in a typical pipeline we would also plot the proportion of features that map to the mitochondrial genome (a tool coming soon to do so!). In the meantime, let's do some QC checks.

We can, and should, ask a number of questions about the quality of our data before conducting any actual analyses. Batch effect, for example carries the potential to alter the conclusions we make. Let's take a look at whether this may be the case here:

> <hands-on-title>Visualize Counts Split by Individual</hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Initial Seurat Object` (output of **Seurat Read10x** {% icon tool %})
> - *"Plot_type_selector"*: `VlnPlot`
> - *"Features"*: `nCount_RNA`
> - *"Group by"*: `Sample.Characteristic.individual.`
> - *"Split by"*: `Sample.Characteristic.individual.`
> - *"Log"*: `Yes`
> - *"Fill by"*: `ident`
{: .hands_on}

![Violin Plot of Counts Split by Individual](../../images/scrna-case_FPE_SeuratTools/nCount_split_by_Individual_vln_plot.png "Violin Plot of counts split by individual.")

This plot shows us the number of cells split by the individual (mouse) from which the cells came from. Now, depending on your experimental design, batch may be represented by something other than individual--like timepoint or even the wet lab researcher who isolated the cells.

Ideally, we would like to see a relatively even distribution of counts for each individual (or batch) but if there isn’t, fear not, we can regress this variable out in a later step.

><tip-title>Plotting Lesson</tip-title>
>In order to accurately assess potential batch effects, use the "group.by" parameter to indicate the variable which differed across experiments.
>If you are analyzing your own data, try plotting another variable which you know differed across experiments or even just samples.
>
{: .tip}

Now let's get an idea of how other variables, like  sex or genotype of the mice, might be represented across our dataset.

## 1. Sex Differences in Counts?

   > <hands-on-title>Visualize Counts Split by Sex</hands-on-title>
  >
  > Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
  > - *"RDS file"*: `Initial Seurat Object` (output of **Seurat Read10x** {% icon tool %})
  > - *"Plot_type_selector"*: `VlnPlot`
  > - *"Features"*: `nCount_RNA`
  > - *"Group by"*: `Sample.Characteristic.sex.`
  > - *"Split by"*: `Sample.Characteristic.sex.`
  > - *"Log"*: `Yes`
  > - *"Fill by"*: `ident`
  {: .hands_on}

![Violin Plot of Counts Split by Sex with males representing more of the counts than females but distributions are about the same across both sexes](../../images/scrna-case_FPE_SeuratTools/nCount_split_by_Sex_vln_plot.png "Violin Plot of counts split by sex.")

## 2. Genotype Differences in Counts?

  > <hands-on-title>Visualize Counts Split by Genotype</hands-on-title>
  >
  > Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
  > - *"RDS file"*: `Initial Seurat Object` (output of **Seurat Read10x** {% icon tool %})
  > - *"Plot_type_selector"*: `VlnPlot`
  > - *"Features"*: `nCount_RNA`
  > - *"Group by"*: `Sample.Characteristic.genotype.`
  > - *"Split by"*: `Sample.Characteristic.genotype.`
  > - *"Log"*: `Yes`
  > - *"Fill by"*: `ident`
  {: .hands_on}

  ![Violin Plot split by Genotype--Mutant versus Control](../../images/scrna-case_FPE_SeuratTools/nCount_split_by_Genotype_vln_plot.png "Violin Plot of counts split by Genotype--Mutant versus Control.")

# Finding Our Filtering Parameters
Now that we have a better understanding of what our data looks like, we can begin identifying those spurious reads and low quality cells and then remove them.

In a standard workflow, we would plot the percentage of mitochondrial reads (perc.mt) against the total number of reads per cell (nCount_RNA) and number of genes per cell (nFeature_RNA) to get an idea of our thresholds.

><comment-title>High Mitochondrial Reads</comment-title>
>High perc.mt will typically indicate stressed out cells (often due to the extraction, sorting, or sample prep protocols).
>
{: .comment}

However, this is not always necessary, and in fact, filtering based on counts and features (indeed, even just counts alone) will often remove the cells with spuriously high mitochondrial transcripts. These tools (and this tutorial) will soon be updated to allow us to do so--in the meantime, please see [Filter, Plot, and Explore single cell RNA-seq data (Seurat, R)]({% link topics/single-cell/tutorials/scrna-case_FilterPlotandExploreRStudio/tutorial.md %}) [Filter, plot and explore single-cell RNA-seq (Scanpy)]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}), or [Filter, plot and explore single-cell RNA-seq data (Scanpy, Python)]({% link topics/single-cell/tutorials/scrna-case-jupyter_basic-pipeline/tutorial.md %}) if you hope to include perc.mt in your own data analysis adventures.

For now, we will use just transcript and gene counts to filter our data. Let's take a look back at our nFeature Violin Plot to pick our gene threshold:

![Violin Plot of Features](../../images/scrna-case_FPE_SeuratTools/nFeature_RNA_vln_plot.png "Violin Plot of features.")

><comment-title>Interpretations</comment-title>
>You can see that very few cells in the dataset contain fewer than ~500 genes. Biologically, this makes sense, and the cells appear to be outliers in the data. As such, we will set our lower threshold of genes (nFeature_RNA) at 500.
{: .comment}

Now, what about transcripts (nCount_RNA)? Let's take a look:

![Violin Plot of Counts](../../images/scrna-case_FPE_SeuratTools/nCount_RNA_vln_plot.png "Violin Plot of counts.")

><comment-title>Interpretations</comment-title>
>This one is a bit more difficult to visualize but we see a severe drop off in the number of cells that contain fewer than 500 and more than 10,000 transcripts. These will be our nCount thresholds that we filter based on.
{: .comment}

These cells won't tell us much biologically, rather, they will contribute noise that we'll want to filter out of the data. With that being said, filtering scRNA-seq data will always be an iterative process--so label your work well and be ready to revisit these thresholds if your analyses seem strange down the line.

# Applying our Thresholds
It’s time to filter our cells by applying the above thresholds!

In order to include more than one parameter by which to filter, use the "Insert Subsets used to filter cells" button below the first parameter box.

> <hands-on-title>Filter Cells</hands-on-title>
>
> Run{% tool [Seurat FilterCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_filter_cells/seurat_filter_cells/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Initial Seurat Object` (output of **Seurat Read10x** {% icon tool %})
> - In *"Subsets used to filter cells"*:
>    - *"Name of Parameter to filter on"*: `nCount_RNA`
> - *"Min value"*: `500.0`
> - *"Max value"*: `10000`
>
> - {% icon param-repeat %} *"Insert Subsets used to filter cells"*
>     - *"Name of Parameter to filter on"*: `nFeature_RNA`
> - *"Min value"*: `500.0`
> - *"Max value"*: `1000000000.0`
>
> - *"Choose the format of the output"*: `RDS with a Seurat object`
>
> **Rename** {% icon galaxy-pencil %} output `Filtered Seurat Object`
{: .hands_on}

In this step we are creating a new Seurat object (notice that the selected output of this tool will be an RDS file as opposed to the png plots we have thus far been creating).

Now, genes that do not appear in any cell, or even in only 1 or 2 cells, may break some analytical tools and will generally not be biologically informative.

Since you’ve removed a whole heap of cells, and the captured genes are sporadic (i.e. a small percentage of the overall transcriptome per cell) this means there are a number of genes still present in your matrix that are not expressed in any of the cells.

The removal of these genes is by no means necessary, but will speed up your analyses. The developers are currently working to enable a means of doing this through the Seurat Tools, but, in the meantime if you are analyzing your own data and would like to filter genes--please see [Filter, Plot, and Explore single cell RNA-seq data (Seurat, R)]({% link topics/single-cell/tutorials/scrna-case_FilterPlotandExploreRStudio/tutorial.md %}) [Filter, plot and explore single-cell RNA-seq (Scanpy)]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}), or [Filter, plot and explore single-cell RNA-seq data (Scanpy, Python)]({% link topics/single-cell/tutorials/scrna-case-jupyter_basic-pipeline/tutorial.md %}).

# Processing
Currently, we still have quite big data. We have two issues here
 1. We already saw in our filtering plots that there are differences in how many transcripts and genes have been counted per cell. This technical variable could, and likely will, distract us from identifying true biological differences.
 2. We like to plot things on 2-dimensional X/Y plots. So, for instance, Gapdh could be on one axis, and Actin could be on another, and then each cell is plotted onto that 2D axis based on how many of each transcript they possess.

Although that would be fine, adding in a 3rd dimension (or, indeed, in our case, a dimension for each of the thousands of genes), is a bit trickier.

So, our next steps will be to transform our big data object into something that is easy to analyse and easy to visualize: this is commonly referred to as preprocessing of the data and a typical scRNA-seq preprocessing pipeline will include the following steps:

## 1. Normalization

What is Normalization?

Normalisation helps reduce the differences between gene and UMI counts by fitting total counts across cells in our data to be comparable to one another. SCTransform regularizes the gene expression profiles via a negative binomial regression while also controlling for overfitting of the data.

> <hands-on-title>Normalize Data</hands-on-title>
>
> Run{% tool [Seurat NormaliseData](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_normalise_data/seurat_normalise_data/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Filtered Seurat Object` (output of **Seurat FilterCells** {% icon tool %})
> - *"Normalisation method"*: `Log Normalise`
>
> **Rename** {% icon galaxy-pencil %} output `Normalised Seurat Object`
{: .hands_on}

## 2. Identifying Variable Genes

What are variable genes?

The datasets have loads of genes, but not all of them vary in expression from cell to cell. For instance, housekeeping genes are defined as not changing much from cell to cell, so we could remove these from our data to simplify our analyses.

The find variable genes step flags genes that *do* vary across cells to expedite future analyses and ensure that we, and Seurat, don't waste time looking for meaningful differences where they don't exist.

> <hands-on-title>Find Variable Genes</hands-on-title>
>
> Run{% tool [Seurat FindVariableGenes](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_variable_genes/seurat_find_variable_genes/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Normalised Seurat Object` (output of **Seurat NormaliseData** {% icon tool %})
> - *"Choose the format of the output"*: `RDS with a Seurat object`
>
> **Rename** {% icon galaxy-pencil %} output `Normalised Seurat Object with Variable Features`
{: .hands_on}

This tool will output two new pieces of data into our Galaxy history:
  1. a new Seurat object with variable features identified and flagged
  2. a tabular file with a list of these variable genes.

This gene list may be used as a sneak peak into understanding what the dataset will look like! We can begin to understand which genes are going to be driving downstream clustering of our cells and maybe even make some decisions about whether we are happy with our filtering based on this list.

## 3. Scale Data

Now we will scale the data.

What is scaling?

This is an important step to set up our data for further dimensionality reduction. It will transform the dataset such that all genes have the same variance and a zero mean. It helps negate sequencing depth differences between samples, since the gene levels across the cells become comparable.

><comment-title>Don't Worry!</comment-title>
> Note, that the differences from scaling etc. are not the values you have at the end - i.e. if your cell has average GAPDH levels, it will not appear as a ‘0’ when you calculate gene differences between clusters.
>
{: .comment}

> <hands-on-title>Scale Data </hands-on-title>
>
> Run{% tool [Seurat ScaleData](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_scale_data/seurat_scale_data/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Normalised Seurat Object with Variable Features` (output of **SeuratFindVariableGenes** {% icon tool %})
> - *"Choose the format of the output"*: `RDS with a Seurat object`
> - *"Genes to use"*: `Seurat FindVariableGenes on data 12: Variable genes tabular file`
> - *"Vars to regress"*: `nCount_RNA`
> - *"Statistical model"*: `Linear model`
>
> **Rename** {% icon galaxy-pencil %} output `Scaled, Preprocessed Seurat Object`
{: .hands_on}

You now have a preprocessed Seurat object!

><comment-title>Regressing Variables</comment-title>
> Take note of the "Vars to regress" argument in the above tool. This function allow us to mitigate the effects of confounding factors in our dataset.
> In true research practice, I often regress out multiple variables including but not limited to perc.mt, cell cycle scoring, and feature count.
> As currently written, this tool only allows us to regress out a single variable: so feel free to pick another to regress and see how it changes the downstream analyses!
{: .comment}

# Dimensionality Reduction
Although we've made our expression values comparable to one another and our overall dataset less computationally demanding, we still have way too many dimensions (n cells x n genes!).

Transcript changes are not usually singular--which is to say, genes function and exist in pathways and groups. It would be easier to analyse our data if we could group these differences. To address this we will run principal component analysis (PCA).

><comment-title>What is PCA?</comment-title>
>Principal components (PCs) are calculated from highly dimensional data to find the most representative spread in the dataset. So in our highly variable gene dimensions, there will be one line (axis) that yields the most spread and variation across the cells. That will be our first principal component.
{: .comment}

We can calculate the first handful of principal components in our data to drastically reduce the number of dimensions:

><tip-title>Running Computationally Demanding Steps on Variable Features </tip-title>
>You'll notice that the **Seurat RunPCA** tool is run using the variable features from the previous step. This signficantly decreases the number of genes, and their expression changes, that must be grouped into principal components by this step.
{: .tip}

> <hands-on-title>Run PCA </hands-on-title>
>
> Run{% tool [Seurat RunPCA](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_run_pca/seurat_run_pca/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Scaled, Preprocessed Seurat Object` (output of **Seurat ScaleData** {% icon tool%})
> - *"Choose the format of the output"*: `RDS with a Seurat object`
> - *"Genes to scale"*: `Seurat FindVariableGenes on data 12: Variable genes tabular file`
>
> **Rename** {% icon galaxy-pencil %} RDS output `PCA Processed Seurat Object`
{: .hands_on}

This tool will output you with four new datasets into your history:
  1. Seurat RDS which includes all of the following PCA metadata
  2. Embeddings: Principal component values for each of the cells in your dataset
  3. Loadings: Prinicial component values for each of the genes in your dataset
  4. Standard deviations of each principal component coordinates

><tip-title>Visualizing PCA</tip-title>
>In order to use the PCA information which was just calculated, we must visualize it. The currently available tools unfortunately do not [YET] carry the capacity to do so, but I will provide the plot so that we may make an informed decision together:
>
>![Elbow Plot of Principal Components (PCs) against the standard deviation (variability in the dataset) they each account for](../../images/scrna-case_FPE_SeuratTools/ElbowPlot.png "Elbow Plot")
>
>We can see that there is really not much variation explained past the 9th PC. So we might save ourselves a great deal of time and muddied data by focusing on the top 15 PCs to be conservative.
>You can also think about it like choosing a threshold of variance explained. Conservatively, 2.5 standard deviations are explained by about 10 of the PCs.
{: .tip}

We’re still looking at around 15 dimensions at this point--likely not the easiest to visualize. To make our lives easier, we need to identify how similar a cell is to another cell, across every cell across each of these dimensions.

For this, we will use the k-nearest neighbor (kNN) graph, to identify which cells are close together and which are not.

The kNN graph plots connections between cells if their distance (when plotted in this 10 dimensional space!) is amongst the k-th smallest distances from that cell to other cells. This will be crucial for identifying clusters, and is necessary for plotting a UMAP--which is what will ultimately allow us to visualize our data in 2 dimensions.

><comment-title>From UMAP developers:</comment-title>
>“Larger neighbor values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15 being a sensible default”.
{: .comment}

Let's now use the 15 PC threshold we chose from the Elbowplot and apply it to find neighbors:

> <hands-on-title>Find Neighbors </hands-on-title>
>
> Run{% tool [Seurat FindNeighbours](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_neighbours/seurat_find_neighbours/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `PCA Processed Seurat Object` (output of **Seurat RunPCA** {% icon tool %})
> - *"Reduction"*: `pca`
> - *"Dimensions"*: `1,2,3,4,5,6,7,8,9,10,11,12,13,14,15`
> - *"Assay"*: `RNA`
>
> **Rename** {% icon galaxy-pencil %} output `Preprocessed Seurat Object with Neighbors`
{: .hands_on}

Now we can use the neighborhood graph to identify clusters of cells whose transcriptional profiles appear most similar to one another: we can identify and label clusters:

> <hands-on-title>Find Clusters </hands-on-title>
>
> Run{% tool [Seurat FindClusters](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_clusters/seurat_find_clusters/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Preprocessed Seurat Object with Neighbors` (output of **Seurat FindNeighbors** {% icon tool %})
> - In *"Advanced Options "*
>   - *"Resolution"*: `0.5`
>
> **Rename** {% icon galaxy-pencil %} output `Preprocessed Seurat Object with Clusters`
{: .hands_on}

This tool will output two new datasets: as usual, a new Seurat object which includes a metadata column denoting which cluster each cell was assigned to, and a csv file of the same information.

Unfortunately, identifying clusters is not as majestic as biologists often think - the math doesn’t necessarily identify true cell clusters. Every algorithm for identifying cell clusters falls short of a biologist knowing their data, knowing what cells should be there, and proving it in the lab.

So, we’re going to make the best of it as a starting point and see what happens! We will define clusters from the kNN graph, based on how many connections cells have with one another. Roughly, this will depend on a resolution parameter for how granular you want to be.

><tip-title>On Clustering Resolution</tip-title>
>The resolution parameter available in the advanced options of this tool allows for you, the bioinformatician, to dictate the granularity of the clusters.
>
>For example, a higher clustering resolution dictates increased granularity, and more stringent clusters. That is--cells must more closely resemble one another in order to be grouped into the same cluster than at a lower clustering resolution.
>
>In general, I find it easiest to think of a higher resolution producing more clusters and conversely, a lower resolution will produce less clusters. This parameter is a useful one that you will use often to help decipher how many true populations of cells are present in your data!
{: .tip}

Now that we have made note within our object of which cells cluster together, we can start to visualize our data! Two major visualizations for this data type are tSNE and UMAP. We can calculate the coordinates for both prior to visualization. For tSNE, the parameter perplexity can be changed to best represent the data, while for UMAP the main change would be to change the kNN graph above itself, via the **Seurat FindNeighbors** tool.

><tip-title>On UMAP</tip-title>
>UMAP is the most recently developed, and most widely used dimensionality reduction for visualization of principal component data. It has been optimized since tSNE to better preserve global structure and is less computationally demanding.
>
{: .tip}

> <hands-on-title>Run UMAP </hands-on-title>
>
> Run{% tool [Seurat UMAP](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_run_umap/seurat_run_umap/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Preprocessed Seurat Object with Clusters` (output of **Seurat FindClusters** {% icon tool %})
> - *"Choose the format of the output"*: `RDS with a Seurat object`
> - *"Dims"*: `1:15`
>
> **Rename** {% icon galaxy-pencil %} output `Final Preprocessed Seurat Object`
{: .hands_on}

You now have a completely preprocessed and ready to be analyzed Seurat object--congratulations!

# Let's Take a Look
Now that we have run dimensionality reduction on our dataset, it is ready for visualization. Let's take a look at what our cells look like in a UMAP projection:

> <hands-on-title>Plot UMAP </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `DimPlot`
> - *"Group by"*: `RNA_nn_res.0.5`
{: .hands_on}

![DimPlot colored by 0.5 resolution clusters with a total of 8 clusters--two noticeably different "island" populations and six clustering more closely to one another](../../images/scrna-case_FPE_SeuratTools/DimPlot_GroupBy_pt5Res.png "DimPlot colored by 0.5 resolution cluster.")

Good work! It looks like with a clustering resolution of 0.5, we are able to identify 8 clusters of cells in our data.

We can also look for expression of particular genes and see how those map to our UMAP projection. This is often useful in getting an initial understanding of which clusters might be representative of which cell types.

> <hands-on-title>Plot Gapdh </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Gapdh`
{: .hands_on}

![FeaturePlot of Gapdh expression shown boradly across the whole dataset](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_Gapdh.png "FeaturePlot: Gapdh")

We just plotted a housekeeping gene, Gapdh, so the broad expression we observe is expected.

In practice, it is helpful to plot known markers of cell types you expect to be in your dataset. This will give you a first look at how your cells are clustered.

For example, we can plot early T-cell marker Il2ra and get an idea of which cells and/or clusters might resemble the early T-cells:

> <hands-on-title>Plot Il2ra </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Il2ra`
{: .hands_on}

![FeaturePlot of Il2ra expression most highlky expressed in the lower island population (cluster 2)](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_Il2ra.png "FeaturePlot: Il2ra")

It is a good idea, when analyzing your own data, to plot some markers of cell types you expect to be present. Later on we can also use these FeaturePlots to visualize manual annotation of clusters.

# Differential Expression Testing: Finding Markers
Because each cluster of cells was grouped based on similar transcriptome profiles, each cluster will inherently differ from one another based on a set of "marker" genes.

Following an initial look at the DimPlots and FeaturePlots, we can take an even closer look at which genes are driving the clustering.

To do so, we'll run Seurat's FindMarkers function, which will compare each identity (in this case cluster) against every other identity within its class (all the other clusters). This function of marker finding is particularly useful in identifying up, or down, regulated genes that drive differences in identity/cluster.

> <hands-on-title>Find Markers </hands-on-title>
>
> Run{% tool [Seurat FindMarkers](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_markers/seurat_find_markers/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
{: .hands_on}

The marker list that has been output by this tool will be useful to us shortly for identifying which cells types are represented by the various clusters.\

><comment-title>On Finding Markers</comment-title>
> Differential expression can be run using any identity class and any two identities within the same class. The tools as of current do not allow for such direct comparisons (but will soon!) and I implore you to try [Filter, Plot, and Explore single cell RNA-seq data (Seurat, R)]({% link topics/single-cell/tutorials/scrna-case_FilterPlotandExploreRStudio/tutorial.md %}) if you hope to conduct marker identification across, say, genotypes.
{: .comment}

# Biological Interpretations
Now it’s the fun bit! We can see where genes are expressed, and start considering and interpreting the biology of it. At this point, it’s really about what information you want to get from your data--the following is only the tip of the iceberg. However, a brief exploration is good, because it may help give you ideas going forward for your own data. Let's start interrogating our data!

Let's take another look at what our clusters look like:

> <hands-on-title>Plot UMAP </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `DimPlot`
> - *"Group by"*: `RNA_nn_res.0.5`
{: .hands_on}

![DimPlot colored by 0.5 resolution cluster showing 8 total clusters](../../images/scrna-case_FPE_SeuratTools/DimPlot_GroupBy_pt5Res.png "DimPlot colored by 0.5 resolution cluster.")

><comment-title>On Cluster Numbering</comment-title>
>Note that Seurat's cluster numbering is based on size alone, so clusters 0 and 1 are not necessarily related, they are just the clusters containing the most cells.
{: .comment}

It would be nice to know what these cells are. This analysis (googling all of the marker genes, both checking where the ones you know are and then going through marker tables we generated) is a fun task for any individual experiment, so we’re going to speed past that and nab the assessment from the original paper!

Here are the markers per cell type that the paper uses to classify the avrious cell types which are expected to be present in the data:

| Markers                 | Cell Type                           |
|-------------------------|-------------------------------------|
| Il2ra                   | Double negative (early T-cell)      |
| Cd8b1, Cd8a, Cd4        | Double positive (middle T-cell)     |
| Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell)|
| Itm2a                   | Mature T-cell                       |

We can plot these markers as a means of discerning which cluster might be representing a given cell type. Let's start with the Mature T-cell marker Itm2a:

> <hands-on-title>Plot Itm2a </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Itm2a`
{: .hands_on}

![FeaturePlot of Itm2a expression most highly localized to cluster 3](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_Itm2a.png "FeaturePlot of Itm2a")

We can see that Cluster 3 seems to be most highly expressing this gene, and when we look back at the marker list we created earlier, Itm2a appears as a marker for cluster 3 with an average log fold change of 3! Therefore, we can quite conficently say that cluster 3 is likely to be representing Mature T-cells!

Now what about the opposite end of the spectrum: the double negative early T-cells? We can see from our table about that Il2ra is a known marker of this cell type, let's see where it's most strongly expressed in the data:

> <hands-on-title>Plot Il2ra </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Il2ra`
{: .hands_on}

![FeaturePlot of Il2ra expression mainly in cluster 2](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_Il2ra.png "FeaturePlot of Il2ra")

Il2ra expression seems to be most prominent in Cluster 2 and as seen in our marker list, has a fold change of 2.79!

It can, and should, act as a sanity check that the most (and least) differentiated cell types expected in the data appear as two "island" clusters.

Now for the intermediate populations--which may be a bit more tricky to deconvolute. Based on our known markers, we see that both the double-positive populations express many of the same genes: Cd8b1, Cd8a, and Cd4. Let's make sure that the remaining clusters (0, 1, 3, 4, 5, 6, and 7) all express these:

> <hands-on-title>Plot Cd8b1 </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Cd8b1`
{: .hands_on}

![FeaturePlot of Cd8b1 expression expressed across each of the "body" clusters (0, 1, 4, 5, 6, and 7)](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_Cd8b1.png "FeaturePlot of Cd8b1")

It looks like clusters 1, 4, 5, and 6 pretty strongly express Cd8b1, now what about Cd8a?

> <hands-on-title>Plot Cd8a </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Cd8a`
{: .hands_on}

![FeaturePlot of Cd8a expression in the same clsuters as Cd8b1 (0, 1, 4, 5, 6, and 7)](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_Cd8a.png "FeaturePlot of Cd8a")

This looks pretty consistent with the Cd8b1 plot, which is expected as these are markers of a double positive population! The true discernment between the two populations will be the level of Cd4 expression:

> <hands-on-title>Plot Cd4 </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Cd4`
{: .hands_on}

![FeaturePlot of Cd4 expression most highly expressed in cluster 1 and lowly expressed in clusters 4, 5, 6, and 7](../../images/scrna-case_FPE_SeuratTools/FeatuerePlot_Cd4.png "FeaturePlot of Cd4")

Looks like the top portion of these clusters, and mainly cluster 1 hold the vast majority of Cd4 expression. This, coupled with Cd4 being identified as a marker of cluster 1 in our marker table above tells us pretty confidently that the upper part of this "body" cluster are likely the late middle t-cells!

Here is a breakdown of the cell type labeling we just accomplished:

| Clusters     | Markers                 | Cell Type                           |
|--------------|-------------------------|-------------------------------------|
| 2            | Il2ra                   | Double negative (early T-cell)      |
| 0            | Il2ra, Cd8b1, Cd8a      | [maybe] Early transitioning T-cell  |
| 4, 5, 6, 7   | Cd8b1, Cd8a, Cd4        | Double positive (middle T-cell)     |
| 1            | Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell)|
| 3            | Itm2a                   | Mature T-cell                       |

Now we can begin to feel a bit more oriented in exploring our data. The clusters are labelled with cell types, and our object has been processed enough such that we may now begin to answer some real biological questions! Now that we know what we’re dealing with, let’s examine the effect of our variable, real science!

## Keep Digging
Are there any differences in genotype? Or in biological terms, is there an impact of growth restriction on T-cell development in the thymus? We can begin to answer this question visually by using the "split.by" parameter in Seurat's plot functions.

> <hands-on-title>Visualize Counts Split by Genotype</hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `VlnPlot`
> - *"Features"*: `nCount_RNA`
> - *"Group by"*: `RNA_nn_res.0.5`
> - *"Split by"*: `Sample.Characteristic.genotype.`
> - *"Log"*: `Yes`
> - *"Fill by"*: `ident`
{: .hands_on}

![VlnPlot colored by cluster and split by genotype showing the mutant clusters appearing more sparse than the control](../../images/scrna-case_FPE_SeuratTools/VlnPlot_GroupedbyCluster_SplitbyGenotype.png "Violin Plot colored by assigned clusters & split by genotype")

We can see that there seems to be a decrease in cellcounts across the celltypes in the het mutant... INTERESTING! What next? We might look further at the transcripts present in both those populations, and perhaps also look at the genotype marker table… So much to investigate! But before we set you off to explore to your heart’s delight, let’s also look at this a bit more technically.

# Technical Assessment
Is our analysis real? Is it right? Well, we can assess that a little bit.

First thing's first, is there a batch effect?

> <hands-on-title>Visualize Counts Split by Individual</hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `DimPlot`
> - *"Group by"*: `RNA_nn_res.0.5`
> - *"Split by"*: `Sample.Characteristic.individual.`
{: .hands_on}

![DimPlot colored by labelled celltype split by individual/batch showing that individual 3 and 4 contribute the vast majority of immature T-cells](../../images/scrna-case_FPE_SeuratTools/DimPlotColorbyClusters_SplitbyIndividual.png "DimPlot colored by assigned clusters split by individual/batch")

While some differences across batch are expected and nothing to be concerned about, the immature T-cells looks to be mainly comprised of Individuals 3 and 4. There might be a bit of batch effect, so you could consider using batch correction on this dataset. However, if we focus our attention on the other cluster - mature T-cells - where there is batch mixing, we can still assess this biologically even without batch correction.

Additionally, we will also look at the confounding effect of sex:

> <hands-on-title>Visualize Counts Split by Sex</hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `DimPlot`
> - *"Group by"*: `RNA_nn_res.0.5`
> - *"Split by"*: `Sample.Characteristic.sex.`
{: .hands_on}

![DimPlot colored by cluster and split by sex showing that there are fewere female cells (a product of fewer female samples being present in the data as a whole)](../../images/scrna-case_FPE_SeuratTools/DimPlot_SplitbySex.png "DimPlot colored by assigned clusters and split by Sex")

We note that the one female sample - unfortunately one of merely three knockout samples - seems to be distributed in the same areas as the knockout samples at large, so luckily, this doesn’t seem to be a confounding factor and we can still learn from our data. Ideally, this experiment would be re-run with either more female samples all around or swapping out this female from the male sample.

Are there any clusters or differences being driven by sequencing depth, a technical factor?

> <hands-on-title>Plot Counts </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `nCount_RNA`
{: .hands_on}

![FeaturePlot colored by counts showing no differences in sequencing depth across the dataset](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_nCount.png "FeaturePlot colored by counts")

There doesn't visually appear to be any differences in sequencing depth across the clusters, but let's check out some of those other variables we grouped by:

> <hands-on-title>Plot Features </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `nCount_RNA`
> - *"split.by"*: `Sample.Characteristic.individual.`
{: .hands_on}

![FeaturePlot colored by counts & Split by Individual showing that Individual 3 seems to have been most deeply sequenced](../../images/scrna-case_FPE_SeuratTools/FeaturePlot_Counts_SplitbyIndividual.png "FeaturePlot colored by counts & split by individual")

There we go! This might explain the dramatic shift in early to middle T-Cell between wildtype and knockout cells--the leftmost early to middle T-cells simply have a higher sequencing depth represented by Individual 3 (UMIs/cell) than the ones on the right side. Well, that explains some of the sub-cluster that we’re seeing in that splurge (specifically this likely accounts for the discernment between clusters 0, 4, 5, 6, and 7).

Luckily, and importantly, we don’t see the double negative or mature T-cells being similarly affected. So, although, this variable of sequencing depth, or moreso, Individual, might be something to regress out somehow, it doesn’t seem to be impacting our dataset such that we cannot draw meaningful insights.

><tip-title>Overprocessing</tip-title>
>The less you can regress/modify your data, in general, the better--you want to stay as true as you can to the raw data, and only use maths to correct your data when you really need to (and not to create insights where there are none!).
{: .tip}

Do you think we processed these samples well enough? We have seen in the previous images that these clusters are not very tight or distinct, so we could consider stronger filtering. Let's take a look at gene expression of a gene we know should not be expressed in tCells as a sanity check:

> <hands-on-title>Plot Hba-a1 </hands-on-title>
>
> Run{% tool [Plot with Seurat](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0) %} with the following parameters:
> - *"RDS file"*: `Final Preprocessed Seurat Object` (output of **Seurat UMAP** {% icon tool %})
> - *"Plot_type_selector"*: `FeaturePlot`
> - *"Features"*: `Hba-a1`
{: .hands_on}

![FeaturePlot of Hba-a1 showing sparse expression across the whole dataset](../../images/scrna-case_FPE_SeuratTo

tools_json:
["EBI SCXA Data Retrieval", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2", "Seurat Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/4.0.4+galaxy0", "Seurat FilterCells", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_filter_cells/seurat_filter_cells/4.0.4+galaxy0", "Plot with Seurat", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_plot/seurat_plot/4.0.4+galaxy0", "Seurat NormaliseData", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_normalise_data/seurat_normalise_data/4.0.4+galaxy0", "Seurat FindVariableGenes", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_variable_genes/seurat_find_variable_genes/4.0.4+galaxy0", "Seurat ScaleData", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_scale_data/seurat_scale_data/4.0.4+galaxy0", "Seurat RunPCA", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_run_pca/seurat_run_pca/4.0.4+galaxy0", "Seurat FindNeighbours", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_neighbours/seurat_find_neighbours/4.0.4+galaxy0", "Seurat FindClusters", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_clusters/seurat_find_clusters/4.0.4+galaxy0", "Seurat UMAP", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_run_umap/seurat_run_umap/4.0.4+galaxy0", "Seurat FindMarkers", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_find_markers/seurat_find_markers/4.0.4+galaxy0"]


topic: single-cell
tutorial: scanpy_parameter_iterator
title: Scanpy Parameter Iterator
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: e655353b0579c2fadcfd880f206069cd
last_modified: 2026-05-15T22:28:16.665526
zenodo_link: https://zenodo.org/record/8011681
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scanpy_parameter_iterator/tutorial.html

content:
The magic of bioinformatic analysis is that we use maths, statistics and complicated algorithms to deal with huge amounts of data to help us investigate biology. However, analysis is not always straightforward – each tool has various parameters to select. Eventually, we can end up with very different outcomes depending on the values we choose. With analysing scRNA-seq data, it’s almost like you need to know about 75% of your data, then make sure your analysis shows that, for you to then be able to identify the 25% new information.

Given the vast array of values that we can specify in the tool parameters, how can we know if the values we choose are the most optimal ones - or at least good enough? Well, we can try different values in our workflow and then compare the outputs to see which is consistent with our understanding of the underlying biology. But can we do this efficiently, at scale, to test multiple values?

And here the Parameter Iterator comes in – it allows us to test different variables quickly and easily. This tutorial will show you how to use Parameter Iterator to generate multiple outputs with different parameter values in one go.

{% snippet faqs/galaxy/tutorial_mode.md %}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get Data
The data used in this tutorial is from a mouse dataset of fetal growth restriction ({% cite Bacon2018 %}). You can download the dataset below or import the history with the starting data.

Here are several ways of getting our toy dataset – choose whichever you like!

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. Import history from: [example input history](https://usegalaxy.eu/u/j.jakiela/h/scanpy-parameter-iterator-1)
>
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the the history to your name of choice.
>
{: .hands_on}

> <hands-on-title>Option 2: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Scanpy_RunPCA_AnnData_object.h5ad
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>    
>
> 3. Rename the dataset if you wish: `Scanpy RunPCA: AnnData object`
>
>    {% snippet  faqs/galaxy/datasets_rename.md %}
>
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}


# Workflow

This tutorial is an extension of the full analysis shown in [Filter, Plot and Explore Single-cell RNA-seq Data]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial in the Single-cell RNA-seq: Case Study series. So if you've been working through it, you can use your dataset from that tutorial here. If you haven’t completed it but you’re interested in how we get to this point, feel free to have a look at the mentioned tutorial.

Our starting data will be the output of **Scanpy RunPCA** {% icon tool %}. It is part of the full analysis tutorial, but we will only focus on a smaller and shortened bit of the [full workflow](https://usegalaxy.eu/u/j.jakiela/w/copy-of-filter-plot-and-explore-single-cell-rna-seq-data-imported-from-uploaded-file-3) to show the application of the Parameter Iterator. Our workflow consists of the following steps:

![Workflow that we are going to use in this tutorial: Scanpy RunPCA, Scanpy ComputeGraph, Scanpy RunTSNE, Scanpy RunUMAP, Scanpy FindCluster, Scanpy PlotEmbed](../../images/scrna-casestudy_parameter-iterator/workflow.png "Shortened workflow that we are going to use in this tutorial.")

\
For the detailed explanation of the tools presented above, check out [this tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}).


# Inputs
Scanpy ParameterIterator tool currently works only for the following parameters:
1.	Number of neighbours to derive kNN graph (for **Scanpy ComputeGraph** {% icon tool %})
2.	Perplexity (for **Scanpy RunTSNE** {% icon tool %})
3.	Resolution (for **Scanpy FindCluster** {% icon tool %})

There are two formats of the input values:
1.	List of all parameter values to be iterated
2.	Step increase values to be iterated


# Number of neighbours to derive kNN graph (for **Scanpy ComputeGraph** {% icon tool %})

We will now use **Scanpy ComputeGraph** {% icon tool %} to derive the k-nearest neighbour (kNN) graph from our PCA values. We can use the Parameter Iterator to check how the different k values of nearest neighbours will affect the final outcome. It is important that **k neighbours is an integer**.

> <warning-title>Float vs integer</warning-title>
> Using ‘Step increase values to be iterated’ as the format of the input values automatically generates float values instead of integers. Float, or floating point numbers, are values with a 'floating' decimal point. To avoid float values, you must use ‘List of all parameter values to be iterated’ as your chosen values.
{: .warning}

The kNN graph will be needed for plotting a UMAP. According to the [UMAP developers](https://github.com/lmcinnes/umap): “Larger neighbor values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15 being a sensible default”. Therefore, let’s pick some values bigger and smaller than 15 to check how it changes the final UMAP. This is where the Parameter Iterator comes in!

> <hands-on-title> Set your values in Parameter Iterator </hands-on-title>
>
> 1. {% tool [Scanpy ParameterIterator](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_parameter_iterator/scanpy_parameter_iterator/0.0.1+galaxy9) %} with the following parameters:
>    - *"Parameter type"*: `n-neighbours`
>    - *"Choose the format of the input values"*: `List of all parameter values to be iterated`
>    - *"User input values"*: `5,10,15,20,25,30,35,40`
>
> 2. **Rename** {% icon galaxy-pencil %} the resulting list of datasets: `Parameter iterated - n-neighbours` (you have to first click on the collection so that you see the datasets, and then rename it)
>
> 3. **Tag** {% icon galaxy-tags %} each dataset with its corresponding value:
>    - navigate to *Show hidden* ({% icon galaxy-show-hidden %} icon)
>    - add tags accordingly - *n-neighbours_10*: `#n-neighbours_10` etc.
>    - If you want to refresh your memory on how to add tags to datasets, have a look here:
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}




The output of the Parameter Iterator is the list of datasets. We will be working on dataset collections quite a lot, so if you want to gain more understanding of collection operations, visit the [corresponding tutorial]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}).

> <hands-on-title> Derive kNN graph with iterated parameter </hands-on-title>
>
> 1. {% tool [Scanpy ComputeGraph](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `Scanpy RunPCA: AnnData object`
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"File with n_neighbours, use with parameter iterator. Overrides the n_neighbors setting"*:
>       - Click on {% icon param-collection %} (*Dataset collection*)
>       - Choose `Parameter iterated - n-neighbours`
>    - *"Use the indicated representation"*: `X_pca`
>    - *"Number of PCs to use"*: `20`
>
> 2. **Rename** {% icon galaxy-pencil %} the resulting list of datasets: `Scanpy ComputeGraph on collection X: Graph object AnnData (n-neighbours)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
{: .hands_on}

You should now see the output is also a collection. If you click on that, you will see Anndata files, differing only by the n-neighbour value.

Now you have two options, either:
 - pick one of the generated output files and proceed to the next tool with another parameter iteration; or
 - continue with the current collection of datasets.

We choose the second option as only then you will be able to see the effect of using different nearest neighbour values. However, the disadvantage of this option is that you have to select only one value for the subsequent parameters in the workflow in order to see the changes in the final plots.

> <comment-title>Why only one Parameter Iteration at a time?</comment-title>
> Iterating the parameters within one tool will give you a list with X datasets: each dataset is the output with the given parameter value. However, if you want to use Parameter Iterator again within another tool, specifying Y parameter values, you **will not** get X x Y datasets as you might expect. Therefore you have to choose **just one** output file to be passed on to the next tool which will use Parameter Iterator again. Alternatively, you can use Parameter Iterator once and run the rest of the tools on dataset collection with just one parameter value.
{: .comment}

Where are we now in our workflow?
![Image showing the step we are at: after Scanpy RunPCA, already run Scanpy ComputeGraph, and before Scanpy RunTSNE, Scanpy RunUMAP, Scanpy FindCluster, Scanpy PlotEmbed](../../images/scrna-casestudy_parameter-iterator/workflow_n-neighbours.png "We used the Parameter Iterator for the k nearest neighbours to derive the kNN graph. Now we’ll complete our small workflow to see the differences at the end.")

> <hands-on-title> Complete the workflow </hands-on-title>
>
> 1. {% tool [Scanpy RunTSNE](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_tsne/scanpy_run_tsne/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy ComputeGraph on collection X: Graph object AnnData (n-neighbours)`
>    - *"Use the indicated representation"*: `X_pca`
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"The perplexity is related to the number of nearest neighbours, select a value between 5 and 50"*: `30`
>
> 1A. **Rename** {% icon galaxy-pencil %} the Anndata object collection output: `Scanpy RunTSNE on collection X: tSNE object AnnData (n-neighbours)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
> 2. {% tool [Scanpy RunUMAP](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_umap/scanpy_run_umap/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy RunTSNE on collection X: tSNE object AnnData (n-neighbours)`
>    - *"Use programme defaults"*: {% icon param-toggle %} `Yes`
>
> 2A. **Rename** {% icon galaxy-pencil %} the Anndata object collection output: `Scanpy RunUMAP on collection X: UMAP object AnnData (n-neighbours)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
> 3. {% tool [Scanpy FindCluster](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_find_cluster/scanpy_find_cluster/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy RunUMAP on collection X: UMAP object AnnData`
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"Resolution, high value for more and smaller clusters"*: `0.6`
>
> 3A. **Rename** {% icon galaxy-pencil %} the Anndata object collection output: `Scanpy FindCluster on collection X: Clusters AnnData (n-neighbours)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
{: .hands_on}

The differences will only appear in the UMAP embedding, so we’ll plot only them. However, when you run your own analysis, you might want to check if there are changes in other embeddings as well.

> <hands-on-title> Plot UMAP embedding </hands-on-title>
>
> 1. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy FindCluster on collection X: Clusters AnnData (n-neighbours)`
>    - *"name of the embedding to plot"*: `umap`
>    - *"color by attributes, comma separated texts"*: `louvain`
>    - *"Use raw attributes if present"*: `No`
>
{: .hands_on}

If you click on the resulting collection you will see several plots. Click on {% icon galaxy-eye %} to see how they differ. Galaxy's {% icon galaxy-scratchbook %} Window Manager, which you can enable (and disable again) from the menu bar can be very helpful for comparing multiple datasets.

![Eight graphs showing the differences between UMAP embeddings caused by different values of k nearest neighbours.](../../images/scrna-casestudy_parameter-iterator/n_neighbours.png "Comparison of UMAP embedding with different values of k nearest neighbours, perplexity set to 30 and resolution to 0.6.")

If you compare the UMAP graphs, you can see the differences that were caused by changing the value of k nearestneighbours. Relying on your biological knowledge, you can now choose which parameter value works best and use it for further analysis.

We will go forward with k value equal to 15. But hang on, we’ve been working on a collection and not a single dataset! How can we access that one dataset with the n-neighbour = 15?

Here is the answer: The datasets that are included in the collections can be accessed separately if you go to your history and click on *Show hidden* ({% icon galaxy-show-hidden %} icon). You can bring each individual dataset to the visible and active datasets by clicking *Unhide*.

> <hands-on-title> Unhide the dataset of interest </hands-on-title>
> 1. In *Show hidden* find the dataset `Scanpy ComputeGraph on data X and data Y: Graph object AnnData` with the tag `#n-neighbours_15` (or any value that you want to go forward with).
> 2. Click on *Unhide*
> 3. Your chosen dataset is now visible amongst the active datasets in your history!
>
{: .hands_on}


# Perplexity (for **Scanpy RunTSNE** {% icon tool %})

Let’s have another look at our wee workflow.
![Image showing the step we are at: after Scanpy RunPCA and Scanpy ComputeGraph from which we will take one dataset to pass on to Scanpy RunTSNE with Parameter Iterator.](../../images/scrna-casestudy_parameter-iterator/workflow_perplexity.png "We will take one dataset from the Scanpy ComputeGraph as the input to Scanpy RunTSNE and use the Parameter Iterator with the perplexity parameter.")


The next tool in our workflow is Scanpy RunTSNE, which contains the perplexity parameter. Although the tool description says that this value should be an integer, we tested it with float values and it works. Therefore, you can use ‘Step increase values to be iterated’. Keep in mind that perplexity should take values between 5 and 50. Let’s run the Parameter Iterator again.

> <hands-on-title> Set your values in Parameter Iterator </hands-on-title>
>
> 1. {% tool [Scanpy ParameterIterator](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_parameter_iterator/scanpy_parameter_iterator/0.0.1+galaxy9) %} with the following parameters:
>    - *"Parameter type"*: `perplexity`
>    - *"Choose the format of the input values"*: `Step increase values to be iterated`
>    - *"Starting value"*: `15`
>    - *"Step"*: `5`
>    - *"Ending value"*: `45`
>
> 2. **Rename** {% icon galaxy-pencil %} the resulting list of datasets: `Parameter iterated - perplexity` (you have to first click on the collection so that you see the datasets, and then rename it)
>
> 3. **Tag** {% icon galaxy-tags %} each dataset with its corresponding value:
>    - navigate to *Show hidden* ({% icon galaxy-show-hidden %} icon)
>    - add tags accordingly - *perplexity_15*: `#perplexity_15` etc.
>    - If you want to refresh your memory on how to add tags to datasets, have a look here:
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    
{: .hands_on}

> <hands-on-title> RunTSNE with iterated parameter </hands-on-title>
> 1. {% tool [Scanpy RunTSNE](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_tsne/scanpy_run_tsne/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `Scanpy ComputeGraph on data X and data Y: Graph object AnnData` with the tag `#n-neighbours_15`
>    - *"Use the indicated representation"*: `X_pca`
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"The perplexity is related to the number of nearest neighbours"*:
>       - Click on {% icon param-collection %} (*Dataset collection*)
>       - Choose `Parameter iterated - perplexity`
>
> 2. **Rename** {% icon galaxy-pencil %} the Anndata object collection output: `Scanpy RunTSNE on collection X: tSNE object AnnData (perplexity)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
{: .hands_on}

Changing the value of perplexity will only affect the tSNE graphs, so we can complete the workflow and compare the tSNE plots to choose the best value for further analysis.

> <hands-on-title> Complete the workflow </hands-on-title>
> 1. {% tool [Scanpy RunUMAP](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_umap/scanpy_run_umap/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy RunTSNE on collection X: tSNE object AnnData (perplexity)`
>    - *"Use programme defaults"*: {% icon param-toggle %} `Yes`
>
> 1A. **Rename** {% icon galaxy-pencil %} the Anndata object collection output: `Scanpy RunUMAP on collection X: UMAP object AnnData (perplexity)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
> 2. {% tool [Scanpy FindCluster](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_find_cluster/scanpy_find_cluster/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy RunUMAP on collection X: UMAP object AnnData (perplexity)`
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"Resolution, high value for more and smaller clusters"*: `0.6`
>
> 3A. **Rename** {% icon galaxy-pencil %} the Anndata object collection output: `Scanpy FindCluster on collection X: Clusters AnnData (perplexity)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
{: .hands_on}

> <warning-title></warning-title>
> In *Use programme defaults* you can specify *Additional suffix to the name of the slot to save the embedding* – if it’s included, PERPLEXITY will be substituted with the value of the perplexity setting. However, in that case you will get an error when using **Scanpy PlotEmbed**: due to the value included in the entry name, the tool will not recognise the correct embedding. Therefore, we leave that box unfilled even though it would make it easier to differentiate between datasets.
{: .warning}


> <hands-on-title> Plot tSNE embedding </hands-on-title>
>
> 1. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy FindCluster on collection X: Clusters AnnData (perplexity)`
>    - *"name of the embedding to plot"*: `tsne`
>    - *"color by attributes, comma separated texts"*: `louvain`
>    - *"Use raw attributes if present"*: `No`
>
{: .hands_on}

Can you see those differences?
![Graphs showing the differences between tSNE embeddings caused by different values of perplexity.](../../images/scrna-casestudy_parameter-iterator/perplexity.png "Comparison of tSNE  embedding with different values of perplexity, k nearest neighbours set to 15 and resolution to 0.6.")


What do you think about those plots? Which value would you choose? Perplexity is probably the least intuitive or consistently functioning parameter, so feel free to learn more from [this nifty blog post](https://apiumhub.com/tech-blog-barcelona/dimensionality-reduction-tsne/#:~:text=.tensorflow.org%2F-,Perplexity,are%20between%205%20and%2050.). We will go forward with a perplexity value equal to 30.

# Resolution (for **Scanpy FindCluster** {% icon tool %})

Where is **Scanpy FindCluster** {% icon tool %} in our workflow?
![Image showing the step we are at: we have chosen values for Scanpy ComputeGraph and Scanpy RunTSNE, then proceeded to Scanpy RunUMAP to get to Scanpy FindCluster.](../../images/scrna-casestudy_parameter-iterator/workflow_resolution.png "We have chosen values for Scanpy ComputeGraph and Scanpy RunTSNE, then proceeded to Scanpy RunUMAP to get to Scanpy FindCluster where we can iterate the resolution parameter.")


The input data for the **Scanpy FindCluster** {% icon tool %} is the output of **Scanpy RunUMAP** {% icon tool %}, so let’s get this single dataset with the parameter values (from previous tools) of our choice, that is n-neighbours=15 and perplexity=30.


> <hands-on-title> Unhide the dataset of interest </hands-on-title>
> 1. In *Show hidden* find the dataset `Scanpy RunUMAP on data X: UMAP object AnnData` with the tags `#n-neighbours_15` and `#perplexity_30` (or any value that you want to go forward with).
> 2. Click on *Unhide*
> 3. Your chosen dataset is now visible amongst the active datasets in your history!
>
{: .hands_on}

The last tool that we can use Parameter Iterator for is **Scanpy FindCluster** {% icon tool %}. We will iterate over the resolution values. In this case, those values can be floats, so you can use either ‘List of all parameter values to be iterated’ or ‘Step increase values to be iterated’. Keep in mind that when it comes to the resolution, a high value means more and smaller clusters.

> <hands-on-title> Set your values in Parameter Iterator </hands-on-title>
>
> 1. {% tool [Scanpy ParameterIterator](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_parameter_iterator/scanpy_parameter_iterator/0.0.1+galaxy9) %} with the following parameters:
>    - *"Parameter type"*: `resolution`
>    - *"Choose the format of the input values"*: `Step increase values to be iterated`
>    - *"Starting value"*: `0.2`
>    - *"Step"*: `0.2`
>    - *"Ending value"*: `1.4`
>
> 2. **Rename** {% icon galaxy-pencil %} the resulting list of datasets: `Parameter iterated - resolution` (you have to first click on the collection so that you see the datasets, and then rename it)
>
> 3. **Tag** {% icon galaxy-tags %} each dataset with its corresponding value:
>    - navigate to *Show hidden* ({% icon galaxy-show-hidden %} icon)
>    - add tags accordingly - *resolution_0.2*: `#resolution_0.2` etc.
>    - If you want to refresh your memory on how to add tags to datasets, have a look here:
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>    
{: .hands_on}

> <hands-on-title> FindCluster with iterated parameter </hands-on-title>
> 1. {% tool [Scanpy FindCluster](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_find_cluster/scanpy_find_cluster/1.8.1+galaxy9) %} with the following parameters:
>   - {% icon param-file %} *"Input object in AnnData/Loom format"*: `Scanpy RunUMAP on data X: UMAP object AnnData` with the tags `#n-neighbours_15` and `#perplexity_30`
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"File with resolution, use with parameter iterator. Overrides the resolution setting"*:
>       - Click on {% icon param-collection %} (*Dataset collection*)
>       - Choose `Parameter iterated - resolution`
>
> 2. **Rename** {% icon galaxy-pencil %} the Anndata object collection output: `Scanpy FindCluster on collection X: Clusters AnnData (resolution)` (you have to first click on the collection so that you see the datasets, and then rename it)
>
{: .hands_on}

You can see the effect of resolution parameter on all embeddings: UMAP, tSNE and PCA so you can plot them all and compare the granularity of the clusters. It is also useful to see how the equal increments affect the clustering and what is the rate of change of the granularity.

> <hands-on-title> Plot the embeddings </hands-on-title>
>
> 1. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy FindCluster on collection X: Clusters AnnData (resolution)`
>    - *"name of the embedding to plot"*: `pca`
>    - *"color by attributes, comma separated texts"*: `louvain`
>    - *"Use raw attributes if present"*: `No`
> *You can re-run {% icon galaxy-refresh %} the same tool again, but change `pca` to `tsne` and then finally to `umap` in order to skip the following two steps.*
>
> 2. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy FindCluster on collection X: Clusters AnnData (resolution)`
>    - *"name of the embedding to plot"*: `tsne`
>    - *"color by attributes, comma separated texts"*: `louvain`
>    - *"Use raw attributes if present"*: `No`
>
> 3. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-collection %} *"Input object in AnnData/Loom format"* (make sure you choose *Dataset collection*): `Scanpy FindCluster on collection X: Clusters AnnData (resolution)`
>    - *"name of the embedding to plot"*: `umap`
>    - *"color by attributes, comma separated texts"*: `louvain`
>    - *"Use raw attributes if present"*: `No`
{: .hands_on}

That’s the last tool in our workflow which uses Parameter Iterator! Let’s have a final look at the generated plots.

![Graphs showing the differences between PCA embeddings caused by different values of resolution.](../../images/scrna-casestudy_parameter-iterator/resolution_pca.png "Comparison of PCA embedding with different values of resolution, with n-neighbours set to 15 and perplexity to 30.")

![Graphs showing the differences between tSNE embeddings caused by different values of resolution.](../../images/scrna-casestudy_parameter-iterator/resolution_tsne.png "Comparison of tSNE embedding with different values of resolution, with n-neighbours set to 15 and perplexity to 30.")

![Graphs showing the differences between UMAP embeddings caused by different values of resolution.](../../images/scrna-casestudy_parameter-iterator/resolution_umap.png "Comparison of UMAP embedding with different values of resolution, with n-neighbours set to 15 and perplexity to 30.")

> <comment-title></comment-title>
> Still not sure which value works best? For more explanation and in-depth analysis, please read through [this tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}). Hopefully it will give you more insight into interpretation of the resulting plots.
{: .comment}


# Additional steps
- It may happen that some of the values you choose will give an error, but some will work fine. In that case, you can use {% icon tool %} **Filter failed datasets** tool to remove datasets with errors from a collection.

- If you still haven’t found an answer that would help you with the parameter iteration in your own analysis, check out [another workflow](https://usegalaxy.eu/published/workflow?id=8f677efac7100097) which has some extra steps, not directly related to our analysis. But it might contain steps that would be helpful for you.


# Conclusion
You might want to compare your results with this [control history](https://usegalaxy.eu/u/j.jakiela/h/scanpy-parameter-iterator-updated), or check out the [workflow](https://usegalaxy.eu/u/j.jakiela/w/scanpy-parameter-iterator-workflow) for this tutorial. You can also continue to analyse this data by returning to the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial.


{% icon congratulations %} You have finished the tutorial! You have learned how to use the Parameter Iterator with the nearest neighbours, perplexity and resolution parameters. You also compared multiple outputs resulting from the analysis using different values at three different steps (Scanpy ComputeGraph, Scanpy RunTSNE and Scanpy FindCluster). Hopefully this tool will help you more quickly assess parameter values, ultimately helping you choose values that both confirm prior knowledge as well as offer new insights on biological data.

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Scanpy ParameterIterator", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_parameter_iterator/scanpy_parameter_iterator/0.0.1+galaxy9", "Scanpy ComputeGraph", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.8.1+galaxy9", "Scanpy RunTSNE", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_tsne/scanpy_run_tsne/1.8.1+galaxy9", "Scanpy RunUMAP", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_umap/scanpy_run_umap/1.8.1+galaxy9", "Scanpy FindCluster", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_find_cluster/scanpy_find_cluster/1.8.1+galaxy9", "Scanpy PlotEmbed", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9"]


topic: single-cell
tutorial: scrna-umis
title: Understanding Barcodes
difficulty: intermediate
time_estimation: 2h
requirements_json: ["type: internal", "scrna-intro"]
tags_json: []
content_hash: 23fa411ff8c08504fe6bf9b0b8e18239
last_modified: 2026-05-15T22:28:16.829936
zenodo_link: https://zenodo.org/record/2573177
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-umis/tutorial.html

content:
Barcodes are small oligonucleotides that are inserted into the captured sequence at a specific point, and provide two pieces of information about the sequence:

 1. Which cell the sequence came from.
 2. Which transcript the sequence came from

When the sequence is mapped against a reference genome, we can then see which gene locus it aligns to and qualitatively assert that, together with the two pieces of information above, the sequence depicts a transcript from a specific gene that originated from a specific cell.

Barcodes come in a variety of formats, and in this tutorial we will be looking at the CEL-Seq2 protocol {% cite Hashimshony2016 %} used in droplet-based single-cell RNA-seq.

## The CEL-Seq2 Protocol

CEL-Seq2 is a paired-end protocol, meaning that two primers bind to opposite ends of a cDNA strand in order to sequence. Each primer has a specific role.

In this case; *Read1* contains the barcoding information followed by the polyT tail of the messenger RNA, and *Read2* contains the actual sequence. Here, Read1 is regarded as the 'forward' strand and Read2 as the 'reverse' strand, though this is more a convention when dealing with paired-end data rather than an indication of the actual strand orientation.

 ![CEL-Seq2 Scheme](../../images/scrna-umis/celseq2_schema.svg "Read1 encapsulates the barcodes, and Read2 the mRNA sequence")



> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Understanding Barcodes

Cell barcodes are designed primarily for delineating one cell from another, such that read transcripts containing different cell barcodes can be trivially said to be derived from different cells.

 ![Cell Barcodes]({% link topics/single-cell/images/scrna-intro/scrna_pbb_barcodes_add.svg %} "Two separate cell barcodes GGG and TCT, added to all read transcripts of two different cells")

Transcript barcodes, meanwhile, are random sets of nucleotides added to each transcript.

 ![Transcript Barcodes]({% link topics/single-cell/images/scrna-umis/scrna_umi_add.svg %} "Random oligonucleotides barcodes added to (cell barcoded) reads")

There are two things to take note of:

 1. The number of duplicates in the transcript barcodes (left)
 2. The number of duplicate read transcripts (right)

Transcript barcodes are often *not* unique. This becomes evident when you consider that there are approximately 200,000 mRNA's in a given mammalian cell ({% cite Shapiro2013 %}) which would require barcode lengths of greater than 9 nucleotides to capture, assuming no sequencing errors.

> <question-title></question-title>
>
> 1. Why is it important to know which cell a read came from?
> 2. Why do we need to barcode a read transcript too? Isn't mapping it against the reference genome enough?
>
> > <solution-title></solution-title>
> >
> > 1. If all our reads encode for a *Red Gene* (as above), we may want to know which cells express *Red Gene* more than others.
> > * e.g. If our *Grey* cell has 10 times more *Red Gene* reads than our *Green* cell, then we know that the *Grey* cell and *Green* cell differ in their expression of *Red Gene* - which might be biologically significant.
> > 2. *Yes* and *no*!
> > * **Yes**: We can indeed align our sequence against a reference genome and obtain the name of the gene it aligns against. This sequence will then contribute to the 'count' of sequences that gene has, and increase the expression of that gene.
> > * **No**: We do not know whether these 'counts' are *unique*. Many of these counts could be duplicates as a result of the amplification process. To explain further, we must look at UMIs and their role in the analysis.
> {: .solution}
>
{: .question}

The purpose of transcript barcodes is to reduce the impact of duplicated reads that occur non-linearly during the amplification process.

For this reason, transcript barcodes do not need to be unique. As long as we know that a given read maps to a specific transcript (i.e. after mapping it to a transcriptome), then we can assess how unique that read is based on:

  1. Cell barcode
  2. Transcript barcode
  3. Mapped location

To fully explore the uniqueness of counts, we must discuss the inclusion of *UMIs* in a single-cell analysis.

# Mitigating duplicate transcript counts with UMIs

One of the major issues with sequencing is that the read fragments require amplification before they can be sequenced. A gene with a single mRNA transcript will not be detected by most sequencers, so it needs to be duplicated 100-1000x times for the sequencer to 'see' it.

Amplification is an imprecise process however, since some reads are amplified more than others, and subsequent amplification can lead to these over-amplified reads being over-amplified even more, leading to an exponential bias of some reads over others.

<small>***Note:*** Cell barcodes are not shown in any of the below examples, we assume they were added to our transcripts previously. </small><br /><br />

## Naive Amplification

 ![Amplification Bias]({% link topics/single-cell/images/scrna-intro/scrna_amplif_errors.svg %} "A cell with two reads from different transcripts being amplified unevenly")



Consider the above example where two reads from different transcripts are amplified unevenly. The resulting frequency table would yield:

> |  | Reads in Cell 1 |
> |--|------------------|
> | Gene Red | 4 |
> | Gene Blue | 0 |
{: .matrix}

But the truth is entirely different (i.e. Gene Red should have 1 count, and Gene Blue should also have 1 count).
How do we correct for this bias?

## Amplification with UMIs

**Unique Molecular Identifiers** (or *UMIs*) constitute the second portion of a barcode, where their role is to *uniquely* count reads such that amplicons of the same read are only counted once, e.g:

 ![Amplification Bias with UMIs]({% link topics/single-cell/images/scrna-intro/scrna_amplif_errors_umis.svg %} "A cell with four unique transcripts, two from Gene Red and two from Gene Blue")

 Here, we see two unique transcripts from Gene Red and two unique transcripts from Gene Blue, each given a (coloured) UMI. After amplification, Gene Red has more reads than Gene Blue. If we were to construct a frequency table as before to count the reads, we would have:

> |  | Reads in Cell 1 |
> |--|-----------------|
> | Gene Red | 6 |
> | Gene Blue | 3 |
{: .matrix}

This information is false, because it shows that Red has twice the expression that Blue does. However, we can reconstitute the true count by considering the UMI information:

> |  | UMI colour  | Reads in Cell 1 |
> |--|-------------|-----------------|
> | Gene Red | Pink | 2 |
> |          | Blue | 4 |
> | Gene Blue | Pink | 1 |
> |           | Green | 2 |
{: .matrix}

From this we can then make the decision to ignore the frequencies of these UMIs, and simply count how many *unique* UMIs we see in each gene:

> |  | Set of UMIs in Gene | UMIs in Cell 1 |
> |--|---------------------|----------------|
> | Gene Red | {Pink, Blue} | 2 |
> | Gene Blue | {Pink, Green} | 2 |
{: .matrix}

This then provides us with the true count of the number of true transcripts for each gene as given by our original figure.

> |  | UMIs in Cell 1 |
> |--|----------------|
> | Gene Red | 2 |
> | Gene Blue | 2 |
{: .matrix}


> <question-title>about UMIs</question-title>
>
> 1. Are UMIs not specific to certain genes? Can the same UMI map to different genes?
> 2. Can the same UMI map to different mRNA molecules of the same gene?
>
> > <solution-title></solution-title>
> >
> > 1. Yes, UMIs are not specific to genes and the same UMI barcode can tag the transcripts of different genes. UMIs are not universal tags, they are just 'added randomness' that help reduce amplification bias.
> > 2. Yes, UMIs are not precise but operate probabilistically. In most cases, two transcripts of the same gene will be tagged by different UMIs. In rarer (but still prevalent) cases, the same UMI will capture different transcripts of the same gene.
> >  * One helpful way to think about how quantification is performed is to observe the following hierarchy of data `Cell Barcode → Gene → UMI`
> >
> >   e.g.
> >
> >  | BC:Cell | Maps to Gene | BC:UMI |
> >  |------|-----|------|
> >  | AAAT | Slx1 | TCA |
> >  | AAAT | Slx2 | GTG |
> >  | AAAT | Gh13 | TCA |
> >  | TTAA | Slx1 | TCA |
> >  | TTAA | Atp3 | CCC |
> >
> > If UMIs were unique to a gene, then the `TCA` UMI barcode would not have reads that map to both *Slx1* and *Gh13* in the same cell (`AAAT`).
> >
> {: .solution}
>
{: .question}

# Barcoding Format

We now know the role of UMIs and cell barcodes, but how do we handle them in the analysis? Let us look at 4 example sequences in our paired-end FASTQ data.

> <hands-on-title>Preparing the Data</hands-on-title>
>
>    1. Create a new history and rename it (*e.g.* 'Inspecting FastQ Files in scRNA batch data')
>    1. Import the following files from [`Zenodo`](https://zenodo.org/record/2573177) or from the data library (ask your instructor)
>    ```
> https://zenodo.org/record/2573177/files/test_barcodes_celseq2_R1.fastq.gz
> https://zenodo.org/record/2573177/files/test_barcodes_celseq2_R2.fastq.gz
>    ```
>
>       {% snippet faqs/galaxy/datasets_import_via_link.md %} <br/>
>
>    1. Build a *Dataset pair* for the two FASTQ files <!-- cannot force this to count from 3 -->
>       - Click on {% icon galaxy-selector %} "Select Items" at the top of the history panel
>       - Check the two boxes next to the R1 and R2 scRNA FASTQ samples
>       - Click *2 of N selected* and choose *Build dataset pair*
>       - Ensure that the forward read is the `R1` sample, and the reverse read is the `R2` sample.
>          - Click 'Swap' otherwise.
>       - Set the name of the pair
>
>    1. Generate a list of reads to filter by creating a plain tabular file containing the following read names:
>        ```
>       J00182:75:HTKJNBBXX:2:1114:12469:11073
>       J00182:75:HTKJNBBXX:2:2222:13301:35690
>       J00182:75:HTKJNBBXX:2:1203:25022:13763
>       J00182:75:HTKJNBBXX:2:1115:8501:46961
>        ```
>    1. Set the datatype of the file as **tabular**
>
{: .hands_on}

At this point we now have a history with two items: our paired FASTQ test data, and a tabular file of read names. We will now apply the tabular file to the FASTQ file and extract only those reads.

> <hands-on-title>Extracting the Reads</hands-on-title>
>
> 1. Extracting our 4 reads
>    * {% tool [Filter sequences by ID](toolshed.g2.bx.psu.edu/repos/peterjc/seq_filter_by_id/seq_filter_by_id/0.2.9) %} with the following parameters:
>      - **Sequence file to be filtered**
>        - Click the *Dataset Collection* icon
>        - Select the FastQ collection if not already selected.
>      - **Filter using the ID list from**:`tabular file`
>        - *Tabular file containing sequence identifiers*:`Pasted Entry`
>      - **Column(s) containing sequence identifiers**
>        - **Select/Unselect all**:(tick the box)
>      - **Output positive matches, negative matches, or both?**:`Just positive matches (ID on list), as a single file`
>
> 1. Change the datatypes of the output pair to `fastqsanger` if not already set.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
> 1. Viewing our 4 reads side-by-side
>    - Activate the **Window Manager** by clicking on the **Enable/Disable Window Manager** icon on the main top toolbar
>    - Click on the newly generated FastQ pair ending in *"with matched ID"* to expand the individual reads
>      - Click on the {% icon galaxy-eye %} symbol of the forward read
>      - Click somewhere outside the white box to close the **Window Manager**
>      - Click on the {% icon galaxy-eye %} symbol of the reverse read
>    - Position/Resize the boxes as desired
>
>    {% snippet faqs/galaxy/features_scratchbook.md %}
{: .hands_on}

<!--
These are reads that all map to ENSDARG00000019692. In [Cell, UMI] format:

(J00182:75:HTKJNBBXX:2:1114:12469:11073|J00182:75:HTKJNBBXX:2:2222:13301:35690|J00182:75:HTKJNBBXX:2:1203:25022:13763|J00182:75:HTKJNBBXX:2:1115:8501:46961)

            Cell  , UMI
1: 46961 -- ACCAGA, GGAAGA
2: 13763 -- GGTAAC, GTCCCA -> same UMI, same cell
3: 35690 -- GGTAAC, GTCCCA -> same UMI, same cell
4: 11073 -- GGTAAC, CGGCGT -> diff UMI, same cell
-->

## Our Four Reads of Interest

Let us examine these four reads of interest which we have just sub-selected using their headers:

> <details-title>Forward Reads:</details-title>
>
>     @J00182:75:HTKJNBBXX:2:1115:8501:46961 1:N:0:ATCACG
>     GGAAGAACCAGATTTTTTTTTTTTTTTTTT
>     +
>     AAFFFJJJJJJJFFFJJJJJJJJJJJJJJJ
>
>     @J00182:75:HTKJNBBXX:2:1203:25022:13763 1:N:0:ATCACG
>     GTCCCAGGTAACTTTTTTTTTTTTTTTTTT
>     +
>     AAFFFJJJJJJJJFFJJJJJJJJJFJ<FF-
>
>     @J00182:75:HTKJNBBXX:2:2222:13301:35690 1:N:0:ATCACG
>     GTCCCAGGTAACTTTTTTTTTTTTTTTTTT
>     +
>     AAFFFJJJJJJJ<AFJJJJJFFJJFJJJFF
>
>     @J00182:75:HTKJNBBXX:2:1114:12469:11073 1:N:0:ATCACG
>     CGGCGTGGTAACTTTTTTTTTTTTTTTTCC
>     +
>     AAFFFJJJJJJJFAFFJJJJJJJJF---<F
>
>
{: .details}

> <details-title>Reverse Reads:</details-title>
>
>     @J00182:75:HTKJNBBXX:2:1115:8501:46961 2:N:0:ATCACG
>     GACCTCTGATCTTTACGAAAGGCCAACGCGTTTTCAGTCTGGACACGGTTCAGCTCCTGTTCATTATTCA
>     +
>     A<<A-777F<AA<AJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
>
>     @J00182:75:HTKJNBBXX:2:1203:25022:13763 2:N:0:ATCACG
>     GCCACCTAATTTCCGTCATCACACTCCTCTCCGTTTTCAACTTGCACAATGCTGTCTCCGCAGAATCCCT
>     +
>     ---<----<A---77-7A-FJ<JJFFJJ<JJAJ7<-FAFFJJFF<FFJJFFAJFA-AFFFJFFFFFJAJJ
>
>     @J00182:75:HTKJNBBXX:2:2222:13301:35690 2:N:0:ATCACG
>     CAATCCTCTCCGTTATCAACTTGCACAATGCTGTCTCCGCAGAATCCCTCCGGATCAGGATCGCTCTCCA
>     +
>     <<A-77--77F<----7AFJ-A--FJJJFAJF-AFAJAJ<JFJ<JJJFFJJJFJJJJJAAFJJJFJJJF-
>
>     @J00182:75:HTKJNBBXX:2:1114:12469:11073 2:N:0:ATCACG
>     ATCCACTTATTGCAAAGCAGAGGACATTGAGTCTCACCTTTTGTCCAGGTCTTCCAATTTCACCCTGCAA
>     +
>     A-77AA-7FF<7FFJFFFJJJJJJJJJJJJJ-AFJJJJJJJFJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
>
{: .details}

What we observe are the standard four lines of any FASTQ file:
  1. Read name starting with `@`
  2. Sequence of nucleotide bases
  3. Separator `+`
  4. Quality string of the nucleotide bases in ASCII

The main source of interest for us is in the (2) sequences of these reads, which somewhere within encode for three crucial pieces of information that we will need to perform quantification:

  1. Cell Barcode
  2. UMI Barcode
  3. Reverse-transcribed mRNA sequence

These can be encoded into the sequences of our paired-end data by any means. In order to know where our barcodes are, we must be familiar with the sequencing primers used in the analysis:


<!-- TODO: Expand the schema to talk about the actual read primers and not just the sequencing primers -->

## Verifying the Barcode Format

As shown in [CEL-Seq2 protocol](#the-cel-seq2-protocol), we have the following encoding:
 * Forward Read:
    * 01-06bp: UMI Barcode
    * 07-12bp: Cell Barcode
    * 13-30bp: Poly-T tail
 * Reverse Read:
    * 01-70bp: mRNA sequence

The encoding of the barcodes on the first read can actually be seen by examining the distribution of bases in a FastQC plot.

> <hands-on-title>Confirming the Barcoding</hands-on-title>
>
> 1. {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} with the following parameters:
>    - {% icon param-collection %} *"Short read data from your current history"*: `Paired FastQ` (the original paired set) You will need to choose 'Dataset collection' to allow this as an input.
>
>    > <comment-title></comment-title>
>    >
>    > We are only interested in the distribution of bases on the Forward read, but it is more convenient to process the data as a pair instead of un-hiding the original dataset
>    {: .comment}
>
> 1. Click on *FastQC on collection :Webpage*
> 1. Click on the {% icon galaxy-eye %} of the Forward read
> 1. Click on the *Per base sequence content* header on the side-bar
{: .hands_on}


 ![FastQC Plot](../../images/scrna-umis/scrna_barcodes_fastq.png "FastQC plot of Forward read")


Here we can see the three distinct regions along the x-axis that correspond to our expected [CEL-Seq2 Schema](#the-cel-seq2-protocol):

 * 01-06bp: smooth, relatively constant bases.
 * 07-12bp: noisy, highly varied distribution of bases.
 * 13-30bp: T-dominated region

We can see that the distribution of nucleotides in the 01-06bp range is relatively more stable than the distribution of nucleotides in the 07-12bp range, which seems to exhibit more extreme variation.

> <question-title></question-title>
>
> Why is this the case? Why is the UMI barcode base distribution smoother than the Cell barcode base distribution?
> >
> > <solution-title></solution-title>
> >
> > There are far more UMIs than cells. Cell barcodes are designed and selected with a specified edit distance, greatly limiting their availability in the data. UMIs are not so well-curated -- i.e it is possible to encounter the same UMI in the same cell multiple times. The more extreme variation in the 7-12bp region is simply caused by a fewer number of samples.
> >
> {: .solution}
{: .question}


# Uniting Barcodes with Sequence

In a sense, we have a disparity in our data: the reverse reads contain the sequences we wish to map, but not the barcodes; the forward reads contain the barcode, but not the sequence. For the forward and reverse reads given above, the information that we really want from both can be summarized in this table:

 | Read | Cell | UMI | Sequence |
 |------|------|-----|----------|
 | <small>@J00182:75:HTKJNBBXX:2:1115:8501:46961</small>  | ACCAGA | GGAAGA | <small>GACCTCTGATCTTTACGAAAGGCCAACGCGTTTTCAGTCTGGACACGGTTCAGCTCCTGTTCATTATTCA</small> |
 | <small>@J00182:75:HTKJNBBXX:2:1203:25022:13763</small> | GGTAAC | GTCCCA | <small>GCCACCTAATTTCCGTCATCACACTCCTCTCCGTTTTCAACTTGCACAATGCTGTCTCCGCAGAATCCCT</small> |
 | <small>@J00182:75:HTKJNBBXX:2:2222:13301:35690</small> | GGTAAC | GTCCCA | <small>CAATCCTCTCCGTTATCAACTTGCACAATGCTGTCTCCGCAGAATCCCTCCGGATCAGGATCGCTCTCCA</small> |
 | <small>@J00182:75:HTKJNBBXX:2:1114:12469:11073</small> | GGTAAC | CGGCGT | <small>ATCCACTTATTGCAAAGCAGAGGACATTGAGTCTCACCTTTTGTCCAGGTCTTCCAATTTCACCCTGCAA</small> |

> <question-title></question-title>
>
> Provided that these reads all map to the same gene:
>
> 1. Which of these reads come from the same cell?
> 2. Which of these reads are PCR duplicates?
>
> > <solution-title></solution-title>
> >
> > 1. Reads:
> >  * `@J00182:75:HTKJNBBXX:2:1203:25022:13763`
> >  * `@J00182:75:HTKJNBBXX:2:2222:13301:35690`
> >  * `@J00182:75:HTKJNBBXX:2:1114:12469:11073`
> >  all have the cell barcode `GGTAAC`.
> >
> > 2. Reads:
> >  * `@J00182:75:HTKJNBBXX:2:1203:25022:13763`
> >  * `@J00182:75:HTKJNBBXX:2:2222:13301:35690`
> >  appear to be PCR duplicates, since they both have the same cell barcode and same UMI.
> >
> > However if we consider their sequences, we can see that they contain different (but overlapping) sequences.
> >
> >     13763:   GCCACCTAATTTCCGTCATCACACTCCTCTCCGTTTTCAACTTGCACAATGCTGTCTCCGCAGAATCCCT
> >     35690:                        CAATCCTCTCCGTTATCAACTTGCACAATGCTGTCTCCGCAGAATCCCTCCGGATCAGGATCGCTCTCCA
> >
> > They describe the same transcript but have sequences from different reads, and therefore both reads *should* be counted as separate reads. Whether or not both these reads are counted as a single read due to their identical barcodes, or counted separately due to their differing sequences depends entirely on the deduplication utility they are fed into it.
> >
> {: .solution}
{: .question}

## Coupling our Data Sources

How should we unite these two source of information into a single location without impacting the data content?

For this we need to take the barcode information from the Forward reads, and stick it into the *header* of the Reverse reads. That way we can align our sequence to the reference and still keep the barcode information associated with the reads.

> <hands-on-title>Barcode Extraction and Annotation of our 4 reads</hands-on-title>
>
> 1. {% tool [UMI-tools extract](toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_extract/umi_tools_extract/1.1.6+galaxy0) %} with the following parameters:
>    - *"Library type"*: `Paired-end Dataset Collection`
>        - {% icon param-collection %} *"Reads in FASTQ format"*: `output` (Our paired set of 4 sequences)
>        - *"Barcode on both reads?"*: `Barcode on first read only`
>    - *"Barcode pattern for first read"*: `NNNNNNCCCCCC`
>    - *"Enable quality filter?"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > - Here we specify the format of our barcodes as `NNNNNNCCCCCC` where the *N*s represent UMI bases and the *C*s represent the cell barcodes.
>    > - In some protocols, actual sequence data can be found in between the cell and UMI barcodes, wherein it is neccesary to represent sequence bases using *X*.
>    > - e.g. A protocol that starts with a 3bp sequence, followed by a 4bp Cell barcode, followed once again by a 10bp sequence, and then finally a 5bp UMI barcode, would require the following barcode format:
>    >   `XXXCCCCXXXXXXXXXXNNNNN`
>    >
>    {: .comment}
> 1. Click the {% icon galaxy-eye %} symbol on the *Reads1: UMI-tools extract* file
> 1. Click somewhere outside the white box to close the **Window Manager**
> 1. Click the {% icon galaxy-eye %} symbol on the *Reads2: UMI-tools extract* file
>
{: .hands_on}

We should now be able to see the following reads:

> <details-title>Forward Reads:</details-title>
>
>     @J00182:75:HTKJNBBXX:2:1115:8501:46961_ACCAGA_GGAAGA 1:N:0:ATCACG
>     TTTTTTTTTTTTTTTTTT
>     +
>     FFFJJJJJJJJJJJJJJJ
>
>     @J00182:75:HTKJNBBXX:2:1203:25022:13763_GGTAAC_GTCCCA 1:N:0:ATCACG
>     TTTTTTTTTTTTTTTTTT
>     +
>     JFFJJJJJJJJJFJ<FF-
>
>     @J00182:75:HTKJNBBXX:2:2222:13301:35690_GGTAAC_GTCCCA 1:N:0:ATCACG
>     TTTTTTTTTTTTTTTTTT
>     +
>     <AFJJJJJFFJJFJJJFF
>
>     @J00182:75:HTKJNBBXX:2:1114:12469:11073_GGTAAC_CGGCGT 1:N:0:ATCACG
>     TTTTTTTTTTTTTTTTCC
>     +
>     FAFFJJJJJJJJF---<F>
>
{: .details}

> <details-title>Reverse Reads:</details-title>
>
>     @J00182:75:HTKJNBBXX:2:1115:8501:46961_ACCAGA_GGAAGA 2:N:0:ATCACG
>     GACCTCTGATCTTTACGAAAGGCCAACGCGTTTTCAGTCTGGACACGGTTCAGCTCCTGTTCATTATTCA
>     +
>     A<<A-777F<AA<AJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
>
>     @J00182:75:HTKJNBBXX:2:1203:25022:13763_GGTAAC_GTCCCA 2:N:0:ATCACG
>     GCCACCTAATTTCCGTCATCACACTCCTCTCCGTTTTCAACTTGCACAATGCTGTCTCCGCAGAATCCCT
>     +
>     ---<----<A---77-7A-FJ<JJFFJJ<JJAJ7<-FAFFJJFF<FFJJFFAJFA-AFFFJFFFFFJAJJ
>
>     @J00182:75:HTKJNBBXX:2:2222:13301:35690_GGTAAC_GTCCCA 2:N:0:ATCACG
>     CAATCCTCTCCGTTATCAACTTGCACAATGCTGTCTCCGCAGAATCCCTCCGGATCAGGATCGCTCTCCA
>     +
>     <<A-77--77F<----7AFJ-A--FJJJFAJF-AFAJAJ<JFJ<JJJFFJJJFJJJJJAAFJJJFJJJF-
>
>     @J00182:75:HTKJNBBXX:2:1114:12469:11073_GGTAAC_CGGCGT 2:N:0:ATCACG
>     ATCCACTTATTGCAAAGCAGAGGACATTGAGTCTCACCTTTTGTCCAGGTCTTCCAATTTCACCCTGCAA
>     +
>     A-77AA-7FF<7FFJFFFJJJJJJJJJJJJJ-AFJJJJJJJFJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
>
{: .details}

Notice the remaining sequence in each of the reads, and that the reverse reads appear to fully encapsulate all the information that we wanted to capture in our table at the beginning of this section.

> <question-title></question-title>
>
> 1. Compare the Forward/Read1 and Reverse/Read2 reads to those prior the extraction. What has happened to the header and sequence of each read?
> 2. Are the Forward reads useful at all?
>
> > <solution-title></solution-title>
> >
> > 1. Comparison:
> >    * Forward:
> >       * Sequence: The `cell` and `umi` sections of the sequence have been removed, leaving behind only the PolyT tail.
> >       * Header: The `cell` and `umi` sections of the sequence have been added as `cell_umi` barcode in the header
> >    * Reverse:
> >       * Sequence: Has not changed.
> >       * Header: The `cell` and `umi` sections of the sequence from the Forward (note: **NOT** Reverse) reads have been added to the header.
> > 2. With the inclusion of the cell and UMI barcodes into the header of our sequence data, we now have all our useful data in the Reverse reads. We can now effectively throw away our Forward reads, as they have no more useful information within them.
> {: .solution}
{: .question}

We have now successfully multiplexed data from several different (cell) sources into a single file that will greatly simplify the mapping/alignment process downstream.

We have also now successfully *de*-multiplexed our data, by decoding each pair of reads into barcoding and sequence parts and making use of the FASTQ format by storing this information within the FASTQ headers and data, respectively.


# Conclusion


With this tutorial we have understood the importance of handling FASTQ data from different sources, and extracting the information we need (barcodes (cell and UMI) and sequence) using **UMI-tools** so that we can perform mapping without losing any context of where the reads are derived from.

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %}).

tools_json:
["Filter sequences by ID", "toolshed.g2.bx.psu.edu/repos/peterjc/seq_filter_by_id/seq_filter_by_id/0.2.9", "FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1", "UMI-tools extract", "toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_extract/umi_tools_extract/1.1.6+galaxy0"]


