# tutorials

topic: single-cell
tutorial: scrna-case_alevin
title: Generating a single cell matrix using Alevin
difficulty: intermediate
requirements_json: []
tags_json: []
content_hash: 64af8ffe7ca1895f245b2a38214af5e4
last_modified: 2026-05-27T09:01:49.745391
zenodo_link: https://zenodo.org/record/4574153
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_alevin/tutorial.html

content:
answer-key
  date: 2024-01-01

input_histories:
- label: UseGalaxy.eu
  history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/cs1pre-processing-with-alevin---input-1

extra:
  example_histories:
  - label: UseGalaxy.eu_Whole_All-7xAnnData
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/locked-whole-samples-generating-a-single-cell-matrix-using-alevin-answer-key
    date: 2025-03-20
  - label: UseGalaxy.eu_Whole_All-7-FASTQ
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/locked-whole-samples-generating-a-single-cell-matrix-using-alevin-input
  - label: UseGalaxy.eu_Downsampled_All-7-FASTQ
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/alevin-tutorial---all-samples---400k
    date: 2025-03-20
  - label: UseGalaxy.eu_Downsampled_Allx7-AnnData
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/generating-a-single-cell-matrix-using-alevin-all-downsample-fastq-processed-to-anndata
    date: 2025-03-20
  downsample_histories:
  - label: UseGalaxy.eu_How_to_Downsample_History
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/pre-processing-with-alevin---part-1---how-to-downsample

objectives:
- Generate a cell by gene matrix for droplet-based single-cell RNA sequencing data
- Interpret quality control (QC) plots to make informed decisions on cell thresholds
- Find relevant information in GTF files for the experiment, and add
  this information into the metadata of the data matrix
time_estimation: 2H
key_points:
- You can create a ready-to-analyse object from FASTQ files, including relevant gene metadata
tags:
- paper-replication
- MIGHTS
contributions:
  authorship:
  - nomadscientist
  - pinin4fjords
  editing:
  - hexylena
  testing:
  - wee-snufkin
requirements:
- type: internal
  topic_name: single-cell
  tutorials:
  - scrna-intro
  - scrna-umis
gitter: Galaxy-Training-Network/galaxy-single-cell
follow_up_training:
- type: internal
  topic_name: single-cell
  tutorials:
  - scrna-case_alevin-combine-datasets
recordings:
- captioners:
  - nomadscientist
  date: '2021-02-15'
  galaxy_version: '21.01'
  length: 30M
  youtube_id: 3ytm2AU6QUc
  speakers:
  - nomadscientist
- youtube_id: BQriffof8yM
  length: 49M
  galaxy_version: 24.1.2.dev0
  date: '2024-09-12'
  speakers:
  - MarisaJL
  captioners:
  - MarisaJL
  bot-timestamp: 1726104172


---



This tutorial will take you from raw FASTQ files to a cell x gene data matrix in AnnData format. What's a data matrix, and what's AnnData format? Well you'll find out! Importantly, this is the first step in processing single cell data in order to start analysing it.

> <details-title>Where am I?</details-title>
>
> You are in one of the four tutorials associated with a Case Study, which replicates and expands on the analysis performed in a manuscript {% cite Bacon2018 %}.
>
> ![case study overview](../../images/scrna-casestudy/CS1-case_study_overview_1.png "Overview of the four parts of the case study, represented by boxes connected with noodles. There is a signpost specifying that you are currently in the first part.")
>
{: .details}

Currently you have a bunch of strings of `ATGGGCTT` etc. in your sequencing files, and what you need to know is how many cells you have and what genes appear in those cells. These steps are the most computationally heavy in the single cell world, as you're starting with 100s of millions of reads, each with 4 lines of text. Later on in analysis, this data becomes simple gene counts such as 'Cell A has 4 GAPDHs', which is a lot easier to store! Because of this data overload, we have downsampled the FASTQ files to speed up the analysis a bit. Saying that, you're still having to map loads of reads to the massive murine genome, so get yourself a cup of coffee and prepare to analyse!

> <warning-title>For the bench scientists and biologists!</warning-title>
> If you're not used to computing, this tutorial will *not* feel intuitive. It's lots of heavy (and necessary) computational steps with little visible reward. You will still absolutely be able to complete it, but it won't make that much sense.
>
> **That is ok!**
>
> Conceptually, the {% icon level %} [Filter, plot & explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial (which comes later) is when you really get to generate fun plots and interpret them scientifically. However, you can't do that until you have pre-processed your data. Some learners like doing that tutorial first, them coming back to learn how to build their input dataset here. So:
>
> - If you're in a *Live course*, follow the path of training materials
> - If you're learning on your own, either get through these pre-processing steps with the belief that plots will get more fun later, or:
> - Try out the {% icon level %} [Filter, plot & explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial *first*, then swing back and do this one.
> It's up to you!
>
{: .warning}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

{% snippet faqs/gtn/gtn_example_histories.md %}

# Generating a matrix

In this section, we will show you the principles of the initial phase of single-cell RNA-seq analysis: generating expression measures in a matrix. We'll concentrate on droplet-based (rather than plate-based) methodology, since this is the process with most differences with respect to conventional approaches developed for bulk RNA-seq.

Droplet-based data consists of three components: cell barcodes, unique molecular identifiers (UMIs) and cDNA reads. To generate cell-wise quantifications we need to:

 * Process cell barcodes, working out which ones correspond to 'real' cells, which to sequencing artefacts, and possibly correct any barcodes likely to be the product of sequencing errors by comparison to more frequent sequences.
 * Map biological sequences to the reference genome or transcriptome.
 * 'De-duplicate' using the UMIs.

This used to be a complex process involving multiple algorithms, or was performed with technology-specific methods (such as 10X's 'Cellranger' tool)  but is now much simpler thanks to the advent of a few new methods. When selecting methodology for your own work you should consider:

 * [STARsolo](https://github.com/alexdobin/STAR) - a droplet-based scRNA-seq-specific variant of the popular genome alignment method STAR. Produces results very close to those of Cellranger (which itself uses STAR under the hood).
 * [Kallisto/ bustools](https://www.kallistobus.tools/) - developed by the originators of the transcriptome quantification method, Kallisto.
 * [Alevin](https://salmon.readthedocs.io/en/latest/alevin.html) - another transcriptome analysis method developed by the authors of the Salmon tool.

We're going to use Alevin {% cite article-Alevin %} for demonstration purposes, but we do not endorse one method over another.

## Get Data

We've provided you with some example data to play with, a small subset of the reads in a mouse dataset of fetal growth restriction (){% cite Bacon2018 %}). You can also find it in the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and in [Array Express](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)), where we downloaded the original FASTQ files.zs This is a study using the Drop-seq chemistry, however this tutorial is almost identical to a 10x chemistry. We will point out the one tool parameter change you will need to run 10x samples. This data is not carefully curated, standard tutorial data - it's real, it's messy, it desperately needs filtering, it has background RNA running around, and most of all it will give you a chance to practice your analysis as if this data were yours.

Down-sampled reads and some associated annotation will be imported in your first step.

> <details-title>Downsampling?</details-title>
>
> The datasets take a while to run in their original size, so we've pre-selected 400,000 lines from the original file to make it run faster. How did we do this?
> - How did I downsample these FASTQ files? You can check out the {% icon galaxy-history %} [ {{ page.downsample_histories[0].label }} ] ( {{ page.input_histories[0].history }}  ) to find out!
>
> - {% icon warning %} If you are in a *live course*, the time to explore this **downsampling history** is not be factored into the schedule. Please instead check it out *after* your course is finished, or if you finish early!
>
{: .details}

Additionally, to map your reads, we have given you a transcriptome to align against (a FASTA) as well as the gene information for each transcript (a gtf) file.

> <details-title>Where did the FASTA and gtf come from?</details-title>
>
> In practice, you can download the latest, most accurate files for your species of interest [from Ensembl](https://www.ensembl.org/info/data/ftp/index.html).
>
{: .details}

{% include _includes/cyoa-choices.html option1="Import History on EU server" option2="Zenodo" default="Import History on EU server"
       text="If you're on the EU server, (if your usegalaxy has an **EU** anywhere in the URL), then the quickest way to Get the Data for this tutorial is via importing a history. Otherwise, you can import from Zenodo." %}

<div class="Import-History-on-EU-server" markdown="1">

> <hands-on-title>Import History from EU server</hands-on-title>
>
> 1. Import the {% icon galaxy-history-input %} *Input history* by following the link below
>
>     {% for h in page.input_histories %}
>       [ {{h.label}} Input History]( {{h.history}} )
>     {% endfor %}
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
{: .hands_on}

</div>

<div class="Zenodo" markdown="1">

> <hands-on-title>Option 2: Import data from Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the Experimental Design table, sequencing reads 1 & 2, the GTF and fasta files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Experimental_Design.tabular
>    {{ page.zenodo_link }}/files/Mus_musculus.GRCm38.100.gtf.gff
>    {{ page.zenodo_link }}/files/Mus_musculus.GRCm38.cdna.all.fa.fasta
>    {{ page.zenodo_link }}/files/SLX-7632.TAAGGCGA.N701.s_1.r_1.fq-400k.fastq
>    {{ page.zenodo_link }}/files/SLX-7632.TAAGGCGA.N701.s_1.r_2.fq-400k.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename {% icon galaxy-pencil %} the datasets: Change `SLX-7632.TAAGGCGA.N701.s_1.r_1.fq-400k` to `N701-Read1` and `SLX-7632.TAAGGCGA.N701.s_1.r_2.fq-400k.fastq` to `N701-Read2`.
>
{: .hands_on}
</div>



> <question-title></question-title>
>
> Have a look at the files you now have in your history.
> 1. Which of the FASTQ files do you think contains the barcode sequences?
> 2. Given the chemistry this study should have, are the barcode/UMI reads the correct length?
> 3. What is the 'N701' referring to?
>
> > <solution-title></solution-title>
> >
> > 1. Read 1 (SLX-7632.TAAGGCGA.N701.s_1.r_1.fq-400k) contains the cell barcode and UMI because it is significantly shorter (indeed, 20 bp!) compared to the longer, r_2 transcript read. For ease, it's better to rename these files N701-Read1 and N701-Read2.
> > 2. You can see Read 1 is only 20 bp long, which for original Drop-Seq is 12 bp for cell barcode and 8 bp for UMI. This is correct! Be warned - 10x Chromium (and many technologies) change their chemistry over time, so particularly when you are accessing public data, you want to check and make sure you have your numbers correct!
> > 3. 'N701' is referring to an index read. This sample was run alongside 6 other samples, each denoted by an Illumina Nextera Index (N70X). Later, this will tell you batch information. If you look at the 'Experimental Design' file, you'll see that the N701 sample was from a male wildtype neonatal thymus.
> {: .solution}
{: .question}

Gene-level, rather than transcript-level, quantification is standard in scRNA-seq, which means that the expression level of alternatively spliced RNA molecules are combined to create gene-level values. Droplet-based scRNA-seq techniques only sample one end each transcript, so lack the full-molecule coverage that would be required to accurately quantify different transcript isoforms.

## Generate a transcript to gene map

To generate gene-level quantifications based on transcriptome quantification, Alevin and similar tools require a conversion between transcript and gene identifiers. We can derive a transcript-gene conversion from the gene annotations available in genome resources such as Ensembl. The transcripts in such a list need to match the ones we will use later to build a binary transcriptome index. If you were using spike-ins, you'd need to add these to the transcriptome and the transcript-gene mapping.

In your example data you will see the murine reference annotation as retrieved from Ensembl in GTF format. This annotation contains gene, exon, transcript and all sorts of other information on the sequences. We will use these to generate the transcript/ gene mapping by passing that information to a tool that extracts just the transcript identifiers we need.

> <question-title></question-title>
>
> Which of the 'attributes' in the last column of the GTF files contains the transcript and gene identifiers?
>
>
>   > <tip-title>Hint</tip-title>
>   >
>   > The file is organised such that the last column (headed 'Group') contains a wealth of information in the format: attribute1 "information associated with attribute 1";attribute2 "information associated with attribute 2" etc.
>   {: .tip}
>
> > <solution-title></solution-title>
> > *gene_id* and *transcript_id* are each followed by "ensembl gene_id" and "ensembl transcript_id"
> {: .solution}
{: .question}

It's now time to *parse* (computing term for separating out important information from a larger file) the GTF file using the [rtracklayer](https://bioconductor.org/packages/release/bioc/html/rtracklayer.html) package in R. This parsing will give us a conversion table with a list of transcript identifiers and their corresponding gene identifiers for counting. Additionally, because we will be generating our own binary index (more later!), we also need to input our FASTA so that it can be filtered to only contain transcriptome information found in the GTF.

> <hands-on-title>Generate a filtered FASTA and transcript-gene map</hands-on-title>
>
> 1. {% tool [GTF2GeneList](toolshed.g2.bx.psu.edu/repos/ebi-gxa/gtf2gene_list/_ensembl_gtf2gene_list/1.52.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Ensembl GTF file"*: `GTF file in the history` {% icon galaxy-history %}
>    - *"Feature type for which to derive annotation"*: `transcript` (Your sequences are transcript sequencing, so this is your starting point)
>    - *"Field to place first in output table"*: `transcript_id` (This is accessing the column you identified above!)
>    - *"Suppress header line in output?"*: {% icon galaxy-toggle %} `Yes` (The next tool (Alevin) does not expect a header)
>    - *"Comma-separated list of field names to extract from the GTF (default: use all fields)"*: `transcript_id,gene_id` (This calls the first column to be the transcript_id, and the second the gene_id. Thus, your key can turn transcripts into genes)
>    - *"Append version to transcript identifiers?"*: {% icon galaxy-toggle %} `Yes` (The Ensembl FASTA files usually have these, and since we need the FASTA transcriptome and the GTF gene information to work together, we need to append these!)
>    - *"Flag mitochondrial features?"*: {% icon galaxy-toggle %} `No`
>    - *"Provide a cDNA file for extracting annotations and/ or possible filtering?"*:  {% icon galaxy-toggle %} `Yes`
>    - {% icon param-file %} *"FASTA-format cDNA/transcript file"*: `Mus_musculus.GRCm38cdna.all.fa` (you may have renamed your differently in your {% icon galaxy-history %} history)
>    - *"Annotation field to match with sequences"*: `transcript_id`
>    - *"Filter the cDNA file to match the annotations?"*: {% icon galaxy-toggle %} `Yes`
>
> 2. Rename {% icon galaxy-pencil %} the annotation table to `Map`
>
> 3. Rename {% icon galaxy-pencil %} the filtered FASTA file to `Filtered_FASTA`
{: .hands_on}

## Generate a transcriptome index & quantify!

Alevin collapses the steps involved in dealing with dscRNA-seq into a single process. Such tools need to compare the sequences in your sample to a reference containing all the likely transcript sequences (a 'transcriptome'). This will contain the biological transcript sequences known for a given species, and perhaps also technical sequences such as 'spike ins' if you have those.

> <details-title>How does Alevin work?</details-title>
>
> To be able to search a transcriptome quickly, Alevin needs to convert the text (FASTA) format sequences into something it can search quickly, called an 'index'. The index is in a binary rather than human-readable format, but allows fast lookup by Alevin. Because the types of biological and technical sequences we need to include in the index can vary between experiments, and because we often want to use the most up-to-date reference sequences from Ensembl or NCBI, we can end up re-making the indices quite often. Making these indices is time-consuming! Have a look at the uncompressed FASTA to see what it starts with.
>
{: .details}

We now have:

* Barcode/ UMI reads
* cDNA reads
* transcript/ gene mapping
* filtered FASTA

We can now run Alevin! However, Alevin will inherently do some of its own filtering and thresholding. We will actually use a better tool later (*emptyDrops*) that needs to work on raw outputs. Therefore, we are going to select options to limit Alevin from filtering the outputs.

> <hands-on-title>Running Alevin</hands-on-title>
>
> 1. {% tool [Alevin](toolshed.g2.bx.psu.edu/repos/bgruening/alevin/alevin/1.10.1+galaxy2) %}
>    - *"Select a reference transcriptome from your history or use a built-in index?"*: `Use one from the history`
>       - You are going to generate the binary index using your filtered FASTA!
>        - In *"Salmon index"*:
>            - {% icon param-file %} *"Transcripts FASTA file"*: `Filtered_FASTA`
>    - *"Single or paired-end reads?"*: `Paired-end`
>        - {% icon param-file %} *"Mate pair 1"*: `N701-Read1`
>        - {% icon param-file %} *"Mate pair 2"*: `N701-Read2`
>    - *"Specify the strandedness of the reads"*: `Infer automatically (A)`
>    - *"Type of single-cell protocol"*: `DropSeq Single Cell protocol`
>    - {% icon param-file %} *"Transcript to gene map file"*: `Map`
>    - In *"Extra output files"*:
>        - {% icon param-check %} `Salmon Quant log file`
>        - {% icon param-check %} `Features used by the CB classification and their counts at each cell level (--dumpFeatures)`
>    - In *"Advanced options"*:
>        - *" Dump cell v transcripts count matrix in MTX format"*: {% icon galaxy-toggle %} `Yes`
>        - *"Fraction of cellular barcodes to keep"*: `1.0` (this prevents Alevin from adding extra thresholds!)
>        - *"Minimum frequency for a barcode to be considered"*: `3` (This is normally set at about 10, but as we've downsampled these datasets, setting it to 10 would pretty much delete them!)
{: .hands_on}

> <tip-title>What if I'm running a 10x sample?</tip-title>
>
> The main parameter that needs changing for a 10X Chromium sample is the 'Protocol' parameter of Alevin. Just select the correct 10x Chemistry there instead.
{: .tip}

> <details-title>Strandedness?</details-title>
>
> The Salmon documentation on 'Fragment Library Types' and running the Alevin command is here: [salmon.readthedocs.io/en/latest/library_type.html](https://salmon.readthedocs.io/en/latest/library_type.html) and [salmon.readthedocs.io/en/latest/alevin.html](https://salmon.readthedocs.io/en/latest/alevin.html). These links will help here, although keep in mind the image there is drawn with the RNA 5' on top, whereas in this scRNA-seq protocol, the polyA is captured by its 3' tail and thus effectively the bottom or reverse strand...)
>
{: .details}

> <comment-title>Alevin file names</comment-title>
>
> You will notice that the names of the output files of Alevin are written in a certain convention, mentioning which tool was used and on which files, for example: *"Alevin on data X, data Y, and others: whitelist"*. Remember that you can always rename the files if you wish! For simplicity, when we refer to those files in the tutorial, we skip the information about tool and only use the second part of the name - in this case it would be simply *"whitelist"*.
{: .comment}

This tool will take a while to run. Alevin produces many file outputs, not all of which we'll use. You can refer to the [Alevin documentation](https://salmon.readthedocs.io/en/latest/alevin.html) if you're curious what they all are, but we're most interested in is:

* the matrix itself (*per-cell gene-count matrix (MTX)* - the count by gene and cell)
* the row (cell/ barcode) identifiers (*row index (CB-ids)*) and
* the column (gene) labels (*column headers (gene-ids)*).


> <question-title></question-title>
>
> After you've run Alevin, {% icon galaxy-eye %} look through all the different output files. Can you find:
> 1. The Mapping Rate?
> 2. How many cells are present in the matrix output?
>
> > <solution-title></solution-title>
> >
> > 1. Inspect {% icon galaxy-eye %} the file {% icon param-file %} *Salmon log file*. You can see the mapping rate is a paltry `48.3268%` (This may vary slightly with tool versions). This is not a great mapping rate. Why might this be? For early Drop-Seq samples and single-cell data in general, you might expect a slightly poorer mapping rate. 10x samples are much better these days! This is real data, not test data, after all!
> > 2. *Peek* at the file {% icon param-file %} *row index (CB-ids)* in your {% icon galaxy-history %}, and you will find it has `22,952` lines. The rows refer to the cells in the cell x gene matrix. According to this (rough) estimate, your sample has 22,952 cells in it! This entirely inaccurate, because it's currently counting background noise as cells. So we need to do some **Quality Control** (QC)!
> >
> {: .solution}
>
{: .question}

{% icon congratulations %} Congratulations - you've made an expression matrix! We could almost stop here. But it's sensible to do some basic QC, and one of the things we can do is look at a barcode rank plot.

# Generating QC Plots

The question we're looking to answer here, is: "do we mostly have a single cell per droplet"? That's what experimenters are normally aiming for, but it's not entirely straightforward to get exactly one cell per droplet. Sometimes almost no cells make it into droplets, other times we have too many cells in each droplet. At a minimum, we should easily be able to distinguish droplets with cells from those without.

> <hands-on-title>Generate a raw barcode QC plot</hands-on-title>
>
> 1. {% tool [Droplet barcode rank plot](toolshed.g2.bx.psu.edu/repos/ebi-gxa/droplet_barcode_plot/_dropletBarcodePlot/1.6.1+galaxy2) %} with the following parameters:
>    - *"Input MTX-format matrix?"*: {% icon galaxy-toggle %} `No`
>    - {% icon param-file %} *"A two-column tab-delimited file, with barcodes in the first column and frequencies in the second"*: Output of Alevin `raw CB classification frequencies`
>    - *"Label to place in plot title"*: `Barcode rank plot (raw barcode frequencies)`
>
> 2. Rename {% icon galaxy-pencil %} the image output `Barcode Plot - raw barcode frequencies`
{: .hands_on}

## Interpreting the QC plots

This is our own formulation of the barcode plot based on a [discussion](https://github.com/COMBINE-lab/salmon/issues/362#issuecomment-490160480) we had with community members. The left hand plots with the smooth lines are the main plots, showing the UMI counts for individual cell barcodes ranked from high to low. The right hand plots are density plots from the first one, and the thresholds are generated either using [dropletUtils](https://bioconductor.org/packages/release/bioc/html/DropletUtils.html) or by the method described in the discussion mentioned above.

![raw droplet barcode plots-400k](../../images/scrna-casestudy/CS1-raw_barcodes-400k.png "400k subsample raw")

Now, the image generated here (400k) isn't the most informative - but you are dealing with a fraction of the reads! If you run the total sample (so identical steps above, but with significantly more time!) you'd get the image below.

![raw droplet barcode plots-total](../../images/scrna-casestudy/CS1-raw_barcodes-total.png "Total sample - 32,579,453 reads - raw")

We expect a sharp drop-off between cell-containing droplets and ones that are empty or contain only cell debris. Then, various proposed thresholds (shown in the different coloured horizontal lines) are calculated by Alevin, to give options of where to put a cut-off line that says "Any barcodes with fewer reads per cell than this cut-off line are discarded, because these are likely background and uninformative." Ideally, this cut-off is clearly at the point where the curve bends, known as the *knee* of the curve.

Now, this data is not an ideal dataset, so for perspective, in an ideal world with a very clean 10x run, data will look a bit more like the following taken from the lung atlas (see the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6653/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6653/)).

![raw droplet barcode plots - lung atlas](../../images/scrna-casestudy/CS1-lung-atlas-barcodes-raw.png "Pretty data - raw")

You can see the clearer 'knee' bend in this significantly larger and cleaner sample, showing the cut-off between empty droplets and cell-containing droplets.

While we could use any of these calculated thresholds to select cells, there are some more sophisticated methods also available. We will try one called *"emptyDrops"*.

# emptyDrops

In experiments with relatively simple characteristics, the 'knee detection' method works relatively well. But some populations (such as our sample!) present difficulties. For instance, sub-populations of small cells may not be distinguished from empty droplets based purely on counts by barcode. Some libraries produce multiple 'knees' for multiple sub-populations. The [emptyDrops](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1662-y) method has become a popular way of dealing with this. emptyDrops still retains barcodes with high counts, but also adds in barcodes that can be statistically distinguished from the ambient profiles, even if total counts are similar.

## Reformatting for emptyDrops

Alevin outputs MTX format. However, emptyDrops runs on SingleCellExperiment (SCE) format. To get there, we will perform the following steps:

 1. Swap Cells & Genes orientation
 2. Add in gene metadata (gene symbols, mitochondrial information)
 3. Convert to a SingleCellExperiment object

### Transform matrix

First, we need to 'transform' the matrix such that cells are in columns and genes are in rows.

> <hands-on-title>Transform matrix</hands-on-title>
>
> 1. {% tool [salmonKallistoMtxTo10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/salmon_kallisto_mtx_to_10x/_salmon_kallisto_mtx_to_10x/0.0.1+galaxy6) %} with the following parameters:
>    - {% icon param-file %} *".mtx-format matrix"*: `per-cell gene-count matrix (MTX)` (output of **Alevin** {% icon tool %})
>    - {% icon param-file %} *"Tab-delimited genes file"*: `column headers (gene-ids)` (output of **Alevin** {% icon tool %})
>    - {% icon param-file %} *"Tab-delimited barcodes file"*: `row index (CB-ids)` (output of **Alevin** {% icon tool %})
>
> 2. Rename {% icon galaxy-pencil %} 'salmonKallistoMtxTo10x....:genes' to `Gene_table`
> 3. Rename {% icon galaxy-pencil %} 'salmonKallistoMtxTo10x....:barcodes' to `Barcode_table`
> 4. Rename {% icon galaxy-pencil %} 'salmonKallistoMtxTo10x....:matrix' to `Matrix_table`
{: .hands_on}

The output is a matrix in the correct orientation for the rest of our tools. However, our matrix is looking a bit sparse - for instance, click on `Gene_table`. I don't know about you, but I'd struggle to have a good biological discussion using only Ensembl gene_ids! What I'd really like is the more understandable 'GAPDH' or other gene acronym, as well as information on mitochondrial genes so that I can assess if my cells were stressed out or not. In order to prepare our data for emptyDrops, we're going to combine this information into an object, and it's easiest to add in that information now.

### Add in Gene metadata

> <question-title></question-title>
>
> Where can we find information about genes?
>
> > <solution-title></solution-title>
> >
> > Our old friend the GTF file!
> >
> {: .solution}
>
{: .question}

> <question-title></question-title>
>
> Which of the 'attributes' in the last column of that file contains the gene acronym?
>
> > <solution-title></solution-title>
> >
> > gene_name
> >
> {: .solution}
>
{: .question}

We're now going to re-run {% icon galaxy-refresh %} the tool that extracts information from our GTF file, but extract gene information instead of transcript information.

> <hands-on-title>Generate gene information</hands-on-title>
>
> 1. {% tool [GTF2GeneList](toolshed.g2.bx.psu.edu/repos/ebi-gxa/gtf2gene_list/_ensembl_gtf2gene_list/1.52.0+galaxy0) %} with the following parameters:
>    - *"Feature type for which to derive annotation"*: `gene`
>    - *"Field to place first in output table"*: `gene_id`
>    - *"Suppress header line in output?"*: {% icon galaxy-toggle %} `Yes`
>    - *"Comma-separated list of field names to extract from the GTF (default: use all fields)"*: `gene_id,gene_name,mito`
>    - *"Append version to transcript identifiers?"*: {% icon galaxy-toggle %} `Yes`
>    - *"Flag mitochondrial features?"*: {% icon galaxy-toggle %} `Yes` - note, this will auto-fill a bunch of acronyms for searching in the GTF for mitochondrial associated genes. This is good!
>    - *"Filter the cDNA file to match the annotations?"*: {% icon galaxy-toggle %} `No` - we don't need to, we're done with the FASTA!
>
> 2. Convert the {% icon param-file %} annotation table (TSV) output to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 3. Rename {% icon galaxy-pencil %} the tabular annotation table to `Gene_Information`
>
{: .hands_on}

Inspect {% icon galaxy-eye %} the {% icon param-file %} `Gene_Information` object in the history. You have made a new key for gene_id, with gene name and a column of mitochondrial information (false = not mitochondrial, true = mitochondrial). We need to add this information into the {% icon param-file %} salmonKallistoMtxTo10x output `Gene_table`. But we need to keep `Gene_table` in the same order, since it is referenced in the `Matrix_table` by row.

> <hands-on-title>Combine MTX Gene Table with Gene Information</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - *"Join"*: `Gene_table`
>    - *"Using column"*: `Column: 1`
>    - *"with"*: `Gene_Information`
>    - *"and column"*: `Column: 1`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>    - *"Keep lines of first input that are incomplete"*: `Yes`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `No`
>
>
>    If you inspect {% icon galaxy-eye %} the object, you'll see we have joined these tables and now have quite a few gene_id repeats. Let's take those out, while keeping the order of the original 'Gene Table'.
>
>
> 2.  {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c4,c5`
>    - *"Delimited by"*: `Tab`
>    - *"From"*: output of **Join two Datasets** {% icon tool %}
>
> 3. Rename output `Annotated_Gene_Table`
{: .hands_on}

Inspect {% icon galaxy-eye %} your `Annotated_Gene_Table`. That's more like it! You now have `gene_id`, `gene_name`, and `mito` information (just without a header to label them). Now let's get back to your journey to emptyDrops and sophisticated thresholding of empty droplets!

## Convert to SCE format

The tool emptyDrops {% cite article-emptyDrops %} works with a specific form of R object called a *"SingleCellExperiment"*. We need to convert our transformed MTX files into that form, using the DropletUtils Read10x tool:

> <hands-on-title>Converting to SingleCellExperiment format</hands-on-title>
>
> 1. {% tool [DropletUtils Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_read_10x/dropletutils_read_10x/1.0.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `Matrix_table`
>    - {% icon param-file %} *"Gene Table"*: `Annotated_Gene_Table`
>    - {% icon param-file %} *"Barcode/cell table"*: `Barcode_table`
>    - *"Should metadata file be added?"*: {% icon galaxy-toggle %} `No`
>
> 2. Rename {% icon galaxy-pencil %} output: `SCE_Object`
{: .hands_on}

Fantastic! Now that our matrix is combined into an object, specifically the SingleCellExperiment format, we can now run emptyDrops! Let's get rid of those background droplets containing no cells!

## Run emptyDrops

> <hands-on-title>Emptydrops</hands-on-title>
>
> 1. {% tool [DropletUtils emptyDrops](toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_empty_drops/dropletutils_empty_drops/1.0.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"SingleCellExperiment rdata object"*: `SCE_Object`
>    - *"Should barcodes estimated to have no cells be removed from the output object?"*: {% icon galaxy-toggle %} `Yes`
>
> 2. Rename {% icon galaxy-pencil %} `serialised SingleCellExperiment` output as `Emptied_Object`
>
{: .hands_on}

> <question-title></question-title>
>
> How many cell barcodes remain after the emptyDrops treatment? Why might that be?
>
>   > <tip-title>Hint</tip-title>
>   > If you *peek* at the `Emptied_Object` in your {% icon galaxy-history %} history, the text in that window says `38 barcodes` or something similar to that - there is an element of random in the algorithm, so yours might differ slightly. Why is this so low?? And why might the number be different?
>   > Consider...is this a complete set of data?
>   {: .tip}
>
>
> > <solution-title></solution-title>
> >
> > Remember this is a subsampled dataset. If you look carefully at the parameters of emptyDrops, you'll see it set a minimum threshold at 100 UMI. If you look at the barcode plots above for the 400k read sample, you'll see this is far too stringent for this subsampled data! To satisfy your curiosity, this minimum threshold would yield around `4332` barcodes for the total sample. Again, the number may vary slightly as the output depends on a large number of random iterations.
> >
> {: .solution}
>
{: .question}

We will nevertheless proceed with your majestic annotated expression matrix of ~38 cells, ready to go for further processing and analysis!

{% include _includes/cyoa-choices.html option1="Scanpy" option2="Seurat" default="Scanpy" text="Which toolsuite do you intend to use to analyse your data? Both Scanpy and Seurat are supported in Galaxy. We probably have slightly more tools for Scanpy in Galaxy at the time of writing, so we will default to that, but pick what works for you!" disambiguation="toolsuite" %}

<div class="Scanpy" markdown="1">

The tool suite called Scanpy ({% cite Wolf2018 %}) uses a format called `AnnData`, which is a variant of a file format called `hdf5`. You will now convert your SCE object to an AnnData one!

# Convert to AnnData object

> <hands-on-title>Converting to AnnData format</hands-on-title>
>
> 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `SingleCellExperiment to AnnData`
>    - {% icon param-file %} *"Input object in sce,rds,rdata.sce format"*: `Emptied_Object`
>
> If the dataset does not show up in the corresponding input field or displays as 'unavailable', don't worry - try dragging the dataset from the history panel and dropping it into the input field. If this still doesn't work, then you can change the datatype to `rdata.sce`.
>
> {% snippet faqs/galaxy/datasets_change_datatype.md %}
>
{: .hands_on}

Last but not least, after all these data conversions, your AnnData object is missing some labelling. If you *peek* at our latest output {% icon param-file %} in your {% icon galaxy-history %} history, and check the *vars* categories, you will find one category labelled `NA.` What is this? To find out, you can use a handy tool.

> <hands-on-title>Inspecting AnnData Objects</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `output_anndata` (output of **SCEasy Converter** {% icon tool %})
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
{: .hands_on}

If you {% icon galaxy-eye %} examine the output {% icon param-file %}, you will find that this `NA.` column is a bunch of True/False outputs.

> <question-title></question-title>
>
> What are this "True/False" referring to? (Hint: Consider what gene information you have added!)
>
> > <solution-title></solution-title>
> >
> > The second time you ran the GTF2GeneList, you extracted `gene_id,gene_name,mito` information and then added that information to your datasets. The `mito` flagging outputs a True if the gene has any mitochondrial information in its GTF listing, or False if it does not. Therefore, this `NA.` should be labelled `mito`!
> >
> {: .solution}
>
{: .question}

## Flag mitochondrial genes

> <hands-on-title> Label Mito Metadata</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `output_anndata` (output of **SCEasy Converter** {% icon tool %})
>    - *"Function to manipulate the object"*: `Rename fileds in AnnData variables`
>        - *"Name of the variables field that you want to change"*: `NA.`
>        - *"New name of the filed in the variables"*: `mito`
>
> 2. Rename {% icon galaxy-pencil %} output `N701-400k-AnnData`
>
{: .hands_on}

Your object is ready to for the scanpy pipeline!

</div>

<div class="Seurat" markdown="1">
<!-- intentionally empty. This comment itself isn't necessary, but we recommend adding the div. -->
</div>

# Conclusion

{% icon congratulations %} Congrats! You have finished the tutorial!

We have:

 * Examined raw read data, annotations and necessary input files for quantification.
 * Run Alevin and applied emptyDrops instead of Alevin's inbuilt thresholds.
 * Deployed barcode rank plots as a way of quickly assessing the signal present in droplet datasets.
 * Converted these datasets to enable downstream analysis.

You might find the {% icon galaxy-history-answer %} *Answer Key Histories* helpful to check or compare with:
  - {% for h in page.answer_histories %}
      [ {{h.label}} ]( {{h.history}} )
    {% endfor %}

<!-- Only currently want to iterate through the first history, but might want others in the future (different servers!) -->

You can also run this entire tutorial via a {% icon galaxy-workflows-activity %} *Workflow*, after performing the **Get data** step initially.
 - [Tutorial Workflow]({% link topics/single-cell/tutorials/scrna-case_alevin/workflows/index.md %})

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://singlecell.usegalaxy.eu/published/workflow?id=9a9bbd8046da7b19&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=0&initialY=0&zoom=0.33"></iframe>

It may be that you want to combine this object with others like it, for instance, maybe you ran 5 samples, and you are starting with 10 FASTQ files... To do this, you can run that same Workflow on all your files! In fact, that's what we do to set up the datasets for the next tutorial. You can explore applying this tutorial to all seven samples in this dataset, either in the downsampled or whole state.

- {% icon warning %} Remember that *if* you are in a course, time for exploring these histories will not be factored into the schedule. Explore these outside of course time!

- {% for h in page.example_histories %}
    [ {{h.label}} ]( {{h.history}} )
  {% endfor %}

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["GTF2GeneList", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/gtf2gene_list/_ensembl_gtf2gene_list/1.52.0+galaxy0", "Alevin", "toolshed.g2.bx.psu.edu/repos/bgruening/alevin/alevin/1.10.1+galaxy2", "Droplet barcode rank plot", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/droplet_barcode_plot/_dropletBarcodePlot/1.6.1+galaxy2", "salmonKallistoMtxTo10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/salmon_kallisto_mtx_to_10x/_salmon_kallisto_mtx_to_10x/0.0.1+galaxy6", "Join two Datasets", "join1", "Cut columns from a table", "Cut1", "DropletUtils Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_read_10x/dropletutils_read_10x/1.0.4+galaxy0", "DropletUtils emptyDrops", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_empty_drops/dropletutils_empty_drops/1.0.4+galaxy0", "SCEasy Converter", "toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1"]


topic: single-cell
tutorial: scrna-case-jupyter_basic-pipeline
title: Filter, plot and explore single-cell RNA-seq data with Scanpy (Python)
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: ["type: internal", "scrna-case_alevin", "scrna-case_alevin-combine-datasets"]
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 4c3316a19aec0c3a1429c4ffe648aff4
last_modified: 2026-05-27T09:01:49.743849
zenodo_link: https://zenodo.org/record/7053673
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case-jupyter_basic-pipeline/tutorial.html

content:
{% snippet topics/single-cell/faqs/notebook_warning.md %}

# Install libraries

This tutorial requies some libraries to be installed which is done below (igraph and louvain are not used directly and are just required for plotting). The ```-q``` parameter hides most of the outputs of the installation in order to make the notebook a bit cleaner. If there are any issues with the installation, then removing this parameter may give you more information about the issue.

```python
pip install scanpy -q
```

```python
pip install igraph -q
```

```python
pip install louvain -q
```

```python
pip install pandas -q
```

---

We can now import the two libraries that we will be using, **scanpy** is the primary library that we will use and will handle all the plotting and data processing. Meanwhile, **pandas** is used briefly for some manual data manipulation.

```python
import scanpy as sc
import pandas as pd
```

# Load Data

You can import files from your Galaxy history directly using the following code. This will depend on what number in your history the final annotated object is. If your object is dataset #1 in your history, then you import it with the following:

```python
mito_counted_anndata = get(1)                   # get an object from Galaxy history
adata = sc.read_h5ad(mito_counted_anndata)      # read in the file as h5ad object
```

Alternatively, if you don't want to get the dataset from your Galaxy history, you can also download the input file from Zenodo, running the code below:

```python
%%bash
wget -nv https://zenodo.org/record/7053673/files/Mito-counted_AnnData
```
```python
adata = sc.read_h5ad("Mito-counted_AnnData")
```

# Filtering

You have generated an annotated AnnData object from your raw scRNA-seq fastq files. However, you have only completed a ‘rough’ filter of your dataset - there will still be a number of ‘cells’ that are actually just background from empty droplets or simply low-quality. There will also be genes that could be sequencing artifacts or that appear with such low frequency that statistical tools will fail to analyse them. This background garbage of both cells and genes not only makes it harder to distinguish real biological information from the noise, but also makes it computationally heavy to analyse. These spurious reads take a lot of computational power to analyse! First on our agenda is to filter this matrix to give us cleaner data to extract meaningful insight from, and to allow faster analysis.

> <question-title></question-title>
>
> 1. What information is stored in your AnnData object? The last tool to generate this object counted the mitochondrial associated genes in your matrix. Where is that data stored?
> 2. While you are figuring that out, how many genes and cells are in your object?
>
>   > <tip-title>Hint</tip-title>
>   > Inspect the Anndata object by printing it with:
>   > ```python
>   > print(adata)
>   >
>   > print(adata.obs)
>   >
>   > print(adata.var)
>   > ```
>   {: .tip}
> > <solution-title></solution-title>
> >
> > 1. If you examine your AnnData object, you'll find a number of different quality control metrics for both cells (**obs**) and genes (**var**).
> >   - For instance, you can see a `n_cells` under **var**, which counts the number of cells that gene appears in.
> >   - In the **obs**, you have both discrete and log-based metrics for `n_genes`, how many genes are counted in a cell, and `n_counts`, how many UMIs are counted per cell. So, for instance, you might count multiple GAPDHs in a cell. Your `n_counts` should thus be higher than `n_genes`.
> >   - But what about the mitochondria?? Within the cells information **obs**, the `total_counts_mito`,  `log1p_total_counts_mito`, and `pct_counts_mito` has been calculated for each cell.
> > 2. You can see by printing the object that the matrix is `31178 x 35734`. This is `obs x vars`, or rather, `cells x genes`, so there are `31178 cells` and `35734 genes` in the matrix.
> >
> {: .solution}
>
{: .question}

## Generate QC Plots

We want to filter our cells, but first we need to know what our data looks like. There are a number of subjective choices to make within scRNA-seq analysis, for instance we now need to make our best informed decisions about where to set our thresholds (more on that soon!). We’re going to plot our data a few different ways. Different bioinformaticians might prefer to see the data in different ways, and here we are only generating some of the myriad of plots you can use. Ultimately you need to go with what makes the most sense to you.

## Creating the Plots

```python
# Violin - genotype - log
sc.pl.violin(
  adata,
  keys=['log1p_total_counts', 'log1p_n_genes_by_counts', 'pct_counts_mito'],
  groupby='genotype',
  save='-genotype-log.png'
)
```


```python
# Violin - sex - log
sc.pl.violin(
  adata,
  keys=['log1p_total_counts', 'log1p_n_genes_by_counts', 'pct_counts_mito'],
  groupby='sex',
  save='-sex-log.png'
)
```

```python
# Violin - batch - log
sc.pl.violin(
  adata,
  keys=['log1p_total_counts', 'log1p_n_genes_by_counts', 'pct_counts_mito'],
  groupby='batch',
  save='-batch-log.png'
)
```

```python
# Scatter - mito x UMIs
sc.pl.scatter(
  adata,
  x='log1p_total_counts',
  y='pct_counts_mito',
  save='-mitoxUMIs.png'
)
```

```python
# Scatter - mito x genes
sc.pl.scatter(
  adata,
  x='log1p_n_genes_by_counts',
  y='pct_counts_mito',
  save='-mitoxgenes.png'
)
```

```python
# Scatter - genes x UMIs
sc.pl.scatter(
  adata,
  x='log1p_total_counts',
  y='log1p_n_genes_by_counts',
  color='pct_counts_mito',
  save='-genesxUMIs.png'
)
```

## Analysing the plots

That’s a lot of information! Let’s attack this in sections and see what questions these plots can help us answer.

> <question-title>Batch Variation</question-title>
>
> Are there differences in sequencing depth across the samples?
> 1. Which plot(s) addresses this?
> 2. How do you interpret it?
>
> > <solution-title></solution-title>
> >
> > 1. The plot `violin - batch - log` will have what you're looking for!
> >     ![Violin - batch - log](../../images/scrna-casestudy-jupyter/violin-batch-log.png "Violin - batch - log (Raw)")
> > 2. Keeping in mind that this is a log scale - which means that small differences can mean large differences - the violin plots probably look pretty similar.
> >    - `N703` and `N707` might be a bit lower on genes and counts (or UMIs), but the differences aren't catastrophic.
> >    - The `pct_counts_mito` looks pretty similar across the batches, so this also looks good.
> >    - Nothing here would cause us to eliminate a sample from our analysis, but if you see a sample looking completely different from the rest, you would need to question why that is and consider eliminating it from your experiment!
> >
> {: .solution}
>
{: .question}

> <question-title>Biological Variables</question-title>
>
> Are there differences in sequencing depth across sex? Genotype?
> 1. Which plot(s) addresses this?
> 2. How do you interpret the `sex` differences?
> 3. How do you interpret the `genotype` differences?
>
> > <solution-title></solution-title>
> >
> > 1. Similar to above, the plots `violin - sex - log` and `violin - genotype - log` will have what you're looking for!
> >      ![Violin - sex - log](../../images/scrna-casestudy-jupyter/violin-sex-log.png "Violin - sex - log (Raw)")
> >      ![Violin - genotype - log](../../images/scrna-casestudy-jupyter/violin-genotype-log.png "Violin - genotype - log (Raw)")
> > 2. There isn’t a major difference in sequencing depth across sex, I would say - though you are welcome to disagree!
> >    - It is clear there are far fewer female cells, which makes sense given that only one sample was female. *Note - that was an unfortunate discovery made long after generating libraries. It's quite hard to identify the sex of a neonate in the lab! In practice, try hard to not let such a confounding factor into your data! You could consider re-running all the following analysis without that female sample, if you wish.*
> > 3. In `Violin - genotype - log`, however, we can see there is a difference. The `knockout` samples clearly have fewer genes and counts. From an experimental point of view, we can consider, does this make sense?
> >    - Would we biologically expect that those cells would be smaller or having fewer transcripts? Possibly, in this case, given that these cells were generated by growth restricted neonatal mice, and in which case we don’t need to worry about our good data, but rather keep this in mind when generating clusters, as we don’t want depth to define clusters, we want biology to!
> >    - On the other hand, it may be that those cells didn’t survive dissociation as well as the healthy ones (in which case we’d expect higher mitochondrial-associated genes, which we don’t see, so we can rule that out!).
> >    - Maybe we unluckily poorly prepared libraries for specifically those knockout samples. There are only three, so maybe those samples are under-sequenced.
> >    - So what do we do about all of this?
> >        - Ideally, we consider re-sequencing all the samples but with a higher concentration of the knockout samples in the library. Any bioinformatician will tell you that the best way to get clean data is in the lab, not the computer! Sadly, absolute best practice isn’t necessarily always a realistic option in the lab - for instance, that mouse line was long gone! - so sometimes, we have to make the best of it. There are options to try and address such discrepancy in sequencing depth. Thus, we’re going to take these samples forward and see if we can find biological insight despite the technical differences.
> >
> {: .solution}
>
{: .question}

Now that we’ve assessed the differences in our samples, we will look at the libraries overall to identify appropriate thresholds for our analysis.

> <question-title>Filter Thresholds: genes</question-title>
>
> What threshold should you set for `log1p_n_genes_by_counts`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. Any plot with `log1p_n_genes_by_counts` would do here, actually! Some people prefer scatterplots to violins.
> >    ![Scatter-genesxmito](../../images/scrna-casestudy-jupyter/scatter-mito-genes.png "Scatter - mito x genes (Raw)")
> >
> > 2. In `Scatter - mito x genes` you can see how cells with `log1p_n_genes_by_counts` up to around, perhaps, `5.7` (around 300 genes) often have high `pct_counts_mito`.
> >   - You can plot this as just `n_counts` and see this same trend at around 300 genes, but with this data the log format is clearer so that's how we're presenting it.
> >   - You could also use the violin plots to come up with the threshold, and thus also take batch into account. It's good to look at the violins as well, because you don't want to accidentally cut out an entire sample (i.e. N703 and N707).
> >   - Some bioinformaticians would recommend filtering each sample individually, but this is difficult in larger scale and in this case (you're welcome to give it a go! You'd have to filter separately and then concatenate), it won't make a notable difference in the final interpretation.
> >
> {: .solution}
>
{: .question}

> <question-title>Filter Thresholds: UMIs</question-title>
>
> What threshold should you set for `log1p_total_counts`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. As before, any plot with `log1p_total_counts` will do! Again, we'll use a scatterplot here, but you can use a violin plot if you wish!
> >    ![Scatter-countsxmito](../../images/scrna-casestudy-jupyter/scatter-mito-umis.png "Scatterplot - mito x UMIs (Raw)")
> >
> > 2. We can see that we will need to set a higher threshold (which makes sense, as you'd expect more UMI's per cell rather than unique genes!). Again, perhaps being a bit aggressive in our threshold, we might choose `6.3`, for instance (which amounts to around 500 counts/cell).
> >   - In an ideal world, you'll see a clear population of real cells separated from a clear population of debris. Many samples, like this one, are under-sequenced, and such separation would likely be seen after deeper sequencing!
> >
> {: .solution}
>
{: .question}

> <question-title>Filter Thresholds: mito</question-title>
>
> What threshold should you set for `pct_counts_mito`?
> 1. Which plot(s) addresses this?
> 2. What number would you pick?
>
> > <solution-title></solution-title>
> >
> > 1. Any plot with `pct_counts_mito` would do here, however the scatterplots are likely the easiest to interpret. We'll use the same as last time.
> >    ![Scatter-countsxmito](../../images/scrna-casestudy-jupyter/scatter-mito-umis.png "Scatterplot - mito x UMIs (Raw)")
> >
> > 2. We can see a clear trend wherein cells that have around 5% mito counts or higher also have far fewer total counts. These cells are low quality, will muddy our data, and are likely stressed or ruptured prior to encapsulation in a droplet. While 5% is quite a common cut-off, this is quite messy data, so just for kicks we'll go more aggressive with a `4.5%`.
> >    - In general, you must adapt all cut-offs to your data - metabolically active cells might have higher mitochondrial RNA in general, and you don't want to lose a cell population because of a cut-off.
> >
> {: .solution}
>
{: .question}

## Applying the Thresholds

It’s now time to apply these thresholds to our data! First, a reminder of how many cells and genes are in your object: ```31178 cells``` and ```35734 genes```. Let’s see how that changes each time!

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **log1p_n_genes_by_counts** > `5.7`
>      - **log1p_total_counts** > `6.3`
>      - **pct_counts_mito** < `4.5%`
> - Everyone else: Choose your own thresholds and compare results!
{: .details}

We will plot the raw data before applying any filters so that we can more clearly see the changes we will make.

```python
# Raw
sc.pl.violin(
  adata,
  keys=['log1p_total_counts', 'log1p_n_genes_by_counts', 'pct_counts_mito'],
  groupby='genotype',
  save='-raw.png'
)
```

```python
genes_filtered_obj = adata[adata.obs['log1p_n_genes_by_counts'] >= 5.7]
genes_filtered_obj = genes_filtered_obj[genes_filtered_obj.obs['log1p_n_genes_by_counts'] <= 20.0]

# Violin - Filterbygenes
sc.pl.violin(
  genes_filtered_obj,
  keys=['log1p_total_counts', 'log1p_n_genes_by_counts', 'pct_counts_mito'],
  groupby='genotype',
  save='-Filterbygenes.png'
)
```

```python
print(genes_filtered_obj)
```

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violinplot-filteronce](../../images/scrna-casestudy-jupyter/violin-raw-filteredgenes.png "Raw vs 1st filter - genes/cell")
> > 1. The only part that seems to change is the `log1p_n_genes_by_counts`.  You can see a flatter bottom to the violin plot - this is the lower threshold set. Ideally, this would create a beautiful violin plot because there would be a clear population of low-gene number cells. Sadly not the case here, but still a reasonable filter.
> > 2. In the printed AnnData information, you can see you now have `17,040 cells x 35,734 genes`.
> >
> {: .solution}
>
{: .question}


```python
counts_filtered_obj = genes_filtered_obj[genes_filtered_obj.obs['log1p_total_counts'] >=  6.3]
counts_filtered_obj = counts_filtered_obj[counts_filtered_obj.obs['log1p_total_counts'] <= 20.0]

# Violin - Filterbycounts
sc.pl.violin(
  counts_filtered_obj,
  keys=['log1p_total_counts', 'log1p_n_genes_by_counts', 'pct_counts_mito'],
  groupby='genotype',
  save='-Filterbycounts.png'
)
```

```python
print(counts_filtered_obj)
```

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violinplot-filtertwice](../../images/scrna-casestudy-jupyter/violin-filteredgenesxfilteredcounts.png "1st filter vs 2nd filter - counts/cell")
> > 1. We will focus on the `log1p_total_counts` as that shows the biggest change. Similar to above, the bottom of the violin shape has flattered due to the threshold.
> > 2. In the printed AnnData information, you can see you now have `8,678 cells x 35,734 genes`.
> >
> {: .solution}
>
{: .question}

```python
mito_filtered_obj = counts_filtered_obj[counts_filtered_obj.obs['pct_counts_mito'] >=  0]
mito_filtered_obj = mito_filtered_obj[mito_filtered_obj.obs['pct_counts_mito'] <= 4.5]

# Violin - Filterbymito
sc.pl.violin(
  mito_filtered_obj,
  keys=['log1p_total_counts', 'log1p_n_genes_by_counts', 'pct_counts_mito'],
  groupby='genotype',
  save='-Filterbymito.png'
)
```

```python
print(mito_filtered_obj)
```

> <question-title></question-title>
>
> 1. Interpret the violin plot
> 2. How many genes & cells do you have in your object now?
>
> > <solution-title></solution-title>
> >
> > ![Violinplot-filtermito](../../images/scrna-casestudy-jupyter/violin-mitofilter.png "Violin plots after filtering genes, counts, and mito content/cell")
> > 1. If we carefully check the axes, we can see that the `pct_counts_mito` has shrunk.
> > 2. In the printed AnnData information, you can see you now have `8,605 cells x 35,734 genes`.
> >
> {: .solution}
>
{: .question}

Here’s a quick overall summary for easy visualisation if you fancy it.

![Filtering summary](../../images/scrna-casestudy-jupyter/violins2gether.png "Filtering summary")

Fantastic work! However, you’ve now removed a whole heap of cells, and since the captured genes are sporadic (i.e. a small percentage of the overall transcriptome per cell) this means there are a number of genes in your matrix that are currently not in any of the remaining cells. Genes that do not appear in any cell, or even in only 1 or 2 cells, will make some analytical tools break and overall will not be biologically informative. So let’s remove them! Note that ```3``` is not necessarily the best number, rather it is a fairly conservative threshold. You could go as high as 10 or more.

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - **min_cells** = `3`
> - Everyone else: Choose your own thresholds and compare results! Note if you go less than 3 (or even remove this step entirely), future tools are likely to fail due to empty gene data.
{: .details}

```python
filtered_obj = mito_filtered_obj.copy()

sc.pp.filter_genes(filtered_obj, min_cells=3)
sc.pp.filter_genes(filtered_obj, max_cells=1000000000)

print(filtered_obj)
```

In practice, you’ll likely choose your thresholds then set up all these filters to run without checking plots in between each one. But it’s nice to see how they work!

Using the final ```filtered_object```, we can summarise the results of our filtering:

|                    | Description | Genes |
| ------------------ | ----------- | ----- |
| Raw                | 31178       | 35734 |
| Filter genes/cell  | 17040       | 35734 |
| Filter counts/cell | 8678        | 35734 |
| Filter mito/cell   | 8605        | 35734 |
| Filter cells/gene  | 8605        | 15395 |

{% icon congratulations %} Congratulations! You have filtered your object! Now it should be a lot easier to analyse.

# Processing

So currently, you have a matrix that is 8605 cells by 15395 genes. This is still quite big data. We have two issues here - firstly, you already know there are differences in how many transcripts and genes have been counted per cell. This technical variable can obscure biological differences. Secondly, we like to plot things on x/y plots, so for instance Gapdh could be on one axis, and Actin can be on another, and you plot cells on that 2-dimensional axis based on how many of each transcript they possess. While that would be fine, adding in a 3rd dimension (or, indeed, in this case, 15393 more dimensions), is a bit trickier! So our next steps are to transform our big data object into something that is easy to analyse and easy to visualise.

```python
output_h5ad = filtered_obj.copy()
sc.pp.normalize_total(output_h5ad)
```

Normalisation helps reduce the differences between gene and UMI counts by fitting total counts to 10,000 per cell. The inherent log-transform (by log(count+1)) aligns the gene expression level better with a normal distribution. This is fairly standard to prepare for any future dimensionality reduction.

Now we need to look at reducing our gene dimensions. We have loads of genes, but not all of them are different from cell to cell. For instance, housekeeping genes are defined as not changing much from cell to cell, so we could remove these from our data to simplify the dataset. We will flag genes that vary across the cells for future analysis.

```python
output_h5ad = sc.pp.log1p(output_h5ad, copy=True)  # below function requires log scaled data
sc.pp.highly_variable_genes(output_h5ad)
```

Next up, we’re going to scale our data so that all genes have the same variance and a zero mean. This is important to set up our data for further dimensionality reduction. It also helps negate sequencing depth differences between samples, since the gene levels across the cells become comparable. Note, that the differences from scaling etc. are not the values you have at the end - i.e. if your cell has average GAPDH levels, it will not appear as a ‘0’ when you calculate gene differences between clusters.

```python
scaled_data = sc.pp.scale(output_h5ad, max_value=10.0, copy=True)
```

{% icon congratulations %} Congratulations! You have processed your object!

> <comment-title></comment-title>
> At this point, we might want to remove or regress out the effects of unwanted variation on our data. A common example of this is the cell cycle, which can affect which genes are expressed and how much material is present in our cells. If you’re interested in learning how to do this, then you can move over to the [Removing the Effects of the Cell Cycle]({% link topics/single-cell/tutorials/scrna-case_cell-cycle/tutorial.md %}) tutorial now – then return here to complete your analysis.
{: .comment}

# Preparing coordinates

We still have too many dimensions. Transcript changes are not usually singular - which is to say, genes were in pathways and in groups. It would be easier to analyse our data if we could more easily group these changes.

## Principal components

Principal components are calculated from highly dimensional data to find the most spread in the dataset. So in our, ```1982``` highly variable gene dimensions, there will be one line (axis) that yields the most spread and variation across the cells. That will be our first principal component. We can calculate the first ```x``` principal components in our data to drastically reduce the number of dimensions.

> <comment-title>1982???</comment-title>
> Where did the `1982` come from?
>
> The quickest way to figure out how many highly variable genes you have, in my opinion, is to re-run ```sc.pp.highly_variable_genes``` function with the added parameter ```subset=True```, therefore: ```sc.pp.highly_variable_genes(output_h5ad, subset=True)```. This subsetting removes any nonvariable genes.
>
> Then you can ```print(output_h5ad)``` and you'll see only 1982 genes. The following processing steps will use only the highly variable genes for their calculations, but depend on keeping all genes in the object. Thus, please use the original output of your ```sc.pp.highly_variable_genes``` function with far more than 1982 genes!, currently stored as ```scaled_data```.
{: .comment}

> <warning-title>Check your AnnData object!</warning-title>
> Run ```print(scaled_data)```
> Your AnnData object should have far more than 1982 genes in it (if you followed our settings and tool versions, you'd have a matrix 8605 × 15395 (cells x genes). Make sure to use that AnnData object output from FindVariableGenes, rather than the 1982 from your testing in the section above labelled '1982'.
{: .warning}

```python
pca_components = sc.tl.pca(scaled_data, n_comps=50, copy=True)
```

Why 50 principal components you ask? Well, we’re pretty confident 50 is an over-estimate. Let's visualise the variance of each principal component.

```python
sc.pl.pca_variance_ratio(pca_components, n_pcs=50, save='-variance-ratio.png')
```

![Variance ratio](../../images/scrna-casestudy-jupyter/pca-variance.png "Variance ratio")

We can see that there is really not much variation explained past component 19. So we might save ourselves a great deal of time and muddied data by focusing on the top ```20``` PCs.

## Neighborhood graph

We’re still looking at around 20 dimensions at this point. We need to identify how similar a cell is to another cell, across every cell across these dimensions. For this, we will use the k-nearest neighbor (kNN) graph, to identify which cells are close together and which are not. The kNN graph plots connections between cells if their distance (when plotted in this 20 dimensional space!) is amonst the k-th smallest distances from that cell to other cells. This will be crucial for identifying clusters, and is necessary for plotting a UMAP. From <a href="https://github.com/lmcinnes/umap">UMAP developers</a>: “Larger neighbor values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the range 5 to 50, with a choice of 10 to 15 being a sensible default”.

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **Number of PCs to use** = `20`
>      - **Maximum number of neighbours used** = `15`
> - Everyone else: Use the PC variance plot to pick your own PC number, and choose your own neighbour maximum as well!
{: .details}

```python
neighbours = sc.pp.neighbors(pca_components, n_neighbors=15, use_rep='X_pca', n_pcs=20, copy=True)
```

## Dimensionality reduction for visualisation

Two major visualisations for this data are tSNE and UMAP. We must calculate the coordinates for both prior to visualisation. For tSNE, the parameter [perplexity](https://www.nature.com/articles/s41467-019-13056-x) can be changed to best represent the data, while for UMAP the main change would be to change the kNN graph above itself, by changing the <b>neighbours.</b>

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest varied numbers so that you can compare results throughout the tutorials.
> - Control
>      - **Perplexity** = `30`
> - Everyone else: Choose your own perplexity, between 5 and 50!
{: .details}

```python
tsne_components = sc.tl.tsne(neighbours, use_rep='X_pca', perplexity=30, copy=True)
```

```python
umap_components = sc.tl.umap(tsne_components, copy=True)
```

{% icon congratulations %} Congratulations! You have prepared your object and created neighborhood coordinates. We can now use those to call some clusters!

# Cell clusters & gene markers

> <question-title></question-title>
>
> Let's take a step back here. What is it, exactly, that you are trying to get from your data? What do you want to visualise, and what information do you need from your data to gain insight?
>
> > <solution-title></solution-title>
> >
> > Really we need two things - firstly, we need to make sure our experiment was set up well. This is to say, our biological replicates should overlap and our variables should, ideally, show some difference. Secondly, we want insight - we want to know which cell types are in our data, which genes drive those cell types, and in this case, how they might be affected by our biological variable of growth restriction. How does this affect the developing cells, and what genes drive this? So let's add in information about cell clusters and gene markers!
> >
> {: .solution}
>
{: .question}

Finally, let’s identify clusters! Unfortunately, it’s not as majestic as biologists often think - the maths doesn’t necessarily identify true cell clusters. Every algorithm for identifying cell clusters falls short of a biologist knowing their data, knowing what cells should be there, and proving it in the lab. Sigh. So, we’re going to make the best of it as a starting point and see what happens! We will define clusters from the kNN graph, based on how many connections cells have with one another. Roughly, this will depend on a resolution parameter for how granular you want to be.

> <details-title>Working in a group? Decision-time!</details-title>
> Oh yes, yet another decision! Single cell analysis is sadly not straight forward.
> - Control
>      - **Resolution, high value for more and smaller clusters** = `0.6`
> - Everyone else: Pick your own number. If it helps, this sample should have a lot of very similar cells in it. It contains developing T-cells, so you aren't expecting massive differences between cells, like you would in, say, an entire embryo, with all sorts of unrelated cell types.
{: .details}

```python
# Find Clusters
clusters = sc.tl.louvain(umap_components, resolution=0.6, copy=True)
```

Nearly plotting time! But one final piece is to add in SOME gene information. Let’s focus on genes driving the clusters.

# FindMarkers

```python
markers_cluster = sc.tl.rank_genes_groups(clusters, groupby="louvain", method='t-test_overestim_var', n_genes=50, copy=True)
```

But we are also interested in differences across genotype, so let’s also check that (note that in this case, it’s turning it almost into bulk RNA-seq, because you’re comparing all cells of a certain genotype against all cells of the other)

```python
markers_genotype = sc.tl.rank_genes_groups(markers_cluster, groupby="genotype", method='t-test_overestim_var', n_genes=50, copy=True)
```

**Note:** The function ```rank_genes_groups``` does not return a DataFrame that we can use but instead metadata about the marker table, so first we need to construct the marker table using this generated metadata. This is done using the following function, however it's not too important to understand what this code does!

```python
def generate_marker_table(adata):
    # extract marker table metadata
    res = adata.uns['rank_genes_groups']

    # generate DataFrame from metadata
    res_df = pd.DataFrame({
                "genes": pd.DataFrame(res["names"]).stack(),
                "scores": pd.DataFrame(res["scores"]).stack(),
                "logfoldchanges": pd.DataFrame(res["logfoldchanges"]).stack(),
                "pvals": pd.DataFrame(res["pvals"]).stack(),
                "pvals_adj": pd.DataFrame(res["pvals_adj"]).stack(),
            })

    # convert row names to columns
    res_df.index.name = 'newhead'
    res_df.reset_index(inplace=True)

    # rename generic column names
    res_df = res_df.rename(columns={'level_0': 'rank', 'level_1':'cluster'})

    # reorder columns
    res_df = res_df.reindex(columns=['cluster', 'rank', 'genes', 'scores', 'logfoldchanges', 'pvals', 'pvals_adj'])

    # insert ref column
    res_df.insert(2, 'ref', 'rest')

    return res_df
```

Now we can generate our marker tables!

```python
# Generate marker tables
cluster_marker_table = generate_marker_table(markers_cluster)
genotype_marker_table = generate_marker_table(markers_genotype)

display(cluster_marker_table.head(4))
display(genotype_marker_table.head(4))
```

Now, there’s a small problem here, which is that if you inspect the output marker tables (above tables), you won’t see gene names, you’ll see Ensembl IDs. While this is a more bioinformatically accurate way of doing this (not every ID has a gene name!), we might want to look at more well-recognised gene names, so let’s pop some of that information in!

```python
# Join two datasets

cluster_joined = pd.merge(cluster_marker_table, markers_cluster.var, left_on='genes', right_on='ID')
genotype_joined = pd.merge(genotype_marker_table, markers_genotype.var, left_on='genes', right_on='ID')

display(cluster_joined.head(5))
display(genotype_joined.head(5))
```

```python
# Cut columns from tables

cluster_markers_named = cluster_joined[['cluster', 'ref', 'rank', 'genes', 'Symbol', 'scores', 'logfoldchanges', 'pvals', 'pvals_adj']]
genotype_markers_named = genotype_joined[['cluster', 'ref', 'rank', 'genes', 'Symbol', 'scores', 'logfoldchanges', 'pvals', 'pvals_adj']]

display(cluster_markers_named.head(5))
display(genotype_markers_named.head(5))
```

Well done! It’s time for the best bit, the plotting!

# Plotting!

It’s time! Let’s plot it all! But first, let’s pick some marker genes from the ```markers_cluster``` list that you made as well. I’ll be honest, in practice, you’d now be spending a lot of time looking up what each gene does (thank you google!). There are burgeoning automated-annotation tools, however, so long as you have a good reference (a well annotated dataset that you’ll use as the ideal). In the mean time, let’s do this the old-fashioned way, and just copy a bunch of the markers in the original paper.

```python
# PCA
sc.pl.embedding(
    markers_cluster,
    basis='pca',  
    color=['louvain','sex','batch','genotype','Il2ra','Cd8b1','Cd8a','Cd4','Itm2a','Aif1','log1p_total_counts'],
    gene_symbols='Symbol',
    use_raw=False,
    save='.png'
)
```

```python
# TSNE
sc.pl.embedding(
    markers_cluster,
    basis='tsne',  
    color=['louvain','sex','batch','genotype','Il2ra','Cd8b1','Cd8a','Cd4','Itm2a','Aif1','log1p_total_counts'],
    gene_symbols='Symbol',
    use_raw=False,
    save='.png'
)
```

```python
# UMAP
sc.pl.embedding(
    markers_cluster,
    basis='umap',  
    color=['louvain','sex','batch','genotype','Il2ra','Cd8b1','Cd8a','Cd4','Itm2a','Aif1','log1p_total_counts'],
    gene_symbols='Symbol',
    use_raw=False,
    save='.png'
)
```

{% icon congratulations %} Congratulations! You now have plots galore!

# Insights into the beyond

Now it’s the fun bit! We can see where genes are expressed, and start considering and interpreting the biology of it. At this point, it’s really about what information you want to get from your data - the following is only the tip of the iceberg. However, a brief exploration is good, because it may help give you ideas going forward with for your own data. Let us start interrogating our data!

## Biological Interpretation

> <question-title>Appearance is everything</question-title>
>
> Which visualisation is the most useful for getting an overview of our data, *pca*, *tsne*, or *umap*?
> ![PCA-tSNE-UMAP](../../images/scrna-casestudy-jupyter/louvain_clustering.png "Louvain clustering by dimension reduction")
>
> > <solution-title></solution-title>
> >
> > You can see why a PCA is generally not enough to see clusters in samples - keep in mind, you're only seeing components 1 and 2! - and therefore why the tSNE and UMAP visualisation dimensionality reductions are so useful. But there is not necessarily a clear winner between tSNE and UMAP, but I think UMAP is slightly clearer with its clusters, so we'll stick with that for the rest of the analysis.
> >
> {: .solution}
>
{: .question}

Note that the cluster numbering is based on size alone - clusters 0 and 1 are not necessarily related, they are just the clusters containing the most cells. It would be nice to know what exactly these cells are. This analysis (googling all of the marker genes, both checking where the ones you know are as well as going through the marker tables you generated!) is a fun task for any individual experiment, so we’re going to speed past that and nab the assessment from the original paper!

| Clusters   | Marker                  | Cell type                            |
| ---------- | ----------------------- | ------------------------------------ |
| 4          | Il2ra                   | Double negative (early T-cell)       |
| 0,1,2,6    | Cd8b1, Cd8a, Cd4        | Double positive (middle T-cell)      |
| 5          | Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell) |
| 3          | Itm2a                   | Mature T-cell                        |
| 7          | Hba-a1                  | RBCs (as impurity here)              |
| -          | Aif1                    | Macrophages                          |



The authors weren’t interested in further annotation of the DP cells, so neither are we. Sometimes that just happens. The maths tries to call similar (ish) sized clusters, whether it is biologically relevant or not. Or, the question being asked doesn’t really require such granularity of clusters.

![Known marker gene locations](../../images/scrna-casestudy-jupyter/known_marker_gene_locations.png "Known marker genes locations")

> <details-title>Working in a group? Important!</details-title>
> If you have deviated from any of the original parameters in this tutorial, you will likely have a different number of clusters. You will, therefore, need to change the ```categories``` parameter in ```rename_categories``` accordingly. Best of luck!
>
{: .details}

## Annotating Clusters

As mentioned at the beginning of the tutorial, you might not get outputs identical to a tutorial if you are running it in a programming environment, but the outputs should still be pretty close. However, you have to double check if the categories (ie. cell types) in the example below correspond to the identified cluster numbers based on gene expression. You might need to change the order of assigned cell types in the *categories* parameter to match the cluster numbers identified by *louvain*.

```python
# Add meaningful names to each category
markers_cluster.rename_categories(key='louvain',
    categories=['DP-M4','DP-M3','T-mat','DN','DP-M2','DP-M1','DP-L','RBCs'])   # categories (cell types) correspond to the identified cluster numbers, ie. [0, 1, 2,...,7]

# Copy AnnData object
markers_cluster_copy = markers_cluster.copy()

# Rename 'louvain' column
markers_cluster_copy.obs = markers_cluster_copy.obs.rename(columns={'louvain': 'cell_type'})

# Scanpy - plot updated object
sc.pl.embedding(
    markers_cluster_copy,
    basis='umap',  
    color=['cell_type','sex','batch','genotype','Il2ra','Cd8b1','Cd8a','Cd4','Itm2a','Aif1','Hba-a1','log1p_total_counts'],
    gene_symbols='Symbol',
    use_raw=False,
    save='-annotated.png'
)
```

![Annotated Embeddings](../../images/scrna-casestudy-jupyter/annotated_umap.png "Our Annotated UMAP")

Now that we know what we’re dealing with, let’s examine the effect of our variable, proper science!

> <question-title>Genotype</question-title>
>
> Are there any differences in genotype? Or in biological terms, is there an impact of growth restriction on T-cell development in the thymus?
>
> ![Genotype Images](../../images/scrna-casestudy-jupyter/genotype_differences.png "Genotype differences")
>
> > <solution-title></solution-title>
> >
> > We can see that DP-L, which seems to be extending away from the DP-M bunch, as well as the mature T-cells (or particularly the top half) are missing some knockout cells. Perhaps there is some sort of inhibition here? INTERESTING! What next? We might look further at the transcripts present in both those populations, and perhaps also look at the genotype marker table... So much to investigate! But before we set you off to explore to your heart's delight, let's also look at this a bit more technically.
> >
> {: .solution}
>
{: .question}

## Technical Assessment

Is our analysis real? Is it right? Well, we can assess that a little bit.

> <question-title>Batch effect</question-title>
>
> Is there a batch effect?
>
> ![Batch effect](../../images/scrna-casestudy-jupyter/batch_effect.png "Batch effect?")
>
> > <solution-title></solution-title>
> >
> > While some shifts are expected and nothing to be concerned about, DP-L looks to be mainly comprised of N705. There might be a bit of batch effect, so you could consider using batch correction on this dataset. However, if we focus our attention on the other cluster - mature T-cells -  where there is batch mixing, we can still assess this biologically even without batch correction.
> > Additionally, we will also look at the confounding effect of sex.
> >
> > ![Sex effect](../../images/scrna-casestudy-jupyter/sex_differences.png "Sex differences")
> >
> > We note that the one female sample - unfortunately one of the mere three knockout samples - seems to be distributed in the same areas as the knockout samples at large, so luckily, this doesn't seem to be a confounding factor and we can still learn from our data. Ideally, this experiment would be re-run with either more female samples all around or swapping out this female from the male sample.
> >
> {: .solution}
>
{: .question}

> <question-title>Depth effect</question-title>
>
> Are there any clusters or differences being driven by sequencing depth, a technical and random factor?
>
> ![Sequencing depth](../../images/scrna-casestudy-jupyter/counts_across_clusters.png "Counts across clusters")
>
> > <solution-title></solution-title>
> >
> > Eureka! This explains the odd DP shift between wildtype and knockout cells - the left side of the DP cells simply have a higher sequencing depth (UMIs/cell) than the ones on the right side. Well, that explains some of the sub-cluster that we're seeing in that splurge. Importantly, we don't see that the DP-L or (mostly) the mature T-cell clusters are similarly affected. So, whilst again, this variable of sequencing depth might be something to regress out somehow, it doesn't seem to be impacting our dataset. The less you can regress/modify your data, in general, the better - you want to stay as true as you can to the raw data, and only use maths to correct your data when you really need to (and not to create insights where there are none!).
> >
> {: .solution}
>
{: .question}

> <question-title>Sample purity</question-title>
>
> Do you think we processed these samples well enough?
>
> ![Sequencing depth](../../images/scrna-casestudy-jupyter/hemoglobin.png "Hemoglobin across clusters")
>
> > <solution-title></solution-title>
> >
> > We have seen in the previous images that these clusters are not very tight or distinct, so we could consider stronger filtering. Additionally, hemoglobin - a red blood cell marker that should NOT be found in T-cells - appears throughout the entire sample in low numbers. This suggests some background in the media the cells were in, and we might consider in the wet lab trying to get a purer, happier sample, or in the dry lab, techniques such as SoupX or others to remove this background. Playing with filtering settings (increasing minimum counts/cell, etc.) is often the place to start in these scenarios.
> >
> {: .solution}
>
{: .question}

> <question-title>Clustering resolution</question-title>
>
> Do you think the clustering is appropriate? i.e. are there single clusters that you think should be separate, and multiple clusters that could be combined?
>
> ![Itm2a Expression](../../images/scrna-casestudy-jupyter/itm2a_across_clusters.png "Itm2a across clusters")
>
> > <solution-title></solution-title>
> >
> > Important to note, lest all bioinformaticians combine forces to attack the biologists: just because a cluster doesn't look like a cluster by eye is NOT enough to say it's not a cluster! But looking at the biology here, we struggled to find marker genes to distinguish the DP population, which we know is also affected by depth of sequencing. That's a reasonable argument that DP-M1, DP-M2, and DP-M3 might not be all that different. Maybe we need more depth of sequencing across all the DP cells, or to compare these explicitly to each other (consider variations on FindMarkers!). However, DP-L is both seemingly leaving the DP cluster and also has fewer knockout cells, so we might go and look at what DP-L is expressing in the marker genes. If we look at T-mat further, we can see that its marker gene - Itm2a - is only expressed in half of the cluster. You might consider sub-clustering this to investigate further, either through changing the resolution or through analysing this cluster alone.
> > If we look at the differences between genotypes alone (so the pseudo-bulk), we can see that most of the genes in that list are actually ribosomal. This might be a housekeeping background, this might be cell cycle related, this might be biological, or all three. You might consider investigating the cycling status of the cells, or even regressing this out (which is what the authors did).
> {: .solution}
>
{: .question}

Ultimately, there are quite a lot ways to analyse the data, both within the confines of this tutorial (the many parameters that could be changed throughout) and outside of it (batch correction, sub-clustering, cell-cycle scoring, inferred trajectories, etc.) Most analyses will still yield the same general output, though: there are fewer knockout cells in the mature T-cell population.

{% icon congratulations %} Congratulations! You have interpreted your plots in several important ways!

# Export your data, and notebook, and figures

It’s now time to export your data! First, we need to get Jupyter to see it as a file.

```python
adata.write('MarkersCluster.h5ad')
```

Now you can export it.

```python
put("MarkersCluster.h5ad")
```

To export your notebook to your Galaxy history, you can use the following. Change the text to be your notebook name. Do not use spaces!

```python
put("name_of_jupyter_notebook.ipynb")
```

Want to export some plots? Choose any (or all) of the plots you saved as files in the folder at the left and put their titles in the following. You can run multiple exports at the same time.

```python
put("figures/plotname.png")
put("figures/plotname.png")
put("figures/plotname.png")
```

# Conclusion

{% icon congratulations %} Congratulations! You’ve made it to the end!

In this tutorial, you moved from technical processing to biological exploration. By analysing real data - both the exciting and the messy! - you have, hopefully, experienced what it’s like to analyse and question a dataset, potentially without clear cut-offs or clear answers. If you were working in a group, you each analysed the data in different ways, and most likely found similar insights. One of the biggest problems in analysing scRNA-seq is the lack of a clearly defined pathway or parameters. You have to make the best call you can as you move through your analysis, and ultimately, when in doubt, try it multiple ways and see what happens!

If, for some reasons anything didn't work in Galaxy JupyterLab environment, please don't get discouraged - we prepared a [Google Colab notebook version](https://colab.research.google.com/drive/1DkCysA77iaFAWoKJ1vwE5_qYV8Su7UsX?usp=sharing) for you as a backup so that you can enjoy the tutorial no matter what!


topic: single-cell
tutorial: scrna-case_alevin-combine-datasets
title: Combining single cell datasets after pre-processing
difficulty: intermediate
time_estimation: 1H
requirements_json: ["type: \"internal\""]
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 05b87d48a443b4f65c8f512fcb1be603
last_modified: 2026-05-27T09:01:49.744665
zenodo_link: https://zenodo.org/records/15090813
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.html

content:
This tutorial will take you from the multiple AnnData outputs of the [previous tutorial](https://humancellatlas.usegalaxy.eu/training-material/topics/transcriptomics/tutorials/scrna-case_alevin/tutorial.html) to a single, combined  AnnData object, ready for all the fun downstream processing. We will also look at how to add in metadata (for instance, SEX or GENOTYPE) for analysis later on.

> <details-title>Where am I?</details-title>
>
> You are in one of the four tutorials associated with a Case Study, which replicates and expands on the analysis performed in a manuscript {% cite Bacon2018 %}.
>
> ![case study overview](../../images/scrna-casestudy/CS2-case_study_overview_2.png "Overview of the four parts of the case study, represented by boxes connected with noodles. There is a signpost specifying that you are currently in the second part.")
>
{: .details}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Get Data
The sample data is a subset of the reads from each of the seven samples in a mouse dataset of fetal growth restriction {% cite Bacon2018 %} (see the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)).

> <details-title>Fun facts about the dataset</details-title>
>
> We are particularly keen for learners to be able to go from raw FASTQ files all the way through analysis. We aren't handing you a curated dataset that we specially modified in order for this tutorial to work.
> - This tutorial's input dataset is the full dataset generated from the {% icon level %} [previous tutorial in this case study]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}).
> - The only difference is that in that previous tutorial, we only analysed one dataset. Our input data, however, is the result of applying that workflow to all seven datasets.
> - You can find this dataset in the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/).
>
{: .details}

{% include _includes/cyoa-choices.html option1="Import History" option2="Zenodo" default="Import History"
       text="If you're on the EU or ORG/USA server, then the quickest way to Get the Data for this tutorial is via importing a history. Otherwise, you can also import from Zenodo - it just might take a moment longer if you're in a live course and everyone is importing the same dataset at the same time!" %}
<div class="Import-History" markdown="1">

> <hands-on-title>Import History</hands-on-title>
>
> 1. Import the {% icon galaxy-history-input %} *Input history* by following the link below
>
>    {% for h in page.input_histories %}
>      [ {{h.label}} Input History]( {{h.history}} )
>    {% endfor %}
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> If you want to import the history to another Galaxy server, check how to do it below!
>
> {% snippet faqs/galaxy/histories_transfer_entire_histories_from_one_galaxy_server_to_another.md %}
>
{: .hands_on}

</div>

<div class="Zenodo" markdown="1">

> <hands-on-title>Data upload for 7 files</hands-on-title>
>
> 1. Create a new history for this tutorial (if you're not importing the history above)
> 2. Import the different AnnData files and the experimental design table from [Zenodo](https://zenodo.org/records/10852529).
>
>    ```
>    {{ page.zenodo_link }}/files/N701-400k-AnnData.h5ad
>    {{ page.zenodo_link }}/files/N702-400k-AnnData.h5ad
>    {{ page.zenodo_link }}/files/N703-400k-AnnData.h5ad
>    {{ page.zenodo_link }}/files/N704-400k-AnnData.h5ad
>    {{ page.zenodo_link }}/files/N705-400k-AnnData.h5ad
>    {{ page.zenodo_link }}/files/N706-400k-AnnData.h5ad
>    {{ page.zenodo_link }}/files/N707-400k-AnnData.h5ad
>    {{ page.zenodo_link }}/files/Experimental_Design.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets
> 4. Check that the AnnData files datatype is `h5ad`, otherwise you will need to change each file to `h5ad`!
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

</div>

{% icon galaxy-eye %} Inspect the {% icon param-file %} `Experimental Design` text file. This shows you how each `N70X` corresponds to a sample, and whether that sample was from a male or female. This will be important metadata to add to our sample, which we will add very similarly to how you added the `gene_name` and `mito` metadata previously!

# Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

{% snippet faqs/gtn/gtn_example_histories.md %}

# Concatenating objects

> <hands-on-title>Concatenating AnnData objects</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `N701-400k`
>    - *"Function to manipulate the object"*: `Concatenate along the observations axis`
>    - {% icon param-file %} *"Annotated data matrix to add"*: `Select all the other matrix files from bottom to top, N702 to N707`
>
>    > <comment-title></comment-title>
>    > If you imported files from Zenodo instead of using the input history, yours might not be in the same order as ours. Since the files will be concatenated in the order that you click, it will be helpful if you click them in the same order, from N702 to N707. This will ensure your samples are given the same batch numbers as we got in this tutorial, which will help when we're adding in metadata later!
>    {: .comment}
>
>    > <warning-title>Don't add N701!</warning-title>
>    > You are adding files to N701, so do not add N701 to itself!
>    {: .warning}
>
>    - *"Join method"*: `Intersection of variables`
>    - *"Key to add the batch annotation to obs"*: `batch`
>    - *"Separator to join the existing index names with the batch category"*: `-`
> 2. Rename {% icon galaxy-pencil %} output `Combined_Object`
{: .hands_on}

Now let's look at what we've done! You can *peek* at the AnnData object in your {% icon galaxy-history %} history. You can also inspect the dataset for further details using a tool.

> <hands-on-title>Inspecting AnnData Objects</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Combined_object`
>    - *"What to inspect?"*: `General information about the object`
> 2. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Combined_object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Combined_object`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
{: .hands_on}

Now have a look at the three {% icon tool %} **Inspect AnnData** outputs.

> <question-title></question-title>
>
> 1. How many cells do you have now?
> 2. Where is `batch` information stored?
>
> > <solution-title></solution-title>
> >
> > 1. If you peek at your dataset, or look at the **General information** {% icon tool %} output, you will find there are `316 cells`, as the matrix is now 316 cells (n_obs) x 35734 genes (n_var). You will also find these numbers in the **obs** {% icon tool %} (cells) and **var** {% icon tool %} (genes) file sizes.
> > 2. Batch information is stored under **Key-indexed observations annotation (obs)**. Different versions of the Manipulate tool might put the `batch` columns in different locations. The tool version in this tutorial puts `batch` in the `8th` column. Batch refers to the order in which the matrices were added. The files are added from the bottom of the history upwards, so be careful how you set up your histories when running this (i.e. if your first dataset is N703 and the second is N701, the `batch` will call N703 `0` and N701 `1`!)
> {: .solution}
>
{: .question}

# Adding cell metadata

I set up the example history with the earliest indices at the bottom.

![The files are numbered such that dataset #1 is N701, dataset #2 is N702, etc., up through #7 as N707. This puts N707 at the top and N701 at the bottom in the Galaxy history.](../../images/scrna-casestudy/CS2-history-files-ascending.png "Correct history ordering for combining datasets in order"){ width=400px }

Therefore, when it is all concatenated together, the `batch` appears as follows:

| Index | Batch | Genotype | Sex |
|------ |--------------------|
| N701 | 0    | wildtype    | male    |
| N702 | 1    | knockout   | male    |
| N703 | 2    | knockout   | female    |
| N704 | 3    | wildtype    | male    |
| N705 | 4    | wildtype    | male    |
| N706 | 5    | wildtype    | male    |
| N707 | 6    | knockout    | male    |

If you used Zenodo to import files, they may not have imported in order (i.e. N701 to N707, ascending). In that case, you will need to tweak the parameters of the next tools appropriately to label your batches correctly!

The two critical pieces of metadata in this experiment are **sex** and **genotype**. I will later want to color my cell plots by these parameters, so I want to add them in now!

## Sex & Genotype metadata

> <hands-on-title>Labelling sex</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Inspect AnnData: Key-indexed observations annotation (obs)** {% icon tool %})
>    - *"1. Replacement"*
>
>         - *"in column"*: `Column: 8` - or whichever column `batch` is in
>         - *"Find pattern"*: `0|1|3|4|5|6`
>         - *"Replace with"*: `male`
>    - **+  Insert Replacement**
>    - *"2. Replacement"*
>
>         - *"in column"*: `Column: 8`
>         - *"Find pattern"*: `2`
>         - *"Replace with"*: `female`
>    - **+  Insert Replacement**
>    - *"3. Replacement"*
>
>         - *"in column"*: `Column: 8`
>         - *"Find pattern"*: `batch`
>         - *"Replace with"*: `sex`
>
>    The output of the {% icon param-file %} **Replace Text** {% icon tool %} tool has many columns. However, we want only the column containing the **sex** information, in order by cell barcode, to add to our AnnData later.
>
> 2. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c8`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: output of **Replace text** {% icon tool %}
>
> 3. Rename {% icon galaxy-pencil %} output `Sex_metadata`
{: .hands_on}

That was so fun, let's do it all again but for genotype!

> <hands-on-title>Labelling genotype</hands-on-title>
>
> 1. {% tool [Replace Text in a specific column](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Inspect AnnData: Key-indexed observations annotation (obs)** {% icon tool %}
>    - *"1. Replacement"*
>
>         - *"in column"*: `Column: 8`
>         - *"Find pattern"*: `0|3|4|5`
>         - *"Replace with"*: `wildtype`
>    - **+ Insert Replacement**
>    - *"2. Replacement"*
>
>         - *"in column"*: `Column: 8`
>         - *"Find pattern"*: `1|2|6`
>         - *"Replace with"*: `knockout`
>    - **+ Insert Replacement**
>    - *"3. Replacement"*
>
>         - *"in column"*: `Column: 8`
>         - *"Find pattern"*: `batch`
>         - *"Replace with"*: `genotype`
>
>    Now we want only the column containing the genotype information - we will ultimately add this into the cell annotation in the AnnData object.
>
> 2. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c8`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: output of **Replace text** {% icon tool %}
>
> 3. Rename {% icon galaxy-pencil %} output `Genotype_metadata`
{: .hands_on}

You might want to do this with all sorts of different metadata - which labs handled the samples, which days they were run, etc. Once you've created and cut all your metadata columns, we can paste them together before adding them into the AnnData object itself.

> <hands-on-title>Combining metadata columns</hands-on-title>
>
> 1. {% tool [Paste two files side by side](Paste1) %} with the following parameters:
>    - {% icon param-file %} *"Paste"*: `Genotype_metadata`
>    - {% icon param-file %} *"and"*: `Sex_metadata`
>    - *"Delimit by"*: `Tab`
> 2. Rename {% icon galaxy-pencil %} output `Cell_Metadata`
{: .hands_on}

Let's add this metadata to the AnnData object!

> <hands-on-title>Adding metadata to AnnData object</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Combined_object`
>    - *"Function to manipulate the object"*: `Add new annotation(s) for observations or variables`
>    - *"What to annotate?"*: `Observations (obs)``
>    - {% icon param-file %} *"Table with new annotations"*: `Cell_Metadata`
{: .hands_on}

Woohoo! We're there! You can *peek* at your new AnnData object in your history {% icon galaxy-history %} to see the additional **Obs** categories to make sure this worked. You should now find a `sex` and `genotype` in the **Obs** listing.

## Add batch metadata

I want to clean up this AnnData object just a bit more first. It would be a lot nicer if 'batch' meant something, rather than 'the order in which the Manipulate AnnData tool added my datasets'.

> <hands-on-title>Labelling batches</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: output of **Manipulate AnnData - Add new annotations** {% icon tool %}
>    - *"Function to manipulate the object"*: `Rename categories of annotation`
>    - *"Key for observations or variables annotation"*: `batch`
>    - *"Comma-separated list of new categories"*: `N701,N702,N703,N704,N705,N706,N707`
>
> 2. Rename {% icon galaxy-pencil %} output `Batched_Object`
>
{: .hands_on}

{% icon congratulations %} Well done! *Peek* at your final {% icon param-file %} `Batched_Object` to see the wealth of information that has been added. You are now ready to move along to further filtering!

# Conclusion

You might find the {% icon galaxy-history-answer %} *Answer Key Histories* helpful to check or compare with:
  - {% for h in page.answer_histories %}
      [ {{h.label}} ]( {{h.history}} )
    {% endfor %}
<!-- Only currently want to iterate through the first history, but might want others in the future (different servers!) -->

You can also run this entire tutorial via a {% icon galaxy-workflows-activity %} *Workflow*, after performing the **Get data** step initially.
 - [Tutorial Workflow]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/workflows/index.md %})

<iframe title="Galaxy Workflow Embed" style="width: 100%; height: 700px; border: none;" src="https://singlecell.usegalaxy.eu/published/workflow?id=d1a1e6070c0a36ca&embed=true&buttons=true&about=false&heading=false&minimap=true&zoom_controls=true&initialX=0&initialY=-20&zoom=0.5"></iframe>

Finally, you may remember that the datasets you analysed in this tutorial were downsampled. The idential analysis can be performed on whole samples, which you can find in the example histories below.
- {% icon warning %} Remember that *if* you are in a course, time for exploring these histories will not be factored into the schedule. Explore these outside of course time!
- {% for h in page.example_histories %}
    [ {{h.label}} ]( {{h.history}} )
  {% endfor %}

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy1", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy1", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy0", "Cut columns from a table", "Cut1", "Replace Text in a specific column", "Paste two files side by side", "Paste1"]


topic: single-cell
tutorial: scrna-case_monocle3-trajectories
title: Inferring single cell trajectories with Monocle3
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["paper-replication", "MIGHTS"]
content_hash: e5c5b7f1e3bb6e9c6c436a2d0045bf24
last_modified: 2026-05-27T09:01:49.747178
zenodo_link: https://zenodo.org/records/10397366
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.html

content:
This tutorial is a follow-up to the ['Single-cell RNA-seq: Case Study']({% link topics/single-cell/index.md %}). We will use the same sample from the previous tutorials. If you haven’t done them yet, it’s highly recommended that you go through them to get an idea how to [prepare a single cell matrix]({% link topics/single-cell/tutorials/scrna-case_alevin/tutorial.md %}), [combine datasets]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md %}) and [filter, plot and process scRNA-seq data]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) to get the data in the form we’ll be working on today.

In this tutorial we will perform trajectory analysis using [monocle3](https://cole-trapnell-lab.github.io/monocle3/). You can find out more about the theory behind trajectory analysis in our [slide deck]({% link topics/single-cell/tutorials/scrna-trajectories/slides.html %}). We have already analysed the trajectory of our sample using the ScanPy toolkit in another tutorial: [Inferring Trajectories using Scanpy]({% link topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.md %}). However, trajectory analysis is quite sensitive and some methods work better for specific datasets. In this tutorial, you will perform the same steps but using a different method for inferring trajectories. You will then compare the results, usability and outcomes! Sounds exciting, let’s dive into that!

{% snippet faqs/galaxy/tutorial_mode.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data
We will continue to work on the case study data from a mouse model of fetal growth restriction {% cite Bacon2018 %} (see [the study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and [the project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)).

In the previous tutorials, we first created an AnnData object and performed downstream analysis on that file. However, Monocle3 uses another datatype which is Cell Data Set (CDS). To be able to infer trajectories in Monocle, we need to transform our AnnData object into CDS file. And guess what - we already have a tutorial for that! We did it in [format conversion tutorial]({% link topics/single-cell/tutorials/scrna-data-ingest/tutorial.md %}), in the [Anndata -> Cell Data Set (CDS) subsection]({% link topics/single-cell/tutorials/scrna-data-ingest/tutorial.md %}#anndata---cell-data-set-cds). To better understand the structure of CDS object and learn how to create it from expression matrix, cell and gene annotations, it is highly recommended that you complete the mentioned tutorial before importing the prepared CDS file.

You have two options for uploading the dataset. Importing via history is often faster.

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. You can import [history](https://usegalaxy.eu/u/j.jakiela/h/anndata---cds-conversion-for-monocle3-tutorial) where we went from AnnData to CDS file. Then you will also have access to extracted cell metadata, gene metadata, and an expression matrix. If you want to get the CDS file alone, you can get it in this [input history](https://usegalaxy.eu/u/j.jakiela/h/cds-input-file-for-monocle3-tutorial).
>
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the the history to your name of choice.
>
{: .hands_on}

><hands-on-title>Option 2: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/records/10397366/files/CDS_input_for_Monocle3_tutorial.rdata
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}


> <details-title>Input files for Monocle3</details-title>
>
> Monocle3 turns the expression matrix, cell and gene annotations into an object called cell_data_set (CDS), which holds single-cell expression data. We provided you with the CDS file already, but it was created by combining the three mentioned elements. Check out [data conversion tutorial]({% link topics/single-cell/tutorials/scrna-data-ingest/tutorial.md %}#anndata---cell-data-set-cds) to see how to do it!
> Here is what [Monocle3 documentation](https://cole-trapnell-lab.github.io/monocle3/docs/starting/) says about the three input files required to create a CDS object:
>    - **expression_matrix**: a numeric matrix of expression values, where rows are genes, and columns are cells. Must have the same number of columns as the cell_metadata has rows and the same number of rows as the gene_metadata has rows.
>    - **cell_metadata**: a data frame, where rows are cells, and columns are cell attributes (such as cell type, culture condition, day captured, etc.)
>    - **gene_metadata**: a data frame, where rows are features (e.g. genes), and columns are gene attributes, such as biotype, gc content, etc. One of its columns should be named "gene_short_name", which represents the gene symbol or simple name (generally used for plotting) for each gene.
>
{: .details}

Once you get the CDS file in your history, let's have a closer look at this dataset to understand it a little bit better.

> <question-title></question-title>
>
> What are the dimensions of the created CDS object?
>
> > <solution-title></solution-title>
> >
> > Just click on that dataset - on the preview you’ll see that the dimensions are 15395 x 8569 - so exactly as we predicted genes x cells!
> > ![The dataset preview shows class: cell_data_set, dim: 15395 8569](../../images/scrna-casestudy-monocle/monocle_dimensions.png "Monocle Object Dimensions")
> >
> {: .solution}
{: .question}


# Monocle3 workflow

The Monocle3 workflow looks like the following, which should seem pretty similar to what you've done throughout the case study.

![Monocle workflow: scRNA-seq dataset, pre-process data (normalise, remove batch effects), non-linear dimensionality reduction (t-SNE, UMAP), cluster cells, compare clusters (identify top markers, targeted contrasts), trajectory analysis](../../images/scrna-casestudy-monocle/monocle3_new_workflow.png "Workflow provided by Monocle3 documentation")

We will follow those steps and see how it all works in practice.


# Pre-processing

In Galaxy, there are currently 2 methods of initial dimensionality reduction included in the pre-processing step: principal component analysis (PCA) and latent semantic indexing (LSI).
Given that PCA is more commonly used, and it allows us to perform further steps on the CDS object, we’ll use this method. There is one parameter here that has a great impact on how our analysis will look - the `dimensionality of the initially reduced space`. This is a highly subjective choice - you will want to test a lot of different parameters on your dataset. After much trial and error, we were able to find the value that made the most sense biologically. Have a look at the image below to see how different values affect the outcomes.

![Six graphs showing the output of depending on the number of dimensions that the space was reduced to during pre-processing. Chosen numbers: 10 (really disjoint cell groups), 50 (disjoint cell groups), 150 (two bigger groups), 200 (cell groups start to come together), 250 (cell groups well aligned into a semicircle), 300 (cell groups start to fold and become biologically irrelevant)](../../images/scrna-casestudy-monocle/num_dim_legend.jpg "Different outputs depending on the number of dimensions that the space was reduced to during pre-processing. Currently the label size in Monocle graphs in Galaxy is a nightmare, so for clarity additional legend was added to this image. We're working on the label size so that you can generate clear, readable and pretty graphs. However in this particular figure, the main point is to show the differences in shapes depending on the num-dim.")

> <question-title></question-title>
>
> Looking at the image above, which value would you choose?
>
> > <solution-title></solution-title>
> >
> > It might be hard to tell at this point without any explanation! Don’t worry, after a few more steps you’ll understand what all those colors mean and how to generate those plots. But look - we want to infer trajectories and find relationships between cells, ideally we would see the development of cells or transition from one type to another. In the graphs where num-dim is from 10 to 200, you see that the clusters are quite disjoint, while we want to see smooth transitions from one to another. I can tell you now that we’ll go ahead with the value of **250**. We're not choosing 300, because the arrangement of the cells on that graph is not really biologically relevant.
> >
> {: .solution}
{: .question}

> <hands-on-title>Pre-processing</hands-on-title>
>
> 1. {% tool [Monocle3 preprocess](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_preprocess/monocle3_preprocess/0.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 create** {% icon tool %}
>    - *"The dimensionality of the reduced space."*: `250`
 >
{: .hands_on}

#  Dimensionality reduction

Now it’s time for the proper dimensionality reduction, to turn the original thousands of dimensions (genes), into a 2-dimensional graph. There are several algorithms to do this: UMAP, tSNE, PCA and LSI (only possible when preprocess_method is set to `LSI`), but due to the same reasons as above, we’ll use UMAP (most common + allows further operations + best results). But I’ll let you see how the outputs from the other algorithms look to convince you that **UMAP** is indeed the best for this dataset. Of course, it's possible that by choosing different pre-processing values, tSNE or PCA plots would look better, so don't be afraid to play around with the parameters and test them!

![Four graphs showing the alignment of cell types depending on the algorithm of dimensionality reduction that was chosen: UMAP, tSNE, PCA, LSI. UMAP shows distinct cell groups, transitioning smoothly from one to another, creating kind of semicircle. tSNE shows distinct cell groups, however no smooth transitions are observed, all groups gathered into one big grouping. PCA shows cell groups whose boundaries are blurred between each other. On LSI graph, the cell types are all mixed together.](../../images/scrna-casestudy-monocle/dim_red_methods.png "The preview of alignment of cell types depending on the algorithm of dimensionality reduction that was chosen: UMAP, tSNE, PCA, LSI. The methods were applied to the output of the PCA-preprocessed data (except LSI method which was called on LSI-preprocessed data). LSI failed in forming distinct cell groups, PCA managed to cluster cells according to their types but tSNE did it more precisely. However, UMAP gave the best results, not only showing distinct cell type groups, but also ordering them in a way that makes sense biologically - DN to T-mat.")

> <hands-on-title>Dimensionality reduction</hands-on-title>
>
> 1. {% tool [Monocle3 reduceDim](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_reducedim/monocle3_reduceDim/0.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 preprocess** {% icon tool %}
>
{: .hands_on}

# Plotting

Alright, now let's have a look at our output! Above you got a sneak peek of how the plot would look, but now you’ll generate the plots on your own!

Thanks to the fact that we provided Monocle3 with annotated data, we can now color the cells by any attribute that was in the cell metadata file! Similarly to the previous tutorial, we’ll color them by cell type, genotype, batch and sex. At least for now...

> <hands-on-title>Plotting</hands-on-title>
>
> 1. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 reduceDim** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `cell_type`
>    - *"If set, display the cell group names directly on the plot. Otherwise include a color legend on the side of the plot."*: {% icon param-toggle %} `No`
> 2. Rename {% icon galaxy-pencil %} the output: `Cell type plot`
>
> 3. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 reduceDim** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `genotype`
>    - *"If set, display the cell group names directly on the plot. Otherwise include a color legend on the side of the plot."*: {% icon param-toggle %} `No`
> 4. Rename {% icon galaxy-pencil %} the output: `Genotype plot`
>
> 5. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 reduceDim** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `batch`
>    - *"If set, display the cell group names directly on the plot. Otherwise include a color legend on the side of the plot."*: {% icon param-toggle %} `No`
> 6. Rename {% icon galaxy-pencil %} the output: `Batch plot`
>
> 7. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 reduceDim** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `sex`
>    - *"If set, display the cell group names directly on the plot. Otherwise include a color legend on the side of the plot."*: {% icon param-toggle %} `No`
> 8. Rename {% icon galaxy-pencil %} the output: `Sex plot`
>
{: .hands_on}

The [Previous tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) discussed in detail the biological interpretation of data, so we will quickly go through similar analysis to see if the results are consistent and if we can draw any new conclusions.


As a reminder, here's the comparision between cell type annotation done in the other tutorial using Scanpy, and the output from the previous step of this Monocle tutorial. The main difference is that Scanpy was used to identify the cell types and assign them to clusters. That data was then passed on to Force-Directed + PAGA algorithms to infer trajectory, and then the arrangement of the cell groups changed a bit. In Monocle, trajectory analysis will be based on the clustering you see now.

![Each cell type forms a distinct group but they lie next to each other: DN, DP-M2, DP-M3, then DP-M4 has DP-M1 on the right and DP-L on the left, and DP-L neighbors with T-mat](../../images/scrna-casestudy-monocle/cell_types.png "Cells coloured in Monocle by their type.")

In the mentioned tutorial, we annotated the cells so that we know what type they are. Above you can see how Monocle used this information to colour cells by their type. It is important to see the 'form' of the graph and how the cell types are arranged within its confines. But why is it important? Well, in Monocle, our trajectory analysis will be based on the same arrangement of the cells that you see now. And if you now recall our choice of the number of dimensions during pre-processing, you'll understand why it was crucial - choosing the right value at the beginning determines the 'shape' of the graph that is then retained for the trajectory analysis. Therefore, the fact that on the plot above we clearly see DN cells on one side of the graph and T-mat on the other, going through DP cells, looks promising. But there is DP-M1 group that suspiciously branches out... Let's investigate that and wait until the trajectory is inferred!

> <question-title>Genotype</question-title>
> Based on our results, can we confirm findings from the previous tutorial that DP-L and mature T-cells (particularly the top half) are missing some knockout cells?
> ![Genotype data plotted on the obtained graph, compared with the cell types graph. Cell types DP-M1, DP-L and T-mat are mainly wildtypes, while for other cell types the distribution is fairly even.](../../images/scrna-casestudy-monocle/genotypes.png "Genotype differences")
>
> > <solution-title></solution-title>
> >
> > Indeed, that's what we see in our graph! But look closer, there is something more...Additionally, we also discovered that the vast majority of DP-M1 is only wildtype cells. That's interesting, isn't it?
> >
> {: .solution}
{: .question}

> <question-title>Batch effect</question-title>
> Can we confirm the previous findings that DP-L looks to be mainly comprised of N705?
> ![Batch data plotted on the obtained graph, compared with the cell types graph. DP-L and DP-M1 seem to consist mostly of N705 and N706, while other cell types have quite even batch distribution.](../../images/scrna-casestudy-monocle/batch_1.png "Checking for batch effect")
>
> > <solution-title></solution-title>
> >
> > Both DP-L and DP-M1 seem to consist mostly of N705 and N706. There might be indeed a bit of batch effect, so you could consider using batch correction on this dataset. In the absence of batch correction, we will focus on those clusters where there is batch mixing for biological interpretation. Finally, we will look at the confounding effect of sex.
> > ![Sex data plotted on the obtained graph, compared with the cell types and genotype graphs. No female cells in DP-L and DP-M1 can be observed, but they appear in other cell types.](../../images/scrna-casestudy-monocle/sex_1.png "Sex distribution across the sample")
> >  There are also no female cells in DP-L and DP-M1. The one female sample - which is one of the mere three knockout samples - seems to be distributed in the same areas as the knockout samples at large. Luckily, this doesn’t seem to be a confounding factor and we can still learn from our data.
> >
> {: .solution}
{: .question}

## Clustering

Don't get confused - we haven't clustered our cells yet, for now we have only plotted them based on cell type annotation. Now it's time to create clusters, which - in an ideal world where all computation picks up the exact biological phenomenons - would yield the same areas as the clusters determined by the Scanpy algorithms. Is this the case here? Do Monocle and Scanpy identify the same clusters?

Monocle uses a technique called "community detection" ({% cite Traag_2019 %}) to group cells. This approach was introduced by {% cite Levine_2015 %} as part of the phenoGraph algorithm.

Monocle also divides the cells into larger, more well separated groups called partitions, using a statistical test from {% cite Wolf_2019 %}, introduced as part of their [PAGA](https://github.com/theislab/paga) algorithm.

> <details-title>Clusters vs partitions</details-title>
>
> Clusters are particularly useful while trying to assign cells to a certain type, because they are based on the similarity in gene expression. The relationships between different clusters are analysed to identify possible trajectories.
>
> Partitions, meanwhile, are larger groups of cells that usually contain several clusters. Trajectory inference is performed only within one partition, so it is essential that all the cells that we want to analyse in pseudotime belong to the same partition.
>
{: .details}

> <hands-on-title>Clustering </hands-on-title>
>
> 1. {% tool [Monocle3 partition](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_partition/monocle3_partition/0.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 reduceDim** {% icon tool %}
>    - *"Resolution of clustering result, specifying the granularity of clusters. Not used by default and the standard igraph louvain clustering algorithm will be used."*: `0.00015`
>    - *"The q-value threshold used to determine the partition of cells."*: `1.0`
>    - The clusters and partitions are now stored in your CDS file. To see them, just plot the output, coloring the cells with the corresponding attributes.
>
> 2. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 partition** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `partition`
> 3. Rename {% icon galaxy-pencil %} the output: `Partition plot`
>
> 4. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 partition** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `cluster`
> 5. Rename {% icon galaxy-pencil %} the output: `Cluster plot`
>
{: .hands_on}
> <tip-title>If the partition does not contain all your cells of interest...</tip-title>
>
> Sometimes it might happen that cells are grouped into several partitions, while you want them all to be in just one in order to perform trajectory analysis on all of them. Then, you can try to increase the `q-value` threshold that is used to determine the partition of cells.
> ![Two graphs showing the difference between q-value used. For q-value=0.05 there are two partitions formed and for q-value=1.0 all cells were assigned to one partition so that there is only one in total.](../../images/scrna-casestudy-monocle/partition_qval.png "q-value threshold affecting the span of partition. Note that 0.05 is the default value.")
>
{: .tip}
> <tip-title>If the granularity of clusters does not make sense...</tip-title>
>
>
> When using standard igraph louvain clustering, the value of `resolution` parameter is by default set to `NULL`, which means that it is determined automatically. If you are not satisfied with the results of the standard igraph louvain clustering, you may set the `resolution` value manually, and thus specify the granularity of the clustering.
> ![Three graphs showing the difference between the resolution of clustering. During automatic determination of this value, there were 6 clusters formed, corresponding to cell types. Manually set resolution=0.001 resulted in 11 clusters so that more than one cluster corresponds to one cell type, and resolution=0.1 resulted in almost 200 very small clusters.](../../images/scrna-casestudy-monocle/clusters_resolution.png "Different granularity of clusters based on the resolution set automatically and manually.")
{: .tip}

> <warning-title>Ambiguous clusters!</warning-title>
> As mentioned above, standard igraph louvain clustering determines the resolution automatically, unless the specific value is provided by the user. Therefore, it sometimes returns slightly different outputs. To ensure that your clusters are reproducible, you might want to pass a certain value to the `resolution` parameter. In case of our data, the resolution value of 0.00015 gave the same results as the best output of igraph louvain clustering, and ensured reproducibility.
> ![Four graphs showing slight changes in clustering: 1) 5 clusters, 2) 5 clusters but with other cells assigned, 3) 4 clusters, 4) 6 clusters.](../../images/scrna-casestudy-monocle/igraph.png "Standard igraph louvain clustering giving different results despite the same input becasue of the automatic determination of resolution vaule.")
>
>
{: .warning}

If we compare the annotated cell types and the clusters that were just formed, we see that they nicely correspond to one another.
![When projected onto a graph, clusters quite accurately corresponding to the cell type.](../../images/scrna-casestudy-monocle/cell_type_vs_cluster.png "Comparision between annotated cell types and formed clusters.")


# Gene expression

We haven't looked at gene expression yet! This step is particularly important when working with data which is not annotated. Then, based on the expression of marker genes, you are able to identify which clusters correspond to which cell types. This is indeed what we did in the previous tutorial using scanpy. We can do the same using Monocle3! Since we work on annotated data, we can directly check if the expressed genes actually correspond to the previously assigned cell types. If they do, that’s great - if two different methods are consistent, that gives us more confidence that our results are valid.
Below is the table that we used in the previous tutorial to identify the cell types.

| Marker | Cell type |
|--------------------|
| Il2ra    | Double negative (early T-cell)    |
| Cd8b1, Cd8a, Cd4    | Double positive (middle T-cell)|
| Cd8b1, Cd8a, Cd4 - high | Double positive (late middle T-cell)|
| Itm2a    | Mature T-cell |
| Aif1    | Macrophages    |
| Hba-a1    | RBC    |

> <hands-on-title>Gene expression</hands-on-title>
>
> 1. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 partition** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `cell_type`
>    - *"A list of gene IDs/short names to plot."*: `Il2ra,Cd8b1,Cd8a,Cd4,Itm2a,Aif1,Hba-a1`
>
{: .hands_on}

![Expression on the genes: Il2ra expressed in DN, Cd8b1, Cd8a expressed in the areas where middle DP, Cd4 mostly expressed in late DP cluster, Itm2a expressed in mature T-cells, Aif1 – no expression, Hba-a1 - very high expression in small branch of DP-M4.](../../images/scrna-casestudy-monocle/gene_expression.png "Expression of the genes across analysed sample")

> <question-title>Genes and cell types</question-title>
> Based on the gene expression graph that we just generated, the table above and your knowledge from the previous tutorial, how would you interpret the results?
>
> > <solution-title></solution-title>
> >
> > - `Il2ra`: expressed in the cluster where DN cells are - an indication of where the trajectory should start
> > - `Cd8b1, Cd8a`: expressed in the areas where middle DP were assigned - great
> > - high `Cd4`: mostly in late DP cluster - as expected
> > - `Itm2a`: expressed in mature T-cells - tells us where the trajectory should end
> > - `Aif1`: nothing here - correct! We filtered out macrophages from the sample
> > - `Hba-a1`: look at this! Very high expression in just one, tiny bit, which hasn't even been grouped into a cluster! So wait, why do we see hemoglobin gene here? Do we have red blood cells in our sample?! Let's investigate that further...
> >
> {: .solution}
{: .question}

> <tip-title>Purity of the sample - Hba-a1 gene</tip-title>
>
> The Hba-a1 gene creates hemoglobin which is found in red blood cells. This is highly expressed in a tiny bit of the middle DP cluster. Interestingly, it forms a a clearly visible, distinct, little branch. Hemoglobin should NOT be found in T-cells. However, if you remember, the gene was found to be expressed in the previous Scanpy tutorial (see the image below). That marker appeared throughout the entire sample in low numbers, suggesting some background contamination of red blood cell debris in the cell samples during library generation. Unlike Scanpy, Monocle algorithms allowed us to gather the cells expressing that gene into a distinct group! That's great!
>
> ![In Monocle graph, the cells expressing hemoglobin gene were grouped into a small branch of DP-M4, while in Scanpy the marker appears throughout the entire sample in low numbers, not allowing to group those cells.](../../images/scrna-casestudy-monocle/hb.png "Hemoglobin across clusters - comparision between Monocle and Scanpy")
>
{: .tip}

# Top marker genes

Here we used a priori knowledge regarding the marker genes. If we wanted to approach this problem in an unsupervised manner, we could use Monocle to tell us the top marker genes in each group of cells. This is very useful if we are trying to identify a cell type, or if we want to find novel marker genes for known cell types.

> <question-title></question-title>
>
> If I cluster cells that are not annotated, can I assign clusters to a cell type based on gene expression using Monocle3?
>
> > <solution-title></solution-title>
> >
> > Theoretically...yes! That’s the point of the clustering and gene expression analysis. However, as of the writing of this tutorial, this function hasn’t been turned into a Galaxy tool yet and is only available in R.
> >
> {: .solution}
>
{: .question}

> <hands-on-title>Top marker genes</hands-on-title>
>
> 1. {% tool [Monocle3 top markers](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_topmarkers/monocle3_topmarkers/0.1.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input Object"*: output of **Monocle3 partition** {% icon tool %}
>    - *"Group cell by"*: `cell_type`
> 2. Rename {% icon galaxy-pencil %} the tabular output: `Top markers table`
> 3. Rename {% icon galaxy-pencil %} the pdf output: `Top markers plot`
>
{: .hands_on}

![A table showing top markers - which genes (and their percentage) are the most expressed in each cell type.](../../images/scrna-casestudy-monocle/top_markers.jpg "Identification of the genes most specifically expressed in groups of cells.")

> <question-title></question-title>
>
> What genes are uniquely expressed in DP-M1?
>
> > <solution-title></solution-title>
> >
> > By looking at the table, you might give the 5 top gene IDs expressed in DP-M1. To save you some time and make the analysis more readable, we converted the gene IDs to gene names and they are as follows: Rps17, Rpl41, Rps26, Rps29, Rps28. They are all ribosomal! [You can do this yourself if you want by following this section of a previous tutorial that [uses the gene names in one object to add to a table of Ensembl IDs]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}#find-gene-markers). These ribosomal differences might be due to housekeeping background, cell cycling, or even something more bioligically interesting...or all three!
> > The plot also indicates other specifically expressed genes, such as Hmgb2, Pclaf, Rpl13, Rps19, Ybx1, Ncl, Hsp90ab1, Npm1.
> >
> > Whenever you want to explore what might be the function of a particular cluster or why it branches out from the trajectory, check the top markers for that cluster to draw biological conclusions. Thank you Maths!
> {: .solution}
>
{: .question}

But what if you want to know how gene expression changes across a trajectory? This is where Monocle is particularly powerful. But in order to do that, we have to infer that trajectory first!

# Learn the trajectory graph

We’re getting closer and closer! The next step is to learn the trajectory graph, which means to fit a principal graph within each partition. In that way, we’ll ‘connect’ the existing clusters by creating a path between them.

> <hands-on-title>Learn graph</hands-on-title>
>
> 1. {% tool [Monocle3 learnGraph](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_learngraph/monocle3_learnGraph/0.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 partition** {% icon tool %}
>    - Again, the graph is now stored in your CDS file. To see it, just plot the output, you can color the cells by any attribute that you want. We'll use cell types to see how they are connected.
> 2. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 learnGraph** {% icon tool %}
>    - *"The cell attribute (e.g. the column of pData(cds)) to map to each cell's color, or one of {cluster, partition, pseudotime}."*: `cell_type`
>
{: .hands_on}

As you can see, the learned trajectory path is just a line connecting the clusters. However, there are some important points to understand here.
If the resolution of the clusters is high, then the trajectory path will be very meticulous, strongly branched and curved. There's a danger here that we might start seeing things that don't really exist.
You can set an option to learn a single tree structure for all the partitions or use the partitions calculated when clustering and identify disjoint graphs in each. To make the right decision, you have to understand how/if the partitions are related and what would make more biolgical sense. In our case, we were only interested in a big partition containing all the cells and we ignored the small 'dot' classified as another partition.
There are many trajectory patterns: linear, cycle, bifurcation, tree and so on. Those patterns might correspond to various biological processes: transition events for different phases, cell cycle, cell differentiation. Therefore, branching points are quite important on the trajectory path. You can always plot them, {% icon param-toggle %} checking the correct box in {% tool Monocle3 plotCells %}.


![A trajectory path, branching out to connect all the clusters and thus show their relationships.](../../images/scrna-casestudy-monocle/learned_trajectory.png "Learned trajectory path")

> <tip-title>Comparing the trajectories</tip-title>
>
> As a reminder, here's the comparision between our trajectory and the one from the previous tutorial, where we used Scanpy for clustering, and then appplied Force-Directed + PAGA algorithms to infer trajectory. As you remember from those tutorials, the arrangement of the cell groups changed a bit during these steps. Indeed - by using the mentioned methods, we get different graphs for clusters and trajectory, while in Monocle the general 'shape' of the graph stays the same from the beginning. 'Leranig the trajectory' step in Monocle is about finding a path, along which the cells can be then ordered in pseudotime.
> ![First graph shows the trajectory inferred by using Scanpy + Force-Directed + PAGA (DN connecting equally with DP-M2 and DP-M3, then going to DP-M1, and branching out to DP-M4, then going down to DP-L and finally turning into T-mat).  Second graph shows the trajectory inferred by using Monocle (DN connected to DP-M2 and DP-M3, then DP-M4 has DP-M1 on the right and DP-L on the left, and DP-L comes to T-mat).](../../images/scrna-casestudy-monocle/scanpy_monocle_trajectories.png "Comparison between the trajectory inferred in the previous 'case study' tutorials (Scanpy + Force-Directed + PAGA) and the trajectory obtained in Monocle.")
{: .tip}

# Pseudotime analysis

Finally, it's time to see our cells in pseudotime! We have already learned a trajectory, now we only have to order the cells along it. Monocle3 requires information on where to start ordering the cells, so we need to provide it with this information. We annotated early T-cells as double negative (DN), so those will be our root cells!

> <details-title>Pseudotime</details-title>
>
> To infer trajectories, we need data from cells at different points along a path of differentiation. The assumption is that in any given sample, some cells are further along a trajectory than others. This inferred temporal dimension is known as pseudotime. Pseudotime measures the cells’ progress through the transition.  See {% cite Reid_2016 %} for more.
>
{: .details}

> <hands-on-title>Ordering the cells along trajectory</hands-on-title>
>
> 1. {% tool [Monocle3 orderCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_ordercells/monocle3_orderCells/0.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 learnGraph** {% icon tool %}
>    - *"The cell phenotype (column in pdata) used to identify root principal nodes."*: `cell_type`
>    - *"The value in the cell phenotype column used to extract root nodes."*: `DN`
>    - Alright - we were waiting for this plot the whole tutorial: once we have the cells ordered, we can finally color them by pseudotime!
>
> 2. {% tool [Monocle3 plotCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 orderCells** {% icon tool %}
> 3. Rename {% icon galaxy-pencil %} the output: `Pseudotime plot`
>
{: .hands_on}

> <tip-title>Other ways to specify the root cells</tip-title>
>
> The method to specify the root cells shown above is not the only one available in Galaxy! However, it is probably the most intuitive one.
> 1. **Annotated cell type as root cells**
>    -  fill *(--cell-phenotype)* with the colname (heading) where the cell types are stored
>    -  fill *(--root-type)* with the name of the cell type that you want to start ordering from
> 2. **Cell ID as root cell**
>    -  fill *(--root-cells)* with the cell ID that you want to start ordering from
> {% comment %}
>   3. **Starting principal points**
>     - repeat the plotting step, find the parameter *label_principal_points* and set its value to {% icon param-toggle %} `Yes`
>     - have a look at the plot and note which principal point best corresponds to the root cells
>     - fill *(--root-pr-nodes)* with the noted value from *label_principal_points*
> {% endcomment %}
{: .tip}

Now we can see how all our hard work has come together to give a final pseudotime trajectory analysis. DN cells gently switching to DP-M which change into DP-L to finally become mature T-cells. Isn't it beautiful? But wait, don't be too enthusiastic - why on earth DP-M1 group branches out? We didn't expect that... What could that mean?

There are a lot of such questions in bioinformatics, and we're always get excited to try to answer them. However, with analysing scRNA-seq data, it's almost like you need to know about 75% of your data to make sure that your analysis is reasonable, before you can identify the 25% new information. Additionally, pseudotime analysis crucially depends on choosing the right analysis and parameter values, as we showed for example with initial dimensionality reduction during pre-processing. The outputs here, at least in our hands, are more sensitive to parameter choice than standard clustering analysis with Scanpy.

![Pseudotime plot, showing the development of T-cells – starting in dark blue on DN cells and ending up on mature T-cells, marked in yellow on pseudotime scale and (going in the opposite direction) DP-M1 branch which is marked in light orange.](../../images/scrna-casestudy-monocle/pseudotime.png "Trajectory analysis - pseudotime")

Last but not least, you can now identify genes that define the inferred trajectories.

> <hands-on-title>Differentially expressed genes</hands-on-title>
>
> 1. {% tool [Monocle3 diffExp](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_diffexp/monocle3_diffExp/0.1.4+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input object in RDS format"*: output of **Monocle3 orderCells** {% icon tool %}
> 2. Rename {% icon galaxy-pencil %} the output: `Differential gene expression table`
>
{: .hands_on}

# Conclusion

{% icon congratulations %} Well done, you’ve made it to the end! You might want to consult your results with this [control history](https://humancellatlas.usegalaxy.eu/u/j.jakiela/h/monoce3-tutorial-workflow) (which also includes AnnData to CDS conversion), and check out the [workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/copy-of-trajectory-analysis-using-monocle3-1) for this tutorial. There is also a separate workflow for [preparing the input files for Monocle3, starting from AnnData](https://humancellatlas.usegalaxy.eu/u/j.jakiela/w/copy-of-trajectory-analysis-using-monocle3), and [full analysis workflow](https://humancellatlas.usegalaxy.eu/u/j.jakiela/w/copy-of-anndata-to-monocle-right-1). You can use them to accelerate analysis of your own data, paying attention to the requirements of the input data that are mentioned in this tutorial.

![A scheme connecting all the tools used in this tutorial.](../../images/scrna-casestudy-monocle/workflow.jpg "Full workflow for this tutorial, including AnnData to CDS format conversion.")

If you're following the Case Study tutorials from the beginning, you have already experienced what it’s like to analyse and question a dataset, potentially without clear cut-offs or clear answers. You now know that trajectory analysis is even more sensitive to parameter values, so it's often trying to find the best set of values that would give the most reasonable results and go in accordance with biology. Moreover, not all trajectory analysis methods are designed to infer all kinds of biological processes - due to the fact that they use different algorithms, some would work better for analysing your sample. Since Monocle is quite widely used for trajectory analysis, it might be a good practice to compare its results with other methods. The more evidence you have to confirm your findings, the more confident you can be about their reliability!

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Monocle3 preprocess", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_preprocess/monocle3_preprocess/0.1.4+galaxy0", "Monocle3 reduceDim", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_reducedim/monocle3_reduceDim/0.1.4+galaxy0", "Monocle3 plotCells", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_plotcells/monocle3_plotCells/0.1.5+galaxy1", "Monocle3 partition", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_partition/monocle3_partition/0.1.4+galaxy0", "Monocle3 top markers", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_topmarkers/monocle3_topmarkers/0.1.5+galaxy0", "Monocle3 learnGraph", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_learngraph/monocle3_learnGraph/0.1.4+galaxy0", "Monocle3 orderCells", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_ordercells/monocle3_orderCells/0.1.4+galaxy0", "Monocle3 diffExp", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_diffexp/monocle3_diffExp/0.1.4+galaxy1"]


topic: single-cell
tutorial: bulk-music-2-preparescref
title: Matrix Exchange Format to ESet | Creating a single-cell RNA-seq reference dataset for deconvolution
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["data management"]
content_hash: 826f4cb639a928d14ccf55f7be5c78ca
last_modified: 2026-05-15T22:28:16.657456
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music-2-preparescref/tutorial.html

content:
After completing the [MuSiC](https://xuranw.github.io/MuSiC/articles/MuSiC.html) {% cite wang2019bulk %} deconvolution tutorial, you are hopefully excited to apply this analysis to data of your choice. Annoyingly, getting data in the right format is often what prevents us from being able to successfully apply analyses. This tutorial is all about reformatting a raw scRNA-seq dataset pulled from a public resource (the EMBL-EBI single cell expression atlas {% cite Moreno2021 %}. Let's get started!


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Metadata Manipulation

First, we will tackle the metadata. We are roughly following the same concept as in the previous bulk deconvolution tutorial, by comparing human pancreas data across a disease variable (type II diabetes vs healthy), but using public datasets to do it.

## Find the data
We explored the [single cell expression atlas](https://www.ebi.ac.uk/gxa/sc/experiments), browsing experiments in order to find a pancreas dataset ({% cite segerstolpe2016single %}). You can [explore this dataset](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-5061/results/tsne) using their browser. These cells come from 6 healthy individuals and 4 individuals with Type II diabetes, so we will create reference Expression Set objects for the total as well as separating out by phenotype, as you may have reason to do this in your analysis (or you may not!).

{% snippet faqs/galaxy/tutorial_mode.md %}

Galaxy has a specific tool for ingesting data from the Single cell expression atlas, so there are no uploads for this tutorial.

> <hands-on-title>Data retrieval</hands-on-title>
>
> 1. {% tool [EBI SCXA Data Retrieval](toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2) %} with the following parameters:
>    - *"SC-Atlas experiment accession"*: `E-MTAB-5061`
>
> Data management is going to be key in this analysis, so trust me now to start adding tags.
> 5. Add to the **EBI SCXA Data Retrieval on E-MTAB-5061 exp_design.tsv** file the following tags: `#ebi #metadata #singlecell`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
{: .hands_on}

This tool will retrieve four files: a barcodes list, a genes list, an experimental design file, and a matrix market format (where columns refer to genes, cells, and quantities). We (mostly) only need the experimental design file, but keep in mind this will have data on all the cells reported by the authors.

> <question-title></question-title>
>
> 1. How many cells are in the sample?
> 2. How many cells were submitted by the authors?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > 1. If you select the {% icon param-file %} **barcodes.tsv** file, you'll find that it contains 2914 lines - this corresponds to the 2914 cells, because each cell is given a barcode.
> > 2. The nature of public repositories is that they ingest data from many places, which means they usually apply a uniform analysis to samples. This rarely means they yield the same cell numbers as the original authors. If you check the {% icon param-file %} **exp_design.tsv** file, which refers to the data submitted by the authors, you'll find it contains 3514 lines - referring to 3514 cells submitted by authors. It's important to know that (currently) these files differ.
> >
> {: .solution}
{: .question}

## Prepare the experimental design file

Let's get rid of a bunch of repetitive columns in the metadata we don't need. You can find out what each column is by inspecting the dataset {% icon galaxy-eye %} in the history window.

> <hands-on-title>Cutting necessary metadata columns</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c4,c6,c8,c10,c14,c20,c24,c26,c30,c32,c34`
>    - {% icon param-file %} *"From"*: `design_tsv` (output of **EBI SCXA Data Retrieval** {% icon tool %})
>
{: .hands_on}

You can inspect the dataset {% icon galaxy-eye %} to see that it's full of annoying "" everywhere, and overly long descriptions of each columns.

![Columns in Galaxy history window contain "Assay" and "" around every word or ID](../../images/bulk-music/annoying_ebimetadata.png "Annoying metadata")

Now, there might be a better way to do this in Galaxy (or you might consider downloading the file locally and changing it in a spreadsheet application or something), but this is what will work to reformat all that annoying text.

> <hands-on-title>Reformatting the metadata</hands-on-title>
>
> 1. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `out_file1` (output of **Cut** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[individual\]"`
>            - *"Replacement"*: `Individual`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[sex\]"`
>            - *"Replacement"*: `Sex`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[age\]"`
>            - *"Replacement"*: `Age`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[body mass index\]"`
>            - *"Replacement"*: `BMI`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `kilogram per square meter`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `HbA1c `
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[clinical information\]"`
>            - *"Replacement"*: `HbA1c`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `%`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[disease\]"`
>            - *"Replacement"*: `Disease`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[single cell quality\]"`
>            - *"Replacement"*: `Single cell quality`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Sample Characteristic\[submitted single cell quality\]"`
>            - *"Replacement"*: `"Submitted single cell quality"`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Factor Value\[inferred cell type - ontology labels\]"`
>            - *"Replacement"*: `Inferred cell type - ontology label`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"Factor Value\[inferred cell type - authors labels\]"`
>            - *"Replacement"*: `Inferred cell type - author labels`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `""`
>            - *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `"`
>            - *"Replacement"*:
>
>    > <comment-title></comment-title>
>    >
>    > What's with the `\` everywhere? That's because the `[]` symbols usually call the code to do something, rather than just read it as a normal character. the `\` prevents this.
>    {: .comment}
>
> 2. Change the datatype to tabular.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

Great, this file is now ready to go! But, it contains all those extra cells that didn't pass filtration with the EBI pipeline and therefore won't exist in the matrix. We need to remove them for future steps to work. We can use our barcodes list to remove the extra cells.

## Prepare the barcodes file

> <hands-on-title>Adding a header</hands-on-title>
>
> 1. {% tool [Add line to file](toolshed.g2.bx.psu.edu/repos/bgruening/add_line_to_file/add_line_to_file/0.1.0) %} with the following parameters:
>    - *"text to add"*: `Cell`
>    - {% icon param-file %} *"input file"*: `barcode_tsv` (output of **EBI SCXA Data Retrieval** {% icon tool %})
>
> 2. Change the datatype to tabular.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
>    > <comment-title></comment-title>
>    >
>    > This is an annoying step we have to do to get the right format, otherwise future steps won't work.
>    {: .comment}
>
{: .hands_on}

## Use the barcodes list to filter out cells in the experimental design file

> <hands-on-title>Joining datasets</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `outfile` (output of **Add line to file** {% icon tool %})
>    - *"using column"*: `c1`
>    - {% icon param-file %} *"with"*: `out_file1` (output of **Regex Find And Replace** {% icon tool %})
>    - *"and column"*: `c1`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `Yes`
>    > <comment-title></comment-title>
>    >
>    > Make sure that you join the files in the same order as above - put the output of Add line to file in first - otherwise your columns will be in a different order for the next step. Everything will still work, but you would need to change the number of the column you remove using Advanced Cut.
>    {: .comment}
{: .hands_on}

> <question-title></question-title>
>
> 1. How many cells are now in your table?
> 2. Is your table ready to go?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > 1. If you select the output dataset in your history, you will find `2915` lines, corresponding to 2914 cells and a header. Success!
> > 2. Not quite - notice how you have two identical columns `Cell` and `Assay`? Let's get rid of one.
> >
> {: .solution}
{: .question}

> <hands-on-title>Remove duplicate columns</hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `out_file1` (output of **Join two Datasets** {% icon tool %})
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"List of Fields"*: `c1`
>        
>    > <comment-title></comment-title>
>    >
>    > Advanced cut works slightly differently in a workflow versus running the tool independently. Independently, there is a list and you can click through the list to note your columns, while in a workflow it appears as a text option and you put each column on a different line. The point is, each number above represents a column, so remove them!
>    {: .comment}
{: .hands_on}

Fantastic! You've completed part 1 - making the single cell metadata file. It should now look like this:

![Columns in the history window of a dataset contain words without any extra symbols or ""](../../images/bulk-music/corrected_ebimetadata.png "Pretty scRNA metadata")

You can use the [workflow for this portion of the tutorial](https://usegalaxy.eu/u/wendi.bacon.training/w/music-deconvolution-data-generation--sc--metadata), and access an [example history](https://usegalaxy.eu/u/wendi.bacon.training/h/music-deconvolution-data-generation--sc--metadata).


# Manipulate the expression matrix

Currently, the matrix data is in a 3-column format common in 10x outputs, where you need the barcodes and the genes files to interpret the matrix. What you actually need is an expression matrix with cells on one axis and genes on another. While we aren't running a Scanpy analysis, we can still use our Scanpy tools to get this format.

## Reformat the matrix

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Scanpy Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.8.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `matrix_mtx` (output of **EBI SCXA Data Retrieval** {% icon tool %})
>    - {% icon param-file %} *"Gene table"*: `genes_tsv` (output of **EBI SCXA Data Retrieval** {% icon tool %})
>    - {% icon param-file %} *"Barcode/cell table"*: `barcode_tsv` (output of **EBI SCXA Data Retrieval** {% icon tool %})
>    - *"Format of output object"*: `AnnData format (h5 for older versions)`
>
> 2. Change the datatype to `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}

Now your precious matrix is stored in the 10x AnnData object. Let's retrieve it!

> <hands-on-title>Inspect the matrix</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `output_h5` (output of **Scanpy Read10x** {% icon tool %})
>    - *"What to inspect?"*: `The full data matrix`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Which are currently the rows in your matrix, cells or genes?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > 1. You may remember from earlier that the sample should have `2914 cells` in it. If you inspect the dataset in your history, you will find that it contains `2915` lines (1 for the header), which means that rows correspond to cells. Unfortunately... that's not what you need.
> >
> {: .solution}
{: .question}

> <hands-on-title>Transpose the matrix</hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `X` (output of **Inspect AnnData** {% icon tool %})
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many genes are in your sample?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > 1. You should have `30,416` lines in it, meaning your sample has `30,415` genes.
> >
> {: .solution}
{: .question}

## Collapse EnsemblIDs

Ok, real talk here. Technically, the best way of analysing anything is by using the EnsemblIDs for any given RNA transcript, because they are more specific than gene names and also cover more of the transcriptome than our gene names...
But...
As biologists, it's very difficult to interpret ENSIDs. And it's an awful shame to get to the end of the MuSiC deconvolution and have all our plots show sad ENS IDs. So, courtesy of the excellent @mtekman, we steal his workflow to collapse the ENS IDs into gene names.

![First table shows 4 rows with different ENS IDs and a second column with geney symbols and some overlap. Arrow pointing to second table shows 3 rows having collapsed the overlap.](../../images/bulk-music/ensid_collapse.png "Collapsing ENS IDs")

> <hands-on-title>Convert from Ensembl to GeneSymbol using workflow</hands-on-title>
>
> 1. Import this [workflow](https://usegalaxy.eu/u/wendi.bacon.training/w/convert-from-ensembl-to-genesymbol-summing-duplicate-genes).
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Run the workflow on your sample with the following parameters:
>    - *"Organism"*: `Human`
>    - {% icon param-file %} *"Expression Matrix (Gene Rows)"*: `output_h5` (output of **Transpose** {% icon tool %})
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

The output will likely be called **Text transformation** and will look like this:

![Alphabetised gene symbols appear in column one with decimal pointed integers in the following columns corresponding to cells](../../images/bulk-music/ensid_output.png "Output of the ENS ID collapsing workflow")

# Construct Expression Set Objects

We're nearly there! We have three more tasks to do: first, we need to create the expression set object with all the phenotypes combined. Then, we also want to create two separate objects - one for healthy and one for diseased as references.

> <hands-on-title>Creating the combined object</hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `out_file` #matrix (output of **Text transformation** {% icon tool %})
>    - {% icon param-file %} *"Phenotype Data"*: `output` (output of **Advanced Cut** {% icon tool %})
>
> 2. Remove the `#metadata #matrix` tags from the output **RData ESet Object**
>
> 3. Add the tag `#combined` to the output **RData ESet Object**
>
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many genes are in your sample now?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > 1. If you select the {% icon galaxy-eye %} of the output **General Info** dataset in the history, you will find it contains 21671 features and 2914 samples, or rather, `21671` genes and `2914` cells. That's a huge reduction in genes thanks to the ENS ID collapsing!
> >
> {: .solution}
{: .question}

> <hands-on-title>Creating the disease-only object</hands-on-title>
>
> 1. {% tool [Manipulate Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_manipulate_eset/music_manipulate_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Expression Set Dataset"*: `out_rds` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Concatenate other Expression Set objects?"*: `No`
>    - *"Subset the dataset?"*: `Yes`
>        - *"By"*: `Filter Samples and Genes by Phenotype Values`
>            - In *"Filter Samples by Condition"*:
>                - {% icon param-repeat %} *"Insert Filter Samples by Condition"*
>                    - *"Name of phenotype column"*: `Disease`
>                    - *"List of values in this column to filter for, comma-delimited"*: `type II diabetes mellitus`
>
> 2. Remove the `#combined` tag from the output **RData ESet Object**
>
> 3. Add the tag `#T2D` to the output **RData ESet Object**
>
>
>
{: .hands_on}

You can either re-run this tool or set it up again to create the healthy-only object.

> <hands-on-title>Creating the healthy-only object</hands-on-title>
>
> 1. {% tool [Manipulate Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_manipulate_eset/music_manipulate_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Expression Set Dataset"*: `out_rds` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Concatenate other Expression Set objects?"*: `No`
>    - *"Subset the dataset?"*: `Yes`
>        - *"By"*: `Filter Samples and Genes by Phenotype Values`
>            - In *"Filter Samples by Condition"*:
>                - {% icon param-repeat %} *"Insert Filter Samples by Condition"*
>                    - *"Name of phenotype column"*: `Disease`
>                    - *"List of values in this column to filter for, comma-delimited"*: `normal`
>
> 2. Remove the `#combined` tag from the output **RData ESet Object**
>
> 3. Add the tag `#healthy` to the output **RData ESet Object**
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are you making a healthy-only and diseased-only reference objects?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > 1. We could imagine that the cells will express different transcript levels, but that the deconvolution tools will have to take some sort of average. Perhaps it might be more accurate to infer like from like, i.e. healthy from healthy? Or perhaps that is skewing the data through a more 'supervised' approach. We're not   quite sure, and it likely depends on the biology, so we're covering all our bases by making sure you can do this every way. (We've tested it on our dataset in all the ways and got the same results, so it doesn't make much of a difference as far as we can tell!)
> >
> {: .solution}
{: .question}

# Conclusion

You have successfully performed, essentially, three workflows. You can find the [workflows for generating the ESet object](https://usegalaxy.eu/u/wendi.bacon.training/w/music-deconvolution-data-generation--sc--matrix--eset) and the [answer key history for this entire tutorial](https://usegalaxy.eu/u/wendi.bacon.training/h/music-deconvolution-data-generation--sc--matrix--eset).

![6 boxes in the workflow editor](../../images/bulk-music/scref-metadata.png "Workflow: Manipulating metadata")

![9 boxes including a subworkflow in the workflow editor](../../images/bulk-music/scref-eset.png "Workflow: Creating the ESet single-cell object")

![8 boxes in the workflow editor](../../images/bulk-music/ens-id-collapse.png "Subworkflow: Collapsing the Ensembl IDs into gene names")

With these workflows, you've created three Expression Set objects, capable of running in the MuSiC Compare tutorial. Now you just need the bulk RNA-seq Expression Set objects!

This tutorial is part of the [https://singlecell.usegalaxy.eu](https://singlecell.usegalaxy.eu) portal ({% cite tekman2020single %}).

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["EBI SCXA Data Retrieval", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/retrieve_scxa/retrieve_scxa/v0.0.2+galaxy2", "Cut", "Cut1", "Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.2", "Add line to file", "toolshed.g2.bx.psu.edu/repos/bgruening/add_line_to_file/add_line_to_file/0.1.0", "Join two Datasets", "join1", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "Scanpy Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_read_10x/scanpy_read_10x/1.8.1+galaxy0", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.7.5+galaxy1", "Transpose", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0+galaxy2", "Construct Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4", "Manipulate Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_manipulate_eset/music_manipulate_eset/0.1.1+galaxy4"]


topic: single-cell
tutorial: scrna-case_JUPYTER-trajectories
title: Inferring single cell trajectories with Scanpy (Python)
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["paper-replication", "MIGHTS"]
content_hash: 97f42da3e1b1fca5dfc55593c36cf92c
last_modified: 2026-05-27T09:01:49.744504
zenodo_link: https://zenodo.org/record/7075718
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.html

content:
# Run the tutorial!

From now on, you can view this tutorial in the Jupyter notebook, which will allow you to read the material and simultaneously execute the code cells! You may have to change certain numbers in the code blocks, so do read carefully. The tutorial is adapted from the [Scanpy Trajectory inference tutorial](https://scanpy.readthedocs.io/en/stable/tutorials/trajectories/paga-paul15.html).

{% snippet topics/single-cell/faqs/notebook_warning.md %}

## Install modules & activate them

```python
pip install scanpy
```
```python
pip install igraph
```
```python
pip install louvain
```
```python
pip install fa2-modified
```
```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as pl
from matplotlib import rcParams
import scanpy as sc
```


## Import dataset

You can now import files from your Galaxy history directly using the following code. This will depend on what number in your history the final annotated object is. If your object is dataset #2 in your history, then you import it as following:

```python
thymusobject = get(2)
```

You now you need to read it in as an h5ad object.

```python
adata = sc.read_h5ad(thymusobject)
```

> <details-title>Any problems with *get()* function?</details-title>
> There shouldn't be any issues with fetching data from your Galaxy history using *get()* function. However, if you experience any problems, you can use the code below to download the input data and be able to follow the tutorial.
> ```python
> %%bash
> wget -nv https://zenodo.org/records/13743145/files/Filtered_anndata.h5ad
> ```
> ```python
> adata = sc.read_h5ad("Filtered_anndata.h5ad")
> ```
{: .details}


## Draw force-directed graph

First, we will calculate a [force-directed graph](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.draw_graph.html), as an alternate to tSNE, which will likely work better for trajectory analysis.

```python
sc.tl.draw_graph(adata)
```

And now time to plot it!
*Note: We're saving to **png**, but you can also choose pdf*


```python
sc.pl.draw_graph(adata, color='cell_type', legend_loc='on data', save = 'Plot1.png')
```

![Plot1-Force-Directed Graph](../../images/scrna-casestudy/draw_graph_faPlot1.png "Plot1-Force-Directed Graph")

Well now this is exciting! Our DP-late is more clearly separating, and we might also suppose that DP-M1, DP-M2, and DP-M3 are actually earlier on in the differentiation towards mature T-cells. And we're only just getting started!


## Diffusion maps

We'll now perform an *optional step*, that basically takes the place of the PCA. Instead of using PCs, we can use [diffusion maps](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.diffmap.html).


```python
sc.tl.diffmap(adata)
```

Now that we have our diffusion map, we need to re-calculate neighbors using the diffusion map instead of the PCs. Then we re-draw and plot a new force directed graph using the new neighbors.

```python
sc.pp.neighbors(adata, n_neighbors=15, use_rep='X_diffmap')
sc.tl.draw_graph(adata)
sc.pl.draw_graph(adata, color='cell_type', legend_loc='on data', save = 'Plot2.png')
```
![Diffusion Map](../../images/scrna-casestudy/draw_graph_faPlot2.png "Diffusion Map")


Oh dear! This doesn't look great. Maybe the DP-M4 cells are a whole other trajectory? That doesn't seem right. Saying that, this spreads out our T-mature cells, which makes a lot more sense when it comes to T-cell biology (we expect T-cells to differentiate into two types of T-cells, Cd8+Cd4- and Cd4+Cd8-). If you wanted to, you could also re-cluster your cells (since you've changed the neighborhood graph on which the clusterisation depends). You could use this:
`sc.tl.louvain(adata, resolution=0.6)`
However, we tried that, and it called far too many clusters given the depth of sequencing in this dataset. Let's stick with our known cell types and move from there.


## Working in a group? Decision-time!
If you are working in a group, you can now divide up a decision here with one *control* and the rest can vary numbers so that you can compare results throughout the tutorials.
- Control
   - Go straight to the PAGA section
- Everyone else:
   - you could re-call clusters `sc.tl.louvain(adata, resolution=0.6)` or use other resolutions! (Tip, go low!)
        - Please note that in this case, you will want to change the PAGA step `sc.pl.paga` to group by `louvain` rather than `cell_type`. You can certainly still plot both, we only didn't because with using our old Louvain calls, the *cell_type* and *louvain* categories are identical.
   - you could undo the diffusion map step by running the following

        `sc.pp.neighbors(adata, n_neighbors=15, use_rep='X_pca')`

        `sc.tl.draw_graph(adata)`
   - you could also change the number of neighbors used in the `pp.neighbors` step (this is the same as the Galaxy tool **Scanpy ComputeGraph**)

- Everyone else: You will want to compare FREQUENTLY with your control team member.

## PAGA

[PAGA](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.paga.html) is used to generalise relationships between groups, or likely clusters, in this case.


```python
sc.tl.paga(adata, groups='cell_type')
```

Now we want to plot our PAGA, but we might also be interested in colouring our plot by genes as well. In this case, remembering that we are dutifully counting our genes by their EnsemblIDs rather than Symbols (which do not exist for all EnsemblIDs), we have to look up our gene of interest (CD4, CD8a) and plot the corresponding IDs.


```python
sc.pl.paga(adata, color=['cell_type', 'ENSMUSG00000023274', 'ENSMUSG00000053977'], title=['Cell type', 'CD4', 'Cd8a'], save = 'Plot4.png')
```

![PAGA](../../images/scrna-casestudy/pagaPlot4.png "PAGA")


Well now that is interesting! This analysis would find that DP-M1 and DP-M4 are both driving towards differentiation, which is not something we had necessarily been able to specify before by just looking at our cluster graphs or applying our biological knowledge.

## Re-draw force-directed graph

Force directed graphs can be initialised randomly, or we can prod it in the right direction. We'll prod it with our PAGA calculations. Note that you could also try prodding it with tSNE or UMAP. A lot of these tools can be used on top of each other or with each other in different ways, this tutorial is just one example. Similarly, you could be using any **obs** information for grouping, so could do this for *louvain* or *cell_type* for instance.

```python
sc.tl.draw_graph(adata, init_pos='paga')

sc.pl.draw_graph(adata, color=['cell_type'], title=['Cluster'], legend_loc='on data', save = 'Plot5.png')
sc.pl.draw_graph(adata, color=['genotype'], title=['Genotype'], save = 'Plot6.png')
sc.pl.draw_graph(adata, color=['ENSMUSG00000023274', 'ENSMUSG00000053977'], title=['CD4', 'Cd8a'], save = 'Plot7.png')
```

![Force-Directed + PAGA - Cell type](../../images/scrna-casestudy/draw_graph_faPlot5.png "Force-Directed + PAGA - Cell type")

![Force-Directed + PAGA - Genotype](../../images/scrna-casestudy/draw_graph_faPlot6.png "Force-Directed + PAGA - Genotype")

![Force-Directed + PAGA - Markers](../../images/scrna-casestudy/draw_graph_faPlot7.png "Force-Directed + PAGA - Markers")

<!---
**Note** - we are aware that something about these graphs has gotten a bit odd in the recent Scanpy updates. Watch this space for a fix! The fix came!
-->


Well aren't those charts interesting! Using the diffusion map to drive the force-directed graph, we see correct ordering of our cells (from DN to DP to T-mature, which was lost with the diffusion map alone) as well as two apparent branches leaving the mature T-cell population, which is what we'd biologically expect. In terms of our experiment, we're seeing a clear trajectory issue whereby the knockout cells are not found along the trajectory into T-mature (which, well, we kind of already figured out with just the cluster analysis, but we can feel even more confident about our results!) More importantly, we can see the T-mature population dividing itself, which we did not see in the clustering via UMAP/tSNE alone, and we can verify that as the leftmost branch has CD4 but the rightmost branch does not. This is suggesting our branchpoint from to CD4+ and CD8+ single positive cells. Exciting! However, it is important to note, that the branches there are quite small and sparsely populated, which can indicate artifact branches (i.e. trajectory analysis does its best to find branches, particularly diffusion map, so you can pretty easily force branches to appear even if they are not biologically real!). However, to be frank, we were surprised not to find this clearer in the main cluster map, as we know that the T-cells should diverge at that point, so if anything this is a relief that our data is believable!

And now, just for fun, we can compare the scatter graph with our PAGA side by side.


```python
sc.pl.paga_compare(
    adata, threshold=0.03, title='', right_margin=0.2, size=10, edge_width_scale=0.5,
    legend_fontsize=12, fontsize=12, frameon=False, edges=True, save=True)
```

![PAGA Compare](../../images/scrna-casestudy/paga_compare.png "PAGA Compare")

<!---
**Note** - we are aware that something about these graphs has gotten a bit odd in the recent Scanpy updates. Watch this space for a fix! The fix came!
-->

## Diffusion pseudotime

We know that our cells are initialising at DN. We can feed that information into our algorithms to then calculate a trajectory.

First, let's name our 'root'.

```python
adata.uns['iroot'] = np.flatnonzero(adata.obs['cell_type']  == 'DN')[0]
```

## Working in a group? Decision-time!
If you called new clusters using the louvain algorithm, you might want to choose one of those clusters to be your root cell instead, so change the `cell_type` above for `louvain` and then name the cluster number. Use the plots you created to help you pick the number!


Onto the [diffusion pseudotime](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.dpt.html), where we are infer multiple time points within the same piece of data!

```python
sc.tl.dpt(adata)
sc.pl.draw_graph(adata, color=['cell_type', 'dpt_pseudotime'], legend_loc='on data', save = 'Plot8.png')
```

![Force-Directed + Pseudotime](../../images/scrna-casestudy/draw_graph_faPlot8.png "Force-Directed + Pseudotime")

This is nice, as it supports our conclusions thus far on the trajectory of the T-cell differentiation. With single-cell, the more ways you can prove to yourself what you're seeing is real, the better! If we did not find consistent results, we would need to delve in further to see if the algorithm produced the artefacts (not all algorithms fit all data!) or the biology suprised us.

Where might we go from here? We might consider playing with our louvain resolutions, to get the two branches to be called as different clusters, and then comparing them to each other for gene differences or genotype differences. We might also use different objects (for instance, what if we regressed out cell cycle genes?) and see if that changes the results. Perhaps we would eliminate the DN double-branch input. Or perhaps that's real, and we should investigate that. What would you do?



## Working in a group? The finale!
Look at each others images! How do yours differ, what decisions were made? Previously, when calling clusters in the [Filter, Plot, Explore tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}), the interpretation at the end is largely consistent, no matter what decisions are made throughout (mostly!). Is this the case with your trajectory analyses? You may find that it is not, which is why pseudotime analysis even more crucially depends on your understanding of the underlying biology (we have to choose the root cells, for instance, or recognise that DN cells should not be found in the middle of the DPs) as well as choosing the right analysis. That's why it is a huge field! With analysing scRNA-seq data, it's almost like you need to know about 75% of your data and make sure your analysis shows that, for you to then identify the 25% new information.

# Export your data, figures, and notebook

It's now time to export your data! First, we need to get it Jupyter to see it as a file.


```python
adata.write('Trajectorythymus.h5ad')
```

Now you can export it, as well as all your lovely plots! If you go into the *figures* folder at the left, you'll see your lovely plots and can choose which ones to export. The following code will push them into your galaxy history. You can also directly download them onto your computer from the file window at the left.

```python
put("Trajectorythymus.h5ad")
```
```python
put("figures/draw_graph_faPlot1.png")
put("figures/draw_graph_faPlot2.png")
put("figures/draw_graph_faPlot5.png")
put("figures/draw_graph_faPlot6.png")
put("figures/draw_graph_faPlot7.png")
put("figures/draw_graph_faPlot8.png")
put("figures/paga_compare.pdf")
put("figures/pagaPlot4.png")
```
The cell below will only work if you haven't changed the name of the notebook. If you renamed it, simply type its new name in the parenthesis.
```python
put("single-cell-scrna-case_JUPYTER-trajectories.ipynb")       
```

This may take a moment, so go check your Galaxy history to make sure your images, anndata object, and notebook (.ipynb) have all made it back into your Galaxy history. Once they are all there, you can exit this browser and return to the Galaxy tutorial!

If things have gone wrong, you can also download this [answer key tutorial]({{ page.zenodo_link }}/files/Trajectories_Answer_Key.ipynb).

# Citation

Please note, this is largely based on the trajectories tutorial found on the Scanpy site itself [https://scanpy-tutorials.readthedocs.io/en/latest/paga-paul15.html](https://scanpy-tutorials.readthedocs.io/en/latest/paga-paul15.html).



# After Jupyter

Congratulations! You've made it through Jupyter!

> <hands-on-title>Closing JupyterLab</hands-on-title>
> 1. Click **User**: **Active Interactive Tools**
> 2. Tick the box of your Jupyter Interactive Tool, and click **Stop**
{: .hands_on}

If you want to run this notebook again, or share it with others, it now exists in your history. You can use this 'finished' version just the same way as you downloaded the directions file and uploaded it into the Jupyter environment.


# Conclusion

Congratulations! You've made it to the end! You might be interested in the [Answer Key History](https://usegalaxy.eu/u/wendi.bacon.training/h/cs4inferring-trajectories-using-python-in-galaxyanswer-key) or the [Answer Key Jupyter Notebook](https://zenodo.org/record/7054806/files/Trajectories_Answer_Key.ipynb?download=1). If, for some reason anything didn't work in Galaxy JupyterLab environment, please don't get discouraged - we prepared a [Google Colab notebook version](https://colab.research.google.com/drive/1-ENlIXM7TAoDGsMam7RyJIusIaBfF9e4?usp=sharing) for you as a backup so that you can enjoy the tutorial no matter what!

In this tutorial, you moved from called clusters to inferred relationships and trajectories using pseudotime analysis. You found an alternative to PCA (diffusion map), an alternative to tSNE (force-directed graph), a means of identifying cluster relationships (PAGA), and a metric for pseudotime (diffusion pseudotime) to identify early and late cells. If you were working in a group, you found that such analysis is slightly more sensitive to your decisions than the simpler filtering/plotting/clustering is. We are inferring and assuming relationships and time, so that makes sense!

To discuss with like-minded scientists, join our [Gitter](https://gitter.im/Galaxy-Training-Network/galaxy-single-cell?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) channel for all things Galaxy-single cell!


topic: single-cell
tutorial: bulk-music-3-preparebulk
title: Bulk matrix to ESet | Creating the bulk RNA-seq dataset for deconvolution
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["transcriptomics", "data management"]
content_hash: 96239fef4f6c119c2a4122f9b8ce4ed6
last_modified: 2026-05-15T22:28:16.658876
zenodo_link: https://zenodo.org/record/7319173
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music-3-preparebulk/tutorial.html

content:
After completing the [MuSiC deconvolution tutorial](https://xuranw.github.io/MuSiC/articles/MuSiC.html) ({% cite wang2019bulk %}), you are hopefully excited to apply this analysis to data of your choice. Annoyingly, getting data in the right format is often what prevents us from being able to successfully apply analyses. This tutorial is all about reformatting a raw bulk RNA-seq dataset pulled from a public resource (the EMBL-EBI Expression atlas ({% cite Moreno2021 %}).  Let's get started!

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Metadata Manipulation

Just as in our scRNA-dataset preparation tutorial, we will tackle the metadata first. We are roughly following the same concept as in the previous bulk deconvolution tutorial, by comparing human pancreas data across a disease variable (type II diabetes vs healthy), but using public datasets to do it.

## Find the data
We explored the [expression atlas](https://www.ebi.ac.uk/gxa/experiments), browsing experiments in order to find the bulk RNA-seq pancreas dataset ({% cite segerstolpe2016single %}). You can [explore this dataset here](https://www.ebi.ac.uk/gxa/experiments/E-MTAB-5060/Results) using their browser. These cells come from 7 healthy individuals and 4 individuals with Type II diabetes, so we will create reference Expression Set objects for the total as well as separating out by phenotype, as you may have reason to do this in your analysis (or you may not!). This dataset is from the same lab that we built our scRNA-seq reference from, so we should get quite accurate results given the same lab made both datasets!

> <hands-on-title> Data upload </hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    {{ page.zenodo_link }}/files/E-MTAB-5060-experiment-design.tsv
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets as needed
>
> 4. Check that the datatype is tabular
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to `experiment-design` the following tags `#metadata #bulk #ebi`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

As before, the metadata object annoyingly has a bunch of unnecessary columns. You can examine this with the {% icon galaxy-eye %} in the Galaxy history. Let's remove them!

![Columns in a table where some contain run info or Sample Characteristic[age] while others are empty.](../../images/bulk-music/bulk-metadata-annoying.png "Ridiculous metadata columns and labels")

{% snippet faqs/galaxy/tutorial_mode.md %}

> <hands-on-title> Remove unnecessary columns </hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `output` (Input dataset)
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"List of Fields"*: `3 5 7 8 9 10 11 12 13 15 16 17 18`
>
>    > <comment-title></comment-title>
>    >
>    > Advanced cut works slightly differently in a workflow versus running the tool independently. Independently, there is a list and you can click through the list to note your columns, while in a workflow it appears as a text option and you put each column on a different line. The point is, each number above represents a column, so remove them!
>    {: .comment}
>
{: .hands_on}

Now let's take care of the excessively wordy header titles - and note that oftentimes various programmes struggle with titles or cells that have any spaces ` ` in them, so removing those now often saves hassle later.

> <comment-title></comment-title>
> You might also remember in the MuSiC tutorial that we can analyse numeric parameters in the metadata (in that case, hbac1c content). Reformatting to ensure numerical values in these columns (i.e. taking the ` years` out of an age cell) is helpful then too.
{: .comment}

> <hands-on-title> Fixing titles </hands-on-title>
>
> 1. {% tool [Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `output` (output of **Advanced Cut** {% icon tool %})
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Sample Characteristic\[age\]`
>            - *"Replacement"*: `Age`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `year`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Sample Characteristic\[body mass index\]`
>            - *"Replacement"*: `BMI`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Sample Characteristic\[disease\]`
>            - *"Replacement"*: `Disease`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Sample Characteristic\[individual\]`
>            - *"Replacement"*: `Individual`
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Sample Characteristic\[sex\]`
>            - *"Replacement"*: `Sex`
>
> 2. Change the datatype to tabular
>    
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
{: .hands_on}

Now examine {% icon galaxy-eye %} your resultant metadata file in the Galaxy history. Better, right?

![5 columns with numerical or string information on Run, Age, BMI, Disease and Sex](../../images/bulk-music/bulk-metadata-pretty.png "Look at the pretty metadata")

This is ready to go, so now we'll reformat the matrix!

# Manipulate the expression matrix

Let's upload the dataset.

> <hands-on-title> Data upload </hands-on-title>
>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    {{ page.zenodo_link }}/files/E-MTAB-5060-raw-counts.tsv
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. Rename the dataset as needed
> 3. Check that the datatype is tabular
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 4. Add to `raw-counts` the following tags `#matrix #bulk #ebi`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

Now examine {% icon galaxy-eye %} your raw counts file in the Galaxy history.

> <question-title></question-title>
>
> 1. Are samples in the rows or columns?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > ![Column 1 contains Gene ID followed by many lines of ENSG####. Column 2 contains the gene names. The following columns contain numerous iterations of ERR#####](../../images/bulk-music/raw-matrix.png "Gene info")
> >
> > 1. By examining the matrix, you can find that genes are the rows while samples are the columns.
> >
> {: .solution}
{: .question}

While it's awesome that there's a gene name column, unfortunately the gene names will be duplicated - different ENS IDs can refer to the same Gene Name. This going to be a problem later. So we need to get this in a format to collapse the ENS IDs, just as we did previously in the scRNA-seq data reference preparation. Sadly, we'll start by removing the column of gene names to prepare for the ENS ID collapse.

> <hands-on-title> Remove gene names column </hands-on-title>
>
> 1. {% tool [Remove columns](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file"*: `raw-counts` (Input dataset)
>    - In *"Select Columns"*:
>        - {% icon param-repeat %} *"Insert Select Columns"*
>            - *"Header name"*: `Gene Name`
>
{: .hands_on}

Now that your data is in a format of having a rows of ENS IDs and samples as columns, you can apply the handy ENS ID collapsing workflow as we did in the scRNA-seq reference. If you have already imported this workflow during the first tutorial, then you can use it again now.

> <hands-on-title> Convert from Ensembl to GeneSymbol using workflow </hands-on-title>
>
> 1. Import this [workflow](https://usegalaxy.eu/u/wendi.bacon.training/w/convert-from-ensembl-to-genesymbol-summing-duplicate-genes).
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. Run the workflow on your sample with the following parameters:
>    - *"Organism"*: `Human`
>    - {% icon param-file %} *"Expression Matrix (Gene Rows)"*: `output` (output of **Remove columns** {% icon tool %})
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
{: .hands_on}

The output will likely be called **Text transformation** and will look like this:

![Alphabetised gene symbols appear in column one with integers in the following columns corresponding to samples](../../images/bulk-music/ensidcollapse_bulk.png "Output of the ENS ID collapsing workflow for bulk dataset")

Success! You've now prepared your metadata and your matrix. It's time to put it together to create the Expression Set objects needed for MuSiC deconvolution.

# Construct Expression Set Objects

We have three more tasks to do: first, we need to create the expression set object with all the phenotypes combined. Then, we will create the two objects we actually need - one for healthy and one for diseased.

> <hands-on-title> Creating the combined object </hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `out_file` #matrix (output of **Text transformation** {% icon tool %})
>    - {% icon param-file %} *"Phenotype Data"*: `out_file1` #metadata (output of **Regex Find And Replace** {% icon tool %})
>
> 2. Remove the `#metadata #matrix` tags from the output **RData ESet Object**
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many genes are in your object?
> 2. How many samples?
> 3. What metadata categories are there?
>
> > <div id="solution-1" class="box-title"><button type="button" aria-controls="solution-1-contents" aria-expanded="true" aria-label="Toggle solution box: "><i class="far fa-eye" aria-hidden="true"></i><span class="visually-hidden"></span> Solution<span role="button" class="fold-unfold fa fa-minus-square"></span></button></div>
> >
> > The trick with all of these questions is to examine {% icon galaxy-eye %} the `General info` output {% icon param-file %} of the **Construct Expression Set Object** tool.
> > ![Lines showing ExpressionSet'; assayData: 34997 features, 7 samples; protocolData: none; phenoData; sampleNames: ERR### (7 total); varLabels: Age BMI Disease Sex; varMetadata: labelDescription; and 3 more useless lines ](../../images/bulk-music/generalinfo.png "General info output")
> >
> > 1. There are `34997` features, which are the genes.
> > 2. There are `7` samples.
> > 3. The metadata categories are the same you prepared earlier, shown here in a category of phenoData: `Age BMI Disease Sex`
> >
> {: .solution}
{: .question}

> <hands-on-title> Creating the disease-only object </hands-on-title>
>
> 1. {% tool [Manipulate Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_manipulate_eset/music_manipulate_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Expression Set Dataset"*: `out_rds` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Concatenate other Expression Set objects?"*: `No`
>    - *"Subset the dataset?"*: `Yes`
>        - *"By"*: `Filter Samples and Genes by Phenotype Values`
>            - In *"Filter Samples by Condition"*:
>                - {% icon param-repeat %} *"Insert Filter Samples by Condition"*
>                    - *"Name of phenotype column"*: `Disease`
>                    - *"List of values in this column to filter for, comma-delimited"*: `type II diabetes mellitus`
>
> 2. Add the tag `#T2D` to the output **RData ESet Object**
>
{: .hands_on}

You can either re-run this tool or set it up again to create the healthy-only object.

> <hands-on-title> Creating the healthy-only object </hands-on-title>
>
> 1. {% tool [Manipulate Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_manipulate_eset/music_manipulate_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Expression Set Dataset"*: `out_rds` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Concatenate other Expression Set objects?"*: `No`
>    - *"Subset the dataset?"*: `Yes`
>        - *"By"*: `Filter Samples and Genes by Phenotype Values`
>            - In *"Filter Samples by Condition"*:
>                - {% icon param-repeat %} *"Insert Filter Samples by Condition"*
>                    - *"Name of phenotype column"*: `Disease`
>                    - *"List of values in this column to filter for, comma-delimited"*: `normal`
>
> 2. Add the tag `#healthy` to the output **RData ESet Object**
>
{: .hands_on}

# Conclusion

{% icon congratulations %} Congrats! You have successfully reformatted the RNA-seq samples into two ESet objects consisting of disease-only or healthy-only samples. You're ready to take all this hard work and start comparing cell compositions in the next tutorial.

You can find the [workflow for generating the ESet object](https://usegalaxy.eu/u/wendi.bacon.training/w/music-deconvolution-data-generation--bulk--eset) and the [answer key history](https://usegalaxy.eu/u/wendi.bacon.training/h/music-deconvolution-data-generation--bulk--eset).

![7 boxes in the workflow editor and a subworkflow box for converting Ensembl to GeneSymbol](../../images/bulk-music/workflow-bulk.png "Workflow: Generating the bulk ESet Objects")

This tutorial is part of the [https://singlecell.usegalaxy.eu](https://singlecell.usegalaxy.eu) portal ({% cite tekman2020single %}).

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.2", "Remove columns", "toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0", "Construct Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4", "Manipulate Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_manipulate_eset/music_manipulate_eset/0.1.1+galaxy4"]


topic: single-cell
tutorial: scrna-raceid
title: Downstream Single-cell RNA analysis with RaceID
difficulty: intermediate
time_estimation: 3H
requirements_json: []
tags_json: ["[]"]
content_hash: 7155197fc9f28713846364e9e82a2e82
last_modified: 2026-05-15T22:28:16.816261
zenodo_link: https://zenodo.org/record/1511582
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-raceid/tutorial.html

content:
<!-- TODO: Subset analysis to validate prior cell labelling -->
<!-- TODO: Replace sort/unique with datamash -->


The data provided here as part of this tutorial analyses single-cell RNA-seq data from a study published by {% cite Gr_n_2016 %}. The data was used to cluster cells from *Lgr5*-positive intestinal stem cells of C57BL6/J mice, with the aim of discovering distinct cell sub-populations and deriving a lineage tree between them to find out how these sub-populations relate (or are derived from) one another.

The input data consists of a single count matrix consisting of ~21,000 genes (rows) and ~400 cells (columns) in tidy data format, generated via [scRNA pre-processing methods]({% link topics/single-cell/tutorials/scrna-preprocessing/tutorial.md %}) using the [CelSeq2 protocol]({% link topics/single-cell/tutorials/scrna-umis/tutorial.md %}).

> <comment-title>Tidy Data</comment-title>
> The [tidy data](https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html) convention prevalent amongst the R data analysis community assigns every value to a variable and an observation. The values are the number of reads which are assigned to a particular gene (a variable) that was measured within a specific cell (an observation).
>
{: .comment}

> <details-title>Count Correction</details-title>
> Normally a count matrix consists of integers, but this matrix has undergone an UMI-to-transcript [count alteration](https://www.nature.com/articles/nmeth.2930#methods) to correct against UMI errors, yielding decimal values instead. This correction is not that necessary in most datasets but it is used here.
>
{: .details}

This tutorial will perform cell clustering and lineage construction, as well as exploring some genes of interest.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Initial Count Matrix

Every single-cell RNA analysis begins with a count matrix, which contains the raw data required for the downstream analysis. Annotation data such as cell phenotype or gene annotations are often used to enrich and validate the analysis. Cell annotations can sometimes be encoded directly in the cell labelling, as we will later explore in this count matrix.

First, let's setup our history and initial dataset.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial and name it "RaceID on scRNA"
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the file from [Zenodo](https://zenodo.org/record/1511582) or from the shared data library
>
>    ```
>    https://zenodo.org/record/1511582/files/intestinalData.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the dataset to *"intestinal"*
> 4. Check that the datatype is a tab-separated file
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many genes are in the count matrix?
> 2. How many cells?
>
> > <solution-title></solution-title>
> >
> > A summary of our dataset is given by expanding the file preview window by clicking on the file name.
> >
> > 1. 20,269 lines, where lines/rows denote our genes.
> > 2. Scroll the mini-preview window to the right to see that the number of cells are 431
> >
> {: .solution}
>
{: .question}



## Inspecting the Cell Labelling

Cell labels usually encode which plate, batch, or cell barcode was used to delineate cell. Sometimes all three sources of information are used, which can be valuable information to have when looking for batch effects.

We can inspect the labels of this dataset by clicking on the {% icon galaxy-eye %} symbol. Immediately we can see that the header of this file contains cell names, following a naming convention that separates the cells into 5 batches: *I5d, II5d, III5d, IV5d*, and *V5d*

Is each batch equally populated? We can investigate this ourselves by extracting the headers, and reformatting them to see how many unique types we can detect:


> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Select first](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/1.1.0) %}  with the following parameters:
>    - {% icon param-file %} *"File to select"*: imported tabular file
>    - *"Operation"*: `Keep first lines`
>    - *"Number of lines"*: `1`
>
> 2. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0) %}
>    - {% icon param-file %} *"Input tabular dataset"*: output of **Select first**
>
>    > <question-title></question-title>
>    >
>    > What did this transpose step do? Why is it necessary?
>    >
>    > > <solution-title></solution-title>
>    > > The sole purpose of the Transpose tool is to switch columns with rows (and vice versa), which will make it easier to inspect and sort data.
>    > {: .solution}
>    {: .question}
>
> 1. {% tool [Text transformation](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sed_tool/1.1.1) %}  with the following parameters:
>    - {% icon param-file %} *"File to process"*: output of **Transpose**
>    - *"SED Program"*: `s/_[0-9]+//`
>
>       The above text is a regular expression used to match on anything that contains a `_` followed by a number, and removing it.
>
> 2. Inspect the generated file and verify that we have the 5 phenotypes without `_` followed by a number
>
>    > <question-title></question-title>
>    >
>    > How many rows remain after the **Text transformation** step?
>    > > <solution-title></solution-title>
>    > > The number of rows has not changed since the last step, but the cell names have lost their numbering and are identified purely by their phenotype.
>    > {: .solution}
>    {: .question}
>
> 1. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0) %}  with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: output of **Text transformation**
>    - *"Group by fields"*: `1`
>    - *"Input file has a header line"* : `Yes`
>    - *"Print header line"*:`No`
>    - *"Sort input"*:`Yes`
>    - *"Print all fields from input file"*:`No`
>    - *"Ignore case when grouping"*:`No`
>    - In *"Operation to perform on each group"*
>       - In *"1: Operation to perform on each group"*
>          - *"Type"*: `count`
>          - *"On column"*: `Column: 1`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many unique cell phenotypes were identified in the cell headers?
> 1. Which cell phenotype is least represented in the count matrix?
>
> > <solution-title></solution-title>
> >
> > 1. There are 5 types of cells in our count matrix: *I5d*, *II5d*, *III5d*, *IV5d*, and *V5d*.
> > 1. There are only 48 *IV5d* cells compared to the other types which have 95 or 96.
> {: .solution}
>
{: .question}

With these types already labelled in the header of our data, we can validate the clustering that we will perform later. Ideally, the cells described by these 5 different labels should group into 5 separate clusters, with varying degrees of proximity to one another.

## Inspecting the Quality of the Count Matrix

Low quality cells and genes are often caught at the pre-processing stage and removed, but sometimes more filtering is required to remove unwanted noise from the data.

A gene with a low number of total counts across multiple cells might be differentially expressed across those cells, however differential gene expression and log fold change do not necessarily denote significant change. 1 count in CellA and 10 counts in CellB yields an LFC of 10, but is not as significant as an LFC of 10 from 100 counts in CellA vs 1000 counts in CellB.


We can refine filtering thresholds by examining how much a histogram of our plots change before and after filtering using standard parameters.

> <hands-on-title>Unique Cell Types</hands-on-title>
>
> 1. {% tool [Initial processing using RaceID](toolshed.g2.bx.psu.edu/repos/iuc/raceid_filtnormconf/raceid_filtnormconf/0.2.3+galaxy0) %}  with the following parameters:
>    - {% icon param-file %} *"Count Matrix"*: `intestinal` (Input dataset)
>    - In *"Filtering"*:
>        - *"Min Transcripts"*: `3000`
>        - *"Min Expression"*: `5`
>        - *"Min Cells"*: `5`
>        - *"Use Defaults?"*: `Yes`
>
>
{: .hands_on}

The tool provides metrics on the library size and the number of detected features.

> <details-title>Library Size and Number of Features (Definitions)</details-title>
>
> ![Library Size and Detected Genes]({% link topics/single-cell/images/scrna-raceid/libsize.svg %} "Library Size and Number of Features")
>
> * **Library Size**: The total number of transcripts in a cell, or a column sum. *The minimum threshold for this can be set by "Min Transcripts" (= 3000)*.
> * **Gene Transcripts**: The total number of transcripts for a gene, or a row sum.
> * **Detectability**: This is how many values above a certain threshold exist in a given row/column.
>   * **Number of Features**: The number of genes detected above a certain threshold.
>     * *>0*: This is the default threshold when the term *Number of Features* is used, because it counts any genes with non-zero counts for that cell, and are therefore 'detected' during sequencing.
>     * *>N*: Depending on how lowly sequenced the cells are, sometimes it is useful to set a higher threshold of detectability to guard against sequencing errors or other noise-related factors. *The minimum threshold for this can be set by "Min Expression" (= 5)*.
>   * **Number of Cells**:
>     * *>0*: This is the default threshold when the term *Detected in X number of Cells* is used, because it counts any cells with non-zero counts for that gene.
>     * *>N*: This raises the detectability threshold to any cell count above N. *The minimum threshold for this can be set by "Min Cells" (= 5)*.
>
{: .details}

Four histograms are generated with the top line giving the raw expression data fed into the tool, and the bottom line giving the filtered data (cells with at least 3,000 transcripts in total and genes with at least 5 transcripts in at least 5 cells).

![Histograms of raw and filtered data]({% link topics/single-cell/images/scrna-raceid/filter_plots.png %} "RaceID Histograms of raw and filtered data")




* (Top-Left) Library Size (total number of transcripts per cell)

* (Top-Right) Feature Set (total number of detectable genes per cell)

* (Bottom-Left) Filtered Library Size (minimum 3000 transcripts per cell)

* (Bottom-Right) Filtered Feature Set

> <details-title>RaceID Histograms</details-title>
>
> The top row shows the count distributions of the Library Size and Number of Features of the raw data using a Log scale on the x-axis (e.g. 2.5 on the Log10 x-axis = $$10^{2.5}$$ = 316 counts).
>
> * (Top-Left) A lower-tail heavy distribution centred around 3-4 Log10 (~3000) counts per cell , with a few cells having library sizes containing a handful of counts (0-1000).
>
> * (Top-Right) Another lower-tail heavy distribution with a peak centred around $$10^{3.5}$$ counts. Cells with a low number of features are hard to compare with other cells due to incomplete data. It is possible that these low feature cells (< 100 genes) are rare types and that we should impute their missing values, but it is often more likely the case that these are simply just low-quality cells that will add noise to the clustering.
>
> The bottom row shows the count distributions of the Library Size and Number of Features of the filtered data
>
> * (Bottom-Left) The lower-tail of our previous distribution has been trimmed off, which gives an even normal-looking distribution centred around $$10^{3.4}$$ transcripts per cell.
>
> * (Bottom-Right) Instead of a distribution we have a single bar that indicates that all of our cells have the exact number of features. The red line displays the number of features across all cells (~ $$10^{3.3}$$).
>
{: .details}

> <comment-title>Choosing Filtering Thresholds</comment-title>
>
> The minimum total filtering threshold of 3000 chosen for this dataset is derived from analysing the *Cross-Contamination Plots* from the [Pre-processing of Single-Cell RNA Data]({% link topics/single-cell/tutorials/scrna-preprocessing/tutorial.md %}).
>
> This threshold is dependent primarily on the capture efficiency of the cells that were sequenced, with some cell types being easier to capture than others. For example, neuron cells would have a lower filtering threshold of ~1500 compared to the ~3000 used for hematopoietic cells.
>
{: .comment}

> <details-title>Why the Same Number of Features?</details-title>
>
> * RaceID normalises the data to compare all cells using the same set of features. Selected features must be *meaningful*, describing or contributing to the biological variation in the data. Therefore features should be the most differentially expressed genes across different cells.
>
> * The cells that still remain after filtering will have some genes with count values of zero. In some cases this will give a median value of zero, which can make normalisation strategies difficult (e.g. dividing by the median value of a gene may require dividing by zero). For this reason, a value of 0.1 is added to the count data so that these features are not lost during the analysis.
>
> * It should be noted that this then assumes that the gene *is* detectable for that cell (i.e. no errors during sequencing), but that the transcript was very lowly expressed.
>
> * The flat square plot (bottom-right) for the post-filtered number of features is square because it shows that the dataset is still a rectangular matrix with C number of cells and G number of genes. For a more 'realistic' distribution of features, re-run the tool with *"Count filtered features greater than or equal to 1"* enabled.
>
{: .details}

> <question-title></question-title>
>
> 1. How many cells remain after filtering?
> 2. How many genes remain after filtering?
> 3. Are these numbers to be expected?
>
> > <solution-title></solution-title>
> >
> > The answer to the first two questions can be seen in log file generated **Initial processing using RaceID**
> >
> > 1. 287 cells remain (66%)
> > 2. 2089 genes remain (10%)
> > 3. **Yes**
> >    * *Cells:*
> >      These are the observations of your dataset, and the more observations you have, the better the model will be. At minimum, 60% of your initial cells should be retained, though this will depend on the quality of your dataset.
> >    * *Genes:*
> >      As the variables of the dataset, the more genes included, the more complex the model. Discovering the few variables that are relevant to the final model, then,  is the aim. Genes *not* differentially expressed between cells will not affect the final model, but serve as stable background metrics against which to measure significantly differentially expressed genes in the initial matrix. It is perfectly acceptable to perform a single-cell RNA-seq analysis with as few as 500 (differentially expressed) genes.
> >
> {: .solution}
>
{: .question}


The filtered distributions are what are expected of a properly filtered and normalised dataset (i.e. a count matrix with all observations having roughly the same number of transcripts, but distributed differently across different sets of common features). With this we can now perform initial clustering to see whether we can cluster any of our cells into distinct cell types.


# Normalising and Clustering Cells

Normalisation permits the comparison of different samples by refactoring out uninformative variability relating to the size of sample, and other sources of unwanted variability. Clustering, which groups or categorises cells based on their similarity, and is a crucial stage in the analysis after normalisation.

The effectiveness of the clustering relies on the normalisation. The ideal method to normalise single cell RNA-seq data is still a field of active research as  it is not a straightforward process due to two potential sources of uncertainty: technological and biological variability.


## Biological Variation

![Sources of variation]({% link topics/single-cell/images/scrna-raceid/cellcycle.svg %} "Sources of unwanted biological variation: (Left) Transcriptional Bursting, and (Right) Cell-cycle Variation")

Transcriptional bursting is a stochastic model for the transcription process in a cell, where transcription does not occur as a smooth or continuous process but occurs in spontaneous and discrete 'bursts' thought to be only loosely associated with chromatin conformation/availability. It is an effect that is not seen in bulk RNA-seq due to the smoothing effect of measuring average gene expression across a tissue. The effect is more pronounced in single-cell and is hard to model against.

On the other hand, cell-cycle variation is well defined and can be modelled against. As the cell grows from the G1 to the M phase, the amount of mRNA transcribed grows, meaning that cells in the later stages of their cycle are more likely to produce more transcripts of a given gene. Such differences can give false variation that would cluster two cells of the same type but at different time-points separately. Fortunately, there are a well-defined set of genes whose expression is known  to co-vary with the cell-cycle, thus this effect can be modelled out.


## Technical Variation

![Sources of variation]({% link topics/single-cell/images/scrna-raceid/technical_variation.svg %} "Sources of unwanted technical variation")

Technical variation appears in three main forms: *Library size variation*, *Amplification bias*, and *Dropout events*.

> <details-title>Technical Variation</details-title>
> 1. **Library size variation** occurs where two cells of the same cell type may produce a different amount of total transcripts than one another (e.g. due to cell-cycle effects, or differing capture efficiencies), but have a similar *proportion* of transcripts for specific genes. For example, a neural cell with 10 counts of SOX2 and a library size of 100 and one with 20 counts and a library size of 200 have the same proportion of SOX2. The two cells have different library sizes, but harbour the same expression for that gene because they are of the same cell type.
>
> 1. **Amplification bias** stems from an uneven amplification of certain transcripts of a cell over others, giving a false number of reads for the number of mRNA molecules actually observed in the cell. Unique Molecular Identifiers can significantly reduce this bias, and are covered more extensively in the [*Understanding Barcodes*]({% link topics/single-cell/tutorials/scrna-umis/tutorial.md %}) hands-on.
>
> 1. **Dropout events** are the zero counts that are prevalent in the data due to the reduced sequencing sensitivity in detecting reads, which yields many false negatives in the detection of genes, often resulting in over 80% of the count values in the count matrix being zero. A major point to take into account is that some of these zeroes are *real* (i.e. no transcripts of that gene were detected in that cell) and some of these are *false* (i.e. the transcripts were never captured due to the low sequencing depth). Modelling this duality in the data and mitigating against it is one of the biggest challenges of normalising single-cell data.
>
{: .details}


## Performing the Clustering

We will attempt to perform some normalisation and clustering using the recommended default settings to see if we can detect different cell types.

Here we assume that there is no unwanted technical or biological variability in the data and that the cells will cluster purely based on their phenotypes. This assumption is not completely without merit, since often the biological signal is strong enough to counter the lesser unwanted variation.


> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Clustering using RaceID](toolshed.g2.bx.psu.edu/repos/iuc/raceid_clustering/raceid_clustering/0.2.3+galaxy0) %}  with the following parameters:
>    - {% icon param-file %} *"Input RaceID RDS"*: `outrdat` (output of **Initial processing using RaceID** {% icon tool %})
>    - In *"Clustering"*:
>        - *"Use Defaults?"*: `Yes`
>    - In *"Outliers"*:
>        - *"Use Defaults?"*: `Yes`
>    - In *"tSNE and FR"*:
>        - *"Use Defaults?"*: `Yes`
>
{: .hands_on}


> <details-title>Clustering on a Distance Matrix</details-title>
>
>
> In order to perform clustering, the proximity of cells to one another are first defined by a distance metric
> such as [Euclidean distance or other](https://en.wikipedia.org/wiki/Metric_(mathematics)).
>
> ![Distances]({% link topics/single-cell/images/scrna-raceid/distance.svg %} "Euclidean distance between three points (R, P, V) across three features (G1, G2, G3)")
>
> The clustering then tries to find groupings of cells using this distance matrix.
>
> > <question-title></question-title>
> >
> > 1. Why are there zeroes along the diagonal of the above example distance matrix?
> > 1. Is there any symmetry in this matrix?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. The distance between a point to itself is - nothing!
> > > 1. The distance between point *a* to point *b* is the same as the distance between point *b* to point *a* using the Euclidean distance metric.
> > >
> > {: .solution }
> >
> {: .question }
>
{: .details}

The first three plots in the PDF report tell us about the stability/reliability of our clusters, and are more important indicators for the quality of our clustering than any of the resultant graph projections, such as PCA or tSNE.

![Stability Plots]({% link topics/single-cell/images/scrna-raceid/sat_jacc.png %} "RaceID Saturation and Jaccard Distance Plots")

* (Top-Left) Dispersion within each cluster for given K values, with mean change in dispersion for said K value overlayed.

* (Top-Right) Same as top-left, but with the actual dispersion plotted instead of the relative change of dispersion.

* (Bottom-Left) Stability for each cluster using [Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index).

> <details-title>RaceID Plots</details-title>
>   * The first plot measures the levels of dispersion within each cluster and displays the mean dispersion over all clusters for each k value. The grey points indicate the mean change in dispersion for that k values, and the red bars show this dispersion as a box plot, which get smaller and smaller until a "saturation point" (blue) is reached where the change in dispersion no longer decreases for an increase in k.
>
>  * For example, if *k=2*, then all cells will be sorted into 2 clusters, and the variance of the gene expression in each cluster will be measured and averaged to give a score for the clustering at that k value. Certain values of *k* may cluster the cells of the same type better, with the expectation that the average dispersion of expression values across all clusters will be minimised for some value of *k*. As *k* increases, the reduction in this dispersion is measured for each increase of *k* until the change in the mean within-cluster dispersion no longer changes. Here we can see that reduction saturates at *k=12*, which is chosen to the be the number of clusters detected in our data for all further analysis.
>
>  * The third plot measures the direct stability of each of the derived (in this case, 12) clusters using the Jaccard distance, which is a fractional quantity that measures the dissimilarity between two sets as measured by overlap divided by the union of both sets.
>
{: .details}

> <details-title>Jaccard Distance</details-title>
>
> Jaccard distance is measured by the intersection of the two or more sets divided by the union of those sets.
>
> $$d_J(X,Y) = 1 - J(X,Y) = \frac{|X \cup  Y| - |X \cap Y|}{|X \cup Y|}$$
> <!-- This seems to be visible when collapsed...? -->
>
> In the case of single-cell data, sets are defined as the cells contained within a given cluster, and the Jaccard similarity score provides a quantitative measure for how distinct a given cluster is, based on its similarity to other sets.
>
> ![Jaccard]({% link topics/single-cell/images/scrna-raceid/jaccard.svg %} "Example Jaccard Distance calculation")
>
{: .details}

Here, for each of the 12 clusters, the (top N) genes expressed by the cells in each cluster are intersected with the (top N) genes expressed by the cells in all other clusters to measure how unique the expression profile is to that specific cluster. The scales given by the plot are actually measuring the dissimilarity between sets, which is one minus the index, therefore a higher y-value is better.

Ideally the Jaccard distance should have above 0.6 in most clusters, but it is acceptable to have one or two more poorly defined clusters.


## Outlier Detection

Outlier detection attempts to refine the initially detected clusters to find smaller (sub-)clusters that could be used to define rarer cell types.

The next three plots attempts to do this by describing the variation of the gene expression, given by; a Background plot, a Sensitivity plot, and an Outlier probability plot.


![Gene Expression Plots]({% link topics/single-cell/images/scrna-raceid/gexpr.png %} "RaceID Gene Expression Plots")


* (Top-Left) Outlier identification via background model based on distribution of transcript counts within a cluster.

* (Top-Right) Outlier cells are detected if the probability for that cell $$c$$ that a minimum number of genes $$G_{min}$$ of observing total counts $$T_{G_{min}}$$ is less than a specific threshold $$P_{thr}$$, as given by the red dotted line.

* (Bottom-Left) Outlier probabilities of all cells in all clusters.


> <details-title>RaceID Outlier Identification</details-title>
> * (Top-Left) A background model is calibrated and outliers are identified based on the distribution of transcript counts within a cluster. The counts for each gene are assumed to follow a negative binomial distribution determined by the average expression of a gene across all cells in a cluster, along with a dispersion parameter.
>     * The dispersion is derived from the average variance-mean dependence, modelled as a logarithmic second order polynomial under the assumption that:
>         1. Most genes are not differentially expressed between clusters
>         1. True biological variability is located within a handful of genes
>
>     * In the background model, the upper and lower (violet and red) regression of the variance on the mean (as approximated by a second-order polynomial in logarithmic space) is higher than the variance of most genes (all grey dots below the red curve). This is expected since they are not differentially expressed, and so the genes above the background regression are therefore significant in the detection of outlier cells. The orange line is the local regression (moving average variance per mean) and is used purely for illustrative purposes. <!-- See: https://github.com/dgrun/RaceID3_StemID2_package/blob/master/R/RaceID.R#L280 -->
>
> * (Top-Right) Outlier cells are detected if the probability for that cell $$c$$ that a minimum number of genes $$G_{min}$$ of observing total counts $$T_{G_{min}}$$ is less than a specific threshold $$P_{thr}$$, as given by the red dotted line.
>     * This is shown in the chart above as the number of outliers as a function of the probability threshold, which is set to $$1 \cdot 10^{-3}$$ by default. Ideally, this threshold should be chosen so that the lower tail of the distribution contains as few outliers as possible (i.e. lower than the steep rise in outliers towards the higher end of the plot) to ensure a maximum sensitivity of this method. If the sensitivity of the sequencing was low, then only a few highly expressed genes would be reliably quantified, so the outlier probability threshold would need to be higher (e.g. up to 1).
>
> * (Bottom-Left) A bar plot of the outlier probabilities of all cells across all clusters. All outlier cells are merged into their own clusters if their similarity exceeds a quantile threshold of the similarity distribution for all pairs of cells within one of the original clusters. After the outlier cells are merged, then the new cluster centres are defined for the original clusters after removing the outliers. Then, each cell is assigned to the nearest cluster centre using k-partitioning.
>
{: .details}


The most differentially expressed genes (below a maximum p-value cutoff) in each of the clusters can be seen in the output file *Clustering using RaceID on data: Cluster - Genes per Cluster*, with 7 columns describing:

 | Column | Description |
 |--------|-------------|
 | **.** | Gene Name |
 | **n** | Cluster number |
 | **mean.ncl** | Mean expression of gene across cells *outside* the cluster |
 | **mean.cl** | Mean expression of gene across cells *inside* the cluster |
 | **fc** | Fold-change of mean expression in the cluster, against all remaining cells |
 | **pv** | Inferred p-value for differential expression |
 | **padj** | Adjusted p-value using the Benjami-Hochberg correction for the false discovery rate |


> <question-title></question-title>
>
> 1. What is the most significant differentially expressed gene in Cluster 5?
> 2. Which cluster has the most genes with a fold-change greater than 30?
>    > <tip-title></tip-title>
>    > One way to answer is this is to scroll through the list and make a mental note of all the genes with a high fold-change value...
>    >
>    > ...but a *better* way is to sort the data in descending order on Column 5 using the {%icon tool %} **Sort** tool.
>    >
>    {: .tip}
>
> > <solution-title></solution-title>
> > 1. The *Genes per Cluster* dataset is already sorted by cluster number and then by the adjusted p-value. Scrolling down to the first gene that is in cluster 5, we see that *Dmbt11* is the most significant gene in that cluster with an adjusted p-value of `2e-30`.
> > 1. Cluster 11 appears to have the most genes with a fold-change greater than 30.
> >
> {: .solution}
>
{: .question}


## Heatmaps

The remainder of the plots are heatmaps derived from k-medoids clustering, showing the similarity between clusters for both the initial clustering and the final (post outlier detection) clustering.

> <details-title>K-medoids and K-means Clustering</details-title>
>
> The k-medoids algorithm relates to the k-means algorithm of clustering, by which
>
> * Initialise:
>    1. A value of k is chosen, e.g *k=3*.
>    1. k points are chosen at random in the dataset (or initialised in some rational way)
> * Step:
>    1. Each datapoint in the dataset is assigned to the closest k point using some distance metric.
>    1. The mean (or in this case, the median) of all points assigned to a specific k point are computed for each k point.
>    1. The k-points take on the new position of the mean (or median) values just computed
>    1. Repeat Step until convergence reached or after N number of iterations
>
> ![K-means](https://upload.wikimedia.org/wikipedia/commons/e/ea/K-means_convergence.gif "K-means convergence (Wikimedia Commons)")
>
> In terms of single-cell data, for *G* genes, each datapoint is a G-dimensional cell. The distances between cells are computed using a [given metric](#performing-the-clustering), and the k points are G-dimensional points indicating the centre of cluster. The k points are then updated for each iteration of the algorithm.
>
> > <question-title></question-title>
> >
> > Why use G dimensions?
> >
> > Why not use 2 dimensions as given in the graphic above?
> >
> > > <solution-title></solution-title>
> > >
> > > Using 2 dimensions will not capture all the variability in the data, and so the clustering will be based on whatever those 2 chosen dimensions are biased towards.
> > >
> > > For example, if we performed a PCA first on the data and used only the first two (most variable) components of the data to perform clustering upon, we might be clustering cells based on cell-cycle variation which might be stronger than the actual biological signal that is perhaps hidden away in the higher PCA components.
> > >
> > {: .solution}
> >
> {: .question}
>
{: .details}


![Heatmaps]({% link topics/single-cell/images/scrna-raceid/heatmaps.png %} "RaceID Heatmaps for initial and final clusters")

The difference shown between the initial and final clustering is sometimes subtle depending which of the clusters the new clusters have been extracted from.


> <question-title></question-title>
>
> 1. Which new clusters have been added?
> 2. From which cells are these new clusters derived from?
>
> > <solution-title></solution-title>
> >
> > 1. `c13` and `c14`
> > 2. `c13` appears between `c8` and `c1`, suggesting that the cells in `c13` share a closer similarity to the cells in these clusters, which they were most likely extracted from. The other new cluster `c14` is between `c11` and `c6`, suggesting similar points of extraction.
> >
> {: .solution}
>
{: .question}

All the following plots are heatmaps for the individual genes expressed in each cluster.

![Gene Heatmaps]({% link topics/single-cell/images/scrna-raceid/geneheatmaps.png %} "RaceID Individual (final) heatmaps for the top 10 significant genes in clusters 1 to 4")

The top 10 defining genes from each cluster (above only `c1`-`c4` are shown) give us an idea of how unique these genes are to the cluster.

> <question-title></question-title>
>
> 1. Which clusters are *Gstm3*, *St3gal4*, and *Gna11* highly expressed in?
> 2. Where is *Eef1a1* expressed?
> 3. Which cluster is poorly defined?
>
> > <solution-title></solution-title>
> >
> > 1. (Top-Left Heatmap) *Gstm3*, *St3gal4*, and *Gna11* are highly expressed in `c1` and `c6`
> > 2. *Eef1a1* is highly expressed everywhere, and so is not a very differentially expressed gene.
> > 3. `c4` appears to be a not so well-defined cluster with no genes specifically bound to it.
> {: .solution}
>
{: .question}



## Visualising All Clusters

The previous section produced plots that spoke about the quality of the clustering without really showing us the clusters projected into an understandable 2D space. To perform this, we must feed the clustered data into the cluster inspection tool.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Cluster Inspection using RaceID](toolshed.g2.bx.psu.edu/repos/iuc/raceid_inspectclusters/raceid_inspectclusters/0.2.3+galaxy0) %}  with the following parameters:
>    - {% icon param-file %} *"Input RaceID RDS"*: `outrdat` (output of **Clustering using RaceID** {% icon tool %})
>    - *"Plot All Clusters?"*: `Yes`
>    - *"Perform Subset Analysis?"*: `No`
>    - *"Examine Genes of Interest"*: `No`
>    - *"Differential Gene Testing"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > This tool can perform multiple different modes of inspection upon clustered scRNA data, and users are encouraged to explore the different modes.
>    {: .comment}
>
{: .hands_on}

The main issue with visualising this data is that as before, we have C cells that serve as our observations which are described by G genes. Representing G dimensional data in the 2 or 3 dimensional plots that we are more familiar falls under the problem of [*dimensional reduction*](https://en.wikipedia.org/wiki/Dimensionality_reduction).

> <details-title>Dimension Reduction</details-title>
> ![Dim red]({% link topics/single-cell/images/scrna-raceid/dimred.svg %} "Reducing a set of 4 observations from 3D to 2D space, whilst approximating the 3D relationships")
>
> Dimension reduction aims to preserve the distances and relationship of the higher dimensional (G-dimensional) data in a lower dimensional (usually 2D) space.
>
{: .details}

Preserving these higher dimensional distances in lower dimensional space is a complex and ongoing challenge in computer science, but there are various commonly-used methods such as PCA and tSNE often encountered in single cell RNA-seq datasets. For more information, see the box below.

> <details-title>PCA, tSNE, and Force-Directed Graphs</details-title>
>
> * **PCA** (Principle Component Analysis)
>   * This linearly separates the N-dimensional data into N distinct components (or 'axes') of variability, sorted in descending order of variability (i.e. the first component explains most of the variation in the data, the second component contributes the second most amount of variation in the data, etc). For more information on how these components are derived, see [*Eigendecomposition of a matrix*](https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix).
>   * PCA has the benefit of being deterministic, uncostly to implement, and usually good enough to find clear differences in the data if the data is not complex.
>
> * **tSNE** (T-distributed Stochastic Neighbour Embedding)
>   * PCA rests on the assumption that each of its N components are independent of one another, but this is often rarely the case and data usually exhibits more complexity than that of a linearly separable dataset. In these situations, tSNE outshines PCA, by modelling a more complex relationship between datapoints, and often produces better looking plots with more clearly defined clusters. This is especially the case in single-cell RNA-seq data which tends to describe a continuous blend of cell phenotypes, instead of discrete rigidly-defined types.
>
> * **Force-Directed** Graphs
>   * These graphs can be better thought of as particle simulations, instead of performing any dimensional reduction, since all that is required is a connected graph. Forces are added to the connections between the nodes on the graph based on the strength of the connection, and then the whole system simulates the interplay of these forces for a number of iterations or until the system comes to rest. In general, these tend to yield much more nicely separated plots than tSNE, but it is not always the case and so it is always good practice to compare both force-directed and tSNE plots.
>
{: .details}

**RaceID** makes use of tSNE and force-directed (Fruchterman-Reingold) graph layouts to space the clusters in a visually meaningful manner to show the separation and relative proximity of clusters to one another.

![Clusters]({% link topics/single-cell/images/scrna-raceid/tsne_umap_fr.png %} "RaceID Initial and Final clusters using tSNE, UMAP and F-R projections")

The figure above displays the initial (left top/bottom) clusters detected during the clustering stage, as well as the final (right top/bottom) clusters determined during the outlier detection stage, projected using tSNE and Fruchterman-Rheingold graph layouts.

> <question-title></question-title>
>
> 1. What has changed between the initial and final plots?
> 1. Which clusters appear to be well defined? Are they consistent between projections?
> 1. Does this agree with the heatmaps we have seen previously?
>
> > <solution-title></solution-title>
> >
> > 1. Two extra clusters are added in the final plots `c13` and `c14`
> > 1. For example, `c11` appears to be an isolated well-defined cluster of cells, distinct in both projections. At the edge of the main cluster body in both projections lies `c1`, but seems to be in closer proximity to `c13` in the tSNE map than in the F-R layout. In both projections, `c2`, `c3`, and `c4` are large noisy clusters, but `c2` and `c4` appear to be closer to one another in the F-R layout.
> > 1. `c1` was better defined by a smaller set of genes than `c2`, `c3`, or `c4,` which listed less differentially expressed genes as their most significant genes.
> >
> {: .solution}
>
{: .question}

### Is this expected?

One pervasive thought when analysing single-cell RNA data, is "*is this actually good clustering?*"

To answer this question, we must understand the nature of the data which *does* contain clear cell type phenotypes, but that these cell phenotypes are driven by continuous cell-cycle and cell developmental processes. As a result it is quite normal to see clusters 'blending' into one another, since these suggest intermediate cell types that are transitioning from one cell type to another.

> <details-title>Continuous Phenotypes vs Discrete Clustering Methods</details-title>
>
> ![Continuous Phenotypes]({% link topics/single-cell/images/scrna-raceid/contpheno.svg %} "The continuous phenotypes along a red blood cell development trajectory")
>
> The gene expression profile for Reticulocytes is distinct from the gene expression profile for the mature Red Blood Cells that they will become, but the cells that are actually undergoing this short-lived transition from Reticulocyte to Red Blood Cell will not fit neatly into the two aforementioned gene expression profiles, instead having their own profile which lies somewhere in between.
>
> As a result it is often false to think of cell clustering as an exercise in classification, but better to be thought of as a desire to find the relatedness between expression profiles. A helpful visual example of this is to not think of assigning cells to different disconnected 'peaks' of expression, but to place cells along an expression peak landscape that describes not only which expression profile they resemble the most, but along which gradient (or development trajectory) they adhere to, allowing transient cell types to be better assigned.
>
> ![Discrete vs Continuous]({% link topics/single-cell/images/scrna-raceid/mountains.svg %} "(Above) Cells assigned to discrete profiles, compared to (Below) Cells placed along a continuous expression profile landscape.")
>
> The clustering of cells with single-cell RNA-seq data, is therefore not a classification problem, but a *manifold* detection one, where the manifold ultimately describes the topology of the data, and allows us to see relatedness between clusters.
>
{: .details}

# Individual Cluster Inspection

To really understand the dynamics in the shifting profiles of gene expressions between different clusters, it is often useful to see specifically which genes are differentially expressed, and which clusters and cells they align to or define the most.

There are three ways to do this in RaceID:

 1. **MA Plot**

    Perform a pairwise comparison between two clusters (or two sets of clusters) to see specifically which genes are differentially expressed between them.

 1. **Subset Cell Analysis**

    If the cell headers have names that contain information prior to the clustering about the different cell phenotypes, then it might be interesting to see if the cells do cluster as expected.

 1. **Specific Expression Plots**

    It may be of interest to look at how specific genes which may be markers for a cell type are expressed across different clusters, with the expectation that they are localised to a specific cluster depending on how specific the marker is.


## Differential Gene Analysis Between Two Clusters

We will generate an [MA Plot](https://en.wikipedia.org/wiki/MA_plot) between the two clusters, which looks at the differences between two samples by comparing the *M* (log ratio) against the *A* (mean average) of the sets.

Here we will compare how the cells in cluster 1 are differentially expressed compared to the cell of cluster 3.

> <hands-on-title>MA plot</hands-on-title>
>
> 1. {% tool [Cluster Inspection using RaceID](toolshed.g2.bx.psu.edu/repos/iuc/raceid_inspectclusters/raceid_inspectclusters/0.2.3+galaxy0) %}  with the following parameters:
>    - {% icon param-file %} *"Input RaceID RDS"*: `outrdat` (output of **Clustering using RaceID** {% icon tool %})
>    - *"Plot All Clusters?"*: `No`
>    - *"Perform Subset Analysis?"*: `No`
>    - *"Examine Genes of Interest"*: `No`
>    - *"Differential Gene Testing"*: `Yes`
>        - In *"Cells in Set A"*:
>            - *"Name of Set"*: `Cells in 1`
>            - *"Selection method"*: `Cluster Numbers`
>                - *"List of clusters"*: `1`
>        - In *"Cells in Set B"*:
>            - *"Name of Set"*: `Cells in 3`
>            - *"Selection method"*: `Cluster Numbers`
>                - *"List of clusters"*: `3`
>        - *"Use Defaults?"*: `Yes`
>
{: .hands_on}

![Cluster Inspection of Cells]({% link topics/single-cell/images/scrna-raceid/clustinspect_cells.png %} "RaceID MA plot of cells in cluster 1 and cluster 3")

The genes shown as grey dots are not labelled because they are not so differentially expressed between the two clusters, but the genes as labelled red dots on the fringes do display significant variability between the clusters.

> <question-title></question-title>
>
> 1. How do we interpret this plot?
> 1. Is *Gstm3* (at position 1.5, -2.7 on the MA plot) more significantly differentially expressed than *Ptma* (at position 3.6, 1.2)?
>
> > <solution-title></solution-title>
> >
> > 1. Interpretation of plot:
> >   * The vertical axis displays the difference between cluster 3 and cluster 1, where the total counts of a gene in cluster 3 are subtracted by the total counts of a gene in cluster 1. Therefore, a gene which is in the negative portion of the vertical axis has more expression in cluster 1.
> >   * The horizontal axis displays the average expression of a gene in both clusters, and so serves as a yardstick to estimate how expressive that gene is overall.
> > 1. *Gstm3* has a lower average expression in both clusters, than *Ptma* which is higher on the horizontal axis. However, *Gstm3* is expressed significantly more in Cluster 1, whereas *Ptma* is expressed only slightly more in Cluster 3. Overall, *Gstm3* is more differentially expressed than *Ptma*, and is up-regulated in Cluster 3, although with fewer total mRNA counts.
> >
> {: .solution}
>
{: .question}


## Differential Gene Expression Across All Clusters

We will now look at some genes of interest to see how prevalent or unique they are across clusters. Usually known marker genes are used to identify clusters by their cell type and not just a number, but any gene of interest can be used if it is believed to characterise a cluster of cells.

Here we will look at the combined expression of *Gstm3*, *St3gal4*, and *Gna11* which all had adjusted P-values of less than $$1 \cdot 10^{-15}$$ in cluster 1.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Cluster Inspection using RaceID](toolshed.g2.bx.psu.edu/repos/iuc/raceid_inspectclusters/raceid_inspectclusters/0.2.3+galaxy0) %}  with the following parameters:
>    - {% icon param-file %} *"Input RaceID RDS"*: `outrdat` (output of **Clustering using RaceID** {% icon tool %})
>    - *"Plot All Clusters?"*: `No`
>    - *"Perform Subset Analysis?"*: `No`
>    - *"Examine Genes of Interest"*: `Yes`
>        - *"Genes to Examine"*: `Gstm3, St3gal4, Gna11`
>        - *"Use Defaults?"*: `Yes`
>    - *"Differential Gene Testing"*: `No`
>
{: .hands_on}


![Expression Plot]({% link topics/single-cell/images/scrna-raceid/goi.png %} "RaceID Expression plot of genes of interest across different cells.")

The above figure shows where the combined expression of *Gstm3*, *St3gal4*, and *Gna11* is centred, which from the tSNE plot (top-left) appears to be concentrated in cluster 6. The log expression plot (top-right) changes the scale so that lesser expression in other clusters is still visible. The bottom images provide the same information but using the F-R layout.


> <question-title></question-title>
>
> Observe the above expression plot and the clustering plot generated during the "*Visualising All Clusters*" step.
>
> Are these genes (the top 3 DE genes from cluster 1) expressed where we expect them to be?
>
> {% snippet faqs/galaxy/features_scratchbook.md %}
>
>
> > <solution-title></solution-title>
> >
> > They appear to overlap `c6` which is in close proximity to `c1`. There are two reasons why this might be the case:
> > 1. `c1` is a small cluster and noisy cluster surrounded by more stably defined neighbours.
> > 2. The three genes are more differentially expressed in `c1` than in `c6`, but they are more highly expressed in `c6`. That is, their expression is more significant in `c1` compared to the rest of the genes in that cluster.
> >
> {: .solution}
>
{: .question}


# Trajectory and Lineage Analysis

It was [mentioned previously](#details-continuous-phenotypes-vs-discrete-clustering-methods) that the clusters displayed are not discrete entities, but are related through some continuous topology as inferred by intermediate cell types.

**StemID** is a tool (part of the **RaceID** package) that makes use of this topology to derive a hierarchy of these cell types by constructing a cell lineage tree, rooted at the cluster(s) believed to best describe multipotent progenitor stem cells, and terminating at the clusters which describe more mature cell types. Cell trajectories are identified as a sequence of links between the medoids of different clusters, where the links between clusters are assigned scores that reflect the level of multipotency of the cell type indicated by the cluster.

## Computing the Lineage Tree

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Lineage computation using StemID](toolshed.g2.bx.psu.edu/repos/iuc/raceid_trajectory/raceid_trajectory/0.2.3+galaxy0) %}  with the following parameters:
>    - {% icon param-file %} *"Input RDS"*: `outrdat` (output of **Clustering using RaceID** {% icon tool %})
>    - In *"Compute transcriptome entropy of each cell"*:
>        - *"Use Defaults?"*: `Yes`
>    - In *"Compute Cell Projections for Randomised Background Distribution"*:
>        - *"Use Defaults?"*: `Yes`
>    - In *"StemID2 Lineage Graph"*:
>        - *"Use Defaults?"*: `Yes`
>
> > <comment-title></comment-title>
> >
> > This tool has three main functions:
> > 1. Estimate the entropy (variability) of a cell
> > 1. Estimate links/trajectories between clusters of cells
> > 1. 

tools_json:
["Select first", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/1.1.0", "Transpose", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0", "Text transformation", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sed_tool/1.1.1", "Datamash", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0", "Initial processing using RaceID", "toolshed.g2.bx.psu.edu/repos/iuc/raceid_filtnormconf/raceid_filtnormconf/0.2.3+galaxy0", "Clustering using RaceID", "toolshed.g2.bx.psu.edu/repos/iuc/raceid_clustering/raceid_clustering/0.2.3+galaxy0", "Cluster Inspection using RaceID", "toolshed.g2.bx.psu.edu/repos/iuc/raceid_inspectclusters/raceid_inspectclusters/0.2.3+galaxy0", "Lineage computation using StemID", "toolshed.g2.bx.psu.edu/repos/iuc/raceid_trajectory/raceid_trajectory/0.2.3+galaxy0", "Lineage Branch Analysis using StemID", "toolshed.g2.bx.psu.edu/repos/iuc/raceid_inspecttrajectory/raceid_inspecttrajectory/0.2.3+galaxy0"]


topic: single-cell
tutorial: scrna-case_trajectories
title: Inferring single cell trajectories with Scanpy
difficulty: intermediate
requirements_json: []
tags_json: []
content_hash: 48a66804e4fc759d9cfa3d7c5ba380f8
last_modified: 2026-05-27T09:01:49.747345
zenodo_link: https://zenodo.org/record/7075718
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-case_trajectories/tutorial.html

content:
example-history
    date: 2023-12-14
  - label: "UseGalaxy.eu-ARCHIVED"
    history: https://singlecell.usegalaxy.eu/u/wendi.bacon.training/h/inferring-trajectories-using-scanpy---example-history
    date: 2024-10-12

input_histories:
  - label: "UseGalaxy.eu"
    history: https://usegalaxy.eu/u/wendi.bacon.training/h/cs4inferred-trajectory-analysis-using-python-jupyter-notebook-in-galaxy---input


objectives:
- Execute multiple plotting methods designed to identify lineage relationships between cells
- Interpret these plots
time_estimation: 3H
key_points:
- Trajectory analysis is less robust than pure plotting methods, as such 'inferred relationships' are a bigger mathematical leap
- As always with single-cell analysis, you must know enough biology to deduce if your analysis is reasonable, before exploring or deducing novel insight
requirements:
-
    type: "internal"
    topic_name: single-cell
    tutorials:
        - scrna-case_alevin
        - scrna-case_alevin-combine-datasets
        - scrna-case_basic-pipeline

follow_up_training:
-
    type: "internal"
    topic_name: single-cell
    tutorials:
        - EBI-retrieval
        - GO-enrichment


tags:
- paper-replication
- MIGHTS

contributions:
  authorship:
    - MarisaJL
    - nomadscientist
    - wee-snufkin
    - mtekman
  editing:
    - pavanvidem
    - hexylena
---


You've done all the hard work of preparing a single-cell matrix, processing it, plotting it, interpreting it, and finding lots of lovely genes. Now you want to infer trajectories, or relationships between cells... you can do that here, using the Galaxy interface, or head over to the [Jupyter notebook version of this tutorial]({% link topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.md %}) to learn how to perform the same analysis using Python.

Traditionally, we thought that differentiating or changing cells jumped between discrete states, so 'Cell A' became 'Cell B' as part of its maturation. However, most data shows otherwise. Generally, there is a spectrum (a 'trajectory', if you will...) of small, subtle changes along a pathway of that differentiation. Trying to analyse cells every 10 seconds can be pretty tricky, so 'pseudotime' analysis takes a single sample and assumes that those cells are all on slightly different points along a path of differentiation. Some cells might be slightly more mature and others slightly less, all captured at the same 'time'.  These cells are sorted accordingly along these pseudotime paths of differentiation to build a continuum of cells from one state to the next.  We therefore 'assume' or 'infer' relationships from this continuum of cells.

We will use the same sample from the previous three tutorials, which contains largely T-cells in the thymus. We know T-cells differentiate in the thymus, so we would assume that we would capture cells at slightly different time points within the same sample. Furthermore, our cluster analysis alone showed different states of T-cells. Now it's time to look further!

> <comment-title>Tutorial from Scanpy</comment-title>
> Please note, this tutorial is largely based on the trajectories tutorial found [on the Scanpy site itself](https://scanpy.readthedocs.io/en/stable/tutorials/trajectories/paga-paul15.html) ({% cite scanpytutorialsTrajectoryInference %}).
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Important tips for easier analysis

{% snippet faqs/galaxy/tutorial_mode.md %}

{% snippet topics/single-cell/faqs/single_cell_omics.md %}

{% snippet faqs/galaxy/analysis_troubleshooting.md sc=true %}

# Prepare datasets

## Get data

We've provided you with experimental data to analyse from a mouse dataset of fetal growth restriction {% cite Bacon2018 %}. This is the full dataset generated from [this tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) (see the [study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and the [project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)). You can find the final dataset in this [input history](https://usegalaxy.eu/u/wendi.bacon.training/h/cs4inferred-trajectory-analysis-using-python-jupyter-notebook-in-galaxy---input) or download from Zenodo below.

{% include _includes/cyoa-choices.html option1="History_import" option2="Zenodo_import" default="History_import"
       text="Importing via History is quickest. Works only on Galaxy EU for now." %}

<div class="History_import" markdown="1">
> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. Import history from: [input history](https://usegalaxy.eu/u/wendi.bacon.training/h/cs4inferred-trajectory-analysis-using-python-jupyter-notebook-in-galaxy---input)
>
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the the history to your name of choice.
>
{: .hands_on}
</div>

<div class="Zenodo_import" markdown="1">
> <hands-on-title>Option 2: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/Final_cell_annotated_object.h5ad
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} the .h5ad object as `Final cell annotated object`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Final cell annotated object" %}
>
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}
</div>

## Filtering for T-cells

One problem with our current dataset is that it's not just T-cells: we found in the previous tutorial that it also contains macrophages. This is a problem, because trajectory analysis will generally try to find relationships between all the cells in the sample. We need to remove those cell types to analyse the trajectory.

> <hands-on-title> Removing macrophages </hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Final cell annotated object` (Input dataset)
>    - *"Function to manipulate the object"*: `Filter observations or variables`
>        - *"What to filter?"*: `Observations (obs)`
>        - *"Type of filtering?"*: `By key (column) values`
>            - *"Key to filter"*: `cell_type`
>            - *"Type of value to filter"*: `Text`
>                - *"Filter"*: `not equal to`
>                - *"Value"*: `Macrophages`
>                -
> 3. **Rename** {% icon galaxy-pencil %} output h5ad `T-cell_object.h5ad`
>
{: .hands_on}

You should now have `8569` cells, as opposed to the `8605` you started with. You've only removed a few cells (the contaminants!), but it makes a big difference in the next steps.

# Force-directed graph

First, we will calculate a [force-directed graph](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.draw_graph.html) (FDG), as an alternate to tSNE, which will likely work better for trajectory analysis.

## Calculate force-directed graph

> <hands-on-title> Draw FDG </hands-on-title>
>
> 1. {% tool [Scanpy RunFDG](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_fdg/scanpy_run_fdg/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `T-cell_object.h5ad` (output of **Manipulate AnnData** {% icon tool %})
>    - *"Use programme defaults"*: {% icon galaxy-toggle %} `No`
>    - *"Graph layout"*: `fa`
>
>    > <comment-title> Graph Layout </comment-title>
>    >
>    > We're using the fa or ForceAtlas2 layout for our FDGs. It is the same layout used in the [Jupyter notebook version of this tutorial]({% link topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.md %}) and works well for our data. As well as choosing the fa layout when we create the FDGs, we will also specify the `draw_graph_fa` embedding when drawing the plots.
>    {: .comment}
>
{: .hands_on}

## Plot the FDG

And now time to plot it!

> <hands-on-title> Plot the FDG </hands-on-title>
>
> 1. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"name of the embedding to plot"*: `draw_graph_fa`
>    - *"color by attributes, comma separated texts"*: `cell_type`
>    - *"Use raw attributes"*: {% icon galaxy-toggle %} `No`
>    - *"Location of legend"*: `On data`
{: .hands_on}


> <question-title></question-title>
>
> 1. What has the FDG done to our clusters of T-cells and what might this suggest about the relationships between these groups?
>
> > <solution-title></solution-title>
> >
> > 1. Well now this is exciting! Our DP-late is more clearly separating, and we might also suppose that DP-M1, DP-M2, and DP-M3 are actually earlier on in the differentiation towards mature T-cells. And we're only just getting started!
> >
> > ![Plot showing the T Mat and DP L clusters have separated from the rest of the cells](../../images/scrna-case_trajectories/TrajectoriesFDG1.png "FDG Plot of our T-cell dataset")
> {: .solution}
>
{: .question}

# Diffusion maps

We'll now perform an *optional step*, that basically takes the place of the standard Principle Component Analysis (PCA). Instead of using PCs, we can use [diffusion maps](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.diffmap.html).

## Draw diffusion map

> <hands-on-title> Draw the Diffusion Map </hands-on-title>
>
> 1. {% tool [Scanpy DiffusionMap](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_diffmap/scanpy_run_diffmap/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"Number of diffusion components to calculate"*: `15`
>
>    > <comment-title> Choosing the number of diffusion components </comment-title>
>    >
>    > We could change the number of diffusion components and end up with a slightly different plot - a bit like if we changed the number of principal components used in the PCA we ran in the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial. 15 seems to work well for this dataset and matches the number used in the Jupyter version of this tutorial, so we'll stick with that.
>    {: .comment}
>
{: .hands_on}

## Re-calculate Nearest Neighbours

Now that we have our diffusion map, we need to re-calculate neighbors using the diffusion map instead of the PCs.

> <hands-on-title> Compute neighbours using diffusion map </hands-on-title>
>
> 1. {% tool [Scanpy ComputeGraph](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `DiffusionMap Anndata` (output of **Scanpy DiffusionMap** {% icon tool %})
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"Use the indicated representation"*: `X_diffmap`
>
{: .hands_on}

> <comment-title></comment-title>
> If you're using the latest versions of these tools (e.g. {% tool [Scanpy ComputeGraph](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.9.3+galaxy0) %}, rather than the ones suggested in the tutorial (e.g. {% tool [Scanpy ComputeGraph](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.8.1+galaxy9) %} then you may need to change one more parameter here to set the `Number of PCs to use` to 15. These are the 15 diffusion components we just calculated, rather than actual PCs.
{: .comment}

## Re-draw the FDG

Now that we've re-calculated the nearest neighbours, we can use these new neighbours to re-draw the FDG to see how this changes the plot.

> <hands-on-title> Plot a new FDG </hands-on-title>
>
> 1. {% tool [Scanpy RunFDG](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_fdg/scanpy_run_fdg/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `Graph object Anndata` (output of **Scanpy ComputeGraph** {% icon tool %})
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"Graph layout"*: `fa`
>
>
> 2. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"name of the embedding to plot"*: `draw_graph_fa`
>    - *"color by attributes, comma separated texts"*: `cell_type`
>    - *"Use raw attributes"*: `No`
>    - *"Location of legend"*: `On data`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Does this plot seem better or worse than before? Remember that we're trying to understand the relationships between our groups of cells in time.
>
> > <solution-title></solution-title>
> >
> > 1. Oh dear! This doesn't look great. Maybe the DP-M4 cells are a whole other trajectory? That doesn't seem right. Saying that, this does spread out our T-mature cells, which makes a lot more sense when it comes to T-cell biology (we expect T-cells to differentiate into two types of T-cells, Cd8+Cd4- and Cd4+Cd8-). If you wanted to, you could also re-cluster your cells (since you've changed the neighborhood graph on which the clusterisation depends). However, we tried that, and it called far too many clusters given the depth of sequencing in this dataset. Let's stick with our known cell types and move from there.
> >
> > ![The T mature and DP L clusters have been stretched out](../../images/scrna-case_trajectories/TrajectoriesFDG2.png "FDG Plot after recalculating neighbours from the diffusion map")
> {: .solution}
>
{: .question}

> <details-title>Working in a group? Decision-time!</details-title>
> If you are working in a group, you can now divide up a decision here with one *control* and the rest can vary in numbers so that you can compare results throughout the tutorials.
> - Control
>   - Go straight to the [Partition-based Graph Abstraction (PAGA) section](#partition-based-graph-abstraction-paga)
> - Everyone else:
>   - you could recluster your cells using {% tool [Scanpy FindCluster](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_find_cluster/scanpy_find_cluster/1.8.1+galaxy0) %} at a different resolution, perhaps lower than the 0.6 we used before (Take a look at the Cell clusters step in the [Filter, Plot and Explore]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial if you need help with this.) Please note that in this case, you will want to change the PAGA step **Scanpy PAGA** to group by `louvain` rather than `cell_type`. You can certainly still plot both, we only didn't because with using our old Louvain calls, the `cell_type` and `louvain` categories are identical.
>   - you could undo the optional diffusion map step by recalculating the neighbours again using `X_pca` instead of `X_diffmap`
>   - you could also try changing the number of neighbors used in that step when running {% tool [Scanpy ComputeGraph](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.8.1+galaxy9) %}
>
> - Everyone else: You will want to compare FREQUENTLY with your control team member.
{: .details}

# Partition-based Graph Abstraction (PAGA)

[PAGA](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.paga.html) is used to generalise relationships between groups, or likely clusters, in this case. It will make it much easier to see the trajectories between our clusters of T-cells.

> <hands-on-title> Plot PAGA </hands-on-title>
>
> 1. {% tool [Scanpy PAGA](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_paga/scanpy_run_paga/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"Name of the clustering"*: `cell_type`
>
>    > <comment-title> Plotting gene expression </comment-title>
>    >
>    > We can now draw our PAGA plot and we might also be interested in colouring our plot by genes as well. In this case, remembering that we are dutifully counting our genes by their EnsemblIDs rather than Symbols (which do not exist for all EnsemblIDs), we have to look up our genes of interest (CD4, CD8a) and plot the corresponding IDs in the next step.
>    {: .comment}
>
> 2. {% tool [Scanpy PlotTrajectory](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_trajectory/scanpy_plot_trajectory/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `PAGA object Anndata` (output of **Scanpy PAGA** {% icon tool %})
>    - *"Layout functions"*: `ForceAtlas2`
>    - *"Location of legend"*: `On data`
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"Name of cell annotation or gene that is used to color the nodes"*: `cell_type,ENSMUSG00000023274,ENSMUSG00000053977`
>
>
>    > <comment-title> Choosing the layout </comment-title>
>    >
>    > We're going to pick the `ForceAtlas2` layout function for our PAGA plots as this is the same type of layout that we used for our FDG.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How have the relationships between our cell clusters changed now?
> 2. Which clusters are expressing our genes of interest, Cd4 and Cd8, at the highest levels?
>
> > <solution-title></solution-title>
> >
> > 1. The way the clusters are arranged has changed a bit now. The M4 cluster is right in the middle of the M1-3 clusters, rather than heading off on its own. The M1 cluster is looking like it is driving towards differentiation, which is not something we had necessarily been able to specify before by just looking at our cluster graphs or applying our biological knowledge.
> > 2. Cd4 and Cd8 expression appear highest in the DP-L cluster. The expression of both Cd4 and Cd8 also appears higher than we might expect in the DP-M4 cluster - perhaps this is a sign that it is closer to the DP-L cluster than it seems in this simple plot.
> >
> > ![DN cluster is on one side of the plot, with the DP L and T mature clusters on the other side. The other clusters are close together in the centre of the plot. Coloured to show higher Cd4 and Cd8 expression in the M4, DP L and T mature clusters.](../../images/scrna-case_trajectories/TrajectoriesPAGA.png "PAGA plots coloured by cell type, Cd4 expression, and Cd8 expression")
> {: .solution}
>
{: .question}

## Re-draw force-directed graph (again!)

Force-directed graphs can be initialised randomly, or we can prod it in the right direction. We'll prod it with our PAGA calculations. Note that you could also try prodding it with tSNE or UMAP. A lot of these tools can be used on top of each other or with each other in different ways, this tutorial is just one example. Similarly, you could be using any **obs** information for grouping, so could do this for *louvain* or *cell_type* for instance.

> <hands-on-title> Initialise FDG using PAGA </hands-on-title>
>
> 1. {% tool [Scanpy RunFDG](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_fdg/scanpy_run_fdg/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `Plotted PAGA Anndata` (output of **Scanpy PlotTrajectory** {% icon tool %})
>    - *"Use programme defaults"*: {% icon param-toggle %} `No`
>    - *"Method to initialise embedding, any key for adata.obsm or choose from the preset methods"*: `paga`
>    - *"Graph layout"*: `fa`
>
> 2. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"name of the embedding to plot"*: `draw_graph_fa`
>    - *"color by attributes, comma separated texts"*: `cell_type`
>    - *"Use raw attributes"*: `No`
>    - *"Location of legend"*: `On data`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How has basing our FDG on the PAGA plot changed the relationships between our cells?
>
> > <solution-title></solution-title>
> >
> > 1. The interesting change here occurs between the Double Negative (DN) and Double Positive Module 4 (DP-M4) cells. Our DP-M4 cells are now heading on a clear trajectory towards differentiation. It looks like we've got the correct ordering of cells from DN through the DP groups and on towards T-mature. We didn't see this in our previous plots.
> >
> > ![The M4 cluster is now closer to the DP L cluster](../../images/scrna-case_trajectories/TrajectoriesFDG3.png "FDG plot initiated from the PAGA plot")
> {: .solution}
>
{: .question}

# Exploring the results

The experiment that produced this data used two different groups of mice - the control or wildtype group and the knockout mice that were missing a gene involved in placental development, which impacted thymus development. Since we know the genotype of the mice from which each sample was collected, we can colour in our plots to see if there are any differences in the cells present in wildtype and knockout mice.

## Plotting by Genotype

The easiest way to do this is just to rerun {% icon galaxy-refresh %} the previous step, but change the attribute we want to use to colour the FDG plot.

> <hands-on-title> Plot by genotype </hands-on-title>
>
> 1. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"name of the embedding to plot"*: `draw_graph_fa`
>    - *"color by attributes, comma separated texts"*: `genotype`
>    - *"Use raw attributes"*: `No`
>    - *"Location of legend"*: `Right margin`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Are there any differences in the distribution of the wildtype and knockout cells?
>
> > <solution-title></solution-title>
> >
> > 1. We’re seeing a clear trajectory issue whereby the knockout cells are not found along the trajectory into T-mature (which, well, we kind of already figured out with just the cluster analysis, but we can feel even more confident about our results!)
> >
> > ![Most clusters are coloured as a mix of both genotypes but the DP L and T mature clusters are mainly coloured as wildtype cells](../../images/scrna-case_trajectories/TrajectoriesFDGGenotype.png "FDG plot coloured by genotype")
> {: .solution}
>
{: .question}

## Plotting Gene Expression

We're also interested in the expression of the two genes that are known to be markers of the two different types of mature T-cells: Cd4 and Cd8. We can colour in our plot to show which cells are expressing these genes.

> <hands-on-title> Plot for gene expression </hands-on-title>
>
> 1. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"name of the embedding to plot"*: `draw_graph_fa`
>    - *"color by attributes, comma separated texts"*: `ENSMUSG00000023274,ENSMUSG00000053977`
>    - *"Use raw attributes"*: `No`
>    - *"Location of legend"*: `On data`
>
>    > <comment-title> Gene Symbols </comment-title>
>    >
>    > We're using the EnsemblIDs during this tutorial, as discussed above. If you like, you could change the names of these plots to the gene symbols by filling in the optional *"Figure title"* field with `Cd4,Cd8`. Make sure that the order of your figure titles matches the order of the EnsemblIDs in the colour by field. ENSMUSG00000023274 is Cd4 and ENSMUSG00000053977 is Cd8.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Does the expression pattern of these genes tell us anything about our cells?
>
> > <solution-title></solution-title>
> >
> > 1. It's clear that both Cd4 and Cd8 are being expressed mainly in the later stages of T-cell development, as they head towards the DP-L cluster - although Cd4 expression is a bit more widespread. This is what we would expect to see in genes that are associated with mature T-cells. You might also be able to spot some differences in the expression of the two genes in our mature T-cell group, but there doesn't seem to be a very clear division between the cells that express Cd4 and those that express Cd8. This is a bit disappointing, as we know that there are two types of mature T-cells, which each express a different gene.
> >
> > ![Cells in the DP L cluster are coloured as having the highest expression levels of both Cd4 and Cd8 while most cells in the M4 cluster already show higher expression of Cd8](../../images/scrna-case_trajectories/TrajectoriesFDGExpression.png "FDG plot showing expression of Cd4 and Cd8")
> {: .solution}
>
{: .question}

# Diffusion pseudotime

Now that we have a reasonable FDG plot for our cells, based on the diffusion map (if used) and PAGA plot, we can place our cells into pseudotime. Pseudotime lets us imagine that instead of looking at a sample of cells taken at a single timepoint, we are looking at cells moving through time. Our sample included cells at different stages of their development, but we can use pseudotime to think of these as different timepoints in the journey of individual cells.

We know that our cells are initialising at DN. We can feed that information into our algorithms by naming DN as the root cell type to then calculate a trajectory starting from these cells.

> <details-title>Working in a group? Decision-time!</details-title>
>
> If you called new clusters using {% tool [Scanpy FindCluster](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_find_cluster/scanpy_find_cluster/1.8.1+galaxy0) %}, you might want to choose one of those clusters to be your root cell instead, so change the `cell_type` for `louvain` and then name the cluster number. Use the plots you created to help you pick the number!
{: .details}

On to the [diffusion pseudotime](https://scanpy.readthedocs.io/en/stable/api/scanpy.tl.dpt.html), where we infer multiple time points within the same piece of data!

> <hands-on-title> DPT Plot </hands-on-title>
>
> 1. {% tool [Scanpy DPT](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_dpt/scanpy_run_dpt/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `FDG object Anndata` (output of **Scanpy RunFDG** {% icon tool %})
>    - *"Name of attribute that defines clustering"*: `cell_type`
>    - *"Name of the clustering that defines the root cell type"*: `DN`
>
>
> 2. {% tool [Scanpy PlotEmbed](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `Diffusion pseudotime inference Anndata` (output of **Scanpy DPT** {% icon tool %})
>    - *"name of the embedding to plot"*: `draw_graph_fa`
>    - *"color by attributes, comma separated texts"*: `cell_type,dpt_pseudotime`
>    - *"Use raw attributes"*: `No`
>    - *"Location of legend"*: `On data`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Does the pseudotime plot match with your expectations of which cells represent earlier or later stages of T-cell development?
>
> > <solution-title></solution-title>
> >
> > 1. When we look at the cell type and DPT plots, we can see that there's a clear progression from our root DN cells, through the various groups of DP cells, into DP-L and then T-mat. This matches with our expectations that the DP-L and DP-mat clusters represent later stages in T-cell development. The DPT plot also confirms that the DP-M4 cluster is heading towards differentiation, which makes sense given its position on our FDG plot.
> >
> > ![Cells in DN cluster are coloured as earliest in pseudotime, which increases through clusters M1 to M4, into cluster DP L and is latest in T mature](../../images/scrna-case_trajectories/TrajectoriesFDGPseudotime.png "FDG plots showing cell types and pseudotime")
> {: .solution}
>
{: .question}

This is nice, as it supports our conclusions thus far on the trajectory of the T-cell differentiation. With single-cell, the more ways you can prove to yourself what you're seeing is real, the better! If we did not find consistent results, we would need to delve in further to see if the cause is the algorithm (not all algorithms fit all data!) or the biology.

Where might we go from here? We might consider playing with our louvain resolutions, to see if we can get the two groups of Cd4+ and Cd8+ cells to be called as different clusters, and then comparing them to each other for gene differences or genotype differences. We might also use different objects (for instance, what if we regressed out cell cycle genes?) and see if that changes the results. What would you do?

> <details-title>Working in a group? The Finale!</details-title>
> Look at each others images! How do yours differ, what decisions were made? Previously, when calling clusters in the [Filter, Plot and Explore Single-cell RNA-seq Data]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}) tutorial, the interpretation at the end is largely consistent, no matter what decisions are made throughout (mostly!). Is this the case with your trajectory analyses? You may find that it is not, which is why pseudotime analysis even more crucially depends on your understanding of the underlying biology (we have to choose the root cells, for instance, or recognise that DN cells should not be found in the middle of the DPs) as well as choosing the right analysis. That's why it is a huge field! With analysing scRNA-seq data, it's almost like you need to know about 75% of your data and make sure your analysis shows that, for you to then identify the 25% new information.
{: .details}

# Conclusion

Congratulations! You've made it to the end! You might be interested in the {% icon workflow %}[workflow](https://humancellatlas.usegalaxy.eu/u/marisa_jl/w/workflow-constructed-from-history-inferring-trajectories-with-scanpy-tutorial-1) for this tutorial or this {% icon galaxy-history %}[Example History](https://humancellatlas.usegalaxy.eu/u/marisa_jl/h/inferring-trajectories-using-scanpy---example-history) which shows the results you should expect to see if you follow this tutorial.

In this tutorial, you moved from called clusters to inferred relationships and trajectories using pseudotime analysis. You found an alternative to PCA (diffusion map), an alternative to tSNE (force-directed graph), a means of identifying cluster relationships (PAGA), and a metric for pseudotime (diffusion pseudotime) to identify early and late cells. If you were working in a group, you found that such analysis is slightly more sensitive to your decisions than the simpler filtering/plotting/clustering is. We are inferring and assuming relationships and time, so that makes sense!

tools_json:
["Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.7.5+galaxy1", "Scanpy RunFDG", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_fdg/scanpy_run_fdg/1.8.1+galaxy9", "Scanpy PlotEmbed", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_embed/scanpy_plot_embed/1.8.1+galaxy9", "Scanpy DiffusionMap", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_diffmap/scanpy_run_diffmap/1.8.1+galaxy9", "Scanpy ComputeGraph", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.8.1+galaxy9", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_compute_graph/scanpy_compute_graph/1.9.3+galaxy0", "Scanpy FindCluster", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_find_cluster/scanpy_find_cluster/1.8.1+galaxy0", "Scanpy PAGA", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_paga/scanpy_run_paga/1.8.1+galaxy9", "Scanpy PlotTrajectory", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_plot_trajectory/scanpy_plot_trajectory/1.8.1+galaxy9", "Scanpy DPT", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_run_dpt/scanpy_run_dpt/1.8.1+galaxy9"]


topic: single-cell
tutorial: scrna-scater-qc
title: Single-cell quality control with scater
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["[]"]
content_hash: 94ca86e710144c46b3f7479f2ee1f01d
last_modified: 2026-05-15T22:28:16.825411
zenodo_link: https://zenodo.org/record/3386291
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-scater-qc/tutorial.html

content:
Single-cell RNA-seq (scRNA-seq) is emerging as a promising technology for analysing variability in cell populations. However, the combination of technical noise and intrinsic biological variability makes detecting technical artefacts particularly challenging. Removal of low-quality cells and detection of technical artefacts is critical for accurate downstream analysis.

A number of factors should be examined before downstream analyses, many of which we'll address here:

- **Low library size**: When cells are very degraded or absent from the library preparation, the number of reads sequenced from that library will be very low. It's important to remove these cells from downstream analyses.

- **Low number of expressed genes**: A low number of expressed genes may be a result of poor-quality cells (e.g. dying, degraded, damaged, etc.), followed by high PCR amplification of the remaining RNA. Again, these cells should be removed from downstream analyses.

- **High mitochondrial gene content**: High concentrations of mitochondrial genes is often a result of damaged cells where the endogenous RNA escapes or degrades. As mitochondria has its own cell membranes, it is often the last DNA/RNA in damaged cells to degrade and hence occurs in high quantities during sequencing.

- **Batch effect**: Large scRNA-seq projects usually need to generate data across multiple batches due to logistical constraints. However, the processing of different batches is often subject to variation, e.g., changes in operator, differences in reagent quality and concentration, the sequencing machine used, etc. This results in systematic differences in the observed expression in cells from different batches, which we refer to as “batch effects”. Batch effects are problematic as they can be major drivers of variation in the data, masking the relevant biological differences and complicating interpretation of the results.

We will use *scater* ({% cite 10.1093/bioinformatics/btw777 %}) to visualise scRNA-seq data, obtaining information about the factors mentioned above, filter out low-quality cells and confirm that filtering has worked. We'll then look at confounding factors such as batch effect to see if the data is biased to any technical artifacts.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}



# Overview
We will use a pre-calculated expression matrix, along with some additional metadata such as lists of mitochondrial genes and annotation of technical information for each sequencing library. We will plot the data and carry out quality control filtering based on the visualise-filter-visualise paradigm of this approach (see workflow below).

![Workflow](../../images/scrna-scater-qc/Figure1.jpg "Workflow demonstrating the visualise-filter-visualise paradigm of the software")


# Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo](https://zenodo.org/record/3386291) or from the shared data library
>
>    ```
>    https://zenodo.org/record/3386291/files/annotation.txt
>    https://zenodo.org/record/3386291/files/counts.txt
>    https://zenodo.org/record/3386291/files/mt_controls.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Visualise the data

Take a look at the uploaded data by clicking on the {% icon galaxy-eye %} symbol for each dataset.

- The `counts.txt` file is a 40-sample expression matrix. Each sample (Cell_001 - Cell_040) is listed as the column headers and the start of each row is a gene name. The rest of the data refers to the number of reads mapped to each gene/sample.
- `annotation.txt` is a file listing experimental information about each cell. Parameters here include `Mutation_Status`, `Cell_Cycle`, and `Treatment`. These will be useful for looking at  batch effects later.
- The `mt_controls.txt` file is a list of mitochondrial genes. This list will be used later to calculate the % of mitochondrial reads in each sequencing library.


> <hands-on-title>Calculate QC metrics</hands-on-title>
>
> 1. {% tool [Scater: Calculate QC metrics](toolshed.g2.bx.psu.edu/repos/iuc/scater_create_qcmetric_ready_sce/scater_create_qcmetric_ready_sce/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Expression matrix in tabular format"*: `counts.txt` (Input dataset)
>    - {% icon param-file %} *"Format dataset describing the features in tabular format"*: `annotation.txt` (Input dataset)
>    - {% icon param-file %} *"Dataset containing the list of the mitochondrial control genes"*: `mt_controls.txt` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > The output of this tool is a SingleCellExperiment object in [Loom](http://loompy.org/) format, which contains all the information from the input files, along with a host of other quality control metrics, calculated from the input data.
>    {: .comment}
>
{: .hands_on}

Next, lets take a look at the data by plotting various properties to see what our data looks like.

> <hands-on-title>Plot library QC</hands-on-title>
>
> 1. {% tool [Scater: plot library QC](toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_dist_scatter/scater_plot_dist_scatter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: Calculate QC metrics** {% icon tool %})
>     - {% icon param-check %} *"Plot on log scale"*: `No`
>
> 2. If we have a large number of cells (500+), set the 'Plot on log scale' option to 'Yes'. This will make it easier to pick cut-offs when dealing with large numbers. When the tool has finished running, click on the {% icon galaxy-eye %} to view the plots. If it doesn't appear in the browser, you may have to download it and view it externally. You should be presented with plots similar to those below.
>    ![Raw data QC plots](../../images/scrna-scater-qc/raw_data.png "Raw data QC plots")
>
>    > <comment-title></comment-title>
>    >
>    > There are four plots, two distribution bar plots and two scatter plots.
>    > The first distribution plot is the number of reads in each library (from a single cell).
>    > The second plot is the distribution of *feature counts* per cell. Feature counts in this case refers to the number of genes expressed in each cell.
>    > The third plot **Scatterplot of reads vs genes** is a combination of the two barplots, in that it plots both the read count and the expressed gene count for each cell.
>    > The final scatterplot is the **% MT genes**, which plots the number of genes expressed verses the percentage of those genes that are mitochondrial genes.
>    {: .comment}
>
>    Let's look at each plot in turn and see what it tells us.
>    - **Read counts**. You can see that there are a few cells that have less than ~200,000 reads, with other cells having up to one million reads. Although 200,000 reads is still quite a lot and we wouldn't want to get rid of so much data, we might want to think about removing cells that only contain a smaller number of reads (say, 100,000).
>    - **Feature counts**. Similar to the read counts plot, we see a few cells that have a very low number of expressed genes (<600), then followed by a more even distribution.
>    - **Scatterplot of reads vs genes**. This takes the information provided in the two distribution plots above and creates a scatterplot from them. The really poor-quality cells are represented by the points near the intersection of the x and y axis, being data with low read count and low gene count. These are the cells we want to remove during filtering.
>    - **% MT genes**. You can see from the plot that there are a few cells outside the main "cloud" of datapoints. Some of these could be removed by filtering out cells with low feature counts, but others might need to be removed by mitochondrial content, such as the cell around 37.5%
>
{: .hands_on}

# Filtering

## Manual filtering

In the **Scater: filter SCE** Galaxy tool there are two filtering methods available. First, there's an "automatic" filtering method that uses PCA to identify outliers cells and remove them from the data. This is particularly useful for very large datasets (hundreds of samples). Second, there's a manual filtering method where users can put a range of filtering parameters, informed by the previous plotting tool.
Here, we'll use the manual filtering method.

> <hands-on-title>Filtering with scater</hands-on-title>
>
> 1. {% tool [Scater: filter SCE](toolshed.g2.bx.psu.edu/repos/iuc/scater_filter/scater_filter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: Calculate QC metrics** {% icon tool %})
>    - *"Type of filter"*: `manual`
>        - *"Number of reads mapped to a gene for it to be counted as expressed"*: `4.0`
>        - *"Minimum library size (mapped reads) to filter cells on"*: `100000`
>        - *"Minimum number of expressed genes to filter cells on"*: `500`
>        - *"Maximum % of mitochondrial genes expressed per cell"*: `35.0`
>
>    > <comment-title></comment-title>
>    >
>    > Let's have a look at the parameters and their values:
>    > 1. *"Number of reads mapped to a gene for it to be counted as expressed"*: by default, only one read needs to be mapped to a gene for it to be counted as "expressed". We can be a little bit more stringent here and increase the number of reads that need to be mapped to a gene for it to be categorised as "expressed".
>    > 2. *"Minimum library size (mapped reads) to filter cells on"*: This value asks how many mapped reads from each cell do you require to be mapped to your genome to be included in downstream analysis. We can see from our plots that we have a few cells that have less than 200,000 reads. 200,000 can still be quite a lot of reads (depending on the experiment), but we can use a smaller number to see what the initial effect of filtering is. Initially, use 100,000 as the value here.
>    > 3. *"Minimum number of expressed genes"*: You can see that some cells only express a few hundred genes, so we'll remove these cells also.
>    > 4. *"Maximum % of mitochondrial genes expressed per cell"*. You can see that as well as having one obvious outlier (~37%).
>    {: .comment}
>
>
{: .hands_on}


> <hands-on-title>Plot library QC after filtering</hands-on-title>
>
> 1. {% tool [Scater: plot library QC](toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_dist_scatter/scater_plot_dist_scatter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: filter SCE** {% icon tool %})
>
{: .hands_on}

![QC plots after manual filtering](../../images/scrna-scater-qc/post_filter.png "QC plots after manual filtering")

> <comment-title></comment-title>
>
> How did the filtering go? Do you think it's done a good job? Have you removed too many cells? Too few cells? About right?
>
> Often, it's a matter of trial and error, where you would start off by being quite lenient (low parameters) and then increasing the stringency until you're happy with the results. Using the initial Calculate QC metrics file, play around with the filtering parameters and visualise the output to see the effect different paramters have.
>
{: .comment}

## Automatic (PCA) filtering

Another filtering approach is to identify outliers in the data and remove them. PCA can be run once a SingleCellExperiment object has been normalised, and outliers cells identified based on the pre-computed quality control metrics within the SingleCellExperiment object.

As we are using a rather small test dataset, it's unlikely that PCA filtering will make any difference; for a larger, noisier dataset this is what we would perform instead:

> <hands-on-title>Task description</hands-on-title>
> 1. {% tool [Scater: filter SCE](toolshed.g2.bx.psu.edu/repos/iuc/scater_filter/scater_filter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: Calculate QC metrics** {% icon tool %})
>    - *"Type of filter"*: `automatic`
>
>    > <comment-title></comment-title>
>    >
>    > The data will be normalised and then PCA ran on it using the following information from the data:
>    > * `pct_counts_top_100_features`
>    > * `total_features_by_counts`
>    > * `pct_counts_feature_control`
>    > * `total_features_feature_control`
>    > * `log10_total_counts_endogenous`
>    > * `log10_total_counts_feature_control`
>    >
>    > When using these filtering approaches, it is sometimes good to try running them in reverse - try PCA filtering first and then if that doesn't remove enough low-quality cells then use the manual filtering. We could actually pipeline them together - use PCA filtering first and then use the output of that to do further manual filtering.
>    {: .comment}
>
{: .hands_on}

As discussed previously, technical artefacts can bias scRNA-seq analyses. Strong batch effects can mask real biological differences in the data, so must be identified and removed from the data. Logging meta-data details such as date of library construction, sequencing batch, sample name, technical replicate, plate number, etc., is essential to identify batch effects in the data. We can use this information to visualise the data to examine it for clustering according to batch, rather than any real biological feature.

> <hands-on-title>PCA plot</hands-on-title>
> 1. {% tool [Scater: PCA plot](toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_pca/scater_plot_pca/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: filter SCE** {% icon tool %})
>    - *"Feature (from annotation file) to colour PCA plot points by"*: `Mutation_Status`
>    - *"Feature (from annotation file) to shape PCA plot points by"*: `Cell_Cycle`
>
>
{: .hands_on}

![PCA plot](../../images/scrna-scater-qc/PCAplot.png "PCA plot")

> <question-title></question-title>
>
> 1. Do you see any obvious batch effect?
> 2. Do any of the categories suggesting some sort of technical artefact?
>
> > <solution-title></solution-title>
> >
> > 1. We can see that the **S** and **G2M** categories in **Cell\_Cycle** cluster away from the rest of the data. Spend some time thinking about whether this might be a batch effect or biologically significant.
> >
> {: .solution}
>
{: .question}


# Conclusion

We have gone through the process of filtering low-quality data from an scRNA-seq expression matrix, using the visualise-filter-visualise paradigm, which proves to be a very effective way of quality-controlling scRNA-seq data. Cells that have low read-coverage, low expression values, or high mitochondrial gene expression have been filtered out. We have then examined ways of looking at confounding factors to examine batch effects in our data.
The workflow available from the "Supporting Materials" of this tutorial can be directly imported and used or adapted to a specific analysis.

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %} and {% cite etherington2019galaxy %})

tools_json:
["Scater: Calculate QC metrics", "toolshed.g2.bx.psu.edu/repos/iuc/scater_create_qcmetric_ready_sce/scater_create_qcmetric_ready_sce/1.12.2", "Scater: plot library QC", "toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_dist_scatter/scater_plot_dist_scatter/1.12.2", "Scater: filter SCE", "toolshed.g2.bx.psu.edu/repos/iuc/scater_filter/scater_filter/1.12.2", "Scater: PCA plot", "toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_pca/scater_plot_pca/1.12.2"]


topic: single-cell
tutorial: bulk-music
title: Bulk RNA Deconvolution with MuSiC
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: ["transcriptomics"]
content_hash: a7505b50470a9e3a94f20273a6a6cffa
last_modified: 2026-05-15T22:28:16.661203
zenodo_link: https://zenodo.org/record/5719228
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music/tutorial.html

content:
Bulk RNA-seq data contains a mixture of transcript signatures from several types of cells. We wish to deconvolve this mixture to obtain estimates of the proportions of cell types within the bulk sample. To do this, we can use single cell RNA-seq data as a reference for estimating the cell type proportions within the bulk data.

<!-- using info from here: https://xuranw.github.io/MuSiC/articles/MuSiC.html -->

In this tutorial, we will use bulk and single-cell RNA-seq data, including matrices of similar tissues from different sources, to illustrate how to infer cell type abundances from bulk RNA-seq.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Bulk RNA-seq Cell Type Deconvolution

The heterogeneity that exists in the cellular composition of bulk RNA-seq can add bias to the results from differential expression analysis. In order to circumvent this limitation, RNA-seq deconvolution aims to infer cell type abundances by modelling the gene expressions levels as 'weighted sums' of cell type specific expression profiles.

> <details-title>More details on 'Sums'</details-title>
>
> **So...** You fancy some maths do you? Good! This is important, as you'll see variations of the phrase 'weighted sums' if you ever look at any papers in the field! Let's think about just the 'sums' here.
> If we think about the total expression of a given gene you might get from bulk RNA-seq, you could also think about it as the sum of the expression of each cell, for example,
>
> *Total = cell<sub>a</sub> expression + cell<sub>b</sub> expression ... (and so forth)*
>
> This is a *'sum'*, because we're adding up all the cells. If we now think about how we expect similar expression from similar cell types, we could change this to the following:
>
> *Total = cell_type<sub>a</sub> expression + cell_type<sub>b</sub> expression*  ... (and so forth)
>
> **BUT WAIT!** That's not strictly true, because the cell types are not all in equal proportion (if only!). So we have to take into account another variable, the proportion a given cell type takes up in a sample. So we now have:
>
> *Total = Proportion<sub>a</sub> x cell_type<sub>a</sub> expression + Proportion<sub>b</sub> x cell_type<sub>b</sub> expression*
>
> So now, we are using the sums of expression based off of the cell proportion. And if you're a mathematician, you might instead put this as
>
> **T = C x P**
> or some fancier formulas...(read more [in-depth here](https://academic.oup.com/bioinformatics/article/34/11/1969/4813737) if you like!)
>
> The point is, if we have an idea of what the average expression should be for each gene (what we can get from single cell RNA-seq data, *C*), and we have the total expression (from the bulk RNA-seq, *T*), then we can infer the cell proportions (*P*).
>
{: .details}

Many different computational methods have been developed to estimate these cell type proportions, but in this tutorial we will be using the [MuSiC](https://xuranw.github.io/MuSiC/articles/MuSiC.html) tool suite ({% cite wang2019bulk %}) to estimate the proportion of individual cell types in our bulk RNA-seq datasets.

### MusiC

MuSiC uses cell-type specific gene expression from single-cell RNA seq data to characterize cell type compositions (proportions) from bulk RNA-seq data in complex tissues. By appropriate weighting of genes showing cross-subject (sample to sample) and cross-cell (cells of the same cell type within a sample) consistency, MuSiC enables the transfer of cell type-specific gene expression information from one dataset to another.

> <question-title></question-title>
>
> 1. What is a weighted sum? *Hint*: What kinds of genes would be best for distinguishing cell types, and what kinds of genes would make it difficult?
>
> > <solution-title></solution-title>
> >
> > 1. So you know what the *Sum* is from above - the total expression of a given gene in a bulk RNA-seq sample depends on the proportion of cell types and the average expression level of each of those cell types (**T = C x P**). However, single cell RNA-seq data is highly variable. Cell-type specific expression in genes with lower variation from sample to sample (i.e. person to person or organism to organism) and cell to cell (i.e. within a sample) will be the most useful for distinguishing cells, while genes that vary heavily (i.e. high in cell type<sub>a</sub> in one sample, but low in cell type<sub>a</sub> in another sample) will be the least useful in accurately distinguishing cells. Therefore, to accurately use the mean expression level in a cell_type, MusiC weights the sums, favouring more consistently expressed genes in cell types.
> >
> {: .solution}
{: .question}

Solid tissues often contain closely related cell types that are difficult to distinguish from one another, a phenomenon known as *"collinearity"*. To deal with collinearity, MuSiC can also employ a tree-guided procedure that recursively zooms in on closely related cell types. Briefly, MuSic first groups similar cell types into the same cluster and estimates cluster proportions, then recursively repeats this procedure within each cluster.

![muse1](../../images/bulk-music/figure_method.jpg "Overview of MuSiC Suite")

# Cell Proportion Estimation

Here we will extract cell proportions from a bulk data of human pancreas data from {%cite fadista2014global %} concerning 56 638 genes across 89 samples, using a single cell human pancreas dataset from {%cite segerstolpe2016single %} containing 25 453 genes across 2209 cells, clustered into 14 cell types, from 6 healthy subjects and 4 with Type-II diabetes (T2D). If the deconvolution is good, and the datasets are compatible with sufficient enough overlap, we should be able to identify the same cell types from the bulk data.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Cell Type inference of Human Pancreas Data"*
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    * Human pancreas bulk RNA datasets (tag: `#bulk`)
>
>      ```
>      https://zenodo.org/record/5719228/files/GSE50244bulkeset.expression.tabular
>      https://zenodo.org/record/5719228/files/GSE50244bulkeset.phenotype.tabular
>      ```
>    * Human pancreas single-cell RNA datasets (tag: `#scrna`)
>      ```
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.expression.tabular
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.phenotype.tabular
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
>
> 4. Check the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each `expression` file a tag corresponding to `#bulk` and `#scrna`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

### Exploring the Datasets

  We are told before we download the data, that:
  * The bulk human pancreas dataset is 89 samples across 56 638 genes
  * The single cell human pancreas datasets is 2209 cells across 24 453 genes.

  But what does this actually look like in the data?

> <comment-title>Inspecting Datasets</comment-title>
>
> Note that at any time you can visually inspect the input datasets yourself by either:
> * Expanding the dataset in the history panel clicking on the name of the dataset
> * Or clicking on {% icon galaxy-eye %} icon to load them into the main window.
{: .comment}

Let's start exploring the datasets.

> <hands-on-title>Exploring the Datasets</hands-on-title>
>
> 1. Inspect the `#scrna` expression file
>
>    > <question-title></question-title>
>    >
>    > 1. What do the rows and columns correspond to?
>    > 2. What do the values at each position mean?
>    > 3. Has the data been normalised?
>    > 4. Why are there so many zeroes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![expr_tab](../../images/bulk-music/peek_tabular_scrna_expr.png "Peeking at the tabular scRNA expression dataset")
>    > >
>    > > 1. Rows correspond to gene names, and columns to cell identifiers
>    > > 2. These are the number of reads or counts for each gene for each cell
>    > > 3. The data has not been normalised since the counts are integer and not decimal
>    > > 4. Single-cell datasets can be very sparse for a variety of reasons relating to dropouts and biological factors. For more information, please see the [introduction single-cell RNA-seq slides#41](../../tutorials/scrna-intro/slides.html).
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Inspect the `#bulk` expression file
>
>    > <question-title></question-title>
>    >
>    > 1. What do the rows and columns correspond to?
>    > 2. In which field is there likely to be overlap with the `#scrna` dataset?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![bulk_tab](../../images/bulk-music/peek_tabular_bulk_expr.png "Peeking at the tabular bulk RNA-seq expression dataset")
>    > >
>    > > 1. As before with the `#scrna` dataset, rows correspond to gene names and columns to sample identifiers.
>    > > 2. The sample identifiers and the cell identifiers are completely different, but the gene names appear to be using the same symbols as the `#scrna` dataset, so the gene field is the common factor here.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Inspect the `#scrna` phenotype file
>
>    > <question-title></question-title>
>    >
>    > 1. Does the phenotypes file describe genes or cells?
>    > 2. What does the `SubjectName` field describe?
>    > 3. What does the `cellType` field describe?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![peek_tabular_scrna_pheno.png](../../images/bulk-music/peek_tabular_scrna_pheno.png "Peeking at the tabular scRNA phenotype dataset")
>    > >
>    > > 1. The first column is the index column, which uses the cell identifiers in the header of the `#scrna` expression file, so the phenotypes file describes the cells.
>    > > 2. `SubjectName` tells us whether the cell (on that row) is labelled as Type-II diabetes or not, "Non T2D" or "T2D" respectively.
>    > > 3. `cellType` tells us which cell type the cell was assigned to. This could be the result of prior-clustering and then labelling, or the cells could be labelled before analysis.
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. Inspect the `#bulk` phenotype file
>
>    > <question-title></question-title>
>    >
>    > 1. Does the phenotypes file describe genes or samples?
>    > 2. Is the `SubjectName` field related to the `SubjectName` field in the `#scrna` phenotypes file?
>    > 3. What other factors are in the phenotypes file?
>    > 4. Is the `tissue` field related to the `cellType` field in the `#scrna` phenotypes file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![peek_tabular_bulk_pheno.png](../../images/bulk-music/peek_tabular_bulk_pheno.png "Peeking at the tabular bulk RNA-seq phenotype dataset")
>    > >
>    > > 1. The first column is the index column, which uses the sample identifiers in the header of the `#bulk` expression file, so the phenotypes file describes the samples.
>    > > 2. The `SubjectName` column uses a completely different set of identifiers from the `#scrna` phenotypes file so they should be assumed to be unrelated.
>    > > 3. We see `age`, `bmi`, `hba1c`, `gender`, `tissue`. HbA1c appears to be a gene of interest related to a known phenotype.
>    > > 4. **This is the question we wish to answer in the deconvolution**. Visually, there appears to be no overlap, but the "pancreatic islets" tissue likely consists of several cell types that show expressions profiles with some affinity  to the single cell types described in the `#scrna` phenotypes file.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

   The bulk RNA-seq phenotype file lists the main factors of interest, and HbA1c appears to be a specific gene associated with a phenotype. It is well known that the beta cell proportions are related to T2D disease status. In the progress of T2D, the number of beta cells decreases. One of the most important tests for T2D is the HbA1c (hemoglobin A1c) test. When the HbA1c level is greater than 6.5%, the patient is diagnosed as T2D. We will look later at the beta cell proportions relationship with HbA1c level in this deconvolution analysis.

## Building the Expression Set objects

{% snippet faqs/galaxy/tutorial_mode.md %}

For now we need to construct our Expression set objects that will be consumed by MuSiC.

> <details-title>Expression Set</details-title>
>
> ![expression_set](../../images/bulk-music/expressionset.png "Image from Stefano Monti")
>
>
> *Expression Set* objects are a container for high-throughput assays and experimental metadata. The *ExpressionSet* class is derived from *eSet*, and requires a matrix named *exprs*.
>
> The *ExpressionSet* class is designed to combine several different sources of information into a single convenient structure. An *ExpressionSet* can be manipulated (e.g., subsetted, copied) conveniently, and is the input or output from many Bioconductor functions.
>
> The data in an *ExpressionSet* is complicated, consisting of:
> - expression data (*exprs*, or assayData; assayData is used to hint at the methods used to access different data components, as we will see below);
> - metadata describing samples in the experiment (phenoData, *pData*)
> - information related to the protocol used for processing each sample and usually extracted from manufacturer files (protocolData);
> - annotations and metadata about the features on the chip or technology used for the experiment (featureData, annotation, *fData*);
> - and a flexible structure to describe the experiment (experimentData).
>
> The ExpressionSet class coordinates all of this data, so that you do not usually have to worry about the details.
>
> For more information please [read the specification](http://www.bioconductor.org/packages/release/bioc/vignettes/Biobase/inst/doc/ExpressionSetIntroduction.pdf) as well as the [image source](https://montilab.github.io/BS831/articles/docs/ExpressionSet.html).
>
{: .details}

Here we shall build two ExpressionSet objects corresponding to the bulk and single-cell datatypes.

### Construct Expression Set Object

> <hands-on-title>Build the Expression Set inputs</hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `GSE50244bulkeset.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `GSE50244bulkeset.phenotype.tabular` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > An ExpressionSet object has many data slots, the principle of which are the experiment data (*exprs*), the phenotype data (*pData*), as well metadata pertaining to experiment information and additional annotations (*fData*).
>    {: .comment}
>
> 2. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `EMTABesethealthy.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `EMTABesethealthy.phenotype.tabular` (Input dataset)
>
{: .hands_on}

#### Inspect Expression Set Object

We will now inspect these objects we just created to see what information we can extract out of them, and how these multiple datasets are summarized within the object.

> <hands-on-title>Inspect and Describe the scRNA ExpressionSet Object</hands-on-title>
>
> 1. Obtain General Info about the data set
>    - {% icon galaxy-eye %} Click on the `#scrna` *General Info* dataset in the history view (output of **Construct Expression Set Object** {% icon tool %})
> 1. Obtain Feature Information about the data set
>    - {% tool [Inspect Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_inspect_eset/music_inspect_eset/0.1.1+galaxy4) %} with the following parameters:
>         > <warning-title>Danger: This tool has needs!</warning-title>
>         > You may need to click the dataset from your history and drag it into the input of this tool. Some browsers don't allow this.
>         {: .warning}
>       - {% icon param-file %} *"ESet Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>       - *"Inspect"*: `Feature Data Table`
>         > <comment-title>Features or Genes?</comment-title>
>         >
>         > "Features" are synonymous with "Genes" in a genomic setting, but data scientists tend to prefer to use the former term, as it can be used in other non-genomic settings.
>         >
>         {: .comment}
> By inspecting the `Feature Data Table`, you should see a list of gene names.
>
> 1. Obtain the dimensions of the data set
>    - {% tool [Inspect Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_inspect_eset/music_inspect_eset/0.1.1+galaxy4) %} with the following parameters:
>      - {% icon param-file %} *"ESet Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>      - *"Inspect"*: `Dimension`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many samples are in the dataset, and how many genes?
> 2. Does this agree with the original input tabular expression data set?
>
> > <solution-title></solution-title>
> >
> > 1. 1097 samples and 25 453 genes
> > 2. Yes!
> >
> {: .solution}
>
{: .question}


## Estimating Cell Type proportions

<!-- Maybe this goes in a comment? -->
Instead of selecting marker genes, MuSiC gives weights to each gene. The weighting scheme is based on cross-subject variation, by up-weighing genes with low variation and down-weighing genes with high variation in a cell type. Here we demonstrate this step-by-step with the human pancreas datasets.

The deconvolution of 89 subjects from {%cite fadista2014global %} is performed with the bulk data GSE50244 expression set and single cell reference EMTAB. The estimation was constrained on 6 major cell types: alpha, beta, delta, gamma, acinar and ductal, which make up over 90% of the whole islet.

### Cell Type estimation with **MuSiC**

The deconvolution process can be performed by simply inputting the bulk RNA-seq & scRNA-seq datasets and then hitting execute. However, in this section we will be tracing the effect of a disease phenotype across both datasets.

Here we will use one of the factors from the bulk RNA-seq phenotypes related to the T2D disease status. Any bulk RNA-seq sample coming from a patient with the `HbA1c` factor above 6.5% would be clinically classified as having T2D, so we want to compare these datasets with non-diseased samples. As diabetes affects Beta cells, we're particularly interested in these cells proportions - mostly to prove that the deconvolution has worked well!

It's also important that input scRNA-seq datasets (ideally) have some representation of cell populations from diseased patients.

> <question-title></question-title>
>
> 1. Why should the scRNA-seq dataset contain diseased cells and healthy cells?
>
> > <solution-title></solution-title>
> >
> > 1. For instance, there might be a weird extra cell type that appears in the disease phenotype. Or perhaps there is a missing cell type in the disease phenotype. Without both healthy & diseased datasets, the cells in the bulk datasets might not be fully identifiable. Saying that, such datasets are not always available - something to keep in mind when interpreting the results!
> >
> {: .solution}
>
{: .question}


> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [MuSiC](toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4) %} with the following parameters:
>    - *MuSiC sometimes does not show up from the tool search box. You may need to look for it under the **Single Cell** heading
>    - {% icon param-file %} *"scRNA Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>    - {% icon param-file %} *"Bulk RNA Dataset"*: `#bulk` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Purpose"*: `Estimate Proportions`
>        - *"Methods to use"*: `MuSiC` and `NNLS`
>        - *"Cell Types Label from scRNA dataset"*: `cellType`
>        - *"Samples Identifier from scRNA dataset"*: `sampleID`
>        - *"Comma list of cell types to use from scRNA dataset"*: `alpha,beta,delta,gamma,acinar,ductal`
>        - *"Phenotype factors"*: `(leave blank)`
>        - *"Excluded phenotype factors"*: `sampleID,SubjectName`
>        - Under *"Show proportions of a disease factor?"*: `Yes`
>          - *"scRNA Phenotype Cell Target"*: `beta`
>          - *"Bulk Phenotype Target"*: `hba1c` (**Factor from Bulk Phenotype**)
>          - *"Bulk Phenotype Target Threshold"*: `6.5`
>          - *"scRNA Sample Disease Group"*: `T2D` (**Ideally a Factor from scRNA Phenotype**)
>          - *"scRNA Sample Disease Group (Scale)"*: `5`
>
>    > <comment-title></comment-title>
>    >
>    > It's important to set a phenotype target threshold, otherwise no cells will be matched for the phenotype target. In this case, when the HbA1c level is greater than 6.5%, the patient is diagnosed as T2D.
>    {: .comment}
>
{: .hands_on}

The estimated proportions are normalized such that the proportions of cell types within each sample sum to 1. MuSic compares itself against a previous method of deconvolution known as Non-negative Least-Squares (NNLS), which MuSic supercededs via its Weighted Non-negative Least-Squares (W-NNLS) methodology. You can remove this if you wish from within the tool parameters when running.

![jitter_plot](../../images/bulk-music/jitter_plot.png "Jitter plot of Estimated Proportions")

In the above image you can see (a) the estimated proportion of cells for each of the 6 declared types, as calculated by MuSiC and the NNLS methods, respectively. In the (b) section, this information is better represented as a box plot to show you the distribution of cell type proportions.

![ctprop](../../images/bulk-music/ctprop_plot.png "(top) Cell type proportions by disease factor, and (bottom) HbA1c factor expression against beta cell type proportion")

As stated previously, it is well known that the beta cell proportions are related to T2D disease status. As T2D progresses, the number of beta cells decreases. In the above image we can see in (a) that we have the same information as previous, but we also distinguish between cells that from patients with T2D status over the Normal cell phenotypes. Section (b) further explores this with a linear regression showing the cell type proportion of cells with HbA1c expression, where we see that there is a significant negative correlation between HbA1c level and beta cell proportions.


> <comment-title></comment-title>
>
>  We can extract the coefficients of this fitting by looking at the `Log of Music Fitting Data` in the `Summaries and Logs` output collection:
>
>  ```
>  Coefficients:
>               Estimate Std. Error t value Pr(>|t|)
>  (Intercept)  0.797148   0.194757   4.093  0.00011 ***
>  age          0.002639   0.001772   1.489  0.14087
>  bmi         -0.013620   0.007276  -1.872  0.06529 .
>  hba1c       -0.061396   0.025403  -2.417  0.01819 *
>  genderMale   0.079874   0.039274   2.034  0.04566 *
>
>  ```
>
>  In addition to HbA1c levels, gender has also correlated with beta cell proportions. This is unsurprising, as more men have diabetes than women and sex is known to impact HbA1c levels.
>
{: .comment}

### Proportions of Cell Type to each Bulk RNA sample

One question we might wish to ask is: what affinity did each of the 6 single cell types have to each of the 89 subjects in the bulk data? For this we can look at the raw data {% icon galaxy-eye %} `MuSiC Estimated Proportions of Cell Types` in the `Proportion Matrices`, to get a glimpse of cell type compositions on a bulk RNA sample level.

Both the MuSiC and the NNLS calculations of this data is best represented in the below heatmap, with RNA samples as rows and cell types as columns:

![subject_heatmap](../../images/bulk-music/subject_heatmap.png "Heatmap of cell type proportions at the RNA sample level.")

> <question-title></question-title>
>
> 1. Which cell types are under-represented in the NNLS method?
> 2. Which cell types do not appear to be present in both?
>
> > <solution-title></solution-title>
> >
> > 1. Here it is evident that the previous NNLS method over-represents the Alpha cell type compared to the MuSiC method which gives more weight to the Beta and Ductal cell types, which were under-represented in the NNLS method.
> > 2. The Delta and Gamma cell types appear empty in both.
> >
> {: .solution}
>
{: .question}


# Estimation of cell type proportions with pre-grouping of cell types

  In the previous section we estimated cell types under the assumption that that the gene expression between cell types was largely independent. However, solid tissues often contain closely related cell types. This correlation of gene expression between these cell types is termed 'collinearity', which makes it difficult to resolve their relative proportions in bulk data.

  To deal with collinearity, MuSiC can also employ a tree-guided procedure that recursively zooms in on closely related cell types.

  Briefly, similar cell types are grouped into the same cluster and their cluster proportions are estimated, then this procedure is recursively repeated within each cluster. At each recursion stage, only genes that have low within-cluster variance are used, as they should be consistent within a cell type. This is critical as the mean expression estimates of genes with high variance are affected by the pervasive bias in cell capture of scRNA-seq experiments, and thus cannot serve as reliable reference.

  To perform this analysis, we will use mouse kidney single-cell RNA-seq data from {% cite park2018single %} described by 16 273 genes over a trimmed subset of 10 000 cells, giving 16 unique cell type (2 of which are novel) across 7 subjects. The bulk RNA-seq dataset is from {% cite beckerman2017transgenic %} and contains mouse kidney tissue described by 19 033 genes over 10 samples.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Dendrogram of Mouse Data"*
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    * Mouse kidney bulk RNA datasets (tag: `#bulk`)
>
>      ```
>      https://zenodo.org/record/5719228/files/Mousebulkeset.expression.tabular
>      https://zenodo.org/record/5719228/files/Mousebulkeset.phenotype.tabular
>      ```
>    * Mouse kidney single-cell RNA datasets (tag: `#scrna`)
>      ```
>      https://zenodo.org/record/5719228/files/Mousesubeset.expression.tabular
>      https://zenodo.org/record/5719228/files/Mousesubeset.phenotype.tabular
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
>
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each `expression` file a tag corresponding to `#bulk` and `#scrna`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

> <details-title>Exploring the Datasets</details-title>
>
> As before, you may choose to explore the bulk and scrna datasets and try to determine their factors from the phenotypes as well as any overlapping fields that will be used to guide the deconvolution.
>
{: .details}

You will need to again create ExpressionSet objects, as before.

### Construct Expression Set Object

> <hands-on-title>Build the Expression Set inputs</hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `Mousebulkeset.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `Mousebulkeset.phenotype.tabular` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > An ExpressionSet object has many data slots, the principle of which are the experiment data (*exprs*), the phenotype data (*pData*), as well metadata pertaining to experiment information and additional annotations (*fData*).
>    {: .comment}
>
> 2. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `Mousesubeset.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `Mousesubeset.phenotype.tabular` (Input dataset)
>
{: .hands_on}

### Colinearity Dendrogram with **MuSiC** to determine cell type similarities

 Determining cell type similarities requires first producing a design matrix as well as a cross-subject mean of relative abundance, using a tree-based clustering method of the cell types we wish to cluster.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [MuSiC](toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"scRNA Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>    - {% icon param-file %} *"Bulk RNA Dataset"*: `#bulk` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Purpose"*: `Compute Dendrogram`
>        - *"Cell Types Label from scRNA Dataset"*: `cellType`
>        - *"Cluster Types Label from scRNA dataset"*: `clusterType`
>        - *"Samples Identifier from scRNA dataset"*: `sampleID`
>        - *"Comma list of cell types to use from scRNA dataset"*:
>           `Endo,Podo,PT,LOH,DCT,CD-PC,CD-IC,Fib,Macro,Neutro,B lymph,T lymph,NK`
>
{: .hands_on}

![dendrogram](../../images/bulk-music/dendrogram.png "Dendrogram of Design Matrix and Cross-Subject Mean of Relative Abundance")

> <question-title></question-title>
>
> 1. What do you notice about the cells clustering?
> 2. How many clusters can you see with a height threshold above 650 in the "Cluster log(Design Matrix)"?
>
> > <solution-title></solution-title>
> >
> > 1. The immune cells are clustered together and the kidney specific cells are clustered together. Notice that DCT and PT are within the same high-level grouping.
> > 2. The cut-off of 650. Here we cut 13 cell types into 4 groups:
> >
> >    ```
> >    C1: Neutro
> >    C2: Podo
> >    C3: Endo, CD-PC, CD-IC, LOH, DCT, PT
> >    C4: Fib, Macro, NK, B lymph, T lymph
> >    ```
> >
> {: .solution}
>
{: .question}


### Heatmap of Cell Type Similarities using **MuSiC**

We shall use the 4 cell type groups determined by the cut off threshold in the above question box. To guide the clustering, we shall upload known epithelial and immune cell markers to improve the more diverse collection of cell types in the C3 and C4 groups.

> <hands-on-title>Upload marker genes and generate heatmap</hands-on-title>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/5719228/files/epith.markers
>    https://zenodo.org/record/5719228/files/immune.markers
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. {% tool [MuSiC](toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4) %} with the following parameters:
>    - **Note**
>      > <warning-title>Shortcut!</warning-title>
>      >
>      > Here we need to re-use all the inputs from the previous **MuSiC** {% icon tool %} step, plus add a few extra. To speed this up, you can simply click on the re-run icon {% icon galaxy-refresh %} under any of its outputs.
>      {: .warning}
>
>    - {% icon param-file %} *"scRNA Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>    - {% icon param-file %} *"Bulk RNA Dataset"*: `#bulk` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Purpose"*: `Compute Dendrogram`
>        - *"Cell Types Label from scRNA Dataset"*: `cellType`
>        - *"Cluster Types Label from scRNA dataset"*: `clusterType`
>        - *"Samples Identifier from scRNA dataset"*: `sampleID`
>        - *"Comma list of cell types to use from scRNA dataset"*:
>           `Endo,Podo,PT,LOH,DCT,CD-PC,CD-IC,Fib,Macro,Neutro,B lymph,T lymph,NK`
>        - In *"Cluster Groups"*:
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C1`
>                - *"Comma list of cell types to use from scRNA dataset"*: `Neutro`
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C2`
>                - *"Comma list of cell types to use from scRNA dataset"*: `Podo`
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C3`
>                - *"Comma list of cell types to use from scRNA dataset"*:
>                   `Endo,CD-PC,LOH,CD-IC,DCT,PT`
>                - *"Marker Gene Group Name"*:`Epithelial`
>                - {% icon param-file %} *"List of Gene Markers"*: `epith.markers` (Input dataset)
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C4`
>                - *"Comma list of cell types to use from scRNA dataset"*:
>                   `Macro,Fib,B lymph,NK,T lymph`
>                - *"Marker Gene Group Name"*:`Immune`
>                - {% icon param-file %} *"List of Gene Markers"*: `immune.markers` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > The C1 (Neutrophil) and C2 (Podocyte) clusters do not use marker genes for the dendrogram clustering in this dataset.
>    {: .comment}
>
{: .hands_on}


![dendro_jitboxheat](../../images/bulk-music/dendro_jitboxheat.png "Jitter and Boxplots of all cell types followed by a heatmap of each RNA sample against cell type. Note that the y-axis for each of the plots above are not constant across cell types.")

> <question-title></question-title>
>
> Most of the expression in the above plot appears to be derived from one cell type.
>
> 1. Which cell type dominates the plot?
> 2. What does this tell you about the bulk RNA?
>
> > <solution-title></solution-title>
> >
> > 1. The PT cells appear to dominate.
> > 2. Most of the expression in the bulk RNA dataset is derived solely from the PT cells, and could be a monogenic cell line.
> >
> {: .solution}
>
{: .question}


# Conclusion


In this tutorial we constructed ExpressionSet objects, inspected and annotated them, and then finally processed them with the MuSiC RNA-Deconvolution analysis suite.

Below is an overview of the workflow that was used throughout this tutorial.

![workflow1](../../images/bulk-music/workflow1.png "Workflow for estimating cell types")

Note how two ExpressionSet objects are constructed: one from bulk RNA-seq tabular assay data, and the other from single-cell RNA-seq tabular assay data. A blind analysis of cell proportion estimation is performed first. Then in the second half of the tutorial, we performed a guided analysis using pre-grouped cell types.


![workflow_dendro.png](../../images/bulk-music/workflow_dendro.png "Workflow for generating a dendrogram and clustering cells upon it")

This tutorial is part of the [https://singlecell.usegalaxy.eu](https://singlecell.usegalaxy.eu) portal ({% cite tekman2020single %}).

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Construct Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4", "Inspect Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_inspect_eset/music_inspect_eset/0.1.1+galaxy4", "MuSiC", "toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4"]


topic: single-cell
tutorial: scrna-preprocessing
title: Pre-processing of Single-Cell RNA Data
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: ["[]"]
content_hash: 9b03d7e0a4875e9def6dff1f78969c4b
last_modified: 2026-05-27T09:01:49.748559
zenodo_link: https://zenodo.org/record/3253142
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-preprocessing/tutorial.html

content:
## Why do Single Cell sequencing?


Single-cell RNA (scRNA) sequencing is the technological successor to classical "bulk" RNA-seq, where samples are no longer defined at the tissue level but at the individual cell level. The bulk RNA-seq methods seen in [previous hands-on material]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) would give the average expression of genes in a sample, whilst overlooking the distinct expression profiles given by the cell sub-populations due to their heterogeneity.

The rise of scRNA sequencing provides the means to explore this heterogeneity by examining samples at the individual cell level, enabling a greater understanding of the development and function of such samples, by the characteristics of their constituent cells.

Consider the heterogeneity of cells sampled from bone marrow, where hematopoietic stem cells can give rise to many different cell types within the same tissue:

![Cell Differentiation of Hematopoietic Stem cells]({% link topics/single-cell/images/scrna-pre-processing/scrna_hematopoiesis.png %} "Cell Differentiation of Hematopoietic Stem cells. Image Source: https://commons.wikimedia.org/wiki/File:Hematopoiesis_simple.svg")

The genes expressed by these cells at different developmental time points can be subtle, but generally can be classified into discrete cell sub-populations or under statistical clustering methods such as PCA or tSNE. Cells in the same cluster exhibit similar profiles of differential expression in the same set of related genes, compared to cells in other clusters. By identifying significant genes in each cluster, cell types and cell developmental hierarchies can be inferred based on the proximity of these clusters to one another.

Other than cell development, there are many more factors that can shape the level of gene expression exhibited by a given cell. Intercellular cell-signalling can block or enhance specific transcripts, the total amount of transcripts of a cell increases with the cell-cycle, or the proximity of a cell within a tissue to nutrients or oxygen.

![Facets of Cellular Identity]({% link topics/single-cell/images/scrna-pre-processing/scrna_nbt3711.png %} "Revealing the vectors of cellular identity with single-cell genomics, Nature Biotechnology, 2016")

This tutorial is in part inspired by aspects of the [Hemberg workflow](https://hemberg-lab.github.io/scRNA.seq.course/) at the Sanger institute, as well as the [CGATOxford workflow](https://github.com/CGATOxford/UMI-tools) which provides the **UMI-tools** suite that we make use of. The barcoding follows the CEL-Seq2 protocol {% cite Hashimshony2016 %},  mentioned in the [*Understanding Barcodes*]({% link topics/single-cell/tutorials/scrna-umis/tutorial.md %}) hands-on, and uses the same lane configuration as utilised by the [Freiburg MPI Grün lab](https://www.ie-freiburg.mpg.de/gruen).


# Analysis Strategy


Most scRNA sequencing techniques use pooled-sequencing approaches to generate a higher throughput of data by performing amplification and sequencing upon multiple cells in the same "pool". From a bioinformatics standpoint, this means that the output FASTQ data from the sequencer is batch-specific and contains all the sequences from multiple cells, where one sample of cells is equal to one batch.

In this tutorial, we will perform pre-processing upon scRNA FASTQ batch data to generate an *N*-by-*M*  count matrix of *N* cells and *M* genes, with each element indicating the level of expression of that gene in a particular cell.

This count matrix is crucial for performing the downstream analysis, where differential gene analysis is performed between cells in order to cluster them into groups denoting their cell type and lineage.

The tutorial is structured in two parts:

1. **Single-Batch Processing**
1. **Multi-Batch Processing**

![Overview of workflow]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow.svg %} "An overview of the entire workflow")

The first part of this tutorial will deal with batches, and use example *FASTQ* data from a single batch, which we will perform [barcode extraction](#barcode-extraction) and annotation upon. Alignment and quality control will also be performed, and we will see how to construct a rudimentary count matrix.

> <comment-title>10x Datasets</comment-title>
>
> 10x Genomics datasets can be processed in a much easier way that is outlined in the [*Pre-processing of 10X Single-Cell RNA Datasets*]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}) tutorial.
>
> However, much of the essential concepts for scRNA-seq pre-processing are not explained, so it is a good idea to familiarise yourself with them in this tutorial.
>
{: .comment}

The second part of this tutorial will deal with merging several output count matrices from multiple single batches generated in the first portion. Here, a set of example count matrices are [merged together](#merging-count-matrices) and quality control performed. This produces a final count matrix valid for downstream analysis.


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Single-Batch Processing

## Data upload and organization

In this tutorial we will be analysing scRNA-seq data of bone marrow cells taken from a single C57 mouse by *Herman et al.* ({% cite Herman2018 %}) and producing a count matrix that we can use for later analysis.

The size of scRNA files (.fastq) are typically in the gigabyte range and are somewhat impractical for training purposes, so we will expedite the analysis by using a smaller subset of actual batch data. We will also be using *Mus Musculus* annotation data (.gtf) from the  [NCBI RefSeq](ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/vertebrate_mammalian/Mus_musculus/) track, as well as a barcodes file (.tsv).


![Workflow Upper]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow_upper.png %} "Single batch processing")


> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and rename it (e.g. scRNA-seq single batch tutorial)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 1. Import the subset FASTQ paired data from [`Zenodo`](https://zenodo.org/record/3253142) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3253142/files/SRR5683689_1.subset.fastq
>    https://zenodo.org/record/3253142/files/SRR5683689_2.subset.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true collection_type="Paired" collection_name_convention="`<name>_<plate>_<batch>` to preserve the sample names, sequencing plate number and batch number." collection_name="Here we will write `C57_P1_B1`" genome="GRCm38/mm10" pairswaptext="`SRR5683689_1` and `SRR5683689_2`" %}
>
> 3. Import the Gene Annotations and Barcodes from [`Zenodo`](https://zenodo.org/record/3253142) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3253142/files/Mus_musculus.GRCm38.93.mm10.UCSC.ncbiRefSeq.gtf
>    https://zenodo.org/record/3253142/files/celseq_barcodes.192.tabular
>    ```
>
> 4. Set the datatype of the `celseq_barcodes.192.tabular` to `tabular`
>
{: .hands_on}


## Barcode Extraction

> <comment-title>Note</comment-title>
>
> Before performing the barcode extraction process, it is recommended that you familiarise yourself with the concepts of designing cell barcodes as given by the [*Plates, Batches, and Barcodes*]({% link topics/single-cell/tutorials/scrna-plates-batches-barcodes/slides.html %}), as well as the [*Understanding Barcodes*]({% link topics/single-cell/tutorials/scrna-umis/tutorial.md %}) hands-on material for an introduction into transcript barcodes.
>
{: .comment}

We will be demultiplexing our FASTQ batch data by performing barcode extraction whilst also making use of the provided barcodes file to filter for *specific* cell barcodes.

> <hands-on-title>Barcode Extraction</hands-on-title>
>
> 1. {% tool [UMI-tools extract](toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_extract/umi_tools_extract/0.5.5.1) %} with the following parameters:
>    - *"Library type"*: `Paired-end Dataset Collection`
>        - {% icon param-collection %} *"Reads in FASTQ format"*: `C57_P1_B1` (Our paired set)
>        - *"Barcode on both reads?"*: `Barcode on first read only`
>    - *"Use Known Barcodes?"*: `Yes`
>        - {% icon param-file %} *"Barcode File"*: `celseq_barcodes.192.tabular` (Input dataset)
>    - *"Barcode pattern for first read"*: `NNNNNNCCCCCC`
>    - *"Enable quality filter?"*: `No`
>
{: .hands_on}

Verifying that the desired UMI and cell barcodes have been extracted from the sequence of the Forward reads and inserted into the header of the Reverse reads is encouraged, using the method outlined in the above hands-on material.


> <question-title></question-title>
>
> How many reads were filtered out, and why?
>
> > <solution-title></solution-title>
> >
> > The input FASTQ files contained reads from all barcodes, including those with sequencing errors, resulting in a larger pool of detected barcodes than those desired. (e.g. Cell barcode `AAATTT` could have single base-pair sequencing errors that could modify it into `ATATTT` or `AAACTT`, etc).
> >
> > This information is included in the Log file of **UMI-tools extract** which contains all the parameters used to run, as well as *INFO* lines that indicate how many reads were read, and how many output. In this case: 134431 reads were retained (>90% of input reads).
> {: .solution}
{: .question}


## Mapping / Alignment

FASTQ files contain sequence information that we wish to map to genes in a genome. Mapping is a relatively straightforward process, and is covered more extensively in the [*Sequence Analysis*]({% link topics/sequence-analysis/index.md %}) tutorials:

1. Select your genome
1. Select your gene annotation file
1. Run the alignment program
1. (Optional) Run MultiQC to assess the quality of the mapping

The FASTQ data was sequenced from mouse data, so to perform the alignment we will need to gather all data relevant to that genome. We will use the latest version (mm10).

The annotation GTF file must match the genome version used, since both use physical coordinates. Each GTF contains all the gene, exon, intron, and other regions of interest that we will use to annotate our reads, should our reads align to any of the regions specified in this file.

For alignment, we will use RNA-STAR for performance and splice-awareness.


> <hands-on-title>Performing the Alignment</hands-on-title>
>
> 1. {% tool [RNA-STAR](toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.7a) %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Single-end`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file"*: `Reads2` (output of **UMI-tools extract** {% icon tool %})
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>        - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>            - {% icon param-file %} *"Select reference genome"*: `Mus Musculus (mm10)` (Mouse)
>            - {% icon param-file %} *"Gene model (gff3,gtf) file for splice junctions"*: `Mus_musculus.GRCm38.93.mm10.UCSC.ncbiRefSeq`
> 1. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.9) %} with the following parameters:
>    - *"Results"*:
>      - *"1: Results"*:
>        - *"Which tool was used to generate logs?"*:`STAR`
>        - *"STAR output"*:
>           - *"1: STAR output"*
>              - *"Type of STAR output?"*:`Log`
>        - *"STAR log output"* :(Select the STAR output file that ends in "log")
> 1. Click on the {% icon galaxy-eye %} symbol on the MultiQC Webpage
>
{: .hands_on}

The purpose of MultiQC is to observe how well our reads were mapped against the reference genome. Many reads are discarded due to being of too low quality, or having ambiguous sequence content that can map them to multiple locations.

> <question-title></question-title>
>
> 1. What percentage of our reads are uniquely mapped? How many millions of reads is this percentage?
> 2. What percentage of our reads are mapped to more than one locus?
> 3. Is our overall mapping 'good' ?
>
> > <solution-title></solution-title>
> >
> > 1. `59.5%` or ~80k reads were successfully mapped
> > 2. `13.6%` are multiply mapped, and `3.7%` were mapped to too many loci
> >   - Multiply mapped means that a read was aligned to more than one gene
> >   - Mapped to too many loci means that a read was aligned to 10 or more loci, and should be ignored.
> > 3. It depends on how good the sequencing protocol is, and how many reads in total were mapped.
> >   - `90%` is amazing, reserved for bulk RNA-seq which typically has high coverage
> >   - `70%` is weak for bulk RNA-seq, but good for single-cell RNA-seq
> >   - This a small subset of a real dataset, but one would expect that 6 million mapped reads would be enough to generate a downstream analysis.
> >
> {: .solution}
{: .question}


## Filtering

Before continuing let us first look back on some of the previous stages:

> <comment-title>Recap of previous stages</comment-title>
>
> 1. *Barcode Extraction*:
>
>     Here we used `umi_tools extract` on our input forward and reverse FASTQ files, and extracted the UMI and cell barcode from the forward read *sequence*, and placed it into the *header* of both forward and reverse FASTQ files. i.e. FASTQ files → Modified FASTQ files
>
> 2. *Mapping*:
>
>     We took the sequencing data from the reverse FASTQ file (with modified headers) and aligned it to the mouse genome, using annotations presented in the GTF file for that genome. i.e. Modified FASTQ file (reverse) → BAM file
>
{: .comment}

### Confirming Reads in the BAM file

We now have a BAM file of our aligned reads, with cell and UMI barcodes embedded in the read headers. We also have the chromosome and base-pair positions of where these reads are aligned. The can be confirmed by peeking into the BAM file:

> <hands-on-title>Confirming the Alignment Data</hands-on-title>
>
>  1. Click on the {% icon galaxy-eye %} symbol of the BAM output from STAR.
>  2. There are many header lines that begin with `@` which we are not interested in.
>  3. Look at 10th read directly below the header lines:
>
>         SRR5683689.38437_GCATTC_CTTCGT	16	chr1	3439991	255	70M	*	0	0	CTTTGAATCTCTTCTTCCCAGCTAGTCATCTTCCTGCTTTTCTCTCTGTCTGTCTGTCTGTCTGTCTGTC	'0'<B<''B77<BFBBBBB7'FBFB0F7FBB<B'''<IFFBF<FBFB<FBBFBB0<BFFFBB0BBFFB<<	NH:i:1 HI:i:1 AS:i:66 nM:i:1
>
>
{: .hands_on}

The fields of the BAM file can be better explained at section 1.4 of [the SAM specification](https://samtools.github.io/hts-specs/SAMv1.pdf), but we will summarise the main fields of interest here:

* `SRR568..._GCATTC_CTTCGT`: The *readname* appended by `_`, the cell barcode, another `_`, and then the UMI barcode.
* `16`: The FLAG value

> <question-title>What does the alignment flag value of 16 tell us about this read?</question-title>
>
>   <!-- TODO This information needs to be integrated into an actual tool -->
>   We can interactively see what the different FLAG values mean by feeding values into the SAM specification to the [Picard web tool](https://broadinstitute.github.io/picard/explain-flags.html)
>
>   > <solution-title></solution-title>
>   > The read aligns to the reverse strand
>   >
>   {: .solution}
>
{: .question}

* `chr1` `3439991`: The position and base-pair of alignment of the first base of the sequence.
* A series of quality fields, with the main contributors being  the sequence and sequence quality strings.
* `NH`: The number of hits for  this read. If it is multiply mapped, then the number of multiples will be shown (here `1`, so not multiply mapped).
* `HI`: Which number this particular read is in the series of (potentially) multi-mapped reads (here `1`, not necessarily meaning the first or 'better').
* `nM`: The number of base mismatches in the alignment of this read to the reference (here `1`).


### Filtering the BAM File

If we perform counting on the current BAM file we will be counting all reads, even the undesirable ones such as those that did not align so optimally.

The main filtering steps performed on our reads so far have been relatively silent due to the 'default' parameters used.
* *UMI-tools Extract* - Filters reads for those only with matching barcodes given by our barcodes file.
* *RNA-STAR* - As seen in the log, we lose 25% of our reads for being too short or being multiply mapped.

Another filtering measure we can apply is to keep reads that we are confident about, e.g those with a minimum number of mismatches to the reference within an acceptable range. Specifically, we want to keep all reads that align to the forward or reverse strand that also have less that 3 mismatches to the reference, and are also mapped only once to the reference.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1) %} with the following parameters:
>    - {% icon param-file %} *"BAM dataset(s) to filter"*: `output_bam` (output of **RNA STAR** {% icon tool %})
>    - In *"Condition"*:
>        - In *"1: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `alignmentFlag`
>                        - *"Filter on this alignment flag"*: `0`
>        - Click on *"Insert Condition"*:
>        - In *"2: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `alignmentFlag`
>                        - *"Filter on this alignment flag"*: `16`
>        - Click on *"Insert Condition"*:
>        - In *"3: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `tag`
>                        - *"Filter on a particular tag"*: `nM:<3`
>                          <small>(**Attention! please use a lowercase 'n' here!**)
>        - Click on *"Insert Condition"*:
>        - In *"4: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `tag`
>                        - *"Filter on a particular tag"*: `NH:<2`
>    - *"Would you like to set rules?"*: `Yes`
>        - *"Enter rules here"*: `(1 | 2) & 3 & 4`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are we filtering only for alignment flags `0` and `16`?
> 2. What do the tag filters `nM:<3` and `NH:<2` do?
> 3. What is happening at the rules stage?
>
> > <solution-title></solution-title>
> >
> > 1. Alignment flags `0` and `16` specify that we wish to only keep reads that align to the forward and reverse strands.
> > 2. We are keeping reads that have a number of mismatches (`nM`) to the reference of less than 3, and has a number of hits (`NH`) across the reference of less than 2 (i.e. it is not a multiply-mapped read).
> > 3. [Boolean expressions](https://en.wikipedia.org/wiki/Boolean_expression) are applied that state that *either* conditions 1 or 2 can happen, in conjunction with rules 3 and 4 happening.
> >
> {: .solution}
{: .question}


## Quantification

Once we have the name of the gene for a specific read, we can count how many of those reads fall into that gene and generate a count matrix.

Counting reads is performed by two commonly-used tools:
1. RNA-STAR
2. FeatureCounts

The **RNA-STAR** tools has the ability to count reads as it maps them. **FeatureCounts** performs the same task, but is capable of counting not just at the Read level, but also at the UMI level too, such that 10 duplicate reads will be counted only once.

Unfortunately, both are currently limited to counting without being able to distinguish between different cells.

> <details-title>Example</details-title>
>
> If we consider the number of reads that align to *GeneA*, the output given by these two tools is as follows:
>
> > | (reads) | RNA STAR | FeatureCounts |
> > |---------|----------|---------------|
> > | GeneA   |   12     |    12         |
> {: .matrix}
>
> But what we actually require is:
>
> > | (reads) | C1 | C2 |
> > |---------|----|----|
> > | GeneA   | 10 |  2 |
> {: .matrix}
>
> or more specifically:
>
> > | (UMIs) | C1 | C2 |
> > |--------|----|----|
> > | GeneA  | 2  |  1 |
> {: .matrix}
>
{: .details}


In order to obtain this desired format, we must use **UMI-tools count** to perform the counting. However, this tool is dependent on **FeatureCounts** to annotate our reads with the one crucial piece of information that is missing from our BAM file: the name of the gene.

> <comment-title>Verifying missing gene name</comment-title>
> You can check this yourself by examining the {% icon galaxy-eye %} of the BAM file *"STAR Alignment file"*
{: .comment}

### Annotating Gene name with FeatureCounts

Let us annotate our BAM file with desired gene tags.

> <hands-on-title>Quantification assist via FeatureCounts</hands-on-title>
>
> 1. {% tool [FeatureCounts](toolshed.g2.bx.psu.edu/repos/iuc/featurecounts/featurecounts/2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Alignment file"*: `mapped_reads` (output of **Filter BAM** {% icon tool %})
>    - *"Specify strand information"*:`Stranded (Forward)`
>    - *"Gene annotation file"*: `in your history`
>        - {% icon param-file %} *"Gene annotation file"*: `Mus_musculus.GRCm38.93.mm10.UCSC.ncbiRefSeq.gtf`
>    - In *"Advanced options"*:
>        - *"Count multi-mapping reads/fragments"*: `Disabled; multi-mapping reads are excluded (default)`
>        - *"Exon-exon junctions"*: `No (default)`
>        - *"Annotates the alignment file with 'XS:Z:'-tags to described per read or read-pair the corresponding assigned feature(s)."*: `Yes`
>
> 2. Examine the output BAM file
>  - Click on the {% icon galaxy-eye %} for the "Feature Counts: Alignment File"
>  - Scroll down past the header lines
>  - Scroll horizontally to the tags, observe the new `XT` tag.
{: .hands_on}

The `XS` and `XT` tags in the BAM file will now form the basis for counting reads.
With all the relevant data now in our BAM file, we can actually perform the counting via `UMI-tools count`.

> <comment-title>Verifying added gene name</comment-title>
> You can once again check this yourself by examining the {% icon galaxy-eye %} of the BAM file *"STAR Alignment file"*
{: .comment}


### Counting Genes / Cell

> <hands-on-title>Final Quantification</hands-on-title>
>
> 1. {% tool [UMI-tools count](toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_count/umi_tools_count/0.5.5.1) %} with the following parameters:
>    - {% icon param-file %} *"Sorted BAM file"*: `out_file1` (output of **FeatureCounts** {% icon tool %})
>    - *"UMI Extract Method"*: `Barcodes are contained at the end of the read separated by a delimiter`
>    - *"Bam is paired-end"*:`No`
>    - *"Method to identify group of reads"*:`Unique`
>    - *"Extra Parameters"*:
>       - *"Deduplicate per gene."*:`XT`
>       - *"Group reads only if they have the same cell barcode."*:`Yes`
>    - *"Prepend a label to all column headers"*:`No modifications`
>
{: .hands_on}


The important parameters to take note of are those given in the *Extra Parameters* where we specify that each of the reads with a `XT:Z` tag in the BAM file will be counted on a per cell basis. Reads sharing the same UMI and cell Barcodes will be de-duplicated into a single count, reducing PCR duplicate bias from the analysis.

At this stage, we now have a tabular file containing genes/features as rows, and cell labels as headers.

> <question-title></question-title>
>
> 1. How many genes do we have in the matrix?
> 2. How many cells?
>
> > <solution-title></solution-title>
> >
> > 1. 2,140 lines
> >    This information can be seen in the file preview window by clicking on the name of the file (**NOT** the {% icon galaxy-eye %} symbol).
> >
> > 2. 180 columns (not including the first column of gene names)
> >    The number of columns can be seen by scrolling the file preview window completely to the right. 192 cell barcodes were given, but due to the subsetted data, only 180 were detected.
> >
> {: .solution}
{: .question}

The generation of a single count matrix is now complete, with the emphasis on the word *single* due to the fact that we often deal in multiple batches of sequencing data.


> <comment-title>Recap of previous stages</comment-title>
>
> 1. *Barcode Extraction*:
>
>     Here we used `UMI_tools extract` on our input forward and reverse FASTQ files, and extracted the UMI and cell barcode from the forward read *sequence*, and placed it into the *header* of both forward and reverse FASTQ files. i.e. FASTQ files → Modified FASTQ files
> 2. *Mapping*:
>
>     We took the sequencing data from the reverse FASTQ file (with modified headers) and aligned it to the mouse genome, using annotations presented in the GTF file for that genome. i.e. Modified FASTQ file (reverse) → BAM file
>
> 3. *Quality Filtering*:
>
>     Reads with alignment mismatches greater than 2 were discarded, and only non multi-mapped reads that mapped to the forward or reverse strand were kept
>
> 4. *Quantification*:
>
>     Gene tags were added to our alignment file, and reads were grouped according those sharing the same gene tag, with further reduction performed by collapsing all reads sharing the same cell and UMI barcode to be counted only once.
>
{: .comment}

This concludes the first part of the tutorial which focused on the transformation of raw FASTQ data from a single batch into a count matrix. The second part of this tutorial guides us through the process of merging multiple processed batches from the first stage, and performing qualitative filtering.


> <details-title>Workflows For All Steps</details-title>
>
> Galaxy provides a workflow that captures the process of all the above stages for a [single pair of FASTQ data](workflows/scrna_pp_celseq.ga):
>
> ![Single Batch Workflow]({{site.baseurl}}/topics/single-cell/images/scrna-pre-processing/scrna_workflow_single.png "Single Batch Workflow")
>
> For multiple batch processing, Galaxy can make use of *Nested Workflows* that in this case can take in a [list of input paired FASTQ data](workflows/scrna_mp_celseq.ga) and process them in parallel.
>
> ![Multiple Batch Workflow]({{site.baseurl}}/topics/single-cell/images/scrna-pre-processing/scrna_workflow_nested.png "Nested Multiple Batch Workflow")
>
{: .details}

For the processing of 10x Genomics datasets, please refer to the [*Pre-processing of 10X Single-Cell RNA Datasets*]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}) tutorial.

# Multi-Batch Processing


![Workflow Lower]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow_lower.png %} "Multiple batch processing")

Handling more than one batch of sequencing data is rather trivial when we take into account our main goals and requirements:

1. For each batch, convert FASTQ reads from into a count matrix.
2. Merge all count matrices into a single count matrix

The first step merely requires us to run the same workflow on each of our batches, using the exact same inputs except for the FASTQ paired data. The second step requires a minimal level of interaction from us; namely using a merge tool and selecting our matrices.


## Data upload and organisation

The count matrix we have generated in the previous section is too sparse to perform any reasonable analysis upon, and constitutes data only of a single batch. Here we will use more populated count matrices from multiple batches, under the assumption that we now know how to generate each individual one of them using the steps provided in the previous section. This data is available at [`Zenodo`](https://zenodo.org/record/3253142).

Once again, file naming is important, and so we will rename our matrix files appropriately to the plate and batch they are supposed to originate from.

> <hands-on-title>Data upload and organisation</hands-on-title>
>
> 1. Create a new history and rename it (*e.g.* scRNA-seq multiple-batch tutorial)
>
> 1. Import the eight matrices (`P1_B1.tsv`, `P1_B2.tsv`, etc.) and the barcodes file from [`Zenodo`](https://zenodo.org/record/3253142) or from the data library (ask your instructor)
>    - Set the datatype of the files to **tabular**
>
>    ```
>    https://zenodo.org/record/3253142/files/P1_B1.tsv
>    https://zenodo.org/record/3253142/files/P1_B2.tsv
>    https://zenodo.org/record/3253142/files/P1_B3.tsv
>    https://zenodo.org/record/3253142/files/P1_B4.tsv
>    https://zenodo.org/record/3253142/files/P2_B5.tsv
>    https://zenodo.org/record/3253142/files/P2_B6.tsv
>    https://zenodo.org/record/3253142/files/P2_B7.tsv
>    https://zenodo.org/record/3253142/files/P2_B8.tsv
>    https://zenodo.org/record/3253142/files/celseq_barcodes.192.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 1. Rename a matrix
>    - Click on {% icon galaxy-pencil %} of the *`P1_B1.tsv`* file
>    - Set the Name field such that it is affixed with "_P1_B1" (e.g. 'multibatch_P1_B1')
>    - Click *Save*
> 1. Repeat for all matrices
>    **Pay attention to the Plate number which changes after Batch 4**
>
{: .hands_on}

## Merging Count Matrices

Before we begin, we must consider that our matrices are not equal -- e.g. Batch1 has 3 cells that describe *Genes{A,B,C,D}* whereas Batch2 has 4 cells that describe *Genes{C,D,E}*.

We have the problem that only *GeneC* and *GeneD* appear in both batches, whilst describing 7 different cells in total.

To resolve this we can perform a "Full Table Join" where the missing data for *GeneE* and *GeneA* in Batch1 and Batch2 respectively are replaced with zeroes:

![Table Join]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/scrna_downstream.svg %} "Full Table Join")

> <comment-title></comment-title>
> For more information on table joins, see [this guide](http://www.sql-join.com/sql-join-types/)
{: .comment}


> <question-title></question-title>
>
> 1. Why is it required to change the column headers in the Full matrix?
> 2. Why were the cell labels in B1 and B2 the same, if they were labelling completely different cells?
>
> > <solution-title></solution-title>
> >
> > 1. Although the cell headers in each batch matrix is the same, the cells they label are *not* the same and need to be relabelled in the final matrix to tell us which batch they originated from.
> > 2. The reason the cell headers are the same is because the cells use the same barcodes, due to fact that the *same* barcodes are sometimes used across *different* batches.
> >
> {: .solution}
>
{: .question}


Let us now merge our matrices from different batches. In order to ensure that our batches are merged in the order that we wish, we should first create a list of datasets so that our matrices are merged in the order given by the list.

> <hands-on-title>Table Merge</hands-on-title>
>
> 1. **Create a Dataset List**
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 1. **Column Join on Collections** {% icon tool %} with the following parameters:
>   - *"Tabular Files"*: (Select the Dataset Collection icon {% icon param-collection %}, and select the Collection from the previous step each of the matrices that you wish to join)
>   - *"Identifier column"*:`1`
>   - *"Number of Header lines in each item"*:`1`
>   - *"Add column name to header"*:`Yes`
>   - *"Fill character"*:`0`
>
{: .hands_on}

The identifier column refers to the column where the gene names are listed. A 1:1 correspondence between matrices is checked, so that the merge does not concatenate the wrong rows between matrices. The *Fill character* provides a default value of 0 for cases where a Gene appears only in one of the matrices as per our example earlier.

Once the merge is complete, we can now peek at our full combined matrix by once again clicking on the file name to see a small summary.

> <question-title></question-title>
>
> Each of these matrices/batches come from the same organism.
>
> 1. How much overlap in their detected genes did you expect?
> 2. How much overlap in their detected genes did you observe?
> 3. Why is this?
>
> > <solution-title></solution-title>
> >
> > 1. Given that they come from the same sample, and each matrix has ~15,000 genes, we would have expected a high overlap between matrices, yielding ~18,000 genes in the combined matrix if we assume a difference of +/- 500 genes per batch.
> >
> > 2. We observe 20,800 genes, giving less overlap between batches of the same organism than we originally thought.
> >
> > 3. The batches were sequenced at different time points along the organisms development, and therefore different genes were expressed/detected at different time points. For early development data, the overlap can be very sparse.
> {: .solution}
{: .question}

In the new combined matrix we see that we have 1536 cells, but this number is greatly overestimated.  This is because *not all batches use the same barcodes*, and yet we applied the full set of 192 barcodes against our FASTQ data during the [*Barcode Extraction*](#barcode-extraction) stage previously.

The reason we do this is to test for cross-contamination between batches, the details of which are better explained in the [*accompanying slides*]({% link topics/single-cell/tutorials/scrna-preprocessing/slides.html %}).


## Guarding against Cross-Contamination

There are multiple possible ways to configure a plate for sequencing multiple batches. Thankfully, Galaxy provides a tool that caters for this, and checks for cross-contamination in any experimental setup. It requires only the following information:

1. A full list of barcodes
2. Which barcodes apply to which batches
3. Which batches apply to which plates

Since the plating protocol we are using is that designed by the Freiburg MPI Grün lab, we will follow their structure.

> <details-title>Plating protocol</details-title>
>
> - Barcodes:
>     These are each 8bp long, with an edit distance of 2, and there 192 of them.
>
>     > |:-------:|:-----------:|
>     > | 001-006 | AACACC AACCTC AACGAG AACTGG AAGCAC AAGCCA |
>     > | 007-012 | AAGGTG AAGTGC ACAAGC ACAGAC ACAGGA ACAGTG |
>     > |    .    |        .  |
>     > |    .    |        .  |
>     > | 180-186 | TTACGC TTCACC TTCCAG TTCGAC TTCTCG TTGCAC |
>     > | 187-192 | TTGCGA TTGCTG TTGGAG TTGGCA TTGGTC TTGTGC |
>     {: .matrix}
>
> - Plates:
>     Here we have 8 batches spread out over 2 plates, with alternate barcode striping.
>
>     > |          | 001-096 | 097-192 | 001-096 | 097-192 |
>     > |----------+---------+---------+---------+---------|
>     > | Plate 1  |  B1     | B2      | B3      | B4      |
>     > | Plate 2  |  B5     | B6      | B7      | B8      |
>     {: .matrix}
>
{: .details}

This plating protocol can be converted into a more textual format, which allows for many variable setups (see *Help* section of *Cross-contamination Barcode Filter* tool).

    [Barcodes → Batches]
    001-096: B1 , B3 , B5 , B7
    097-192: B2 , B4 , B6 , B8

    [Plates → Batches]
    1: B1 , B2 , B3 , B4
    2: B5 , B6 , B7 , B8

Let us now apply this protocol to our count matrix, and look for any cross-contamination.

> <hands-on-title>Barcode Filtering</hands-on-title>
>
> Select **Cross-contamination Barcode Filter** {%icon tool %} with the following parameters:
>  - *"Input Matrix"*:`Column Join output` (merged matrices)
>  - *"Complete Barcodes"*:`celseq_barcodes.192.tabular` (barcodes file)
>  - *"Plate Protocol"*:`Custom`
>     - *"Under 'Barcode Format'"*:
>        - Select `+ Insert Barcode Format`:
>           - *"1: Barcode Format"*:
>              - *"Barcode Range: Start"*:`1`
>              - *"Barcode Range: End"*:`96`
>              - *"Batches utilizing this Range"*:`1,3,5,7`
>        - Select `+ Insert Barcode Format`:
>           - *"2: Barcode Format"*:
>              - *"Barcode Range: Start"*:`97`
>              - *"Barcode Range: End"*:`192`
>              - *"Batches utilizing this Range"*:`2,4,6,8`
>     - *"Under 'Plate Format'"*:
>        - Select `+ Insert Plate Format`:
>           - *"1: Plate Format"*:
>              - *"Plate Number"*:`1`
>              - *"Batches within this Plate Number"*:`1,2,3,4`
>        - Select `+ Insert Plate Format`:
>           - *"2: Plate Format"*:
>              - *"Plate Number"*:`2`
>              - *"Batches within this Plate Number"*:`5,6,7,8`
>  - Expand the *"RegEx Parameters"* section:
>     - *"RegEx to extract Plate, Batch, and Barcodes from headers"*:`.*P(\\d)_B(\\d)_([ACTG]+)`
>     - *"RegEx to replace Plate, Batch, and Barcodes from headers"*:`P\\1_B\\2_\\3`
>
{: .hands_on}

> <comment-title>Regular Expressions</comment-title>
> The [regular expression](https://www.regular-expressions.info/quickstart.html) (RegEx) used in the final steps of the above *Hands-On* is required in order to tell us how to capture the important information in the cell headers contained in brackets `(` `)`, where `\\d` denotes an expected digit, and `[ACTG]+` denotes 1 or more characters matching A or C or T or G.
>
> The information captured in the brackets `(` `)` can then be placed in the desired arrangement, where `Place \\1 Matches \\2 Here \\3` would place the first `\\d` after "Place ", the second after "Matches ", and so on.
>
{: .comment }

The plot that follows tells us everything we need to know about each of our batches. Each batch is essentially tested against the full set of barcodes in order to assert that only the desired or 'Real' barcodes have been sequenced.

### Cross-contamination Plot

![Contamination Plots]({% link topics/single-cell/images/scrna-pre-processing/scrna_crosscontamination.png %} "The Pre-filter and Post-filter plots")

Two things to take note of:

1. In the pre-filter plot, we can see how only half of the sequences in each batch map to half the barcodes. This shows very little cross-contamination, and proves that our data is real.
2. The post-filter plot essentially removes the false barcodes from each batch and retains only the 'Real' barcodes.

> <question-title></question-title>
>
> 1. The count matrix that is output from this tool has only half the number of cells as the original input count matrix. Why is this?
> 1. Which batches yield worrying levels of cross-contamination?
> 1. Which batches should we remove from all further analysis?
>
> > <solution-title></solution-title>
> >
> > 1. Because only half the barcodes in each batch were real. The *UMI-tools extract* took the entire barcodes file to filter against each batch, and the *UMI-tools count* also took the entire barcodes file to count against each batch. Naturally, each batch produced 192 cells, even though 96 were real. As a result of joining each of these matrices we ended up with a count-matrix of $$8 * 192 = 1536$$ cells. The cross-contamination tool removes the false barcodes (50% in each batch), resulting in $$768$$ cells.
> > 1. Batch 4 and Batch 6 both appear to have a significant number of mid-to-high range counts in cells under the *False Positives* section in these batches. This means that the cell barcodes that we should *not* be detecting in those batches, did in fact detect cells.
> > 1. Batch 4 does still have a good number of *True Positives*, despite the many mid-level *False Positives* so we could still perhaps use the cells from that Batch. Batch 6 on the other hand appears to derive most of its counts from the *False Positives*, and therefore is likely not so suitable for further analysis.
> {: .solution}
{: .question}

All *False Positives* from all batches are filtered out, leaving only the *True Positives* (or 'Real' barcodes) in the remaining filtered matrix. It is up to the user to filter further based on the contamination information that they derive from this step. With this, a QC filtered count-matrix is produced that can then be used for further downstream analysis.

<!--
Factoid: We can convert the number of UMIs to the number of molecules using a transformation script.
-->

# Conclusion


In this tutorial we have learned the importance of barcoding; namely how to define, extract, and annotate them from our reads and into the read headers, in order to preserve them during mapping. We have discovered how these barcoded reads are transformed into counts, where the cell barcode and UMI barcode are used to denote individual cells and to correct against reads that are PCR duplicates. Finally, we have learned how to combine separate batch data as well as being able to check and correct against cross-contamination.

![Recap of workflow]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow.svg %} "A recap of the entire workflow")

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %}).

tools_json:
["UMI-tools extract", "toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_extract/umi_tools_extract/0.5.5.1", "RNA-STAR", "toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.7a", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.9", "Filter BAM datasets on a variety of attributes", "toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1", "FeatureCounts", "toolshed.g2.bx.psu.edu/repos/iuc/featurecounts/featurecounts/2.0.1", "UMI-tools count", "toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_count/umi_tools_count/0.5.5.1"]


topic: single-cell
tutorial: scrna-data-ingest
title: Converting between common single cell data formats
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["data management", "data import"]
content_hash: fe1dac0ef1ba5eecaf7e9cb937af8c61
last_modified: 2026-05-27T09:01:49.747711
zenodo_link: https://zenodo.org/record/4574153
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-data-ingest/tutorial.html

content:
You finally decided to analyse some single cell data, you got your files either from the lab or publicly available sources, you opened the first tutorial available on Galaxy Training Network and... you hit the wall! The format of your files is not compatible with the one used in tutorial! Have you been there?
This tutorial was created to help you overcome that problem and ensure data interoperability in single cell analysis. Once you get your data into Galaxy in the right format, that's already 50% of success. Additionally, by using format conversion, you will be able to use different packages presented in tutorials that may require different datatypes.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Single cell datatypes

To start with, here are the most common formats and datatypes that you might come across if you work with single cell data:
- **Tabular** - simply using TSV, CSV or TXT formats to store expression matrix as well as cell and gene metadata.
- **MTX** - it's just a sparse matrix format with genes on the rows and cells on the columns as output by Cell Ranger.
- **HDF5** - Hierarchical Data Format - can store datasets and groups. A dataset is a  a multidimensional array of data elements, together with supporting metadata. A group is a structure for organizing objects in an HDF5 file. This format allows for storing both the count matrices and all metadata in a single file rather than having separate features, barcodes and matrix files.
- **AnnData objects** - [anndata](https://anndata.readthedocs.io/en/latest/) is a Python package for handling annotated data matrices. In Galaxy, you'll see AnnData objects in **h5ad** format, which is based on the standard HDF5 (h5) format. There are lots of Python tools that work with this format, such as Scanpy, MUON, Cell Oracle, SquidPy, etc.
- **Loom** - it is simply an HDF5 file that contains specific groups containing the main matrix as well as row and column attributes and can be read by any language supporting HDF5. [Loompy](https://linnarssonlab.org/loompy/) has been released as a Python API to interact with loom files, and [loomR](https://github.com/mojaveazure/loomR) is its implementation in R.
- **Zarr** - a Python package providing an implementation of compressed, chunked, N-dimensional arrays, designed for use in parallel computing. The Zarr file format offers powerful compression options, supports multiple data store backends, and can read/write your NumPy arrays.
- **Seurat objects** - a representation of single-cell expression data for R, in Galaxy you might see them in **rdata** format.
- **Single Cell Experiment (SCE) object** - defines a S4 class for storing data from single-cell experiments and provides a more formalized approach towards construction and accession of data. The S4 system is one of R's systems for object oriented programing. In Galaxy you might see SCE objects in **rdata** format.
- **CellDataSet (CDS) object** - the main class used by Monocle to hold single cell expression data. In Galaxy you might see CDS objects in **rdata** format.

<!---
TO FURTHER IMPROVE THE TUTORIAL:
include images showing the structure of those files
-->

{% snippet faqs/galaxy/tutorial_mode.md %}

# Data import
As you can see above, there are multiple ways to store single cell data. Therefore, there are also many ways how you can get that data!
Obviously, before any format conversion, we need to import the data. In our tutorials we often use [Zenodo](https://zenodo.org/) links, but you can also upload the files directly from your computer. There are also publicly available resources which you can easily access through public atlases, such as [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home) or [Human Cell Atlas data portal](https://data.humancellatlas.org/). We created a [dedicated tutorial]({% link topics/single-cell/tutorials/EBI-retrieval/tutorial.md %}) to show how to use those atlases to retrieve data. But today we're here to focus on data conversion!

# SCEasy Tool
In Galaxy Toolshed there is a wonderful tool called {% tool [SCEasy](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} which allows you to convert between common single cell formats, such as:
- AnnData to CellDataSet (CDS)
- AnnData to Seurat
- Loom to AnnData
- Loom to SingleCellExperiment (SCE)
- SingleCellExperiment (SCE) to AnnData
- SingleCellExperiment (SCE) to Loom
- Seurat to AnnData
- Seurat to SingleCellExperiment (SCE)

> <warning-title>Two SCEasy tools</warning-title>
>  
> As of the writing of this tutorial, the updated SCEasy tool is called {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} and it's only available on *usegalaxy.eu*. The second tool is called {% tool [SCEasy convert](toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1) %} and it works on *usegalaxy.org*, however has limited conversion options. Both tools should be visible on *singlecell.usegalaxy.eu* if you try to find them in the search box.
>
{: .warning}

In this tutorial you will see multiple examples of SCEasy Tool in action. However, sometimes it is useful to know how to do this conversion manually or at least to know how it all works and better understand the structure of the files. Therefore, we also have examples showing how to convert objects manually or prepare the input files for some of our single cell tutorials and workflows.

# Seurat -> AnnData
As mentioned above, there are two SCEasy tools currently available. When we speak about conversion to AnnData, the latest tool {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} converts Seurat into the latest version of AnnData object. However, currently it is not possible to use that output with downstream EBI single-cell tools because they only support older versions of the AnnData. If you need to use those tools, the solution is to take advantage of the second SCEasy tool {% tool [SCEasy convert](toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1) %} which has limited conversion options but can generate AnnData files compatible with the EBI single-cell tools that we are going to use next in the workflow.
Our Seurat starting file was generated by pulling data from [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home) and transforming it into the desired format, what was shown in the [data import tutorial]({% link topics/single-cell/tutorials/EBI-retrieval/tutorial.md %}). It will be our toy dataset.

> <hands-on-title>Get toy data</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo](https://zenodo.org/records/10397653)
>
>    ```
>    https://zenodo.org/records/10397653/files/Seurat_object.rdata
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>    
> 3. Alternatively, you can import history where we created the Seurat object: [Input history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/ebi-scxa---anndata-scanpy-or-seurat-object-1)
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 4. **Rename** {% icon galaxy-pencil %} the dataset `Seurat object`
> 5. Check that the datatype is `rdata`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="rdata" %}
>
{: .hands_on}

You will now have the option of using one tool to do the conversion, or going through a more complex conversion to an anndata object usable in the [**Filter, plot and explore single-cell RNA-seq data with Scanpy** tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %})

{% include _includes/cyoa-choices.html option1="Quick one tool method" option2="Convert to AnnData object" default="Quick one tool method"
       text="Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!" disambiguation="seurat2anndata" %}

<div class="Quick-one-tool-method" markdown="1">

## Quick one tool method

> <hands-on-title> SCEasy Converter </hands-on-title>
>
> 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `Seurat to AnnData`
>    - {% icon param-file %} *"Input object in rds,rdata format"*: `Seurat object`
> 2. **Rename** {% icon galaxy-pencil %} the output: `AnnData object`
>    
{: .hands_on}

And that's it! Please note that the output file is the newest AnnData version and is not compatible with EBI tools used in [Filter, Plot, Explore workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/copy-of-filter-plot-and-explore-single-cell-rna-seq-data-imported-from-uploaded-file-3).

</div>

<div class="Convert-to-AnnData-object" markdown="1">

## Convert to AnnData object compatible with Filter, Plot, Explore workflow

Here, we use the other SCEasy tool which generates older version of the AnnData which is compatible with the EBI single-cell tools that we are going to use next in the workflow.

> <hands-on-title> Seurat to AnnData with SCEasy convert </hands-on-title>
>
> 1. {% tool [SCEasy convert](toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1) %} with the following parameters:
>    - *"Direction of conversion"*: `Seurat to AnnData`
>    - {% icon param-file %} *"Input object in Seurat RDS format"*: `Seurat object`
>      (if the dataset does not show up in the corresponding input field, just drag the dataset from the history panel and drop into the input field)
>    - *"Name of the assay to be transferred"*: `RNA`
>    - *"Data type of the assay to be transferred"*: `data`
>   
> 2. **Rename** {% icon galaxy-pencil %} the output `Converted EBI-compatible AnnData file`.
>
> 3. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>    
{: .hands_on}

Now, if we do small modification to metadata, we can use the generated dataset as an input file in the workflow we created in the [previous tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %})! Let's see how it works:

> <hands-on-title> Modify AnnData object</hands-on-title>
>
> 1. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92) %}
> 2. Make sure you are using version **1.8.1+galaxy92** of the tool (change by clicking on {% icon tool-versions %} Versions button)
> 3. Set the following parameters:
>    - {% icon param-file %} In *"Input object in hdf5 AnnData format"*: `Converted EBI-compatible AnnData file`
>    - In *"Change field names in AnnData var"*:
>        - {% icon param-repeat %} *"Insert Change field names in AnnData var"*
>            - *"Original name"*: `name`
>            - *"New name"*: `Symbol`
>    - *"Gene symbols field in AnnData"*: `Symbol`
>    - In *"Flag genes that start with these names"*:
>        - {% icon param-repeat %} *"Insert Flag genes that start with these names"*
>            - *"Starts with"*: `mt-`
>            - *"Var name"*: `mito`
> 4. Rename {% icon galaxy-pencil %} output `AnnData for Filter, Plot, Explore workflow`
>
{: .hands_on}

The object is ready to start its journey through the [Filter, Plot, Explore workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/copy-of-filter-plot-and-explore-single-cell-rna-seq-data-imported-from-uploaded-file-3). All thanks to one conversion tool - how awesome that is!

</div>

# AnnData -> Seurat

Let's get an **AnnData object** that we can further work on. It's the object used in [many tutorials]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}), so check it out if you're curious.

> <hands-on-title>Get toy data</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    If you do this tutorial just for learning purposes, you can download the downsampled dataset which will be much quicker to process:
>    
>    ```
>    https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
>    ```
>    
>    If you want to use the full dataset used in the other single-cell case study tutorials, here it is! Please note, it will take much longer to process it, so we will only show the conversions on the downsampled objects.
>    
>    ```
>    https://zenodo.org/record/7053673/files/Mito-counted_AnnData
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. **Rename** {% icon galaxy-pencil %} the datasets `Downsampled AnnData object`
> 5. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}


{% include _includes/cyoa-choices.html option1="Quick one step method" option2="Manual conversion" default="Quick one step method"
       text="Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!" disambiguation="anndata2seurat" %}

<div class="Quick-one-step-method" markdown="1">

## Quick one step method

> <hands-on-title> SCEasy Converter </hands-on-title>
>
> 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `AnnData to Seurat`
>    - {% icon param-file %} *"Input object in h5ad,h5 format"*: `Downsampled AnnData object`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Seurat object`
>    
{: .hands_on}

</div>

<div class="Manual-conversion" markdown="1">

## Manual conversion

Most of our manual conversions involve extracting tables from different data objects and importing them into the target object.
First, we will extract observations (cell metadata) and the full matrix from our AnnData.

> <hands-on-title> Inspect AnnData </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>   
> 2. **Rename** {% icon galaxy-pencil %} the output `Observations`.
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. **Rename** {% icon galaxy-pencil %} the output `Matrix`.
>    
{: .hands_on}


> <question-title></question-title>
>
> What are the rows, and what are the columns in the retrieved Matrix?
>
> > <solution-title></solution-title>
> >
> > If you just click on the `Matrix` dataset, you will see a preview showing barcodes in the first column, and genes in the first row.
> >
> {: .solution}
>
{: .question}

However, the next tool we need expects a matrix wherein the genes are listed in the first column and the barcodes are listed in the first row. Therefore, we need to transpose the current matrix.

> <hands-on-title> Transpose the matrix </hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `Matrix`
>
{: .hands_on}

And now we are ready to input that data into **DropletUtils** tool, which will separate this matrix into the cells, genes, and matrix tabular files needed to build a Seurat object.

> <hands-on-title> DropletUtils </hands-on-title>
>
> 1. {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %} with the following parameters:
>    - *"Format for the input matrix"*: `Tabular`
>    - {% icon param-file %} *"Count Data"*: output of **Transpose** {% icon tool %}
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `DefaultDrops`
>            - *"Expected Number of Cells"*: `338`
>            - *"Upper Quantile"*: `1.0`
>            - *"Lower Proportion"*: `0.0`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - *"Random Seed"*: `100`
>
{: .hands_on}

Finally, let's combine those files that we have just generated and turn them into the Seurat object!

> <hands-on-title> Create Seurat object </hands-on-title>
>
> 1. {% tool [Seurat Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/3.2.3+galaxy0) %} with the following parameters:
>    - *"Choose the format of the input"*: `10X-type MTX`
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `DropletUtils 10X Matrices`
>    - *"Gene table"*: `DropletUtils 10X Genes`
>    - *"Barcode/cell table"*: `DropletUtils 10X Barcodes`
>    - {% icon param-file %} *"Cell Metadata"*: `Observations`
>    - *"Choose the format of the output"*: `RDS with a Seurat object`
>
> 2. **Rename** {% icon galaxy-pencil %} the output `Converted Seurat object`.
>    
{: .hands_on}

As usual, you can check the [example history](https://usegalaxy.eu/u/j.jakiela/h/anndata-to-seurat-conversion) and the dedicated [workflow](https://usegalaxy.eu/u/j.jakiela/w/anndata---seurat-conversion).

</div>

# AnnData -> SingleCellExperiment (SCE)
We will work on the same AnnData object so if you create a new history for this exercise, you can either get this file from Zenodo again or just copy this dataset from the previous history.

> <hands-on-title>Get toy data</hands-on-title>
>
> 1. Create a new history for this section
> 2. Import the files from [Zenodo](https://zenodo.org/record/10391629)
>
>    ```
>    https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
>    ```
>
{: .hands_on}

{% snippet faqs/galaxy/histories_copy_dataset.md %}


{% include _includes/cyoa-choices.html option1="Quick single tool method" option2="Convert manually" default="Quick single tool method"
       text="Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!" disambiguation="anndata2sce" %}

 <div class="Quick-single-tool-method" markdown="1">

## Quick single tool method

> <hands-on-title> SCEasy Converter </hands-on-title>
>
> 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `AnnData to Seurat`
>    - {% icon param-file %} *"Input object in h5ad,h5 format"*: `Downsampled AnnData object`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Seurat object`
> 3. We will use SCEasy again! {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `Seurat to SingleCellexperiment`
>    - {% icon param-file %} *"Input object in rds,rdata format"*: `Seurat object`
> 4. **Rename** {% icon galaxy-pencil %} the output: `SCE object`
>
{: .hands_on}

</div>

<div class="Convert-manually" markdown="1">

## Convert manually

First, we will extract observations and the full matrix from our AnnData.

> <tip-title>Skip some steps!</tip-title>
>
> If you are following this entire tutorial (rather than using a specific section necessary!), you can stay in the your previous history and just reuse outputs to build different single cell objects!
{: .tip}


> <hands-on-title> Inspect AnnData </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>   
> 2. **Rename** {% icon galaxy-pencil %} the output `Observations`.
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. **Rename** {% icon galaxy-pencil %} the output `Matrix`.
>    
{: .hands_on}


> <question-title></question-title>
>
> What are the rows, and what are the columns in the retrieved Matrix?
>
> > <solution-title></solution-title>
> >
> > If you just click on the `Matrix` dataset, you will see a preview, showing barcodes in the first column, while genes are in the first row.
> >
> {: .solution}
>
{: .question}


However, the next tool we need expects a matrix wherein the genes are listed in the first column and the barcodes are listed in the first row. Therefore, we need to transpose the current matrix.

> <hands-on-title> Transpose the matrix </hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `Matrix`
>
{: .hands_on}

And now we are ready to input that data to **DropletUtils** tool.

> <hands-on-title> DropletUtils </hands-on-title>
>
> 1. {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %} with the following parameters:
>    - *"Format for the input matrix"*: `Tabular`
>    - {% icon param-file %} *"Count Data"*: output of **Transpose** {% icon tool %}
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `DefaultDrops`
>            - *"Expected Number of Cells"*: `338`
>            - *"Upper Quantile"*: `1.0`
>            - *"Lower Proportion"*: `0.0`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - *"Random Seed"*: `100`
>
{: .hands_on}

Finally, let's combine those files that we have just generated and turn them into the SingleCellExperiment!

> <hands-on-title> Create SCE object </hands-on-title>
>
> 1. {% tool [DropletUtils Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_read_10x/dropletutils_read_10x/1.0.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `DropletUtils 10X Matrices`
>    - {% icon param-file %} *"Gene table"*: `DropletUtils 10X Genes`
>    - {% icon param-file %} *"Barcode/cell table"*: `DropletUtils 10X Barcodes`
>    - *"Should metadata file be added?"*: {% icon param-toggle %} `Yes`
>        - {% icon param-file %} *"Metadata file"*: `Observations`
>        - *"Cell ID column"*: `index`
>     
> 2. **Rename** {% icon galaxy-pencil %} the output `Converted SCE object`.
>
{: .hands_on}

As usual, you can check the [example history](https://usegalaxy.eu/u/j.jakiela/h/anndata-to-singlecellexperiment-sce-conversion) and the dedicated [workflow](https://usegalaxy.eu/u/j.jakiela/w/anndata-to-singlecellexperiment-sce-conversion).

</div>

# Anndata -> Cell Data Set (CDS)
Cell Data Set (CDS) format is usually used when working with a package called Monocle3 ({% cite monocle3 %}). Below we show two methods on how to transform AnnData to CDS object, one of which creates an input file for [Monocle 3 tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}).

{% include _includes/cyoa-choices.html option1="Create CDS input for Monocle3 tutorial" option2="General AnnData to CDS conversion" default="General AnnData to CDS conversion"
       text="You can choose whether you want just to transform AnnData to CDS or to create CDS input file for Monocle3 tutorial to proceed with the downstream analysis described in that tutorial. Please note that depending on your dataset, you might need to refer to the first method which uses both annotated and unprocessed matrices. If you did some pre-processing on your AnnData object, then you might need to choose the first method anyway since Monocle3 performs its own pre-processing, so we would also need an unprocessed expression matrix alongside annotated (pre-processed) AnnData. That method is more expanded and specific, while the general one just shows the main principle of the conversion." disambiguation="anndata2cds" %}

<div class="Create-CDS-input-for-Monocle3-tutorial" markdown="1">

## CDS input for Monocle3 tutorial (use for pre-processed data)
The [dedicated tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}) shows how to perform trajectory analysis using Monocle3 which is the next step in the single-cell case study tutorial series, right after [pre-processing tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md %}) and [analysing the metadata]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}). To keep the continuity of the series, we will continue to work on the case study data from a mouse model of fetal growth restriction {% cite Bacon2018 %} (see [the study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and [the project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)). After successfully completing this section with the mentioned dataset, you can use it directly in the [Monocle3 tutorial workflow](https://humancellatlas.usegalaxy.eu/u/j.jakiela/w/copy-of-trajectory-analysis-using-monocle3-1).
If you work on your own data, you might also need to follow this method since it shows how to deal with already pre-processed datasets. Since Monocle3 performs its own pre-processing, you will need both annotated and unprocessed matrices.

## Get data
Monocle3 works great with annotated data, so we will make use of our annotated AnnData object, generated in the [previous tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}). We will also need a ‘clean’ expression matrix, extracted from the AnnData object just before we started the processing.
You have two options for uploading these datasets. Importing via history is often faster.

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. Import history from: [input history](https://humancellatlas.usegalaxy.eu/u/j.jakiela/h/monocle3-input-files)
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the history to your name of choice.
>
{: .hands_on}


><hands-on-title>Option 2: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo](https://zenodo.org/records/7078524)
>
>    ```
>    https://zenodo.org/records/7078524/files/AnnData_before_processing.h5ad
>    https://zenodo.org/records/7078524/files/Annotated_AnnData.h5ad
>    ```
>
> 3. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}


## Extracting annotations
To run Monocle, we need cell metadata, gene metadata, and an expression matrix file of genes by cells. (In theory, the expression matrix alone could do, but then we wouldn’t have all those useful annotations that we worked on so hard in the previous tutorials!). In order to get these files, we will extract the gene and cell annotations from our AnnData object.

> <question-title></question-title>
>
> How many lines do you expect to be in the gene and cell metadata files?
>
> > <solution-title></solution-title>
> >
> > If you click on the step with uploaded annotated AnnData file, you will see on a small preview that this object has 8605 observations and 15395 variables, so we expect to get a cell metadata file with 8605 lines and gene metadata file with 15395 lines (without headers of course!).
> >
> {: .solution}
>
{: .question}


> <hands-on-title>Extracting annotations</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Annotated_AnnData`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
> 2. Rename {% icon galaxy-pencil %} the observations annotation `Extracted cell annotations (obs)`
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Annotated_AnnData`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 4. Rename {% icon galaxy-pencil %} the annotation of variables `Extracted gene annotations (var)`
>
>
{: .hands_on}

Quick and easy, isn’t it? However, we need to make some minor changes before we can input these files into the Monocle toolsuite.

## Cell metadata
Our current dataset is not just T-cells: as you might remember from the last tutorial, we identified a cluster of macrophages as well. This might be a problem, because the trajectory algorithm will try to find relationships between all the cells (even if they are not necessarily related!), rather than only the T-cells that we are interested in. We need to remove those unwanted cell types to make the analysis more accurate.

The **Manipulate AnnData** tool allows you to filter observations or variables, and that would be the most obvious way to remove those cells. However, given that we don't need an AnnData object, it's a lot quicker to edit a table rather than manipulate an AnnData object. Ultimately, we need cell metadata, gene metadata and expression matrix files that have macrophages removed, and that have the correct metadata that Monocle looks for. With some table manipulation, we’ll end up with three separate files, ready to be passed onto Monocle3.


> <question-title></question-title>
>
> Where is the information about cell types stored?
>
> > <solution-title></solution-title>
> >
> > We have already extracted the cell annotations file - in one of the columns you can find the information about cell type, assigned to each cell.
> > ![Cell annotations along the top, n_genes, n_counts, louvain, cell_type with a cell barcode and subsequent metadata as each row](../../images/scrna-casestudy-monocle/example_cell_annotations.png "Example cell annotations")
> >
> {: .solution}
>
{: .question}

Click on `Extracted cell annotations (obs)` file to see a small preview window. This shows you that the column containing the cell types has number 22.  We’ll need that to filter out unwanted cell types!

> <warning-title>Check the column number!</warning-title>
> If you are working on a different dataset, the number of the ‘cell_type’ column might be different, so make sure you check it on a preview and use the correct number!
{: .warning}

> <hands-on-title>Filter out macrophages</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Extracted cell annotations (obs)`
>    - *"With following condition"*: `c22!='Macrophages'`
>    - *"Number of header lines to skip"*: `1`
>    - That’s it - our cell annotation file is ready for Monocle! Let’s rename it accordingly.
> 2. **Rename** {% icon galaxy-pencil %} the output: `Cells input data for Monocle3`
>
>    > <details-title>Parameters</details-title>
>    >
>    > - `c22` means column no. 22 - that's the column with cell types, and it will be filtered for the macrophages
>    > - `!=` means 'not equal to' - we want to keep the cell types which ARE NOT macrophages
>    {: .details}
>
>    > <tip-title>Other unwanted cell types</tip-title>
>    >
>    > It might happen that during clustering you’ll find another cell type that you want to get rid of for the trajectory analysis. Then simply re-run this tool on already filtered file and change ‘Macrophages’ to another unwanted cell type.
>    {: .tip}
{: .hands_on}

## Gene annotations
Sometimes certain functionalities require a specific indication of where the data should be taken from. Monocle3 tools expect that the genes column is named ‘gene_short_name’. Let's check what the name of that column is in our dataset currently.

> <question-title></question-title>
>
> 1. Where can you check the header of a column containing genes names?
> 2. What is the name of this column?
>
> > <solution-title></solution-title>
> >
> > 1. Our extracted gene annotations file! Either by clicking on the eye icon {% icon solution %} or having a look at the small preview window.
> > 2. In our dataset the gene names are stored in a column called ‘Symbol’ - we need to change that!
> > ![The dataset in the history has a preview window showing the columns of the extracted gene annotation with each gene as a row and the metadata - index, ID, symbol - as the column names](../../images/scrna-casestudy-monocle/window_in_history.png "Preview window in the history")
> >
> {: .solution}
>
{: .question}

Let’s click on the `Extracted gene annotations (var)` file to see a small preview. We can see that the gene names are in the third column with a header `Symbol`. Keep that in mind - we’ll use that in a second!

> <hands-on-title>Changing the column name</hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `Extracted gene annotations (var)`
>    - *"using column"*: `c3` or `Column: 3`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Symbol`
>            - *"Replacement"*: `gene_short_name`
> 2. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>     - Voila! That’s the gene input for Monocle! Just a quick rename...
> 3. **Rename** {% icon galaxy-pencil %} the output: `Genes input data for Monocle3`
>
{: .hands_on}

## Expression matrix
Last, but not least! And in fact, the most important! The expression matrix contains all the values representing the expression level of a particular gene in a cell. This is why in theory the expression matrix is the only input file required by Monocle3. Without annotation files the CDS data can still be generated - it will be quite bare and rather unhelpful for interpretation, but it's possible to process.
So, the values in the expression matrix are just numbers. But do you remember that we have already done some processing such as normalisation and the calculation of principal components in the AnnData object in the previous tutorial? That affected our expression matrix. Preprocessing is one of the steps in the Monocle3 workflow, so we want to make sure that the calculations are done on a ‘clean’ expression matrix. If we apply too many operations on our raw data, it will be too ‘deformed’ to be reliable. The point of the analysis is to use algorithms that make the enormous amount of data understandable in order to draw meaningful conclusions in accordance with biology.
So how do we do that?

> <question-title></question-title>
>
> 1. How many cells and genes are there in the `Anndata_before_processing` file?
> 2. How many lines are there in `Cells input data for Monocle3`?
> 3. How many lines are there in `Genes input data for Monocle3`?
>
> > <solution-title></solution-title>
> > You can answer all the questions just by clicking on the given file and looking at the preview window.
> > 1. [n_obs x n_vars] = 31178 x 35734, so there are 31178 cells and 35734 genes.
> > 2. 8570 lines, including a header, which makes 8569 cells.
> > 3. 15396 lines, including a header, which makes 15395 genes.
> >
> {: .solution}
>
{: .question}

As you can see, there are way more genes and cells in the unprocessed AnnData file, so the expression matrix is much bigger than we need it to be. If the genes and cells we prepared for Monocle3 are not the same as in the expression matrix, Monocle3 will crash. Therefore, we have to filter that big, clean matrix and adjust it to our already prepared genes and cells files. But first, let’s extract the matrix from the unprocessed AnnData object.

> <hands-on-title>Extracting matrix</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `AnnData_before_processing`
>    - *"What to inspect?"*: `The full data matrix`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Unprocessed expression matrix`
>
{: .hands_on}

If you have a look at the preview of `Unprocessed expression matrix`, you’ll see that the first column contains the cell barcodes, while the first row - the gene IDs. We would like to keep only the values corresponding to the cells and genes that are included in `Cells input data for Monocle3` and `Genes input data for Monocle3`. How do we do it? First, we compare the cell barcodes from `Cells input data for Monocle3` to those in `Unprocessed expression matrix` and ask Galaxy to keep the values of the matrix for which the barcodes in both files are the same. Then, we’ll do the same for gene IDs. We will cut the first columns from `Cells input data for Monocle3` and `Genes input data for Monocle3` to be able to compare those columns side by side with the matrix file.

> <hands-on-title>Cutting out the columns</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `Cells input data for Monocle3`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Cells IDs`
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `Genes input data for Monocle3`
> 4. **Rename** {% icon galaxy-pencil %} the output: `Genes IDs`
>
{: .hands_on}

> <hands-on-title> Filter matrix values by cell barcodes</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `Cells IDs`
>    - *"using column"*: `c1`or `Column: 1`
>    - {% icon param-file %} *"with"*: `Unprocessed expression matrix`
>    - *"and column"*: `c1`or `Column: 1`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>    - *"Keep lines of first input that are incomplete"*: `Yes`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `Yes`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Pre-filtered matrix (by cells)`
>
{: .hands_on}

Look at the preview of the output file. First of all, you can see that there are 8570 lines (8569 cells) instead of 31178 cells that were present in the matrix. That’s exactly what we wanted to achieve - now we have raw information for the T-cells that we have filtered. However, the step that we have already performed left us with the matrix whose first and second columns are the same - let’s get rid of one of those!

> <hands-on-title>Remove duplicate column (cells IDs)</hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `Pre-filtered matrix (by cells)`
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"List of Fields"*: `c1`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Filtered matrix (by cells)`
>
{: .hands_on}

Now we will perform the same steps, but for gene IDs. But gene IDs are currently in the first row, so we need to transpose the matrix, and from there we can repeat the same steps as above for Gene IDs.

> <hands-on-title>Filter matrix by gene IDs</hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `Filtered matrix (by cells)`
>    - The matrix is now ready to be filtered by gene IDs!
> 2. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `Genes IDs`
>    - *"using column"*: `c1` or `Column: 1`
>    - {% icon param-file %} *"with"*: output of **Transpose** {% icon tool %}
>    - *"and column"*: `c1` or `Column: 1`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>    - *"Keep lines of first input that are incomplete"*: `Yes`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `Yes`
> 3. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: output of **Join two Datasets** {% icon tool %}
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"List of Fields"*: `c1`
>    -  Monocle3 requires that in the matrix rows are genes, and columns are cells - that is what we've got, so there is no need to transpose matrix again. The expression matrix is ready! Let's just rename it...
> 4. **Rename** {% icon galaxy-pencil %} the output: `Expression matrix for Monocle3`
>
{: .hands_on}

{% icon congratulations %} Finally! We have prepared all the files to pass them onto the Monocle3 workflow!

## Creating CDS object
Monocle3 turns the expression matrix, cell and gene annotations into an object called cell_data_set (CDS), which holds single-cell expression data.

> <details-title>Input files</details-title>
>
> Here is what [Monocle3 documentation](https://cole-trapnell-lab.github.io/monocle3/docs/starting/) says about the required three input files:
>    - **expression_matrix**: a numeric matrix of expression values, where rows are genes, and columns are cells. Must have the same number of columns as the cell_metadata has rows and the same number of rows as the gene_metadata has rows.
>    - **cell_metadata**: a data frame, where rows are cells, and columns are cell attributes (such as cell type, culture condition, day captured, etc.)
>    - **gene_metadata**: a data frame, where rows are features (e.g. genes), and columns are gene attributes, such as biotype, gc content, etc. One of its columns should be named "gene_short_name", which represents the gene symbol or simple name (generally used for plotting) for each gene.
>
{: .details}

> <hands-on-title>Create CDS object</hands-on-title>
>
>    > <details-title>Data format</details-title>
>    >
>    > You can provide expression matrix as TSV, CSV, MTX or RDS file, while genes and cells metadata as TSV, CSV or RDS files. In our case all three files are tabular, so we will set the format to TSV.
>    {: .details}
> 1. {% tool [Monocle3 create](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_create/monocle3_create/0.1.4+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Expression matrix, genes as rows, cells as columns. Required input. Provide as TSV, CSV or RDS."*: `Expression matrix for Monocle3`
>    - *"Format of expression matrix"*: `TSV`
>    - {% icon param-file %} *"Per-cell annotation, optional. Row names must match the column names of the expression matrix. Provide as TSV, CSV or RDS."*: `Cells input data for Monocle3`
>    - *"Format of cell metadata"*: `TSV`
>    - {% icon param-file %} *"Per-gene annotation, optional. Row names must match the row names of the expression matrix. Provide as TSV, CSV or RDS."*: `Genes input data for Monocle3`
>    - *"Format of gene annotation"*: `TSV`
>
> 2. **Rename** {% icon galaxy-pencil %} the output: `CDS input for Monocle3 tutorial`
{: .hands_on}

It was quite a long conversion, but we did it! If you're interested, the "Tip" below describes how we could possibly speed up the process using alternative tools.

> <tip-title> Alternative tools </tip-title>
>
> Generally in coding you can use different ways to achieve the same result. Similarly, in Galaxy you can use different tools but the outcome will be the same.
> For example, we could have used some tools on our starting AnnData object to remove macrophages and rename the column header, and afterwards extract observations and variables. It is a good practice to use methods from the libraries (such as Scanpy below) rather than dealing with text files rows and columns. The shown approach works well, but it might be problematic with much bigger datasets sizes. Then, you can use the following route:
> 1. {% tool [Scanpy FilterCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_filter_cells/scanpy_filter_cells/1.8.1+galaxy93) %} on annotated Anndata to remove macrophages.
> 2. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92) %} to change the genes name from *Symbol* to *gene_short_name*.
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} to extract genes and cells (to be used for Monocle).
> 4. Follow further steps to filter unprocessed matrix.
>
> You've probably used the mentioned tools before, so **challenge yourself** and try to replicate the process using the tools above!
{: .tip}

You might want to consult your results with this [control history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/anndata---cds-conversion-for-monocle3-tutorial), or check out the [workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/copy-of-trajectory-analysis-using-monocle3).
And now you have your input file ready to start the [Monocle3 tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %})!

</div>

<div class="General-AnnData-to-CDS-conversion" markdown="1">

## General AnnData to CDS conversion
In fact, you can do this conversion just in one step - check the {% icon tip %} **Tip** below!

> <tip-title>Conversion in a single step</tip-title>
>
> As mentioned in the previous section, SCEasy tool can do this conversion in a single step. Let's try that out!
>
>   > <hands-on-title> SCEasy Converter </hands-on-title>
>   >
>   > 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>   >    - *"Convert From / To"*: `AnnData to CellDataSet`
>   >    - {% icon param-file %} *"Input object in h5ad,h5 format"*: `Downsampled AnnData object`
>   > 2. **Rename** {% icon galaxy-pencil %} the output: `CDS object`
>   >
>   {: .hands_on}
>
{: .tip}

Below we will also show the manual conversion, just to make you familiar with the structure of the files and operations needed.
We will continue working on previously used dataset, so you can copy it from your history or download from Zenodo.

> <hands-on-title>Get toy data again</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. You can either copy the previously used dataset from your history:
>
>    {% snippet faqs/galaxy/histories_copy_dataset.md %}
>
>    Or, alternatively, download the dataset from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
>    ```
>
> 3. **Rename** {% icon galaxy-pencil %} the datasets `Downsampled AnnData object`
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}

Now we just need to extract information about cells, genes and an expression matrix.

> <hands-on-title> Inspect AnnData </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
> **Rename**  {% icon galaxy-pencil %} the output `Cell barcodes (obs)`.
>
> 2. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> **Rename**  {% icon galaxy-pencil %} the output `Genes (var)`.
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `output` (Input dataset)
>    - *"What to inspect?"*: `The full data matrix`
>   
> **Rename**  {% icon galaxy-pencil %} the output `Expression matrix`.
>      
{: .hands_on}

Hold on here! As mentioned, if you're converting your files to CDS, you'll probably be working with Monocle. There is one function in downstream analysis in Monocle that requires a specific name of the column containing gene symbols, and that is `gene_short_name`. If you use Galaxy buttons for the analysis, you won't be able to change that name after you create CDS file, so a good piece of advice is to rename it at this stage. There is no harm in doing this, and it might save you some time and frustration later on. You only need to check which column contains the gene symbols and what is its header - you can check that in the preview window, simply by clicking on the `Genes` dataset. In our case, that's column 3 and its name is `Symbol`. Let's change that!

> <hands-on-title>Changing the column name</hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `Genes`
>    - *"using column"*: `c3` or `Column: 3`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Symbol`
>            - *"Repl

tools_json:
["SCEasy", "toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2", "SCEasy Converter", "SCEasy convert", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1", "AnnData Operations", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0", "Transpose", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy0", "DropletUtils", "toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2", "Seurat Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/3.2.3+galaxy0", "DropletUtils Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_read_10x/dropletutils_read_10x/1.0.4+galaxy0", "Filter", "Filter1", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.2", "Cut", "Cut1", "Join two Datasets", "join1", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0+galaxy2", "Monocle3 create", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_create/monocle3_create/0.1.4+galaxy2", "Scanpy FilterCells", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_filter_cells/scanpy_filter_cells/1.8.1+galaxy93", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy1", "FASTQ to Tabular", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_to_tabular/fastq_to_tabular/1.1.5", "Select last", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/1.1.0", "Tabular to FASTQ", "toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fastq/tabular_to_fastq/1.1.5"]


