# tutorials

topic: single-cell
tutorial: scrna-scater-qc
title: Single-cell quality control with scater
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["[]"]
content_hash: 94ca86e710144c46b3f7479f2ee1f01d
last_modified: 2026-05-15T22:28:16.825411
zenodo_link: https://zenodo.org/record/3386291
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-scater-qc/tutorial.html

content:
Single-cell RNA-seq (scRNA-seq) is emerging as a promising technology for analysing variability in cell populations. However, the combination of technical noise and intrinsic biological variability makes detecting technical artefacts particularly challenging. Removal of low-quality cells and detection of technical artefacts is critical for accurate downstream analysis.

A number of factors should be examined before downstream analyses, many of which we'll address here:

- **Low library size**: When cells are very degraded or absent from the library preparation, the number of reads sequenced from that library will be very low. It's important to remove these cells from downstream analyses.

- **Low number of expressed genes**: A low number of expressed genes may be a result of poor-quality cells (e.g. dying, degraded, damaged, etc.), followed by high PCR amplification of the remaining RNA. Again, these cells should be removed from downstream analyses.

- **High mitochondrial gene content**: High concentrations of mitochondrial genes is often a result of damaged cells where the endogenous RNA escapes or degrades. As mitochondria has its own cell membranes, it is often the last DNA/RNA in damaged cells to degrade and hence occurs in high quantities during sequencing.

- **Batch effect**: Large scRNA-seq projects usually need to generate data across multiple batches due to logistical constraints. However, the processing of different batches is often subject to variation, e.g., changes in operator, differences in reagent quality and concentration, the sequencing machine used, etc. This results in systematic differences in the observed expression in cells from different batches, which we refer to as “batch effects”. Batch effects are problematic as they can be major drivers of variation in the data, masking the relevant biological differences and complicating interpretation of the results.

We will use *scater* ({% cite 10.1093/bioinformatics/btw777 %}) to visualise scRNA-seq data, obtaining information about the factors mentioned above, filter out low-quality cells and confirm that filtering has worked. We'll then look at confounding factors such as batch effect to see if the data is biased to any technical artifacts.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}



# Overview
We will use a pre-calculated expression matrix, along with some additional metadata such as lists of mitochondrial genes and annotation of technical information for each sequencing library. We will plot the data and carry out quality control filtering based on the visualise-filter-visualise paradigm of this approach (see workflow below).

![Workflow](../../images/scrna-scater-qc/Figure1.jpg "Workflow demonstrating the visualise-filter-visualise paradigm of the software")


# Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo](https://zenodo.org/record/3386291) or from the shared data library
>
>    ```
>    https://zenodo.org/record/3386291/files/annotation.txt
>    https://zenodo.org/record/3386291/files/counts.txt
>    https://zenodo.org/record/3386291/files/mt_controls.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
{: .hands_on}

# Visualise the data

Take a look at the uploaded data by clicking on the {% icon galaxy-eye %} symbol for each dataset.

- The `counts.txt` file is a 40-sample expression matrix. Each sample (Cell_001 - Cell_040) is listed as the column headers and the start of each row is a gene name. The rest of the data refers to the number of reads mapped to each gene/sample.
- `annotation.txt` is a file listing experimental information about each cell. Parameters here include `Mutation_Status`, `Cell_Cycle`, and `Treatment`. These will be useful for looking at  batch effects later.
- The `mt_controls.txt` file is a list of mitochondrial genes. This list will be used later to calculate the % of mitochondrial reads in each sequencing library.


> <hands-on-title>Calculate QC metrics</hands-on-title>
>
> 1. {% tool [Scater: Calculate QC metrics](toolshed.g2.bx.psu.edu/repos/iuc/scater_create_qcmetric_ready_sce/scater_create_qcmetric_ready_sce/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Expression matrix in tabular format"*: `counts.txt` (Input dataset)
>    - {% icon param-file %} *"Format dataset describing the features in tabular format"*: `annotation.txt` (Input dataset)
>    - {% icon param-file %} *"Dataset containing the list of the mitochondrial control genes"*: `mt_controls.txt` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > The output of this tool is a SingleCellExperiment object in [Loom](http://loompy.org/) format, which contains all the information from the input files, along with a host of other quality control metrics, calculated from the input data.
>    {: .comment}
>
{: .hands_on}

Next, lets take a look at the data by plotting various properties to see what our data looks like.

> <hands-on-title>Plot library QC</hands-on-title>
>
> 1. {% tool [Scater: plot library QC](toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_dist_scatter/scater_plot_dist_scatter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: Calculate QC metrics** {% icon tool %})
>     - {% icon param-check %} *"Plot on log scale"*: `No`
>
> 2. If we have a large number of cells (500+), set the 'Plot on log scale' option to 'Yes'. This will make it easier to pick cut-offs when dealing with large numbers. When the tool has finished running, click on the {% icon galaxy-eye %} to view the plots. If it doesn't appear in the browser, you may have to download it and view it externally. You should be presented with plots similar to those below.
>    ![Raw data QC plots](../../images/scrna-scater-qc/raw_data.png "Raw data QC plots")
>
>    > <comment-title></comment-title>
>    >
>    > There are four plots, two distribution bar plots and two scatter plots.
>    > The first distribution plot is the number of reads in each library (from a single cell).
>    > The second plot is the distribution of *feature counts* per cell. Feature counts in this case refers to the number of genes expressed in each cell.
>    > The third plot **Scatterplot of reads vs genes** is a combination of the two barplots, in that it plots both the read count and the expressed gene count for each cell.
>    > The final scatterplot is the **% MT genes**, which plots the number of genes expressed verses the percentage of those genes that are mitochondrial genes.
>    {: .comment}
>
>    Let's look at each plot in turn and see what it tells us.
>    - **Read counts**. You can see that there are a few cells that have less than ~200,000 reads, with other cells having up to one million reads. Although 200,000 reads is still quite a lot and we wouldn't want to get rid of so much data, we might want to think about removing cells that only contain a smaller number of reads (say, 100,000).
>    - **Feature counts**. Similar to the read counts plot, we see a few cells that have a very low number of expressed genes (<600), then followed by a more even distribution.
>    - **Scatterplot of reads vs genes**. This takes the information provided in the two distribution plots above and creates a scatterplot from them. The really poor-quality cells are represented by the points near the intersection of the x and y axis, being data with low read count and low gene count. These are the cells we want to remove during filtering.
>    - **% MT genes**. You can see from the plot that there are a few cells outside the main "cloud" of datapoints. Some of these could be removed by filtering out cells with low feature counts, but others might need to be removed by mitochondrial content, such as the cell around 37.5%
>
{: .hands_on}

# Filtering

## Manual filtering

In the **Scater: filter SCE** Galaxy tool there are two filtering methods available. First, there's an "automatic" filtering method that uses PCA to identify outliers cells and remove them from the data. This is particularly useful for very large datasets (hundreds of samples). Second, there's a manual filtering method where users can put a range of filtering parameters, informed by the previous plotting tool.
Here, we'll use the manual filtering method.

> <hands-on-title>Filtering with scater</hands-on-title>
>
> 1. {% tool [Scater: filter SCE](toolshed.g2.bx.psu.edu/repos/iuc/scater_filter/scater_filter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: Calculate QC metrics** {% icon tool %})
>    - *"Type of filter"*: `manual`
>        - *"Number of reads mapped to a gene for it to be counted as expressed"*: `4.0`
>        - *"Minimum library size (mapped reads) to filter cells on"*: `100000`
>        - *"Minimum number of expressed genes to filter cells on"*: `500`
>        - *"Maximum % of mitochondrial genes expressed per cell"*: `35.0`
>
>    > <comment-title></comment-title>
>    >
>    > Let's have a look at the parameters and their values:
>    > 1. *"Number of reads mapped to a gene for it to be counted as expressed"*: by default, only one read needs to be mapped to a gene for it to be counted as "expressed". We can be a little bit more stringent here and increase the number of reads that need to be mapped to a gene for it to be categorised as "expressed".
>    > 2. *"Minimum library size (mapped reads) to filter cells on"*: This value asks how many mapped reads from each cell do you require to be mapped to your genome to be included in downstream analysis. We can see from our plots that we have a few cells that have less than 200,000 reads. 200,000 can still be quite a lot of reads (depending on the experiment), but we can use a smaller number to see what the initial effect of filtering is. Initially, use 100,000 as the value here.
>    > 3. *"Minimum number of expressed genes"*: You can see that some cells only express a few hundred genes, so we'll remove these cells also.
>    > 4. *"Maximum % of mitochondrial genes expressed per cell"*. You can see that as well as having one obvious outlier (~37%).
>    {: .comment}
>
>
{: .hands_on}


> <hands-on-title>Plot library QC after filtering</hands-on-title>
>
> 1. {% tool [Scater: plot library QC](toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_dist_scatter/scater_plot_dist_scatter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: filter SCE** {% icon tool %})
>
{: .hands_on}

![QC plots after manual filtering](../../images/scrna-scater-qc/post_filter.png "QC plots after manual filtering")

> <comment-title></comment-title>
>
> How did the filtering go? Do you think it's done a good job? Have you removed too many cells? Too few cells? About right?
>
> Often, it's a matter of trial and error, where you would start off by being quite lenient (low parameters) and then increasing the stringency until you're happy with the results. Using the initial Calculate QC metrics file, play around with the filtering parameters and visualise the output to see the effect different paramters have.
>
{: .comment}

## Automatic (PCA) filtering

Another filtering approach is to identify outliers in the data and remove them. PCA can be run once a SingleCellExperiment object has been normalised, and outliers cells identified based on the pre-computed quality control metrics within the SingleCellExperiment object.

As we are using a rather small test dataset, it's unlikely that PCA filtering will make any difference; for a larger, noisier dataset this is what we would perform instead:

> <hands-on-title>Task description</hands-on-title>
> 1. {% tool [Scater: filter SCE](toolshed.g2.bx.psu.edu/repos/iuc/scater_filter/scater_filter/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: Calculate QC metrics** {% icon tool %})
>    - *"Type of filter"*: `automatic`
>
>    > <comment-title></comment-title>
>    >
>    > The data will be normalised and then PCA ran on it using the following information from the data:
>    > * `pct_counts_top_100_features`
>    > * `total_features_by_counts`
>    > * `pct_counts_feature_control`
>    > * `total_features_feature_control`
>    > * `log10_total_counts_endogenous`
>    > * `log10_total_counts_feature_control`
>    >
>    > When using these filtering approaches, it is sometimes good to try running them in reverse - try PCA filtering first and then if that doesn't remove enough low-quality cells then use the manual filtering. We could actually pipeline them together - use PCA filtering first and then use the output of that to do further manual filtering.
>    {: .comment}
>
{: .hands_on}

As discussed previously, technical artefacts can bias scRNA-seq analyses. Strong batch effects can mask real biological differences in the data, so must be identified and removed from the data. Logging meta-data details such as date of library construction, sequencing batch, sample name, technical replicate, plate number, etc., is essential to identify batch effects in the data. We can use this information to visualise the data to examine it for clustering according to batch, rather than any real biological feature.

> <hands-on-title>PCA plot</hands-on-title>
> 1. {% tool [Scater: PCA plot](toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_pca/scater_plot_pca/1.12.2) %}  with the following parameters:
>    - {% icon param-file %} *"Input SingleCellLoomExperiment dataset"*: `output_loom` (output of **Scater: filter SCE** {% icon tool %})
>    - *"Feature (from annotation file) to colour PCA plot points by"*: `Mutation_Status`
>    - *"Feature (from annotation file) to shape PCA plot points by"*: `Cell_Cycle`
>
>
{: .hands_on}

![PCA plot](../../images/scrna-scater-qc/PCAplot.png "PCA plot")

> <question-title></question-title>
>
> 1. Do you see any obvious batch effect?
> 2. Do any of the categories suggesting some sort of technical artefact?
>
> > <solution-title></solution-title>
> >
> > 1. We can see that the **S** and **G2M** categories in **Cell\_Cycle** cluster away from the rest of the data. Spend some time thinking about whether this might be a batch effect or biologically significant.
> >
> {: .solution}
>
{: .question}


# Conclusion

We have gone through the process of filtering low-quality data from an scRNA-seq expression matrix, using the visualise-filter-visualise paradigm, which proves to be a very effective way of quality-controlling scRNA-seq data. Cells that have low read-coverage, low expression values, or high mitochondrial gene expression have been filtered out. We have then examined ways of looking at confounding factors to examine batch effects in our data.
The workflow available from the "Supporting Materials" of this tutorial can be directly imported and used or adapted to a specific analysis.

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %} and {% cite etherington2019galaxy %})

tools_json:
["Scater: Calculate QC metrics", "toolshed.g2.bx.psu.edu/repos/iuc/scater_create_qcmetric_ready_sce/scater_create_qcmetric_ready_sce/1.12.2", "Scater: plot library QC", "toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_dist_scatter/scater_plot_dist_scatter/1.12.2", "Scater: filter SCE", "toolshed.g2.bx.psu.edu/repos/iuc/scater_filter/scater_filter/1.12.2", "Scater: PCA plot", "toolshed.g2.bx.psu.edu/repos/iuc/scater_plot_pca/scater_plot_pca/1.12.2"]


topic: single-cell
tutorial: bulk-music
title: Bulk RNA Deconvolution with MuSiC
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: ["transcriptomics"]
content_hash: a7505b50470a9e3a94f20273a6a6cffa
last_modified: 2026-05-15T22:28:16.661203
zenodo_link: https://zenodo.org/record/5719228
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/bulk-music/tutorial.html

content:
Bulk RNA-seq data contains a mixture of transcript signatures from several types of cells. We wish to deconvolve this mixture to obtain estimates of the proportions of cell types within the bulk sample. To do this, we can use single cell RNA-seq data as a reference for estimating the cell type proportions within the bulk data.

<!-- using info from here: https://xuranw.github.io/MuSiC/articles/MuSiC.html -->

In this tutorial, we will use bulk and single-cell RNA-seq data, including matrices of similar tissues from different sources, to illustrate how to infer cell type abundances from bulk RNA-seq.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Bulk RNA-seq Cell Type Deconvolution

The heterogeneity that exists in the cellular composition of bulk RNA-seq can add bias to the results from differential expression analysis. In order to circumvent this limitation, RNA-seq deconvolution aims to infer cell type abundances by modelling the gene expressions levels as 'weighted sums' of cell type specific expression profiles.

> <details-title>More details on 'Sums'</details-title>
>
> **So...** You fancy some maths do you? Good! This is important, as you'll see variations of the phrase 'weighted sums' if you ever look at any papers in the field! Let's think about just the 'sums' here.
> If we think about the total expression of a given gene you might get from bulk RNA-seq, you could also think about it as the sum of the expression of each cell, for example,
>
> *Total = cell<sub>a</sub> expression + cell<sub>b</sub> expression ... (and so forth)*
>
> This is a *'sum'*, because we're adding up all the cells. If we now think about how we expect similar expression from similar cell types, we could change this to the following:
>
> *Total = cell_type<sub>a</sub> expression + cell_type<sub>b</sub> expression*  ... (and so forth)
>
> **BUT WAIT!** That's not strictly true, because the cell types are not all in equal proportion (if only!). So we have to take into account another variable, the proportion a given cell type takes up in a sample. So we now have:
>
> *Total = Proportion<sub>a</sub> x cell_type<sub>a</sub> expression + Proportion<sub>b</sub> x cell_type<sub>b</sub> expression*
>
> So now, we are using the sums of expression based off of the cell proportion. And if you're a mathematician, you might instead put this as
>
> **T = C x P**
> or some fancier formulas...(read more [in-depth here](https://academic.oup.com/bioinformatics/article/34/11/1969/4813737) if you like!)
>
> The point is, if we have an idea of what the average expression should be for each gene (what we can get from single cell RNA-seq data, *C*), and we have the total expression (from the bulk RNA-seq, *T*), then we can infer the cell proportions (*P*).
>
{: .details}

Many different computational methods have been developed to estimate these cell type proportions, but in this tutorial we will be using the [MuSiC](https://xuranw.github.io/MuSiC/articles/MuSiC.html) tool suite ({% cite wang2019bulk %}) to estimate the proportion of individual cell types in our bulk RNA-seq datasets.

### MusiC

MuSiC uses cell-type specific gene expression from single-cell RNA seq data to characterize cell type compositions (proportions) from bulk RNA-seq data in complex tissues. By appropriate weighting of genes showing cross-subject (sample to sample) and cross-cell (cells of the same cell type within a sample) consistency, MuSiC enables the transfer of cell type-specific gene expression information from one dataset to another.

> <question-title></question-title>
>
> 1. What is a weighted sum? *Hint*: What kinds of genes would be best for distinguishing cell types, and what kinds of genes would make it difficult?
>
> > <solution-title></solution-title>
> >
> > 1. So you know what the *Sum* is from above - the total expression of a given gene in a bulk RNA-seq sample depends on the proportion of cell types and the average expression level of each of those cell types (**T = C x P**). However, single cell RNA-seq data is highly variable. Cell-type specific expression in genes with lower variation from sample to sample (i.e. person to person or organism to organism) and cell to cell (i.e. within a sample) will be the most useful for distinguishing cells, while genes that vary heavily (i.e. high in cell type<sub>a</sub> in one sample, but low in cell type<sub>a</sub> in another sample) will be the least useful in accurately distinguishing cells. Therefore, to accurately use the mean expression level in a cell_type, MusiC weights the sums, favouring more consistently expressed genes in cell types.
> >
> {: .solution}
{: .question}

Solid tissues often contain closely related cell types that are difficult to distinguish from one another, a phenomenon known as *"collinearity"*. To deal with collinearity, MuSiC can also employ a tree-guided procedure that recursively zooms in on closely related cell types. Briefly, MuSic first groups similar cell types into the same cluster and estimates cluster proportions, then recursively repeats this procedure within each cluster.

![muse1](../../images/bulk-music/figure_method.jpg "Overview of MuSiC Suite")

# Cell Proportion Estimation

Here we will extract cell proportions from a bulk data of human pancreas data from {%cite fadista2014global %} concerning 56 638 genes across 89 samples, using a single cell human pancreas dataset from {%cite segerstolpe2016single %} containing 25 453 genes across 2209 cells, clustered into 14 cell types, from 6 healthy subjects and 4 with Type-II diabetes (T2D). If the deconvolution is good, and the datasets are compatible with sufficient enough overlap, we should be able to identify the same cell types from the bulk data.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Cell Type inference of Human Pancreas Data"*
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    * Human pancreas bulk RNA datasets (tag: `#bulk`)
>
>      ```
>      https://zenodo.org/record/5719228/files/GSE50244bulkeset.expression.tabular
>      https://zenodo.org/record/5719228/files/GSE50244bulkeset.phenotype.tabular
>      ```
>    * Human pancreas single-cell RNA datasets (tag: `#scrna`)
>      ```
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.expression.tabular
>      https://zenodo.org/record/5719228/files/EMTABesethealthy.phenotype.tabular
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
>
> 4. Check the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each `expression` file a tag corresponding to `#bulk` and `#scrna`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

### Exploring the Datasets

  We are told before we download the data, that:
  * The bulk human pancreas dataset is 89 samples across 56 638 genes
  * The single cell human pancreas datasets is 2209 cells across 24 453 genes.

  But what does this actually look like in the data?

> <comment-title>Inspecting Datasets</comment-title>
>
> Note that at any time you can visually inspect the input datasets yourself by either:
> * Expanding the dataset in the history panel clicking on the name of the dataset
> * Or clicking on {% icon galaxy-eye %} icon to load them into the main window.
{: .comment}

Let's start exploring the datasets.

> <hands-on-title>Exploring the Datasets</hands-on-title>
>
> 1. Inspect the `#scrna` expression file
>
>    > <question-title></question-title>
>    >
>    > 1. What do the rows and columns correspond to?
>    > 2. What do the values at each position mean?
>    > 3. Has the data been normalised?
>    > 4. Why are there so many zeroes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![expr_tab](../../images/bulk-music/peek_tabular_scrna_expr.png "Peeking at the tabular scRNA expression dataset")
>    > >
>    > > 1. Rows correspond to gene names, and columns to cell identifiers
>    > > 2. These are the number of reads or counts for each gene for each cell
>    > > 3. The data has not been normalised since the counts are integer and not decimal
>    > > 4. Single-cell datasets can be very sparse for a variety of reasons relating to dropouts and biological factors. For more information, please see the [introduction single-cell RNA-seq slides#41](../../tutorials/scrna-intro/slides.html).
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Inspect the `#bulk` expression file
>
>    > <question-title></question-title>
>    >
>    > 1. What do the rows and columns correspond to?
>    > 2. In which field is there likely to be overlap with the `#scrna` dataset?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![bulk_tab](../../images/bulk-music/peek_tabular_bulk_expr.png "Peeking at the tabular bulk RNA-seq expression dataset")
>    > >
>    > > 1. As before with the `#scrna` dataset, rows correspond to gene names and columns to sample identifiers.
>    > > 2. The sample identifiers and the cell identifiers are completely different, but the gene names appear to be using the same symbols as the `#scrna` dataset, so the gene field is the common factor here.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Inspect the `#scrna` phenotype file
>
>    > <question-title></question-title>
>    >
>    > 1. Does the phenotypes file describe genes or cells?
>    > 2. What does the `SubjectName` field describe?
>    > 3. What does the `cellType` field describe?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![peek_tabular_scrna_pheno.png](../../images/bulk-music/peek_tabular_scrna_pheno.png "Peeking at the tabular scRNA phenotype dataset")
>    > >
>    > > 1. The first column is the index column, which uses the cell identifiers in the header of the `#scrna` expression file, so the phenotypes file describes the cells.
>    > > 2. `SubjectName` tells us whether the cell (on that row) is labelled as Type-II diabetes or not, "Non T2D" or "T2D" respectively.
>    > > 3. `cellType` tells us which cell type the cell was assigned to. This could be the result of prior-clustering and then labelling, or the cells could be labelled before analysis.
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. Inspect the `#bulk` phenotype file
>
>    > <question-title></question-title>
>    >
>    > 1. Does the phenotypes file describe genes or samples?
>    > 2. Is the `SubjectName` field related to the `SubjectName` field in the `#scrna` phenotypes file?
>    > 3. What other factors are in the phenotypes file?
>    > 4. Is the `tissue` field related to the `cellType` field in the `#scrna` phenotypes file?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ![peek_tabular_bulk_pheno.png](../../images/bulk-music/peek_tabular_bulk_pheno.png "Peeking at the tabular bulk RNA-seq phenotype dataset")
>    > >
>    > > 1. The first column is the index column, which uses the sample identifiers in the header of the `#bulk` expression file, so the phenotypes file describes the samples.
>    > > 2. The `SubjectName` column uses a completely different set of identifiers from the `#scrna` phenotypes file so they should be assumed to be unrelated.
>    > > 3. We see `age`, `bmi`, `hba1c`, `gender`, `tissue`. HbA1c appears to be a gene of interest related to a known phenotype.
>    > > 4. **This is the question we wish to answer in the deconvolution**. Visually, there appears to be no overlap, but the "pancreatic islets" tissue likely consists of several cell types that show expressions profiles with some affinity  to the single cell types described in the `#scrna` phenotypes file.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

   The bulk RNA-seq phenotype file lists the main factors of interest, and HbA1c appears to be a specific gene associated with a phenotype. It is well known that the beta cell proportions are related to T2D disease status. In the progress of T2D, the number of beta cells decreases. One of the most important tests for T2D is the HbA1c (hemoglobin A1c) test. When the HbA1c level is greater than 6.5%, the patient is diagnosed as T2D. We will look later at the beta cell proportions relationship with HbA1c level in this deconvolution analysis.

## Building the Expression Set objects

{% snippet faqs/galaxy/tutorial_mode.md %}

For now we need to construct our Expression set objects that will be consumed by MuSiC.

> <details-title>Expression Set</details-title>
>
> ![expression_set](../../images/bulk-music/expressionset.png "Image from Stefano Monti")
>
>
> *Expression Set* objects are a container for high-throughput assays and experimental metadata. The *ExpressionSet* class is derived from *eSet*, and requires a matrix named *exprs*.
>
> The *ExpressionSet* class is designed to combine several different sources of information into a single convenient structure. An *ExpressionSet* can be manipulated (e.g., subsetted, copied) conveniently, and is the input or output from many Bioconductor functions.
>
> The data in an *ExpressionSet* is complicated, consisting of:
> - expression data (*exprs*, or assayData; assayData is used to hint at the methods used to access different data components, as we will see below);
> - metadata describing samples in the experiment (phenoData, *pData*)
> - information related to the protocol used for processing each sample and usually extracted from manufacturer files (protocolData);
> - annotations and metadata about the features on the chip or technology used for the experiment (featureData, annotation, *fData*);
> - and a flexible structure to describe the experiment (experimentData).
>
> The ExpressionSet class coordinates all of this data, so that you do not usually have to worry about the details.
>
> For more information please [read the specification](http://www.bioconductor.org/packages/release/bioc/vignettes/Biobase/inst/doc/ExpressionSetIntroduction.pdf) as well as the [image source](https://montilab.github.io/BS831/articles/docs/ExpressionSet.html).
>
{: .details}

Here we shall build two ExpressionSet objects corresponding to the bulk and single-cell datatypes.

### Construct Expression Set Object

> <hands-on-title>Build the Expression Set inputs</hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `GSE50244bulkeset.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `GSE50244bulkeset.phenotype.tabular` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > An ExpressionSet object has many data slots, the principle of which are the experiment data (*exprs*), the phenotype data (*pData*), as well metadata pertaining to experiment information and additional annotations (*fData*).
>    {: .comment}
>
> 2. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `EMTABesethealthy.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `EMTABesethealthy.phenotype.tabular` (Input dataset)
>
{: .hands_on}

#### Inspect Expression Set Object

We will now inspect these objects we just created to see what information we can extract out of them, and how these multiple datasets are summarized within the object.

> <hands-on-title>Inspect and Describe the scRNA ExpressionSet Object</hands-on-title>
>
> 1. Obtain General Info about the data set
>    - {% icon galaxy-eye %} Click on the `#scrna` *General Info* dataset in the history view (output of **Construct Expression Set Object** {% icon tool %})
> 1. Obtain Feature Information about the data set
>    - {% tool [Inspect Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_inspect_eset/music_inspect_eset/0.1.1+galaxy4) %} with the following parameters:
>         > <warning-title>Danger: This tool has needs!</warning-title>
>         > You may need to click the dataset from your history and drag it into the input of this tool. Some browsers don't allow this.
>         {: .warning}
>       - {% icon param-file %} *"ESet Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>       - *"Inspect"*: `Feature Data Table`
>         > <comment-title>Features or Genes?</comment-title>
>         >
>         > "Features" are synonymous with "Genes" in a genomic setting, but data scientists tend to prefer to use the former term, as it can be used in other non-genomic settings.
>         >
>         {: .comment}
> By inspecting the `Feature Data Table`, you should see a list of gene names.
>
> 1. Obtain the dimensions of the data set
>    - {% tool [Inspect Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_inspect_eset/music_inspect_eset/0.1.1+galaxy4) %} with the following parameters:
>      - {% icon param-file %} *"ESet Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>      - *"Inspect"*: `Dimension`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many samples are in the dataset, and how many genes?
> 2. Does this agree with the original input tabular expression data set?
>
> > <solution-title></solution-title>
> >
> > 1. 1097 samples and 25 453 genes
> > 2. Yes!
> >
> {: .solution}
>
{: .question}


## Estimating Cell Type proportions

<!-- Maybe this goes in a comment? -->
Instead of selecting marker genes, MuSiC gives weights to each gene. The weighting scheme is based on cross-subject variation, by up-weighing genes with low variation and down-weighing genes with high variation in a cell type. Here we demonstrate this step-by-step with the human pancreas datasets.

The deconvolution of 89 subjects from {%cite fadista2014global %} is performed with the bulk data GSE50244 expression set and single cell reference EMTAB. The estimation was constrained on 6 major cell types: alpha, beta, delta, gamma, acinar and ductal, which make up over 90% of the whole islet.

### Cell Type estimation with **MuSiC**

The deconvolution process can be performed by simply inputting the bulk RNA-seq & scRNA-seq datasets and then hitting execute. However, in this section we will be tracing the effect of a disease phenotype across both datasets.

Here we will use one of the factors from the bulk RNA-seq phenotypes related to the T2D disease status. Any bulk RNA-seq sample coming from a patient with the `HbA1c` factor above 6.5% would be clinically classified as having T2D, so we want to compare these datasets with non-diseased samples. As diabetes affects Beta cells, we're particularly interested in these cells proportions - mostly to prove that the deconvolution has worked well!

It's also important that input scRNA-seq datasets (ideally) have some representation of cell populations from diseased patients.

> <question-title></question-title>
>
> 1. Why should the scRNA-seq dataset contain diseased cells and healthy cells?
>
> > <solution-title></solution-title>
> >
> > 1. For instance, there might be a weird extra cell type that appears in the disease phenotype. Or perhaps there is a missing cell type in the disease phenotype. Without both healthy & diseased datasets, the cells in the bulk datasets might not be fully identifiable. Saying that, such datasets are not always available - something to keep in mind when interpreting the results!
> >
> {: .solution}
>
{: .question}


> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [MuSiC](toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4) %} with the following parameters:
>    - *MuSiC sometimes does not show up from the tool search box. You may need to look for it under the **Single Cell** heading
>    - {% icon param-file %} *"scRNA Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>    - {% icon param-file %} *"Bulk RNA Dataset"*: `#bulk` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Purpose"*: `Estimate Proportions`
>        - *"Methods to use"*: `MuSiC` and `NNLS`
>        - *"Cell Types Label from scRNA dataset"*: `cellType`
>        - *"Samples Identifier from scRNA dataset"*: `sampleID`
>        - *"Comma list of cell types to use from scRNA dataset"*: `alpha,beta,delta,gamma,acinar,ductal`
>        - *"Phenotype factors"*: `(leave blank)`
>        - *"Excluded phenotype factors"*: `sampleID,SubjectName`
>        - Under *"Show proportions of a disease factor?"*: `Yes`
>          - *"scRNA Phenotype Cell Target"*: `beta`
>          - *"Bulk Phenotype Target"*: `hba1c` (**Factor from Bulk Phenotype**)
>          - *"Bulk Phenotype Target Threshold"*: `6.5`
>          - *"scRNA Sample Disease Group"*: `T2D` (**Ideally a Factor from scRNA Phenotype**)
>          - *"scRNA Sample Disease Group (Scale)"*: `5`
>
>    > <comment-title></comment-title>
>    >
>    > It's important to set a phenotype target threshold, otherwise no cells will be matched for the phenotype target. In this case, when the HbA1c level is greater than 6.5%, the patient is diagnosed as T2D.
>    {: .comment}
>
{: .hands_on}

The estimated proportions are normalized such that the proportions of cell types within each sample sum to 1. MuSic compares itself against a previous method of deconvolution known as Non-negative Least-Squares (NNLS), which MuSic supercededs via its Weighted Non-negative Least-Squares (W-NNLS) methodology. You can remove this if you wish from within the tool parameters when running.

![jitter_plot](../../images/bulk-music/jitter_plot.png "Jitter plot of Estimated Proportions")

In the above image you can see (a) the estimated proportion of cells for each of the 6 declared types, as calculated by MuSiC and the NNLS methods, respectively. In the (b) section, this information is better represented as a box plot to show you the distribution of cell type proportions.

![ctprop](../../images/bulk-music/ctprop_plot.png "(top) Cell type proportions by disease factor, and (bottom) HbA1c factor expression against beta cell type proportion")

As stated previously, it is well known that the beta cell proportions are related to T2D disease status. As T2D progresses, the number of beta cells decreases. In the above image we can see in (a) that we have the same information as previous, but we also distinguish between cells that from patients with T2D status over the Normal cell phenotypes. Section (b) further explores this with a linear regression showing the cell type proportion of cells with HbA1c expression, where we see that there is a significant negative correlation between HbA1c level and beta cell proportions.


> <comment-title></comment-title>
>
>  We can extract the coefficients of this fitting by looking at the `Log of Music Fitting Data` in the `Summaries and Logs` output collection:
>
>  ```
>  Coefficients:
>               Estimate Std. Error t value Pr(>|t|)
>  (Intercept)  0.797148   0.194757   4.093  0.00011 ***
>  age          0.002639   0.001772   1.489  0.14087
>  bmi         -0.013620   0.007276  -1.872  0.06529 .
>  hba1c       -0.061396   0.025403  -2.417  0.01819 *
>  genderMale   0.079874   0.039274   2.034  0.04566 *
>
>  ```
>
>  In addition to HbA1c levels, gender has also correlated with beta cell proportions. This is unsurprising, as more men have diabetes than women and sex is known to impact HbA1c levels.
>
{: .comment}

### Proportions of Cell Type to each Bulk RNA sample

One question we might wish to ask is: what affinity did each of the 6 single cell types have to each of the 89 subjects in the bulk data? For this we can look at the raw data {% icon galaxy-eye %} `MuSiC Estimated Proportions of Cell Types` in the `Proportion Matrices`, to get a glimpse of cell type compositions on a bulk RNA sample level.

Both the MuSiC and the NNLS calculations of this data is best represented in the below heatmap, with RNA samples as rows and cell types as columns:

![subject_heatmap](../../images/bulk-music/subject_heatmap.png "Heatmap of cell type proportions at the RNA sample level.")

> <question-title></question-title>
>
> 1. Which cell types are under-represented in the NNLS method?
> 2. Which cell types do not appear to be present in both?
>
> > <solution-title></solution-title>
> >
> > 1. Here it is evident that the previous NNLS method over-represents the Alpha cell type compared to the MuSiC method which gives more weight to the Beta and Ductal cell types, which were under-represented in the NNLS method.
> > 2. The Delta and Gamma cell types appear empty in both.
> >
> {: .solution}
>
{: .question}


# Estimation of cell type proportions with pre-grouping of cell types

  In the previous section we estimated cell types under the assumption that that the gene expression between cell types was largely independent. However, solid tissues often contain closely related cell types. This correlation of gene expression between these cell types is termed 'collinearity', which makes it difficult to resolve their relative proportions in bulk data.

  To deal with collinearity, MuSiC can also employ a tree-guided procedure that recursively zooms in on closely related cell types.

  Briefly, similar cell types are grouped into the same cluster and their cluster proportions are estimated, then this procedure is recursively repeated within each cluster. At each recursion stage, only genes that have low within-cluster variance are used, as they should be consistent within a cell type. This is critical as the mean expression estimates of genes with high variance are affected by the pervasive bias in cell capture of scRNA-seq experiments, and thus cannot serve as reliable reference.

  To perform this analysis, we will use mouse kidney single-cell RNA-seq data from {% cite park2018single %} described by 16 273 genes over a trimmed subset of 10 000 cells, giving 16 unique cell type (2 of which are novel) across 7 subjects. The bulk RNA-seq dataset is from {% cite beckerman2017transgenic %} and contains mouse kidney tissue described by 19 033 genes over 10 samples.

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial *"Deconvolution: Dendrogram of Mouse Data"*
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    * Mouse kidney bulk RNA datasets (tag: `#bulk`)
>
>      ```
>      https://zenodo.org/record/5719228/files/Mousebulkeset.expression.tabular
>      https://zenodo.org/record/5719228/files/Mousebulkeset.phenotype.tabular
>      ```
>    * Mouse kidney single-cell RNA datasets (tag: `#scrna`)
>      ```
>      https://zenodo.org/record/5719228/files/Mousesubeset.expression.tabular
>      https://zenodo.org/record/5719228/files/Mousesubeset.phenotype.tabular
>      ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
>
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 5. Add to each `expression` file a tag corresponding to `#bulk` and `#scrna`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

> <details-title>Exploring the Datasets</details-title>
>
> As before, you may choose to explore the bulk and scrna datasets and try to determine their factors from the phenotypes as well as any overlapping fields that will be used to guide the deconvolution.
>
{: .details}

You will need to again create ExpressionSet objects, as before.

### Construct Expression Set Object

> <hands-on-title>Build the Expression Set inputs</hands-on-title>
>
> 1. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `Mousebulkeset.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `Mousebulkeset.phenotype.tabular` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > An ExpressionSet object has many data slots, the principle of which are the experiment data (*exprs*), the phenotype data (*pData*), as well metadata pertaining to experiment information and additional annotations (*fData*).
>    {: .comment}
>
> 2. {% tool [Construct Expression Set Object](toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"Assay Data"*: `Mousesubeset.expression.tabular` (Input dataset)
>    - {% icon param-file %} *"Phenotype Data"*: `Mousesubeset.phenotype.tabular` (Input dataset)
>
{: .hands_on}

### Colinearity Dendrogram with **MuSiC** to determine cell type similarities

 Determining cell type similarities requires first producing a design matrix as well as a cross-subject mean of relative abundance, using a tree-based clustering method of the cell types we wish to cluster.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [MuSiC](toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4) %} with the following parameters:
>    - {% icon param-file %} *"scRNA Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>    - {% icon param-file %} *"Bulk RNA Dataset"*: `#bulk` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Purpose"*: `Compute Dendrogram`
>        - *"Cell Types Label from scRNA Dataset"*: `cellType`
>        - *"Cluster Types Label from scRNA dataset"*: `clusterType`
>        - *"Samples Identifier from scRNA dataset"*: `sampleID`
>        - *"Comma list of cell types to use from scRNA dataset"*:
>           `Endo,Podo,PT,LOH,DCT,CD-PC,CD-IC,Fib,Macro,Neutro,B lymph,T lymph,NK`
>
{: .hands_on}

![dendrogram](../../images/bulk-music/dendrogram.png "Dendrogram of Design Matrix and Cross-Subject Mean of Relative Abundance")

> <question-title></question-title>
>
> 1. What do you notice about the cells clustering?
> 2. How many clusters can you see with a height threshold above 650 in the "Cluster log(Design Matrix)"?
>
> > <solution-title></solution-title>
> >
> > 1. The immune cells are clustered together and the kidney specific cells are clustered together. Notice that DCT and PT are within the same high-level grouping.
> > 2. The cut-off of 650. Here we cut 13 cell types into 4 groups:
> >
> >    ```
> >    C1: Neutro
> >    C2: Podo
> >    C3: Endo, CD-PC, CD-IC, LOH, DCT, PT
> >    C4: Fib, Macro, NK, B lymph, T lymph
> >    ```
> >
> {: .solution}
>
{: .question}


### Heatmap of Cell Type Similarities using **MuSiC**

We shall use the 4 cell type groups determined by the cut off threshold in the above question box. To guide the clustering, we shall upload known epithelial and immune cell markers to improve the more diverse collection of cell types in the C3 and C4 groups.

> <hands-on-title>Upload marker genes and generate heatmap</hands-on-title>
> 1. Import the files from [Zenodo]({{ page.zenodo_link }}) or from
>    the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`):
>
>    ```
>    https://zenodo.org/record/5719228/files/epith.markers
>    https://zenodo.org/record/5719228/files/immune.markers
>    ```
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. {% tool [MuSiC](toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4) %} with the following parameters:
>    - **Note**
>      > <warning-title>Shortcut!</warning-title>
>      >
>      > Here we need to re-use all the inputs from the previous **MuSiC** {% icon tool %} step, plus add a few extra. To speed this up, you can simply click on the re-run icon {% icon galaxy-refresh %} under any of its outputs.
>      {: .warning}
>
>    - {% icon param-file %} *"scRNA Dataset"*: `#scrna` (output of **Construct Expression Set Object** {% icon tool %})
>    - {% icon param-file %} *"Bulk RNA Dataset"*: `#bulk` (output of **Construct Expression Set Object** {% icon tool %})
>    - *"Purpose"*: `Compute Dendrogram`
>        - *"Cell Types Label from scRNA Dataset"*: `cellType`
>        - *"Cluster Types Label from scRNA dataset"*: `clusterType`
>        - *"Samples Identifier from scRNA dataset"*: `sampleID`
>        - *"Comma list of cell types to use from scRNA dataset"*:
>           `Endo,Podo,PT,LOH,DCT,CD-PC,CD-IC,Fib,Macro,Neutro,B lymph,T lymph,NK`
>        - In *"Cluster Groups"*:
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C1`
>                - *"Comma list of cell types to use from scRNA dataset"*: `Neutro`
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C2`
>                - *"Comma list of cell types to use from scRNA dataset"*: `Podo`
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C3`
>                - *"Comma list of cell types to use from scRNA dataset"*:
>                   `Endo,CD-PC,LOH,CD-IC,DCT,PT`
>                - *"Marker Gene Group Name"*:`Epithelial`
>                - {% icon param-file %} *"List of Gene Markers"*: `epith.markers` (Input dataset)
>            - {% icon param-repeat %} *"Insert Cluster Groups"*
>                - *"Cluster ID"*: `C4`
>                - *"Comma list of cell types to use from scRNA dataset"*:
>                   `Macro,Fib,B lymph,NK,T lymph`
>                - *"Marker Gene Group Name"*:`Immune`
>                - {% icon param-file %} *"List of Gene Markers"*: `immune.markers` (Input dataset)
>
>    > <comment-title></comment-title>
>    >
>    > The C1 (Neutrophil) and C2 (Podocyte) clusters do not use marker genes for the dendrogram clustering in this dataset.
>    {: .comment}
>
{: .hands_on}


![dendro_jitboxheat](../../images/bulk-music/dendro_jitboxheat.png "Jitter and Boxplots of all cell types followed by a heatmap of each RNA sample against cell type. Note that the y-axis for each of the plots above are not constant across cell types.")

> <question-title></question-title>
>
> Most of the expression in the above plot appears to be derived from one cell type.
>
> 1. Which cell type dominates the plot?
> 2. What does this tell you about the bulk RNA?
>
> > <solution-title></solution-title>
> >
> > 1. The PT cells appear to dominate.
> > 2. Most of the expression in the bulk RNA dataset is derived solely from the PT cells, and could be a monogenic cell line.
> >
> {: .solution}
>
{: .question}


# Conclusion


In this tutorial we constructed ExpressionSet objects, inspected and annotated them, and then finally processed them with the MuSiC RNA-Deconvolution analysis suite.

Below is an overview of the workflow that was used throughout this tutorial.

![workflow1](../../images/bulk-music/workflow1.png "Workflow for estimating cell types")

Note how two ExpressionSet objects are constructed: one from bulk RNA-seq tabular assay data, and the other from single-cell RNA-seq tabular assay data. A blind analysis of cell proportion estimation is performed first. Then in the second half of the tutorial, we performed a guided analysis using pre-grouped cell types.


![workflow_dendro.png](../../images/bulk-music/workflow_dendro.png "Workflow for generating a dendrogram and clustering cells upon it")

This tutorial is part of the [https://singlecell.usegalaxy.eu](https://singlecell.usegalaxy.eu) portal ({% cite tekman2020single %}).

{% snippet topics/single-cell/faqs/user_community_join.md %}

tools_json:
["Construct Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_construct_eset/music_construct_eset/0.1.1+galaxy4", "Inspect Expression Set Object", "toolshed.g2.bx.psu.edu/repos/bgruening/music_inspect_eset/music_inspect_eset/0.1.1+galaxy4", "MuSiC", "toolshed.g2.bx.psu.edu/repos/bgruening/music_deconvolution/music_deconvolution/0.1.1+galaxy4"]


topic: single-cell
tutorial: scrna-preprocessing
title: Pre-processing of Single-Cell RNA Data
difficulty: intermediate
time_estimation: 3h
requirements_json: []
tags_json: ["[]"]
content_hash: 9b03d7e0a4875e9def6dff1f78969c4b
last_modified: 2026-05-27T09:01:49.748559
zenodo_link: https://zenodo.org/record/3253142
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-preprocessing/tutorial.html

content:
## Why do Single Cell sequencing?


Single-cell RNA (scRNA) sequencing is the technological successor to classical "bulk" RNA-seq, where samples are no longer defined at the tissue level but at the individual cell level. The bulk RNA-seq methods seen in [previous hands-on material]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) would give the average expression of genes in a sample, whilst overlooking the distinct expression profiles given by the cell sub-populations due to their heterogeneity.

The rise of scRNA sequencing provides the means to explore this heterogeneity by examining samples at the individual cell level, enabling a greater understanding of the development and function of such samples, by the characteristics of their constituent cells.

Consider the heterogeneity of cells sampled from bone marrow, where hematopoietic stem cells can give rise to many different cell types within the same tissue:

![Cell Differentiation of Hematopoietic Stem cells]({% link topics/single-cell/images/scrna-pre-processing/scrna_hematopoiesis.png %} "Cell Differentiation of Hematopoietic Stem cells. Image Source: https://commons.wikimedia.org/wiki/File:Hematopoiesis_simple.svg")

The genes expressed by these cells at different developmental time points can be subtle, but generally can be classified into discrete cell sub-populations or under statistical clustering methods such as PCA or tSNE. Cells in the same cluster exhibit similar profiles of differential expression in the same set of related genes, compared to cells in other clusters. By identifying significant genes in each cluster, cell types and cell developmental hierarchies can be inferred based on the proximity of these clusters to one another.

Other than cell development, there are many more factors that can shape the level of gene expression exhibited by a given cell. Intercellular cell-signalling can block or enhance specific transcripts, the total amount of transcripts of a cell increases with the cell-cycle, or the proximity of a cell within a tissue to nutrients or oxygen.

![Facets of Cellular Identity]({% link topics/single-cell/images/scrna-pre-processing/scrna_nbt3711.png %} "Revealing the vectors of cellular identity with single-cell genomics, Nature Biotechnology, 2016")

This tutorial is in part inspired by aspects of the [Hemberg workflow](https://hemberg-lab.github.io/scRNA.seq.course/) at the Sanger institute, as well as the [CGATOxford workflow](https://github.com/CGATOxford/UMI-tools) which provides the **UMI-tools** suite that we make use of. The barcoding follows the CEL-Seq2 protocol {% cite Hashimshony2016 %},  mentioned in the [*Understanding Barcodes*]({% link topics/single-cell/tutorials/scrna-umis/tutorial.md %}) hands-on, and uses the same lane configuration as utilised by the [Freiburg MPI Grün lab](https://www.ie-freiburg.mpg.de/gruen).


# Analysis Strategy


Most scRNA sequencing techniques use pooled-sequencing approaches to generate a higher throughput of data by performing amplification and sequencing upon multiple cells in the same "pool". From a bioinformatics standpoint, this means that the output FASTQ data from the sequencer is batch-specific and contains all the sequences from multiple cells, where one sample of cells is equal to one batch.

In this tutorial, we will perform pre-processing upon scRNA FASTQ batch data to generate an *N*-by-*M*  count matrix of *N* cells and *M* genes, with each element indicating the level of expression of that gene in a particular cell.

This count matrix is crucial for performing the downstream analysis, where differential gene analysis is performed between cells in order to cluster them into groups denoting their cell type and lineage.

The tutorial is structured in two parts:

1. **Single-Batch Processing**
1. **Multi-Batch Processing**

![Overview of workflow]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow.svg %} "An overview of the entire workflow")

The first part of this tutorial will deal with batches, and use example *FASTQ* data from a single batch, which we will perform [barcode extraction](#barcode-extraction) and annotation upon. Alignment and quality control will also be performed, and we will see how to construct a rudimentary count matrix.

> <comment-title>10x Datasets</comment-title>
>
> 10x Genomics datasets can be processed in a much easier way that is outlined in the [*Pre-processing of 10X Single-Cell RNA Datasets*]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}) tutorial.
>
> However, much of the essential concepts for scRNA-seq pre-processing are not explained, so it is a good idea to familiarise yourself with them in this tutorial.
>
{: .comment}

The second part of this tutorial will deal with merging several output count matrices from multiple single batches generated in the first portion. Here, a set of example count matrices are [merged together](#merging-count-matrices) and quality control performed. This produces a final count matrix valid for downstream analysis.


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Single-Batch Processing

## Data upload and organization

In this tutorial we will be analysing scRNA-seq data of bone marrow cells taken from a single C57 mouse by *Herman et al.* ({% cite Herman2018 %}) and producing a count matrix that we can use for later analysis.

The size of scRNA files (.fastq) are typically in the gigabyte range and are somewhat impractical for training purposes, so we will expedite the analysis by using a smaller subset of actual batch data. We will also be using *Mus Musculus* annotation data (.gtf) from the  [NCBI RefSeq](ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/vertebrate_mammalian/Mus_musculus/) track, as well as a barcodes file (.tsv).


![Workflow Upper]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow_upper.png %} "Single batch processing")


> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and rename it (e.g. scRNA-seq single batch tutorial)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 1. Import the subset FASTQ paired data from [`Zenodo`](https://zenodo.org/record/3253142) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3253142/files/SRR5683689_1.subset.fastq
>    https://zenodo.org/record/3253142/files/SRR5683689_2.subset.fastq
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true collection_type="Paired" collection_name_convention="`<name>_<plate>_<batch>` to preserve the sample names, sequencing plate number and batch number." collection_name="Here we will write `C57_P1_B1`" genome="GRCm38/mm10" pairswaptext="`SRR5683689_1` and `SRR5683689_2`" %}
>
> 3. Import the Gene Annotations and Barcodes from [`Zenodo`](https://zenodo.org/record/3253142) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3253142/files/Mus_musculus.GRCm38.93.mm10.UCSC.ncbiRefSeq.gtf
>    https://zenodo.org/record/3253142/files/celseq_barcodes.192.tabular
>    ```
>
> 4. Set the datatype of the `celseq_barcodes.192.tabular` to `tabular`
>
{: .hands_on}


## Barcode Extraction

> <comment-title>Note</comment-title>
>
> Before performing the barcode extraction process, it is recommended that you familiarise yourself with the concepts of designing cell barcodes as given by the [*Plates, Batches, and Barcodes*]({% link topics/single-cell/tutorials/scrna-plates-batches-barcodes/slides.html %}), as well as the [*Understanding Barcodes*]({% link topics/single-cell/tutorials/scrna-umis/tutorial.md %}) hands-on material for an introduction into transcript barcodes.
>
{: .comment}

We will be demultiplexing our FASTQ batch data by performing barcode extraction whilst also making use of the provided barcodes file to filter for *specific* cell barcodes.

> <hands-on-title>Barcode Extraction</hands-on-title>
>
> 1. {% tool [UMI-tools extract](toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_extract/umi_tools_extract/0.5.5.1) %} with the following parameters:
>    - *"Library type"*: `Paired-end Dataset Collection`
>        - {% icon param-collection %} *"Reads in FASTQ format"*: `C57_P1_B1` (Our paired set)
>        - *"Barcode on both reads?"*: `Barcode on first read only`
>    - *"Use Known Barcodes?"*: `Yes`
>        - {% icon param-file %} *"Barcode File"*: `celseq_barcodes.192.tabular` (Input dataset)
>    - *"Barcode pattern for first read"*: `NNNNNNCCCCCC`
>    - *"Enable quality filter?"*: `No`
>
{: .hands_on}

Verifying that the desired UMI and cell barcodes have been extracted from the sequence of the Forward reads and inserted into the header of the Reverse reads is encouraged, using the method outlined in the above hands-on material.


> <question-title></question-title>
>
> How many reads were filtered out, and why?
>
> > <solution-title></solution-title>
> >
> > The input FASTQ files contained reads from all barcodes, including those with sequencing errors, resulting in a larger pool of detected barcodes than those desired. (e.g. Cell barcode `AAATTT` could have single base-pair sequencing errors that could modify it into `ATATTT` or `AAACTT`, etc).
> >
> > This information is included in the Log file of **UMI-tools extract** which contains all the parameters used to run, as well as *INFO* lines that indicate how many reads were read, and how many output. In this case: 134431 reads were retained (>90% of input reads).
> {: .solution}
{: .question}


## Mapping / Alignment

FASTQ files contain sequence information that we wish to map to genes in a genome. Mapping is a relatively straightforward process, and is covered more extensively in the [*Sequence Analysis*]({% link topics/sequence-analysis/index.md %}) tutorials:

1. Select your genome
1. Select your gene annotation file
1. Run the alignment program
1. (Optional) Run MultiQC to assess the quality of the mapping

The FASTQ data was sequenced from mouse data, so to perform the alignment we will need to gather all data relevant to that genome. We will use the latest version (mm10).

The annotation GTF file must match the genome version used, since both use physical coordinates. Each GTF contains all the gene, exon, intron, and other regions of interest that we will use to annotate our reads, should our reads align to any of the regions specified in this file.

For alignment, we will use RNA-STAR for performance and splice-awareness.


> <hands-on-title>Performing the Alignment</hands-on-title>
>
> 1. {% tool [RNA-STAR](toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.7a) %} with the following parameters:
>    - *"Single-end or paired-end reads"*: `Single-end`
>        - {% icon param-file %} *"RNA-Seq FASTQ/FASTA file"*: `Reads2` (output of **UMI-tools extract** {% icon tool %})
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>        - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>            - {% icon param-file %} *"Select reference genome"*: `Mus Musculus (mm10)` (Mouse)
>            - {% icon param-file %} *"Gene model (gff3,gtf) file for splice junctions"*: `Mus_musculus.GRCm38.93.mm10.UCSC.ncbiRefSeq`
> 1. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.9) %} with the following parameters:
>    - *"Results"*:
>      - *"1: Results"*:
>        - *"Which tool was used to generate logs?"*:`STAR`
>        - *"STAR output"*:
>           - *"1: STAR output"*
>              - *"Type of STAR output?"*:`Log`
>        - *"STAR log output"* :(Select the STAR output file that ends in "log")
> 1. Click on the {% icon galaxy-eye %} symbol on the MultiQC Webpage
>
{: .hands_on}

The purpose of MultiQC is to observe how well our reads were mapped against the reference genome. Many reads are discarded due to being of too low quality, or having ambiguous sequence content that can map them to multiple locations.

> <question-title></question-title>
>
> 1. What percentage of our reads are uniquely mapped? How many millions of reads is this percentage?
> 2. What percentage of our reads are mapped to more than one locus?
> 3. Is our overall mapping 'good' ?
>
> > <solution-title></solution-title>
> >
> > 1. `59.5%` or ~80k reads were successfully mapped
> > 2. `13.6%` are multiply mapped, and `3.7%` were mapped to too many loci
> >   - Multiply mapped means that a read was aligned to more than one gene
> >   - Mapped to too many loci means that a read was aligned to 10 or more loci, and should be ignored.
> > 3. It depends on how good the sequencing protocol is, and how many reads in total were mapped.
> >   - `90%` is amazing, reserved for bulk RNA-seq which typically has high coverage
> >   - `70%` is weak for bulk RNA-seq, but good for single-cell RNA-seq
> >   - This a small subset of a real dataset, but one would expect that 6 million mapped reads would be enough to generate a downstream analysis.
> >
> {: .solution}
{: .question}


## Filtering

Before continuing let us first look back on some of the previous stages:

> <comment-title>Recap of previous stages</comment-title>
>
> 1. *Barcode Extraction*:
>
>     Here we used `umi_tools extract` on our input forward and reverse FASTQ files, and extracted the UMI and cell barcode from the forward read *sequence*, and placed it into the *header* of both forward and reverse FASTQ files. i.e. FASTQ files → Modified FASTQ files
>
> 2. *Mapping*:
>
>     We took the sequencing data from the reverse FASTQ file (with modified headers) and aligned it to the mouse genome, using annotations presented in the GTF file for that genome. i.e. Modified FASTQ file (reverse) → BAM file
>
{: .comment}

### Confirming Reads in the BAM file

We now have a BAM file of our aligned reads, with cell and UMI barcodes embedded in the read headers. We also have the chromosome and base-pair positions of where these reads are aligned. The can be confirmed by peeking into the BAM file:

> <hands-on-title>Confirming the Alignment Data</hands-on-title>
>
>  1. Click on the {% icon galaxy-eye %} symbol of the BAM output from STAR.
>  2. There are many header lines that begin with `@` which we are not interested in.
>  3. Look at 10th read directly below the header lines:
>
>         SRR5683689.38437_GCATTC_CTTCGT	16	chr1	3439991	255	70M	*	0	0	CTTTGAATCTCTTCTTCCCAGCTAGTCATCTTCCTGCTTTTCTCTCTGTCTGTCTGTCTGTCTGTCTGTC	'0'<B<''B77<BFBBBBB7'FBFB0F7FBB<B'''<IFFBF<FBFB<FBBFBB0<BFFFBB0BBFFB<<	NH:i:1 HI:i:1 AS:i:66 nM:i:1
>
>
{: .hands_on}

The fields of the BAM file can be better explained at section 1.4 of [the SAM specification](https://samtools.github.io/hts-specs/SAMv1.pdf), but we will summarise the main fields of interest here:

* `SRR568..._GCATTC_CTTCGT`: The *readname* appended by `_`, the cell barcode, another `_`, and then the UMI barcode.
* `16`: The FLAG value

> <question-title>What does the alignment flag value of 16 tell us about this read?</question-title>
>
>   <!-- TODO This information needs to be integrated into an actual tool -->
>   We can interactively see what the different FLAG values mean by feeding values into the SAM specification to the [Picard web tool](https://broadinstitute.github.io/picard/explain-flags.html)
>
>   > <solution-title></solution-title>
>   > The read aligns to the reverse strand
>   >
>   {: .solution}
>
{: .question}

* `chr1` `3439991`: The position and base-pair of alignment of the first base of the sequence.
* A series of quality fields, with the main contributors being  the sequence and sequence quality strings.
* `NH`: The number of hits for  this read. If it is multiply mapped, then the number of multiples will be shown (here `1`, so not multiply mapped).
* `HI`: Which number this particular read is in the series of (potentially) multi-mapped reads (here `1`, not necessarily meaning the first or 'better').
* `nM`: The number of base mismatches in the alignment of this read to the reference (here `1`).


### Filtering the BAM File

If we perform counting on the current BAM file we will be counting all reads, even the undesirable ones such as those that did not align so optimally.

The main filtering steps performed on our reads so far have been relatively silent due to the 'default' parameters used.
* *UMI-tools Extract* - Filters reads for those only with matching barcodes given by our barcodes file.
* *RNA-STAR* - As seen in the log, we lose 25% of our reads for being too short or being multiply mapped.

Another filtering measure we can apply is to keep reads that we are confident about, e.g those with a minimum number of mismatches to the reference within an acceptable range. Specifically, we want to keep all reads that align to the forward or reverse strand that also have less that 3 mismatches to the reference, and are also mapped only once to the reference.

> <hands-on-title>Task description</hands-on-title>
>
> 1. {% tool [Filter BAM datasets on a variety of attributes](toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1) %} with the following parameters:
>    - {% icon param-file %} *"BAM dataset(s) to filter"*: `output_bam` (output of **RNA STAR** {% icon tool %})
>    - In *"Condition"*:
>        - In *"1: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `alignmentFlag`
>                        - *"Filter on this alignment flag"*: `0`
>        - Click on *"Insert Condition"*:
>        - In *"2: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `alignmentFlag`
>                        - *"Filter on this alignment flag"*: `16`
>        - Click on *"Insert Condition"*:
>        - In *"3: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `tag`
>                        - *"Filter on a particular tag"*: `nM:<3`
>                          <small>(**Attention! please use a lowercase 'n' here!**)
>        - Click on *"Insert Condition"*:
>        - In *"4: Condition"*:
>            - In *"Filter"*:
>                - In *"1: Filter"*:
>                    - *"Select BAM property to filter on"*: `tag`
>                        - *"Filter on a particular tag"*: `NH:<2`
>    - *"Would you like to set rules?"*: `Yes`
>        - *"Enter rules here"*: `(1 | 2) & 3 & 4`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why are we filtering only for alignment flags `0` and `16`?
> 2. What do the tag filters `nM:<3` and `NH:<2` do?
> 3. What is happening at the rules stage?
>
> > <solution-title></solution-title>
> >
> > 1. Alignment flags `0` and `16` specify that we wish to only keep reads that align to the forward and reverse strands.
> > 2. We are keeping reads that have a number of mismatches (`nM`) to the reference of less than 3, and has a number of hits (`NH`) across the reference of less than 2 (i.e. it is not a multiply-mapped read).
> > 3. [Boolean expressions](https://en.wikipedia.org/wiki/Boolean_expression) are applied that state that *either* conditions 1 or 2 can happen, in conjunction with rules 3 and 4 happening.
> >
> {: .solution}
{: .question}


## Quantification

Once we have the name of the gene for a specific read, we can count how many of those reads fall into that gene and generate a count matrix.

Counting reads is performed by two commonly-used tools:
1. RNA-STAR
2. FeatureCounts

The **RNA-STAR** tools has the ability to count reads as it maps them. **FeatureCounts** performs the same task, but is capable of counting not just at the Read level, but also at the UMI level too, such that 10 duplicate reads will be counted only once.

Unfortunately, both are currently limited to counting without being able to distinguish between different cells.

> <details-title>Example</details-title>
>
> If we consider the number of reads that align to *GeneA*, the output given by these two tools is as follows:
>
> > | (reads) | RNA STAR | FeatureCounts |
> > |---------|----------|---------------|
> > | GeneA   |   12     |    12         |
> {: .matrix}
>
> But what we actually require is:
>
> > | (reads) | C1 | C2 |
> > |---------|----|----|
> > | GeneA   | 10 |  2 |
> {: .matrix}
>
> or more specifically:
>
> > | (UMIs) | C1 | C2 |
> > |--------|----|----|
> > | GeneA  | 2  |  1 |
> {: .matrix}
>
{: .details}


In order to obtain this desired format, we must use **UMI-tools count** to perform the counting. However, this tool is dependent on **FeatureCounts** to annotate our reads with the one crucial piece of information that is missing from our BAM file: the name of the gene.

> <comment-title>Verifying missing gene name</comment-title>
> You can check this yourself by examining the {% icon galaxy-eye %} of the BAM file *"STAR Alignment file"*
{: .comment}

### Annotating Gene name with FeatureCounts

Let us annotate our BAM file with desired gene tags.

> <hands-on-title>Quantification assist via FeatureCounts</hands-on-title>
>
> 1. {% tool [FeatureCounts](toolshed.g2.bx.psu.edu/repos/iuc/featurecounts/featurecounts/2.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Alignment file"*: `mapped_reads` (output of **Filter BAM** {% icon tool %})
>    - *"Specify strand information"*:`Stranded (Forward)`
>    - *"Gene annotation file"*: `in your history`
>        - {% icon param-file %} *"Gene annotation file"*: `Mus_musculus.GRCm38.93.mm10.UCSC.ncbiRefSeq.gtf`
>    - In *"Advanced options"*:
>        - *"Count multi-mapping reads/fragments"*: `Disabled; multi-mapping reads are excluded (default)`
>        - *"Exon-exon junctions"*: `No (default)`
>        - *"Annotates the alignment file with 'XS:Z:'-tags to described per read or read-pair the corresponding assigned feature(s)."*: `Yes`
>
> 2. Examine the output BAM file
>  - Click on the {% icon galaxy-eye %} for the "Feature Counts: Alignment File"
>  - Scroll down past the header lines
>  - Scroll horizontally to the tags, observe the new `XT` tag.
{: .hands_on}

The `XS` and `XT` tags in the BAM file will now form the basis for counting reads.
With all the relevant data now in our BAM file, we can actually perform the counting via `UMI-tools count`.

> <comment-title>Verifying added gene name</comment-title>
> You can once again check this yourself by examining the {% icon galaxy-eye %} of the BAM file *"STAR Alignment file"*
{: .comment}


### Counting Genes / Cell

> <hands-on-title>Final Quantification</hands-on-title>
>
> 1. {% tool [UMI-tools count](toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_count/umi_tools_count/0.5.5.1) %} with the following parameters:
>    - {% icon param-file %} *"Sorted BAM file"*: `out_file1` (output of **FeatureCounts** {% icon tool %})
>    - *"UMI Extract Method"*: `Barcodes are contained at the end of the read separated by a delimiter`
>    - *"Bam is paired-end"*:`No`
>    - *"Method to identify group of reads"*:`Unique`
>    - *"Extra Parameters"*:
>       - *"Deduplicate per gene."*:`XT`
>       - *"Group reads only if they have the same cell barcode."*:`Yes`
>    - *"Prepend a label to all column headers"*:`No modifications`
>
{: .hands_on}


The important parameters to take note of are those given in the *Extra Parameters* where we specify that each of the reads with a `XT:Z` tag in the BAM file will be counted on a per cell basis. Reads sharing the same UMI and cell Barcodes will be de-duplicated into a single count, reducing PCR duplicate bias from the analysis.

At this stage, we now have a tabular file containing genes/features as rows, and cell labels as headers.

> <question-title></question-title>
>
> 1. How many genes do we have in the matrix?
> 2. How many cells?
>
> > <solution-title></solution-title>
> >
> > 1. 2,140 lines
> >    This information can be seen in the file preview window by clicking on the name of the file (**NOT** the {% icon galaxy-eye %} symbol).
> >
> > 2. 180 columns (not including the first column of gene names)
> >    The number of columns can be seen by scrolling the file preview window completely to the right. 192 cell barcodes were given, but due to the subsetted data, only 180 were detected.
> >
> {: .solution}
{: .question}

The generation of a single count matrix is now complete, with the emphasis on the word *single* due to the fact that we often deal in multiple batches of sequencing data.


> <comment-title>Recap of previous stages</comment-title>
>
> 1. *Barcode Extraction*:
>
>     Here we used `UMI_tools extract` on our input forward and reverse FASTQ files, and extracted the UMI and cell barcode from the forward read *sequence*, and placed it into the *header* of both forward and reverse FASTQ files. i.e. FASTQ files → Modified FASTQ files
> 2. *Mapping*:
>
>     We took the sequencing data from the reverse FASTQ file (with modified headers) and aligned it to the mouse genome, using annotations presented in the GTF file for that genome. i.e. Modified FASTQ file (reverse) → BAM file
>
> 3. *Quality Filtering*:
>
>     Reads with alignment mismatches greater than 2 were discarded, and only non multi-mapped reads that mapped to the forward or reverse strand were kept
>
> 4. *Quantification*:
>
>     Gene tags were added to our alignment file, and reads were grouped according those sharing the same gene tag, with further reduction performed by collapsing all reads sharing the same cell and UMI barcode to be counted only once.
>
{: .comment}

This concludes the first part of the tutorial which focused on the transformation of raw FASTQ data from a single batch into a count matrix. The second part of this tutorial guides us through the process of merging multiple processed batches from the first stage, and performing qualitative filtering.


> <details-title>Workflows For All Steps</details-title>
>
> Galaxy provides a workflow that captures the process of all the above stages for a [single pair of FASTQ data](workflows/scrna_pp_celseq.ga):
>
> ![Single Batch Workflow]({{site.baseurl}}/topics/single-cell/images/scrna-pre-processing/scrna_workflow_single.png "Single Batch Workflow")
>
> For multiple batch processing, Galaxy can make use of *Nested Workflows* that in this case can take in a [list of input paired FASTQ data](workflows/scrna_mp_celseq.ga) and process them in parallel.
>
> ![Multiple Batch Workflow]({{site.baseurl}}/topics/single-cell/images/scrna-pre-processing/scrna_workflow_nested.png "Nested Multiple Batch Workflow")
>
{: .details}

For the processing of 10x Genomics datasets, please refer to the [*Pre-processing of 10X Single-Cell RNA Datasets*]({% link topics/single-cell/tutorials/scrna-preprocessing-tenx/tutorial.md %}) tutorial.

# Multi-Batch Processing


![Workflow Lower]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow_lower.png %} "Multiple batch processing")

Handling more than one batch of sequencing data is rather trivial when we take into account our main goals and requirements:

1. For each batch, convert FASTQ reads from into a count matrix.
2. Merge all count matrices into a single count matrix

The first step merely requires us to run the same workflow on each of our batches, using the exact same inputs except for the FASTQ paired data. The second step requires a minimal level of interaction from us; namely using a merge tool and selecting our matrices.


## Data upload and organisation

The count matrix we have generated in the previous section is too sparse to perform any reasonable analysis upon, and constitutes data only of a single batch. Here we will use more populated count matrices from multiple batches, under the assumption that we now know how to generate each individual one of them using the steps provided in the previous section. This data is available at [`Zenodo`](https://zenodo.org/record/3253142).

Once again, file naming is important, and so we will rename our matrix files appropriately to the plate and batch they are supposed to originate from.

> <hands-on-title>Data upload and organisation</hands-on-title>
>
> 1. Create a new history and rename it (*e.g.* scRNA-seq multiple-batch tutorial)
>
> 1. Import the eight matrices (`P1_B1.tsv`, `P1_B2.tsv`, etc.) and the barcodes file from [`Zenodo`](https://zenodo.org/record/3253142) or from the data library (ask your instructor)
>    - Set the datatype of the files to **tabular**
>
>    ```
>    https://zenodo.org/record/3253142/files/P1_B1.tsv
>    https://zenodo.org/record/3253142/files/P1_B2.tsv
>    https://zenodo.org/record/3253142/files/P1_B3.tsv
>    https://zenodo.org/record/3253142/files/P1_B4.tsv
>    https://zenodo.org/record/3253142/files/P2_B5.tsv
>    https://zenodo.org/record/3253142/files/P2_B6.tsv
>    https://zenodo.org/record/3253142/files/P2_B7.tsv
>    https://zenodo.org/record/3253142/files/P2_B8.tsv
>    https://zenodo.org/record/3253142/files/celseq_barcodes.192.tabular
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 1. Rename a matrix
>    - Click on {% icon galaxy-pencil %} of the *`P1_B1.tsv`* file
>    - Set the Name field such that it is affixed with "_P1_B1" (e.g. 'multibatch_P1_B1')
>    - Click *Save*
> 1. Repeat for all matrices
>    **Pay attention to the Plate number which changes after Batch 4**
>
{: .hands_on}

## Merging Count Matrices

Before we begin, we must consider that our matrices are not equal -- e.g. Batch1 has 3 cells that describe *Genes{A,B,C,D}* whereas Batch2 has 4 cells that describe *Genes{C,D,E}*.

We have the problem that only *GeneC* and *GeneD* appear in both batches, whilst describing 7 different cells in total.

To resolve this we can perform a "Full Table Join" where the missing data for *GeneE* and *GeneA* in Batch1 and Batch2 respectively are replaced with zeroes:

![Table Join]({% link topics/single-cell/images/scrna-scanpy-pbmc3k/scrna_downstream.svg %} "Full Table Join")

> <comment-title></comment-title>
> For more information on table joins, see [this guide](http://www.sql-join.com/sql-join-types/)
{: .comment}


> <question-title></question-title>
>
> 1. Why is it required to change the column headers in the Full matrix?
> 2. Why were the cell labels in B1 and B2 the same, if they were labelling completely different cells?
>
> > <solution-title></solution-title>
> >
> > 1. Although the cell headers in each batch matrix is the same, the cells they label are *not* the same and need to be relabelled in the final matrix to tell us which batch they originated from.
> > 2. The reason the cell headers are the same is because the cells use the same barcodes, due to fact that the *same* barcodes are sometimes used across *different* batches.
> >
> {: .solution}
>
{: .question}


Let us now merge our matrices from different batches. In order to ensure that our batches are merged in the order that we wish, we should first create a list of datasets so that our matrices are merged in the order given by the list.

> <hands-on-title>Table Merge</hands-on-title>
>
> 1. **Create a Dataset List**
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
> 1. **Column Join on Collections** {% icon tool %} with the following parameters:
>   - *"Tabular Files"*: (Select the Dataset Collection icon {% icon param-collection %}, and select the Collection from the previous step each of the matrices that you wish to join)
>   - *"Identifier column"*:`1`
>   - *"Number of Header lines in each item"*:`1`
>   - *"Add column name to header"*:`Yes`
>   - *"Fill character"*:`0`
>
{: .hands_on}

The identifier column refers to the column where the gene names are listed. A 1:1 correspondence between matrices is checked, so that the merge does not concatenate the wrong rows between matrices. The *Fill character* provides a default value of 0 for cases where a Gene appears only in one of the matrices as per our example earlier.

Once the merge is complete, we can now peek at our full combined matrix by once again clicking on the file name to see a small summary.

> <question-title></question-title>
>
> Each of these matrices/batches come from the same organism.
>
> 1. How much overlap in their detected genes did you expect?
> 2. How much overlap in their detected genes did you observe?
> 3. Why is this?
>
> > <solution-title></solution-title>
> >
> > 1. Given that they come from the same sample, and each matrix has ~15,000 genes, we would have expected a high overlap between matrices, yielding ~18,000 genes in the combined matrix if we assume a difference of +/- 500 genes per batch.
> >
> > 2. We observe 20,800 genes, giving less overlap between batches of the same organism than we originally thought.
> >
> > 3. The batches were sequenced at different time points along the organisms development, and therefore different genes were expressed/detected at different time points. For early development data, the overlap can be very sparse.
> {: .solution}
{: .question}

In the new combined matrix we see that we have 1536 cells, but this number is greatly overestimated.  This is because *not all batches use the same barcodes*, and yet we applied the full set of 192 barcodes against our FASTQ data during the [*Barcode Extraction*](#barcode-extraction) stage previously.

The reason we do this is to test for cross-contamination between batches, the details of which are better explained in the [*accompanying slides*]({% link topics/single-cell/tutorials/scrna-preprocessing/slides.html %}).


## Guarding against Cross-Contamination

There are multiple possible ways to configure a plate for sequencing multiple batches. Thankfully, Galaxy provides a tool that caters for this, and checks for cross-contamination in any experimental setup. It requires only the following information:

1. A full list of barcodes
2. Which barcodes apply to which batches
3. Which batches apply to which plates

Since the plating protocol we are using is that designed by the Freiburg MPI Grün lab, we will follow their structure.

> <details-title>Plating protocol</details-title>
>
> - Barcodes:
>     These are each 8bp long, with an edit distance of 2, and there 192 of them.
>
>     > |:-------:|:-----------:|
>     > | 001-006 | AACACC AACCTC AACGAG AACTGG AAGCAC AAGCCA |
>     > | 007-012 | AAGGTG AAGTGC ACAAGC ACAGAC ACAGGA ACAGTG |
>     > |    .    |        .  |
>     > |    .    |        .  |
>     > | 180-186 | TTACGC TTCACC TTCCAG TTCGAC TTCTCG TTGCAC |
>     > | 187-192 | TTGCGA TTGCTG TTGGAG TTGGCA TTGGTC TTGTGC |
>     {: .matrix}
>
> - Plates:
>     Here we have 8 batches spread out over 2 plates, with alternate barcode striping.
>
>     > |          | 001-096 | 097-192 | 001-096 | 097-192 |
>     > |----------+---------+---------+---------+---------|
>     > | Plate 1  |  B1     | B2      | B3      | B4      |
>     > | Plate 2  |  B5     | B6      | B7      | B8      |
>     {: .matrix}
>
{: .details}

This plating protocol can be converted into a more textual format, which allows for many variable setups (see *Help* section of *Cross-contamination Barcode Filter* tool).

    [Barcodes → Batches]
    001-096: B1 , B3 , B5 , B7
    097-192: B2 , B4 , B6 , B8

    [Plates → Batches]
    1: B1 , B2 , B3 , B4
    2: B5 , B6 , B7 , B8

Let us now apply this protocol to our count matrix, and look for any cross-contamination.

> <hands-on-title>Barcode Filtering</hands-on-title>
>
> Select **Cross-contamination Barcode Filter** {%icon tool %} with the following parameters:
>  - *"Input Matrix"*:`Column Join output` (merged matrices)
>  - *"Complete Barcodes"*:`celseq_barcodes.192.tabular` (barcodes file)
>  - *"Plate Protocol"*:`Custom`
>     - *"Under 'Barcode Format'"*:
>        - Select `+ Insert Barcode Format`:
>           - *"1: Barcode Format"*:
>              - *"Barcode Range: Start"*:`1`
>              - *"Barcode Range: End"*:`96`
>              - *"Batches utilizing this Range"*:`1,3,5,7`
>        - Select `+ Insert Barcode Format`:
>           - *"2: Barcode Format"*:
>              - *"Barcode Range: Start"*:`97`
>              - *"Barcode Range: End"*:`192`
>              - *"Batches utilizing this Range"*:`2,4,6,8`
>     - *"Under 'Plate Format'"*:
>        - Select `+ Insert Plate Format`:
>           - *"1: Plate Format"*:
>              - *"Plate Number"*:`1`
>              - *"Batches within this Plate Number"*:`1,2,3,4`
>        - Select `+ Insert Plate Format`:
>           - *"2: Plate Format"*:
>              - *"Plate Number"*:`2`
>              - *"Batches within this Plate Number"*:`5,6,7,8`
>  - Expand the *"RegEx Parameters"* section:
>     - *"RegEx to extract Plate, Batch, and Barcodes from headers"*:`.*P(\\d)_B(\\d)_([ACTG]+)`
>     - *"RegEx to replace Plate, Batch, and Barcodes from headers"*:`P\\1_B\\2_\\3`
>
{: .hands_on}

> <comment-title>Regular Expressions</comment-title>
> The [regular expression](https://www.regular-expressions.info/quickstart.html) (RegEx) used in the final steps of the above *Hands-On* is required in order to tell us how to capture the important information in the cell headers contained in brackets `(` `)`, where `\\d` denotes an expected digit, and `[ACTG]+` denotes 1 or more characters matching A or C or T or G.
>
> The information captured in the brackets `(` `)` can then be placed in the desired arrangement, where `Place \\1 Matches \\2 Here \\3` would place the first `\\d` after "Place ", the second after "Matches ", and so on.
>
{: .comment }

The plot that follows tells us everything we need to know about each of our batches. Each batch is essentially tested against the full set of barcodes in order to assert that only the desired or 'Real' barcodes have been sequenced.

### Cross-contamination Plot

![Contamination Plots]({% link topics/single-cell/images/scrna-pre-processing/scrna_crosscontamination.png %} "The Pre-filter and Post-filter plots")

Two things to take note of:

1. In the pre-filter plot, we can see how only half of the sequences in each batch map to half the barcodes. This shows very little cross-contamination, and proves that our data is real.
2. The post-filter plot essentially removes the false barcodes from each batch and retains only the 'Real' barcodes.

> <question-title></question-title>
>
> 1. The count matrix that is output from this tool has only half the number of cells as the original input count matrix. Why is this?
> 1. Which batches yield worrying levels of cross-contamination?
> 1. Which batches should we remove from all further analysis?
>
> > <solution-title></solution-title>
> >
> > 1. Because only half the barcodes in each batch were real. The *UMI-tools extract* took the entire barcodes file to filter against each batch, and the *UMI-tools count* also took the entire barcodes file to count against each batch. Naturally, each batch produced 192 cells, even though 96 were real. As a result of joining each of these matrices we ended up with a count-matrix of $$8 * 192 = 1536$$ cells. The cross-contamination tool removes the false barcodes (50% in each batch), resulting in $$768$$ cells.
> > 1. Batch 4 and Batch 6 both appear to have a significant number of mid-to-high range counts in cells under the *False Positives* section in these batches. This means that the cell barcodes that we should *not* be detecting in those batches, did in fact detect cells.
> > 1. Batch 4 does still have a good number of *True Positives*, despite the many mid-level *False Positives* so we could still perhaps use the cells from that Batch. Batch 6 on the other hand appears to derive most of its counts from the *False Positives*, and therefore is likely not so suitable for further analysis.
> {: .solution}
{: .question}

All *False Positives* from all batches are filtered out, leaving only the *True Positives* (or 'Real' barcodes) in the remaining filtered matrix. It is up to the user to filter further based on the contamination information that they derive from this step. With this, a QC filtered count-matrix is produced that can then be used for further downstream analysis.

<!--
Factoid: We can convert the number of UMIs to the number of molecules using a transformation script.
-->

# Conclusion


In this tutorial we have learned the importance of barcoding; namely how to define, extract, and annotate them from our reads and into the read headers, in order to preserve them during mapping. We have discovered how these barcoded reads are transformed into counts, where the cell barcode and UMI barcode are used to denote individual cells and to correct against reads that are PCR duplicates. Finally, we have learned how to combine separate batch data as well as being able to check and correct against cross-contamination.

![Recap of workflow]({% link topics/single-cell/images/scrna-pre-processing/scrna_workflow.svg %} "A recap of the entire workflow")

This tutorial is part of the https://singlecell.usegalaxy.eu portal ({% cite tekman2020single %}).

tools_json:
["UMI-tools extract", "toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_extract/umi_tools_extract/0.5.5.1", "RNA-STAR", "toolshed.g2.bx.psu.edu/repos/iuc/rgrnastar/rna_star/2.7.7a", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.9", "Filter BAM datasets on a variety of attributes", "toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1", "FeatureCounts", "toolshed.g2.bx.psu.edu/repos/iuc/featurecounts/featurecounts/2.0.1", "UMI-tools count", "toolshed.g2.bx.psu.edu/repos/iuc/umi_tools_count/umi_tools_count/0.5.5.1"]


topic: single-cell
tutorial: scrna-data-ingest
title: Converting between common single cell data formats
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["data management", "data import"]
content_hash: fe1dac0ef1ba5eecaf7e9cb937af8c61
last_modified: 2026-05-27T09:01:49.747711
zenodo_link: https://zenodo.org/record/4574153
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-data-ingest/tutorial.html

content:
You finally decided to analyse some single cell data, you got your files either from the lab or publicly available sources, you opened the first tutorial available on Galaxy Training Network and... you hit the wall! The format of your files is not compatible with the one used in tutorial! Have you been there?
This tutorial was created to help you overcome that problem and ensure data interoperability in single cell analysis. Once you get your data into Galaxy in the right format, that's already 50% of success. Additionally, by using format conversion, you will be able to use different packages presented in tutorials that may require different datatypes.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Single cell datatypes

To start with, here are the most common formats and datatypes that you might come across if you work with single cell data:
- **Tabular** - simply using TSV, CSV or TXT formats to store expression matrix as well as cell and gene metadata.
- **MTX** - it's just a sparse matrix format with genes on the rows and cells on the columns as output by Cell Ranger.
- **HDF5** - Hierarchical Data Format - can store datasets and groups. A dataset is a  a multidimensional array of data elements, together with supporting metadata. A group is a structure for organizing objects in an HDF5 file. This format allows for storing both the count matrices and all metadata in a single file rather than having separate features, barcodes and matrix files.
- **AnnData objects** - [anndata](https://anndata.readthedocs.io/en/latest/) is a Python package for handling annotated data matrices. In Galaxy, you'll see AnnData objects in **h5ad** format, which is based on the standard HDF5 (h5) format. There are lots of Python tools that work with this format, such as Scanpy, MUON, Cell Oracle, SquidPy, etc.
- **Loom** - it is simply an HDF5 file that contains specific groups containing the main matrix as well as row and column attributes and can be read by any language supporting HDF5. [Loompy](https://linnarssonlab.org/loompy/) has been released as a Python API to interact with loom files, and [loomR](https://github.com/mojaveazure/loomR) is its implementation in R.
- **Zarr** - a Python package providing an implementation of compressed, chunked, N-dimensional arrays, designed for use in parallel computing. The Zarr file format offers powerful compression options, supports multiple data store backends, and can read/write your NumPy arrays.
- **Seurat objects** - a representation of single-cell expression data for R, in Galaxy you might see them in **rdata** format.
- **Single Cell Experiment (SCE) object** - defines a S4 class for storing data from single-cell experiments and provides a more formalized approach towards construction and accession of data. The S4 system is one of R's systems for object oriented programing. In Galaxy you might see SCE objects in **rdata** format.
- **CellDataSet (CDS) object** - the main class used by Monocle to hold single cell expression data. In Galaxy you might see CDS objects in **rdata** format.

<!---
TO FURTHER IMPROVE THE TUTORIAL:
include images showing the structure of those files
-->

{% snippet faqs/galaxy/tutorial_mode.md %}

# Data import
As you can see above, there are multiple ways to store single cell data. Therefore, there are also many ways how you can get that data!
Obviously, before any format conversion, we need to import the data. In our tutorials we often use [Zenodo](https://zenodo.org/) links, but you can also upload the files directly from your computer. There are also publicly available resources which you can easily access through public atlases, such as [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home) or [Human Cell Atlas data portal](https://data.humancellatlas.org/). We created a [dedicated tutorial]({% link topics/single-cell/tutorials/EBI-retrieval/tutorial.md %}) to show how to use those atlases to retrieve data. But today we're here to focus on data conversion!

# SCEasy Tool
In Galaxy Toolshed there is a wonderful tool called {% tool [SCEasy](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} which allows you to convert between common single cell formats, such as:
- AnnData to CellDataSet (CDS)
- AnnData to Seurat
- Loom to AnnData
- Loom to SingleCellExperiment (SCE)
- SingleCellExperiment (SCE) to AnnData
- SingleCellExperiment (SCE) to Loom
- Seurat to AnnData
- Seurat to SingleCellExperiment (SCE)

> <warning-title>Two SCEasy tools</warning-title>
>  
> As of the writing of this tutorial, the updated SCEasy tool is called {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} and it's only available on *usegalaxy.eu*. The second tool is called {% tool [SCEasy convert](toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1) %} and it works on *usegalaxy.org*, however has limited conversion options. Both tools should be visible on *singlecell.usegalaxy.eu* if you try to find them in the search box.
>
{: .warning}

In this tutorial you will see multiple examples of SCEasy Tool in action. However, sometimes it is useful to know how to do this conversion manually or at least to know how it all works and better understand the structure of the files. Therefore, we also have examples showing how to convert objects manually or prepare the input files for some of our single cell tutorials and workflows.

# Seurat -> AnnData
As mentioned above, there are two SCEasy tools currently available. When we speak about conversion to AnnData, the latest tool {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} converts Seurat into the latest version of AnnData object. However, currently it is not possible to use that output with downstream EBI single-cell tools because they only support older versions of the AnnData. If you need to use those tools, the solution is to take advantage of the second SCEasy tool {% tool [SCEasy convert](toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1) %} which has limited conversion options but can generate AnnData files compatible with the EBI single-cell tools that we are going to use next in the workflow.
Our Seurat starting file was generated by pulling data from [Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/home) and transforming it into the desired format, what was shown in the [data import tutorial]({% link topics/single-cell/tutorials/EBI-retrieval/tutorial.md %}). It will be our toy dataset.

> <hands-on-title>Get toy data</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo](https://zenodo.org/records/10397653)
>
>    ```
>    https://zenodo.org/records/10397653/files/Seurat_object.rdata
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>    
> 3. Alternatively, you can import history where we created the Seurat object: [Input history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/ebi-scxa---anndata-scanpy-or-seurat-object-1)
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 4. **Rename** {% icon galaxy-pencil %} the dataset `Seurat object`
> 5. Check that the datatype is `rdata`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="rdata" %}
>
{: .hands_on}

You will now have the option of using one tool to do the conversion, or going through a more complex conversion to an anndata object usable in the [**Filter, plot and explore single-cell RNA-seq data with Scanpy** tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %})

{% include _includes/cyoa-choices.html option1="Quick one tool method" option2="Convert to AnnData object" default="Quick one tool method"
       text="Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!" disambiguation="seurat2anndata" %}

<div class="Quick-one-tool-method" markdown="1">

## Quick one tool method

> <hands-on-title> SCEasy Converter </hands-on-title>
>
> 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `Seurat to AnnData`
>    - {% icon param-file %} *"Input object in rds,rdata format"*: `Seurat object`
> 2. **Rename** {% icon galaxy-pencil %} the output: `AnnData object`
>    
{: .hands_on}

And that's it! Please note that the output file is the newest AnnData version and is not compatible with EBI tools used in [Filter, Plot, Explore workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/copy-of-filter-plot-and-explore-single-cell-rna-seq-data-imported-from-uploaded-file-3).

</div>

<div class="Convert-to-AnnData-object" markdown="1">

## Convert to AnnData object compatible with Filter, Plot, Explore workflow

Here, we use the other SCEasy tool which generates older version of the AnnData which is compatible with the EBI single-cell tools that we are going to use next in the workflow.

> <hands-on-title> Seurat to AnnData with SCEasy convert </hands-on-title>
>
> 1. {% tool [SCEasy convert](toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1) %} with the following parameters:
>    - *"Direction of conversion"*: `Seurat to AnnData`
>    - {% icon param-file %} *"Input object in Seurat RDS format"*: `Seurat object`
>      (if the dataset does not show up in the corresponding input field, just drag the dataset from the history panel and drop into the input field)
>    - *"Name of the assay to be transferred"*: `RNA`
>    - *"Data type of the assay to be transferred"*: `data`
>   
> 2. **Rename** {% icon galaxy-pencil %} the output `Converted EBI-compatible AnnData file`.
>
> 3. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>    
{: .hands_on}

Now, if we do small modification to metadata, we can use the generated dataset as an input file in the workflow we created in the [previous tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %})! Let's see how it works:

> <hands-on-title> Modify AnnData object</hands-on-title>
>
> 1. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92) %}
> 2. Make sure you are using version **1.8.1+galaxy92** of the tool (change by clicking on {% icon tool-versions %} Versions button)
> 3. Set the following parameters:
>    - {% icon param-file %} In *"Input object in hdf5 AnnData format"*: `Converted EBI-compatible AnnData file`
>    - In *"Change field names in AnnData var"*:
>        - {% icon param-repeat %} *"Insert Change field names in AnnData var"*
>            - *"Original name"*: `name`
>            - *"New name"*: `Symbol`
>    - *"Gene symbols field in AnnData"*: `Symbol`
>    - In *"Flag genes that start with these names"*:
>        - {% icon param-repeat %} *"Insert Flag genes that start with these names"*
>            - *"Starts with"*: `mt-`
>            - *"Var name"*: `mito`
> 4. Rename {% icon galaxy-pencil %} output `AnnData for Filter, Plot, Explore workflow`
>
{: .hands_on}

The object is ready to start its journey through the [Filter, Plot, Explore workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/copy-of-filter-plot-and-explore-single-cell-rna-seq-data-imported-from-uploaded-file-3). All thanks to one conversion tool - how awesome that is!

</div>

# AnnData -> Seurat

Let's get an **AnnData object** that we can further work on. It's the object used in [many tutorials]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}), so check it out if you're curious.

> <hands-on-title>Get toy data</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo]({{ page.zenodo_link }})
>
>    If you do this tutorial just for learning purposes, you can download the downsampled dataset which will be much quicker to process:
>    
>    ```
>    https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
>    ```
>    
>    If you want to use the full dataset used in the other single-cell case study tutorials, here it is! Please note, it will take much longer to process it, so we will only show the conversions on the downsampled objects.
>    
>    ```
>    https://zenodo.org/record/7053673/files/Mito-counted_AnnData
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. **Rename** {% icon galaxy-pencil %} the datasets `Downsampled AnnData object`
> 5. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}


{% include _includes/cyoa-choices.html option1="Quick one step method" option2="Manual conversion" default="Quick one step method"
       text="Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!" disambiguation="anndata2seurat" %}

<div class="Quick-one-step-method" markdown="1">

## Quick one step method

> <hands-on-title> SCEasy Converter </hands-on-title>
>
> 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `AnnData to Seurat`
>    - {% icon param-file %} *"Input object in h5ad,h5 format"*: `Downsampled AnnData object`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Seurat object`
>    
{: .hands_on}

</div>

<div class="Manual-conversion" markdown="1">

## Manual conversion

Most of our manual conversions involve extracting tables from different data objects and importing them into the target object.
First, we will extract observations (cell metadata) and the full matrix from our AnnData.

> <hands-on-title> Inspect AnnData </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>   
> 2. **Rename** {% icon galaxy-pencil %} the output `Observations`.
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. **Rename** {% icon galaxy-pencil %} the output `Matrix`.
>    
{: .hands_on}


> <question-title></question-title>
>
> What are the rows, and what are the columns in the retrieved Matrix?
>
> > <solution-title></solution-title>
> >
> > If you just click on the `Matrix` dataset, you will see a preview showing barcodes in the first column, and genes in the first row.
> >
> {: .solution}
>
{: .question}

However, the next tool we need expects a matrix wherein the genes are listed in the first column and the barcodes are listed in the first row. Therefore, we need to transpose the current matrix.

> <hands-on-title> Transpose the matrix </hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `Matrix`
>
{: .hands_on}

And now we are ready to input that data into **DropletUtils** tool, which will separate this matrix into the cells, genes, and matrix tabular files needed to build a Seurat object.

> <hands-on-title> DropletUtils </hands-on-title>
>
> 1. {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %} with the following parameters:
>    - *"Format for the input matrix"*: `Tabular`
>    - {% icon param-file %} *"Count Data"*: output of **Transpose** {% icon tool %}
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `DefaultDrops`
>            - *"Expected Number of Cells"*: `338`
>            - *"Upper Quantile"*: `1.0`
>            - *"Lower Proportion"*: `0.0`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - *"Random Seed"*: `100`
>
{: .hands_on}

Finally, let's combine those files that we have just generated and turn them into the Seurat object!

> <hands-on-title> Create Seurat object </hands-on-title>
>
> 1. {% tool [Seurat Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/3.2.3+galaxy0) %} with the following parameters:
>    - *"Choose the format of the input"*: `10X-type MTX`
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `DropletUtils 10X Matrices`
>    - *"Gene table"*: `DropletUtils 10X Genes`
>    - *"Barcode/cell table"*: `DropletUtils 10X Barcodes`
>    - {% icon param-file %} *"Cell Metadata"*: `Observations`
>    - *"Choose the format of the output"*: `RDS with a Seurat object`
>
> 2. **Rename** {% icon galaxy-pencil %} the output `Converted Seurat object`.
>    
{: .hands_on}

As usual, you can check the [example history](https://usegalaxy.eu/u/j.jakiela/h/anndata-to-seurat-conversion) and the dedicated [workflow](https://usegalaxy.eu/u/j.jakiela/w/anndata---seurat-conversion).

</div>

# AnnData -> SingleCellExperiment (SCE)
We will work on the same AnnData object so if you create a new history for this exercise, you can either get this file from Zenodo again or just copy this dataset from the previous history.

> <hands-on-title>Get toy data</hands-on-title>
>
> 1. Create a new history for this section
> 2. Import the files from [Zenodo](https://zenodo.org/record/10391629)
>
>    ```
>    https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
>    ```
>
{: .hands_on}

{% snippet faqs/galaxy/histories_copy_dataset.md %}


{% include _includes/cyoa-choices.html option1="Quick single tool method" option2="Convert manually" default="Quick single tool method"
       text="Choose below if you just want to convert your object quickly or see how it all happens behind the scenes!" disambiguation="anndata2sce" %}

 <div class="Quick-single-tool-method" markdown="1">

## Quick single tool method

> <hands-on-title> SCEasy Converter </hands-on-title>
>
> 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `AnnData to Seurat`
>    - {% icon param-file %} *"Input object in h5ad,h5 format"*: `Downsampled AnnData object`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Seurat object`
> 3. We will use SCEasy again! {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>    - *"Convert From / To"*: `Seurat to SingleCellexperiment`
>    - {% icon param-file %} *"Input object in rds,rdata format"*: `Seurat object`
> 4. **Rename** {% icon galaxy-pencil %} the output: `SCE object`
>
{: .hands_on}

</div>

<div class="Convert-manually" markdown="1">

## Convert manually

First, we will extract observations and the full matrix from our AnnData.

> <tip-title>Skip some steps!</tip-title>
>
> If you are following this entire tutorial (rather than using a specific section necessary!), you can stay in the your previous history and just reuse outputs to build different single cell objects!
{: .tip}


> <hands-on-title> Inspect AnnData </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>   
> 2. **Rename** {% icon galaxy-pencil %} the output `Observations`.
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `The full data matrix`
>
> 4. **Rename** {% icon galaxy-pencil %} the output `Matrix`.
>    
{: .hands_on}


> <question-title></question-title>
>
> What are the rows, and what are the columns in the retrieved Matrix?
>
> > <solution-title></solution-title>
> >
> > If you just click on the `Matrix` dataset, you will see a preview, showing barcodes in the first column, while genes are in the first row.
> >
> {: .solution}
>
{: .question}


However, the next tool we need expects a matrix wherein the genes are listed in the first column and the barcodes are listed in the first row. Therefore, we need to transpose the current matrix.

> <hands-on-title> Transpose the matrix </hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `Matrix`
>
{: .hands_on}

And now we are ready to input that data to **DropletUtils** tool.

> <hands-on-title> DropletUtils </hands-on-title>
>
> 1. {% tool [DropletUtils](toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2) %} with the following parameters:
>    - *"Format for the input matrix"*: `Tabular`
>    - {% icon param-file %} *"Count Data"*: output of **Transpose** {% icon tool %}
>    - *"Operation"*: `Filter for Barcodes`
>        - *"Method"*: `DefaultDrops`
>            - *"Expected Number of Cells"*: `338`
>            - *"Upper Quantile"*: `1.0`
>            - *"Lower Proportion"*: `0.0`
>        - *"Format for output matrices"*: `Bundled (barcodes.tsv, genes.tsv, matrix.mtx)`
>        - *"Random Seed"*: `100`
>
{: .hands_on}

Finally, let's combine those files that we have just generated and turn them into the SingleCellExperiment!

> <hands-on-title> Create SCE object </hands-on-title>
>
> 1. {% tool [DropletUtils Read10x](toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_read_10x/dropletutils_read_10x/1.0.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Expression matrix in sparse matrix format (.mtx)"*: `DropletUtils 10X Matrices`
>    - {% icon param-file %} *"Gene table"*: `DropletUtils 10X Genes`
>    - {% icon param-file %} *"Barcode/cell table"*: `DropletUtils 10X Barcodes`
>    - *"Should metadata file be added?"*: {% icon param-toggle %} `Yes`
>        - {% icon param-file %} *"Metadata file"*: `Observations`
>        - *"Cell ID column"*: `index`
>     
> 2. **Rename** {% icon galaxy-pencil %} the output `Converted SCE object`.
>
{: .hands_on}

As usual, you can check the [example history](https://usegalaxy.eu/u/j.jakiela/h/anndata-to-singlecellexperiment-sce-conversion) and the dedicated [workflow](https://usegalaxy.eu/u/j.jakiela/w/anndata-to-singlecellexperiment-sce-conversion).

</div>

# Anndata -> Cell Data Set (CDS)
Cell Data Set (CDS) format is usually used when working with a package called Monocle3 ({% cite monocle3 %}). Below we show two methods on how to transform AnnData to CDS object, one of which creates an input file for [Monocle 3 tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}).

{% include _includes/cyoa-choices.html option1="Create CDS input for Monocle3 tutorial" option2="General AnnData to CDS conversion" default="General AnnData to CDS conversion"
       text="You can choose whether you want just to transform AnnData to CDS or to create CDS input file for Monocle3 tutorial to proceed with the downstream analysis described in that tutorial. Please note that depending on your dataset, you might need to refer to the first method which uses both annotated and unprocessed matrices. If you did some pre-processing on your AnnData object, then you might need to choose the first method anyway since Monocle3 performs its own pre-processing, so we would also need an unprocessed expression matrix alongside annotated (pre-processed) AnnData. That method is more expanded and specific, while the general one just shows the main principle of the conversion." disambiguation="anndata2cds" %}

<div class="Create-CDS-input-for-Monocle3-tutorial" markdown="1">

## CDS input for Monocle3 tutorial (use for pre-processed data)
The [dedicated tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}) shows how to perform trajectory analysis using Monocle3 which is the next step in the single-cell case study tutorial series, right after [pre-processing tutorial]({% link topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md %}) and [analysing the metadata]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}). To keep the continuity of the series, we will continue to work on the case study data from a mouse model of fetal growth restriction {% cite Bacon2018 %} (see [the study in Single Cell Expression Atlas](https://www.ebi.ac.uk/gxa/sc/experiments/E-MTAB-6945/results/tsne) and [the project submission](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-6945/)). After successfully completing this section with the mentioned dataset, you can use it directly in the [Monocle3 tutorial workflow](https://humancellatlas.usegalaxy.eu/u/j.jakiela/w/copy-of-trajectory-analysis-using-monocle3-1).
If you work on your own data, you might also need to follow this method since it shows how to deal with already pre-processed datasets. Since Monocle3 performs its own pre-processing, you will need both annotated and unprocessed matrices.

## Get data
Monocle3 works great with annotated data, so we will make use of our annotated AnnData object, generated in the [previous tutorial]({% link topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md %}). We will also need a ‘clean’ expression matrix, extracted from the AnnData object just before we started the processing.
You have two options for uploading these datasets. Importing via history is often faster.

> <hands-on-title>Option 1: Data upload - Import history</hands-on-title>
>
> 1. Import history from: [input history](https://humancellatlas.usegalaxy.eu/u/j.jakiela/h/monocle3-input-files)
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the history to your name of choice.
>
{: .hands_on}


><hands-on-title>Option 2: Data upload - Add to history</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the AnnData object from [Zenodo](https://zenodo.org/records/7078524)
>
>    ```
>    https://zenodo.org/records/7078524/files/AnnData_before_processing.h5ad
>    https://zenodo.org/records/7078524/files/Annotated_AnnData.h5ad
>    ```
>
> 3. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}


## Extracting annotations
To run Monocle, we need cell metadata, gene metadata, and an expression matrix file of genes by cells. (In theory, the expression matrix alone could do, but then we wouldn’t have all those useful annotations that we worked on so hard in the previous tutorials!). In order to get these files, we will extract the gene and cell annotations from our AnnData object.

> <question-title></question-title>
>
> How many lines do you expect to be in the gene and cell metadata files?
>
> > <solution-title></solution-title>
> >
> > If you click on the step with uploaded annotated AnnData file, you will see on a small preview that this object has 8605 observations and 15395 variables, so we expect to get a cell metadata file with 8605 lines and gene metadata file with 15395 lines (without headers of course!).
> >
> {: .solution}
>
{: .question}


> <hands-on-title>Extracting annotations</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Annotated_AnnData`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
> 2. Rename {% icon galaxy-pencil %} the observations annotation `Extracted cell annotations (obs)`
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Annotated_AnnData`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> 4. Rename {% icon galaxy-pencil %} the annotation of variables `Extracted gene annotations (var)`
>
>
{: .hands_on}

Quick and easy, isn’t it? However, we need to make some minor changes before we can input these files into the Monocle toolsuite.

## Cell metadata
Our current dataset is not just T-cells: as you might remember from the last tutorial, we identified a cluster of macrophages as well. This might be a problem, because the trajectory algorithm will try to find relationships between all the cells (even if they are not necessarily related!), rather than only the T-cells that we are interested in. We need to remove those unwanted cell types to make the analysis more accurate.

The **Manipulate AnnData** tool allows you to filter observations or variables, and that would be the most obvious way to remove those cells. However, given that we don't need an AnnData object, it's a lot quicker to edit a table rather than manipulate an AnnData object. Ultimately, we need cell metadata, gene metadata and expression matrix files that have macrophages removed, and that have the correct metadata that Monocle looks for. With some table manipulation, we’ll end up with three separate files, ready to be passed onto Monocle3.


> <question-title></question-title>
>
> Where is the information about cell types stored?
>
> > <solution-title></solution-title>
> >
> > We have already extracted the cell annotations file - in one of the columns you can find the information about cell type, assigned to each cell.
> > ![Cell annotations along the top, n_genes, n_counts, louvain, cell_type with a cell barcode and subsequent metadata as each row](../../images/scrna-casestudy-monocle/example_cell_annotations.png "Example cell annotations")
> >
> {: .solution}
>
{: .question}

Click on `Extracted cell annotations (obs)` file to see a small preview window. This shows you that the column containing the cell types has number 22.  We’ll need that to filter out unwanted cell types!

> <warning-title>Check the column number!</warning-title>
> If you are working on a different dataset, the number of the ‘cell_type’ column might be different, so make sure you check it on a preview and use the correct number!
{: .warning}

> <hands-on-title>Filter out macrophages</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Extracted cell annotations (obs)`
>    - *"With following condition"*: `c22!='Macrophages'`
>    - *"Number of header lines to skip"*: `1`
>    - That’s it - our cell annotation file is ready for Monocle! Let’s rename it accordingly.
> 2. **Rename** {% icon galaxy-pencil %} the output: `Cells input data for Monocle3`
>
>    > <details-title>Parameters</details-title>
>    >
>    > - `c22` means column no. 22 - that's the column with cell types, and it will be filtered for the macrophages
>    > - `!=` means 'not equal to' - we want to keep the cell types which ARE NOT macrophages
>    {: .details}
>
>    > <tip-title>Other unwanted cell types</tip-title>
>    >
>    > It might happen that during clustering you’ll find another cell type that you want to get rid of for the trajectory analysis. Then simply re-run this tool on already filtered file and change ‘Macrophages’ to another unwanted cell type.
>    {: .tip}
{: .hands_on}

## Gene annotations
Sometimes certain functionalities require a specific indication of where the data should be taken from. Monocle3 tools expect that the genes column is named ‘gene_short_name’. Let's check what the name of that column is in our dataset currently.

> <question-title></question-title>
>
> 1. Where can you check the header of a column containing genes names?
> 2. What is the name of this column?
>
> > <solution-title></solution-title>
> >
> > 1. Our extracted gene annotations file! Either by clicking on the eye icon {% icon solution %} or having a look at the small preview window.
> > 2. In our dataset the gene names are stored in a column called ‘Symbol’ - we need to change that!
> > ![The dataset in the history has a preview window showing the columns of the extracted gene annotation with each gene as a row and the metadata - index, ID, symbol - as the column names](../../images/scrna-casestudy-monocle/window_in_history.png "Preview window in the history")
> >
> {: .solution}
>
{: .question}

Let’s click on the `Extracted gene annotations (var)` file to see a small preview. We can see that the gene names are in the third column with a header `Symbol`. Keep that in mind - we’ll use that in a second!

> <hands-on-title>Changing the column name</hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `Extracted gene annotations (var)`
>    - *"using column"*: `c3` or `Column: 3`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Symbol`
>            - *"Replacement"*: `gene_short_name`
> 2. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>     - Voila! That’s the gene input for Monocle! Just a quick rename...
> 3. **Rename** {% icon galaxy-pencil %} the output: `Genes input data for Monocle3`
>
{: .hands_on}

## Expression matrix
Last, but not least! And in fact, the most important! The expression matrix contains all the values representing the expression level of a particular gene in a cell. This is why in theory the expression matrix is the only input file required by Monocle3. Without annotation files the CDS data can still be generated - it will be quite bare and rather unhelpful for interpretation, but it's possible to process.
So, the values in the expression matrix are just numbers. But do you remember that we have already done some processing such as normalisation and the calculation of principal components in the AnnData object in the previous tutorial? That affected our expression matrix. Preprocessing is one of the steps in the Monocle3 workflow, so we want to make sure that the calculations are done on a ‘clean’ expression matrix. If we apply too many operations on our raw data, it will be too ‘deformed’ to be reliable. The point of the analysis is to use algorithms that make the enormous amount of data understandable in order to draw meaningful conclusions in accordance with biology.
So how do we do that?

> <question-title></question-title>
>
> 1. How many cells and genes are there in the `Anndata_before_processing` file?
> 2. How many lines are there in `Cells input data for Monocle3`?
> 3. How many lines are there in `Genes input data for Monocle3`?
>
> > <solution-title></solution-title>
> > You can answer all the questions just by clicking on the given file and looking at the preview window.
> > 1. [n_obs x n_vars] = 31178 x 35734, so there are 31178 cells and 35734 genes.
> > 2. 8570 lines, including a header, which makes 8569 cells.
> > 3. 15396 lines, including a header, which makes 15395 genes.
> >
> {: .solution}
>
{: .question}

As you can see, there are way more genes and cells in the unprocessed AnnData file, so the expression matrix is much bigger than we need it to be. If the genes and cells we prepared for Monocle3 are not the same as in the expression matrix, Monocle3 will crash. Therefore, we have to filter that big, clean matrix and adjust it to our already prepared genes and cells files. But first, let’s extract the matrix from the unprocessed AnnData object.

> <hands-on-title>Extracting matrix</hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `AnnData_before_processing`
>    - *"What to inspect?"*: `The full data matrix`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Unprocessed expression matrix`
>
{: .hands_on}

If you have a look at the preview of `Unprocessed expression matrix`, you’ll see that the first column contains the cell barcodes, while the first row - the gene IDs. We would like to keep only the values corresponding to the cells and genes that are included in `Cells input data for Monocle3` and `Genes input data for Monocle3`. How do we do it? First, we compare the cell barcodes from `Cells input data for Monocle3` to those in `Unprocessed expression matrix` and ask Galaxy to keep the values of the matrix for which the barcodes in both files are the same. Then, we’ll do the same for gene IDs. We will cut the first columns from `Cells input data for Monocle3` and `Genes input data for Monocle3` to be able to compare those columns side by side with the matrix file.

> <hands-on-title>Cutting out the columns</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `Cells input data for Monocle3`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Cells IDs`
> 3. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - {% icon param-file %} *"From"*: `Genes input data for Monocle3`
> 4. **Rename** {% icon galaxy-pencil %} the output: `Genes IDs`
>
{: .hands_on}

> <hands-on-title> Filter matrix values by cell barcodes</hands-on-title>
>
> 1. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `Cells IDs`
>    - *"using column"*: `c1`or `Column: 1`
>    - {% icon param-file %} *"with"*: `Unprocessed expression matrix`
>    - *"and column"*: `c1`or `Column: 1`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>    - *"Keep lines of first input that are incomplete"*: `Yes`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `Yes`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Pre-filtered matrix (by cells)`
>
{: .hands_on}

Look at the preview of the output file. First of all, you can see that there are 8570 lines (8569 cells) instead of 31178 cells that were present in the matrix. That’s exactly what we wanted to achieve - now we have raw information for the T-cells that we have filtered. However, the step that we have already performed left us with the matrix whose first and second columns are the same - let’s get rid of one of those!

> <hands-on-title>Remove duplicate column (cells IDs)</hands-on-title>
>
> 1. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: `Pre-filtered matrix (by cells)`
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"List of Fields"*: `c1`
> 2. **Rename** {% icon galaxy-pencil %} the output: `Filtered matrix (by cells)`
>
{: .hands_on}

Now we will perform the same steps, but for gene IDs. But gene IDs are currently in the first row, so we need to transpose the matrix, and from there we can repeat the same steps as above for Gene IDs.

> <hands-on-title>Filter matrix by gene IDs</hands-on-title>
>
> 1. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `Filtered matrix (by cells)`
>    - The matrix is now ready to be filtered by gene IDs!
> 2. {% tool [Join two Datasets](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: `Genes IDs`
>    - *"using column"*: `c1` or `Column: 1`
>    - {% icon param-file %} *"with"*: output of **Transpose** {% icon tool %}
>    - *"and column"*: `c1` or `Column: 1`
>    - *"Keep lines of first input that do not join with second input"*: `Yes`
>    - *"Keep lines of first input that are incomplete"*: `Yes`
>    - *"Fill empty columns"*: `No`
>    - *"Keep the header lines"*: `Yes`
> 3. {% tool [Advanced Cut](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"File to cut"*: output of **Join two Datasets** {% icon tool %}
>    - *"Operation"*: `Discard`
>    - *"Cut by"*: `fields`
>        - *"List of Fields"*: `c1`
>    -  Monocle3 requires that in the matrix rows are genes, and columns are cells - that is what we've got, so there is no need to transpose matrix again. The expression matrix is ready! Let's just rename it...
> 4. **Rename** {% icon galaxy-pencil %} the output: `Expression matrix for Monocle3`
>
{: .hands_on}

{% icon congratulations %} Finally! We have prepared all the files to pass them onto the Monocle3 workflow!

## Creating CDS object
Monocle3 turns the expression matrix, cell and gene annotations into an object called cell_data_set (CDS), which holds single-cell expression data.

> <details-title>Input files</details-title>
>
> Here is what [Monocle3 documentation](https://cole-trapnell-lab.github.io/monocle3/docs/starting/) says about the required three input files:
>    - **expression_matrix**: a numeric matrix of expression values, where rows are genes, and columns are cells. Must have the same number of columns as the cell_metadata has rows and the same number of rows as the gene_metadata has rows.
>    - **cell_metadata**: a data frame, where rows are cells, and columns are cell attributes (such as cell type, culture condition, day captured, etc.)
>    - **gene_metadata**: a data frame, where rows are features (e.g. genes), and columns are gene attributes, such as biotype, gc content, etc. One of its columns should be named "gene_short_name", which represents the gene symbol or simple name (generally used for plotting) for each gene.
>
{: .details}

> <hands-on-title>Create CDS object</hands-on-title>
>
>    > <details-title>Data format</details-title>
>    >
>    > You can provide expression matrix as TSV, CSV, MTX or RDS file, while genes and cells metadata as TSV, CSV or RDS files. In our case all three files are tabular, so we will set the format to TSV.
>    {: .details}
> 1. {% tool [Monocle3 create](toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_create/monocle3_create/0.1.4+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Expression matrix, genes as rows, cells as columns. Required input. Provide as TSV, CSV or RDS."*: `Expression matrix for Monocle3`
>    - *"Format of expression matrix"*: `TSV`
>    - {% icon param-file %} *"Per-cell annotation, optional. Row names must match the column names of the expression matrix. Provide as TSV, CSV or RDS."*: `Cells input data for Monocle3`
>    - *"Format of cell metadata"*: `TSV`
>    - {% icon param-file %} *"Per-gene annotation, optional. Row names must match the row names of the expression matrix. Provide as TSV, CSV or RDS."*: `Genes input data for Monocle3`
>    - *"Format of gene annotation"*: `TSV`
>
> 2. **Rename** {% icon galaxy-pencil %} the output: `CDS input for Monocle3 tutorial`
{: .hands_on}

It was quite a long conversion, but we did it! If you're interested, the "Tip" below describes how we could possibly speed up the process using alternative tools.

> <tip-title> Alternative tools </tip-title>
>
> Generally in coding you can use different ways to achieve the same result. Similarly, in Galaxy you can use different tools but the outcome will be the same.
> For example, we could have used some tools on our starting AnnData object to remove macrophages and rename the column header, and afterwards extract observations and variables. It is a good practice to use methods from the libraries (such as Scanpy below) rather than dealing with text files rows and columns. The shown approach works well, but it might be problematic with much bigger datasets sizes. Then, you can use the following route:
> 1. {% tool [Scanpy FilterCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_filter_cells/scanpy_filter_cells/1.8.1+galaxy93) %} on annotated Anndata to remove macrophages.
> 2. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92) %} to change the genes name from *Symbol* to *gene_short_name*.
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} to extract genes and cells (to be used for Monocle).
> 4. Follow further steps to filter unprocessed matrix.
>
> You've probably used the mentioned tools before, so **challenge yourself** and try to replicate the process using the tools above!
{: .tip}

You might want to consult your results with this [control history](https://singlecell.usegalaxy.eu/u/j.jakiela/h/anndata---cds-conversion-for-monocle3-tutorial), or check out the [workflow](https://singlecell.usegalaxy.eu/u/j.jakiela/w/copy-of-trajectory-analysis-using-monocle3).
And now you have your input file ready to start the [Monocle3 tutorial]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %})!

</div>

<div class="General-AnnData-to-CDS-conversion" markdown="1">

## General AnnData to CDS conversion
In fact, you can do this conversion just in one step - check the {% icon tip %} **Tip** below!

> <tip-title>Conversion in a single step</tip-title>
>
> As mentioned in the previous section, SCEasy tool can do this conversion in a single step. Let's try that out!
>
>   > <hands-on-title> SCEasy Converter </hands-on-title>
>   >
>   > 1. {% tool [SCEasy Converter](toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2) %} with the following parameters:
>   >    - *"Convert From / To"*: `AnnData to CellDataSet`
>   >    - {% icon param-file %} *"Input object in h5ad,h5 format"*: `Downsampled AnnData object`
>   > 2. **Rename** {% icon galaxy-pencil %} the output: `CDS object`
>   >
>   {: .hands_on}
>
{: .tip}

Below we will also show the manual conversion, just to make you familiar with the structure of the files and operations needed.
We will continue working on previously used dataset, so you can copy it from your history or download from Zenodo.

> <hands-on-title>Get toy data again</hands-on-title>
>
> 1. Create a new history for this tutorial
>
> 2. You can either copy the previously used dataset from your history:
>
>    {% snippet faqs/galaxy/histories_copy_dataset.md %}
>
>    Or, alternatively, download the dataset from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/record/10391629/files/Downsampled_annotated_AnnData.h5ad
>    ```
>
> 3. **Rename** {% icon galaxy-pencil %} the datasets `Downsampled AnnData object`
> 4. Check that the datatype is `h5ad`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="h5ad" %}
>
{: .hands_on}

Now we just need to extract information about cells, genes and an expression matrix.

> <hands-on-title> Inspect AnnData </hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
> **Rename**  {% icon galaxy-pencil %} the output `Cell barcodes (obs)`.
>
> 2. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Downsampled AnnData object`
>    - *"What to inspect?"*: `Key-indexed annotation of variables/features (var)`
>
> **Rename**  {% icon galaxy-pencil %} the output `Genes (var)`.
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `output` (Input dataset)
>    - *"What to inspect?"*: `The full data matrix`
>   
> **Rename**  {% icon galaxy-pencil %} the output `Expression matrix`.
>      
{: .hands_on}

Hold on here! As mentioned, if you're converting your files to CDS, you'll probably be working with Monocle. There is one function in downstream analysis in Monocle that requires a specific name of the column containing gene symbols, and that is `gene_short_name`. If you use Galaxy buttons for the analysis, you won't be able to change that name after you create CDS file, so a good piece of advice is to rename it at this stage. There is no harm in doing this, and it might save you some time and frustration later on. You only need to check which column contains the gene symbols and what is its header - you can check that in the preview window, simply by clicking on the `Genes` dataset. In our case, that's column 3 and its name is `Symbol`. Let's change that!

> <hands-on-title>Changing the column name</hands-on-title>
>
> 1. {% tool [Column Regex Find And Replace](toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.2) %} with the following parameters:
>    - {% icon param-file %} *"Select cells from"*: `Genes`
>    - *"using column"*: `c3` or `Column: 3`
>    - In *"Check"*:
>        - {% icon param-repeat %} *"Insert Check"*
>            - *"Find Regex"*: `Symbol`
>            - *"Repl

tools_json:
["SCEasy", "toolshed.g2.bx.psu.edu/repos/iuc/sceasy_convert/sceasy_convert/0.0.7+galaxy2", "SCEasy Converter", "SCEasy convert", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/sceasy_convert/sceasy_convert/0.0.5+galaxy1", "AnnData Operations", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+galaxy92", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.3+galaxy0", "Transpose", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy0", "DropletUtils", "toolshed.g2.bx.psu.edu/repos/iuc/dropletutils/dropletutils/1.10.0+galaxy2", "Seurat Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/seurat_read10x/seurat_read10x/3.2.3+galaxy0", "DropletUtils Read10x", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/dropletutils_read_10x/dropletutils_read_10x/1.0.4+galaxy0", "Filter", "Filter1", "Column Regex Find And Replace", "toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.2", "Cut", "Cut1", "Join two Datasets", "join1", "Advanced Cut", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.1.0+galaxy2", "Monocle3 create", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/monocle3_create/monocle3_create/0.1.4+galaxy2", "Scanpy FilterCells", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_filter_cells/scanpy_filter_cells/1.8.1+galaxy93", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.8+galaxy1", "FASTQ to Tabular", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_to_tabular/fastq_to_tabular/1.1.5", "Select last", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/1.1.0", "Tabular to FASTQ", "toolshed.g2.bx.psu.edu/repos/devteam/tabular_to_fastq/tabular_to_fastq/1.1.5"]


topic: single-cell
tutorial: scrna-ncbi-anndata
title: Converting NCBI Data to the AnnData Format
difficulty: intermediate
time_estimation: 1H
requirements_json: []
tags_json: ["data management", "data import"]
content_hash: f67c05d0ef0b1ef1cabd80b0caf5a852
last_modified: 2026-05-27T09:01:49.747864
zenodo_link: https://zenodo.org/record/10101768
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-ncbi-anndata/tutorial.html

content:
The goal of this tutorial is to take raw NCBI data from some published research, convert the raw data into the AnnData format then add metadata to the object so that it can be used for further processing / analysis. Here we will look at the steps to obtain, understand, and manipulate the data in order for it to be properly processed.

The [NCBI Database](https://www.ncbi.nlm.nih.gov/) is a large repository for storing and accessing various types of biological data such as genome sequences, protein sequences, academic literature, etc. NCBI is a invaluable tool used for researchers and scientists, as a result it is important to know how to understand and utilise NCBI's data resources.

# Obtaining the Data

The first step is the obtain the data. For this tutorial, we will use data from the paper {% cite Song2022 %}. The data for this research is stored in the Gene Expression Omnibus (GEO).

{% include _includes/cyoa-choices.html option1="Manual" option2="Zenodo" default="Zenodo"
       text="If you have experience finding and downloading data from GEO then you can use the Zenodo link to load the data directly into Galaxy, if you don't have experience with GEO then you can manually download, prepare, and load the data into Galaxy." %}


<div class="Manual" markdown="1">
> <hands-on-title>Download and extract the data from GEO</hands-on-title>
>
> 1. Using a web browser navigate to the GEO repository for the paper
>    ```
>    https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE176031
>    ```
> 2. Copy the ```(http)``` link located in the supplemental materials section of the page
>
> 3. Import the data into Galaxy
>
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Change the datatype to `tar`
>
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tar" %}
>
> 5. {% tool [Unzip](toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"input_file"*: `tar` file you just imported
{: .hands_on}
</div>

<div class="Zenodo" markdown="1">
> <hands-on-title>Downloading the data from Zenodo</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the following files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    {{ page.zenodo_link }}/files/GSM5353214_PA_AUG_PB_1A_S1_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353215_PA_AUG_PB_1B_S2_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353216_PA_PB1A_Pool_1_3_S50_L002_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353217_PA_PB1A_Pool_2_S107_L004_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353218_PA_PB1B_Pool_1_2_S74_L003_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353219_PA_PB1B_Pool_2_S24_L001_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353220_PA_PB1B_Pool_3_S51_L002_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353221_PA_PB2A_Pool_1_3_S25_L001_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353222_PA_PB2B_Pool_1_3_S52_L002_dge.txt
>    {{ page.zenodo_link }}/files/GSM5353223_PA_PB2B_Pool_2_S26_L001_dge.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the datatype is `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}
</div>

We now have the raw gene expression data that we will process. However, we will need to manually add some metadata, which requires finding out some more information about our files.


> <hands-on-title>Finding the metadata</hands-on-title>
>
> 1. Follow  the link for the paper ([https://pubmed.ncbi.nlm.nih.gov/35013146/](https://pubmed.ncbi.nlm.nih.gov/35013146/)), where you can see a link for accessing the full text. Accessing the full text takes us the following page [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8748675/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8748675/)
>
> 2. Downloading the paper and looking under **Methods** we can see that there are 2 biopsy samples for patients 1-3. For this tutorial we will be processing the first 3 patient's data, therefore we will need to find more information about the biopsy samples.
> ![Snippet of paper about patient biopsies](../../images/scrna-ncbi-anndata/patient_samples_paper.png "Snippet of paper about patient biopsies")
>
> 3. On the NCBI page containing the paper we can scroll down to see a **Supplementary Materials** section. Under this section we should see various `.xlsx` files (Excel spreadsheets). These will contain metadata about the work done in the paper.
>
> 4. Looking through each file we can eventually find the spreadsheet ```41467_2021_27322_MOESM2_ESM.xlsx``` which contains information about each patient including the biopsy sample IDs and tumor specimens. So we will want to download this in order to extract out the required information!
>
> ![Snippet of clinical info](../../images/scrna-ncbi-anndata/clinical_info_spreadsheet.png "Snippet of clinical info")
>
> 5. Download Supplementary Dataset 1 (```41467_2021_27322_MOESM2_ESM.xlsx```).
>
{: .hands_on}

This one spreadsheet contains all the information we need to further understand our data.

## Understanding the Data

If we look at that excel file `41467_2021_27322_MOESM2_ESM.xlsx`, we can see multiple sheets.

> <question-title></question-title>
>
> 1. How many different patients are in this study?
>
> > <solution-title></solution-title>
> >
> > 1. Navigating to ```Clinical_info``` shows us that the data consists of 11 different patients, with some patients having multiple different samples of different tumor specimens.
> >
> {: .solution}
{: .question}

For this tutorial, we are only going to process the first 3 patients. A summary of the relevant data can be seen below:

| PatientID | Biopsy samples | Tumor Specimen |
|:----------|:---------------|:---------------|
| Patient 1 | AUG_PB1A       | left-mid       |
|           | AUG_PB1B       | right-mid      |
| Patient 2 | MAY_PB1A       | right-mid      |
|           | MAY_PB1B       | right-apex     |
| Patient 3 | MAY_PB2A       | right-anterior |
|           | MAY_PB2B       | right-mid      |

We can see that each patient has 2 different samples from different tumor locations. This is important information that needs to be added to our dataset so that we can separate it during analysis, if needed.

Unfortunately, the names of the files don't exactly match the data in our excel sheet. This can make finding the right samples a bit difficult! A summary of which samples match which file can be found below:

- **AUG_PB1A** ---> **PA_AUG_PB_1A**
- **AUG_PB1B** ---> **PA_AUG_PB_1B**
- **MAY_PB1A** ---> **PA_PB1A**
- **MAY_PB1B** ---> **PA_PB1B**
- **MAY_PB2A** ---> **PA_PB2A**
- **MAY_PB2B** ---> **PA_PB2B**

You may also notice that we have multiple sample files with the same name suffixed with `_Pool_X`. These are *replicates*, where multiple samples are taken of the same tumor area in order to get a more comprehensive view of the gene data of the tumor. We will need to combine these replicates during processing.

These replicate files can be confirmed by looking under the **Methods** section of the paper where it states that one to four arrays were used per sample.

![Snippet of paper about sample arrays](../../images/scrna-ncbi-anndata/replicates_paper.png "Snippet of paper about sample arrays")


<div class="Manual" markdown="1">

We now need to find the files relating to each patient. Selecting the output {% icon param-file %} **Unzip** tool shows you 53 files in a {% icon param-collection %} dataset collection. We will work with the 10 files associated with our target patients.

> <hands-on-title>Unhiding 10 target files</hands-on-title>
>
> 1. In the {% icon galaxy-history %} Galaxy history panel, select the *Include hidden* icon {% icon galaxy-show-hidden %} to see the 53 files in your dataset collection.
>
> 2. Scroll down to visualise the following 10 samples (likely in opposite order to this!):
>    ```
>    GSM5353214_PA_AUG_PB_1A_S1_dge.txt
>    GSM5353215_PA_AUG_PB_1B_S2_dge.txt
>    GSM5353216_PA_PB1A_Pool_1_3_S50_L002_dge.txt
>    GSM5353217_PA_PB1A_Pool_2_S107_L004_dge.txt
>    GSM5353218_PA_PB1B_Pool_1_2_S74_L003_dge.txt
>    GSM5353219_PA_PB1B_Pool_2_S24_L001_dge.txt
>    GSM5353220_PA_PB1B_Pool_3_S51_L002_dge.txt
>    GSM5353221_PA_PB2A_Pool_1_3_S25_L001_dge.txt
>    GSM5353222_PA_PB2B_Pool_1_3_S52_L002_dge.txt
>    GSM5353223_PA_PB2B_Pool_2_S26_L001_dge.txt
>    ```
> 3. For each of those 10 samples, select the *Unhide* icon {% icon galaxy-show-hidden %}. This will move them from {% icon galaxy-show-hidden %} *Hidden* to *Active*.
>
> 4. Navigate back to the {% icon galaxy-show-active %} *Active* datasets list, where you will now see your 10 target samples.
>
> 5. In the {% icon galaxy-history %} Galaxy history panel, click the {% icon galaxy-selector %} *Select items* to allow you to select multiple datasets from your history
>
> 6.  Select all 10 samples
>
> 7. Select the highlighted rectangle `10 of X` (the X may be different depending on what you've imported!) to give you a menu, from which you select `Change data type`
>
> 8. Select the dropdown menu and either type in or scroll to find `tabular.gz`
>
{: .hands_on}
</div>

# Creating the AnnData object

The next step is to convert all of the raw files into AnnData objects, this can be done one at a time or all at once by selecting ```multiple datasets``` when choosing the input and highlighting all the raw data files.

> <tip-title>Tag your datasets</tip-title>
> Before starting to process the data, it is generally a good idea to add tags to each sample in order to keep track of what data is being processed. Below is some of the imported data with added tags for the patient and sample id. We suggest you replicate this in your history.
>
> ![Imported data with tags](../../images/scrna-ncbi-anndata/metadata.png "Imported data with tags")
>
>
{: .tip}
{% snippet faqs/galaxy/datasets_add_tag.md %}
> <hands-on-title>Convert raw data to AnnData</hands-on-title>
>
> 1. {% tool [Import Anndata](toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.10.9+galaxy0) %} with the following parameters:
>    - *"Create anndata from"*: `Tabular, CSV, TSV`
>         - *"Annotated data matrix"*
>           - {% icon param-files %} *Multiple datasets*: `Select all imported files`
>         - *"Does the first column store the row names?"*: `Yes`
>
> 2. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `select any one of the AnnData files`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
{: .hands_on}

Examine {% icon galaxy-eye %} the {% icon param-file %} *"Inspect AnnData"* output. You will find a list of genes - however, the `obs` (observations) layer in the AnnData object should store cell data. The `var` (variables) should store the genes data. We need to transpose all of our AnnData objects. We will speed up the process by selecting all of the AnnData objects and processing them at once.

> <comment-title>Check your data</comment-title>
> Whilst for this specific data, the object needed to be transposed, this won't always be the case! The easiest way is to check the `obs` data of the AnnData object using the {% icon tool %} **Inspect AnnData** tool. The {% icon param-file %} `obs` file should have a column containing short nucleotide sequences (something like **CGGAAGTGATAC**). If that's the case, then the data doesn't need to be transposed, because those are the cell barcodes!
{: .comment}

> <hands-on-title>Transpose AnnData objects</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>
>    - *"Annotated data matrix"*
>         - {% icon param-files %} *Multiple datasets*: `Select all AnnData files`
>    - *"Function to manipulate the object"*: `Transpose the data matrix, leaving observations and variables interchanged`
>
{: .hands_on}

Now, we have all the AnnData objects with the data in the correct orientation. We can combine them into a single AnnData object, which will make the data easier to work with. This combination operation will add an additional column called `batch` which tells us which AnnData object each bit of data came from. This will be useful for further processing!

> <hands-on-title>Combine AnnData objects</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Select first Manipulate AnnData (transpose) output`
>    - *"Function to manipulate the object"*: `Concatenate along the observations axis`
>         - {% icon param-file %} *"Annotated data matrix to add"*: `Select all other Manipulate AnnData (transpose) outputs`
>         - *"Join method"*: `Intersection of variables`
>         - *"Key to add the batch annotation to obs"*: `batch`
>         - *"Separator to join the existing index names with the batch category"*: `-`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Combined Object`
>
{: .hands_on}

Now we have all of our gene expression data stored in a single AnnData object! We just need to add our metadata!

# Annotating the data

The next step is to annotate our data using the information gathered from the excel sheet earlier. We will do this by leveraging the `batch` column generated when the objects were combined for each individual patient/sample (indicated by the batch number). We will add in the relevant annotations. First, let's extract the observation data so we can manipulate it.

> <hands-on-title></hands-on-title>
>
> 1. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Combined Object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Observation data`
>
{: .hands_on}

For each new column, we need to replace the relevant batch numbers with the appropriate metadata. After that, we can cut the modified column out so it is separated. We will repeat this process for each piece of metadata we want to add. All the relevant data can be found in the first table of this tutorial.

Let's now add the replicate column which tells us which cells are part of pools of the same patient and tumor location.

> <hands-on-title>Create replicate metadata</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} *in a specific column* with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Observation data`
>    - *"1: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `2|4|7|8`
>         - *"Replace with"*: `poolA`
>    - **+ Insert Replacement**
>    - *"2: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `3|5|9`
>         - *"Replace with"*: `poolB`
>    - **+ Insert Replacement**
>    - *"3: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `6`
>         - *"Replace with"*: `poolC`
>    - **+ Insert Replacement**
>    - *"4: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `0|1`
>         - *"Replace with"*: `NA`
>    - **+ Insert Replacement**
>    - *"5: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `batch`
>         - *"Replace with"*: `replicate`
>
> 2. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c2`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: `output of Replace Text`
>
> 3. **Rename** {% icon galaxy-pencil %} output `Replicate Metadata`
>
{: .hands_on}

Next we will add the metadata indicating which patient each row came from.

> <hands-on-title>Create patient data</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Observation data`
>    - *"1: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `0|1`
>         - *"Replace with"*: `patient1`
>    - **+ Insert Replacement**
>    - *"2: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `2|3|4|5|6`
>         - *"Replace with"*: `patient2`
>    - **+ Insert Replacement**
>    - *"3: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `7|8|9`
>         - *"Replace with"*: `patient3`
>    - **+ Insert Replacement**
>    - *"5: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `batch`
>         - *"Replace with"*: `patient`
>
> 2. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c2`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: `output of Replace Text`
>
> 3. **Rename** {% icon galaxy-pencil %} output `Patient Metadata`
>
{: .hands_on}

We will now add a column to indicate which sample each row came from using the sample ID's described earlier.

> <hands-on-title>Create sample ID metadata</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Observation data`
>    - *"1: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `0`
>         - *"Replace with"*: `AUG_PB1A`
>    - **+ Insert Replacement**
>    - *"2: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `1$`
>         - *"Replace with"*: `AUG_PB1B`
>    - **+ Insert Replacement**
>    - *"3: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `(2$)|3`
>         - *"Replace with"*: `MAY_PB1A`
>    - **+ Insert Replacement**
>    - *"4: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `4|5|6`
>         - *"Replace with"*: `MAY_PB1B`
>    - **+ Insert Replacement**
>    - *"5: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `7`
>         - *"Replace with"*: `MAY_PB2A`
>    - **+ Insert Replacement**
>    - *"6: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `8|9`
>         - *"Replace with"*: `MAY_PB2B`
>    - **+ Insert Replacement**
>    - *"7: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `batch`
>         - *"Replace with"*: `specimenID`
>
> 2. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c2`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: `output of Replace Text`
>
> 3. **Rename** {% icon galaxy-pencil %} output `Specimen Metadata`
>
{: .hands_on}

> <details-title>$ parameter</details-title>
> You may have noticed that some of the parameters in the previous tool used the $ symbol. This is due to how the {% tool Replace Text %} tool works. The tool will replace and **update** the data for every insert operation, meaning that lines that have already been updated could be updated again.
>
> Since we are replacing our batch ID's with identifiers that include numbers, we want to prevent them from being updated again. Appending the pattern with the `$` symbol tells the tool to only replace the pattern if it is not followed by any other character (so the 1 in `AUG_PB1A` won't be replaced as it is followed by an 'A').
{: .details}

Finally we will add the tumor column which indicates which tumor sample each row belongs to.

> <hands-on-title>Create tumor metadata</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Observation data`
>    - *"1: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `0`
>         - *"Replace with"*: `left-mid`
>    - **+ Insert Replacement**
>    - *"2: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `1|2|3|8|9`
>         - *"Replace with"*: `right-mid`
>    - **+ Insert Replacement**
>    - *"3: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `4|5|6`
>         - *"Replace with"*: `right-apex`
>    - **+ Insert Replacement**
>    - *"4: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `7`
>         - *"Replace with"*: `right-anterior`
>    - **+ Insert Replacement**
>    - *"5: Replacement"*
>         - *"in column"*: `Column: 2`
>         - *"Find pattern"*: `batch`
>         - *"Replace with"*: `tumorSpecimen`
>
> 2. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c2`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: `output of Replace Text`
>
> 3. **Rename** {% icon galaxy-pencil %} output `Tumor Metadata`
>
{: .hands_on}

With all the individual metadata columns created, we can now combine them together to make a single tabular file containing our metadata. Since the ```Paste``` operation only allows us to combine two columns at once we will need to run the tool a few times to add all the columns together.

> <hands-on-title>Combine metadata</hands-on-title>
>
> 1. {% tool [Paste](Paste1) %} with the following parameters:
>    - {% icon param-file %} *"Paste"*: `Replicate Metadata`
>    - {% icon param-file %} *"and"*: `Patient Metadata`
>    - *"Delimit by"*: `Tab`
>
> 2. {% tool [Paste](Paste1) %} with the following parameters:
>    - {% icon param-file %} *"Paste"*: `Output of previous Paste`
>    - {% icon param-file %} *"and"*: `Specimen Metadata`
>    - *"Delimit by"*: `Tab`
>
> 3. {% tool [Paste](Paste1) %} with the following parameters:
>    - {% icon param-file %} *"Paste"*: `Output of previous Paste`
>    - {% icon param-file %} *"and"*: `Tumor Metadata`
>    - *"Delimit by"*: `Tab`
>
> 4. **Rename** {% icon galaxy-pencil %} output `Cell Metadata`
>
{: .hands_on}

After processing the ```Cell Metadata``` table should look like the following:

![Cell Metadata Table](../../images/scrna-ncbi-anndata/cell_metadata.png "Cell Metadata Table")

With the metadata table ready, we can add it to our original combined object!

> <hands-on-title>Add metadata to AnnData object</hands-on-title>
>
> 1. {% tool [Manipulate AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Combined Object`
>    - *"Function to manipulate the object"*: `Add new annotation(s) for observations of variables`
>         - *"What to annotate?"*: `Observations (obs)`
>         - {% icon param-file %} *"Table with new annotations"*: `Cell Metadata`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Annotated Object`
>
{: .hands_on}

{% icon congratulations %} Congratulations! You have successfully interpreted the milieu of files in a published dataset, transformed the data into a usable format, and added all the relevant annotations to the AnnData object! All that's left to do is to add some quality control metrics using automated tools!

# Adding quality control metrics

With the manual annotations added, we need to do some further processing to add some statistical metadata about the genes. This is done automatically by running two different tools.

First, we will run the ```Scanpy FilterCells``` tool without actually filtering. This tool will add some metadata about the counts and numbers of expressed genes.

> <hands-on-title>Add initial metadata</hands-on-title>
>
> 1. {% tool [Scanpy FilterCells](toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_filter_cells/scanpy_filter_cells/1.9.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in AnnData/Loom format"*: `Annotated Object`
>    - *"Name of the column in `anndata.var` that contains gene name"*: `_index`
>
{: .hands_on}

The final tool, {% icon tool %} **AnnData Operations**, will add the rest of our metadata. This is mostly information about the mitochondrial genes present in the object, roughly counted by flagging genes that start with **MT-**.

> <hands-on-title>Add final metadata</hands-on-title>
>
> 1. {% tool [AnnData Operations](toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input object in hdf5 AnnData format"*: `Output of Scanpy FilterCells`
>    - **+ Insert Flag genes that start with these names**
>    - *"1: Parameters to select cells to keep"*
>         - *"starts with"*: `MT-`
>         - *"Var name"*: `mito`
>
> 2. **Rename** {% icon galaxy-pencil %} output `Final Object`
>
> 3. {% tool [Inspect AnnData](toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Annotated data matrix"*: `Final Object`
>    - *"What to inspect?"*: `Key-indexed observations annotation (obs)`
>
{: .hands_on}

With that run we should be finished! Check the {% icon param-file %} `obs` file to see all the metadata that we've added in throughout the tutorial.

> <warning-title>Check your final object!</warning-title>
> The above tools will sometimes run without error but not produce the correct results - just because they run doesn't mean all the parameters and inputs were set properly! When {% icon galaxy-eye %} inspecting the {% icon param-file %} `obs` file of the final AnnData object, you should see that all the columns contain data. If any of the columns are blank, then something has gone wrong!
{: .warning}

{% icon congratulations %} Congratulations! Now your data is ready for further processing or analysis! You can compare your analysis results with this [example history](https://usegalaxy.eu/u/videmp/h/ncbi-to-anndata) and redo the whole analysis with a single click using this [workflow](https://usegalaxy.eu/u/videmp/w/ncbi-to-anndata).

tools_json:
["Unzip", "toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy0", "Import Anndata", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_import/anndata_import/0.10.9+galaxy0", "Inspect AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_inspect/anndata_inspect/0.10.9+galaxy0", "Manipulate AnnData", "toolshed.g2.bx.psu.edu/repos/iuc/anndata_manipulate/anndata_manipulate/0.10.9+galaxy0", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.3+galaxy1", "Cut", "Cut1", "Paste", "Paste1", "Scanpy FilterCells", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/scanpy_filter_cells/scanpy_filter_cells/1.9.3+galaxy0", "AnnData Operations", "toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.9.3+galaxy0"]


topic: digital-humanities
tutorial: open-refine-tutorial
title: OpenRefine Tutorial for researching cultural data
difficulty: introductory
time_estimation: 2H
tools_json: ["OpenRefine", "interactive_tool_openrefine"]
requirements_json: ["type: internal"]
tags_json: []
content_hash: 40a4675a612cbdf3d06e97fd00dbc32e
last_modified: 2026-05-27T09:01:48.035501
zenodo_link: https://doi.org/10.5281/zenodo.17047254
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/digital-humanities/tutorials/open-refine-tutorial/tutorial.html

content:
This tutorial shows how to use **OpenRefine** in Galaxy to clean and visualize data from the **humanities and social sciences**. It has two parts:
- **Introduction to OpenRefine**, based on {% cite Hooland_2013 %} and adapted for Galaxy.
- **Introduction to running Galaxy workflows** to visualize cleaned data and extract specific information.


## What is OpenRefine?

**OpenRefine** is a free, open-source “data wrangler” built for messy, heterogeneous, evolving datasets. It imports common formats (CSV/TSV, Excel, JSON, XML) and domain-specific ones used across GLAM (Galleries, Libraries, Archives and Museums) and official statistics (MARC, RDF serializations, PC-Axis).

It is **non-destructive**—OpenRefine does not alter your source files, but works on copies and saves projects locally. Facets and filters enable you to audit categories, surface outliers, and triage inconsistencies without writing code. Its **clustering** tools consolidate near-duplicates using both key-collision methods (fingerprint, n-gram, phonetic) and edit-distance/nearest-neighbour methods (Levenshtein, PPM) so you can standardize names and places at scale while keeping human oversight.

For enrichment, OpenRefine speaks the **Reconciliation API** to match local values to external authorities (e.g. **Wikidata**, **ROR**) and, optionally, retrieve richer metadata. Transformations—both point-and-click and **GREL** formulas—are recorded as a stepwise, undoable history that you can export as JSON and re-apply to other datasets, enabling reproducible cleaning and easy peer review. Finished tables can be exported cleanly to **CSV/TSV**, ODS/XLS(X), SQL statements, templated JSON, Google Sheets, or can be exported back to Galaxy.

## From Cleaning to Analysis in Galaxy

Once your dataset has been cleaned with OpenRefine, you often want to analyze it further or visualize specific aspects. This is where **Galaxy Workflows** become essential: they enable you to build reproducible pipelines that operate on your curated data, transitioning from one-off cleaning to structured analysis.

## What are Galaxy Workflows?

**Galaxy Workflows** are structured, step-by-step pipelines that you build and run entirely in the browser—either extracted from a recorded analysis *history* or assembled in the visual editor. They can be annotated, shared, published, imported, and rerun, making them ideal for teaching, collaboration, and reproducible research.

A captured analysis is easy to share: export the workflow as JSON (**`.ga`**: tools, parameters, and Input/Output) or export a provenance-rich run as a **[Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/)** bundling the definition with inputs, outputs, and invocation metadata. This lowers the barrier to entry (no local installs; web User-Interface (UI) with pre-installed tools and substantial compute) while preserving best practices (histories track tool versions and parameters; workflows are easily re-applied to new data).

For findability and credit, the community uses **[WorkflowHub](https://workflowhub.eu/)**—a curated registry that supports multiple workflow technologies (including Galaxy) and promotes **FAIR** principles; it offers Spaces/Teams, permissions, versioning, and **DOIs via DataCite**, with metadata linking to identifiers like **[ORCID](https://orcid.org/)** so contributions enter scholarly knowledge graphs and are properly acknowledged.

In practice, you can iterate on a workflow in a familiar Graphic-User-Interface (GUI), export the exact definition or a run package, and deposit it where peers can discover, reuse, review, and cite it—closing the loop between simple authoring and robust scholarly dissemination.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Hands on: Get the data

We will work with a slightly adapted dataset from the **[Powerhouse Museum](https://powerhouse.com.au/)** (Australia’s largest museum group) containing a metadata collection. The museum shared the dataset online before giving API access to its collection. We slightly adapted the dataset and uploaded it to Zenodo for long-term reuse. The tabular file (**36.4 MB**) includes **14 columns** for **75,811** objects, released under a **[Creative Commons Attribution Share Alike (CCASA) license](http://creativecommons.org/licenses/by-nc/2.5/au/)**. We will answer three questions: From **which category** does the museum have the most objects? From **which year** does the museum have the most objects? And **what objects does the museum have from that year?**

**Why this dataset?** It is credible, openly published, and realistically messy—ideal for practising problems scholars encounter at scale. Records include a **Categories** field populated from the **Powerhouse Museum Object Names Thesaurus (PONT)**, a controlled vocabulary reflecting Australian usage. The tutorial deliberately surfaces common quality issues—blank values that are actually stray whitespace, duplicate rows, and multi-valued cells separated by the pipe character `|` (including edge cases where **double pipes** `||` inflate row counts)—so we can practice systematic inspection before any analysis. Without careful atomization and clustering, these irregularities would bias statistics, visualizations, and downstream reconciliation.

We suggest that you download the data from the Zenodo record as explained below. This helps us with the reproducibility of the results.

> <hands-on-title>Upload your data</hands-on-title>
>
> 1. Create a new history for this tutorial and name it "Powerhouse Museum — OpenRefine"
> 2. Import the files from [Zenodo]({{page.zenodo_link}}):
>
>    ```
>    https://zenodo.org/records/17047254/files/phm_collection_adapted.tsv
>    https://zenodo.org/records/17047254/files/stopwords-en.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Ensure that the datatype of "phm_collection_adapted" is "tsv". Otherwise, use convert datatype.
> 4. Verify that the datatype of "stopwords-en" is "txt". If not, convert the datatype.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

In this first part, we will focus on working with the metadata from the Powerhouse Museum. 

# Use OpenRefine to explore and clean your dataset

Access OpenRefine as an interactive tool in Galaxy and explore your data.

## Start OpenRefine

> <hands-on-title>Opening the dataset with OpenRefine</hands-on-title>
>
> 1. Open the tool {% tool [OpenRefine](interactive_tool_openrefine) %}: Working with messy data
>    - *"Input file in tabular format"*:  `openrefine-phm-collection.tsv`
>
> 2. Click on "Run Tool".
>
>    ![OpenRefine tool interface in Galaxy](images/openrefine.png)
>
> 3. After around 30 seconds, a red dot appears over the interactive tools section on the left panel. Click on "interactive tools". A new window opens. Make sure to wait until you see the symbol with an arrow pointing outside the box, which indicates that you can start OpenRefine in a new tab. Now you can open OpenRefine by clicking on its name.
>
>    ![Open OpenRefine tool as an Interactive tool](images/interactive_tools.png)
>
> 4. Here, you can see the OpenRefine GUI. Click on `Open Project`.
>
>    ![Open OpenRefine interface](images/openrefine_interface.png)
>
> 5. Click on `Galaxy file`. If the file does not appear, you may have started OpenRefine before it was fully loaded. Retry steps 3 and 4, and the file should be visible.
>
>    ![Open OpenRefine Open Project as an input](images/openrefine_open_project.png)
>
> 6. You can see the data that has been loaded for you.
>
>    ![Open OpenRefine GUI](images/openrefine_gui.png)
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many rows does this table have?
>
> > <solution-title></solution-title>
> >
> > 1. 75809
> >
> {: .solution}
{: .question}

Great, now that the dataset is in OpenRefine, we can start cleaning it.

## Remove duplicates

In large datasets, errors are common. Some basic cleaning exercises can help enhance the data quality. One of those steps is to remove duplicate entries. 

> <hands-on-title>Removing duplicates</hands-on-title>
>
> 1. Click on the triangle on the left of `Record ID`.
>
>    ![Sort Record ID](images/sort.png)
>
> 2. Click on `Sort...`.
>
> 3. Select `numbers` and click on `OK`.
>
>    ![Sort Record ID options](images/sort2.png)
>
> 4. Above the table, click on `Sort` and select `Reorder rows permanently`.
>
>    ![Sort Record ID reorder permanently](images/sort3.png)
>
> 5. Click on the triangle left of the `Record ID` column. Hover over `Edit cells` and select `Blank down`.
>
>    ![Blank down Record ID](images/sort4.png)
>
> 6. Click on the triangle left of the `Record ID` column. Hover over `Facet`, then move your mouse to `Customized facets` and select `Facet by blank (null or empty string)`.
>
>    ![Facet by blank Record ID](images/sort5.png)
>
> 7. On the left, a new option appears under `Facet/Filter` with the title `Record ID`. It shows two choices, `true` and `false`. Click on `true`.
>
>    ![Facet by blank true Record ID](images/sort6.png)
>
> 8. Click on the triangle to the left of the column called `All`. Hover over `Edit rows`, and select `remove matching rows`.
>
>    ![Remove matching rows Record ID](images/deduplicate.png)
>
> 9. Close the `Record ID` under `Facet/Filter` by clicking on the cross (x) to see all rows.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many rows have been removed?
>
> > <solution-title></solution-title>
> >
> > 1. 84
> >
> {: .solution}
{: .question}

The dataset no longer contains duplicates based on the Record ID. However, we need to perform further cleaning to enhance the dataset.

## Use GREL

There are many ways to manipulate your dataset in OpenRefine. One of them is the Google Refine Expression Language (GREL). With the help of GREL, you can, for example, create custom facets or add columns by fetching URLs. We will use it to find and replace errors. For more information, refer to the [GREL documentation](https://openrefine.org/docs/manual/expressions).

Take a look at the `Categories` column of your dataset. Most objects were attributed to various categories, separated by "\|". However, several fields contain "\|\|" instead of "\|" as a separator. We want to unify those.

> <hands-on-title>Find and replace typos using GREL</hands-on-title>
>
> To remove the occurance of double pipe "\|\|" from the file we can do the following:
> 1. Click on the triangle on the left of `Categories` and select `Text filter`.
> 2. On the left, using the `Facet/Filter` section, search for the occurrence of \| and \|\|. There are 71061 rows with "\|" and 9 rows with "\|\|". We would like to remove these nine lines, as they were added by mistake.
> 3. Click on the triangle on the left of `Categories`, hover over `Edit cells`, and click on `Transform...`.
> 4. In the new window, use the following text `value.replace('||', '|')` as "Expression" and click on `OK`.
>
>    ![Custom text transform on column Categories](images/filter_grel3.png)
>
>    The expression replaces \|\| with \|. If you search for the occurrence of \|\| again, you will no longer get any results. 
>
>    Many different categories describe the object. You may notice duplicates categorising the same object twice.
>    We also want to remove those to ensure we only have unique categories that describe a single object.
>
> 5. Click on the triangle on the left of `Categories`, hover over `Edit cells`, and click on `Transform...`.
>
>    ![Edit cells Categories](images/filter_grel.png)
>
>    ![Transform Categories](images/filter_grel2.png)
>
> 6. In the new window, use the following text `value.split('|').uniques().join('|')` as "Expression" and click on `OK`.
>
{: .hands_on}

These expressions split categories at the pipe separator and join the unique ones within this column. As a result, duplicate categories for one object are deleted.

> <question-title></question-title>
>
> 1. How many cells had duplicated categories?
>
> > <solution-title></solution-title>
> >
> > 1. 1,668
> >
> {: .solution}
{: .question}

## Atomization

Once the duplicate records have been removed, we can examine the content of the "Categories" column more closely. Different categories are separated from each other by pipe (\|). 
Each entry can be assigned to more than one category. To leverage those keywords, the values in the Categories column must be split into individual cells using the pipe character.

> <hands-on-title>Atomization</hands-on-title>
>
> 1. Click on the triangle on the left of `Categories`, hover over `Edit cells`, and click on `Split multi-valued cells...`.
>
>    ![Atomization of Categories](images/split_multi_valued_cells.png)
>
> 2. Define the `Separator` as `\|` (pipe). Click on `OK`.
>
{: .hands_on}

Are you ready for a little challenge? Let's investigate the categories column of the museum items.

> <question-title></question-title>
>
> 1. How many rows do you have after atomizing the categories column?
> 2. How many entries do not have any category?
>
> > <solution-title></solution-title>
> >
> > 1. 168,476
> > 2. Click on the triangle on the left of `Categories` and hover over `facet` and move your mouse over `Customized facets`, and click on `Facet by blank (null or empty string)`. The `true` value for blank entries is 447.
> >    
> >   ![Facet Blank of atomized Categories](images/facet_categories_blank.png)
> >
> {: .solution}
{: .question}

## Faceting

Now that the `Categories` field is cleaned, we can check the occurrence of categories with various facets.

> <hands-on-title>Faceting</hands-on-title>
>
> 1. Click on the triangle on the left of `Categories`, hover over `Facet`, and click on `Text facet`.
> 2. On the left panel, it mentions the total number of choices. The default value of `count limit` is low for this dataset, and we should increase it. Click on `Set choice count limit`.
>
>    ![Text faceting of atomized Categories](images/text_facet.png)
>
> 3. Enter `5000` as the new limit and click on `Ok`.
>
>    ![Increasing the limit of text facetring](images/text_facet2.png)
>
> 4. Now, you see all categories. Click on `count` to see the categories sorted in descending order.
>
>    ![Text faceting of atomized Categories sorted by count](images/text_facet3.png)
>
{: .hands_on}

You can now see, from which category the museum has the most objects, one of our initial questions about the dataset.

> <question-title></question-title>
>
> 1. What are the top 3 categories? How many items are associated with each of them?
>
> > <solution-title></solution-title>
> >
> > 1. Numismatics (8011), Ceramics (7389), and Clothing and Dress (7279).
> >    Congratulations, you have just answered our first question: from which category does the museum have the most objects?
> >    It is numismatic objects, meaning coins. This makes a lot of sense; coins have a long history and convey a lot of information. They are therefore very interesting for researchers.
> >    Moreover, they are robust and compact, making them durable and relatively easy for museums to store.
> >
> {: .solution}
{: .question}


## Clustering

The clustering allows you to solve issues regarding case inconsistencies, incoherent use of either the singular or plural form, and simple spelling mistakes. We apply those to the object categories for the next step of cleaning.

> <hands-on-title>Clustering of similar categories</hands-on-title>
>
> 1. Click on the `Cluster` button on the left in the `Facet/Filter` tab.
> 2. Use `Key collision` as clustering method. Change the Keying function to `n-Gram fingerprint` and change the n-Gram size to `3`.
>
>    ![Cluster and edit column Categories](images/cluster.png)
>
> 3. Click on the `Cluster` button in the middle window.
>
>    ![Clustered and merged similar Categories](images/cluster2.png)
>
> 4. Here, you can see different suggestions from OpenRefine to cluster different categories and merge them into one. In our tutorial, we merge all the suggestions by clicking on `Select all` and then clicking on `Merge selected and re-cluster`.
>
> 5. Now, you can close the clustering window by clicking on `close`.
>
>    Be careful with clustering! Some settings are very aggressive, so you might end up clustering values that do not belong together!
>
>    Now that the different categories have been clustered individually, we can reassemble them in the respective object single cell.
>    
> 6. Click the Categories triangle and hover over the `Edit cells` and click on `Join multi-valued cells`.
> 7. Choose the pipe character (`|`) as a separator and click on `OK`.
>
>    ![Join multi-valued cells on Categories](images/join.png)
>    
> The rows now look like before, with a multi-valued Categories field.
>
{: .hands_on}

You have now successfully split, cleaned and re-joined the various categories of objects in the museum's metadata! Congratulations.
As before the splitting of columns, we are now back to 75725 rows.
When you are satisfied with your data, choose whether to export the dataset to your Galaxy history or download it directly to your computer.

## Exporting your data back to Galaxy

Exporting your data back to Galaxy allows you to analyse or visualise it with further tools in the platform.
But OpenRefine also allows you to export your operation history, detailing all the steps you took in JSON format.
This way, you can import it later and reproduce the exact same analysis. To do so:

> <hands-on-title>Exporting the OpenRefine history</hands-on-title>
> 
> 1. Click on `Undo/Redo` on the left panel.
> 2. Click on `Extract...`.
>
>    ![Extract OpenRefine](images/extract_tasks.png)
>
> 3. Click on the steps that you want to extract. Here, we selected everything.
> 4. Click on `Export`. Give your file a name to save it on your computer.
>
> ![Extract OpenRefine GUI](images/extract_tasks2.png)
>
{: .hands_on}

However, you will also ensure that you save your data. You can download your cleaned dataset in various formats, such as CSV, TSV, and Excel, within OpenRefine. 
For further analysis or visualisation, we suggest you export it to your Galaxy history.

> <hands-on-title>Exporting the results and history</hands-on-title>
>
> 1. Click on `Export` at the top of the table.
> 2. Select `Galaxy exporter`. Wait a few seconds. In a new page, you will see a text as follows: "Dataset has been exported to Galaxy, please close this tab". When you see this, you can close that tab.
>
>    ![Export results of OpenRefine](images/export_results3.png)
>
> 3. You can now close the OpenRefine interactive tool. For that, go to your history with the orange item `OpenRefine on data [and a number]`. This is your interactive tool. Click "OK" on the small square (it says "Stop this interactive tool" when you mouse over it). You do not need it for your next steps.
> 4. You can find a new dataset (named something like "openrefine-Galaxy file.tsv") in your Galaxy History (with a green background). It contains your cleaned dataset for further analysis.
> 5. You can click on the eye icon ({% icon galaxy-eye %}) and investigate the table.
>
>    ![Cleaned dataset](images/dataset_cleaned.png)
>
{: .hands_on}

Awesome work! However, you may recall that we still have two unanswered questions about our data: From which year does the museum have the most objects? And what objects does the museum have from that year?

# Run a Galaxy Workflow on your cleaned data

Congratulations, you have successfully cleaned your data and improved its quality!
But what can you do with it now?
This depends on your research objectives. For us, it is interesting to extract further information from the data.
To make it easier for you, we have created a workflow that links all the tools needed for this analysis.
We wanted to know which year the museum had the most objects and what they were.
You can follow along and answer those questions with us, or explore the Galaxy tools on your own, to adapt the analysis to your needs.
In this case, be sure to check out our other tutorials, particularly the introductory ones.

## How to find and run existing workflows

> <hands-on-title>Run a Galaxy workflow on your dataset</hands-on-title>
>
> There are different ways to import or create a workflow to Galaxy. For example, you can import a workflow from the registered workflows on [WorkflowHub](https://workflowhub.eu/) which is a registry for describing, sharing, and publishing scientific computational workflows. To do that, you have to navigate to the [WorkflowHub](https://workflowhub.eu/) and find the workflow of interest. In this tutorial, we are working [with this workflow](https://workflowhub.eu/workflows/1884?version=1). When you open the link to this workflow on WorkflowHub, you see the following page:
>
> ![Workflowhub page](images/workflowhub.png)
>
> Please click on the `Run on Galaxy` button on top right. After doing this, you will be redirected to your Galaxy account and see the workflow automatically in your middle panel as follows:
>
> ![Workflow imported to Galaxy](images/workflow.png)
>
> This is one way of importing a workflow to your account.
> 
> Let's assume you have done this and imported the workflow to your Galaxy account.
> 1. You can find all workflows available to you by clicking on the Workflows Icon ({% icon galaxy-workflows-activity %}) on the left panel.
>
>    ![Workflows button](images/workflows.png)
>
> 2. Then, you can select and run different workflows (if you have any workflows in your account). Here, let's click on the Run button ({% icon workflow-run %}) of the workflow we provided to you in this tutorial.
>
>    ![Select this workflow](images/select_workflow.png)
>
> 3. Determine the inputs as follows:
>    
>    Input: `openrefine-Galaxy file.tsv`—This is the file you cleaned in OpenRefine.
>    
>    stop_words_english: `stop_words_english.txt`, which is the file we provided to you in this tutorial.
>
>    ![Determine the inputs of the workflow](images/workflow_inputs.png)
>
> 4. Click on the `Run Workflow` button at the top.
> 5. You can follow the stages of different jobs (computational tasks). They will be created, scheduled, executed, and completed. When everything is green, your workflow has run fully and the results are ready.
>
>    ![Overview of the workflow](images/workflow_overview.png)
>
{: .hands_on}

What can you see here? To follow along, we made all substeps of the task available as outputs. 
To answer our question of which year most elements in the museum derive from, we first cut the column of production time from the table.
You can see this in the file `Cut on Data (Number)`.
From this, we filter only the dates that derive from specific years, not year ranges. (See `Filter Tabular on Data (Number)`.) 
You can click on the arrow in the circle button of this dataset (`Run job again`) to see what exact input was used to exclude year ranges.
Regular expressions help clean remaining inconsistencies in the dataset. (Dataset: `Column Regex Find And Replace on Data (Number)`)
Sorting the production date in descending order, as done in the dataset `Sort on data (lowest Number)`, reveals that one faulty dataset, which is supposed to have been created in 2041, is part of the table. 
We remove it in the next step with the tool `Remove beginning`.

The tool **Datamash** allows for summarising how many elements arrived at the museum in each year.  (Dataset: `Datamash on data (Number)`.)
After we apply this tool, the dataset is no longer 7738 lines long, but only 259. 
This is because the number of times each year appeared in the table was summed up in a second column.
Sorting in ascending order (`Sort on data (Number)`) shows a chronological dataset with the earliest entries in the beginning and the most recent entries at the end of the table.
 We can easily visualise this in a (particularly crowded) bar chart directly within Galaxy. (`Bar chart on data (Number)`)
But this is not the most optimal view to show us which year most objects derive from.

To determine from which year most objects originate, we use another sorting order (`Sort on Data (highest number)`). 

> <question-title></question-title>
>
> 1. From what year does the museum have the most objects?
>
> > <solution-title></solution-title>
> >
> > 1. The dataset `Sort on data (highest number)` shows the number of objects by year, sorted from most to least. 288 items are noted for the year 1969 in the first row. This is the year from which the museum has the most (clearly datable) objects.
> >
> {: .solution}
{: .question}

The next four steps parse this year as a conditional statement step by step. (`Select first on data (Number)`, `Cut on data (Number)`, `Parse parameter value on data (Number)` and `Compose text parameter value`.)  
This means, even if you upload another dataset, the highest number is always selected and taken as an input for the next steps.

Based on this input, which is determined by the year with the highest input, `Search in textfiles on data (Number)` searches for object descriptions from the 288 objects of the most prominent year. 
The table is very rich in information, but not that easy to digest.
To make the table more accessible, we create a word cloud of the object descriptions with the offered stop word list.
If you click on the stop word list we provided, you see what "fill words" are excluded from the word cloud. In essence, only words conveying meaning remain.
This helps us quickly determine what kinds of objects the museum has from this popular year.
The dataset `Word cloud image` shows that most objects from the museum are negatives from Davis Mist, a famous Australian photographer, who created them that year and donated them to the museum.

![Word cloud of objects' descriptions](images/display_1969.png)

# Conclusion

Congratulations! You used OpenRefine to clean your data and ran a workflow from Galaxy with your results! You now know how to perform basic steps in Galaxy, run OpenRefine as an interactive tool, and transfer data from Galaxy to OpenRefine and back. On the way, you have learned basic data cleaning techniques, such as facetting, to enhance the quality of your data. To extract further information from the cleaned data, running a pre-designed workflow showed you a glimpse into Galaxy. Of course, you can always conduct your own analysis using the tools most useful to you, instead.


topic: digital-humanities
tutorial: Whisper_speech_recognition
title: Transcribing Audio and Video files with Automated Speech Recognition
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["audio", "video"]
content_hash: 6f35c8a608b342a77075d9f13944f6fb
last_modified: 2026-05-27T09:01:48.029840
zenodo_link: https://zenodo.org/records/17949386
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/digital-humanities/tutorials/Whisper_speech_recognition/tutorial.html

content:
Audio and media files are a rich source in the social sciences and the humanities. 
One example is oral history, in which historians interview contemporary witnesses to access information beyond written sources. 
{% cite Cole2022 %} gives an introduction to the topic and lays out potential challenges. But once you have your audio or audiovisual material, 
how can you make it accessible for structured analysis? You will often need to transcribe the media content into machine-readable text first.
This is a task Galaxy can help with.
The platform contains several tools for Automatic Speech Recognition (ASR). 
From uploading and converting to suitable file types to transcriptions and post-processing, Galaxy has you covered.

This tutorial aims to make the audio track of a video machine-readable for further processing. 
It is the pre-processing step to get you started. 
Once you have covered this, you can further analyse your material more thoroughly. This step will not be covered in this tutorial.

We use a video of the 1960 United States presidential debate between John F. Kennedy and Richard Nixon.
It was broadcast on television, and the recording is now in the public domain. We will use {% tool WhisperX %} to transcribe the material. 
The advantage of {% tool WhisperX %} over {% tool Whisper %}, another tool available on Galaxy, is its speaker diarization. 
This means the tool recognises different speakers and (tries to) allocate the text passages accordingly. 
To make it easier to distinguish who said what, we later replace the tool's naming convention, for example, "SPEAKER_00," with the person's name. 
Then we extract passages from Kennedy's and Nixon's speeches for further analysis. 


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Upload your Files

You can upload data in various ways. Here are some examples:

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from the shared data library (`GTN - Material` -> `{{ page.topic_name }}`
>     -> `{{ page.title }}`) or from [Zenodo](https://zenodo.org/records/17949386):
>
>    ```
>    https://zenodo.org/records/17949386/files/1960_kennedy-nixon_1.mp4
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>    
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. Check the Datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

We upload one file: the video of the presidential debate we want to transcribe. 

Once you click start, your upload should begin. It will turn green once it is done.
If you follow this workflow with one of your own files, make sure to check the file's data type. 
If your file's format is not supported as input by {% tool WhisperX %} or {% tool Whisper %}, you can use {% tool FFMPEG %} to convert it to a format that the tool of your choice supports. 
You can check which file types a tool supports by clicking on the tool of your choice, for example, {% tool WhisperX %}, and clicking on "accepted formats" below the upload section, where you choose the tool input.


## Get to know your Input

You should now have one item in your history.
But how can you watch the video? If you try to access it via the view option, a download button pops up. 
But you do not need to download the file - you can watch it directly within Galaxy:

> <hands-on-title> Watch the Video </hands-on-title>
>
1. In the **Activity bar**, click on **Visualisations** {% icon galaxy-visualise %}.
2. Search for **Media Player** and **click on it**.
3. In the drop-down menu, select `1960_kennedy-nixon_1.mp4`. A black screen appears.
4. Click on the **Play icon** in the middle and take a look at the video.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How long is the video?
> 2. What is the colour of the video?
>
> > <solution-title></solution-title>
> >
> > 1. 16:11
> > 2. The video is in black and white.
> >
> {: .solution}
>
{: .question}

This video shows the first televised presidential debate held between Richard Nixon and John F. Kennedy in 1960. 
{% cite Druckman_2003 %} looks at the media history of this document, in case you want to know more about it.
We will not analyse its aesthetics or importance, but focus on the spoken words.
You can use Galaxy for automated speech recognition of the video.


# Transcribe your Media File(s)
Galaxy has different tools to transcribe your media files: {% tool Whisper %}
and {% tool WhisperX %}.
If you want to transcribe a monologue or do not care about who said what, you can choose Whisper. 
To differentiate between various speakers, use WhisperX instead. 
This is what we will do in this tutorial to create our video transcript. 

Please make sure to comply with your local copyright and privacy laws regarding recordings and sensitive data, if applicable. 
You might also need to obtain informed consent before using this tool on some files. 
Read more, for example, on [the regulations in Germany, here](https://forschungsdaten.info/themen/rechte-und-pflichten/datenschutzrecht/).

> <hands-on-title> Create the Video Transcription </hands-on-title>
>
> 1. {% tool [Speech to Text with Diarization](toolshed.g2.bx.psu.edu/repos/bgruening/whisperx/whisperx/3.4.2+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Select audio or video file"*: `output` (Input dataset)
>    - *"Speech to Text Model"*: `Medium (~2x faster than the large model)`
>    - *"Language"*: `English`
>    - *"Output Format"*: `Text`
>
>    > <comment-title> Choose a suitable model </comment-title>
>    >
>    > There are several models to choose from. This [overview](https://whisper-api.com/blog/models/) can help you decide. The bigger the model, the higher the accuracy, but also the greater the computational time for transcription and its carbon footprint.
>    > Despite the tool's option to auto-detect the language, we suggest selecting a language if you can. This saves computing resources.
>    > The `Medium` model selected here showed the best results for this recording. The large model was too sensitive and returned many errors, while the smaller ones were not accurate enough. Files with different qualities and in other languages may differ.
>    > For those, you can check Eamonn Bell's [workflow on comparing Whisper Models](https://usegalaxy.eu/u/eamonnbell/w/compare-two-whisper-models-on-a-set-of-audio-files) or build your own for WhisperX based on this.
>    {: .comment}
>
{: .hands_on}


Now, take a look at the finished transcript as soon as the job turns green.

> <question-title></question-title>
>
> 1. How many different speakers does your transcript show?
>
> > <solution-title></solution-title>
> >
> > 1. There are 4 different speakers, named `SPEAKER_00` through `SPEAKER_03`.
> >
> {: .solution}
>
{: .question}

Your transcript differs slightly because Whisper and WhisperX are not deterministic, meaning their outputs are not standardised. 
You can run it several times and will get a slightly different output each time.
Another thing to be aware of is small errors that can occur in speaker diarization.
The longer a person speaks on the recording, the easier it is for WhisperX to allocate a speaker. 
If someone speaks only a single sentence in the whole recording, it might mean it is not recognised as a different speaker. 
As a result, some passages can be wrongly allocated, as happened in the last bit of the recording:

![Screenshot of wrongly allocated text in the WhisperX example](images/SC_wrong_allocation.png "Wrong speaker allocation in the example transcript.")

The above screenshot shows an extract at the end of the transcript. 
You can also watch that part of the recording in the media player starting at 14:35. 
It is the moment when correspondents and reporters introduce themselves in one line each.
The video clearly shows four different journalists speaking. WhisperX, however, names them all SPEAKER_03 and fails to distinguish between them at this point. 
As we are more interested in the presidential candidates than the journalists, this poses no issue to us. But keep this in mind for your own transcripts.

It is cumbersome to keep track of who is who in this version of the transcript. 
We will make it more obvious by renaming the speakers with their given names with the help of Regular Expressions (RegEx).

# Allocate the Speakers by using Regular Expressions

We will start by allocating the moderator. 
In this example transcript, the moderator was recorded as `SPEAKER_03`. 
This might differ slightly from your text. 
If you want to make sure, you can either listen to the video and check the transcript, or you can search for who speaks the first line in the transcript and use this alias if it differs from `SPEAKER_03`, and replace it with `Moderator`.

## Allocating the Moderator

> <hands-on-title> Allocating the Moderator </hands-on-title>
>
> 1. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `output_txt` (output of **Speech to Text with Diarization** {% icon tool %})
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `SPEAKER_03`
>            - *"Replace with"*: `Moderator`
>            - *"Find and Replace text in"*: `entire line`
>
>
>    {% snippet faqs/galaxy/analysis_regular_expressions.md %}
>
> 2. Rename your output file (once it is green) to `Transcribed_Mod` to signal that this text was transcribed and includes the allocated moderator.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

The tool goes through the text line by line, finds passages that match `SPEAKER_03`, and replaces them with `Moderator`. 
When the dataset turns green, we can check whether our renaming worked. This step makes it easier to point out when the moderator spoke.

We will repeat this step three more times to allocate all the speakers mentioned in the transcript. 
Always use the latest text file to ensure we end up with a single text file where all speakers are clearly distinguished.

## Allocating Kennedy

We are continuing in chronological order, so the next speaker after the moderator is Kennedy.
In the example, his lines are tagged with `SPEAKER_00`. 
We use this (or the name used in your text) and replace it with `Kennedy` by redoing this step.

> <hands-on-title> Allocating Kennedy </hands-on-title>
>
> 1. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Transcribed_Mod` (output of **Replace** {% icon tool %})
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `SPEAKER_00`
>            - *"Replace with"*: `Kennedy`
>            - *"Find and Replace text in"*: `entire line`
>
> 2. Rename your output file to `Transcribed_Mod_Ken`.
{: .hands_on}

The text should now also clearly indicate which lines Kennedy spoke. We will continue to allocate Nixon's passages as well.

## Allocating Nixon

Nixon's first line is: `Senator Kennedy.` You can search for it to see if it was also attributed to `SPEAKER_01`, as in this example. 
If not, use the respective name as the `Find pattern` in the tool instead.

> <hands-on-title> Allocating Nixon </hands-on-title>
>
> 1. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Transcribed_Mod_Ken` (output of **Replace** {% icon tool %})
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `SPEAKER_01`
>            - *"Replace with"*: `Nixon`
>            - *"Find and Replace text in"*: `entire line`
>
> 2. Rename your output file to `Transcribed_Mod_Ken_Nix`.
{: .hands_on}

## Allocating Fleming

The only journalist clearly named in this example was Bob Fleming. 
He probably spoke a longer passage that clearly distinguished him from the others.
We also allocate his passage, as before. 
His first line is: `Senator, the Vice President in his campaign has said that you are naive and at times immature.`

> <hands-on-title> Allocating Fleming </hands-on-title>
>
> 1. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Transcribed_Mod_Ken_Nix` (output of **Replace** {% icon tool %})
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `SPEAKER_02`
>            - *"Replace with"*: `Fleming`
>            - *"Find and Replace text in"*: `entire line`
>
> 2. Rename your output file to `Transcribed_all`.
{: .hands_on}

Congratulations, now all speakers are allocated.
When we compare this file with the initial transcript, all speakers are now allocated, making it easier to see who spoke what. 
There should be no more passages saying `SPEAKER_xx` in your text. 
This may be all you need, but you may want to go further and work only with selected text. 
In this example, we want to get a sense of the topics Nixon and Kennedy addressed during their time on screen. 
Therefore, we will first select all passages allocated to them and then clean them.

# Select the Presidential Candidate’s Passages

In this step, we will use the `Transcribed_all` file to select only the lines from Nixon and Kennedy that are most relevant to this example.

## Select Nixon's Passages

First, we search the document for mentions of Richard Nixon, which are now marked as `[Nixon]`. 
Despite the tool's name, `Search`, it actually selects the lines that fit a certain expression, in our case, the speaker allocation for Nixon. 

> <hands-on-title> Search Nixon's passages </hands-on-title>
>
> 1. {% tool [Search in textfiles](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `Transcribed_all` (output of **Replace** {% icon tool %})
>    - *"Regular Expression"*: `\[Nixon\]`
>
> 2. Rename your output file to `Nixon_raw`.
>
>    > <comment-title> Using RegEx here </comment-title>
>    >
>    > As the input is set as another Regular Expression, we need to "escape" the brackets by putting `\` in front of them to make the selection work.
>    {: .comment}
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How many lines were selected for Nixon?
>
> > <solution-title></solution-title>
> >
> > 1. 83
> >
> {: .solution}
>
{: .question}


## Select Kennedy's Passages

We will redo this step to extract the relevant parts of John F. Kennedy's speech.

> <hands-on-title> Search Kennedy's passages </hands-on-title>
>
> 1. {% tool [Search in textfiles](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `Transcribed_all` (output of **Replace** {% icon tool %})
>    - *"Regular Expression"*: `\[Kennedy\]`
>
> 2. Rename your output file to `Kennedy_raw`.
>
>    > <comment-title> Troubleshooting </comment-title>
>    >
>    > Make sure to select `Transcribed_all` as your input. If you select `Nixon_raw` instead, your output will be empty as that file no longer contains any passages spoken by Kennedy.
>    {: .comment}
>
{: .hands_on}


This output might already suffice, depending on what you want to do with the texts. 
If you want to use it further, some text cleaning might be in order.

# Clean the Presidential Candidate’s Passages

If you want to create a Wordcloud, for example, the information on who spoke what might be redundant and falsify your visualisation.
Therefore, we remove this passage for both speakers.

## Clean Nixon's Passages

Since all of the passages in this file are now from Nixon, the allocation saying `[Nixon]: ` at the beginning of each line is redundant.
We remove it in the next step.

> <hands-on-title> Clean Nixon's passages </hands-on-title>
>
> 1. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Nixon_raw` (output of **Search in textfiles** {% icon tool %})
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `[Nixon]: `
>            - *"Find and Replace text in"*: `entire line`
>
> 2. Rename your output file to `Nixon_cleaned`.
>
>    > <comment-title>Troubleshooting</comment-title>
>    >
>    > Make sure to add the space after the colon to clean the text properly. If you forget, each line will start with a space! 
>    {: .comment}
>
>
{: .hands_on}

The result is the transcript of the text Nixon spoke during the debate. You can use it for further in-depth analysis or visualise it.
To get the same for Kennedy, we have to redo the same step on Kennedy's text.


## Clean Kennedy's Passages

> <hands-on-title> Clean Kennedy's passages </hands-on-title>
>
> 1. {% tool [Replace](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Kennedy_raw` (output of **Search in textfiles** {% icon tool %})
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `[Kennedy]: `
>            - *"Find and Replace text in"*: `entire line`
>
> 2. Rename your output file to `Kennedy_cleaned`.
>
>    > <comment-title>Troubleshooting</comment-title>
>    >
>    > Also here, remember to add the space after the colon. It is easy to forget.
>    {: .comment}
>
{: .hands_on}

Congratulation! You should now have two cleaned texts: one with Kennedy's text and one with Nixon's text.
If you want, you can use other tools in Galaxy for further analysis or visualisation.
The {% tool Word cloud %}, for example, can show you what the most frequent words in the respective speeches were.

# Conclusion

With the example of the first televised presidential debate of Richard Nixon and John F. Kennedy in 1960, this tutorial introduced you to automated speech recognition using Galaxy.
We introduced several tools to transcribe your audio and video files into machine-readable text.
Building on this, we explored how using Regular Expressions can help us clean the automated text and extract specific passages, in our case, the lines from Nixon and Kennedy, respectively. 

Which audio and video files do you want to test this on?

tools_json:
["Speech to Text with Diarization", "toolshed.g2.bx.psu.edu/repos/bgruening/whisperx/whisperx/3.4.2+galaxy1", "Replace", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2", "Search in textfiles", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.5+galaxy2"]


topic: digital-humanities
tutorial: text_mining_chinese
title: Text-Mining Differences in Chinese Newspaper Articles
difficulty: intermediate
time_estimation: 1H
requirements_json: ["type: \"internal\""]
tags_json: ["text mining"]
content_hash: 5d0538f45ec1d51d41af882d93d99b56
last_modified: 2026-05-27T09:01:48.036962
zenodo_link: https://doi.org/10.5281/zenodo.14899614
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/digital-humanities/tutorials/text_mining_chinese/tutorial.html

content:
The British Hong Kong Government censored Chinese newspapers before their publication in the colony in the 1930s ({% cite Ng2022 %}).
Replacement characters like × visibly marked those redactions, making them obvious even to those who did not read any Chinese.

![Example of a Chinese article with symbol × marking censored characters]({% link topics/digital-humanities/tutorials/text_mining_chinese/images/Example_x_censored_Chinese_article.svg %} "Example of a Chinese article with symbol × marking censored characters")

The schematic example in Figure 1, adapted from {% cite Schneider2024 %}, shows what such a censored Chinese article looked like. It is read from right to left and top to bottom. The two more prominent lines on the right are the article title, and the following text is the article's main body. It contains the character × several times, indicating various instances where it was censored.

Despite this obvious form of censorship, no research is looking into what Chinese characters the × replaced. My dissertation ({% cite Schneider2024 %}), which informs this workflow, started at this point. Through extensive research, I found several articles censored in the Hong Kong edition of Da gong bao (大公報) as uncensored versions. Those mostly came from Chinese editions printed in mainland China, where different censorship regulations applied. Those articles from China were not censored and openly showed the characters redacted in the Hong Kong versions. An example of a censored article could be {% cite Anon..16.10.1938_5598 %} and of the uncensored version {% cite TKP.16.10.1938_18864 %}.

This tutorial uses text mining to compare two newspaper articles, a censored and an uncensored text to answer what characters were censored in a Chinese newspaper published in Hong Kong in the 1930s.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Upload the texts

The machine-readable versions of the Chinese newspaper articles I originally used in my dissertation come from a proprietary database and can not be shared here. Instead, I generated a dummy article with a similar setup in GPT and manually adapted the censorship symbols based on my research. As a result, those dummy articles differ in style and punctuation, as the articles in my research. Therefore, the input files are two texts in traditional Chinese. The first is censored, containing ×, and the second one is uncensored and does not contain replacement symbols. Both texts slightly differ in their layout, which will be unified later.

> <hands-on-title> Upload the Texts </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://zenodo.org/records/14899614)
>
>
>    ```
>    https://zenodo.org/records/14899614/files/Example_Chinese_newspaper_censored.txt
>    https://zenodo.org/records/14899614/files/Example_Chinese_newspaper_uncensored.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Rename the datasets, if necessary
> 4. Check that the datatype is `txt`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
{: .hands_on}

> <question-title>What do the uploaded texts look like?</question-title>
>
> 1. Name at least one visual difference you notice between the two texts. (No need to understand their content here.)
>
> > <solution-title></solution-title>
> >
> > 1. Visual differences you notice might be:
> > - The texts have different headlines
> > - The uncensored text contains more paragraphs
> > - The censored text contains the symbol × several times
> > - The censored text contains additional symbols at the end
> > - The texts use slightly different punctuation (impressive catch!)
> {: .solution}
>
{: .question}


# Clean both texts

We pre-process and clean both texts to make the comparison easier and more apparent. Regular Expressions (Regex) help unify the text and find and replace certain text parts. Here, the Regex also helps to restructure the text.


> <details-title> More on Regular Expressions (optional) </details-title>
>
> Regular Expressions are a powerful tool to modify text automatically based on word patterns. To read more about Regex's specifics, see its [documentation](https://docs.python.org/3/library/re.html).
> You can check what your input matches in [Regex 101](https://regex101.com). Enter the Regular Expression you want to try in the top field. Insert a sample text below to see what content your Regex catches and to find out how to adapt it.
>
{: .details}

We will use Regular Expressions in a tool called "Replace text" with four different sub-steps. Those will vary if you upload different texts. Apply this step first to the censored and then to the uncensored text to get two cleaned texts.

> <hands-on-title> Cleaning the Text with Regular Expressions </hands-on-title>
>
> 1. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Example_Chinese_newspaper_censored.txt` (censored text you uploaded)
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `\r`
>            - *"Additional sed commands before replacement"*: `:a;N;$!ba;`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `\n`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `\s`
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `(.)`
>            - *"Replace with:"*: `\1\n`
>    - Click *"Run Tool"*.
>    
>    Now repeat those steps with the uncensored text.
>
>    > <comment-title>Explaining the above Regular Expressions</comment-title>
>    > Regular expressions can not only find particular words, as you might be familiar with from regular text editors.
>    > It is more powerful and can find particular patterns, for example, only capitalised words or all numbers.
>    > In this step, we mostly delete unnecessary placeholders.
>    > The first pattern we want to find is `\r`. It catches a specific form of invisible line breaks that would create unwanted gaps in the comparison later.
>    > We delete those by leaving the optional "Replace with" field blank.
>    > The additional sed commands before replacement `:a;N;$!ba;` catch all blank spaces with this tool.
>    > It is necessary only once to ensure that particular end-of-line characters are removed consistently.
>    > Similarly, `\n` marks linebreaks. We also delete those by leaving the optional "Replace with" field blank.
>    > The next expression we search for is `\s`. Those are spaces as you see them between words on your computer. We delete those.
>    > As a result, there are no gaps in our text anymore.
>    > In the last step, we want to choose each character with `(.)` and reformat it. We want to have one character per line. Therefore, we replace all characters with `\1\n`. `\1` means the responding character, and `\n` adds a linebreak after each.
 >    > The result is a clean and reformatted text.
>    {: .comment}
>
>    {% snippet faqs/galaxy/analysis_regular_expressions.md %}
>
{: .hands_on}

Remember to apply those steps to both the censored and the uncensored text. Rename both new texts in a meaningful way. Choose, for example, "Cleaned Censored Text" and "Cleaned Uncensored Text".

> <question-title>Take a look at the texts</question-title>
>
> 1. What do the texts look like now? How have they changed?
>
> > <solution-title></solution-title>
> >
> > 1. Before, the texts had many sentences per line. Now, both texts show one character per line and have more lines. The layout is entirely different.
> >
> {: .solution}
>
{: .question}



# Compare the censored and uncensored text

We can now compare the two cleaned texts. This will visualise the differences between the two texts and mark them by colour. Make sure to upload the cleaned, censored text with the replacement characters like ‘×’ first. As text two, upload the cleaned, uncensored text without the replacement characters.  This version (HTML version) creates an HTML file, which colour codes differences as additions (green) or extractions (red) when comparing the texts.

## Create a _diff_ file for researchers

> <hands-on-title> Comparing the texts using <em>diff</em> tool </hands-on-title>
>
> 1. {% tool [diff](toolshed.g2.bx.psu.edu/repos/bgruening/diff/diff/3.10+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"First input file"*: `Cleaned Censored Text` (output of **Replace Text** {% icon tool %})
>    - {% icon param-file %} *"Second input file"*: `Cleaned Uncensored Text` (output of **Replace Text** {% icon tool %})
>    - *"Choose a report format"*: `Generates an HTML report to visualize the differences`
>    - Click *"Run Tool"*
{: .hands_on}

The result is two files: an HTML file and a raw output as a by-product.

> <question-title> Take a look at the HTML report</question-title>
>
> 1. What can you see in lines 19 / 21?
>
> > <solution-title></solution-title>
> >
> > 1. Line 19  on the left shows a red comma on the left side of the table. It corresponds with line 21 on the right, which contains a colon in green. This means the punctuation in the file differs. The censored version contains a comma, while the uncensored one includes a colon.
> >
> {: .solution}
>
{: .question}

The HTML file could look like this:

![Screenshot of the diff tool comparing the censored and uncensored text]({% link topics/digital-humanities/tutorials/text_mining_chinese/images/Diff_WF_HTML.jpg %} "Example of the HTML file comparing the censored and uncensored text")

It shows what passages differ in the two texts. Red parts indicate deletions, and green areas indicate additions.
If passages are identical in both texts, they remain uncoloured.
This output is very convenient for researchers, as it shows differences quickly. 
If all you want to do is compare the two texts - you have achieved your goal and can stop here. Congratulations! 

If you want to extract the censored passages in the next steps, this HTML output is not helpful for further technical processing with Galaxy. For this, we run this _diff_ tool a second time with slightly changed parameters. The output is the basis for our further analysis steps.


## Create a _diff_ file for further processing
This step runs the text comparison line by line again to create another raw file that the computer can work with.
It is less intuitive to understand at first glance. Again, select the cleaned, censored text containing `×` first and the uncensored text second.
But this time, choose another report format.

> <hands-on-title> Run another <em>diff</em> tool </hands-on-title>
>
> 1. {% tool [diff](toolshed.g2.bx.psu.edu/repos/bgruening/diff/diff/3.10+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"First input file"*: `Cleaned Censored Text` (output of **Replace Text** {% icon tool %})
>    - {% icon param-file %} *"Second input file"*: `Cleaned Uncensored Text` (output of **Replace Text** {% icon tool %})
>    - *"Choose a report format"*: `Text file, side-by-side (-y)`
>    - Click *"Run Tool"*
>
>    > <comment-title> The output format </comment-title>
>    > As you can see above, the output of this file is a text file, compared to the HTML output in the last step.
>    {: .comment}
>
>2. Rename your file `diff raw`.
{: .hands_on}


> <question-title>Look at the diff raw file</question-title>
>
> 1. How does this file differ from the HTML file in the last step?
>
> > <solution-title>Answer</solution-title>
> >
> > 1. This output is a text file. Changes between the two texts are not coloured.
> > Instead, they are marked by additional symbols like '>' or '<'.
> >
> {: .solution}
>
{: .question}

Great, we can use this file for our next steps.

# Select only censored lines
Now we want to extract only specific lines. To determine what content was redacted in the first text, we filter the last steps' raw output for lines containing the censorship symbol ×. 

> <hands-on-title> Select text </hands-on-title>
>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-file %} *"Select lines"*: `diff_raw` (output of **diff** {% icon tool %})
>    - *"that are"*: `Matching`
>    - *"the pattern"*: `×`
>    - Click *"Run Tool"*
>
>2. Rename your file `Selected lines`.
>
>
{: .hands_on}


> <question-title>What output do you get?</question-title>
>
> 1. How many lines contain ×?
>
> > <solution-title></solution-title>
> >
> > 1. 13
> >
> {: .solution}
>
{: .question}

Check the datatype of this file, it should be `txt`.
The tools we use in the next steps need tabular files instead. 
Therefore, we will change the datatype of this file to `tabular` to enable the next steps:

> <hands-on-title>Change Datatype</hands-on-title>
>
> 1. **Set the datatype** of `Selected lines` {% icon galaxy-pencil %} to `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

When you have changed the format successfully, you should see × in Column 1 and the Chinese characters in Column 9.


# Ensure consistent file format

After filtering for censored lines, we add a sub-step to ensure smooth computation. The previous setup could cause an error if the characters filtered in the last step were erased. Then the extracted file would lack the last column, causing an error. This is invisible to the researchers in the file. The compute step covers this potential error and ensures all necessary columns exist.

> <hands-on-title> Compute to ensure all columns exist </hands-on-title>
>
> 1. {% tool [Compute](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `Selected lines` (output of **Select** {% icon tool %})
>    - *"Input has a header line with column names?"*: `No`
>        - In *"Expressions"*:
>                - *"Add expression"*: `c9`
>                - *"Mode of the operation"*: `Replace`
>                    - *"Use new column to replace column number"*: `9`
>    - In *"Error handling"*:
>        - *"If an expression cannot be computed for a row"*: `Produce an empty column value for the row`
>    - Click *"Run Tool"*
>
> 2. Rename the result to `Censored lines`.
>
{: .hands_on}



# Summarise your findings

This step summarises how often each character appeared in the table previously.

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.9+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `Censored lines` (output of **Compute** {% icon tool %})
>    - *"Group by fields"*: `9`
>    - *"Sort input"*: `Yes`
>    - In *"Operation to perform on each group"*:
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `count`
>            - *"On column"*: `column 9`
>   - Click *"Run Tool"*
>
> 2. Rename the output file `Quantified Results`.
>
{: .hands_on}


> <question-title></question-title>
>
> 1. How many lines does the file have now?
>
> > <solution-title></solution-title>
> >
> > 1. Three lines showing different characters: 寇, 敵 and 日.
> {: .solution}
>
{: .question}


# Sort your findings

Particularly if you get a long list in the last step, sorting the results from the most to least frequent characters is necessary.
If you are only interested in the quantitative results, this can be your final output.

> <hands-on-title> Sort </hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `Quantified Results` (output of **Datamash** {% icon tool %})
>    - *"Sort on column"*: `Column 2`
>    - *"in"*: `Descending order`
>    - *"using sort flavor"*: `Fast numeric sort`
>    - Click *"Run Tool"*
>
> 2. Rename the output file `Sorted Results`. 
>
>    > <comment-title> How to sort? </comment-title>
>    >
>    > Select column `c2` because it contains the character frequency.
>    {: .comment}
>
{: .hands_on}

> <question-title> Check your results </question-title>
>
> 1. How often did the most frequently censored character appear?
>
> > <solution-title></solution-title>
> >
> > 1. The character 敵, meaning enemy, was censored 10 times in the first text.
> >
> {: .solution}
>
{: .question}

Why would the British Hong Kong Government consistently censor this character? Jump to the conclusion to find out!

# Cut out the censored characters only

If you want to visualise your results, this step gets you there. We use the file `Censored lines` that we created in the "Ensure consistent file format" step.
Unlike the last step, this tabular file contains all 13 lines in 9 columns.
From this file, we cut out all uncensored characters. The result is a single column with multiple rows of Chinese characters.
It allows scaling words by frequency in the word cloud in the next step. As a result, characters that appear more often appear bigger, making the results evident at first sight.

> <hands-on-title> Select the censored characters </hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c9`
>    - {% icon param-file %} *"From"*: `Censored lines` (output of **Compute** {% icon tool %})
>    - Click *"Run Tool"*
>
> 2. Rename the output file `Uncensored characters`.
>
>    > <details-title> What do I select here? (optional) </details-title>
>    >
>    > `c9` means column 9. It contains the uncensored characters from text two and is, therefore, cut out in this step.
>    {: .details}
>
{: .hands_on}

# Generate a word cloud

The last step is to visualise the results within a word cloud. It shows which characters were censored in the first text. The bigger the word, the more often it appeared in the text.

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [Generate a word cloud](toolshed.g2.bx.psu.edu/repos/bgruening/wordcloud/wordcloud/1.9.6+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `Uncensored characters` (output of **Cut** {% icon tool %})
>    - *"Do you want to select a special font?": `Select from a list of fonts`: `Noto Sans Traditional Chinese`
>    - *"Smallest font size to show"*: `8`
>    - *"Color option"*: `Colormap`
>    - *"Ratio of times to try horizontal fitting as opposed to vertical"*: `1.0`
>    - *"Scaling of words by frequency (0 - 1)"*: `0.9`
>    - Click *"Run Tool"*
>
> 2. Rename the output file `Wordcloud of uncensored characters`. 
>
>    > <details-title> Optimise your word cloud (optional) </details-title>
>    >
>    > You can choose different colours to suit your needs. The higher the "Ratio of times to try horizontal fitting as opposed to vertical" is towards "1", the more likely the character or word will appear horizontally.
>    > "Scaling of words by frequency (0 - 1)" allows you to scale the words according to their frequency. The smaller this number, the more equal-sized the characters in your word cloud will be, no matter their number.
>    {: .details}
>
{: .hands_on}

Your word cloud should look similar to this:

![Screenshot of the above Workflow in Galaxy]({% link topics/digital-humanities/tutorials/text_mining_chinese/images/Wordcloud_censored_characters.png %})


# Conclusion

This tutorial used text mining to extract censored characters from a Chinese newspaper.

![Screenshot of the above Workflow in Galaxy]({% link topics/digital-humanities/tutorials/text_mining_chinese/images/Workflow_Screenshot.jpg %} "Screenshot of the above workflow showing each of the steps explained above")

The uploaded dummy texts differed in several ways. They used slightly different punctuation, and some sentences and characters differed. The most obvious difference is that the second text was published uncensored in China, whereas the original, published in Hong Kong, contained censorship symbols. This allowed us to extract what characters were censored in the text from Hong Kong.

Within this workflow, we first unified the layout of both texts, showing one character per line for an easier comparison with the *diff* tool. The tool marked the differences between the two texts in colour. Afterwards, we extracted only lines censored with ×. The extraction of the results ran in two strands: One was counting and sorting the results. This will answer what characters the British Hong Kong Government censored in Chinese newspapers in the 1930s: Based on the (simplified) dummy texts, the characters were 敵 (enemy), 寇 (brave) and 日 (Japan). The character for *enemy* dominates and was censored five times more often than the character for *brave*.

What do those findings tell us? The British Hong Kong Government avoided publishing newspapers with a strong stand against Japan. Why? The British colony of Hong Kong, with a large Chinese population, is located very close to the Chinese mainland. Especially after the Japanese army invaded China in the summer of 1937, the British had to walk a tightrope. They tried to support the Chinese efforts without offending Japan. As a British outpost, Hong Kong had little military power and would not withstand a Japanese attack for long. Therefore, the British tried to appease the Japanese Government and avoid an attack. Calling them brave or enemies openly would have been dangerous. The one redaction of 日 (Japan) is very uncommon. This shows that the censorship practices were adaptable and not always unified. Censoring Hong Kong's newspapers to avoid anti-Japanese content is therefore a practical example of how the British Government's appeasement policies were implemented locally. This newspaper comparison is consistent with the findings in archival sources that I also researched for my dissertation ({% cite Schneider2024 %}) and lays the censored characters open for the first time.

tools_json:
["Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy3", "diff", "toolshed.g2.bx.psu.edu/repos/bgruening/diff/diff/3.10+galaxy1", "Select lines that match an expression", "Grep1", "Compute", "toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1+galaxy0", "Datamash", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.9+galaxy0", "Sort", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy3", "Cut", "Cut1", "Generate a word cloud", "toolshed.g2.bx.psu.edu/repos/bgruening/wordcloud/wordcloud/1.9.6+galaxy0"]


topic: digital-humanities
tutorial: introduction_to_dh
title: Introduction to Digital Humanities in Galaxy
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["text mining"]
content_hash: fcb608ccfa87c8194e0e88603f1272f0
last_modified: 2026-05-27T09:01:48.030664
zenodo_link: https://doi.org/10.5281/zenodo.17053220
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/digital-humanities/tutorials/introduction_to_dh/tutorial.html

content:
Loosely building on {% cite Richardson2003 %}, this tutorial compares two editions of the poem "The Sorrows of Yamba".[^1]
The first couple of steps derive from [A short introduction to Galaxy]({% link topics/introduction/tutorials/galaxy-intro-short/tutorial.html %}).

"The Sorrows of Yamba" was published in 1795 and was among the most popular antislavery poems. However, the version published by Hannah More in the Cheap Repository Tracts series was not the only version of the poem that circulated. Also, Moore's authorship on the topic is contested ({% cite Richardson2003 %}). But while we leave this debate to the experts, the different versions of the poem offer a great opportunity to delve into how digital tools can help us compare texts more quickly. We will do this in the following tutorial.

While Richardson compared the poems by hand, we use his example to introduce how Galaxy can help you with your text analysis.
This tutorial covers the basics of Galaxy, from logging in and uploading text to using the first tools.
We will clean the two poem versions and check the texts from a distance by comparing their number of lines and characters, and visualizing both in a word cloud.
Then, we take a closer look. For an easier comparison, we reformat both texts and compare them line by line and side by side. As the word cloud shows, "death" is a dominant theme in the first poem, so we extract all lines including "death" for further in-depth analysis. This helps us gain a better understanding of where those articles differ and is applicable to many other texts you may want to compare.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get started in Galaxy

## Create an account on Galaxy
To use Galaxy's full potential, you must register and create an account. You can skip this step if you already have a Galaxy account.

{% snippet faqs/galaxy/account_create.md %}

Alternatively, you can log in using a single sign-on of your choice, for example, from [IAM4NFDI](https://iam.services.base4nfdi.de/faq_ENG/) on [Galaxy Europe](https://usegalaxy.eu/).

 ![Screenshot of Galaxy Europe register window with the IAM4NFDI login button highlighted](../../images/iam4nfdi.png)

## Log in to Galaxy

> <hands-on-title>Log in to Galaxy</hands-on-title>
> 1. Open your favourite browser (Chrome, Safari, Edge or Firefox as your browser, not Internet Explorer!)
> 2. Browse to your Galaxy instance, for example [Galaxy Europe](https://usegalaxy.eu/)
> 3. Log in with your credentials
>
> ![Screenshot of Galaxy Europe with the register or login button highlighted](../../images/Galaxyeulogin.png)
>
>   > <comment-title>Different Galaxy servers</comment-title>
>   >  This is an image of Galaxy Europe, located at [usegalaxy.eu](https://usegalaxy.eu)
>   >
>   > The particular Galaxy server you are using may look slightly different and have a different web address.
>   >
>   > You can also find more possible Galaxy servers at the top of this tutorial in **Available on these Galaxies**
>   {: .comment}
{: .hands_on}

The Galaxy homepage is divided into four sections (panels):
* The Activity Bar on the left: _This is where you will navigate to the resources in Galaxy (Tools {% icon tool %}, Workflows {% icon galaxy-workflows-activity %}, Histories {% icon galaxy-history-storage-choice %}, etc.)_
* Currently active "Activity Panel" on the left: _By default, the {% icon tool %} **Tools** activity will be active and its panel will be expanded_
* Viewing panel in the middle: _The main area for context for your analysis_
* History of analysis and files on the right: _Shows your "current" history; i.e.: Where any new files for your analysis will be stored_

![Screenshot of the Galaxy interface with aforementioned structure]({% link topics/introduction/images/galaxy_interface.png %})

The first time you use Galaxy, your history panel is empty.

## Name your current history

Your "History" is on the panel on the right. It is a record of the actions you have taken.

> <hands-on-title>Name history</hands-on-title>
> 1. Go to the **History** panel (on the right)
> 2. Click {% icon galaxy-pencil %} (**Edit**) next to the history name (which by default is "Unnamed history")
>
>    ![Screenshot of the galaxy interface with the history name being edited, it currently reads "Unnamed history", the default value. An input box is below it.]({% link shared/images/rename_history.png %}){:width="250px"}
>
>    > <comment-title></comment-title>
>    >
>    > In some previous versions of Galaxy, you will need to click the history name to rename it as shown here:
>    > ![Screenshot of the galaxy interface with the history name being edited, it currently reads "Unnamed history", the default value.](../../../../shared/images/rename_history_old.png){:width="320px"}
>    {: .comment}
>
> 3. Type in a new name, for example, "My Analysis"
> 4. Click **Save**
>
> > <comment-title>Renaming not an option?</comment-title>
> > If renaming does not work, you may not be logged in, so try logging in to Galaxy first. Anonymous users can only have one history, and they cannot rename it.
> {: .comment}
>
{: .hands_on}

## Upload a file to Galaxy

The "Activity Bar" is located on the leftmost part of the interface.

> <hands-on-title>Upload a file</hands-on-title>
> 1. At the top of the **Activity Bar**, click the {% icon galaxy-upload %} **Upload** activity
>
>    ![upload data button shown in the galaxy interface]({% link topics/introduction/images/upload-data.png %})
>
>    This brings up a box:
>
>    ![the Galaxy upload dialogue, the 'regular' tab is active with a large textarea to paste subsequent URL]({% link topics/introduction/images/upload-box.png %})
>
> 3. Click **Paste/Fetch data**
> 4. Paste in the address of both files in the Zenodo folder:
>    ```
>    https://zenodo.org/records/17053220/files/SoY_Cheap_Repo_Source.txt
>    https://zenodo.org/records/17053220/files/SoY_Univ_Mag_Source.txt
>    ```
>
> 5. Click **Start**
> 6. Click **Close**
>
>
> **Option 2:** On [usegalaxy.eu](https://usegalaxy.eu), you can alternatively import the Zenodo files directly from a *data library* within Galaxy:
>
> 1. At the top of the **Activity Bar**, click the {% icon galaxy-upload %} **Upload** activity
> 2. Click on the bottom of the newly opened window on **Choose from repository**.
> 3. Enter **"Zenodo"** in the search bar and click on the folder **"Zenodo"**.
> 4. Enter **Training material for Galaxy tutorial "Introduction to Digital Humanities in Galaxy"** in the search bar and select the items.
> 7. Click **Select**
> 8. Click **Start**
> 9. Click **Close**
{: .hands_on}


Your uploaded file is now in your current history.
When the file has been uploaded to Galaxy, it will turn green.

> <comment-title></comment-title>
> After this, you will see your first history item (called a "dataset") in Galaxy's right panel. It will go through
> the grey (preparing/queued) and yellow (running) states to become green (success).
>
{: .comment}

The contents of the file will be displayed in the central Galaxy panel. If the dataset is large, you will see a warning message which explains that only the first megabyte is shown.

> <hands-on-title>View the text files content</hands-on-title>
> 1. Click the {% icon galaxy-eye %} (eye) icon next to the dataset name, to look at the file content
>
>    ![galaxy history view showing a single dataset mutant_r1.fastq. Display link is being hovered.](../../images/eye-icon.png){:width="520px"}
>
> 2. Check the datatype - is it **txt**? Then you are all set. Otherwise, adapt the datatype.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 3. **Add a tag** to each dataset, corresponding to the file's origin.
>    - One saying `#cheap` for the file from the cheap repository (SoY_Cheap_Repo_Source.txt)
>    - The other one `#universal` for the second one (SoY_Univ_Mag_Source.txt)
>    - Don't forget the hashtags
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}


What are those files?

You can see two text files; they are two versions of the poem "The Sorrows of Yamba".
The file "SoY_Cheap_Repo_Source.txt" is a poem version of Sorrows of Yamba, which was published in the [Cheap Repository](https://archive.org/details/McGillLibrary-PN970_C52_no_42a-1732).
The file "SoY_Univ_Mag_Source.txt" is another version of the poem, first published in the Universal Magazine in 1797.

Both files start with "Text adapted from:" and a different hyperlink. The second paragraph for both texts begins with "the sorrows of yamba," but the files continue differently. While one gives the year, the other is immediately followed by more text.
Both texts are already pre-cleaned and are entirely in lower case, but still contain punctuation.

It is obvious that the texts have similarities, but they are not identical. Now comes the fun part: Using Galaxy to compare your files. To do that, we first need to clean both files.

# Clean your Texts

## Delete the hyperlink


When looking at the two datasets, you will notice they still contain the hyperlink from their source.
As this is metadata and not the text we want to compare, we delete it at the beginning of both files.


> <hands-on-title> Delete Hyperlink in Text One </hands-on-title>
>
> 1. Click on **Tools** {% icon tool %} in the left panel
> 2. Search for {% tool [Remove beginning](Remove beginning1) %} and pass the following parameters:
>    - *"Remove first"*: `1` (lines)
>    - {% icon param-file %} *"from"*: `1: SoY_Cheap_Repo_Source.txt`
> 3. Click on **Run Tool** {% icon workflow-run %}
>
>    > <comment-title> What does this tool do? </comment-title>
>    >
>    > Remove beginning deletes a selected number of lines from your file. In this case, removing the first line is enough.
>    {: .comment}
>
{: .hands_on}

When the job is finished and appears green in your history, click on its name.

> <question-title></question-title>
>
> 1. Check how many lines the file now contains?
> 2. How does this differ from the original file you uploaded?
>
> > <solution-title></solution-title>
> >
> > 1. The file now contains only one line.
> > 2. The originally uploaded text contained two lines. You removed one with this step.
> >
> {: .solution}
>
{: .question}

As a result, only the poem's text remains, while the source was removed for text one.
Galaxy names the files after the task used to create that step. While this can be helpful, we change the name to a clearer filename.

> <hands-on-title> Rename the output</hands-on-title>
>
> 1. **Change the name** {% icon galaxy-pencil %} of the output of this tool, which removed the beginning of `SoY_Cheap_Repo_Source.txt`
>    - Rename it to `SoY_Cheap_Repo.txt`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}


We also use this tool on the second file.


> <hands-on-title> Delete Hyperlink in Text Two </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Remove beginning](Remove beginning1) %} with the following parameters:
>    - *"Remove first"*: `1` (lines)
>    - {% icon param-file %} *"from"*: `2: SoY_Univ_Mag_Source.txt`
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
> 
> 2. Once it is finished, rename this file to `SoY_Univ_Mag.txt`.
>
{: .hands_on}



Click on the finished dataset that just appeared in your history. Check that it starts with the poem text and that the hyperlink is removed.
To quickly see which version of the poems we have, we rename both datasets with clearer names and add tags based on the text origin.
The hashtag propagates the tags, so all further outputs from this dataset contain the same hashtag, making it much easier to identify what text we are currently working with.


Depending on how detailed you want to compare your texts, we suggest further unifying them. In the next step, we remove all the punctuation with one command.


## Remove punctuation

Regular Expressions (RegEx) allow you to search for particular patterns in your text. They can be a massive help if you want to extract or remove them with minimal work. In our two poems, the punctuation is not unified; therefore, we want to remove it from both using regular expressions (RegEx). If comparing the punctuation of texts is also relevant to you, you can skip this step. Make sure to select the text version from the Cheap Repository that we have earlier removed the hyperlink from.

> <hands-on-title> Remove Punctuation in Poem One </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Replace text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `SoY_Cheap_Repo.txt`
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `[[:punct:]]`
>            - *"Replace with"*: (leave this empty)
>
>    {% snippet faqs/galaxy/analysis_regular_expressions.md %}
>
> 2. Rename your output file (once it is green) to `SoY_Cheap_Repo_cleaned.txt`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}



And we repeat the same for the second text. Remember to use the redo button if you want to save some time.

Also in text two, we search for the pattern `[[:punct:]]` and omit a replacement, meaning that all punctuation marks will be deleted. Ensure that you select the text version from the Universal Magazine, from which we previously removed the hyperlink.

> <hands-on-title> Remove Punctuation in Poem Two </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Replace text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `SoY_Univ_Mag.txt`
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `[[:punct:]]`
>            - *"Replace with"*: (leave this empty)
>
> 2. **Rename** {% icon galaxy-pencil %} the output file to `SoY_Univ_Mag_cleaned.txt`
>
>
{: .hands_on}

To get an idea of how the two cleaned texts compare, we check out their metadata.

# Different ways to compare the texts

There are various ways to compare two texts. 
You could, for example, compare them quantitatively with the tool {% tool Line/Word/Character count %}. 
The output of the tool would show, that the texts have different lengths. 
While both files are only two lines long, the poem from the cheap repository is 1139 words long, more than double the amount of the second poem. 

Another option for comparing the texts is by visualising their content. 
We will use a word cloud next to do just that. 


## Compare visually

A picture says more than 1000 words! Accordingly, we aim to delve deeper into the actual content of both texts. 
Particularly for larger corpora, a word cloud can be a nice way to get a first idea of what a text is about. 

> <hands-on-title> Visualize the Content of Poem One </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Generate a word cloud](toolshed.g2.bx.psu.edu/repos/bgruening/wordcloud/wordcloud/1.9.6+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `SoY_Cheap_Repo_cleaned.txt` (output of **Replace Text** {% icon tool %})
>    - *"Do you want to select a special font?"*: `Use the default DroidSansMono font`
>    - *"Color option"*: `Colormap`
>    - *"Scaling of words by frequency (0 - 1)"*: `0.8`
>
> 2. **Rename** {% icon galaxy-pencil %} the output file to `Wordcloud Poem One`.
>
>    > <comment-title> Adapting the Word Cloud </comment-title>
>    >
>    > The word cloud has many different features. You can upload a stop word list that should be excluded from the visualization, or play around with other parameters like the text size. Rerun {%icon dataset-rerun %} the tool with some changed parameters and see what happens.
>    {: .comment}
>
{: .hands_on}

The word cloud for the second text is created in the same way. 
We suggest using the rerun {%icon dataset-rerun %} button on `Wordcloud Poem One`, to reun the tool with `SoY_Univ_Mag_cleaned.txt` as an input and create a wordcloud for it. That way we can make sure to use identical input parameters (e.g. font, color option, etc.) which helps for easy comparison.
This should look as follows:

> <hands-on-title> Visualize the Content of Poem Two </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Generate a word cloud](toolshed.g2.bx.psu.edu/repos/bgruening/wordcloud/wordcloud/1.9.6+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `SoY_Univ_Mag_cleaned.txt` (output of **Replace Text** {% icon tool %})
>    - *"Do you want to select a special font?"*: `Use the default DroidSansMono font`
>    - *"Color option"*: `Colormap`
>    - *"Scaling of words by frequency (0 - 1)"*: `0.8`
>
> 2. **Rename** {% icon galaxy-pencil %} the output file to `Wordcloud Poem Two`.

>    > <comment-title> Uniqueness of the Word Cloud </comment-title>
>    >
>    > The word cloud from this tool looks a little different each time you run it. The layout may vary even when you redo it with the exact same text and inputs.
>    {: .comment}
>
{: .hands_on}

Comparing items from your history is easiest when you enable the window manager and view both images side by side.

{% snippet faqs/galaxy/features_scratchbook.md %}


> <question-title></question-title>
>
> 1. What is the most prominent word in each of the clouds?
> 2. How do the Word Clouds for Poem One and Poem Two compare?
>
> > <solution-title></solution-title>
> >
> > 1. The most prominent word in the word cloud created from the cheap repository is "yamba", while the one from the universal text is "death".
> > 2. The word cloud from the cheap repository has four words that appear most prominent and are much bigger - and therefore more frequent in the text. They are "yamba", "now", "death" and "ye". The most prominent words in the universal text are "death", "yamba" and "africs". They appear a bit smaller than the words from the cheap repository, suggesting a lower frequency.
> >
> > ![Word Cloud of Cheap Repository Version](../../images/wc_cheap.png "Word Cloud of Cheap Repository Version")
> > ![Word Cloud of Universal Text Version](../../images/wc_universal.png "Word Cloud of Universal Text Version")
> {: .solution}
>
{: .question}

You can disable the window manager again by clicking on the item, then you will see your datasets again in your middle panel, once you click on its eye {% icon galaxy-eye %} symbol.

The visualisation suggests that the text's metrics, which we checked with the line and character count, and their messages differ. The cheap repository text addresses the reader with multiple mentions of "ye", you, which is rare in the second poem. In the universal poem, death is more central than yamba, which is the other way around in the cheap repository text.

With this text's length and just two poems, this is, of course, something you can find out by reading both texts. However, this distant reading approach can give you important preliminary insights to guide your close reading, particularly with bigger corpora.

Of course, the word cloud insights are just a first glance and do not allow a proper analysis; 
we need to compare both texts properly. 

## Compare texts side by side

But what is a good way to compare those texts in depth? 
We suggest comparing them side by side and line by line. 
For that, we adapt the layout once more.

### Replace spaces with line breaks to prepare side-by-side comparison

We used {% tool Replace text in entire line %} to remove the punctuation earlier. 
This time, we use the same tool to replace characters in our files. 
To get a convenient layout that shows one word per line, we use Regular Expressions to replace the spaces (\s) with line breaks (\n). 
That way, each word gets displayed in a different line, which prepares the detailed comparison in the next step.

> <hands-on-title> Changing Layout of Poem One </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Replace text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `SoY_Cheap_Repo_cleaned.txt`
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `\s`
>            - *"Replace with:"*: `\n`
>
>
>    > <comment-title> How do I understand the RegEx commands? </comment-title>
>    >
>    > Don't worry, if you have never used Regular Expressions. Several websites help you find out what patterns to detect and how to catch the passages you need. For now, you can just add the symbols that stand for the space `\s` and the line break `\n`.
>    {: .comment}
>
> 2. **Rename** {% icon galaxy-pencil %} this text `SoY_Cheap_Repo_word_per_line.txt`.
>
{: .hands_on}

When you click on the eye {% icon galaxy-eye %} icon of the data set in the history, when the dataset turns green, you can see that it now contains one word per line. 
To match this, we rerun {%icon dataset-rerun %} the step with the same parameters also for the second poem.

> <hands-on-title> Changing Layout of Poem Two </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Replace text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `SoY_Univ_Mag_cleaned.txt`
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>            - *"Find pattern"*: `\s`
>            - *"Replace with:"*: `\n`
>
> 2. **Rename** {% icon galaxy-pencil %} this text `SoY_Univ_Mag_word_per_line.txt`.
>
{: .hands_on}



> <question-title></question-title>
>
> 1. How many lines long are the poems now?
>
> > <solution-title>  </solution-title>
> >
> > 1. When you click on the two names of the two new datasets you just worked on, you see that one is now 539, the other 1139 lines long. The number of lines matches the amount of words you might have detected with the tool `Line/Word/Character count`.
> >
> {: .solution}
>
{: .question}

Now, both poems show one word per line, which is the perfect setup to compare them side by side. 
Use a tool called `diff` to visualise this. 
To reproduce the setup from this tutorial, make sure to select the version from the Cheap Repository as the first input file and the one from the Universal Magazine as the second input file.

### Using *diff* to compare texts side by side

> <hands-on-title> Compare the Poems </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [diff](toolshed.g2.bx.psu.edu/repos/bgruening/diff/diff/3.10+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"First input file"*: `SoY_Cheap_Repo_word_per_line.txt`
>    - {% icon param-file %} *"Second input file"*: `SoY_Univ_Mag_word_per_line.txt`
>    - *"Choose a report format"*: `Generates an HTML report to visualize the differences`
>    - *"Choose report output format"*: `Side by side`
>
>    > <comment-title> Different Report Formats </comment-title>
>    >
>    > The `diff` tool allows you to create different outputs, depending on your goal. In this case, the HTML report contains colours to highlight the changes between both texts, making it really useful for researchers to quickly identify. If you want to extract information automatically, the option `text file, side by side` could also be helpful.
>    {: .comment}
>
{: .hands_on}

We get two new files as a result. The HTML report and the raw output it is based on, in txt format.

> <question-title></question-title>
>
> 1. What is the first difference between the two texts visualized in the HTML report?
>
> > <solution-title></solution-title>
> >
> > 1. Lines 6-40 of the cheap poem are marked in green. They are not part of the universal poem. The couple of lines before and after are identical.
> >
> {: .solution}
>
{: .question}

In the HTML report, you can quickly identify deletions (in red) and additions (in green) between both texts.
You can also see smaller details, which you might quickly miss manually. 
Lines 63/64 and 28/29, respectively, show that also changes within one word (prisoner / prisner) are detected. 
You can furthermore see how the perspective was changed between the poems. 
While line 359-361 in the cheap repository text states "they sell us", the other text states "they sell them" (l. 298-300), suggesting the reader is (no longer) among the group which is sold. 
You can go through the HTML file and detect further changes in language and length.

Seeing this, you might want to go into detail with the respective themes once more. As "death" was central in both word clouds, we will extract sentences containing this word so you can analyze them further. 
The recent files without punctuation and one word per line are not the easiest form for this. 
Instead, we use an earlier version from our history.

# Extract specific sentences

## Breaking text into sentences

We return to Regular Expressions a third time, but this time we use a different tool with further functionalities. 
The tools helps us to divide the text into more lines, to make it easier to extract sentences containing the word "death".

Full stops  help us to indicate a sentence, which will not be perfectly accurate but sufficient for this case. 
Of course, you could spend more time on this and make it neater. 
Make sure to use not the last input but the poems without a hyperlink, but including punctuation. 
This will not work if the text contains no more full stops.

> <hands-on-title> Rearrange Poem One </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Replace parts of text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `SoY_Cheap_Repo.txt`
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `\.`
>            - *"Replace with"*: `\.\n`
>            - *"Find-Pattern is a regular expression"*: `Yes`
>            - *"Replace all occurences of the pattern"*: `Yes`
>            - *"Find and Replace text in"*: `entire line`
>
>    > <comment-title> What do those inputs mean? </comment-title>
>    >
>    > A full stop `.` has its own meaning in Regular Expressions. It stands for all elements. To show that we do not mean all characters but actually a full stop, we need to escape it in RegEx by putting `\.` instead of `.` if we mean a full stop. We want to add a line break afterwards, which we already learned is indicated as `\n`. The replacement pattern, therefore, is `\.\n`. As a result, the text will now be split in multiple lines.
>    {: .comment}
>
> 2. **Rename** {% icon galaxy-pencil %} your resulting file to `SoY_Cheap_Repo_sent_per_line.txt`.
>
{: .hands_on}



when you have finished this step, remember to rerun {%icon dataset-rerun %} this tool for the second poem 

> <hands-on-title> Rearrange Poem Two </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Replace parts of text ](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `SoY_Univ_Mag.txt`
>    - In *"Find and Replace"*:
>        - {% icon param-repeat %} *"Insert Find and Replace"*
>            - *"Find pattern"*: `\.`
>            - *"Replace with"*: `\.\n`
>            - *"Find-Pattern is a regular expression"*: `Yes`
>            - *"Replace all occurences of the pattern"*: `Yes`
>            - *"Find and Replace text in"*: `entire line`
>
> 2. **Rename** {% icon galaxy-pencil %} your resulting file to `SoY_Univ_Mag_sent_per_line.txt` for easier distinction.
>
{: .hands_on}

As a result, you get two files, each split at full stops. How can you now extract the sentences that are relevant to you?


## Extract sentences containing 'death'

Use {% tool Search in textfiles %} to select all lines containing the word "death".

> <hands-on-title> Extract particular sentences </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Search in textfiles](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `SoY_Cheap_Repo_sent_per_line.txt`
>    - *"Regular Expression"*: `death`
>
>    > <comment-title> Further Functionalities </comment-title>
>    >
>    > You can see that the tool has many parameters you can tweak.  The ones not mentioned here are kept at the default input, like `Match` and `Perl,` which is the kind of RegEx applied. But you could also select all lines that do not contain "death" by selecting `Do not match` or extracting lines before or after the line containing the content you chose.
>    {: .comment}
>
> 2. **Rename** {% icon galaxy-pencil %} your output `SoY_Cheap_Repo_death.txt`
>
{: .hands_on}



And for the last time, we rerun {%icon dataset-rerun %} this step for the second poem.

> <hands-on-title> Extract particular sentences from Poem Two </hands-on-title>
>
> 1. Run {% icon workflow-run %} {% tool [Search in textfiles](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.5+galaxy2) %} with the following parameters:
>    - {% icon param-file %} *"Select lines from"*: `SoY_Univ_Mag_sent_per_line.txt`
>    - *"Regular Expression"*: `death`
>
> 2. **Rename** {% icon galaxy-pencil %} your output `SoY_Univ_Mag_death.txt`.
>
{: .hands_on}


When you enable the window manager at the top bar, you can click on the eye {% icon galaxy-eye %} symbols of your last two outputs and visualize them side by side in two different windows. Six and seven lines from the poems contain the term, respectively. You could analyze them in detail now to see where they differ. While the first lines are nearly identical, the last ones are completely different in both versions of the poem. An intriguing insight for further analysis. No wonder the poems and their many editions have sparked the interest of many researchers.

If you only analyze those two poems, you might find it easier to do those steps manually. But particularly, if you create a workflow out of this, you can reproduce this process with only a few clicks, saving you considerable work.

Learn how to [extract a workflow from the above analysis]({% link topics/galaxy-interface/tutorials/history-to-workflow/tutorial.html %}).

Alternatively, you can make your analysis more complex and extract further differences between the poems automatically to adapt the above analysis. For inspiration, check out
the [advanced tutorial on Text-Mining]({% link topics/digital-humanities/tutorials/text_mining_chinese/tutorial.html %}).

# Conclusion

Congratulations! You just finished your first analysis with Galaxy, well done! The tutorial covered the basic setup of Galaxy and how you can register, log in and upload your material. You are now familiar with terms in Galaxy, like history, dataset, tool, etc. We used several tools, learned to rerun them and how we can see the outputs in different ways. We used various versions of Regular Expressions to rearrange and clean your text. We also reshaped the text to compare it with the `diff` tool. In the end, we extracted notable sentences for further close reading.
The workflow created from this history would look as follows:

 ![Screenshot of Workflow extracted from the Tutorial Introduction to DH](../../images/WF_Intro_to_DH.png)

With all this knowledge in mind, you can now continue with one of our other tutorials or experiment with your own input. Enjoy!


[^1]: Thanks to Lilli Fortmeier for suggesting this use case!

tools_json:
["Remove beginning", "Remove beginning1", "Replace text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/9.5+galaxy3", "Generate a word cloud", "toolshed.g2.bx.psu.edu/repos/bgruening/wordcloud/wordcloud/1.9.6+galaxy0", "diff", "toolshed.g2.bx.psu.edu/repos/bgruening/diff/diff/3.10+galaxy1", "Replace parts of text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_find_and_replace/9.5+galaxy2", "Search in textfiles", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_grep_tool/9.5+galaxy2"]


