Nextflow¶
Nextflow is the most widely used workflow manager in bioinformatics.
This guide shows how to register a Nextflow run with inputs & outputs for the example of the nf-core/scrnaseq pipeline by running a Python script.
The approach could be automated by deploying the script via
a serverless environment trigger (e.g., AWS Lambda)
a post-run script on the Seqera Platform
What steps are executed by the nf-core/scrnaseq pipeline?
!lamin init --storage ./test-nextflow --name test-nextflow
Show code cell output
→ connected lamindb: testuser1/test-nextflow
Run the pipeline¶
Let’s download the input data from an S3 bucket.
import lamindb as ln
input_path = ln.UPath("s3://lamindb-test/scrnaseq_input")
input_path.download_to("scrnaseq_input")
→ connected lamindb: testuser1/test-nextflow
And run the nf-core/scrnaseq pipeline.
# the test profile uses all downloaded input files as an input
!nextflow run nf-core/scrnaseq -r 2.7.1 -profile docker,test -resume --outdir scrnaseq_output
Show code cell output
N E X T F L O W ~ version 24.04.4
Pulling nf-core/scrnaseq ...
downloaded from https://github.com/nf-core/scrnaseq.git
WARN: It appears you have never run this project before -- Option `-resume` is ignored
Launching `https://github.com/nf-core/scrnaseq` [pensive_bartik] DSL2 - revision: 4171377f40 [2.7.1]
Downloading plugin nf-validation@1.1.4
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/scrnaseq v2.7.1-g4171377
------------------------------------------------------
Core Nextflow options
revision : 2.7.1
runName : pensive_bartik
containerEngine : docker
launchDir : /home/runner/work/nextflow-lamin/nextflow-lamin/docs
workDir : /home/runner/work/nextflow-lamin/nextflow-lamin/docs/work
projectDir : /home/runner/.nextflow/assets/nf-core/scrnaseq
userName : runner
profile : docker,test
configFiles :
Input/output options
input : https://github.com/nf-core/test-datasets/raw/scrnaseq/samplesheet-2-0.csv
outdir : scrnaseq_output
Mandatory arguments
aligner : star
protocol : 10XV2
Skip Tools
skip_emptydrops : true
Reference genome options
fasta : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/GRCm38.p6.genome.chr19.fa
gtf : https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/scrnaseq for your analysis please cite:
* The pipeline
https://doi.org/10.5281/zenodo.3568187
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/scrnaseq/blob/master/CITATIONS.md
------------------------------------------------------
[03/0dc4ba] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_X)
[67/59bacc] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:FASTQC_CHECK:FASTQC (Sample_Y)
[af/7068d5] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:GTF_GENE_FILTER (GRCm38.p6.genome.chr19.fa)
[77/b96d71] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_GENOMEGENERATE (GRCm38.p6.genome.chr19.fa)
[98/bc0c48] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_ALIGN (Sample_X)
[31/6b668e] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:STARSOLO:STAR_ALIGN (Sample_Y)
[60/459949] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)
[48/e554dd] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_X)
[46/289b09] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_X)
[63/45b351] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_X)
[14/d316f6] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_Y)
[50/38a960] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_SEURAT (Sample_Y)
[10/f4d1c6] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_Y)
[d8/1e0fb9] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:MTX_TO_H5AD (Sample_Y)
[21/ffb58c] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MULTIQC
[fc/1dcd66] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:CONCAT_H5AD (1)
[8f/726714] Submitted process > NFCORE_SCRNASEQ:SCRNASEQ:MTX_CONVERSION:CONCAT_H5AD (2)
-[nf-core/scrnaseq] Pipeline completed successfully-
What is the full run command for the test profile?
nextflow run nf-core/scrnaseq -r 2.7.1 \
-profile docker \
-resume \
--outdir scrnaseq_output \
--input 'scrnaseq_input/samplesheet-2-0.csv' \
--skip_emptydrops \
--fasta 'https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/GRCm38.p6.genome.chr19.fa' \
--gtf 'https://github.com/nf-core/test-datasets/raw/scrnaseq/reference/gencode.vM19.annotation.chr19.gtf' \
--aligner 'star' \
--protocol '10XV2' \
--max_cpus 2 \
--max_memory '6.GB' \
--max_time '6.h'
Run the registration script¶
After the pipeline has completed, a Python script registers inputs & outputs in LaminDB.
nf-core/scrnaseq run registration¶
import argparse
import lamindb as ln
import json
import re
from pathlib import Path
def parse_arguments() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("--input", type=str, required=True)
parser.add_argument("--output", type=str, required=True)
return parser.parse_args()
def register_pipeline_io(input_dir: str, output_dir: str, run: ln.Run) -> None:
"""Register input and output artifacts for an `nf-core/scrnaseq` run."""
input_artifacts = ln.Artifact.from_dir(input_dir, run=False)
ln.save(input_artifacts)
run.input_artifacts.set(input_artifacts)
ln.Artifact(f"{output_dir}/multiqc", description="multiqc report", run=run).save()
ln.Artifact(
f"{output_dir}/star/mtx_conversions/combined_filtered_matrix.h5ad",
description="filtered count matrix",
run=run,
).save()
def register_pipeline_metadata(output_dir: str, run: ln.Run) -> None:
"""Register nf-core run metadata stored in the 'pipeline_info' folder."""
ulabel = ln.ULabel(name="nextflow").save()
run.transform.ulabels.add(ulabel)
# nextflow run id
content = next(Path(f"{output_dir}/pipeline_info").glob("execution_report_*.html")).read_text()
match = re.search(r"run id \[([^\]]+)\]", content)
nextflow_id = match.group(1) if match else ""
run.reference = nextflow_id
run.reference_type = "nextflow_id"
# execution report and software versions
for file_pattern, description, run_attr in [
("execution_report*", "execution report", "report"),
("nf_core_pipeline_software*", "software versions", "environment"),
]:
artifact = ln.Artifact(
next(Path(f"{output_dir}/pipeline_info").glob(file_pattern)),
description=f"nextflow run {description} of {nextflow_id}",
visibility=0,
run=False,
).save()
setattr(run, run_attr, artifact)
# nextflow run parameters
params_path = next(Path(f"{output_dir}/pipeline_info").glob("params*"))
with params_path.open() as params_file:
params = json.load(params_file)
ln.Param(name="params", dtype="dict").save()
run.params.add_values({"params": params})
run.save()
args = parse_arguments()
scrnaseq_transform = ln.Transform(
name="scrna-seq",
version="2.7.1",
type="pipeline",
reference="https://github.com/nf-core/scrnaseq",
).save()
run = ln.Run(transform=scrnaseq_transform).save()
register_pipeline_io(args.input, args.output, run)
register_pipeline_metadata(args.output, run)
!python register_scrnaseq_run.py --input scrnaseq_input --output scrnaseq_output
Show code cell output
→ connected lamindb: testuser1/test-nextflow
! this creates one artifact per file in the directory - consider ln.Artifact(dir_path) to get one artifact for the entire directory
! folder is outside existing storage location, will copy files from scrnaseq_input to /home/runner/work/nextflow-lamin/nextflow-lamin/docs/test-nextflow/scrnaseq_input
Data lineage¶
The output data could now be accessed (in a different notebook/script) for analysis with full lineage.
matrix_af = ln.Artifact.get(description__icontains="filtered count matrix")
matrix_af.view_lineage()
View transforms & runs on the hub¶
View the database content¶
ln.view()
Show code cell output
Artifact
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_by_id | updated_at | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 5 | YCeKhRcowS6CYKHP0000 | None | True | filtered count matrix | None | .h5ad | None | 659819 | jBZHd6wUH87dQsrRihLAWQ | NaN | None | md5 | AnnData | 1 | True | 1 | 1.0 | 1.0 | 1 | 2024-10-03 07:55:31.070759+00:00 |
| 4 | S7cYZssGvVCZ3z3O0000 | None | True | multiqc report | None | None | 9676756 | Kp1R_IYSRmq2yNRsEcvtVw | 59.0 | None | md5-d | None | 1 | True | 1 | 1.0 | 1.0 | 1 | 2024-10-03 07:55:31.050658+00:00 | |
| 3 | 8PsW4PEMuk8v4v1O0000 | None | True | None | scrnaseq_input/samplesheet.csv | .csv | None | 236 | QXMVrT5ZucmidIxbYJ9KHA | NaN | None | md5 | None | 1 | True | 1 | NaN | NaN | 1 | 2024-10-03 07:55:31.006629+00:00 |
| 2 | CaUQ5CX452tSEzd00000 | None | True | None | scrnaseq_input/S10_L001_R1_001.fastq.gz | .fastq.gz | None | 1727503 | UrpdRtwcAhl3QV7xfzI29w | NaN | None | md5 | None | 1 | True | 1 | NaN | NaN | 1 | 2024-10-03 07:55:31.006052+00:00 |
| 1 | ZgtqjHJ59cYqX9KT0000 | None | True | None | scrnaseq_input/S10_L001_R2_001.fastq.gz | .fastq.gz | None | 4259756 | W-dGV6rDQMWXfGSAv_xL0g | NaN | None | md5 | None | 1 | True | 1 | NaN | NaN | 1 | 2024-10-03 07:55:31.005189+00:00 |
Run
| uid | started_at | finished_at | is_consecutive | reference | reference_type | transform_id | report_id | environment_id | parent_id | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||
| 1 | ZhrTEiUyrj7DHmO2Vxou | 2024-10-03 07:55:30.976320+00:00 | None | None | pensive_bartik | nextflow_id | 1 | 6 | 7 | None | 1 |
Storage
| uid | root | description | type | region | instance_uid | run_id | created_by_id | updated_at | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| 1 | nPzHGbT9BAte | /home/runner/work/nextflow-lamin/nextflow-lami... | None | local | None | 7JUvfoPu6nFp | None | 1 | 2024-10-03 07:48:08.917584+00:00 |
Transform
| uid | version | is_latest | name | key | description | type | source_code | hash | reference | reference_type | _source_code_artifact_id | created_by_id | updated_at | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 1 | IG5ARBQ5Fmj60000 | 2.7.1 | True | scrna-seq | None | None | pipeline | None | None | https://github.com/nf-core/scrnaseq | None | None | 1 | 2024-10-03 07:55:30.972957+00:00 |
ULabel
| uid | name | description | reference | reference_type | run_id | created_by_id | updated_at | |
|---|---|---|---|---|---|---|---|---|
| id | ||||||||
| 1 | WTwrm2QQ | nextflow | None | None | None | None | 1 | 2024-10-03 07:55:31.076699+00:00 |
User
| uid | handle | name | updated_at | |
|---|---|---|---|---|
| id | ||||
| 1 | DzTjkKse | testuser1 | Test User1 | 2024-10-03 07:48:08.914167+00:00 |
Show code cell content
# clean up the test instance:
!rm -rf test-nextflow
!lamin delete --force test-nextflow
• deleting instance testuser1/test-nextflow