Nextflow Pipelines
Introduction
This page explains how to run the OmicsBox Engine pipelines with Nextflow: how to start a pipeline, configure it, resume an interrupted run and find the results. For the list of pipelines and their requirements, see Pipelines.
In the commands below, replace <workflow> with the identifier of a pipeline, for example functional_annotation or gene_level_analysis. The identifiers are listed in the repository wiki.
Run a pipeline
You can run a pipeline from a local copy of the repository, or straight from GitHub.
From a local copy
Use this option when you want to read or modify the pipeline code.
git clone https://github.com/BioBam/omicsbox-engine-workflows.git
cd omicsbox-engine-workflows
-
Export the configuration template. This writes
<workflow>.configto the current folder and exits, without starting any analysis:nextflow run nextflow/workflows/<workflow>/<workflow>.nf --dump_config -
Edit
<workflow>.config: set the input files and the output folder, and adjust the options you need. See Configure a pipeline. -
Run the pipeline with your configuration:
nextflow run nextflow/workflows/<workflow>/<workflow>.nf \ -c <workflow>.config \ -resume
Straight from GitHub
Nextflow downloads the pipeline for you, so the configuration file is the only file you keep.
-
Export the configuration template:
nextflow run BioBam/omicsbox-engine-workflows \ -r main \ -main-script nextflow/workflows/<workflow>/<workflow>.nf \ --dump_config -
Edit
<workflow>.config, as above. -
Run the pipeline:
nextflow run BioBam/omicsbox-engine-workflows \ -r main \ -main-script nextflow/workflows/<workflow>/<workflow>.nf \ -c <workflow>.config \ -resume
-r main selects the published version, and -main-script selects the pipeline within the repository.
Configure a pipeline
The template has 2 blocks:
paramsholds the pipeline settings: the input files, the output folder (local_folder,./resultsby default) and the chart format. Some steps have their own group of settings, such as an optional file that one step accepts.processhas one entry per step. Each entry lists the options passed to that step'somicsboxcommand, one per line. Comment out a line to go back to that option's default, or uncomment one to enable it. The comment on each line gives the accepted values, the default and any dependency.
Options depend on each other
Some options are only read when another option is enabled, and some modes disable the options of other modes. A file set in the params block may also need a matching option switched on in the step. Before changing an option, read its comment and change the related lines with it.
Always edit the full list of options of a step. If you replace it with a shorter list, the options you leave out are dropped for that step.
Override settings on the command line
Any pipeline setting can also be passed on the command line, which is useful for running the same configuration on a different dataset:
nextflow run nextflow/workflows/gene_level_analysis/gene_level_analysis.nf \
-c gene_level_analysis.config \
--input_paired_end='data/*_{1,2}.fastq.gz' \
--local_folder=/scratch/run_042/results
- Pipeline settings take 2 dashes (
--input_fasta). Nextflow's own options take 1 (-c,-resume). - Put quotes around file patterns such as
'data/*_{1,2}.fastq.gz'. Without them, the shell expands the pattern before Nextflow sees it. - Settings inside a step group use a dot, for example
--trimmomatic.adapters=/path/to/adapters.fa. - A value on the command line overrides the same value in your configuration file.
Resume a run
Add -resume whenever you run a pipeline again, for example after fixing an input path or a failed step. Steps whose inputs and settings have not changed are reused instead of being run again, so you do not wait or pay twice for the same cloud analysis.
Nextflow keeps the information it needs to resume in the folder you launch the pipeline from. Resuming therefore only works when you launch it again from the same folder.
Use one folder per pipeline
Launch each pipeline, and ideally each dataset, from its own folder. -resume continues the last run started in the current folder, whichever pipeline it was. Mixing pipelines in one folder makes them re-run every step and can write their results into the same output folder.
To resume an older run instead of the last one, list the runs with nextflow log and pass the session ID to -resume <session-id>.
Results
Results are saved in <local_folder>/<session-id>/, with one numbered subfolder per step:
results/
└── <session-id>/
├── 01_<step_name>/
├── 02_<step_name>/
└── …
A resumed run keeps its session ID, so it keeps filling the same results folder.
While the pipeline runs, Nextflow also keeps a work/ folder with each step's files and logs, including the exact omicsbox command that was run. The results are hard links to those files, not copies, so they take no extra disk space. You can delete work/ after the run without losing the results, but you then lose the ability to resume it.
More information
The repository wiki has a page for each pipeline, with its steps and outputs, plus a detailed execution and configuration guide and a guide to modifying the pipelines.