Solutions

2. Knowing where to find a pipeline and which one to use.

Exercises

2.1. How many pipelines are currently available in nf-core? How many are under development, released, and archived?
- As of 6/04/2021: 50 pipelines are available, of which 18 under development, 28 relased and 4 archived.

2.2 Find the pipeline doing ATAC-seq data analysis in nf-core: link

  • What is the current/latest version of the pipeline?

    • 1.2.1 (6/04/2021)

  • How many versions are available to download?

    • 5 versions (as of 6/04/2021): current (1.2.1), 1.2.0, 1.1.0, 1.0.0, dev.

  • How many and which paramater(s) is(are) required to run the pipeline?

    • Only 1 required parameter: --input (Path to comma-separated file containing information about the samples in the experiment)

  • What is the default output directory’s name?

    • ./results (parameter --outdir)

  • What happens if you do not specify a profile (-profile)? If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH. More information is available here.

2.3 In the nextflow-io awesome pipelines, look for the featured BABS-aDNASeq workflow: link.

  • What tool is used for calling variants?

    • samtools mpileup

  • What version of Nextflow is it advised to use?

    • version 0.30.2 (Note that the current version is 20.10.0 (6/04/2021))

  • How do you download the BABS-aDNASeq pipeline locally?

    • git clone https://github.com/crickbabs/BABS-aDNASeq

3. Import a pipeline

Exercises

3.1 Try to pull the pipeline from your command line.
- nextflow pull nextflow-io/rnaseq-nf

3.2 The latest version of the pipeline is implemented using DSL2, the new syntax extension enables Nextflow to use modules. Imagine that you would like to run the last DSL1 version of the pipeline (v1.2), which command could you use to pull this specific version?
- nextflow pull nextflow-io/rnaseq-nf -r v1.2

3.3 Nextflow enables to pull any specific tag, release or commit. Now try to use the same option to pull the pipeline from (1) a given branch and at a (2) specific git commit.
- nextflow pull nextflow-io/rnaseq-nf -r master
- nextflow run nextflow-io/rnaseq-nf -r 98ffd10a76

4. Execute a pipeline

Exercises

4.1 Run the rnaseq-nf pipeline.

  • nextflow run nextflow-io/rnaseq-nf

Exercises

4.2 Run the pipeline with with Docker containers. Find a parameter that you need to add when running the pipeline on the command line by using nextflow run -h

  • nextflow run nextflow-io/rnaseq-nf -with-docker

4.3 With the reproducibility aspect in mind, at a certain point we want to be sure that we are running a specific version of the pipeline. Use the command that you created in the previous exercise and extend it so it runs a specific released version of the pipeline (v1.2).

  • nextflow run nextflow-io/rnaseq-nf -with-docker -r v1.2

4.4 (advanced) Besides Docker containers, it is also possible to run the pipeline with conda or singularity. Edit the command, so it uses the appropriate environment manager.

  • Conda: the state-of-the-art way of running with conda is by using the -with-conda parameter. However, if the conda environment file (here conda.yml) is not present in the directory where you are currently running the pipeline from, it will give an error. Therefore, the solution would be to include the path to the conda.yml file: nextflow run nextflow-io/rnaseq-nf -with-conda path/to/conda.yml.

  • Singularity: (if Singularity installed): nextflow run nextflow-io/rnaseq-nf -with-singularity

5. Locate and describe the output after running a pipeline.

Exercises

5.1 Have a look in the folder from where you ran the pipeline. Which directories and files have been created?

  • .nextflow.log, results/ and work/

5.2 All the output generated by the pipeline will be stored in the work/ directory. This folder contains a bunch of subfolders that store the (intermediate) output of the pipeline. Based on the output of running the pipeline, can you find out in which folder the results of the multiqc step is being stored?

  • Solution: ./work/3d/17a04c… the actual folder name is much longer, but the short version is given here. These names are automatically generated based on a hash.

5.3 Go to the directory that contains the outputs of a given step e.g. RNASEQ:QUANT.

  • Find out which files are present in that folder. What does the -> symbol mean?

    • .command.* (e.g. .command.log that contains the output of running the tool, .command.sh that contains the command being run on the command-line, …), the -> symbol referst to the fact that input files are being linked from another folder. The output files are present in the folder ggal_gut/.

  • Find for that process which command was given to the computer.

    • .command.sh

  • Advanced: how can you change the behaviour of storing the files in the output directory? (hint have a look in the documentation and search for a parameter publishDir)?

    • Advanced: each module defines the process that is actually being run. Underneath that definition, the publishDir tells us something about how and where the output is being stored, more specifically the mode argument allows us to copy the results from the work/... directory to the outdir which is in our case results/.

6. Run a pipeline with different parameters.

  1. Pipeline parameters:

Exercises

6.1 Try to modify the name of the folder where results are dumped by using a different parameter on the command-line.

  • nextflow run nextflow-io/rnaseq-nf --outdir 'myAwesomeResults' -with-docker

6.2 What other parameters can you modify in the rnaseq-nf pipeline?

  • The reads, transcriptome, outdir and multiqc parameters.

  1. Nextflow parameters

Exercises

6.3 Nextflow has a built-in parameter that allows us to run the pipeline in the background (hence no need of nohup or screen) and one that allows us to resume a pipeline when it failed at a given process. These are very helpful and time-saving features. Which parameters do we need to add on the command-line to request such behaviour?

  • -resume and -bg

7. Configuration files

Exercises

7.1 Run with profile (standard & -with-docker) or (standard,docker) or (standard,conda)

  • nextflow run nextflow-io/rnaseq-nf -profiles standard -with-docker. Actually, by default the local executor will be chosen and it is hence not necessary to select the standard profile.

  • nextflow run nextflow-io/rnaseq-nf -profiles standard,docker

  • nextflow run nextflow-io/rnaseq-nf -profiles standard,conda

7.2 (advanced) Knowing the above, change the parameters of the standard profile so it takes different reads. How does the pipeline react, which reads are being used as inputs?

  • To change anything in the configuration file, the nextflow.config file needs to be in the same directory from where we are running the pipeline. For this, you can use the following command: nextflow clone <pipeline-name> to download the pipeline locally. Then, open an editor and change the nextflow.config file so it contains the following:

standard {
    process.container = 'quay.io/nextflow/rnaseq-nf:latest'
    params.reads = "$baseDir/data/ggal/ggal_liver_{1,2}.fq"
 }