Preface

In the field of biotechnology, fastp, a software led and developed by Dr. Shifu Chen, founder and CTO of HaploX, has set a paradigm for technology translation. Since its inception, this ultra-fast FASTQ data preprocessing tool has become a phenomenal tool in the field of bioinformatics, owing to its innovative algorithmic architecture and continuous iterative capability. According to Google Scholar, the fastp research paper has garnered over 28,000 citations, firmly ranking at the top of academic influence in China's biotechnology sector over the past decade.

Software Features

01

Fastp is a multithreaded multifunctional preprocessor for FASTQ streams. It accepts single-end or paired-end FASTQ data as inputs and outputs the processed data along with the QC metric reports. Figure 1 shows how fastp processes paired-end FASTQ data.

Figure 1. Paired-end data processing workflow of fastp

The workflow can be simply divided into a decompressor, a preprocessor, and a compressor. The input-paired FASTQ files are decompressed individually to read packs, and each pack consists of fixed read records. Each worker thread picks the odd or even read packs one by one, processes the reads, makes some statistics, and outputs the clean data to the compressor in the same order.

Two worker threads are used for demonstration, but usually, much more worker threads (usually 3-16) are used to make preprocessing faster. The classical producer/consumer thread model is applied, and specifically, the input and output read packs are stored in a single-producer-single-consumer (SPSC) list for thread-safe communications. This SPSC list is implemented without any thread locks to support high-performance interthread communication. As shown in Figure 2, for a certain read pack, it is fixed that in which worker thread the read pack will be processed. This feature keeps the output and input data in the same order, making the output completely reproducible, which means the resulting output files will be identical if the command is run twice. Most features shown in Figure 2 were introduced in the first publication on fastp. Some features, such as paired-end merging and deduplication, have been recently introduced. Applying paired-end merging is relatively simple after the overlapping analysis is complete.

02

Removing redundant reads is a necessary step in NGS data analysis pipelines. Previous deduplication tools usually require reads to be first aligned to a reference genome, which renders them inefficient and unsuitable for certain applications that do not involve sequence alignment. The new fastp implements a fast, accurate, and memory-efficient FASTQ-level deduplication. Figure 2 briefly illustrates the method by which fastp removes duplicate reads.

Figure 2. How fastp determines whether a read is unique or duplicated

As shown in Figure 3, many bloom filter arrays (e.g., three) are used, and each has L bits. A hash function is defined accordingly for each array. A hash function maps a read sequence into an integer number p in [0, L); therefore, a read R will be mapped to p1, p2, and p3. If Array1[p1], Array2[p2], and Array3[p3] are all positive, then R is marked as duplicated; otherwise, Array1[p1], Array2[p2], and Array3[p3] are set to be positive. For paired-end reads, the read pairs are combined first and then treated as same as single-end reads.