igdiscover

1.

Install Conda by following the conda installation instructions as appropriate for your system. You will need to choose between a “Miniconda” and “Anaconda” installation. We recommend Miniconda as the download is smaller. If you are in a hurry, these two commands are usually sufficient to install Miniconda on Linux (read the linked document for macOS instructions):

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

When the installer asks you about modifying the PATH in your .bashrc file, answer yes.

2.

Close the terminal window and open a new one. Then test whether conda is installed correctly by running

conda --version

If you see the conda version number, it worked.

3.

Set up Conda so that it can access the bioconda channel. For that, follow the instructions on the bioconda website or simply run these commands:

conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge

4.

Install IgDiscover with this command:

conda create -n igdiscover igdiscover

This will create a new so-called “environment” for IgDiscover (retry if it fails). Whenever you want to run IgDiscover, you will need to activate the environment with this command:

source activate igdiscover

5.

Make sure you have activated the igdiscover environment. Then test whether IgDiscover is correctly installed with this command:

igdiscover --version

If you see the version number of IgDiscover, it worked! If an error message appears that says "The 'networkx' distribution was not found and is required by snakemake", install networkx manually with:

pip install networkx==2.1

Then retry to check the igdiscover version.

6.

You can now run IgDiscover on the test data set to familiarize yourself with how it works.

1.

Install Python 3.5 or newer. It most likely is already installed on your system, but in Debian/Ubuntu, you can get it with

sudo apt-get install python3

2.

Create the directory where binaries will be installed. We assume $HOME/.local/bin here, but this can be anywhere as long as they are in your $PATH.

mkdir -p ~/.local/bin

Add this line to the end of your ~/.bashrc file:

export PATH=$HOME/.local/bin:$PATH

Then either start a new shell or run source ~/.bashrc to get the changes.

3.

Install MUSCLE. This is available as a package in Ubuntu:

sudo apt-get install muscle

If your distribution does not have a 'muscle' package or if you are not allowed to run sudo:

wget -O - http://www.drive5.com/muscle/downloads3.8.31/muscle3.8.31_i86linux64.tar.gz | tar xz
mv muscle3.8.31_i86linux64 ~/.local/bin/

4.

Install PEAR:

wget http://sco.h-its.org/exelixis/web/software/pear/files/pear-0.9.6-bin-64.tar.gz
tar xvf pear-0.9.6-bin-64.tar.gz
mv pear-0.9.6-bin-64/pear-0.9.6-bin-64 ~/.local/bin/pear

5.

Install IgBLAST:

wget ftp://ftp.ncbi.nih.gov/blast/executables/igblast/release/1.4.0/ncbi-igblast-1.4.0-x64-linux.tar.gz
tar xvf ncbi-igblast-1.4.0-x64-linux.tar.gz
mv ncbi-igblast-1.4.0/bin/igblast? ~/.local/bin/

IgBLAST requires some data files that must be downloaded separately. The following commands put the files into ~/.local/igdata:

mkdir ~/.local/igdata
cd ~/.local/igdata
wget -r -nH --cut-dirs=4 ftp://ftp.ncbi.nih.gov/blast/executables/igblast/release/internal_data
wget -r -nH --cut-dirs=4 ftp://ftp.ncbi.nih.gov/blast/executables/igblast/release/database/
wget -r -nH --cut-dirs=4 ftp://ftp.ncbi.nih.gov/blast/executables/igblast/release/optional_file/

Also, you must set the $IGDATA environment variable to point to the directory with data files. Add this line to your ~/.bashrc:

export IGDATA=$HOME/.local/igdata

Then run source ~/.bashrc to get the changes.

7.

Optionally, install flash:

wget -O FLASH-1.2.11.tar.gz http://sourceforge.net/projects/flashpage/files/FLASH-1.2.11.tar.gz/download
tar xf FLASH-1.2.11.tar.gz
cd FLASH-1.2.11
make
mv flash ~/.local/bin/

1.

A FASTA or FASTQ file with single-end reads or two FASTQ files with paired-end reads (also, the files must be gzip-compressed)

2.

A database of V/D/J genes (three FASTA files named V.fasta, D.fasta, J.fasta)

3.

A configuration file that describes the library

1.

Create and initialize the analysis directory.

First, pick a name for your analysis. We will use myexperiment in the following. Then run igdiscover init:

igdiscover init myexperiment

A dialog will appear and ask for the file with the first (forward) reads. Find your compressed FASTQ file that contains them and select it. Typical file names may be Library1_S1_L001_R1_001.fastq.gz or mylibrary.1.fastq.gz. You do not need to choose the second read file! It is found automatically.

Next, choose the directory with your database. The directory must contain the three files V.fasta, D.fasta, J.fasta. These files contain the V, D, J gene sequences, respectively. Even if have have only light chains in your data, a D.fasta file needs to be provided, just use one with the heavy chain D gene sequences.

If you do not want a graphical user interface, use the two command-line parameters --db and --reads1 to provide this information instead:

igdiscover init --db path/to/my/database/ --reads1 mylibrary.1.fastq.gz myexperiment

Again, the second reads file will be found automatically. Use --single-reads instead of --reads1 if you have single-end reads or a dataset with already merged reads. For --single-reads, a FASTA file (not only FASTQ) is also allowed. In any case, an analysis directory named myexperiment will have been created.

2.

Adjust the configuration file

The previous step created a configuration file named myexperiment/igdiscover.yaml, which you may need to adjust. In particular, the number of discovery rounds is set to 3 by default, which takes a long time. Reducing this to 2 or even 1 often works just as well.

3.

Run the analysis

Change into the newly created analysis directory and run the analysis:

igdiscover run

Depending on the size of your library, your computer, and the number of iterations, this will now take from a few hours to a day. See the running IgDiscover section for more fine-grained control over what to run and how to resume the process if something failed.

• Paired-end reads in gzipped FASTQ format,
• Single-end reads in gzipped FASTQ format,
• Single-end reads in gzipped FASTA format.

• The forward primer sequence. This is optional.
• A random barcode (molecular identifier). This is optional. Set the configuration option barcode_length_5p to 0 if you don’t have random barcodes or if you don’t want the program to use them.
• Optionally, a run of G nucleotides. This is an artifact of the RACE protocol (Rapid amplification of cDNA ends). If you have this, set race_g to true in the configuration file.
• 5' UTR
• Leader
• Re-arranged V, D and J gene sequences for heavy chains; only V and J for light chains
• An optional random barcode. Set the configuration option barcode_length_3p to the length of this barcode. You can currently not have both a 5' and a 3' barcode.
• The reverse primer. This is optional.

1.

Lines starting with the # symbol are comments (they are ignored)

2.

A configuration option that is meant to be switched on or off will say something like stranded: false if it is off. Change this to stranded: true to switch the option on (and vice versa).

3.

The primer sequences are given as a list, and must be written in a certain way - one sequence per line, and a - (dash) in front, like so:

forward_primers:
- ACGTACGTACGT
- AACCGGTTAACC

Even if you have only one primer sequence, you still need to use this syntax.

• Ctrl+C was pressed on the keyboard
• A full harddisk
• If running on a cluster, the program may have been terminated because it exceeded its allocated time
• Too little RAM
• Power loss

1.

Change the configuration setting.

2.

Delete the file that needs to be re-generated. Assume it is filename

3.

Run igdiscover run filename to re-create the file. Only that file will be created, not the ones that usually would be created afterwards.

4.

Optionally, run igdiscover run (without a file name this time) to update the remaining files (those that depend on the file that was just updated).

igdiscover.yaml

The configuration file. Make sure to adjust this to your needs as described above.

reads.1.fastq.gz, reads.2.fastq.gz

Symbolic links to the raw paired-end reads.

database/

The input V/D/J database (as three FASTA files). The files are a copy of the ones you selected when running igdiscover init.

reads/

Processed reads (merged, de-duplicated etc.)

iteration-xx/

Iteration-specific analysis directory, where “xx” is a number starting from 01. Each iteration is run in one of these directories. The first iteration (in iteration-01) uses the original input database, which is also found in the database/ directory. The database is updated and then used as input for the next iteration.

final/

After the last iteration, IgBLAST is run again on the input sequences, but using the final database (the one created in the very last iteration). This directory contains all the results, such as plots of the repertoire profiles. If you set the number of iterations to 0 in the configuration file, this directory is the only one that is created.

final/database/(V,D,J).fasta

These three files represent the final, individualized V/D/J database found by IgDiscover. The D and J files are copies of the original starting database; they are not updated by IgDiscover.

final/dendrogram_(V,D,J).pdf

These three PDF files contain dendrograms of the V, D and J sequences in the individualized database.

final/assigned.tab.gz

V/D/J gene assignments and other information for each sequence. The file is created by parsing the IgBLAST output in the igblast.txt.gz file. This is a table that contains one row for each input sequence. See below for a detailed description of the columns.

final/filtered.tab.gz

Filtered V/D/J gene assignments. This is the same as the assigned.tab file mentioned above, but with low-quality assignments filtered out. Run igdiscover filter --help to see the filtering criteria.

final/expressed_(V,D,J).tab, final/expressed_(V,D,J).pdf

The V, D and J gene expression counts. Some assignments are filtered out to reduce artifacts. In particular, an allele-ratio filter of 10% is applied. For D genes, only those with an E-value of at most 1E-4 and a coverage of at least 70% are counted. See also the help for the igdiscover count subcommand, which is used to create these files.

The .tab file contains the counts as a table, while the PDF file contains a plot of the same values.

These tables also exist in the iteration-specific directories (iteration-xx). For those, note that the numbers do not include the genes that were discovered in that iteration. For example, iteration-01/expressed_V.tab shows only expression counts of the V genes in the starting database.

final/errorhistograms.pdf

A PDF with one page per V gene/allele. Each page shows a histogram of the percentage differences for that gene.

final/clusterplots/

This is a directory that contains one PNG file for each discovered gene/allele. Each image shows a clusterplot of all the sequences assigned to that gene. Note that the shown clusterplots are by default restricted to showing only at most 300 sequences, while the actual clustering used by IgDiscover uses 1000 sequences.

iteration-xx/candidates.tab

A table with candidate novel V alleles (or genes). This is a list of sequences found through the windowing strategy or linkage cluster analysis, as discussed in our paper. See the full description of candidates.tab.

iteration-xx/read_names_map.tab

For each candidate novel V allele listed in candidates.tab, this file contains one row that lists which sequences went into generating this candidate. Only the exact matches are listed, that is, the number of listed sequence names should be equal to the value in the exact column. Each line in this file contains tab-separated values. The first is name of the candidate, the others are the names of the sequences. Some of these sequences may be consensus sequences if barcode grouping was enabled, so in that case, this will not be a read name.

iteration-xx/new_V_germline.fasta, iteration-xx/new_V_pregermline.fasta

The discovered list of V genes for this iteration. The file is created from the candidates.tab file by applying either the germline or pre-germline filter. The file resulting from application of the germline filter is used in the last iteration only. The file resulting from application of the pre-germline filter is used in earlier iterations.

iteration-xx/annotated_V_germline.tab, iteration-xx/annotated_V_pregermline.tab

A version of the candidates.tab file that is annotated with extra columns that describe why a candidate was filtered out. See the description of this file.

reads/1-limited.1.fastq.gz, reads/1-limited.1.fastq.gz

Input reads file limited to the first N entries. This is just a symbolic link to the input file if the limit configuration option is not set.

reads/2-merged.fastq.gz

Reads merged with PEAR or FLASH

reads/3-forward-primer-trimmed.fastq.gz

Merged reads with 5' primer sequences removed. (This file is automatically removed when it is not needed anymore.)

reads/4-trimmed.fastq.gz

Merged reads with 5' and 3' primer sequences removed.

reads/5-filtered.fasta

Merged, primer-trimmed sequences converted to FASTA, and too short sequences removed. (This file is automatically removed when it is not needed anymore.)

reads/sequences.fasta.gz

Fully pre-processed sequences. That is, filtered sequences without duplicates (using VSEARCH)

stats/reads.txt

Statistics of pre-processed sequences.

stats/readlengths.txt, stats/readlengths.pdf

Histogram of the lengths of pre-processed sequences (created from reads/sequences.fasta)

count

How many copies of input sequence this query sequence represents. Copied from the ;size=3; entry in the FASTA header field that is added by VSEARCH -derep_fulllength.

V_gene, D_gene, J_gene

V/D/J gene match for the query sequence

stop

whether the sequence contains a stop codon (either “yes” or “no”)

V_covered, D_covered, J_covered

percentage of bases of the reference gene that is covered by the bases of the query sequence

V_evalue, D_evalue, J_evalue

E-value of V/D/J hit

FR1_SHM, CDR1_SHM, FR2_SHM, CDR2_SHM, FR3_SHM, V_SHM, J_SHM

rate of somatic hypermutation (actually, an error rate)

V_errors, J_errors

Absolute number of errors (differences) in the V and J gene match

UTR

Sequence of the 5' UTR (the part before the V gene match up to, but not including, the start codon)

leader

Leader sequence (the part between UTR and the V gene match)

CDR1_nt, CDR1_aa, CDR2_nt, CDR2_aa, CDR3_nt, CDR3_aa

nucleotide and amino acid sequence of CDR1/2/3

V_nt, V_aa

Nucleotide and amino acid sequence of V gene match

V_CDR3_start

Start coordinate of CDR3 within V_nt. Set to zero if no CDR3 was detected. Comparisons involving the V gene ignore those V bases that are part of the CDR3.

V_end, VD_junction, D_region, DJ_junction, J_start

nucleotide sequences for various match regions

name, barcode, race_G, genomic_sequence

see the following explanation

1.

Split the 5' end barcode from the sequence (if barcode length is zero, this will be empty), put it in the barcode column.

2.

Remove the initial run of G bases from the remaining sequence, put that in the race_G column.

3.

The remainder is put into the genomic_sequence column.

4.

If there is a V gene match, take the sequence before it and split it up in the following way. Search for the start codon and write the part before it into the UTR column. Write the part starting with the start column into the leader column.

• The J gene was not assigned
• A stop was codon found
• The V gene coverage is less than 90%
• The J gene coverage is less than 60%
• The V gene E-value is greater than 10-3

name

The name of the candidate gene. See novel gene names.

source

The original database gene to which the sequences from this row were originally assigned. All candidates coming from the same source gene are grouped together.

chain

Chain type: VH for heavy, VK for light chain lambda, VL for light chain kappa

cluster

From which type of cluster or clusters the consensus was computed. If there are multiple clusters that give rise to the same consensus sequence, they are all listed here, separated by semicolon. A cluster name such as 2-4 is for a percentage difference window: Such a cluster consists of all sequences assigned to the source gene that have a percentage difference to it between 2 and 4 percent.

A cluster name such as cl3 describes a cluster generated through linkage cluster analysis. The clusters are simply named cl1, cl2, cl3 etc. If any cluster number seems to be missing (such as when cl1 and cl3 occur, but not cl2), then this means that the cluster led to an ambiguous consensus sequence that has been filtered out. Since the cl clusters are created from a random subsample of the data (in order to keep computation time down), they are never larger than the size of the subsample (currently 1000).

The name db represents a cluster that is identical to the database sequence. If no actual cluster corresponding to the database sequence is found, but the database sequence is expressed, a db cluster is inserted artificially in order to make sure that the sequence is not lost. The cluster name all represents the set of all sequences assigned to the source gene. This means that an unambiguous consensus could be computed from all the sequences. Typically, this happens during later iterations when there are no more novel sequences among the sequences assigned to the database gene.

cluster_size

The number of sequences from which the consensus was computed. Equivalently, the size of the cluster set (all clusters described in this row). Sequences that are in multiple clusters at the same time are counted only once.

Js

The number of unique J genes associated with the sequences in the cluster set.

Consensus sequences are computed only from V gene sequences, but each V gene sequence is part of a full V/D/J sequence. We therefore know for each V sequence which J gene it was found with. This number says how many different J genes were found for all sequences that the consensus in this row was computed from.

CDR3s

The number of unique CDR3 sequences associated with the sequences in the cluster set. See also the description for the Js column. This number says how many different CDR3 sequences were found for all sequences that the consensus in this row was computed from.

exact

The number of exact occurrences of the consensus sequence among all sequences assigned to the source gene, ignoring the 3' junction region.

To clarify: While the consensus sequence is computed only from a subset of sequences assigned to a source gene, all sequences assigned to the source gene are searched for exact occurrences of that consensus sequence.

When comparing sequences, they are first truncated at the 3' end by removing those (typically 8) bases that correspond to the CDR3 region.

barcodes_exact

How many unique barcode sequences were used by the sequences in the set of exact sequences (described above).

Ds_exact

How many unique D genes were used by the sequences in the set of exact sequences (described above). Only those D gene assignments are included in this count for which the number of errors is zero, the E-value is at most a given threshold, and for which the number of covered bases is at least a given percentage.

Js_exact

How many unique J genes were used by the sequences in the set of exact sequences (described above).

CDR3s_exact

How many unique CDR3 sequences were used by the sequences in the set of exact sequences (described above).

clonotypes

The estimated number of clonotypes within the set of exact sequences (which is described above). The value is computed by clustering the unique CDR3 sequences associated with all exact occurrences, allowing up to six differences (mismatches, insertions, deletions) and then counting the number of resulting clusters.

database_diff

The number of differences between the consensus sequence and the sequence of the source gene. (Given as edit distance, that is insertion, deletion, mismatch count as one difference each.)

has_stop

Indicates whether the consensus sequence contains a stop codon.

looks_like_V

Whether the consensus sequence “looks like” a true V gene (1 if yes, 0 if no). Currently, this checks whether the 5' end of the sequence matches a known V gene motif.

CDR3_start

Where the CDR3 starts within the discovered V gene sequence. This uses the most common CDR3 start location among the sequences from which this consensus is derived.

consensus

The consensus sequence itself.

N_bases

Number of N bases in the consensus

too_low_dbdiff

The number of differences between this candidate and the database is lower than the required number.

too_many_N_bases

The candidate contains too many N nucleotide wildcard characters.

too_low_CDR3s_exact

The CDR3s_exact value for this candidate is lower than required.

too_high_CDR3_shared_ratio

The CDR3_shared_ratio is higher than the configured threshold.

too_low_Js_exact

The Js_exact value is lower than the configured threshold.

has_stop

The filter configuration disallows stop codons, but this candidate has one and is not whitelisted.

too_low_cluster_size

The cluster_size of this candidate is lower than the configured threshold, and the candidate is not whitelisted.

is_duplicate

A filtering criterion not listed above applies to this candidate. This covers all the filters that need to compare candidates to each other: cross-mapping ratio, clonotype allele ratio, exact ratio, Ds_exact ratio.

commonv

Find common V genes between two different antibody libraries

upstream

Cluster upstream sequences (UTR and leader) for each gene

dendrogram

Draw a dendrogram of sequences in a FASTA file.

rename

Rename sequences in a target FASTA file using a template FASTA file

union

Compute union of sequences in multiple FASTA files

filter

Filter a table with IgBLAST results

count

Count and plot V, D, J gene usage

group

Group sequences by barcode and V/J assignment and print each group’s consensus (unused in IgDiscover)

germlinefilter

Create new V gene database from V gene candidates using the germline and pre-germline filter criteria.

discover

Discover candidate new V genes within a single antibody library

clusterplot

For each V gene, plot a clustermap of the sequences assigned to it

errorplot

Plot histograms of differences to reference V gene

• Discard sequences with N bases
• Discard sequences that come from a consensus over too few source sequences
• Discard sequences with too few unique CDR3s (CDR3s_exact column)
• Discard sequences with too few unique Js (Js_exact column)
• Discard sequences identical to one of the database sequences (if DB given)
• Discard sequences that do not match a set of known good motifs
• Discard sequences that contain a stop codon (has_stop column)
• Discard near-duplicate sequences
• Discard cross-mapping artifacts
• Discard sequences whose “allele ratio” is too low.

• are not checked for the cluster size criterion,
• do not need to match a set of known good motifs,
• are never considered near-duplicates (but they are checked for cross-mapping and for the allele ratio),
• are allowed to contain a stop codon.

• The two sequences have a distance of 1,
• they are both in the database for that particular iteration (only then can cross-mapping occur)
• the ratio between the expression levels of the two sequences (using the cluster_size field in the candidates.tab file) is less than the value cross_mapping_ratio defined in the configuration file (0.02 by default).

Analysis directory

The directory that was created with igdiscover init. Separate ones are created for each experiment. When you used igdiscover init myexperiment, the analysis directory would be myexperiment/.

Starting database

The initial list of V/D/J genes. These are expected to be in FASTA format and are copied into the database/ directory within each analysis directory.

• the linkage cluster analysis plots in iteration-03/clusterplots/,
• the error histograms in iteration-03/errorhistograms.pdf, which contain the windowed cluster analysis figures.
• Details about the individualized database in new_V_germline.tab in tab-separated-value format

1.

If you haven’t done so, install miniconda. See the first steps of the regular installation instructions. Do not install IgDiscover, yet!

2.

Clone the IgDiscover repository:

git clone https://github.com/NBISweden/IgDiscover.git

(Use the git@ URL instead if you are a developer.)

4.

Create a new Conda environment using the environment.yml file in the repository:

cd IgDiscover
conda env create -n igdiscover -f environment.yml

You can choose a different environment name by changing the name after the -n parameter. This may be necessary, when you already have a regular (non-developer) IgDiscover installation in an igdiscover environment that you don’t want to overwrite.

5.

Activate the environment:

source activate igdiscover

(Or use whichever name you chose above.)

6.

Install IgDiscover in “editable” mode:

python3 -m pip install -e .

• tests/run.sh
• python3 setup.py sdist
• twine upload sdist/igdiscover-0.10.tar.gz
• Update bioconda recipe

1.

Delete miniconda: Run the command which conda. The output will be something like /home/myusername/miniconda3/bin/conda. The part before bin/conda is the miniconda installation directory. Delete that folder. In this case, you would need to delete miniconda3 in /home/myusername.

2.

Run pip3 uninstall igdiscover. If this runs successfully and prints some messages about removing files, then repeat the same command! Do this until you get a message telling you that the package cannot be uninstalled because it is not installed.

3.

Repeat the previous step, but with pip3 uninstall sqt.

4.

If you have a directory named .local within your home directory, you may want to rename it: mv .local dot-local-backup You can also delete it, but there is a small risk that other software (not IgDiscover) uses that directory. The directory is hidden, so a normal ls will not show it. Use ls -la while in your home directory to see it.

5.

If you have ever used sudo to install IgDiscover, you may have an installation in /usr/local/. You can try to remove it with sudo pip3 uninstall igdiscover.

6.

Delete the cloned Git repository if you have one. This is the directory in which you run git pull.

• The IgBLAST cache is now disabled by default. We assume that, in most cases, datasets will not be re-run with the exact same parameters, and then it only fills up the disk. Delete your cache with rm -r ~/.cache/igdiscover to reclaim the space. To enable the cache, create a file ~/.config/igdiscover.conf with the contents use_cache: true.
• If you choose to enable the cache, results from the PEAR merging step will now also be cached. See also the caching documentation.
• Added detection of chimeras to the (pre-)germline filters. Any novel allele that can be explained as a chimera of two unmodified reference alleles is marked in the new_V_germline.tab file. This is a bit sensitive, so the candidate is currently not discarded.
• Two additional files annotated_V_germline.tab and annotated_V_pregermline.tab are created in each iteration during the germline filtering step. These are identical to the candidates.tab file, except that they contain a why_filtered column that describes why a sequence was filtered. See the documentation for this feature.
• A more realistic test dataset (v0.5), now based on human instead of rhesus data, was prepared. The testing instructions have been updated accordingly.
• J discovery has been tuned to give fewer truncated sequences.
• Statistics are written to stats/stats.json.
• V SHM distribution plots are created automatically and written written to v-shm-distributions.pdf in each iteration folder.
• An igdiscover dbdiff subcommand was added that can compare two FASTA files.

• When computing a consensus sequence, allow some sequences to be truncated in the 3' end. Many of the discovered novel V alleles were truncated by one nucleotide in the 3' end because IgBLAST does not always extend the alignment to the end of the V sequence. If these slightly too short V sequences were in the majority, their consensus would lead to a truncated sequence as well. The new consensus algorithm allows for this effect at the 3' end and can therefore more often than previously find the full sequence. Example:

TACTGTGCGAGAGA (seq 1)
TACTGTGCGAGAGA (seq 2)
TACTGTGCGAGAG- (seq 3)
TACTGTGCGAG--- (seq 4)
TACTGTGCGAG--- (seq 5)
TACTGTGCGAGAG  (previous consensus)
TACTGTGCGAGAGA (new consensus)

• Add a column database_changes to the new_V_germline.tab file that describes how the novel sequence differs from the database sequence. Example: 93C>T; 114A>G
• Allow filtering by CDR3_shared_ratio and do so by default (needs documentation)
• Cache the edit distance when computing the distance matrix. Speeds up the discover command slightly.
• discover: Use more than six CPU cores if available
• igblast: Print progress every minute

• Implemented allele ratio filtering for J gene discovery
• J genes are discovered as part of the pipeline (previously, one needed to run the discoverj script manually)
• In each iteration, dendrograms are now created not only for V genes, but also for D and J genes. The file names are dendrogram_D.pdf, dendrogram_J.pdf
• The V dendrograms are now in dendrogram_V.pdf (no longer V_dendrogram.pdf). This puts all the dendrograms together when looking at the files in the iteration directory.
• The V_usage.tab and V_usage.pdf files are no longer created. Instead, expressed_V.tab and expressed_V.pdf are created. These contain similar information, but an allele-ratio filter is used to filter out artifacts.
• Similarly, expressed_D.tab and expressed_J.tab and their .pdf counterparts are created in each iteration.
• Removed parse subcommand (functionality is in the igblast subcommand)
• New CDR3 detection method (only heavy chain sequences): CDR3 start/end coordinates are pre-computed using the database V and J sequences. Increases detection rate to 99% (previously less than 90%).
• Remove the ability to check discovered genes for required motifs. This has never worked well.
• Add a column clonotypes to the candidates.tab that tries to count how many clonotypes are associated with a single candidate (using only exact occurrences). This is intended to replace the CDR3s_exact column.
• Add an exact_ratio to the germline filtering options. This checks the ratio between the exact V occurrence counts (exact column) between alleles.
• Germline filtering option allele_ratio was renamed to clonotypes_ratio
• Implement a cache for IgBLAST results. When the same dataset is re-analyzed, possibly with different parameters, the cached results are used instead of re-running IgBLAST, which saves a lot of time. If the V/D/J database or the IgBLAST version has changed, results are not re-used.

• Add a barcodes_exact column to the candidates table. It gives the number of unique barcode sequences that were used by the sequences in the set of exact sequences. Also, add a configuration setting barcode_consensus that can turn off consensus taking of barcode groups, which needs to be set to false for barcodes_exact to work.
• Add a Ds_exact column to candidates table.
• Add a D_coverage configuration option.
• The pre-processing filtering step no longer reads in the full table of IgBLAST assignments, but filters the table piece by piece. Memory usage for this step therefore does not depend anymore on the dataset size and should always be below 1 GB.
• The functionality of the parse subcommand has been integrated into the igblast subcommand. This means that igdiscover igblast now directly outputs a result table (assigned.tab). This makes it easier to use that subcommand directly instead of only via the workflow.
• The igblast subcommand now always runs makeblastdb by itself and deletes the BLAST database afterwards. This reduces clutter and ensures the database is always up to date.
• Remove the library_name configuration setting. Instead, the library_name is now always the same as the name of analysis directory.

• Add an “allele ratio” criterion to the germline filter to further reduce the number of false positives. The filter is activated by default and can be configured through the allele_ratio setting in the configuration file. See the documentation for how it works.
• Ignore the CDR3-encoding bases whenever comparing two V gene sequences.
• Avoid finding 5'-truncated V genes by extending found hits towards the 5' end.
• By default, candidate sequences are no longer merged if they are nearly identical. That is, the differences setting within the two germline filter configuration sections is now set to zero by default. Previously, we believed the merging would remove some false positives, but it turns out we also miss true positives. It also seems that with the other changes in this version we also no longer get the particular false positives the setting was supposed to catch.
• Implement an experimental discoverj script for J gene discovery. It is curently not run automatically as part of igdiscover run. See igdiscover discoverj --help for how to run it manually.
• Add a config subcommand, which can be used to change the configuration file from the command-line.
• Add a V_CDR3_start column to the assigned.tab/filtered.tab tables. It describes where the CDR3 starts within the V sequence.
• Similarly, add a CDR3_start column to the new_V_germline.tab file describing where the CDR3 starts within a discovered V sequence. It is computed by using the most common CDR3 start of the sequences within the cluster.
• Rename the compose subcommand to germlinefilter.
• The init subcommand automatically fixes certain problems in the input database (duplicate sequences, empty records, duplicate sequence names). Previously, it would complain, but the user would have to fix the problems themselves.
• Move source code to GitHub
• Set up automatic code testing (continuous integration) via Travis
• Many documentation improvements

• The FASTA files of the input V/D/J gene lists now need to be named V.fasta, D.fasta and J.fasta. The species name is no longer part of the file name. This should reduce confusion when working with species not supported by IgBLAST.
• The species: configuration setting in the configuration can (and should) now be left empty. Its only use was that it is passed to IgBLAST, but since IgDiscover provides IgBLAST with its own V/D/J sequences anyway, it does not seem to make a difference.
• A “cross-mapping” detection has been added, which should reduce the number of false positives. See the documentation for an explanation.
• Novel sequences identical to a database sequence no longer get the _S1234 suffix.
• No longer trim trim the initial G run in sequences (due to RACE) by default. It is now a configuration setting.
• Add cdr3_location configuration setting: It allows to set whether to use a CDR3 in addition to the barcode for grouping sequences.
• Create a groups.tab.gz file by default (describing the de-barcoded groups)
• The pre-processing filter is now configurable. See the preprocessing_filter section in the configuration file.
• Many improvements to the documentation
• Extended and fixed unit tests. These are now run via a CI system.
• Statistics in JSON format are written to stats/stats.json.
• IgBLAST 1.5.0 output can now be parsed. Parsing is also faster by 25%.
• More helpful warning message when no sequences were discovered in an iteration.
• Drop support for Python 3.3.

• V sequences of the input database are now whitelisted by default. The meaning of the whitelist configuration option has changed: If set to false, those sequences are no longer whitelisted. To whitelist additional sequences, create a whitelist.fasta file as before.
• Sequences with stop codons are now filtered out by default.
• Use more stringent germline filtering parameters by default.

• It is now possible to install and run IgDiscover on OS X. Appropriate Conda packages are available on bioconda.
• Add column has_stop to candidates.tab, which indicates whether the candidate sequence contains a stop codon.
• Add a configuration option that makes it possible to disable the 5' motif check by setting check_motifs: false (the looks_like_V column is ignored in this case).
• Make it possible to whitelist known sequences: If a found gene candidate appears in that list, the sequence is included in the list of discovered sequences even when it would otherwise not pass filtering criteria. To enable this, just add a whitelist.fasta file to the project directory before starting the analysis.
• The criteria for germline filter and pre-germline filter are now configurable: See germline_filter and pre_germline_filter sections in the configuration file.
• Different runs of IgDiscover with the same parameters on the same input files will now give the same results. See the seed parameter in the configuration, also on how to get non-reproducible results as before.
• Both the germline and pre-germline filter are now applied in each iteration. Instead of the new_V_database.fasta file, two files named new_V_germline.fasta and new_V_pregermline.fasta are created.
• The compose subcommand now outputs a filtered version of the candidates.tab file in addition to a FASTA file. The table contains columns closest_whitelist, which is the name of the closest whitelist sequence, and whitelist_diff, which is the number of differences to that whitelist sequence.

• Optionally, sequences are not renamed in the assigned.tab file, but retain their original name as in the FASTA or FASTQ file. Set rename: false in the configuration file to get this behavior.
• Started an “advanced” section in the manual.

• IgDiscover can now also detect kappa and lambda light chain V genes (VK, VL)