Provided by: lucy_1.20-5_amd64 
NAME
lucy - Assembly Sequence Cleanup Program
SYNOPSIS
lucy [-pass_along min_value max_value med_value]
[-range area1 area2 area3] [-alignment area1 area2 area3]
[-vector vector_sequence_file splice_site_file]
[-cdna [minimum_span maximum_error initial_search_range]] [-keep]
[-size vector_tag_size] [-threshold vector_cutoff]
[-minimum good_sequence_length] [-debug [filename]]
[-output sequence_filename quality_filename]
[-error max_avg_error max_error_at_ends]
[-window window_size max_avg_error
[window_size max_avg_error ...]]
[-bracket window_size max_avg_error]
[-quiet] [-inform_me] [-xtra cpu_threads]
sequence_file quality_file [2nd_sequence_file]
DESCRIPTION
Lucy is a utility that prepares raw DNA sequence fragments for sequence assembly, possibly using
the TIGR Assembler. Raw DNA sequence fragments are obtained from DNA sequencing machines, such as
those from the Applied Biosystems Inc. (ABI). Lucy accepts three input data files: the sequences
file, the quality file, and (optionally) a second sequence file for comparison purposes. All three
files should be in the plain FASTA format.
The first sequence file and its accompanying quality file are obtained from other utility
programs, such as phred(1), which reads the sequencing machine chromatograph outputs and generates
sequence base calls in a sequence file together with a quality assessment file for each base call
made. The optional second sequence file usually comes directly from the sequencing machine itself
and are used to reassure/enhance the quality of the sequences.
Lucy makes no assumption about the order of sequences in the three input files. As long as all
necessary information can be found, DNA sequences and quality sequences can be in different order.
A sequence without its quality assessment companion or vice versa will be reported as an error.
The second sequence file is allowed to have missing sequences that appear only in the first
sequence file. Sequences that appear only in the second sequence file will be reported and ignored
by lucy.
The operations of Lucy are divided into 7 phases:
Phase 1:
Count the number of input sequences in the first sequence file, and create internal data
structures accordingly. Lucy allocates memory dynamically, so there is no preset limit on
the size of input files. The size of input files is only limited by your available computer
memory.
Note that lucy's static memory requirement grows very slowly with the number of input
sequences, at roughly 60 bytes plus the sequence name storage for each input sequence. This
is because lucy does not store any sequence data in the memory after processing them.
Therefore, by all practical considerations lucy can handle any number of input sequences.
The dynamic memory requirement of lucy is proportional to the longest sequence in the
input, not the number of sequences, but its actual size varies from processing phases to
phases.
Phase 2:
Read all sequence information, including name, length, and positions. To save memory, lucy
does not load all sequences data into main memory at once. Instead, it uses direct file
addressing to access each sequence only when it is needed. Therefore, it is very important
that the content of the input files stays unchanged during runtime.
Phase 3:
Read the quality information for sequences, and compute good quality regions, i.e., regions
within sequences that have higher quality values and can be trusted to be correct. Lucy
determines a "clean" range which has an average probability of error (per base) that is no
greater than the probability specified by the -error option (or the default, if -error is
not used). Note that secondary sequence extension (next step) is performed after quality
trimming. Because of this, it is possible that the final "clear" range (after vector
trimming) will have a probability of error which is greater than the specified value.
Phase 4:
Read the second sequence file, compare its sequences to the first sequence file, and extend
good quality regions if they both agree. If the second sequence file is not provided, this
step is skipped. Note that this sequence comparison phase will not in anyway shorten the
good quality region determined by the previous phase; it will only extend it if possible.
It is very important that the second sequence file does not come from the same base calling
software as the first sequence file, and is base-called with different algorithms.
Otherwise, if the two sequence files are identical, lucy will extend sequences all the way
to both ends, and completely ruin the purpose of quality trimming done in the previous
phase.
Usually, the first sequence file comes from phred(1) with the companion quality file, and
the second sequence file comes directly from the original ABI base calling software with
the sequencing machines.
Phase 5:
Locate splice sites on ends of sequences. In this phase, lucy tries to compare all input
sequences against splice site sequences in a splice site file which defines the vector
sequences near the insertion point on the vector. If splice site sequences are found on any
input sequence, they will be excluded from the good quality region so that the sequence
assembly program will not mistakenly take them into account when trying to reassemble the
sequences. Note that lucy assumes all input sequences are read from the same direction and
matching the direction of the splice site sequences. Therefore, the forward and reverse
read sequences of a clone should not be mixed together in a single input file. If such a
mixture of forward and reverse read sequences is unavoidable, lucy can be run twice to
check in both directions, once with the forward splice site sequences, the other time with
the reverse splice site sequences. See the description of option -vector below for more
details.
By popular demand, a poly-A/T trimming feature has been built into lucy. It is designated
Phase 5a and is an optional step. See the options -cdna and -keep below for details of
their usage.
Phase 6:
Remove vector insert sequences. In this phase, all input sequences are checked against a
full length vector sequence in a vector file, and sequences that are vector inserts
themselves will be detected and removed. Lucy uses a quick fragment match method to check
for vector sequences. Both the target vector sequence and the input sequences are converted
into fragments (range from 8 to 16 bases long, default is 10), and matching fragments are
detected quickly. Vector sequences are detected when they contain more matches to vector
fragments in their good quality region (already excluded of splice site sequences done
previously) than a normal, non-vector sequence can possibly match by chance. The default
cutoff threshold is 20%. A sequence which contains over 20% match to the vector will be
considered a vector insert and discarded.
Phase 7:
Produce output sequences for fragment assembly. In the final phase, Lucy produces two
output files, the cleaned sequence file with markers for good quality regions, and a
companion quality file. Optionally, lucy can also generate a cleavage information file
(i.e. the good quality region information) which can be used to update database.
Each sequence in the output sequence file begins with a header that includes its name, three pass
along clone length values to the fragment assembly program, and a left and right marker denoting
the begin and end of the good quality, vector free region. The following is an example of lucy
output:
>GCCAA03TF 1500 3000 2000 43 490
AGCCAAGTTTGCAGCCCTGCAGGTCGACTCTAGAGGATCCCCAGGATGATCAGCCACATT
GGGACTGAGACACGGCCCAAACTCCTACGGGAGGCAGCAGTGGGGAATCTTGCGCAATGG
GCGAAAGCCTGACGCAGCCATGCCGCGTGAATGATGAAGGTCTTAGGATTGTAAAATTCT
TTCACCGGGGACGATAATGACGGTACCCGGAGAAGAAGCCCCGGCTAACTTCGTGCCAGC
...
OPTIONS
Note that lucy checks only the first letter of each option, so all options below can be
represented by just typing the first letter, e.g. -p for -pass_along.
-pass_along min_value max_value med_value
The three pass along values of minimum, maximum and medium clone lengths are given using
this option. Lucy does not interpret these values; they are used by some sequence assembly
programs, such as TIGR Assembler. These values are directly copied over to the output
sequence file. The default values are 0, 0, and 0.
-range area1 area2 area3
This option is used in combination with the following option -alignment. It defines the
three splice site checking areas which may need different strengths of splice site
alignment. The quality of the base calls is usually poor at the beginning of a sequence but
gradually improves when moving into the sequence read. Therefore, when looking for splice
sites, stronger and stronger alignment measurements are needed to cope with the quality
change. The default range values are 40, 60 and 100, i.e., lucy will check for splice sites
at the first 200 DNA bases. If splice site is not found at the first 200 bases, the next
100 (=area3) bases will be checked, with a total checking length of 300. Once a splice site
is found, the rest of the sequence after the splice site is searched for the other end of
the splice site, if any, to guard against short inserts.
-alignment area1 area2 area3
This option is used in combination with the previous option. It defines the three
different alignment strengths for the three areas. An alignment within each area must be
equal or longer than these values before it is considered a match of the splice site.
Default values are 8, 12 and 16 for the first 40, 60 and 100 bases, respectively.
-vector vector_sequence_file splice_site_file
This option provides the complete vector sequence file and a partial splice site sequence
file. Lucy expects to see one single (probably long) sequence of the vector that is used
to do cloning in the vector file, and two splice site sequences before and after the
insertion point on the vector in the splice site file. The splice site sequences are
usually 150 bases in length, with a 50 bases overlay right around the vector insertion
point. Their actual lengths are not very critical. For example, the followings are the
PUC18 splice site sequences that can be used by lucy:
>PUCsplice.for.begin
gattaagttgggtaacgccagggttttcccagtcacgacgttgtaaaacg
acggccagtgccaagcttgcatgcctgcaggtcgactctagaggatcccc
gggtaccgagctcgaattcgtaatcatggtcatagctgtttcctgtgtga
>PUCsplice.for.end
acggccagtgccaagcttgcatgcctgcaggtcgactctagaggatcccc
gggtaccgagctcgaattcgtaatcatggtcatagctgtttcctgtgtga
aattgttatccgctcacaattccacacaacatacgagccggaagcataaa
With two splice site sequences as above, lucy assumes all sequences from the input files
are read in the same direction as the splice site sequences. If that is not true and the
input consists of sequences from both forward and reverse reads of a clone, there are two
options. One can either separate the sequences into forward and reverse read sets and run
them through lucy with correct splice site sequences. One can also run lucy with a combined
splice site file with both the forward and reverse splice site sequence pairs. That is, if
lucy sees four splice site sequences, it will assume that a bidirectional splice site
trimming has been ordered. For example, the following reverse PUC18 splice site sequences
can be appended to the forward splice sequences above to instruct lucy to do bidirectional
trimmings:
>PUCsplice.rev.begin
tttatgcttccggctcgtatgttgtgtggaattgtgagcggataacaatt
tcacacaggaaacagctatgaccatgattacgaattcgagctcggtaccc
ggggatcctctagagtcgacctgcaggcatgcaagcttggcactggccgt
>PUCsplice.rev.end
tcacacaggaaacagctatgaccatgattacgaattcgagctcggtaccc
ggggatcctctagagtcgacctgcaggcatgcaagcttggcactggccgt
cgttttacaacgtcgtgactgggaaaaccctggcgttacccaacttaatc
Bidirectional trimmings will run about two times slower because each sequence is compared
against two sets of splice site sequences when only one set is actually needed. It is
possible that random alignments with the other (unneeded) set will result in somehow a
shortened good quality region of a sequence. However, bidirectional trimmings can guarantee
that there are no vector fragments in the good quality region even when the assumed
direction of some sequence reads is wrong.
During phase 6 contaminant removal, lucy will automatically reverse complement the full
length vector sequence and check for both its forward and reverse inserts, so only one
vector sequence is needed in the vector file. Lucy can also get the vector and splice site
file names from the environment variables VECTOR_FILE and SPLICE_FILE, if they are not
given at the command line.
-cdna [minimum_span maximum_error initial_search_range]
Since the release of lucy, many people have requested that a poly-A/T trimming feature be
built into lucy for the convenience of people doing cDNA sequencing. This option is added
for that purpose. By default lucy will not do this step, unless this option is given. This
option can be given alone without any parameter, in that case the default values will be
used, or it can be given with all three parameters. The minimum_span defines the minimum
length of continuous poly-A or T before lucy believes that it has found them. The
maximum_error denotes the maximum number of errors allowed before a new continuous poly-A/T
region stops. If mismatch error count goes beyond maximum_error, then lucy believes that
it has reached the end of the poly-A/T tail/head. Note that each new continuous poly-A/T
region that is more than minimum_span long will reset the error counter to zero, therefore
an interleaving number of maximum_error followed by minimum_span can keep the poly-A/T
region expanding. The last parameter initial_search_range denotes the range from the ends
of a sequence within which a minimum_span number of continuous poly-A/T's have to be found,
otherwise lucy believes that it cannot find poly-A/T regions for the sequence. Note that
the ends of the sequence are related to the clear region after vector splice sites
trimming, not to the actual physical ends of the sequence. The default values of the three
parameter are minimum_span=10, maximum_error=3 and initial_search_range=50. Warning: these
three parameters have to be set carefully in order to avoid throwing good sequences away in
the middle of a sequence, or to let poly-A/T regions slip by at the ends of a sequence.
-keep When -cdna option is turned on, lucy will trim all poly-A/T fragments it finds. This is
good for EST clustering purposes, where you don't want the poly-A/T fragments to stay.
However, if you want to see the EST sequence in its entirety to know its direction, it is
not helpful to trim poly-A/T away. The -keep option, when used in combination with the
-cdna option, will preserve the poly-A/T tails/heads at ends of each EST sequence to keep
them as tags indicating the direction of the EST sequence.
-size vector_tag_size
This option is used in combination with the following option. It defines the size of
fragments for checking vector using the fragment matching algorithm. The default value is
10 bases. The range of acceptable values are 8 to 16.
-threshold vector_cutoff
The option is used in combination with the previous option. It defines the threshold of
similarity between a sequence and the vector for it to be considered a vector insert. Since
splice sites are not included in vector screening, any sequence which has a higher than
normal similarity to the vector sequence will be considered a vector itself and discarded.
The default value of cutoff is 20% of the good quality region.
-minimum good_sequence_length
After all kinds of checking, comparing and trimming, the good region of a sequence must
still be long enough than the minimum length for it to be considered useful to the sequence
assembly program. We do not want our sequence assembly program to be bothered by many
small, trashy fragments. The default minimum good sequence length is 100 bases.
-debug [filename]
This option, if given, tells lucy to produce a sequence cleavage information file for
reference or for updating the database. The default file name is "lucy.debug", which can be
overridden by the given file name.
-output sequence_filename quality_filename
This option defines the output sequence and quality file names. If not given, the default
file names lucy uses are "lucy.seq" and "lucy.qul".
-error max_avg_error max_error_at_ends
There are three main steps in the quality trimming performed by lucy. The first involves
removing low-quality bases from each end of the sequence, using the criteria specified by
the -bracket option. The second involves finding regions of the sequence where the
probability of error meets all of the criteria specified by the -window option. After
these regions are found, the third step is to trim each of them to the largest region
having an average probability of error no greater than the max_avg_error specified by the
-error option. Finally, the largest region meeting all of the criteria is chosen as the
final "clean" range.
Two parameters are specified with this option: max_avg_error is the maximum acceptable
average probability of error over the final clean range. max_error_at_ends is the maximum
probability of error that is allowed for the 2 bases at each end of the final clean range.
The defaults are 0.025 and 0.02, respectively, if -error is not specified.
Note: A base's estimated probability of error is calculated from the quality value that is
assigned by the base caller. The quality value (Q) is defined as:
Q = -10 * log10(Probability of error)
-window window_size max_avg_error [window_size max_avg_error ...]
This option affects the quality trimming of the sequence (see the description of the -error
option, above). It specifies one or more window sizes, and a maximum allowable average
probability of error for each of those window sizes. If more than one window size is
specified, they must be specified in decreasing order by window size. The maximum number
of windows that may be specified is 20.
Lucy uses a sliding window algorithm to find regions of the sequence within which the
average probability of error, within any window of the specified size, is no greater than
the specified max_avg_error. Regions which meet all of the specified window criteria are
then trimmed again using the criteria specified by the -error option, and the final "clean"
range is the largest region that meets all of the criteria.
If the -window option is not specified, then lucy uses 2 windows by default, of 50 and 10
bases. The default maximum allowable probabilities of error in the two windows are listed
below:
50-base window: 0.08
10-base window: 0.3
-bracket window_size max_avg_error
This option controls the initial quality trimming step, which is the removal of low-quality
bases from both ends of the sequence. lucy looks for the first and last window of size
window_size having an average probability of error no greater than max_avg_error. The
subsequence which extends from the first base of the first such window to the last base of
the last window is then examined further to find the clean range. Bases which precede the
first window or follow the last window are excluded from the clean range (so the two
terminal windows bracket the clean range).
The defaults for window_size and max_avg_error are 10 and 0.02.
-quiet Tells lucy to shut up and only report serious errors it finds. :)
-inform_me
Asks lucy to report sequences by names that have been thrown out due to low quality values,
or salvaged due to comparison to the 2nd sequence file.
-xtra cpu_threads
If you have multiple CPUs in your computer, you can dramatically increase lucy's speed by
allowing lucy to run multiple execution threads concurrently. For example, if you have a
dual-CPU computer, you can give the option -xtra 2 to cut lucy's execution time roughly in
half. By default, lucy will run just one thread if this option is not given. The maximum
number of allowable threads is 32. Note that this option is only available with the multi-
threaded lucy version 1.16p. There is also a 1.16s version that does not do multi-
threading.
ENVIRONMENT
The environment variable VECTOR_FILE defines the vector file name, and the environment variable
SPLICE_FILE defines the splice site file name. Both variables are used when the user does not
specify them using the -vector option.
SEE ALSO
TIGR_Assembler(1), grim(1), everm.sp(1), phred(1), TraceTuner(1), and ethyl.pl(1).
BUGS
No known bugs for the program at this moment. Some of the manual pages mentioned above do not
exist.
CAVEATS
The "no bugs" claim above can never be true. This is a new program built mostly from scratch, and
there must be bugs somewhere, somehow. Please direct all bug reports to the authors.
ACRONYM
Lucy stands for Less Useful Chunks Yank, an awkward combination of words in order to make it a
member of the family with phred, the base caller, ethyl, the old scripting system lucy replaced,
and ricky, the database linking and communication driving software of lucy for use in TIGR.
AUTHOR
Lucy was written by Hui-Hsien Chou and Michael Holmes at The Institute for Genomic Research, with
help and suggestions from Granger Sutton, Anna Glodek, John Scott, and Terry Shea. Michael Holmes
is currently responsible for lucy. Please direct any suggestions, bug reports, etc. to
mholmes@tigr.org.