Ubuntu Manpage: md5deep - Compute and compare MD5 message digests

NAME

       md5deep - Compute and compare MD5 message digests
       sha1deep - Compute and compare SHA-1 message digests
       sha256deep - Compute and compare SHA-256 message digests
       sha3deep - Compute and compare SHA-3-256 message digests
       tigerdeep - Compute and compare Tiger message digests
       whirlpooldeep - Compute and compare Whirlpool message digests

SYNOPSIS

       md5deep -v | -V | -h
       md5deep  [-m|-M|-x|-X <file>]  [-a|-A <hash>] [-f <file>] [-p <size>] [-i <size>] [-tnwzresS0lbkqZud] [-F
       <bum>] [-o <fbcplsde>]  [-j <num>] [[FILES]

DESCRIPTION

Computes the hashes, or message digest, for any number of files while optionally recursively digging
through the directory structure. Can also take a list of known hashes and display the filenames of input
files whose hashes either do or do not match any of the known hashes. Errors are reported to standard
error. If no FILES are specified, reads from standard input.

-p <size>
Piecewise mode. Breaks files into chunks before hashing. Chunks may be specified using IEC
multipliers b, k, m, g, t, p, or e. (Never let it be said that the author didn't plan ahead!)
This mode cannot be used with the -z mode.

-i|-I <size>
Size threshold mode. Only hash files smaller than the given the threshold. In -i mode, simply
omits those files larger than the threshold. In -I mode, displays all files, but uses asterisks
for the hashes of files larger than the threshold. Sizes may be specified using IEC multipliers
b, k, m, g, t, p, or e.

-r Enables recursive mode. All subdirectories are traversed. Please note that recursive mode cannot
be used to examine all files of a given file extension. For example, calling md5deep -r *.txt will
examine all files in directories that end in .txt.

-e Displays a progress indicator and estimate of time remaining for each file being processed. Time
estimates for files larger than 4GB are not available on Windows. This mode may not be used with
th -p mode.

-m <file>
Enables matching mode. The file given should be a list of known hashes. The input files are
examined one at a time, and only those files that match the list of known hashes are output. This
flag may be used more than once to add multiple sets of known hashes. Acceptable formats for lists
of known hashes are plain (such as those generated by md5deep or md5sum), Hashkeeper files, iLook,
and the National Software Reference Library (NSRL) as produced by the National Institute for
Standards in Technology.
If standard input is used with the -m flag, displays "stdin" if the input matches one of the
hashes in the list of known hashes. If the hash does not match, the program displays no output.
This flag may not be used in conjunction with the -x, -X, or -A flags. See the section "UNICODE
SUPPORT" below.

-x <file>
Same as the -m flag above, but does negative matching. That is, only those files NOT in the list
of known hashes are displayed.
This flag may not be used in conjunction with the -m, -M, or -a flags. See the section "UNICODE
SUPPORT" below.

-M and -X <file>
Same as -m and -x above, but displays the hash for each file that does (or does not) match the
list of known hashes.

-a <hash>
Adds a single hash to the list of known hashes used for matching mode, and if not already enabled,
enables matching mode. Adding single hashes cannot, by itself, be used to print the hashes of
matching files like the -M flag does. When used in conjunction with the -w flag, the filename
displayed is just the hash submitted on the command line.
This flag may not be used in conjunction with the -x, -X, or -A flags.

-A <hash>
Same as -a above, but does negative matching. This flag may not be used in conjunction with the
-m, -M, or -A flags.

-f <file>
Takes a list of files to be hashed from the specified file. Each line is assumed to be a filename.
This flag can only be used once per invocation. If it's used a second time, the second instance
will clobber the first.
Note that you can still use other flags, such as the -m or -x modes, and submit additional FILES
on the command line.

-w During any of the matching modes (-m,-M,-x,or -X), displays the filename of the known hash that
matched the input file. See the section "UNICODE SUPPORT" below.

-t Display a timestamp in GMT with each result. On Windows this timestamp will be the file's creation
time. On all other systems it should be the file's change time.

-n During any of the matching modes (-m,-M,-x,or -X), displays only the filenames of any known hashes
that were not matched by any of the input files.

-s Enables silent mode. All error messages are suppressed.

-S Like silent mode, but still displays warnings on improperly formatted hashes in the list of known
hashes.

-z Enables file size mode. Prepends the hash with a ten digit representation of the size of each file
processed. If the file size is greater than 9999999999 bytes (about 9.3GB) the program displays
9999999999 for the size.

-q Quiet mode. File names are omitted from the output. Each hash is still followed by two spaces
before the newline.

-Z Produces output in Triage format. Each line contains the file's size, a tab, a hash of the first
512 bytes, a tab, the hash of the complete file, a tab, and the file name. These values are
intended in increasing order of specificity. That is, two files with different sizes cannot
possibly match. This is a fast comparison and should be done first. Next, two files with different
partial hashes cannot possibly match. This is often faster than hashing the whole file. Finally,
if those two pieces align, then it's worth reading and hashing the entire file.

-0 Uses a NULL character (/0) to terminate each line instead of a newline. Useful for processing
filenames with strange characters.

-l Enables relative file paths. Instead of printing the absolute path for each file, displays the
relative file path as indicated on the command line. This flag may not be used in conjunction with
the -b flag.

-b Enables bare mode. Strips any leading directory information from displayed filenames. This flag
may not be used in conjunction with the -l flag.

-k Enables asterisk mode. An asterisk is inserted in lieu of a second space between the filename and
the hash, just like md5sum in its binary (-b) mode.

-c Enables comma separated values output, or CSV mode. This mode has the side effect of removing the
10 digit size limitation from -z mode. Also note that asterisks from -k mode are not displayed
when in CSV mode.

-o <bcpflsd>
Enables expert mode. Allows the user specify which (and only which) types of files are processed.
Directory processing is still controlled with the -r flag. The expert mode options allowed are:
f - Regular files
b - Block Devices
c - Character Devices
p - Named Pipes
l - Symbolic Links
s - Sockets
d - Solaris Doors
e - Windows PE executables

-jnn Controls multi-threading. By default the program will create one producer thread to scan the file
system and one hashing thread per CPU core. Multi-threading causes output filenames to be in non-
deterministic order, as files that take longer to hash will be delayed while they are hashed. If a
deterministic order is required, specify -j0 to disable multi-threading

-d Output in Digital Forensics XML (DFXML) format.

-u Quote Unicode output. For example, the snowman is shown as U+C426.

-F<bum>
Specifies the input mode that is used to read files. The default is -Fb (buffered I/O) which reads
files with fopen(). Specifying -Fu will use unbuffered I/O and read the file with open().
Specifying -Fm will use memory-mapped I/O which will be faster on some platforms, but which
(currently) will not work with files that produce I/O errors.

-h Show a help screen and exit.

-v Show the version number and exit.

-V Show copyright information and exit.

UNICODE SUPPORT

       As  of  version 3.0 the program supports Unicode characters in filenames on Microsoft Windows systems for
       filenames specified on the command line with globbing (e.g. *), for files specified with the -f of  files
       to hash, and for files read from directories using the -r option.

       By  default  all program input and output should be in UTF-8.  The program automatically converts this to
       UTF-16 for opening files).

       On Unix/Linux/MacOS, you should use a terminal emulator that  supports  UTF-8  and  UTF-8  characters  in
       filenames will be properly displayed.

       On  Windows,  the  programs  do  not display Unicode characters on the console.  You must either redirect
       output to a file and open the file with Wordpad (which can display Unicode), or you must specify  the  -u
       option to quote Unicode using standard U+XXXX notation.

       Currently the file name of a file containing known hashes may not be specified as a unicode filename, but
       you can specify the name using tab completion or an asterisk (e.g. md5deep -m *.txt where there  is  only
       one file with a .txt extension).

RETURN VALUE

       Returns a bit-wise value based on the success of the operation and the status of any matching operations.

       0      Success.  Note  that  the program considers itself successful even when it encounters read errors,
              permission denied errors, or finds directories when not in recursive mode.

       1      Unused hashes. Under any of the matching modes, returns this value if one or  more  of  the  known
              hashes was not matched by any of the input files.

       2      Unmatched  inputs. Under any of the matching modes, returns this value if one or more of the input
              values did not match any of the known hashes.

       64     User error, such as trying to do both positive and negative matching at the same time.

       128    Internal error, such as memory corruption or  uncaught  cycle.   All  internal  errors  should  be
              reported to the developer! See the section "Reporting Bugs" below.

AUTHOR

       md5deep was written by Jesse Kornblum, research@jessekornblum.com and Simson Garfinkel.

KNOWN ISSUES

       Using  the  -r  flag cannot be used to recursively process all files of a given extension in a directory.
       This is a feature, not a bug.  If you need to do this, use the find(1) command.

REPORTING BUGS

       We take all bug reports very seriously. Any bug that jeopardizes the forensic integrity of  this  program
       could  have  serious  consequences  on  people's  lives.  When  submitting a bug report, please include a
       description of the problem, how you found it, and your contact information.

       Send bug reports to the author at the address above.

COPYRIGHT

       This program is a work of the US Government. In accordance with 17 USC 105, copyright protection  is  not
       available  for  any  work  of the US Government.  This program is PUBLIC DOMAIN. Portions of this program
       contain code that is licensed under the terms of the General Public License (GPL).  Those portions retain
       their original copyright and license. See the file COPYING for more details.

       There is NO warranty for this program; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.