NAME

       simhash - file similarity hash tool

SYNOPSIS

       simhash [ -s nshingles ] [ -f nfeatures ] [ file ]
       simhash [ -s nshingles ] [ -f nfeatures ] -w file ...
       simhash [ -s nshingles ] [ -f nfeatures ] -m file ...
       simhash -c hashfile hashfile

DESCRIPTION

       This  program is used to compute and compare similarity hashes of files.  A similarity hash is a chunk of
       data that has the property that some distance metric between  files  is  proportional  to  some  distance
       metric between the hashes.  Typically the similarity hash will be much smaller than the file itself.

       The  algorithm  used by simhash is Manassas' "shingleprinting" algorithm (see BIBLIOGRAPHY below): take a
       hash of every m-byte subsequence of the file, and retain the n  of  these  hashes  that  are  numerically
       smallest.  The size of the intersection of the hash sets of two files gives a statistically good estimate
       of the similarity of the files as a whole.

       In  its  default mode, simhash will compute the similarity hash of its file argument (or stdin) and write
       this hash to its standard output.  When invoked with the -w argument (see below),  simhash  will  compute
       similarity  hashes  of all of its file arguments in "batch mode".  When invoked with the -m argument (see
       below), simhash will compare all the given files using similarity hashes in "match mode".  Finally,  when
       invoked  with  the  -c  argument  (see  below),  simhash will report the degree of similarity between two
       hashes.

OPTIONS

       -f feature-count
              When computing a similarity hash, retain at most feature-count significant hashes from the  target
              file.   The  default  is  128  features.   Larger  feature  counts  will give higher resolution in
              differences between files, will increase the size of the similarity  hash  proportionally  to  the
              feature count, and will increase similarity hash computation time slightly.

       -s shingle-size
              When  computing  a  similarity  hash, use hashes of samples consisting of shingle-size consecutive
              bytes drawn from the target file.  The default is 8 bytes, the minimum is 4 bytes.  Larger shingle
              sizes will emphasize the differences  between  files  more  and  will  slow  the  similarity  hash
              computation proportionally to the shingle size.

       -c hashfile1 hashfile2
              Display  the  distance  (normalized  to  the  range  0..1)  between  the similarity hash stored in
              hashfile1 and the similarity hash stored in hashfile2.

       -w file ...
              Write the similarity hash of each of the file arguments to file.sim.

       -m file ...
              Compute the similarity hash of each of the file arguments, and  output  a  similarity  matrix  for
              those files.

AUTHOR

       Bart Massey <bart@cs.pdx.edu>

BUGS

       This  currently  uses  CRC32 for the hashing.  A Rabin Fingerprint should be offered as a slightly slower
       but more reliable alternative.

       The shingleprinting algorithm works for text files and fairly well for other  sequential  filetypes,  but
       does not work well for image files.   The latter both are 2D and often undergo odd transformations.

BIBLIOGRAPHY

       Mark  Manasse,  Microsoft  Research Silicon Valley.  Finding similar things quickly in large collections.
       http://research.microsoft.com/research/sv/PageTurner/similarity.htm

       Andrei Z. Broder.  On the resemblance and containment of documents.  In  Compression  and  Complexity  of
       Sequences       (SEQUENCES'97),       pages       21-29.      IEEE      Computer      Society,      1998.
       ftp://ftp.digital.com/pub/DEC/SRC/publications/broder/positano-final-wpnums.pdf

       Andrei Z. Broder.  Some applications of Rabin's fingerprinting method.  Published in R. Capocelli, A.  De
       Santis,  U.  Vaccaro  eds.,  Sequences  II:  Methods  in  Communications, Security, and Computer Science,
       Springer-Verlag, 1993.  http://athos.rutgers.edu/~muthu/broder.ps

                                                 3 January 2007                                       SIMHASH(1)