NAME

       sim - find similarities in C, Java, Pascal, Modula-2, Lisp, Miranda, or text files

SYNOPSIS

       sim_c [ -[defFiMnpPRsST] -r N -t N -w N -o F ] file ... [ / [ file ... ] ]
       sim_c ...
       sim_java ...
       sim_pasc ...
       sim_m2 ...
       sim_lisp ...
       sim_mira ...
       sim_text ...

DESCRIPTION

       Sim_c  reads  the  C  files file ...  and looks for segments of text that are similar; two
       segments of program text are similar if they only differ in layout,  comment,  identifiers
       and the contents of numbers, strings and characters.  If any runs of sufficient length are
       found, they are reported on standard output; the number of significant tokens in  the  run
       is given between square brackets.

       Sim_java  does  the  same for Java, sim_pasc for Pascal, sim_m2 for Modula-2, sim_mira for
       Miranda, and sim_lisp for Lisp.  Sim_text works on  arbitrary  text;  it  is  occasionally
       useful on shell scripts.

       The  program  can  be  used  for  finding  copied  pieces of code in purportedly unrelated
       programs (with -s or -S), or for finding accidentally duplicated code in  larger  projects
       (with -f).

       If  a  /  is present between the input files, the latter are divided into a group of "new"
       files (before the /) and a group of "old" files; if there is no /, all  files  are  "new".
       Old files are never compared to each other.

       Since  the  similarity  tester reads the files several times, it cannot read from standard
       input.

       There are the following options:

       -d     The output is in a diff(1)-like format instead of the default 2-column format.

       -e     Each file is compared to each file in isolation; this will  find  all  similarities
              between all texts involved, regardless of duplicates.

       -f     Runs  are  restricted  to segments with balancing parentheses, to isolate potential
              routine bodies (not in text).

       -F     The names of routines in calls are required to match exactly (not in text).

       -i     The names of the files to be compared are read from  standard  input,  including  a
              possible  /; the file names must be one to a line.  This option allows a very large
              number of file names to be specified; it differs from the @  facility  provided  by
              some  compilers  in  that it handles file names only, and does not recognize option
              arguments.

       -M     Memory usage information is displayed on standard error output.

       -n     Similarities found are only summarized, not displayed.

       -o F   The output is written to the file named F.

       -p     The output is given in similarity percentages; see below; implies -e and -s.

       -P     As -p but more extensive; implies -e and -s.

       -r N   The minimum run length is set to N units; the  default  is  24  tokens,  except  in
              sim_text, where it is 8 words.

       -R     Directories  in  the input list are entered recursively, and all files they contain
              are involved in the comparison.

       -s     The contents of a file are not compared to itself (-s for "not self").

       -S     The contents of the new files are compared to the old  files  only  -  not  between
              themselves.

       -t N   In  combination  with  the  -p option, sets the threshold (in percents) below which
              similarities will not be reported; the default is 1, except in sim_text,  where  it
              is 20.

       -T     A more terse and uniform form of output is produced, which may be more suitable for
              postprocessing.

       -w N   The page width used is set to N columns; the default is 80.

       --     (A secret option, which prints the input as the similarity  checker  sees  it,  and
              then stops.)

       The -p option results in lines of the form
               F consists for x % of G material
       meaning  that  x  %  of  F's  text can also be found in G.  Note that this relation is not
       symmetric; it is in fact quite possible for one file to consist for 100  %  of  text  from
       another  file,  while  the  other file consists for only 1 % of text of the first file, if
       their lengths differ enough.  Each file is reported only once in the position of the F  in
       the above line.  This simplifies the identification of a set of files A[1] ... A[n], where
       the concatenation of these files is also present.  This restriction can be lifted by using
       the -P option instead.  A threshold can be set using the -t option; this option is ignored
       under -P.  Note that the granularity of the recognized text is still governed  by  the  -r
       option or its default.

       Sim_text accepts  s p a c e d   t e x t  as normal text.

       The  program can handle UNICODE file names under Windows.  This is relevant only under the
       -R option, since there is no way to give UNICODE file names from the command line.

       Care has been taken to keep all internal processes linear in the length of the input, with
       the  exception of the matching process which is almost linear, using a hash table; various
       other tables are used for speed-up.  If, however, there  is  not  enough  memory  for  the
       tables, they are discarded in order of unimportance, under which conditions the algorithms
       revert to their quadratic nature.

EXAMPLES

       The call
               sim_c *.c
       highlights duplicate code in the directory.  (It  is  useful  to  remove  generated  files
       first.)  A call
               sim_c -f -F *.c
       can pinpoint them further.

       A call
               sim_text -e -p -s new/* / old/*
       compares  each  file  in new/* to each subsequent file in new/* and old/*, and if any pair
       has more that 20% in common, that fact is reported.  Usually a similarity of 30%  or  more
       is significant; lower than 20% is probably coincidence; and in between is doubtful.

       A call
               sim_text -e -n -s -r100 new/* / old/*
       compares  the same files, and reports large common segments.  Both approaches are good for
       plagiarism detection.

LIMITATIONS

       Repetitive input is the bane of similarity checking.  If  we  have  a  file  containing  4
       copies of similar text,
           A1 A2 A3 A4
       where  the numbers serve only to distinguish the similar copies, there are 7 similarities:
       A1=A2, A1=A3, A1=A4, A2=A3, A2=A4, A3=A4, and A1A2=A3A4, even discarding  the  overlapping
       A1A2A3=A2A3A4.  Of these, only 3 are meaningful: A1=A2, A2=A3, and A3=A4.  And for a table
       with 20 lines similar to each other, not unusual in a program, there are 715 similarities,
       of which at most 19 are meaningful.  Reporting all 715 of them is clearly unacceptable.

       To remedy this, finding the similarities is performed as follows: For each position in the
       text, the largest segment is found, of which a non-overlapping copy  occurs  in  the  text
       following it.  That segment and its copy are reported and scanning resumes at the position
       just after the segment.  For the above example this results in the similarities  A1A2=A3A4
       and  A3=A4, which is quite satisfactory, and for N similar segments roughly log N messages
       are given.

       A drawback of this heuristic is that the output is sensitive to the  order  of  the  input
       files.  If we have two files
           file1 = A1, file2 = A2A3
       then  the order "file1 file2" gives "A1=A2, A2=A3" and "file2 file1" gives "A2=A3, A3=A1";
       but both reports convey the same information.

BUGS

       Since it uses lex(1) on some  systems,  it  may  crash  on  any  weird  construction  that
       overflows lex's internal buffers.

AUTHOR

       Dick Grune, Vrije Universiteit, Amsterdam; dick@dickgrune.com.

                                            2012/05/02                                     SIM(1)