Provided by: dieharder_3.31.1-7_i386 bug

NAME

       dieharder   -  A  testing  and  benchmarking  tool  for  random  number
       generators.

SYNOPSIS

       dieharder [-a] [-d dieharder test number] [-f filename] [-B]
                 [-D output flag [-D output flag] ... ] [-F] [-c separator]
                 [-g generator number or -1] [-h] [-k ks_flag] [-l]
                 [-L overlap] [-m multiply_p] [-n ntuple]
                 [-p number of p samples] [-P Xoff]
                 [-o filename] [-s seed strategy] [-S random number seed]
                 [-n ntuple] [-p number of p samples] [-o filename]
                 [-s seed strategy] [-S random number seed]
                 [-t number of test samples] [-v verbose flag]
                 [-W weak] [-X fail] [-Y Xtrategy]
                 [-x xvalue] [-y yvalue] [-z zvalue]

dieharder OPTIONS

       -a runs all the tests with standard/default options to create a
              user-controllable report.  To  control  the  formatting  of  the
              report,  see  -D below.  To control the power of the test (which
              uses default values for tsamples that cannot generally be varied
              and psamples which generally can) see -m below as a "multiplier"
              of the default number of psamples (used only in a -a run).

       -d test number -  selects specific diehard test.

       -f filename - generators 201 or 202 permit either raw binary or
              formatted ASCII numbers to be read in from a file  for  testing.
              generator  200  reads  in  raw  binary numbers from stdin.  Note
              well: many tests with default parameters require a lot of rands!
              To  see  a  sample  of the (required) header for ASCII formatted
              input, run

                       dieharder -o -f example.input -t 10

              and then examine the  contents  of  example.input.   Raw  binary
              input  reads  32  bit  increments  of the specified data stream.
              stdin_input_raw accepts a pipe from a raw binary stream.

       -B binary mode (used with -o below) causes output rands to  be  written
       in raw binary, not formatted ascii.

       -D output flag - permits fields to be selected for inclusion in
              dieharder  output.   Each flag can be entered as a binary number
              that turns on a specific output field or header or by flag name;
              flags  are aggregated.  To see all currently known flags use the
              -F command.

       -F - lists all known flags by name and number.

       -c table separator  -  where  separator  is  e.g.  ','  (CSV)  or  '  '
       (whitespace).

       -g generator number - selects a specific generator for testing.  Using
              -g  -1  causes  all  known  generators  to be printed out to the
              display.

       -h prints context-sensitive help -- usually Usage (this message) or a
              test synopsis if entered as e.g. dieharder -d 3 -h.

       -k ks_flag - ks_flag

              0 is fast but slightly sloppy for psamples > 4999 (default).

              1 is MUCH  slower  but  more  accurate  for  larger  numbers  of
              psamples.

              2  is  slower still, but (we hope) accurate to machine precision
              for any number of psamples up to some as yet  unknown  numerical
              upper  limit  (it  has  been  tested out to at least hundreds of
              thousands).

              3 is kuiper  ks,  fast,  quite  inaccurate  for  small  samples,
              deprecated.

       -l list all known tests.

       -L overlap

              1 (use overlap, default)

              0 (don't use overlap)

              in  operm5  or  other  tests  that  support overlapping and non-
              overlapping sample modes.

       -m multiply_p - multiply default # of psamples in -a(ll) runs to crank
              up the resolution of failure.  -n ntuple - set ntuple length for
              tests  on  short bit strings that permit the length to be varied
              (e.g. rgb bitdist).

       -o filename - output -t count random numbers from current generator  to
       file.

       -p count - sets the number of p-value samples per test (default 100).

       -P  Xoff  -  sets  the  number  of  psamples  that will cumulate before
       deciding
              that a generator is "good" and really, truly passes even a -Y  2
              T2D run.  Currently the default is 100000; eventually it will be
              set from AES-derived  T2D  test  failure  thresholds  for  fully
              automated reliable operation, but for now it is more a "boredom"
              threshold set by how long one might reasonably want to  wait  on
              any given test run.

       -S seed - where seed is a uint.  Overrides the default random seed
              selection.  Ignored for file or stdin input.

       -s strategy - if strategy is the (default) 0, dieharder reseeds (or
              rewinds)  once at the beginning when the random number generator
              is selected and then never again.  If strategy is  nonzero,  the
              generator  is reseeded or rewound at the beginning of EACH TEST.
              If -S seed was specified, or a file is used,  this  means  every
              test  is  applied  to  the  same  sequence  (which is useful for
              validation and testing of dieharder, but not a good way to  test
              rngs).  Otherwise a new random seed is selected for each test.

       -t count - sets the number of random entities used in each test, where
              possible.   Be  warned  --  some  tests have fixed sample sizes;
              others are variable but have practical  minimum  sizes.   It  is
              suggested  you  begin  with the values used in -a and experiment
              carefully on a test by test basis.

       -W weak - sets the "weak" threshold to make the test(s) more or less
              forgiving during e.g. a  test-to-destruction  run.   Default  is
              currently 0.005.

       -X fail - sets the "fail" threshold to make the test(s) more or less
              forgiving  during  e.g.  a  test-to-destruction run.  Default is
              currently 0.000001, which is basically "certain failure  of  the
              null  hypothesis",  the  desired  mode of reproducible generator
              failure.

       -Y Xtrategy - the Xtrategy flag controls  the  new  "test  to  failure"
       (T2F)
              modes.  These flags and their modes act as follows:

                0  -  just run dieharder with the specified number of tsamples
              and psamples, do not dynamically modify a run based on  results.
              This is the way it has always run, and is the default.

                1  - "resolve ambiguity" (RA) mode.  If a test returns "weak",
              this is an undesired result.  What does that  mean,  after  all?
              If  you  run  a  long  test series, you will see occasional weak
              returns  for  a  perfect  generators  because  p  is   uniformly
              distributed  and will appear in any finite interval from time to
              time.  Even if a test run returns more than one weak result, you
              cannot  be  certain that the generator is failing.  RA mode adds
              psamples (usually in blocks of 100) until the test  result  ends
              up solidly not weak or proceeds to unambiguous failure.  This is
              morally equivalent to running the test several times to see if a
              weak result is reproducible, but eliminates the bias of personal
              judgement in the process since the default failure threshold  is
              very small and very unlikely to be reached by random chance even
              in many runs.

              This option should only be used with -k 2.

                2 - "test to destruction" mode.  Sometimes you  just  want  to
              know  where  or if a generator will .I ever fail a test (or test
              series).  -Y 2 causes psamples to be added 100 at a time until a
              test  returns an overall pvalue lower than the failure threshold
              or a specified maximum number of psamples (see -P) is reached.

              Note well!  In this mode one may well fail due to the  alternate
              null  hypothesis  --  the  test  itself is a bad test and fails!
              Many dieharder tests, despite our best efforts, are  numerically
              unstable  or  have only approximately known target statistics or
              are straight up asymptotic results, and will eventually return a
              failing result even for a gold-standard generator (such as AES),
              or for the hypercautious the XOR generator with AES,  threefish,
              kiss,  all  loaded  at once and xor'd together.  It is therefore
              safest to use this mode .I comparatively, executing a T2D run on
              AES to get an idea of the test failure threshold(s) (something I
              will eventually do and publish on the web so  everybody  doesn't
              have  to do it independently) and then running it on your target
              generator.  Failure with numbers of psamples within an order  of
              magnitude  of  the  AES thresholds should probably be considered
              possible test failures, not  generator  failures.   Failures  at
              levels significantly less than the known gold standard generator
              failure thresholds are, of  course,  probably  failures  of  the
              generator.

              This option should only be used with -k 2.

       -v verbose flag -- controls the verbosity of the output for debugging
              only.   Probably of little use to non-developers, and developers
              can read the enum(s) in dieharder.h and the test sources to  see
              which flag values turn on output on which routines.  1 is result
              in a highly detailed trace of program activity.

       -x,-y,-z number - Some tests have parameters that can safely be varied
              from their default value.  For example, in the diehard birthdays
              test,  one  can  vary  the  number  of length, which can also be
              varied.  -x 2048 -y 30 alters these two values but should  still
              run  fine.   These  parameters  should  be documented internally
              (where they exist) in the e.g. -d 0 -h visible notes.

              NOTE WELL: The assessment(s) for  the  rngs  may,  in  fact,  be
              completely incorrect or misleading.  There are still "bad tests"
              in dieharder, although we are working to fix  and  improve  them
              (and  try to document them in the test descriptions visible with
              -g testnumber -h).  In particular, 'Weak' pvalues  should  occur
              one  test  in two hundred, and 'Failed' pvalues should occur one
              test in a million with the default thresholds -  that's  what  p
              MEANS.  Use them at your Own Risk!  Be Warned!

              Or  better  yet,  use the new -Y 1 and -Y 2 resolve ambiguity or
              test to destruction modes above, comparing to  similar  runs  on
              one  of  the as-good-as-it-gets cryptographic generators, AES or
              threefish.

DESCRIPTION

       dieharder

       Welcome to the current snapshot of the dieharder random number  tester.
       It  encapsulates  all of the Gnu Scientific Library (GSL) random number
       generators (rngs) as  well  as  a  number  of  generators  from  the  R
       statistical  library,  hardware  sources  such  as  /dev/*random, "gold
       standard"  cryptographic  quality  generators   (useful   for   testing
       dieharder  and for purposes of comparison to new generators) as well as
       generators contributed by users or  found  in  the  literature  into  a
       single harness that can time them and subject them to various tests for
       randomness.  These tests are variously drawn  from  George  Marsaglia's
       "Diehard  battery  of  random  number tests", the NIST Statistical Test
       Suite, and again from other sources such as  personal  invention,  user
       contribution, other (open source) test suites, or the literature.

       The  primary  point  of  dieharder  is to make it easy to time and test
       (pseudo)random number generators, including both software and  hardware
       rngs,  with  a  fully  open  source  tool.   In  addition  to providing
       "instant" access to testing  of  all  built-in  generators,  users  can
       choose  one of three ways to test their own random number generators or
       sources:  a unix pipe of a raw binary (presumed  random)  bitstream;  a
       file  containing  a (presumed random) raw binary bitstream or formatted
       ascii uints or floats; and embedding your generator in dieharder's GSL-
       compatible   rng  harness  and  adding  it  to  the  list  of  built-in
       generators.  The stdin and file input methods are  described  below  in
       their  own  section,  as  is  suggested  "best practice" for newbies to
       random number generator testing.

       An important motivation for using dieharder is  that  the  entire  test
       suite  is  fully  Gnu  Public  License (GPL) open source code and hence
       rather than being prohibited from "looking  underneath  the  hood"  all
       users  are  openly  encouraged to critically examine the dieharder code
       for errors, add new tests or generators or user interfaces, or  use  it
       freely  as is to test their own favorite candidate rngs subject only to
       the constraints of the GPL.  As a result  of  its  openness,  literally
       hundreds  of  improvements and bug fixes have been contributed by users
       to date, resulting in a far stronger and more reliable test suite  than
       would  have  been  possible with closed and locked down sources or even
       open sources (such as STS) that lack the dynamical  feedback  mechanism
       permitting corrections to be shared.

       Even  small  errors  in test statistics permit the alternative (usually
       unstated) null hypothesis to become an important factor in rng  testing
       -- the unwelcome possibility that your generator is just fine but it is
       the test that is failing.  One extremely useful feature of dieharder is
       that  it  is  at  least  moderately  self  validating.  Using the "gold
       standard" aes and threefish cryptographic generators, you  can  observe
       how  these  generators  perform  on  dieharder runs to the same general
       degree of accuracy that you wish to  use  on  the  generators  you  are
       testing.   In  general,  dieharder  tests that consistently fail at any
       given level of precision (selected with e.g. -a -m 10) on both  of  the
       gold  standard  rngs (and/or the better GSL generators, mt19937, gfsr4,
       taus) are probably unreliable at that precision and it would hardly  be
       surprising if they failed your generator as well.

       Experts  in  statistics are encouraged to give the suite a try, perhaps
       using any of the example calls below at first and then using it  freely
       on  their  own  generators  or as a harness for adding their own tests.
       Novices (to either statistics or random number generator  testing)  are
       strongly  encouraged  to read the next section on p-values and the null
       hypothesis and running the test suite a few times with a  more  verbose
       output report to learn how the whole thing works.

QUICK START EXAMPLES

       Examples  for  how  to  set  up  pipe  or  file  input are given below.
       However, it is recommended that a user play with some of the  built  in
       generators  to gain familiarity with dieharder reports and tests before
       tackling their own favorite generator or file full of  possibly  random
       numbers.

       To  see  dieharder's  default  standard  test  report  for  its default
       generator (mt19937) simply run:

          dieharder -a

       To increase the resolution of possible failures of the standard  -a(ll)
       test,  use  the -m "multiplier" for the test default numbers of pvalues
       (which are selected more to make a full test run take  an  hour  or  so
       instead  of  days than because it is truly an exhaustive test sequence)
       run:

          dieharder -a -m 10

       To test a different generator (say the gold  standard  AES_OFB)  simply
       specify the generator on the command line with a flag:

          dieharder -g 205 -a -m 10

       Arguments  can  be in any order.  The generator can also be selected by
       name:

          dieharder -g AES_OFB -a

       To apply only the diehard opso test to the AES_OFB  generator,  specify
       the test by name or number:

          dieharder -g 205 -d 5

       or

          dieharder -g 205 -d diehard_opso

       Nearly  every  aspect  or  field in dieharder's output report format is
       user-selectable by means of display option  flags.   In  addition,  the
       field  separator  character  can  be  selected  by the user to make the
       output particularly easy for them to parse (-c ' ') or  import  into  a
       spreadsheet (-c ',').  Try:

          dieharder -g 205 -d diehard_opso -c ',' -D test_name -D pvalues

       to see an extremely terse, easy to import report or

          dieharder  -g  205 -d diehard_opso -c ' ' -D default -D histogram -D
       description

       to see a verbose report good for a  "beginner"  that  includes  a  full
       description of each test itself.

       Finally, the dieharder binary is remarkably autodocumenting even if the
       man page is not available. All users should try the following  commands
       to see what they do:

          dieharder -h

       (prints the command synopsis like the one above).

          dieharder -a -h
          dieharder -d 6 -h

       (prints the test descriptions only for -a(ll) tests or for the specific
       test indicated).

          dieharder -l

       (lists all known tests, including how reliable rgb thinks that they are
       as things stand).

          dieharder -g -1

       (lists all known rngs).

          dieharder -F

       (lists  all  the currently known display/output control flags used with
       -D).

       Both beginners and experts should be aware that the assessment provided
       by  dieharder  in  its  standard  report  should be regarded with great
       suspicion.  It is entirely possible for a generator to "pass" all tests
       as  far  as  their  individual  p-values  are concerned and yet to fail
       utterly when considering them all together.  Similarly, it is  probable
       that  a rng will at the very least show up as "weak" on 0, 1 or 2 tests
       in a typical -a(ll) run, and may even "fail" 1 test one such run in  10
       or  so.   To  understand  why this is so, it is necessary to understand
       something of rng testing, p-values, and the null hypothesis!

P-VALUES AND THE NULL HYPOTHESIS

       dieharder returns "p-values".  To understand what a p-value is and  how
       to use it, it is essential to understand the null hypothesis, H0.

       The  null  hypothesis  for  random  number  generator  testing is "This
       generator is a perfect random number generator, and for any  choice  of
       seed  produces  a infinitely long, unique sequence of numbers that have
       all the expected statistical  properties  of  random  numbers,  to  all
       orders".   Note  well  that we know that this hypothesis is technically
       false for all software generators as they are periodic and do not  have
       the  correct  entropy  content  for  this  statement  to  ever be true.
       However, many hardware generators  fail  a  priori  as  well,  as  they
       contain  subtle  bias  or correlations due to the deterministic physics
       that underlies them.  Nature is often unpredictable but  it  is  rarely
       random and the two words don't (quite) mean the same thing!

       The  null  hypothesis  can be practically true, however.  Both software
       and hardware generators can be "random"  enough  that  their  sequences
       cannot  be  distinguished from random ones, at least not easily or with
       the available tools (including dieharder!) Hence the null hypothesis is
       a practical, not a theoretically pure, statement.

       To  test  H0  ,  one uses the rng in question to generate a sequence of
       presumably random numbers.  Using these numbers one  can  generate  any
       one  of a wide range of test statistics -- empirically computed numbers
       that are considered random samples that may or  may  not  be  covariant
       subject  to  H0,  depending  on whether overlapping sequences of random
       numbers are used to generate successive samples  while  generating  the
       statistic(s), drawn from a known distribution.  From a knowledge of the
       target distribution of the statistic(s) and the  associated  cumulative
       distribution  function  (CDF)  and  the empirical value of the randomly
       generated statistic(s), one can read off the probability  of  obtaining
       the  empirical result if the sequence was truly random, that is, if the
       null hypothesis is true and the  generator  in  question  is  a  "good"
       random  number  generator!   This  probability is the "p-value" for the
       particular test run.

       For example, to test a coin (or a sequence of  bits)  we  might  simply
       count the number of heads and tails in a very long string of flips.  If
       we assume that the coin is a "perfect coin", we expect  the  number  of
       heads and tails to be binomially distributed and can easily compute the
       probability of getting any particular number of heads and tails.  If we
       compare  our recorded number of heads and tails from the test series to
       this distribution and find that the probability of getting the count we
       obtained  is very low with, say, way more heads than tails we'd suspect
       the coin wasn't a perfect coin.  dieharder applies this very test (made
       mathematically  precise)  and  many  others  that  operate on this same
       principle to the string of random bits produced by the rng being tested
       to provide a picture of how "random" the rng is.

       Note  that  the  usual dogma is that if the p-value is low -- typically
       less than 0.05 -- one "rejects" the null hypothesis.  In a word, it  is
       improbable that one would get the result obtained if the generator is a
       good one.  If it  is  any  other  value,  one  does  not  "accept"  the
       generator  as good, one "fails to reject" the generator as bad for this
       particular test.  A "good random number generator" is hence one that we
       haven't been able to make fail yet!

       This  criterion  is, of course, naive in the extreme and cannot be used
       with dieharder!  It makes just as much sense to reject a generator that
       has p-values of 0.95 or more!  Both of these p-value ranges are equally
       unlikely on any given test run, and should be returned for (on average)
       5%  of all test runs by a perfect random number generator.  A generator
       that fails to produce p-values less than 0.05 5%  of  the  time  it  is
       tested  with different seeds is a bad random number generator, one that
       fails the test of the null hypothesis.  Since  dieharder  returns  over
       100  pvalues  by  default per test, one would expect any perfectly good
       rng to "fail" such a naive test around five times by this criterion  in
       a single dieharder run!

       The  p-values  themselves,  as  it  turns out, are test statistics!  By
       their nature, p-values should be uniformly  distributed  on  the  range
       0-1.   In  100+  test  runs  with  independent seeds, one should not be
       surprised to obtain 0, 1, 2, or even  (rarely)  3  p-values  less  than
       0.01.   On  the other hand obtaining 7 p-values in the range 0.24-0.25,
       or seeing that 70 of the p-values are greater than 0.5 should make  the
       generator  highly  suspect!   How  can  a user determine when a test is
       producing "too many" of any particular value range for p?  Or too few?

       Dieharder does it for you, automatically.  One can in  fact  convert  a
       set  of  p-values into a p-value by comparing their distribution to the
       expected one, using a  Kolmogorov-Smirnov  test  against  the  expected
       uniform distribution of p.

       These  p-values  obtained  from looking at the distribution of p-values
       should in turn be uniformly  distributed  and  could  in  principle  be
       subjected  to still more KS tests in aggregate.  The distribution of p-
       values for a good generator should be idempotent, even across different
       test statistics and multiple runs.

       A  failure  of the distribution of p-values at any level of aggregation
       signals trouble.  In fact, if  the  p-values  of  any  given  test  are
       subjected  to  a KS test, and those p-values are then subjected to a KS
       test, as we add more p-values to either level we  will  either  observe
       idempotence  of  the  resulting  distribution of p to uniformity, or we
       will observe idempotence to a single p-value of zero!  That is, a  good
       generator  will  produce a roughly uniform distribution of p-values, in
       the specific sense that the p-values of the distributions  of  p-values
       are  themselves  roughly  uniform  and  so on ad infinitum, while a bad
       generator will produce a non-uniform distribution of p-values,  and  as
       more  p-values drawn from the non-uniform distribution are added to its
       KS test, at some point the failure will be absolutely unmistakeable  as
       the resulting p-value approaches 0 in the limit.  Trouble indeed!

       The question is, trouble with what?  Random number tests are themselves
       complex computational objects, and there is a  probability  that  their
       code  is  incorrectly framed or that roundoff or other numerical -- not
       methodical  --  errors  are  contributing  to  a  distortion   of   the
       distribution  of  some  of  the p-values obtained.  This is not an idle
       observation; when one works on writing random number generator  testing
       programs,  one  is  always testing the tests themselves with "good" (we
       hope) random number generators so that egregious failures of  the  null
       hypothesis  signal  not  a bad generator but an error in the test code.
       The null hypothesis above is correctly framed from a theoretical  point
       of  view,  but  from a real and practical point of view it should read:
       "This generator is a perfect  random  number  generator,  and  for  any
       choice  of  seed produces a infinitely long, unique sequence of numbers
       that have all the expected statistical properties of random numbers, to
       all  orders  and  this  test  is  a  perfect test and returns precisely
       correct p-values from the test  computation."   Observed  "failure"  of
       this  joint null hypothesis H0' can come from failure of either or both
       of these disjoint components, and comes from the  second  as  often  or
       more  often  than  the first during the test development process.  When
       one cranks up the "resolution" of the test (discussed next) to where  a
       generator  starts  to  fail  some test one realizes, or should realize,
       that development never ends and  that  new  test  regimes  will  always
       reveal new failures not only of the generators but of the code.

       With  that  said, one of dieharder's most significant advantages is the
       control that it gives you over a critical  test  parameter.   From  the
       remarks  above, we can see that we should feel very uncomfortable about
       "failing" any given random number generator on the basis of  a  5%,  or
       even  a  1%,  criterion,  especially  when  we  apply a test suite like
       dieharder that returns over 100 (and climbing) distinct  test  p-values
       as  of  the  last  snapshot.   We  want  failure  to be unambiguous and
       reproducible!

       To accomplish this, one can simply crank up its resolution.  If we  ran
       any  given  test against a random number generator and it returned a p-
       value of (say) 0.007328, we'd be perfectly justified in wondering if it
       is  really  a good generator.  However, the probability of getting this
       result isn't really all that small -- when one uses dieharder for hours
       at a time numbers like this will definitely happen quite frequently and
       mean nothing.  If one runs the same test again (with a  different  seed
       or  part  of the random sequence) and gets a p-value of 0.009122, and a
       third time and gets 0.002669 -- well, that's three 1% (or  less)  shots
       in  a  row and that should happen only one in a million times.  One way
       to clearly resolve failures, then, is to  increase  the  number  of  p-
       values  generated in a test run.  If the actual distribution of p being
       returned by the test is not uniform, a KS test will eventually return a
       p-value  that  is  not some ambiguous 0.035517 but is instead 0.000000,
       with the latter produced time after time as we rerun.

       For this reason, dieharder is extremely conservative  about  announcing
       rng  "weakness" or "failure" relative to any given test.  It's internal
       criterion for these things are currently p < 0.5% or p > 99.5% weakness
       (at the 1% level total) and a considerably more stringent criterion for
       failure: p < 0.05% or p >  99.95%.   Note  well  that  the  ranges  are
       symmetric -- too high a value of p is just as bad (and unlikely) as too
       low, and it is critical to flag it, because it is quite possible for  a
       rng  to be too good, on average, and not to produce enough low p-values
       on the full spectrum of dieharder  tests.   This  is  where  the  final
       kstest is of paramount importance, and where the "histogram" option can
       be very useful to help you visualize the failure in the distribution of
       p -- run e.g.:

         dieharder [whatever] -D default -D histogram

       and you will see a crude ascii histogram of the pvalues that failed (or
       passed) any given level of test.

       Scattered reports of weakness or  marginal  failure  in  a  preliminary
       -a(ll)  run should therefore not be immediate cause for alarm.  Rather,
       they are tests to repeat, to watch out for, to push the rng  harder  on
       using  the -m option to -a or simply increasing -p for a specific test.
       Dieharder permits one to increase the number of p-values generated  for
       any  test,  subject  only  to the availability of enough random numbers
       (for file based tests) and time, to make failures unambiguous.  A  test
       that  is  truly  weak  at -p 100 will almost always fail egregiously at
       some larger value of psamples, be it -p 1000 or  -p  100000.   However,
       because dieharder is a research tool and is under perpetual development
       and testing, it is strongly suggested  that  one  always  consider  the
       alternative  null  hypothesis  --  that the failure is a failure of the
       test code in dieharder itself in some limit of  large  numbers  --  and
       take  at  least  some  steps (such as running the same test at the same
       resolution on a "gold standard" generator) to ensure that  the  failure
       is indeed probably in the rng and not the dieharder code.

       Lacking  a  source  of  perfect  random  numbers to use as a reference,
       validating the tests themselves is not easy and always leaves one  with
       some  ambiguity  (even  aes or threefish).  During development the best
       one can usually do is to rely heavily on these "presumed  good"  random
       number  generators.   There  are  a  number  of generators that we have
       theoretical reasons to expect to be extraordinarily good  and  to  lack
       correlations out to some known underlying dimensionality, and that also
       test out extremely well quite  consistently.   By  using  several  such
       generators  and  not  just one, one can hope that those generators have
       (at the very least) different correlations and should not all uniformly
       fail a test in the same way and with the same number of p-values.  When
       all of these generators consistently fail a test at a  given  level,  I
       tend  to  suspect  that  the  problem  is  in  the  test  code, not the
       generators, although it is very  difficult  to  be  certain,  and  many
       errors in dieharder's code have been discovered and ultimately fixed in
       just this way by myself or others.

       One advantage of dieharder is that it  has  a  number  of  these  "good
       generators"  immediately available for comparison runs, courtesy of the
       Gnu Scientific Library and user contribution (notably David Bauer,  who
       kindly  encapsulated aes and threefish).  I use AES_OFB, Threefish_OFB,
       mt19937_1999, gfsr4, ranldx2  and  taus2  (as  well  as  "true  random"
       numbers  from  random.org)  for  this purpose, and I try to ensure that
       dieharder will "pass" in particular the -g 205 -S 1 -s 1  generator  at
       any reasonable p-value resolution out to -p 1000 or farther.

       Tests (such as the diehard operm5 and sums test) that consistently fail
       at these high resolutions are flagged as being  "suspect"  --  possible
       failures  of  the  alternative null hypothesis -- and they are strongly
       deprecated!  Their results should not be used  to  test  random  number
       generators  pending  agreement  in  the  statistics  and  random number
       community that those tests are  in  fact  valid  and  correct  so  that
       observed  failures  can indeed safely be attributed to a failure of the
       intended null hypothesis.

       As I  keep  emphasizing  (for  good  reason!)  dieharder  is  community
       supported.   I therefore openly ask that the users of dieharder who are
       expert in statistics to help  me  fix  the  code  or  algorithms  being
       implemented.   I  would  like  to  see  this  test  suite ultimately be
       validated by the general statistics community in hard use  in  an  open
       environment,  where  every  possible  failure  of the testing mechanism
       itself is subject to scrutiny and eventual correction.  In this way  we
       will  eventually  achieve  a  very powerful suite of tools indeed, ones
       that may well give us very specific information not just about  failure
       but  of  the  mode  of  failure  as  well, just how the sequence tested
       deviates from randomness.

       Thus far, dieharder has benefitted  tremendously  from  the  community.
       Individuals have openly contributed tests, new generators to be tested,
       and fixes for existing tests that were revealed by their own work  with
       the  testing  instrument.   Efforts are underway to make dieharder more
       portable so that it will build on more platforms  and  faster  so  that
       more thorough testing can be done.  Please feel free to participate.

FILE INPUT

       The  simplest  way  to  use  dieharder  with an external generator that
       produces raw binary (presumed random) bits is to pipe  the  raw  binary
       output  from  this  generator (presumed to be a binary stream of 32 bit
       unsigned integers) directly into dieharder, e.g.:

         cat /dev/urandom | ./dieharder -a -g 200

       Go ahead and try this example.  It will run the entire dieharder  suite
       of  tests  on  the  stream  produced  by  the  linux built-in generator
       /dev/urandom (using /dev/random is not recommended as it is too slow to
       test in a reasonable amount of time).

       Alternatively,  dieharder can be used to test files of numbers produced
       by a candidate random number generators:

         dieharder -a -g 201 -f random.org_bin

       for raw binary input or

         dieharder -a -g 202 -f random.org.txt

       for formatted ascii input.

       A formatted ascii input file can accept either uints (integers  in  the
       range  0  to  2^31-1, one per line) or decimal uniform deviates with at
       least ten significant digits (that can be multiplied by UINT_MAX = 2^32
       to  produce  a  uint  without  dropping  precition), also one per line.
       Floats with fewer digits will almost  certainly  fail  bitlevel  tests,
       although they may pass some of the tests that act on uniform deviates.

       Finally,  one  can  fairly  easily wrap any generator in the same (GSL)
       random number harness used internally by dieharder and simply  test  it
       the  same  way  one  would  any  other internal generator recognized by
       dieharder.  This is strongly recommended where it is possible,  because
       dieharder  needs  to  use  a lot of random numbers to thoroughly test a
       generator.  A built in generator can simply let dieharder determine how
       many  it  needs  and  generate them on demand, where a file that is too
       small will "rewind" and render the test results where a  rewind  occurs
       suspect.

       Note  well  that file input rands are delivered to the tests on demand,
       but if the test needs more than are available  it  simply  rewinds  the
       file  and  cycles  through  it  again,  and again, and again as needed.
       Obviously this significantly reduces the sample space and can  lead  to
       completely  incorrect  results  for the p-value histograms unless there
       are enough rands to run EACH test without repetition (it is harmless to
       reuse the sequence for different tests).  Let the user beware!

BEST PRACTICE

       A  frequently asked question from new users wishing to test a generator
       they are working on for fun or profit (or both) is "How  should  I  get
       its  output  into  dieharder?"   This  is  a  nontrivial  question,  as
       dieharder consumes enormous numbers of random numbers in  a  full  test
       cycle,  and  then  there are features like -m 10 or -m 100 that let one
       effortlessly demand 10 or 100 times as many to stress a  new  generator
       even more.

       Even  with  large file support in dieharder, it is difficult to provide
       enough random numbers in a file to really make dieharder happy.  It  is
       therefore strongly suggested that you either:

       a)  Edit the output stage of your random number generator and get it to
       write its production to stdout as a  random  bit  stream  --  basically
       create  32  bit  unsigned  random  integers  and write them directly to
       stdout as e.g. char data or raw binary.  Note that this is not the same
       as  writing  raw floating point numbers (that will not be random at all
       as a bitstream) and that "endianness" of the uints  should  not  matter
       for  the  null  hypothesis  of  a "good" generator, as random bytes are
       random in any order.  Crank the  generator  and  feed  this  stream  to
       dieharder in a pipe as described above.

       b) Use the samples of GSL-wrapped dieharder rngs to similarly wrap your
       generator (or calls to your generator's  hardware  interface).   Follow
       the  examples in the ./dieharder source directory to add it as a "user"
       generator in the  command  line  interface,  rebuild,  and  invoke  the
       generator  as  a  "native" dieharder generator (it should appear in the
       list produced by -g -1 when done correctly).  The advantage of doing it
       this  way  is  that  you  can  then  (if  your  new generator is highly
       successful) contribute it back to the dieharder project  if  you  wish!
       Not to mention the fact that it makes testing it very easy.

       Most  users  will probably go with option a) at least initially, but be
       aware that b)  is  probably  easier  than  you  think.   The  dieharder
       maintainers  may  be  able  to  give you a hand with it if you get into
       trouble, but no promises.

WARNING!

       A warning for those who are testing files of random numbers.  dieharder
       is  a  tool  that  tests  random number generators, not files of random
       numbers!  It is extremely inappropriate to try to "certify" a  file  of
       random  numbers  as being random just because it fails to "fail" any of
       the dieharder tests in e.g. a dieharder -a run.  To put it bluntly,  if
       one rejects all such files that fail any test at the 0.05 level (or any
       other), the one thing one can be  certain  of  is  that  the  files  in
       question  are  not  random,  as  a truly random sequence would fail any
       given test at the 0.05 level 5% of the time!

       To put it another way, any file of numbers produced by a generator that
       "fails to fail" the dieharder suite should be considered "random", even
       if it contains sequences that might well "fail" any given test at  some
       specific  cutoff.  One has to presume that passing the broader tests of
       the generator itself, it was determined that the p-values for the  test
       involved  was  globally  correctly distributed, so that e.g. failure at
       the 0.01 level occurs neither more nor less than 1%  of  the  time,  on
       average,  over  many  many  tests.   If one particular file generates a
       failure at this level, one can therefore safely presume that  it  is  a
       random  file  pulled from many thousands of similar files the generator
       might create that have the correct  distribution  of  p-values  at  all
       levels of testing and aggregation.

       To  sum  up,  use  dieharder to validate your generator (via input from
       files or an embedded stream).  Then by all means use your generator  to
       produce files or streams of random numbers.  Do not use dieharder as an
       accept/reject tool to validate the files themselves!

EXAMPLES

       To demonstrate all tests, run on the default GSL rng, enter:

         dieharder -a

       To demonstrate a test of an external generator of a raw  binary  stream
       of bits, use the stdin (raw) interface:

         cat /dev/urandom | dieharder -g 200 -a

       To use it with an ascii formatted file:

         dieharder -g 202 -f testrands.txt -a

       (testrands.txt should consist of a header such as:

        #==================================================================
        # generator mt19937_1999  seed = 1274511046
        #==================================================================
        type: d
        count: 100000
        numbit: 32
        3129711816
          85411969
        2545911541

       etc.).

       To use it with a binary file

         dieharder -g 201 -f testrands.bin -a

       or

         cat testrands.bin | dieharder -g 200 -a

       An  example that demonstrates the use of "prefixes" on the output lines
       that make it relatively easy to filter off the different parts  of  the
       output  report  and chop them up into numbers that can be used in other
       programs or in spreadsheets, try:

         dieharder -a -c ',' -D default -D prefix

DISPLAY OPTIONS

       As of version 3.x.x, dieharder  has  a  single  output  interface  that
       produces  tabular  data  per  test, with common information in headers.
       The display control options and flags can  be  used  to  customize  the
       output to your individual specific needs.

       The  options are controlled by binary flags.  The flags, and their text
       versions, are displayed if you enter:

         dieharder -F

       by itself on a line.

       The flags can be entered all at once  by  adding  up  all  the  desired
       option  flags.   For example, a very sparse output could be selected by
       adding the flags for the test_name (8) and the associated pvalues (128)
       to get 136:

         dieharder -a -D 136

       Since  the flags are cumulated from zero (unless no flag is entered and
       the default is used) you could accomplish the same display via:

         dieharder -a -D 8 -D pvalues

       Note that you can enter flags by value or by name, in any  combination.
       Because  people  use dieharder to obtain values and then with to export
       them into spreadsheets (comma separated values) or into filter scripts,
       you can chance the field separator character.  For example:

         dieharder -a -c ',' -D default -D -1 -D -2

       produces  output  that  is ideal for importing into a spreadsheet (note
       that one can subtract field values from the base set of fields provided
       by the default option as long as it is given first).

       An  interesting  option  is  the -D prefix flag, which turns on a field
       identifier prefix to make it easy to filter  out  particular  kinds  of
       data.   However,  it  is equally easy to turn on any particular kind of
       output to the exclusion of others directly by means of the flags.

       Two other flags of interest  to  novices  to  random  number  generator
       testing  are  the  -D histogram (turns on a histogram of the underlying
       pvalues, per test)  and  -D  description  (turns  on  a  complete  test
       description, per test).  These flags turn the output table into more of
       a series of "reports" of each test.

PUBLICATION RULES

       dieharder is entirely original code and can be  modified  and  used  at
       will by any user, provided that:

         a) The original copyright notices are maintained and that the source,
       including all modifications, is made publically available at  the  time
       of  any derived publication.  This is open source software according to
       the  precepts  and  spirit  of  the  Gnu  Public  License.    See   the
       accompanying    file   COPYING,   which   also   must   accompany   any
       redistribution.

         b) The primary author of the code (Robert G. Brown) is  appropriately
       acknowledged and referenced in any derived publication.  It is strongly
       suggested that George Marsaglia and the Diehard suite and  the  various
       authors  of  the  Statistical  Test  Suite  be  similarly acknowledged,
       although this suite shares no actual code with these random number test
       suites.

         c)   Full   responsibility   for   the   accuracy,  suitability,  and
       effectiveness of the program rests with the users and/or modifiers.  As
       is clearly stated in the accompanying copyright.h:

       THE  COPYRIGHT  HOLDERS  DISCLAIM  ALL  WARRANTIES  WITH REGARD TO THIS
       SOFTWARE, INCLUDING  ALL  IMPLIED  WARRANTIES  OF  MERCHANTABILITY  AND
       FITNESS,  IN  NO  EVENT  SHALL  THE COPYRIGHT HOLDERS BE LIABLE FOR ANY
       SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR  ANY  DAMAGES  WHATSOEVER
       RESULTING  FROM  LOSS  OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
       CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING  OUT  OF  OR  IN
       CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

ACKNOWLEDGEMENTS

       The  author of this suite gratefully acknowledges George Marsaglia (the
       author of the diehard test suite)  and  the  various  authors  of  NIST
       Special  Publication 800-22 (which describes the Statistical Test Suite
       for  testing   pseudorandom   number   generators   for   cryptographic
       applications),  for excellent descriptions of the tests therein.  These
       descriptions enabled this suite to be developed with a GPL.

       The author also wishes to reiterate that the academic  correctness  and
       accuracy   of   the   implementation   of   these  tests  is  his  sole
       responsibility and not that of  the  authors  of  the  Diehard  or  STS
       suites.   This is especially true where he has seen fit to modify those
       tests from their strict original descriptions.

COPYRIGHT

       GPL 2b; see the file  COPYING  that  accompanies  the  source  of  this
       program.  This is the "standard Gnu General Public License version 2 or
       any  later  version",  with  the  one   minor   (humorous)   "Beverage"
       modification listed below.  Note that this modification is probably not
       legally defensible and can be followed really pretty much according  to
       the honor rule.

       As  to my personal preferences in beverages, red wine is great, beer is
       delightful, and Coca Cola or coffee or tea or even milk  acceptable  to
       those  who for religious or personal reasons wish to avoid stressing my
       liver.

       The Beverage Modification to the GPL:

       Any satisfied user of this software shall,  upon  meeting  the  primary
       author(s)  of  this  software  for the first time under the appropriate
       circumstances, offer to buy him  or  her  or  them  a  beverage.   This
       beverage may or may not be alcoholic, depending on the personal ethical
       and moral views of the offerer.  The beverage cost need not exceed  one
       U.S.  dollar  (although  it certainly may at the whim of the offerer:-)
       and may be accepted or declined with no further obligation on the  part
       of  the  offerer.   It  is  not necessary to repeat the offer after the
       first meeting, but it can't hurt...