focal (1) dieharder.1.gz

Provided by: dieharder_3.31.1.1-1build1_amd64 bug

NAME

       dieharder - A testing and benchmarking tool for random number generators.

SYNOPSIS

       dieharder [-a] [-d dieharder test number] [-f filename] [-B]
                 [-D output flag [-D output flag] ... ] [-F] [-c separator]
                 [-g generator number or -1] [-h] [-k ks_flag] [-l]
                 [-L overlap] [-m multiply_p] [-n ntuple]
                 [-p number of p samples] [-P Xoff]
                 [-o filename] [-s seed strategy] [-S random number seed]
                 [-n ntuple] [-p number of p samples] [-o filename]
                 [-s seed strategy] [-S random number seed]
                 [-t number of test samples] [-v verbose flag]
                 [-W weak] [-X fail] [-Y Xtrategy]
                 [-x xvalue] [-y yvalue] [-z zvalue]

dieharder OPTIONS

       -a runs all the tests with standard/default options to create a
              user-controllable  report.  To control the formatting of the report, see -D below.  To control the
              power of the test (which uses default values for tsamples that  cannot  generally  be  varied  and
              psamples  which  generally  can)  see -m below as a "multiplier" of the default number of psamples
              (used only in a -a run).

       -d test number -  selects specific diehard test.

       -f filename - generators 201 or 202 permit either raw binary or
              formatted ASCII numbers to be read in from a file for testing.  generator 200 reads in raw  binary
              numbers from stdin.  Note well: many tests with default parameters require a lot of rands!  To see
              a sample of the (required) header for ASCII formatted input, run

                       dieharder -o -f example.input -t 10

              and then examine the contents of example.input.  Raw binary input reads 32 bit increments  of  the
              specified data stream.  stdin_input_raw accepts a pipe from a raw binary stream.

       -B binary mode (used with -o below) causes output rands to be written in raw binary, not formatted ascii.

       -D output flag - permits fields to be selected for inclusion in
              dieharder  output.   Each  flag  can be entered as a binary number that turns on a specific output
              field or header or by flag name; flags are aggregated.  To see all currently known flags  use  the
              -F command.

       -F - lists all known flags by name and number.

       -c table separator - where separator is e.g. ',' (CSV) or ' ' (whitespace).

       -g generator number - selects a specific generator for testing.  Using
              -g -1 causes all known generators to be printed out to the display.

       -h prints context-sensitive help -- usually Usage (this message) or a
              test synopsis if entered as e.g. dieharder -d 3 -h.

       -k ks_flag - ks_flag

              0 is fast but slightly sloppy for psamples > 4999 (default).

              1 is MUCH slower but more accurate for larger numbers of psamples.

              2  is  slower  still, but (we hope) accurate to machine precision for any number of psamples up to
              some as yet unknown numerical upper limit (it  has  been  tested  out  to  at  least  hundreds  of
              thousands).

              3 is kuiper ks, fast, quite inaccurate for small samples, deprecated.

       -l list all known tests.

       -L overlap

              1 (use overlap, default)

              0 (don't use overlap)

              in operm5 or other tests that support overlapping and non-overlapping sample modes.

       -m multiply_p - multiply default # of psamples in -a(ll) runs to crank
              up  the  resolution of failure.  -n ntuple - set ntuple length for tests on short bit strings that
              permit the length to be varied (e.g. rgb bitdist).

       -o filename - output -t count random numbers from current generator to file.

       -p count - sets the number of p-value samples per test (default 100).

       -P Xoff - sets the number of psamples that will cumulate before deciding
              that a generator is "good" and really, truly passes even a -Y 2 T2D run.  Currently the default is
              100000; eventually it will be set from AES-derived T2D test failure thresholds for fully automated
              reliable operation, but for now it is more a  "boredom"  threshold  set  by  how  long  one  might
              reasonably want to wait on any given test run.

       -S seed - where seed is a uint.  Overrides the default random seed
              selection.  Ignored for file or stdin input.

       -s strategy - if strategy is the (default) 0, dieharder reseeds (or
              rewinds)  once at the beginning when the random number generator is selected and then never again.
              If strategy is nonzero, the generator is reseeded or rewound at the beginning of EACH TEST.  If -S
              seed  was  specified,  or  a  file  is used, this means every test is applied to the same sequence
              (which is useful for validation and testing of dieharder, but  not  a  good  way  to  test  rngs).
              Otherwise a new random seed is selected for each test.

       -t count - sets the number of random entities used in each test, where
              possible.  Be warned -- some tests have fixed sample sizes; others are variable but have practical
              minimum sizes.  It is suggested you begin with the values used in -a and experiment carefully on a
              test by test basis.

       -W weak - sets the "weak" threshold to make the test(s) more or less
              forgiving during e.g. a test-to-destruction run.  Default is currently 0.005.

       -X fail - sets the "fail" threshold to make the test(s) more or less
              forgiving  during  e.g.  a  test-to-destruction  run.   Default  is  currently  0.000001, which is
              basically "certain failure of the null hypothesis", the desired  mode  of  reproducible  generator
              failure.

       -Y Xtrategy - the Xtrategy flag controls the new "test to failure" (T2F)
              modes.  These flags and their modes act as follows:

                0  -  just  run dieharder with the specified number of tsamples and psamples, do not dynamically
              modify a run based on results.  This is the way it has always run, and is the default.

                1 - "resolve ambiguity" (RA) mode.  If a test returns "weak", this is an undesired result.  What
              does  that  mean,  after all?  If you run a long test series, you will see occasional weak returns
              for a perfect generators because p is uniformly distributed and will appear in any finite interval
              from  time  to  time.  Even if a test run returns more than one weak result, you cannot be certain
              that the generator is failing.  RA mode adds psamples (usually in blocks of 100)  until  the  test
              result ends up solidly not weak or proceeds to unambiguous failure.  This is morally equivalent to
              running the test several times to see if a weak result is reproducible, but eliminates the bias of
              personal  judgement  in  the  process  since  the default failure threshold is very small and very
              unlikely to be reached by random chance even in many runs.

              This option should only be used with -k 2.

                2 - "test to destruction" mode.  Sometimes you just want to know where or if a generator will .I
              ever  fail  a  test (or test series).  -Y 2 causes psamples to be added 100 at a time until a test
              returns an overall pvalue lower than the failure  threshold  or  a  specified  maximum  number  of
              psamples (see -P) is reached.

              Note well!  In this mode one may well fail due to the alternate null hypothesis -- the test itself
              is a bad test and fails!  Many dieharder tests, despite our best efforts, are numerically unstable
              or have only approximately known target statistics or are straight up asymptotic results, and will
              eventually return a failing result even for a gold-standard generator (such as AES),  or  for  the
              hypercautious  the XOR generator with AES, threefish, kiss, all loaded at once and xor'd together.
              It is therefore safest to use this mode .I comparatively, executing a T2D run on  AES  to  get  an
              idea  of  the  test failure threshold(s) (something I will eventually do and publish on the web so
              everybody doesn't have to do it independently) and then  running  it  on  your  target  generator.
              Failure  with  numbers  of  psamples  within  an  order  of magnitude of the AES thresholds should
              probably be considered possible  test  failures,  not  generator  failures.   Failures  at  levels
              significantly  less  than  the  known  gold  standard generator failure thresholds are, of course,
              probably failures of the generator.

              This option should only be used with -k 2.

       -v verbose flag -- controls the verbosity of the output for debugging
              only.  Probably of  little  use  to  non-developers,  and  developers  can  read  the  enum(s)  in
              dieharder.h  and the test sources to see which flag values turn on output on which routines.  1 is
              result in a highly detailed trace of program activity.

       -x,-y,-z number - Some tests have parameters that can safely be varied
              from their default value.  For example, in the diehard birthdays test, one can vary the number  of
              length,  which  can  also  be  varied.  -x 2048 -y 30 alters these two values but should still run
              fine.  These parameters should be documented internally (where they exist) in the  e.g.  -d  0  -h
              visible notes.

              NOTE  WELL:  The  assessment(s)  for the rngs may, in fact, be completely incorrect or misleading.
              There are still "bad tests" in dieharder, although we are working to fix and improve them (and try
              to  document  them in the test descriptions visible with -g testnumber -h).  In particular, 'Weak'
              pvalues should occur one test in two hundred, and 'Failed' pvalues should  occur  one  test  in  a
              million with the default thresholds - that's what p MEANS.  Use them at your Own Risk!  Be Warned!

              Or  better  yet,  use  the new -Y 1 and -Y 2 resolve ambiguity or test to destruction modes above,
              comparing to similar runs on one  of  the  as-good-as-it-gets  cryptographic  generators,  AES  or
              threefish.

DESCRIPTION

       dieharder

       Welcome  to  the  current snapshot of the dieharder random number tester.  It encapsulates all of the Gnu
       Scientific Library (GSL) random number generators (rngs) as well as a number of  generators  from  the  R
       statistical  library,  hardware  sources  such  as  /dev/*random,  "gold  standard" cryptographic quality
       generators (useful for testing dieharder and for purposes of comparison to new  generators)  as  well  as
       generators  contributed  by users or found in the literature into a single harness that can time them and
       subject them to various tests for randomness.  These tests are variously drawn  from  George  Marsaglia's
       "Diehard  battery  of random number tests", the NIST Statistical Test Suite, and again from other sources
       such as personal invention, user contribution, other (open source) test suites, or the literature.

       The primary point of dieharder is to make it easy to time  and  test  (pseudo)random  number  generators,
       including  both  software  and  hardware  rngs,  with a fully open source tool.  In addition to providing
       "instant" access to testing of all built-in generators, users can choose one of three ways to test  their
       own random number generators or sources:  a unix pipe of a raw binary (presumed random) bitstream; a file
       containing a (presumed random) raw binary bitstream or formatted ascii uints  or  floats;  and  embedding
       your  generator  in  dieharder's  GSL-compatible  rng  harness  and  adding  it  to  the list of built-in
       generators.  The stdin and file input methods are described below in their own section, as  is  suggested
       "best practice" for newbies to random number generator testing.

       An  important  motivation  for  using dieharder is that the entire test suite is fully Gnu Public License
       (GPL) open source code and hence rather than being prohibited from  "looking  underneath  the  hood"  all
       users  are  openly  encouraged  to  critically  examine  the  dieharder code for errors, add new tests or
       generators or user interfaces, or use it freely as is to test their own favorite candidate  rngs  subject
       only  to the constraints of the GPL.  As a result of its openness, literally hundreds of improvements and
       bug fixes have been contributed by users to date, resulting in a far  stronger  and  more  reliable  test
       suite  than  would  have  been possible with closed and locked down sources or even open sources (such as
       STS) that lack the dynamical feedback mechanism permitting corrections to be shared.

       Even small errors in test statistics permit the alternative (usually unstated) null hypothesis to  become
       an  important  factor in rng testing -- the unwelcome possibility that your generator is just fine but it
       is the test that is failing.  One extremely useful feature of dieharder is that it is at least moderately
       self  validating.   Using the "gold standard" aes and threefish cryptographic generators, you can observe
       how these generators perform on dieharder runs to the same general degree of accuracy that  you  wish  to
       use  on  the generators you are testing.  In general, dieharder tests that consistently fail at any given
       level of precision (selected with e.g. -a -m 10) on both of the gold standard rngs (and/or the better GSL
       generators,  mt19937,  gfsr4,  taus)  are  probably  unreliable  at that precision and it would hardly be
       surprising if they failed your generator as well.

       Experts in statistics are encouraged to give the suite a try, perhaps using  any  of  the  example  calls
       below  at  first  and  then  using it freely on their own generators or as a harness for adding their own
       tests.  Novices (to either statistics or random number generator testing) are strongly encouraged to read
       the  next  section on p-values and the null hypothesis and running the test suite a few times with a more
       verbose output report to learn how the whole thing works.

QUICK START EXAMPLES

       Examples for how to set up pipe or file input are given below.  However, it is recommended  that  a  user
       play  with  some  of  the built in generators to gain familiarity with dieharder reports and tests before
       tackling their own favorite generator or file full of possibly random numbers.

       To see dieharder's default standard test report for its default generator (mt19937) simply run:

          dieharder -a

       To increase the resolution of possible failures of the standard -a(ll) test, use the -m "multiplier"  for
       the  test  default numbers of pvalues (which are selected more to make a full test run take an hour or so
       instead of days than because it is truly an exhaustive test sequence) run:

          dieharder -a -m 10

       To test a different generator (say the gold standard AES_OFB) simply specify the generator on the command
       line with a flag:

          dieharder -g 205 -a -m 10

       Arguments can be in any order.  The generator can also be selected by name:

          dieharder -g AES_OFB -a

       To apply only the diehard opso test to the AES_OFB generator, specify the test by name or number:

          dieharder -g 205 -d 5

       or

          dieharder -g 205 -d diehard_opso

       Nearly  every  aspect or field in dieharder's output report format is user-selectable by means of display
       option flags.  In addition, the field separator character can be selected by the user to make the  output
       particularly easy for them to parse (-c ' ') or import into a spreadsheet (-c ',').  Try:

          dieharder -g 205 -d diehard_opso -c ',' -D test_name -D pvalues

       to see an extremely terse, easy to import report or

          dieharder -g 205 -d diehard_opso -c ' ' -D default -D histogram -D description

       to see a verbose report good for a "beginner" that includes a full description of each test itself.

       Finally,  the  dieharder  binary is remarkably autodocumenting even if the man page is not available. All
       users should try the following commands to see what they do:

          dieharder -h

       (prints the command synopsis like the one above).

          dieharder -a -h
          dieharder -d 6 -h

       (prints the test descriptions only for -a(ll) tests or for the specific test indicated).

          dieharder -l

       (lists all known tests, including how reliable rgb thinks that they are as things stand).

          dieharder -g -1

       (lists all known rngs).

          dieharder -F

       (lists all the currently known display/output control flags used with -D).

       Both beginners and experts should be aware that the assessment provided  by  dieharder  in  its  standard
       report  should  be  regarded with great suspicion.  It is entirely possible for a generator to "pass" all
       tests as far as their individual p-values are concerned and yet to fail utterly when considering them all
       together.   Similarly,  it  is  probable that a rng will at the very least show up as "weak" on 0, 1 or 2
       tests in a typical -a(ll) run, and may even "fail" 1 test one such run in 10 or so.   To  understand  why
       this is so, it is necessary to understand something of rng testing, p-values, and the null hypothesis!

P-VALUES AND THE NULL HYPOTHESIS

       dieharder  returns  "p-values".   To  understand  what a p-value is and how to use it, it is essential to
       understand the null hypothesis, H0.

       The null hypothesis for random number generator testing is "This generator is  a  perfect  random  number
       generator,  and  for  any choice of seed produces a infinitely long, unique sequence of numbers that have
       all the expected statistical properties of random numbers, to all orders".  Note well that we  know  that
       this hypothesis is technically false for all software generators as they are periodic and do not have the
       correct entropy content for this statement to ever be true.  However, many  hardware  generators  fail  a
       priori  as  well,  as  they  contain  subtle  bias  or correlations due to the deterministic physics that
       underlies them.  Nature is often unpredictable but it is rarely random and the two  words  don't  (quite)
       mean the same thing!

       The  null  hypothesis  can  be  practically  true, however.  Both software and hardware generators can be
       "random" enough that their sequences cannot be distinguished from random ones, at  least  not  easily  or
       with  the  available  tools  (including  dieharder!)  Hence  the  null  hypothesis  is a practical, not a
       theoretically pure, statement.

       To test H0 , one uses the rng in question to generate a sequence of  presumably  random  numbers.   Using
       these numbers one can generate any one of a wide range of test statistics -- empirically computed numbers
       that are considered random samples that may or may not be covariant subject to H0, depending  on  whether
       overlapping  sequences  of  random  numbers  are used to generate successive samples while generating the
       statistic(s), drawn from a known distribution.  From a  knowledge  of  the  target  distribution  of  the
       statistic(s)  and  the  associated  cumulative distribution function (CDF) and the empirical value of the
       randomly generated statistic(s), one can read off the probability of obtaining the  empirical  result  if
       the sequence was truly random, that is, if the null hypothesis is true and the generator in question is a
       "good" random number generator!  This probability is the "p-value" for the particular test run.

       For example, to test a coin (or a sequence of bits) we might simply count the number of heads  and  tails
       in  a very long string of flips.  If we assume that the coin is a "perfect coin", we expect the number of
       heads and tails to be binomially distributed and can  easily  compute  the  probability  of  getting  any
       particular number of heads and tails.  If we compare our recorded number of heads and tails from the test
       series to this distribution and find that the probability of getting the count we obtained  is  very  low
       with, say, way more heads than tails we'd suspect the coin wasn't a perfect coin.  dieharder applies this
       very test (made mathematically precise) and many others that operate on this same principle to the string
       of random bits produced by the rng being tested to provide a picture of how "random" the rng is.

       Note  that the usual dogma is that if the p-value is low -- typically less than 0.05 -- one "rejects" the
       null hypothesis.  In a word, it is improbable that one would get the result obtained if the generator  is
       a good one.  If it is any other value, one does not "accept" the generator as good, one "fails to reject"
       the generator as bad for this particular test.  A "good random number generator" is  hence  one  that  we
       haven't been able to make fail yet!

       This  criterion  is, of course, naive in the extreme and cannot be used with dieharder!  It makes just as
       much sense to reject a generator that has p-values of 0.95 or more!  Both of  these  p-value  ranges  are
       equally  unlikely on any given test run, and should be returned for (on average) 5% of all test runs by a
       perfect random number generator.  A generator that fails to produce p-values less than  0.05  5%  of  the
       time  it  is tested with different seeds is a bad random number generator, one that fails the test of the
       null hypothesis.  Since dieharder returns over 100 pvalues by default per  test,  one  would  expect  any
       perfectly  good rng to "fail" such a naive test around five times by this criterion in a single dieharder
       run!

       The p-values themselves, as it turns out, are test statistics!   By  their  nature,  p-values  should  be
       uniformly  distributed  on  the  range  0-1.  In 100+ test runs with independent seeds, one should not be
       surprised to obtain 0, 1, 2, or even (rarely) 3 p-values less than 0.01.  On the other hand  obtaining  7
       p-values  in  the range 0.24-0.25, or seeing that 70 of the p-values are greater than 0.5 should make the
       generator highly suspect!  How can a user determine when a test is producing "too many" of any particular
       value range for p?  Or too few?

       Dieharder  does  it  for you, automatically.  One can in fact convert a set of p-values into a p-value by
       comparing their distribution to the expected one, using a Kolmogorov-Smirnov test  against  the  expected
       uniform distribution of p.

       These  p-values  obtained  from  looking  at  the  distribution  of  p-values should in turn be uniformly
       distributed and could in principle be subjected to still more KS tests in aggregate.  The distribution of
       p-values  for  a  good generator should be idempotent, even across different test statistics and multiple
       runs.

       A failure of the distribution of p-values at any level of aggregation signals trouble.  In fact,  if  the
       p-values  of  any  given  test  are subjected to a KS test, and those p-values are then subjected to a KS
       test, as we add more p-values to either level  we  will  either  observe  idempotence  of  the  resulting
       distribution  of p to uniformity, or we will observe idempotence to a single p-value of zero!  That is, a
       good generator will produce a roughly uniform distribution of p-values, in the specific sense that the p-
       values  of  the  distributions of p-values are themselves roughly uniform and so on ad infinitum, while a
       bad generator will produce a non-uniform distribution of p-values, and as more p-values  drawn  from  the
       non-uniform  distribution  are  added  to  its  KS  test,  at  some  point the failure will be absolutely
       unmistakeable as the resulting p-value approaches 0 in the limit.  Trouble indeed!

       The question is, trouble with what?  Random number tests are themselves  complex  computational  objects,
       and  there  is a probability that their code is incorrectly framed or that roundoff or other numerical --
       not methodical -- errors are contributing to a distortion of the distribution of  some  of  the  p-values
       obtained.   This  is  not  an idle observation; when one works on writing random number generator testing
       programs, one is always testing the tests themselves with "good" (we hope) random  number  generators  so
       that  egregious failures of the null hypothesis signal not a bad generator but an error in the test code.
       The null hypothesis above is correctly framed from a theoretical point of  view,  but  from  a  real  and
       practical point of view it should read: "This generator is a perfect random number generator, and for any
       choice of seed produces a infinitely long,  unique  sequence  of  numbers  that  have  all  the  expected
       statistical  properties  of  random  numbers,  to  all orders and this test is a perfect test and returns
       precisely correct p-values from the test computation."  Observed "failure" of this joint null  hypothesis
       H0'  can  come  from failure of either or both of these disjoint components, and comes from the second as
       often or more often than the first  during  the  test  development  process.   When  one  cranks  up  the
       "resolution"  of the test (discussed next) to where a generator starts to fail some test one realizes, or
       should realize, that development never ends and that new test regimes will always reveal new failures not
       only of the generators but of the code.

       With  that  said,  one of dieharder's most significant advantages is the control that it gives you over a
       critical test parameter.  From the remarks above, we can see that we should feel very uncomfortable about
       "failing"  any  given  random  number generator on the basis of a 5%, or even a 1%, criterion, especially
       when we apply a test suite like dieharder that returns over 100 (and climbing) distinct test p-values  as
       of the last snapshot.  We want failure to be unambiguous and reproducible!

       To  accomplish  this,  one can simply crank up its resolution.  If we ran any given test against a random
       number generator and it returned a p-value of (say) 0.007328, we'd be perfectly justified in wondering if
       it  is  really  a  good generator.  However, the probability of getting this result isn't really all that
       small -- when one uses dieharder for hours at a time numbers  like  this  will  definitely  happen  quite
       frequently  and  mean  nothing.   If  one  runs the same test again (with a different seed or part of the
       random sequence) and gets a p-value of 0.009122, and a third time and gets 0.002669 -- well, that's three
       1%  (or  less)  shots  in  a  row and that should happen only one in a million times.  One way to clearly
       resolve failures, then, is to increase the number of p-values generated in a test  run.   If  the  actual
       distribution  of  p being returned by the test is not uniform, a KS test will eventually return a p-value
       that is not some ambiguous 0.035517 but is instead 0.000000, with the latter produced time after time  as
       we rerun.

       For  this  reason,  dieharder  is  extremely  conservative  about  announcing rng "weakness" or "failure"
       relative to any given test.  It's internal criterion for these things are currently p < 0.5% or p > 99.5%
       weakness  (at the 1% level total) and a considerably more stringent criterion for failure: p < 0.05% or p
       > 99.95%.  Note well that the ranges are symmetric -- too high a value of p is just as bad (and unlikely)
       as  too  low,  and  it  is critical to flag it, because it is quite possible for a rng to be too good, on
       average, and not to produce enough low p-values on the full spectrum of dieharder tests.  This  is  where
       the  final kstest is of paramount importance, and where the "histogram" option can be very useful to help
       you visualize the failure in the distribution of p -- run e.g.:

         dieharder [whatever] -D default -D histogram

       and you will see a crude ascii histogram of the pvalues that failed (or passed) any given level of test.

       Scattered reports of weakness or marginal failure in a preliminary -a(ll) run  should  therefore  not  be
       immediate cause for alarm.  Rather, they are tests to repeat, to watch out for, to push the rng harder on
       using the -m option to -a or simply increasing -p for a specific test.  Dieharder permits one to increase
       the  number of p-values generated for any test, subject only to the availability of enough random numbers
       (for file based tests) and time, to make failures unambiguous.  A test that is truly weak at -p 100  will
       almost  always  fail  egregiously at some larger value of psamples, be it -p 1000 or -p 100000.  However,
       because dieharder is a research tool and is under perpetual  development  and  testing,  it  is  strongly
       suggested  that  one  always consider the alternative null hypothesis -- that the failure is a failure of
       the test code in dieharder itself in some limit of large numbers -- and take at least some steps (such as
       running  the  same test at the same resolution on a "gold standard" generator) to ensure that the failure
       is indeed probably in the rng and not the dieharder code.

       Lacking a source of perfect random numbers to use as a reference, validating the tests themselves is  not
       easy  and always leaves one with some ambiguity (even aes or threefish).  During development the best one
       can usually do is to rely heavily on these "presumed good" random number generators.  There are a  number
       of  generators  that  we  have  theoretical  reasons  to  expect  to  be extraordinarily good and to lack
       correlations out to some known underlying dimensionality, and that also test  out  extremely  well  quite
       consistently.  By using several such generators and not just one, one can hope that those generators have
       (at the very least) different correlations and should not all uniformly fail a test in the same  way  and
       with  the  same  number  of  p-values.   When all of these generators consistently fail a test at a given
       level, I tend to suspect that the problem is in the test code, not the generators, although  it  is  very
       difficult to be certain, and many errors in dieharder's code have been discovered and ultimately fixed in
       just this way by myself or others.

       One advantage of dieharder is that it has a number of these "good generators" immediately  available  for
       comparison  runs,  courtesy of the Gnu Scientific Library and user contribution (notably David Bauer, who
       kindly encapsulated aes and threefish).  I use AES_OFB, Threefish_OFB, mt19937_1999, gfsr4,  ranldx2  and
       taus2  (as  well  as  "true  random"  numbers from random.org) for this purpose, and I try to ensure that
       dieharder will "pass" in particular the -g 205 -S 1 -s 1 generator at any reasonable  p-value  resolution
       out to -p 1000 or farther.

       Tests  (such  as  the  diehard operm5 and sums test) that consistently fail at these high resolutions are
       flagged as being "suspect" -- possible failures of the  alternative  null  hypothesis  --  and  they  are
       strongly deprecated!  Their results should not be used to test random number generators pending agreement
       in the statistics and random number community that those tests are in fact  valid  and  correct  so  that
       observed failures can indeed safely be attributed to a failure of the intended null hypothesis.

       As  I  keep emphasizing (for good reason!) dieharder is community supported.  I therefore openly ask that
       the users of dieharder who are expert in  statistics  to  help  me  fix  the  code  or  algorithms  being
       implemented.   I  would  like  to  see  this test suite ultimately be validated by the general statistics
       community in hard use in an open environment, where every  possible  failure  of  the  testing  mechanism
       itself  is  subject  to  scrutiny and eventual correction.  In this way we will eventually achieve a very
       powerful suite of tools indeed, ones that may well give us  very  specific  information  not  just  about
       failure but of the mode of failure as well, just how the sequence tested deviates from randomness.

       Thus  far, dieharder has benefitted tremendously from the community.  Individuals have openly contributed
       tests, new generators to be tested, and fixes for existing tests that were revealed  by  their  own  work
       with  the testing instrument.  Efforts are underway to make dieharder more portable so that it will build
       on more platforms and faster so that more thorough testing can be done.  Please feel free to participate.

FILE INPUT

       The simplest way to use dieharder with an external generator that produces raw binary  (presumed  random)
       bits  is  to  pipe  the  raw  binary output from this generator (presumed to be a binary stream of 32 bit
       unsigned integers) directly into dieharder, e.g.:

         cat /dev/urandom | ./dieharder -a -g 200

       Go ahead and try this example.  It will run the entire dieharder suite of tests on the stream produced by
       the linux built-in generator /dev/urandom (using /dev/random is not recommended as it is too slow to test
       in a reasonable amount of time).

       Alternatively, dieharder can be used to test files of numbers  produced  by  a  candidate  random  number
       generators:

         dieharder -a -g 201 -f random.org_bin

       for raw binary input or

         dieharder -a -g 202 -f random.org.txt

       for formatted ascii input.

       A formatted ascii input file can accept either uints (integers in the range 0 to 2^31-1, one per line) or
       decimal uniform deviates with at least ten significant digits (that can be multiplied by UINT_MAX =  2^32
       to  produce  a uint without dropping precition), also one per line.  Floats with fewer digits will almost
       certainly fail bitlevel tests, although they may pass some of the tests that act on uniform deviates.

       Finally, one can fairly easily wrap any generator in the same (GSL) random number harness used internally
       by  dieharder  and  simply  test  it  the  same  way one would any other internal generator recognized by
       dieharder.  This is strongly recommended where it is possible, because dieharder needs to use  a  lot  of
       random  numbers  to thoroughly test a generator.  A built in generator can simply let dieharder determine
       how many it needs and generate them on demand, where a file that is too small will  "rewind"  and  render
       the test results where a rewind occurs suspect.

       Note well that file input rands are delivered to the tests on demand, but if the test needs more than are
       available it simply rewinds the file and cycles through  it  again,  and  again,  and  again  as  needed.
       Obviously  this  significantly  reduces the sample space and can lead to completely incorrect results for
       the p-value histograms unless there are enough rands to run EACH test without repetition (it is  harmless
       to reuse the sequence for different tests).  Let the user beware!

BEST PRACTICE

       A  frequently  asked  question  from new users wishing to test a generator they are working on for fun or
       profit (or both) is "How should I get its output into dieharder?"  This  is  a  nontrivial  question,  as
       dieharder  consumes  enormous numbers of random numbers in a full test cycle, and then there are features
       like -m 10 or -m 100 that let one effortlessly demand 10 or 100 times as many to stress a  new  generator
       even more.

       Even  with large file support in dieharder, it is difficult to provide enough random numbers in a file to
       really make dieharder happy.  It is therefore strongly suggested that you either:

       a) Edit the output stage of your random number generator and get it to write its production to stdout  as
       a random bit stream -- basically create 32 bit unsigned random integers and write them directly to stdout
       as e.g. char data or raw binary.  Note that this is not the same as writing raw  floating  point  numbers
       (that  will not be random at all as a bitstream) and that "endianness" of the uints should not matter for
       the null hypothesis of a "good" generator, as random bytes are random in any order.  Crank the  generator
       and feed this stream to dieharder in a pipe as described above.

       b)  Use  the  samples  of  GSL-wrapped  dieharder rngs to similarly wrap your generator (or calls to your
       generator's hardware interface).  Follow the examples in the ./dieharder source directory to add it as  a
       "user" generator in the command line interface, rebuild, and invoke the generator as a "native" dieharder
       generator (it should appear in the list produced by -g -1 when done correctly).  The advantage  of  doing
       it  this  way is that you can then (if your new generator is highly successful) contribute it back to the
       dieharder project if you wish!  Not to mention the fact that it makes testing it very easy.

       Most users will probably go with option a) at least initially, but be aware that b)  is  probably  easier
       than  you  think.   The  dieharder  maintainers  may  be  able to give you a hand with it if you get into
       trouble, but no promises.

WARNING!

       A warning for those who are testing files of random numbers.  dieharder  is  a  tool  that  tests  random
       number generators, not files of random numbers!  It is extremely inappropriate to try to "certify" a file
       of random numbers as being random just because it fails to "fail" any of the dieharder tests  in  e.g.  a
       dieharder  -a run.  To put it bluntly, if one rejects all such files that fail any test at the 0.05 level
       (or any other), the one thing one can be certain of is that the files in question are not  random,  as  a
       truly random sequence would fail any given test at the 0.05 level 5% of the time!

       To  put  it  another  way, any file of numbers produced by a generator that "fails to fail" the dieharder
       suite should be considered "random", even if it contains sequences that might well "fail" any given  test
       at  some  specific cutoff.  One has to presume that passing the broader tests of the generator itself, it
       was determined that the p-values for the test involved was globally correctly distributed, so  that  e.g.
       failure  at  the  0.01 level occurs neither more nor less than 1% of the time, on average, over many many
       tests.  If one particular file generates a failure at this level, one can therefore safely  presume  that
       it  is a random file pulled from many thousands of similar files the generator might create that have the
       correct distribution of p-values at all levels of testing and aggregation.

       To sum up, use dieharder to validate your generator (via input from files or an embedded  stream).   Then
       by  all  means use your generator to produce files or streams of random numbers.  Do not use dieharder as
       an accept/reject tool to validate the files themselves!

EXAMPLES

       To demonstrate all tests, run on the default GSL rng, enter:

         dieharder -a

       To demonstrate a test of an external generator of a raw binary  stream  of  bits,  use  the  stdin  (raw)
       interface:

         cat /dev/urandom | dieharder -g 200 -a

       To use it with an ascii formatted file:

         dieharder -g 202 -f testrands.txt -a

       (testrands.txt should consist of a header such as:

        #==================================================================
        # generator mt19937_1999  seed = 1274511046
        #==================================================================
        type: d
        count: 100000
        numbit: 32
        3129711816
          85411969
        2545911541

       etc.).

       To use it with a binary file

         dieharder -g 201 -f testrands.bin -a

       or

         cat testrands.bin | dieharder -g 200 -a

       An  example  that  demonstrates the use of "prefixes" on the output lines that make it relatively easy to
       filter off the different parts of the output report and chop them up into numbers that  can  be  used  in
       other programs or in spreadsheets, try:

         dieharder -a -c ',' -D default -D prefix

DISPLAY OPTIONS

       As  of  version  3.x.x, dieharder has a single output interface that produces tabular data per test, with
       common information in headers.  The display control options and flags can be used to customize the output
       to your individual specific needs.

       The  options  are  controlled  by binary flags.  The flags, and their text versions, are displayed if you
       enter:

         dieharder -F

       by itself on a line.

       The flags can be entered all at once by adding up all the desired option  flags.   For  example,  a  very
       sparse  output  could  be  selected  by adding the flags for the test_name (8) and the associated pvalues
       (128) to get 136:

         dieharder -a -D 136

       Since the flags are cumulated from zero (unless no flag is entered and the default  is  used)  you  could
       accomplish the same display via:

         dieharder -a -D 8 -D pvalues

       Note  that  you can enter flags by value or by name, in any combination.  Because people use dieharder to
       obtain values and then with to export them into spreadsheets (comma  separated  values)  or  into  filter
       scripts, you can chance the field separator character.  For example:

         dieharder -a -c ',' -D default -D -1 -D -2

       produces  output  that is ideal for importing into a spreadsheet (note that one can subtract field values
       from the base set of fields provided by the default option as long as it is given first).

       An interesting option is the -D prefix flag, which turns on a field identifier prefix to make it easy  to
       filter  out  particular  kinds  of  data.   However, it is equally easy to turn on any particular kind of
       output to the exclusion of others directly by means of the flags.

       Two other flags of interest to novices to random number generator testing are the -D histogram (turns  on
       a  histogram  of  the  underlying  pvalues,  per  test)  and  -D  description  (turns  on a complete test
       description, per test).  These flags turn the output table into more of a series  of  "reports"  of  each
       test.

PUBLICATION RULES

       dieharder is entirely original code and can be modified and used at will by any user, provided that:

         a)  The  original copyright notices are maintained and that the source, including all modifications, is
       made publically available at the time of any derived publication.  This is open source software according
       to the precepts and spirit of the Gnu Public License.  See the accompanying file COPYING, which also must
       accompany any redistribution.

         b) The primary author of the code (Robert G. Brown) is appropriately acknowledged and referenced in any
       derived  publication.   It  is  strongly  suggested  that  George Marsaglia and the Diehard suite and the
       various authors of the Statistical Test Suite be similarly acknowledged, although this  suite  shares  no
       actual code with these random number test suites.

         c)  Full  responsibility for the accuracy, suitability, and effectiveness of the program rests with the
       users and/or modifiers.  As is clearly stated in the accompanying copyright.h:

       THE COPYRIGHT HOLDERS DISCLAIM ALL WARRANTIES  WITH  REGARD  TO  THIS  SOFTWARE,  INCLUDING  ALL  IMPLIED
       WARRANTIES  OF  MERCHANTABILITY  AND  FITNESS,  IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY
       SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA  OR
       PROFITS,  WHETHER  IN  AN  ACTION  OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
       CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

ACKNOWLEDGEMENTS

       The author of this suite gratefully acknowledges George Marsaglia (the author of the diehard test  suite)
       and  the  various  authors of NIST Special Publication 800-22 (which describes the Statistical Test Suite
       for testing pseudorandom number generators for cryptographic applications), for excellent descriptions of
       the tests therein.  These descriptions enabled this suite to be developed with a GPL.

       The  author  also wishes to reiterate that the academic correctness and accuracy of the implementation of
       these tests is his sole responsibility and not that of the authors of the Diehard or STS suites.  This is
       especially true where he has seen fit to modify those tests from their strict original descriptions.

       GPL  2b;  see  the  file  COPYING that accompanies the source of this program.  This is the "standard Gnu
       General Public License version 2 or  any  later  version",  with  the  one  minor  (humorous)  "Beverage"
       modification  listed  below.   Note  that this modification is probably not legally defensible and can be
       followed really pretty much according to the honor rule.

       As to my personal preferences in beverages, red wine is great, beer  is  delightful,  and  Coca  Cola  or
       coffee  or  tea  or  even  milk  acceptable  to those who for religious or personal reasons wish to avoid
       stressing my liver.

       The Beverage Modification to the GPL:

       Any satisfied user of this software shall, upon meeting the primary author(s) of this  software  for  the
       first  time  under  the  appropriate  circumstances,  offer  to  buy him or her or them a beverage.  This
       beverage may or may not be alcoholic, depending on the personal ethical and moral views of  the  offerer.
       The  beverage  cost  need  not  exceed  one  U.S.  dollar  (although  it certainly may at the whim of the
       offerer:-) and may be accepted or declined with no further obligation on the part of the offerer.  It  is
       not necessary to repeat the offer after the first meeting, but it can't hurt...