NAME

       Data::Stream::Bulk - N at a time iteration API

VERSION

       version 0.11

SYNOPSIS

               # get a bulk stream from somewere
               my $s = Data::Stream::Bulk::Foo->new( ... );

               # can be used like this:
               until ( $s->is_done ) {
                       foreach my $item ( $s->items ) {
                               process($item);
                       }
               }

               # or like this:
               while( my $block = $s->next ) {
                       foreach my $item ( @$block ) {
                               process($item);
                       }
               }

DESCRIPTION

       This module tries to find middle ground between one at a time and all at once processing
       of data sets.

       The purpose of this module is to avoid the overhead of implementing an iterative api when
       this isn't necessary, without breaking forward compatibility in case that becomes
       necessary later on.

       The API optimizes for when a data set typically fits in memory and is returned as an
       array, but the consumer cannot assume that the data set is bounded.

       The API is destructive in order to minimize the chance that resultsets are leaked due to
       improper usage.

API

   Required Methods
       The API requires two methods to be implemented:

       is_done
           Should return true if the stream is exhausted.

           As long as this method returns a false value (not done) "next" could potentially
           return another block.

       next
           Returns the next block.

           Note that "next" is not guaranteed to return an array reference, even if "is_done"
           returned false prior to calling it.

   Convenience Methods
       items
           This method calls "next" and dereferences the result if there are pending items.

       all Force evaluation of the entire resultset.

           Note that for large data sets this might cause swap thrashing of various other
           undesired effects. Use with caution.

       cat @streams
           Concatenates this stream with @streams, returning a single stream.

       list_cat @tail
           Returns a possibly cleaned up list of streams.

           Used by "cat".

           Overridden by Data::Stream::Bulk::Array, Data::Stream::Bulk::Cat and
           Data::Stream::Bulk::Nil to implement some simple short circuiting.

       filter $filter
           Applies a per-block block filter to the stream.

           Returns a possibly new stream with the filtering layered.

           $filter is invoked once per block and should return an array reference to the filtered
           block.

       chunk $chunk_size
           Chunks the input stream so that each block returned by "next" will have at least
           $chunk_size items.

       loaded
           Should be overridden to return true if all the items are already realized (e.g.  in
           the case of Data::Stream::Bulk::Array).

           Returns false by default.

           When true calling "all" is supposed to be safe (memory usage should be in the same
           order of magnitude as stream's own usage).

           This is typically useful when tranforming an array is easier than transorming a stream
           (e.g. optional duplicate filtering).

CLASSES

       Data::Stream::Bulk::Array
           This class is not a stream at all, but just one block. When the data set easily fits
           in memory this class can be used, while retaining forward compatibility with larger
           data sets.

       Data::Stream::Bulk::Callback
           Callback driven iteration.

       Data::Stream::Bulk::Chunked
           Wrapper to return larger blocks from an existing stream.

       Data::Stream::Bulk::DBI
           Bulk fetching of data from DBI statement handles.

       Data::Stream::Bulk::DBIC
           DBIx::Class::ResultSet iteration.

       Data::Stream::Bulk::FileHandle
           Iterates over lines in a text file.

       Data::Stream::Bulk::Nil
           An empty result set.

       Data::Stream::Bulk::Cat
           A concatenation of several streams.

       Data::Stream::Bulk::Filter
           A filter wrapping a stream.

SEE ALSO

       HOP::Stream, Iterator, Class::Iterator etc for one by one iteration

       DBI, DBIx::Class::ResultSet

       POE::Filter

       Data::Page

       Parallel::Iterator

       <http://en.wikipedia.org/wiki/MapReduce>, LISP, and all that other kool aid

TODO

       Sorted streams
           Add a hint for sorted streams (like "loaded" but as an attribute in the base role).

           Introduce a "merge" operation for merging of sorted streams.

           Optimize "unique" to make use of sorting hints for constant space uniquing.

       More utility functions
           To assist in proccessing and creating streams.

       Coercion tables
           Moose::Util::TypeConstraints

VERSION CONTROL

       This module is maintained using git. You can get the latest version from
       <http://github.com/nothingmuch/data-stream-bulk/>.

AUTHOR

       Yuval Kogman <nothingmuch@woobling.org>

COPYRIGHT AND LICENSE

       This software is copyright (c) 2012 by Yuval Kogman.

       This is free software; you can redistribute it and/or modify it under the same terms as
       the Perl 5 programming language system itself.