NAME

       SQL::SplitStatement::Tokenizer - A simple SQL tokenizer.

SYNOPSIS

        use SQL::SplitStatement::Tokenizer qw(tokenize_sql);

        my $query= q{SELECT 1 + 1};
        my @tokens= tokenize_sql($query);

        # @tokens now contains ('SELECT', ' ', '1', ' ', '+', ' ', '1')

DESCRIPTION

       SQL::SplitStatement::Tokenizer is a simple tokenizer for SQL queries. It does not claim to
       be a parser or query verifier. It just creates sane tokens from a valid SQL query.

       It supports SQL with comments like:

        -- This query is used to insert a message into
        -- logs table
        INSERT INTO log (application, message) VALUES (?, ?)

       Also supports '', "" and "\'" escaping methods, so tokenizing queries like the one below
       should not be a problem:

        INSERT INTO log (application, message)
        VALUES ('myapp', 'Hey, this is a ''single quoted string''!')

API

       tokenize_sql
               use SQL::SplitStatement::Tokenizer qw(tokenize_sql);

               my @tokens = tokenize_sql($query);
               my $tokens = tokenize_sql($query);

               $tokens = tokenize_sql( $query, $remove_white_tokens );

           "tokenize_sql" can be imported to current namespace on request. It receives a SQL
           query, and returns an array of tokens if called in list context, or an arrayref if
           called in scalar context.

           If $remove_white_tokens is true, white spaces only tokens will be removed from result.

ACKNOWLEDGEMENTS

       ·   Igor Sutton Lopes for writing SQL::Tokenizer, which this was forked from.

       ·   Evan Harris, for implementing Shell comment style and SQL operators.

       ·   Charlie Hills, for spotting a lot of important issues I haven't thought.

       ·   Jonas Kramer, for fixing MySQL quoted strings and treating dot as punctuation
           character correctly.

       ·   Emanuele Zeppieri, for asking to fix SQL::Tokenizer to support dollars as well.

       ·   Nigel Metheringham, for extending the dollar signal support.

       ·   Devin Withers, for making it not choke on CR+LF in comments.

       ·   Luc Lanthier, for simplifying the regex and make it not choke on backslashes.

AUTHOR

       Copyright (c) 2007, 2008, 2009, 2010, 2011 Igor Sutton Lopes "<IZUT@cpan.org>". All rights
       reserved.

       Copyright (c) 2021 Veesh Goldman "<veesh@cpan.org>"

       This module is free software; you can redistribute it and/or modify it under the same
       terms as Perl itself.