Provided by: libcatmandu-perl_1.0700-1_all 
NAME
Catmandu::Fix - a Catmandu class used for data transformations
SYNOPSIS
# From the command line
$ catmandu convert JSON --fix 'add_field(foo,bar)' < data.json
$ catmandu convert YAML --fix 'upcase(job); remove_field(test)' < data.yml
$ catmandu convert CSV --fix 'sort_field(tags)' < data.csv
$ catmandu run /tmp/myfixes.txt
$ catmandu convert OAI --url http://biblio.ugent.be/oai --fix /tmp/myfixes.txt
# From Perl
use Catmandu;
my $fixer = Catmandu->fixer('upcase(job)','remove_field(test)');
my $fixer = Catmandu->fixer('/tmp/myfixes.txt');
# Convert data
my $arr = $fixer->fix([ ... ]);
my $hash = $fixer->fix({ ... });
my $importer = Catmandu->importer('YAML', file => 'data.yml');
my $fixed_importer = $fixer->fix($importer);
# Inline fixes
use Catmandu::Fix::upcase as => 'my_upcase';
use Catmandu::Fix::remove_field as => 'my_remove';
my $hash = { 'job' => 'librarian' , deep => { nested => '1'} };
my_upcase($hash,'job');
my_remove($hash,'deep.nested');
DESCRIPTION
A Catmandu::Fix is a Perl package that can transform data. These packages are used for
easy data manipulation by non programmers. The main intention is to use fixes on the
command line or in Fix scripts. A small DSL language is available to execute many Fix
command on a stream of data.
When a "fix" argument is given to a Catmandu::Importer, Catmandu::Exporter or
Catmandu::Store then the transformations are executed on every item in the stream.
FIX LANGUAGE
A Fix script is a collection of one or more Fix commands. The fixes are executed on every
record in the dataset. If this command is executed on the command line:
$ catmandu convert JSON --fix 'upcase(title); add_field(deep.nested.field,1)' < data.json
then all the title fields will be upcased and a new deeply nested field will be added:
{ "title":"foo" }
{ "title":"bar" }
becomes:
{ "title":"FOO" , "deep":{"nested":{"field":1}} }
{ "title":"BAR" , "deep":{"nested":{"field":1}} }
Using the command line, Fix commands need a semicolon (;) as separator. All these commands
can also be written into a Fix script where semicolons are not required:
$ catmandu convert JSON --fix script.fix < data.json
where "script.fix" contains:
upcase(title)
add_field(deep.nested.field,1)
Conditionals can be used to provide the logic when to execute fixes:
if exists(error)
set_field(valid, 0)
end
if exists(error)
set_field(is_valid, 0)
elsif exists(warning)
set_field(is_valid, 1)
log(...)
else
set_field(is_valid, 1)
end
unless all_match(title, "PERL")
add_field(is_perl, "noooo")
end
exists(error) and set_field(is_valid, 0)
exists(error) && set_field(is_valid, 0)
exists(title) or log('title missing')
exists(title) || log('title missing')
Binds are used to manipulate the context in which Fixes are executed. E.g. execute a fix
on every item in a list:
# 'demo' is an array of hashes
bind list(path:demo)
add_field(foo,bar)
end
# do is an alias for bind
do list(path:demo)
add_field(foo,bar)
end
To delete records from a stream of data the "reject" Fix can be used:
reject() # Reject all in the stream
if exists(foo)
reject() # Reject records that contain a 'foo' field
end
reject exists(foo) # Reject records that contain a 'foo' field
The opposite of "reject" is "select":
select() # Keep all records in the stream
select exists(foo) # Keep only the records that contain a 'foo' field
Comments in Fix scripts are all lines (or parts of a line) that start with a hash (#):
# This is ignored
add_field(test,123) # This is also a comment
PATHS
Most of the Fix commandsuse paths to point to values in a data record. E.g. 'foo.2.bar' is
a key 'bar' which is the 3-rd value of the key 'foo'.
A special case is when you want to point to all items in an array. In this case the
wildcard '*' can be used. E.g. 'foo.*' points to all the items in the 'foo' array.
For array values there are special wildcards available:
* $append - Add a new item at the end of an array
* $prepend - Add a new item at the start of an array
* $first - Syntactic sugar for index '0' (the head of the array)
* $last - Syntactic sugar for index '-1' (the tail of the array)
E.g.
# Create { mods => { titleInfo => [ { 'title' => 'a title' }] } };
add_field('mods.titleInfo.$append.title', 'a title');
# Create { mods => { titleInfo => [ { 'title' => 'a title' } , { 'title' => 'another title' }] } };
add_field('mods.titleInfo.$append.title', 'another title');
# Create { mods => { titleInfo => [ { 'title' => 'foo' } , { 'title' => 'another title' }] } };
add_field('mods.titleInfo.$first.title', 'foo');
# Create { mods => { titleInfo => [ { 'title' => 'foo' } , { 'title' => 'bar' }] } };
add_field('mods.titleInfo.$last.title', 'bar');
OPTIONS
fixes
An array of fixes. Catmandu::Fix which will execute every fix in consecutive order. A fix
can be the name of a Catmandu::Fix::* routine, or the path to a plain text file containing
all the fixes to be executed. Required.
preprocess
If set to 1, fix files or inline fixes will first be preprocessed as a moustache template.
See "variables" below for an example. Default is 0, no preprocessing.
variables
An optional hashref of variables that are used to preprocess the fix files or inline fixes
as a moustache template. Setting the "variables" option also sets "preprocess" to 1.
my $fixer = Catmandu::Fix->new(
variables => {x => 'foo', y => 'bar'},
fixes => ['add_field({{x}},{{y}})'],
);
my $data = {};
$fixer->fix($data);
# $data is now {foo => 'bar'}
METHODS
fix(HASH)
Execute all the fixes on a HASH. Returns the fixed HASH.
fix(ARRAY)
Execute all the fixes on every element in the ARRAY. Returns an ARRAY of fixes.
fix(Catmandu::Iterator)
Execute all the fixes on every item in an Catmandu::Iterator. Returns a (lazy) iterator on
all the fixes.
fix(sub {})
Executes all the fixes on a generator function. Returns a new generator with fixed data.
log
Return the current logger. See Catmandu for activating the logger in your main code.
CODING
One can extend the Fix language by creating own custom-made fixes. Two methods are
available to create an own Fix function:
* Quick and easy: create a class that implements a C<fix> method.
* Advanced: create a class that emits Perl code that will be evaled by the Fix module.
Both methods will be explained shortly.
Quick and easy
A Fix function is a Perl class in the "Catmandu::Fix" namespace that implements a "fix"
method. The "fix" methods accepts a Perl hash as input and returns a (fixed) Perl hash as
output. As an example, the code belows implements the "meow" Fix which inserts a 'meow'
field with value 'purrrrr'.
package Catmandu::Fix::meow;
use Moo;
sub fix {
my ($self,$data) = @_;
$data->{meow} = 'purrrrr';
$data;
}
1;
Given this Perl class, the following fix statement can be used in your application:
# Will add 'meow' = 'purrrrr' to the data
meow()
Use the quick and easy method when your fixes are not dependent on reading or writing data
from/to a JSON path. Your Perl classes need to implement their own logic to read or write
data into the given Perl hash.
Fix arguments are passed as arguments to the "new" function of the Perl class. As in
# In the fix file...
meow('test123', -count => 4)
# ...will be translated into this pseudo code
my $fix = Catmandu::Fix::meow->new('test123', '-count', 4);
Using Moo these arguments can be catched with Catmandu::Fix::Has package:
package Catmandu::Fix::meow;
use Catmandu::Sane;
use Moo;
use Catmandu::Fix::Has;
has msg => (fix_arg => 1); # required parameter 1
has count => (fix_opt => 1, default => sub { 4 }); # optional parameter 'count' with default value 4
sub fix {
my ($self,$data) = @_;
$data->{meow} = $self->msg x $self->count;
$data;
}
1;
Using this code the fix statement can be used like:
# Will add 'meow' = 'purrpurrpurrpurr'
meow('purr', -count => 4)
Advanced
The advanced method is required when one needs to read or write values from/to deeply
nested JSON paths. One could parse JSON paths using the quick and easy Perl class above,
but this would require a lot of inefficient for-while loops. The advanced method emits
Perl code that gets compiled. This compiled code is evaled against all Perl hashes in the
unput.The best way to learn this method is by inspecting some example Fix commands.
To ease the implementation of Fixed that emit Perl code some helper methods are created.
Many Fix functions require a transformation of one or more values on a JSON Path. The
Catmandu::Fix::SimpleGetValue provides an easy way to create such as script. In the
example below we'll set the value at a JSON Path to 'purrrrr':
package Catmandu::Fix::purrrrr;
use Catmandu::Sane;
use Moo;
use Catmandu::Fix::Has;
has path => (fix_arg => 1);
with 'Catmandu::Fix::SimpleGetValue';
sub emit_value {
my ($self, $var, $fixer) = @_;
"${var} = 'purrrrr';";
}
1;
Run this command as:
# Set the value(s) of an existing path to 'purrr'
purrrrr(my.deep.nested.path)
purrrrr(all.my.values.*)
Notice how the "emit_value" of the Catmandu::Fix::purrrrr package returns Perl code and
doesn't operate directy on the Perl data. The parameter $var contains only the name of a
temporary variable that will hold the value of the JSON path after compiling the code into
Perl.
Use Catmandu::Fix::Has to add more arguments to this fix:
package Catmandu::Fix::purrrrr;
use Catmandu::Sane;
use Moo;
use Catmandu::Fix::Has;
has path => (fix_arg => 1);
has msg => (fix_opt => 1 , default => sub { 'purrrrr' });
with 'Catmandu::Fix::SimpleGetValue';
sub emit_value {
my ($self, $var, $fixer) = @_;
my $msg = $fixer->emit_string($self->msg);
"${var} = ${msg};";
}
1;
Run this command as:
# Set the value(s) of an existing path to 'okido'
purrrrr(my.deep.nested.path, -msg => 'okido')
purrrrr(all.my.values.*, -msg => 'okido')
Notice how the "emit_value" needs to quote the "msg" option using the emit_string
function.
INTERNAL METHODS
This module provides several methods for writing fix packages. Usage can best be
understood by reading the code of existing fix packages.
capture
emit_block
emit_clone
emit_clear_hash_ref
emit_create_path
emit_declare_vars
emit_delete_key
emit_fix
emit_fixes
emit_foreach
emit_foreach_key
emit_get_key
emit_reject
emit_retain_key
this method is DEPRECATED.
emit_set_key
emit_string
emit_value
emit_walk_path
generate_var
split_path
SEE ALSO
Catmandu::Fixable, Catmandu::Importer, Catmandu::Exporter, Catmandu::Store, Catmandu::Bag