Provided by: fitsh_0.9.4-1_amd64 
NAME
lfit - general purpose evaluation and regression analysis tool
SYNOPSIS
lfit [method of analysis] [options] <input> [-o, --output <output>]
DESCRIPTION
The program `lfit` is a standalone command line driven tool designed for both interactive and batch
processed data analysis and regression. In principle, the program may run in two modes. First, `lfit`
supports numerous regression analysis methods that can be used to search for "best fit" parameters of
model functions in order to model the input data (which are read from one or more input files in
tabulated form). Second, `lfit` is capable to read input data and performs various arithmetic operations
as it is specified by the user. Basically this second mode is used to evaluate the model functions with
the parameters presumably derived by the actual regression methods (and in order to complete this
evaluation, only slight changes are needed in the command line invocation arguments).
OPTIONS
General options:
-h, --help
Gives general summary about the command line options.
--long-help, --help-long
Gives a detailed list of command line options.
--wiki-help, --help-wiki, --mediawiki-help, --help-mediawiki
Gives a detailed list of command line options in Mediawiki format.
--version, --version-short, --short-version
Gives some version information about the program.
--functions, --list-functions, --function-list
Lists the available arithmetic operations and built-in functions supported by the program.
--wiki-functions, --functions-wiki
Lists the available arithmetic operations and built-in functions supported by the program in
Mediawiki format.
--examples
Prints some very basic examples for the program invocation.
Common options for regression analysis:
-v, --variable, --variables <list-of-variables>
Comma-separated list of regression variables. In case of non-linear regression analysis, all of
these fit variables are expected to have some initial values (specified as <name>=<value>),
otherwise the initial values are set to be zero. Note that in the case of some of the
regression/analysis methods, additional parameters should be assigned to these fit/regression
variables. See the section "Regression analysis methods" for additional details.
-c, --column, --columns <independent>[:<column index>],...
Comma-separated list of independet variable names as read from the subsequent columns of the
primary input data file. If the independent variables are not in sequential order in the input
file, the optional column indices should be defined for each variable, by separating the column
index with a colon after the name of the variable. In the case of multiple input files and data
blocks, the user should assign the individual independent variables and the respective column
names and definitions for each file (see later, Sec. "Multiple data blocks").
-f, --function <model function>
Model function of the analysis in a symbolic form. This expression for the model function should
contain built-in arithmetic operators, built-in functions, user-defined macros (see -x, --define)
or functions provided by the dynamically loaded external modules (see -d, --dynamic). The model
function can depend on both the fit/regression variables (see -v, --variables) and the independent
variables read from the input file (see -c, --columns). In the case of multiple input files and
data blocks, the user should assign the respective model functions for each data block (see
later). Note that some of the analysis methods expects the model function to be either
differentiable or linear in the fit/regression variables. See "Regression analysis methods" later
on about more details.
-y, --dependent <dependent expression>
The dependent variable of the regression analysis, in a form of an arithmetic expression. This
expression for the dependent variable can depend only on the variables read from the input file
(see -c, --columns). In the case of multiple input files and data blocks, the user should assign
the respective dependent expressions for each data block (see later).
-o, --output <output file>
Name of the output file into which the fit results (the values for the fit/regression variables)
are written.
Common options for function evaluation:
-f, --function <function to evaluate>[...]
List of functions to be evaluated. More expressions can be specified by either separating the
subsequent expressions by a comma or by specifying more -f, --function options in the command
line.
Note that the two basic modes of `lfit` are distinguished only by the presence or the absence of the -y,
--dependent command line argument. In other words, there isn't any explicit command line argument which
specify the mode of `lfit`. If the -y, --dependent command line argument is omitted, `lfit` runs in
function evaluation mode, otherwise the program runs in regression analysis mode.
-o, --output <output file>
Name of the output file in which the results of the function evaluation are written.
Regression analysis methods:
-L, --clls, --linear
The default mode of `lfit`, the classical linear least squares (CLLS) method. The model functions
specified after -f, --function are expected to be both differentiable and linear with respect to
the fit/regression variables. Otherwise, `lfit` detects the non-differentiable and non-linear
property of the model function(s) and refuses the analysis. In this case, other types of
regression analysis methods can be applied depending our needs, for instance the
Levenberg-Marquardtalgorithm (NLLM, see -N, --nllm) or the downhill simplex minimization (DHSX,
see -D, --dhsx).
-N, --nllm, --nonlinear
This option implies a regression involving the nonlinear Levenberg-Marquardt (NLLM) minimization
algorithm. The model function(s) specified after -f, --function are expected to be differentiable
with respect to the fit/regression variables. Otherwise, `lfit` detects the non-differentiable
property and refuses the analysis. There some fine-tune parameters of the Levenberg-Marquardt
algorithm, see also the secion "Fine-tuning of regression analysis methods" for more details how
these additional regression parameters can be set. Note that all of the fit/regression variables
should have a proper initial value, defined in the command line argument -v, --variable (see also
there).
-U, --lmnd
Levenberg-Marquardt minimization with numerical partial derivatives (LMND). Same as the NLLM
method, with the exception of that the partial derivatives of the model function(s) are calculated
numerically. Therefore, the model function(s) may contain functions of which partial derivatives
are not known in an analytic form. The differences used in the computations of the partial
derivatives should be declared by the user, see also the command line option -q, --differences.
-D, --dhsx, --downhill
This option implies a regression involving the nonlinear downhill simplex (DHSX) minimization
algorithm. The user should specify the proper inital values and their uncertainties as
<name>=<initial>:<uncertainty>, unless the "fisher" option is passed to the -P, --parameters
command line argument (see later in the section "Fine-tuning of regression analysis methods"). In
the first case, the initial size of the simplex is based on the uncertainties provided by the user
while in the second case, the initial simplex is derived from the eigenvalues and eigenvectors of
the Fisher covariance matrix. Note that the model functions must be differentiable in the latter
case.
-M, --mcmc
This option implies the method of Markov Chain Monte-Carlo (MCMC). The model function(s) can be
arbitrary in the point of differentiability. However, each of the fit/regression variables must
have an initial assumption for their uncertainties which must be specified via the command line
argument -v, --variable. The user should specify the proper inital values and uncertainties of
these as <name>=<initial>:<uncertainty>. In the actual implementation of `lfit`, each variable has
an uncorrelated Gaussian a priori distribution with the specified uncertainty. The MCMC algorithm
has some fine-tune parameters, see the section "Fine-tuning of regression analysis methods" for
more details.
-K, --mchi, --chi2
With this option one can perform a "brute force" Chi^2 minimization by evaluating the value of the
merit function of Chi^2 on a grid of the fit/regression variables. In this case the grid size and
resolution must be specified in a specific form after the -v, --variable command line argument.
Namely each of the fit/regression variables intended to be varied on a grid must have a format of
<name>=[<min>:<step>:<max>] while the other ones specified as <name>=<value> are kept fixed. The
output of this analysis will be a series of lines with N+1 columns, where the values of
fit/regression variables are followed by the value of the merit function. Note that all of the
declared fit/regression variables are written to the output, including the ones which are fixed
(therefore the output is somewhat redundant).
-E, --emce
This option implies the method of "refitting to synthetic data sets", or "error Monte-Carlo
estimation" (EMCE). This method must have a primarily assigned minimization algorithm (that can be
any of the CLLS, NLLM or DHSX methods). First, the program searches the best fit values for the
fit/regression variables involving the assigned primary minimization algorithm and reports these
best fit variables. Then, additional synthetic data sets are generated around this set of best fit
variables and the minimization is repeated involving the same primary method. The synthetic data
sets are generated independently for each input data block, taking into account the fit residuals.
The noise added to the best fit data is generated from the power spectrum of the residuals.
-X, --xmmc
This option implies an improved/extended version of the Markov Chain Monte-Carlo analysis (XMMC).
The major differences between the classic MCMC and XMMC methods are the following. 1/ The
transition distribution is derived from the Fisher covariance matrix. 2/ The program performs an
initial minimization of the merit function involving the method of downhill simplex. 3/ Various
sanity checks are performed in order to verify the convergence of the Markov chains (including the
comparison of the actual and theoretical transition probabilities, the computation of the
autocorrelation lengths of each fit/regression variable series and the comparison of the
statistical and Fisher covariance).
-A, --fima
Fisher information matrix analysis (FIMA). With this analysis method one can estimate the
uncertainties and correlations of the fit/regression variables involving the method of Fisher
matrix analysis. This method does not minimize the merit functions by adjusting the fit/regression
variables, instead, the initial values (specified after the -v, --variables option) are expected
to be the "best fit" ones.
Fine-tuning of regression analysis methods:
-e, --error <error expression>
Expression for the uncertainties. Note that zero or negative uncertainty is equivalent to zero
weight, i.e. input lines with zero or negative errors are discarded from the fit.
-w, --weight <weight expression>
Expression for the weights. The weight is simply the reciprocal of the uncertainty. The default
error/uncertainty (and therefore the weight) is unity. Note that most of the analysis/regression
methods are rather sensitive to the uncertainties since the merit function also depends on these.
-P, --parameters <regression parameters>
This option is followed by a set of optional fine-tune parameters, that is different for each
primary regression analysis method:
default, defaults
Use the default fine-tune parameters for the given regression method.
clls, linear
Use the classic linear least squares method as the primary minimization algorithm of the EMCE
method. Like in the case of the CLLS regression analysis (see -L, --clls), the model function(s)
must be both differentiable and linear with respect to the fit/regression variables.
nllm, nonlinear
Use the non-linear Levenberg-Marquardt minimization algorithm as the primary minimization
algorithm of the EMCE method. Like in the case of the NLLM regression analysis (see -N, --nllm),
the model function(s) must be differentiable with respect to the fit/regression variables.
lmnd Use the non-linear Levenberg-Marquardt minimization algorithm as the primary minimization
algorithm of the EMCE method. Like in the case of -U, --lmnd regression method, the parametric
derivatives of the model function(s) are calculated by a numerical approximation (see also -U,
--lmnd and -q, --differences for additional details).
dhsx, downhill
Use the downhill simplex (DHSX) minimization as the primary minimization algorithm of the EMCE
method. Unless the additional 'fisher' option is specified directly, like in the default case of
the DHSX regression method, the user should specify the uncertainties of the fit/regression
variables that are used as an initial size of the simplex.
mc, montecarlo
Use a primitive Monte-Carlo diffusion minimization technique as the primary minimization algorithm
of the EMCE method. The user should specify the uncertainties of the fit/regression variables
which are then used to generate the Monte-Carlo transitions. This primary minimization technique
is rather nasty (very slow), so its usage is not recommended.
fisher In the case of the DHSX regression method or in the case of the EMCE method when the primary
minimization is the downhill simplex algorithm, the initial size of the simplex is derived from
the Fisher covariance approximation evaluated at the point represented by the initial values of
the fit/regression variables. Since the derivation of the Fisher covariance requires the knowledge
of the partial derivatives of the model function(s) with respect to the fit/regression variables,
the(se) model function(s) must be differentiable. On the other hand, the user do not have to
specify the initial uncertainties after the -v, --variables option since these uncertainties
derived automatically from the Fisher covariance.
skip In the case of EMCE and XMMC method, the initial minimization is skipped.
lambda=<value>
Initial value for the "lambda" parameter of the Levenberg-Marquardt algorithm.
multiply=<value>
Value of the "lambda multiplicator" parameter of the Levenberg-Marquardt algorithm.
iterations=<max.iterations>
Number of iterations during the Levenberg-Marquardt algorithm.
accepted
Count the accepted transitions in the MCMC and XMMC methods (default).
nonaccepted
Count the total (accepted plus non-accepted) transitions in the MCMC and XMMC methods.
gibbs Use the Gibbs sampler in the MCMC method.
adaptive
Use the adaptive XMMC algorithm (i.e. the Fisher covariance is re-computed after each accepted
transition).
window=<window size>
Window size for calculating the autocorrelation lengths for the Markov chains (these
autocorrelation lengths are reported only in the case of XMMC method). The default value is 20,
which is fine in the most cases since the typical autocorrelation lengths are between 1 and 2 for
nice convergent chains.
-q, --difference <variablename>=<difference>[,...]
The analysis method of LMND (Levenberg-Marquardt minimization using numerical derivatives, see -U,
--lmnd) requires the differences that are used during the computations of the partial derivatives
of the model function(s). With this option, one can specify these differences.
-k, --separate <variablename>[,...]
In the case of non-linear regression methods (for instance, DHSX or XMMC) the fit/regression
variables in which the model functions are linear can be separated from the nonlinear part and
therefore make the minimization process more robust and reliable. Since the set of variables in
which the model functions are linear is ambiguous, the user should explicitly specify this
supposedly linear subset of regression variables. (For instance, the model function
"a*b*x+a*cos(x)+b*sin(x)+c*x^2" is linear in both "(a,c)" and "(b,c)" parameter vectors but it is
non-linear in "(a,b,c)".) The program checks whether the specified subset of regression variables
is a linear subset and reports a warning if not. Note that the subset of separated linear
variables (defined here) and the subset of the fit/regression variables affected by linear
constraints (see also section "Constraints") must be disjoint.
--perturbations <noise level>, --perturbations <key>=<noise level>[,...]
Additional white noise to be added to each EMCE synthetic data sets. Each data block (referred
here by the approprate data block keys, see also section "Multiple data blocks") may have
different white noise levels. If there is only one data block, this command line argument is
followed only by a single number specifying the white noise level.
Additional parameters for Monte-Carlo analysis:
-s, --seed <random seed>
Seed for the random number generator. By default this seed is 0, thus all of the Monte-Carlo
regression analyses (EMCE, MCMC, XMMC and the optional generator for the FIMA method) generate
reproducible parameter distributions. A positive value after this option yields alternative random
seeds while all negative values result in an automatic random seed (derived from various available
sources, such as /dev/[u]random, system time, hardware MAC address and so), therefore
distributions generated involving this kind of automatic random seed are not reproducible.
-i, --[mcmc,emce,xmmc,fima]-iterations <iterations>
The actual number of Monte-Carlo iterations for the MCMC, EMCE, XMMC methods. Additionally, the
FIMA method is capable to generate a mock Gaussian distribution of the parameter with the same
covariance as derived by the Fisher analysis. The number of points in this mock distribution is
also specified by this command line option.
Clipping outlier data points:
-r, --sigma, --rejection-level <level>
Rejection level in the units of standard deviations.
-n, --iterations <number of iterations>
Maximum number of iterations in the outlier clipping cycles. The actual number of outlier points
can be traced by increasing the verbosity of the program (see -V, --verbose).
--[no-]weighted-sigma
During the derivation of the standard deviation, the contribution of the data points data points
can be weighted by the respective weights/error bars (see also -w, --weight or -e, --error in the
section "Fine-tuning of regression analysis methods"). If no weights/error bars are associated to
the data points (i.e. both -w, --weight or -e, --error options are omitted), this option will have
no practical effect.
Note that in the actual version of `lfit`, only the CLLS, NLLM and LMND regression methods support the
above discussed way of outlier clipping.
Multiple data blocks:
-i<key> <input file name>
Input file name for the data block named as <key>.
-c<key> <independent>[:<column index>],...
Column definitions (see also -c, --columns) for the given data block named as <key>.
-f<key> <model function>
Expression for the model function assigned to the data block named as <key>.
-y<key> <dependent expression>
Expression of the dependent variable for the data block named as <key>.
-e<key> <errors>
Expression of the uncertainties for the data block named as <key>.
-w<key> <weights>
Expression of the weights for the data block named as <key>. Note that like in the case of -e,
--errors and -w, --weights, only one of the -e<key>, -w<key> arguments should be specified.
Constraints:
-t, --constraint, --constraints <expression>{=<>}<expression>[,...]
List of fit and domain constraints between the regression variables. Each fit constraint
expression must be linear in the fit/regression variables. The program checks the linearity of the
fit constraints and reports an error if any of the constraints are non-linear. A domain
constraint can be any expression involving arbitrary binary arithmetic relation (such as strict
greater than: '>', strict less than: '<', greater or equal to: '>=' and less or requal to: '<=').
Constraints can be specified either by a comma-separated list after a single command line argument
of -t, --constraints or by multiple of these command line arguments.
-v, --variable <name>:=<value>
Another form of specifying constraints. The variable specifications after -v, --variable can also
be used to define constraints by writing ":=" instead of "=" between the variable name and initial
value. Thus, -v <name>:=<value> is equivalent to -v <name>=<value> -t <name>=<value>.
User-defined functions:
-x, --define, --macro <name>(<parameters>)=<definition expression>
With this option, the user can define additional functions (also called macros) on the top of the
built-in functions and operators, dynamically loadaded functions and previously defined macros.
Note that each such user-defined function must be stand-alone, i.e. external variables (such as
fit/regression variables and independent variables) cannot be part of the definition expression,
only the parameters of these functions.
Dynamically loaded extensions and functions:
-d, --dynamic <library>:<array>[,...]
Load the dynamically linked library (shared object) named <library> and import the global
`lfit`-compatible set of functions defined in the arrays specified after the name of the library.
The arrays must have to be declared with the type of 'lfitfunction', as it is defined in the file
"lfit.h". Each record in this array contains information about a certain imported function, namely
the actual name of this function, flags specifying whether the function is differentiable and/or
linear in its regression parameters, the number of regression variables and independent variables
and the actual C subroutine that implements the evaulation of the function (and the optional
computation of the partial derivatives). The module 'linear.c' and 'linear.so' provides a simple
example that implements the "line(a,b,x)=a*x+b" function. This example function has two regression
variables ("a" and "b") and one independent variable ("x") and the function itself is linear in
the regression variables.
More on outputs:
-z, --columns-output <column indices>
Column indices where the results are written in evaluation mode. If this option is omitted, the
results of the function evaluation are written sequentally. Otherwise, the input file is written
to the output and the appropriate columns (specified here) are replaced by the respective results
of the function evaluation. Thus, although the default column order is sequential, there is a
significant difference between omitting this option and specifying "-z 1,2,...,N". In the first
case, the output file contains only the results of the function evaluations, while in the latter
case, the first N columns of the original file are replaced with the results.
--errors, --error-line, --error-columns
Print the uncertainties of the fit/regression variables.
-F, --format <variable name>=<format>[,...]
Format of the output in printf-style for each fit/regression variable(see printf(3)). The default
format is %12.6g (6 signifiant figures).
-F, --format <format>[,...]
Format of the output in evaluation mode. The default format is %12.6g (6 signifiant figures).
-C, --correlation-format <format>
Format of the correlation matrix elements. The default format is %6.3f (3 significant figures).
-g, --derived-variable[s] <variable name>=<expression>[,...]
Some of the regression and analysis methods are capable to compute the uncertainties and
correlations for derived regression variables. These additional (and therefore not independent)
variables can be defined with this command line option. In the definition expression one should
use only the fit/regression variables (as defined by the -v, --variables command line argument).
The output format of these variables can also be specified by the -F, --format command line
argument.
-u, --output-fitted <filename>
Neme of an output file into which those lines of the input are written that were involved in the
final regression. This option is useful in the case of outlier clipping in order to see what was
the actual subset of input data that was used in the fit (see also the -n, --iterations and -r,
--sigma options).
-j, --output-rejected <filename>
Neme of an output file into which those lines of the input are written that were rejected from the
final regression. This option is useful in the case of outlier clipping in order to see what was
the actual subset of input data where the dependent variable represented outlier points (see also
the -n, --iterations and -r, --sigma options).
-a, --output-all <filename>
File containing the lines of the input file that were involved in the complete regression
analysis. This file is simply the original file, only the commented and empty lines are omitted.
-p, --output-expression <filename>
In this file the model function is written in which the fit/regression variables are replaced by
their best-fit values.
-l, --output-variables <filename>
List of the names and values of the fit/regression variables in the same format as used after the
-v, --variables command line argument. The content of this file can therefore be passed to
subsequent invocations of `lfit`.
--delta
Write the individual differences between the independent variables and the evaluated best fit
model function values for each line in the output files specified by the -u, --output-fitted, -j,
--output-rejected and -a, --output-all command line options.
--delta-comment
Same as --delta, but the differences are written as a comment (i.e. separated by a '##' from the
original input lines).
--residual
Write the final fit residual to the output file (after the list of the best-fit values for the
fit/regression variables).
REPORTING BUGS
Report bugs to <apal@szofi.net>, see also https://fitsh.net/.
COPYRIGHT
Copyright © 1996, 2002, 2004-2008, 2009-2020; Pal, Andras <apal@szofi.net>