NAME

       doc_tutorials_import_md - How-To: kdb import

   Introduction
       The kdb tool allows users to interact with Elektra's Key Database via the command line.
       This tutorial explains the import function of kdb. This command lets you import Keys from
       the Elektra Key Database.

       The command to use kdb import is:

       kdb import [options] destination [format]

       In this command, destination is where the imported Keys should be stored below. For
       instance, kdb import system/imported would store all the keys below system/imported. This
       command takes Keys from stdin to store them into the Elektra Key Database. Typically, it
       is used with a pipe to read in the Keys from a file.

   Format
       The format argument can be a very powerful option to use with kdb import. The format
       argument allows a user to specify which plug-in is used to import the Keys into the Key
       Database. The user can specify any storage plug-in to serve as the format for the Keys to
       be imported. For instance, if a user wanted to import a /etc/hosts file into KDB without
       mounting it, they could use the command cat /etc/hosts | kdb import system/hosts hosts.
       This command would essentially copy the current hosts file into KDB, like mounting it.
       Unlike mounting it, changes to the Keys would not be reflected in the hosts file and vise
       versa.

   Dump Format
       If no format is specified, the format dump will be used instead. The dump format is the
       standard way of expressing Keys and all their relevant information. This format is
       intended to be used only within Elektra. The dump format is a good means of backing up
       Keys from the Key Database for use with Elektra later such as reimporting them later. As
       of this writing, dump is the only way to fully preserve all parts of the KeySet.

       It is very important to note that the dump does not rename keys by design. If a user
       exports a KeySet using dump using a command such as kdb export system/backup > backup.ecf,
       they can only import that keyset back into system/backup using a command like cat
       backup.ecf | kdb import system/backup.

   Options
       The kdb import command only takes one special option:

           -s --strategy <name>            which is used to specify a strategy

   Strategies
       For kdb import, you can specify a strategy to use if Keys already exist in the specified
       destination

       The current list of strategies are:

           preserve                                        any keys already in the destination will not be overwritten

           overwrite                                       any keys already in the destination will be overwritten if a new key has the same name

           cut                                                     all keys already in the destination will be removed, then new keys will be imported

       If no strategy is specified, the command defaults to the preserve strategy as to not be
       destructive to any previous keys.

   Example
           cat backup.ecf | kdb import system/backup

       This command would import all keys stored in the file backup.ecf into the Key Database
       under system/backup.

       In this example, backup.ecf was exported from the KeySet using the dump format by using
       the command: kdb export system/backup > backup.ecf

       backup.ecf contains all the information about the keys below system/backup: $cat
       backup.ecf kdbOpen 1 ksNew 3 keyNew 19 0 system/backup/key1 keyMeta 7 1 binary keyEnd
       keyNew 19 0 system/backup/key2 keyMeta 7 1 binary keyEnd keyNew 19 0 system/backup/key3
       keyMeta 7 1 binary keyEnd ksEnd

       Before the import command, system/backup does not exists and no keys are contained there.
       After the import command, running the command kdb ls system/backup prints:
       system/backup/key1 system/backup/key2 system/backup/key3 ages. Both the storage plugins
       and Elektra's core can conform to C99. Because the resolver plugin is the very first in
       the chain of plugins, it is guaranteed that no useless work is done.

   Initial kdbGet Problem
       Because Elektra provides self-contained configuration, kdbOpen() has to retrieve settings
       in the {bootstrapping} process below {system/elektra} as explained in {bootstrapping}.
       Because of the new way to keep track of removed keys, the internally executed kdbGet()
       creates a problem. Without countermeasures even the first kdbGet() of a user requesting
       the configuration below {system/elektra} fails because the resolver finds out that the
       configuration is already up to date. The configuration delivered by the user is empty at
       this point. As a result, the empty configuration will be appointed and returned to the
       user.

       A simple way to resolve this issue is to reload the default backend after the internal
       configuration was fetched. Reloading resets the timestamps and kdbGet() works as expected.

   kdbSet}
       Not the performance, but robust and reliable behaviour is the most important issue for
       kdbSet(). The design was chosen so that some additional in-memory comparisons are
       preferred to a suboptimal sequence of [syscall]{syscalls}. The algorithm makes sure that
       keys are written out only if it is necessary because applications can call kdbSet() with
       an unchanged KeySet. For the code to decide this, performance is important.

   Properties
       kdbSet() [guarantee]{guarantees} the following properties:

       {enumerate}

       Modifications to permanent storage are only made when the configuration was changed.

       When errors occur, every plugin gets a chance to rollback its changes as described in
       {exception safety}.

       If every plugin does this correctly, the whole KeySet is propagated to permanent storage.
       Otherwise nothing is changed in the key database. Plugins developed during the thesis meet
       this requirement.

       {enumerate}

       The synopsis of the function is: {lstlisting} int kdbSet(KDB *handle, KeySet *returned,
       Key * parentKey); {lstlisting}

       The user passes the configuration using the KeySet returned. The key set will not be
       changed by kdbSet(). The parentKey provides a way to limit which part of the configuration
       is written out. For example, the parentKey {user/sw/apps/myapp} will induce kdbSet() to
       only modify the key databases below {user/sw/apps/myapp} even if the KeySet returned also
       contains more configuration. Note that all backends with no keys in returned but that are
       below parentKey will completely wipe out their key database. The KDB handle contains the
       necessary data structures as shown in Figure~fig:architecture}.

   Search for Changes
       {wrapfigure}{r}{0.5} {-40pt} {center} [trim = 0 65 0 0, clip=true,
       width=6cm]{resolver_set} {center} {-20pt} {{kdbSet()} Algorithm} {fig:resolver_set}
       {-10pt} {wrapfigure}

       As a first step, kdbSet() {divides} the configuration passed in by the user to the key
       sets in the Split object. kdbSet() searches for every key if the {sync flag} is checked.
       Then kdbSet() decides if a key was removed from a backend by comparing the actual size of
       the key set with the size stored from the last kdbGet() call. We see that it is necessary
       to call kdbGet() first before invocations of kdbSet() are allowed.

       We know that data of a backend has to be written out if at least one key was changed or
       removed. If no backend has any changes, the algorithm will terminate at this point. The
       careful reader notices that the process involves no file operations.

   Duplicated Key Sets
       If some backends need synchronisation, the algorithm continues by filtering out all
       backends in the Split object that do not have changes. At this point, the Split object has
       a list of backends with their respective key sets.

       {deep duplicate}

       Plugins in kdbSet() can change values. Other than in kdbGet(), the user is not interested
       in these changes. Instead, the values are transformed to be suitable for the storage. To
       make sure that the changed values are not passed to the user, the algorithm continues with
       a {deep duplication} of all key sets in the Split object.

   Resolver
       All plugins of each included backend are executed one by one up to the resolver plugin. If
       this succeeds, the resolver plugin is responsible for committing these changes. After the
       successful commit, [error code]{error codes} of plugins are ignored. Only logging and
       notification plugins are affected.

   Atomic Replacement
       For this thesis only file-based storage with atomic properties were developed. The
       replacement of a file with another file that has not yet been written is not trivial. The
       straightforward way is to lock a file and start writing to it. But this approach can
       result in broken or partially finished files in events like ''out of disc space'', signals
       or other asynchronous aborts of the program.

       A temporary file solves this problem, because in problematic events the original file
       stays untouched. When the temporary file is written out properly, it is renamed and the
       original configuration file is overwritten. But another concurrent invocation of kdbSet()
       can try to do the same with the result that one of the newly written files is lost.

       To avoid this problem, locks are needed again. It is not possible to lock the
       configuration file itself because it will be unlinked when the temporary file is renamed.
       So a third file for locking is needed. The resolver currently implements this approach.

       An alternative to this approach without locks is to completely rely on the modification
       time. The modification time typically has only a resolution of one second. So any changes
       within that time slot will not be recognised. For this approach, however, the name of
       every temporary file must be unique because concurrent kdbSet() invocations each try to
       create one. The temporary file must also be unlinked in case of a rollback. The opened
       temporary file can be passed to the storage plugins using a file name in the directory
       {/dev/fd}. This approach may be more practical than the currently implemented way because
       it does not need the additional lock file{Nevertheless, the other way was chosen to test
       if the algorithm is exception safe as described in {exception safety}.}.

   Errors
       The plugins within kdbSet() can fail for a variety of reasons. [conflict]{Conflicts} occur
       most frequently. A conflict means that during executions of kdbGet() and kdbSet() another
       program has changed the key database. In order not to lose any data, kdbSet() fails
       without doing anything. In conflict situations Elektra leaves the programmer no choice.
       The programmer has to retrieve the configuration using kdbGet() again to be up to date
       with the key database. Afterwards it is up to the application to decide which
       configuration to use. In this situation it is the best to ask the user, by showing him the
       description and reason of the error, how to continue:

       {enumerate}

       {conflicts}

       Save the configuration again. The changes of the other program will be lost in this case.

       The key database can also be left unchanged as the other program wrote it. After using
       kdbGet() the application is already up to date with the new configuration. All
       configuration changes the user made before will be lost.

       The application can try to merge the key sets to get the best result. If no key is changed
       on both sides the result is clear, otherwise the application has to decide if the own or
       the other configuration should be favoured. The result of the merged key sets has to be
       written out with kdbSet().

       Merging the key sets can be done with ksAppend(). The source parameter is the preferred
       configuration. Note that the downside of the third option is that a merged configuration
       can be an not validating configuration.

       {enumerate}

       Sometimes a concrete key causes the problem that the whole key set cannot be stored. That
       can happen on validation or because of type errors. Such errors are usually caused by a
       mistake made by the user. So the user is responsible for changing the settings to make it
       valid again. In such situations, the {internal cursor} of the KeySet returned will point
       to the problematic key.

       A completely different approach is to export the configuration when kdbSet() returned an
       error code. The user can then edit, change or merge this configuration with more powerful
       tools. Finally, the user can import the configuration into the global key database. The
       export and import mechanism is called ''streaming'' and will be explained in {streaming}.

   Introduction
       The kdb tool allows users to interact with Elektra's Key Database via the command line.
       This tutorial explains the import function of kdb. This command lets you import Keys from
       the Elektra Key Database.

       The command to use kdb import is:

       kdb import [options] destination [format]

       In this command, destination is where the imported Keys should be stored below. For
       instance, kdb import system/imported would store all the keys below system/imported. This
       command takes Keys from stdin to store them into the Elektra Key Database. Typically, it
       is used with a pipe to read in the Keys from a file.

   Format
       The format argument can be a very powerful option to use with kdb import. The format
       argument allows a user to specify which plug-in is used to import the Keys into the Key
       Database. The user can specify any storage plug-in to serve as the format for the Keys to
       be imported. For instance, if a user wanted to import a /etc/hosts file into KDB without
       mounting it, they could use the command cat /etc/hosts | kdb import system/hosts hosts.
       This command would essentially copy the current hosts file into KDB, like mounting it.
       Unlike mounting it, changes to the Keys would not be reflected in the hosts file and vise
       versa.

   Dump Format
       If no format is specified, the format dump will be used instead. The dump format is the
       standard way of expressing Keys and all their relevant information. This format is
       intended to be used only within Elektra. The dump format is a good means of backing up
       Keys from the Key Database for use with Elektra later such as reimporting them later. As
       of this writing, dump is the only way to fully preserve all parts of the KeySet.

       It is very important to note that the dump does not rename keys by design. If a user
       exports a KeySet using dump using a command such as kdb export system/backup > backup.ecf,
       they can only import that keyset back into system/backup using a command like cat
       backup.ecf | kdb import system/backup.

   Options
       The kdb import command only takes one special option:

           -s --strategy <name>            which is used to specify a strategy

   Strategies
       For kdb import, you can specify a strategy to use if Keys already exist in the specified
       destination

       The current list of strategies are:

           preserve                                        any keys already in the destination will not be overwritten

           overwrite                                       any keys already in the destination will be overwritten if a new key has the same name

           cut                                                     all keys already in the destination will be removed, then new keys will be imported

       If no strategy is specified, the command defaults to the preserve strategy as to not be
       destructive to any previous keys.

   Example
           cat backup.ecf | kdb import system/backup

       This command would import all keys stored in the file backup.ecf into the Key Database
       under system/backup.

       In this example, backup.ecf was exported from the KeySet using the dump format by using
       the command: kdb export system/backup > backup.ecf

       backup.ecf contains all the information about the keys below system/backup: $cat
       backup.ecf kdbOpen 1 ksNew 3 keyNew 19 0 system/backup/key1 keyMeta 7 1 binary keyEnd
       keyNew 19 0 system/backup/key2 keyMeta 7 1 binary keyEnd keyNew 19 0 system/backup/key3
       keyMeta 7 1 binary keyEnd ksEnd

       Before the import command, system/backup does not exists and no keys are contained there.
       After the import command, running the command kdb ls system/backup prints:
       system/backup/key1 system/backup/key2 system/backup/key3