[Registry] Elektra 0.8.15 Release

[Markus Raab]

Hello everyone,

In case you do not yet know about it, here is an abstract about Elektra:

Elektra serves as a universal and secure framework to access configuration
parameters in a global, hierarchical key database.
Elektra provides a mature, consistent and easily comprehensible API.
Its modularity effectively avoids code duplication across applications
and tools regarding configuration tasks. Elektra abstracts from
cross-platform-related issues and allows applications to be aware of other
applications' configurations, leveraging easy application integration.

See [http://libelektra.org](http://libelektra.org)

## Overview

This is one of the largest release up to now. It includes many
user-visible improvements. Some highlights:

- Mounting is vastly improved:
think about Debian's "dpkg" to "apt"-like functionality
- In previous releases you could already specify the structure
of the configuration. Now you can also automatically enforce
this property.
- We split the shared library `libelektra` into smaller parts.
Now users can link against the parts of the library they need.
- As always, the ABI and API is fully forward-compatible.
- The release contains improvements in the
[bootstrapping process]
(https://github.com/ElektraInitiative/libelektra/blob/master/doc/decisions/bootstrap.md).
- We improved the `ini`, `rename` and `crypto` plugins.
- The tool `kdb` now supports bookmarks and profiles.
- The new tool `kdb editor` allows you to edit KDB configuration in
your favorite text editor.
- We are glad of the new packages for Debian, Arch Linux and OpenWRT.

The same text as follows is also available
[here as html](http://doc.libelektra.org/news/1ab4a560-c286-46d2-a058-1a8e7e208fe8.html)


## Global Mount

Sometimes you simply want some functionality for the whole key database.
For example, you want to enable logging or notification of configuration
changes. In previous versions, you had to change every mountpoint
individually. Even more problematic, every mountpoint created its
individual logs and notifications, without any way for someone to know
if an application has issued its last log/notification.

These problems are now solved by global plugins. The same plugins are
reused for this purpose. Also the mounting works nearly in the same way,
you only have to omit the configuration file and the mountpoint:

kdb global-mount syslog journald dbus

Voilà, from now on every configuration change gets logged to syslog
and journald. Additionally, every application gets notified via dbus.

If you want it more verbose for debugging, you can easily do so by:

kdb global-mount logchange counter

Which gives you detailed information to standard output which keys
were changed/edited/deleted. Additionally, Elektra counts how often
the `counter` plugin is invoked.

The underlying work for the global plugins, i.e. hooks in the core
libraries and the `list` plugin that allows us to use many plugins
without bloating the core was done by Thomas Waser.


It was already possible in earlier versions of Elektra to specify the
configuration of your program. Until now, this specification could be
mainly used to to generate code as described
[here]
(https://github.com/ElektraInitiative/libelektra/tree/master/src/tools/gen).
This release adds two more interesting options:

1. the spec plugin, and
2. the spec mount.



## Spec Plugin

The most important global plugin that is now newly introduced with this
release (thanks to Thomas Waser) is the `spec` plugin. By default it
will be added for every global-mount. So for a new installation make
sure you executed at least once:

kdb global-mount

The spec plugin is a global plugin that copies metadata from the
`spec`-namespace to other namespaces. That means, it reads the
specification, and makes sure that the configuration conforms to it. The
actual validation is done by the many validation plugins already present.

Lets start by saying a key is a long and must have at least the
value 10:

kdb setmeta spec/example/longkey check/type long

Then we can create a key in a different namespace and see if the `spec`
plugin applies the meta-data correctly:

kdb set /example/longkey 25
kdb lsmeta /example/longkey

Should now at least print `check/type`.
By itself, this is useful for documentation of keys.
For example, the application developer says:

kdb setmeta spec/example/longkey description "Do not change"
kdb setmeta spec/example/longkey example 30

The user can retrieve this documentation by:

kdb getmeta /example/longkey description

But we want `check/type` to be not only a documentation, but also
enforced.


## Spec Mount

Using `kdb setmeta` extensively and always looking out that all plugins
are mounted correctly is error-prone. So instead, one can directly mount
the plugins as specified. For the example above one simply needs:

kdb setmeta spec/example mountpoint example.ecf
kdb spec-mount /example

Now, when we try to modify `/example/longkey` it will be validated:

kdb set /example/longkey a

Error (#52) [...] long [not] matched [...] a

Based on the present meta-data, the correct plugins (in this case `type`
because of the metadata `check/type`) will be loaded.

We can also create a whole specification file, first mount the
specification and then the mountpoint according the specification, e.g
when we have `battery.ini` in the folder `$(dirname $(kdb file spec))`
with following content:

[]
mountpoint = battery.ini
infos/plugins = ini

[level]
check/enum = 'critical', 'low', 'high', 'full'

Then we can use:

# mount the file itself:
kdb mount battery.ini spec/example/battery ni
# make sure all plugins are present (e.g. enum for check/enum):
kdb spec-mount /example/battery

Now lets verify if it worked:

kdb lsmeta /example/battery/level
# we see it has a check/enum
kdb getmeta /example/battery/level check/enum
# now we know allowed values
kdb set /example/battery/level low
# success, low is ok!
kdb set /example/battery/level wrong
# fails, not one of the allowed values!

The main idea of the spec-mount is: search a plugin for every
specification (meta-data) found in the spec-namespace.




## General Mount Improvements

In earlier versions `kdb mount` failed when plugin dependencies were
not satisfied. Now dependencies will automatically be fulfilled, e.g.

kdb mount /etc/modules system/modules line

In earlier versions you would have got an error because of the missing
`null` plugin. Now it simply adds the needed plugins.

The plugins given in the command-line used to be real plugins. Now also
so called providers are accepted.

To make providers useful we need to actually know which plugin is the
best candidate. To get the information we added a `infos/status` clause
in the contract. In this clause the plugin developer adds many details
how well the plugin is tested, reviewed, documented, maintained and so
on. Based on this information, the best suited plugin will be chosen.
For example, you now can use:

kdb mount /etc/security/limits.conf system/limits augeas \
lens=Limits.lns logging

And the best suitable logger will automatically be chosen.

The configuration variable `/sw/kdb/current/plugins` now allows
us to pass plugin configuration with the same syntax as the plugin
specification passed on the commandline. A subtle difference is that
thus the shell-splitting of arguments is missing, it is not possible to
include whitespaces in the plugin configuration that way.

Now it is also possible to include the same plugin multiple times and
also give them individual names. This feature is essential for
script-based plugins, e.g. you now might add:

kdb mount file.x /example/mountpoint \
lua#resolver script=resolver.lua \
lua#storage script=storage.lua

Furthermore, `kdb mount` now supports recommendations, which can be
enabled with `--with-recommends`. E.g. supplied to the mount command
using augeas above, comments will automatically transformed to meta-data
to avoid cluttering of the real configuration.





## Library Split

Up to now, Elektra consisted only of a single shared library,
`libelektra.so`. Not all symbols in it were relevant to end users, for
example, some were only needed by plugins. Others were only proposed and
not yet part of the stable API. And finally, other symbols were not needed
in some situations, e.g. the plugins do not need the `kdb` interface.

Thus, we split `libelektra.so`, so that only coherent parts are in the
same library:

- `libelektra-core.so` only contains the `KeySet` data structure and
module loading. Every binary using Elektra should link against it.
- `libelektra-kdb.so` contains the missing `KDB` symbols. Together with
the `core` they contain everything declared in `kdb.h`.
Michael Zehender plans to have multiple variants of `libelektra-kdb.so`
that use different kinds of concurrency.
Headerfile: `<kdb.h>`
- `libelektra-ease.so` adds functionality missing in `core` to make the
life for C programmers easier.
New headerfile: `kdbease.h`
- `libelektra-proposal.so` adds functionality proposed for `core`. It
directly uses internal structures of `core`, thus they always need to
have exactly the same version.
Headerfile: `kdbproposal.h`


The source code is now better organized by the introduction of a `libs`
folder. The explanation of every library can be found in
[/src/libs/]
(https://github.com/ElektraInitiative/libelektra/tree/master/src/libs).

The rationale of the library split is documented
[here]
(https://github.com/ElektraInitiative/libelektra/blob/master/doc/decisions/library_split.md).
Shortly put, it was needed for further evolution and allowing us to grow
and enhance without getting a fat core.

Thanks to Manuel Mausz for fixing many bugs related to the library split
and also adapting all the bindings for it.

### Benchmark

To show that the split does not make Elektra slower, Mihael Pranjić made
a small benchmark. The real time of [benchmarks/large](/benchmarks/large.c)
and revision 634ad924109d3cf5d9f83c33aacfdd341b8de17a

1. kdb-static: Median :0.8800
2. kdb-full: Median :0.8700
3. kdb: Median :0.8700

So it seems that the split does not influence the time of running
elektrified processes.

*Times were measured with time(1) and the median is calculated for 21
runs of [benchmarks/large](/benchmarks/large.c). This was done using
[scripts/benchmark_libsplit.sh](/scripts/benchmark_libsplit.sh)*


## Compatibility

As always, the ABI and API is fully forward-compatible, i.e. programs
compiled against an older 0.8 version of Elektra will continue to work
(ABI) and you will be able to recompile every program without errors
(API).

We added `keyGetNamespace` which allows us to handle all namespaces
in a switch and to iterate all namespaces. This is essential, when
a new namespace is added, thus then the compiler can warn you about
every part where the new namespace is not yet considered. We currently
do not plan to add a new namespace, but better be safe than sorry.

The internal function `keyCompare` now also detects any meta-data
change.

libtools was nearly rewritten. Even though it is mostly API compatible
(you should not use the low-level `Backend` anymore but instead use the
`BackendBuilder`), it is certainly not ABI compatible. If you have an
undefined symbol: `_ZN3kdb5tools7Backend9addPluginESsNS_6KeySetE` you need
to recompile your tool. Even the merging part has ABI incompatibility
(different size of `_ZTVN3kdb5tools7merging14NewKeyStrategyE`).
Unfortunately, we still cannot guarantee compatibility in `libtools`,
further changes are planned (e.g. implementing mounting of lazy plugins).

The python(2) and lua interfaces changed, an additional argument (the
plugin configuration) is passed to `open`.

The INI plugin was rewritten, so many options changed in incompatible ways.

The default format plugin (e.g. for import/export) is no longer hardcoded
to be `dump`. Instead KDB_DEFAULT_STORAGE will be used.
To get KDB_DEFAULT_STORAGE you can use the constants plugin:

kdb mount-info
kdb get system/info/constants/cmake/KDB_DEFAULT_STORAGE

Thanks to Manuel Mausz plugins do no longer export any method other than
`elektraPluginSymbol`. It now will fail if you directly linked against
plugins and did not correctly use their public interface. Please
use the module loading and access functions via the contract.

The CMake and Pkgconfig Files now only link against `elektra-core` and
`elektra-kdb`. If you used some symbols not present in `kdb.h`, your
application might not work anymore.

`libelektra.so` is still present for compatibility reasons. It should
not be used for new applications. Some unimportant parts, however,
moved to the "sugar" libraries. E.g. the symbol `elektraKeyCutNamePart`
is no longer part of `libelektra.so`.


### Bootstrapping

When you use `kdbOpen` in Elektra as first step it reads mountpoint
configuration from `/elektra`. This step is the only hardcoded one,
and all other places of the KDB's tree can be customized as desired
via mounting.

The bootstrapping now changed, so that `/elektra` is not part of the
default configuration `default.ecf` (or as configured with `KDB_DB_FILE`),
but instead we use `elektra.ecf` (or as configured with `KDB_DB_INIT`).
This behaviour solves the problem that `default.ecf` was read twice
(and even differently, once for `/elektra` and once for `/`).

To be fully compatible with previous mountpoints, Elektra will still read
mountpoints from `default.ecf` as long as `elektra.ecf` is not present.

To migrate the mountpoints to the new method, simply use:

kdb upgrade-bootstrap

which basically exports `system/elektra/mountpoints`, then does `kdb rm
-r system/elektra/mountpoints` so that `default.ecf` will be without an
mountpoint and thus the compatibility mode does not apply anymore. As
last step it will import again what it exported before.

[Details are here]
(https://github.com/ElektraInitiative/libelektra/blob/master/doc/decisions/bootstrap.md)




## Plugins

We already highlighted the new `spec` plugin, but also other plugins
were improved at many places. Small other changes are:

- Conditionals now also support `assign/condition` syntax,
thanks to Thomas Waser
- Lua and Python are not tagged experimental anymore.
They now correctly add their configuration to the open-call.
- The plugin `yajl` (the json parser and generator) now also accepts the
type `string` and yields better warnings on wrong types.
- Improved error message in the `type` plugin.

Larger changes were done in the following plugins:

### INI

The INI plugin was rewritten and a huge effort was taken so that it
fully-roundtrips and additionally preserves all comments and ordering.
Currently, it is brand new. It is planned that it will replace `dump`
in the future.

Currently is has some minor problems when used as KDB_DEFAULT_STORAGE.
You can avoid most problems by mounting a different file into root:

kdb mount root.ini /

Read [here about the details]
(https://github.com/ElektraInitiative/libelektra/tree/master/src/plugins/ini).

A huge thanks to Thomas Waser.

### Rename

Thanks to Thomas Waser `rename` is now fixed for cascading keys.
Additionally, it does not change the `sync` status of the keys so
that notification plugins work properly afterwards.

### Crypto

The crypto plugin is a facility for securing sensitive Keys by symmetric
encryption of the value. It acts as a filter plugin and it will only
operate on Keys, which have the meta-key „crypto/encrypt“ set.

The key derivation is still work-in-progress, so the plugin does not
work with kdb yet. A planned method for key derivation is to utilize
the gpg-agent.

For now the plugin requires the following Keys within the plugin
configuration in order to work properly:

1. /crypto/key - the cryptographic key (binary 256 bit long)
2. /crypto/iv - the initialization vector (binary 128 bit long)

Please note that this method of key input is for testing purposes only
and should never be used in a productive environment!

Thanks to Peter Nirschl, especially the patience for also supporting
different platforms where dependencies need to be handled differently.


## KDB

A technical preview of a new tool was added: `kdb editor` allows you
to edit any part of Elektra's configuration with any editor and any
syntax. It uses 3-way merging and other stable technology, but it
currently does not provides a way to abort editing. So you only should
use it with care.

The tool `kdb list` now searches in the rpath for libraries and thus
will also find plugins not present at compile time (using `glob`).
Additionally, it sorts the plugins by `infos/status` score, which can
also be printed with `-v`. The last plugins printed are the ones ranked
highest.

When running as root, `kdb` will now use the `system` namespace when
writing configuration to cascading key names.

Long paths are cumbersome to enter in the CLI. Thus one can define
bookmarks now. Bookmarks are key-names that start with `+`. They are
only recognized by the `kdb` tool or tools that explicitly have support
for it. Applications should not depend on the presence of a bookmark. For
example, if you set the bookmark kdb:

kdb set user/sw/elektra/kdb/#0/current/bookmarks
kdb set user/sw/elektra/kdb/#0/current/bookmarks/kdb
user/sw/elektra/kdb/#0/current

You are able to use:

kdb ls +kdb/bookmarks
kdb set +kdb/format ini


The kdb tool got much more robust when the initial configuration is broken,
no man page viewer is present or Elektra was installed wrongly.

The `--help` usage is unified and improved.

The new keyname naming conventions are now used for
configuration of the `kdb`-tool itself: `/sw/elektra/kdb/#0/%/`
and `/sw/elektra/kdb/#0/current/` are additionally read. The option
`-p`/`--profile` is now supported for every command, it allows to fetch
configuration from `/sw/elektra/kdb/#0/<profile>/` instead of `current`.
KDB is more robust when it cannot fetch its own configuration due to
KDB errors.





## Coding Guidelines

Thanks to Kurt Micheli the code guidelines are
[now properly documented]
(https://github.com/ElektraInitiative/libelektra/blob/master/doc/CODING.md).
Thanks to René Schwaiger we also provide a style file for clang-format.

Furthermore we unified and fixed the source:
- only use @ for doxygen commands
- always use elektra* functions for allocation
- added a file header for every file

## C++11 migration

Since we now only use C++11, we applied `clang-modernize` which simplified
many loops and replaced many `0` to `nullptr`. Additionally, we added
`override` and `default` at many places.

We removed all places where we had `ifdefs` to use `auto_ptr` if no
modern smart pointer is available.

Because of these changes there is no easy way to compile Elektra without
C++11 support, except by avoiding the C++ parts all together.

The C++ `KeySet` now also supports a `get` to retrieve whole containers
at once. By specializing `KeySetTypeWrapper` you can support your own
containers. Currently only `map<string, T>` is supported (T is any type
supported by `Key::get`).

If you haven't removed it from your flags already, there is no use
anymore to set `ENABLE_CXX11`.

## Documentation

The documentation was improved vastly. Most thanks to Kurt Micheli who
did a lot of editing and fixed many places throughout the documentation
Also thanks to Michael Zehender who added two paragraphs in the main
README.md.

Keynames of applications should be called `/sw/org/app/#0/current`,
where `current` is the default profile (non given). `org` and
`app` is supposed to not contain `/` and be completely lowercase.
Keynames are documented [here](/doc/help/elektra-key-names.md).
[See also here](/doc/tutorials/application-integration.md). The main
reason for long paths is the provided flexibility in the future
(e.g. to use profiles and have a compatible path for new major versions
of configuration). By using bookmarks, users should not be confronted
by it too often.

- many man pages were improved
- many typos were fixed, thanks to Pino Toscano!
- Fix documentation for kdb list, thanks to Christian Berrer
- Compilation variants are explained better,
thanks to Peter Nirschl for pointing out what was missing
- document ronn as dependency, thanks to Michael Zehender
- fix broken links, thanks to Daniel Bugl

Thanks to Kurt Micheli, developers are now warned during compilation on
broken links in Markdown. The output format is the same as for gcc.
Additionally, the markdown documentation of plugins now get a proper
title in the pdf and html output of doxygen.


## Qt-Gui 0.0.10

Raffael Pancheri again updated qt-gui with many nice improvements:

- update existing nodes in case the config was changed outside the gui
- safely update tree
- add update signal to metadata
- fix crash when closing the GUI
- move Actions in separate file to make main.qml less clustered
- plugins do not output messages at startup
- `BackendBuilder` is now used, which automatically takes cares of the correct
ordering of plugins
- Qt 5.4 compatibility is still ensured
- C++11 is now used in Qt-Gui, too

## Packaging and Build System

Elektra 0.8.14 now in Debian with qt-gui, man pages,
thanks to Pino Toscano
[read more here]
(https://packages.qa.debian.org/e/elektra/news/20151215T000031Z.html)

Thanks to Gustavo Alvarez for updating and splitting the packages on
Arch Linux!

Thanks to [Harald Geyer](http://friends.ccbib.org/harald/supporting/),
Elektra is now packaged for OpenWRT. We fixed a number of
cross-compilation issues and now officially support building against musl
libc, with one minor limitation: RPATH works differently on musl so you
need to install all plugins directly in /usr/lib/ or set LD_LIBRARY_PATH.
Report any bugs in
[Harald's OpenWRT packaging issue tracker]
(https://github.com/haraldg/packages).

- export errors/symbols are now called `elektra-export-symbols` and
`elektra-export-symbols` and can be installed using `INSTALL_BUILD_TOOLS`
(by default off). This is needed for cross-compilation.
Thanks to Harald Geyer for reporting.
- some header files are renamed because they endlessly included themselves
if the header files were found in wrong order.
Thanks to Harald Geyer for reporting.
- fixed compilation when built on more than 20 cores with >= -j15,
thanks to Gustavo Alvarez for reporting and Manuel Mausz for analyzing
- lua 5.1 now works too (except for iterators),
thanks to Harald Geyer for reporting.
thanks to Manuel Mausz for adding a new FindLua.cmake
- pdf builds do not fail due to half written files,
reported by René Schwaiger and fixed by Kurt Micheli

Read about [other packages here]
(https://github.com/ElektraInitiative/libelektra#packages).

## Fixes and Improvements

- 3 way merge now properly deals with binary data,
thanks to Felix Berlakovich
- getenv: fix wrapping on powerpc, thanks to Pino Toscano
- markdownlinkconverter: fix char/int mismatch, thanks to Pino Toscano
- wresolver: use KDB_MAX_PATH_LENGTH instead of PATH_MAX,
thanks to Pino Toscano
- Cleaning up #ifdefs that break statements, thanks to Romero Malaquias
- Daniel Bugl tested the INI plugin
- cmake list_filter was broken because of different behaviour in
cmake_parse_arguments, thanks to Christian Berrer for reporting
- g++5.3 is now supported
- gtest does not link against pthread if not needed
- testcases that are built with BUILD_SHARED also successfully work
- kdb list works when libs are in same path as plugins,
thanks to Harald Geyer for reporting
- fix Mac OS issues,
thanks to Peter Nirschl, René Schwaiger and Mihael Pranjic
- fix resolver-baseflag docu, thanks to Harald Geyer for reporting
- do not create wrong directories called `(` and `)` in source,
thanks to René Schwaiger
- fix cmake for systems where iconv is not part of libc,
thanks to Michael Zehender and Peter KÃŒmmel (for FindIconv.cmake)
- fix segfault in libgetenv if root keys are present
- lua: fix Key:tostring(), thanks to Manuel Mausz
- add list of [supported bindings]
(https://github.com/ElektraInitiative/libelektra/tree/master/src/bindings),
thanks to Manuel Mausz


## Get It!

You can download the release from
[here](http://www.libelektra.org/ftp/elektra/releases/elektra-0.8.15.tar.gz)
and now also [here on github]
(https://github.com/ElektraInitiative/ftp/tree/master/releases/elektra-0.8.15.tar.gz)

- name: elektra-0.8.15.tar.gz
- size: 2338297
- md5sum: 33ec1e5982fb7fbd8893bf7b579b80f0
- sha1: 6b1fdd5aa5aaad6ba377b4bb5ef437e0c85319ff
- sha256: 6a406986cecb8d4a44485ced118ee803bc039b0824b72298e123b4dd47eb0b22
- sha512:
86a408dd546b33e3b437f92f415de7aee6a235189f9eab0762b3f44ab4c453ee369a53de10a9f5b0df1b446460b12c57c6b8b77c282648ec2a49f2328d9af13d



This release tarball now is also available
[signed by me using gpg]
(http://www.libelektra.org/ftp/elektra/releases/elektra-0.8.15.tar.gz.gpg)

already built API-Docu can be found [here]
(http://doc.libelektra.org/api/0.8.15/html/)


## Stay tuned! ##

Subscribe to the
[RSS feed](http://doc.libelektra.org/news/feed.rss)
to always get the release notifications.

For any questions and comments, please contact the
[Mailing List](https://lists.sourceforge.net/lists/listinfo/registry-list)
the issue tracker [on github](http://git.libelektra.org/issues)
or by mail ***@markus-raab.org.

[Permalink to this NEWS entry](http://doc.libelektra.org/news/1ab4a560-c286-46d2-a058-1a8e7e208fe8.html)

For more information, see [http://libelektra.org](http://libelektra.org)

Best regards,
Markus