Testing Augeas lenses with augparse on Debian GNU/Linux

Augeas is awesome

As intensive Puppet users, updating only certain key/values of system configuration files is extremely practical.

And this is what augeas does by convert existing files into an « abstraction », working on a virtual representation of the file made available as a tree and then writing changes back into the file.

# augtool print /files/etc/passwd
/files/etc/passwd
/files/etc/passwd/root
/files/etc/passwd/root/password = "x"
/files/etc/passwd/root/uid = "0"
/files/etc/passwd/root/gid = "0"
/files/etc/passwd/root/name = "root"
/files/etc/passwd/root/home = "/root"
/files/etc/passwd/root/shell = "/bin/bash"

Yet Augeas is very sensitive to syntax errors

Alas, sometimes Augeas doesn’t work and won’t say why. The augtool command -part of the augeas-tools Debian package- will just silently fail reading a file.

In that kind of case, you can of course resort to a well-known method « Cut 50% of content and see if the bug occurs, otherwise cut the other 50%, and repeat until you find the bug. »

But in the age of advanced testing methods, this is silly and there is a better method.

Introducing augparse

Augeas works with « lenses » which are in charge of converting the file into a tree and vice-versa.

Augparse -also a member of the augeas-tools package- is the tool you can use to write and test lenses.

On Debian, the lenses are provided with their corresponding tests, found in/usr/share/augeas/lenses/{dist/}tests

# cat /usr/share/augeas/lenses/dist/tests/test_webmin.aug
module Test_webmin =

let conf = "port=10000
realm=Webmin Server
denyfile=\.pl$
"

test Webmin.lns get conf =
{ "port" = "10000" }
{ "realm" = "Webmin Server" }
{ "denyfile" = "\.pl$" }

In this example, we see how the augparse syntax works.

First, the corresponding module is emptied for a clean test environment
Then the let keyword is used to define a multiline string, representing the file’s content.
Finally a test is declared, using a lens executed against the string.

Finding syntax errors with augparse

Now, finding where your file breaks is really easy:

Make a copy of the test file corresponding to the file you want to parse. Ex: test_webmin.aug
Edit this copy to add your actual configuration file content. Replace the first let variable value with the file’s content, but carefully remove or escape quotes.
Replace the rest of the test. Replace all other variables and tests by a single test running against your own content. Do not attempt to match a given tree (i.e. {« key » = « value »}), simply match a question mark, which will output the parsed tree. Ex: test Webmin.lns get conf = ?
Use augparse to run the test. You will test your actual configuration file’s content and you shall be rewarded with error lines, invalid characters, or the actual tree if the file is OK.

Hands on

We had a postfix which refused to be parsed.

I copied /usr/share/augeas/lenses/dist/tests/test_postfix_master.aug to /tmp and replaced its content by:

module Test_postfix_master =

let conf = "# AUTO GENERATED FILE
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page 
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       -       -       -       smtpd
  -o content_filter=smtp-amavis:[127.0.0.1]:10024
  -o receive_override_options=no_address_mappings
# ... SNIP MASTER.CF OTHER CONTENT ...
"
test Postfix_Master.lns get conf = ?

At this point and after fixing quotes escaping, I encountered the following error when running:

# augparse /tmp/test_postfix_master.aug

/tmp/test_postfix_master.aug:117.0-.36:exception thrown in test
/tmp/test_postfix_master.aug:117.5-.32:exception: Iterated lens matched less than it should
    Lens: /usr/share/augeas/lenses/dist/postfix_master.aug:49.17-.40:
    Error encountered at 16:0 (599 characters into string)
    <      -       -       smtpd\n|=|  -o content_filter=smtp-ama>

    Tree generated so far:
    /#comment[1] = "AUTO GENERATED FILE"
/#comment[2] = "Postfix master process configuration file.  For details on the format"
/#comment[3] = "of the file, see the master(5) manual page"
/#comment[4] = "=========================================================================="
/#comment[5] = "service type  private unpriv  chroot  wakeup  maxproc command + args"
/#comment[6] = "(yes)   (yes)   (yes)   (never) (100)"
/#comment[7] = "=========================================================================="
/smtp
/smtp/type = "inet"
/smtp/private = "n"
/smtp/unprivileged = "-"
/smtp/chroot = "-"
/smtp/wakeup = "-"
/smtp/limit = "-"
/smtp/command = "smtpd"

And there we are, the invalid line for augeas was -o content_filter=smtp-amavis:[127.0.0.1]:10024, removing the brackets worked.

After that, our puppet run worked fine thanks to augeas reading correctly the file. Thanks, lens debugging!