wiki:ConfigParseDesign

This document is a draft and is a subject to substantial changes

Design for re-implementation of Kea Configuration Parsers

Introduction

Kea uses a collection of classes, a.k.a. ''configuration parsers'', to collect the configuration data from the pre-parsed JSON structures and set the new configuration of the Kea deamons. The configuration parsers for DHCP components were implemented within the BIND10 framework and inherited the concepts of configuration parsing from DNS modules. As the number of Kea configuration parameters grew, it became troublesome to add new configuration parameters and maintain the parsers. This document identifies the issues with the current implementation of the configuration parsers and proposes a design to address them.

Issues with Current Configuration Parsers

1. Unused spec Files

Spec files were introduced in BIND10 to define the structures of the configuration information used for different modules. The RESTful API exposed parameters defined in the spec files to the client application, called bindctl, which was used to modify the parameters. Since, bindctl didn't expose parameters which didn't exist in the spec files, there was no need for the parsers to validate whether the configuration structure adhered to the spec files. When the BIND10 framework was removed, Kea deamons have become standalone applications using JSON files as the configuration storage. Every deamon is now fully responsible for reading the configuration from the file and validating that the data types and the hierarchy of the parameters are correct. The legacy parsers have no support for validating the configuration against the spec file. As a result, some configuration parsers implement sanity checks for unsupported or invalid parameters but this requires modification of the configuration parser for each new parameter. Also, hardcoding the supported parameters makes it troublesome to keep the configuration documentation up to date and provides no means for the user to view the configuration options.

Briefly:

  • The spec file should be used to generate the documentation for the configuration parameters of the deamons
  • The spec file should be used to validate that the hierarchy of the parameters and their types is correct

2. Parsing Order Does Matter

Some of the configuration parameters exposed by Kea are interdependent. For example, Kea allows for defining custom option formats, a.k.a. option definitions. The definition includes the option code, type of values conveyed in the option fields etc. This definition doesn't include the actual data to be sent in the option. The option data is specified elsewhere in the configuration file. The option-data parser must verify if the data specified by the user adheres to the option format before accepting the data. This implies that the option definition must be known upfront and therefore the option definition must be parsed before the option data. Currently, the order in which the parsers are invoked is hardcoded and the code launching the configuration parsers is quite convoluted.

Briefly:

  • The order in which the configuration parsers are executed should not be harcoded but rather defined in the spec file

3. Abuse of ''commit''

All configuration parsers derive from the common base class and implement two functions: build and commit. The build is meant to read the configuration information from the JSON data structures, validate it and store in the temporary storage. The commit was invoked for all parsers to move the configuration from the temporary storage to the Configuration Manager which holds the whole server configuration. However, due to interdependencies between various configuration parameters it was necessary to execute commit on some parsers so as the other parsers have access to the committed information. This violates the original concept of separation of the build and commit phases, according to which the ''commit'' should not trigger exceptions.

Briefly:

  • The commit phase should be exception safe
  • The commit should not be executed by the configuration parsers but rather by the configuration manager after all configuration parsers are done parsing.

4. No rollback capability

As described in the previous section, some configuration parsers need to commit changes to provide access to the configuration information to other parsers. This leads to the states of ''partially committed'' information which disrupts the configuration data integrity. To avoid this problem, the code which triggers the server configuration, stores the currently used configuration information in the local data structures, and if the configuration fails the server is reconfigured with stored data. The Configuration Manager doesn't provide any capability to manage the process of rolling back changes and for each newly defined parameter the ''workaround'' code has to be updated to guarantee that al changes are rolled back when appropriate.

Briefly:

  • The configuration mechanism must use the rollback mechanism described here

The Design

Overview

The major pain in the maintenance of the current configuration mechanism is the difficulty of adding new configuration parameters, as it requires modification of the code in multiple places to guarantee the correct parsing order, to include the parameter in the rollback procedure and to make sure that the appropriate parser is executed for the parameter being added. Although it is impossible to add a new configuration parameter without a change to the code (need to extend the Configuration Manager to store the new type of information and provide the validation criteria for the new parameter), the modifications should be reduced to the minimum. The design described here proposes that the spec file is a common place where the new parameters are created. The spec files should contain references to the configuration parsers so as they are executed in the appropriate order and for the appropriate parameters.

The spec File

The spec file is selected as a central repository for the parameters for two reasons. First, the spec files are already created for each module and contain the necessary information. Second, the spec files use the JSON format which can be parsed by the existing Kea libraries, similar to the configuration files.

One of the goals set in this document is that the spec file should specify a parsing order for all the parameters. This obviously excludes maps as they are not meant to be ordered structures (according to JSON specifications). However, Kea doesn't require maps to be ordered anyway. All future parameters should be held in the lists if they require a specific order of parsing. Currently, the contents of the spec file are not segregated according to the order in which they should be parsed. This means that the spec files will need to be revised and re-ordered as needed.

Each parameter in the configuration requires some special processing. For example, the renew timer must not be greater than the rebind timer so it is not sufficient to process timers individually and check that they are valid numbers but both parameters have to be compared to verify that the relation between them is fulfilled. This validation will need to be performed by the dedicated parsers. In order to minimize the changes to include the new parser in the code, the parsers will be associated with the parameter names in the spec file.

Let's consider the fragment of the spec file which defines an option definition for the DHCPv4 daemon:

      { "item_name": "option-def",
        "item_type": "list",
        "item_optional": false,
        "item_default": [],
        "list_item_spec":
        {
          "item_name": "single-option-def",
          "item_type": "map",
          "item_optional": false,
          "item_default": {},
          "map_item_spec": [
          {
            "item_name": "name",
            "item_type": "string",
            "item_optional": false,
            "item_default": ""
          },

          { "item_name": "code",
            "item_type": "integer",
            "item_optional": false,
            "item_default": 0
          },

          { "item_name": "type",
            "item_type": "string",
            "item_optional": false,
            "item_default": ""
          },

          { "item_name": "array",
            "item_type": "boolean",
            "item_optional": false,
            "item_default": False
          },

          { "item_name": "record-types",
            "item_type": "string",
            "item_optional": false,
            "item_default": ""
          },

          { "item_name": "space",
            "item_type": "string",
            "item_optional": false,
            "item_default": ""
          },

          { "item_name": "encapsulate",
            "item_type": "string",
            "item_optional": false,
            "item_default": ""
          } ]
        }
      },

The ''item_name'' values can be associated with the appropriate parser objects in the code. For example: there may be the new class called OptionDefParser associated with the ''option-def'' item in the spec file, the SingleOptionDef parser associated with the ''single-option-def'' item and so on. When the configuration is triggered, the code would walk through the spec and the configuration file and trigger appropriate configuration parsers for the appropriate configuration parameters. Note that this feature doesn't require any changes to the spec file.

Associating Parser with Configuration Item

In order to associate the appropriate parsers with the configuration parameters there is a need to create a central repository of the parsers. This will be implemented as a singleton object ParserMgr exposing factory functions returning instances of the configuration parsers:

ParserPtr parser = ParserMgr::instance().createParser("foo-parameter");
parser->parse(context);

where "foo-parameter" is the ''item_name'' of the parameter in the spec file. The context holds the information required by the parser such as: portion of the configuration to be parsed, portion of the spec file and the path to the configuration element being parsed. If there is no custom parser associated with the ''foo-parameter'' the default parser will be returned. The ParserMgr will need to load the spec file initially to create a default parser when there is no specialized one. Note that the default parser returned depends on the data type of the particular configuration parameter: list, map, integer, boolean, string etc.

Note that some parameters may appear more than once in the configuration hierarchy. For example, the ''renew-timer'' appears in the global list of parameters and under the subnet configuration - to override the global setting on per-subnet basis. In some cases, there may be a need to differentiate the parsers for the parameters having the same name. In that case, the parser object should be associated with the full ''path'' to the parameter in the hierarchy:

ParserPtr parser = ParserMgr::instance().createParser("subnet4/renew-timer");
parser->parse(context);

This implies that the configuration parsing logic will need to store the information about the full path to the currently parsed object so as the appropriate parser can be selected using the fully qualified path to the parameter.

Basic Configuration Parser

All configuration parsers must derive (directly or indirectly) from a common base class Parser. This class exposes the following interface:

  • Parser: constructor used by the ParserMgr to create an instance of a parser
  • parse(context) - a virtual function parsing the data element. It invokes the registered parsers for the child elements as indicated by the spec file (and the ParserManager). It also invokes the parseThis virtual function which provides a parser-specific implementation of the parsing algorithm. This function must only be overriden by the default parsers like: ListParser, MapParser, StringParser, IntParser, BoolParser which are the base classes for the specialized parsers. The context parameter contains the information required by the parser like:
    • portion of the spec file to be used for parsing the particular data element,
    • configuration element to be parsed
    • absolute path to the configuration element to be parsed, e.g. ''subnet4/renew-timer''
  • parseThis - a virtual function to be overridden in the specialized parsers which provides a specific implementation of the data parsing for the particular configuration item.
  • validate - a non-virtual function testing the configuration item against the spec, e.g. testing if the appropriate parameters are specified for the item. This function is always called by the parse function before invoking child parsers.

When the new parameter is added and the new parser is implemented, the parseThis function has to be overriden. This function reads the parameters specified for the particular data element and updates the ''staging'' configuration of the CfgMgr (Configuration Manager) accordingly. If the parameters being parsed failed to validate the parseThis function throws an exception which propagates to the method triggering the re-configuration. This method will rollback the changes to the CfgMgr introduced by the configuration parsers ran so far. The configuration parser is responsible for including the position of the data element on which parsing failed in the error message.

There is a set of standard parsers deriving from the Parser class which implement parsing of elements for various data types: boolean, string, map, list etc. The specialized classes should derive from them.

Processing Configuration

The configuration parameters must be processed in the specific order defined in the spec file. Elements in the spec file are processed recursively and for each element in the spec file the corresponding data element in the configuration hierarchy is obtained. At this point, it is checked if the element is mandatory or not. If it is mandatory and not specified, an exception is thrown. For each element which is found in the configuration the parser is created and parse function is executed.

A pseudo code for the ListParser class deriving from the Parser base class:

void ListParser::parse(context) {
    ElementPtr list = context->config_data;
    for each item in list {
        // Get the appropriate parser from the ParserMgr. This may return
        // a default parser or a specialized one.
        ParserPtr parser = ParserMgr::createParser(context->config_path);
        child_context->path = context_path + '/' + context->spec->item_name;
        child_context->config_data = item;
        child_context->spec = context->spec->list_item_spec;
        // Invoke a parser for the sub-element.
        parser->parse(child_context);
    }
    // This will really just execute the parseThis function or a derivative.
    Parser::parse(context);
}

Note that in most cases it will not be necessary to override the ListParser class as it simply invokes a bunch of parsers for each element on the list. The specialized parsers will need to be created for the particular elements that belong to the list. For example: there will not be a need to create a parser to process the list of active interfaces because this will be handled by the default ListParser class. But, there will be a need to create a specialized parser (derived from StringParser) which will be invoked for each interface and which will validate whether the particular interface name is valid and will add it to the staging configuration of the CfgMgr.

Classes similar to ListParser will be created for other data types.

Automatic Documentation Generation

TBD: Describe how the spec file will be used to automatically update the Kea User Guide with the configuration parameters supported.

Last modified 12 months ago Last modified on Nov 25, 2016, 6:54:13 PM