wiki:ControlAPIDesign

Command Control Channel Design

Introduction

This document describes a design of the Control API in Kea. This API is used to manage Kea servers' configuration, control the state of the servers (e.g. shutdown or startup a server) and retrieve information about servers' states (e.g. how many leases have been assigned). The API allows for secure remote access to the Kea servers, i.e. the encryption and authentication mechanisms are used if required by the Kea configuration.

This design is intended to satisfy the requirements listed in the Requirements Document.

This is ongoing work and the document is a subject to updates.

Sections marked as Historical are not a part of the actual design. They are held in this document as a reference to past design discussions, which resulted in choosing alternative solutions.

System Overview

The component diagram includes two systems: System A and System B. The former runs the Kea services, the latter runs a Control API Client (CAC). It is possible to run CAC on the same machine as Kea services, but in the typical case they will be installed on separate machines. Kea comprises three services: DHCPv4, DHCPv6 and D2. They are already shipped in the Kea package. DHCPv4 and DHCPv6 services can be controlled via commands sent over unix domain sockets. This is currently not supported for the D2 service, but is planned to be supported in one of the future releases.

The design introduces new component, the Kea Control Agent (CA). It aggregates commands sent from the remote clients and forwards them to the appropriate Kea services. CA runs HTTP service on the CAC facing link. A HTTP request received from the CAC encapsulate JSON-formatted commands within the HTTP payload. The CA strips the HTTP headers and retrieves the command. It then retrieves a command name from the JSON structure, as well as the name/class of a service which the command pertains to. Most of the commands are simply forwarded to one of the services. However, some commands should be handled by the CA on its own. A common example of the command that is not forwarded is a "startup" command, which instructs the CA to launch a particular Kea service.

Note: "startup" command is not explicitly listed as a requirement for Kea 1.2.0 release, but it serves as a good example of a command to be handled by CA. This command will be implemented in one of the future releases.

When the CA receives a response from the service to which the command was forwarded, it creates a HTTP response by encapsulating the received JSON response with HTTP headers. If the command is handled by the CA on its own, the CA is responsible for creating a JSON response.

The third party HTTP server ("nginx" on our diagram) serves as a reverse proxy which forwards requests to the CA. The primary role of the proxy server is to provide a secure communication channel (HTTP + TLS) between CA running on System A, and the CAC on System B. The CA itself doesn't provide any encryption and authentication mechanisms. The communication between the proxy and the CA is unencrypted and not authenticated.

It is not necessary to install proxy server when securing the communication is not required. The CA can be configured to expose the HTTP service to the CAC directly. However, because the communication is not secured, any malicious client can take control over the Kea services. A possible way to mitigate this problem is to setup firewall rules to restrict the group of hosts that can communicate with the webservice.

Extended JSON Command Structure

In most cases, when the CA receives a command from the CAC, it will forward this command to one of the Kea servers for processing. When the response from the server is received, CA will simply send it encapsulated in the HTTP response to the CAC. Command names aren't unique between the Kea services. Thus, the CA needs some additional information from the CAC to forward the command to the appropriate service. In some cases commands are not forwarded but they are handled by the CA on its own. An example is the "startup" command which instructs the CA to start a specified Kea service, e.g. DHCPv4 server.

Note: "service-startup" is not explicitly listed in the requirements for the Kea 1.2.0 release.

Regardless if the command is forwarded or handled by the CA directly, the CAC must specify the service (or class of the services) which the command pertains to. The exceptions are these commands that do not trigger actions on the particular Kea services, e.g. commands that are specific to CA (e.g. reload CA's configuration) or some global administrative commands. None of those are currently specified, but can be added in the future.

An example of a command which specifies the service name is:

{
    "service": "dhcp4",
    "command": "foo", 
    "arguments": {
        "param_foo": "value1",
        "param_bar": "value2",
        ...
    }
}

In the example above, the command "foo" pertains to the DHCPv4 service. Other possible values are "dhcp6" or "d2".

In the example above the "param_foo" and "param_bar" are simple strings, but in general they may contain much more complex structures. Consider add-subnet4 command which adds a new subnet to the configuration. The single parameter "subnet" will contain a map describing the new subnet just as it looks in the Kea configuration file.

Currently we're assuming that a single instance of the particular Kea service is being controlled by the CA. It is envisaged that in the future CA will be able to manage multiple services of the same type running on the particular machine, e.g. multiple DHCPv4 servers. In such case, the "service" parameter will specify that the command pertains to all managed instances of a given service. Such command will be forwarded to all instances and the received responses are combined within a single JSON list. For example:

[
    {
        "pid": 8080,
        "result": 0,
        "error": "All ok"
    },
    {
        "pid": 8081,
        "result": 0,
        "error": "All ok"
    }
]

Note that "pid" values are added to the responses to identify the process which responded to the particular command.

As already mentioned, in the Kea 1.2.0, the CA can only manage a single instance of the particular service. Thus, the response list will contain at most one result, e.g.:

[
    {
        "pid": 8080,
        "result": 0,
        "error": "All ok"
    }
]

The following part:

    {
        "pid": 8080,
        "result": 0,
        "error": "All ok"
    }

The square brackets [ ] are added by the CA.

In the future, any command that pertains to a single instance of a particular service will contain a "pid" parameter which holds a process id of this instance, instead of "service" parameter. This parameter, however, is not to be supported in the Kea 1.2.0 release.

Restructuring CommandMgr

CommandMgr is a singleton object currently serving three purposes:

  • maintain associations of supported commands with the appropriate handler functions,
  • open and maintain the connection (via unix domain socket) with control API clients,
  • report supported commands by responding to 'list-commands' command.

With the advent of Kea 1.2.0 release the command handlers can be implemented both in the core Kea code and optional hook libraries. The new component, CA, will also use hooks mechanism to provide external implementations of some of the commands.

Neither CA nor hook libraries attached to the Kea services require UNIX domain sockets functionality of the CommandMgr, but they do require ability to maintain the mapping of commands to their handlers. Thus, the functions of the CommandMgr must be decoupled as presented on the figure below.

The CommandMgrBase provides methods for registering, deregistering command handlers. The CommandMgrBase::processCommand finds appropriate handlers for a processed commands and executes them.

The CommandMgrHooked extends CommandMgr::processCommand with new hook points 'control_command_receive' and 'control_commands_list'.

The CommandMgr derives from CommandMgrHooked and implements mechanisms responsible for communication via unix sockets.

New Hook Points

It is desired that Kea servers and Control Agent can handle additional commands implemented within hook libraries. These commands may be specific to a particular deployment and thus can't be natively supported by Kea. The new hook point 'control_command_receive' will be added within the CommandMgrHooked::processCommand and will be common for all applications and libraries using/deriving from this class. The callouts for this hook point will be executed when the command is received by the process and before the process handles the command on its own. If a hook library handles the command on its own, it should return a 'next action' flag set to 'skip'. The CommandMgrHooked::processCommand will assume that the command has been handled and will return the response to the client. If the 'skip' status is not set, the !CommandMgrHooked::processCommand will handle the command if the appropriate handler has been registered.

The 'control_command_receive' callouts will be executed with the following arguments:

  • name: command, type: ConstElementPtr, direction: in
  • name: response, type: ElementPtr, direction: out

The list-commands is a special command which returns commands supported by the module. If the server receiving this command has hook libraries implementing additional commands attached, the returned list should combine the natively supported commands and these supported by the hook libraries. The !CommandMgrHooked::processCommand adds a special hook point 'control_commands_list' which should be implemented by the hook libraries providing new commands. The callouts for this hook point will be executed with the following argument:

  • name: commands_list, type: ElementPtr, direction: out

The commands_list is a pointer to the JSON list of commands supported by the hook library. The hook library should insert the list of commands into this structure.

The !CommandMgrHooked::processCommand "glues" commands returned by the hook libraries with the natively supported commands and returned them to the control client.

Hook libraries should use CommandMgrBase as a base class for implementing their own instances of Command Manager, i.e. each hook library has its own Command Manager instance used to manage commands supported by this library.

Handling Commands within Control Agent

The Kea Control Agent (CA) is a process which receives commands from the clients and either forwards them to the respective Kea services or handles them on its own. CA determines if it should handle the command on its own by checking if the command handler has been registered for it.

When the CA receives a command, it first executes 'control_command_receive' callouts for this command. If the callouts return 'skip' status it simply returns the response to the CAC. Otherwise, the CA checks if it should handle the command on its own or forward to one of the Kea servers. This is done by checking if the command handler for this command has been registered. If it has been registered, this handler is executed to perform the command and generate response. If there is no handler for this command registered, the command is forwarded to the service specified as "service" parameter within JSON command.

Control Agent

This section presents the design of the Control Agent.

Configuration

The CA shares configuration format with all other Kea services. The CA can either use the dedicated configuration file or simply read the CA specific configuration from the configuration file common for all services. The following is a basic configuration of the CA.

"Control-agent":
{
    "http-host": "localhost",
    "http-port": 8000,
    "control-sockets":
    {
        "dhcp4-server":
        {
            "socket-type": "unix",
            "socket-name": "/path/to/the/unix/socket-v4"
        },

        "dhcp6-server":
        {
            "socket-type": "unix",
            "socket-name": "/path/to/the/unix/socket-v6"
        },

        "d2-server":
        {
            "socket-type": "unix",
            "socket-name": "/path/to/the/unix/socket-d2"
        }
    },

   "hooks-libraries": [
   {
        "library": "/opt/local/control-agent-commands.so",
        "parameters": {
            "param1": "foo"
         }
    }
 ]

},

"Logging":
{
  "loggers": [
    {
      "name": "kea-ctrl-agent",
      "output_options": [
          {
            "output": "/Users/marcin/devel/kea-build/var/log/kea-ctrl-agent.log"
          }
      ],
      "severity": "INFO",
      "debuglevel": 0
    }
  ]
}

The configuration example provided above specifies the host and port on which the HTTP service is available. It also specifies unix domain sockets which are used for communication with the respective Kea servers. If the socket configuration for the particular service is not specified, the control socket for this service is not opened. The "socket-type": "unix" is currently the only value supported by the CA.

Debate About Architecture

One of the major problems to be resolved by the design was to determine how to implement reliable and secure HTTP service, and avoid spending too much engineering time on creation and maintenance of the proprietary code. The aspects that dominated the discussion between the team members were:

  • Kea is written in C++. Can Control Agent be implemented in other language, with a better support for creating web services?
  • Should Kea crypto library be extended to provide a secure HTTP connection (over TLS) or we rather want to rely on third party code to solve the issue of authentication and encyrption?
  • If third party code is used, is this ok that it comes with dependencies on OpenSSL?
  • Should Control Agent support HTTP/1.1 or HTTP/2?
  • Should Control API provide authentication via HTTP BA (Basic Authentication)?

Starting from the least contentious aspects the team has decided that:

  • HTTP BA would be useful if Control API provided multiple administration roles. Given there is only one administration role, the authentication problem can be resolved by using TLS certificates installed on the limited number of hosts which are authorized to use the API.
  • HTTP/2 seems to be much more complex than HTTP/1.1 and not fully deployed, so we rather want to use HTTP/1.1. Because there is backward compatibility between version 1.1 and 2.0, the 2.0 can be implemented later if we desire.
  • Kea Control Agent can be implemented in another language (not C++) if the benefits outweigh the issues with additional complexity related to multiple languages and more dependencies.

Before the team was able to conclude what other language would best work for this the new aspects were considered, which implied that C++ can actually be a better choice than anything else. They are:

  • Kea already provides a lot of useful code for the Control Agent, e.g. CommandMgr?, logging, configuration parsing (currently transitioned to use Bison),
  • If the security layer can be provided by the use of HTTP proxy server (nginx, Apache or other) the most complex part (security) is already solved elsewhere. This leaves us with only a problem of providing a minimal HTTP server implementation on the Kea side.
  • There are some example solutions available online for the HTTP service implementation in C++. One of them is boost::asio based solution, another one is the FastCGI based solution.

The proof of concepts for both solutions have been created and put into the internal Kea repository on the following branches:

  • experiments/http
  • experiments/fcgi

Note that only ISC employees currently have access to these branches.

In the opinion of the author of this design the boost::asio based solution is better because it doesn't introduce new dependencies (FastCGI not required). In addition, it is asynchronous solution relying on asynchronous reads/writes, making it easier to multiplex between multiple connections (HTTP, multiple Unix sockets) and support asynchronous events, such as interval timers etc.

Finally, the team concluded that we should not attempt to implement TLS in Kea crypto library because both OpenSSL and Botan have awkward API which, if misused, can introduce security vulnerabilities in Kea. Therefore, we agreed that we should rely on the third party software (e.g. nginx or Apache etc) to provide a reverse proxy function. The major advantage of this choice is that we offload responsibility for security layer to third party. We also allow the users to select a HTTP server of their liking to be used as a proxy.

Proposed Architecture

The simplified class diagram of the Control Agent service implemented using boost::asio (see previous section) is depicted below.

The HTTP service is built around the HttpListener object which creates a TCP socket and accepts new connections on this socket. Once the connection comes in, the HttpConnection object is created to represent this new connection. The HttpConnectionPool serves as a container for all active connections. If the CA process has to shut down, all connections from this pool are closed before it shuts down. Once the HTTP transaction is completed, the connection used for this transaction is also closed and removed from the pool.

HttpConnection is used to read and parse HTTP requests received. HttpConnection creates an instance of the HttpRequestParser which implements a state machine used for parsing the HTTP requests. Each connection comes with its own instance of the parser. The parser stores an instance of the derivation of the HttpRequest object, which is used to interpret the received request once it has been parsed. The Control Agent uses PostHttpRequestJson class as a request interpreter as it provides means for interpreting JSON structures encapsulated within HTTP. It also checks that the received message is HTTP POST and that the "Content-Type" is "application/json".

The interpreted requests are held within the HttpRequest objects and are used to generate HttpResponse, which represent responses to be sent to the CAC. The HttpRequestCreator class provide basic tool set for generating HTTP responses. The derived classes provide specialization of this class to generate the body for these responses. For example, the CtrlAgentResponseCreator, being a part of the CA application will generate the HttpResponses containing JSON body representing answers to the commands issued by CAC. This class will obviously use a derivation of the CommandMgrBase to map JSON queries to command handlers.

The HTTP specific classes will be placed in a new library: libkea-http so as they can be reused in other components in the future. These classes are marked in yellow on the diagram above. The classes external to the new HTTP library are marked in blue.

HTTP Parser State Diagram

The HttpRequestParser is the complex part of the libkea-http library. Its state model is based on the implementation of the sample HTTP server in boost. The following diagram presents a simplified state model of the HTTP parser.

Note that almost all states on this diagram can transition to "parse failure" state, but these transitions are not shown on this diagram to keep the diagram clear. The transition to "parse failure" occurs when unexpected character is found in the stream. For example, the parser received CR character and expects LF. If LF is not found, parsing fails.

For better understanding of the transitions between the states we provide a sample HTTP message which is parsed using this model.

POST /foo/bar HTTP/1.0\r\n
Content-Type: application/json\r\n"
Content-Length: 45\r\n
User-Agent: Kea/1.2 Command \r\n"
 Control \r\n"
\tClient\r\n\r\n"
{ "service": "dhcp4", "command": "shutdown" }

The first line contains HTTP method, URI and version number. This is followed by 3 HTTP headers: "Content-Type", "Content-Length" and "User-Agent". The "User-Agent" has line breaks marked by Leading White Space (LWS) characters (space and tab). The HTTP message body contains command using JSON notation.

Documenting Supported Commands

We have discussed whether the Command API could use .spec (JSON) files to define supported commands. We considered whether the .spec files could be used as input for generating documentation as well as for configuring CommandMgr. We concluded that it brings a lot of complexity and puts additional burden on the implementors of the hook libraries with a minimal gain. Thus, the current design excludes this feature from the Kea 1.2.0 release.

Nevertheless, the format of the .spec file is convenient to document the commands we're planning to support within this design. Hence, we paste it here to demonstrate the basic commands we're aiming to support in Kea 1.2.0.

{
    "module": "dhcp4-server",
    "commands": [
         {
             "command": "shutdown",
             "since_version": "0.9.2",
             "in_subversion": "",
             "description": "Terminates the server process.",
             "arguments": [ ]
         },
         {
             "command": "reload-config",
             "since_version": "1.0.0",
             "in_subversion": "",
             "description": "Instructs the server to reload configuration from a file.",
             "arguments": [
                 {
                     "argument": "location",
                     "type": "text",
                     "mandatory": false
                 }
             ]
         },
         {
             "command": "get-config",
             "since_version": "1.2.0",
             "in_subversion": "",
             "description": "Retrieves entire server configuration.",
             "arguments": [ ]
         },
         {
             "command": "set-config",
             "since_version": "1.2.0",
             "in_subversion": "",
             "description": "Instructs the server to use new configuration. The configuration is not persisted to disk until write-config is invoked.",
             "arguments": [
                 {
                     "argument": "config",
                     "type": "text",
                     "mandatory": true
                 }
             ]
         },
         {
             "command": "test-config",
             "since_version": "1.2.0",
             "in_subversion": "",
             "description": "Performs dry run of the server configuration. It is used to test that the new configuration is valid.",
             "arguments": [
                 {
                     "argument": "config",
                     "type": "text",
                     "mandatory": true
                 }
             ]
         },
         {
             "command": "write-config",
             "since_version": "1.2.0",
             "in_subversion": "",
             "description": "Persists currently used configuration to disk.",
             "arguments": [
                 {
                     "argument": "location",
                     "type": "text",
                     "mandatory": true
                 }
             ]
         },
         {
             "command": "leases-reclaim",
             "since_version": "1.2.0",
             "in_subversion": "",
             "description": "Instructs the server to reclaim all expired leases.",
             "arguments": [ ]
         },
         {
             "command": "list-commands",
             "since_version": "1.0.0",
             "in_subversion": "",
             "description": "Returns all commands supported by the server and those that are planned to be supported in the future releases.",
             "arguments": [ ]
         }
    ]
}

Tasks

The following is the list of implementation and documentation tasks required for the Control API feature in Kea 1.2.0:

  1. #5100 - Decouple CommandMgr's unix sockets from commands handling and Implement 'control_command_receive' hook point.
  2. Implement basic commands (the actual ones to be implemented for 1.2.0 release is TBD)
  3. #5074 - Move DController and other generic classes to a separate library so as it can be reused by CA.
  4. #5075 - Create stub Control Agent
  5. #5076 - Implement configuration parsing within Control Agent (Bison)
  6. #3175 - d2::StateModel class(es) to lib::util
  7. #5077 - Implement HTTP requests parser in libkea-http library
  8. #5088 - Create HttpResponse and HttpResponseCreator classes
  9. #5094 - Implement TCPAcceptor class in linkea-asiolink
  10. #5099 - Create HttpListener, HttpConnection and HttpConnectionPool
  11. #5107 - Create CtrlAgentResponseCreator class and link it with CommandMgrBase
  12. #5078 - Control Agent: Create connections via unix sockets in Control Agent and forward received commands
  13. #5108 - Update keactrl to support kea-ctrl-agent startup, shutdown and reconfiguration.
  14. #5079 - Document the solution

Considerations Regarding Technology Selection for Control Agent (Historical)

The Kea Control Agent (CA) implements a web service that Command API Clients (CAC) use to deliver commands to the controlled system. The team considered whether CA should be implemented in C++, to be consistent with the source code of other Kea modules, or it can be implemented in another language, which provides better framework for building web services and is better suited for writing web applications in general. The team eventually decided that the advantages provided by other languages outweigh the desire to avoid additional dependencies and having a common code base for the Kea modules.

The following table lists candidate technologies arbitrarily selected for the implementation of the CA. The assumptions which led to picking this set of technologies for comparison were:

  • Technology provides some framework for building web services with minimal effort,
  • Technology is known to be in widespread use, has good support and wide community,
  • ISC staff has some experience with the technology,
  • The programming language associated with the technology has a straightforward syntax,
  • The language supports object oriented programming.

Note that Perl has been excluded from this list as it can be awkward to use with object oriented programming. Ruby on Rails has been excluded as it is known to have hard to understand syntax for the beginners and seems to be too heavy for the simple purpose of the CA.

PythonPHPJavaNodejs
AvailabilityHighHighHighHigh (mostly via Nodejs repos or website)
Integration with HTTP serversyes (via module)yes (via module)yes (Tomcat, Jetty...)partial (via reverse proxy)
Built-in HTTP serveryes (server class)yes (since version PHP5)in packages (via Jersey, JavaEE)yes (via Express framework)
Support for JSONyesyes (as of PHP 5.2)in librariesyes
Unix socketsyesyesvia external libraryyes
Unit testsyes (unittest framework)phpunit (phpunit.de)JUnitBuilt-in assert or external Mocha framework
SQL supportMySQL via pyPI or mysql-dev. Postgres from postgres packagesyesyes (JDBC)via npm
IssuesWhat version to choose: 2.7 or 3.0+?PHP was invented for pages renderingMany dependencies, optional packages, complex IDESome packages may be available from external sources, or unavailable
ProsISC has some experience in PythonMature language, high availabilityCross platform, many libraries and featuresClear syntax, easy installation, community, ISC has some expertise
Last modified 11 months ago Last modified on Jan 9, 2017, 9:57:33 AM

Attachments (5)

Download all attachments as: .zip