wiki:ZoneLoadingAPIDesign

BIND 10 Zone File Loading API Design

0. About This Document

This document summarizes initial ideas on possible implementation designs for the complete zone loader, taking into account the requirement document: http://bind10.isc.org/wiki/ZoneLoadingRequirements

This is provided mainly for further discussions, rather than a complete/concrete proposal. Class/function names are tentative.

1. Overview

The basic idea is to port the BIND 9's lexer/parser implementation as much as possible, while modernizing internal details by, e.g, using exceptions when appropriate or class inheritance for customizations, and (of course) with more detailed tests and documentation.

BIND 9 implements zone loading using the following main components:

  • isc_lex: a generic lexer module
  • dns_master_load* functions: parsing master file and generate RR(set)s in its native forms.
  • callback functions for dns_master_load*: these are customization point and are caller-specific functions to use the generated RRsets or handle error cases.
  • dns_zone: a monster module handling various zone management. in the context of loader, its role is to perform post-load validations.

My suggestion is to port these to BIND 10 as follows:

  • Introduce a separate lexer class, say, MasterLexer. It's the BIND 10 version of isc_lex. Unlike isc_lex, MasterLexer focuses on master file parsing, considering we can use other richer C++/boost libraries for general purpose parsing. This class is responsible for understanding quoted string, multiline-notation, etc, and is essentially expected to provide the "next string" from an input stream/file.
  • Introduce a set of free functions, say, masterLoad (maybe revising existing ones in libdns++). This is the BIND 10 version of dns_master_load* and responsible for parsing various forms of textual RRs (detect TTL, detect origin, handle arbitrary order of TTL and class, $-directives such as $ORIGIN, $INCLUDE etc). It can work either in the strict or lenient mode: when being strict, any error will immediately stop loading (probably) with an exception; when being lenient, it internally catches "parse error" type of exceptions, leaves a log message or something, then try to continue the load.
  • Introduce a base LoaderContext class. This is the BIND 10 version of callbacks. It has add(), error(), warn() methods, which are for installing RRsets and handling errors in the context-specific manner.
  • Introduce a generic class to represent a set of RRsets, say, RRsetCollection. This is a generic container and the stored set of RRsets does not necessarily form a valid zone (e.g. there doesn't necessarily have to be an SOA at the "origin"), but will be used to represent a single zone for the purpose of zone loading/checking. It provides at least a simple find() method to find an RRset for the given name and type (and maybe class) and a way to iterate over all RRsets. We'll make it an abstract base class and allow the user module to implement specific version of it; libdatasrc will make it a wrapper of the updating working on building the zone.
  • Introduce a separate free function, say, validateZone(). It takes the origin name and an RRsetCollection object, and performs common validity checks on the collection as zone content of the given origin name. Loader and checker applications can use it after loading the zone.

We'll also need to make some adjustments/extensions to existing APIs:

  • the "from text" constructor of Rdata classes and createRdata() (from std::string) will have to take MasterLexer and (maybe optionally) an origin name.
  • the "from text" constructor of the Name class will have to take a (maybe optional) origin name parameter.
  • exception classes in libdns++ will be reorganized so that exceptions due to textual errors will be a subclass of a single DNSParserError exception. It will help implement the "lenient" mode of masterLoad() above.
  • we need to implement ZoneUpdater for the in-memory data source and deprecate the "load" method of the finder class. We also need to make it more flexible in terms of how the sequence of RRsets is provided: a single RRset may be provided by multiple add()s, and an RRSIG may be added before the RRset it covers.

2. Sample Code

Using these building blocks described in the previous section, a possible implementation of the loader feature would be as follows. Again, it's not necessarily a proposal, but for providing more concrete ideas of the interfaces.

2.1 Classes/Functions for Loading

class MasterLexer {
public:
    // token types: basically "string" and some special types of tokens
    enum TokenType { END_OF_FILE, END_OF_LINE, STRING, ... };

    MasterLexer(const char* const filename); // trivial constructor
    MasterLexer(std::istream* input_stream); // ditto, but from stream

    // Read one token from input, and tell the caller what it is.
    TokenType readToken();

    // when the next token is a string, return its value.
    const std::string& getStringToken() const;

    // These will be used to locate the position in file when an error happens.
    size_t getSourceLine() const;
    size_t getSourceBytes() const;
    const std::string& getSourceName() const;
};

// The base class of LoaderContext
class LoaderContextBase {
public:
    // Add one RRset to context-specific storage (such as a concrete data
    // source)
    virtual void add(RRsetPtr rrset) = 0;

    // Callback called when the parser finds an error.
    // source_name is the file name when the loader is given a file
    // source_line is the line in the file/stream on/around which the error
    // is detected.
    // source_byte is the offset from the beginning of the file/stream in bytes
    // at/around which the error is detected.
    // msg is a free-format string that describes the error.
    virtual void error(const std::string& source_name,
                       size_t source_line, size_t source_byte,
                       const string& msg) = 0;

    // Similar to error(), but for less critical issues.
    virtual void warn(const std::string& source_name,
                      size_t source_line, size_t source_byte,
                      const std::string& msg) = 0;
};

void
masterLoad(const char* const master_file,
           const Name& zone_origin, RRClass zone_class,
           LoaderContextBase& context, ...) 
{
    MasterLexer lexer(master_file);

    while (lexer.getToken() != MasterLexer::END_OF_FILE) {
        RRsetPtr rrset;
        try {
            lexer.getToken();
            name = lexer.getStringToken();
            lexer.getToken();
            rrtype = lexer.getStringToken();
            lexer.getToken();
            rrclass = lexer.getStringToken();
            // ...
            rrset = RRsetPtr(new RRset(name, rrtype, rrclass, rrttl));
        } catch (const DNSParserError& ex) {
            context.error(lexer.getSourceName(), lexer.getSourceLine(),
                          lexer.getSourceBytes(),
                          "failed to parse an RR: " + ex.what());
            // when lenient skip until EOL; otherwise throw
        }

        // parse RDATA
        try {
            rrset->addRdata(createRdata(rrset->getType(), rrset->getClass(),
                                        lexer, origin));
        } catch (const DNSParserError& ex) {
            context.error(lexer.getSourceName(), lexer.getSourceLine(),
                          lexer.getSourceBytes(),
                          "failed to parse an RDATA: " + ex.what());
            // when lenient skip until EOL; otherwise throw
        }

        // On building a complete RRset:
        context.add(rrset);
    }
}

2.2 Classes/Functions for Validating

// A base class of RRsetCollection.  As noted in the description, this
// collection can be a generic set of RRsets and doesn't have to form a valid
// zone.
class RRsetCollectionBase {
public:
    // Returns the RRset in the container that has the given name and RRtype
    // (exact match).  If not found, return NULL.
    virtual const AbstractRRset* find(const Name& name, RRType rrtype) const
    = 0;

    // Calls callback for each RRset contained in the collection.  Works as
    // a kind of iterator.
    virtual void foreach(boost::function<void(const AbstractRRset&)> callback)
        const = 0;
};

// In libdns++ we'd add a simple implementation of RRsetCollectionBase.
// We'd probably use a straightforward representation using some STL container.
class RRsetCollection : public RRsetCollectionBase {
    // In addition to virtual methods, this one would have the following:

    // Create a new collection from a file in the master file format, which
    // may or may not form a valid zone.
    RRsetCollection(const char* const input_file);

    // update or search the collection, for search, both mutable or immutable.
    void addRRset(RRsetPtr rrset);
    void removeRRset(const Name& name, RRClass rrclass, RRType rrtype);
    RRsetPtr find(const Name& name, RRClass rrclass, RRType rrtype);
    ConstRRsetPtr find(const Name& name, RRClass rrclass, RRType rrtype) const;
};

void
validateZone(const Name& origin, RRClass zone_class,
             const RRsetCollectionBase& rrsets,
             boost::function<void(const string&) error_callback)
{
    const AbstractRRset* rrset;
    if ((rrset = rrsets.find(origin, RRType::SOA())) == NULL) {
        error_callback("zone doesn't have an SOA");
        // When 'strict' continue; otherwise throw
    }
    if ((rrset = rrsets.find(origin, RRType::NS())) == NULL) {
        error_callback("zone doesn't have an NS at apex");
        // When 'strict' continue; otherwise throw
    }
    // and so on...
}

2.3 Loader with Validation for Data Sources

namespace isc { namespace datasrc {

// Data source loader context: its add is a wrapper for
// ZoneUpdater::addRRset().  The error()/warn() will produce a log message.
class LoaderContext : public isc::dns::LoaderContextBase {
public:
    LoaderContext(ZoneUpdater& updater) : updater_(updater) {}
    virtual void add(RRsetPtr rrset) {
        updater_.addRRset(*rrset);
    }

    virtual void error(const std::string& source_name,
                       size_t source_line, size_t source_byte,
                       const string& msg) const
    {
        logger.error(DATASRC_LOADER_ERROR).arg(source_name).arg(source_line).
            arg(source_byte).arg(msg)
    }
};

// RRset collection for a zone being built in a data source: it provides
// the find interface through the updater's zone finder.
class RRsetCollection : public RRsetCollectionBase {
public:
    RRsetCollection(ZoneUpdater& updater);
    virtual const AbstractRRset* find(const Name& name, RRType type) const {
        return (updater.getFinder().find(name, type,
                                         NO_WILDCARD|FIND_GLUE_OK).get());
    }
    // foreach() cannot be implemented with current set of interfaces.
    // we'll probably need to be able to get an iterator from the updater, too.
};

// load zone content into a specified data source from a master file
void
loadZone(DataSourceClient& client,
         const isc::dns::Name& zone_origin,
         const char* const master_file)
{
    updater = client.getUpdater(zone_origin, true);
    isc::dns::masterLoad(master_file, zone_origin, client.getClass(),
                         LoaderContext(*updater));
    isc::dns::validateZone(RRsetCollection(*updater));
    updater->commit();
};

// load zone content into a specified data source from another data source
void
loadZone(DataSourceClient& client,
         DataSourceClient& backend_client,
         const isc::dns::Name& zone_origin)
{
    updater = client.getUpdater(zone_origin, true);
    backend_iterator = backend_client.getIterator(zone_origin);
    ConstRRsetPtr rrset;
    while ((rrset = backend_iterator->getNextRRset()) != NULL) {
        updater->addRRset(*rrset);
    }
    isc::dns::validateZone(RRsetCollection(*updater));
    updater->commit();
}

}}                              // end of datasrc namespace

2.4 Validation after Xfrin (Python)

The xfrin module doesn't need a "loader", but will need to validate the new version of the zone. The code would look like this:

    # on completion of updating the zone, do this.
    # any errors will be reported in logs, but xfrin will accept it.
    isc.dns.validate_zone(isc.datasrc.RRsetCollection(updater))
    updater.commit()
Last modified 5 years ago Last modified on Oct 18, 2012, 5:48:45 PM