Classes of the Data Source Library

0. About This Document

This memo describes major classes used in the data source library, mainly focusing on handling in-memory cache with consideration of the shared memory support. It will give an overview of the entire design architecture and some specific details of how these classes are expected to be used.

Before reading, the higher level inter-module protocol should be understood:

1. Overall Relationships between Classes

The following diagram shows major classes in the data source library related to in-memory caches and their relationship.

Major design decisions of this architecture are:

  • Keep each class as concise as possible, each focusing on one or small set of responsibilities. Smaller classes are generally easier to understand (at the cost of understanding how they work in the "big picture" of course) and easier to test.
  • On a related point, minimize dependency to any single class. A monolithic class on which many others are dependent is generally difficult to maintain because you'll need to ensure a change to the monolithic class doesn't break anything on any other classes.
  • Use polymorphism for any "fluid" behavior, and hide specific details under abstract interfaces so implementation details won't be directly referenced from any other part of the library. Specifically, the underlying memory segment type (local, mapped, and possibly others) and the source of in-memory data (master file or other data source) are hidden via a kind of polymorphism.
  • Separate classes directly used by applications from classes that implement details. Make the former classes as generic as possible, agnostic about implementation specific details such as the memory segment type (or, ideally and where possible, whether it's for in-memory cache or the underlying data source).

The following give a summarized description of these classes.

  • ConfigurableClientList: The front end to application classes. An application that uses the data source library generally maintains one or more ConfigurableClientList object (usually one per RR class, or when we support views, probably one per view). This class is a container of sets of data source related classes, providing accessor to these classes and also acting as a factory of other related class objects. Note: Due to internal implementation reasons, there is a base class for ConfigurableClientList named ClientList in the C++ version, and applications are expected to use the latter. But conceptually ConfigurableClientList is an independent value class; the inheritance is not for polymorphism. Note also that the Python version doesn't have the base class.
  • DataSourceInfo: this is a straightforward tuple of set of class objects corresponding to a single data source, including DataSourceClient, CacheConfig, and ZoneTableSegment. ConfigurableClientList maintains a list of DataSourceInfo, one for each data source specified in its configuration.
  • DataSourceClient: The front end class to applications for a single data source. Applications will get a specific DataSourceClient object by ConfigurableClientList::find(). DataSourceClient itself is a set of factories for various operations on the data source such as lookup or update.
  • CacheConfig: library internal representation of in-memory cache configuration for a data source. It knows which zones are to be cached and where the zone data (RRs) should come from, either from a master file or other data source. With this knowledge it will create an appropriate LoadAction object. Note that CacheConfig isn't aware of the underlying memory segment type for the in-memory data. It's intentionally separated from this class (see the conciseness and minimal-dependency design decisions above).
  • ZoneTableSegment: when in-memory cache is enabled, it provides memory-segment-type independent interface to the in-memory data. This is an abstract base class (see polymorphism in the design decisions) and inherited by segment-type specific subclasses: ZoneTableSegmentLocal and ZoneTableSegmentMapped (and possibly others). Any subclass of ZoneTableSegment is expected to maintain the specific type of MemorySegment object.
  • ZoneWriter: a frontend utility class for applications to update in-memory zone data (currently it can only load a whole zone and replace any existing zone content with a new one, but this should be extended so it can handle partial updates). Applications will get a specific ZoneWriter object by ConfigurableClientList::getCachedZoneWriter(). ZoneWriter is constructed with ZoneableSegment and LoadAction. Since these are abstract classes, ZoneWriter doesn't have to be aware of "fluid" details. It's only responsible for "somehow" preparing ZoneData for a new version of a specified zone using LoadAction, and installing it in the ZoneTable (which can be accessed via ZoneTableSegment).
  • DataSourceStatus: created by ConfigurableClientList::getStatus(), a straightforward tuple that represents some status information of a specific data source managed in the ConfigurableClientList. getStatus() generates DataSourceStatus for all data sources managed in it, and returns them as a vector.
  • ZoneTableAccessor, ZoneTableIterator: frontend classes to get access to the conceptual "zone table" (a set of zones) stored in a specific data source. In particular, ZoneTableIterator allows applications to iterate over all zones (by name) stored in the specific data source. Applications will get a specific ZoneTableAccessor object by ConfigurableClientList::getZoneTableAccessor(), and get an iterator object by calling getIterator on the accessor. These are abstract classes and provide unified interfaces independent from whether it's for in-memory cached zones or "real" underlying data source. But the initial implementation only provides the in-memory cache version of subclass (see the next item).
  • ZoneTableAccessorCache, ZoneTableIteratorCache: implementation classes of ZoneTableAccessor and ZoneTableIterator for in-memory cache. They refer to CacheConfig to get a list of zones to be cached.
  • ZoneTableHeader, ZoneTable: top-level interface to actual in-memory data. These were separated based on a prior version of the design ( where ZoneTableHeader may contain multiple ZoneTables. It's one-to-one relationship in the latest version (of implementation), so we could probably unify them as a cleanup.
  • ZoneData: representing the in-memory content of a single zone. ZoneTable contains (zero, one or) multiple ZoneData objects.
  • RdataSet: representing the in-memory content of (data of) a single RRset. ZoneData contains RdataSets corresponding to the RRsets stored in the zone.
  • LoadAction: a "polymorphic" functor that implements loading zone data into memory. It hides from its user (i.e., ZoneWriter) details about the source of the data: master file or other data source (and perhaps some others). The "polymorphism" is actually realized as different implementations of the functor interface, not class inheritance (but conceptually the effect and goal is the same). Note: there's a proposal to replace LoadAction with a revised ZoneDataLoader, although the overall concept doesn't change. See Trac ticket #2912.
  • ZoneDataLoader and ZoneDataUpdater: helper classes for the LoadAction functor(s). These work independently from the source of data, taking a sequence of RRsets objects, converting them into the in-memory data structures (RdataSet), and installing them into a newly created ZoneData object.

2. Sequence for Auth Module Using Local Memory Segment

In the remaining sections, we explain how the classes shown in the previous section work together through their methods for commonly intended operations.

The following sequence diagram shows the case for the authoritative DNS server module to maintain "local" in-memory data. Note that "auth" is a conceptual "class" (not actually implemented as a C++ class) to represent the server application behavior. For the purpose of this document that should be sufficient. The same note applies to all examples below.

  1. On startup, the auth module creates a ConfigurableClientList for each RR class specified in the configuration for "data_sources" module. It then calls ConfigurableClientList::configure() for the given configuration of that RR class.
  1. For each data source, ConfigurableClientList creates a CacheConfig object with the corresponding cache related configuration.
  1. If in-memory cache is enabled for the data source, ZoneTableSegment is also created. In this scenario the cache type is specified as "local" in the configuration, so a functor creates ZoneTableSegmentLocal as the actual instance. In this case its ZoneTable is immediately created, too.
  1. ConfigurableClientList checks if the created ZoneTableSegment is writable. It is always so for "local" type of segments. So ConfigurableClientList immediately loads zones to be cached into memory. For each such zone, it first gets the appropriate LoadAction through CacheConfig, then creates ZoneWriter with the LoadAction, and loads the data using the writer.
  1. If the auth module receives a "reload" command for a cached zone from other module (xfrin, an end user, etc), it calls ConfigurableClientList::getCachedZoneWriter to load and install the new version of the zone. The same loading sequence takes place except that the user of the writer is the auth module. Also, the old version of the zone data is destroyed at the end of the process.

3. Sequence for Auth Module Using Mapped Memory Segment

This is an example for the authoritative server module that uses mapped type memory segment for in-memory data.

  1. The sequence is the same to the point of creating CacheConfig.
  1. But in this case a ZoneTableSegmentMapped object is created based on the configuration for the cache type. This type of ZoneTableSegment is initially empty, and isn't even associated with a MemorySgement (and therefore considered "non writable").
  1. ConfigurableClientList checks if the zone table segment is writable to know whether to load zones into memory by itself, but since ZoneTableSegment::isWritable() returns false, it skips the loading.
  1. the auth module gets the status of each data source, and notices there's a "WAITING" state of segment. So it subscribes to the "Memmgr" group on a command session and waits for an update from the memory manager (memmgr) module. (See also the note at the end of the section)
  1. when the auth module receives an update command from memmgr, it calls ConfigurableClientList::resetMemorySegment() with the command argument and the segment mode of "READ_ONLY". Note that the auth module handles the command argument as mostly opaque data; it's not expected to be details of segment type-specific behavior.
  1. ConfigurableClientList::resetMemorySegment() subsequently calls reset() method on the corresponding ZoneTableSegment with the given parameters. In the case of ZoneTableSegmentMapped, it creates a new MemorySegment object for the mapped type, which internally maps the specific file into memory. memmgr is expected to have prepared all necessary data in the file, so all the data are immediately ready for use (i.e., there shouldn't be no explicit load operation).
  1. When a change is made in the mapped data, memmgr will send another update command with parameters for new mapping. The auth module calls ConfigurableClientList::resetMemorySegment(), and the underlying memory segment is swapped with a new one. The old memory segment object is destroyed at this point. Note that this "destroy" just means unmapping the memory region; the data stored in the file are intact.
  1. If the auth module happens to receive a reload command from other module, it could call ConfigurableClientList::getCachedZoneWriter() to reload the data by itself, just like in the previous section. In this case, however, the writability check of getCachedZoneWriter() fails (the segment was created as READ_ONLY, so is considered non writable), so loading won't happen.

Note: while less likely in practice, it's possible that the same auth module uses both "local" and "mapped" (and even others) type of segments for different data sources. In such cases the sequence is either the one in this or previous section depending on the specified segment type in the configuration. The auth module itself isn't aware of per segment-type details, but change the behavior depending on the segment state of each data source at step 4 above: if it's "WAITING", it means the auth module needs help from memmgr (that's all the auth module should know; it shouldn't be bothered with further details such as mapped file names); if it's something else, the auth module doesn't have to do anything further.

4. Sequence for Memmgr Module Initialization Using Mapped Memory Segment

This sequence shows the common initialization sequence for the memory manager (memmgr) module using a mapped type memory segment. This is a mixture of the sequences shown in Sections 2 and 3.

  1. Initial sequence is the same until the application module (memmgr) calls ConfigurableClientList::getStatus() as that for the previous section.
  1. The memmgr module identifies the data sources whose in-memory cache type is "mapped". (Unlike other application modules, the memmgr should know what such types means due to its exact responsibility). For each such data source, it calls ConfigurableClientList::resetMemorySegment with the READ_WRITE mode and other mapped-type specific parameters. memmgr should be able to generate the parameters from its own configuration and other data source specific information (such as the RR class and data source name).
  1. The ConfigurableClientList class calls ZoneTableSegment::reset() on the corresponding zone table segment with the given parameters. In this case, since the mode is READ_WRITE, a new ZoneTable will be created (assuming this is a very first time initialization; if there's already a zone table in the segment, it will be used).
  1. The memmgr module then calls ConfigurableClientList::getZoneTableAccessor(), and calls the getItertor() method on it to get a list of zones for which zone data are to be loaded into the memory segment.
  1. The memmgr module loads the zone data for each such zone. This sequence is the same as shown in Section 2.
  1. On loading all zone data, the memmgr module sends an update command to all interested modules (such as auth) in the segment, and waits for acknowledgment from all of them.
  1. Then it calls ConfigurableClientList::resetMemorySegment() for this data source with almost the same parameter as step 2 above, but with a different mapped file name. This will make a swap of the underlying memory segment with a new mapping. The old MemorySegment object will be destroyed, but as explained in the previous section, it simply means unmapping the file.
  1. The memmgr loads the zone data into the newly mapped memory region by repeating the sequence shown in step 5.
  1. The memmgr repeats all this sequence for data sources that use "mapped" segment for in-memory cache. Note: it could handle multiple data sources in parallel, e.g., while waiting for acknowledgment from other modules.

5. Sequence for Memmgr Module to Reload a Zone Using Mapped Memory Segment

This example is a continuation of the previous section, describing how the memory manager reloads a zone in mapped memory segment.

  1. When the memmgr module receives a reload command from other module, it calls ConfigurableClientList::getCachedZoneWriter() for the specified zone name. This method checks the writability of the segment, and since it's writable (as memmgr created it in the READ_WRITE mode), getCachedZoneWriter() succeeds and returns a ZoneWriter.
  1. The memmgr module uses the writer to load the new version of zone data. There is nothing specific to mapped-type segment here.
  1. The memmgr module then sends an update command to other modules that would share this version, and waits for acknowledgment from all of them.
  1. On getting acknowledgments, the memmgr module calls ConfigurableClientList::resetMemorySegment() with the parameter specifying the other mapped file. This will swap the underlying MemorySegment with a newly created one, mapping the other file.
  1. The memmgr updates this segment, too, so the two files will contain the same version of data.
Last modified 5 years ago Last modified on Jun 13, 2013, 8:48:31 PM

Attachments (6)

Download all attachments as: .zip