wiki:datasrc_config

Data source configuration

As we want to have multiple data sources available in parallel, we need to be able to configure them somehow. From this configuration, we need following:

  • Be able to specify multiple data sources, including their specific configuration parameters.
  • Define some matching strategy deciding which data source should be used to handle each query.
  • In case the in-memory is not going to be in shared memory (and it seems it will not come soon), allow for XfrOut? not to use in-memory, but use the underlying source it was loaded from instead.
  • Allow iterating through all the possible data sources for XfrIn?/DDNS to find the correct one for update.
  • Make it simple enough so users understand what is happening.
  • Make sure the configuration doesn't create serious performance problems.

In-Memory specialities

The In-Memory data source is special in many ways, therefore it probably needs special handling.

First, the data source needs to load specific zones from somewhere, possibly from multiple different data sources. We may specify this in the In-Memory configuration, or in configuration of each specific data source.

Also, the In-Memory should be used only in the Auth server and somehow skipped in the others. One possibility is to list all the data sources where the data come from and the selection mechanism would skip it, or it could delegate the updater/iterator requests to the originating data source.

Agreed-on configuration format

We'll have a list of data sources, ordered by their priority. When searching for a correct data source, we'll be going through the whole list, looking for the best match. From the equal best matches, we take the first one encountered (eg. if we get an exact match, we can terminate right away, if there are two data sources with the same zone, the one closer to the beginning of the list is taken.

To solve the problem with in-memory data source, each data source has a 'cache' option. This allows loading some of the zones of the data source into an in-memory data source. Each data source that has a cache turned on has its own In-Memory data source, which is checked prior to asking the data source itself. The cache options are following:

  • off: No in-memory zone is created at all. The data source is checked directly.
  • all: All the zones found in the data source are loaded into the in-memory and the data source is not checked at all.
  • origins: Only the names of the zone origins are cached in memory. The list is used to find the closest zone in the data source and only if the one makes sense, the data source is asked to provide it (eg. when the zone is a better match than we already have).
  • list of zones: Instead of a string, a list of strings is provided. Each one represents the origin of the zone to be cached. Also, the list of zone origins is taken into the memory. First, the list of known zones is considered. Then when we need to get the zone, we either ask the data source or the in-memory, depending on if it is cached.

Note that we need a way to refresh the zone list sometimes. We may want to have a command for a start, but we'll need some way to detect the change automatically eventually.

If there are multiple In-Memory implementations in future, we may want to provide some kind of 'cache-params' option.

It would look something like this:

'DataSources': [
  {
    'type': 'SQLite3',
    'cache': [
      'example.org',
      'example.com'
    ],
    'params': {
      'file': '/some/file.sqlite3'
    }
  },
  {
    'type': 'MySQL',
    'cache': 'off',
    'params': {
      'host': 'localhost',
      'user': 'bind10',
      'password': 'very secret'
    }
  },
  {
    'type': 'MasterFiles',
    'cache': 'all',
    'params': {
      'example.cz': '/some/file.zone',
      'example.cn': '/another/file.zone'
    }
  }
]

The 'params' section is a free-form and depends on the data source used.

The 'MasterFiles?' data source is a special one. It does not work as a data source and works only to load zone files into the memory. The 'cache' option must be set to 'all' for it.

Some components will enable caches for them (eg. Auth), while others will not do caching at all (XfrIn?, for example).

Multiple classes

We need to be able to specify data sources for different classes. To do so, we would have a named set with the names being the classes and containing the respective configurations, like this:

'DataSourceConfiguration': {
  'IN': {
    ...
  },
  'CH': {
    ...
  }
}
Last modified 5 years ago Last modified on May 31, 2012, 11:53:21 AM