Changes between Version 5 and Version 6 of datasrc_config


Ignore:
Timestamp:
May 31, 2012, 11:53:21 AM (5 years ago)
Author:
vorner
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • datasrc_config

    v5 v6  
    2020Also, the In-Memory should be used only in the Auth server and somehow skipped in the others. One possibility is to list all the data sources where the data come from and the selection mechanism would skip it, or it could delegate the updater/iterator requests to the originating data source.
    2121
    22 == Ideas for configurations ==
     22== Agreed-on configuration format ==
    2323
    24 Let's have a look at some possibilities how the configuration might look like.
     24We'll have a list of data sources, ordered by their priority. When searching for a correct data source, we'll be going through the whole list, looking for the best match. From the equal best matches, we take the first one encountered (eg. if we get an exact match, we can terminate right away, if there are two data sources with the same zone, the one closer to the beginning of the list is taken.
    2525
    26 === Meta-data-source approach ===
     26To solve the problem with in-memory data source, each data source has a 'cache' option. This allows loading some of the zones of the data source into an in-memory data source. Each data source that has a cache turned on has its own In-Memory data source, which is checked prior to asking the data source itself. The cache options are following:
     27 * '''off''': No in-memory zone is created at all. The data source is checked directly.
     28 * '''all''': All the zones found in the data source are loaded into the in-memory and the data source is not checked at all.
     29 * '''origins''': Only the names of the zone origins are cached in memory. The list is used to find the closest zone in the data source and only if the one makes sense, the data source is asked to provide it (eg. when the zone is a better match than we already have).
     30 * '''list of zones''': Instead of a string, a list of strings is provided. Each one represents the origin of the zone to be cached. Also, the list of zone origins is taken into the memory. First, the list of known zones is considered. Then when we need to get the zone, we either ask the data source or the in-memory, depending on if it is cached.
    2731
    28 This approach assumes the back-end uses the meta data source.
     32Note that we need a way to refresh the zone list sometimes. We may want to have a command for a start, but we'll need some way to detect the change automatically eventually.
    2933
    30 The idea is, there's a place for single data source configuration. It may be the In-Memory. Or the SQLite3. Or the Meta. A meta would contain a list of sub-data sources and a matching strategy. The Meta data sources could be nested, allowing for more complex matching patterns.
     34If there are multiple In-Memory implementations in future, we may want to provide some kind of 'cache-params' option.
    3135
    32 The most natural approach for solving the In-Memory problems would be that In-Memory would contain a list of data sources too. Each such slave data source would be „slurped“ on load time to the In-Memory in Auth, and it would be considered a Meta data source in case of other servers (XfrIn/XfrOut).
     36It would look something like this:
    3337
    34 Here are some examples of how the configuration could look like:
    35 
    36 '''A server with In-Memory only'''
    3738{{{
    38 'DataSource': {
    39   'type': 'Memory',
    40   'slaves': [
    41     {
    42       'type': 'Sqlite3',
    43       'file': '/some/file.sqlite3'
    44     },
    45     {
    46       'type': 'ZoneFile',
    47       'file': '/some/file.zone'
    48     }
    49   ]
    50 }
    51 }}}
    52 
    53 '''A server with multiple database back-ends'''
    54 {{{
    55 'DataSource': {
    56   'type': 'Meta',
    57   'strategy': 'BestMatch',
    58   'slaves': [
    59     {
    60       'type': 'Sqlite3',
    61       'file': '/some/file.sqlite3'
    62     },
    63     {
    64       'type': 'MySQL',
    65       'db': 'bind10',
    66       'user': 'bind10',
    67       'passwd': "Don't tell anybody"
    68     }
    69   ]
    70 }
    71 }}}
    72 
    73 '''A server with some high-load zones in In-Memory and many more low-load zones in DBs'''
    74 
    75 This one does not touch the database if it is one of the high-load zones. In that case, it does not try to find a better match in the database. But if it is not there, it does a search through everything.
    76 {{{
    77 'DataSource': {
    78   'type': 'Meta',
    79   'strategy': 'FirstMatch',
    80   'slaves': [
    81      {
    82        'type': 'Memory',
    83        'slaves': [
    84          {
    85            'type': 'ZoneFile',
    86            '/some/file.zone'
    87          }
    88        ]
    89      },
    90      {
    91        'type': 'Meta',
    92        'strategy': 'BestMatch',
    93        'slaves': [
    94          {
    95            'type': 'Sqlite3',
    96            'file': '/some/file.sqlite3'
    97          },
    98          {
    99            'type': 'MySQL',
    100            'db': 'bind10',
    101            'user': 'bind10',
    102            'passwd': "Don't tell anybody"
    103          }
    104        ]
    105      }
    106   ]
    107 }
    108 }}}
    109 
    110 === The „Huge list“ approach ===
    111 
    112 We list all the data sources in a list. To each of them, we add some processing information which will state under what conditions we consider what we have as final and stop.
    113 
    114 The valid conditions would be:
    115 
    116  * ExactMatch
    117  * PartialMatch
    118  * Auth ‒ this will stop processing if it is in the authoritative server. The rest is for the others only.
    119  * Always
    120 
    121 The In-Memory is more naturally solved by specifying list of „cached“ zones in each data source. The In-Memory does not have any of its own configuration and is ignored in others than Auth.
    122 
    123 It might be better demonstrated on an example:
    124 
    125 '''A server with an In-Memory only'''
    126 {{{
    127 'datasources': [
     39'DataSources': [
    12840  {
    129     'type': 'Memory',
    130     'stop-if': 'Auth'
    131   },
    132   {
    133     'type': 'ZoneFile',
    134     'file': '/some/file.zone',
    135     'cache-zones': 'all',
    136     'stop-if': 'ExactMatch'
    137   },
    138   {
    139     'type': 'Sqlite3',
    140     'file': '/some/file.sqlite3',
    141     'cache-zones': [
     41    'type': 'SQLite3',
     42    'cache': [
    14243      'example.org',
    14344      'example.com'
    14445    ],
    145     'stop-if': 'ExactMatch'
     46    'params': {
     47      'file': '/some/file.sqlite3'
     48    }
     49  },
     50  {
     51    'type': 'MySQL',
     52    'cache': 'off',
     53    'params': {
     54      'host': 'localhost',
     55      'user': 'bind10',
     56      'password': 'very secret'
     57    }
     58  },
     59  {
     60    'type': 'MasterFiles',
     61    'cache': 'all',
     62    'params': {
     63      'example.cz': '/some/file.zone',
     64      'example.cn': '/another/file.zone'
     65    }
    14666  }
    14767]
    14868}}}
    14969
    150 '''A server with multiple database back-ends'''
    151 {{{
    152 'datasources': [
    153   {
    154     'type': 'Sqlite3',
    155     'file': '/some/file.sqlite3',
    156     'stop-if': 'ExactMatch',
    157     'cache-zones': 'none'
    158   },
    159   {
    160     'type': 'MySQL',
    161     'db': 'bind10',
    162     'user': 'bind10',
    163     'passwd': "Don't tell anybody",
    164     'stop-if': 'ExactMatch',
    165     'cache-zones': 'none'
    166   }
    167 ]
    168 }}}
     70The 'params' section is a free-form and depends on the data source used.
    16971
    170 '''A server with some high-load zones in In-Memory and many more low-load zones in DBs'''
    171 {{{
    172 'datasources': [
    173   {
    174     'type': 'Memory',
    175     'stop-if': 'PartialMatch'
    176   },
    177   {
    178     'type': 'Sqlite3',
    179     'file': '/some/file.sqlite3'
    180     'stop-if': 'ExactMatch',
    181     'cache-zones': 'none'
    182   },
    183   {
    184     'type': 'MySQL',
    185     'db': 'bind10',
    186     'user': 'bind10',
    187     'passwd': "Don't tell anybody",
    188     'stop-if': 'Auth',
    189     'cache-zones': 'none'
    190   },
    191   {
    192     # This one is to fill the in-memory only (and to provide XfrOut of the zones)
    193     'type': 'ZoneFile',
    194     'file': '/some/file.zone',
    195     'cache-zones': 'all',
    196     'stop-if': 'ExactMatch'
    197   }
    198 ]
    199 }}}
     72The 'MasterFiles' data source is a special one. It does not work as a data source and works only to load zone files into the memory. The 'cache' option must be set to 'all' for it.
    20073
    201 Note that many options could be omitted as defaults. What are the right defaults could be a question, but I think it would be {{{'stop-if': 'ExactMatch', 'cache-zones': 'none'}}}.
    202 
    203 === The separate data sources and matching config ===
    204 
    205 In this approach, we separate the data source configuration from the description on how to match them. There are two advantages:
    206 
    207  * The „free-form“ part of configuration (as each data source will have different set of options) is minimized.
    208  * We can have data sources that are not listed in the matching (for some special purposes, as sources of data for in-memory or just being disabled for the time).
    209 
    210 It seems easier to solve the problem of In-Memory by listing data source names and zones to load from them in the In-Memory config. The In-Memory could then delegate the requests for updater/iterator to the underlying data source. Alternatively, we could have multiple matching lists for auth and the rest.
    211 
    212 '''A server with an In-Memory only'''
    213 {{{
    214 'datasources': {
    215   'memory': {
    216     'type': 'Memory',
    217     'load-zones': {
    218       'zonefiles': 'all',
    219       'sqlite3': [
    220         'example.org',
    221         'example.com'
    222       ]
    223     }
    224   },
    225   'zonefiles': {
    226     'type': 'ZoneFile',
    227     'file': '/some/file.zone'
    228   },
    229   'sqlite3': {
    230     'type': 'Sqlite3',
    231     'file': '/some/file.sqlite3'
    232   }
    233 },
    234 'match-order': [
    235   {
    236     'datasource': 'memory',
    237     'stop-if': 'ExactMatch'
    238   }
    239 ]
    240 }}}
    241 
    242 '''A server with multiple database back-ends'''
    243 {{{
    244 'datasources': {
    245   'sqlite3': {
    246     'type': 'Sqlite3',
    247     'file': '/some/file.sqlite3'
    248   },
    249   'mysql': {
    250     'type': 'MySQL',
    251     'db': 'bind10',
    252     'user': 'bind10',
    253     'passwd': "Don't tell anybody"
    254   }
    255 },
    256 'match-order': [
    257   {
    258     'datasource': 'sqlite3',
    259     'stop-if': 'ExactMatch'
    260   },
    261   {
    262     'datasource': 'mysql',
    263     'stop-if': 'ExactMatch'
    264   }
    265 ]
    266 }}}
    267 
    268 '''A server with some high-load zones in In-Memory and many more low-load zones in DBs'''
    269 {{{
    270 'datasources': {
    271   'memory': {
    272     'type': 'Memory',
    273     'load-zones': {
    274       'zone-files': 'all'
    275     }
    276   },
    277   'sqlite3': {
    278     'type': 'Sqlite3',
    279     'file': '/some/file.sqlite3'
    280   },
    281   'mysql': {
    282     'type': 'MySQL',
    283     'db': 'bind10',
    284     'user': 'bind10',
    285     'passwd': "Don't tell anybody"
    286   },
    287   'zone-files': {
    288     'type': 'ZoneFile',
    289     'file': '/some/file.zone'
    290   }
    291 }
    292 'match-order': [
    293   {
    294     'datasource': 'memory',
    295     'stop-if': 'PartialMatch'
    296   },
    297   {
    298     'datasource': 'sqlite3',
    299     'stop-if': 'ExactMatch'
    300   },
    301   {
    302     'datasource': 'mysql',
    303     'stop-if': 'ExactMatch'
    304   }
    305 ]
    306 }}}
    307 
    308 Alternatively, with the separate lists, the ``'datasources'`` part would be the same, but then we would have something like:
    309 {{{
    310 'match-order-auth': [
    311   {
    312     'datasource': 'memory',
    313     'stop-if': 'PartialMatch'
    314   },
    315   {
    316     'datasource': 'sqlite3',
    317     'stop-if': 'ExactMatch'
    318   },
    319   {
    320     'datasource': 'mysql',
    321     'stop-if': 'ExactMatch'
    322   }
    323 ]
    324 'match-order-other': [
    325   {
    326     'datasource': 'zone-files',
    327     'stop-if': 'PartialMatch'
    328   },
    329   {
    330     'datasource': 'sqlite3',
    331     'stop-if': 'ExactMatch'
    332   },
    333   {
    334     'datasource': 'mysql',
    335     'stop-if': 'ExactMatch'
    336   }
    337 ]
    338 }}}
     74Some components will enable caches for them (eg. Auth), while others will not do caching at all (XfrIn, for example).
    33975
    34076== Multiple classes ==
     
    34581'DataSourceConfiguration': {
    34682  'IN': {
    347     'datasources': {
    348       ...
    349     },
    350     'match-order': [
    351       ...
    352     ]
     83    ...
    35384  },
    35485  'CH': {
    355     'datasources': {
    356       ...
    357     },
    358     'match-order': [
    359       ...
    360     ]
     86    ...
    36187  }
    36288}