wiki:StatsModule

Statistics modules

  1. Introduction
  2. Stats
    1. Procedure of the stats module
    2. Collecting and reporting items
    3. Command to request statistics data
    4. How stats requests to each module
    5. How to identify how many instances of same module are working
    6. Partial statistics updates
    7. Requirements for the target module returning statistics data
    8. Available commands
  3. StatsHttpd
    1. Basic design and concepts
    2. Sequence diagram
    3. XML files
  4. StatsSnmp
    1. Basic Design
    2. Related files
    3. OID in SNMP
    4. Configuration for Net-SNMP
    5. MIB structure
    6. Name and data type mapping between MIB file and spec file
  5. TODO related to statistics modules

Introduction

There are three statistics modules in BIND 10. One is the main module Stats(b10-stats), and others are sub modules StatsHttpd(b10-stats-httpd) and StatsSnmp(b10-stats-snmp). Stats collects statistics data from target modules like Auth, Boss and itself, and reports the data via bindctl. StatsHttpd is the HTTP/XML interface and responds the statistics data from Stats in HTTP/XML. On the other hand, StatsSnmp is the SNMP interface and responds the statistics data in SNMP. StatsHttpd is an independent daemon but StatsSnmp is an script working with Net-SNMP(snmpd).

					        +--------------+
		                           +--->|   bindctl    |
		statistics flows           |	+--------------+
+---------------+     +--------------+     |    +--------------+
| Target modules|---->|    Stats     |---->+--->|  StatsHttpd  |
|(Auth, Boss ..)|     +--------------+     |    +--------------+
+---------------+                          |    +--------------+
					   +--->|  StatsSnmp   |
					   	+--------------+

Stats

  • The stats module collects stats data from each module.
  • The stats module reports statistics information via bindctl.
  • The stats module and target modules communicate by using CC protocol.
  • The stats module requests statistics data to target modules periodically, then target modules reply the data to the stats module.
  • The stats module collects data and aggregates it.
    +------- Target modules -----+
    | +------+ +------+ +------+ |
    | | Boss | | Auth | | etc. | | <- *1
    | +------+ +------+ +------+ |
    +-----^--------^--------^----+
          |        |        |
          +--[CC protocol]--+
                   |               <- *2
                   |
            +--------------+
            |    Stats     |       <- *3
            +--------------+
                   |               <- *4
                   v
          +-----------------+
          |    Cmd-ctrld    |      <- *5
          +-----------------+
    
    *1 The target modules are listening for the stats modules' requests.
    *2 The stats module requests statistics data to each module by using CC protocol.
    *3 The stats module collects data and aggregate it.
    *4 The stats module reports stats message with aggregated data.
    *5 Commands related to statistics are invoked by BIND 10 user.
    

Procedure of the stats module

Basic procedure

Startup procedure

  1. Boss starts the stats module and target modules.
  2. Spawned the stats module creates ConfigManager object and contacts config manager process.
  3. Stats starts to subscribe to stats channel "Stats".
              spawns
    +------+ -------> +-------+ creates a ConfigManager object
    | Boss |    	  | Stats |
    +------+ <------- +-------+
                ack
    

Collecting procedure

  1. The stats module requests statistics data to target modules periodically.
  2. The target modules answer the request and send the data.
  3. The stats module receives that data and aggregates it.
    +------+  requests
    | Boss | <------- +-------+ requesting to each module
    +------+  	  | Stats |  / receives data and aggregates
    +------+ <------- +-------+
    | Auth |  requests
    +------+
    

Reporting procedure

  1. When commands related to the stats module like "show" are invoked via bindctl, stats daemon reports a result as formatted statistics data via bindctl. If more than one seconds past since the last request to each module, the stats module requests each module statistics data and then shows the latest result. Otherwise, the stats module just shows statistics data which it has. Thus, regarding whether the stats module requests each module on this command, the stats module should memorize the last time it requested.
            invoke
            command                send command
             ------> +-----------+ -----------> +-------+
    [ User ]         | Cmd-Ctrld |              | Stats |
             <------ +-----------+ <----------- +-------+
            print                  send result
            to stdout         (aggregated and formatted)
    

Config-change procedure

  1. Configuration updates and commands from cmd-ctrld come from config manager to the stats module.
                   set_config		      config_update
    +-----------+ -----------> +---------------+ -----------> +-------+
    | Cmd-ctrld |  		   | ConfigManager |   		  | Stats |
    +-----------+ <----------- +---------------+ <----------- +-------+
    	          ack   		      	  ack
    

Shutdown procedure

  1. When Boss is shutting down or "shutdown" command for stats module is invoked, the stats module is killed.
           kills
+------+ -------> +-------+
| Boss |    	  | Stats |
+------+ <------- +-------+
            ack

Collecting and reporting items

The details of statistics items of each module are in each actual spec file: stats.spec, auth.spec, bob.spec and so on.

Stats
stats.spec
Boss
bob.spec
Auth
auth.spec

Command to request statistics data

getstats is a command invoked by the stats module to request statistics data to the target module. The command is only for inter-communication use between the stats module and each target module. Thus the command isn't designed to be invoked via bindctl. So the command spec doesn't need to be defined in spec file of each module.

This command has no argument. But in future release, the stats module will specify the names of statistics data which the stats module wants to know in a argument. In current implement, the target module should return all statistics data which it owns if this command is invoked.

The channel used in CC-session for obtaining statistics data is target module's one. For example, in a case of requesting to the auth module Auth channel is used.

When a target module is requested by the stats module, the module returns statistics data which the target module owns. Before returning statistics data, the module can check the validity of it by using such a function as validateStatistics (C++ variant) and validate_statistics (Python variant). The returned value including the return code from the module should be created by the function: createAnswer(C++ variant) or create_answer(Python variant). These are provided from CC-session library. Refer other API documents or the source codes for more details of these functions. The first argument of the function means a return code. Return code 0 means success, None-zero means an error. In the second argument, content of the statistics data is included. In case of an error, the target module should return an error message in the second argument in a text form.

When stats gets statistics data from the target module, stats should check the return code and then check whether the structure of the statistics data is valid or not. Checking should be done by using the statistics spec of the spec file of the module. Such statistics spec of each module can be obtained from ConfigManager. If the stats module finds invalid statistics data, it ignores that and output STATS_RECEIVED_INVALID_STATISTICS_DATA message into a log file.

If it's valid, the stats module itself should internally replace only the value of the corresponding statistics data with a new value. If the stats module doesn't internally have the value of the corresponding data, it should just add as a new value into the internal variable. The stats module never sums the existent value and a new value. Because such value is handled in the stats module as not a relative value but a absolute value. When no statistics data is returned with a return code 0 which means success, the stats should do nothing and go requesting to the next module.

This is an example python code in the target module to which the stats module requests:

...
# set pairs of statistics names and its data into a dict-type variable
statistics_data = ...
...
def command_handler(self, command, args):
...
    # if "getstats" command is invoked
    elif command == "getstats":
        # the dict-type variable statistics_data should be valid with statistics
        # spec, so check validity of it. This checking is not mandatory.
        valid = self.ccs.get_module_spec().validate_statistics(False, statistics_data)
        if valid:
            # return all of statistics data which the target module owns
            return isc.config.ccsession.create_answer(0, statistics_data)
...

This is an example python code in the stats module which requests to the target module:

...
for module_name in module_list:
    cmd = isc.config.ccsession.create_command("getstats", None)
    seq = self.cc_session.group_sendmsg(cmd, module_name)
    try:
        answer, env = self.cc_session.group_recvmsg(False, seq)
        if answer:
            rcode, args = isc.config.ccsession.parse_answer(answer)
            if rcode == 0:
                errors = self.validate_statistics(args):
                if errors:
                    # specfile is incorrect
                else:
                    self.update_statistics_data(args)
                    ...
    except isc.cc.session.SessionTimeout:
        pass
...

How stats requests to each module

The stats module sequentially requests to a set of modules. After the stats module finished requesting to the first module, stats starts requesting to the second one. The stats module isn't intended to use thread for parallel processing of requesting. If the stats module can get no answer from the target module or SessionTimeout exception is raised, the stats module might give up requesting to the module and request to the next module. The stats module gets the list of module names to which stats requests from ConfigManager. The stats module can request to only the target modules which have statistics spec in their spec files. the stats module has its own statistics spec in its spec file but the case of the stats module is exceptional. the stats module itself internally update its statistics data. It never request to itself via CC-session. When stats finished requesting all modules specified in the config, the stats module sleeps for some time and then restarts requesting to the first module. If there is a target module from which stats failed to obtain statistics due to an error at the former round, the stats module requests that module at the second or later round. Such polling interval in seconds should be configurable in the stats config. Polling interval should be an integer and greater than 0. If 0 is set, the stats module never requests. If the value less than 0 is set, the stats module ignores that.

This is an example configuration of the stats module which the interval seconds to request presents:

    "config_data": [
...
      { "item_name": "poll-interval",
        "item_type": "integer",
        "item_optional": true,
        "item_default": 60
      },
...

How to identify how many instances of same module are working

Some of modules can be propagated into multiple instances. e.g. Auth and Resolver. In that case, the stats module exactly should get statistics data from each instance. Before requesting statistics data to each target module, the stats module needs to know which module has multiple instances. For getting that information, the stats module asks the boss module show_processes via a !Boss channel. This example of the result of show_processes is:

> Boss show_processes
[
...
    [
        20061,
        "b10-auth",
        "Auth"
    ],
...
    [
        20103,
        "b10-auth-2",
        "Auth"
    ],
...
]

In this example, it's found that a least two Auth instances are running. Due to such information from this command, the stats module can identify how many instances of same module are running by counting instances running under Boss. Stats should consolidate statistics data from each instance and then show statistics data per module.

Statistics data collected from each module should be all preserved while the the stats module is working. For example, some of modules or instances might be shut down or restarted unexpectedly. Even in such case, the stats module should preserve statistics data collected from the module before its shutdown.

Partial statistics updates

The stats module should support "partial statistics updates". Here "partial statistics updates" mean: each target module returns only statistics data which are updated inside the target module since the last time it returned, and then the stats module receives it and updates only that corresponding data internally. For example, when at first the target module returns the following data to the stats module:

{
  "counterX": 1,
  "counterY": 2,
  "counterZ": 3
}

then the stats module receives it and updates internally to same statistics data for the module. Secondary, when the target module returns the following data:

{
  "counterX": 4,
  "counterY": 5
}

then the stats data receives it again and statistics data which the stats module has are:

{
  "counterX": 4,
  "counterY": 5,
  "counterZ": 3
}

Thirdly, when the "counterZ" value is updated inside the target module and when it returns the following data to the stats module:

{
  "counterZ": 100
}

finally the stats module internally updates to the following data:

{
  "counterX": 4,
  "counterY": 5,
  "counterZ": 100
}

Thus the stats module shouldn't touch the value until the corresponding data come from the target module. But all collected statistics data are reset to default values if the stats module restarts. This way of updating is for reducing the amount of statistics data sent in the message queue.

Requirements for the target module returning statistics data

  • getstats command should be implemented in the module so that the stats can collect statistics data via the command. This command is invoked via channel which the target module is listening on e.g. Auth, Boss and so on.
  • The module should return all of statistics data which the module owns. (In future release, the module will return statistics data of names specified by the stats module.)
  • Before returning, the module can check the validity of statistics data by using such a function as validateStatistics (C++ variant) and validate_statistics (Python variant). These are provided from CC-session library. This checking is mandatory.
  • The module should return by the function: createAnswer(C++ variant) or create_answer(Python variant).
  • If the command is successfully done, the module should return (0, statistics data).
  • If the target module caught an error when returning, the module should return (1, 'error message'), the return code is 1 and the error message is included detail of the error. Error message is text type.
  • The structure of statistics data should be defined in statistics spec in its own spec file e.g. auth.spec. The type of statistics data is map(hash). The key is a element name of statistics data and the value contains the value of the element. For reference here is an example as followings. The statistics spec defined in a spec file is:
        ...
        "statistics": [
          {
            "item_name": "xcounter",
            "item_type": "integer",
            "item_optional": false,
            "item_default": 0,
            "item_title": "X counter",
            "item_description": "The X counter is ... "
          },
          {
            "item_name": "ycounter",
            "item_type": "integer",
            "item_optional": false,
            "item_default": 0,
            "item_title": "Y counter",
            "item_description": "The Y counter is ..."
          }
       ]
    
    Statistics data in JSON format flowing in CC-Session is:
    { "xcounter": 1235,
      "ycounter": 5678 }
    
  • Requirements of functions and spec file which the module uses should refer to target modules documents and API documents.
  • The target module may support "partial statistics updates" for reducing the amount of data sent in the message queue above.

Available commands

Following commands are available in the stats module. All commands of the stats module can be defined in spec file stats.spec. These commands are intended to be invoked in bindctl.

"status" command
Show status of the stats daemon
"show" command
Show the specified/all statistics data
"showschema" command
Show the specified/all statistics schema
"shutdown" command
Shut down the stats module
"help" command
Show help

Examples of command via bindctl:

> Stats show
{
    "Auth": {
        "queries.tcp": 81120,
        "queries.udp": 81120
    },
    "Boss": {
        "boot_time": "2010-07-26T02:33:41Z"
    },
    "Stats": {
        "boot_time": "2010-07-26T02:32:36Z",
        "last_update_time": "2010-07-27T05:27:05Z",
        "lname": "4c4cf3c4_b@covm",
        "report_time": "2010-07-27T05:27:06Z",
        "timestamp": 1280208426.29731
    }
}
>
> Stats show owner=Stats name=lname
{
    "Stats": {
        "lname": "4c4cf3c4_b@covm"
    }
}

Python samples using msgq library::

A case of 'show' command with argument:

import isc.cc.session
import isc.config.ccsession

sess = isc.cc.session.Session()
sess.group_subscribe("Stats", "*")
# For example, show the value of specified name
cmd = isc.config.ccsession.create_command('show',
                                          {"owner": "Auth", "name": "queries.udp"}
                                         )
seq = sess.group_sendmsg(cmd, "Stats")
msg, env = sess.group_recvmsg(False, seq)
if 'result' in msg:
    ret, arg = isc.config.ccsession.parse_answer(msg)
    print(arg)
sess.group_reply(env, isc.config.ccsession.create_answer(0))
sess.group_unsubscribe("Stats", "*")
sess.close()

Compatibility of supported commands

command via bindctl using ccsession lib
status supported supported
show supported supported
showschmea supported supported
shutdown supported supported
help supported not supported

StatsHttpd

Basic design and concepts

  • The HTTP server process is separated from the current stats-daemon process.
  • The HTTP server is executed and exited by the Boss process.
  • When the HTTP server is requested by a HTTP client, the server requests statistics data to the stats daemon, and responds it to the client in XML format.
  • CC session is used in communication between these two processes.
  • HTTP-server libraries in Python are used in implementation of the HTTP server.
  • Options like the listening address and the port number are specified in the spec file of the HTTP server and the HTTP server obtains its configuration from the config manager(Cfgmgr). (These options cannot be specified in command-line options of the HTTP server.)
  • The HTTP server is checking the status of the stats daemon until the server receives the message "Alive" from the daemon. (The request to check the status will be timed out while the daemon is not yet running.)

Sequence diagram

Procedures and interactions in XML/HTTP interface http_interface.png

XML files

This interface produces some XML files. These are examples:

StatsSnmp

Basic Design

StatsSnmp is a EXPERIMENTAL script named b10-stats-snmp and is NOT released yet. b10-stats-snmp works together with the NET-SNMP agent. It is independent from the normal BIND 10 modules. Add the pass_persist directive into the end of your snmpd.conf and make the snmpd process reload the config file. And then a set of statistics data in BIND 10 in SNMP will be received if BIND 10 is installed properly, and if the SNMP-related command like snmpwalk or snmpget is invoked with proper arguments.

Related files

  • b10-stats-snmp -- A script working under NET-SNMP manager.
  • gen-mibfile.py -- A tool generating the MIB file(ISC-BIND10-MIB.txt) for BIND 10 by reading from each spec file.
  • ISC-BIND10-MIB.txt -- A MIB file auto-generated by the above tool gen-mibfile.py.

OID in SNMP

The OID used in these programs is .1.3.6.1.4.1.2021.255. The parent OID ucdavis (.1.3.6.1.4.1.2021) is prepared for experimental purposes. So it can be replaced with another OID for a enterprise purpose in future.

Configuration for Net-SNMP

b10-stats-snmp is invoked by Net-SNMP manager as above. In order to make the pass_persist in snmpd.conf enabled, The administrator may need to edit snmpd.conf by hands after installing BIND 10. The administrator may need to add other configurations depending on some other needs. The configuration additionally needed in snmpd.conf for BIND 10 is like the following.

pass_persist .1.3.6.1.4.1.2021.255 /usr/local/bin/b10-stats-snmp

This configuration can be easily obtained by execution like the following.

# ./b10-stats-snmp --help 2>>/etc/snmp/snmpd.conf

You have to reload the snmpd process after editing snmpd.conf for enabling the change. The following is an example for the way to do that.

# kill -HUP `cat /var/run/snmpd.pid`

For obtaining a set of statistics for BIND 10 via SNMP, you may use the snmpwalk command or the snmpget command which are bundled with NET-SNMP. You can execute these commands like the followings if BIND 10 processes are running on the same host.

% snmpwalk -c public -v1 localhost ISC-BIND10-MIB::statistics
ISC-BIND10-MIB::authQueriestcp.0 = INTEGER: 2
ISC-BIND10-MIB::authQueriesudp.0 = INTEGER: 1
ISC-BIND10-MIB::bossBoottime.0 = STRING: 2011-12-02T10:23:07Z
ISC-BIND10-MIB::statsReporttime.0 = STRING: 2011-12-02T10:24:28Z
ISC-BIND10-MIB::statsBoottime.0 = STRING: 2011-12-02T10:23:09Z
ISC-BIND10-MIB::statsLastupdatetime.0 = STRING: 2011-12-02T10:24:09Z
ISC-BIND10-MIB::statsTimestamp.0 = Timeticks: (1322821468) 153 days, 2:30:14.68
ISC-BIND10-MIB::statsLname.0 = STRING: bar@foo
% snmpget -c public -v1 localhost ISC-BIND10-MIB::authQueriesudp.0
ISC-BIND10-MIB::authQueriesudp.0 = INTEGER: 1

MIB structure

MIB structure used in the SNMP feature is the following. statistics(255) is located under ucdavis(2021). Other MIB files such as UCD-SNMP-MIB.txt may be also required at same time but some required MIBs will be installed together when NET-SNMP is installed.

+--statistics(255)
   |
   +--auth(0)
   |  |
   |  +-- -R-- Integer32 authQueriestcp(0)
   |  +-- -R-- Integer32 authQueriesudp(1)
   |  |
   |  +--authQueriesperzoneTable(2)
   |     |
   |     +--authZonesEntry(0)
   |        |
   |        +-- -R-- String    authZonename(0)
   |        |        Textual Convention: DisplayString
   |        |        Size: 0..255
   |        +-- -R-- Integer32 authQueriesudp(1)
   |        +-- -R-- Integer32 authQueriestcp(2)
   |
   +--boss(1)
   |  |
   |  +-- -R-- String    bossBoottime(0)
   |           Textual Convention: DisplayString
   |           Size: 0..255
   |
   +--stats(2)
      |
      +-- -R-- String    statsReporttime(0)
      |        Textual Convention: DisplayString
      |        Size: 0..255
      +-- -R-- String    statsBoottime(1)
      |        Textual Convention: DisplayString
      |        Size: 0..255
      +-- -R-- String    statsLastupdatetime(2)
      |        Textual Convention: DisplayString
      |        Size: 0..255
      +-- -R-- TimeTicks statsTimestamp(3)
      +-- -R-- String    statsLname(4)
               Textual Convention: DisplayString
               Size: 0..255

The above structure can be obtained by the snmptranslate command which is bundled with NET-SNMP. you can invoke like the following.

% snmptranslate -Tp "ISC-BIND10-MIB::statistics"

An example MIB file ISC-BIND10-MIB.txt is attached in this page. This will be auto-generated by gen-mibfile.py, so you don't need to make from scratch.

Name and data type mapping between MIB file and spec file

The data type mapping between the MIB and the spec file is the following. gen-mibfile.py generates a MIB file according to the following table. Some special characters in the item name like _ and . cannot be used in MIB file, so such characters will be removed by generation of gen-mibfile.py. But confliction between the duplicate item names may happen after such removals. That may become issues regarding identifying name in SNMP. We should furthermore consider name mapping between the MIB file and the spec file.

Name in MIB Data type in MIB Owner module name in spec file item_name in spec file Data type in spec file
authQueriestcp INTEGER Auth queries.tcp integer
authQueriesudp INTEGER Auth queries.udp integer
bossBoottime STRING Boss boot_time string
statsReporttime STRING Stats report_time string
statsBoottime STRING Stats boot_time string
statsLastupdatetime STRING Stats last_update_time string
statsTimestamp Timeticks Stats timestamp real
statsLname STRING Stats lname string

TODO related to statistics modules

  • Add more collecting and reporting items
  • If the incremental counting is required for the stats module, we'll discuss and implement then. It isn't implemented in the initial version of stats module.
  • Consider name mapping between the MIB file and the spec file
  • Consider prevention of duplicate item names in MIB
  • Consider how to change the MIB file dynamically. e.g. Consider the situation where Auth module is disabled and Resolver module is enabled.
  • Consider a SNMP daemon for BIND 10 independent from the NET-SNMP agent and able to be replaced with the b10-stats-snmp script
Last modified 5 years ago Last modified on Aug 28, 2012, 2:13:40 AM

Attachments (6)

Download all attachments as: .zip