Opened 7 months ago

Closed 3 months ago

#5441 closed defect (fixed)

network{4,6}-{add,del} malfunction

Reported by: wlodekwencel Owned by: tomek
Priority: high Milestone: Kea1.4
Component: hooks Version: git
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DHCP Feature Depending on Ticket:
Estimated Difficulty: 0 Add Hours to Ticket: 3
Total Hours: 4 Internal?: no

Description

all 4 commands:
network4-add
network4-del
network6-add
network6-del
are malfunctioning in Kea. Command -add does NOT add shared-network, and command -del does NOT remove it.

  1. Add commands parse config correctly and return success:
    {
      "arguments": {
        "shared-networks": [
          {
            "name": "name-abc"
          }
        ]
      },
      "result": 0,
      "text": "A new IPv6 shared network 'name-abc' added"
    }
    

logs shows subnet added:

[kea-dhcp6.commands/23457] COMMAND_RECEIVED Received command 'network6-add'
DEBUG [kea-dhcp6.callouts/23457] HOOKS_CALLOUTS_BEGIN begin all callouts for hook $network6_add
INFO  [kea-dhcp6.dhcpsrv/23457] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 2001:db8:a::/64 with params t1=1000, t2=2000, preferred-lifetime=3000, valid-lifetime=4000, rapid-commit is disabled
DEBUG [kea-dhcp6.callouts/23457] HOOKS_CALLOUT_CALLED hooks library with index 1 has called a callout on hook $network6_add that has address 0x7f05f03b0da0 (callout duration: 0.635 ms)
DEBUG [kea-dhcp6.callouts/23457] HOOKS_CALLOUTS_COMPLETE completed callouts for hook $network6_add (total callouts duration: 0.635 ms)

and if command "network6-get" is used it returns correct configuration:

{
  "arguments": {
    "shared-networks": [
      {
        "interface": "eth2",
        "name": "name-abc",
        "option-data": [],
        "preferred-lifetime": 0,
        "rapid-commit": false,
        "rebind-timer": 0,
        "relay": {
          "ip-address": "::"
        },
        "renew-timer": 0,
        "reservation-mode": "all",
        "subnet6": [
          {
            "id": 1,
            "interface": "eth2",
            "option-data": [],
            "pd-pools": [],
            "pools": [
              {
                "option-data": [],
                "pool": "2001:db8:a::1/128"
              }
            ],
            "preferred-lifetime": 3000,
            "rapid-commit": false,
            "rebind-timer": 2000,
            "relay": {
              "ip-address": "::"
            },
            "renew-timer": 1000,
            "reservation-mode": "all",
            "subnet": "2001:db8:a::/64",
            "valid-lifetime": 4000
          }
        ],
        "valid-lifetime": 0
      }
    ]
  },
  "result": 0,
  "text": "Info about IPv6 shared network 'name-abc' returned"
}

Kea is unable to assign address from 'added' subnet:

DEBUG [kea-dhcp6.packets/23457] DHCP6_SUBNET_SELECTION_FAILED duid=[00:03:00:01:66:55:44:33:22:11], tid=0xe23d01: failed to select subnet for the client
DEBUG [kea-dhcp6.leases/23457] DHCP6_PROCESS_IA_NA_REQUEST duid=[00:03:00:01:66:55:44:33:22:11], tid=0xe23d01: server is processing IA_NA option with iaid=21235 and hint=(no hint)
DEBUG [kea-dhcp6.options/23457] DHCP6_ADD_STATUS_CODE_FOR_IA duid=[00:03:00:01:66:55:44:33:22:11], tid=0xe23d01: adding Status Code to IA with iaid=21235: NoAddrsAvail(2) "Server could not select subnet for this client"
  1. Command -del also does return success, but it's not removing network from configuration. Kea still assign addresses from removed networks.

Kea is configured with one shared network which network6-list shows:

{
  "arguments": {
    "shared-networks": [
      {
        "name": "name-abc"
      }
    ]
  },
  "result": 0,
  "text": "1 IPv6 network found"
}

network6-del results with:

{
  "arguments": {
    "shared-networks": [
      {
        "name": "name-abc"
      }
    ]
  },
  "result": 0,
  "text": "IPv6 shared network 'name-abc' deleted"
}

Logs:

NFO  [kea-dhcp6.commands/24297] COMMAND_RECEIVED Received command 'network6-del'
DEBUG [kea-dhcp6.callouts/24297] HOOKS_CALLOUTS_BEGIN begin all callouts for hook $network6_del
DEBUG [kea-dhcp6.callouts/24297] HOOKS_CALLOUT_CALLED hooks library with index 1 has called a callout on hook $network6_del that has address 0x7f25952e99c0 (callout duration: 0.109 ms)
DEBUG [kea-dhcp6.callouts/24297] HOOKS_CALLOUTS_COMPLETE completed callouts for hook $network6_del (total callouts duration: 0.109 ms)

and network6-list report that network was removed:

{
  "arguments": {
    "shared-networks": []
  },
  "result": 3,
  "text": "0 IPv6 networks found"
}

But kea still is able to assign address from 'removed' network.

  1. Problem occur for kea4 and kea6
  1. Attached to tickets are logs, commands and config files both for -del and -add commands

Subtickets

Attachments (3)

del.zip (4.8 KB) - added by wlodekwencel 7 months ago.
add.zip (5.8 KB) - added by wlodekwencel 7 months ago.
5441.tar.gz (1.5 KB) - added by tomek 4 months ago.
config, commands used during repro

Download all attachments as: .zip

Change History (15)

Changed 7 months ago by wlodekwencel

Changed 7 months ago by wlodekwencel

comment:1 Changed 6 months ago by tomek

  • Milestone changed from Kea-proposed to Kea1.4
  • Priority changed from medium to high

comment:2 Changed 6 months ago by tomek

  • Owner set to tomek
  • Status changed from new to assigned

comment:3 Changed 6 months ago by tomek

  • Owner changed from tomek to Unassigned
  • Status changed from assigned to reviewing

After some discussion on jabber, I decided to implement subnets-action parameter for networkX-del. This optional parameter governs what to do with subnets: keep or delete them.

Proposed changelog entry:

2X.	[bug,func]	tomek
	A bug has been fixed in network4-add and network6-add commands.
	If the shared network being added contains subnets, those
	subnets are now added properly. A new parameter subnets-action
	has been added to the network4-del and network6-del commands.
	It governs what to do with the subnets within the network
	being deleted: keep or delete.
	(Trac #5441, git abcd)

The code is on premium repo, however there are changes in kea repo that document the new parameter.

The code is now ready for review. This is a fix in subnet_cmds, so it affects subscription package only.

comment:4 Changed 5 months ago by fdupont

  • Owner changed from Unassigned to fdupont

comment:5 Changed 5 months ago by fdupont

  • Add Hours to Ticket changed from 0 to 1
  • Owner changed from fdupont to tomek

Someone from QA should confirm this addresses their concern.

Change year to 2018...

Kea patch is OK.

Argument naming is not very consistent, e.g for delNetwork networks should be networks_cfg

Reading the code it is OK. It passes tests (*). ChangeLog is OK too.

PS (*): Got again (once every ~10 runs) a crash in ca:

[ RUN      ] CtrlAgentCommandMgrTest.forwardListCommands
ca_unittests(37988,0x7fff984df340) malloc: *** error for object 0x7fb01be03150: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
/bin/sh: line 1: 37988 Abort trap: 6           /bin/sh ../../../../libtool --mode=execute ${dir}$tst
FAIL: ca_unittests

If someone else got it too we should open a 1.4-final ticket to track it.

comment:6 Changed 5 months ago by tomek

Parameters renamed as suggested. Ok, waiting for Wlodek to confirm.

comment:7 Changed 5 months ago by tomek

  • Owner changed from tomek to wlodekwencel

comment:8 Changed 5 months ago by wlodekwencel

  • Add Hours to Ticket changed from 1 to 3
  • Owner changed from wlodekwencel to tomek
  • Total Hours changed from 0 to 4

I'm forced to push this ticket back to Tomek because fix for network6-add does not work.
command issued:

{
	"command": "network6-add",
	"arguments": {
		"shared-networks": [{
			"name": "name-abc",
			"interface": "eth2",
			"preferred-lifetime": 3000,
			"rebind-timer": 2000,
			"renew-timer": 1000,
			"valid-lifetime": 4000,
			"subnet6": [{
				"id": 1,
				"interface": "eth2",
				"pools": [{
					"pool": "2001:db8:1::1-2001:db8:1::10"
				}],
				"preferred-lifetime": 3000,
				"rebind-timer": 2000,
				"renew-timer": 1000,
				"reservation-mode": "all",
				"subnet": "2001:db8:1::/64",
				"valid-lifetime": 4000
			}]
		}]
	}
}

As the result I got

{
  "arguments": {
    "shared-networks": [
      {
        "name": "name-abc"
      }
    ]
  },
  "result": 0,
  "text": "A new IPv6 shared network 'name-abc' added"
}

But kea fail to assign leases again:
DHCP6_SUBNET_SELECTION_FAILED duid=[00:03:00:01:f6:f5:f4:f3:f2:01], tid=0x1d56a4: failed to select subnet for the client
after sending command network6-get result was rather interesting:

{
  "arguments": {
    "shared-networks": [
      {
        "interface": "eth2",
        "name": "name-abc",
        "option-data": [],
        "preferred-lifetime": 0,
        "rapid-commit": false,
        "rebind-timer": 0,
        "relay": {
          "ip-address": "::"
        },
        "renew-timer": 0,
        "reservation-mode": "all",
        "subnet6": [
          {
            "id": 1,
            "interface": "eth2",
            "option-data": [],
            "pd-pools": [],
            "pools": [
              {
                "option-data": [],
                "pool": "2001:db8:1::1-2001:db8:1::10"
              }
            ],
            "preferred-lifetime": 3000,
            "rapid-commit": false,
            "rebind-timer": 2000,
            "relay": {
              "ip-address": "::"
            },
            "renew-timer": 1000,
            "reservation-mode": "all",
            "subnet": "2001:db8:1::/64",
            "valid-lifetime": 4000
          }
        ],
        "valid-lifetime": 0
      }
    ]
  },
  "result": 0,
  "text": "Info about IPv6 shared network 'name-abc' returned"
}

timers in shared-network level are different than in configuration send via 'network6-add', so I've created another tests, checking interface-id and relay configuration:

  1. interface-id

configuration send via network6-add:

{
	"command": "network6-add",
	"arguments": {
		"shared-networks": [{
			"name": "name-abc",
			"interface-id": "interface-abc",
			"preferred-lifetime": 3000,
			"rebind-timer": 2000,
			"renew-timer": 1000,
			"valid-lifetime": 4000,
			"subnet6": [{
				"id": 1,
				"pools": [{
					"pool": "2001:db8:1::1-2001:db8:1::10"
				}],
				"preferred-lifetime": 3000,
				"rebind-timer": 2000,
				"renew-timer": 1000,
				"reservation-mode": "all",
				"subnet": "2001:db8:1::/64",
				"valid-lifetime": 4000
			}]
		}]
	}
}

configuration received via network6-get:

{
  "arguments": {
    "shared-networks": [
      {
        "name": "name-abc",
        "option-data": [],
        "preferred-lifetime": 0,
        "rapid-commit": false,
        "rebind-timer": 0,
        "relay": {
          "ip-address": "::"
        },
        "renew-timer": 0,
        "reservation-mode": "all",
        "subnet6": [
          {
            "id": 1,
            "interface-id": "interface-abc",
            "option-data": [],
            "pd-pools": [],
            "pools": [
              {
                "option-data": [],
                "pool": "2001:db8:1::1-2001:db8:1::10"
              }
            ],
            "preferred-lifetime": 3000,
            "rapid-commit": false,
            "rebind-timer": 2000,
            "relay": {
              "ip-address": "::"
            },
            "renew-timer": 1000,
            "reservation-mode": "all",
            "subnet": "2001:db8:1::/64",
            "valid-lifetime": 4000
          }
        ],
        "valid-lifetime": 0
      }
    ]
  },
  "result": 0,
  "text": "Info about IPv6 shared network 'name-abc' returned"
  1. realy

configuration send via network6-add:

{
	"command": "network6-add",
	"arguments": {
		"shared-networks": [{
			"name": "name-abc",
			"relay": {
				"ip-address": "2001:db8::abcd"
			},
			"preferred-lifetime": 3000,
			"rebind-timer": 2000,
			"renew-timer": 1000,
			"valid-lifetime": 4000,
			"subnet6": [{
				"id": 1,
				"pools": [{
					"pool": "2001:db8:1::1-2001:db8:1::10"
				}],
				"preferred-lifetime": 3000,
				"rebind-timer": 2000,
				"renew-timer": 1000,
				"reservation-mode": "all",
				"subnet": "2001:db8:1::/64",
				"valid-lifetime": 4000
			}]
		}]
	}
}

configuration received via network6-get:

{
  "arguments": {
    "shared-networks": [
      {
        "name": "name-abc",
        "option-data": [],
        "preferred-lifetime": 0,
        "rapid-commit": false,
        "rebind-timer": 0,
        "relay": {
          "ip-address": "::"
        },
        "renew-timer": 0,
        "reservation-mode": "all",
        "subnet6": [
          {
            "id": 1,
            "option-data": [],
            "pd-pools": [],
            "pools": [
              {
                "option-data": [],
                "pool": "2001:db8:1::1-2001:db8:1::10"
              }
            ],
            "preferred-lifetime": 3000,
            "rapid-commit": false,
            "rebind-timer": 2000,
            "relay": {
              "ip-address": "2001:db8::abcd"
            },
            "renew-timer": 1000,
            "reservation-mode": "all",
            "subnet": "2001:db8:1::/64",
            "valid-lifetime": 4000
          }
        ],
        "valid-lifetime": 0
      }
    ]
  },
  "result": 0,
  "text": "Info about IPv6 shared network 'name-abc' returned"
}

Some of the configured values are missing (interface-id and relay was moved into subnet level). And I couldn't get any lease from each of those configurations. Very similar issue is with network4-add.

Another thing is a regression:
Now Kea is unable to locate network via it's name:
configuration:

{
    "Dhcp6":
    {
        "renew-timer":1000,
        "rebind-timer":2000,
        "preferred-lifetime":3000,
        "valid-lifetime":4000,
        "interfaces-config":
        {
            "interfaces":["eth2"]
        }
        ,
        "subnet6":[],
        "hooks-libraries":[
        {
            "library":"/home/test/installed/git/lib/hooks/libdhcp_subnet_cmds.so"
        }
        ],
        "lease-database":
        {
            "type":"memfile"
        }
        ,
        "control-socket":
        {
            "socket-type":"unix",
            "socket-name":"/home/test/installed/git/var/kea/control_socket"
        }
        ,
        "shared-networks":[
        {
            "name":"name-abc",
            "interface-id":"interface-abc",
            "subnet6":[
            {
                "subnet":"2001:db8:a::/64",
                "pools":[
                {
                    "pool":"2001:db8:a::1-2001:db8:a::1"
                }
                ]
            }
            ,
            
            {
                "subnet":"2001:db8:b::/64",
                "option-data":[
                {
                    "csv-format":true,
                    "code":23,
                    "data":"2001:db8::1,2001:db8::2",
                    "name":"dns-servers",
                    "space":"dhcp6"
                }
                ],
                "pools":[
                {
                    "pool":"2001:db8:b::1-2001:db8:b::1"
                }
                ]
            }
            ]
        }
        ,
        
        {
            "name":"name-xyz",
            "interface":"eth2",
            "subnet6":[
            {
                "subnet":"2001:db8:c::/64",
                "pools":[
                {
                    "pool":"2001:db8:c::1-2001:db8:c::1"
                }
                ]
            }
            ,
            
            {
                "subnet":"2001:db8:d::/64",
                "pools":[
                {
                    "pool":"2001:db8:d::1-2001:db8:d::1"
                }
                ]
            }
            ]
        }
        ,
        
        {
            "name":"name-something",
            "relay":
            {
                "ip-address":"2001:db8::1234"
            }
            ,
            "subnet6":[
            {
                "subnet":"2001:db8:e::/64",
                "pools":[
                {
                    "pool":"2001:db8:e::1-2001:db8:e::1"
                }
                ]
            }
            ,
            
            {
                "subnet":"2001:db8:f::/64",
                "pools":[
                {
                    "pool":"2001:db8:f::1-2001:db8:f::1"
                }
                ]
            }
            ]
        }
        ]
    }
    ,
    "Logging":
    {
        "loggers":[
        {
            "name":"kea-dhcp6",
            "output_options":[
            {
                "output":"/home/test/installed/git/var/kea/kea.log"
            }
            ],
            "debuglevel":99,
            "severity":"DEBUG"
        }
        ]
    }
    
}

command:
{"command":"network6-get","arguments":{"name":"name-xyz"}}
result:

{
  "result": 3,
  "text": "No 'name-xyz' shared network found"
}

and that makes testing -del command impossible

comment:9 Changed 4 months ago by tomek

There are three separate things reported here:

  1. Failing to select a subnet. I tested this manually and it seems to be working. Wlodek sent me the commands he used. I marginally modified them (different interface name) and they are working for me. I don't know what happened. Perhaps you updated the repository, but forgot to do a separate git pull in premium/ dir?
  1. Invalid (zeroed) values for timers, relay information and interface-id. Yes, I confirm this is a bug (or missing functionality). However, the problem lies in the network configuration parser, which is part of the open source Kea. We can fix it, but won't be able to release it in hooks release. The code will go out in kea 1.4. We should open a new bug for this, though.
  1. network6-del not working. It does, but it keeps the subnets by default. There's a new parameter called subnets-action. If not specified, the default is to delete the network, but keep the subnets. This is by design to minimise the risk of people deleting too much of their configuration.

Anyway, here are the steps I did to try to reproduce the issue:

  1. started kea-dhcp6 with 5441.json (no shared networks there)
  1. started client, it didn't get any addresses. Stopped client.
  1. ran this: socat UNIX:/tmp/kea6-ctrl-socket - < cmds/network6-add

This added a shared network with a subnet in it.

  1. started client again, it did get an address. Stopped client.
  1. ran this: socat UNIX:/tmp/kea6-ctrl-socket - < cmds/network6-list Kea listed a single network.
  1. ran this: socat UNIX:/tmp/kea6-ctrl-socket - < cmds/network6-get

Kea returned details of the shared network with subnet6 in it. Note that, as explained above, timers, relay information and interface-id was zeroed out.

  1. ran this: socat UNIX:/tmp/kea6-ctrl-socket - < cmds/network6-add

This removed the networks and the subnet within it. Note there's subnets-action parameter set to delete. If not specified, the default is "keep". This means that, unless specified, the network is deleted, but the subnet is not.

  1. started client, it didn't get address anymore. Stopped client.
  1. Confirmed that network6-list returns an empty list.
  1. Confirmed that network6-get does not return any subnet.

This is with the Kea code from master and the premium/ code from trac5441, with the last commit being addc287ac2407d406bdba48db504f56ace65cc2197a (commit dated Jan 12th).

Note, there is small change on master on trac5441, but it pertains only to doc update.

Changed 4 months ago by tomek

config, commands used during repro

comment:10 Changed 4 months ago by tomek

  • Owner changed from tomek to wlodekwencel

comment:11 Changed 3 months ago by wlodekwencel

  • Owner changed from wlodekwencel to tomek

tests re-run, all good, ready to merge.

comment:12 Changed 3 months ago by tomek

  • Resolution set to fixed
  • Status changed from reviewing to closed

Whew. It has been merged. Closing

Note: See TracTickets for help on using tickets.