Opened 7 years ago

Closed 6 years ago

#1833 closed defect (duplicate)

XFROUT_RECEIVE_FILE_DESCRIPTOR_ERROR logged 16 thousand times in two seconds on shutdown

Reported by: jreed Owned by:
Priority: medium Milestone:
Component: xfrout Version:
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DNS Feature Depending on Ticket:
Estimated Difficulty: 3 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no

Description

I restarted bind10 on n10 today in and two seconds I had XFROUT_RECEIVE_FILE_DESCRIPTOR_ERROR logged 16663 times.

Maybe this related to #1537.

This was on a normal and working "Boss shutdown":

2012-03-22 20:23:19.q DEBUG [b10-zonemgr.zonemgr] ZONEMGR_RECEIVE_SHUTDOWN received SHUTDOWN command
2012-03-22 20:23:19.q DEBUG [b10-auth.auth] AUTH_RECEIVED_COMMAND command 'shutdown' received
2012-03-22 20:23:19.q DEBUG [b10-auth.auth] AUTH_SHUTDOWN asked to stop, doing so
2012-03-22 20:23:19.q DEBUG [b10-auth.cc] CC_DISCONNECT disconnecting from message queue daemon
2012-03-22 20:23:19.q INFO  [b10-stats.stats] STATS_RECEIVED_SHUTDOWN_COMMAND shutdown command received
2012-03-22 20:23:19.q DEBUG [b10-auth.cc] CC_DISCONNECT disconnecting from message queue daemon
2012-03-22 20:23:19.q ERROR [b10-xfrout.xfrout] XFROUT_RECEIVE_FILE_DESCRIPTOR_ERROR error receiving the file descriptor for an XFR connection
...
2012-03-22 20:23:20.q ERROR [b10-xfrout.xfrout] XFROUT_RECEIVE_FILE_DESCRIPTOR_ERROR error receiving the file descriptor for an XFR connection
2012-03-22 20:23:20.q ERROR [b10-xfrout.config] CONFIG_SESSION_STOPPING_FAILED error sending stopping message: [Errno 32] Broken pipe
2012-03-22 20:23:20.q INFO  [b10-boss.boss] BIND10_PROCESS_ENDED process 39858 of b10-stats ended with status 0
2012-03-22 20:23:20.q INFO  [b10-boss.boss] BIND10_PROCESS_ENDED process 39857 of b10-zonemgr ended with status 0

Subtickets

Change History (9)

comment:1 Changed 6 years ago by jinmei

I suggest we solve this by #1867 and #1868.

comment:2 Changed 6 years ago by jelte

  • Milestone changed from New Tasks to Sprint-20120417

comment:3 follow-up: Changed 6 years ago by vorner

So, should we pull the two mentioned tickets into the current sprint?

comment:4 in reply to: ↑ 3 Changed 6 years ago by jinmei

Replying to vorner:

So, should we pull the two mentioned tickets into the current sprint?

maybe, if my proposal makes sense.

comment:5 Changed 6 years ago by jelte

  • Milestone changed from Sprint-20120417 to Year 3 Task Backlog

OK, I'm pulling this out of the current sprint, as it looks like those two task should fix it. I'm keeping the ticket alive for the purpose of checking whether they have done so (we can close it then)

comment:6 Changed 6 years ago by jreed

This needs to be a priority.

On a second production system, this was logged 69,915,570 times and counting, over 9GB filled up the disk. It logged it 31,018 times in one second.

As a workaround, I turned off xfrout.

> config remove Boss/components b10-xfrout
> config commit
Error: [Errno 32] Broken pipe
Configuration not committed
> config commit
>

But Boss show_processes still shows it, but configuration is gone, so I shut it down manually:

...
    [
        26432, 
        "b10-xfrout"
    ],
... 
> config remove Boss/components b10-xfrout
Error: b10-xfrout not found in named_set /Boss/components
> Xfrout shutdown

The logging had XFROUT_XFR_TRANSFER_FAILED for client 79.134.255.103 for TLDs ac through zw.
Ten minutes later the XFROUT_RECEIVE_FILE_DESCRIPTOR_ERROR problems kicked in.

2012-05-02 08:21:06.456 INFO  [b10-xfrout.xfrout] XFROUT_XFR_TRANSFER_FAILED AXF
R client 79.134.255.103:53238: transfer of ac/IN failed, rcode: NOTAUTH
2012-05-02 08:21:06.841 INFO  [b10-xfrout.xfrout] XFROUT_XFR_TRANSFER_FAILED AXF
R client 79.134.255.103:41272: transfer of ad/IN failed, rcode: NOTAUTH
...  
2012-05-02 08:22:40.426 INFO  [b10-xfrout.xfrout] XFROUT_XFR_TRANSFER_FAILED AXF
R client 79.134.255.103:39160: transfer of zm/IN failed, rcode: NOTAUTH
2012-05-02 08:22:40.809 INFO  [b10-xfrout.xfrout] XFROUT_XFR_TRANSFER_FAILED AXF
R client 79.134.255.103:47128: transfer of zw/IN failed, rcode: NOTAUTH
2012-05-02 08:32:41.466 FATAL [b10-auth.auth] AUTH_SERVER_FAILED server failed:
Can't assign requested address
2012-05-02 08:32:41.467 ERROR [b10-xfrout.xfrout] XFROUT_RECEIVE_FILE_DESCRIPTOR
_ERROR error receiving the file descriptor for an XFR connection
2012-05-02 08:32:41.467 ERROR [b10-xfrout.xfrout] XFROUT_RECEIVE_FILE_DESCRIPTOR
_ERROR error receiving the file descriptor for an XFR connection
...

comment:7 Changed 6 years ago by jreed

On the original production system, it happened again only on shutdown: 16298 times in about one second. (I do use Xfrout on that system.)

comment:8 Changed 6 years ago by jinmei

As discussed in #988, this should be resolved with that ticket.
Closing.

comment:9 Changed 6 years ago by jinmei

  • Resolution set to duplicate
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.