When connecting via TELNET to a running KLH10-simulated OS instance from a FreeBSD machine, I have found that the connection was being dropped after two hours. The following is what I learned about FreeBSD, and how I resolved the issue.
After an upgrade of my KLH10-hosting FreeBSD platform from FreeBSD 12-RELEASE to FreeBSD 13-RELEASE, TELNET connections began dropping after two hours of inactivity. I had not associated this with the FreeBSD upgrade, and didn't take time to diagnose the problem right away. I did determine that the disconnections did not occur when originating the TELNET connection from Linux-based machines. At that point I opted to create a relay-login account on Linux to perform the TELNET, and I would investigate later.
Today is "Later."
I've whipped out the tcpdump(8) and monitored the TELNET session.
In the output and texts below, the following hostname and operating systems are at issue:
TELNET Client host: shell.mrynet.com running FreeBSD 13.0-RELEASE
TELNET Server host: tops20.mrynet.com running TOPS-20 under the KLH10 PDP-10 simulator.
The actual OS hosting the KLH10 PDP-10 simulator is not at issue, since the raw packets are passed to the TOPS-20 OS and are not otherwise acted upon by the hosting OS.
The following tcpdump(8) output was captured on the KLH10 simulator host with:
# tcpdump -nnSX -i em2 host tops20.mrynet.com and port 23
(em2 is a dedicated ethernet interface name on the simulator host logically attached to the KLH10 simulator):
21:53:43.950967 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1102, win 65535, length 0
21:53:43.981740 IP tops20.mrynet.com.telnet > shell.mrynet.com.21707: Flags [P.], seq 1102:1131, ack 105, win 1356, length 29
21:53:43.982567 IP tops20.mrynet.com.telnet > shell.mrynet.com.21707: Flags [P.], seq 1131:1132, ack 105, win 1356, length 1
21:53:43.982697 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
23:53:44.029075 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
23:54:59.029593 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
23:56:14.029869 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
23:57:29.036743 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
23:58:44.044758 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
23:59:59.058763 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
00:01:14.071775 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
00:02:29.117760 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [.], ack 1132, win 65535, length 0
00:03:44.133015 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [R.], seq 105, ack 1132, win 0, length 0
00:03:44.134285 IP tops20.mrynet.com.telnet > shell.mrynet.com.21707: Flags [.], ack 105, win 0, length 0
00:03:44.134385 IP shell.mrynet.com.21707 > tops20.mrynet.com.telnet: Flags [R], seq 2319642573, win 0, length 0
Shown in the above log section are the following activities:
21:53:43.982697 is the timestamp of the last TELNET connection transmission, this one being an ACK from the TELNET client.
23:53:44.029075 is the timestamp of the FreeBSD TELNET client-side host sending a keep-alive probe on the TELNET connection to the TOPS-20 TELNET server emulated OS. This is odd, since the TELNET protocol itself does not open the TCP connection with keep-alive (SO_KEEPALIVE) enabled, and TCP connections themselves are not required to employ such functionality. The keep-alive packet is sent by the FreeBSD TELNET client's Host, and not the TELNET client itself. It is being used as a connection keep-alive probe utilizing an ACK-flagged packet with no data payload. The TOPS-20 OS does not respond to this packet.
23:54:59.029593 \
23:56:14.029869 |
23:57:29.036743 |
23:58:44.044758 > These are additional duplicate keep-alive packets sent by the FreeBSD TELNET client host.
23:59:59.058763 |
00:01:14.071775 |
00:02:29.117760 /
00:03:44.133015 This packet is sent by the the FreeBSD TELNET client's host to finally shut down the TCP connection carrying the TELNET communications. It is sent as a result of the FreeBSD client host not having received responses to the keep-alive packets.
00:03:44.134285 is the response from the TOPS-20 OS dutifully complying with the connection shutdown. TOPS-20 will have notified the TELNET server of the connection being closed.
00:03:44.134385 is the final packet from the the FreeBSD TELNET client host closing the TCP connection.
While I do not know why this has suddenly become an issue when upgrading my FreeBSD systems, I have tracked down the functionality within FreeBSD that controls these keep-alive packets being sent on established TCP connections.
The four relevant FreeBSD sysctl(8) settings are in the net.inet.tcp sysctl(3) grouping. They are listed below along with their default values:
net.inet.tcp.always_keepalive: 1 (boolean)
net.inet.tcp.keepidle: 7200000 (msec)
net.inet.tcp.keep.intvl: 75000 (msec)
net.inet.tcp.keepcnt: 8 (packets)
The descriptions of these settings, from the Freebsd tcp(4) man page, are:
net.inet.tcp.always_keepalive
When set, assume that SO_KEEPALIVE is set on all TCP connections, the kernel will periodically send a packet to the remote host to verify the connection is still up.
net.inet.tcp.keepidle
Amount of time, in milliseconds, that the connection must be idle before keep-alive probes (if enabled) are sent.
net.inet.tcp.keepintvl
The interval, in milliseconds, between keep-alive probes sent to remote machines, when no response is received on a keep-alive probe.
net.inet.tcp.keepcnt
Number of probes sent, with no response, before a connection is dropped.
These settings, and their default values, result in the following explanation of the packets at issue:
- Since net.inet.tcp.always_keepalive is enabled, FreeBSD will perform the keep-alive operation on all TCP connections being originated. In this case, the TELNET client host will perform the keep-alive process on the TELNET connection.
- The keep-alive probes begin at exactly 7200000 milliseconds (7200 seconds, or 120 minutes) of idle time (net.inet.tcp.keepidle) on the connection.
- The FreeBSD TELNET CLIENT host (shell.mrynet.com), and not the TELNET client itself, sends 8 (net.inet.tcp.keepcnt) keep-alive packets at 75000 millisecond (75 second) intervals (net.inet.tcp.keepintvl).
- After no response to the the 8th packet, for a total of 10 minutes of keep-alive attempts, the FreeBSD TELNET Client host closes the TCP TELNET connection.
- There was a total of 7800 seconds (130 minutes) of time since the TELNET connection was used until the connection was closed by the FreeBSD client host.
The TCP specification does not state that any keep-alive protocol should be employed. And it hasn't been for TELNET connections previously.
The resolution (workaround?) for this situation is to turn off the always_keepalive setting on the host running the TELNET client to connect to the TOPS-20 system instance.
I do not know if this is purely specific to FreeBSD, but I do know that it isn't occurring with the Ubuntu Linux kernel 5.4.0.
Regardless, the specific action to fix this is to run the following command on the TCP-connection-originating FreeBSD host:
# sysctl -w net.inet.tcp.always_keepalive=0
