缘起

一个奇特的标题 i40e-ovs-tc . 起初的现象是k8s集群终端无法 ssh登陆，3台物理机机器都这样，提示 kex_exchange_identification: read: Connection reset by peer，通过 bmc web界面查看到疯狂刷屏的内核日志告警 "i40e 0000:18:00.0: Invalid traffic class"。

修复过程

同事提醒尝试通过 alt+f1～alt+f6 切换，效果不佳，依然疯狂刷屏，登陆很困难，半盲打状态下输入了账号密码登陆了，重启了sshd，似乎有时能登陆了，先尝试一波更新i40e的驱动。

更新i40e驱动

驱动地址，最新版本为2.17.4

https://www.intel.com/content/www/us/en/download/18026/24411/intel-network-adapter-driver-for-pcie-40-gigabit-ethernet-network-connections-under-linux.html

大概装了这么一些包
yum -y install rpm-build rpmdevtools

开始 # rpmbuild -tb i40e-2.17.4.tar.gz，遇到报错

error: Failed build dependencies:

kernel-devel is needed by i40e-2.16.11-1.x86_64

yum install gcc kernel-header kernel-devel

继续运行，继续报错，

+ STATUS=0 + '[' 0 -ne 0 ']' + cd i40e-2.16.11 + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w . + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.uPhM53 + umask 022 + cd /root/rpmbuild/BUILD + cd i40e-2.16.11 + make -C src clean make: Entering directory `/root/rpmbuild/BUILD/i40e-2.16.11/src' common.mk:82: *** Kernel header files not in any of the expected locations. common.mk:83: *** Install the appropriate kernel development package, e.g. common.mk:84: *** kernel-devel, for building kernel modules and try again. Stop. make: Leaving directory `/root/rpmbuild/BUILD/i40e-2.16.11/src' error: Bad exit status from /var/tmp/rpm-tmp.uPhM53 (%build)

最终一股脑全装上

yum install pcre-devel zlib-devel openssl-devel make cmake gcc gcc-c++ bison ncurses-devel rpm-build -y

yum groupinstall "Development Tools"

大概有遇到这个报错

# Kernel Search Path # All the places we look for kernel source KSP := /lib/modules/${BUILD_KERNEL}/source \ /lib/modules/${BUILD_KERNEL}/build \ /usr/src/linux-${BUILD_KERNEL} \ /usr/src/linux-$(${BUILD_KERNEL} | sed 's/-.*//') \ /usr/src/kernel-headers-${BUILD_KERNEL} \ /usr/src/kernel-source-${BUILD_KERNEL} \ /usr/src/linux-$(${BUILD_KERNEL} | sed 's/$[0-9]*\.[0-9]*$\..*/\1/') \ /usr/src/linux \ /usr/src/kernels/${BUILD_KERNEL} \ /usr/src/kernels

解决方法，创建软连接

ln -s /usr/src/kernels/3.10.0-1160.45.1.el7.x86_64/ /usr/src/linux

再次继续编译

rpmbuild -tb i40e-2.17.4.tar.gz

编译成功，最后部分输出日志大致如下

<Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/i40e-2.17.4-1.x86_64 Wrote: /root/rpmbuild/RPMS/x86_64/i40e-2.17.4-1.x86_64.rpm Wrote: /root/rpmbuild/RPMS/x86_64/auxiliary-1.0.0-1.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.n2andN + umask 022 + cd /root/rpmbuild/BUILD + cd i40e-2.17.4 + rm -rf /root/rpmbuild/BUILDROOT/i40e-2.17.4-1.x86_64 + exit 0

更新驱动
rpm -Uvh i40e-2.17.4-1.x86_64.rpm
rpm -Uvh auxiliary-1.0.0-1.x86_64.rpm

查看 modinfo i40e
modinfo i40e
filename: /lib/modules/3.10.0-1160.el7.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
version: 2.8.20-k
似乎一直没变，后来手动make，make install一轮
更换 rmmod i40e; modprobe i40e
dracut --force

终于完成驱动更新了，版本从2.8.20-k更新为2.17.4

ovs 和 tc

可是开机仍然报错 Invalid traffic class，查了一下代码，应该和 tc 有关

drivers/net/ethernet/intel/i40e/i40e_main.c

不懂程序，不懂网络的我晕了，赶紧请教同事，竟然是一个过渡版本的bug， ovs 里面开了一个网卡 oc offload策略，但是发现有的网卡工作不好。

就是上图蓝色框里面的东东，关闭它即可。

ovs-vsctl remove Open_vSwitch . other_config hw-offload ovs-vsctl remove Open_vSwitch . other_config tc-policy systemctl restart openvswitch

差不多没有遇到 Invalid traffic class 刷屏了，还有1台会出现，需要改环境变量，这里不说了，继续有坑。

ksoftriqd 疯狂吃cpu，没空查了，最终把集群环境铲掉重建了。

i40e-ovs-tc

缘起

修复过程

更新i40e驱动

ovs 和 tc

相关文章：

文章评论