家庭网络巴巴博弈

深入交流 117 views 0 replies
#1 ·

最近在家笔记本插网线突然无法上网了，但是连接 WLAN 却一切正常，最后发现原因非常傻逼 :a_grinning_face_with_sweat:

症状

  • DHCP 模式下，NetworkManager 无法激活网络
  • 配置静态 IP 后，无法访问网关，但是可以访问链路层直连的交换机
  • 交换机上同一接口接入 Windows PC 可以正常 DHCP 上网

背景

网络拓扑

L2 拓扑|551x500

ONU

  • 型号：TEWA-768G
  • 设置：
    服务 模式 接口
    Internet 桥接 eth0
    IPTV 桥接 eth0.43

r1

  • 型号：H3C NX30Pro
  • OS：OpenWrt 24.10.4
  • 设置：
    r1|462x470

s2

  • 型号：Redmi AX6000
  • OS: OpenWrt 24.10.4
  • 设置：二层交换所有接口

iptv-box

  • PPPoE 接入 IPTV 网络

ws1

  • Fedora 43 笔记本，出问题的设备

ws2

  • Windows 11 PC

worker1

  • 型号：ThinkPad E470
  • Rocky 9 服务器

前情提要

本来 IPTV 是 ONU 上单独接口提供的，最近和朋友联机玩 Satisfactory，用旧笔记本（worker1）搭的独立服务器（可以挂机生产 😇）

因为我家弱电箱到电视柜只预埋了一根双绞线，本来是 IPTV 独占的，所以我改成了 ONU 在 Internet 口 vlan 43 桥接 IPTV 上游，然后 r1 拆开 vlan 并桥接到 br-lan

Troubleshooting

查看 NetworkManager 日志，发现 DHCP 超时导致激活失败

NetworkManager 日志 
Feb 21 17:58:43 ws1 NetworkManager[1340]: <info> [1771667923.5363] manager: (enp0s31f6): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Feb 21 17:58:43 ws1 NetworkManager[1340]: <info> [1771667923.5364] device (enp0s31f6): state change: unmanaged -> unavailable (reason 'managed', managed-type: 'external')
Feb 21 21:30:40 ws1 NetworkManager[1340]: <info> [1771680640.8620] device (enp0s31f6): carrier: link connected
Feb 21 21:30:40 ws1 NetworkManager[1340]: <info> [1771680640.8624] device (enp0s31f6): state change: unavailable -> disconnected (reason 'carrier-changed', managed-type: '>
Feb 21 21:31:14 ws1 NetworkManager[1340]: <info> [1771680674.1726] device (enp0s31f6): Activation: starting connection 'enp0s31f6' (0a349357-55da-3dec-a942-21c73818f8cb)
Feb 21 21:31:14 ws1 NetworkManager[1340]: <info> [1771680674.1727] device (enp0s31f6): state change: disconnected -> prepare (reason 'none', managed-type: 'full')
Feb 21 21:31:14 ws1 NetworkManager[1340]: <info> [1771680674.1732] device (enp0s31f6): state change: prepare -> config (reason 'none', managed-type: 'full')
Feb 21 21:31:14 ws1 NetworkManager[1340]: <info> [1771680674.1743] device (enp0s31f6): state change: config -> ip-config (reason 'none', managed-type: 'full')
Feb 21 21:31:14 ws1 NetworkManager[1340]: <info> [1771680674.1749] dhcp4 (enp0s31f6): activation: beginning transaction (timeout in 45 seconds)
Feb 21 21:31:59 ws1 NetworkManager[1340]: <info> [1771680719.7963] device (enp0s31f6): state change: ip-config -> failed (reason 'ip-config-unavailable', managed-type: 'fu>
Feb 21 21:31:59 ws1 NetworkManager[1340]: <warn> [1771680719.7969] device (enp0s31f6): Activation: failed for connection 'enp0s31f6'
Feb 21 21:31:59 ws1 NetworkManager[1340]: <info> [1771680719.7970] device (enp0s31f6): state change: failed -> disconnected (reason 'none', managed-type: 'full')
Feb 21 21:31:59 ws1 NetworkManager[1340]: <info> [1771680719.8198] dhcp4 (enp0s31f6): canceled DHCP transaction

在 s2 上抓包，发现有 DHCP request 但是没有 response

tcpdump 
root@s2:~# tcpdump -i br-lan -nn port 67 or port 68
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:06:45.152450 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:06:47.152175 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:06:49.598975 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:06:53.781919 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:07:02.074461 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:07:18.476678 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:07:46.827972 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:07:48.828047 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:07:51.422797 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:07:55.587902 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:08:03.601729 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
14:08:19.919914 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 287
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

在 r1 上抓包，发现 r1 收到了 DHCP request 而且发出了 reply，但是显然没有转发到 s2，而且还有另一个 DHCP server 回复 🤯

tcpdump 
root@r1:~# tcpdump -i br-lan -nn -vv -e port 67 or port 68
tcpdump: listening on br-lan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:01:15.782149 4a:de:9d:1d:d4:db > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 323: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 309)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 4a:de:9d:1d:d4:db, length 281, xid 0xd03f7c74, secs 1, Flags [none] (0x0000)
	  Client-Ethernet-Address 4a:de:9d:1d:d4:db
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message (53), length 1: Discover
	    Client-ID (61), length 7: ether 4a:de:9d:1d:d4:db
	    Parameter-Request (55), length 17: 
	      Subnet-Mask (1), Time-Zone (2), Domain-Name-Server (6), Hostname (12)
	      Domain-Name (15), MTU (26), BR (28), Classless-Static-Route (121)
	      Default-Gateway (3), Static-Route (33), YD (40), YS (41)
	      NTP (42), Unknown (119), Classless-Static-Route-Microsoft (249), Unknown (252)
	      RP (17)
	    MSZ (57), length 2: 576
	    Hostname (12), length 3: "ws1"
15:01:15.782921 fa:3b:9d:5b:ab:d8 > 4a:de:9d:1d:d4:db, ethertype IPv4 (0x0800), length 342: (tos 0xc0, ttl 64, id 35446, offset 0, flags [none], proto UDP (17), length 328)
    192.168.1.1.67 > 192.168.1.241.68: [bad udp cksum 0x8588 -> 0xb78b!] BOOTP/DHCP, Reply, length 300, xid 0xd03f7c74, secs 1, Flags [none] (0x0000)
	  Your-IP 192.168.1.241
	  Server-IP 192.168.1.1
	  Client-Ethernet-Address 4a:de:9d:1d:d4:db
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message (53), length 1: Offer
	    Server-ID (54), length 4: 192.168.1.1
	    Lease-Time (51), length 4: 43200
	    RN (58), length 4: 21600
	    RB (59), length 4: 37800
	    Subnet-Mask (1), length 4: 255.255.255.0
	    BR (28), length 4: 192.168.1.255
	    Default-Gateway (3), length 4: 192.168.1.1
	    Domain-Name-Server (6), length 4: 192.168.1.1
	    Domain-Name (15), length 3: "lan"
15:01:15.849689 a8:bd:de:cd:39:56 > 4a:de:9d:1d:d4:db, ethertype IPv4 (0x0800), length 365: (tos 0xc0, ttl 1, id 7554, offset 0, flags [none], proto UDP (17), length 351)
    10.218.32.1.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 323, xid 0xd03f7c74, secs 1, Flags [none] (0x0000)
	  Your-IP 10.218.37.64
	  Gateway-IP 10.218.32.1
	  Client-Ethernet-Address 4a:de:9d:1d:d4:db
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message (53), length 1: Offer
	    Subnet-Mask (1), length 4: 255.255.224.0
	    Default-Gateway (3), length 4: 10.218.32.1
	    Domain-Name-Server (6), length 8: 218.30.19.50,61.134.1.5
	    Lease-Time (51), length 4: 2400
	    RN (58), length 4: 1200
	    RB (59), length 4: 2100
	    Server-ID (54), length 4: 10.218.32.1
	    Unknown (125), length 31: 40.53.16.83.72.67.84.67.73.80.84.86.68.72.67.80.65.65.65.40.53.9.48.48.57.48.48.57.48.48.57

推测 10.218.32.1/19 应该是 r1 eth1.43 上来自 IPTV 上游的 DHCP 泄漏，但是和 DHCP reply 无法到达 s2 没有直接关系，应该是 r1 的二层转发有问题

查看 r1 的 ARP 表

root@r1:~# arp
IP address       HW type     Flags       HW address            Mask     Device
...
192.168.1.50     0x1         0x2         4a:de:9d:1d:d4:db     *        br-lan
192.168.1.241    0x1         0x0         4a:de:9d:1d:d4:db     *        br-lan
192.168.1.49     0x1         0x2         4a:de:9d:1d:d4:db     *        br-lan
198.18.0.1       0x1         0x2         4a:de:9d:1d:d4:db     *        br-lan

有 ws1 的 MAC，但是为什么无法转发呢？ 🫠

于是我去咨询 Claude，它让我贴 r1 的 ip link show

root@r1:~# ip link show
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 4a:de:9d:1d:d4:db brd ff:ff:ff:ff:ff:ff
46: eth1.43@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 4a:de:9d:1d:d4:db brd ff:ff:ff:ff:ff:ff

草，原来 r1 的 wan 口 MAC 和 ws1 是一样的，r1 的 DHCP reply 每次都发给自己了 🤣

问题根源

安装 ONU 的时候，我用 ws1 直接接入 ONU 拨号了。ONU 拨号有 MAC 绑定，一定时间内只允许一个 MAC 拨号（防止多拨？），所以配置 r1 的时候复制了 ws1 的 MAC

root@r1:~# cat /etc/config/network 
...
config device
	option name 'eth1'
	option macaddr '4a:de:9d:1d:d4:db'

这样正常情况下不会有问题，但是由于我配置 IPTV，把 eth1.43 桥接到 br-lan，导致 eth1.43 和 ws1 在同一个广播域，Openwrt vlan 设备自动继承物理设备 MAC 地址，所以发给 ws1 的 ethernet frame 全都被 r1 drop 了

解决方法

把 r1 eth1 的 MAC 地址改掉就行了（

拨号 MAC 绑定等几分钟就自动解除了

不过刚才注意到 IPTV 上游 DHCP 泄漏到局域网了，而且这样局域网和上游二层桥接了，实在不太安全，所以修改 r1 设置如下：
r1-alt|354x500

给 iptv-box 配置静态 IP 192.168.1.254，在 r1 拨号访问 IPTV 网络，并配置策略路由，只将来自 iptv-box 的流量转发至 pppoe-iptv

同时还需要配置 igmpproxy 转发 iptv-box 的组播流量，否则无法观看直播（需要打开 br-lan 的 igmp snooping 避免组播流量浪费带宽）

❤️wave_gif🤠️👍️
9