On Dawn, it can be hard to figure out where some of the traffic is coming from and apply appropriate Cilium Network Policies. Specifically, traffic originating within the cluster is often given the world identity by Cilium, even though it is clearly coming from within the cluster - cross-node traffic, and often traffic from the k8s API, frequently ends up this way.
This is quite frustrating when applying cilium network policies - we have to allow the world identity for things that just shouldn't need it.
Possible that some tweaks to the K3s and/or configuration on Dawn will be helpful. Here's what i've tested out:
- turned on Cilium EBPF masquerading and routing instead of Host (didn't help)
- use
egress-selector-mode: cluster for k3s (didn't help)
- dropped Cilium MTU to 1400 from 1450. Openstack network MTU is 1450. Suspect that Cilium's own VXLAN wrapping was adding 50 bytes on top of its 1450 packet size, which was then getting dropped by Openstack and causing the packet identities to be dropped by Cilium. This has caused Cilium to start correctly identifying traffic sources (e.g. remote-node and
kube-apiserver identities are now showing up when they were not before) but only partially, so a lot of traffic still dropped.
- moved back to
egress-selector-mode: agent (the default) (nothing changed)
On Dawn, it can be hard to figure out where some of the traffic is coming from and apply appropriate Cilium Network Policies. Specifically, traffic originating within the cluster is often given the
worldidentity by Cilium, even though it is clearly coming from within the cluster - cross-node traffic, and often traffic from the k8s API, frequently ends up this way.This is quite frustrating when applying cilium network policies - we have to allow the world identity for things that just shouldn't need it.
Possible that some tweaks to the K3s and/or configuration on Dawn will be helpful. Here's what i've tested out:
egress-selector-mode: clusterfor k3s (didn't help)kube-apiserveridentities are now showing up when they were not before) but only partially, so a lot of traffic still dropped.egress-selector-mode: agent(the default) (nothing changed)