11
progmem
4y

Kubernetes is a breeze they said. Now I‘m sitting here for several hours trying to find out why my pods randomly fail to resolve domain names.

Coming along my adventure: broken systemd configs, systemd-resolved stub causing loops, broken k3s modules and finding out that busybox‘s nslookup is broken for versions greater than v1.28.4.

50 issues later, I figured out that the dude who setup the corporate network (where the machine in question is located) uses two nameservers: one to resolve the internal routes and one for all the external domains. Luckily, coredns randomly picks a nameserver for each request. Therefore, sometimes queries for external domains reach the nameserver dedicated to the internal network which then answers with NXDOMAIN.

I hate networking so so much...

Comments
  • 1
    K8S the hard way is shit to debug.

    I was the one who took that shit to the face, when my company made the move to k8s. The coredns config patch I used to upstrean to 8.8.8.8 is still in use. And its causing weird namespace resolve problem....
  • 0
    @AtuM yeah its running on Ubuntu. Default system tools seem to be k8s unfriendly. Rest of the story is a mixture of a weird network setup, wrong assumptions and consequently configuration issues on my side.
  • 0
    I've never heard someone say k8 is easy personally. I've always heard the opposite and avoid it until you can't.
  • 0
    K8s is like *nux. It is stupidly easy when you get a good grip on it.
Add Comment