#cilium — Public Fediverse posts

6 Stunden später: die subtilen Errors im Kubernetes Log und random Pods, die wegen Timeouts wegsterben, haben eine Ursache: Duplicated Node PodCIDRs.

6 Server hatten jeweils 2 identische Subnetze. Schon lustig, wenn man eine IP pingt, den Server mit der IP runter fährt, und trotzdem ein ICMP Reply bekommt.

Ich hab zwar noch keinen Schimmer, warum Cilium aus dem IP-Pool Blöcke doppelt vergeben hat, aber die IP Allocations sehen nun alle wieder ordentlich aus.

#kubernetes #cilium

@hegerdes Is it wise to use GUAs when rolling out encryption (wireguard/IPSec)? I've experimented a few times with #Cilium and ULAs and Wireguard encryption. It works, but was sometimes unpredictable and would require further investigation.

Let's continue the Proxmox + Tofu + Talos + Cilium adventure, with two little footnotes. "Devil is in the details!"

First: Talos "inlineManifests" behavior.

When you add some inlineManifests to your Talos MachineConfig and push that MachineConfig, the manifests get applied immediately. Yay!

However, when you update or remove some inlineManifests and push the MachineConfig ... Nothing happens. Talos does a full (potentially destructive!) reconcile only when executing a cluster upgrade. (This is pretty well explained in the Talos docs[1])

This means that our initial installation of CIlium will work immediately, but subsequent configuration changes won't work (the YAML won't be applied) until we run a "talosctl upgrade-k8s". (Pro-tip: make sure to specify "--to" with the current k8s version, otherwise it'll execute a "real" upgrade which implies downloading new images and restarting the whole control plane one component at a time - which takes a while.)

So, are we there yet?

Not quite!

The second issue: each time I'd do a "tofu plan", it would tell me that something had changed. Which is kind of annoying. If you don't change your Tofu configuration, variables, etc, normally, you'd expect "tofu plan" to tell you a reassuring:

No changes. Your infrastructure matches the configuration.

So, what is going on? 🤔

[1] https://docs.siderolabs.com/kubernetes-guides/advanced-guides/inlinemanifests#how-talos-handles-manifest-resources

#terraform #talos #opentofu #homelab #kubernetes #cilium

Also, I want the K8S cluster to support IPV6, which meant replacing Talos' default CNI (Flannel) with Cilium.

(OK, it might be possible to support IPv6 with Flannel on Talos, but the Talos docs say very little about how to customize Flannel, and I wanted Cilium for other reasons too - e.g. LoadBalancer support with L2 announcements, replacing kube-proxy...)

This means declaring "cni: none" in the Talos machine config, and then either:

1) manually installing Cilium after provisioning the cluster

2) finding a way to automatically install Cilium when the cluster is provisioned.

Of course I went for option 2, right :-)

Which leads us to a rabbit hole of multiple options:

1) wait for the cluster to be up (=K8S API is functional) and then use the Helm provider to create a helm_release resource on the cluster

Problem: there is no easy and clean way to wait for the cluster to be up.

Talos has a talos_cluster_health resource, but this one waits for all nodes to be "Ready", which isn't going to happen since the CNI hasn't been deployed yet. (There is a skip_kubernetes_checks option but it doesn't seem to help.)

Declaring something like a kubernetes_nodes resource in Tofu sort of works, ... until you reprovision the cluster. Then you realize that you can't even do a "tofu plan" because Tofu tries to refresh that resources' status, which requires the cluster to be up. So, this is a non-starter.

2) use Talos "inlineManifests" feature, which instructs talos to apply a bunch of YAML to the cluster when it's provisioned

Problem: this requires Cilium YAML manifests; and the way I install it is typically with the Helm chart.

Solution: use a helm_template data source to do the equivalent of the "helm template" command, and render the Cilium chart into ready-to-apply YAML manifests.

Next problem: the Cilium Helm chart is very sophisticated, and depends on Capabilities.KubeVersion - in other words, when we invoke the helm_template resource, we need to pass it the correct kube_version.

Next solution: that version is available in talos_machine_configuration resources.

And with that (and a good amount of Cilium configuration!) our cluster comes up fully functional!

#kubernetes #talos #proxmox #cilium #opentofu

I need to move off ingress-nginx because it's mothballed. I'm already using #Cilium for CNI, so I figured I'd switch to that. Yesterday was discovering that Cilium's gateway api implementation doesn't play nice with MetalLB. I figured that as Cilium BGP control plane can perform the same function, I might as well simplify a bit and replace MetalLB, so I did that this evening. Went pretty smoothly (except for a *weird* choice they made in their API wrt selectors, but at least it's documented) so I went back to try the service I was experimentally moving to gateway API. But it turns out that it doesn't support filters yet so I can't migrate the integrated authentication 😭 Looks like it's been on the todo list for a few years without moving, so it's a tossup now whether to wait a couple of releases to see if it shows up, or use something else. #kubernetes

#homelab and #k3s made further progress, they now run #cilium for networking.

Why? Because I like Cilium and eBPF.

Compare to my full blown #k8s cluster on 10 bare metal supermicro systems, this was a tremendous amount easier. In total I spent about two hours to get k3s and Cilium to play nice.

All of this is Ansible based. I am reusing the official k3s orchestration, and a personalized Cilium role.

Dabbled with enabling #IPv6 on my #cilium based #k3s cluster this morning. Seems that it /is/ possible to enable without a full cluster/node rebuild*.

Mostly went fine, prefix, prefix mask, masq set to off. After poking a couple of the Cilium Pods new Pods got an IPv6 addr. ...but couldn't ping anything. Traffic made it out based on what Hubble was showing, but not the reverse.

Enabled v6 masquerading, and it all started to work, yay. Suspect I need to try setting up a static route on my router for this to work.

I have a couple pods w/ quirky networking so they got unhappy. v6 IP, dns query replying w/ AAAA but no dice as they really only have v4 connectivity.

Back off for now but promising that it could work.

*.spec.PodCIDR(s) are immutable on v1.Node resources, but cilium in it's default configuration doesn't get it's PodCIDR from there in the default config.

Its easy to hate on #k8s if you're a developer now working in microservices at your job. Microservices intensify the division of labor to an inconcievable degree, alienating workers from not just from the products of our labor, but especially from each other. And the Agile culture that comes with it is simply Taylorism perfected

But the truth is, #Cilium + #eBPF is simply one of the coolest things happening in software rn. A rose garden sprouting in the cracks of our neo-Haussmannian boulevards

One last oddity from my NetworkPolicy project over the last few days.....

I am getting the following in my hubble logs:

Feb 22 20:48:28.333: :: (ID:16777244) <> ff02::1:ff99:2a81 (ID:16777244) Unknown L3 target address DROPPED (ICMPv6 NeighborSolicitation)
Feb 22 20:48:29.325: fe80::b85f:80ff:fed7:6193 (ID:2435) <> ff02::16 (ID:16777244) Invalid source ip DROPPED (ICMPv6 143(0))
Feb 22 20:48:29.325: fe80::b85f:80ff:fed7:6193 (ID:2435) <> ff02::2 (ID:16777244) Invalid source ip DROPPED (ICMPv6 RouterSolicitation)
Feb 22 20:49:43.117: :: (ID:9705) <> ff02::16 (ID:16777244) Invalid source ip DROPPED (ICMPv6 143(0))
Feb 22 20:49:43.213: :: (ID:16777244) <> ff02::1:ffaf:3d08 (ID:16777244) Unknown L3 target address DROPPED (ICMPv6 NeighborSolicitation)

@rachel The Hubble-generated dashboards (from cilium/hubble) tend to work better than third-party ones. The official Grafana integration at grafana.com/grafana/dashboards/16611 is solid for flow visibility.

For DNS-specific monitoring, dashboard 16612 covers Hubble DNS metrics well.

Avoid anything built for pre-1.14 Cilium — the metric names changed significantly.

#kubernetes #cilium #grafana #monitoring

Well... I have setup my demo lab on cheap mini pc and mikrotik router without much preparation in maybe 2 hours.

now tune the scripts, maybe throw in terraform/tofu instead of semi-manual vm creation (talos is installed automatically via PXE)

#talos #kubernetes #k8s #cilium #pxe #proxmox #nerd

Finally resurrected my talos homelab cluster. Did not get much attention after my vacation last summer.
Moved the rack, no clean shutdown, different network, everything broken.. 🙈
Next: upgrades and fix headscale setup.
#talos #k8s #kubernetes #cilium #nerd #homelab

Phew. For some reason, Cilium major updates are the one update in my Homelab which always has me biting my nails. But it looks like all went well, and I'm now on 1.19.

#HomeLab #Cilium

Happy to contribute to #Cilium (#documentation).

Good tools deserve good docs. ✨

https://github.com/cilium/cilium/pull/44204

Let's see where this goes!

@homelab #talos #k8s #cilium

Posted part two of my homelab k8s cluster series:

https://unixorn.github.io/post/homelab/k8s/02-k8s-cilium-r53-and-cert-manager/

This one covers using cert-manager to create certificates for domains hosted on Route 53 and setting up a basic https service using Cilium and also automatically redirecting http to https.

I set up a #talos #k8s cluster with #cilium on #proxmox over the holiday break.

I documented how to set one up on my blog at https://unixorn.github.io/post/homelab/k8s/01-talos-with-cilium-cni-on-proxmox/

#selfhosted #homelab @homelab

This is part one of a series.

Cilium native routing means the same L2 and has to know how to route Pod IPs - okay so you make the Pod CIDR _inside_ of the LAN subnet? It's v6 so clashes are unlikely, we can gamble on that.

...but I have the nodes BGP peering with the router. Which means each node advertises (and gets a route table entry in the router) with the `/80` that is their Pod range.

That means anything _else_ on the LAN subnet that falls inside those `/80`s will get routed to the node as a gateway right?

You can't say, reserve a range for pods in the LAN subnet and downsize it from `/64` cause then SLAAC won't work right?

Very confused about how native routing is supposed to work.

#Cilium #CiliumCNI

Just published a deep dive on locking down my self-hosted #Mastodon instance on bare metal #Kubernetes. 🛡️

I used #Cilium Network Policies to implement a Zero Trust model—blocking internal LAN access while allowing federation. The post covers: ✅ Safely deploying with PolicyAuditMode ✅ Troubleshooting MinIO (toServices vs toEndpoints) ✅ Handling Redis via FQDN

Read the full write-up: https://wael.nasreddine.com/kubernetes/mastodon-locking-down-with-net.html

#kubernetes #cilium #sre #infosec

Finally done building my first single-node and single-IP bare-metal #kubernetes cluster (at OVHCloud) with #cilium and #traefik. Way harder than the "normal" cloud clusters! It's not like I had anything else to do the past two weeks... (p.s. #getfedihired plz thanks?)

Cilium continues to be amazing, and it was worth the deeper dive. But I'm really really ready to shrink my yak stack and get back to the #dataEngineering for #fedora.

"Kubernetes is overkill for a Home Lab", exhibit 493 (made up number): I'm trying to make Pi-Hole egress to the internet via the loadbalancer IP, rather than the node IP (default). To do that, I need to implement Cilium Egress Gateway, which is horribly complicated, and will be used for a one-off. But if I don't do so, I need to review all my firewall rules, which only allow very few IP addresses in my network to query external DNS servers. Ugh. I'm starting to reconsider my sanity. What am I getting myself into? :picardfacepalm:

Edit: nothing like a good night of sleep. I found a solution using traffic policies in my router, so I no longer need the complexity of the k8s egress gateway. 👍

#HomeLab #TalosLinux #Kubernetes #Cilium #PiHole @homelab

@maxheadroom nee, wegen #cilium 🐝 I guess

'A Cilium Introduction: Back to Bee-Sics - Nico Vibert & Dan Finneran, Isovalent' https://www.youtube.com/watch?v=KZzNm5ntRbo (2024) #cilium #k8s

Всё переписать: запускаем Cilium dualstack overlay поверх IPv6 underlay

На связи Борис Литвиненко из команды NOCDEV в Yandex Infrastructure — наша группа занимается автоматизацией сетей всего Яндекса. В прошлом году я уже рассказал о том, как и почему наши сетевые сервисы живут на отдельном железе с развёрнутым k8s‑кластером, избегая кольцевых зависимостей с остальной инфраструктурой компании. Среди прочего на тот момент мы использовали Calico VXLAN — с нашим разнообразным железом нам был нужен overlay, а Cilium тогда не умел работать с IPv6 для VXLAN. Однако всё меняется. В 2025 году попытка включить Calico eBPF в нашем K8s‑кластере вылилась в запуск Cilium, в новом релизе которого как раз добавили поддержку IPv6 underlay для VXLAN. Поэтому сегодня я расскажу, как мы вместе с Кириллом Глушенковым @kglushen протестировали новый Cilium 1.18 — а заодно перешли с самописных salt‑рецептов для kubeadm на kubespray, столкнулись с не такими уж простыми особенностями dualstack, а ещё немного повайбкодили. Под катом — все наши приключения с нелинейным сюжетом.

https://habr.com/ru/companies/yandex/articles/945460/

#cilium #k8s #ipv6 #overlay #dualstack #yandex #ebpf #underlay

eBPF & Security: возможности, угрозы и способы защиты

Технология eBPF не нова. Её используют повсеместно, ведь она упрощает написание кода для ядра ОС. классно и удобно, а главное безопасно! Но, как оказывается на практике, не все так гладко… Это не только удобное средство для написания кода, но и новые потенциальные векторы для атак. Поэтому давайте подробно разберём, как она работает, и как можно избежать потенциальных проблем. Для меня как безопасника интереснее всего использование eBPF сервисами и инструментами в продакшене. Именно там открываются возможные пути обхода для злоумышленников. Меня зовут Лев Хакимов, я DevOps и Kubernetes Security Lead в

https://habr.com/ru/companies/oleg-bunin/articles/941560/

#ebpf #linux_kernel #cilium #falco #трассировка #rootkit #анализ_угроз #maps #io_uring #безопасность

Stop Using the Wrong #CNI in 2025: #Flannel vs #Calico vs #Cilium

https://pdelarco.medium.com/stop-using-the-wrong-cni-flannel-vs-calico-vs-cilium-in-2025-c11b42ce05a3

#Kubernetes

Kubernetes: шестимесячный марафон по прокачке dBrain.cloud

За последние полгода наши инженеры проделали значительную работу по развитию и оптимизации платформы dBrain.cloud , ключевым инструментом которой является Kubernetes. В этой статье мы подробно расскажем об изменениях, произошедших в инфраструктуре, о внедрении новых функций и фишках, которые, мы надеемся, будут интересны DevOps-сообществу.

https://habr.com/ru/companies/dbraincloud/articles/931798/

#gpu #slicing #s3 #cilium #metallb #ipam #cgroup_v2 #kubernetes #dbrain #devops

A lesson learned for #cilium and #certmanager on #kubernetes

One shall never forget all necessary http routes and most importantly the enableGatewayAPI flag.

This one also helped: https://kubito.dev/posts/gateway-api-cert-manager/

Discover how eBPF, Cilium, and Tetragon enhance container security with real-time kernel-level insights. Learn to combine these tools with SBOMs for robust security monitoring. Boost your skills with OS-SCi education programs! #ContainerSecurity #eBPF #Cilium #Tetragon #SBOMs https://dub.sh/cZVQvk6

Недостатки Istio по сравнению с Cilium: подробное объяснение

В этой статье мы разберём основные недостатки Istio в сравнении с Cilium Service Mesh, чтобы даже начинающий разработчик мог понять, в чём разница и почему некоторые команды выбирают Cilium вместо Istio.

https://habr.com/ru/articles/903736/

#cilium #istio #kubernetes

#Question

What #CNI do you recommend for #HomeLab?

I have experience with:
- #Istio
- #Calico
- #Cilium *
- #Flannel *

*: I didn't configured them manually, they came with #k3s and I left them on defaults

I finally managed to create a new cluster with #Talos, and want to configure CNI for it. 😊

Securing #Microservices with #Cilium and #Istio 👇 #Azure #AKS #Kubernetes https://techcommunity.microsoft.com/blog/azurenetworkingblog/securing-microservices-with-cilium-and-istio/4381646

wawaweewa #cilium #ingress adventure time!

https://docs.cilium.io/en/stable/network/servicemesh/ingress/
https://docs.cilium.io/en/stable/network/lb-ipam/

look ma! no #MetalLB !

Note to self: When converting a Kubernetes cluster with Cilium as CNI to replace MetalLB with Cilium's new L2 announcements, you need to tweak some settings in your Cilium installation. Especially enabling Cilium to act as a kube-proxy replacement (if you are not already doing so) and enabling the l2 announcements. Which means kube-proxy needs to be disabled in k3s.

In other words: k3s on my Raspi4 is now providing a loadbalancer to blocky using Cilium's L2 announcements...

#kubernetes #k3s #cilium #homelab #metalLB #loadbalancer #dns #blocky #hellyeah

How do #IFT & cilium structure impact #SensoryNeuron responses? @alison_marie1 @SenguptaLab &co reveal complex contributions of IFT & #cilium structure to #Celegans chemosensory neuron responses & modulation of olfactory behavior #PLOSBiology https://plos.io/4eMYz1f

For those of you running #Talos for your #Kubernetes clusters, how has your experience with #cilium been? Is the article on the #sidero site still the best source? https://www.talos.dev/v1.8/kubernetes-guides/network/deploying-cilium/

Are there any gotchas not mentioned here?

So I've managed to finally get #Traefik working with #CertManager.

It took lots of frustration, a sidequest around attempting replace Traefik with the #Cilium Gateway API implementation, to lots of annoyance and frustration, broken iptables, but we finally got back to pretty much where we started and things started to fall in place from here.

So the good news is by separating certificates from Traefik, we can now get Traefik doing HA. Why you ask? Just cause.

#Kubernetes #Docker #homelab

Сетевые плагины (CNI) в Kubernetes

Сеть Kubernetes — это сложная и увлекательная тема, наполненная множеством подвижных частей. Одним из ключевых компонентов, обеспечивающих сетевую связность и взаимодействие различных элементов кластера, является CNI (Container Networking Interface). CNI - это спецификация, разработанная CNCF (Cloud Native Computing Foundation) для стандартизации процесса подключения сетевых интерфейсов к контейнерам. CNI обеспечивает гибкость и адаптивность сетевой инфраструктуры, позволяя интегрировать различные сетевые решения в Kubernetes. Давайте подробнее разберем, что такое Container Network Interface?

https://habr.com/ru/companies/amvera/articles/834482/

#CNI #сетевые_плагины_kubernetes #сеть_kubernetes #Концепции_CNI #Функции_CNI #vxlan #CNI_с_Cilium #cilium #Установка_Cilium #сетевые_политики_Kubernetes

#Cilium is really awesome piece of tech. Runs flawlessly as a pure CNI. But can do so much more. Ingress controller, #gatewayAPI, service mesh, ...

#kubernetes #container #linux

Has anyone used #talos #kubernetes with #cilium as #CNI and successfully gotten the IPAM part of cilium working? This basically should do what #metallb does, announce "additional" IPs via L2 or BGP and provide them as loadbalancer IPs inside the cluster.

I have the machine running, cilium is running, I defined the Cilium IPPool and my #traefik service got an external IP address from the ippool's range. I have ingressroutes ready and DNS set up for the ingressroute hostnames to point to that IP address.

But I cannot reach anything on that IP, curl tells me that it could not connect to the server...

Looks like nothing is listening on that IP.

@zas Also, #kong KIC has _great_ support for the #GatewayAPI (even the preview stuff like #grpc) as far as I can tell.

I haven't tried it, because BYOCNI made it harder to switch off #cilium, and I have a talk to do ;-)

https://docs.konghq.com/kubernetes-ingress-controller/latest/get-started/

@zas Yeah, I've been trying to set up an #AKS #k8s #cluster with #cilium using the #GatewayAPI exclusively instead of my #traefik #ingress and it's tricky, not because of the "BYOCNI" part, but because *most* helm charts still don't support Gateway -- but do support Ingress.

Sadly, the #Ingress2Gateway project got so sidetracked supporting vendor extensions that the current version can't even generate plain HTTP Routes from ingresses unless you're using @istiomesh

DIY: Ваше собственное облако на базе Kubernetes (часть 2)

Продолжаем серию постов про то как построить своё собственное облако в экосистеме Kubernetes. В прошлой статье мы разобрали как можно подготовить базовый дистрибутив Kubernetes на базе Talos Linux и Flux CD. Теперь нам предстоит обсудить возможность запуска виртуальных машин и всего что для этого необходимо, а это в первую очередь хранилище и сеть. Мы поговорим про такие технологии как KubeVirt, LINSTOR и Kube-OVN Для начала мне стоит рассказать зачем вообще нужны виртуальные машины, почему бы нам не ограничиться только-лишь контейнерами? Всё дело в том, что контейнеры в ядре Linux не дают должного уровня изоляции. Несмотря на то, что с каждым годом ситуация становится всё лучше, тем не менее довольно часто мы сталкиваемся с уязвимостями, позволяющими покинуть песочницу контейнера и повысить свои привилегии в системе.

https://habr.com/ru/companies/aenix/articles/797323/

#kubernetes #kubevirt #linstor #kubeovn #cilium

A simple problem of getting vault on one machine to talk to a mysql pod on another machine got me into researching #cilium's BGP support and making various weird setups over #tailnet.

Or, I could have just added my VM to the tailnet directly 🤔