Search
1000 results for “Gentoo_eV”
-
Today #Inea, my ISP, had a major fiber fault — apparently ranging from the Dębiec district of Poznań, all the way south to Czempiń (something like 30 km). What did I discover due to that?
Firstly, I couldn't find any way to find information on current faults anywhere on their website. Finally, I've decided to use the "chat" with their "virtual advisor" (i.e. some stupid LLM). Immediately after opening it and agreeing to the terms of service, it threw a list of current faults at me. Couldn't they have put that somewhere in the website instead?
Secondly, I had a hard time getting #tethering with my phone to work. I couldn't figure out why #DHCP wasn't working. And neither #systemd, nor #Android, provides any useful way to debug this shit, so I've finally tried Wireshark. Using it, I was able to tell what IP the phone is using, gave my PC an address in the same range, and DHCP miraculously started working.
In fact, I do believe it isn't the first time that DHCP didn't work for me until I gave some random IP address to the computer. I admit that I'm not a networking expert, nor would I even call myself a sysadmin, but WTF?!
-
Today #Inea, my ISP, had a major fiber fault — apparently ranging from the Dębiec district of Poznań, all the way south to Czempiń (something like 30 km). What did I discover due to that?
Firstly, I couldn't find any way to find information on current faults anywhere on their website. Finally, I've decided to use the "chat" with their "virtual advisor" (i.e. some stupid LLM). Immediately after opening it and agreeing to the terms of service, it threw a list of current faults at me. Couldn't they have put that somewhere in the website instead?
Secondly, I had a hard time getting #tethering with my phone to work. I couldn't figure out why #DHCP wasn't working. And neither #systemd, nor #Android, provides any useful way to debug this shit, so I've finally tried Wireshark. Using it, I was able to tell what IP the phone is using, gave my PC an address in the same range, and DHCP miraculously started working.
In fact, I do believe it isn't the first time that DHCP didn't work for me until I gave some random IP address to the computer. I admit that I'm not a networking expert, nor would I even call myself a sysadmin, but WTF?!
-
Today #Inea, my ISP, had a major fiber fault — apparently ranging from the Dębiec district of Poznań, all the way south to Czempiń (something like 30 km). What did I discover due to that?
Firstly, I couldn't find any way to find information on current faults anywhere on their website. Finally, I've decided to use the "chat" with their "virtual advisor" (i.e. some stupid LLM). Immediately after opening it and agreeing to the terms of service, it threw a list of current faults at me. Couldn't they have put that somewhere in the website instead?
Secondly, I had a hard time getting #tethering with my phone to work. I couldn't figure out why #DHCP wasn't working. And neither #systemd, nor #Android, provides any useful way to debug this shit, so I've finally tried Wireshark. Using it, I was able to tell what IP the phone is using, gave my PC an address in the same range, and DHCP miraculously started working.
In fact, I do believe it isn't the first time that DHCP didn't work for me until I gave some random IP address to the computer. I admit that I'm not a networking expert, nor would I even call myself a sysadmin, but WTF?!
-
Today #Inea, my ISP, had a major fiber fault — apparently ranging from the Dębiec district of Poznań, all the way south to Czempiń (something like 30 km). What did I discover due to that?
Firstly, I couldn't find any way to find information on current faults anywhere on their website. Finally, I've decided to use the "chat" with their "virtual advisor" (i.e. some stupid LLM). Immediately after opening it and agreeing to the terms of service, it threw a list of current faults at me. Couldn't they have put that somewhere in the website instead?
Secondly, I had a hard time getting #tethering with my phone to work. I couldn't figure out why #DHCP wasn't working. And neither #systemd, nor #Android, provides any useful way to debug this shit, so I've finally tried Wireshark. Using it, I was able to tell what IP the phone is using, gave my PC an address in the same range, and DHCP miraculously started working.
In fact, I do believe it isn't the first time that DHCP didn't work for me until I gave some random IP address to the computer. I admit that I'm not a networking expert, nor would I even call myself a sysadmin, but WTF?!
-
The FreeBSD-native-ish home lab and network
For many years my setup was pretty simple: A FreeBSD home server running on my old laptop. It runs everything I need to be present on the internet, an email server, a web server (like the one you’ve accessed right now to see this blog post) and a public chat server (XMPP/Jabber) so I can be in touch with friends.
For my home network, I had a basic Access Point and a basic Router.
Lately, my setup has become more… intense. I have IPv6 thanks to Hurricane Electric, the network is passed to my home network (which we’ll talk about in a bit), a home network with multiple VLANs, since friends who come home also need WiFi.
I decided to blog about the details, hoping it would help someone in the future.
I’ll start with the simplest one.
The Home Server
I’ve been running home servers for a long time. I believe that every person/family needs a home server. Forget about buying your kids iPads and Smartphones. Their first devices should be a real computer (sorry Apple, iOS devices are still just a toy) like a desktop/laptop and a home server. The home server doesn’t need to be on the public internet, but mine is, for variety of reasons. This blog being one of them.
I get a static IP address from my ISP, Ucom. After the management change that happened couple of years ago, Ucom has become a very typical ISP (think shitty), but they are the only ones that provide a static IP address, instead of setting it on your router, where you have to do port forwarding.
My home server, hostnamed pingvinashen (meaning the town of the penguins, named after the Armenian cartoon) run FreeBSD. Historically this machine has run Debian, Funtoo, Gentoo and finally FreeBSD.
Hardware wise, here’s what it is:
root@pingvinashen:~ # dmidecode -s system-product-nameLatitude E5470root@pingvinashen:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHzroot@pingvinashen:~ # sysctl hw.physmemhw.physmem: 17016950784root@pingvinashen:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 420G 178G 242G - - 64% 42% 1.00x ONLINE -
While most homelabbers use hardware virtualization, I think that resources are a tight thing, and should be managed properly. Any company that markets itself as “green/eco-friendly” and uses hardware virtualization should do calculations using a pen and paper and prove if going native would save power/resources or not. (sometimes it doesn’t, usually it does)
I use containers, the old-school ones, Jails to be more specific.
I manage jails using Jailer, my own tool, that tries to stay out of your way when working with Jails.
Here are my current jails:
root@pingvinashen:~ # jailer listNAME STATE JID HOSTNAME IPv4 GWantranig Active 1 antranig.bsd.am 192.168.10.42/24 192.168.10.1antranigv Active 2 antranigv.bsd.am 192.168.10.52/24 192.168.10.1git Stoppedhuginn0 Active 4 huginn0.bsd.am 192.168.10.34/24 192.168.10.1ifconfig Active 5 ifconfig.bsd.am 192.168.10.33/24 192.168.10.1lucy Active 6 lucy.vartanian.am 192.168.10.37/24 192.168.10.1mysql Active 7 mysql.antranigv.am 192.168.10.50/24 192.168.10.1newsletter Active 8 newsletter.bsd.am 192.168.10.65/24 192.168.10.1oragir Active 9 oragir.am 192.168.10.30/24 192.168.10.1psql Active 10 psql.pingvinashen.am 192.168.10.3/24 192.168.10.1rss Active 11 rss.bsd.am 192.168.10.5/24 192.168.10.1sarian Active 12 sarian.am 192.168.10.53/24 192.168.10.1syuneci Active 13 syuneci.am 192.168.10.60/24 192.168.10.1znc Active 14 znc.bsd.am 192.168.10.152/24 192.168.10.1
You already get a basic idea of how things are. Each of my blogs (Armenian and English) has its own Jail. Since I’m using WordPress, I need a database, so I have a MySQL jail (which ironically runs MariaDB) inside of it.
I also have a Git server, running gitea, which is down at the moment as I’m doing maintanence. The Git server (and many other services) requires PostgreSQL, hence the existence of a PostgreSQL jail. I run huginn for automation (RSS to Telegram, RSS to XMPP). My sister has her own blog, using WordPress, so that’s a Jail of its own. Same goes about my fiancée.
Other Jails are Newsletter using Listmonk, Sarian (the Armenian instance of lobste.rs) and a personal ZNC server.
As an avid RSS advocate, I also have a RSS Jail, which runs Miniflux. Many of my friends use this service.
Oragir is an instance of WriteFreely, as I advocate public blogging and ActivityPub. Our community uses that too.
The web server that forwards all this traffic from the public to the Jails is nginx. All it does is
proxy_passas needed. It runs on the host.Other services that run on the host are DNS (BIND9), an email service running
OpenSMTPd(which will be moved to a Jail soon), the chat service runningprosody(which will be moved to a Jail soon) and finally, WireGuard, because I love VPNs.Finally, there’s a IPv6-over-IPv4 tunnel that I use to obtain IPv6 thanks to Hurricane Electric.
Yes, I have a firewall, I use
pf(4).For the techies in the room, here’s what my
rc.conflooks like.# cat /etc/rc.conf# Defaultsclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"#local_unbound_enable="YES"sshd_enable="YES"moused_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="pingvinashen.am"# Networkingdefaultrouter="37.157.221.1"gateway_enable="YES"ifconfig_em0="up"vlans_em0="37 1000" # 1000 -> WAN; 37 -> Home Routerifconfig_em0_1000="inet 37.157.221.130 netmask 255.255.255.0"ifconfig_em0_37="inet 192.168.255.2 netmask 255.255.255.0"static_routes="home"route_home="-net 172.16.100.0/24 -gateway 192.168.255.1"cloned_interfaces="bridge0 bridge6 bridge10"ifconfig_bridge10="inet 192.168.10.1 netmask 255.255.255.0"## IPv6ipv6_gateway_enable="YES"gif_interfaces="gif0"gifconfig_gif0="37.157.221.130 216.66.84.46"ifconfig_gif0="inet6 2001:470:1f14:ef::2 2001:470:1f14:ef::1 prefixlen 128"ipv6_defaultrouter="2001:470:1f14:ef::1"ifconfig_em0_37_ipv6="inet6 2001:470:7914:7065::2 prefixlen 64"ipv6_static_routes="home guest"ipv6_route_home="-net 2001:470:7914:6a76::/64 -gateway 2001:470:7914:7065::1"ipv6_route_guest="-net 2001:470:7914:6969::/64 -gateway 2001:470:7914:7065::1"ifconfig_bridge6_ipv6="inet6 2001:470:1f15:e4::1 prefixlen 64"ifconfig_bridge6_aliases="inet6 2001:470:1f15:e4::25 prefixlen 64 \inet6 2001:470:1f15:e4::80 prefixlen 64 \inet6 2001:470:1f15:e4::5222 prefixlen 64 \inet6 2001:470:1f15:e4:c0fe::53 prefixlen 64 \"# VPNwireguard_enable="YES"wireguard_interfaces="wg0"# Firewallpf_enable="YES"# Jailsjail_enable="YES"jailer_dir="zfs:zroot/jails"# DNSnamed_enable="YES"# Mailsmtpd_enable="YES"smtpd_config="/usr/local/etc/smtpd.conf"# XMPPprosody_enable="YES"turnserver_enable="YES"# Webnginx_enable="YES"tor_enable="YES"
The
gif0interface is a IPv6-over-IPv4 tunnel. I have static routes to my home network, so I don’t go to my server over the ISP every time. This also gives me the ability to get IPv6 in my home network that is routed via my home server.As you have guessed from this config file, I do have VLANs setup. So let’s get into that.
The Home Network
First of all, here’s a very cheap diagram
I have the following VLANs setup on the switch.
VLAN IDPurpose1Switch Management1000pingvinashen (home server) WAN1001evn0 (home router) WAN37pingvinashen ↔ evn042Internal Management100Home LAN69Home GuestHere are the active ports
PortVLANsPurpose24untagged: 1Switch management, connects to Port 222untagged: 1000pingvinashen WAN, from ISP21untagged: 1001Home WAN, from ISP20tagged: 1000, 37To pingvinashen, portem019untagged: 1001To home router, portigb118tagged: 42, 100, 69, 99To home router, portigb217untagged: 37To home router, portigb016tagged: 42, 100, 69To Lenovo T480s15untagged: 100To Raspberri Pi 42untagged: 99From Port 24, for switch management1untagged: 42; tagged: 100, 69; PoETo UAP AC ProThe home router, hostnamed
evn0(named after the IATA code of Yerevan’s Zvartnots International Airport) runs FreeBSD as well, the hardware is the followingroot@evn0:~ # dmidecode -s system-product-nameAPU2root@evn0:~ # sysctl hw.modelhw.model: AMD GX-412TC SOC root@evn0:~ # sysctl hw.physmemhw.physmem: 4234399744root@evn0:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 12.5G 9.47G 3.03G - - 67% 75% 1.00x ONLINE -
The home router does… well, routing. It also does DHCP, DNS, SLAAC, and can act as a syslog server.
Here’s what the
rc.conflooks likeclear_tmp_enable="YES"sendmail_enable="NONE"syslogd_flags="-a '172.16.100.0/24:*' -H"zfs_enable="YES"dumpdev="AUTO"hostname="evn0.illuriasecurity.com"pf_enable="YES"gateway_enable="YES"ipv6_gateway_enable="YES"sshd_enable="YES"# Get an IP address from the ISP's GPONifconfig_igb1="DHCP"# Internal routes with pingvinashenifconfig_igb0="inet 192.168.255.1 netmask 255.255.255.0"ifconfig_igb0_ipv6="inet6 2001:470:7914:7065::1 prefixlen 64"static_routes="pingvinashen"route_pingvinashen="-net 37.157.221.130/32 -gateway 192.168.255.2"ipv6_defaultrouter="2001:470:7914:7065::2"# Home Mgmt, Switch Mgmt, Home LAN, Home Guestifconfig_igb2="up"vlans_igb2="42 99 100 69"ifconfig_igb2_42="inet 172.31.42.1 netmask 255.255.255.0"ifconfig_igb2_99="inet 172.16.99.1 netmask 255.255.255.0"ifconfig_igb2_100="inet 172.16.100.1 netmask 255.255.255.0"ifconfig_igb2_100_ipv6="inet6 2001:470:7914:6a76::1 prefixlen 64"ifconfig_igb2_69="inet 192.168.69.1 netmask 255.255.255.0"ifconfig_igb2_69_ipv6="inet6 2001:470:7914:6969::1 prefixlen 64"# DNS and DHCPnamed_enable="YES"dhcpd_enable="YES"named_flags=""# NTPntpd_enable="YES"# Router Advertisement and LLDPrtadvd_enable="YES"lldpd_enable="YES"lldpd_flags=""
Here’s
pf.conf, because security is important.ext_if="igb1"bsd_if="igb0"int_if="igb2.100"guest_if="igb2.69"mgmt_if="igb2.42"sw_if="igb2.99"ill_net="172.16.0.0/16"nat pass on $ext_if from $int_if:network to any -> ($ext_if)nat pass on $ext_if from $mgmt_if:network to any -> ($ext_if)nat pass on $ext_if from $guest_if:network to any -> ($ext_if)set skip on { lo0 }block in allpass on $int_if from $int_if:network to anypass on $mgmt_if from $mgmt_if:network to anypass on $sw_if from $sw_if:network to anypass on $guest_if from $guest_if:network to anyblock quick on $guest_if from any to { $int_if:network, $mgmt_if:network, $ill_net, $sw_if:network }pass in on illuria0 from $ill_net to { $ill_net, $mgmt_if:network }pass inet proto icmppass inet6 proto icmp6pass out all keep stateI’m sure there are places to improve, but it gets the job done and keeps the guest network isolated.
Here’s
rtadvd.conf, for my IPv6 folksigb2.100:\ :addr="2001:470:7914:6a76::":prefixlen#64:\ :rdnss="2001:470:7914:6a76::1":\ :dnssl="evn0.loc.illuriasecurity.com,loc.illuriasecurity.com":igb2.69:\ :addr="2001:470:7914:6969::":prefixlen#64:\ :rdnss="2001:470:7914:6969::1":
For DNS, I’m running BIND, here’s the important parts
listen-on { 127.0.0.1; 172.16.100.1; 172.16.99.1; 172.31.42.1; 192.168.69.1; };listen-on-v6 { 2001:470:7914:6a76::1; 2001:470:7914:6969::1; };allow-query { 127.0.0.1; 172.16.100.0/24; 172.31.42.0/24; 192.168.69.0/24; 2001:470:7914:6a76::/64; 2001:470:7914:6969::/64;};And for DHCP, here’s what it looks like
subnet 172.16.100.0 netmask 255.255.255.0 { range 172.16.100.100 172.16.100.150; option domain-name-servers 172.16.100.1; option subnet-mask 255.255.255.0; option routers 172.16.100.1; option domain-name "evn0.loc.illuriasecurity.com"; option domain-search "loc.illuriasecurity.com evn0.loc.illuriasecurity.com";}host zvartnots { hardware ethernet d4:57:63:f1:5a:36; fixed-address 172.16.100.7;}host unifi0 { hardware ethernet 58:9c:fc:93:d1:0b; fixed-address 172.31.42.42;}
[…]subnet 172.31.42.0 netmask 255.255.255.0 { range 172.31.42.100 172.31.42.150; option domain-name-servers 172.31.42.1; option subnet-mask 255.255.255.0; option routers 172.31.42.1;}subnet 192.168.69.0 netmask 255.255.255.0 { range 192.168.69.100 192.168.69.150; option domain-name-servers 192.168.69.1; option subnet-mask 255.255.255.0; option routers 192.168.69.1;}So you’re wondering, what’s this
unifi0? Well, that brings us toT480s
This laptop has been gifted to me by [REDACTED] for my contributions to the Armenian government (which means when a server goes down and no one knows how to fix it, they called me and I showed up)
Here’s the hardware
root@t480s:~ # dmidecode -s system-versionThinkPad T480sroot@t480s:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHzroot@t480s:~ # sysctl hw.physmemhw.physmem: 25602347008root@t480s:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 224G 109G 115G - - 44% 48% 1.00x ONLINE -
The T480s has access to VLAN 100, 42, 69, but the host itself has access only to VLAN 100 (LAN), while the jails can exist on other VLANs.
So I have a Jail named
unifi0that runs the Unifi Management thingie.Here’s what
rc.confof the host looks likeclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"sshd_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="t480s.evn0.loc.illuriasecurity.com"ifconfig_em0="up -rxcsum -txcsum"vlans_em0="100 42 69"ifconfig_em0_100="up"ifconfig_em0_42="up"ifconfig_em0_69="up"cloned_interfaces="bridge0 bridge100 bridge42 bridge69"create_args_bridge100="ether 8c:16:45:82:b4:10"ifconfig_bridge100="addm em0.100 SYNCDHCP"ifconfig_bridge100_ipv6="inet6 auto_linklocal"rtsold_flags="-i -F -m bridge100"rtsold_enable="YES"create_args_bridge42=" ether 8c:16:45:82:b4:42"create_args_bridge69=" ether 8c:16:45:82:b4:69"ifconfig_bridge42="addm em0.42"ifconfig_bridge69="addm em0.69"jail_enable="YES"jailer_dir="zfs:zroot/jailer"ifconfig_bridge0="inet 10.1.0.1/24 up"ngbuddy_enable="YES"ngbuddy_private_if="nghost0"dhcpd_enable="YES"lldpd_enable="YES"
I used Jailer to create the
unifi0jail, here’s what thejail.conflooks like# vim: set syntax=sh:exec.clean;allow.raw_sockets;mount.devfs;unifi0 { $id = "6"; devfs_ruleset = 10; $bridge = "bridge42"; $domain = "evn0.loc.illuriasecurity.com"; vnet; vnet.interface = "epair${id}b"; exec.prestart = "ifconfig epair${id} create up"; exec.prestart += "ifconfig epair${id}a up descr vnet-${name}"; exec.prestart += "ifconfig ${bridge} addm epair${id}a up"; exec.start = "/sbin/ifconfig lo0 127.0.0.1 up"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown jail"; exec.poststop = "ifconfig ${bridge} deletem epair${id}a"; exec.poststop += "ifconfig epair${id}a destroy"; host.hostname = "${name}.${domain}"; path = "/usr/local/jailer/unifi0"; exec.consolelog = "/var/log/jail/${name}.log"; persist; mount.fdescfs; mount.procfs;}Here are the important parts inside the jail
root@t480s:~ # cat /usr/local/jailer/unifi0/etc/rc.confifconfig_epair6b="SYNCDHCP"sendmail_enable="NONE"syslogd_flags="-ss"mongod_enable="YES"unifi_enable="YES"root@t480s:~ # cat /usr/local/jailer/unifi0/etc/start_if.epair6b ifconfig epair6b ether 58:9c:fc:93:d1:0b
Don’t you love it that you can see what’s inside the jail from the host? God I love FreeBSD!
Did I miss anything? I hope not.
Oh, for the homelabbers out there, the T480s is the one that runs things like Jellyfin if needed.
Finally, the tiny
Raspberry Pi 4, Model B
I found this in a closed, so I decided to run it for TimeMachine.
I guess all you care about is
rc.confhostname="tm0.evn0.loc.illuriasecurity.com"ifconfig_DEFAULT="DHCP inet6 accept_rtadv"sshd_enable="YES"sendmail_enable="NONE"sendmail_submit_enable="NO"sendmail_outbound_enable="NO"sendmail_msp_queue_enable="NO"growfs_enable="YES"powerd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"rtsold_enable="YES"samba_server_enable="YES"
And the Samba Configuration
[global]# Network settingsworkgroup = WORKGROUPserver string = Samba Server %vnetbios name = RPi4# Logginglog file = /var/log/samba4/log.%mmax log size = 50log level = 0# Authenticationsecurity = userencrypt passwords = yespassdb backend = tdbsammap to guest = Bad Usermin protocol = SMB2max protocol = SMB3# Apple Time Machine settingsvfs objects = catia fruit streams_xattrfruit:metadata = streamfruit:resource = streamfruit:encoding = nativefruit:locking = nonefruit:time machine = yes# File System supportea support = yeskernel oplocks = nokernel share modes = noposix locking = nomangled names = nosmbd max xattr size = 2097152# Performance tuningread raw = yeswrite raw = yesgetwd cache = yesstrict locking = no# Miscellaneouslocal master = nopreferred master = nodomain master = nowins support = no[tm]comment = Time Machine RPi4path = /usr/local/timemachine/%Ubrowseable = yesread only = novalid users = antranigvvfs objects = catia fruit streams_xattrfruit:time machine = yesfruit:advertise_fullsync = truefruit:time machine max size = 800G # Adjust the size according to your needscreate mask = 0600directory mask = 0700
That’s pretty much it.
Conclusion
I love running homebrew servers, home networks and home labs. I love that (almost) everything is FreeBSD. The switch itself runs Linux, and the Unifi Access Point also runs Linux, both of which I’m pretty happy with.
While most homelabbers used ESXi in the past, I’m happy to see that most people are moving to open source solutions like Proxmox and Xen, but I think that FreeBSD Jails and bhyve is much better. I still don’t have a need for bhyve at the moment, but I would use it if I needed hardware virtualization.
Most homelabbers would consider the lack of Web/GUI interfaces as a con, but I think that it’s a pro. If I need to “replicate” this network, all I need to do is to copy some text files and modify some IP addresses / Interface names.
I hope this was informative and that it would be useful for anyone in the future.
That’s all folks…
Reply via email.
https://antranigv.am/posts/2024/06/freebsd-server-network-homelab/
#Containers #Dell #DellLatitudeE5470 #FreeBSD #homeServer #HowTo #Jailer #Jails #macOS #Networking #pf #Samba #Unifi #Unix #VNET
-
The FreeBSD-native-ish home lab and network
For many years my setup was pretty simple: A FreeBSD home server running on my old laptop. It runs everything I need to be present on the internet, an email server, a web server (like the one you’ve accessed right now to see this blog post) and a public chat server (XMPP/Jabber) so I can be in touch with friends.
For my home network, I had a basic Access Point and a basic Router.
Lately, my setup has become more… intense. I have IPv6 thanks to Hurricane Electric, the network is passed to my home network (which we’ll talk about in a bit), a home network with multiple VLANs, since friends who come home also need WiFi.
I decided to blog about the details, hoping it would help someone in the future.
I’ll start with the simplest one.
The Home Server
I’ve been running home servers for a long time. I believe that every person/family needs a home server. Forget about buying your kids iPads and Smartphones. Their first devices should be a real computer (sorry Apple, iOS devices are still just a toy) like a desktop/laptop and a home server. The home server doesn’t need to be on the public internet, but mine is, for variety of reasons. This blog being one of them.
I get a static IP address from my ISP, Ucom. After the management change that happened couple of years ago, Ucom has become a very typical ISP (think shitty), but they are the only ones that provide a static IP address, instead of setting it on your router, where you have to do port forwarding.
My home server, hostnamed pingvinashen (meaning the town of the penguins, named after the Armenian cartoon) run FreeBSD. Historically this machine has run Debian, Funtoo, Gentoo and finally FreeBSD.
Hardware wise, here’s what it is:
root@pingvinashen:~ # dmidecode -s system-product-nameLatitude E5470root@pingvinashen:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHzroot@pingvinashen:~ # sysctl hw.physmemhw.physmem: 17016950784root@pingvinashen:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 420G 178G 242G - - 64% 42% 1.00x ONLINE -
While most homelabbers use hardware virtualization, I think that resources are a tight thing, and should be managed properly. Any company that markets itself as “green/eco-friendly” and uses hardware virtualization should do calculations using a pen and paper and prove if going native would save power/resources or not. (sometimes it doesn’t, usually it does)
I use containers, the old-school ones, Jails to be more specific.
I manage jails using Jailer, my own tool, that tries to stay out of your way when working with Jails.
Here are my current jails:
root@pingvinashen:~ # jailer listNAME STATE JID HOSTNAME IPv4 GWantranig Active 1 antranig.bsd.am 192.168.10.42/24 192.168.10.1antranigv Active 2 antranigv.bsd.am 192.168.10.52/24 192.168.10.1git Stoppedhuginn0 Active 4 huginn0.bsd.am 192.168.10.34/24 192.168.10.1ifconfig Active 5 ifconfig.bsd.am 192.168.10.33/24 192.168.10.1lucy Active 6 lucy.vartanian.am 192.168.10.37/24 192.168.10.1mysql Active 7 mysql.antranigv.am 192.168.10.50/24 192.168.10.1newsletter Active 8 newsletter.bsd.am 192.168.10.65/24 192.168.10.1oragir Active 9 oragir.am 192.168.10.30/24 192.168.10.1psql Active 10 psql.pingvinashen.am 192.168.10.3/24 192.168.10.1rss Active 11 rss.bsd.am 192.168.10.5/24 192.168.10.1sarian Active 12 sarian.am 192.168.10.53/24 192.168.10.1syuneci Active 13 syuneci.am 192.168.10.60/24 192.168.10.1znc Active 14 znc.bsd.am 192.168.10.152/24 192.168.10.1
You already get a basic idea of how things are. Each of my blogs (Armenian and English) has its own Jail. Since I’m using WordPress, I need a database, so I have a MySQL jail (which ironically runs MariaDB) inside of it.
I also have a Git server, running gitea, which is down at the moment as I’m doing maintanence. The Git server (and many other services) requires PostgreSQL, hence the existence of a PostgreSQL jail. I run huginn for automation (RSS to Telegram, RSS to XMPP). My sister has her own blog, using WordPress, so that’s a Jail of its own. Same goes about my fiancée.
Other Jails are Newsletter using Listmonk, Sarian (the Armenian instance of lobste.rs) and a personal ZNC server.
As an avid RSS advocate, I also have a RSS Jail, which runs Miniflux. Many of my friends use this service.
Oragir is an instance of WriteFreely, as I advocate public blogging and ActivityPub. Our community uses that too.
The web server that forwards all this traffic from the public to the Jails is nginx. All it does is
proxy_passas needed. It runs on the host.Other services that run on the host are DNS (BIND9), an email service running
OpenSMTPd(which will be moved to a Jail soon), the chat service runningprosody(which will be moved to a Jail soon) and finally, WireGuard, because I love VPNs.Finally, there’s a IPv6-over-IPv4 tunnel that I use to obtain IPv6 thanks to Hurricane Electric.
Yes, I have a firewall, I use
pf(4).For the techies in the room, here’s what my
rc.conflooks like.# cat /etc/rc.conf# Defaultsclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"#local_unbound_enable="YES"sshd_enable="YES"moused_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="pingvinashen.am"# Networkingdefaultrouter="37.157.221.1"gateway_enable="YES"ifconfig_em0="up"vlans_em0="37 1000" # 1000 -> WAN; 37 -> Home Routerifconfig_em0_1000="inet 37.157.221.130 netmask 255.255.255.0"ifconfig_em0_37="inet 192.168.255.2 netmask 255.255.255.0"static_routes="home"route_home="-net 172.16.100.0/24 -gateway 192.168.255.1"cloned_interfaces="bridge0 bridge6 bridge10"ifconfig_bridge10="inet 192.168.10.1 netmask 255.255.255.0"## IPv6ipv6_gateway_enable="YES"gif_interfaces="gif0"gifconfig_gif0="37.157.221.130 216.66.84.46"ifconfig_gif0="inet6 2001:470:1f14:ef::2 2001:470:1f14:ef::1 prefixlen 128"ipv6_defaultrouter="2001:470:1f14:ef::1"ifconfig_em0_37_ipv6="inet6 2001:470:7914:7065::2 prefixlen 64"ipv6_static_routes="home guest"ipv6_route_home="-net 2001:470:7914:6a76::/64 -gateway 2001:470:7914:7065::1"ipv6_route_guest="-net 2001:470:7914:6969::/64 -gateway 2001:470:7914:7065::1"ifconfig_bridge6_ipv6="inet6 2001:470:1f15:e4::1 prefixlen 64"ifconfig_bridge6_aliases="inet6 2001:470:1f15:e4::25 prefixlen 64 \inet6 2001:470:1f15:e4::80 prefixlen 64 \inet6 2001:470:1f15:e4::5222 prefixlen 64 \inet6 2001:470:1f15:e4:c0fe::53 prefixlen 64 \"# VPNwireguard_enable="YES"wireguard_interfaces="wg0"# Firewallpf_enable="YES"# Jailsjail_enable="YES"jailer_dir="zfs:zroot/jails"# DNSnamed_enable="YES"# Mailsmtpd_enable="YES"smtpd_config="/usr/local/etc/smtpd.conf"# XMPPprosody_enable="YES"turnserver_enable="YES"# Webnginx_enable="YES"tor_enable="YES"
The
gif0interface is a IPv6-over-IPv4 tunnel. I have static routes to my home network, so I don’t go to my server over the ISP every time. This also gives me the ability to get IPv6 in my home network that is routed via my home server.As you have guessed from this config file, I do have VLANs setup. So let’s get into that.
The Home Network
First of all, here’s a very cheap diagram
I have the following VLANs setup on the switch.
VLAN IDPurpose1Switch Management1000pingvinashen (home server) WAN1001evn0 (home router) WAN37pingvinashen ↔ evn042Internal Management100Home LAN69Home GuestHere are the active ports
PortVLANsPurpose24untagged: 1Switch management, connects to Port 222untagged: 1000pingvinashen WAN, from ISP21untagged: 1001Home WAN, from ISP20tagged: 1000, 37To pingvinashen, portem019untagged: 1001To home router, portigb118tagged: 42, 100, 69, 99To home router, portigb217untagged: 37To home router, portigb016tagged: 42, 100, 69To Lenovo T480s15untagged: 100To Raspberri Pi 42untagged: 99From Port 24, for switch management1untagged: 42; tagged: 100, 69; PoETo UAP AC ProThe home router, hostnamed
evn0(named after the IATA code of Yerevan’s Zvartnots International Airport) runs FreeBSD as well, the hardware is the followingroot@evn0:~ # dmidecode -s system-product-nameAPU2root@evn0:~ # sysctl hw.modelhw.model: AMD GX-412TC SOC root@evn0:~ # sysctl hw.physmemhw.physmem: 4234399744root@evn0:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 12.5G 9.47G 3.03G - - 67% 75% 1.00x ONLINE -
The home router does… well, routing. It also does DHCP, DNS, SLAAC, and can act as a syslog server.
Here’s what the
rc.conflooks likeclear_tmp_enable="YES"sendmail_enable="NONE"syslogd_flags="-a '172.16.100.0/24:*' -H"zfs_enable="YES"dumpdev="AUTO"hostname="evn0.illuriasecurity.com"pf_enable="YES"gateway_enable="YES"ipv6_gateway_enable="YES"sshd_enable="YES"# Get an IP address from the ISP's GPONifconfig_igb1="DHCP"# Internal routes with pingvinashenifconfig_igb0="inet 192.168.255.1 netmask 255.255.255.0"ifconfig_igb0_ipv6="inet6 2001:470:7914:7065::1 prefixlen 64"static_routes="pingvinashen"route_pingvinashen="-net 37.157.221.130/32 -gateway 192.168.255.2"ipv6_defaultrouter="2001:470:7914:7065::2"# Home Mgmt, Switch Mgmt, Home LAN, Home Guestifconfig_igb2="up"vlans_igb2="42 99 100 69"ifconfig_igb2_42="inet 172.31.42.1 netmask 255.255.255.0"ifconfig_igb2_99="inet 172.16.99.1 netmask 255.255.255.0"ifconfig_igb2_100="inet 172.16.100.1 netmask 255.255.255.0"ifconfig_igb2_100_ipv6="inet6 2001:470:7914:6a76::1 prefixlen 64"ifconfig_igb2_69="inet 192.168.69.1 netmask 255.255.255.0"ifconfig_igb2_69_ipv6="inet6 2001:470:7914:6969::1 prefixlen 64"# DNS and DHCPnamed_enable="YES"dhcpd_enable="YES"named_flags=""# NTPntpd_enable="YES"# Router Advertisement and LLDPrtadvd_enable="YES"lldpd_enable="YES"lldpd_flags=""
Here’s
pf.conf, because security is important.ext_if="igb1"bsd_if="igb0"int_if="igb2.100"guest_if="igb2.69"mgmt_if="igb2.42"sw_if="igb2.99"ill_net="172.16.0.0/16"nat pass on $ext_if from $int_if:network to any -> ($ext_if)nat pass on $ext_if from $mgmt_if:network to any -> ($ext_if)nat pass on $ext_if from $guest_if:network to any -> ($ext_if)set skip on { lo0 }block in allpass on $int_if from $int_if:network to anypass on $mgmt_if from $mgmt_if:network to anypass on $sw_if from $sw_if:network to anypass on $guest_if from $guest_if:network to anyblock quick on $guest_if from any to { $int_if:network, $mgmt_if:network, $ill_net, $sw_if:network }pass in on illuria0 from $ill_net to { $ill_net, $mgmt_if:network }pass inet proto icmppass inet6 proto icmp6pass out all keep stateI’m sure there are places to improve, but it gets the job done and keeps the guest network isolated.
Here’s
rtadvd.conf, for my IPv6 folksigb2.100:\ :addr="2001:470:7914:6a76::":prefixlen#64:\ :rdnss="2001:470:7914:6a76::1":\ :dnssl="evn0.loc.illuriasecurity.com,loc.illuriasecurity.com":igb2.69:\ :addr="2001:470:7914:6969::":prefixlen#64:\ :rdnss="2001:470:7914:6969::1":
For DNS, I’m running BIND, here’s the important parts
listen-on { 127.0.0.1; 172.16.100.1; 172.16.99.1; 172.31.42.1; 192.168.69.1; };listen-on-v6 { 2001:470:7914:6a76::1; 2001:470:7914:6969::1; };allow-query { 127.0.0.1; 172.16.100.0/24; 172.31.42.0/24; 192.168.69.0/24; 2001:470:7914:6a76::/64; 2001:470:7914:6969::/64;};And for DHCP, here’s what it looks like
subnet 172.16.100.0 netmask 255.255.255.0 { range 172.16.100.100 172.16.100.150; option domain-name-servers 172.16.100.1; option subnet-mask 255.255.255.0; option routers 172.16.100.1; option domain-name "evn0.loc.illuriasecurity.com"; option domain-search "loc.illuriasecurity.com evn0.loc.illuriasecurity.com";}host zvartnots { hardware ethernet d4:57:63:f1:5a:36; fixed-address 172.16.100.7;}host unifi0 { hardware ethernet 58:9c:fc:93:d1:0b; fixed-address 172.31.42.42;}
[…]subnet 172.31.42.0 netmask 255.255.255.0 { range 172.31.42.100 172.31.42.150; option domain-name-servers 172.31.42.1; option subnet-mask 255.255.255.0; option routers 172.31.42.1;}subnet 192.168.69.0 netmask 255.255.255.0 { range 192.168.69.100 192.168.69.150; option domain-name-servers 192.168.69.1; option subnet-mask 255.255.255.0; option routers 192.168.69.1;}So you’re wondering, what’s this
unifi0? Well, that brings us toT480s
This laptop has been gifted to me by [REDACTED] for my contributions to the Armenian government (which means when a server goes down and no one knows how to fix it, they called me and I showed up)
Here’s the hardware
root@t480s:~ # dmidecode -s system-versionThinkPad T480sroot@t480s:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHzroot@t480s:~ # sysctl hw.physmemhw.physmem: 25602347008root@t480s:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 224G 109G 115G - - 44% 48% 1.00x ONLINE -
The T480s has access to VLAN 100, 42, 69, but the host itself has access only to VLAN 100 (LAN), while the jails can exist on other VLANs.
So I have a Jail named
unifi0that runs the Unifi Management thingie.Here’s what
rc.confof the host looks likeclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"sshd_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="t480s.evn0.loc.illuriasecurity.com"ifconfig_em0="up -rxcsum -txcsum"vlans_em0="100 42 69"ifconfig_em0_100="up"ifconfig_em0_42="up"ifconfig_em0_69="up"cloned_interfaces="bridge0 bridge100 bridge42 bridge69"create_args_bridge100="ether 8c:16:45:82:b4:10"ifconfig_bridge100="addm em0.100 SYNCDHCP"ifconfig_bridge100_ipv6="inet6 auto_linklocal"rtsold_flags="-i -F -m bridge100"rtsold_enable="YES"create_args_bridge42=" ether 8c:16:45:82:b4:42"create_args_bridge69=" ether 8c:16:45:82:b4:69"ifconfig_bridge42="addm em0.42"ifconfig_bridge69="addm em0.69"jail_enable="YES"jailer_dir="zfs:zroot/jailer"ifconfig_bridge0="inet 10.1.0.1/24 up"ngbuddy_enable="YES"ngbuddy_private_if="nghost0"dhcpd_enable="YES"lldpd_enable="YES"
I used Jailer to create the
unifi0jail, here’s what thejail.conflooks like# vim: set syntax=sh:exec.clean;allow.raw_sockets;mount.devfs;unifi0 { $id = "6"; devfs_ruleset = 10; $bridge = "bridge42"; $domain = "evn0.loc.illuriasecurity.com"; vnet; vnet.interface = "epair${id}b"; exec.prestart = "ifconfig epair${id} create up"; exec.prestart += "ifconfig epair${id}a up descr vnet-${name}"; exec.prestart += "ifconfig ${bridge} addm epair${id}a up"; exec.start = "/sbin/ifconfig lo0 127.0.0.1 up"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown jail"; exec.poststop = "ifconfig ${bridge} deletem epair${id}a"; exec.poststop += "ifconfig epair${id}a destroy"; host.hostname = "${name}.${domain}"; path = "/usr/local/jailer/unifi0"; exec.consolelog = "/var/log/jail/${name}.log"; persist; mount.fdescfs; mount.procfs;}Here are the important parts inside the jail
root@t480s:~ # cat /usr/local/jailer/unifi0/etc/rc.confifconfig_epair6b="SYNCDHCP"sendmail_enable="NONE"syslogd_flags="-ss"mongod_enable="YES"unifi_enable="YES"root@t480s:~ # cat /usr/local/jailer/unifi0/etc/start_if.epair6b ifconfig epair6b ether 58:9c:fc:93:d1:0b
Don’t you love it that you can see what’s inside the jail from the host? God I love FreeBSD!
Did I miss anything? I hope not.
Oh, for the homelabbers out there, the T480s is the one that runs things like Jellyfin if needed.
Finally, the tiny
Raspberry Pi 4, Model B
I found this in a closed, so I decided to run it for TimeMachine.
I guess all you care about is
rc.confhostname="tm0.evn0.loc.illuriasecurity.com"ifconfig_DEFAULT="DHCP inet6 accept_rtadv"sshd_enable="YES"sendmail_enable="NONE"sendmail_submit_enable="NO"sendmail_outbound_enable="NO"sendmail_msp_queue_enable="NO"growfs_enable="YES"powerd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"rtsold_enable="YES"samba_server_enable="YES"
And the Samba Configuration
[global]# Network settingsworkgroup = WORKGROUPserver string = Samba Server %vnetbios name = RPi4# Logginglog file = /var/log/samba4/log.%mmax log size = 50log level = 0# Authenticationsecurity = userencrypt passwords = yespassdb backend = tdbsammap to guest = Bad Usermin protocol = SMB2max protocol = SMB3# Apple Time Machine settingsvfs objects = catia fruit streams_xattrfruit:metadata = streamfruit:resource = streamfruit:encoding = nativefruit:locking = nonefruit:time machine = yes# File System supportea support = yeskernel oplocks = nokernel share modes = noposix locking = nomangled names = nosmbd max xattr size = 2097152# Performance tuningread raw = yeswrite raw = yesgetwd cache = yesstrict locking = no# Miscellaneouslocal master = nopreferred master = nodomain master = nowins support = no[tm]comment = Time Machine RPi4path = /usr/local/timemachine/%Ubrowseable = yesread only = novalid users = antranigvvfs objects = catia fruit streams_xattrfruit:time machine = yesfruit:advertise_fullsync = truefruit:time machine max size = 800G # Adjust the size according to your needscreate mask = 0600directory mask = 0700
That’s pretty much it.
Conclusion
I love running homebrew servers, home networks and home labs. I love that (almost) everything is FreeBSD. The switch itself runs Linux, and the Unifi Access Point also runs Linux, both of which I’m pretty happy with.
While most homelabbers used ESXi in the past, I’m happy to see that most people are moving to open source solutions like Proxmox and Xen, but I think that FreeBSD Jails and bhyve is much better. I still don’t have a need for bhyve at the moment, but I would use it if I needed hardware virtualization.
Most homelabbers would consider the lack of Web/GUI interfaces as a con, but I think that it’s a pro. If I need to “replicate” this network, all I need to do is to copy some text files and modify some IP addresses / Interface names.
I hope this was informative and that it would be useful for anyone in the future.
That’s all folks…
Reply via email.
https://antranigv.am/posts/2024/06/freebsd-server-network-homelab/
#Containers #Dell #DellLatitudeE5470 #FreeBSD #homeServer #HowTo #Jailer #Jails #macOS #Networking #pf #Samba #Unifi #Unix #VNET
-
The FreeBSD-native-ish home lab and network
For many years my setup was pretty simple: A FreeBSD home server running on my old laptop. It runs everything I need to be present on the internet, an email server, a web server (like the one you’ve accessed right now to see this blog post) and a public chat server (XMPP/Jabber) so I can be in touch with friends.
For my home network, I had a basic Access Point and a basic Router.
Lately, my setup has become more… intense. I have IPv6 thanks to Hurricane Electric, the network is passed to my home network (which we’ll talk about in a bit), a home network with multiple VLANs, since friends who come home also need WiFi.
I decided to blog about the details, hoping it would help someone in the future.
I’ll start with the simplest one.
The Home Server
I’ve been running home servers for a long time. I believe that every person/family needs a home server. Forget about buying your kids iPads and Smartphones. Their first devices should be a real computer (sorry Apple, iOS devices are still just a toy) like a desktop/laptop and a home server. The home server doesn’t need to be on the public internet, but mine is, for variety of reasons. This blog being one of them.
I get a static IP address from my ISP, Ucom. After the management change that happened couple of years ago, Ucom has become a very typical ISP (think shitty), but they are the only ones that provide a static IP address, instead of setting it on your router, where you have to do port forwarding.
My home server, hostnamed pingvinashen (meaning the town of the penguins, named after the Armenian cartoon) run FreeBSD. Historically this machine has run Debian, Funtoo, Gentoo and finally FreeBSD.
Hardware wise, here’s what it is:
root@pingvinashen:~ # dmidecode -s system-product-nameLatitude E5470root@pingvinashen:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHzroot@pingvinashen:~ # sysctl hw.physmemhw.physmem: 17016950784root@pingvinashen:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 420G 178G 242G - - 64% 42% 1.00x ONLINE -
While most homelabbers use hardware virtualization, I think that resources are a tight thing, and should be managed properly. Any company that markets itself as “green/eco-friendly” and uses hardware virtualization should do calculations using a pen and paper and prove if going native would save power/resources or not. (sometimes it doesn’t, usually it does)
I use containers, the old-school ones, Jails to be more specific.
I manage jails using Jailer, my own tool, that tries to stay out of your way when working with Jails.
Here are my current jails:
root@pingvinashen:~ # jailer listNAME STATE JID HOSTNAME IPv4 GWantranig Active 1 antranig.bsd.am 192.168.10.42/24 192.168.10.1antranigv Active 2 antranigv.bsd.am 192.168.10.52/24 192.168.10.1git Stoppedhuginn0 Active 4 huginn0.bsd.am 192.168.10.34/24 192.168.10.1ifconfig Active 5 ifconfig.bsd.am 192.168.10.33/24 192.168.10.1lucy Active 6 lucy.vartanian.am 192.168.10.37/24 192.168.10.1mysql Active 7 mysql.antranigv.am 192.168.10.50/24 192.168.10.1newsletter Active 8 newsletter.bsd.am 192.168.10.65/24 192.168.10.1oragir Active 9 oragir.am 192.168.10.30/24 192.168.10.1psql Active 10 psql.pingvinashen.am 192.168.10.3/24 192.168.10.1rss Active 11 rss.bsd.am 192.168.10.5/24 192.168.10.1sarian Active 12 sarian.am 192.168.10.53/24 192.168.10.1syuneci Active 13 syuneci.am 192.168.10.60/24 192.168.10.1znc Active 14 znc.bsd.am 192.168.10.152/24 192.168.10.1
You already get a basic idea of how things are. Each of my blogs (Armenian and English) has its own Jail. Since I’m using WordPress, I need a database, so I have a MySQL jail (which ironically runs MariaDB) inside of it.
I also have a Git server, running gitea, which is down at the moment as I’m doing maintanence. The Git server (and many other services) requires PostgreSQL, hence the existence of a PostgreSQL jail. I run huginn for automation (RSS to Telegram, RSS to XMPP). My sister has her own blog, using WordPress, so that’s a Jail of its own. Same goes about my fiancée.
Other Jails are Newsletter using Listmonk, Sarian (the Armenian instance of lobste.rs) and a personal ZNC server.
As an avid RSS advocate, I also have a RSS Jail, which runs Miniflux. Many of my friends use this service.
Oragir is an instance of WriteFreely, as I advocate public blogging and ActivityPub. Our community uses that too.
The web server that forwards all this traffic from the public to the Jails is nginx. All it does is
proxy_passas needed. It runs on the host.Other services that run on the host are DNS (BIND9), an email service running
OpenSMTPd(which will be moved to a Jail soon), the chat service runningprosody(which will be moved to a Jail soon) and finally, WireGuard, because I love VPNs.Finally, there’s a IPv6-over-IPv4 tunnel that I use to obtain IPv6 thanks to Hurricane Electric.
Yes, I have a firewall, I use
pf(4).For the techies in the room, here’s what my
rc.conflooks like.# cat /etc/rc.conf# Defaultsclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"#local_unbound_enable="YES"sshd_enable="YES"moused_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="pingvinashen.am"# Networkingdefaultrouter="37.157.221.1"gateway_enable="YES"ifconfig_em0="up"vlans_em0="37 1000" # 1000 -> WAN; 37 -> Home Routerifconfig_em0_1000="inet 37.157.221.130 netmask 255.255.255.0"ifconfig_em0_37="inet 192.168.255.2 netmask 255.255.255.0"static_routes="home"route_home="-net 172.16.100.0/24 -gateway 192.168.255.1"cloned_interfaces="bridge0 bridge6 bridge10"ifconfig_bridge10="inet 192.168.10.1 netmask 255.255.255.0"## IPv6ipv6_gateway_enable="YES"gif_interfaces="gif0"gifconfig_gif0="37.157.221.130 216.66.84.46"ifconfig_gif0="inet6 2001:470:1f14:ef::2 2001:470:1f14:ef::1 prefixlen 128"ipv6_defaultrouter="2001:470:1f14:ef::1"ifconfig_em0_37_ipv6="inet6 2001:470:7914:7065::2 prefixlen 64"ipv6_static_routes="home guest"ipv6_route_home="-net 2001:470:7914:6a76::/64 -gateway 2001:470:7914:7065::1"ipv6_route_guest="-net 2001:470:7914:6969::/64 -gateway 2001:470:7914:7065::1"ifconfig_bridge6_ipv6="inet6 2001:470:1f15:e4::1 prefixlen 64"ifconfig_bridge6_aliases="inet6 2001:470:1f15:e4::25 prefixlen 64 \inet6 2001:470:1f15:e4::80 prefixlen 64 \inet6 2001:470:1f15:e4::5222 prefixlen 64 \inet6 2001:470:1f15:e4:c0fe::53 prefixlen 64 \"# VPNwireguard_enable="YES"wireguard_interfaces="wg0"# Firewallpf_enable="YES"# Jailsjail_enable="YES"jailer_dir="zfs:zroot/jails"# DNSnamed_enable="YES"# Mailsmtpd_enable="YES"smtpd_config="/usr/local/etc/smtpd.conf"# XMPPprosody_enable="YES"turnserver_enable="YES"# Webnginx_enable="YES"tor_enable="YES"
The
gif0interface is a IPv6-over-IPv4 tunnel. I have static routes to my home network, so I don’t go to my server over the ISP every time. This also gives me the ability to get IPv6 in my home network that is routed via my home server.As you have guessed from this config file, I do have VLANs setup. So let’s get into that.
The Home Network
First of all, here’s a very cheap diagram
I have the following VLANs setup on the switch.
VLAN IDPurpose1Switch Management1000pingvinashen (home server) WAN1001evn0 (home router) WAN37pingvinashen ↔ evn042Internal Management100Home LAN69Home GuestHere are the active ports
PortVLANsPurpose24untagged: 1Switch management, connects to Port 222untagged: 1000pingvinashen WAN, from ISP21untagged: 1001Home WAN, from ISP20tagged: 1000, 37To pingvinashen, portem019untagged: 1001To home router, portigb118tagged: 42, 100, 69, 99To home router, portigb217untagged: 37To home router, portigb016tagged: 42, 100, 69To Lenovo T480s15untagged: 100To Raspberri Pi 42untagged: 99From Port 24, for switch management1untagged: 42; tagged: 100, 69; PoETo UAP AC ProThe home router, hostnamed
evn0(named after the IATA code of Yerevan’s Zvartnots International Airport) runs FreeBSD as well, the hardware is the followingroot@evn0:~ # dmidecode -s system-product-nameAPU2root@evn0:~ # sysctl hw.modelhw.model: AMD GX-412TC SOC root@evn0:~ # sysctl hw.physmemhw.physmem: 4234399744root@evn0:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 12.5G 9.47G 3.03G - - 67% 75% 1.00x ONLINE -
The home router does… well, routing. It also does DHCP, DNS, SLAAC, and can act as a syslog server.
Here’s what the
rc.conflooks likeclear_tmp_enable="YES"sendmail_enable="NONE"syslogd_flags="-a '172.16.100.0/24:*' -H"zfs_enable="YES"dumpdev="AUTO"hostname="evn0.illuriasecurity.com"pf_enable="YES"gateway_enable="YES"ipv6_gateway_enable="YES"sshd_enable="YES"# Get an IP address from the ISP's GPONifconfig_igb1="DHCP"# Internal routes with pingvinashenifconfig_igb0="inet 192.168.255.1 netmask 255.255.255.0"ifconfig_igb0_ipv6="inet6 2001:470:7914:7065::1 prefixlen 64"static_routes="pingvinashen"route_pingvinashen="-net 37.157.221.130/32 -gateway 192.168.255.2"ipv6_defaultrouter="2001:470:7914:7065::2"# Home Mgmt, Switch Mgmt, Home LAN, Home Guestifconfig_igb2="up"vlans_igb2="42 99 100 69"ifconfig_igb2_42="inet 172.31.42.1 netmask 255.255.255.0"ifconfig_igb2_99="inet 172.16.99.1 netmask 255.255.255.0"ifconfig_igb2_100="inet 172.16.100.1 netmask 255.255.255.0"ifconfig_igb2_100_ipv6="inet6 2001:470:7914:6a76::1 prefixlen 64"ifconfig_igb2_69="inet 192.168.69.1 netmask 255.255.255.0"ifconfig_igb2_69_ipv6="inet6 2001:470:7914:6969::1 prefixlen 64"# DNS and DHCPnamed_enable="YES"dhcpd_enable="YES"named_flags=""# NTPntpd_enable="YES"# Router Advertisement and LLDPrtadvd_enable="YES"lldpd_enable="YES"lldpd_flags=""
Here’s
pf.conf, because security is important.ext_if="igb1"bsd_if="igb0"int_if="igb2.100"guest_if="igb2.69"mgmt_if="igb2.42"sw_if="igb2.99"ill_net="172.16.0.0/16"nat pass on $ext_if from $int_if:network to any -> ($ext_if)nat pass on $ext_if from $mgmt_if:network to any -> ($ext_if)nat pass on $ext_if from $guest_if:network to any -> ($ext_if)set skip on { lo0 }block in allpass on $int_if from $int_if:network to anypass on $mgmt_if from $mgmt_if:network to anypass on $sw_if from $sw_if:network to anypass on $guest_if from $guest_if:network to anyblock quick on $guest_if from any to { $int_if:network, $mgmt_if:network, $ill_net, $sw_if:network }pass in on illuria0 from $ill_net to { $ill_net, $mgmt_if:network }pass inet proto icmppass inet6 proto icmp6pass out all keep stateI’m sure there are places to improve, but it gets the job done and keeps the guest network isolated.
Here’s
rtadvd.conf, for my IPv6 folksigb2.100:\ :addr="2001:470:7914:6a76::":prefixlen#64:\ :rdnss="2001:470:7914:6a76::1":\ :dnssl="evn0.loc.illuriasecurity.com,loc.illuriasecurity.com":igb2.69:\ :addr="2001:470:7914:6969::":prefixlen#64:\ :rdnss="2001:470:7914:6969::1":
For DNS, I’m running BIND, here’s the important parts
listen-on { 127.0.0.1; 172.16.100.1; 172.16.99.1; 172.31.42.1; 192.168.69.1; };listen-on-v6 { 2001:470:7914:6a76::1; 2001:470:7914:6969::1; };allow-query { 127.0.0.1; 172.16.100.0/24; 172.31.42.0/24; 192.168.69.0/24; 2001:470:7914:6a76::/64; 2001:470:7914:6969::/64;};And for DHCP, here’s what it looks like
subnet 172.16.100.0 netmask 255.255.255.0 { range 172.16.100.100 172.16.100.150; option domain-name-servers 172.16.100.1; option subnet-mask 255.255.255.0; option routers 172.16.100.1; option domain-name "evn0.loc.illuriasecurity.com"; option domain-search "loc.illuriasecurity.com evn0.loc.illuriasecurity.com";}host zvartnots { hardware ethernet d4:57:63:f1:5a:36; fixed-address 172.16.100.7;}host unifi0 { hardware ethernet 58:9c:fc:93:d1:0b; fixed-address 172.31.42.42;}
[…]subnet 172.31.42.0 netmask 255.255.255.0 { range 172.31.42.100 172.31.42.150; option domain-name-servers 172.31.42.1; option subnet-mask 255.255.255.0; option routers 172.31.42.1;}subnet 192.168.69.0 netmask 255.255.255.0 { range 192.168.69.100 192.168.69.150; option domain-name-servers 192.168.69.1; option subnet-mask 255.255.255.0; option routers 192.168.69.1;}So you’re wondering, what’s this
unifi0? Well, that brings us toT480s
This laptop has been gifted to me by [REDACTED] for my contributions to the Armenian government (which means when a server goes down and no one knows how to fix it, they called me and I showed up)
Here’s the hardware
root@t480s:~ # dmidecode -s system-versionThinkPad T480sroot@t480s:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHzroot@t480s:~ # sysctl hw.physmemhw.physmem: 25602347008root@t480s:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 224G 109G 115G - - 44% 48% 1.00x ONLINE -
The T480s has access to VLAN 100, 42, 69, but the host itself has access only to VLAN 100 (LAN), while the jails can exist on other VLANs.
So I have a Jail named
unifi0that runs the Unifi Management thingie.Here’s what
rc.confof the host looks likeclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"sshd_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="t480s.evn0.loc.illuriasecurity.com"ifconfig_em0="up -rxcsum -txcsum"vlans_em0="100 42 69"ifconfig_em0_100="up"ifconfig_em0_42="up"ifconfig_em0_69="up"cloned_interfaces="bridge0 bridge100 bridge42 bridge69"create_args_bridge100="ether 8c:16:45:82:b4:10"ifconfig_bridge100="addm em0.100 SYNCDHCP"ifconfig_bridge100_ipv6="inet6 auto_linklocal"rtsold_flags="-i -F -m bridge100"rtsold_enable="YES"create_args_bridge42=" ether 8c:16:45:82:b4:42"create_args_bridge69=" ether 8c:16:45:82:b4:69"ifconfig_bridge42="addm em0.42"ifconfig_bridge69="addm em0.69"jail_enable="YES"jailer_dir="zfs:zroot/jailer"ifconfig_bridge0="inet 10.1.0.1/24 up"ngbuddy_enable="YES"ngbuddy_private_if="nghost0"dhcpd_enable="YES"lldpd_enable="YES"
I used Jailer to create the
unifi0jail, here’s what thejail.conflooks like# vim: set syntax=sh:exec.clean;allow.raw_sockets;mount.devfs;unifi0 { $id = "6"; devfs_ruleset = 10; $bridge = "bridge42"; $domain = "evn0.loc.illuriasecurity.com"; vnet; vnet.interface = "epair${id}b"; exec.prestart = "ifconfig epair${id} create up"; exec.prestart += "ifconfig epair${id}a up descr vnet-${name}"; exec.prestart += "ifconfig ${bridge} addm epair${id}a up"; exec.start = "/sbin/ifconfig lo0 127.0.0.1 up"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown jail"; exec.poststop = "ifconfig ${bridge} deletem epair${id}a"; exec.poststop += "ifconfig epair${id}a destroy"; host.hostname = "${name}.${domain}"; path = "/usr/local/jailer/unifi0"; exec.consolelog = "/var/log/jail/${name}.log"; persist; mount.fdescfs; mount.procfs;}Here are the important parts inside the jail
root@t480s:~ # cat /usr/local/jailer/unifi0/etc/rc.confifconfig_epair6b="SYNCDHCP"sendmail_enable="NONE"syslogd_flags="-ss"mongod_enable="YES"unifi_enable="YES"root@t480s:~ # cat /usr/local/jailer/unifi0/etc/start_if.epair6b ifconfig epair6b ether 58:9c:fc:93:d1:0b
Don’t you love it that you can see what’s inside the jail from the host? God I love FreeBSD!
Did I miss anything? I hope not.
Oh, for the homelabbers out there, the T480s is the one that runs things like Jellyfin if needed.
Finally, the tiny
Raspberry Pi 4, Model B
I found this in a closed, so I decided to run it for TimeMachine.
I guess all you care about is
rc.confhostname="tm0.evn0.loc.illuriasecurity.com"ifconfig_DEFAULT="DHCP inet6 accept_rtadv"sshd_enable="YES"sendmail_enable="NONE"sendmail_submit_enable="NO"sendmail_outbound_enable="NO"sendmail_msp_queue_enable="NO"growfs_enable="YES"powerd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"rtsold_enable="YES"samba_server_enable="YES"
And the Samba Configuration
[global]# Network settingsworkgroup = WORKGROUPserver string = Samba Server %vnetbios name = RPi4# Logginglog file = /var/log/samba4/log.%mmax log size = 50log level = 0# Authenticationsecurity = userencrypt passwords = yespassdb backend = tdbsammap to guest = Bad Usermin protocol = SMB2max protocol = SMB3# Apple Time Machine settingsvfs objects = catia fruit streams_xattrfruit:metadata = streamfruit:resource = streamfruit:encoding = nativefruit:locking = nonefruit:time machine = yes# File System supportea support = yeskernel oplocks = nokernel share modes = noposix locking = nomangled names = nosmbd max xattr size = 2097152# Performance tuningread raw = yeswrite raw = yesgetwd cache = yesstrict locking = no# Miscellaneouslocal master = nopreferred master = nodomain master = nowins support = no[tm]comment = Time Machine RPi4path = /usr/local/timemachine/%Ubrowseable = yesread only = novalid users = antranigvvfs objects = catia fruit streams_xattrfruit:time machine = yesfruit:advertise_fullsync = truefruit:time machine max size = 800G # Adjust the size according to your needscreate mask = 0600directory mask = 0700
That’s pretty much it.
Conclusion
I love running homebrew servers, home networks and home labs. I love that (almost) everything is FreeBSD. The switch itself runs Linux, and the Unifi Access Point also runs Linux, both of which I’m pretty happy with.
While most homelabbers used ESXi in the past, I’m happy to see that most people are moving to open source solutions like Proxmox and Xen, but I think that FreeBSD Jails and bhyve is much better. I still don’t have a need for bhyve at the moment, but I would use it if I needed hardware virtualization.
Most homelabbers would consider the lack of Web/GUI interfaces as a con, but I think that it’s a pro. If I need to “replicate” this network, all I need to do is to copy some text files and modify some IP addresses / Interface names.
I hope this was informative and that it would be useful for anyone in the future.
That’s all folks…
Reply via email.
https://antranigv.am/posts/2024/06/freebsd-server-network-homelab/
#Containers #Dell #DellLatitudeE5470 #FreeBSD #homeServer #HowTo #Jailer #Jails #macOS #Networking #pf #Samba #Unifi #Unix #VNET
-
The FreeBSD-native-ish home lab and network
For many years my setup was pretty simple: A FreeBSD home server running on my old laptop. It runs everything I need to be present on the internet, an email server, a web server (like the one you’ve accessed right now to see this blog post) and a public chat server (XMPP/Jabber) so I can be in touch with friends.
For my home network, I had a basic Access Point and a basic Router.
Lately, my setup has become more… intense. I have IPv6 thanks to Hurricane Electric, the network is passed to my home network (which we’ll talk about in a bit), a home network with multiple VLANs, since friends who come home also need WiFi.
I decided to blog about the details, hoping it would help someone in the future.
I’ll start with the simplest one.
The Home Server
I’ve been running home servers for a long time. I believe that every person/family needs a home server. Forget about buying your kids iPads and Smartphones. Their first devices should be a real computer (sorry Apple, iOS devices are still just a toy) like a desktop/laptop and a home server. The home server doesn’t need to be on the public internet, but mine is, for variety of reasons. This blog being one of them.
I get a static IP address from my ISP, Ucom. After the management change that happened couple of years ago, Ucom has become a very typical ISP (think shitty), but they are the only ones that provide a static IP address, instead of setting it on your router, where you have to do port forwarding.
My home server, hostnamed pingvinashen (meaning the town of the penguins, named after the Armenian cartoon) run FreeBSD. Historically this machine has run Debian, Funtoo, Gentoo and finally FreeBSD.
Hardware wise, here’s what it is:
root@pingvinashen:~ # dmidecode -s system-product-nameLatitude E5470root@pingvinashen:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHzroot@pingvinashen:~ # sysctl hw.physmemhw.physmem: 17016950784root@pingvinashen:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 420G 178G 242G - - 64% 42% 1.00x ONLINE -
While most homelabbers use hardware virtualization, I think that resources are a tight thing, and should be managed properly. Any company that markets itself as “green/eco-friendly” and uses hardware virtualization should do calculations using a pen and paper and prove if going native would save power/resources or not. (sometimes it doesn’t, usually it does)
I use containers, the old-school ones, Jails to be more specific.
I manage jails using Jailer, my own tool, that tries to stay out of your way when working with Jails.
Here are my current jails:
root@pingvinashen:~ # jailer listNAME STATE JID HOSTNAME IPv4 GWantranig Active 1 antranig.bsd.am 192.168.10.42/24 192.168.10.1antranigv Active 2 antranigv.bsd.am 192.168.10.52/24 192.168.10.1git Stoppedhuginn0 Active 4 huginn0.bsd.am 192.168.10.34/24 192.168.10.1ifconfig Active 5 ifconfig.bsd.am 192.168.10.33/24 192.168.10.1lucy Active 6 lucy.vartanian.am 192.168.10.37/24 192.168.10.1mysql Active 7 mysql.antranigv.am 192.168.10.50/24 192.168.10.1newsletter Active 8 newsletter.bsd.am 192.168.10.65/24 192.168.10.1oragir Active 9 oragir.am 192.168.10.30/24 192.168.10.1psql Active 10 psql.pingvinashen.am 192.168.10.3/24 192.168.10.1rss Active 11 rss.bsd.am 192.168.10.5/24 192.168.10.1sarian Active 12 sarian.am 192.168.10.53/24 192.168.10.1syuneci Active 13 syuneci.am 192.168.10.60/24 192.168.10.1znc Active 14 znc.bsd.am 192.168.10.152/24 192.168.10.1
You already get a basic idea of how things are. Each of my blogs (Armenian and English) has its own Jail. Since I’m using WordPress, I need a database, so I have a MySQL jail (which ironically runs MariaDB) inside of it.
I also have a Git server, running gitea, which is down at the moment as I’m doing maintanence. The Git server (and many other services) requires PostgreSQL, hence the existence of a PostgreSQL jail. I run huginn for automation (RSS to Telegram, RSS to XMPP). My sister has her own blog, using WordPress, so that’s a Jail of its own. Same goes about my fiancée.
Other Jails are Newsletter using Listmonk, Sarian (the Armenian instance of lobste.rs) and a personal ZNC server.
As an avid RSS advocate, I also have a RSS Jail, which runs Miniflux. Many of my friends use this service.
Oragir is an instance of WriteFreely, as I advocate public blogging and ActivityPub. Our community uses that too.
The web server that forwards all this traffic from the public to the Jails is nginx. All it does is
proxy_passas needed. It runs on the host.Other services that run on the host are DNS (BIND9), an email service running
OpenSMTPd(which will be moved to a Jail soon), the chat service runningprosody(which will be moved to a Jail soon) and finally, WireGuard, because I love VPNs.Finally, there’s a IPv6-over-IPv4 tunnel that I use to obtain IPv6 thanks to Hurricane Electric.
Yes, I have a firewall, I use
pf(4).For the techies in the room, here’s what my
rc.conflooks like.# cat /etc/rc.conf# Defaultsclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"#local_unbound_enable="YES"sshd_enable="YES"moused_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="pingvinashen.am"# Networkingdefaultrouter="37.157.221.1"gateway_enable="YES"ifconfig_em0="up"vlans_em0="37 1000" # 1000 -> WAN; 37 -> Home Routerifconfig_em0_1000="inet 37.157.221.130 netmask 255.255.255.0"ifconfig_em0_37="inet 192.168.255.2 netmask 255.255.255.0"static_routes="home"route_home="-net 172.16.100.0/24 -gateway 192.168.255.1"cloned_interfaces="bridge0 bridge6 bridge10"ifconfig_bridge10="inet 192.168.10.1 netmask 255.255.255.0"## IPv6ipv6_gateway_enable="YES"gif_interfaces="gif0"gifconfig_gif0="37.157.221.130 216.66.84.46"ifconfig_gif0="inet6 2001:470:1f14:ef::2 2001:470:1f14:ef::1 prefixlen 128"ipv6_defaultrouter="2001:470:1f14:ef::1"ifconfig_em0_37_ipv6="inet6 2001:470:7914:7065::2 prefixlen 64"ipv6_static_routes="home guest"ipv6_route_home="-net 2001:470:7914:6a76::/64 -gateway 2001:470:7914:7065::1"ipv6_route_guest="-net 2001:470:7914:6969::/64 -gateway 2001:470:7914:7065::1"ifconfig_bridge6_ipv6="inet6 2001:470:1f15:e4::1 prefixlen 64"ifconfig_bridge6_aliases="inet6 2001:470:1f15:e4::25 prefixlen 64 \inet6 2001:470:1f15:e4::80 prefixlen 64 \inet6 2001:470:1f15:e4::5222 prefixlen 64 \inet6 2001:470:1f15:e4:c0fe::53 prefixlen 64 \"# VPNwireguard_enable="YES"wireguard_interfaces="wg0"# Firewallpf_enable="YES"# Jailsjail_enable="YES"jailer_dir="zfs:zroot/jails"# DNSnamed_enable="YES"# Mailsmtpd_enable="YES"smtpd_config="/usr/local/etc/smtpd.conf"# XMPPprosody_enable="YES"turnserver_enable="YES"# Webnginx_enable="YES"tor_enable="YES"
The
gif0interface is a IPv6-over-IPv4 tunnel. I have static routes to my home network, so I don’t go to my server over the ISP every time. This also gives me the ability to get IPv6 in my home network that is routed via my home server.As you have guessed from this config file, I do have VLANs setup. So let’s get into that.
The Home Network
First of all, here’s a very cheap diagram
I have the following VLANs setup on the switch.
VLAN IDPurpose1Switch Management1000pingvinashen (home server) WAN1001evn0 (home router) WAN37pingvinashen ↔ evn042Internal Management100Home LAN69Home GuestHere are the active ports
PortVLANsPurpose24untagged: 1Switch management, connects to Port 222untagged: 1000pingvinashen WAN, from ISP21untagged: 1001Home WAN, from ISP20tagged: 1000, 37To pingvinashen, portem019untagged: 1001To home router, portigb118tagged: 42, 100, 69, 99To home router, portigb217untagged: 37To home router, portigb016tagged: 42, 100, 69To Lenovo T480s15untagged: 100To Raspberri Pi 42untagged: 99From Port 24, for switch management1untagged: 42; tagged: 100, 69; PoETo UAP AC ProThe home router, hostnamed
evn0(named after the IATA code of Yerevan’s Zvartnots International Airport) runs FreeBSD as well, the hardware is the followingroot@evn0:~ # dmidecode -s system-product-nameAPU2root@evn0:~ # sysctl hw.modelhw.model: AMD GX-412TC SOC root@evn0:~ # sysctl hw.physmemhw.physmem: 4234399744root@evn0:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 12.5G 9.47G 3.03G - - 67% 75% 1.00x ONLINE -
The home router does… well, routing. It also does DHCP, DNS, SLAAC, and can act as a syslog server.
Here’s what the
rc.conflooks likeclear_tmp_enable="YES"sendmail_enable="NONE"syslogd_flags="-a '172.16.100.0/24:*' -H"zfs_enable="YES"dumpdev="AUTO"hostname="evn0.illuriasecurity.com"pf_enable="YES"gateway_enable="YES"ipv6_gateway_enable="YES"sshd_enable="YES"# Get an IP address from the ISP's GPONifconfig_igb1="DHCP"# Internal routes with pingvinashenifconfig_igb0="inet 192.168.255.1 netmask 255.255.255.0"ifconfig_igb0_ipv6="inet6 2001:470:7914:7065::1 prefixlen 64"static_routes="pingvinashen"route_pingvinashen="-net 37.157.221.130/32 -gateway 192.168.255.2"ipv6_defaultrouter="2001:470:7914:7065::2"# Home Mgmt, Switch Mgmt, Home LAN, Home Guestifconfig_igb2="up"vlans_igb2="42 99 100 69"ifconfig_igb2_42="inet 172.31.42.1 netmask 255.255.255.0"ifconfig_igb2_99="inet 172.16.99.1 netmask 255.255.255.0"ifconfig_igb2_100="inet 172.16.100.1 netmask 255.255.255.0"ifconfig_igb2_100_ipv6="inet6 2001:470:7914:6a76::1 prefixlen 64"ifconfig_igb2_69="inet 192.168.69.1 netmask 255.255.255.0"ifconfig_igb2_69_ipv6="inet6 2001:470:7914:6969::1 prefixlen 64"# DNS and DHCPnamed_enable="YES"dhcpd_enable="YES"named_flags=""# NTPntpd_enable="YES"# Router Advertisement and LLDPrtadvd_enable="YES"lldpd_enable="YES"lldpd_flags=""
Here’s
pf.conf, because security is important.ext_if="igb1"bsd_if="igb0"int_if="igb2.100"guest_if="igb2.69"mgmt_if="igb2.42"sw_if="igb2.99"ill_net="172.16.0.0/16"nat pass on $ext_if from $int_if:network to any -> ($ext_if)nat pass on $ext_if from $mgmt_if:network to any -> ($ext_if)nat pass on $ext_if from $guest_if:network to any -> ($ext_if)set skip on { lo0 }block in allpass on $int_if from $int_if:network to anypass on $mgmt_if from $mgmt_if:network to anypass on $sw_if from $sw_if:network to anypass on $guest_if from $guest_if:network to anyblock quick on $guest_if from any to { $int_if:network, $mgmt_if:network, $ill_net, $sw_if:network }pass in on illuria0 from $ill_net to { $ill_net, $mgmt_if:network }pass inet proto icmppass inet6 proto icmp6pass out all keep stateI’m sure there are places to improve, but it gets the job done and keeps the guest network isolated.
Here’s
rtadvd.conf, for my IPv6 folksigb2.100:\ :addr="2001:470:7914:6a76::":prefixlen#64:\ :rdnss="2001:470:7914:6a76::1":\ :dnssl="evn0.loc.illuriasecurity.com,loc.illuriasecurity.com":igb2.69:\ :addr="2001:470:7914:6969::":prefixlen#64:\ :rdnss="2001:470:7914:6969::1":
For DNS, I’m running BIND, here’s the important parts
listen-on { 127.0.0.1; 172.16.100.1; 172.16.99.1; 172.31.42.1; 192.168.69.1; };listen-on-v6 { 2001:470:7914:6a76::1; 2001:470:7914:6969::1; };allow-query { 127.0.0.1; 172.16.100.0/24; 172.31.42.0/24; 192.168.69.0/24; 2001:470:7914:6a76::/64; 2001:470:7914:6969::/64;};And for DHCP, here’s what it looks like
subnet 172.16.100.0 netmask 255.255.255.0 { range 172.16.100.100 172.16.100.150; option domain-name-servers 172.16.100.1; option subnet-mask 255.255.255.0; option routers 172.16.100.1; option domain-name "evn0.loc.illuriasecurity.com"; option domain-search "loc.illuriasecurity.com evn0.loc.illuriasecurity.com";}host zvartnots { hardware ethernet d4:57:63:f1:5a:36; fixed-address 172.16.100.7;}host unifi0 { hardware ethernet 58:9c:fc:93:d1:0b; fixed-address 172.31.42.42;}
[…]subnet 172.31.42.0 netmask 255.255.255.0 { range 172.31.42.100 172.31.42.150; option domain-name-servers 172.31.42.1; option subnet-mask 255.255.255.0; option routers 172.31.42.1;}subnet 192.168.69.0 netmask 255.255.255.0 { range 192.168.69.100 192.168.69.150; option domain-name-servers 192.168.69.1; option subnet-mask 255.255.255.0; option routers 192.168.69.1;}So you’re wondering, what’s this
unifi0? Well, that brings us toT480s
This laptop has been gifted to me by [REDACTED] for my contributions to the Armenian government (which means when a server goes down and no one knows how to fix it, they called me and I showed up)
Here’s the hardware
root@t480s:~ # dmidecode -s system-versionThinkPad T480sroot@t480s:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHzroot@t480s:~ # sysctl hw.physmemhw.physmem: 25602347008root@t480s:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 224G 109G 115G - - 44% 48% 1.00x ONLINE -
The T480s has access to VLAN 100, 42, 69, but the host itself has access only to VLAN 100 (LAN), while the jails can exist on other VLANs.
So I have a Jail named
unifi0that runs the Unifi Management thingie.Here’s what
rc.confof the host looks likeclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"sshd_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="t480s.evn0.loc.illuriasecurity.com"ifconfig_em0="up -rxcsum -txcsum"vlans_em0="100 42 69"ifconfig_em0_100="up"ifconfig_em0_42="up"ifconfig_em0_69="up"cloned_interfaces="bridge0 bridge100 bridge42 bridge69"create_args_bridge100="ether 8c:16:45:82:b4:10"ifconfig_bridge100="addm em0.100 SYNCDHCP"ifconfig_bridge100_ipv6="inet6 auto_linklocal"rtsold_flags="-i -F -m bridge100"rtsold_enable="YES"create_args_bridge42=" ether 8c:16:45:82:b4:42"create_args_bridge69=" ether 8c:16:45:82:b4:69"ifconfig_bridge42="addm em0.42"ifconfig_bridge69="addm em0.69"jail_enable="YES"jailer_dir="zfs:zroot/jailer"ifconfig_bridge0="inet 10.1.0.1/24 up"ngbuddy_enable="YES"ngbuddy_private_if="nghost0"dhcpd_enable="YES"lldpd_enable="YES"
I used Jailer to create the
unifi0jail, here’s what thejail.conflooks like# vim: set syntax=sh:exec.clean;allow.raw_sockets;mount.devfs;unifi0 { $id = "6"; devfs_ruleset = 10; $bridge = "bridge42"; $domain = "evn0.loc.illuriasecurity.com"; vnet; vnet.interface = "epair${id}b"; exec.prestart = "ifconfig epair${id} create up"; exec.prestart += "ifconfig epair${id}a up descr vnet-${name}"; exec.prestart += "ifconfig ${bridge} addm epair${id}a up"; exec.start = "/sbin/ifconfig lo0 127.0.0.1 up"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown jail"; exec.poststop = "ifconfig ${bridge} deletem epair${id}a"; exec.poststop += "ifconfig epair${id}a destroy"; host.hostname = "${name}.${domain}"; path = "/usr/local/jailer/unifi0"; exec.consolelog = "/var/log/jail/${name}.log"; persist; mount.fdescfs; mount.procfs;}Here are the important parts inside the jail
root@t480s:~ # cat /usr/local/jailer/unifi0/etc/rc.confifconfig_epair6b="SYNCDHCP"sendmail_enable="NONE"syslogd_flags="-ss"mongod_enable="YES"unifi_enable="YES"root@t480s:~ # cat /usr/local/jailer/unifi0/etc/start_if.epair6b ifconfig epair6b ether 58:9c:fc:93:d1:0b
Don’t you love it that you can see what’s inside the jail from the host? God I love FreeBSD!
Did I miss anything? I hope not.
Oh, for the homelabbers out there, the T480s is the one that runs things like Jellyfin if needed.
Finally, the tiny
Raspberry Pi 4, Model B
I found this in a closed, so I decided to run it for TimeMachine.
I guess all you care about is
rc.confhostname="tm0.evn0.loc.illuriasecurity.com"ifconfig_DEFAULT="DHCP inet6 accept_rtadv"sshd_enable="YES"sendmail_enable="NONE"sendmail_submit_enable="NO"sendmail_outbound_enable="NO"sendmail_msp_queue_enable="NO"growfs_enable="YES"powerd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"rtsold_enable="YES"samba_server_enable="YES"
And the Samba Configuration
[global]# Network settingsworkgroup = WORKGROUPserver string = Samba Server %vnetbios name = RPi4# Logginglog file = /var/log/samba4/log.%mmax log size = 50log level = 0# Authenticationsecurity = userencrypt passwords = yespassdb backend = tdbsammap to guest = Bad Usermin protocol = SMB2max protocol = SMB3# Apple Time Machine settingsvfs objects = catia fruit streams_xattrfruit:metadata = streamfruit:resource = streamfruit:encoding = nativefruit:locking = nonefruit:time machine = yes# File System supportea support = yeskernel oplocks = nokernel share modes = noposix locking = nomangled names = nosmbd max xattr size = 2097152# Performance tuningread raw = yeswrite raw = yesgetwd cache = yesstrict locking = no# Miscellaneouslocal master = nopreferred master = nodomain master = nowins support = no[tm]comment = Time Machine RPi4path = /usr/local/timemachine/%Ubrowseable = yesread only = novalid users = antranigvvfs objects = catia fruit streams_xattrfruit:time machine = yesfruit:advertise_fullsync = truefruit:time machine max size = 800G # Adjust the size according to your needscreate mask = 0600directory mask = 0700
That’s pretty much it.
Conclusion
I love running homebrew servers, home networks and home labs. I love that (almost) everything is FreeBSD. The switch itself runs Linux, and the Unifi Access Point also runs Linux, both of which I’m pretty happy with.
While most homelabbers used ESXi in the past, I’m happy to see that most people are moving to open source solutions like Proxmox and Xen, but I think that FreeBSD Jails and bhyve is much better. I still don’t have a need for bhyve at the moment, but I would use it if I needed hardware virtualization.
Most homelabbers would consider the lack of Web/GUI interfaces as a con, but I think that it’s a pro. If I need to “replicate” this network, all I need to do is to copy some text files and modify some IP addresses / Interface names.
I hope this was informative and that it would be useful for anyone in the future.
That’s all folks…
Reply via email.
https://antranigv.am/posts/2024/06/freebsd-server-network-homelab/
#Containers #Dell #DellLatitudeE5470 #FreeBSD #homeServer #HowTo #Jailer #Jails #macOS #Networking #pf #Samba #Unifi #Unix #VNET
-
The FreeBSD-native-ish home lab and network
For many years my setup was pretty simple: A FreeBSD home server running on my old laptop. It runs everything I need to be present on the internet, an email server, a web server (like the one you’ve accessed right now to see this blog post) and a public chat server (XMPP/Jabber) so I can be in touch with friends.
For my home network, I had a basic Access Point and a basic Router.
Lately, my setup has become more… intense. I have IPv6 thanks to Hurricane Electric, the network is passed to my home network (which we’ll talk about in a bit), a home network with multiple VLANs, since friends who come home also need WiFi.
I decided to blog about the details, hoping it would help someone in the future.
I’ll start with the simplest one.
The Home Server
I’ve been running home servers for a long time. I believe that every person/family needs a home server. Forget about buying your kids iPads and Smartphones. Their first devices should be a real computer (sorry Apple, iOS devices are still just a toy) like a desktop/laptop and a home server. The home server doesn’t need to be on the public internet, but mine is, for variety of reasons. This blog being one of them.
I get a static IP address from my ISP, Ucom. After the management change that happened couple of years ago, Ucom has become a very typical ISP (think shitty), but they are the only ones that provide a static IP address, instead of setting it on your router, where you have to do port forwarding.
My home server, hostnamed pingvinashen (meaning the town of the penguins, named after the Armenian cartoon) run FreeBSD. Historically this machine has run Debian, Funtoo, Gentoo and finally FreeBSD.
Hardware wise, here’s what it is:
root@pingvinashen:~ # dmidecode -s system-product-nameLatitude E5470root@pingvinashen:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHzroot@pingvinashen:~ # sysctl hw.physmemhw.physmem: 17016950784root@pingvinashen:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 420G 178G 242G - - 64% 42% 1.00x ONLINE -
While most homelabbers use hardware virtualization, I think that resources are a tight thing, and should be managed properly. Any company that markets itself as “green/eco-friendly” and uses hardware virtualization should do calculations using a pen and paper and prove if going native would save power/resources or not. (sometimes it doesn’t, usually it does)
I use containers, the old-school ones, Jails to be more specific.
I manage jails using Jailer, my own tool, that tries to stay out of your way when working with Jails.
Here are my current jails:
root@pingvinashen:~ # jailer listNAME STATE JID HOSTNAME IPv4 GWantranig Active 1 antranig.bsd.am 192.168.10.42/24 192.168.10.1antranigv Active 2 antranigv.bsd.am 192.168.10.52/24 192.168.10.1git Stoppedhuginn0 Active 4 huginn0.bsd.am 192.168.10.34/24 192.168.10.1ifconfig Active 5 ifconfig.bsd.am 192.168.10.33/24 192.168.10.1lucy Active 6 lucy.vartanian.am 192.168.10.37/24 192.168.10.1mysql Active 7 mysql.antranigv.am 192.168.10.50/24 192.168.10.1newsletter Active 8 newsletter.bsd.am 192.168.10.65/24 192.168.10.1oragir Active 9 oragir.am 192.168.10.30/24 192.168.10.1psql Active 10 psql.pingvinashen.am 192.168.10.3/24 192.168.10.1rss Active 11 rss.bsd.am 192.168.10.5/24 192.168.10.1sarian Active 12 sarian.am 192.168.10.53/24 192.168.10.1syuneci Active 13 syuneci.am 192.168.10.60/24 192.168.10.1znc Active 14 znc.bsd.am 192.168.10.152/24 192.168.10.1
You already get a basic idea of how things are. Each of my blogs (Armenian and English) has its own Jail. Since I’m using WordPress, I need a database, so I have a MySQL jail (which ironically runs MariaDB) inside of it.
I also have a Git server, running gitea, which is down at the moment as I’m doing maintanence. The Git server (and many other services) requires PostgreSQL, hence the existence of a PostgreSQL jail. I run huginn for automation (RSS to Telegram, RSS to XMPP). My sister has her own blog, using WordPress, so that’s a Jail of its own. Same goes about my fiancée.
Other Jails are Newsletter using Listmonk, Sarian (the Armenian instance of lobste.rs) and a personal ZNC server.
As an avid RSS advocate, I also have a RSS Jail, which runs Miniflux. Many of my friends use this service.
Oragir is an instance of WriteFreely, as I advocate public blogging and ActivityPub. Our community uses that too.
The web server that forwards all this traffic from the public to the Jails is nginx. All it does is
proxy_passas needed. It runs on the host.Other services that run on the host are DNS (BIND9), an email service running
OpenSMTPd(which will be moved to a Jail soon), the chat service runningprosody(which will be moved to a Jail soon) and finally, WireGuard, because I love VPNs.Finally, there’s a IPv6-over-IPv4 tunnel that I use to obtain IPv6 thanks to Hurricane Electric.
Yes, I have a firewall, I use
pf(4).For the techies in the room, here’s what my
rc.conflooks like.# cat /etc/rc.conf# Defaultsclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"#local_unbound_enable="YES"sshd_enable="YES"moused_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="pingvinashen.am"# Networkingdefaultrouter="37.157.221.1"gateway_enable="YES"ifconfig_em0="up"vlans_em0="37 1000" # 1000 -> WAN; 37 -> Home Routerifconfig_em0_1000="inet 37.157.221.130 netmask 255.255.255.0"ifconfig_em0_37="inet 192.168.255.2 netmask 255.255.255.0"static_routes="home"route_home="-net 172.16.100.0/24 -gateway 192.168.255.1"cloned_interfaces="bridge0 bridge6 bridge10"ifconfig_bridge10="inet 192.168.10.1 netmask 255.255.255.0"## IPv6ipv6_gateway_enable="YES"gif_interfaces="gif0"gifconfig_gif0="37.157.221.130 216.66.84.46"ifconfig_gif0="inet6 2001:470:1f14:ef::2 2001:470:1f14:ef::1 prefixlen 128"ipv6_defaultrouter="2001:470:1f14:ef::1"ifconfig_em0_37_ipv6="inet6 2001:470:7914:7065::2 prefixlen 64"ipv6_static_routes="home guest"ipv6_route_home="-net 2001:470:7914:6a76::/64 -gateway 2001:470:7914:7065::1"ipv6_route_guest="-net 2001:470:7914:6969::/64 -gateway 2001:470:7914:7065::1"ifconfig_bridge6_ipv6="inet6 2001:470:1f15:e4::1 prefixlen 64"ifconfig_bridge6_aliases="inet6 2001:470:1f15:e4::25 prefixlen 64 \inet6 2001:470:1f15:e4::80 prefixlen 64 \inet6 2001:470:1f15:e4::5222 prefixlen 64 \inet6 2001:470:1f15:e4:c0fe::53 prefixlen 64 \"# VPNwireguard_enable="YES"wireguard_interfaces="wg0"# Firewallpf_enable="YES"# Jailsjail_enable="YES"jailer_dir="zfs:zroot/jails"# DNSnamed_enable="YES"# Mailsmtpd_enable="YES"smtpd_config="/usr/local/etc/smtpd.conf"# XMPPprosody_enable="YES"turnserver_enable="YES"# Webnginx_enable="YES"tor_enable="YES"
The
gif0interface is a IPv6-over-IPv4 tunnel. I have static routes to my home network, so I don’t go to my server over the ISP every time. This also gives me the ability to get IPv6 in my home network that is routed via my home server.As you have guessed from this config file, I do have VLANs setup. So let’s get into that.
The Home Network
First of all, here’s a very cheap diagram
I have the following VLANs setup on the switch.
VLAN IDPurpose1Switch Management1000pingvinashen (home server) WAN1001evn0 (home router) WAN37pingvinashen ↔ evn042Internal Management100Home LAN69Home GuestHere are the active ports
PortVLANsPurpose24untagged: 1Switch management, connects to Port 222untagged: 1000pingvinashen WAN, from ISP21untagged: 1001Home WAN, from ISP20tagged: 1000, 37To pingvinashen, portem019untagged: 1001To home router, portigb118tagged: 42, 100, 69, 99To home router, portigb217untagged: 37To home router, portigb016tagged: 42, 100, 69To Lenovo T480s15untagged: 100To Raspberri Pi 42untagged: 99From Port 24, for switch management1untagged: 42; tagged: 100, 69; PoETo UAP AC ProThe home router, hostnamed
evn0(named after the IATA code of Yerevan’s Zvartnots International Airport) runs FreeBSD as well, the hardware is the followingroot@evn0:~ # dmidecode -s system-product-nameAPU2root@evn0:~ # sysctl hw.modelhw.model: AMD GX-412TC SOC root@evn0:~ # sysctl hw.physmemhw.physmem: 4234399744root@evn0:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 12.5G 9.47G 3.03G - - 67% 75% 1.00x ONLINE -
The home router does… well, routing. It also does DHCP, DNS, SLAAC, and can act as a syslog server.
Here’s what the
rc.conflooks likeclear_tmp_enable="YES"sendmail_enable="NONE"syslogd_flags="-a '172.16.100.0/24:*' -H"zfs_enable="YES"dumpdev="AUTO"hostname="evn0.illuriasecurity.com"pf_enable="YES"gateway_enable="YES"ipv6_gateway_enable="YES"sshd_enable="YES"# Get an IP address from the ISP's GPONifconfig_igb1="DHCP"# Internal routes with pingvinashenifconfig_igb0="inet 192.168.255.1 netmask 255.255.255.0"ifconfig_igb0_ipv6="inet6 2001:470:7914:7065::1 prefixlen 64"static_routes="pingvinashen"route_pingvinashen="-net 37.157.221.130/32 -gateway 192.168.255.2"ipv6_defaultrouter="2001:470:7914:7065::2"# Home Mgmt, Switch Mgmt, Home LAN, Home Guestifconfig_igb2="up"vlans_igb2="42 99 100 69"ifconfig_igb2_42="inet 172.31.42.1 netmask 255.255.255.0"ifconfig_igb2_99="inet 172.16.99.1 netmask 255.255.255.0"ifconfig_igb2_100="inet 172.16.100.1 netmask 255.255.255.0"ifconfig_igb2_100_ipv6="inet6 2001:470:7914:6a76::1 prefixlen 64"ifconfig_igb2_69="inet 192.168.69.1 netmask 255.255.255.0"ifconfig_igb2_69_ipv6="inet6 2001:470:7914:6969::1 prefixlen 64"# DNS and DHCPnamed_enable="YES"dhcpd_enable="YES"named_flags=""# NTPntpd_enable="YES"# Router Advertisement and LLDPrtadvd_enable="YES"lldpd_enable="YES"lldpd_flags=""
Here’s
pf.conf, because security is important.ext_if="igb1"bsd_if="igb0"int_if="igb2.100"guest_if="igb2.69"mgmt_if="igb2.42"sw_if="igb2.99"ill_net="172.16.0.0/16"nat pass on $ext_if from $int_if:network to any -> ($ext_if)nat pass on $ext_if from $mgmt_if:network to any -> ($ext_if)nat pass on $ext_if from $guest_if:network to any -> ($ext_if)set skip on { lo0 }block in allpass on $int_if from $int_if:network to anypass on $mgmt_if from $mgmt_if:network to anypass on $sw_if from $sw_if:network to anypass on $guest_if from $guest_if:network to anyblock quick on $guest_if from any to { $int_if:network, $mgmt_if:network, $ill_net, $sw_if:network }pass in on illuria0 from $ill_net to { $ill_net, $mgmt_if:network }pass inet proto icmppass inet6 proto icmp6pass out all keep stateI’m sure there are places to improve, but it gets the job done and keeps the guest network isolated.
Here’s
rtadvd.conf, for my IPv6 folksigb2.100:\ :addr="2001:470:7914:6a76::":prefixlen#64:\ :rdnss="2001:470:7914:6a76::1":\ :dnssl="evn0.loc.illuriasecurity.com,loc.illuriasecurity.com":igb2.69:\ :addr="2001:470:7914:6969::":prefixlen#64:\ :rdnss="2001:470:7914:6969::1":
For DNS, I’m running BIND, here’s the important parts
listen-on { 127.0.0.1; 172.16.100.1; 172.16.99.1; 172.31.42.1; 192.168.69.1; };listen-on-v6 { 2001:470:7914:6a76::1; 2001:470:7914:6969::1; };allow-query { 127.0.0.1; 172.16.100.0/24; 172.31.42.0/24; 192.168.69.0/24; 2001:470:7914:6a76::/64; 2001:470:7914:6969::/64;};And for DHCP, here’s what it looks like
subnet 172.16.100.0 netmask 255.255.255.0 { range 172.16.100.100 172.16.100.150; option domain-name-servers 172.16.100.1; option subnet-mask 255.255.255.0; option routers 172.16.100.1; option domain-name "evn0.loc.illuriasecurity.com"; option domain-search "loc.illuriasecurity.com evn0.loc.illuriasecurity.com";}host zvartnots { hardware ethernet d4:57:63:f1:5a:36; fixed-address 172.16.100.7;}host unifi0 { hardware ethernet 58:9c:fc:93:d1:0b; fixed-address 172.31.42.42;}
[…]subnet 172.31.42.0 netmask 255.255.255.0 { range 172.31.42.100 172.31.42.150; option domain-name-servers 172.31.42.1; option subnet-mask 255.255.255.0; option routers 172.31.42.1;}subnet 192.168.69.0 netmask 255.255.255.0 { range 192.168.69.100 192.168.69.150; option domain-name-servers 192.168.69.1; option subnet-mask 255.255.255.0; option routers 192.168.69.1;}So you’re wondering, what’s this
unifi0? Well, that brings us toT480s
This laptop has been gifted to me by [REDACTED] for my contributions to the Armenian government (which means when a server goes down and no one knows how to fix it, they called me and I showed up)
Here’s the hardware
root@t480s:~ # dmidecode -s system-versionThinkPad T480sroot@t480s:~ # sysctl hw.modelhw.model: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHzroot@t480s:~ # sysctl hw.physmemhw.physmem: 25602347008root@t480s:~ # zpool listNAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTzroot 224G 109G 115G - - 44% 48% 1.00x ONLINE -
The T480s has access to VLAN 100, 42, 69, but the host itself has access only to VLAN 100 (LAN), while the jails can exist on other VLANs.
So I have a Jail named
unifi0that runs the Unifi Management thingie.Here’s what
rc.confof the host looks likeclear_tmp_enable="YES"syslogd_flags="-ss"sendmail_enable="NONE"sshd_enable="YES"ntpd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"hostname="t480s.evn0.loc.illuriasecurity.com"ifconfig_em0="up -rxcsum -txcsum"vlans_em0="100 42 69"ifconfig_em0_100="up"ifconfig_em0_42="up"ifconfig_em0_69="up"cloned_interfaces="bridge0 bridge100 bridge42 bridge69"create_args_bridge100="ether 8c:16:45:82:b4:10"ifconfig_bridge100="addm em0.100 SYNCDHCP"ifconfig_bridge100_ipv6="inet6 auto_linklocal"rtsold_flags="-i -F -m bridge100"rtsold_enable="YES"create_args_bridge42=" ether 8c:16:45:82:b4:42"create_args_bridge69=" ether 8c:16:45:82:b4:69"ifconfig_bridge42="addm em0.42"ifconfig_bridge69="addm em0.69"jail_enable="YES"jailer_dir="zfs:zroot/jailer"ifconfig_bridge0="inet 10.1.0.1/24 up"ngbuddy_enable="YES"ngbuddy_private_if="nghost0"dhcpd_enable="YES"lldpd_enable="YES"
I used Jailer to create the
unifi0jail, here’s what thejail.conflooks like# vim: set syntax=sh:exec.clean;allow.raw_sockets;mount.devfs;unifi0 { $id = "6"; devfs_ruleset = 10; $bridge = "bridge42"; $domain = "evn0.loc.illuriasecurity.com"; vnet; vnet.interface = "epair${id}b"; exec.prestart = "ifconfig epair${id} create up"; exec.prestart += "ifconfig epair${id}a up descr vnet-${name}"; exec.prestart += "ifconfig ${bridge} addm epair${id}a up"; exec.start = "/sbin/ifconfig lo0 127.0.0.1 up"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown jail"; exec.poststop = "ifconfig ${bridge} deletem epair${id}a"; exec.poststop += "ifconfig epair${id}a destroy"; host.hostname = "${name}.${domain}"; path = "/usr/local/jailer/unifi0"; exec.consolelog = "/var/log/jail/${name}.log"; persist; mount.fdescfs; mount.procfs;}Here are the important parts inside the jail
root@t480s:~ # cat /usr/local/jailer/unifi0/etc/rc.confifconfig_epair6b="SYNCDHCP"sendmail_enable="NONE"syslogd_flags="-ss"mongod_enable="YES"unifi_enable="YES"root@t480s:~ # cat /usr/local/jailer/unifi0/etc/start_if.epair6b ifconfig epair6b ether 58:9c:fc:93:d1:0b
Don’t you love it that you can see what’s inside the jail from the host? God I love FreeBSD!
Did I miss anything? I hope not.
Oh, for the homelabbers out there, the T480s is the one that runs things like Jellyfin if needed.
Finally, the tiny
Raspberry Pi 4, Model B
I found this in a closed, so I decided to run it for TimeMachine.
I guess all you care about is
rc.confhostname="tm0.evn0.loc.illuriasecurity.com"ifconfig_DEFAULT="DHCP inet6 accept_rtadv"sshd_enable="YES"sendmail_enable="NONE"sendmail_submit_enable="NO"sendmail_outbound_enable="NO"sendmail_msp_queue_enable="NO"growfs_enable="YES"powerd_enable="YES"# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disabledumpdev="AUTO"zfs_enable="YES"rtsold_enable="YES"samba_server_enable="YES"
And the Samba Configuration
[global]# Network settingsworkgroup = WORKGROUPserver string = Samba Server %vnetbios name = RPi4# Logginglog file = /var/log/samba4/log.%mmax log size = 50log level = 0# Authenticationsecurity = userencrypt passwords = yespassdb backend = tdbsammap to guest = Bad Usermin protocol = SMB2max protocol = SMB3# Apple Time Machine settingsvfs objects = catia fruit streams_xattrfruit:metadata = streamfruit:resource = streamfruit:encoding = nativefruit:locking = nonefruit:time machine = yes# File System supportea support = yeskernel oplocks = nokernel share modes = noposix locking = nomangled names = nosmbd max xattr size = 2097152# Performance tuningread raw = yeswrite raw = yesgetwd cache = yesstrict locking = no# Miscellaneouslocal master = nopreferred master = nodomain master = nowins support = no[tm]comment = Time Machine RPi4path = /usr/local/timemachine/%Ubrowseable = yesread only = novalid users = antranigvvfs objects = catia fruit streams_xattrfruit:time machine = yesfruit:advertise_fullsync = truefruit:time machine max size = 800G # Adjust the size according to your needscreate mask = 0600directory mask = 0700
That’s pretty much it.
Conclusion
I love running homebrew servers, home networks and home labs. I love that (almost) everything is FreeBSD. The switch itself runs Linux, and the Unifi Access Point also runs Linux, both of which I’m pretty happy with.
While most homelabbers used ESXi in the past, I’m happy to see that most people are moving to open source solutions like Proxmox and Xen, but I think that FreeBSD Jails and bhyve is much better. I still don’t have a need for bhyve at the moment, but I would use it if I needed hardware virtualization.
Most homelabbers would consider the lack of Web/GUI interfaces as a con, but I think that it’s a pro. If I need to “replicate” this network, all I need to do is to copy some text files and modify some IP addresses / Interface names.
I hope this was informative and that it would be useful for anyone in the future.
That’s all folks…
Reply via email.
https://antranigv.am/posts/2024/06/freebsd-server-network-homelab/
#Containers #Dell #DellLatitudeE5470 #FreeBSD #homeServer #HowTo #Jailer #Jails #macOS #Networking #pf #Samba #Unifi #Unix #VNET
-
Yes source based distro's have been around since the very beginning - in fact, MCC Interim Linux and #SLS weren't far from that mark, except that they merely tried to make it a bit more convenient by packaging up tarballs to be exploded during installation. And there's always #LFS.
If you think about Slackpkg - and you consider that you can actually re-install the entire system by compiling every single component of the default (full) install with the evocation of a single command, followed by the customization of your entire system by installing every kind of software imaginable through the use of #sbopkg or some other automated, dependency resolving package manager that uses #SlackBuilds (which are downloaded, then exectuted, and subsequently download the latest release of he software package desired, which is in turn compiled, packaged, and exploded) - you actually have a fully source based distro installed on your box.
That's right - Slackware is (can be forced to be) an entirely source based distro installed on your device.
And choosing to convert from a point release to Slackware -current switches you from a point release to a #Rolling_Release distro.
*Debian Testing, aka at this time, Trixie is a rolling release. #Arch_Linux is a rolling release, SourceMage and Lunar Linux are source based distros based on #Sorcerer_Linux, the original fully source based Linux distro released when Linux was only about 8yrs old in 2000, and the #Gentoo or #Funtoo source based Linux distros.
SystemD my ass. That has nothing to do with nothing in that conversation - it's completely non-sequitur and truth be told, most source based distros (Arch, Gentoo) support the type of init system that *YOU CHOOSE. For Debiantards such as myself, well..... There's #Devuan - and that's very refreshing to actually have control over your system again with true init scripts. But I rarely use Devuan, even though I've been associated with the initiative since its inception, after leaving the #Mageia team several years ago.
As I state in almost all of my profiles, I'm a Slacker, since 1993 (Slackware Linux), and I'm also a bit of a #Debiantard. On the BSD side, after leaving #Jolix (386BSD) for Slackware, I've pretty much settled on either #OpenBSD or #Dragonfly_BSD, w/the awesome #HAMMER2 FS. I still have a lot of love for #FreeBSD and of course #NetBSD - where I spend a lot of time in my proper #Korn Shell....
But what the heck does any of this have to do with a comparison of using Gentoo Linux being akin to using SystemD?
I don't like SystemD - but if you're a realist, that doesn't mean you forgo using distros that only have that init tooling. You just roll with the punches and keep following the innovations that support you - NO ONE STILL RUNS WINDOWS XP in production - at least, no one outside of state mental hospitals, that's just insane to do in a forward facing business environment.
But a lot of companies do leverage OpenRC, SysVinit, etc., instead of SystemD - that's not going away, and SystemD itself and Poetering have their own up and coming challengers.
SystemD is (supposed to be, originally) a way to boot your box. Yes, it's indeed encroached upon other landscapes since, but not all of those constructs are even considered by many mainstream distros - it's not a fact of life. Other init systems thrive in the UNIX world to this day and will continue to do so.
Likewise, Source based Linux distros are just one among many distros that exist, and may or may not leverage SystemD as their init systems - to really get a good grasp of this, I recommend doing a few Arch Linux installs - with and without SystemD as the base init system. Heck, even Debian still supports your regular, good old #syslog, and at every turn during your updates, reminds you how to keep it enabled since the whole journalctl crap just isn't as elegant, IMO.
Personally, I think more concurrent options are usually better - space is cheap. Storage no longer costs a dollar a meg. or worse, like it was when I was a kid, a few thousand dollars a meg. That's right... MegaByte - Not TB for penny's!
Okay so now I'm waiting to hear back from the OP and see just what the heck they meant when I got triggered. In the meantime....
Enjoy installing and using #Sorcerer_Linux, or the subesquent forks of it's surviving lineage like #SourceMage and #Lunar_Linux - you're now a part of mainstream source-basedLinux History once you do 🤘 💀 🤘
#tallship #Linux #FOSS #distros #Sorcerer
⛵️
.
RE: https://social.sdf.org/users/tallship/statuses/111957857148746923
-
Yes source based distro's have been around since the very beginning - in fact, MCC Interim Linux and #SLS weren't far from that mark, except that they merely tried to make it a bit more convenient by packaging up tarballs to be exploded during installation. And there's always #LFS.
If you think about Slackpkg - and you consider that you can actually re-install the entire system by compiling every single component of the default (full) install with the evocation of a single command, followed by the customization of your entire system by installing every kind of software imaginable through the use of #sbopkg or some other automated, dependency resolving package manager that uses #SlackBuilds (which are downloaded, then exectuted, and subsequently download the latest release of he software package desired, which is in turn compiled, packaged, and exploded) - you actually have a fully source based distro installed on your box.
That's right - Slackware is (can be forced to be) an entirely source based distro installed on your device.
And choosing to convert from a point release to Slackware -current switches you from a point release to a #Rolling_Release distro.
*Debian Testing, aka at this time, Trixie is a rolling release. #Arch_Linux is a rolling release, SourceMage and Lunar Linux are source based distros based on #Sorcerer_Linux, the original fully source based Linux distro released when Linux was only about 8yrs old in 2000, and the #Gentoo or #Funtoo source based Linux distros.
SystemD my ass. That has nothing to do with nothing in that conversation - it's completely non-sequitur and truth be told, most source based distros (Arch, Gentoo) support the type of init system that *YOU CHOOSE. For Debiantards such as myself, well..... There's #Devuan - and that's very refreshing to actually have control over your system again with true init scripts. But I rarely use Devuan, even though I've been associated with the initiative since its inception, after leaving the #Mageia team several years ago.
As I state in almost all of my profiles, I'm a Slacker, since 1993 (Slackware Linux), and I'm also a bit of a #Debiantard. On the BSD side, after leaving #Jolix (386BSD) for Slackware, I've pretty much settled on either #OpenBSD or #Dragonfly_BSD, w/the awesome #HAMMER2 FS. I still have a lot of love for #FreeBSD and of course #NetBSD - where I spend a lot of time in my proper #Korn Shell....
But what the heck does any of this have to do with a comparison of using Gentoo Linux being akin to using SystemD?
I don't like SystemD - but if you're a realist, that doesn't mean you forgo using distros that only have that init tooling. You just roll with the punches and keep following the innovations that support you - NO ONE STILL RUNS WINDOWS XP in production - at least, no one outside of state mental hospitals, that's just insane to do in a forward facing business environment.
But a lot of companies do leverage OpenRC, SysVinit, etc., instead of SystemD - that's not going away, and SystemD itself and Poetering have their own up and coming challengers.
SystemD is (supposed to be, originally) a way to boot your box. Yes, it's indeed encroached upon other landscapes since, but not all of those constructs are even considered by many mainstream distros - it's not a fact of life. Other init systems thrive in the UNIX world to this day and will continue to do so.
Likewise, Source based Linux distros are just one among many distros that exist, and may or may not leverage SystemD as their init systems - to really get a good grasp of this, I recommend doing a few Arch Linux installs - with and without SystemD as the base init system. Heck, even Debian still supports your regular, good old #syslog, and at every turn during your updates, reminds you how to keep it enabled since the whole journalctl crap just isn't as elegant, IMO.
Personally, I think more concurrent options are usually better - space is cheap. Storage no longer costs a dollar a meg. or worse, like it was when I was a kid, a few thousand dollars a meg. That's right... MegaByte - Not TB for penny's!
Okay so now I'm waiting to hear back from the OP and see just what the heck they meant when I got triggered. In the meantime....
Enjoy installing and using #Sorcerer_Linux, or the subesquent forks of it's surviving lineage like #SourceMage and #Lunar_Linux - you're now a part of mainstream source-basedLinux History once you do 🤘 💀 🤘
#tallship #Linux #FOSS #distros #Sorcerer
⛵️
.
RE: https://social.sdf.org/users/tallship/statuses/111957857148746923
-
Yes source based distro's have been around since the very beginning - in fact, MCC Interim Linux and #SLS weren't far from that mark, except that they merely tried to make it a bit more convenient by packaging up tarballs to be exploded during installation. And there's always #LFS.
If you think about Slackpkg - and you consider that you can actually re-install the entire system by compiling every single component of the default (full) install with the evocation of a single command, followed by the customization of your entire system by installing every kind of software imaginable through the use of #sbopkg or some other automated, dependency resolving package manager that uses #SlackBuilds (which are downloaded, then exectuted, and subsequently download the latest release of he software package desired, which is in turn compiled, packaged, and exploded) - you actually have a fully source based distro installed on your box.
That's right - Slackware is (can be forced to be) an entirely source based distro installed on your device.
And choosing to convert from a point release to Slackware -current switches you from a point release to a #Rolling_Release distro.
*Debian Testing, aka at this time, Trixie is a rolling release. #Arch_Linux is a rolling release, SourceMage and Lunar Linux are source based distros based on #Sorcerer_Linux, the original fully source based Linux distro released when Linux was only about 8yrs old in 2000, and the #Gentoo or #Funtoo source based Linux distros.
SystemD my ass. That has nothing to do with nothing in that conversation - it's completely non-sequitur and truth be told, most source based distros (Arch, Gentoo) support the type of init system that *YOU CHOOSE. For Debiantards such as myself, well..... There's #Devuan - and that's very refreshing to actually have control over your system again with true init scripts. But I rarely use Devuan, even though I've been associated with the initiative since its inception, after leaving the #Mageia team several years ago.
As I state in almost all of my profiles, I'm a Slacker, since 1993 (Slackware Linux), and I'm also a bit of a #Debiantard. On the BSD side, after leaving #Jolix (386BSD) for Slackware, I've pretty much settled on either #OpenBSD or #Dragonfly_BSD, w/the awesome #HAMMER2 FS. I still have a lot of love for #FreeBSD and of course #NetBSD - where I spend a lot of time in my proper #Korn Shell....
But what the heck does any of this have to do with a comparison of using Gentoo Linux being akin to using SystemD?
I don't like SystemD - but if you're a realist, that doesn't mean you forgo using distros that only have that init tooling. You just roll with the punches and keep following the innovations that support you - NO ONE STILL RUNS WINDOWS XP in production - at least, no one outside of state mental hospitals, that's just insane to do in a forward facing business environment.
But a lot of companies do leverage OpenRC, SysVinit, etc., instead of SystemD - that's not going away, and SystemD itself and Poetering have their own up and coming challengers.
SystemD is (supposed to be, originally) a way to boot your box. Yes, it's indeed encroached upon other landscapes since, but not all of those constructs are even considered by many mainstream distros - it's not a fact of life. Other init systems thrive in the UNIX world to this day and will continue to do so.
Likewise, Source based Linux distros are just one among many distros that exist, and may or may not leverage SystemD as their init systems - to really get a good grasp of this, I recommend doing a few Arch Linux installs - with and without SystemD as the base init system. Heck, even Debian still supports your regular, good old #syslog, and at every turn during your updates, reminds you how to keep it enabled since the whole journalctl crap just isn't as elegant, IMO.
Personally, I think more concurrent options are usually better - space is cheap. Storage no longer costs a dollar a meg. or worse, like it was when I was a kid, a few thousand dollars a meg. That's right... MegaByte - Not TB for penny's!
Okay so now I'm waiting to hear back from the OP and see just what the heck they meant when I got triggered. In the meantime....
Enjoy installing and using #Sorcerer_Linux, or the subesquent forks of it's surviving lineage like #SourceMage and #Lunar_Linux - you're now a part of mainstream source-basedLinux History once you do 🤘 💀 🤘
#tallship #Linux #FOSS #distros #Sorcerer
⛵️
.
RE: https://social.sdf.org/users/tallship/statuses/111957857148746923
-
#ZFS on #Linux observations:
1. ZFS on #Solaris is awesome.
2. My experience with ZFS on Linux has been terrible.
I'm using a Dell #R720 configured as a NAS server, with a Dell PERC H310 controller that natively supports JBOD, running Gentoo Linux. The Dell replaced a succession of two SunFire X4540s, both of which were absolutely rock-solid as NAS servers (until their system controller boards failed) and never once had a ZFS error reported except when a drive physically failed. With the R720, I get hot and cold running errors reported. I'm using all Samsung 870 Evo solid-state drives, in two #RAIDZ arrays, one of eight drives and one of six. I am at this very moment in the process of cleaning up the arrays ... again.
What I can't figure out is why.
— Is ZFS on Linux really that terrible?
— Does ZFS on Linux just somehow not work well with SSDs?
— Does the PERC controller in the R720 not work well with SSDs?
I wasn't originally running SSDs in this array; my first attempt was using 2.5" spinning rust drives. I rapidly discovered two things:
1. As far as I can determine, all 2.5" mechanical hard drives 2TB or larger on the market are SMR drives;
2. OH MY GOD, SMR DRIVES (especially, I am told, in ZFS) ARE UTTERLY FUCKING HORRIBLE except on WORM (read once, write many) applications in which you don't really care how slow the original write is. RAIDZ write performance on the Dell on brand new 2.5" SMR drives was four to six times slower than RAIDZ write performance on the X4540 with older and slower CMR drives on older and slower SCSI/SAS controllers. Despite newer, "faster" drives on a newer, faster controller, the SMR array was utterly unusable.
Now, I'm not experiencing any problems with SSDs in any of my other systems, Windows or Linux, INCLUDING the R720, except with ZFS. The boot drives on the R720 are an mdraid mirror formatted XFS and have never thrown a single error.
So this is really leading me to wonder a crucial question:
Is there something I don't know about #ZFSonLinux that causes it to not work well with #SSD drives? Do I need to just forget about running ZFS on my NAS and let the PERC controller create hardware RAID5 volumes?
(And if anyone wonders "why don't you just run a commercial NAS appliance?", well, I tried that route. I tried one of the very latest generation QNAP servers that run ZFS storage on a Linux OS. Oh my god, I can't even begin to speak to how horribly bastardized it was. QNAP may well be a good NAS choice if you only care about Windows and SMB and never ever want to look under the hood or try to accomplish anything except through the web front-end, and don't already have an existing backup solution that you want to continue using.) -
#ZFS on #Linux observations:
1. ZFS on #Solaris is awesome.
2. My experience with ZFS on Linux has been terrible.
I'm using a Dell #R720 configured as a NAS server, with a Dell PERC H310 controller that natively supports JBOD, running Gentoo Linux. The Dell replaced a succession of two SunFire X4540s, both of which were absolutely rock-solid as NAS servers (until their system controller boards failed) and never once had a ZFS error reported except when a drive physically failed. With the R720, I get hot and cold running errors reported. I'm using all Samsung 870 Evo solid-state drives, in two #RAIDZ arrays, one of eight drives and one of six. I am at this very moment in the process of cleaning up the arrays ... again.
What I can't figure out is why.
— Is ZFS on Linux really that terrible?
— Does ZFS on Linux just somehow not work well with SSDs?
— Does the PERC controller in the R720 not work well with SSDs?
I wasn't originally running SSDs in this array; my first attempt was using 2.5" spinning rust drives. I rapidly discovered two things:
1. As far as I can determine, all 2.5" mechanical hard drives 2TB or larger on the market are SMR drives;
2. OH MY GOD, SMR DRIVES (especially, I am told, in ZFS) ARE UTTERLY FUCKING HORRIBLE except on WORM (read once, write many) applications in which you don't really care how slow the original write is. RAIDZ write performance on the Dell on brand new 2.5" SMR drives was four to six times slower than RAIDZ write performance on the X4540 with older and slower CMR drives on older and slower SCSI/SAS controllers. Despite newer, "faster" drives on a newer, faster controller, the SMR array was utterly unusable.
Now, I'm not experiencing any problems with SSDs in any of my other systems, Windows or Linux, INCLUDING the R720, except with ZFS. The boot drives on the R720 are an mdraid mirror formatted XFS and have never thrown a single error.
So this is really leading me to wonder a crucial question:
Is there something I don't know about #ZFSonLinux that causes it to not work well with #SSD drives? Do I need to just forget about running ZFS on my NAS and let the PERC controller create hardware RAID5 volumes?
(And if anyone wonders "why don't you just run a commercial NAS appliance?", well, I tried that route. I tried one of the very latest generation QNAP servers that run ZFS storage on a Linux OS. Oh my god, I can't even begin to speak to how horribly bastardized it was. QNAP may well be a good NAS choice if you only care about Windows and SMB and never ever want to look under the hood or try to accomplish anything except through the web front-end, and don't already have an existing backup solution that you want to continue using.) -
#ZFS on #Linux observations:
1. ZFS on #Solaris is awesome.
2. My experience with ZFS on Linux has been terrible.
I'm using a Dell #R720 configured as a NAS server, with a Dell PERC H310 controller that natively supports JBOD, running Gentoo Linux. The Dell replaced a succession of two SunFire X4540s, both of which were absolutely rock-solid as NAS servers (until their system controller boards failed) and never once had a ZFS error reported except when a drive physically failed. With the R720, I get hot and cold running errors reported. I'm using all Samsung 870 Evo solid-state drives, in two #RAIDZ arrays, one of eight drives and one of six. I am at this very moment in the process of cleaning up the arrays ... again.
What I can't figure out is why.
— Is ZFS on Linux really that terrible?
— Does ZFS on Linux just somehow not work well with SSDs?
— Does the PERC controller in the R720 not work well with SSDs?
I wasn't originally running SSDs in this array; my first attempt was using 2.5" spinning rust drives. I rapidly discovered two things:
1. As far as I can determine, all 2.5" mechanical hard drives 2TB or larger on the market are SMR drives;
2. OH MY GOD, SMR DRIVES (especially, I am told, in ZFS) ARE UTTERLY FUCKING HORRIBLE except on WORM (read once, write many) applications in which you don't really care how slow the original write is. RAIDZ write performance on the Dell on brand new 2.5" SMR drives was four to six times slower than RAIDZ write performance on the X4540 with older and slower CMR drives on older and slower SCSI/SAS controllers. Despite newer, "faster" drives on a newer, faster controller, the SMR array was utterly unusable.
Now, I'm not experiencing any problems with SSDs in any of my other systems, Windows or Linux, INCLUDING the R720, except with ZFS. The boot drives on the R720 are an mdraid mirror formatted XFS and have never thrown a single error.
So this is really leading me to wonder a crucial question:
Is there something I don't know about #ZFSonLinux that causes it to not work well with #SSD drives? Do I need to just forget about running ZFS on my NAS and let the PERC controller create hardware RAID5 volumes?
(And if anyone wonders "why don't you just run a commercial NAS appliance?", well, I tried that route. I tried one of the very latest generation QNAP servers that run ZFS storage on a Linux OS. Oh my god, I can't even begin to speak to how horribly bastardized it was. QNAP may well be a good NAS choice if you only care about Windows and SMB and never ever want to look under the hood or try to accomplish anything except through the web front-end, and don't already have an existing backup solution that you want to continue using.) -
#ZFS on #Linux observations:
1. ZFS on #Solaris is awesome.
2. My experience with ZFS on Linux has been terrible.
I'm using a Dell #R720 configured as a NAS server, with a Dell PERC H310 controller that natively supports JBOD, running Gentoo Linux. The Dell replaced a succession of two SunFire X4540s, both of which were absolutely rock-solid as NAS servers (until their system controller boards failed) and never once had a ZFS error reported except when a drive physically failed. With the R720, I get hot and cold running errors reported. I'm using all Samsung 870 Evo solid-state drives, in two #RAIDZ arrays, one of eight drives and one of six. I am at this very moment in the process of cleaning up the arrays ... again.
What I can't figure out is why.
— Is ZFS on Linux really that terrible?
— Does ZFS on Linux just somehow not work well with SSDs?
— Does the PERC controller in the R720 not work well with SSDs?
I wasn't originally running SSDs in this array; my first attempt was using 2.5" spinning rust drives. I rapidly discovered two things:
1. As far as I can determine, all 2.5" mechanical hard drives 2TB or larger on the market are SMR drives;
2. OH MY GOD, SMR DRIVES (especially, I am told, in ZFS) ARE UTTERLY FUCKING HORRIBLE except on WORM (read once, write many) applications in which you don't really care how slow the original write is. RAIDZ write performance on the Dell on brand new 2.5" SMR drives was four to six times slower than RAIDZ write performance on the X4540 with older and slower CMR drives on older and slower SCSI/SAS controllers. Despite newer, "faster" drives on a newer, faster controller, the SMR array was utterly unusable.
Now, I'm not experiencing any problems with SSDs in any of my other systems, Windows or Linux, INCLUDING the R720, except with ZFS. The boot drives on the R720 are an mdraid mirror formatted XFS and have never thrown a single error.
So this is really leading me to wonder a crucial question:
Is there something I don't know about #ZFSonLinux that causes it to not work well with #SSD drives? Do I need to just forget about running ZFS on my NAS and let the PERC controller create hardware RAID5 volumes?
(And if anyone wonders "why don't you just run a commercial NAS appliance?", well, I tried that route. I tried one of the very latest generation QNAP servers that run ZFS storage on a Linux OS. Oh my god, I can't even begin to speak to how horribly bastardized it was. QNAP may well be a good NAS choice if you only care about Windows and SMB and never ever want to look under the hood or try to accomplish anything except through the web front-end, and don't already have an existing backup solution that you want to continue using.) -
Alright new instance new #introduction time. I should really just copy this somewhere so I don't have to keep retyping it, given how much I move.
I'm #queer, and #polyamorous who's about 30 years old at the time of writing. I do tend to hop around fediverse instances, but I'd like to stop that at some point. I liked Friendica a lot, but my last 2 instances died very suddenly, and I decided against trying a third time. My more stationary account is @[email protected], which should be up indefinitely if I don't decide to stop giving omg.lol my money (unlikely, they deserve it!)
I'm a #gamer, and have been for as long as I can remember. I play #FinalFantasyXIV, #GuildWars2, and #WurmOnline for #MMORPG, and I rotate between a few different non MMOs. I tend to play a lot of #VisualNovel, #Mahjong on Riichi City (and hopefully some on FFXIV once cross-DC queues hit NA!), and a lot of #NSFW games. If you have any suggestions on those don't hesitate to DM me, I'm always looking for more!
For some technical hobbies, I'm into #3DPrinting, and I've dabbled in #Soldering to make my own #MechanicalKeyboard. Didn't design it, but I did put it together without a kit :D
Generally I'm very into #FOSS, and #HomeLab. I've also been a #Linux user since 2010. Over the years I've used several flavors of Ubuntu, Debian, EndeavourOS, Garuda Linux, Arch Linux, and currently a #Gentoo user. Also #Emacs is the best piece of software to ever be released.
I'm a #Writer, though I'm currently being very slow writing this novella. Hoping to turn it into a VN, but I can't draw to save my life, and it's real expensive to commission that many images. Might just do a light novel-style thing and throw an image in with each chapter.
For crafts I like #Knitting, and I can #Crochet though do that very infrequently because it hurts. I also do #TabletWeaving, though I'm fairly new at that.
That's just about everything! I do occasionally boost porn, and will probably post my own at some point. Images will be CW'd, text likely will not be, as it causes some distress figuring out what to CW -
Alright new instance new #introduction time. I should really just copy this somewhere so I don't have to keep retyping it, given how much I move.
I'm #queer, and #polyamorous who's about 30 years old at the time of writing. I do tend to hop around fediverse instances, but I'd like to stop that at some point. I liked Friendica a lot, but my last 2 instances died very suddenly, and I decided against trying a third time. My more stationary account is @[email protected], which should be up indefinitely if I don't decide to stop giving omg.lol my money (unlikely, they deserve it!)
I'm a #gamer, and have been for as long as I can remember. I play #FinalFantasyXIV, #GuildWars2, and #WurmOnline for #MMORPG, and I rotate between a few different non MMOs. I tend to play a lot of #VisualNovel, #Mahjong on Riichi City (and hopefully some on FFXIV once cross-DC queues hit NA!), and a lot of #NSFW games. If you have any suggestions on those don't hesitate to DM me, I'm always looking for more!
For some technical hobbies, I'm into #3DPrinting, and I've dabbled in #Soldering to make my own #MechanicalKeyboard. Didn't design it, but I did put it together without a kit :D
Generally I'm very into #FOSS, and #HomeLab. I've also been a #Linux user since 2010. Over the years I've used several flavors of Ubuntu, Debian, EndeavourOS, Garuda Linux, Arch Linux, and currently a #Gentoo user. Also #Emacs is the best piece of software to ever be released.
I'm a #Writer, though I'm currently being very slow writing this novella. Hoping to turn it into a VN, but I can't draw to save my life, and it's real expensive to commission that many images. Might just do a light novel-style thing and throw an image in with each chapter.
For crafts I like #Knitting, and I can #Crochet though do that very infrequently because it hurts. I also do #TabletWeaving, though I'm fairly new at that.
That's just about everything! I do occasionally boost porn, and will probably post my own at some point. Images will be CW'd, text likely will not be, as it causes some distress figuring out what to CW -
Alright new instance new #introduction time. I should really just copy this somewhere so I don't have to keep retyping it, given how much I move.
I'm #queer, and #polyamorous who's about 30 years old at the time of writing. I do tend to hop around fediverse instances, but I'd like to stop that at some point. I liked Friendica a lot, but my last 2 instances died very suddenly, and I decided against trying a third time. My more stationary account is @[email protected], which should be up indefinitely if I don't decide to stop giving omg.lol my money (unlikely, they deserve it!)
I'm a #gamer, and have been for as long as I can remember. I play #FinalFantasyXIV, #GuildWars2, and #WurmOnline for #MMORPG, and I rotate between a few different non MMOs. I tend to play a lot of #VisualNovel, #Mahjong on Riichi City (and hopefully some on FFXIV once cross-DC queues hit NA!), and a lot of #NSFW games. If you have any suggestions on those don't hesitate to DM me, I'm always looking for more!
For some technical hobbies, I'm into #3DPrinting, and I've dabbled in #Soldering to make my own #MechanicalKeyboard. Didn't design it, but I did put it together without a kit :D
Generally I'm very into #FOSS, and #HomeLab. I've also been a #Linux user since 2010. Over the years I've used several flavors of Ubuntu, Debian, EndeavourOS, Garuda Linux, Arch Linux, and currently a #Gentoo user. Also #Emacs is the best piece of software to ever be released.
I'm a #Writer, though I'm currently being very slow writing this novella. Hoping to turn it into a VN, but I can't draw to save my life, and it's real expensive to commission that many images. Might just do a light novel-style thing and throw an image in with each chapter.
For crafts I like #Knitting, and I can #Crochet though do that very infrequently because it hurts. I also do #TabletWeaving, though I'm fairly new at that.
That's just about everything! I do occasionally boost porn, and will probably post my own at some point. Images will be CW'd, text likely will not be, as it causes some distress figuring out what to CW -
Alright new instance new #introduction time. I should really just copy this somewhere so I don't have to keep retyping it, given how much I move.
I'm #queer, and #polyamorous who's about 30 years old at the time of writing. I do tend to hop around fediverse instances, but I'd like to stop that at some point. I liked Friendica a lot, but my last 2 instances died very suddenly, and I decided against trying a third time. My more stationary account is @[email protected], which should be up indefinitely if I don't decide to stop giving omg.lol my money (unlikely, they deserve it!)
I'm a #gamer, and have been for as long as I can remember. I play #FinalFantasyXIV, #GuildWars2, and #WurmOnline for #MMORPG, and I rotate between a few different non MMOs. I tend to play a lot of #VisualNovel, #Mahjong on Riichi City (and hopefully some on FFXIV once cross-DC queues hit NA!), and a lot of #NSFW games. If you have any suggestions on those don't hesitate to DM me, I'm always looking for more!
For some technical hobbies, I'm into #3DPrinting, and I've dabbled in #Soldering to make my own #MechanicalKeyboard. Didn't design it, but I did put it together without a kit :D
Generally I'm very into #FOSS, and #HomeLab. I've also been a #Linux user since 2010. Over the years I've used several flavors of Ubuntu, Debian, EndeavourOS, Garuda Linux, Arch Linux, and currently a #Gentoo user. Also #Emacs is the best piece of software to ever be released.
I'm a #Writer, though I'm currently being very slow writing this novella. Hoping to turn it into a VN, but I can't draw to save my life, and it's real expensive to commission that many images. Might just do a light novel-style thing and throw an image in with each chapter.
For crafts I like #Knitting, and I can #Crochet though do that very infrequently because it hurts. I also do #TabletWeaving, though I'm fairly new at that.
That's just about everything! I do occasionally boost porn, and will probably post my own at some point. Images will be CW'd, text likely will not be, as it causes some distress figuring out what to CW -
It's spring time, which means it's time for me to get bored with the operating system on my daily driver and start to re-investigate my options. The oncoming onslaught of age verification laws that look like they were written by badly trained AI is also helping preempt this thought, since Linux distributions will of course be forced to react in some manner.
I'm currently running Project Bluefin, which is an opinionated variation of the Universal Blue project. I'm a fan, and I'm not investigating change because of anything holding me back in the current front.
Ground rules:
- I've run Fedora for almost 2 decades. While I certainly can work in Ubuntu/derivatives, I'd prefer not to simply due to toolset familiarity.
- I know, Arch exists. I don't want to. Did Gentoo before Pentium was a thing and compile time flags actually mattered. Don't need to do that again unless some other compelling reason.
- If I'm going to go all the way in on managing the tool, I'll go all the way to Linux from Scratch. At least that way I've built the OS from ground up, so there's actual benefit to compiling everything from scratch - not imagined performance gains on i7's with 64GB ram.
- If you choose "Something Else," leave a comment with the suggestion and discuss/justify why you think I should move. I'm not against new things, but it won't replace my daily driver "just because, bro."
- Don't do the distro war thing. It was tiring on IRC in 2002, it's exhausting 25 years later. Sell your opinion on it's own merits, not by flaming other things.
Boosts welcome. It'll be a fun time.
#linux #fedora #universalblue #projectbluefin #linuxfromscratch #choices #youdidntreadthisfardidyou #maybedanedid #poll
-
It's spring time, which means it's time for me to get bored with the operating system on my daily driver and start to re-investigate my options. The oncoming onslaught of age verification laws that look like they were written by badly trained AI is also helping preempt this thought, since Linux distributions will of course be forced to react in some manner.
I'm currently running Project Bluefin, which is an opinionated variation of the Universal Blue project. I'm a fan, and I'm not investigating change because of anything holding me back in the current front.
Ground rules:
- I've run Fedora for almost 2 decades. While I certainly can work in Ubuntu/derivatives, I'd prefer not to simply due to toolset familiarity.
- I know, Arch exists. I don't want to. Did Gentoo before Pentium was a thing and compile time flags actually mattered. Don't need to do that again unless some other compelling reason.
- If I'm going to go all the way in on managing the tool, I'll go all the way to Linux from Scratch. At least that way I've built the OS from ground up, so there's actual benefit to compiling everything from scratch - not imagined performance gains on i7's with 64GB ram.
- If you choose "Something Else," leave a comment with the suggestion and discuss/justify why you think I should move. I'm not against new things, but it won't replace my daily driver "just because, bro."
- Don't do the distro war thing. It was tiring on IRC in 2002, it's exhausting 25 years later. Sell your opinion on it's own merits, not by flaming other things.
Boosts welcome. It'll be a fun time.
#linux #fedora #universalblue #projectbluefin #linuxfromscratch #choices #youdidntreadthisfardidyou #maybedanedid #poll
-
It's spring time, which means it's time for me to get bored with the operating system on my daily driver and start to re-investigate my options. The oncoming onslaught of age verification laws that look like they were written by badly trained AI is also helping preempt this thought, since Linux distributions will of course be forced to react in some manner.
I'm currently running Project Bluefin, which is an opinionated variation of the Universal Blue project. I'm a fan, and I'm not investigating change because of anything holding me back in the current front.
Ground rules:
- I've run Fedora for almost 2 decades. While I certainly can work in Ubuntu/derivatives, I'd prefer not to simply due to toolset familiarity.
- I know, Arch exists. I don't want to. Did Gentoo before Pentium was a thing and compile time flags actually mattered. Don't need to do that again unless some other compelling reason.
- If I'm going to go all the way in on managing the tool, I'll go all the way to Linux from Scratch. At least that way I've built the OS from ground up, so there's actual benefit to compiling everything from scratch - not imagined performance gains on i7's with 64GB ram.
- If you choose "Something Else," leave a comment with the suggestion and discuss/justify why you think I should move. I'm not against new things, but it won't replace my daily driver "just because, bro."
- Don't do the distro war thing. It was tiring on IRC in 2002, it's exhausting 25 years later. Sell your opinion on it's own merits, not by flaming other things.
Boosts welcome. It'll be a fun time.
#linux #fedora #universalblue #projectbluefin #linuxfromscratch #choices #youdidntreadthisfardidyou #maybedanedid #poll
-
It's spring time, which means it's time for me to get bored with the operating system on my daily driver and start to re-investigate my options. The oncoming onslaught of age verification laws that look like they were written by badly trained AI is also helping preempt this thought, since Linux distributions will of course be forced to react in some manner.
I'm currently running Project Bluefin, which is an opinionated variation of the Universal Blue project. I'm a fan, and I'm not investigating change because of anything holding me back in the current front.
Ground rules:
- I've run Fedora for almost 2 decades. While I certainly can work in Ubuntu/derivatives, I'd prefer not to simply due to toolset familiarity.
- I know, Arch exists. I don't want to. Did Gentoo before Pentium was a thing and compile time flags actually mattered. Don't need to do that again unless some other compelling reason.
- If I'm going to go all the way in on managing the tool, I'll go all the way to Linux from Scratch. At least that way I've built the OS from ground up, so there's actual benefit to compiling everything from scratch - not imagined performance gains on i7's with 64GB ram.
- If you choose "Something Else," leave a comment with the suggestion and discuss/justify why you think I should move. I'm not against new things, but it won't replace my daily driver "just because, bro."
- Don't do the distro war thing. It was tiring on IRC in 2002, it's exhausting 25 years later. Sell your opinion on it's own merits, not by flaming other things.
Boosts welcome. It'll be a fun time.
#linux #fedora #universalblue #projectbluefin #linuxfromscratch #choices #youdidntreadthisfardidyou #maybedanedid #poll
-
It's spring time, which means it's time for me to get bored with the operating system on my daily driver and start to re-investigate my options. The oncoming onslaught of age verification laws that look like they were written by badly trained AI is also helping preempt this thought, since Linux distributions will of course be forced to react in some manner.
I'm currently running Project Bluefin, which is an opinionated variation of the Universal Blue project. I'm a fan, and I'm not investigating change because of anything holding me back in the current front.
Ground rules:
- I've run Fedora for almost 2 decades. While I certainly can work in Ubuntu/derivatives, I'd prefer not to simply due to toolset familiarity.
- I know, Arch exists. I don't want to. Did Gentoo before Pentium was a thing and compile time flags actually mattered. Don't need to do that again unless some other compelling reason.
- If I'm going to go all the way in on managing the tool, I'll go all the way to Linux from Scratch. At least that way I've built the OS from ground up, so there's actual benefit to compiling everything from scratch - not imagined performance gains on i7's with 64GB ram.
- If you choose "Something Else," leave a comment with the suggestion and discuss/justify why you think I should move. I'm not against new things, but it won't replace my daily driver "just because, bro."
- Don't do the distro war thing. It was tiring on IRC in 2002, it's exhausting 25 years later. Sell your opinion on it's own merits, not by flaming other things.
Boosts welcome. It'll be a fun time.
#linux #fedora #universalblue #projectbluefin #linuxfromscratch #choices #youdidntreadthisfardidyou #maybedanedid #poll
-
if Kyber also available in Kleopatra? Thanks
IMHO below:
Who cares? It's often said that the videotape format war and the high-definition optical disc format war were won by whatever the porn industry favoured. I don't know whether this claim about video format wars is true, but I would argue that the LibrePGP vs. RFC 9580 "war" will be won by whatever is favoured by the predominant public key distribution platform.
Currently, that platform is https://keys.openpgp.org which I don't expect to support LibrePGP anytime soon, because it relies on sequoia-openpgp which I don't expect to support the RFC behind LibrePGP as long as it's just an Active Internet-Draft. At least, Sequoia-PGP supported RFC 9580 after it was released in July 2024 (link). So, why should they handle LibrePGP any differently? tbh, even if the RFC behind LibrePGP has the status of a "RFC - Proposed Standard" which I expect it to never get, I don't think Sequoia-PGP will support it.
I for once will rotate my keypair and opt for Sequoia-PGP once this Gentoo bug has been solved. My reasons:
- I expect for RFC 9580 to remain favoured on the predominant public key platform.
- I cannot rely on the GnuPG manpage (link).
-
if Kyber also available in Kleopatra? Thanks
IMHO below:
Who cares? It's often said that the videotape format war and the high-definition optical disc format war were won by whatever the porn industry favoured. I don't know whether this claim about video format wars is true, but I would argue that the LibrePGP vs. RFC 9580 "war" will be won by whatever is favoured by the predominant public key distribution platform.
Currently, that platform is https://keys.openpgp.org which I don't expect to support LibrePGP anytime soon, because it relies on sequoia-openpgp which I don't expect to support the RFC behind LibrePGP as long as it's just an Active Internet-Draft. At least, Sequoia-PGP supported RFC 9580 after it was released in July 2024 (link). So, why should they handle LibrePGP any differently? tbh, even if the RFC behind LibrePGP has the status of a "RFC - Proposed Standard" which I expect it to never get, I don't think Sequoia-PGP will support it.
I for once will rotate my keypair and opt for Sequoia-PGP once this Gentoo bug has been solved. My reasons:
- I expect for RFC 9580 to remain favoured on the predominant public key platform.
- I cannot rely on the GnuPG manpage (link).
-
if Kyber also available in Kleopatra? Thanks
IMHO below:
Who cares? It's often said that the videotape format war and the high-definition optical disc format war were won by whatever the porn industry favoured. I don't know whether this claim about video format wars is true, but I would argue that the LibrePGP vs. RFC 9580 "war" will be won by whatever is favoured by the predominant public key distribution platform.
Currently, that platform is https://keys.openpgp.org which I don't expect to support LibrePGP anytime soon, because it relies on sequoia-openpgp which I don't expect to support the RFC behind LibrePGP as long as it's just an Active Internet-Draft. At least, Sequoia-PGP supported RFC 9580 after it was released in July 2024 (link). So, why should they handle LibrePGP any differently? tbh, even if the RFC behind LibrePGP has the status of a "RFC - Proposed Standard" which I expect it to never get, I don't think Sequoia-PGP will support it.
I for once will rotate my keypair and opt for Sequoia-PGP once this Gentoo bug has been solved. My reasons:
- I expect for RFC 9580 to remain favoured on the predominant public key platform.
- I cannot rely on the GnuPG manpage (link).
-
if Kyber also available in Kleopatra? Thanks
IMHO below:
Who cares? It's often said that the videotape format war and the high-definition optical disc format war were won by whatever the porn industry favoured. I don't know whether this claim about video format wars is true, but I would argue that the LibrePGP vs. RFC 9580 "war" will be won by whatever is favoured by the predominant public key distribution platform.
Currently, that platform is https://keys.openpgp.org which I don't expect to support LibrePGP anytime soon, because it relies on sequoia-openpgp which I don't expect to support the RFC behind LibrePGP as long as it's just an Active Internet-Draft. At least, Sequoia-PGP supported RFC 9580 after it was released in July 2024 (link). So, why should they handle LibrePGP any differently? tbh, even if the RFC behind LibrePGP has the status of a "RFC - Proposed Standard" which I expect it to never get, I don't think Sequoia-PGP will support it.
I for once will rotate my keypair and opt for Sequoia-PGP once this Gentoo bug has been solved. My reasons:
- I expect for RFC 9580 to remain favoured on the predominant public key platform.
- I cannot rely on the GnuPG manpage (link).
-
if Kyber also available in Kleopatra? Thanks
IMHO below:
Who cares? It's often said that the videotape format war and the high-definition optical disc format war were won by whatever the porn industry favoured. I don't know whether this claim about video format wars is true, but I would argue that the LibrePGP vs. RFC 9580 "war" will be won by whatever is favoured by the predominant public key distribution platform.
Currently, that platform is https://keys.openpgp.org which I don't expect to support LibrePGP anytime soon, because it relies on sequoia-openpgp which I don't expect to support the RFC behind LibrePGP as long as it's just an Active Internet-Draft. At least, Sequoia-PGP supported RFC 9580 after it was released in July 2024 (link). So, why should they handle LibrePGP any differently? tbh, even if the RFC behind LibrePGP has the status of a "RFC - Proposed Standard" which I expect it to never get, I don't think Sequoia-PGP will support it.
I for once will rotate my keypair and opt for Sequoia-PGP once this Gentoo bug has been solved. My reasons:
- I expect for RFC 9580 to remain favoured on the predominant public key platform.
- I cannot rely on the GnuPG manpage (link).