Files

T

darmanandClaude Opus 4.8 6bf0eeab04 pihole: fix gravity writes, declare the blocklists

FTL could not write gravity.db, reporting "attempt to write a readonly
database". The database file was writable; the directory was not. sqlite
creates a sibling gravity.db-journal for every write transaction, so FTL
needs to CREATE files in /var/lib/pihole, and the tmpfiles rule left it
root-owned. The error names the database rather than the directory, which
sends you looking at the file and the filesystem, neither of which is at
fault.

Own the directory as 1000 instead -- the pihole user FTL drops to after
the entrypoint's root phase. Podman is rootful here with no userns
remapping, so the number is the same inside and out; on the host it
collides with darman, harmlessly.

The blocklists are now declared in this module and seeded by a oneshot,
because /var/lib/pihole is not declarative and a reflash took gravity
with it. INSERT OR IGNORE keyed on the URL is idempotent so it can run on
every boot, while the expensive rebuild only runs when gravity is empty.
Adding a list to the Nix attribute needs a manual `pihole -g` -- that is
deliberate, since the rebuild downloads every list and is slow on a Pi.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-20 21:52:01 +02:00

13 KiB

Raw Blame History

homelab

Flake-based NixOS config. Hosts: jupiter (ZimaBlade, NAS + services), neptun (netcup VPS: public reverse proxy, Authentik, headscale), mercury (Raspberry Pi 3B+, DNS/DHCP), terra (desktop).

Structure

flake.nix                  # inputs + nixosConfigurations (jupiter, neptun, kexec, ...)
common.nix                 # shared base: user, ssh, nix, firewall, timezone
services/                  # one reusable module per service, by category
  media/     jellyfin, audiobookshelf, the *arrs, sabnzbd, seerr, ...
  network/   caddy, samba, avahi, pihole, unbound
  vpn/       tailscale, headscale (control server), headplane (its web UI)
  identity/  authentik (OIDC provider, from the authentik-nix flake)
  dev/       gitea
  desktop/   hyprland
  containers.nix           # podman backend, shared across categories
hosts/
  jupiter/                 # ZimaBlade NAS
    configuration.nix      #   host bits + imports common + the services it runs
    disk-config.nix        #   disko: eMMC partitions
    hardware-configuration.nix
    secrets.nix            #   sops-nix wiring
    vm.nix                 #   VirtualBox test image (jupiter-vbox)
  neptun/                     # netcup public reverse proxy + tailnet node
    configuration.nix  disk-config.nix  hardware-configuration.nix  secrets.nix
secrets/                   # age-encrypted sops files, one per host
scripts/                   # deploy, edit_secrets

Hosts compose by importing common.nix + whichever services/* modules they run. Each service module opens its own firewall ports.

Test in VirtualBox (no hardware needed)

nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
VBoxManage import result/*.ova --vsys 0 --vmname jupiter-vbox
VBoxManage startvm jupiter-vbox --type headless

First install on the ZimaBlade — nixos-anywhere + disko

Wipes the OS disk and installs the flake over SSH. No USB needed if the box already runs Linux (ZimaOS) reachable by root SSH — nixos-anywhere kexecs into an installer, partitions via disko, installs.

⚠️ The OS disk in disk-config.nix is WIPED. Set device to the OS disk ONLY (by-id). Back up / physically identify the NAS data disk first — it must NOT appear in disko. lsblk -o NAME,SERIAL,SIZE,MODEL to identify.

Set the real OS disk id in hosts/jupiter/disk-config.nix (ls -l /dev/disk/by-id), and the data-disk mount in configuration.nix.
Add your login SSH pubkey to users.users.darman.openssh.authorizedKeys.keys.

Set the real samba password:

export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
nix shell nixpkgs#sops -c sops secrets/jupiter.yaml   # edit, commit

Stage the pre-generated host key so sops can decrypt on boot #1 (private key lives off-repo in ~/.config/homelab/jupiter/):

install -Dm600 ~/.config/homelab/jupiter/ssh_host_ed25519_key \
  /tmp/extra/etc/ssh/ssh_host_ed25519_key
install -Dm644 ~/.config/homelab/jupiter/ssh_host_ed25519_key.pub \
  /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub

Run from your laptop:
```
nix run github:nix-community/nixos-anywhere -- \
  --flake .#jupiter \
  --extra-files /tmp/extra \
  --generate-hardware-config nixos-generate-config ./hosts/jupiter/hardware-configuration.nix \
  --target-host root@<zimablade-ip>
```
--extra-files plants the host key before first boot (its age identity is already a recipient in .sops.yaml, so /run/secrets/samba_password decrypts on boot #1). --generate-hardware-config pulls the target's real kernel modules into the placeholder. Commit the result. Reboot into NixOS.

Manual alternative (USB ISO): boot installer, disko the disk, then nixos-install --flake .#jupiter.

Deploy (the `./deploy` wrapper)

All arguments mandatory — no default host, no default config.

./deploy kexec   <host>            # headless kexec into a RAM installer (RO-root box)
./deploy install <config> <host>   # first install; wipes OS disk, ships host key
./deploy switch  <config> <host>   # rebuild + activate on a running host
./deploy boot|test <config> <host> # stage for next boot / activate without boot entry
./deploy image   <config>          # build an SD-card image (mercury)
./deploy flash   <config> <dev>    # build SD image, write it, drop the sops age key

switch/boot/test prompt for darman's password (wheelNeedsPassword). <config> is a nixosConfigurations name (jupiter, neptun). Its pre-generated SSH host key lives at ~/.config/homelab/<config>/ssh_host_ed25519_key.

Examples:

./deploy switch jupiter jupiter.sol
./deploy install neptun 159.195.64.117

Rollback: nixos-rebuild switch --rollback on the host, or pick a prior generation at boot.

Post-deploy steps (per host)

Things the flake cannot do for you. Skipping these leaves a host that builds and boots but doesn't work.

Every host, immediately after a first install

ssh darman@<host> sudo -v          # DO NOT SKIP

users.mutableUsers is true, so /etc/shadow is written once, when the user is created. If the sops secret wasn't readable at that moment the account gets ! (locked) permanently — deploy switch will never fix it, because the activation script only sets a password for users not already in /etc/shadow. Combined with wheelNeedsPassword = true and PermitRootLogin = "no" that means no way to escalate, and recovery is physical: netcup's rescue system for neptun, or pulling the SD card for mercury. Verify sudo while you still have another way in.

neptun (netcup VPS)

Edge firewall. In netcup's panel, inbound ACCEPT for TCP 22/80/443/2222 and a rule accepting inbound UDP. The firewall is stateless: without the UDP rule every DNS and NTP reply is dropped, and nothing on the box reports an error — it looks like headscale crash-looping on its DERP fetch and Caddy failing ACME. grep -A1 '^Udp:' /proc/net/snmp showing InDatagrams 0 is the tell. Rules apply on VM restart, not on save. This is safe: nixos-fw is stateful and default-deny, so it remains the real policy. Also open UDP 3478 (STUN) and 41641 (tailscale direct).
Authentik creates akadmin on first start; log in at https://auth.mgaction.town with authentik_bootstrap_password from sops. The username is hardcoded upstream and the bootstrap runs once — later changes to the env vars are ignored. To use your own admin instead: create a user, add it to the authentik Admins group (superuser is a group flag in Authentik, there is no per-user one), verify it works in a private window, then deactivate akadmin — do not rename or delete it. The bootstrap blueprint keys on username: akadmin with state: created, so if no user by that name exists it simply makes a new one on the next reconcile.
Bootstrap the tailnet (headscale starts with an empty database):
```
sudo headscale users create darman
sudo headscale preauthkeys create --user darman --reusable --expiration 24h
```
Put that key in every host's sops file as tailscale_authkey and rebuild.
Headplane API key — defaults to 90d, after which headplane silently stops listing nodes:
```
sudo headscale apikeys create --expiration 999d      # -> headplane_headscale_api_key
```
Headplane OIDC. In Authentik create an OAuth2/OpenID provider (confidential, redirect https://vpn.mgaction.town/admin/oidc/callback, a signing key must be selected or discovery exposes no JWKS) and an application with slug headplane — the slug is what makes the issuer .../application/o/headplane/ in services/vpn/headplane.nix. Client ID goes in that file, client secret into sops.
Headscale OIDC (optional — pre-auth keys work without it). A second Authentik provider/application, slug headscale, redirect https://vpn.mgaction.town/oidc/callback (headscale's own, not headplane's under /admin). Client ID in services/vpn/headscale.nix, secret into sops as headscale_oidc_client_secret. ⚠️ headscale runs OIDC discovery at startup and a failure is fatal — an issuer pointing at an application that doesn't exist yet means the control server won't boot, taking the whole tailnet's control plane with it. Always verify first:
```
curl -s https://auth.mgaction.town/application/o/headscale/.well-known/openid-configuration
```
Users created by OIDC login are distinct from headscale users create ones: headplane matches the OIDC sub claim against the user's providerId, CLI-made users have none, and 0.28 dropped both map_legacy_users and node reassignment — so moving an existing node to an OIDC user means re-enrolling it.

jupiter

chown -R gitea:gitea /mnt/data/AppData/gitea after the first deploy (the repos were copied in over CIFS as darman:users).
Re-enrolling after the headscale database was recreated: tailscaled keeps its old node key and reports Running, and the autoconnect unit exits early on that state without ever sending the new pre-auth key. Force it:
```
sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect
```

mercury (Raspberry Pi 3B+)

./deploy flash mercury /dev/sdX writes the dedicated age key to the root partition. Without ~/.config/homelab/mercury/age.txt it silently skips that step and no secret decrypts on the box — check ls /run/secrets after first boot.
It boots from an SD card, so config changes are ./deploy switch mercury <ip> (an aarch64 build — needs extra-platforms + binfmt on the laptop, see the gotchas in CLAUDE.md) rather than a reflash.
Suspect the card first when binaries crash with Illegal instruction or services fail inexplicably. Failing flash returns corrupt data with no I/O errors in dmesg:
```
sudo nix-store --verify --check-contents      # add --repair to fix
```
A card that has corrupted one path will corrupt more. Replace it and reflash.
A reflash wipes /var/lib/pihole, taking the gravity database with it. The blocklists themselves are declared in services/network/pihole.nix, and the pihole-adlists unit re-seeds them on boot and rebuilds gravity when it finds it empty — so this heals itself, but the first boot after a reflash spends several minutes downloading lists. Query history and dynamic DHCP leases are genuinely lost (static leases are declarative). Check with:
```
systemctl status pihole-adlists
sudo podman exec pihole pihole-FTL sqlite3 /etc/pihole/gravity.db \
  "SELECT address,enabled FROM adlist; SELECT COUNT(*) FROM gravity;"
```
Blocked DNS queries: 0 in the pihole logs means gravity is empty — DNS resolves fine, nothing is filtered.
mercury's own resolv.conf is deliberately public resolvers, not its own pihole (resolveLocalQueries = false, see CLAUDE.md) — so .sol names do not resolve on mercury itself. That is expected, not a fault.
LAN .sol names resolve on tailnet members only because headscale sets override_local_dns = false. With upstream's default of true, every node's resolv.conf is replaced with MagicDNS and .sol returns NXDOMAIN everywhere — along with losing ad blocking, since queries stop reaching pihole at all.

Adding a service

Copy the whoami block in oci-containers.containers, swap image/ports/volumes. Native NixOS module exists for many apps (Nextcloud, Jellyfin, Grafana...) — prefer services.<app> over a container when available. Add a caddy virtualHosts block to expose it.

Notes

Backend is Podman with dockerCompat — docker CLI works, no daemon.
Samba keeps its own password DB. services.samba never sets it; a systemd oneshot (samba-smbpasswd) provisions it. Host reads the password from /run/secrets/samba_password (sops-nix); the VM falls back to plaintext /etc/samba/smb-password.
Secrets: secrets/jupiter.yaml is age-encrypted (safe to commit) to two recipients in .sops.yaml — the admin key (edit on laptop, ~/.config/sops/age/keys.txt) and the jupiter host key (derived from its SSH host key via ssh-to-age, decrypts at runtime). Private keys live off-repo and are gitignored. Rotate/add recipients with sops updatekeys.
Data disk: plain fileSystems."/mnt/data" in configuration.nix — kept out of disko so it is never formatted. Reference by by-id / by-uuid.
system.stateVersion = 26.05, install-time schema. Do NOT bump on upgrades.
Terraform is not used: a single bare-metal box has no provider API. disko + nixos-anywhere cover provisioning natively.

13 KiB Raw Blame History