darman/homelab
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
6a037d557c2779c4c1cebcc920032d04ee4f032b
homelab/README.md
T

23 KiB
Raw Blame History

homelab

Flake-based NixOS config. Hosts: jupiter (ZimaBlade, NAS + services), neptun (netcup VPS: public reverse proxy, Authentik, headscale), mercury (Raspberry Pi 3B+, DNS/DHCP), terra (desktop), mars (on-site, single-purpose: Hermes Agent only).

Structure

flake.nix                  # inputs + nixosConfigurations (jupiter, neptun, kexec, ...)
common.nix                 # shared base: user, ssh, nix, firewall, timezone
services/                  # one reusable module per service, by category
  media/     jellyfin, audiobookshelf, the *arrs, sabnzbd, seerr, ...
  network/   caddy, samba, avahi, pihole, unbound
  vpn/       tailscale, headscale (control server), headplane (its web UI)
  identity/  authentik (OIDC provider, from the authentik-nix flake)
  dev/       gitea
  desktop/   hyprland
  containers.nix           # podman backend, shared across categories
hosts/
  jupiter/                 # ZimaBlade NAS
    configuration.nix      #   host bits + imports common + the services it runs
    disk-config.nix        #   disko: eMMC partitions
    hardware-configuration.nix
    secrets.nix            #   sops-nix wiring
    vm.nix                 #   VirtualBox test image (jupiter-vbox)
  neptun/                     # netcup public reverse proxy + tailnet node
    configuration.nix  disk-config.nix  hardware-configuration.nix  secrets.nix
  mars/                     # on-site, single-purpose: Hermes Agent only
    configuration.nix  disk-config.nix  hardware-configuration.nix  secrets.nix
    hermes-agent.nix          #   Hermes Agent (moved here from jupiter)
secrets/                   # age-encrypted sops files, one per host
scripts/                   # deploy, edit_secrets

Hosts compose by importing common.nix + whichever services/* modules they run. Each service module opens its own firewall ports.

Gitea event relay

Mars includes a small HMAC-validating relay for Gitea webhooks. It forwards the authenticated request body unchanged, along with Gitea event and delivery headers, to Hermes over localhost. The relay has no event, repository, action, payload, or prompt policy; Hermes owns interpretation and response behavior. Jupiter's Gitea provisioning service registers the webhook idempotently at http://mars.orbit.sol:8645/gitea.

Before deploying either host, add the same random gitea_hermes_webhook_secret value to both secrets/mars.yaml and secrets/jupiter.yaml with sops --set. The value is intentionally not included in the repository.

Test in VirtualBox (no hardware needed)

nix build .#nixosConfigurations.jupiter-vbox.config.system.build.virtualBoxOVA
VBoxManage import result/*.ova --vsys 0 --vmname jupiter-vbox
VBoxManage startvm jupiter-vbox --type headless

Login darman / test. Forward ports with VBoxManage modifyvm ... --natpf1.

First install on the ZimaBlade — nixos-anywhere + disko

Wipes the OS disk and installs the flake over SSH. No USB needed if the box already runs Linux (ZimaOS) reachable by root SSH — nixos-anywhere kexecs into an installer, partitions via disko, installs.

⚠️ The OS disk in disk-config.nix is WIPED. Set device to the OS disk ONLY (by-id). Back up / physically identify the NAS data disk first — it must NOT appear in disko. lsblk -o NAME,SERIAL,SIZE,MODEL to identify.

  1. Set the real OS disk id in hosts/jupiter/disk-config.nix (ls -l /dev/disk/by-id), and the data-disk mount in configuration.nix.
  2. Add your login SSH pubkey to users.users.darman.openssh.authorizedKeys.keys.
  3. Set the real samba password: 
    export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
nix shell nixpkgs#sops -c sops secrets/jupiter.yaml   # edit, commit
  4. Stage the pre-generated host key so sops can decrypt on boot #1 (private key lives off-repo in ~/.config/homelab/jupiter/): 
    install -Dm600 ~/.config/homelab/jupiter/ssh_host_ed25519_key \
  /tmp/extra/etc/ssh/ssh_host_ed25519_key
install -Dm644 ~/.config/homelab/jupiter/ssh_host_ed25519_key.pub \
  /tmp/extra/etc/ssh/ssh_host_ed25519_key.pub
  5. Run from your laptop: 
    nix run github:nix-community/nixos-anywhere -- \
  --flake .#jupiter \
  --extra-files /tmp/extra \
  --generate-hardware-config nixos-generate-config ./hosts/jupiter/hardware-configuration.nix \
  --target-host root@<zimablade-ip>
    --extra-files plants the host key before first boot (its age identity is already a recipient in .sops.yaml, so /run/secrets/samba_password decrypts on boot #1). --generate-hardware-config pulls the target's real kernel modules into the placeholder. Commit the result. Reboot into NixOS.

Manual alternative (USB ISO): boot installer, disko the disk, then nixos-install --flake .#jupiter.

First install on mars

mars is an older x86_64 box (unknown provenance, "got from work"), on-site, running Hermes Agent only (see hosts/mars/hermes-agent.nix — moved there from jupiter). Its age recipient, host key (~/.config/homelab/mars/ssh_host_ed25519_key), and secrets/mars.yaml are already set up, with darman_password/samba_password/opencode_go_api_key/ telegram_bot_token/hermes_dashboard_oidc_client_secret carried over from jupiter's old instance. Two things are still placeholders and MUST be filled in before installing:

  1. OS disk id in hosts/mars/disk-config.nix (ls -l /dev/disk/by-id once you have console/installer access on the box) — same REPLACE-ME in hosts/mars/configuration.nix's comment refers to the same disk, but only disk-config.nix's device actually needs editing (grub's own device list comes from disko, see that file's comment).
  2. tailscale_authkey in secrets/mars.yaml — generate a fresh one (see "Bootstrap the tailnet" under neptun below) rather than reusing an old key; reusable pre-auth keys still expire.

Boot mode is assumed legacy BIOS (grub, not systemd-boot) — unconfirmed; check [ -d /sys/firmware/efi ] once you're at the machine and see hosts/mars/disk-config.nix's header comment if it turns out to be UEFI.

Otherwise the flow is identical to the ZimaBlade steps above:

nix run github:nix-community/nixos-anywhere -- \
  --flake .#mars \
  --extra-files /tmp/extra \
  --generate-hardware-config nixos-generate-config ./hosts/mars/hardware-configuration.nix \
  --target-host root@<mars-ip>

(stage the host key into /tmp/extra/etc/ssh/ first, same as step 4 there). Manual alternative (USB ISO): boot installer, disko the disk, then nixos-install --flake .#mars.

First install on terra — no-USB findiso reinstall (replacing CachyOS)

terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running CachyOS with a writable root and Limine as its bootloader (not systemd-boot — see step 4). Everything is already prepped in this repo: real OS-disk id in disk-config.nix, real login pubkey in common.nix, terra's age recipient in .sops.yaml, its host key pre-generated at ~/.config/homelab/terra/, and secrets/terra.yaml already holds real darman_password / tailscale_authkey values. Nothing to fill in — just run it.

⚠️ ./scripts/deploy kexec-local does NOT work on terra — do not use it. Confirmed on real hardware: the jump hangs completely (kexec's own device_shutdown() pass runs — SCSI disks sync fine — then the machine goes dark and never comes back; journalctl --list-boots showed a ~15 minute gap before the next boot, i.e. a hard hang needing a manual power cycle, not a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang during kexec's device-shutdown pass with no clean way to hand control back before the jump — same class of issue as jupiter's reboot=pci warm-reboot workaround, just fatal here instead of merely slow. The path below instead triggers a real ACPI reboot through firmware POST — a materially different code path that never runs kexec's device-shutdown pass at all.

⚠️ The OS disk (ata-KINGSTON_SA400S37480G_50026B738072F6C6) is WIPED. The dev-data disks (/mnt/hdd_01 ext4, /mnt/ssd_01 LVM) and the leftover ntfs disks are not in disko and are untouched — but double check with lsblk -o NAME,SERIAL,SIZE,MODEL before proceeding if the box's disks have changed since disk-config.nix was written.

One command does the whole thing — no need to sudo it yourself, it self-elevates:

./scripts/deploy install terra localhost

scripts/deploy detects it isn't already inside a live installer (checks uname -n) and instead:

  1. Asks where to stage the iso file — never auto-picks, because the wrong disk here is destroyed mid-install (HOMELAB_INSTALLER_STAGE_DIR skips the prompt for scripted use). It refuses if that path resolves to a disk disk-config.nix is about to wipe, if it can't work out which physical disk the path is on at all (fail-closed — LVM and RAID can span several), or if it's on btrfs (stage-1 mounts a btrfs volume's top level, so a path inside a subvolume never resolves and you boot to an emergency shell). On terra, /mnt/hdd_01 is the right answer; the CachyOS root is btrfs on the OS disk and is rejected on both counts.

  2. Prints exactly what it is about to do — OS disk, staging disk, boot entry — and waits for you to type yes. --yes skips it; that is what the ISO passes when it re-runs the command itself.

  3. Builds installer-iso's kernel + initrd + iso image, checks both target partitions have room, then copies the kernel/initrd and terra's pre-generated ssh host key to the boot partition (found via bootctl --print-boot-path, not assumed to be /boot) and the iso to the staging dir.

  4. Arms a one-shot boot of it with findiso= + homelab.install=terra + homelab.keypart=<PARTUUID> on the kernel cmdline, and reboots — a real systemctl reboot, not kexec. Two mechanisms, picked automatically:

    • systemd-boot (jupiter, neptun, and terra once NixOS is on it): a bootctl set-oneshot loader entry.
    • anything else — terra today runs Limine, which reports One-shot entry control: ✗ and has no equivalent: the firmware's own BootNext, pointing at a temporary UEFI entry that EFI-stub-boots the kernel straight off the ESP. Created with --create-only so it never enters BootOrder, which means it is reachable exactly once and nothing else changes.

    Either way the box falls back to its normal bootloader if the attempt fails — nothing is made permanent before the install succeeds. The temporary UEFI entry is deleted by the installer as soon as it boots.

The booted installer clones the repo (homelab-checkout.service, needs network — it's public now, no credentials involved) and then homelab-auto-install.service reads homelab.install=terra back off /proc/cmdline, mounts homelab.keypart= to pick terra's host key back up into /root/.config/homelab/terra/, and re-runs the exact same ./scripts/deploy install terra localhost itself — now genuinely inside the installer, so it takes the disko + nixos-install branch instead of preparing again. That key is what seeds /etc/ssh on the new system, which is what lets /run/secrets/* decrypt on boot #1; it has to travel this way because the ISO is built from a public repo and deliberately carries no credentials. The copy on the boot partition dies with the disko wipe minutes later.

The whole thing is unattended after the initial reboot; ssh into homelab-installer (same pubkey as the ISO everywhere else) to watch progress — journalctl -u homelab-checkout -u homelab-auto-install -f.

The checkout step resolves git.mgaction.town, which goes through mercury's pihole on the LAN. If mercury is down, the installer boots fine but never gets the repo — fix DNS and systemctl restart homelab-checkout.

When it's done, reboot again into the freshly installed NixOS. Then, same as any other host:

ssh darman@terra sudo -v          # DO NOT SKIP — see below

darman is created with mutableUsers = true, so /etc/shadow is written once. If the sops secret wasn't readable at that moment the account gets ! (locked) permanently and no deploy switch will fix it — verify sudo works while you still have physical console access as a fallback.

Deploy (the ./deploy wrapper)

All arguments mandatory — no default host, no default config.

./deploy kexec   <config> <host>   # headless kexec into a RAM installer (RO-root box)
./deploy install <config> <host> [--yes]   # first install; wipes OS disk, ships host key
./deploy switch  <config> <host>   # rebuild + activate on a running host
./deploy boot|test <config> <host> # stage for next boot / activate without boot entry
./deploy image   <config>          # build an SD-card image (mercury)
./deploy flash   <config> <dev>    # build SD image, write it, drop the sops age key

switch/boot/test prompt for darman's password (wheelNeedsPassword). <config> is a nixosConfigurations name (jupiter, neptun). Its pre-generated SSH host key lives at ~/.config/homelab/<config>/ssh_host_ed25519_key.

Examples:

./deploy switch jupiter jupiter.sol
./deploy install neptun 159.195.64.117

Rollback: nixos-rebuild switch --rollback on the host, or pick a prior generation at boot.

Post-deploy steps (per host)

Things the flake cannot do for you. Skipping these leaves a host that builds and boots but doesn't work.

Every host, immediately after a first install

ssh darman@<host> sudo -v          # DO NOT SKIP

users.mutableUsers is true, so /etc/shadow is written once, when the user is created. If the sops secret wasn't readable at that moment the account gets ! (locked) permanently — deploy switch will never fix it, because the activation script only sets a password for users not already in /etc/shadow. Combined with wheelNeedsPassword = true and PermitRootLogin = "no" that means no way to escalate, and recovery is physical: netcup's rescue system for neptun, or pulling the SD card for mercury. Verify sudo while you still have another way in.

neptun (netcup VPS)

  1. Edge firewall. In netcup's panel, inbound ACCEPT for TCP 22/80/443/2222 and a rule accepting inbound UDP. The firewall is stateless: without the UDP rule every DNS and NTP reply is dropped, and nothing on the box reports an error — it looks like headscale crash-looping on its DERP fetch and Caddy failing ACME. grep -A1 '^Udp:' /proc/net/snmp showing InDatagrams 0 is the tell. Rules apply on VM restart, not on save. This is safe: nixos-fw is stateful and default-deny, so it remains the real policy. Also open UDP 3478 (STUN) and 41641 (tailscale direct).
  2. Authentik creates akadmin on first start; log in at https://auth.mgaction.town with authentik_bootstrap_password from sops. The username is hardcoded upstream and the bootstrap runs once — later changes to the env vars are ignored. To use your own admin instead: create a user, add it to the authentik Admins group (superuser is a group flag in Authentik, there is no per-user one), verify it works in a private window, then deactivate akadmin — do not rename or delete it. The bootstrap blueprint keys on username: akadmin with state: created, so if no user by that name exists it simply makes a new one on the next reconcile.
  3. Bootstrap the tailnet (headscale starts with an empty database): 
    sudo headscale users create darman
sudo headscale preauthkeys create --user darman --reusable --expiration 24h
    Put that key in every host's sops file as tailscale_authkey and rebuild.
  4. Headplane API key — defaults to 90d, after which headplane silently stops listing nodes: 
    sudo headscale apikeys create --expiration 999d      # -> headplane_headscale_api_key
  5. Headplane OIDC. In Authentik create an OAuth2/OpenID provider (confidential, redirect https://vpn.mgaction.town/admin/oidc/callback, a signing key must be selected or discovery exposes no JWKS) and an application with slug headplane — the slug is what makes the issuer .../application/o/headplane/ in services/vpn/headplane.nix. Client ID goes in that file, client secret into sops.
  6. Headscale OIDC (optional — pre-auth keys work without it). A second Authentik provider/application, slug headscale, redirect https://vpn.mgaction.town/oidc/callback (headscale's own, not headplane's under /admin). Client ID in services/vpn/headscale.nix, secret into sops as headscale_oidc_client_secret. ⚠️ headscale runs OIDC discovery at startup and a failure is fatal — an issuer pointing at an application that doesn't exist yet means the control server won't boot, taking the whole tailnet's control plane with it. Always verify first: 
    curl -s https://auth.mgaction.town/application/o/headscale/.well-known/openid-configuration
    Users created by OIDC login are distinct from headscale users create ones: headplane matches the OIDC sub claim against the user's providerId, CLI-made users have none, and 0.28 dropped both map_legacy_users and node reassignment — so moving an existing node to an OIDC user means re-enrolling it.

jupiter

  • chown -R gitea:gitea /mnt/data/AppData/gitea after the first deploy (the repos were copied in over CIFS as darman:users).
  • Re-enrolling after the headscale database was recreated: tailscaled keeps its old node key and reports Running, and the autoconnect unit exits early on that state without ever sending the new pre-auth key. Force it: 
    sudo tailscale logout && sudo systemctl restart tailscaled-autoconnect

mars

  • Confirm the OIDC redirect still resolves. hermes-agent.nix reuses jupiter's old Authentik application (slug hermes, redirect https://hermes.mgaction.town/auth/callback) unchanged — nothing to reconfigure in Authentik, just verify neptun's hermes.mgaction.town vhost (now pointed at mars.orbit.sol:9119) actually reaches the dashboard once mars is up and joined the tailnet.
  • Carrying forward old chat history/memories: mars starts with a fresh Hermes state dir (/var/lib/hermes/.hermes). jupiter's old instance data is backed up at /mnt/data/AppData/hermes.bak-2026-08-21 — rsync it over (via the /mnt/jupiter samba mount) before the first switch if you want it preserved instead of starting clean.

mercury (Raspberry Pi 3B+)

  • ./deploy flash mercury /dev/sdX writes the dedicated age key to the root partition. Without ~/.config/homelab/mercury/age.txt it silently skips that step and no secret decrypts on the box — check ls /run/secrets after first boot.
  • It boots from an SD card, so config changes are ./deploy switch mercury <ip> (an aarch64 build — needs extra-platforms + binfmt on the laptop, see the gotchas in CLAUDE.md) rather than a reflash.
  • Suspect the card first when binaries crash with Illegal instruction or services fail inexplicably. Failing flash returns corrupt data with no I/O errors in dmesg: 
    sudo nix-store --verify --check-contents      # add --repair to fix
    A card that has corrupted one path will corrupt more. Replace it and reflash.
  • A reflash wipes /var/lib/pihole, taking the gravity database with it. The blocklists themselves are declared in services/network/pihole.nix, and the pihole-adlists unit re-seeds them on boot and rebuilds gravity when it finds it empty — so this heals itself, but the first boot after a reflash spends several minutes downloading lists. Query history and dynamic DHCP leases are genuinely lost (static leases are declarative). Check with: 
    systemctl status pihole-adlists
sudo podman exec pihole pihole-FTL sqlite3 /etc/pihole/gravity.db \
  "SELECT address,enabled FROM adlist; SELECT COUNT(*) FROM gravity;"
    Blocked DNS queries: 0 in the pihole logs means gravity is empty — DNS resolves fine, nothing is filtered.
  • mercury's own resolv.conf is deliberately public resolvers, not its own pihole (resolveLocalQueries = false, see CLAUDE.md) — so .sol names do not resolve on mercury itself. That is expected, not a fault.
  • mercury is load-bearing for the whole tailnet's DNS. headscale sets override_local_dns = true with pihole as the only global nameserver, so every node — including a phone on mobile data — resolves through it and gets ad blocking and .sol names anywhere. The flip side is that mercury (or the home connection) going down costs name resolution on every device, not just .sol. Recovery on a stranded device is turning Tailscale off. There is deliberately no public fallback in nameservers.global: tailscale treats that list as a set, so a second entry would let queries slip past the filter whenever mercury is slow. neptun and mercury opt out with --accept-dns=false — mercury because it would otherwise resolve through itself, neptun because a public reverse proxy must not depend on a Pi at home to renew its certificates.
  • Tailnet names are *.orbit.sol, LAN names are *.sol. Both work everywhere on the tailnet because tailscale matches DNS routes by longest suffix, so orbit.sol reaches MagicDNS even though everything else goes to pihole. Never name a LAN host orbit: pihole's address=/<host>.sol/<ip> lines match a name and everything beneath it, which would swallow the entire tailnet zone.

Adding a service

Copy the whoami block in oci-containers.containers, swap image/ports/volumes. Native NixOS module exists for many apps (Nextcloud, Jellyfin, Grafana...) — prefer services.<app> over a container when available. Add a caddy virtualHosts block to expose it.

Notes

  • Backend is Podman with dockerCompat — docker CLI works, no daemon.
  • Samba keeps its own password DB. services.samba never sets it; a systemd oneshot (samba-smbpasswd) provisions it. Host reads the password from /run/secrets/samba_password (sops-nix); the VM falls back to plaintext /etc/samba/smb-password.
  • Secrets: secrets/jupiter.yaml is age-encrypted (safe to commit) to two recipients in .sops.yaml — the admin key (edit on laptop, ~/.config/sops/age/keys.txt) and the jupiter host key (derived from its SSH host key via ssh-to-age, decrypts at runtime). Private keys live off-repo and are gitignored. Rotate/add recipients with sops updatekeys.
  • Data disk: plain fileSystems."/mnt/data" in configuration.nix — kept out of disko so it is never formatted. Reference by by-id / by-uuid.
  • system.stateVersion = 26.05, install-time schema. Do NOT bump on upgrades.
  • Terraform is not used: a single bare-metal box has no provider API. disko + nixos-anywhere cover provisioning natively.
Reference in New Issue View Git Blame Copy Permalink