Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e538788907 | ||
|
|
0ea90200b4 | ||
|
|
fd8328d7b3 | ||
|
|
2a27d2cf4b |
@@ -107,6 +107,60 @@ kept its ssh host key. Run it after ANY change to the kexec paths.
|
|||||||
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
|
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
|
||||||
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
|
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
|
||||||
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
|
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
|
||||||
|
- **terra: `./scripts/deploy kexec-local` hangs hard — do not use it there.** Confirmed
|
||||||
|
on real hardware: kexec's `device_shutdown()` pass runs (SCSI disks sync fine in the
|
||||||
|
log), then the machine goes dark and never comes back — `journalctl --list-boots`
|
||||||
|
showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power
|
||||||
|
cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known
|
||||||
|
to hang during kexec's device-shutdown pass with no clean handoff before the jump —
|
||||||
|
same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of
|
||||||
|
slow. Use `./scripts/deploy install terra localhost` instead (README's "First install
|
||||||
|
on terra" section) — it detects it isn't inside a live installer yet and reboots via
|
||||||
|
a real `systemctl reboot` + systemd-boot one-shot `findiso=` entry, not kexec.
|
||||||
|
- **`./scripts/deploy install <config> localhost`'s behavior depends on `uname -n`**
|
||||||
|
(`is_live_installer()`): on a real running OS it builds `installer-iso`, stages it
|
||||||
|
locally, and reboots into it (`local_install_prepare_and_reboot()`); only inside
|
||||||
|
`nixos-installer` (kexec) or `homelab-installer` (installer-iso) does it actually run
|
||||||
|
disko + `nixos-install`. `installer-iso`'s `homelab-auto-install.service` closes the
|
||||||
|
loop: it reads `homelab.install=<config>` back off `/proc/cmdline` (set by the prepare
|
||||||
|
step) and re-runs the identical command itself once `homelab-checkout.service` has
|
||||||
|
cloned the repo — the whole reinstall is one command and unattended after the first
|
||||||
|
reboot. It confirms (type `yes`) before rebooting, like `flash`/`kexec-local`; `--yes`
|
||||||
|
skips that and is what the ISO passes itself. It always ASKS where to stage the iso
|
||||||
|
file (never auto-picks — the wrong disk here is destroyed mid-install);
|
||||||
|
`HOMELAB_INSTALLER_STAGE_DIR` skips the prompt for scripted use. Both this and
|
||||||
|
`kexec-local` self-elevate via `sudo` (`require_root()`) rather than requiring you to
|
||||||
|
prefix the command yourself.
|
||||||
|
- **The staging-dir guard must fail CLOSED, and `findmnt` needs `--nofsroot`**: on btrfs
|
||||||
|
`findmnt -no SOURCE` prints `/dev/sdb2[/@]`, which `lsblk` cannot open, so a naive
|
||||||
|
parent-device lookup comes back empty. Treating empty as "different disk" silently
|
||||||
|
allowed staging the iso on the very disk disko then wiped — terra's CachyOS root is
|
||||||
|
exactly that layout, so it hit the live case. `disks_backing()` (`lsblk -rnso
|
||||||
|
NAME,TYPE`) returns EVERY whole-disk ancestor because LVM/RAID can span several
|
||||||
|
(`/mnt/ssd_01` → sdd + sde), and an empty result is a hard error, not a pass. btrfs
|
||||||
|
staging is refused outright: stage-1 mounts a btrfs volume's TOP level, so an iso
|
||||||
|
inside a subvolume is unreachable via `findiso=`.
|
||||||
|
- **`findiso=` must keep its leading slash**: stage-1 tests `[ -e "/findiso$isoPath" ]`,
|
||||||
|
so stripping the mountpoint prefix off a stagedir whose mountpoint is `/` yields
|
||||||
|
`var/tmp/x.iso` → `/findisovar/tmp/x.iso` → emergency shell, after you have already
|
||||||
|
rebooted out of the working OS.
|
||||||
|
- **The auto-install needs the host key shipped to it, and a `$HOME`**: the ISO is built
|
||||||
|
from a public repo with no credentials, so `local_install_prepare_and_reboot()` copies
|
||||||
|
the key onto the boot partition and passes that partition's PARTUUID as
|
||||||
|
`homelab.keypart=`; the service mounts it and drops the key in
|
||||||
|
`/root/.config/homelab/<config>/` before running the install. Also, systemd does NOT
|
||||||
|
set `$HOME` for a system service without `User=` (`SetLoginEnvironment=` defaults to
|
||||||
|
false), and `scripts/deploy` runs under `set -u` — hence `$KEYDIR` instead of a bare
|
||||||
|
`$HOME`, plus `Environment=HOME=/root` on the unit.
|
||||||
|
- **`installer-iso` must enable `experimental-features` itself.** `installation-cd-minimal`
|
||||||
|
leaves them unset, so `nix run` and `nixos-install --flake` both die with "experimental
|
||||||
|
Nix feature 'nix-command' is disabled". The nixos-images `kexec` installer sets them
|
||||||
|
itself, which is why the same `install <config> localhost` branch worked after
|
||||||
|
`kexec-local` but not from the ISO.
|
||||||
|
- **disko/nixos-anywhere run as `nix run .#disko` / `.#nixos-anywhere`**, from this
|
||||||
|
flake's locked inputs — not `nix run github:...`. They execute while a disk is being
|
||||||
|
wiped, so the revision must be the reviewed one in `flake.lock`, and it has to resolve
|
||||||
|
without network.
|
||||||
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
|
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
|
||||||
plain `fileSystems` in `configuration.nix`.
|
plain `fileSystems` in `configuration.nix`.
|
||||||
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
|
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
|
||||||
|
|||||||
@@ -84,15 +84,27 @@ an installer, partitions via disko, installs.
|
|||||||
Manual alternative (USB ISO): boot installer, `disko` the disk, then
|
Manual alternative (USB ISO): boot installer, `disko` the disk, then
|
||||||
`nixos-install --flake .#jupiter`.
|
`nixos-install --flake .#jupiter`.
|
||||||
|
|
||||||
## First install on terra — in-place kexec (replacing CachyOS)
|
## First install on terra — no-USB findiso reinstall (replacing CachyOS)
|
||||||
|
|
||||||
terra is the desktop you're typing on, currently running CachyOS with a
|
terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running
|
||||||
writable root — no ZimaOS-style read-only-root problem, no second machine
|
CachyOS with a writable root and systemd-boot. Everything is already prepped
|
||||||
needed. Everything is already prepped in this repo: real OS-disk id in
|
in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in
|
||||||
`disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in
|
`common.nix`, terra's age recipient in `.sops.yaml`, its host key
|
||||||
`.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and
|
pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already
|
||||||
`secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey`
|
holds real `darman_password` / `tailscale_authkey` values. Nothing to fill
|
||||||
values. Nothing to fill in — just run it.
|
in — just run it.
|
||||||
|
|
||||||
|
> ⚠️ **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.**
|
||||||
|
> Confirmed on real hardware: the jump hangs completely (kexec's own
|
||||||
|
> `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes
|
||||||
|
> dark and never comes back; `journalctl --list-boots` showed a **~15 minute**
|
||||||
|
> gap before the next boot, i.e. a hard hang needing a manual power cycle, not
|
||||||
|
> a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang
|
||||||
|
> during kexec's device-shutdown pass with no clean way to hand control back
|
||||||
|
> before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot
|
||||||
|
> workaround, just fatal here instead of merely slow. The path below instead
|
||||||
|
> triggers a real ACPI reboot through firmware POST — a materially different
|
||||||
|
> code path that never runs kexec's device-shutdown pass at all.
|
||||||
|
|
||||||
> ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
|
> ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
|
||||||
> dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
|
> dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
|
||||||
@@ -100,23 +112,58 @@ values. Nothing to fill in — just run it.
|
|||||||
> `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
|
> `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
|
||||||
> changed since `disk-config.nix` was written.
|
> changed since `disk-config.nix` was written.
|
||||||
|
|
||||||
1. From a root shell on terra itself:
|
One command does the whole thing — no need to `sudo` it yourself, it
|
||||||
```
|
self-elevates:
|
||||||
sudo ./scripts/deploy kexec-local --yes
|
|
||||||
```
|
|
||||||
Stages a RAM installer and kexecs into it. The console drops for ~1-2 min
|
|
||||||
then comes back logged in as `nixos-installer` — same ssh host key, so
|
|
||||||
`known_hosts` still matches if you're watching over ssh instead of the
|
|
||||||
physical console.
|
|
||||||
2. Still targeting terra (now `localhost`/`127.0.0.1` from the installer's own
|
|
||||||
shell):
|
|
||||||
```
|
```
|
||||||
./scripts/deploy install terra localhost
|
./scripts/deploy install terra localhost
|
||||||
```
|
```
|
||||||
`localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko + `nixos-install`
|
`scripts/deploy` detects it isn't already inside a live installer (checks
|
||||||
directly against `/mnt`. Ships terra's pre-generated host key so
|
`uname -n`) and instead:
|
||||||
`/run/secrets/*` decrypts on boot #1.
|
|
||||||
3. Reboot into NixOS. Then, same as any other host:
|
1. Asks where to stage the iso file — never auto-picks, because the wrong disk
|
||||||
|
here is destroyed mid-install (`HOMELAB_INSTALLER_STAGE_DIR` skips the
|
||||||
|
prompt for scripted use). It **refuses** if that path resolves to a disk
|
||||||
|
`disk-config.nix` is about to wipe, if it can't work out which physical disk
|
||||||
|
the path is on at all (fail-closed — LVM and RAID can span several), or if
|
||||||
|
it's on btrfs (stage-1 mounts a btrfs volume's *top level*, so a path inside
|
||||||
|
a subvolume never resolves and you boot to an emergency shell). On terra,
|
||||||
|
`/mnt/hdd_01` is the right answer; the CachyOS root is btrfs on the OS disk
|
||||||
|
and is rejected on both counts.
|
||||||
|
2. Prints exactly what it is about to do — OS disk, staging disk, boot entry —
|
||||||
|
and waits for you to type `yes`. `--yes` skips it; that is what the ISO
|
||||||
|
passes when it re-runs the command itself.
|
||||||
|
3. Builds `installer-iso`'s kernel + initrd + iso image, checks both target
|
||||||
|
partitions have room, then copies the kernel/initrd **and terra's
|
||||||
|
pre-generated ssh host key** to the boot partition (found via
|
||||||
|
`bootctl --print-boot-path`, not assumed to be `/boot`) and the iso to the
|
||||||
|
staging dir.
|
||||||
|
4. Writes a systemd-boot **one-shot** entry (`bootctl set-oneshot` — reverts to
|
||||||
|
the normal default after this one boot, even if it fails) with `findiso=` +
|
||||||
|
`homelab.install=terra` + `homelab.keypart=<PARTUUID>` on its kernel
|
||||||
|
cmdline, and reboots — a real `systemctl reboot`, not kexec.
|
||||||
|
|
||||||
|
The booted installer clones the repo (`homelab-checkout.service`, needs
|
||||||
|
network — it's public now, no credentials involved) and then
|
||||||
|
`homelab-auto-install.service` reads `homelab.install=terra` back off
|
||||||
|
`/proc/cmdline`, mounts `homelab.keypart=` to pick terra's host key back up
|
||||||
|
into `/root/.config/homelab/terra/`, and re-runs the exact same
|
||||||
|
`./scripts/deploy install terra localhost` itself — now genuinely inside the
|
||||||
|
installer, so it takes the disko + `nixos-install` branch instead of preparing
|
||||||
|
again. That key is what seeds `/etc/ssh` on the new system, which is what lets
|
||||||
|
`/run/secrets/*` decrypt on boot #1; it has to travel this way because the ISO
|
||||||
|
is built from a **public** repo and deliberately carries no credentials. The
|
||||||
|
copy on the boot partition dies with the disko wipe minutes later.
|
||||||
|
|
||||||
|
The whole thing is unattended after the initial reboot; ssh into
|
||||||
|
`homelab-installer` (same pubkey as the ISO everywhere else) to watch
|
||||||
|
progress — `journalctl -u homelab-checkout -u homelab-auto-install -f`.
|
||||||
|
|
||||||
|
> The checkout step resolves `git.mgaction.town`, which goes through mercury's
|
||||||
|
> pihole on the LAN. If mercury is down, the installer boots fine but never
|
||||||
|
> gets the repo — fix DNS and `systemctl restart homelab-checkout`.
|
||||||
|
|
||||||
|
When it's done, reboot again into the freshly installed NixOS. Then, same as
|
||||||
|
any other host:
|
||||||
```
|
```
|
||||||
ssh darman@terra sudo -v # DO NOT SKIP — see below
|
ssh darman@terra sudo -v # DO NOT SKIP — see below
|
||||||
```
|
```
|
||||||
@@ -132,7 +179,7 @@ All arguments mandatory — no default host, no default config.
|
|||||||
|
|
||||||
```
|
```
|
||||||
./deploy kexec <config> <host> # headless kexec into a RAM installer (RO-root box)
|
./deploy kexec <config> <host> # headless kexec into a RAM installer (RO-root box)
|
||||||
./deploy install <config> <host> # first install; wipes OS disk, ships host key
|
./deploy install <config> <host> [--yes] # first install; wipes OS disk, ships host key
|
||||||
./deploy switch <config> <host> # rebuild + activate on a running host
|
./deploy switch <config> <host> # rebuild + activate on a running host
|
||||||
./deploy boot|test <config> <host> # stage for next boot / activate without boot entry
|
./deploy boot|test <config> <host> # stage for next boot / activate without boot entry
|
||||||
./deploy image <config> # build an SD-card image (mercury)
|
./deploy image <config> # build an SD-card image (mercury)
|
||||||
|
|||||||
Generated
+77
-18
@@ -179,9 +179,63 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"nix-vm-test": {
|
||||||
|
"inputs": {
|
||||||
|
"nixpkgs": [
|
||||||
|
"nixos-anywhere",
|
||||||
|
"nixpkgs"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1783265394,
|
||||||
|
"narHash": "sha256-cq4YfNFGYzp0NItZP8tEC7xUI8OSgY4fj75AU/NSaPM=",
|
||||||
|
"owner": "numtide",
|
||||||
|
"repo": "nix-vm-test",
|
||||||
|
"rev": "1a587212d2ac8b669c6c32499015f996506b6ba5",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "numtide",
|
||||||
|
"repo": "nix-vm-test",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"nixos-anywhere": {
|
||||||
|
"inputs": {
|
||||||
|
"disko": [
|
||||||
|
"disko"
|
||||||
|
],
|
||||||
|
"nix-vm-test": "nix-vm-test",
|
||||||
|
"nixos-images": [
|
||||||
|
"nixos-images"
|
||||||
|
],
|
||||||
|
"nixos-stable": [
|
||||||
|
"nixpkgs"
|
||||||
|
],
|
||||||
|
"nixpkgs": [
|
||||||
|
"nixpkgs"
|
||||||
|
],
|
||||||
|
"treefmt-nix": "treefmt-nix"
|
||||||
|
},
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1784704708,
|
||||||
|
"narHash": "sha256-xuzg2Xfalr1X5ah5IPgqB8Vyif+2je1qkCtGxCB+rm0=",
|
||||||
|
"owner": "nix-community",
|
||||||
|
"repo": "nixos-anywhere",
|
||||||
|
"rev": "91fc9b70fc295258c366cce8627efb6f185fd9fb",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "nix-community",
|
||||||
|
"repo": "nixos-anywhere",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
"nixos-images": {
|
"nixos-images": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"nixos-stable": "nixos-stable",
|
"nixos-stable": [
|
||||||
|
"nixpkgs"
|
||||||
|
],
|
||||||
"nixos-unstable": "nixos-unstable"
|
"nixos-unstable": "nixos-unstable"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
@@ -198,23 +252,6 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"nixos-stable": {
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1783389287,
|
|
||||||
"narHash": "sha256-0xIy4dVLqq47rA+mRy0hXDfjhQd4E5PoIns/RmB7nR4=",
|
|
||||||
"ref": "nixos-26.05",
|
|
||||||
"rev": "0ad6f47ea4fe188f4bc8f0380f93ae8523337c6c",
|
|
||||||
"shallow": true,
|
|
||||||
"type": "git",
|
|
||||||
"url": "https://github.com/NixOS/nixpkgs"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"ref": "nixos-26.05",
|
|
||||||
"shallow": true,
|
|
||||||
"type": "git",
|
|
||||||
"url": "https://github.com/NixOS/nixpkgs"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nixos-unstable": {
|
"nixos-unstable": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1782175435,
|
"lastModified": 1782175435,
|
||||||
@@ -421,6 +458,7 @@
|
|||||||
"disko": "disko",
|
"disko": "disko",
|
||||||
"home-manager": "home-manager",
|
"home-manager": "home-manager",
|
||||||
"mediamanager-nix": "mediamanager-nix",
|
"mediamanager-nix": "mediamanager-nix",
|
||||||
|
"nixos-anywhere": "nixos-anywhere",
|
||||||
"nixos-images": "nixos-images",
|
"nixos-images": "nixos-images",
|
||||||
"nixpkgs": "nixpkgs_2",
|
"nixpkgs": "nixpkgs_2",
|
||||||
"nixpkgs-unstable": "nixpkgs-unstable",
|
"nixpkgs-unstable": "nixpkgs-unstable",
|
||||||
@@ -480,6 +518,27 @@
|
|||||||
"url": "ssh://gitea@git.mgaction.town:2222/darman/TOME.git"
|
"url": "ssh://gitea@git.mgaction.town:2222/darman/TOME.git"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"treefmt-nix": {
|
||||||
|
"inputs": {
|
||||||
|
"nixpkgs": [
|
||||||
|
"nixos-anywhere",
|
||||||
|
"nixpkgs"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1784369104,
|
||||||
|
"narHash": "sha256-47cxbcZODibHv3rELFQ9vZly0vUNkND/atn/U7HLeb0=",
|
||||||
|
"owner": "numtide",
|
||||||
|
"repo": "treefmt-nix",
|
||||||
|
"rev": "df3c0640565d04a0261253cdd89fce78ec50168a",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "numtide",
|
||||||
|
"repo": "treefmt-nix",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
"uv2nix": {
|
"uv2nix": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"nixpkgs": [
|
"nixpkgs": [
|
||||||
|
|||||||
@@ -16,9 +16,24 @@
|
|||||||
url = "github:Mic92/sops-nix";
|
url = "github:Mic92/sops-nix";
|
||||||
inputs.nixpkgs.follows = "nixpkgs";
|
inputs.nixpkgs.follows = "nixpkgs";
|
||||||
};
|
};
|
||||||
|
# NB: no `inputs.nixpkgs.follows` here — nixos-images has no `nixpkgs`
|
||||||
|
# input (it takes nixos-stable / nixos-unstable), so declaring one only
|
||||||
|
# printed "override for a non-existent input" on every nix command.
|
||||||
nixos-images = {
|
nixos-images = {
|
||||||
url = "github:nix-community/nixos-images";
|
url = "github:nix-community/nixos-images";
|
||||||
|
inputs.nixos-stable.follows = "nixpkgs";
|
||||||
|
};
|
||||||
|
# Pinned as an input rather than `nix run github:...` from scripts/deploy.
|
||||||
|
# Both it and disko run at the exact moment a disk is being wiped, so the
|
||||||
|
# revision has to come from flake.lock — reviewed, reproducible, and
|
||||||
|
# resolvable from the local store — instead of whatever upstream master
|
||||||
|
# happens to be that day (which also fails outright with no network).
|
||||||
|
nixos-anywhere = {
|
||||||
|
url = "github:nix-community/nixos-anywhere";
|
||||||
inputs.nixpkgs.follows = "nixpkgs";
|
inputs.nixpkgs.follows = "nixpkgs";
|
||||||
|
inputs.nixos-stable.follows = "nixpkgs"; # 26.05 already IS stable
|
||||||
|
inputs.disko.follows = "disko";
|
||||||
|
inputs.nixos-images.follows = "nixos-images";
|
||||||
};
|
};
|
||||||
home-manager = {
|
home-manager = {
|
||||||
url = "github:nix-community/home-manager/release-26.05";
|
url = "github:nix-community/home-manager/release-26.05";
|
||||||
@@ -48,15 +63,23 @@
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
outputs = { self, nixpkgs, disko, sops-nix, nixos-images, home-manager, mediamanager-nix, authentik-nix, ... }@inputs:
|
outputs = { self, nixpkgs, disko, nixos-anywhere, sops-nix, nixos-images, home-manager, mediamanager-nix, authentik-nix, ... }@inputs:
|
||||||
let
|
let
|
||||||
system = "x86_64-linux";
|
system = "x86_64-linux";
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
packages.${system}.tome = nixpkgs.legacyPackages.${system}.callPackage ./pkgs/tome.nix {
|
packages.${system} = {
|
||||||
|
tome = nixpkgs.legacyPackages.${system}.callPackage ./pkgs/tome.nix {
|
||||||
src = inputs.tome;
|
src = inputs.tome;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# Re-exported so `./scripts/deploy` can run them as `nix run .#disko` /
|
||||||
|
# `nix run .#nixos-anywhere`, at the revision flake.lock pins. See the
|
||||||
|
# nixos-anywhere input above for why that matters.
|
||||||
|
disko = disko.packages.${system}.disko;
|
||||||
|
nixos-anywhere = nixos-anywhere.packages.${system}.nixos-anywhere;
|
||||||
|
};
|
||||||
|
|
||||||
nixosConfigurations = {
|
nixosConfigurations = {
|
||||||
# Real host — install on the ZimaBlade.
|
# Real host — install on the ZimaBlade.
|
||||||
# disko owns the OS-disk partitioning + filesystems (see disk-config.nix).
|
# disko owns the OS-disk partitioning + filesystems (see disk-config.nix).
|
||||||
@@ -162,21 +185,120 @@
|
|||||||
];
|
];
|
||||||
};
|
};
|
||||||
|
|
||||||
# Bootable USB recovery installer with our SSH key + sshd + DHCP.
|
# Bootable USB recovery installer with our SSH key + sshd + DHCP. Clones
|
||||||
|
# the (now public) homelab repo fresh at every boot to /root/homelab —
|
||||||
|
# always current master, so the same USB stick stays useful across
|
||||||
|
# install/rescue occasions without ever needing a rebuild. No
|
||||||
|
# rsync/copy-the-repo-over step: boot it, ssh in,
|
||||||
|
# `cd /root/homelab && ./scripts/deploy install ...`.
|
||||||
|
# Reusable for any host's manual-USB install path (jupiter, terra, ...).
|
||||||
# Build the ISO:
|
# Build the ISO:
|
||||||
# nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
|
# nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
|
||||||
# dd it to a USB stick, boot the ZimaBlade from it, SSH in, ./deploy install.
|
# dd it to a USB stick, boot the target from it, SSH in, ./deploy install.
|
||||||
installer-iso = nixpkgs.lib.nixosSystem {
|
installer-iso = nixpkgs.lib.nixosSystem {
|
||||||
inherit system;
|
inherit system;
|
||||||
modules = [
|
modules = [
|
||||||
(nixpkgs + "/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix")
|
(nixpkgs + "/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix")
|
||||||
({ ... }: {
|
({ pkgs, ... }: {
|
||||||
services.openssh.enable = true;
|
services.openssh.enable = true;
|
||||||
services.openssh.settings.PermitRootLogin = "prohibit-password";
|
services.openssh.settings.PermitRootLogin = "prohibit-password";
|
||||||
users.users.root.openssh.authorizedKeys.keys = [
|
users.users.root.openssh.authorizedKeys.keys = [
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZpkPVhzi1zG5JI9hWyUgdyvNIQbp4ts4jw3idpMhhN erik@laptop"
|
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZpkPVhzi1zG5JI9hWyUgdyvNIQbp4ts4jw3idpMhhN erik@laptop"
|
||||||
];
|
];
|
||||||
networking.hostName = "jupiter-installer";
|
networking.hostName = "homelab-installer";
|
||||||
|
environment.systemPackages = [ pkgs.git ];
|
||||||
|
|
||||||
|
# installation-cd-minimal leaves experimental-features unset, so
|
||||||
|
# the ISO's nix.conf has no `nix-command`/`flakes` at all (unlike
|
||||||
|
# the nixos-images kexec installer, which sets
|
||||||
|
# extra-experimental-features itself — which is why the same
|
||||||
|
# `install <config> localhost` branch works after kexec-local but
|
||||||
|
# not here). Without this, both `nix run .#disko` and
|
||||||
|
# `nixos-install --flake` die with "experimental Nix feature
|
||||||
|
# 'nix-command' is disabled".
|
||||||
|
nix.settings.experimental-features = [ "nix-command" "flakes" ];
|
||||||
|
|
||||||
|
# Fresh clone of a PUBLIC repo — no credentials baked into the
|
||||||
|
# ISO. require_tracked() in scripts/deploy still works fine here
|
||||||
|
# (this IS a real git checkout, unlike the old baked-`self`
|
||||||
|
# approach), but retry manually with `systemctl restart
|
||||||
|
# homelab-checkout` if DHCP was still coming up at boot.
|
||||||
|
systemd.services.homelab-checkout = {
|
||||||
|
description = "Clone the homelab repo to /root/homelab";
|
||||||
|
after = [ "network-online.target" ];
|
||||||
|
wants = [ "network-online.target" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
path = [ pkgs.git ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot";
|
||||||
|
RemainAfterExit = true;
|
||||||
|
};
|
||||||
|
script = ''
|
||||||
|
rm -rf /root/homelab
|
||||||
|
git clone --depth 1 https://git.mgaction.town/darman/homelab.git /root/homelab
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
# Finishes a local_install_prepare_and_reboot() run (scripts/deploy)
|
||||||
|
# unattended: that function stages this ISO, points a systemd-boot
|
||||||
|
# one-shot entry at it with `homelab.install=<config>` on the kernel
|
||||||
|
# cmdline, and reboots. Once booted here, this re-runs the exact same
|
||||||
|
# `./scripts/deploy install <config> localhost` command — now genuinely
|
||||||
|
# inside the installer (hostname homelab-installer), so is_live_installer
|
||||||
|
# takes the disko+nixos-install branch instead of preparing again.
|
||||||
|
# A manual boot of this ISO with no such cmdline param is a no-op.
|
||||||
|
systemd.services.homelab-auto-install = {
|
||||||
|
description = "Auto-run the homelab install if homelab.install= was passed on the kernel cmdline";
|
||||||
|
after = [ "homelab-checkout.service" ];
|
||||||
|
requires = [ "homelab-checkout.service" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
path = [ pkgs.gnugrep pkgs.util-linux pkgs.coreutils pkgs.git ];
|
||||||
|
serviceConfig.Type = "oneshot";
|
||||||
|
# systemd does NOT set $HOME for a system service without User=
|
||||||
|
# (systemd.exec(5): SetLoginEnvironment= "defaults to true if
|
||||||
|
# User=, DynamicUser= or PAMName= are set, false otherwise"), and
|
||||||
|
# scripts/deploy runs under `set -u`. Without this the whole
|
||||||
|
# unattended run died on the bare $HOME expansion with an
|
||||||
|
# "unbound variable" that reads like a bug in the script.
|
||||||
|
serviceConfig.Environment = "HOME=/root";
|
||||||
|
script = ''
|
||||||
|
cfg=$(grep -o 'homelab\.install=[^ ]*' /proc/cmdline | cut -d= -f2 || true)
|
||||||
|
if [ -z "$cfg" ]; then
|
||||||
|
echo "no homelab.install= on the kernel cmdline — nothing to auto-install"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The host key scripts/deploy seeds /etc/ssh with (so sops can
|
||||||
|
# decrypt on boot #1) cannot live in this ISO: it is built from
|
||||||
|
# a PUBLIC repo and the private keys are deliberately off-repo.
|
||||||
|
# local_install_prepare_and_reboot() therefore drops it on the
|
||||||
|
# boot partition and passes that partition's PARTUUID here.
|
||||||
|
# That copy dies with the disko wipe a few minutes later.
|
||||||
|
keypart=$(grep -o 'homelab\.keypart=[^ ]*' /proc/cmdline | cut -d= -f2 || true)
|
||||||
|
if [ -n "$keypart" ]; then
|
||||||
|
mkdir -p /run/homelab-key
|
||||||
|
if mount -o ro "/dev/disk/by-partuuid/$keypart" /run/homelab-key; then
|
||||||
|
src=/run/homelab-key/homelab-installer
|
||||||
|
if [ -f "$src/ssh_host_ed25519_key" ]; then
|
||||||
|
echo "picking up $cfg's host key from PARTUUID=$keypart"
|
||||||
|
install -Dm600 "$src/ssh_host_ed25519_key" \
|
||||||
|
"/root/.config/homelab/$cfg/ssh_host_ed25519_key"
|
||||||
|
install -Dm644 "$src/ssh_host_ed25519_key.pub" \
|
||||||
|
"/root/.config/homelab/$cfg/ssh_host_ed25519_key.pub"
|
||||||
|
else
|
||||||
|
echo "warning: no host key at $src — the install will refuse" >&2
|
||||||
|
fi
|
||||||
|
umount /run/homelab-key
|
||||||
|
else
|
||||||
|
echo "warning: could not mount PARTUUID=$keypart for the host key" >&2
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "auto-installing $cfg (homelab.install= on the kernel cmdline)"
|
||||||
|
cd /root/homelab
|
||||||
|
exec ./scripts/deploy install "$cfg" localhost --yes
|
||||||
|
'';
|
||||||
|
};
|
||||||
})
|
})
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
|
|||||||
+294
-17
@@ -17,11 +17,32 @@
|
|||||||
# /var/tmp) must be exec-capable and hold
|
# /var/tmp) must be exec-capable and hold
|
||||||
# ~3x the tarball.
|
# ~3x the tarball.
|
||||||
# Then run `install <config> localhost`.
|
# Then run `install <config> localhost`.
|
||||||
# ./deploy install <config> <host> first install. Wipes the OS disk. Ships the
|
# ./deploy install <config> <host> [--yes]
|
||||||
|
# first install. Wipes the OS disk. Ships the
|
||||||
# host's sops key. <host>=localhost/127.0.0.1
|
# host's sops key. <host>=localhost/127.0.0.1
|
||||||
# skips nixos-anywhere/ssh and runs disko +
|
# skips nixos-anywhere/ssh and runs disko +
|
||||||
# nixos-install directly against /mnt (use
|
# nixos-install directly against /mnt — but
|
||||||
# after `kexec-local`, or on a live ISO).
|
# ONLY once actually inside a live installer
|
||||||
|
# (hostname nixos-installer, from kexec, or
|
||||||
|
# homelab-installer, from installer-iso).
|
||||||
|
# Run from the REAL running OS instead (e.g.
|
||||||
|
# a box where kexec-local doesn't work),
|
||||||
|
# it builds installer-iso, stages its
|
||||||
|
# kernel/initrd + the host key on the boot
|
||||||
|
# partition and the iso file on a non-OS-disk
|
||||||
|
# partition, sets a systemd-boot one-shot
|
||||||
|
# entry with homelab.install=<config> +
|
||||||
|
# homelab.keypart=<PARTUUID> on its kernel
|
||||||
|
# cmdline, and reboots — a real ACPI reboot,
|
||||||
|
# not a kexec jump. The booted installer's
|
||||||
|
# homelab-auto-install.service reads those
|
||||||
|
# cmdline params, picks the host key back up
|
||||||
|
# and re-runs this exact command itself once
|
||||||
|
# its repo checkout (homelab-checkout.service)
|
||||||
|
# succeeds, finishing the install unattended.
|
||||||
|
# It confirms before rebooting; --yes skips
|
||||||
|
# that (it is what the ISO passes itself).
|
||||||
|
# See CLAUDE.md.
|
||||||
# ./deploy switch <config> <host> rebuild + activate on a running host.
|
# ./deploy switch <config> <host> rebuild + activate on a running host.
|
||||||
# ./deploy boot <config> <host> stage for next boot, don't activate now.
|
# ./deploy boot <config> <host> stage for next boot, don't activate now.
|
||||||
# ./deploy test <config> <host> activate without adding a boot entry.
|
# ./deploy test <config> <host> activate without adding a boot entry.
|
||||||
@@ -46,16 +67,49 @@
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
shopt -s nullglob
|
shopt -s nullglob
|
||||||
|
|
||||||
|
# Captured before anything shifts/parses $@, so require_root() below can
|
||||||
|
# re-exec the ORIGINAL invocation under sudo — inside a function, "$@"/"$1"
|
||||||
|
# refer to the function's own args (empty here), not the script's, so this
|
||||||
|
# has to be a global array instead of relying on positional-parameter scoping.
|
||||||
|
SCRIPT_ARGS=("$@")
|
||||||
|
|
||||||
# Locate the repo root (flake dir) regardless of where this script lives on disk.
|
# Locate the repo root (flake dir) regardless of where this script lives on disk.
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$(realpath "$0")")" && pwd)"
|
SCRIPT_PATH="$(realpath "$0")" # absolute — "$0" itself may be relative,
|
||||||
|
# and require_root() re-execs after cd "$REPO"
|
||||||
|
SCRIPT_DIR="$(dirname "$SCRIPT_PATH")"
|
||||||
REPO="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || dirname "$SCRIPT_DIR")"
|
REPO="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || dirname "$SCRIPT_DIR")"
|
||||||
cd "$REPO"
|
cd "$REPO"
|
||||||
export PATH="/nix/var/nix/profiles/default/bin:$PATH"
|
export PATH="/nix/var/nix/profiles/default/bin:$PATH"
|
||||||
|
|
||||||
|
# Off-repo material keyed by <config>: pre-generated SSH host keys (install)
|
||||||
|
# and per-config sops age keys (flash).
|
||||||
|
#
|
||||||
|
# Resolved defensively rather than as a bare $HOME, because this script also
|
||||||
|
# runs from installer-iso's homelab-auto-install.service, and systemd does not
|
||||||
|
# set $HOME for a system service without User= (systemd.exec(5):
|
||||||
|
# SetLoginEnvironment= "defaults to true if User=, DynamicUser= or PAMName= are
|
||||||
|
# set, false otherwise"). Under `set -u` that aborted the whole unattended run
|
||||||
|
# with an "unbound variable" that read like a bug in this script.
|
||||||
|
KEYDIR="${HOMELAB_KEY_DIR:-${HOME:-/root}/.config/homelab}"
|
||||||
|
|
||||||
die() { echo "error: $*" >&2; exit 1; }
|
die() { echo "error: $*" >&2; exit 1; }
|
||||||
|
|
||||||
need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; }
|
need() { command -v "$1" >/dev/null 2>&1 || die "missing required tool: $1"; }
|
||||||
|
|
||||||
|
# Self-elevate instead of dying: re-exec this exact invocation under sudo.
|
||||||
|
# -E preserves the environment (HOMELAB_* overrides, Proton Pass vault vars)
|
||||||
|
# across the re-exec. A no-op once already root.
|
||||||
|
require_root() {
|
||||||
|
[ "$(id -u)" = 0 ] && return 0
|
||||||
|
echo ">> $1 needs root — re-executing under sudo" >&2
|
||||||
|
# $KEYDIR is derived from $HOME, and whether sudo carries $HOME across
|
||||||
|
# depends on the local sudoers policy (env_reset/always_set_home). Pin the
|
||||||
|
# resolved value so the re-exec looks for host keys where the invoking user
|
||||||
|
# has them, not under /root.
|
||||||
|
export HOMELAB_KEY_DIR="$KEYDIR"
|
||||||
|
exec sudo -E -- "$SCRIPT_PATH" "${SCRIPT_ARGS[@]}"
|
||||||
|
}
|
||||||
|
|
||||||
# Exactly one path matching a glob, or die. `ls glob | head -1` silently yields
|
# Exactly one path matching a glob, or die. `ls glob | head -1` silently yields
|
||||||
# an empty string when nothing matches (head exits 0, so set -e never fires) and
|
# an empty string when nothing matches (head exits 0, so set -e never fires) and
|
||||||
# the failure only surfaces later as a confusing tar/dd error.
|
# the failure only surfaces later as a confusing tar/dd error.
|
||||||
@@ -63,9 +117,21 @@ one_match() {
|
|||||||
local what="$1"; shift
|
local what="$1"; shift
|
||||||
local f=("$@") # caller expands the glob (nullglob is on)
|
local f=("$@") # caller expands the glob (nullglob is on)
|
||||||
[ "${#f[@]}" -gt 0 ] || die "no $what found — did the build actually produce one?"
|
[ "${#f[@]}" -gt 0 ] || die "no $what found — did the build actually produce one?"
|
||||||
|
# Say so instead of silently taking [0]: a stale result-sd/ symlink from an
|
||||||
|
# earlier config is exactly how you flash the wrong image without a word.
|
||||||
|
[ "${#f[@]}" -eq 1 ] \
|
||||||
|
|| echo ">> warning: ${#f[@]} candidates for $what, using ${f[0]} (rm the stale ones)" >&2
|
||||||
printf '%s\n' "${f[0]}"
|
printf '%s\n' "${f[0]}"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Every whole-disk device backing a block device or a mounted path, one per
|
||||||
|
# line. LVM/RAID/LUKS can sit on several at once (verified on terra:
|
||||||
|
# /mnt/ssd_01 -> sdd AND sde), so a single lookup is not enough. Empty output
|
||||||
|
# means "could not determine" — which callers must treat as unsafe, not as OK.
|
||||||
|
disks_backing() {
|
||||||
|
lsblk -rnso NAME,TYPE "$1" 2>/dev/null | awk '$2 == "disk" { print "/dev/" $1 }'
|
||||||
|
}
|
||||||
|
|
||||||
# Sets tb / cpio / bbox — the kexec tarball plus the static cpio+gzip that
|
# Sets tb / cpio / bbox — the kexec tarball plus the static cpio+gzip that
|
||||||
# kexec-run.sh needs on PATH to rebuild its initrd.
|
# kexec-run.sh needs on PATH to rebuild its initrd.
|
||||||
#
|
#
|
||||||
@@ -92,14 +158,208 @@ kexec_artifacts() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# True inside one of the throwaway live-installer environments this repo
|
||||||
|
# produces (kexec's nixos-installer, or installer-iso's homelab-installer) —
|
||||||
|
# i.e. `install <config> localhost` should wipe/install right here. False on
|
||||||
|
# any real running OS, where the same command instead means "prepare and
|
||||||
|
# reboot into an installer for THIS box" (see local_install_prepare_and_reboot).
|
||||||
|
is_live_installer() {
|
||||||
|
case "$(uname -n)" in
|
||||||
|
nixos-installer | homelab-installer) return 0 ;;
|
||||||
|
*) return 1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# `install <config> localhost` run on a REAL running OS (not already inside a
|
||||||
|
# live installer): builds installer-iso, stages its kernel/initrd + the host's
|
||||||
|
# pre-generated ssh key on the boot partition and the iso file on a non-OS
|
||||||
|
# disk, points a systemd-boot one-shot entry at them with
|
||||||
|
# homelab.install=<config> + homelab.keypart=<PARTUUID> on the kernel cmdline,
|
||||||
|
# and reboots — a real ACPI reboot through firmware POST, deliberately NOT a
|
||||||
|
# kexec jump (see terra's kexec-local gotcha in CLAUDE.md). The booted
|
||||||
|
# installer's homelab-auto-install.service reads those params, picks the host
|
||||||
|
# key back up and re-runs this exact `install <config> localhost` command
|
||||||
|
# itself (now genuinely inside the installer) once homelab-checkout.service has
|
||||||
|
# fetched the repo, finishing the job unattended.
|
||||||
|
local_install_prepare_and_reboot() {
|
||||||
|
local config="$1" hostkey="$2" assume_yes="$3"
|
||||||
|
require_root "preparing a local reinstall"
|
||||||
|
[ -d /sys/firmware/efi ] || die "not booted UEFI — the one-shot boot entry needs systemd-boot"
|
||||||
|
need bootctl
|
||||||
|
need nix
|
||||||
|
need lsblk
|
||||||
|
need findmnt
|
||||||
|
need awk
|
||||||
|
need realpath
|
||||||
|
need stat
|
||||||
|
need df
|
||||||
|
|
||||||
|
# systemd-boot keeps its entries on $BOOT — the XBOOTLDR partition when there
|
||||||
|
# is one, the ESP otherwise — which is not always /boot. Hardcoding /boot on
|
||||||
|
# a box that mounts its ESP elsewhere just creates a directory on the root
|
||||||
|
# filesystem, and then reboots into an entry the firmware never sees.
|
||||||
|
local boot
|
||||||
|
boot="$(bootctl --print-boot-path 2>/dev/null)" \
|
||||||
|
|| die "bootctl couldn't locate the boot partition — is systemd-boot installed here?"
|
||||||
|
[ -d "$boot/loader/entries" ] \
|
||||||
|
|| die "$boot/loader/entries doesn't exist — systemd-boot isn't installed on this box"
|
||||||
|
|
||||||
|
# No default/auto-picked location — the wrong disk here is destroyed
|
||||||
|
# mid-install (see the OS-disk check below), so this always asks rather
|
||||||
|
# than guessing. HOMELAB_INSTALLER_STAGE_DIR skips the prompt for scripted
|
||||||
|
# use, but is otherwise just as explicit a choice as typing it in.
|
||||||
|
local stagedir="${HOMELAB_INSTALLER_STAGE_DIR:-}"
|
||||||
|
if [ -z "$stagedir" ]; then
|
||||||
|
echo ">> currently mounted filesystems:"
|
||||||
|
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT
|
||||||
|
read -rp ">> path to stage the installer iso on (must NOT be on the OS disk being wiped): " stagedir
|
||||||
|
fi
|
||||||
|
[ -n "$stagedir" ] || die "no staging path given"
|
||||||
|
[ -d "$stagedir" ] \
|
||||||
|
|| die "staging dir $stagedir doesn't exist — needs to be an existing partition that is NOT the OS disk being wiped"
|
||||||
|
# Absolute + symlink-free: findiso= below is computed by stripping the
|
||||||
|
# mountpoint prefix off this, and a relative answer at the prompt would
|
||||||
|
# produce a path the initrd can never resolve.
|
||||||
|
stagedir="$(realpath "$stagedir")"
|
||||||
|
|
||||||
|
# Refuse if the staging partition turns out to live on the same disk
|
||||||
|
# disko is about to wipe — the iso file (and the running installer
|
||||||
|
# loopback-mounted from it) would be destroyed mid-install.
|
||||||
|
local osdisk osdisk_real stage_src stage_fstype stage_disks d
|
||||||
|
osdisk="$(nix eval --raw ".#nixosConfigurations.$config.config.disko.devices.disk" \
|
||||||
|
--apply 'd: (builtins.head (builtins.attrValues d)).device' 2>/dev/null)" \
|
||||||
|
|| die "couldn't read the OS disk device from hosts/$config/disk-config.nix"
|
||||||
|
osdisk_real="$(readlink -f "$osdisk")"
|
||||||
|
|
||||||
|
# --nofsroot matters: on btrfs, findmnt prints the subvolume as
|
||||||
|
# `/dev/sdb2[/@]`, which is not a path lsblk can open. Without it the lookup
|
||||||
|
# came back empty and the guard below was skipped entirely — i.e. it silently
|
||||||
|
# allowed staging on the very disk about to be wiped. terra's current
|
||||||
|
# CachyOS root is exactly that layout.
|
||||||
|
stage_src="$(findmnt -no SOURCE --nofsroot --target "$stagedir")" \
|
||||||
|
|| die "$stagedir doesn't resolve to a mounted filesystem"
|
||||||
|
# `|| true` so the explicit check below is what reports the problem: lsblk
|
||||||
|
# exits nonzero on a device it can't parse, and under `set -e` + pipefail a
|
||||||
|
# bare assignment from a failing substitution kills the script silently,
|
||||||
|
# right past the fail-closed message.
|
||||||
|
stage_disks="$(disks_backing "$stage_src" || true)"
|
||||||
|
# Fail closed. "Couldn't determine the disk" is not "different disk".
|
||||||
|
[ -n "$stage_disks" ] \
|
||||||
|
|| die "couldn't determine which physical disk $stagedir ($stage_src) is on — refusing to guess, since being wrong destroys the install mid-flight"
|
||||||
|
for d in $stage_disks; do
|
||||||
|
if [ "$d" = "$osdisk_real" ]; then
|
||||||
|
die "$stagedir is on the OS disk ($osdisk -> $osdisk_real) that install would wipe — re-run and pick a different disk"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# stage-1 resolves findiso= by mounting each blkid-visible partition and
|
||||||
|
# testing `-e /findiso$isoPath` (nixos/modules/system/boot/stage-1-init.sh).
|
||||||
|
# For btrfs it mounts the volume's TOP level, so a path that lives inside a
|
||||||
|
# subvolume (/@/...) is simply not there and the box boots to an emergency
|
||||||
|
# shell — after it has already rebooted out of the working OS.
|
||||||
|
stage_fstype="$(findmnt -no FSTYPE --target "$stagedir")"
|
||||||
|
[ "$stage_fstype" != btrfs ] \
|
||||||
|
|| die "$stagedir is btrfs: findiso= mounts the volume's top level, so a path inside a subvolume never resolves. Stage on a non-btrfs partition (ext4/vfat/ntfs)."
|
||||||
|
|
||||||
|
# Last chance to back out. This is the most destructive command in the
|
||||||
|
# script — it reboots the machine you are typing at and the wipe that
|
||||||
|
# follows is unattended — so it confirms just like `flash` and `kexec-local`
|
||||||
|
# do, both of which are less final than this.
|
||||||
|
if [ "$assume_yes" != "--yes" ]; then
|
||||||
|
echo ">> about to REINSTALL this machine from scratch:"
|
||||||
|
echo " hostname: $(uname -n)"
|
||||||
|
echo " config: $config"
|
||||||
|
echo " OS disk: $osdisk"
|
||||||
|
echo " -> $osdisk_real ** WIPED, unattended, after the reboot **"
|
||||||
|
# Unquoted on purpose: collapses the one-per-line list onto one line.
|
||||||
|
echo " staging: $stagedir (on $(echo $stage_disks))"
|
||||||
|
echo " entry: $boot/loader/entries/homelab-installer.conf (one-shot)"
|
||||||
|
read -rp ">> type 'yes' to build the installer, reboot into it and wipe $osdisk_real: " ok
|
||||||
|
[ "$ok" = yes ] || die "aborted"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ">> building installer-iso (kernel + initrd + iso image)"
|
||||||
|
local kernel initrd isodir iso mnt_point iso_relpath boot_src boot_partuuid
|
||||||
|
kernel="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.kernel)/bzImage"
|
||||||
|
initrd="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.initialRamdisk)/initrd"
|
||||||
|
isodir="$(nix build --no-link --print-out-paths .#nixosConfigurations.installer-iso.config.system.build.isoImage)"
|
||||||
|
iso="$(one_match 'installer iso' "$isodir"/iso/*.iso)"
|
||||||
|
|
||||||
|
# A short write is not visible until the reboot, when findiso finds a
|
||||||
|
# truncated iso and drops to an emergency shell. Check first — `install`
|
||||||
|
# prints no progress and the iso is ~1GB.
|
||||||
|
local need_stage need_boot avail_stage avail_boot
|
||||||
|
need_stage="$(stat -Lc %s "$iso")"
|
||||||
|
need_boot="$(( $(stat -Lc %s "$kernel") + $(stat -Lc %s "$initrd") + $(stat -Lc %s "$hostkey") ))"
|
||||||
|
avail_stage="$(df -B1 --output=avail "$stagedir" | tail -1 | tr -d ' ')"
|
||||||
|
avail_boot="$(df -B1 --output=avail "$boot" | tail -1 | tr -d ' ')"
|
||||||
|
[ "$avail_stage" -ge "$(( need_stage + 64 * 1024 * 1024 ))" ] \
|
||||||
|
|| die "$stagedir has $(( avail_stage / 1024 / 1024 ))MB free, the iso needs $(( need_stage / 1024 / 1024 ))MB — pick another partition"
|
||||||
|
[ "$avail_boot" -ge "$(( need_boot + 16 * 1024 * 1024 ))" ] \
|
||||||
|
|| die "$boot has $(( avail_boot / 1024 / 1024 ))MB free, kernel+initrd need $(( need_boot / 1024 / 1024 ))MB"
|
||||||
|
|
||||||
|
echo ">> staging kernel/initrd/host key on $boot, iso image on $stagedir"
|
||||||
|
install -Dm644 "$kernel" "$boot/homelab-installer/bzImage"
|
||||||
|
install -Dm644 "$initrd" "$boot/homelab-installer/initrd"
|
||||||
|
install -Dm644 "$iso" "$stagedir/homelab-installer.iso"
|
||||||
|
|
||||||
|
# The ISO is built from a PUBLIC repo and deliberately carries no
|
||||||
|
# credentials, so the host key has to travel with the staged installer or
|
||||||
|
# the auto-install run has nothing to seed /etc/ssh with — and without that,
|
||||||
|
# sops can't decrypt on boot #1, /etc/shadow gets written once with a locked
|
||||||
|
# darman, and no later `deploy switch` can fix it (README).
|
||||||
|
#
|
||||||
|
# $boot lives on the OS disk, so disko destroys this copy minutes later. The
|
||||||
|
# mode is advisory on vfat (permissions come from the mount's fmask, 0077 on
|
||||||
|
# a NixOS/systemd-boot ESP) — it is the wipe, not the mode, doing the work.
|
||||||
|
install -Dm600 "$hostkey" "$boot/homelab-installer/ssh_host_ed25519_key"
|
||||||
|
install -Dm644 "$hostkey.pub" "$boot/homelab-installer/ssh_host_ed25519_key.pub"
|
||||||
|
boot_src="$(findmnt -no SOURCE --nofsroot --target "$boot")" \
|
||||||
|
|| die "couldn't resolve $boot to a device"
|
||||||
|
boot_partuuid="$(lsblk -no PARTUUID "$boot_src" 2>/dev/null | head -1 | tr -d ' ' || true)"
|
||||||
|
[ -n "$boot_partuuid" ] \
|
||||||
|
|| die "couldn't read a PARTUUID for $boot ($boot_src) — the installer needs it to find the host key"
|
||||||
|
|
||||||
|
# findiso= is a path relative to whatever partition the initrd finds it on
|
||||||
|
# (it mounts every blkid-visible partition looking for it), not to `/`, if
|
||||||
|
# $stagedir is a subdirectory of a bigger filesystem rather than a mountpoint
|
||||||
|
# itself. It must KEEP its leading slash: stage-1 tests `-e /findiso$isoPath`,
|
||||||
|
# so a bare `var/tmp/x.iso` becomes `/findisovar/tmp/x.iso` and never matches.
|
||||||
|
# Prefixing then squeezing handles both ends: stagedir == the mountpoint
|
||||||
|
# (strip leaves "") and mnt_point == "/" (strip leaves a relative path).
|
||||||
|
mnt_point="$(findmnt -no TARGET --target "$stagedir")"
|
||||||
|
iso_relpath="$(printf '/%s/%s' "${stagedir#"$mnt_point"}" homelab-installer.iso | tr -s /)"
|
||||||
|
|
||||||
|
cat >"$boot/loader/entries/homelab-installer.conf" <<EOF
|
||||||
|
title Homelab Installer ($config, findiso)
|
||||||
|
linux /homelab-installer/bzImage
|
||||||
|
initrd /homelab-installer/initrd
|
||||||
|
options nohibernate root=fstab loglevel=4 lsm=landlock,yama,bpf findiso=$iso_relpath homelab.install=$config homelab.keypart=$boot_partuuid
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo ">> one-shot boot into the installer, then rebooting — it will finish this install itself"
|
||||||
|
bootctl set-oneshot homelab-installer.conf
|
||||||
|
systemctl reboot
|
||||||
|
}
|
||||||
|
|
||||||
# Flakes only see git-tracked files: an untracked hosts/<config>/ is silently
|
# Flakes only see git-tracked files: an untracked hosts/<config>/ is silently
|
||||||
# invisible to `nix build`/`nixos-install`, which then fails obscurely or builds
|
# invisible to `nix build`/`nixos-install`, which then fails obscurely or builds
|
||||||
# a stale config. Check before doing anything destructive.
|
# a stale config. Check before doing anything destructive.
|
||||||
require_tracked() {
|
require_tracked() {
|
||||||
local config="$1" cfgfile="hosts/$1/configuration.nix"
|
local config="$1" cfgfile="hosts/$1/configuration.nix" f
|
||||||
[ -e "$cfgfile" ] || die "no $cfgfile in the repo"
|
[ -e "$cfgfile" ] || die "no $cfgfile in the repo"
|
||||||
git -C "$REPO" ls-files --error-unmatch "$cfgfile" >/dev/null 2>&1 \
|
# No .git at all (e.g. a tarball export of the repo, no working tree), or no
|
||||||
|| die "$cfgfile is untracked — 'git add hosts/$config' first (flakes ignore untracked files)"
|
# git binary, means there's nothing that CAN be untracked — nothing to check.
|
||||||
|
# Only skip on that, not on any other git failure.
|
||||||
|
command -v git >/dev/null 2>&1 || return 0
|
||||||
|
git -C "$REPO" rev-parse --is-inside-work-tree >/dev/null 2>&1 || return 0
|
||||||
|
# Every .nix in hosts/<config>/, not just configuration.nix: an untracked
|
||||||
|
# disk-config.nix is exactly as invisible to the flake, and it is the file
|
||||||
|
# that decides which disk gets wiped.
|
||||||
|
for f in "hosts/$config"/*.nix; do
|
||||||
|
git -C "$REPO" ls-files --error-unmatch "$f" >/dev/null 2>&1 \
|
||||||
|
|| die "$f is untracked — 'git add hosts/$config' first (flakes ignore untracked files)"
|
||||||
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
# The password field of a Proton Pass item ("--field password" prints the bare
|
# The password field of a Proton Pass item ("--field password" prints the bare
|
||||||
@@ -236,7 +496,7 @@ case "$cmd" in
|
|||||||
# This is a one-way trip on the machine you are typing at, so every check
|
# This is a one-way trip on the machine you are typing at, so every check
|
||||||
# that can fail is done BEFORE the point of no return, and nothing that the
|
# that can fail is done BEFORE the point of no return, and nothing that the
|
||||||
# jump depends on is cleaned up behind it (see the trap discussion below).
|
# jump depends on is cleaned up behind it (see the trap discussion below).
|
||||||
[ "$(id -u)" = 0 ] || die "kexec-local must run as root (sudo ./deploy kexec-local)"
|
require_root "kexec-local"
|
||||||
|
|
||||||
assume_yes=""
|
assume_yes=""
|
||||||
[ "${2:-}" = "--yes" ] && assume_yes=1
|
[ "${2:-}" = "--yes" ] && assume_yes=1
|
||||||
@@ -340,22 +600,36 @@ case "$cmd" in
|
|||||||
;;
|
;;
|
||||||
|
|
||||||
install)
|
install)
|
||||||
config="${2:-}"; host="${3:-}"
|
config="${2:-}"; host="${3:-}"; assume_yes="${4:-}"
|
||||||
{ [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy install <config> <host>"
|
{ [ -n "$config" ] && [ -n "$host" ]; } || die "usage: ./deploy install <config> <host> [--yes]"
|
||||||
hostkey="$HOME/.config/homelab/$config/ssh_host_ed25519_key"
|
# $KEYDIR, not a bare $HOME — see its definition. This same check runs
|
||||||
|
# inside installer-iso, where homelab-auto-install.service has no $HOME and
|
||||||
|
# has just dropped the key into /root/.config/homelab/<config>/.
|
||||||
|
hostkey="$KEYDIR/$config/ssh_host_ed25519_key"
|
||||||
[ -f "$hostkey" ] || die "missing host key: $hostkey"
|
[ -f "$hostkey" ] || die "missing host key: $hostkey"
|
||||||
[ -d "./hosts/$config" ] || die "no ./hosts/$config directory in the repo"
|
[ -d "./hosts/$config" ] || die "no ./hosts/$config directory in the repo"
|
||||||
require_tracked "$config"
|
require_tracked "$config"
|
||||||
|
|
||||||
if [ "$host" = "localhost" ] || [ "$host" = "127.0.0.1" ]; then
|
if [ "$host" = "localhost" ] || [ "$host" = "127.0.0.1" ]; then
|
||||||
|
if ! is_live_installer; then
|
||||||
|
# Not already inside a live installer: build one, stage it, one-shot
|
||||||
|
# boot into it, and let it finish this exact command itself. See
|
||||||
|
# local_install_prepare_and_reboot above and CLAUDE.md.
|
||||||
|
local_install_prepare_and_reboot "$config" "$hostkey" "$assume_yes"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
# Local install: no ssh, no nixos-anywhere. Run after `kexec-local` (or
|
# Local install: no ssh, no nixos-anywhere. Run after `kexec-local` (or
|
||||||
# from a live ISO) so /mnt is free to wipe — this IS the box, no second
|
# from a live ISO) so /mnt is free to wipe — this IS the box, no second
|
||||||
# machine in the loop, so skip straight to disko + nixos-install.
|
# machine in the loop, so skip straight to disko + nixos-install.
|
||||||
[ "$(id -u)" = 0 ] || die "local install must run as root"
|
require_root "local install"
|
||||||
[ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix"
|
[ -f "./hosts/$config/disk-config.nix" ] || die "no ./hosts/$config/disk-config.nix"
|
||||||
|
|
||||||
echo ">> disko .#$config onto this box's OS disk (WILL be wiped)"
|
echo ">> disko .#$config onto this box's OS disk (WILL be wiped)"
|
||||||
nix run github:nix-community/disko -- \
|
# `.#disko`, not github:nix-community/disko — the revision comes from this
|
||||||
|
# repo's flake.lock rather than upstream master-of-the-day, and resolves
|
||||||
|
# from the local store. See the nixos-anywhere input in flake.nix.
|
||||||
|
nix run ".#disko" -- \
|
||||||
--mode disko "./hosts/$config/disk-config.nix"
|
--mode disko "./hosts/$config/disk-config.nix"
|
||||||
|
|
||||||
echo ">> installing sops host key so it can decrypt on boot #1"
|
echo ">> installing sops host key so it can decrypt on boot #1"
|
||||||
@@ -387,11 +661,11 @@ case "$cmd" in
|
|||||||
echo ">> root ssh password from Proton Pass ($root_item)"
|
echo ">> root ssh password from Proton Pass ($root_item)"
|
||||||
export SSHPASS="$root_pw"
|
export SSHPASS="$root_pw"
|
||||||
unset root_pw
|
unset root_pw
|
||||||
nix run github:nix-community/nixos-anywhere -- \
|
nix run ".#nixos-anywhere" -- \
|
||||||
--env-password "${anywhere[@]}"
|
--env-password "${anywhere[@]}"
|
||||||
unset SSHPASS
|
unset SSHPASS
|
||||||
else
|
else
|
||||||
nix run github:nix-community/nixos-anywhere -- "${anywhere[@]}"
|
nix run ".#nixos-anywhere" -- "${anywhere[@]}"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
@@ -463,7 +737,7 @@ case "$cmd" in
|
|||||||
# partition at /var/lib/sops-nix/age.txt so sops decrypts on first boot.
|
# partition at /var/lib/sops-nix/age.txt so sops decrypts on first boot.
|
||||||
# (The Pi's vfat partition isn't mounted at runtime, so the key can't live
|
# (The Pi's vfat partition isn't mounted at runtime, so the key can't live
|
||||||
# there.) Key stays off-repo, out of the nix store, and out of the image.
|
# there.) Key stays off-repo, out of the nix store, and out of the image.
|
||||||
keyfile="$HOME/.config/homelab/$config/age.txt"
|
keyfile="$KEYDIR/$config/age.txt"
|
||||||
if [ -f "$keyfile" ]; then
|
if [ -f "$keyfile" ]; then
|
||||||
echo ">> installing sops age key onto the root partition"
|
echo ">> installing sops age key onto the root partition"
|
||||||
sudo partprobe "$dev" 2>/dev/null || sudo blockdev --rereadpt "$dev" 2>/dev/null || true
|
sudo partprobe "$dev" 2>/dev/null || sudo blockdev --rereadpt "$dev" 2>/dev/null || true
|
||||||
@@ -476,10 +750,13 @@ case "$cmd" in
|
|||||||
| sort -rn | head -1 | cut -d' ' -f2)"
|
| sort -rn | head -1 | cut -d' ' -f2)"
|
||||||
[ -n "$rootpart" ] || die "no ext4 root partition found on $dev — place $keyfile at /var/lib/sops-nix/age.txt manually"
|
[ -n "$rootpart" ] || die "no ext4 root partition found on $dev — place $keyfile at /var/lib/sops-nix/age.txt manually"
|
||||||
mnt="$(mktemp -d)"
|
mnt="$(mktemp -d)"
|
||||||
|
# Unmount + remove even if the install fails, so a retry doesn't trip
|
||||||
|
# over the card still being mounted on a stale temp dir.
|
||||||
|
trap 'sudo umount "$mnt" 2>/dev/null || true; rmdir "$mnt" 2>/dev/null || true' EXIT
|
||||||
sudo mount "$rootpart" "$mnt"
|
sudo mount "$rootpart" "$mnt"
|
||||||
sudo install -Dm600 "$keyfile" "$mnt/var/lib/sops-nix/age.txt"
|
sudo install -Dm600 "$keyfile" "$mnt/var/lib/sops-nix/age.txt"
|
||||||
sudo sync
|
sudo sync
|
||||||
sudo umount "$mnt"; rmdir "$mnt"
|
sudo umount "$mnt"; rmdir "$mnt"; trap - EXIT
|
||||||
echo ">> age key installed (/var/lib/sops-nix/age.txt)"
|
echo ">> age key installed (/var/lib/sops-nix/age.txt)"
|
||||||
fi
|
fi
|
||||||
echo ">> done — insert the card into the Pi and boot."
|
echo ">> done — insert the card into the Pi and boot."
|
||||||
|
|||||||
Reference in New Issue
Block a user