terra: kexec-local hangs hard on real hardware, switch docs to USB installer

Confirmed on real hardware: kexec's device_shutdown() pass runs (SCSI disks
sync fine in the log) then the machine goes dark for good — journalctl
--list-boots showed a ~15min gap before the next boot, a genuine hang needing
a manual power cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT):
discrete AMD GPUs are known to hang during kexec's device-shutdown pass with
no clean handoff before the jump, same class of issue as jupiter's
reboot=pci warm-reboot workaround, just fatal here instead of slow.

README's terra install section now leads with the USB installer path instead
(build ISO, dd to USB, rsync the repo over, disko + nixos-install locally).
CLAUDE.md's gotchas list gets the same warning. installer-iso is renamed from
jupiter-installer to homelab-installer since it's genuinely host-agnostic,
and now ships git.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-24 01:11:52 +02:00
co-authored by Claude Sonnet 5
parent 5845c29a44
commit 2a27d2cf4b
3 changed files with 47 additions and 19 deletions
+8
View File
@@ -107,6 +107,14 @@ kept its ssh host key. Run it after ANY change to the kexec paths.
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board); - **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped). 16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
- **terra: `./scripts/deploy kexec-local` hangs hard — do not use it there.** Confirmed
on real hardware: kexec's `device_shutdown()` pass runs (SCSI disks sync fine in the
log), then the machine goes dark and never comes back — `journalctl --list-boots`
showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power
cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known
to hang during kexec's device-shutdown pass with no clean handoff before the jump —
same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of
slow. Use the USB installer path instead (README's "First install on terra" section).
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are - **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
plain `fileSystems` in `configuration.nix`. plain `fileSystems` in `configuration.nix`.
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
+34 -16
View File
@@ -84,39 +84,57 @@ an installer, partitions via disko, installs.
Manual alternative (USB ISO): boot installer, `disko` the disk, then Manual alternative (USB ISO): boot installer, `disko` the disk, then
`nixos-install --flake .#jupiter`. `nixos-install --flake .#jupiter`.
## First install on terra — in-place kexec (replacing CachyOS) ## First install on terra — USB installer (replacing CachyOS)
terra is the desktop you're typing on, currently running CachyOS with a terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running
writable root — no ZimaOS-style read-only-root problem, no second machine CachyOS. Everything is already prepped in this repo: real OS-disk id in
needed. Everything is already prepped in this repo: real OS-disk id in
`disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in `disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in
`.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and `.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and
`secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey` `secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey`
values. Nothing to fill in — just run it. values. Nothing to fill in — just run it.
> ⚠️ **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.**
> Confirmed on real hardware: the jump hangs completely (kexec's own
> `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes
> dark and never comes back; `journalctl --list-boots` showed a **~15 minute**
> gap before the next boot, i.e. a hard hang needing a manual power cycle, not
> a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang
> during kexec's device-shutdown pass with no clean way to hand control back
> before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot
> workaround, just fatal here instead of merely slow. Untried mitigations
> (unbind/`rmmod amdgpu` from a bare VT first, kernel params) might work but
> aren't worth the risk of repeating a hard hang on a daily-driver box for a
> one-time install — use the USB path below instead.
> ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The > ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
> dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs > dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
> disks are not in disko and are untouched — but double check with > disks are not in disko and are untouched — but double check with
> `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have > `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
> changed since `disk-config.nix` was written. > changed since `disk-config.nix` was written.
1. From a root shell on terra itself: 1. Build the installer ISO and write it to a USB stick (from your laptop):
``` ```
sudo ./scripts/deploy kexec-local --yes nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
sudo dd if=result/iso/*.iso of=/dev/sdX bs=4M status=progress conv=fsync
``` ```
Stages a RAM installer and kexecs into it. The console drops for ~1-2 min 2. Boot terra from the USB stick. It comes up with sshd + DHCP + your laptop
then comes back logged in as `nixos-installer` — same ssh host key, so pubkey authorized for root, hostname `homelab-installer`.
`known_hosts` still matches if you're watching over ssh instead of the 3. From your laptop, copy the repo onto the booted installer (it doesn't have
physical console. its own credentials for the private gitea repo, so push it over instead of
2. Still targeting terra (now `localhost`/`127.0.0.1` from the installer's own cloning from there):
shell):
``` ```
rsync -av /mnt/hdd_01/data/Dev/homelab/ root@<terra-installer-ip>:/root/homelab/
```
4. On terra (ssh'd in as root, or at the physical console):
```
cd /root/homelab
./scripts/deploy install terra localhost ./scripts/deploy install terra localhost
``` ```
`localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko + `nixos-install` `localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko +
directly against `/mnt`. Ships terra's pre-generated host key so `nixos-install` directly against `/mnt`. Ships terra's pre-generated host
`/run/secrets/*` decrypts on boot #1. key so `/run/secrets/*` decrypts on boot #1.
3. Reboot into NixOS. Then, same as any other host: 5. Reboot into NixOS (remove the USB stick first). Then, same as any other
host:
``` ```
ssh darman@terra sudo -v # DO NOT SKIP — see below ssh darman@terra sudo -v # DO NOT SKIP — see below
``` ```
+5 -3
View File
@@ -163,20 +163,22 @@
}; };
# Bootable USB recovery installer with our SSH key + sshd + DHCP. # Bootable USB recovery installer with our SSH key + sshd + DHCP.
# Reusable for any host's manual-USB install path (jupiter, terra, ...).
# Build the ISO: # Build the ISO:
# nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage # nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
# dd it to a USB stick, boot the ZimaBlade from it, SSH in, ./deploy install. # dd it to a USB stick, boot the target from it, SSH in, ./deploy install.
installer-iso = nixpkgs.lib.nixosSystem { installer-iso = nixpkgs.lib.nixosSystem {
inherit system; inherit system;
modules = [ modules = [
(nixpkgs + "/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix") (nixpkgs + "/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix")
({ ... }: { ({ pkgs, ... }: {
services.openssh.enable = true; services.openssh.enable = true;
services.openssh.settings.PermitRootLogin = "prohibit-password"; services.openssh.settings.PermitRootLogin = "prohibit-password";
users.users.root.openssh.authorizedKeys.keys = [ users.users.root.openssh.authorizedKeys.keys = [
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZpkPVhzi1zG5JI9hWyUgdyvNIQbp4ts4jw3idpMhhN erik@laptop" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZpkPVhzi1zG5JI9hWyUgdyvNIQbp4ts4jw3idpMhhN erik@laptop"
]; ];
networking.hostName = "jupiter-installer"; networking.hostName = "homelab-installer";
environment.systemPackages = [ pkgs.git ];
}) })
]; ];
}; };