diff --git a/CLAUDE.md b/CLAUDE.md index 171a8a3..5478638 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -107,6 +107,14 @@ kept its ssh host key. Run it after ANY change to the kexec paths. - **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board); eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the 16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped). +- **terra: `./scripts/deploy kexec-local` hangs hard — do not use it there.** Confirmed + on real hardware: kexec's `device_shutdown()` pass runs (SCSI disks sync fine in the + log), then the machine goes dark and never comes back — `journalctl --list-boots` + showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power + cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known + to hang during kexec's device-shutdown pass with no clean handoff before the jump — + same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of + slow. Use the USB installer path instead (README's "First install on terra" section). - **disko wipes only the OS disk** named in `hosts//disk-config.nix`; data disks are plain `fileSystems` in `configuration.nix`. - `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the diff --git a/README.md b/README.md index 9bd420b..43f81c8 100644 --- a/README.md +++ b/README.md @@ -84,39 +84,57 @@ an installer, partitions via disko, installs. Manual alternative (USB ISO): boot installer, `disko` the disk, then `nixos-install --flake .#jupiter`. -## First install on terra — in-place kexec (replacing CachyOS) +## First install on terra — USB installer (replacing CachyOS) -terra is the desktop you're typing on, currently running CachyOS with a -writable root — no ZimaOS-style read-only-root problem, no second machine -needed. Everything is already prepped in this repo: real OS-disk id in +terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running +CachyOS. Everything is already prepped in this repo: real OS-disk id in `disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in `.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and `secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey` values. Nothing to fill in — just run it. +> ⚠️ **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.** +> Confirmed on real hardware: the jump hangs completely (kexec's own +> `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes +> dark and never comes back; `journalctl --list-boots` showed a **~15 minute** +> gap before the next boot, i.e. a hard hang needing a manual power cycle, not +> a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang +> during kexec's device-shutdown pass with no clean way to hand control back +> before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot +> workaround, just fatal here instead of merely slow. Untried mitigations +> (unbind/`rmmod amdgpu` from a bare VT first, kernel params) might work but +> aren't worth the risk of repeating a hard hang on a daily-driver box for a +> one-time install — use the USB path below instead. + > ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The > dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs > disks are not in disko and are untouched — but double check with > `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have > changed since `disk-config.nix` was written. -1. From a root shell on terra itself: +1. Build the installer ISO and write it to a USB stick (from your laptop): ``` - sudo ./scripts/deploy kexec-local --yes + nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage + sudo dd if=result/iso/*.iso of=/dev/sdX bs=4M status=progress conv=fsync ``` - Stages a RAM installer and kexecs into it. The console drops for ~1-2 min - then comes back logged in as `nixos-installer` — same ssh host key, so - `known_hosts` still matches if you're watching over ssh instead of the - physical console. -2. Still targeting terra (now `localhost`/`127.0.0.1` from the installer's own - shell): +2. Boot terra from the USB stick. It comes up with sshd + DHCP + your laptop + pubkey authorized for root, hostname `homelab-installer`. +3. From your laptop, copy the repo onto the booted installer (it doesn't have + its own credentials for the private gitea repo, so push it over instead of + cloning from there): ``` + rsync -av /mnt/hdd_01/data/Dev/homelab/ root@:/root/homelab/ + ``` +4. On terra (ssh'd in as root, or at the physical console): + ``` + cd /root/homelab ./scripts/deploy install terra localhost ``` - `localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko + `nixos-install` - directly against `/mnt`. Ships terra's pre-generated host key so - `/run/secrets/*` decrypts on boot #1. -3. Reboot into NixOS. Then, same as any other host: + `localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko + + `nixos-install` directly against `/mnt`. Ships terra's pre-generated host + key so `/run/secrets/*` decrypts on boot #1. +5. Reboot into NixOS (remove the USB stick first). Then, same as any other + host: ``` ssh darman@terra sudo -v # DO NOT SKIP — see below ``` diff --git a/flake.nix b/flake.nix index b092588..809720c 100644 --- a/flake.nix +++ b/flake.nix @@ -163,20 +163,22 @@ }; # Bootable USB recovery installer with our SSH key + sshd + DHCP. + # Reusable for any host's manual-USB install path (jupiter, terra, ...). # Build the ISO: # nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage - # dd it to a USB stick, boot the ZimaBlade from it, SSH in, ./deploy install. + # dd it to a USB stick, boot the target from it, SSH in, ./deploy install. installer-iso = nixpkgs.lib.nixosSystem { inherit system; modules = [ (nixpkgs + "/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix") - ({ ... }: { + ({ pkgs, ... }: { services.openssh.enable = true; services.openssh.settings.PermitRootLogin = "prohibit-password"; users.users.root.openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZpkPVhzi1zG5JI9hWyUgdyvNIQbp4ts4jw3idpMhhN erik@laptop" ]; - networking.hostName = "jupiter-installer"; + networking.hostName = "homelab-installer"; + environment.systemPackages = [ pkgs.git ]; }) ]; };