terra: kexec-local hangs hard on real hardware, switch docs to USB installer
Confirmed on real hardware: kexec's device_shutdown() pass runs (SCSI disks sync fine in the log) then the machine goes dark for good — journalctl --list-boots showed a ~15min gap before the next boot, a genuine hang needing a manual power cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known to hang during kexec's device-shutdown pass with no clean handoff before the jump, same class of issue as jupiter's reboot=pci warm-reboot workaround, just fatal here instead of slow. README's terra install section now leads with the USB installer path instead (build ISO, dd to USB, rsync the repo over, disko + nixos-install locally). CLAUDE.md's gotchas list gets the same warning. installer-iso is renamed from jupiter-installer to homelab-installer since it's genuinely host-agnostic, and now ships git. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -107,6 +107,14 @@ kept its ssh host key. Run it after ANY change to the kexec paths.
|
|||||||
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
|
- **jupiter**: `boot.kernelParams = [ "reboot=pci" ]` (warm reboot hangs on that board);
|
||||||
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
|
eMMC initrd modules pinned in `configuration.nix` (generate-config misses them); the
|
||||||
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
|
16TB×2 **RAID0** data lives on `/mnt/data` with `nofail`, kept OUT of disko (never wiped).
|
||||||
|
- **terra: `./scripts/deploy kexec-local` hangs hard — do not use it there.** Confirmed
|
||||||
|
on real hardware: kexec's `device_shutdown()` pass runs (SCSI disks sync fine in the
|
||||||
|
log), then the machine goes dark and never comes back — `journalctl --list-boots`
|
||||||
|
showed a ~15min gap before the next boot, i.e. a genuine hang needing a manual power
|
||||||
|
cycle, not a slow jump. Near-certainly amdgpu (RX 6800 XT): discrete AMD GPUs are known
|
||||||
|
to hang during kexec's device-shutdown pass with no clean handoff before the jump —
|
||||||
|
same class of issue as jupiter's `reboot=pci` workaround, just fatal here instead of
|
||||||
|
slow. Use the USB installer path instead (README's "First install on terra" section).
|
||||||
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
|
- **disko wipes only the OS disk** named in `hosts/<h>/disk-config.nix`; data disks are
|
||||||
plain `fileSystems` in `configuration.nix`.
|
plain `fileSystems` in `configuration.nix`.
|
||||||
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
|
- `nixos-anywhere`/kexec needs a writable root; **ZimaOS root is read-only**, hence the
|
||||||
|
|||||||
@@ -84,39 +84,57 @@ an installer, partitions via disko, installs.
|
|||||||
Manual alternative (USB ISO): boot installer, `disko` the disk, then
|
Manual alternative (USB ISO): boot installer, `disko` the disk, then
|
||||||
`nixos-install --flake .#jupiter`.
|
`nixos-install --flake .#jupiter`.
|
||||||
|
|
||||||
## First install on terra — in-place kexec (replacing CachyOS)
|
## First install on terra — USB installer (replacing CachyOS)
|
||||||
|
|
||||||
terra is the desktop you're typing on, currently running CachyOS with a
|
terra is a Ryzen 9 5900X / Radeon RX 6800 XT desktop, currently running
|
||||||
writable root — no ZimaOS-style read-only-root problem, no second machine
|
CachyOS. Everything is already prepped in this repo: real OS-disk id in
|
||||||
needed. Everything is already prepped in this repo: real OS-disk id in
|
|
||||||
`disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in
|
`disk-config.nix`, real login pubkey in `common.nix`, terra's age recipient in
|
||||||
`.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and
|
`.sops.yaml`, its host key pre-generated at `~/.config/homelab/terra/`, and
|
||||||
`secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey`
|
`secrets/terra.yaml` already holds real `darman_password` / `tailscale_authkey`
|
||||||
values. Nothing to fill in — just run it.
|
values. Nothing to fill in — just run it.
|
||||||
|
|
||||||
|
> ⚠️ **`./scripts/deploy kexec-local` does NOT work on terra — do not use it.**
|
||||||
|
> Confirmed on real hardware: the jump hangs completely (kexec's own
|
||||||
|
> `device_shutdown()` pass runs — SCSI disks sync fine — then the machine goes
|
||||||
|
> dark and never comes back; `journalctl --list-boots` showed a **~15 minute**
|
||||||
|
> gap before the next boot, i.e. a hard hang needing a manual power cycle, not
|
||||||
|
> a slow jump). Near-certainly amdgpu: discrete AMD GPUs are known to hang
|
||||||
|
> during kexec's device-shutdown pass with no clean way to hand control back
|
||||||
|
> before the jump — same class of issue as jupiter's `reboot=pci` warm-reboot
|
||||||
|
> workaround, just fatal here instead of merely slow. Untried mitigations
|
||||||
|
> (unbind/`rmmod amdgpu` from a bare VT first, kernel params) might work but
|
||||||
|
> aren't worth the risk of repeating a hard hang on a daily-driver box for a
|
||||||
|
> one-time install — use the USB path below instead.
|
||||||
|
|
||||||
> ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
|
> ⚠️ The OS disk (`ata-KINGSTON_SA400S37480G_50026B738072F6C6`) is WIPED. The
|
||||||
> dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
|
> dev-data disks (`/mnt/hdd_01` ext4, `/mnt/ssd_01` LVM) and the leftover ntfs
|
||||||
> disks are not in disko and are untouched — but double check with
|
> disks are not in disko and are untouched — but double check with
|
||||||
> `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
|
> `lsblk -o NAME,SERIAL,SIZE,MODEL` before proceeding if the box's disks have
|
||||||
> changed since `disk-config.nix` was written.
|
> changed since `disk-config.nix` was written.
|
||||||
|
|
||||||
1. From a root shell on terra itself:
|
1. Build the installer ISO and write it to a USB stick (from your laptop):
|
||||||
```
|
```
|
||||||
sudo ./scripts/deploy kexec-local --yes
|
nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
|
||||||
|
sudo dd if=result/iso/*.iso of=/dev/sdX bs=4M status=progress conv=fsync
|
||||||
```
|
```
|
||||||
Stages a RAM installer and kexecs into it. The console drops for ~1-2 min
|
2. Boot terra from the USB stick. It comes up with sshd + DHCP + your laptop
|
||||||
then comes back logged in as `nixos-installer` — same ssh host key, so
|
pubkey authorized for root, hostname `homelab-installer`.
|
||||||
`known_hosts` still matches if you're watching over ssh instead of the
|
3. From your laptop, copy the repo onto the booted installer (it doesn't have
|
||||||
physical console.
|
its own credentials for the private gitea repo, so push it over instead of
|
||||||
2. Still targeting terra (now `localhost`/`127.0.0.1` from the installer's own
|
cloning from there):
|
||||||
shell):
|
|
||||||
```
|
```
|
||||||
|
rsync -av /mnt/hdd_01/data/Dev/homelab/ root@<terra-installer-ip>:/root/homelab/
|
||||||
|
```
|
||||||
|
4. On terra (ssh'd in as root, or at the physical console):
|
||||||
|
```
|
||||||
|
cd /root/homelab
|
||||||
./scripts/deploy install terra localhost
|
./scripts/deploy install terra localhost
|
||||||
```
|
```
|
||||||
`localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko + `nixos-install`
|
`localhost`/`127.0.0.1` skips nixos-anywhere/ssh and runs disko +
|
||||||
directly against `/mnt`. Ships terra's pre-generated host key so
|
`nixos-install` directly against `/mnt`. Ships terra's pre-generated host
|
||||||
`/run/secrets/*` decrypts on boot #1.
|
key so `/run/secrets/*` decrypts on boot #1.
|
||||||
3. Reboot into NixOS. Then, same as any other host:
|
5. Reboot into NixOS (remove the USB stick first). Then, same as any other
|
||||||
|
host:
|
||||||
```
|
```
|
||||||
ssh darman@terra sudo -v # DO NOT SKIP — see below
|
ssh darman@terra sudo -v # DO NOT SKIP — see below
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -163,20 +163,22 @@
|
|||||||
};
|
};
|
||||||
|
|
||||||
# Bootable USB recovery installer with our SSH key + sshd + DHCP.
|
# Bootable USB recovery installer with our SSH key + sshd + DHCP.
|
||||||
|
# Reusable for any host's manual-USB install path (jupiter, terra, ...).
|
||||||
# Build the ISO:
|
# Build the ISO:
|
||||||
# nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
|
# nix build .#nixosConfigurations.installer-iso.config.system.build.isoImage
|
||||||
# dd it to a USB stick, boot the ZimaBlade from it, SSH in, ./deploy install.
|
# dd it to a USB stick, boot the target from it, SSH in, ./deploy install.
|
||||||
installer-iso = nixpkgs.lib.nixosSystem {
|
installer-iso = nixpkgs.lib.nixosSystem {
|
||||||
inherit system;
|
inherit system;
|
||||||
modules = [
|
modules = [
|
||||||
(nixpkgs + "/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix")
|
(nixpkgs + "/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix")
|
||||||
({ ... }: {
|
({ pkgs, ... }: {
|
||||||
services.openssh.enable = true;
|
services.openssh.enable = true;
|
||||||
services.openssh.settings.PermitRootLogin = "prohibit-password";
|
services.openssh.settings.PermitRootLogin = "prohibit-password";
|
||||||
users.users.root.openssh.authorizedKeys.keys = [
|
users.users.root.openssh.authorizedKeys.keys = [
|
||||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZpkPVhzi1zG5JI9hWyUgdyvNIQbp4ts4jw3idpMhhN erik@laptop"
|
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZpkPVhzi1zG5JI9hWyUgdyvNIQbp4ts4jw3idpMhhN erik@laptop"
|
||||||
];
|
];
|
||||||
networking.hostName = "jupiter-installer";
|
networking.hostName = "homelab-installer";
|
||||||
|
environment.systemPackages = [ pkgs.git ];
|
||||||
})
|
})
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
|
|||||||
Reference in New Issue
Block a user