add native VictoriaMetrics single-node monitoring to Jupiter
scrape node_exporter on Jupiter, Mars, Neptun, Mercury, and Terra over the tailnet
scrape Mercury every 15 seconds with a 10-second timeout; scrape the other node-exporter targets every 5 seconds
scrape VictoriaMetrics self-metrics
retain metrics for 15 days
store the TSDB on Jupiter's data array at /mnt/data/AppData/victoriametrics, with an explicit mount dependency
Validation
git diff --check passed
Nix evaluation was not rerun because this runner has no nix executable
no deployment was performed
## Summary
- add native VictoriaMetrics single-node monitoring to Jupiter
- scrape node_exporter on Jupiter, Mars, Neptun, Mercury, and Terra over the tailnet
- scrape Mercury every 15 seconds with a 10-second timeout; scrape the other node-exporter targets every 5 seconds
- scrape VictoriaMetrics self-metrics
- retain metrics for 15 days
- store the TSDB on Jupiter's data array at `/mnt/data/AppData/victoriametrics`, with an explicit mount dependency
## Validation
- `git diff --check` passed
- Nix evaluation was not rerun because this runner has no `nix` executable
- no deployment was performed
Addressed #1 (comment): changed VictoriaMetrics retentionPeriod to "15d".\n\nCommit: 082cbaff2a\nValidation: git diff --check passed; Nix evaluation was unavailable because nix is not installed on this runner.
Addressed https://git.mgaction.town/darman/homelab/pulls/1#issuecomment-119: changed VictoriaMetrics retentionPeriod to "15d".\n\nCommit: https://git.mgaction.town/darman/homelab/commit/082cbaff2a4a120b001cd8e92182faed4640127b\nValidation: git diff --check passed; Nix evaluation was unavailable because nix is not installed on this runner.
Addressed #1 (comment): resolved the master conflict by retaining both the VictoriaMetrics and Gitea relay imports on feat/mars-victoriametrics.
Commit: dc037312b3
Validation: git diff --check passed; no conflict markers remain. Nix evaluation was unavailable because nix is not installed; Python discovery reported no tests.
Addressed https://git.mgaction.town/darman/homelab/pulls/1#issuecomment-122: resolved the master conflict by retaining both the VictoriaMetrics and Gitea relay imports on feat/mars-victoriametrics.
Commit: https://git.mgaction.town/darman/homelab/commit/dc037312b3d878259781413b6c39c3f885f1668e
Validation: git diff --check passed; no conflict markers remain. Nix evaluation was unavailable because nix is not installed; Python discovery reported no tests.
Addressed #1 (comment): resolved the new master conflict by keeping the VictoriaMetrics import and removing the deleted Gitea relay import.
Commit: fb1226f9b5
Validation: filter test suites passed, Python compilation passed, git diff --check passed; Nix evaluation was unavailable because nix is not installed.
Addressed https://git.mgaction.town/darman/homelab/pulls/1#issuecomment-125: resolved the new master conflict by keeping the VictoriaMetrics import and removing the deleted Gitea relay import.
Commit: https://git.mgaction.town/darman/homelab/commit/fb1226f9b5a4e17bc4566c9cf7c8015ab1a0a22b
Validation: filter test suites passed, Python compilation passed, git diff --check passed; Nix evaluation was unavailable because nix is not installed.
Implemented review requests in commit https://git.mgaction.town/darman/homelab/commit/5764e6c: changed the VictoriaMetrics scrape interval from 60s to 5s and added terra.orbit.sol:9100 with host=terra. Resolved comments 131 and 132. No comments were deliberately left unaddressed. Validation: git diff --check passed; Nix evaluation was unavailable because nix is not installed in this environment.
Implemented review requests in commit https://git.mgaction.town/darman/homelab/commit/5764e6c: changed the VictoriaMetrics scrape interval from 60s to 5s and added terra.orbit.sol:9100 with host=terra. Resolved comments 131 and 132. No comments were deliberately left unaddressed. Validation: git diff --check passed; Nix evaluation was unavailable because nix is not installed in this environment.
Addressed #1 (comment): moved VictoriaMetrics from mars to jupiter, including the local scrape/self-metrics labels.\n\nCommit: dfe8504402\nValidation: git diff --check passed; Nix evaluation was unavailable because nix is not installed on this runner.
Addressed https://git.mgaction.town/darman/homelab/pulls/1#issuecomment-136: moved VictoriaMetrics from mars to jupiter, including the local scrape/self-metrics labels.\n\nCommit: https://git.mgaction.town/darman/homelab/commit/dfe8504402b887a2457be33f32b66facc0974b8e\nValidation: git diff --check passed; Nix evaluation was unavailable because nix is not installed on this runner.
Addressed review comment 139: the remote node-exporter target now points to mars.orbit.sol:9100 and is labelled mars.\n\nCommit: b6aec1e307\n\nComments 131 and 132 were deliberately not acted on because they were already resolved. Validation: git diff --check passed; nix eval could not run because nix is not installed on this worker.
Addressed review comment 139: the remote node-exporter target now points to mars.orbit.sol:9100 and is labelled mars.\n\nCommit: https://git.mgaction.town/darman/homelab/commit/b6aec1e307c8c19a9e7b38e9eb18e57e0a9b9616\n\nComments 131 and 132 were deliberately not acted on because they were already resolved. Validation: git diff --check passed; nix eval could not run because nix is not installed on this worker.
Review pass. The module evaluates clean on jupiter and checkConfig produces a validated checked-config derivation, so the scrape config is verified at build time. All five scrape targets do run node-exporter and tailscale, and the firewall reasoning in the header comment checks out (8428 is not in jupiter's allowedTCPPorts; trustedInterfaces is ["tailscale0" "lo"]).
Two blocking-ish items inline, plus one housekeeping note:
Title and body are stale. The title still says mars:, but dfe8504 moved this to jupiter and the diff only touches hosts/jupiter/. The body says Mars, says 30 days (it is 15d since 082cbaf), and does not mention terra. If this gets merged from the web UI, the wrong title lands on master — please refresh both.
Also worth noting, not blocking: nothing consumes these metrics yet (no Grafana, no Caddy vhost), so the header comment's "for the homelab dashboard" is aspirational for now. Fine as a follow-up.
Review pass. The module evaluates clean on jupiter and `checkConfig` produces a validated `checked-config` derivation, so the scrape config is verified at build time. All five scrape targets do run node-exporter and tailscale, and the firewall reasoning in the header comment checks out (8428 is not in jupiter's `allowedTCPPorts`; `trustedInterfaces` is `["tailscale0" "lo"]`).
Two blocking-ish items inline, plus one housekeeping note:
**Title and body are stale.** The title still says `mars:`, but `dfe8504` moved this to jupiter and the diff only touches `hosts/jupiter/`. The body says Mars, says 30 days (it is `15d` since `082cbaf`), and does not mention terra. If this gets merged from the web UI, the wrong title lands on master — please refresh both.
Also worth noting, not blocking: nothing consumes these metrics yet (no Grafana, no Caddy vhost), so the header comment's "for the homelab dashboard" is aspirational for now. Fine as a follow-up.
This puts the TSDB on jupiter's eMMC, which I think is the one real blocker here.
The module hardcodes -storageDataPath=/var/lib/${cfg.stateDir} and runs with DynamicUser = true, so the data actually lands in /var/lib/private/victoriametrics on the 29G OS disk. The stateDir option cannot help — it is always relative to /var/lib/.
Every other stateful service on jupiter lives on the array under /mnt/data/AppData. More to the point, services/media/prowlarr.nix:32 documents this exact trap from the last time it bit this repo:
systemd derives RequiresMountsFor from the unit's own paths, which here is only /var/lib/prowlarr on the eMMC — so without this prowlarr starts happily with the array absent and writes its state onto the 29G OS disk.
Same shape here, same DynamicUser -> /var/lib/private/ path, so the existing idiom drops straight in:
Please also add the RequiresMountsFor pin that prowlarr.nix calls out — that is the half people forget, and without it this silently falls back to the eMMC whenever the array is not assembled.
This matters more than usual right now: 6 targets at a 5s interval is a continuous small-write workload aimed at the box's wear-limited disk, which was at 85% full earlier today.
This puts the TSDB on jupiter's eMMC, which I think is the one real blocker here.
The module hardcodes `-storageDataPath=/var/lib/${cfg.stateDir}` and runs with `DynamicUser = true`, so the data actually lands in `/var/lib/private/victoriametrics` on the 29G OS disk. The `stateDir` option cannot help — it is always relative to `/var/lib/`.
Every other stateful service on jupiter lives on the array under `/mnt/data/AppData`. More to the point, `services/media/prowlarr.nix:32` documents this exact trap from the last time it bit this repo:
> systemd derives RequiresMountsFor from the unit's own paths, which here is only `/var/lib/prowlarr` on the eMMC — so without this prowlarr starts happily with the array absent and writes its state onto the 29G OS disk.
Same shape here, same `DynamicUser` -> `/var/lib/private/` path, so the existing idiom drops straight in:
```nix
fileSystems."/var/lib/private/victoriametrics" = {
device = "/mnt/data/AppData/victoriametrics";
fsType = "none";
options = [ "bind" "nofail" ];
};
```
Please also add the `RequiresMountsFor` pin that prowlarr.nix calls out — that is the half people forget, and without it this silently falls back to the eMMC whenever the array is not assembled.
This matters more than usual right now: 6 targets at a 5s interval is a continuous small-write workload aimed at the box's wear-limited disk, which was at 85% full earlier today.
With no scrape_timeout set, this inherits the 10s default — which is longer than the 5s interval. VictoriaMetrics resolves that by clamping the timeout down to the interval, so it silently becomes 5s.
That is fine for the wired hosts, but mercury is a Pi scraped over the tailnet. Anything slower than 5s becomes a gap in the series rather than a late sample, and you will not get an obvious error for it — just missing points.
Either set an explicit scrape_timeout, or give mercury its own job with a longer interval and leave 5s for the rest.
With no `scrape_timeout` set, this inherits the 10s default — which is longer than the 5s interval. VictoriaMetrics resolves that by clamping the timeout down to the interval, so it silently becomes 5s.
That is fine for the wired hosts, but mercury is a Pi scraped over the tailnet. Anything slower than 5s becomes a gap in the series rather than a late sample, and you will not get an obvious error for it — just missing points.
Either set an explicit `scrape_timeout`, or give mercury its own job with a longer interval and leave 5s for the rest.
Addressed comment 143: bind-mounted VictoriaMetrics state to /mnt/data/AppData/victoriametrics and added RequiresMountsFor=/mnt/data.
Addressed comment 144: moved Mercury to its own scrape job at 15s with a 10s timeout; the other node-exporter targets remain at 5s.
Refreshed the PR title and body to describe Jupiter, Terra, 15-day retention, and the current validation status.
Deliberately left the dashboard/Grafana/Caddy note as a follow-up because it was marked non-blocking and no consumer was requested in this change.
Validation: git diff --check passed. Nix evaluation could not run because nix is unavailable on this runner. No deployment was performed.
Implemented and pushed commit [eec17b77df1f3e305d5f1b41d957a350e3502cea](https://git.mgaction.town/darman/homelab/commit/eec17b77df1f3e305d5f1b41d957a350e3502cea).
- Addressed comment 143: bind-mounted VictoriaMetrics state to `/mnt/data/AppData/victoriametrics` and added `RequiresMountsFor=/mnt/data`.
- Addressed comment 144: moved Mercury to its own scrape job at 15s with a 10s timeout; the other node-exporter targets remain at 5s.
- Refreshed the PR title and body to describe Jupiter, Terra, 15-day retention, and the current validation status.
- Deliberately left the dashboard/Grafana/Caddy note as a follow-up because it was marked non-blocking and no consumer was requested in this change.
Validation: `git diff --check` passed. Nix evaluation could not run because nix is unavailable on this runner. No deployment was performed.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
darman
Summary
/mnt/data/AppData/victoriametrics, with an explicit mount dependencyValidation
git diff --checkpassednixexecutablechange retentionPeriod to "15d"
Addressed #1 (comment): changed VictoriaMetrics retentionPeriod to "15d".\n\nCommit:
082cbaff2a\nValidation: git diff --check passed; Nix evaluation was unavailable because nix is not installed on this runner.It seems there are conflicts with master, resolve them please.
Addressed #1 (comment): resolved the master conflict by retaining both the VictoriaMetrics and Gitea relay imports on feat/mars-victoriametrics.
Commit:
dc037312b3Validation: git diff --check passed; no conflict markers remain. Nix evaluation was unavailable because nix is not installed; Python discovery reported no tests.
More conflicts, resolve again
Addressed #1 (comment): resolved the new master conflict by keeping the VictoriaMetrics import and removing the deleted Gitea relay import.
Commit:
fb1226f9b5Validation: filter test suites passed, Python compilation passed, git diff --check passed; Nix evaluation was unavailable because nix is not installed.
@@ -0,0 +14,4 @@listenAddress = ":8428";prometheusConfig = {global.scrape_interval = "60s";reduce interval to 5s
@@ -0,0 +19,4 @@scrape_configs = [{job_name = "node-exporter";static_configs = [add terra here as well
Implemented review requests in commit https://git.mgaction.town/darman/homelab/commit/5764e6c: changed the VictoriaMetrics scrape interval from 60s to 5s and added terra.orbit.sol:9100 with host=terra. Resolved comments 131 and 132. No comments were deliberately left unaddressed. Validation: git diff --check passed; Nix evaluation was unavailable because nix is not installed in this environment.
move vicotriametrics to jupiter
Addressed #1 (comment): moved VictoriaMetrics from mars to jupiter, including the local scrape/self-metrics labels.\n\nCommit:
dfe8504402\nValidation: git diff --check passed; Nix evaluation was unavailable because nix is not installed on this runner.@@ -0,0 +26,4 @@}{targets = [ "jupiter.orbit.sol:9100" ];labels.host = "jupiter";this should point to mars now
Addressed review comment 139: the remote node-exporter target now points to mars.orbit.sol:9100 and is labelled mars.\n\nCommit:
b6aec1e307\n\nComments 131 and 132 were deliberately not acted on because they were already resolved. Validation: git diff --check passed; nix eval could not run because nix is not installed on this worker.Review pass. The module evaluates clean on jupiter and
checkConfigproduces a validatedchecked-configderivation, so the scrape config is verified at build time. All five scrape targets do run node-exporter and tailscale, and the firewall reasoning in the header comment checks out (8428 is not in jupiter'sallowedTCPPorts;trustedInterfacesis["tailscale0" "lo"]).Two blocking-ish items inline, plus one housekeeping note:
Title and body are stale. The title still says
mars:, butdfe8504moved this to jupiter and the diff only toucheshosts/jupiter/. The body says Mars, says 30 days (it is15dsince082cbaf), and does not mention terra. If this gets merged from the web UI, the wrong title lands on master — please refresh both.Also worth noting, not blocking: nothing consumes these metrics yet (no Grafana, no Caddy vhost), so the header comment's "for the homelab dashboard" is aspirational for now. Fine as a follow-up.
@@ -0,0 +10,4 @@{services.victoriametrics = {enable = true;retentionPeriod = "15d";This puts the TSDB on jupiter's eMMC, which I think is the one real blocker here.
The module hardcodes
-storageDataPath=/var/lib/${cfg.stateDir}and runs withDynamicUser = true, so the data actually lands in/var/lib/private/victoriametricson the 29G OS disk. ThestateDiroption cannot help — it is always relative to/var/lib/.Every other stateful service on jupiter lives on the array under
/mnt/data/AppData. More to the point,services/media/prowlarr.nix:32documents this exact trap from the last time it bit this repo:Same shape here, same
DynamicUser->/var/lib/private/path, so the existing idiom drops straight in:Please also add the
RequiresMountsForpin that prowlarr.nix calls out — that is the half people forget, and without it this silently falls back to the eMMC whenever the array is not assembled.This matters more than usual right now: 6 targets at a 5s interval is a continuous small-write workload aimed at the box's wear-limited disk, which was at 85% full earlier today.
@@ -0,0 +14,4 @@listenAddress = ":8428";prometheusConfig = {global.scrape_interval = "5s";With no
scrape_timeoutset, this inherits the 10s default — which is longer than the 5s interval. VictoriaMetrics resolves that by clamping the timeout down to the interval, so it silently becomes 5s.That is fine for the wired hosts, but mercury is a Pi scraped over the tailnet. Anything slower than 5s becomes a gap in the series rather than a late sample, and you will not get an obvious error for it — just missing points.
Either set an explicit
scrape_timeout, or give mercury its own job with a longer interval and leave 5s for the rest.mars: add VictoriaMetrics monitoringto jupiter: add VictoriaMetrics monitoringImplemented and pushed commit eec17b77df1f3e305d5f1b41d957a350e3502cea.
/mnt/data/AppData/victoriametricsand addedRequiresMountsFor=/mnt/data.Validation:
git diff --checkpassed. Nix evaluation could not run because nix is unavailable on this runner. No deployment was performed.