darman/homelab
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

jupiter: add VictoriaMetrics monitoring #1

Merged
darman merged 8 commits from feat/mars-victoriametrics into master 2026-08-24 04:48:46 +02:00
Conversation 18 Commits 8 Files Changed 2
+59
2 changed files with 59 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit ea9be6fb8a - Show all commits
hosts/mars/configuration.nix
+1
View File
@@ -12,6 +12,7 @@
../../services/containers.nix ../../services/containers.nix
../../services/vpn/tailscale.nix ../../services/vpn/tailscale.nix
../../services/monitoring/node-exporter.nix ../../services/monitoring/node-exporter.nix
../../services/monitoring/victoriametrics.nix
]; ];
networking.hostName = "mars"; networking.hostName = "mars";
services/monitoring/victoriametrics.nix
+58
View File
@@ -0,0 +1,58 @@
{ ... }:
# VictoriaMetrics single-node store for the homelab dashboard. It listens on
# all interfaces, but tailscale.nix makes tailscale0 the only trusted ingress;
# the host firewall therefore keeps :8428 off the LAN and public interfaces.
#
# The scrape targets are the node_exporter instances enabled by
# services/monitoring/node-exporter.nix on every real host. MagicDNS names use
# the tailnet's orbit.sol suffix (see services/vpn/headscale.nix).
{
services.victoriametrics = {
enable = true;
retentionPeriod = "30d";
luna marked this conversation as resolved
darman commented 2026-08-24 04:02:42 +02:00
Review
Copy Link
Copy Source

This puts the TSDB on jupiter's eMMC, which I think is the one real blocker here.

The module hardcodes -storageDataPath=/var/lib/${cfg.stateDir} and runs with DynamicUser = true, so the data actually lands in /var/lib/private/victoriametrics on the 29G OS disk. The stateDir option cannot help — it is always relative to /var/lib/.

Every other stateful service on jupiter lives on the array under /mnt/data/AppData. More to the point, services/media/prowlarr.nix:32 documents this exact trap from the last time it bit this repo:

systemd derives RequiresMountsFor from the unit's own paths, which here is only /var/lib/prowlarr on the eMMC — so without this prowlarr starts happily with the array absent and writes its state onto the 29G OS disk.

Same shape here, same DynamicUser -> /var/lib/private/ path, so the existing idiom drops straight in:

fileSystems."/var/lib/private/victoriametrics" = {
  device = "/mnt/data/AppData/victoriametrics";
  fsType = "none";
  options = [ "bind" "nofail" ];
};

Please also add the RequiresMountsFor pin that prowlarr.nix calls out — that is the half people forget, and without it this silently falls back to the eMMC whenever the array is not assembled.

This matters more than usual right now: 6 targets at a 5s interval is a continuous small-write workload aimed at the box's wear-limited disk, which was at 85% full earlier today.

This puts the TSDB on jupiter's eMMC, which I think is the one real blocker here. The module hardcodes `-storageDataPath=/var/lib/${cfg.stateDir}` and runs with `DynamicUser = true`, so the data actually lands in `/var/lib/private/victoriametrics` on the 29G OS disk. The `stateDir` option cannot help — it is always relative to `/var/lib/`. Every other stateful service on jupiter lives on the array under `/mnt/data/AppData`. More to the point, `services/media/prowlarr.nix:32` documents this exact trap from the last time it bit this repo: > systemd derives RequiresMountsFor from the unit's own paths, which here is only `/var/lib/prowlarr` on the eMMC — so without this prowlarr starts happily with the array absent and writes its state onto the 29G OS disk. Same shape here, same `DynamicUser` -> `/var/lib/private/` path, so the existing idiom drops straight in: ```nix fileSystems."/var/lib/private/victoriametrics" = { device = "/mnt/data/AppData/victoriametrics"; fsType = "none"; options = [ "bind" "nofail" ]; }; ``` Please also add the `RequiresMountsFor` pin that prowlarr.nix calls out — that is the half people forget, and without it this silently falls back to the eMMC whenever the array is not assembled. This matters more than usual right now: 6 targets at a 5s interval is a continuous small-write workload aimed at the box's wear-limited disk, which was at 85% full earlier today.
listenAddress = ":8428";
prometheusConfig = {
global.scrape_interval = "60s";
luna marked this conversation as resolved
darman commented 2026-08-24 04:02:42 +02:00
Review
Copy Link
Copy Source

With no scrape_timeout set, this inherits the 10s default — which is longer than the 5s interval. VictoriaMetrics resolves that by clamping the timeout down to the interval, so it silently becomes 5s.

That is fine for the wired hosts, but mercury is a Pi scraped over the tailnet. Anything slower than 5s becomes a gap in the series rather than a late sample, and you will not get an obvious error for it — just missing points.

Either set an explicit scrape_timeout, or give mercury its own job with a longer interval and leave 5s for the rest.

With no `scrape_timeout` set, this inherits the 10s default — which is longer than the 5s interval. VictoriaMetrics resolves that by clamping the timeout down to the interval, so it silently becomes 5s. That is fine for the wired hosts, but mercury is a Pi scraped over the tailnet. Anything slower than 5s becomes a gap in the series rather than a late sample, and you will not get an obvious error for it — just missing points. Either set an explicit `scrape_timeout`, or give mercury its own job with a longer interval and leave 5s for the rest.
scrape_configs = [
{
job_name = "node-exporter";
static_configs = [
luna marked this conversation as resolved
darman commented 2026-08-24 03:29:58 +02:00
Review
Copy Link
Copy Source

add terra here as well

add terra here as well
{
targets = [ "127.0.0.1:9100" ];
labels.host = "mars";
}
{
targets = [ "jupiter.orbit.sol:9100" ];
labels.host = "jupiter";
}
{
targets = [ "neptun.orbit.sol:9100" ];
labels.host = "neptun";
}
{
targets = [ "mercury.orbit.sol:9100" ];
labels.host = "mercury";
}
];
}
{
job_name = "victoriametrics";
static_configs = [
{
targets = [ "127.0.0.1:8428" ];
labels.host = "mars";
}
];
}
];
};
};
# Start after Tailscale has had a chance to establish MagicDNS. This is only
# ordering, not a hard dependency: VictoriaMetrics still starts locally if
# another host or the tailnet is temporarily unavailable.
systemd.services.victoriametrics.after = [ "tailscaled-autoconnect.service" ];
}