Where We Left Off Part 19 was two and a half weeks of rebuilding nixos2 on new hardware after the old box started screaming at me. Big post, lots of gaps found, a punch list at the end. This one was supposed to be small. One update, one service, could be done before I ran to lunch.   Could build a new ISO later. It ended up taking lunch and getting wrapped up later in the evening.  The One Update I Actually Planned Gitea 28.0.0 landed at the end of September. It's a big release: they dropped the historical 1. prefix, so it's 28.0.0 instead of 1.28.0, and added audit logging, bot accounts, HTTPS deploy tokens and an Actions queue view. More to the point for me, it shipped with a long list of security fixes, a good chunk of them around mirrors, migrations and egress. Since my Gitea instance mirrors out to Codeberg, that's not a section of the release notes I wanted to sit on. nixos2 already pulls Gitea from unstable through an overlay (that's the version-mismatch fix from Part 19), so in theory this was: run the usual update, let it pick up 28, check the mirrors, move on with my day. So I ran update. Detour One: Nextcloud Hits End of Life The first box I hit was nixos, and it didn't make it past evaluation. Not because of Gitea. Because of Nextcloud: error: Refusing to evaluate package 'nextcloud-32.0.15' ... because it is marked as insecure Known issues: - Nextcloud version 32.0.15 is EOL Fair enough. Per Nextcloud's maintenance schedule, 32 was only supported through the end of September, and nixpkgs on 26.05 marks EOL Nextcloud versions as insecure, so it flat-out refuses to build them. The error helpfully offers permittedInsecurePackages as an escape hatch, and for a box that's LAN and Tailscale only, it would have worked. It's also exactly the kind of "temporary" fix that's still sitting in my config a year from now. Not doing that. The real fix is to move up a major version. 26.05 ships nextcloud33 and nextcloud34, but Nextcloud only supports upgrading one major at a time, so it's two hops: 32 → 33, rebuild, check it came up clean, then 33 → 34. services.nextcloud.package = pkgs.nextcloud33; The NixOS Nextcloud module handles the occ upgrade itself through nextcloud-setup.service, so in theory it's just a package bump and a rebuild. In practice, nextcloud-setup failed. And the store path in the failure was nextcloud-32.0.11. Not 33. Not even the 32.0.15 from the original error. 32.0.11 is older than what I started with. That was the first clue something else was going on: the rebuild had run out of disk space partway through. My rebuild script forces a reboot at the end, so I came back up on an older generation entirely, the one from before any of this started. I ran my usual cleanup, garbage collected, freed up space, and went to try again. Detour Two: The Disk That Kept Filling Up After cleanup, df showed 75GB free on a 457GB drive. Fine. I went back to fixing Nextcloud, got it through both hops, landed on 34.0.4, occ status happy, maintenance mode off. Then I bumped home-manager (more on that in a second) and the rebuild ran out of space again. A home-manager bump doesn't eat 50GB. Something else was going on. First suspect: /tmp being a RAM disk, so builds hit "no space" even with plenty of room on /. Nope, findmnt /tmp came back empty; it's on the real disk. Second suspect: the Nix store. Also nope: 21G /nix 17G /var 376G /home 376GB in /home. On a server. Where I don't really do anything in /home. A quick look for anything big that had changed recently found it immediately: sudo find /home -xdev -type f -size +500M -mmin -360 -printf '%s\t%TT\t%p\n' | sort -n | tail -20 Three .syncthing.*.tmp files, each 17 to 18GB, sitting in ~/temp. Partial Syncthing transfers of some very large backup files. About 54GB being pulled in, which lined up almost exactly with the space I'd "lost." And that wasn't even the big one: 29G /home/ppb1701/temp/current 71G /home/ppb1701/temp/archive 187G /home/ppb1701/temp/.stversions 187GB of .stversions. If you haven't run into it, that's Syncthing's file versioning. When a file gets replaced or deleted on another device, the old copy gets moved into .stversions instead of just being thrown away. Which is a great feature for documents. For backup files that get regenerated at 20 to 30GB a pop, it means every new backup also shoves the previous one into an archive that nothing was cleaning out. Each refresh: another stack of old copies. The kicker: I set that share up a long time ago for something I'd mostly stopped doing months back. The use case dropped off. The share didn't. It just kept quietly pulling every backup that landed in that folder on my main machine and keeping every copy it replaced. No elaborate fix here. I removed the share, deleted the folder, and went from 91% full to this: Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p2 457G 39G 396G 9% If I bring it back, it'll be a much narrower share, on the 5.8TB drive that's sitting at 5% used instead of the system NVMe, with the big disposable stuff excluded in .stignore from day one and versioning capped with something like trashcan and a short cleanoutDays. The machine that actually needs all those files has plenty of room for them. The server doesn't need to be a second copy of everything. A small side note on the debugging itself: several of the commands I was pasting in had trailing # comment notes on them, and my shell happily passed #, want, and 33.x to grep as filenames. Strip the comments before pasting. Lesson relearned. Detour Three: The Home Manager Pin, Again While all this was going on, the nixos rebuild log had been quietly telling me something else: unpacking 'https://github.com/nix-community/home-manager/archive/release-25.11.tar.gz' into the Git cache nixos has been on 26.05 since Part 18. Home-manager was still pinned to release-25.11. I had to go back and reread Part 19 to remember why it was pinned at all. The answer: home-manager had been on master, master drifted incompatible with stable nixpkgs 25.11, and the fix was to pin home-manager to the matching release branch. I did that everywhere: nixos, nixos2, the ISO config. Which was correct at the time. Then nixos moved to 26.05 and the pin stayed at 25.11, now drifted the other direction. The pin was never really "use 25.11." It was "match the system channel." I wrote down the value and lost the reason. Bumped it to release-26.05, rebuilt (with actual disk space this time), and checked that it took: nix-instantiate --eval '<nixpkgs/nixos>' -A config.home-manager.users.ppb1701.home.version.release "26.05". Good. Then nixos2. Same eval came back "26.05", and then: ❯ nixos-version 25.11.12484.b6018f87da91 (Xantusia) So, uhhhhhh. nixos2 was still on 25.11 entirely, which was well past its support window by now anyway. The channel list already said 26.05. Turns out nix-channel --list shows what a channel points at, not what's actually been downloaded, so the real check is: cat /nix/var/nix/profiles/per-user/root/channels/nixos/.version It took a few loops of rebuilding to get nixos2 onto 26.05, with services hanging on stop or restart and the switch backing out more than once, but it eventually stuck. Grepping both repos for anything still saying 25.11 also turned up the line that would have brought all of this back on the next fresh install: install-nixos.sh was still telling me to add the release-25.11 home-manager channel. Fixed that, cleaned up a couple of stale comments claiming Collabora needs unstable (it's in 26.05 stable now), updated the ISO builder VM, and built a fresh ISO. Finally, Gitea Here's the part I actually came for. systemctl status gitea --no-pager | grep -o 'gitea-[0-9][0-9.]*' | head -1 gitea-28.0.0 (The first time, I tried nix-shell -p gitea --version, which very confidently told me I was running Nix 2.34.8. --version went to nix-shell, not Gitea. Same trick as Part 18 applies: Gitea's a systemd service and isn't on $PATH, so ask the service what it's running.) 28.0.0, from the unstable overlay. For comparison, stable 26.05 ships: nix-instantiate --eval '<nixpkgs>' -A gitea.version "1.27.3" So the overlay stays. My database has already migrated to 28's schema, and an older binary will just refuse to start, which is the exact same migration-version wall from Part 19. Since the security fixes and the biggest breaking change in this release are both about Git network operations (they now go through an internal proxy with new egress rules), I wanted to make sure mirroring still worked before calling it done. On a mirrored repo, Settings → Synchronize now: the timestamp updated. Mirrors survived the new proxy. Then I noticed what actually happened. I got 28 because the overlay pulls whatever nixos-unstable has at that moment. This time that's what I wanted. Next time it might be a major version I haven't read the breaking changes for yet, showing up in the middle of a rebuild I was doing for a completely different reason. Kind of like Nextcloud just did to me. So the overlay is now pinned to a specific nixpkgs commit instead of the moving tarball: REV=$(curl -sL https://channels.nixos.org/nixos-unstable/git-revision) nix-prefetch-url --unpack https://github.com/NixOS/nixpkgs/archive/$REV.tar.gz nix-instantiate --eval -E "(import (fetchTarball \"https://github.com/NixOS/nixpkgs/archive/$REV.tar.gz\") {}).gitea.version" The last command checks what Gitea version that commit actually carries before touching the config, so I can't accidentally pin myself to something older than my database. It said 28.0.0. Then, in system.nix: nixpkgs.overlays = [ (final: prev: { # Gitea pinned to a specific nixpkgs-unstable commit (28.0.0). # Stable 26.05 ships 1.27.x; DB is already migrated to 28, so no downgrading. # To upgrade on purpose: # REV=$(curl -sL https://channels.nixos.org/nixos-unstable/git-revision) # nix-prefetch-url --unpack https://github.com/NixOS/nixpkgs/archive/$REV.tar.gz # check: nix-instantiate --eval -E "(import (fetchTarball \"https://github.com/NixOS/nixpkgs/archive/$REV.tar.gz\") {}).gitea.version" # Drop this overlay once stable ships 28.x (likely 26.11). gitea = (import (fetchTarball { url = "https://github.com/NixOS/nixpkgs/archive/151fa4e8ddfdd8dd25d945ad94ed54a13de9f6e4.tar.gz"; sha256 = "0rm5v6n0kgqq5xrc34imn8nxx0y2xlmhpa0nag9sk6m6ys9slaij"; }) { system = prev.system; }).gitea; }) ]; Same idea as the nurl comment in the Vaultwarden block: the upgrade steps live right above the thing they upgrade, so future me finds them in the repo instead of in my notes. Gitea now changes version when I change those two lines, and not otherwise. 28.1.0 is already out, as it happens. I'll get to it on purpose. So the one update I actually planned is the one I ended up locking down. I'm choosing to find that funny. Lessons Learned A pin outlives its reason. release-25.11 was right when I set it, because it matched the system channel. The day the channel moved, it became wrong, and nothing said so until a rebuild log mentioned it in passing. Leave the why next to the value, not just the value. An unpinned overlay is an upgrade you never agreed to. Tracking unstable for one package is fine. Tracking a moving unstable means a major version, schema migration and all, can show up during any rebuild. Pin it to a commit and bump it deliberately. EOL warnings in nixpkgs are a feature. Annoying in the moment, but the refusal to build Nextcloud 32 is exactly why I'm on a supported version instead of finding out about it later. "Out of disk space" during a rebuild isn't necessarily Nix's fault. The store was 21GB. Look at what's actually using the space before assuming garbage collection is the answer. Abandoned sync shares don't stop syncing just because you stopped using them. And with versioning on, they don't just keep up; they keep everything. If a share's purpose has dropped off, remove the share, not just the habit. nix-channel --list tells you where a channel points, not what you have. The .version file in the channel profile is the actual answer. Both servers are on 26.05, home-manager matches everywhere including the install script, Nextcloud is on 34, Gitea's on 28 and pinned, the ISO is rebuilt, and the system drive went from 91% to 9%. Not bad for an update I expected to take twenty minutes. I've stopped making time estimates in this series. That seems like the real lesson. ← Back to Series The full configs are on Codeberg: nixos and nixos2. If you're following along on Mastodon: @ppb1701@ppb.social