Add SD image pipeline, documentation overhaul, and fix module issues

- Add automatic SD image builds for Raspberry Pi via Forgejo Actions - Enable binfmt emulation on cryodev-main for aarch64 cross-builds - Add sd-image.nix module to cryodev-pi configuration - Create comprehensive docs/ structure with installation guides - Split installation docs into: first-install (server), reinstall, new-client (Pi) - Add lib/utils.nix and apps/rebuild from synix - Fix headplane module for new upstream API (tale/headplane) - Fix various module issues (mailserver stateVersion, option conflicts) - Add placeholder secrets.yaml files for both hosts - Remove old INSTRUCTIONS.md (content moved to docs/)
2026-03-11 08:41:58 +01:00 · 2026-03-11 08:41:58 +01:00 · 5ba78886d2
commit 5ba78886d2
parent a5261d8ff0
44 changed files with 3570 additions and 609 deletions
--- a/docs/services/forgejo.md
+++ b/docs/services/forgejo.md
@ -0,0 +1,149 @@
+# Forgejo
+
+Forgejo is a self-hosted Git service (fork of Gitea) with built-in CI/CD Actions.
+
+## References
+
+- [Forgejo Documentation](https://forgejo.org/docs/)
+- [Forgejo Actions](https://forgejo.org/docs/latest/user/actions/)
+
+## Setup
+
+### DNS
+
+Set a CNAME record for `git.cryodev.xyz` pointing to your main domain.
+
+### Configuration
+
+```nix
+# hosts/cryodev-main/services/forgejo.nix
+{ config, ... }:
+{
+  services.forgejo = {
+    enable = true;
+    settings = {
+      server = {
+        DOMAIN = "git.cryodev.xyz";
+        ROOT_URL = "https://git.cryodev.xyz";
+      };
+      mailer = {
+        ENABLED = true;
+        FROM = "forgejo@cryodev.xyz";
+      };
+    };
+  };
+}
+```
+
+## Forgejo Runner
+
+The runner executes CI/CD pipelines defined in `.forgejo/workflows/`.
+
+### Get Runner Token
+
+1. Go to Forgejo Admin Panel
+2. Navigate to Actions > Runners
+3. Create a new runner and copy the token
+
+### Add to Secrets
+
+```bash
+sops hosts/cryodev-main/secrets.yaml
+```
+
+```yaml
+forgejo-runner:
+  token: "your-runner-token"
+```
+
+### Configuration
+
+```nix
+{
+  sops.secrets."forgejo-runner/token" = { };
+  
+  services.gitea-actions-runner = {
+    instances.default = {
+      enable = true;
+      url = "https://git.cryodev.xyz";
+      tokenFile = config.sops.secrets."forgejo-runner/token".path;
+      labels = [ "ubuntu-latest:docker://node:20" ];
+    };
+  };
+}
+```
+
+## CI/CD Workflows
+
+### deploy-rs Workflow
+
+`.forgejo/workflows/deploy.yaml`:
+
+```yaml
+name: Deploy
+on:
+  push:
+    branches: [main]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Install Nix
+        uses: cachix/install-nix-action@v24
+        
+      - name: Deploy
+        env:
+          SSH_PRIVATE_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
+        run: |
+          mkdir -p ~/.ssh
+          echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_ed25519
+          chmod 600 ~/.ssh/id_ed25519
+          nix run .#deploy
+```
+
+## Administration
+
+### Create Admin User
+
+```bash
+sudo -u forgejo forgejo admin user create \
+  --username admin \
+  --password changeme \
+  --email admin@cryodev.xyz \
+  --admin
+```
+
+### Reset User Password
+
+```bash
+sudo -u forgejo forgejo admin user change-password \
+  --username USER \
+  --password NEWPASS
+```
+
+## Troubleshooting
+
+### Check Service Status
+
+```bash
+sudo systemctl status forgejo
+sudo systemctl status gitea-runner-default
+```
+
+### View Logs
+
+```bash
+sudo journalctl -u forgejo -f
+sudo journalctl -u gitea-runner-default -f
+```
+
+### Database Issues
+
+Forgejo uses SQLite by default. Database location:
+
+```bash
+ls -la /var/lib/forgejo/data/
+```
--- a/docs/services/headplane.md
+++ b/docs/services/headplane.md
@ -0,0 +1,107 @@
+# Headplane
+
+Headplane is a web-based admin interface for Headscale.
+
+## References
+
+- [GitHub](https://github.com/tale/headplane)
+
+## Setup
+
+### DNS
+
+Set a CNAME record for `headplane.cryodev.xyz` pointing to your main domain.
+
+### Generate Secrets
+
+**Cookie Secret** (for session management):
+
+```bash
+nix-shell -p openssl --run 'openssl rand -hex 16'
+```
+
+**Agent Pre-Auth Key** (for Headplane's built-in agent):
+
+```bash
+# First, create a dedicated user
+sudo headscale users create headplane-agent
+
+# Then create a reusable pre-auth key
+sudo headscale preauthkeys create --expiration 99y --reusable --user headplane-agent
+```
+
+### Add to Secrets
+
+Edit `hosts/cryodev-main/secrets.yaml`:
+
+```bash
+sops hosts/cryodev-main/secrets.yaml
+```
+
+```yaml
+headplane:
+  cookie_secret: "your-generated-hex-string"
+  agent_pre_authkey: "your-preauth-key"
+```
+
+### Configuration
+
+```nix
+# hosts/cryodev-main/services/headplane.nix
+{ config, ... }:
+{
+  sops.secrets."headplane/cookie_secret" = { };
+  sops.secrets."headplane/agent_pre_authkey" = { };
+  
+  services.headplane = {
+    enable = true;
+    settings = {
+      server = {
+        cookie_secret_file = config.sops.secrets."headplane/cookie_secret".path;
+      };
+      headscale = {
+        url = "https://headscale.cryodev.xyz";
+      };
+      agent = {
+        enable = true;
+        authkey_file = config.sops.secrets."headplane/agent_pre_authkey".path;
+      };
+    };
+  };
+}
+```
+
+## Usage
+
+Access Headplane at `https://headplane.cryodev.xyz`.
+
+### Features
+
+- View and manage users
+- View connected nodes
+- Manage routes and exit nodes
+- View pre-auth keys
+
+## Troubleshooting
+
+### Check Service Status
+
+```bash
+sudo systemctl status headplane
+```
+
+### View Logs
+
+```bash
+sudo journalctl -u headplane -f
+```
+
+### Agent Not Connecting
+
+Verify the agent pre-auth key is valid:
+
+```bash
+sudo headscale preauthkeys list --user headplane-agent
+```
+
+If expired, create a new one and update the secrets file.
--- a/docs/services/headscale.md
+++ b/docs/services/headscale.md
@ -0,0 +1,116 @@
+# Headscale
+
+Headscale is an open-source, self-hosted implementation of the Tailscale control server.
+
+## References
+
+- [Website](https://headscale.net/stable/)
+- [GitHub](https://github.com/juanfont/headscale)
+- [Example configuration](https://github.com/juanfont/headscale/blob/main/config-example.yaml)
+
+## Setup
+
+### DNS
+
+Set a CNAME record for `headscale.cryodev.xyz` pointing to your main domain.
+
+### Configuration
+
+```nix
+# hosts/cryodev-main/services/headscale.nix
+{
+  services.headscale = {
+    enable = true;
+    openFirewall = true;
+  };
+}
+```
+
+## Usage
+
+### Create a User
+
+```bash
+sudo headscale users create <USERNAME>
+```
+
+### List Users
+
+```bash
+sudo headscale users list
+```
+
+### Create Pre-Auth Key
+
+```bash
+sudo headscale preauthkeys create --expiration 99y --reusable --user <USER_ID>
+```
+
+The pre-auth key is used by clients to automatically authenticate and join the tailnet.
+
+### List Nodes
+
+```bash
+sudo headscale nodes list
+```
+
+### Delete a Node
+
+```bash
+sudo headscale nodes delete -i <NODE_ID>
+```
+
+### Rename a Node
+
+```bash
+sudo headscale nodes rename -i <NODE_ID> new-name
+```
+
+## ACL Configuration
+
+Access Control Lists define which nodes can communicate with each other.
+
+### Validate ACL File
+
+```bash
+sudo headscale policy check --file /path/to/acl.hujson
+```
+
+### Example ACL
+
+```json
+{
+  "acls": [
+    {
+      "action": "accept",
+      "src": ["*"],
+      "dst": ["*:*"]
+    }
+  ]
+}
+```
+
+## Troubleshooting
+
+### Check Service Status
+
+```bash
+sudo systemctl status headscale
+```
+
+### View Logs
+
+```bash
+sudo journalctl -u headscale -f
+```
+
+### Test DERP Connectivity
+
+```bash
+curl -I https://headscale.cryodev.xyz/derp
+```
+
+## Integration
+
+- [Headplane](headplane.md) - Web UI for managing Headscale
+- [Tailscale Client](tailscale.md) - Connect clients to Headscale
--- a/docs/services/mailserver.md
+++ b/docs/services/mailserver.md
@ -0,0 +1,147 @@
+# Mailserver
+
+NixOS mailserver module providing a complete email stack with Postfix and Dovecot.
+
+## References
+
+- [Simple NixOS Mailserver](https://gitlab.com/simple-nixos-mailserver/nixos-mailserver)
+
+## Setup
+
+### DNS Records
+
+| Type | Hostname | Value |
+|------|----------|-------|
+| A | `mail` | `<SERVER_IP>` |
+| AAAA | `mail` | `<SERVER_IPV6>` |
+| MX | `@` | `10 mail.cryodev.xyz.` |
+| TXT | `@` | `"v=spf1 mx ~all"` |
+| TXT | `_dmarc` | `"v=DMARC1; p=none"` |
+
+DKIM records are generated automatically after first deployment.
+
+### Generate Password Hashes
+
+```bash
+nix-shell -p mkpasswd --run 'mkpasswd -sm bcrypt'
+```
+
+### Add to Secrets
+
+```bash
+sops hosts/cryodev-main/secrets.yaml
+```
+
+```yaml
+mailserver:
+  accounts:
+    admin: "$2y$05$..."
+    forgejo: "$2y$05$..."
+```
+
+### Configuration
+
+```nix
+# hosts/cryodev-main/services/mailserver.nix
+{ config, ... }:
+{
+  sops.secrets."mailserver/accounts/admin" = { };
+  sops.secrets."mailserver/accounts/forgejo" = { };
+  
+  mailserver = {
+    enable = true;
+    fqdn = "mail.cryodev.xyz";
+    domains = [ "cryodev.xyz" ];
+    
+    loginAccounts = {
+      "admin@cryodev.xyz" = {
+        hashedPasswordFile = config.sops.secrets."mailserver/accounts/admin".path;
+      };
+      "forgejo@cryodev.xyz" = {
+        hashedPasswordFile = config.sops.secrets."mailserver/accounts/forgejo".path;
+        sendOnly = true;
+      };
+    };
+  };
+}
+```
+
+## DKIM Setup
+
+After first deployment, get the DKIM public key:
+
+```bash
+sudo cat /var/dkim/cryodev.xyz.mail.txt
+```
+
+Add this as a TXT record:
+
+| Type | Hostname | Value |
+|------|----------|-------|
+| TXT | `mail._domainkey` | `v=DKIM1; k=rsa; p=...` |
+
+## Testing
+
+### Send Test Email
+
+```bash
+echo "Test" | mail -s "Test Subject" recipient@example.com
+```
+
+### Check Mail Queue
+
+```bash
+sudo postqueue -p
+```
+
+### View Logs
+
+```bash
+sudo journalctl -u postfix -f
+sudo journalctl -u dovecot2 -f
+```
+
+### Test SMTP
+
+```bash
+openssl s_client -connect mail.cryodev.xyz:587 -starttls smtp
+```
+
+### Verify DNS Records
+
+- [MXToolbox](https://mxtoolbox.com/)
+- [Mail-tester](https://www.mail-tester.com/)
+
+## Troubleshooting
+
+### Emails Not Sending
+
+Check Postfix status:
+
+```bash
+sudo systemctl status postfix
+```
+
+Check firewall (ports 25, 465, 587 must be open):
+
+```bash
+sudo iptables -L -n | grep -E '25|465|587'
+```
+
+### DKIM Failing
+
+Verify the DNS record matches the generated key:
+
+```bash
+dig TXT mail._domainkey.cryodev.xyz
+```
+
+### SPF Failing
+
+Verify SPF record:
+
+```bash
+dig TXT cryodev.xyz
+```
+
+Should return: `"v=spf1 mx ~all"`
--- a/docs/services/netdata.md
+++ b/docs/services/netdata.md
@ -0,0 +1,181 @@
+# Netdata Monitoring
+
+Netdata provides real-time performance monitoring with parent/child streaming.
+
+## Architecture
+
+```
+┌─────────────────┐     Stream over      ┌─────────────────┐
+│  cryodev-pi     │  ───────────────────>│  cryodev-main   │
+│  (Child Node)   │     Tailscale VPN    │  (Parent Node)  │
+└─────────────────┘                      └─────────────────┘
+                                                  │
+                                                  v
+                                         https://netdata.cryodev.xyz
+```
+
+## References
+
+- [Netdata Documentation](https://learn.netdata.cloud/)
+- [Streaming Configuration](https://learn.netdata.cloud/docs/streaming/streaming-configuration-reference)
+
+## Parent Node (cryodev-main)
+
+### DNS
+
+Set a CNAME record for `netdata.cryodev.xyz` pointing to your main domain.
+
+### Generate Stream API Key
+
+```bash
+uuidgen
+```
+
+### Configuration
+
+```nix
+# hosts/cryodev-main/services/netdata.nix
+{ config, ... }:
+{
+  sops.secrets."netdata/stream-api-key" = { };
+  
+  sops.templates."netdata-stream.conf" = {
+    content = ''
+      [${config.sops.placeholder."netdata/stream-api-key"}]
+      enabled = yes
+      default history = 3600
+      default memory mode = ram
+      health enabled by default = auto
+      allow from = *
+    '';
+    owner = "netdata";
+  };
+  
+  services.netdata = {
+    enable = true;
+    configDir."stream.conf" = config.sops.templates."netdata-stream.conf".path;
+  };
+}
+```
+
+## Child Node (cryodev-pi)
+
+### Generate Child UUID
+
+```bash
+uuidgen
+```
+
+### Add to Secrets
+
+```bash
+sops hosts/cryodev-pi/secrets.yaml
+```
+
+```yaml
+netdata:
+  stream:
+    child-uuid: "your-generated-uuid"
+```
+
+Note: The stream API key must match the parent's key. You can either:
+1. Share the same secret between hosts (complex with SOPS)
+2. Hardcode a known API key in both configurations
+
+### Configuration
+
+```nix
+# hosts/cryodev-pi/services/netdata.nix
+{ config, constants, ... }:
+{
+  sops.secrets."netdata/stream/child-uuid" = { };
+  
+  sops.templates."netdata-stream.conf" = {
+    content = ''
+      [stream]
+      enabled = yes
+      destination = ${constants.hosts.cryodev-main.ip}:19999
+      api key = YOUR_STREAM_API_KEY
+      send charts matching = *
+    '';
+    owner = "netdata";
+  };
+  
+  services.netdata = {
+    enable = true;
+    configDir."stream.conf" = config.sops.templates."netdata-stream.conf".path;
+  };
+}
+```
+
+## Email Alerts
+
+Configure Netdata to send alerts via the mailserver:
+
+```nix
+{
+  services.netdata.configDir."health_alarm_notify.conf" = pkgs.writeText "notify.conf" ''
+    SEND_EMAIL="YES"
+    EMAIL_SENDER="netdata@cryodev.xyz"
+    DEFAULT_RECIPIENT_EMAIL="admin@cryodev.xyz"
+  '';
+}
+```
+
+## Usage
+
+### Access Dashboard
+
+Open `https://netdata.cryodev.xyz` in your browser.
+
+### View Child Nodes
+
+Child nodes appear in the left sidebar under "Nodes".
+
+### Check Streaming Status
+
+On parent:
+```bash
+curl -s http://localhost:19999/api/v1/info | jq '.hosts'
+```
+
+On child:
+```bash
+curl -s http://localhost:19999/api/v1/info | jq '.streaming'
+```
+
+## Troubleshooting
+
+### Check Service Status
+
+```bash
+sudo systemctl status netdata
+```
+
+### View Logs
+
+```bash
+sudo journalctl -u netdata -f
+```
+
+### Child Not Streaming
+
+1. Verify network connectivity:
+   ```bash
+   tailscale ping cryodev-main
+   nc -zv <parent-ip> 19999
+   ```
+
+2. Check API key matches between parent and child
+
+3. Verify firewall allows port 19999 on parent
+
+### High Memory Usage
+
+Adjust history settings in `netdata.conf`:
+
+```ini
+[global]
+history = 1800  # seconds to retain
+memory mode = ram
+```
--- a/docs/services/sops.md
+++ b/docs/services/sops.md
@ -0,0 +1,174 @@
+# SOPS Secret Management
+
+Atomic secret provisioning for NixOS using [sops-nix](https://github.com/Mic92/sops-nix).
+
+## Overview
+
+Secrets are encrypted with `age` using SSH host keys, ensuring:
+- No plaintext secrets in the repository
+- Secrets are decrypted at activation time
+- Each host can only decrypt its own secrets
+
+## Setup
+
+### 1. Get Host's Age Public Key
+
+After a host is installed, extract its age key from the SSH host key:
+
+```bash
+nix-shell -p ssh-to-age --run 'ssh-keyscan -t ed25519 <HOST_IP> | ssh-to-age'
+```
+
+Or locally on the host:
+
+```bash
+nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
+```
+
+### 2. Configure .sops.yaml
+
+Add the host key to `.sops.yaml`:
+
+```yaml
+keys:
+  - &admin_key age1e8p35795htf7twrejyugpzw0qja2v33awcw76y4gp6acnxnkzq0s935t4t
+  - &main_key age1...  # cryodev-main
+  - &pi_key age1...    # cryodev-pi
+
+creation_rules:
+  - path_regex: hosts/cryodev-main/secrets.yaml$
+    key_groups:
+      - age:
+        - *admin_key
+        - *main_key
+        
+  - path_regex: hosts/cryodev-pi/secrets.yaml$
+    key_groups:
+      - age:
+        - *admin_key
+        - *pi_key
+```
+
+### 3. Create Secrets File
+
+```bash
+sops hosts/<hostname>/secrets.yaml
+```
+
+This opens your editor. Add secrets in YAML format:
+
+```yaml
+tailscale:
+  auth-key: "tskey-..."
+  
+some-service:
+  password: "secret123"
+```
+
+## Usage in Modules
+
+### Declaring Secrets
+
+```nix
+{ config, ... }:
+{
+  sops.secrets.my-secret = {
+    # Optional: set owner/group
+    owner = "myservice";
+    group = "myservice";
+  };
+}
+```
+
+### Using Secrets
+
+Reference the secret path in service configuration:
+
+```nix
+{
+  services.myservice = {
+    passwordFile = config.sops.secrets.my-secret.path;
+  };
+}
+```
+
+### Using Templates
+
+For secrets that need to be embedded in config files:
+
+```nix
+{
+  sops.secrets."netdata/stream-api-key" = { };
+  
+  sops.templates."netdata-stream.conf" = {
+    content = ''
+      [stream]
+      enabled = yes
+      api key = ${config.sops.placeholder."netdata/stream-api-key"}
+    '';
+    owner = "netdata";
+  };
+  
+  services.netdata.configDir."stream.conf" = 
+    config.sops.templates."netdata-stream.conf".path;
+}
+```
+
+## Common Secrets
+
+### cryodev-main
+
+```yaml
+mailserver:
+  accounts:
+    forgejo: "$2y$05$..."  # bcrypt hash
+    admin: "$2y$05$..."
+
+forgejo-runner:
+  token: "..."
+
+headplane:
+  cookie_secret: "..."       # openssl rand -hex 16
+  agent_pre_authkey: "..."   # headscale preauthkey
+
+tailscale:
+  auth-key: "tskey-..."
+```
+
+### cryodev-pi
+
+```yaml
+tailscale:
+  auth-key: "tskey-..."
+
+netdata:
+  stream:
+    child-uuid: "..."  # uuidgen
+```
+
+## Generating Secret Values
+
+| Secret | Command |
+|--------|---------|
+| Mailserver password | `nix-shell -p mkpasswd --run 'mkpasswd -sm bcrypt'` |
+| Random hex token | `nix-shell -p openssl --run 'openssl rand -hex 16'` |
+| UUID | `uuidgen` |
+| Tailscale preauth | `sudo headscale preauthkeys create --expiration 99y --reusable --user default` |
+
+## Updating Keys
+
+After modifying `.sops.yaml`, update existing secrets files:
+
+```bash
+sops --config .sops.yaml updatekeys hosts/<hostname>/secrets.yaml
+```
+
+## Troubleshooting
+
+### "No matching keys found"
+
+Ensure the host's age key is in `.sops.yaml` and you've run `updatekeys`.
+
+### Secret not decrypting on host
+
+Check that `/etc/ssh/ssh_host_ed25519_key` exists and matches the public key in `.sops.yaml`.
--- a/docs/services/tailscale.md
+++ b/docs/services/tailscale.md
@ -0,0 +1,117 @@
+# Tailscale Client
+
+Tailscale clients connect to the self-hosted Headscale server to join the mesh VPN.
+
+## References
+
+- [Tailscale Documentation](https://tailscale.com/kb)
+- [Headscale Client Setup](https://headscale.net/running-headscale-linux/)
+
+## Setup
+
+### Generate Auth Key
+
+On the Headscale server (cryodev-main):
+
+```bash
+sudo headscale preauthkeys create --expiration 99y --reusable --user default
+```
+
+### Add to Secrets
+
+```bash
+sops hosts/<hostname>/secrets.yaml
+```
+
+```yaml
+tailscale:
+  auth-key: "your-preauth-key"
+```
+
+### Configuration
+
+```nix
+# In your host configuration
+{ config, ... }:
+{
+  sops.secrets."tailscale/auth-key" = { };
+  
+  services.tailscale = {
+    enable = true;
+    authKeyFile = config.sops.secrets."tailscale/auth-key".path;
+    extraUpFlags = [
+      "--login-server=https://headscale.cryodev.xyz"
+    ];
+  };
+}
+```
+
+## Usage
+
+### Check Status
+
+```bash
+tailscale status
+```
+
+### View IP Address
+
+```bash
+tailscale ip
+```
+
+### Ping Another Node
+
+```bash
+tailscale ping <hostname>
+```
+
+### SSH to Another Node
+
+```bash
+ssh user@<hostname>
+# or using Tailscale IP
+ssh user@100.64.0.X
+```
+
+## MagicDNS
+
+With Headscale's MagicDNS enabled, you can reach nodes by hostname:
+
+```bash
+ping cryodev-pi
+ssh steffen@cryodev-main
+```
+
+## Troubleshooting
+
+### Check Service Status
+
+```bash
+sudo systemctl status tailscaled
+```
+
+### View Logs
+
+```bash
+sudo journalctl -u tailscaled -f
+```
+
+### Re-authenticate
+
+If the node is not connecting:
+
+```bash
+sudo tailscale up --login-server=https://headscale.cryodev.xyz --force-reauth
+```
+
+### Node Not Appearing in Headscale
+
+Check the auth key is valid:
+
+```bash
+# On Headscale server
+sudo headscale preauthkeys list --user default
+```
+
+Verify the login server URL is correct in the client configuration.