cryodev/docs/getting-started/first-install.md

# Initial Installation (x86_64 Server)

This guide describes the **initial installation** of a new x86_64 server (e.g. cryodev-main).

> **For Raspberry Pi:** See [Creating an SD Image](sd-image.md).

## Overview

During initial installation there is a chicken-and-egg problem:
- SOPS secrets are encrypted with the SSH host key
- The SSH host key is only generated during installation
- Therefore: **Install without secrets first, then configure secrets**

### Process

```
1. Disable services (that require secrets)
2. Install NixOS
3. Extract SSH host key, configure SOPS, create immediately available secrets
4. Enable stage-1 services and deploy (Headscale, Forgejo, Mail, Nginx)
5. Generate remaining secrets (Tailscale, Headplane, Forgejo Runner)
6. Enable stage-2 services and perform final deployment
```

## Step 1: Prepare Host Configuration

> If the host already exists in `hosts/` and `flake.nix`, skip 1.1-1.2.

### 1.1 Create Host from Template

```bash
nix run .#create -- -t generic-server -n <hostname>
```

The script:
- Copies the template to `hosts/<hostname>/`
- Sets the hostname in `networking.nix`
- Creates an empty `secrets.yaml`
- Adds the files to Git

### 1.2 Register in flake.nix

```nix
nixosConfigurations = {
  <hostname> = mkNixosConfiguration "x86_64-linux" [ ./hosts/<hostname> ];
};
```

Also adjust `hardware.nix` and `disks.sh` for the target hardware.

### 1.4 Temporarily Disable Services

All services that reference SOPS secrets must be disabled for the initial installation. Otherwise the installation will fail because the secrets cannot yet be decrypted.

In `hosts/<hostname>/services/default.nix`, comment out the corresponding imports:

```nix
{
  imports = [
    # Disabled until SOPS secrets are configured:
    # ./forgejo.nix     # requires: forgejo-runner/token, forgejo/mail-pw
    # ./headplane.nix   # requires: headplane/cookie_secret, headplane/agent_pre_authkey
    # ./mailserver.nix  # requires: mailserver/accounts/*
    # ./tailscale.nix   # requires: tailscale/auth-key

    # These services do not require secrets:
    ./headscale.nix
    ./netdata.nix
    ./nginx.nix
    ./openssh.nix
    ./sops.nix
  ];
}
```

Additionally, in `hosts/<hostname>/services/sops.nix`, comment out the secret definitions:

```nix
sops = {
  defaultSopsFile = ../secrets.yaml;
  # secrets = {
  #   "forgejo-runner/token" = { };
  #   "tailscale/auth-key" = { };
  # };
};
```

### 1.5 Test the Configuration

```bash
nix eval .#nixosConfigurations.<hostname>.config.system.build.toplevel.name
```

## Step 2: Perform Installation

### 2.1 Boot NixOS ISO

Boot from the [NixOS Minimal ISO](https://nixos.org/download/#nixos-iso) (USB/CD).

### 2.2 Set Up Network and SSH

```bash
passwd                  # Set root password for SSH access
ip a                    # Determine IP address
```

Optionally connect via SSH (more convenient):

```bash
ssh -o StrictHostKeyChecking=no root@<IP>
```

### 2.3 Install

```bash
nix --experimental-features "nix-command flakes" run \
  git+<REPO_URL>#apps.x86_64-linux.install -- \
  -n <hostname> \
  -r <REPO_URL>
```

Alternatively, if the repository has already been cloned to `/tmp/nixos`:

```bash
nix --experimental-features "nix-command flakes" run /tmp/nixos#install -- -n <hostname>
```

> **Note:** The disk ID in `hosts/<hostname>/disks.sh` must match the hardware.
> Verify with `ls -la /dev/disk/by-id/`.

The script:
1. Clones the repository (when using `-r`)
2. Partitions the disk (via `disks.nix` or `disks.sh`)
3. Generates `hardware.nix` (if not present)
4. Installs NixOS

### 2.4 Reboot

```bash
reboot
```

## Step 3: Configure SOPS Secrets

After the first boot, log in (password: `changeme`, change immediately with `passwd`).

### 3.1 Convert SSH Host Key to Age Key

On the **new server**:

```bash
nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
```

Note the output (e.g. `age1abc123...`).

Alternatively, remotely:

```bash
nix-shell -p ssh-to-age --run 'ssh-keyscan -p 2299 -t ed25519 <IP> | ssh-to-age'
```

### 3.2 Update .sops.yaml

On the **development machine**, add the new host key to `.sops.yaml`:

```yaml
keys:
  - &steffen_key age1e8p...          # steffen (local)
  - &hostname_key age1abc...         # Key from step 3.1

creation_rules:
  - path_regex: hosts/<hostname>/secrets.yaml$
    key_groups:
      - age:
          - *steffen_key
          - *hostname_key
```

### 3.3 Create Secrets

Open the secrets file:

```bash
sops hosts/<hostname>/secrets.yaml
```

The following table shows all secrets for **cryodev-main** and how they are generated:

#### Immediately Available Secrets

These secrets have no dependencies and can be generated directly:

| Secret | Command |
|--------|---------|
| `headplane/cookie_secret` | `openssl rand -hex 16` |
| `mailserver/accounts/admin` | `mkpasswd -sm bcrypt` (remember the password!) |
| `mailserver/accounts/forgejo` | `mkpasswd -sm bcrypt` (remember the password!) |
| `forgejo/mail-pw` | Plaintext password matching the bcrypt hash of `mailserver/accounts/forgejo` |

#### Secrets That Require Running Services

These secrets can only be created after step 4. **Do not add them yet** -- they will be added later.

| Secret | Command | Prerequisite |
|--------|---------|--------------|
| `tailscale/auth-key` | See steps 4.1-4.2 | Headscale is running |
| `headplane/agent_pre_authkey` | See steps 4.1-4.2 | Headscale is running |
| `forgejo-runner/token` | Forgejo Admin Panel > Actions > Runners > Create Runner | Forgejo is running |

#### Example secrets.yaml (Plaintext Before Encryption)

```yaml
headplane:
    cookie_secret: "a1b2c3d4e5f6..."
mailserver:
    accounts:
        admin: "$2b$05$..."
        forgejo: "$2b$05$..."
forgejo:
    mail-pw: "the-plaintext-password"
```

### 3.4 Gradually Re-enable Services -- Stage 1

> **Important:** Services that require Headscale or Forgejo secrets (Tailscale,
> Headplane, Forgejo Runner) must **not** be enabled yet, as these
> secrets can only be generated once those services are running.

On the **development machine**, in `hosts/<hostname>/services/default.nix`, enable
the services **without external dependencies**:

```nix
{
  imports = [
    # Stage 1: Services without external dependencies
    ./forgejo.nix
    ./headscale.nix
    ./mailserver.nix
    ./netdata.nix
    ./nginx.nix
    ./openssh.nix
    ./sops.nix

    # Stage 2: Enable only after step 4
    # ./forgejo-runner.nix  # requires: forgejo-runner/token (Forgejo)
    # ./headplane.nix       # requires: headplane/agent_pre_authkey (Headscale)
    # ./tailscale.nix       # requires: tailscale/auth-key (Headscale)
  ];
}
```

### 3.5 Deploy (Stage 1)

```bash
nix run .#deploy -- -n <hostname>
```

This uses the configuration from `deploy.json`. Alternatively, deploy manually:

```bash
NIX_SSHOPTS="-p 2299" nixos-rebuild switch --flake .#<hostname> \
  --target-host <user>@<IP> --sudo --ask-sudo-password
```

After this deployment, Headscale, Forgejo, Mailserver, and Nginx are running.

### 3.6 Create Forgejo Admin Account

On first start, Forgejo has no users. Create an admin account via CLI
(on the **server**):

```bash
forgejo admin user create \
  --username <username> \
  --email <email>@<domain> \
  --password <password> \
  --admin
```

> **Note:** The `forgejo` shell alias is provided by the module and automatically
> runs the command as the `forgejo` user with the correct config.
> If the alias is not available, start a new shell (`bash` or `zsh`).
>
> Since `DISABLE_REGISTRATION = true` is set, new accounts
> can only be created via CLI.

## Step 4: Generate Remaining Secrets and Enable All Services

After the server is running with Headscale and Forgejo:

1. **Create Headscale users** (on the server):

   ```bash
   sudo headscale users create default
   sudo headscale users create headplane-agent
   ```

2. **Determine user IDs** (needed for the preauth keys):

   ```bash
   sudo headscale users list
   ```

   The output shows the numeric IDs (e.g. `1` for default, `2` for headplane-agent).

3. **Generate preauth keys** (using the IDs from step 2):

   ```bash
   # For Tailscale (use the user ID of "default")
   sudo headscale preauthkeys create --expiration 99y --reusable --user <ID>

   # For Headplane Agent (use the user ID of "headplane-agent")
   sudo headscale preauthkeys create --expiration 99y --user <ID>
   ```

4. **Create the Forgejo Runner token** via the Forgejo Admin Panel:
   Administration > Actions > Runners > Create new Runner

5. **Add the remaining secrets**:

   ```bash
   sops hosts/<hostname>/secrets.yaml
   ```

   Add the missing secrets:

   ```yaml
   tailscale:
       auth-key: "tskey-..."
   forgejo-runner:
       token: "..."
   headplane:
       agent_pre_authkey: "..."
   ```

6. **Enable stage-2 services** in `hosts/<hostname>/services/default.nix`:

   ```nix
   {
     imports = [
       ./forgejo.nix
       ./forgejo-runner.nix
       ./headplane.nix
       ./headscale.nix
       ./mailserver.nix
       ./netdata.nix
       ./nginx.nix
       ./openssh.nix
       ./sops.nix
       ./tailscale.nix
     ];
   }
   ```

6. **Deploy again**:

   ```bash
   nix run .#deploy -- -n <hostname>
   ```

## Next Steps

- [SOPS Reference](../services/sops.md) -- Detailed documentation on secret management
- [Creating an SD Image](sd-image.md) -- Install Raspberry Pi
- [Set Up CD](../deployment/cd.md) -- Automatic deployment