5ba78886d2
- Add automatic SD image builds for Raspberry Pi via Forgejo Actions - Enable binfmt emulation on cryodev-main for aarch64 cross-builds - Add sd-image.nix module to cryodev-pi configuration - Create comprehensive docs/ structure with installation guides - Split installation docs into: first-install (server), reinstall, new-client (Pi) - Add lib/utils.nix and apps/rebuild from synix - Fix headplane module for new upstream API (tale/headplane) - Fix various module issues (mailserver stateVersion, option conflicts) - Add placeholder secrets.yaml files for both hosts - Remove old INSTRUCTIONS.md (content moved to docs/)
181 lines
3.7 KiB
Markdown
181 lines
3.7 KiB
Markdown
# Netdata Monitoring
|
|
|
|
Netdata provides real-time performance monitoring with parent/child streaming.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────┐ Stream over ┌─────────────────┐
|
|
│ cryodev-pi │ ───────────────────>│ cryodev-main │
|
|
│ (Child Node) │ Tailscale VPN │ (Parent Node) │
|
|
└─────────────────┘ └─────────────────┘
|
|
│
|
|
v
|
|
https://netdata.cryodev.xyz
|
|
```
|
|
|
|
## References
|
|
|
|
- [Netdata Documentation](https://learn.netdata.cloud/)
|
|
- [Streaming Configuration](https://learn.netdata.cloud/docs/streaming/streaming-configuration-reference)
|
|
|
|
## Parent Node (cryodev-main)
|
|
|
|
### DNS
|
|
|
|
Set a CNAME record for `netdata.cryodev.xyz` pointing to your main domain.
|
|
|
|
### Generate Stream API Key
|
|
|
|
```bash
|
|
uuidgen
|
|
```
|
|
|
|
### Configuration
|
|
|
|
```nix
|
|
# hosts/cryodev-main/services/netdata.nix
|
|
{ config, ... }:
|
|
{
|
|
sops.secrets."netdata/stream-api-key" = { };
|
|
|
|
sops.templates."netdata-stream.conf" = {
|
|
content = ''
|
|
[${config.sops.placeholder."netdata/stream-api-key"}]
|
|
enabled = yes
|
|
default history = 3600
|
|
default memory mode = ram
|
|
health enabled by default = auto
|
|
allow from = *
|
|
'';
|
|
owner = "netdata";
|
|
};
|
|
|
|
services.netdata = {
|
|
enable = true;
|
|
configDir."stream.conf" = config.sops.templates."netdata-stream.conf".path;
|
|
};
|
|
}
|
|
```
|
|
|
|
## Child Node (cryodev-pi)
|
|
|
|
### Generate Child UUID
|
|
|
|
```bash
|
|
uuidgen
|
|
```
|
|
|
|
### Add to Secrets
|
|
|
|
```bash
|
|
sops hosts/cryodev-pi/secrets.yaml
|
|
```
|
|
|
|
```yaml
|
|
netdata:
|
|
stream:
|
|
child-uuid: "your-generated-uuid"
|
|
```
|
|
|
|
Note: The stream API key must match the parent's key. You can either:
|
|
1. Share the same secret between hosts (complex with SOPS)
|
|
2. Hardcode a known API key in both configurations
|
|
|
|
### Configuration
|
|
|
|
```nix
|
|
# hosts/cryodev-pi/services/netdata.nix
|
|
{ config, constants, ... }:
|
|
{
|
|
sops.secrets."netdata/stream/child-uuid" = { };
|
|
|
|
sops.templates."netdata-stream.conf" = {
|
|
content = ''
|
|
[stream]
|
|
enabled = yes
|
|
destination = ${constants.hosts.cryodev-main.ip}:19999
|
|
api key = YOUR_STREAM_API_KEY
|
|
send charts matching = *
|
|
'';
|
|
owner = "netdata";
|
|
};
|
|
|
|
services.netdata = {
|
|
enable = true;
|
|
configDir."stream.conf" = config.sops.templates."netdata-stream.conf".path;
|
|
};
|
|
}
|
|
```
|
|
|
|
## Email Alerts
|
|
|
|
Configure Netdata to send alerts via the mailserver:
|
|
|
|
```nix
|
|
{
|
|
services.netdata.configDir."health_alarm_notify.conf" = pkgs.writeText "notify.conf" ''
|
|
SEND_EMAIL="YES"
|
|
EMAIL_SENDER="netdata@cryodev.xyz"
|
|
DEFAULT_RECIPIENT_EMAIL="admin@cryodev.xyz"
|
|
'';
|
|
}
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Access Dashboard
|
|
|
|
Open `https://netdata.cryodev.xyz` in your browser.
|
|
|
|
### View Child Nodes
|
|
|
|
Child nodes appear in the left sidebar under "Nodes".
|
|
|
|
### Check Streaming Status
|
|
|
|
On parent:
|
|
```bash
|
|
curl -s http://localhost:19999/api/v1/info | jq '.hosts'
|
|
```
|
|
|
|
On child:
|
|
```bash
|
|
curl -s http://localhost:19999/api/v1/info | jq '.streaming'
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Check Service Status
|
|
|
|
```bash
|
|
sudo systemctl status netdata
|
|
```
|
|
|
|
### View Logs
|
|
|
|
```bash
|
|
sudo journalctl -u netdata -f
|
|
```
|
|
|
|
### Child Not Streaming
|
|
|
|
1. Verify network connectivity:
|
|
```bash
|
|
tailscale ping cryodev-main
|
|
nc -zv <parent-ip> 19999
|
|
```
|
|
|
|
2. Check API key matches between parent and child
|
|
|
|
3. Verify firewall allows port 19999 on parent
|
|
|
|
### High Memory Usage
|
|
|
|
Adjust history settings in `netdata.conf`:
|
|
|
|
```ini
|
|
[global]
|
|
history = 1800 # seconds to retain
|
|
memory mode = ram
|
|
```
|