node-logs

Files (3)

• SKILL.md
• get-node-logs-via-api.sh · scripts/
• get-node-logs.sh · scripts/

SKILL.md (first lines)

# Node Logs

## Read this first: what an empty result means

Every command here can come back with nothing, and the reason matters more than
the emptiness. So both scripts print exactly one `status:` line and it is always
the **last line of the output**, with any explanation above it — one rule, every
outcome. (A bad argument is rejected on stderr with exit 2 before any report, so
there is no status line to read in that case.)

| `status:` | Meaning | What you may conclude |
|-----------|---------|-----------------------|
| `ok` | Source read, lines matched | Read the lines |
| `no_match` | Source read fine, filter matched nothing | **Nothing about the node.** Bounded non-evidence for that one query |
| `source_error` | The log source itself failed (see stderr) | Nothing — the output above it is partial |
| `filter_error` | Your pattern was rejected; nothing was searched | Nothing — fix the pattern and rerun |
| `not_found` / `not_a_log_file` / `empty_source` | The path is wrong, is a directory, or the handler is off | Nothing about the node's health |
| `query_unsupported` | This kubelet cannot serve journald over the API | Nothing — use a file, or tier 1/2 |
| `forbidden` | The agent lacks `nodes/proxy` | Nothing — an access fact, not a node fact |

`no_match` is the one that gets misread. "No PLEG messages in the last hour" is
not "PLEG is healthy": journald retention, a `--since` that misses the event, a
unit that never logged to this source, and a pattern that does not match this
component's wording all produce it. Report what you queried, not just that it
was empty. The header's `scanned:` count is what tells you whether anything was
actually read.

## Pick an access path

Try them in this order and say which one produced the evidence.

### Tier 1 — `host_script` (preferred)

The node is a bound SSH host: check `host_list` by node name or IP. No debug pod,
lowest latency.

**One caveat that produces empty results:** journalctl only shows the *system*
journal to root and to members of `adm` / `systemd-journal`. An ordinary SSH
account gets its own messages and **exit 0** — so a system unit looks like it
logged nothing. The script detects that account state, says so in a `note:`, and
repeats it in the `no_match` explanation. When you see it, the answer is not
"the unit was quiet": rerun via `node_script`, which enters the host namespaces
as root.

```
host_script: host="<host>", skill="node-logs", script="get-node-logs.sh", args="<args>"
```

### Tier 2 — `node_script`

The node is not a bound SSH host. Runs `get-node-logs.sh` inside the host's
namespaces via a privileged debug pod. Same arguments, same output.

```
node_script: node="<node>", skill="node-logs", script="get-node-logs.sh", args="<args>"
```

### Tier 3 — `local_script` against the kubelet log endpoint

Only when tier 1 and tier 2 are both impossible: no SSH binding **and** the debug
pod will not start (the node is NotReady, tainted, out of resources, or the image
cannot be pulled there). This path uses the agent's own kubectl, so it needs
nothing from the node beyond a responsive kubelet.

```
local_script: cluster="<cluster>", skill="node-logs",
              script="get-node-logs-via-api.sh", args="--node <node> --list"
```

**Set `cluster`** to the target cluster's credential name (from `cluster_list`);
`local_script` resolves it to a kubeconfig and exports `KUBECONFIG` for the
script. Do not pass `--kubeconfig` in `args`.

It is a genuinely weaker source, in three specific ways — all three matter when
you report:

- **It serves FILES under `/var/log`, not the journal.** If the node's journald
  does not forward to syslog/messages, kubelet and containerd messages are not
  reachable this way *at all*. A `journal/` entry in the listing means journald
  is persistent, but those files are binary and only `journalctl` can read them —
  which is exactly what tier 1 and 2 give you.
- **No server-side tail, no time filter, no ranges.** A file read transfers the
  whole file. `--head-bytes` can bound it, but it truncates from the *start* of
  the file, i.e. keeps the OLDEST bytes — useless for "what happened 10 minutes
  ago". Prefer a small, specific file over a multi-gigabyte syslog.
- **The journald query path is usually off.** `--query <unit>` uses the kubelet
  node log query (k8s ≥1.30 beta, opt-in alpha in 1.27–1.29). When the feature is
  off the kubelet answers with the `/var/log` **directory listing** and HTTP 200 —
  no error at all. The script detects that and reports `query_unsupported`
  instead of counting HTML as log lines. Always start with `--list`.

## `get-node-logs.sh` (tiers 1 and 2)

Source — one of, repeatable:

| Parameter | Description |
|-----------|-------------|
| `--unit UNIT` | Systemd unit (`containerd`, `kubelet`). Repeat to OR several units in one pass. |
| `--file PATH` | Log file path (`/var/log/messages`). Repeat for several files. |

Time window (journalctl only):

| Parameter | Default | Description |
|-----------|---------|-------------|
| `--since T` | `1h ago` | Window start. Dropped when `--boot` is given, so the boot is not clipped. |
| `--until T` | now | Window end. **Requires an explicit `--since`** — otherwise the default hour would end before it. |
| `--boot ID` | — | `0` = current boot, `-1` = previous, or a boot ID |
| `--list-boots` | — | List boots with IDs and time ranges, then exit |

`T` accepts journalctl syntax (`30m ago`, `today`, `2026-08-18 14:00:00`), RFC3339
(`2026-08-18T06:00:00Z` — converted for you, journalctl rejects that form) or an
epoch (`@1755526800`). The header echoes both what you passed and what was
actually queried.

Filtering — case-insensitive, repeatable, applied in the same single pass:

| Parameter | Description |
|-----------|-------------|
| `--grep PATTERN` | **Extended regular expression.** `pod-a\|pod-b` means a OR b. Repeat for more alternatives. |

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw