cluster-events

Files (2)

• SKILL.md
• event-timestamps-and-fields.md · references/

SKILL.md (first lines)

# Cluster Events Analysis

Use this flow to analyze cluster-wide events for identifying issues, patterns, and correlations across resources.

**Scope:** This skill is for **analysis and diagnosis only**. It helps you understand what is happening across the cluster by examining events. Do NOT attempt to fix issues directly — identify root causes and either use a specific diagnostic skill or report findings to the user.

## Diagnostic Flow

### 0. Normalise the fields first — an event has two layouts

`Event` exists in two API groups with almost disjoint field names, and the `v1` object
additionally carries `series` for a continuing event. Reading only one layout loses
half the events silently, so every query below opens with the same four definitions.
Copy them as-is:

```
def pad: if test("Z$") | not
         then error("timestamps must be UTC and end in Z; got \(.) — convert the offset first")
         else capture("^(?<b>[^.Z]+)(\\.(?<f>[0-9]+))?Z$")
              | .b + "." + (((.f // "") + "000000")[0:6]) + "Z" end;   # fixed width, for comparison
def cut: sub("\\.[0-9]+";"");                                          # whole seconds, for display

active: (.series.lastObservedTime // .lastTimestamp // .deprecatedLastTimestamp // .eventTime)
start:  (.firstTimestamp // .deprecatedFirstTimestamp // .eventTime)
count:  (.series.count // .count // .deprecatedCount // 1)
object: (.involvedObject // .regarding // {})
```

Four things about these are load-bearing, and each was measured rather than assumed:

- **`series` comes first** in `active` and `count`. On a series event the `v1`
  `lastTimestamp` and `count` are frozen at first write; `series.*` carries the truth.
- **`start` has no `series` term.** A series has no separate start.
- **Compare padded, display cut.** `MicroTime` values carry a fraction and
  `"…09:00:00.500000Z" >= "…09:00:00Z"` is false, because `.` sorts below `Z`.
- **The window must be UTC.** `pad` refuses an offset or a missing zone by name rather
  than mis-sorting it.

Full reasoning, the measurements behind each, and what breaks if the ordering changes:
`references/event-timestamps-and-fields.md`. Read it before editing any query here.

### 0a. Fix the time window first, and measure what the cluster actually kept

Events expire. A cluster keeps roughly the last hour by default, so "the events
that exist" is not the same as "what happened" — and the difference is invisible
in the output. Answering "any recent warnings?" from an unbounded listing produces
a conclusion with no reproducible boundary, and one that reads as "nothing went
wrong" when it can only mean "nothing that is still retained".

So establish two things before interpreting anything:

- **The window you were asked about.** Take it from the request ("in the last 20
  minutes", "around 09:53Z"). If the request does not imply one, ask — do not
  substitute "whatever is retained".
- **The window the cluster can answer for.** Measure it:

```bash
kubectl get events -A -o json | jq -r '
  def pad: if test("Z$") | not
           then error("timestamps must be UTC and end in Z; got \(.) — convert the offset first")
           else capture("^(?<b>[^.Z]+)(\\.(?<f>[0-9]+))?Z$")
                | .b + "." + (((.f // "") + "000000")[0:6]) + "Z" end;
  def cut: sub("\\.[0-9]+";"");
  [(.items // [])[]] as $e
  | ($e | map(.series.lastObservedTime // .lastTimestamp // .deprecatedLastTimestamp // .eventTime)
        | map(select(. != null)) | sort_by(pad) | map(cut)) as $active
  | ($e | map(.firstTimestamp // .deprecatedFirstTimestamp // .eventTime)
        | map(select(. != null)) | sort_by(pad) | map(cut)) as $started
  | if ($active | length) == 0 then "no events retained at all"
    else "\($e | length) events retained
  oldest observed activity: \($active[0])
  oldest start still present: \($started[0])
  newest activity: \($active[-1])" end'
```

These are OBSERVATIONS, not a retention boundary. Read them as lower bounds:

- **oldest observed activity** proves the cluster still holds something that recent.
  It does **not** locate the TTL cutoff — with a one-hour TTL and a single event at
  09:55, this reads 09:55 while everything back to 09:00 is still within retention and
  merely uneventful. Treating it as the cutoff would report "the evidence for 09:30 is
  deleted" when the truth is "nothing happened at 09:30".
- **oldest start still present** says some currently-active condition began that long
  ago. It says nothing about coverage in between: anything that started AND finished
  there is gone without a trace.

So for any window earlier than the oldest observed activity, coverage is **unknown** —
"no events" there could mean quiet or deleted, and this data cannot tell them apart.
Say which one you cannot rule out rather than picking one. If the answer depends on it,
go to logs or monitoring, which retain on their own schedule.

The actual cutoff is the apiserver's `--event-ttl` (default 1h), which is usually not
readable from inside the cluster — `kubectl -n kube-system get pod -l
component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].command}'` works on
some deployments and not on managed control planes. If you can read it, the cutoff is
`now - ttl` and everything above becomes a check rather than a guess. If you cannot,
say the window is bounded by an unknown TTL rather than by the oldest event you saw.

### 0b. Confirm the namespace exists, if the question names one

`kubectl get events -n <ns>` returns an empty list and **exit code 0** for a
namespace that does not exist. Measured: no error, no warning, just
`"items": []`. A typo in the namespace is therefore indistinguishable from a
namespace with no warnings, and both read as "nothing wrong there".

```bash
kubectl get namespace <ns> -o name
```

If that fails, the events listing that follows proves nothing. Report the bad
namespace rather than its empty result.

### 1. List Warning events inside the window

`kubectl` has no server-side time filter for events — `--field-selector
'lastTimestamp>...'` is rejected as an invalid selector — so the window is applied
client-side:

```bash

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw