Snippets¶

Short things worth writing down once so I stop re-deriving them at 3am. No narrative, no build-up — the command, what it does, and the caveat.

Longer reasoning lives in posts. Everything here is open source and copy-pasteable.

Find what a process is actually waiting on¶

# Live syscall latency histogram for one PID, no restart, no instrumentation.
sudo bpftrace -e '
  tracepoint:raw_syscalls:sys_enter /pid == '"$PID"'/ { @s[tid] = nsecs; }
  tracepoint:raw_syscalls:sys_exit  /pid == '"$PID"' && @s[tid]/ {
    @us[ksym(args->id)] = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]);
  }'

Run it for 30 seconds before theorising. The answer is usually one syscall you were not thinking about.

Which files does this service open most?¶

Good for finding filter candidates before you turn on verbose auditing.

sudo timeout 60s strace -f -qq -e trace=openat -p "$(pgrep -d, -f php-fpm)" 2>&1 \
  | sed -nE 's/.*openat\(.*"([^"]+)".*/\1/p' \
  | sort | uniq -c | sort -nr | head -30

strace costs real performance — do this on one host, not the fleet.

Postgres: find the unindexed query before adding cache¶

SELECT calls, round(mean_exec_time::numeric, 1) AS avg_ms,
       round(total_exec_time::numeric) AS total_ms, query
  FROM pg_stat_statements
 WHERE query NOT ILIKE '%pg_stat%'
 ORDER BY total_exec_time DESC
 LIMIT 15;

Sort by total time, not mean. A 4ms query run 900k times beats a 2s query run twice, and only one of them is worth your afternoon.

Systemd unit that actually restarts on failure¶

[Service]
Restart=always
RestartSec=2
# Without these two, repeated crashes hit the default burst limit and
# systemd gives up silently — the unit sits in "failed" until someone looks.
StartLimitIntervalSec=0

# Kill the whole cgroup, not just the main pid.
KillMode=control-group
TimeoutStopSec=30

nftables: default-deny that does not lock you out¶

# Validate BEFORE loading. `nft -c` parses without applying.
sudo nft -c -f /etc/nftables.conf && sudo nft -f /etc/nftables.conf

If you are applying over SSH, wrap it so a mistake self-reverts:

sudo systemd-run --on-active=120 --timer-property=AccuracySec=1s \
  nft flush ruleset
# You now have 2 minutes to confirm you can still connect.
# Still connected? Cancel it:  sudo systemctl stop run-r*.timer

kubectl: what is actually eating the node¶

kubectl get pods -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,'\
'CPU_REQ:.spec.containers[*].resources.requests.cpu,'\
'MEM_REQ:.spec.containers[*].resources.requests.memory,NODE:.spec.nodeName' \
  --field-selector spec.nodeName="$NODE" --sort-by=.metadata.namespace

Requests, not usage — this is what the scheduler reserved, which is what makes a node "full" while top looks idle.

Reproducible container digest in CI¶

# Never deploy a tag. Tags move; digests do not.
DIGEST=$(docker buildx imagetools inspect "$IMAGE:$TAG" \
  --format '{{ "{{" }}.Manifest.Digest{{ "}}" }}')
kubectl set image deploy/app "app=$IMAGE@$DIGEST"