Log Analyzer

Overview

Log viewers filter lines. This skill finds patterns.

Use it when an agent runtime, tool process, gateway, scheduler, or other service has produced enough log output that individual grep hits no longer explain the system behavior. The analyzer parses standard log lines, normalizes repeated messages, groups errors by component and tool, and produces a structured anomaly report that is suitable for debugging sessions or cron digests.

The script is intentionally lightweight and portable: it uses only Python's standard library and works on any text log with timestamp, level, optional component, and message fields. It also has a best-effort fallback for unstructured logs.

Quick Start

cd agent-ops/log-analyzer
python scripts/analyze_logs.py scan --log-file agent.log --since 24h

For per-tool failure rates from a Hermes profile's session DB (structured exit_code signals, NOT regex-over-content):

python scripts/state_failures.py                 # last 7 days, dashboard
python scripts/state_failures.py --days 30 --json
python scripts/state_failures.py --quiet         # cron: silent when healthy

See references/state-failure-monitor.md for provenance (adapted concept from hermes-dojo), the spike evidence for why structured signals matter, and interpretation notes.

To write JSON for later Markdown rendering:

python scripts/analyze_logs.py scan --log-file agent.log --since 24h --output scan.json
python scripts/analyze_logs.py report --scan scan.json --output report.md

For cron-compatible anomaly detection:

python scripts/analyze_logs.py scan --log-file agent.log --since 24h --quiet

--quiet exits 0 and prints nothing when no anomalies are found. If anomalies exist, it prints the JSON report so the scheduler can deliver the digest.

What It Detects

1. Error clusters — the same normalized error message repeated 3+ times within the selected time window. URLs, IP addresses, and numbers are replaced with placeholders before grouping so repeated failures with changing IDs still cluster.
2. Rate limit hits — HTTP 429, rate limit, rate_limit, too many requests, and quota exceeded patterns. The analyzer groups them by provider when it can detect a provider name.
3. Timeout patterns — timeout, timed out, deadline exceeded, and connection timeout. Results are grouped by detected tool name and include example URLs where present.
4. Tool failures — error lines grouped by tool name extracted from patterns such as tool: terminal, tool_call: web_search, tool terminal failed, or lines emitted by a tools component.
5. Session crashes — fatal errors, unhandled exceptions, stack traces, Traceback, Exception, and segfault markers. Multiline stack traces are grouped as one crash entry with nearby context.
6. Component breakdown — error and warning counts by component such as gateway, agent, tools, cron, or unknown.
7. Error timeline — error counts bucketed by hour to reveal spikes and regressions after deploys or scheduled jobs.

See references/anomaly-types.md for detection criteria and interpretation.

Log Format Support

The parser handles standard log lines shaped like:

2026-07-06 12:30:45 ERROR [gateway] Connection refused
2026-07-06T12:30:45Z ERROR gateway: Connection refused
12:30:45 ERROR Connection refused

It recognizes ERROR, WARN, WARNING, INFO, DEBUG, FATAL, and CRITICAL levels. Components may appear in square brackets after the level or as component: after the level. Time-only lines are anchored to the current day (or to the supplied default date when called as a library). Unstructured lines fall back to best-effort line-by-line scanning, so obvious ERROR/WARN strings are still counted even when the timestamp cannot be parsed.

See references/log-formats.md for examples and guidance on adding custom patterns.

CLI Commands

scan — analyze a log file for patterns

python scripts/analyze_logs.py scan --log-file LOGFILE [--since TIME] [--output report.json] [--quiet]

Options:

• --log-file LOGFILE — required path to the log file.
• --since TIME — optional time window: minutes/hours/days/weeks, e.g. 30m, 1h, 24h, 7d, 2w; default is all lines.
• --output report.json — write JSON to a file instead of stdout.
• --quiet — cron mode: suppress output when no anomalies are found.

report — render Markdown from scan JSON

python scripts/analyze_logs.py report --scan scan.json [--output report.md]

The report contains overview counts, one section per anomaly type, a component breakdown table, and an hourly error timeline.

tail — smart tail for recent lines

python scripts/analyze_logs.py tail --log-file LOGFILE [--lines N] [--since TIME]

The smart tail prints plain text with markers suitable for chat delivery:

• normal lines: no marker
• warnings: ⚠️
• errors: ❌
• repeated recent errors: 🔥 when the same normalized error is seen 3+ times

Output Format

scan emits JSON with these top-level fields:

• log_file, lines_analyzed, time_window
• anomalies.error_clusters, anomalies.rate_limits, anomalies.timeouts, anomalies.tool_failures, anomalies.crashes
• component_breakdown
• error_timeline
• total_errors, total_warnings, has_anomalies

report converts that JSON into Markdown:

# Log Analysis Report

## Overview
- Log file: agent.log
- Lines analyzed: 1542
- Time window: 24h
- Total errors: 12 | warnings: 20
- Anomalies detected: 5 types

Scheduled Summary Integration

For a scheduled digest, run scan in quiet mode and include the output only when it is non-empty:

python scripts/analyze_logs.py scan --log-file /var/log/agent.log --since 24h --quiet --output /tmp/log-scan.json
if [ -s /tmp/log-scan.json ]; then
  python scripts/analyze_logs.py report --scan /tmp/log-scan.json
fi

A scheduled-summary job can append the Markdown output under a "Log anomalies" heading. Keep the analysis window aligned with the summary window (for example, 24 hours for a daily digest) so counts do not overlap or disappear.

For a state.db failure-rate digest (weekly is a sensible cadence given the volume of sessions):

python scripts/state_failures.py --quiet --days 7

This prints nothing (exit 0) when the window has zero failures, so it can be used as a no_agent cron job that only pings when something is wrong. See references/state-failure-monitor.md for what the categories mean (timeout is the most actionable on real data).

Common Pitfalls

1. Log rotation breaks time windows. If yesterday's file was rotated out, a --since 24h scan over only the current file may miss early-window failures. Point the scheduler at the active file plus rotated file, or concatenate the relevant files before scanning.
2. Multiline stack traces need the first line. The parser groups indented stack-trace continuation lines under the preceding parsed log line. If a log collector strips the first Traceback or ERROR line, the remaining stack frames become unstructured context.
3. Expected errors can be false positives. Retries, probing, and health checks may intentionally emit warnings or connection failures. Treat clusters as "investigate" signals, not automatic incidents.
4. First run on a large historical log can overwhelm output. Start with --since 24h, inspect the report, then widen the window if needed.
5. Changing message formats can split clusters. If an application changes an error string during a deploy, pre- and post-deploy failures may appear as two clusters even when the root cause is the same.
6. Time-only logs depend on the scan date. 12:30:45 ERROR ... lines do not contain a date. For historical files, prefer full timestamps.
7. Comma-millisecond Python-logging lines silently defeat --since. Lines shaped 2026-07-06 12:30:45,123 INFO module: ... (Hermes agent/gateway/errors logs) are NOT matched by the built-in timestamp regexes. They fall to the unstructured fallback with timestamp=null, so --since cannot filter them and the whole unrotated file is scanned. Symptoms: error/traceback counts wildly exceed the true windowed count, every cluster reports "First: unknown / Last: unknown", the error timeline says "No timestamped errors detected", and crashes are inflated (each stack-trace continuation line counts as a separate crash). Fix: for Hermes profile logs use the windowed counter scripts/hermes_log_window.py (handles the comma-ms format, de-duplicates multiline tracebacks by counting Traceback (most recent call last) first-lines only) and see references/hermes-profile-audit.md for pulling skill-usage/tool-call counts from the profile's state.db.
8. Skill-usage counts are not in agent.log. tool skill_view completed (0.05s, 13890 chars) lines do not carry the skill-name argument. Query the profile's state.db messages table — the tool_calls JSON column holds the full arguments. messages.timestamp is a Unix epoch (REAL), not ISO. See references/hermes-profile-audit.md for a ready-to-run snippet.
9. The error timeline buckets by hour-of-day, not date. On a multi-day window (--since 7d), errors from 08:00 on different days merge into one 08:00 bar. Use the clusters (which carry full timestamps) for multi-day forensics; treat the timeline as a time-of-day profile.
10. Crash detection is substring-based. The crash regex matches words like fatal or Exception anywhere in a line, so mentions inside INFO lines (e.g. "retry succeeded after TimeoutException", "non-fatal warning") count as crash signals and can set has_anomalies. Treat crash counts as leads to eyeball, and tune the regex if your logs legitimately chat about exceptions at INFO level.

What This Skill Is NOT

• Not a log viewer: use tail, less, or a log UI when you need raw line inspection.
• Not a log shipper: it does not forward logs to storage or observability systems.
• Not a SIEM: it does not correlate identities, networks, or security events.
• Not real-time monitoring: tail is a recent-line analyzer, not a daemon or alerting service.
• Not a root-cause oracle: it highlights patterns so an agent or operator can investigate faster.

Verification Checklist

• Run python scripts/analyze_logs.py --help and confirm subcommands load.
• Scan a synthetic log with repeated errors and confirm has_anomalies is true.
• Render Markdown from the scan JSON and confirm all anomaly sections appear.
• Run python -m pytest tests/test_analyze_logs.py -v from this skill directory.
• Run python -m pytest tests/test_state_failures.py -v from this skill directory (state.db failure-rate monitor; the spike regression test — content merely mentioning error words must not count as failure — lives here).
• For cron use, test a no-anomaly log with --quiet and confirm stdout is empty with exit code 0.
• For state.db cron use, python scripts/state_failures.py --quiet against a live profile DB exits 0 and prints nothing when the window is clean.