Incident at 14:23

Investigate a production incident by digging through application logs.

A advanced Linux challenge worth 20 points. Solve it hands-on in a real Linux environment in your browser - no local setup, no fake shells.

The Challenge

INCIDENT REPORT: Payment service started throwing errors at exactly 14:23:07. Users cannot complete purchases.

The service logs are in /tmp/logs/payment.log

Your investigation tasks:

1. Find all ERROR lines that appear at 14:23 or later → save to /tmp/errors_after.txt
2. Count how many times "connection refused" appears in the file → write the number to /tmp/conn_refused_count.txt
3. Find the first FATAL line and write the entire line to /tmp/fatal_line.txt
4. Extract all unique error codes (pattern: ERR-[0-9]+) → save to /tmp/error_codes.txt

This is exactly the investigation flow used in real production incidents.

Commands to learn: grep -n, grep -E, grep -o, awk, sort -u

Run check when done.