Multi-Stage Incident Simulation

Diagnose a cascading three-failure incident and write a postmortem.

A advanced Linux challenge worth 20 points. Solve it hands-on in a real Linux environment in your browser - no local setup, no fake shells.

The Challenge

Pre-generated logs from a 30-minute incident are at /var/log/incident/. At T+0 disk filled. At T+8 the app OOM-killed. At T+15 systemd restart failed due to a missing env file. All three failures are connected.

Your tasks:

• Find disk fill event: grep "No space left" /var/log/incident/syslog
• Find OOM kill: grep -i "killed process" /var/log/incident/kern.log
• Find systemd restart failure: grep "Failed" /var/log/incident/app-service.log
• Find missing env file: cat /var/log/incident/systemd-failed.txt
• Trace the causal chain in /tmp/postmortem.txt:

TIMELINE: (3 timestamped events) ROOT_CAUSE: (one sentence) CONTRIBUTING: (one sentence) FIX: (what was done) PREVENTION: (what stops this next time)

• Run check to validate