Massed Compute

Freeze Problem Scope

Define exactly what’s broken before you start investigating. Write a one-sentence symptom description:

SYMPTOM: "User login returns 500 error on POST /api/auth/login"

Include timestamps, affected users, and error patterns. Resist the urge to investigate multiple issues simultaneously.

Reproduce the Issue

Create minimal steps to trigger the problem from a clean state:

# Example reproduction steps
curl -X POST https://app.example.com/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"[email protected]","password":"test123"}'

# Expected: 200 OK with JWT token
# Actual: 500 Internal Server Error

Document commit SHA, software versions, and configuration that triggers the issue.

Separate Investigation Layers

Debug one layer at a time to avoid confusion:

• Network: Can you reach the service? (ping, telnet, curl)
• Authentication: Are credentials valid? (API keys, certificates)
• Data: Is the database accessible? (connection, schema, permissions)
• Service: Is the application running? (process status, health checks)
• Configuration: Are settings correct? (environment variables, config files)

Gather Read-Only Evidence

Collect system information without making changes:

# System status
systemctl status YOUR_SERVICE
journalctl -u YOUR_SERVICE --since today

# Resource usage
top -n 1
df -h
free -m

# Network connectivity
ss -tulpn | grep :80
curl -I http://localhost:8080/health

Form One Hypothesis

Based on evidence, create exactly one testable theory:

HYPOTHESIS: "Database connection pool exhausted due to leaked connections"

Make it specific and measurable. Avoid compound hypotheses like “database is slow AND memory is low.”

Run Minimal Experiment

Test your hypothesis by changing or inspecting one thing only:

# Example: Check database connection count
psql -h localhost -U app_user -d production \
  -c "SELECT count(*) FROM pg_stat_activity WHERE datname='production';"

# If pool limit is 20 and you see 19+ connections, hypothesis confirmed

Document the experiment and its result before making any fixes.

Apply Minimal Fix

Make the smallest change that addresses the root cause:

# Example: Restart service to reset connection pool
sudo systemctl restart YOUR_SERVICE

# Or increase pool size in config
# DATABASE_POOL_SIZE=30 (was 20)

Avoid shotgun fixes. Change one parameter, then test.

Verify the Solution

Re-run your original reproduction steps:

# Re-test the broken scenario
curl -X POST https://app.example.com/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"[email protected]","password":"test123"}'

# Should now return 200 OK

If you have automated tests, run them. Confirm the symptom is resolved.

Document and Record

Create a permanent record with this format:

# Incident: Login 500 errors - 2026-07-06

**Symptom:** User login returns 500 error on POST /api/auth/login
**Root Cause:** Database connection pool exhausted (20/20 connections leaked)
**Fix:** Restarted service + increased pool size to 30 connections
**Prevention:** Add connection pool monitoring alert at 80% utilization

Add this to your project’s troubleshooting documentation or incident log.