Workflow Automation

1. User reports production down
2. /incident start production-outage
3. Playbook automatically:
   - Creates dedicated incident channel
   - Notifies on-call team
   - Posts initial checklist
   - Updates status page
4. Team follows checklist:
   - Check service health
   - Review recent deployments
   - Analyze error logs
   - Implement fix
   - Verify resolution
5. Close incident with retrospective

Name: Production Incident Response
Description: Handle production outages and critical bugs
Public/Private: Public to team

✓ Initial Response
  □ Acknowledge incident (assigned to: On-call engineer)
  □ Create status page incident
  □ Notify stakeholders via #incidents
  □ Identify impacted services

✓ Investigation
  □ Check monitoring dashboards
  □ Review recent deployments
  □ Analyze error logs
  □ Identify root cause

✓ Resolution
  □ Implement fix or rollback
  □ Verify services restored
  □ Update status page: Resolved
  □ Post all-clear message

✓ Post-Incident
  □ Schedule retrospective
  □ Document timeline
  □ Create follow-up tasks
  □ Update runbooks

When run starts:
  → Create channel: incident-{run-number}
  → Invite: @on-call, @engineering-lead
  → Post message: "Incident response initiated. Status updates will be posted here."
  → Webhook: POST to status page API
  
When status changes to "Resolved":
  → Post message: "Incident resolved. Retrospective scheduled."
  → Webhook: Update status page

/playbook run "Production Incident Response"

User posts: "production is down"
→ Playbook suggests starting incident response

Active Runs (3)
┌─────────────────────────────────────────┐
│ 🔴 Production API Down (#427)           │
│    Started: 15 min ago by @john         │
│    Status: Investigating                │
│    Tasks: 4/12 complete                 │
└─────────────────────────────────────────┘

│ 🟡 Deploy v2.5.0 (#426)                 │
│    Started: 2 hours ago by @deploy-bot  │
│    Status: In Progress                  │
│    Tasks: 8/10 complete                 │
└─────────────────────────────────────────┘

Closed Runs (147)
- Database Migration (#425) - 3 hours ago
- Customer Escalation (#424) - 1 day ago
- Release v2.4.9 (#423) - 2 days ago

□ Task description
  Additional context and links
  Assigned to: @username
  Due: 2 hours from run start

□ Run health check
  /healthcheck production

□ Notify stakeholders
  Webhook: POST to notification service

Action: Create Channel
Name Pattern: incident-{run-number}
Type: Public/Private
Announce in: #incidents

Action: Invite Users
Users:
  - @on-call-engineer
  - @engineering-manager
Optional:
  - @cto (for severity: critical)

Action: Post Message
Channel: Run channel
Message: |
  🚨 **Incident Response Initiated**
  
  Severity: {severity}
  Reporter: {reporter}
  
  Please follow the checklist in the right sidebar.

Action: Trigger Webhook
URL: https://status.company.com/api/incidents
Method: POST
Payload:
  title: {run-name}
  status: investigating
  started_at: {start-time}

# Webhook endpoint for starting runs
POST /plugins/playbooks/api/v0/runs

# Example: Start from monitoring alert
curl -X POST https://mattermost.com/plugins/playbooks/api/v0/runs \
  -H "Authorization: Bearer token" \
  -d '{
    "playbook_id": "playbook-id",
    "name": "Production Alert: High Error Rate",
    "description": "Alert triggered by Datadog"
  }'

Time to Acknowledge: 5 minutes
Time to Resolution: 2 hours 15 minutes
Impacted Users: ~5,000
Revenue Impact: $12,000

10:15 AM - Incident reported
10:17 AM - On-call acknowledged  
10:22 AM - Root cause identified
10:45 AM - Fix deployed to staging
11:30 AM - Fix deployed to production
12:30 PM - Incident resolved

What went well?
→ Fast acknowledgment time
→ Clear communication with stakeholders
→ Comprehensive monitoring helped identify issue quickly

What could be improved?
→ Staging environment didn't catch the bug
→ Rollback procedure was unclear
→ Need better load testing

Action Items:
→ Improve staging environment fidelity
→ Document rollback procedure
→ Add load testing to CI/CD pipeline

Retrospective:
  Questions:
    - What was the root cause?
    - How did we discover the issue?
    - What went well during response?
    - What could we improve?
    - What preventive measures should we take?
  Metrics:
    - Time to Acknowledge (minutes)
    - Time to Resolution (minutes)
    - Number of People Involved
    - Customer Impact (1-5 scale)

Incident Response (Last 30 Days)
─────────────────────────────────
Total Incidents: 23
Avg. Time to Acknowledge: 8 min
Avg. Time to Resolve: 1h 45min

Severity Breakdown:
  Critical: 3 (13%)
  High: 8 (35%)
  Medium: 12 (52%)

Top Root Causes:
  1. Deployment issues (8)
  2. Database problems (6)
  3. External dependencies (5)