Keyboard shortcuts

Press ← or → to navigate between chapters

Press ? to show this help

Press Esc to hide this help

Kusama Validator Outage Post-Mortem: Era 8,219

Validator: rotko.net/ksm02 Stash: DKKax6uZkiNPfd2ATd8cJhyi3c1KZD24VDdoWG9CfTmwgSp

Summary Complete validator outage occurred at era 8,219, lasting 6

consecutive sessions. Root cause: physical power disconnection on bkk04 (7950X) during maintenance combined with monitoring blind spot.

Impact

  • 6 consecutive sessions(over 2eras) offline (no signing/voting/backing/block production)
  • Complete loss of validator duties during outage period

Root Cause Analysis

  1. Hardware failure: Power cable disconnection on bkk04 during routine maintenance. 1U form factor precludes redundant PSU configuration.
  2. Monitoring failure: Node excluded from Prometheus configuration, creating visibility gap. No alerts triggered during outage.

Immediate Actions

  • Migrated validation from failed bkk04 (7950X) to higher performance server (9950X on bkk03)
  • Scheduled onsite assessment for 2025-06-20

Corrective Measures Implemented

  1. Dual monitoring system:
    • Direct node metrics monitoring (CPU, memory, network, sync status)
    • Performance monitoring via forked turboflakes API with custom Prometheus exporter (era points, backing performance, block production)
  2. Redundancy: Two independent monitoring paths ensure failure detection even if one system fails
  3. Configuration audit: All active validators now properly configured in both monitoring systems

Future Prevention & Automated Failover

  • Enhanced physical maintenance protocols to prevent cable disconnections
  • Automated configuration validation for monitoring systems
  • Automated Failover System (in development):
    • Trigger: MVR=1 (100% missed votes) for single session
    • 30-second polling interval via turboflakes API
    • Automatic session key rotation to backup node
    • Maximum 1-2 session downtime (rotation delay only)
    • 30-minute cooldown prevents flip-flopping
    • Eliminates manual detection and reaction window at possibly error prone timeslots

The dual monitoring approach ensures immediate detection of failures. Automated failover implementation will reduce outage duration from hours to single session, providing true high availability for our validation services.