RPC Full Outage
Incident Report for WalletConnect
Postmortem

Summary

From ~830am to 1130am CET our RPC was fully down. Reason was that the network as accidentally cut when a VPC was removed. All customers of WalletConnect’s RPC were affected. Customers (dapps/wallets) who use another RPC were not affected.

What Happened

WalletConnect had sunset their version 1 including the hosted infrastructure. Services were deployed in AWS with only little use of off-the-shelf components including NAT.

v1 services were also deployed in a VPC monolith and some new services including the RPC were also deployed in that VPC monolith.

When operator removed v1 EC2 instances they unknowingly also removed the NAT preventing the RPC from talking to the outside world.

Resolution was to move the RPC service into its own VPC.

What went well

  • alarms fired immediately
  • operator was immediately acting

Action items

  • all services / tooling to move off VPC monolith
Posted Jun 30, 2023 - 09:42 UTC

Resolved
This incident has been resolved.
Posted Jun 30, 2023 - 09:42 UTC
Investigating
Our RPC is currently down. Relay and other systems are fully functioning.
Posted Jun 30, 2023 - 06:40 UTC
This incident affected: RPC.