Summary

From ~830am to 1130am CET our RPC was fully down. Reason was that the network as accidentally cut when a VPC was removed. All customers of WalletConnect’s RPC were affected. Customers (dapps/wallets) who use another RPC were not affected.

What Happened

WalletConnect had sunset their version 1 including the hosted infrastructure. Services were deployed in AWS with only little use of off-the-shelf components including NAT.

v1 services were also deployed in a VPC monolith and some new services including the RPC were also deployed in that VPC monolith.

When operator removed v1 EC2 instances they unknowingly also removed the NAT preventing the RPC from talking to the outside world.

Resolution was to move the RPC service into its own VPC.

What went well

• alarms fired immediately
• operator was immediately acting

Action items

• all services / tooling to move off VPC monolith