Write-up
Stuck Start Compute Operations

On two occasions in the past week, Neon customers in AWS/us-east-1 were unable to create or start inactive databases, for a total period of 5.5 hours.  Customers with running databases were unaffected.

The root cause related to our ability to assign IP addresses to new database instances. We have taken preventative measures with our IP allocation, subnet size and VPC configuration to avoid re-occurrence. We have also redirected some load from this region to others.

We have a longer-term plan to replace our current Kubernetes architecture, which is in progress. We are continuing to work on action items following these incidents and will post a further update with more details on our blog, in the coming days.