AWS Amazon Elastic Compute Cloud (Sydney) Status

Now that we are fully recovered, we wanted to provide a brief summary of the issue. Starting at 4:07 PM PST, customers began to experience increased error rates and latencies for the network-related APIs in the AP-SOUTHEAST-2 Region. Launches of new EC2 instances also experienced increased failure rates as a result of this issue. Connectivity to existing instances was not affected by this event. We immediately began investigating the root cause and identified that the data store used by the subsystem responsible for the Virtual Private Cloud (VPC) regional state was impaired. While the investigation into the issue was started immediately, it took us longer to understand the full extent of the issue and determine a path to recovery. We determined that the data store needed to be restored to a point before the issue began. We began the data store restoration process, which took a few hours and by 10:50 PM PST, we had fully restored the primary node in the affected data store. At this stage, we began to see recovery in instance launches within the AP-SOUTHEAST-2 Region, restoring many customer applications and services to a healthy state. We continued to bring the data store back to a fully operational state and by 11:20 PM PST, all API error rates and latencies had fully recovered. Other AWS services - including AppStream, Elastic Load Balancing, ElastiCache, Relational Database Service, Amazon WorkSpaces and Lambda – were also affected by this event. We apologize for any inconvenience this event may have caused as we know how critical our services are to our customers. We are never satisfied with operational performance of our services that is anything less than perfect, and will do everything we can to learn from this event and drive improvement across our services.

We can confirm that all error rates and latencies have returned to normal levels. The issue has been resolved and the service is operating normally.

We have completed the restoration of the affected data store but are still working towards re-enabling writes. We have seen an improvement in successful launches over the last 20 minutes and expect that to continue as we work towards full recovery.

We continue to make steady progress towards the restoration of the affected data store and are currently within the 2 hours ETA published above.

We wanted to provide you with more details on the issue causing increased API error rates and latencies in the AP-SOUTHEAST-2 Region. A data store used by a subsystem responsible for the configuration of Virtual Private Cloud (VPC) networks is currently offline and the engineering team are working to restore it. While the investigation into the issue was started immediately, it took us longer to understand the full extent of the issue and determine a path to recovery. We determined that the data store needed to be restored to a point before the issue began. In order to do this restore, we needed to disable writes. Error rates and latencies for the networking-related APIs will continue until the restore has been completed and writes re-enabled. We are working through the recovery process now. With issues like this, it is always difficult to provide an accurate ETA, but we expect to complete the restore process within the next 2 hours and begin to allow API requests to proceed once again. We will continue to keep you updated if that ETA changes. Connectivity to existing instances is not impacted. Also, launch requests that refer to regional objects like subnets that already exist will succeed at this stage, as they do not depend on the affected subsystem. If you know the subnet ID, you can use that to launch instances within the region. We apologize for the impact and continue to work towards full resolution.

We continue to experience increased API error rates for the EC2 APIs in the AP-SOUTHEAST-2 Region. We have confirmed the root cause, and are working on multiple paths toward recovering the subsytem that is impaired, which is responsible for networking related API calls. This issue mainly affects EC2 RunInstance, and VPC related API requests. Customers using the EC2 Management Console may experience errors Describing Resources, as well as making mutating API requests. Connectivity to existing instances in the AP-SOUTHEAST-2 remains unaffected.

We have identified the root cause of the issue causing increased API error rates and latencies in the AP-SOUTHEAST-2 Region and continue working towards resolution. This issue mainly affects EC2 RunInstances and VPC related API requests. Customer using the EC2 Management Console will also experience error rates for instance and network related functions. Connectivity to existing instances remains unaffected.

We are investigating increased API error rates and latencies in the AP-SOUTHEAST-2 Region. Connectivity to existing instances is not impacted.

We have identified the root cause of the issue causing increased error rates and latencies for the EC2 and EBS APIs in the AP-SOUTHEAST-2 Region. Existing instances are not affected by this issue. We are working towards resolution of the issue.

Between 4:52 PM and 6:22 PM PST we experienced increased error rates and latencies for the EC2 and EBS APIs in the AP-SOUTHEAST-2 Region. The issue has been resolved and the service is operating normally.

We are seeing recovery for the increased error rates and latencies for the EC2 and EBS APIs in the AP-SOUTHEAST-2 Region. We continue to monitor error rates and latencies as we work towards full recovery.

We are investigating increased Console and API error rates in the AP-SOUTHEAST-2 Region.

We have identified the root cause of the issue causing increased error rates and latencies for the EC2 and EBS APIs in the AP-SOUTHEAST-2 Region. Existing instances are not affected by this issue. We are working towards to resolution of the issue.

We are investigating increased Console and API error rates in the AP-SOUTHEAST-2 Region.

We have resolved the network connectivity issue affecting some non-Nitro instances in a single Availability Zone in the AP-SOUTHEAST-2 Region. For instances that are still experiencing connectivity issues, we recommend issuing a reboot or stop/start command from the EC2 Management Console or EC2 API. The issue has been resolved and the service is operating normally.

We can confirm network connectivity issues for some non-Nitro instances in a single Availability Zone in the AP-SOUTHEAST-2 Region. Connectivity for affected instances can be restored using a stop/start command from the EC2 Management Console or EC2 API. We continue to work towards restoring network connectivity for the affected instances.

We are investigating network connectivity issues for some non-Nitro instances in a single Availability Zone in the AP-SOUTHEAST-2 Region.

We can confirm that instances have experienced a power event within a single Availability Zone in the AP-SOUTHEAST-2 Region. Error rates for the EC2 APIs have improved and launches of new EC2 instances are succeeding within the other Availability Zones in the Region.

We have restored power to the affected Availability Zone and are working to restore connectivity to the affected instances.

We continue to work on restoring connectivity to all instances in the affected Availability Zone. Launches of new EC2 instances are now starting to succeed in the affected Availability Zone but we are experiencing delays in updating instance state data.

We continue to make progress in restoring connectivity to all instances in the affected Availability Zone. Launches of new EC2 instances are now succeeding in all Availability Zones and instance state data is no longer delayed in the affected Availability Zone.

Connectivity has been restored for the majority of the instances in the affected Availability Zone and the APIs continue to operate normally. The remaining impacted instances are taking longer to recover. For immediate recovery, we recommend customers that are able to, launch replacement instances.

On June 4th at 10:25 PM PDT a significant number of EC2 instances and EBS volumes within a single Availability Zone in the AP-SOUTHEAST-2 Region experienced a loss of power. Beginning at this same time, EC2 API calls in the AP-SOUTHEAST-2 Region experienced increased error rates and latencies as well as delays in propagation of instance state data in the affected Availability Zone. Instances and volumes in the other Availability Zones in the AP-SOUTHEAST-2 Region were unaffected. At 11:46 PM PDT , power was restored to the facility and instances and volumes started to recover. At 1:00 AM PDT , 80% of the affected instances and volumes had been recovered by our automated systems. At 2:45 AM PDT the increased error rates and latencies for the EC2 APIs and the delayed propagation of instance state data were fully resolved. A couple of unexpected issues prevented our automated systems from recovering the remaining instances and volumes. The team was able to fix these issues, and by 8:00 AM PDT , all but a small number of instances and volumes were recovered. Since 8:00 AM PDT our teams have been working to recover these remaining instances and volumes. Most of these instances are hosted on hardware which was adversely affected by the loss of power. While we will continue to work to recover any affected instances or volumes, we recommend replacing any remaining affected instances or volumes if possible.

On June 4th at 10:25 PM PDT a significant number of EC2 instances and EBS volumes within a single Availability Zone in the AP-SOUTHEAST-2 Region experienced a loss of power. Beginning at this same time, EC2 API calls in the AP-SOUTHEAST-2 Region experienced increased error rates and latencies as well as delays in propagation of instance state data in the affected Availability Zone. Instances and volumes in the other Availability Zones were unaffected. At 11:46 PM PDT , power was restored to the facility and instances and volumes started to recover. By June 5th at 1:00 AM PDT , 80% of the affected instances and volumes had been recovered by our automated systems. At 2:45 AM PDT the increased error rates and latencies for the EC2 APIs and the delayed propagation of instance state data were fully resolved. A couple of unexpected issues prevented our automated systems from recovering the remaining instances and volumes. The team was able to fix these issues, and by 8:00 AM PDT , all but a small number of instances and volumes were recovered. Since 8:00 AM PDT our teams have been working to recover these remaining instances and volumes. Most of these instances are hosted on hardware which was adversely affected by the loss of power. While we will continue to work to recover any affected instances or volumes, we recommend replacing any remaining affected instances or volumes if possible.

Connectivity has been restored for the majority of the instances in the affected Availability Zone and the APIs continue to operate normally. The remaining impacted instances are taking longer to recover. For immediate recovery, we recommend customers that are able to, launch replacement instances.

We continue to make progress in restoring connectivity to all instances in the affected Availability Zone. Launches of new EC2 instances are now succeeding in all Availability Zones and instance state data is no longer delayed in the affected Availability Zone.

We continue to work on restoring connectivity to all instances in the affected Availability Zone. Launches of new EC2 instances are now starting to succeed in the affected Availability Zone but we are experiencing delays in updating instance state data.

We have restored power to the affected Availability Zone and are working to restore connectivity to the affected instances.

We can confirm that instances have experienced a power event within a single Availability Zone in the AP-SOUTHEAST-2 Region. Error rates for the EC2 APIs have improved and launches of new EC2 instances are succeeding within the other Availability Zones in the Region.

We continue to investigate connectivity issues for some instances in a single Availability Zone and increased API error rates for the EC2 APIs in the AP-SOUTHEAST-2 Region.

We are investigating increased connectivity issues for EC2 instances in the AP-SOUTHEAST-2 Region.

We are investigating increased connectivity issues for EC2 instances in the AP-SOUTHEST-2 Region.

Between 1:40 PM and 1:48 PM PDT we experienced an issue with an Internet provider close to our AP-SOUTHEAST-2 Region. The issue may have impacted connectivity between some customer networks and the Region. Connectivity within the Region was not impacted. The issue has been resolved and connectivity has been restored.

Between 12:27 PM and 2:01 PM PDT we experienced increased API error rates for instance launches and tagging actions in the AP-SOUTHEAST-2 Region. The issue has been resolved and the service is operating normally.

Between 12:27 PM and 2:01 PM PDT we experienced increased API error rates for instance launches and tagging actions in the AP-SOUTHEAST-2 Region. The issue has been resolved and the service is operating normally.

We are continuing to investigate increased API error rates for instance launches and tagging actions in the AP-SOUTHEAST-2 Region.

We are continuing to investigate increased API error rates for instance launches and tagging actions in the AP-SOUTHEAST-2 Region.

We are continuing to investigate increased API error rates for instance launches and tagging actions in the AP-SOUTHEAST-2 Region.

We are investigating increased error rates for new instance launches in the AP-SOUTHEAST-2 Region.

We are investigating increased error rates for new instance launches in the AP-SOUTHEAST-2 Region.

Between 7:16 PM and 8:50 PM PDT, a small number of EBS volumes in a single availability zone in the AP-SOUTHEAST-2 region experienced degraded performance, and between 7:16 PM and 8:17 PM PDT the AP-SOUTHEAST-2 region experienced elevated EBS API error rates. The issues have been resolved and the service is now operating normally.

Between 7:16 PM and 8:50 PM PDT, a small number of EBS volumes in a single availability zone in the AP-SOUTHEAST-2 region experienced degraded performance, and between 7:16 PM and 8:17 PM PDT the AP-SOUTHEAST-2 region experienced elevated EBS API error rates. The issues have been resolved and the service is now operating normally.

Most EBS volumes are now performing normally and EBS API error rates in the AP-SOUTHEAST-2 region are normal. We are continuing to investigate a small remaining number of EBS volumes experiencing degraded performance in a single availability zone.

Most EBS volumes are now performing normally and EBS API error rates in the AP-SOUTHEAST-2 region are normal. We are continuing to investigate a small remaining number of EBS volumes experiencing degraded performance in a single availability zone.

Most EBS volumes are now performing normally and EBS API error rates in the AP-SOUTHEAST-2 region are normal. We are continuing to investigate a small remaining number of EBS volumes experiencing degraded performance in a single availability zone.

Most EBS volumes are now performing normally and EBS API error rates in the AP-SOUTHEAST-2 region are normal. We are continuing to investigate a small remaining number of EBS volumes experiencing degraded performance in a single availability zone.

We continue to investigate degraded performance for a small number of EBS volumes in a single AZ and elevated EBS API error rates in the AP-SOUTHEAST-2 region

We continue to investigate degraded performance for a small number of EBS volumes in a single AZ and elevated EBS API error rates in the AP-SOUTHEAST-2 region

We are investigating degraded performance for some volumes in a single AZ in the ap-southeast-2 region

We are investigating degraded performance for some volumes in a single AZ in the ap-southeast-2 region

We are investigating degraded performance for some volumes in a single AZ in the ap-southeast-2 region

Between 8:13 PM PDT and 8:54 PM PDT we experienced increased error rates and latencies for DescribeReservedInstances, DescribeReservedInstancesOfferings, and DescribeReservedInstancesListings API calls in the AP-SOUTHEAST-2 region. The issue has been resolved and the service is operating normally.

Between 8:13 PM PDT and 8:54 PM PDT we experienced increased error rates and latencies for DescribeReservedInstances, DescribeReservedInstancesOfferings, and DescribeReservedInstancesListings API calls in the AP-SOUTHEAST-2 region. The issue has been resolved and the service is operating normally.

We are investigating increased error rates and latencies for the EC2 Reserved Instance APIs in the AP-SOUTHEAST-2 region.

We are investigating increased error rates and latencies for the EC2 Reserved Instance APIs in the AP-SOUTHEAST-2 region.

Between 2:42 AM and 3:23 AM PDT we experienced increased API error rates and latencies for DescribeInstanceStatus in the AP-SOUTHEAST-2 region. The issue has been resolved and the service is operating normally.

Between 2:42 AM and 3:23 AM PDT we experienced increased API error rates and latencies for DescribeInstanceStatus in the AP-SOUTHEAST-2 region. The issue has been resolved and the service is operating normally.

We are investigating increased API error rates and latencies in the AP-SOUTHEAST-2 region.

We are investigating increased API error rates and latencies in the AP-SOUTHEAST-2 region.

Between 6:36 AM and 6:47 AM PDT, Internet connectivity to a small number of instances in the AP-SOUTHEAST-2 region was impaired. The issue has been resolved and the service is operating normally.

Between 6:36 AM and 6:47 AM PDT, Internet connectivity to a small number of instances in the AP-SOUTHEAST-2 region was impaired. The issue has been resolved and the service is operating normally.

Between 6:36 AM and 6:47 AM PDT, Internet connectivity to a small number of instances in the AP-SOUTHEAST-2 region was impaired. The issue has been resolved and the service is operating normally.

Between Mar 31 11:27 PM PDT and 11:59 PM PDT internet connectivity to a small number of instances in the AP-SOUTHEAST-2 region was impaired. The issue has been resolved and the service is operating normally.

We are investigating connectivity issues for EC2 in the AP-SOUTHEAST-2 region.

Between 1:21 PM and 1:48 PM PDT, internet connectivity to a small number of instances in the AP-SOUTHEAST-2 region was impaired. The issue has been resolved and the service is operating normally.