CloudStatus

Iron.io Status Alerts

Iron.io
Iron service degradation

Nov 15, 02:59 PSTResolved - This incident has been resolved.Nov 15, 00:57 PSTMonitoring - A fix has been implemented and we are monitoring the results.

Iron.io
DNS issue with an upstream provider

Sep 20, 10:07 PDT Resolved - This incident has been resolved.Sep 20, 08:06 PDT Identified - Users may experience connection errors or delays. We have confirmed there is a DNS issue with an upstream provider and are monitoring the situation closely.

Iron.io
IronWorker issue with Scheduled tasks: aws network died on few instances

Sep 8, 07:21 PDT Resolved - This incident has been resolved.Sep 8, 07:09 PDT Monitoring - A fix has been implemented and we are monitoring the results.Sep 8, 06:46 PDT Identified - The issue has been identified and a fix is being implemented.

Iron.io
mq-aws-eu-west-1-1

Jun 14, 00:32 PDT Resolved - This incident has been resolved.Jun 13, 15:51 PDT Investigating - Investigating issues with mq-aws-eu-west-1-1

Iron.io
Aws network issues are causing IronWorker service degradation

Apr 27, 03:14 PDT Resolved - This incident has been resolved.Apr 27, 01:25 PDT Identified - Aws network issues are causing Ironworker service degradation

Iron.io
IronWorker Service degradation

Mar 23, 12:00 PDT Resolved - This incident has been resolved.Mar 23, 09:46 PDT Identified - Our engineering team is actively resolving the slow processing tasks on the shared public cluster. They are undertaking necessary steps to improve the system.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 15:16 PSTResolved - The system has now almost fully caught up. We're continuing to scan for any residual jobs that may have not run but all should have ran, or be queued up to run shortly. Thank you for your patience as AWS recovered their core services. We will be evaluating options of running core iaas outside of AWS.Feb 28, 13:54 PSTMonitoring - Job processing is almost fully up to speed again. It may take awhile to get through the backlog of jobs.Feb 28, 13:32 PSTUpdate - We are now seeing recovery of IronWorker and working through backlogs of jobs.Feb 28, 12:59 PSTUpdate - We see jobs going through again... none should be lost but they will be queued up since the issues started this morning.Feb 28, 12:58 PSTUpdate - Update from AWS. We are quickly trying to restore our services as well: Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.Feb 28, 12:48 PSTUpdate - Unfortunately the issue has now cascaded to over 45 AWS services causing unrecoverable issues upstream. At this point, we have to wait on AWS and then begin a fully multi-cloud initiative.Feb 28, 12:02 PSTUpdate - We are considering bypassing s3 but even then, Docker hub is down and would block any Upstream updating of code packages as they are all built with Docker.Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 13:54 PSTMonitoring - Job processing is almost fully up to speed again. It may take awhile to get through the backlog of jobs.Feb 28, 13:32 PSTUpdate - We are now seeing recovery of IronWorker and working through backlogs of jobs.Feb 28, 12:59 PSTUpdate - We see jobs going through again... none should be lost but they will be queued up since the issues started this morning.Feb 28, 12:58 PSTUpdate - Update from AWS. We are quickly trying to restore our services as well: Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.Feb 28, 12:48 PSTUpdate - Unfortunately the issue has now cascaded to over 45 AWS services causing unrecoverable issues upstream. At this point, we have to wait on AWS and then begin a fully multi-cloud initiative.Feb 28, 12:02 PSTUpdate - We are considering bypassing s3 but even then, Docker hub is down and would block any Upstream updating of code packages as they are all built with Docker.Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 13:32 PSTUpdate - We are now seeing recovery of IronWorker and working through backlogs of jobs.Feb 28, 12:59 PSTUpdate - We see jobs going through again... none should be lost but they will be queued up since the issues started this morning.Feb 28, 12:58 PSTUpdate - Update from AWS. We are quickly trying to restore our services as well: Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.Feb 28, 12:48 PSTUpdate - Unfortunately the issue has now cascaded to over 45 AWS services causing unrecoverable issues upstream. At this point, we have to wait on AWS and then begin a fully multi-cloud initiative.Feb 28, 12:02 PSTUpdate - We are considering bypassing s3 but even then, Docker hub is down and would block any Upstream updating of code packages as they are all built with Docker.Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 12:59 PSTUpdate - We see jobs going through again... none should be lost but they will be queued up since the issues started this morning.Feb 28, 12:58 PSTUpdate - Update from AWS. We are quickly trying to restore our services as well: Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.Feb 28, 12:48 PSTUpdate - Unfortunately the issue has now cascaded to over 45 AWS services causing unrecoverable issues upstream. At this point, we have to wait on AWS and then begin a fully multi-cloud initiative.Feb 28, 12:02 PSTUpdate - We are considering bypassing s3 but even then, Docker hub is down and would block any Upstream updating of code packages as they are all built with Docker.Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 12:48 PSTUpdate - Unfortunately the issue has now cascaded to over 45 AWS services causing unrecoverable issues upstream. At this point, we have to wait on AWS and then begin a fully multi-cloud initiative.Feb 28, 12:02 PSTUpdate - We are considering bypassing s3 but even then, Docker hub is down and would block any Upstream updating of code packages as they are all built with Docker.Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 12:02 PSTUpdate - We are considering bypassing s3 but even then, Docker hub is down and would block any Upstream updating of code packages as they are all built with Docker.Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass their S3 service completely. We will build around it in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 10:08 PSTUpdate - Reported S3 issues in US-East: https://status.aws.amazon.com/ Trying to bypass S3. We will build around S3 in the future.Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 09:57 PSTUpdate - https://news.ycombinator.com/item?id=13755673Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
AWS S3 is down now causing job processing issues... stand by please while we try and reroute around it

Feb 28, 09:57 PSTIdentified - The issue has been identified and a fix is being implemented.

Iron.io
Slower than usual job processing from 7:20am PT to 7:40am PT [resolved]

Feb 28, 08:46 PSTResolved - This issue was resolved at 7:40am PT, we are creating a record to log the times. A routine deployment caused job processing to slow down for about 20 minutes this morning. We caught the slowdown and fixed the issue restoring processing speeds. All jobs ran as expected, some with a slight delay. The Iron.io Ops Team

Iron.io
Slower than usual job processing from 7:20am PT to 7:40am PT [resolved]

Feb 28, 07:22 PSTResolved - This issue was resolved at 7:40am PT, we are creating a record to log the times. A routine deployment caused an unexpected slow down in job processing for about 20 minutes this morning. We caught the slowdown and fixed the issue restoring processing speeds. All jobs ran but some with a slight delay. We are improving our canary deployments and internal load testing to discover these issues before they reach production. The Iron.io Ops Team

Iron.io
IronWorker Service degradation

Feb 2, 11:44 PSTResolved - The issue is now resolved. We're working on a limiting process for the API calls that were putting strain on our databases and should have something soon to avoid this from happening again in the future. If you have any specific questions please contact support@iron.io or your dedicated support channel.Feb 2, 11:23 PSTUpdate - We've discovered the source of the database slowdown and are taking corrective actions to build a permanent fix into the system so this does not happen again. Thanks for your patience while we work through slower than usual queue times. The system did not lose jobs, but rather saw slowdowns in queue times.Feb 2, 10:47 PSTMonitoring - The database has stabilized and we are sweeping projects for queued tasks. You may notice some tasks that ran long. Many of them probably finished just did not exit properly and mark as "complete". You can verify by viewing the task logs. We'll update again as we monitor the system.Feb 2, 10:36 PSTUpdate - The cause of the slowdowns is in a backend database under high load. We're searching for the source of the load and as soon as that's found we'll update this page. That said we've restarted the DB and it's mostly recovered so tasks should start flowing again. Stand by.Feb 2, 10:04 PSTInvestigating - We have identified a system slow down with IronWorker. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Feb 2, 11:23 PSTUpdate - We've discovered the source of the database slowdown and are taking corrective actions to build a permanent fix into the system so this does not happen again. Thanks for your patience while we work through slower than usual queue times. The system did not lose jobs, but rather saw slowdowns in queue times.Feb 2, 10:47 PSTMonitoring - The database has stabilized and we are sweeping projects for queued tasks. You may notice some tasks that ran long. Many of them probably finished just did not exit properly and mark as "complete". You can verify by viewing the task logs. We'll update again as we monitor the system.Feb 2, 10:36 PSTUpdate - The cause of the slowdowns is in a backend database under high load. We're searching for the source of the load and as soon as that's found we'll update this page. That said we've restarted the DB and it's mostly recovered so tasks should start flowing again. Stand by.Feb 2, 10:04 PSTInvestigating - We have identified a system slow down with IronWorker. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Feb 2, 10:47 PSTMonitoring - The database has stabilized and we are sweeping projects for queued tasks. You may notice some tasks that ran long. Many of them probably finished just did not exit properly and mark as "complete". You can verify by viewing the task logs. We'll update again as we monitor the system.Feb 2, 10:36 PSTUpdate - The cause of the slowdowns is in a backend database under high load. We're searching for the source of the load and as soon as that's found we'll update this page. That said we've restarted the DB and it's mostly recovered so tasks should start flowing again. Stand by.Feb 2, 10:04 PSTInvestigating - We have identified a system slow down with IronWorker. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Feb 2, 10:36 PSTUpdate - The cause of the slowdowns is in a backend database under high load. We're searching for the source of the load and as soon as that's found we'll update this page. That said we've restarted the DB and it's mostly recovered so tasks should start flowing again. Stand by.Feb 2, 10:04 PSTInvestigating - We have identified a system slow down with IronWorker. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Feb 2, 10:04 PSTInvestigating - We have identified a system slow down with IronWorker. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Public cluster push queues are delayed but still sending

Jan 18, 17:07 PSTResolved - The issue has been resolved and all messages in push queues have now been flushed and delivered. Note: no messages were lost, but many were significantly delayed in their delivery to endpoints. Additionally, we've identified and patched the root causes that have been causing delays in push queues over the past few weeks. This should significantly improve push queue latency. We apologize for any application issues this may have caused. We take these issues seriously and are working around the clock to improve the service across the board for users of all paid tiers. That said, if you need guaranteed reliability and performance, and SLA's in place, please contact us about our enterprise dedicated cluster offering. To discuss dedicated clusters, or anything else about this incident, you can contact us through our in-app messenger, or by emailing support@iron.io.Jan 18, 13:29 PSTUpdate - We are preparing to deploy fixes to improve the push queue delays, please stand by.Jan 18, 08:03 PSTIdentified - We’re working on resolving the issues as quickly as possible.

Iron.io
Public cluster push queues are delayed but still sending

Jan 18, 17:07 PSTResolved - The issue has been resolved and all messages in push queues have now been flushed and delivered. Note: no messages were lost, but many were significantly delayed in their delivery to endpoints. Additionally, we've identified and patched the root causes that have been causing delays in push queues over the past few weeks. We've found some edge cases around users creating hundreds of thousands of push queues with no subscribers causing the push processor to get caught up working on nothing for a long time causing delays for other customers. We patched things so that the push processor will bypass queues in those cases. This should significantly improve push queue performance. We apologize for any application issues this may have caused. We take these issues seriously and are working around the clock to improve the service across the board for users of all paid tiers. That said, if you need guaranteed reliability and performance, and SLA's in place, please contact us about our enterprise dedicated cluster offering. To discuss dedicated clusters, or anything else about this incident, you can contact us through our in-app messenger, or by emailing support@iron.io.Jan 18, 13:29 PSTUpdate - We are preparing to deploy fixes to improve the push queue delays, please stand by.Jan 18, 08:03 PSTIdentified - We’re working on resolving the issues as quickly as possible.

Iron.io
Public cluster push queues are delayed but still sending

Jan 18, 13:29 PSTUpdate - We are preparing to deploy fixes to improve the push queue delays, please stand by.Jan 18, 08:03 PSTIdentified - We’re working on resolving the issues as quickly as possible.

Iron.io
Public cluster push queues are delayed but still sending

Jan 18, 08:03 PSTIdentified - We’re working on resolving the issues as quickly as possible.

Iron.io
Delay in Tasks with Priority 0

Dec 21, 11:17 PSTResolved - This incident has been resolved.Dec 21, 09:54 PSTMonitoring - We have identified the issue and implemented a solutionDec 21, 08:53 PSTIdentified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.

Iron.io
Delay in Tasks with Priority 0

Dec 21, 09:54 PSTMonitoring - We have identified the issue and implemented a solutionDec 21, 08:53 PSTIdentified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.

Iron.io
Delay in Tasks with Priority 0

Dec 21, 08:53 PSTIdentified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.

Iron.io
IronMQ v2 us-east Issues

Dec 13, 15:14 PSTResolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Dec 13, 14:25 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
IronMQ v2 us-east Issues

Dec 13, 14:25 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue identified in mq-aws-us-east-1-2.iron.io

Nov 30, 10:20 PSTResolved - This incident has been resolved.Nov 30, 07:56 PSTMonitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Nov 30, 07:32 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue identified in mq-aws-us-east-1-2.iron.io

Nov 30, 07:56 PSTMonitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Nov 30, 07:32 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue identified in mq-aws-us-east-1-2.iron.io

Nov 30, 07:32 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Nov 22, 11:52 PSTResolved - This incident has been resolved.Nov 22, 10:48 PSTMonitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Nov 22, 10:43 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Nov 22, 10:48 PSTMonitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Nov 22, 10:43 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Nov 22, 10:43 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Nov 22, 09:37 PSTResolved - This incident has been resolved.Nov 22, 09:29 PSTMonitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Nov 22, 09:15 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Nov 22, 09:29 PSTMonitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Nov 22, 09:15 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Nov 22, 09:15 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Nov 22, 09:15 PSTInvestigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
IronCache system issue

Nov 16, 07:59 PSTResolved - We have resolved the IronCache issue and all systems are operational. We will continue to monitor status.Nov 16, 07:07 PSTMonitoring - The Iron Operations team has updated the system, and metrics are returning to normal. Iron will continue to monitor.Nov 16, 07:01 PSTInvestigating - Our engineering team has identified an issue with IronCache and is working on the issue. We will post an update as soon as information becomes available.

Iron.io
IronCache system issue

Nov 16, 07:07 PSTMonitoring - The Iron Operations team has updated the system, and metrics are returning to normal. Iron will continue to monitor.Nov 16, 07:01 PSTInvestigating - Our engineering team has identified an issue with IronCache and is working on the issue. We will post an update as soon as information becomes available.

Iron.io
IronCache system issue

Nov 16, 07:01 PSTInvestigating - Our engineering team has identified an issue with IronCache and is working on the issue. We will post an update as soon as information becomes available.

Iron.io
Issue affecting MQv2 in us-east-1

Nov 4, 19:44 PDT Resolved - This incident has been resolved.Nov 4, 18:37 PDT Identified - We have identified the issue coming from an upstream vendor. Our Engineering team is working to get this resolved.Nov 4, 18:05 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Issue affecting MQv2 in us-east-1

Nov 4, 18:37 PDT Identified - We have identified the issue coming from an upstream vendor. Our Engineering team is working to get this resolved.Nov 4, 18:05 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Issue affecting MQv2 in us-east-1

Nov 4, 18:05 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IO domain DNS failure

Oct 31, 05:28 PDT Resolved - This incident has been resolved.Oct 31, 03:26 PDT Monitoring - We have identified Iron.io services as being affected by the larger DNS outage affecting the entire .io top-level domain. We continue to monitor the situation.

Iron.io
IO domain DNS failure

Oct 31, 03:26 PDT Monitoring - We have identified Iron.io services as being affected by the larger DNS outage affecting the entire .io top-level domain. We continue to monitor the situation.

Iron.io
IO domain DNS failure

Oct 28, 16:01 PDT Postmortem - Today, 10-28-2016, AWS confirmed that Route 53 DNS services were interrupted. This primarily affected domains that end in .io. The was causing nameservers to return NXDOMAIN intermittently for domains that do exist. The issue has been resolved. For more detailed information, please see https://news.ycombinator.com/item?id=12813065Oct 28, 16:01 PDT Resolved - This incident has been resolved.Oct 28, 16:01 PDT Investigating - Wee are recovering after an upstream provider had a DNS issue

Iron.io
DNS Service degradation

Oct 21, 13:49 PDT Resolved - This incident has been resolved.Oct 21, 10:14 PDT Monitoring - We have identified Iron.io services as being affected by the larger DNS outage affecting the Internet this morning. The larger DNS issue is being caused by a DDoS attack affecting core Internet infrastructure. Customers should expect intermittent service as DNS services are restored.

Iron.io
DNS Service degradation

Oct 21, 10:14 PDT Monitoring - We have identified Iron.io services as being affected by the larger DNS outage affecting the Internet this morning. The larger DNS issue is being caused by a DDoS attack affecting core Internet infrastructure. Customers should expect intermittent service as DNS services are restored.

Iron.io
DNS Service degradation

Oct 21, 07:00 PDT Resolved - This issue has been resolved and the service is operating normally.Oct 21, 06:45 PDT Monitoring - We have identified Iron.io services as being affected by the larger DNS outage affecting the Internet this morning. The larger DNS issue is being caused by a DDoS attack affecting core Internet infrastructure. Customers should expect intermittent service as DNS services are restored. We continue to monitor the situation.

Iron.io
DNS Service degradation

Oct 21, 06:45 PDT Monitoring - We have identified Iron.io services as being affected by the larger DNS outage affecting the Internet this morning. The larger DNS issue is being caused by a DDoS attack affecting core Internet infrastructure. Customers should expect intermittent service as DNS services are restored. We continue to monitor the situation.

Iron.io
Iron system issue

Oct 14, 10:51 PDT Resolved - We have resolved the issue and all systems are returning back to operational. We will continue to monitor status.Oct 14, 09:50 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of working the issue.Oct 14, 09:50 PDT Update - There is no ETA at the moment, but we have identified a solution and are in the process of working the issue.Oct 14, 08:10 PDT Investigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Oct 14, 09:50 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of working the issue.Oct 14, 09:50 PDT Update - There is no ETA at the moment, but we have identified a solution and are in the process of working the issue.Oct 14, 08:10 PDT Investigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Oct 14, 08:10 PDT Investigating - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.

Iron.io
Higher than expected latency in IronWorker API.

Oct 4, 12:53 PDT Resolved - This incident has been resolved.Oct 4, 12:48 PDT Monitoring - A fix has been implemented and we are monitoring the results.Oct 4, 12:33 PDT Identified - The issue has been identified and a fix is being implemented.

Iron.io
Higher than expected latency in IronWorker API.

Oct 4, 12:48 PDT Monitoring - A fix has been implemented and we are monitoring the results.Oct 4, 12:33 PDT Identified - The issue has been identified and a fix is being implemented.

Iron.io
Higher than expected latency in IronWorker API.

Oct 4, 12:33 PDT Identified - The issue has been identified and a fix is being implemented.

Iron.io
Iron system issue

Sep 20, 08:33 PDT Resolved - We have identified the issue and resolved it. Services affected should be functional again.Sep 20, 08:27 PDT Monitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Sep 20, 08:15 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Sep 20, 08:27 PDT Monitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Sep 20, 08:15 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Iron system issue

Sep 20, 08:15 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Public Cluster Latency

Sep 16, 07:20 PDT Resolved - Latency on the IronWorker Public cluster have returned to normal levels.Sep 16, 07:02 PDT Investigating - Operations has identified higher than normal latency on the public IronWorker cluster. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Public Cluster Latency

Sep 16, 07:02 PDT Investigating - Operations has identified higher than normal latency on the public IronWorker cluster. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Public Cluster Latency

Sep 16, 07:02 PDT Investigating - Operations has identified higher than normal latency on the public IronWorker cluster. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 22:44 PDT Resolved - All clusters and instances of the IronWorker system are fully operational at this point in time. Any tasks that may have been delayed on public clusters are being requeued. We will continue to monitor the system throughout the night and weekend.Sep 2, 20:58 PDT Update - Our systems are continuing to stabilize. We've launched additional capacity in runners, and tasks are processing. However, there are still higher than typical wait periods. All tasks have been replicated and preserved. Any delayed tasks are being requeued and processed.Sep 2, 18:42 PDT Update - Additional issues have arisen, and Operations continues to investigate. We project to have a new ETA within a matter hours.Sep 2, 16:48 PDT Update - Tasks are processing but performance is still degraded. We are looking at an ETA of about 2 hours until we believe we will have the issues fully resolved.Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 22:44 PDT Resolved - All clusters and instances of the IronWorker system are fully operational at this point in time. Any tasks that may have been delayed on public clusters are being requeued. We will continue to closely monitor the system throughout the night and weekend.Sep 2, 20:58 PDT Update - Our systems are continuing to stabilize. We've launched additional capacity in runners, and tasks are processing. However, there are still higher than typical wait periods. All tasks have been replicated and preserved. Any delayed tasks are being requeued and processed.Sep 2, 18:42 PDT Update - Additional issues have arisen, and Operations continues to investigate. We project to have a new ETA within a matter hours.Sep 2, 16:48 PDT Update - Tasks are processing but performance is still degraded. We are looking at an ETA of about 2 hours until we believe we will have the issues fully resolved.Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 20:58 PDT Update - Our systems are continuing to stabilize. We've launched additional capacity in runners, and tasks are processing. However, there are still higher than typical wait periods. All tasks have been replicated and preserved. Any delayed tasks are being requeued and processed.Sep 2, 18:42 PDT Update - Additional issues have arisen, and Operations continues to investigate. We project to have a new ETA within a matter hours.Sep 2, 16:48 PDT Update - Tasks are processing but performance is still degraded. We are looking at an ETA of about 2 hours until we believe we will have the issues fully resolved.Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 18:42 PDT Update - Additional issues have arisen, and Operations continues to investigate. We project to have a new ETA within a matter hours.Sep 2, 16:48 PDT Update - Tasks are processing but performance is still degraded. We are looking at an ETA of about 2 hours until we believe we will have the issues fully resolved.Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 18:42 PDT Update - Additional issues have arisen, and Operations continues to investigate. We project to have a new ETA within a matter hourSep 2, 16:48 PDT Update - Tasks are processing but performance is still degraded. We are looking at an ETA of about 2 hours until we believe we will have the issues fully resolved.Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 18:42 PDT Update - Additional issues have arisen, and Operations continues to investigate. We project to have a new ETA within a matter hours.Sep 2, 16:48 PDT Update - Tasks are processing but performance is still degraded. We are looking at an ETA of about 2 hours until we believe we will have the issues fully resolved.Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 16:48 PDT Update - Tasks are processing but performance is still degraded. We are looking at an ETA of about 2 hours until we believe we will have the issues fully resolved.Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 15:02 PDT Update - Performance problems have appeared today with the IronWorker service due to resource issues in a primary datastore for IronWorker tasks. We are currently addressing the issue by expanding the clusters and are currently in the process of building indexes now. As soon as these clusters are prepared, we will failover to it so that the performance problem will be resolved.Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 12:51 PDT Monitoring - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 12:51 PDT Monitoring - A fix has been implemented and we are monitoring the results.Sep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 12:32 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. More information will be posted as it is availableSep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 11:50 PDT Update - There is no ETA at this time, but the engineering and operations teams have found the cause of the issue and are working on resolving it.Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 10:24 PDT Identified - While there is no ETA at the moment, engineering has identified the issue and are in the resolution process.Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker Service degradation

Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Increased error rate in SWAPI

Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Increased error rate in SWAPI

Sep 2, 08:52 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
On 8/30 From 6:05am PST - 11:10am PST Some IronWorker customers experienced a decreased performance. The issue has been resolved and we are continuing to monitor.

Aug 30, 12:09 PDT Resolved - This incident has been resolved.Aug 30, 11:33 PDT Monitoring - A fix has been implemented and we are monitoring the results.

Iron.io
On 8/30 From 6:05am PST - 11:10am PST Some IronWorker customers experienced a decreased performance. The issue has been resolved and we are continuing to monitor.

Aug 30, 11:33 PDT Monitoring - A fix has been implemented and we are monitoring the results.

Iron.io
Timeout on HUD-e interface when trying to access queues in MQv3 eu-west

Aug 5, 13:17 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Aug 5, 13:16 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Timeout on HUD-e interface when trying to access queues in MQv3 eu-west

Aug 5, 13:16 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronMQ V3 is degraded

Jul 29, 13:09 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Jul 29, 02:46 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronMQ V3 is degraded

Jul 29, 02:46 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronMQ v3 (AWS US-East) issue

Jul 26, 14:48 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Jul 26, 08:11 PDT Monitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.

Iron.io
IronMQ v3 (AWS US-East) issue

Jul 26, 14:48 PDT Resolved - Service has been restored to functional status. Due to an issue with connections not closing properly on the hosts in the original public v3 cluster, this required a necessary failover to a new v3 cluster. During this timeframe, some customer queues and messages were lost due to the deteriorated health of the original cluster. Upon failing over to the new MQv3 cluster, most customers were able to see their queues and messages. In the days following service restoration, we have been able to migrate customer messages off of the failed cluster. Unfortunately, some messages have not been retrievable. Development Operations will continue all efforts to migrate all available messages to the new v3 cluster, and expect this laborious procedure will continue over the next few days. All customers whose queues and messages were recoverable have now been moved onto the new cluster. Development Operations is identifying the list of customers whose messages were not recoverable, in order to confirm message status.Jul 26, 08:11 PDT Monitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.

Iron.io
IronMQ v3 (AWS US-East) issue

Jul 26, 08:11 PDT Monitoring - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.

Iron.io
Increased 500 and 400 responses

Jul 25, 16:22 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Jul 25, 16:05 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Increased 500 and 400 responses

Jul 25, 16:05 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Service degradation of MQv3 public

Jul 23, 07:33 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Jul 22, 21:14 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.Jul 22, 20:36 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Service degradation of MQv3 public

Jul 22, 21:14 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.Jul 22, 20:36 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Service degradation of MQv3 public

Jul 22, 20:36 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Error 500 and slowness in MQv3 Public US East

Jul 16, 19:02 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Jul 16, 16:24 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Error 500 and slowness in MQv3 Public US East

Jul 16, 16:24 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
10% of scheduled tasks ran twice during a discreet time window

Jul 5, 11:54 PDT Resolved - Between 5:48pm, June 30th, and 11:48am, July 1st, an incident with the scheduler caused 10% of scheduled tasks to be executed twice. Engineering is implementing a networking safeguard to prevent a reoccurrence. Edit

Iron.io
10% of scheduled tasks ran twice during a discreet time window

Jul 5, 11:54 PDT Resolved - Between 5:48pm, June 30th, and 11:48am, July 1st, an incident with the scheduler caused 10% of scheduled tasks to be executed twice. Engineering is implementing a networking safeguard to prevent a reoccurrence.

Iron.io
Test of Status Page notification system

Jun 28, 16:07 PDT Resolved - This incident has been resolved.Jun 28, 16:07 PDT Investigating - Test - please ignore

Iron.io
Maintenance work

Jun 28, 00:36 PDT Resolved - Maintenance is completeJun 28, 00:25 PDT Monitoring - A fix has been implemented and we are monitoring the results.

Iron.io
Maintenance work

Jun 28, 00:25 PDT Monitoring - A fix has been implemented and we are monitoring the results.

Iron.io
IronWorker & MQv2 degraded performance due to stability issues with upstream provider

Jun 24, 22:48 PDT Resolved - We have seen system health return to normal. Rackspace has notified us that their system health is also improving. IronWorker and MQv2 are fully functional.Jun 24, 22:45 PDT Update - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Jun 24, 22:34 PDT Update - We have been notified by Rackspace that they have partially recovered. The system health of IronWorker and MQv2 are looking normal again. We are waiting on more confirmation from Rackspace before we mark this as fully resolved.Jun 24, 22:06 PDT Monitoring - We have received notice from Rackspace that they are currently experiencing issues in one of their data centres according to this notification: https://status.rackspace.com/index/viewincidents?group=22&start=1466827016&guid=3116 . It is possible that IronWorker tasks may occasionally fail to post or fail or update status.

Iron.io
IronWorker & MQv2 degraded performance due to stability issues with upstream provider

Jun 24, 22:45 PDT Update - We have identified the issue and implemented a solution. We expect to post our return to normal status momentarily.Jun 24, 22:34 PDT Update - We have been notified by Rackspace that they have partially recovered. The system health of IronWorker and MQv2 are looking normal again. We are waiting on more confirmation from Rackspace before we mark this as fully resolved.Jun 24, 22:06 PDT Monitoring - We have received notice from Rackspace that they are currently experiencing issues in one of their data centres according to this notification: https://status.rackspace.com/index/viewincidents?group=22&start=1466827016&guid=3116 . It is possible that IronWorker tasks may occasionally fail to post or fail or update status.

Iron.io
IronWorker & MQv2 degraded performance due to stability issues with upstream provider

Jun 24, 22:34 PDT Update - We have been notified by Rackspace that they have partially recovered. The system health of IronWorker and MQv2 are looking normal again. We are waiting on more confirmation from Rackspace before we mark this as fully resolved.Jun 24, 22:06 PDT Monitoring - We have received notice from Rackspace that they are currently experiencing issues in one of their data centres according to this notification: https://status.rackspace.com/index/viewincidents?group=22&start=1466827016&guid=3116 . It is possible that IronWorker tasks may occasionally fail to post or fail or update status.

Iron.io
IronWorker & MQv2 degraded performance due to stability issues with upstream provider

Jun 24, 22:06 PDT Monitoring - We have received notice from Rackspace that they are currently experiencing issues in one of their data centres according to this notification: https://status.rackspace.com/index/viewincidents?group=22&start=1466827016&guid=3116 . It is possible that IronWorker tasks may occasionally fail to post or fail or update status.

Iron.io
IronWorker & MQv2 degraded performance due to stability issues with upstream provider

Jun 24, 22:06 PDT Monitoring - We have received notice from Rackspace that they are currently experiencing issues in their IAD data centre according to this notification: https://status.rackspace.com/index/viewincidents?group=22&start=1466827016&guid=3116 . It is possible that IronWorker tasks may occasionally fail to post or fail or update status.

Iron.io
IronWorker & MQv2 degraded performance due to issues with Rackspace and OR stability

Jun 24, 22:06 PDT Monitoring - We have received notice from Rackspace that they are currently experiencing issues in their IAD data centre according to this notification: https://status.rackspace.com/index/viewincidents?group=22&start=1466827016&guid=3116 . It is possible that IronWorker tasks may occasionally fail to post or fail or update status.

Iron.io
IronWorker mem1 Service Degradation due to AWS Issues.

Jun 9, 14:17 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Jun 9, 13:34 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker mem1 Service Degradation due to AWS Issues.

Jun 9, 13:34 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Due to AWS Virginia API problems IronWorker customers may experience degradation of service.

Jun 9, 13:34 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Delayed push queue messages on MQv2 US East

Jun 9, 11:41 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Jun 9, 06:30 PDT Monitoring - Issue has been fixed, but we still migrating old messages.Jun 8, 19:12 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Delayed push queue messages on MQv2 US East

Jun 9, 06:30 PDT Monitoring - Issue has been fixed, but we still migrating old messages.Jun 8, 19:12 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Delayed push queue messages on MQv2 US East

Jun 8, 19:12 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Degradation Of Service On Mem1 Cluster on Public Service

May 18, 10:43 PDT Resolved - The service is back to the operational state.May 18, 09:27 PDT Update - We are currently investigating the issue.May 18, 09:21 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Degradation Of Service On Mem1 Cluster on Public Service

May 18, 09:27 PDT Update - We are currently investigating the issue.May 18, 09:21 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Degradation Of Service On Mem1 Cluster

May 18, 09:27 PDT Update - We are currently investigating the issue.May 18, 09:21 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Partial outage

May 18, 09:27 PDT Update - We are currently investigating the issue.May 18, 09:21 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Partial outage

May 18, 09:21 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
tars The messages in the queue are not bing processed

May 17, 08:08 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.May 17, 08:02 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
The messages in the queue are not bing processed

May 17, 08:08 PDT Resolved - After investigation we have concluded that all queues are operating normally. No customers were affected.May 17, 08:02 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
tars The messages in the queue are not bing processed

May 17, 08:02 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker issues in AWS

May 15, 18:59 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.May 15, 18:26 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. We expect to post an update momentarily.May 15, 17:23 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.May 15, 15:38 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker issues in AWS

May 15, 18:26 PDT Update - Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. We expect to post an update momentarily.May 15, 17:23 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.May 15, 15:38 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker issues in AWS

May 15, 17:23 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.May 15, 15:38 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
IronWorker issues in AWS

May 15, 15:38 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
tars instances not being launched in AWS from SD. says “queuing” but never actually launches

May 15, 15:38 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
mq-v3-aws-eu-west issues failed over to backup

May 7, 02:28 PDT Resolved - This incident has been resolved.

Iron.io
Iron Worker system issue, Performance degraded on public cluster

May 4, 08:55 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.May 4, 08:51 PDT Identified - There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix.May 4, 07:37 PDT Update - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.May 4, 05:59 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Iron Worker system issue, Performance degraded on public cluster

May 4, 07:37 PDT Update - Our engineering team is diligently working on the issue at hand. We will post an update as soon as information becomes available.May 4, 05:59 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Iron Worker system issue, Performance degraded on public cluster

May 4, 05:59 PDT Investigating - We have identified a system issue. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Iron is experiencing system issues

Apr 21, 14:55 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Apr 21, 14:22 PDT Investigating - We have identified a system issue with IronWorker. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Iron is experiencing system issues

Apr 21, 14:22 PDT Investigating - We have identified a system issue with IronWorker. This is a high priority issue, and our Operations team is actively investigating. We will post an update as soon as information becomes available.

Iron.io
Increased latency to enqueue IronWorker tasks

Apr 20, 12:04 PDT Resolved - We have resolved the issue and all systems are operational. We will continue to monitor status.Apr 20, 11:05 PDT Investigating - We have identified an issue with IronWorker. This is a high priority issue, and our Operations team is actively investigation mode. We will post an update as soon as information becomes available.

Iron.io
Increased latency to enqueue IronWorker tasks

Apr 20, 11:05 PDT Investigating - We have identified an issue with IronWorker. This is a high priority issue, and our Operations team is actively investigation mode. We will post an update as soon as information becomes available.

Iron.io
Auth Service outage

Apr 16, 04:20 PDT Resolved - This incident has been resolved.Apr 16, 03:46 PDT Identified - The issue has been identified and a fix is being implemented.

Iron.io
Auth Service outage

Apr 16, 03:46 PDT Identified - The issue has been identified and a fix is being implemented.

Iron.io
IronWorker API increased number of errors

Apr 1, 11:55 PDT Resolved - This incident has been resolved.Apr 1, 00:28 PDT Monitoring - A fix has been implemented and we are monitoring the results.Mar 31, 23:25 PDT Investigating - We are currently investigating this issue.

Iron.io
IronWorker API increased number of errors

Apr 1, 00:28 PDT Monitoring - A fix has been implemented and we are monitoring the results.Mar 31, 23:25 PDT Investigating - We are currently investigating this issue.

Iron.io
IronWorker API increased number of errors

Mar 31, 23:25 PDT Investigating - We are currently investigating this issue.

Iron.io
IronWorker degraded performance

Mar 31, 14:09 PDT Resolved - This incident has been resolved.Mar 31, 12:42 PDT Monitoring - We have implemented a fix and are monitoring the system.

Iron.io
IronWorker degraded performance

Mar 31, 12:42 PDT Monitoring - We have implemented a fix and are monitoring the system.

Iron.io
IronWorker degraded performance

Mar 8, 10:02 PDT Resolved - This incident has been resolved.Mar 8, 08:17 PDT Monitoring - We have identified the issue and have deployed a fix. Operations should be normal, and we will continue to monitor the situation.Mar 8, 06:46 PDT Investigating - We looking into an issue with IronWorker. This is a high priority issue; we do not have any updates at this time, our engineers are actively investigating. We will post an update as soon as we have more information.Mar 8, 06:10 PDT Monitoring - We have identified the issue, and implemented a solution. We will continue to monitor systems for the next few hours.Mar 8, 05:21 PDT Investigating - We are investigating an issue with IronWorker. This is a high priority issue; we do not have any updates at this time, our engineers are actively troubleshooting the issue. We will post an update as soon as we have more information.

Iron.io
mq-aws-us-east-1-1 is recovering from an earlier issue

Mar 2, 15:02 PDT Resolved - This incident has been resolved.Mar 2, 14:40 PDT Investigating - We are currently investigating this issue.

Iron.io
IronWorker on Mem1 is Seeing Downgraded Performance

Feb 23, 13:05 PDT Resolved - This incident has been resolved.Feb 23, 12:07 PDT Identified - The issue has been identified and a fix is being implemented.Feb 23, 12:06 PDT Monitoring - We are investigating performance issues on mem1

Iron.io
We are investigating the performance on mq-aws-us-east-1-1

Feb 22, 16:00 PDT Resolved - This incident has been resolved.Feb 22, 15:39 PDT Investigating - We are currently investigating this issue.

Iron.io
Workers In Queue for Extended Periods

Feb 16, 11:47 PDT Resolved - This incident has been resolved.Feb 16, 09:45 PDT Update - We have implemented a solution and are in the process of implementing. Tasks are returning to normal, but will take a bit of time for full recovery.Feb 16, 08:41 PDT Identified - We have identified the issue and are in the process of implementing a fix.

Iron.io
IronWorker Public Cluster (us-east-1 )

Feb 10, 21:44 PDT Resolved - IronWorker tasks are processing normally and within expected durations.Feb 10, 21:36 PDT Update - IronWorker tasks are processing normally and within expected durations.Feb 10, 20:15 PDT Update - We are working on a resolution to the delays processing tasks in the IronWorker service operating in AWS US-East (us-east-1)Feb 10, 19:11 PDT Monitoring - We are investigating delays in processing of tasks in the IronWorker service operating in AWS US-East (us-east-1) and with one or more dedicated clusters. We are monitoring workloads and will report status at regular intervals.

Iron.io
IronMQ v3 degraded performance

Feb 10, 13:53 PDT Postmortem - A copy of the postmortem can be found here: http://get.iron.io/gavrsFeb 10, 12:44 PDT Resolved - This incident has been resolved.Feb 9, 01:45 PDT Investigating - Monitoring IronMQ v3 (AWS US-East) service is available and operational. All queues have been restored. All messages that may have been unavailable are now available. We will continue to monitor the situation as well as prepare an incident report. Feb 9, 21:11 PST Update IronMQ v3 (AWS US-East) service is available and operational. All queues have been restored. Messages that were previously in queue are currently being made available. We expect this process to continue for several more hours. (Please note that there may be duplicate messages but we expect this to be rare.) Feb 9, 19:00 PST Update IronMQv3 public is operational and are continuing to bring unavailable queues back online. Feb 9, 16:43 PST Update All queues and messages are persistent. Any queues or messages that are offline are in the process of being made available. We will keep you updated on the progress. Feb 9, 12:21 PST Update We are still in the process of recovery. Queues and messages should be returning so if you don't see yours yet, please stay tuned. Feb 9, 12:07 PST Update The issue we were experiencing has been identified and is in the process of recovery. Feb 9, 11:05 PST Update The issue we were experiencing has been identified and is in the process of recovery. Service is being restored while adhering to FIFO priority, all queued messages are being delivered but there may be some delay before the system is back to normal. Feb 9, 10:11 PST Update There is no current ETA at the moment, but we have identified a solution and are in the process of implementing a fix. We will update here once more information becomes available. Feb 9, 09:44 PST Update There is no ETA at the moment, but we have identified a solution and are in the process of implementing a fix. We will update here once more information becomes available. Feb 9, 08:27 PST Identified The issue has been identified and a fix is being implemented. Feb 9, 04:30 PST Investigating We are currently investigating this issue. Feb 9, 01:45 PST

Iron.io
IronMQ rax-ord degraded performance

Feb 3, 06:57 PDT Resolved - We have identified the issue, and implemented a solution. All systems are operating within normal limits.Feb 3, 05:31 PDT Investigating - We are currently investigating this issue.

Iron.io
IronMQ degraded performance

Feb 1, 20:33 PDT Resolved - This incident has been resolved.Feb 1, 15:13 PDT Update - We have identified a solution and are in the process of implementing it.Feb 1, 14:11 PDT Update - We are experiencing a partial outage related to IronMQ v3, AWS US East, and the HUD-e dashboard. There is no ETA at the moment, but our Engineering team has identified the issue and is actively working on a resolution. We will update you here, and on the status page, once we have more information.Feb 1, 12:44 PDT Identified - We are experiencing a partial outage related to IronMQ v3, AWS US East, and the HUD-e dashboard. There is no ETA at the moment, but our Engineering team has identified the issue and is actively working on a resolution. We will update you here when more information becomes available.Feb 1, 12:12 PDT Investigating - We are experiencing higher than normal load on the public IronMQ v3 service, for AWS US East, resulting in degradation of performance.

Iron.io
IronMQ v1 aws-us-east outage

Jan 31, 00:12 PDT Resolved - This incident has been resolved.

Iron.io
IronCache temporary unavailable due to infrastructure update

Jan 29, 00:26 PDT Resolved - This incident has been resolved.Jan 29, 00:02 PDT Identified - The issue has been identified and a fix is being implemented.

Iron.io
IronWorker degraded performance

Jan 25, 12:06 PDT Resolved - This incident has been resolved.Jan 25, 07:22 PDT Identified - We are experiencing higher than normal load on our Public Cluster. This could result in longer than normal queue times for tasks queued on the Public Cluster.

Iron.io
IronWorker degraded performance

Jan 22, 12:45 PDT Resolved - Systems are operating normally.Jan 22, 12:13 PDT Update - The IronWorker performance degradation issue is isolated to our Public IronWorker cluster.Jan 22, 12:05 PDT Identified - We have identified an issue with IronWorker that is degrading service performance. There is no ETA at the moment, but our Engineering team has identified the issue and is actively working on a resolution. We will update you here, once we have more information.

Iron.io
IronMQ v3 mq-aws-us-east-1-1.iron.io 20160115

Jan 15, 11:28 PDT Resolved - Systems are operating normally.Jan 15, 10:29 PDT Identified - We have identified a service degradation issue with mq-aws-us-east-1-1.iron.io. We have engineers actively working to resolve the issue.

Iron.io
IronMQ 500 issue

Dec 21, 11:55 PDT Resolved - The issue has been resolved.Dec 21, 10:29 PDT Identified - We have identified the issue causing 500 errors, and are in the progress on a solution.Dec 21, 08:18 PDT Investigating - We are currently investigating an issue with IronMQ return 500 status codes.

Iron.io
Iron HUD

Nov 6, 08:03 PDT Resolved - This incident has been resolved.Nov 4, 16:58 PDT Monitoring - We have identified and have fixed the majority of customers users have been experiencing with our HUD. Some integration issues remain. As always, if you are experiencing issues with the service, please contact us at support@iron.ioNov 4, 10:46 PDT Update - We have identified the migration issues and are transitioning our authentication system now. Only the Iron HUD is affected. All services are operational.Nov 4, 10:24 PDT Identified - We are investigating HUD login issues issues after performing a HUD migration. No other services are affected at this time.

Iron.io
IronMQ, HUD

Oct 31, 09:48 PDT Resolved - This incident has been resolved.Oct 30, 11:00 PDT Identified - We have identified several high load customers on the shared public cluster. We are working with them to reduce their resource rate to an acceptable level so as to not affect other customers on the shared cluster.Oct 30, 07:29 PDT Investigating - We are currently investigating issues with IronMQ and the HUD dashboard.

Iron.io
IronMQ v2 Elevated Error Rate

Oct 29, 15:53 PDT Resolved - All systems are operating normally.Oct 29, 14:49 PDT Monitoring - We believe we have identified the issue with one of our upstream providers. Service levels should be returning to normal. If you continue to experience issues, please contact us at support@iron.io.Oct 29, 14:38 PDT Investigating - We are investigating elevated error rates in IronMQ v2.

Iron.io
IronMQ Service Disruption

Oct 11, 21:29 PDT Resolved - This incident has been resolved.Oct 11, 17:12 PDT Identified - An upstream provider used by IronMQ is reporting a service disruption. We are monitoring and evaluating fixes at this time.

Iron.io
IronMQ, IronWorker degraded performance

Oct 3, 07:25 PDT Resolved - This incident has been resolved. Downstream customers experienced a DDoS, causing delayed push queue times. All other services were not impacted and have been operational. Currently, push queues have been resolved and should be performing as expected.Oct 2, 11:48 PDT Investigating - We are currently investigating this issue.

Iron.io
IronMQ v1 degraded performance

Oct 2, 08:28 PDT Resolved - This incident has been resolved.Oct 2, 05:47 PDT Monitoring - A fix has been implemented and we are monitoring the results.Oct 2, 05:38 PDT Investigating - We are currently investigating this issue.

Iron.io
IronWorker - Elevated Error Rates

Oct 1, 17:02 PDT Resolved - Issues with our upstream provider have been resolved. Tasks should now be processing at a normal rate. If you are seeing continued issues, please contact us at support@iron.io.Oct 1, 16:47 PDT Monitoring - We have identified an issue with our upstream infrastructure provider, and are working to restore service to IronWorker. Currently, tasks are running, but may be delayed while existing queued tasks are scheduled.

Iron.io
Elevated IronWorker Start Times

Aug 13, 14:50 PDT Resolved - Networking issues with EC2 Classic have been resolved. We are closely monitoring and moving users over to the new VPC networks over the next few days. There will not be any interruptions to your service.Aug 13, 09:08 PDT Identified - We're still investigating network issues with the IronWorker deployments that are in the EC2 Classic network. Please email support@iron.io if you are experiencing issues with your IronWorker jobs.Aug 12, 22:02 PDT Monitoring - We are continuing to investigate. Jobs are running normally. As always, if you are experiencing issues with the service, please contact us at support@iron.ioAug 12, 18:48 PDT Investigating - We are currently investigating an issue with elevated IronMQ Push queue times and IronWorker start times. We will provide updates as it continues.

Iron.io
IronWorker Standard Cluster delays

Aug 11, 17:55 PDT Resolved - This incident has been resolved.Aug 11, 17:21 PDT Monitoring - We found and resolved the issue. Monitoring now.Aug 11, 17:21 PDT Identified - We are experiencing some slowdowns in job processing on our public IronWorker cluster. This is *not* affecting dedicated cluster customers.

Iron.io
IronWorker degraded performance

Aug 11, 01:14 PDT Resolved - This incident has been resolved.Aug 10, 06:40 PDT Monitoring - A fix has been implemented and we are monitoring the results.Aug 10, 00:15 PDT Investigating - Some tasks are taking a bit longer to start due to issues with S3 (503: Service Unavailable)