CloudStatus

Codeship Status Alerts

Codeship
Codeship builds are run normally, but UI is slow to update with test step progress

Nov 14, 18:15 UTC Resolved - Cleared out workers queue. Build steps should now be appearing in realtime.Nov 14, 17:55 UTC Investigating - We are currently investigating this issue.

Codeship
Codeship builds are run normally, but UI is slow to update with test step progress

Nov 14, 17:55 UTC Investigating - We are currently investigating this issue.

Codeship
Brief spike of build failures

Nov 7, 18:45 UTC Resolved - This incident has been resolved.Nov 7, 17:14 UTC Monitoring - Due to upstream issues with our build infrastructure, we had a brief spike of build failures. We are keeping our eyes on the situation and will update accordingly with more information.

Codeship
Brief spike of build failures

Nov 7, 17:14 UTC Monitoring - Due to upstream issues with our build infrastructure, we had a brief spike of build failures. We are keeping our eyes on the situation and will update accordingly with more information.

Codeship
Heroku Deploy Issues With Codeship Basic

Nov 3, 18:21 UTC Resolved - We have identified the root issue and Heroku builds should now be running normally.Nov 3, 14:54 UTC Monitoring - We are keeping our eyes on the issues with Heroku deploy keys and communicating with Heroku about this. Please let us know if you see any issues with your Heroku deploys.Nov 3, 13:05 UTC Investigating - We are experiencing further issues with Heroku deploys on Codeship Basic failing with a duplicate or invalid key error. We are investigating now and will add more information soon.

Codeship
Heroku Deploy Issues With Codeship Basic

Nov 3, 14:54 UTC Monitoring - We are keeping our eyes on the issues with Heroku deploy keys and communicating with Heroku about this. Please let us know if you see any issues with your Heroku deploys.Nov 3, 13:05 UTC Investigating - We are experiencing further issues with Heroku deploys on Codeship Basic failing with a duplicate or invalid key error. We are investigating now and will add more information soon.

Codeship
Heroku Deploy Issues With Codeship Basic

Nov 3, 13:05 UTC Investigating - We are experiencing further issues with Heroku deploys on Codeship Basic failing with a duplicate or invalid key error. We are investigating now and will add more information soon.

Codeship
Investigating "key is already in use" error with Codeship Basic Heroku Deployment

Nov 2, 23:59 UTC Resolved - We're going to mark this incident as resolved for now and continue the conversation with Heroku. In the meantime, please email support@codeship.com if you are running into this issue and we will manually reset your project SSH key.Nov 2, 23:38 UTC Update - Still communicating with Heroku on issue.Nov 2, 22:54 UTC Update - Communicating with Heroku to investigate whether key usage rules have changed recently.Nov 2, 22:24 UTC Investigating - We are currently investigating this issue.

Codeship
Investigating "key is already in use" error with Codeship Basic Heroku Deployment

Nov 2, 23:38 UTC Update - Still communicating with Heroku on issue.Nov 2, 22:54 UTC Update - Communicating with Heroku to investigate whether key usage rules have changed recently.Nov 2, 22:24 UTC Investigating - We are currently investigating this issue.

Codeship
Investigating "key is already in use" error with Codeship Basic Heroku Deployment

Nov 2, 22:54 UTC Update - Communicating with Heroku to investigate whether key usage rules have changed recently.Nov 2, 22:24 UTC Investigating - We are currently investigating this issue.

Codeship
Investigating "key is already in use" error with Codeship Basic Heroku Deployment

Nov 2, 22:24 UTC Investigating - We are currently investigating this issue.

Codeship
Heroku-related build issues

Oct 31, 15:32 UTC Resolved - Heroku is reporting all build issues now solved.Oct 31, 12:34 UTC Monitoring - Heroku is reporting issues with builds across major regions: https://status.heroku.com/ This will impact Codeship builds that must communicate with Heroku. We are monitoring the situation and will update our status accordingly.

Codeship
Heroku-related build issues

Oct 31, 12:34 UTC Monitoring - Heroku is reporting issues with builds across major regions: https://status.heroku.com/ This will impact Codeship builds that must communicate with Heroku. We are monitoring the situation and will update our status accordingly.

Codeship
Slow performance and increased error rates for images pulled from Docker Hub

Oct 26, 15:12 UTC Resolved - Docker has marked the issue as resolved. https://status.docker.com/Oct 26, 12:25 UTC Identified - We're monitoring slow performance and an elevated error rate for builds pulling images from Docker Hub. Follow the GitHub issue for more information https://github.com/docker/hub-feedback/issues/1225 If your build is impacted, a restart may mitigate the issue, as it it intermittent.

Codeship
Slow performance and increased error rates for images pulled from Docker Hub

Oct 26, 12:25 UTC Identified - We're monitoring slow performance and an elevated error rate for builds pulling images from Docker Hub. Follow the GitHub issue for more information https://github.com/docker/hub-feedback/issues/1225 If your build is impacted, a restart may mitigate the issue, as it it intermittent.

Codeship
Implementing update to Codeship service

Oct 16, 22:25 UTC Resolved - This incident has been resolved.Oct 16, 21:37 UTC Monitoring - Engineering team requires an update to be made to our service. While issues are not anticipated we will be monitoring closely.

Codeship
Implementing update to Codeship service

Oct 16, 21:37 UTC Monitoring - Engineering team requires an update to be made to our service. While issues are not anticipated we will be monitoring closely.

Codeship
Investigating 5XX errors

Oct 16, 21:16 UTC Resolved - This incident has been resolved.Oct 16, 21:05 UTC Investigating - We are currently investigating our exposure to Heroku's reported latency issues.

Codeship
Investigating 5XX errors

Oct 16, 21:05 UTC Investigating - We are currently investigating our exposure to Heroku's reported latency issues.

Codeship
Cloning repositories for new Pro and Basic projects

Oct 2, 21:18 UTC Resolved - This incident has been resolved.Oct 2, 20:24 UTC Identified - Issue identified. Deploying fix and working to reset ssh keys on affected projects.Oct 2, 19:43 UTC Investigating - Investigating issues with cloning repositories for new Pro and Basic projects

Codeship
Cloning repositories for new Pro and Basic projects

Oct 2, 20:24 UTC Identified - Issue identified. Deploying fix and working to reset ssh keys on affected projects.Oct 2, 19:43 UTC Investigating - Investigating issues with cloning repositories for new Pro and Basic projects

Codeship
Cloning repositories for new Pro and Basic projects

Oct 2, 19:43 UTC Investigating - Investigating issues with cloning repositories for new Pro and Basic projects

Codeship
Codeship Pro, Connectivity issues for Docker Hub

Sep 20, 18:23 UTC Resolved - Docker is reporting all systems as operational.Sep 20, 16:11 UTC Update - Docker mentions the network issues are resolved, but they are monitoring performance.Sep 20, 14:56 UTC Monitoring - Docker is experiencing intermittent connectivity issues with their upstream network provider. This can cause failing builds on Codeship Pro, if you depend on Docker Hub for pulling or pushing images. Please see https://status.docker.com/ for details.

Codeship
Codeship Pro, Connectivity issues for Docker Hub

Sep 20, 16:11 UTC Update - Docker mentions the network issues are resolved, but they are monitoring performance.Sep 20, 14:56 UTC Monitoring - Docker is experiencing intermittent connectivity issues with their upstream network provider. This can cause failing builds on Codeship Pro, if you depend on Docker Hub for pulling or pushing images. Please see https://status.docker.com/ for details.

Codeship
Codeship Pro, Connectivity issues for Docker Hub

Sep 20, 14:56 UTC Monitoring - Docker is experiencing intermittent connectivity issues with their upstream network provider. This can cause failing builds on Codeship Pro, if you depend on Docker Hub for pulling or pushing images. Please see https://status.docker.com/ for details.

Codeship
Experiencing Slowness on Codeship Website

Sep 19, 22:30 UTC Resolved - This incident has been resolved.Sep 19, 22:19 UTC Monitoring - Website has returned to normal levels of responsiveness as we continue to monitor situation.Sep 19, 21:50 UTC Investigating - Builds are running normally but we are experiencing slowness with our website.

Codeship
Experiencing Slowness on Codeship Website

Sep 19, 22:19 UTC Monitoring - Website has returned to normal levels of responsiveness as we continue to monitor situation.Sep 19, 21:50 UTC Investigating - Builds are running normally but we are experiencing slowness with our website.

Codeship
Experiencing Slowness on Codeship Website

Sep 19, 21:50 UTC Investigating - Builds are running normally but we are experiencing slowness with our website.

Codeship
AWS is reporting increased error rates for S3

Sep 14, 20:23 UTC Resolved - AWS has resolved their S3 issue.Sep 14, 19:41 UTC Identified - Taking steps to mitigate our exposure to current S3 issues.Sep 14, 19:16 UTC Investigating - We are investigating impact to our service quality

Codeship
AWS is reporting increased error rates for S3

Sep 14, 19:41 UTC Identified - Taking steps to mitigate our exposure to current S3 issues.Sep 14, 19:16 UTC Investigating - We are investigating impact to our service quality

Codeship
AWS is reporting increased error rates for S3

Sep 14, 19:16 UTC Investigating - We are investigating impact to our service quality

Codeship
Infrastructure Upgrades

Sep 10, 11:27 UTC Completed - Maintenance is now completed, both Codeship Pro and Codeship Basic are operating as expected.Sep 10, 11:19 UTC Verifying - We've made the required changes to our infrastructure and are now verifying everything works as expected. Queued builds that were triggered while Codeship was in maintenance mode will now get run.Sep 10, 10:18 UTC In progress - Scheduled maintenance is currently in progress. There will be a delay in build processing and our UI will be unavailable for a period of time.Sep 10, 08:19 UTC Scheduled - We will be undergoing scheduled maintenance during this time.Sep 10, 08:19 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 10, 08:18 UTC Scheduled - We have to postpone maintenance for roughly 1 hour (to ~9am UTC ). Builds will run regularly in the meantime and we'll update you as soon as maintenance starts.Sep 10, 08:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 7, 16:49 UTC Scheduled - We identified an upgrade to our Redis instance as a required step when reviewing a recent service outage (see https://www.codeshipstatus.com/incidents/mg5r0wd4qtlk). As this upgrade requires some downtime of our main application and will cause delays in build processing we are scheduling for a very low volume period, starting Sunday September 9th, 08.00 UTC . We expect the maintenance to last roughly 90 minutes.

Codeship
Infrastructure Upgrades

Sep 10, 11:19 UTC Verifying - We've made the required changes to our infrastructure and are now verifying everything works as expected. Queued builds that were triggered while Codeship was in maintenance mode will now get run.Sep 10, 10:18 UTC In progress - Scheduled maintenance is currently in progress. There will be a delay in build processing and our UI will be unavailable for a period of time.Sep 10, 08:19 UTC Scheduled - We will be undergoing scheduled maintenance during this time.Sep 10, 08:19 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 10, 08:18 UTC Scheduled - We have to postpone maintenance for roughly 1 hour (to ~9am UTC ). Builds will run regularly in the meantime and we'll update you as soon as maintenance starts.Sep 10, 08:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 7, 16:49 UTC Scheduled - We identified an upgrade to our Redis instance as a required step when reviewing a recent service outage (see https://www.codeshipstatus.com/incidents/mg5r0wd4qtlk). As this upgrade requires some downtime of our main application and will cause delays in build processing we are scheduling for a very low volume period, starting Sunday September 9th, 08.00 UTC . We expect the maintenance to last roughly 90 minutes.

Codeship
Infrastructure Upgrades

Sep 10, 10:18 UTC In progress - Scheduled maintenance is currently in progress. There will be a delay in build processing and our UI will be unavailable for a period of time.Sep 10, 08:19 UTC Scheduled - We will be undergoing scheduled maintenance during this time.Sep 10, 08:19 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 10, 08:18 UTC Scheduled - We have to postpone maintenance for roughly 1 hour (to ~9am UTC ). Builds will run regularly in the meantime and we'll update you as soon as maintenance starts.Sep 10, 08:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 7, 16:49 UTC Scheduled - We identified an upgrade to our Redis instance as a required step when reviewing a recent service outage (see https://www.codeshipstatus.com/incidents/mg5r0wd4qtlk). As this upgrade requires some downtime of our main application and will cause delays in build processing we are scheduling for a very low volume period, starting Sunday September 9th, 08.00 UTC . We expect the maintenance to last roughly 90 minutes.

Codeship
Infrastructure Upgrades

Sep 10, 08:19 UTC Scheduled - We will be undergoing scheduled maintenance during this time.Sep 10, 08:19 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 10, 08:18 UTC Scheduled - We have to postpone maintenance for roughly 1 hour (to ~9am UTC ). Builds will run regularly in the meantime and we'll update you as soon as maintenance starts.Sep 10, 08:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 7, 16:49 UTC Scheduled - We identified an upgrade to our Redis instance as a required step when reviewing a recent service outage (see https://www.codeshipstatus.com/incidents/mg5r0wd4qtlk). As this upgrade requires some downtime of our main application and will cause delays in build processing we are scheduling for a very low volume period, starting Sunday September 9th, 08.00 UTC . We expect the maintenance to last roughly 90 minutes.

Codeship
Infrastructure Upgrades

Sep 10, 08:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep 7, 16:49 UTC Scheduled - We identified an upgrade to our Redis instance as a required step when reviewing a recent service outage (see https://www.codeshipstatus.com/incidents/mg5r0wd4qtlk). As this upgrade requires some downtime of our main application and will cause delays in build processing we are scheduling for a very low volume period, starting Sunday September 9th, 08.00 UTC . We expect the maintenance to last roughly 90 minutes.

Codeship
Infrastructure Upgrades

Sep 7, 16:49 UTC Scheduled - We identified an upgrade to our Redis instance as a required step when reviewing a recent service outage (see https://www.codeshipstatus.com/incidents/mg5r0wd4qtlk). As this upgrade requires some downtime of our main application and will cause delays in build processing we are scheduling for a very low volume period, starting Sunday September 9th, 08.00 UTC . We expect the maintenance to last roughly 90 minutes.

Codeship
Website not loading

Sep 9, 08:41 UTC Resolved - We have identified and resolved an issue that caused our website to be unavailable from 08:20 till 08:33 UTC .

Codeship
Build allocation issues

Sep 5, 20:31 UTC Resolved - This issue is now resolved.Sep 5, 19:27 UTC Update - We are aware of a number of "stuck" builds that are preventing queues from clearing out. Currently working to unblock them.Sep 5, 18:18 UTC Update - Intermittent build delays may be encountered as we continue to monitor and work towards full resolution.Sep 5, 18:07 UTC Monitoring - Builds are now stabilized. Will be monitoring closely.Sep 5, 17:03 UTC Identified - We have identified the issue and are continuing to work to resolve the matter.Sep 5, 15:30 UTC Investigating - Looking into builds not getting properly started.

Codeship
Build allocation issues

Sep 5, 19:27 UTC Update - We are aware of a number of "stuck" builds that are preventing queues from clearing out. Currently working to unblock them.Sep 5, 18:18 UTC Update - Intermittent build delays may be encountered as we continue to monitor and work towards full resolution.Sep 5, 18:07 UTC Monitoring - Builds are now stabilized. Will be monitoring closely.Sep 5, 17:03 UTC Identified - We have identified the issue and are continuing to work to resolve the matter.Sep 5, 15:30 UTC Investigating - Looking into builds not getting properly started.

Codeship
Build allocation issues

Sep 5, 18:18 UTC Update - Intermittent build delays may be encountered as we continue to monitor and work towards full resolution.Sep 5, 18:07 UTC Monitoring - Builds are now stabilized. Will be monitoring closely.Sep 5, 17:03 UTC Identified - We have identified the issue and are continuing to work to resolve the matter.Sep 5, 15:30 UTC Investigating - Looking into builds not getting properly started.

Codeship
Build allocation issues

Sep 5, 18:07 UTC Monitoring - Builds are now stabilized. Will be monitoring closely.Sep 5, 17:03 UTC Identified - We have identified the issue and are continuing to work to resolve the matter.Sep 5, 15:30 UTC Investigating - Looking into builds not getting properly started.

Codeship
Build allocation issues

Sep 5, 17:03 UTC Identified - We have identified the issue and are continuing to work to resolve the matter.Sep 5, 15:30 UTC Investigating - Looking into builds not getting properly started.

Codeship
Build allocation issues

Sep 5, 15:30 UTC Investigating - Looking into builds not getting properly started.

Codeship
Logs not loading on Codeship Basic

Sep 1, 19:26 UTC Resolved - Logs should now be loading after reverting a change.Sep 1, 19:17 UTC Investigating - We are investigating logs not loading on Codeship Basic.

Codeship
Logs not loading on Codeship Basic

Sep 1, 19:17 UTC Investigating - We are investigating logs not loading on Codeship Basic.

Codeship
Heroku outage impacting Codeship

Aug 31, 15:38 UTC Resolved - Heroku is now reporting that the outage is resolved.Aug 31, 13:40 UTC Monitoring - Heroku is currently experiencing a significant outage (https://status.heroku.com/incidents/1273) that may make it difficult to reach Codeship, or may cause page load or UI problems interacting with your builds.

Codeship
Heroku outage impacting Codeship

Aug 31, 13:40 UTC Monitoring - Heroku is currently experiencing a significant outage (https://status.heroku.com/incidents/1273) that may make it difficult to reach Codeship, or may cause page load or UI problems interacting with your builds.

Codeship
Codeship Pro build issues

Aug 21, 15:48 UTC Resolved - We rolled back a faulty release that caused several dozen Codeship Pro builds to fail with no explanation, over the last hour. Re-running these builds should work as intended and we will continue to monitor the situation.

Codeship
Experiencing Issues with Codeship Signups from Heroku

Aug 18, 22:11 UTC Resolved - This issue is now resolved. Newly registered users from Heroku should now be able to access their dashboard.Aug 18, 21:51 UTC Investigating - We are aware of the 500 error message on the Codeship Dashboard for Heroku users who have signed up in the last few hours. We are currently investing.

Codeship
Experiencing Issues with Codeship Signups from Heroku

Aug 18, 21:51 UTC Investigating - We are aware of the 500 error message on the Codeship Dashboard for Heroku users who have signed up in the last few hours. We are currently investing.

Codeship
Database Maintenance

Jul 31, 08:31 UTC Scheduled - We've been notified by our hosting provider, that they will need to perform database maintenance for our main application database on August 17th. The maintenance will start at 21.00 UTC and last for approximately 30 minutes.

Codeship
Database Maintenance

Jul 31, 08:31 UTC Scheduled - We've been notified by our hosting provider, that they will need to perform database maintenance for our main application database on August 17th. The maintenance will start at 21.00 UTC and last for approximately 30 minutes.

Codeship
Database Maintenance

Aug 16, 06:36 UTC Completed - The scheduled maintenance has been completed.Aug 16, 06:00 UTC In progress - Codeship is undergoing scheduled database maintenance. Application and builds will be unavailable.Jul 31, 08:31 UTC Scheduled - We've been notified by our hosting provider, that they will need to perform database maintenance for our main application database on August 17th. The maintenance will start at 21.00 UTC and last for approximately 30 minutes.

Codeship
Database Maintenance

Aug 16, 06:00 UTC In progress - Codeship is undergoing scheduled database maintenance. Application and builds will be unavailable.Jul 31, 08:31 UTC Scheduled - We've been notified by our hosting provider, that they will need to perform database maintenance for our main application database on August 17th. The maintenance will start at 21.00 UTC and last for approximately 30 minutes.

Codeship
Codeship Basic, Delays with build allocation

Aug 9, 17:44 UTC Resolved - This incident is now resolved.Aug 9, 15:13 UTC Monitoring - Continuing to monitor as build processing has stabilized on our Basic service.Aug 9, 13:30 UTC Investigating - We're looking into delays with build processing on Codeship Basic. Builds run but are roughly 15 minutes delayed at the moment.

Codeship
Codeship Basic, Delays with build allocation

Aug 9, 15:13 UTC Monitoring - Continuing to monitor as build processing has stabilized on our Basic service.Aug 9, 13:30 UTC Investigating - We're looking into delays with build processing on Codeship Basic. Builds run but are roughly 15 minutes delayed at the moment.

Codeship
Codeship Basic, Delays with build allocation

Aug 9, 15:13 UTC Monitoring - Continuing to monitor as build processing has stabilized on our Basic service.Aug 9, 13:30 UTC Investigating - We're looking into delays with build processing on Codeship Basic. Builds run but are roughly 15 minutes delayed at the moment.

Codeship
Codeship Basic, Delays with build allocation

Aug 9, 13:30 UTC Investigating - We're looking into delays with build processing on Codeship Basic. Builds run but are roughly 15 minutes delayed at the moment.

Codeship
Codeship Pro, Delayed builds for projects on the "Small" instance size

Aug 9, 09:36 UTC Resolved - This incident is now resolved. We'll keep monitoring build VM provisioning closely, but build allocation times are within their normal range.Aug 9, 08:51 UTC Monitoring - Our engineering team has taken steps to remedy this and we're now monitoring performance closely.Aug 9, 08:38 UTC Investigating - We're looking into an issue with provisioning of build instances for the "Small" instance size causing delays for projects using those instances. Projects using a different instance size are not affected by this incident.

Codeship
Codeship Pro, Delayed builds for projects on the "Small" instance size

Aug 9, 08:51 UTC Monitoring - Our engineering team has taken steps to remedy this and we're now monitoring performance closely.Aug 9, 08:38 UTC Investigating - We're looking into an issue with provisioning of build instances for the "Small" instance size causing delays for projects using those instances. Projects using a different instance size are not affected by this incident.

Codeship
Codeship Pro, Delayed builds for projects on the "Small" instance size

Aug 9, 08:38 UTC Investigating - We're looking into an issue with provisioning of build instances for the "Small" instance size causing delays for projects using those instances. Projects using a different instance size are not affected by this incident.

Codeship
GitHub, Major service outage

Jul 31, 17:43 UTC Resolved - Github has reported a return to normal operation.Jul 31, 16:37 UTC Monitoring - GitHub is reporting a major service outage via https://status.github.com which affects any Codeship projects connected to a GitHub repository. We're monitoring the situation and will update this page if we get any new information.

Codeship
GitHub, Major service outage

Jul 31, 16:37 UTC Monitoring - GitHub is reporting a major service outage via https://status.github.com which affects any Codeship projects connected to a GitHub repository. We're monitoring the situation and will update this page if we get any new information.

Codeship
Delayed/Failing Builds on Codeship Pro

Jul 26, 23:45 UTC Resolved - Builds should be running normally on Codeship Pro again.Jul 26, 20:16 UTC Monitoring - Continuing to monitor as Codeship Pro builds are stabilizing.Jul 26, 19:33 UTC Identified - Quay.io outage has intensified and as a result we are noting a major outage of Codeship Pro builds. Currently investigating workarounds to Quay.io dependency.Jul 26, 14:16 UTC Monitoring - Codeship Pro builds are no longer delayed. We're monitoring performance and will update our status page should we see an increase in queue times again.Jul 26, 13:43 UTC Identified - We're experiencing delays with Codeship Pro builds because of the upstream dependency on Quay.io. We're looking at different possibilities to remedy this.

Codeship
Delayed/Failing Builds on Codeship Pro

Jul 26, 20:16 UTC Monitoring - Continuing to monitor as Codeship Pro builds are stabilizing.Jul 26, 19:33 UTC Identified - Quay.io outage has intensified and as a result we are noting a major outage of Codeship Pro builds. Currently investigating workarounds to Quay.io dependency.Jul 26, 14:16 UTC Monitoring - Codeship Pro builds are no longer delayed. We're monitoring performance and will update our status page should we see an increase in queue times again.Jul 26, 13:43 UTC Identified - We're experiencing delays with Codeship Pro builds because of the upstream dependency on Quay.io. We're looking at different possibilities to remedy this.

Codeship
Delayed/Failing Builds on Codeship Pro

Jul 26, 19:33 UTC Identified - Quay.io outage has intensified and as a result we are noting a major outage of Codeship Pro builds. Currently investigating workarounds to Quay.io dependency.Jul 26, 14:16 UTC Monitoring - Codeship Pro builds are no longer delayed. We're monitoring performance and will update our status page should we see an increase in queue times again.Jul 26, 13:43 UTC Identified - We're experiencing delays with Codeship Pro builds because of the upstream dependency on Quay.io. We're looking at different possibilities to remedy this.

Codeship
Delayed Builds on Codeship Pro

Jul 26, 19:33 UTC Identified - Quay.io outage has intensified and as a result we are noting a major outage of Codeship Pro builds. Currently investigating workarounds to Quay.io dependency.Jul 26, 14:16 UTC Monitoring - Codeship Pro builds are no longer delayed. We're monitoring performance and will update our status page should we see an increase in queue times again.Jul 26, 13:43 UTC Identified - We're experiencing delays with Codeship Pro builds because of the upstream dependency on Quay.io. We're looking at different possibilities to remedy this.

Codeship
Delayed Builds on Codeship Pro

Jul 26, 14:16 UTC Monitoring - Codeship Pro builds are no longer delayed. We're monitoring performance and will update our status page should we see an increase in queue times again.Jul 26, 13:43 UTC Identified - We're experiencing delays with Codeship Pro builds because of the upstream dependency on Quay.io. We're looking at different possibilities to remedy this.

Codeship
Delayed Builds on Codeship Pro

Jul 26, 13:43 UTC Identified - We're experiencing delays with Codeship Pro builds because of the upstream dependency on Quay.io. We're looking at different possibilities to remedy this.

Codeship
Codeship Basic build delays

Jul 18, 22:09 UTC Resolved - Builds should be running normally on Codeship Basic again.Jul 18, 19:33 UTC Monitoring - We are continuing to monitor the build queues on Codeship Basic.Jul 18, 18:25 UTC Update - We are currently recovering from earlier problems with cloud capacity from our upstream provider for Codeship Basic. Although the system is returning to normal, there are residual delays in builds running as we work through the backlog.Jul 18, 17:05 UTC Identified - We are currently experiencing problems with cloud capacity from our upstream provider for Codeship Basic resulting in delays in builds running.Jul 18, 16:34 UTC Investigating - We’re investigating build delays with Codeship Basic.

Codeship
Codeship Basic build delays

Jul 18, 19:33 UTC Monitoring - We are continuing to monitor the build queues on Codeship Basic.Jul 18, 18:25 UTC Update - We are currently recovering from earlier problems with cloud capacity from our upstream provider for Codeship Basic. Although the system is returning to normal, there are residual delays in builds running as we work through the backlog.Jul 18, 17:05 UTC Identified - We are currently experiencing problems with cloud capacity from our upstream provider for Codeship Basic resulting in delays in builds running.Jul 18, 16:34 UTC Investigating - We’re investigating build delays with Codeship Basic.

Codeship
Codeship Basic build delays

Jul 18, 18:25 UTC Update - We are currently recovering from earlier problems with cloud capacity from our upstream provider for Codeship Basic. Although the system is returning to normal, there are residual delays in builds running as we work through the backlog.Jul 18, 17:05 UTC Identified - We are currently experiencing problems with cloud capacity from our upstream provider for Codeship Basic resulting in delays in builds running.Jul 18, 16:34 UTC Investigating - We’re investigating build delays with Codeship Basic.

Codeship
Codeship Basic build delays

Jul 18, 17:05 UTC Identified - We are currently experiencing problems with cloud capacity from our upstream provider for Codeship Basic resulting in delays in builds running.Jul 18, 16:34 UTC Investigating - We’re investigating build delays with Codeship Basic.

Codeship
Codeship Basic build delays

Jul 18, 16:34 UTC Investigating - We’re investigating build delays with Codeship Basic.

Codeship
Codeship Basic build delays

Jul 18, 16:34 UTC Investigating - We’re investigating build delays with Codeship Basic.

Codeship
DNS name resolution failures

Jul 10, 20:49 UTC Resolved - This incident has been resolved.Jul 10, 19:42 UTC Monitoring - The issue was resolved by our DNS provider. We are working with them to identify root cause.Jul 10, 19:13 UTC Investigating - We are investigating DNS name resolution failures for codeship.com domains, i.e. app.codeship.com

Codeship
DNS name resolution failures

Jul 10, 19:42 UTC Monitoring - The issue was resolved by our DNS provider. We are working with them to identify root cause.Jul 10, 19:13 UTC Investigating - We are investigating DNS name resolution failures for codeship.com domains, i.e. app.codeship.com

Codeship
DNS name resolution failures

Jul 10, 19:13 UTC Investigating - We are investigating DNS name resolution failures for codeship.com domains, i.e. app.codeship.com

Codeship
Codeship Pro builds not triggering

Jun 21, 18:16 UTC Resolved - Builds are now running normally.Jun 21, 17:51 UTC Monitoring - We had a brief service issue with Codeship Pro builds not triggering. This is now resolved and builds should be running as normal, but we are continuing to monitor the situation.

Codeship
Codeship Pro builds not triggering

Jun 21, 17:51 UTC Monitoring - We had a brief service issue with Codeship Pro builds not triggering. This is now resolved and builds should be running as normal, but we are continuing to monitor the situation.

Codeship
Queued Builds on Codeship Basic

Jun 19, 13:19 UTC Resolved - This incident is now resolved. Depending on account limits there might still be some queued builds, but those will be process as fast as possible. Any new incoming builds are process right away (unless account limits cause the build to get queued).Jun 19, 12:49 UTC Monitoring - We've mostly caught up with the demand and are working through the remaining queued builds. We should be back operating normally soon.Jun 19, 12:19 UTC Update - We're adding new capacity to our build pool as fast as possible to deal with both the backlog of waiting builds as well as the increased load during US working hours.Jun 19, 11:42 UTC Identified - We traced the issue back to capacity issues with AWS and are looking at possible workarounds. We're further spawning more instances and see a slight decrease in queue times for builds.Jun 19, 11:16 UTC Update - We're starting additional instances to deal with the queued builds and are further looking into this.Jun 19, 10:54 UTC Investigating - We're looking into an issue with longer than usual queue times for builds on Codeship Basic.

Codeship
Queued Builds on Codeship Basic

Jun 19, 12:49 UTC Monitoring - We've mostly caught up with the demand and are working through the remaining queued builds. We should be back operating normally soon.Jun 19, 12:19 UTC Update - We're adding new capacity to our build pool as fast as possible to deal with both the backlog of waiting builds as well as the increased load during US working hours.Jun 19, 11:42 UTC Identified - We traced the issue back to capacity issues with AWS and are looking at possible workarounds. We're further spawning more instances and see a slight decrease in queue times for builds.Jun 19, 11:16 UTC Update - We're starting additional instances to deal with the queued builds and are further looking into this.Jun 19, 10:54 UTC Investigating - We're looking into an issue with longer than usual queue times for builds on Codeship Basic.

Codeship
Queued Builds on Codeship Basic

Jun 19, 12:19 UTC Update - We're adding new capacity to our build pool as fast as possible to deal with both the backlog of waiting builds as well as the increased load during US working hours.Jun 19, 11:42 UTC Identified - We traced the issue back to capacity issues with AWS and are looking at possible workarounds. We're further spawning more instances and see a slight decrease in queue times for builds.Jun 19, 11:16 UTC Update - We're starting additional instances to deal with the queued builds and are further looking into this.Jun 19, 10:54 UTC Investigating - We're looking into an issue with longer than usual queue times for builds on Codeship Basic.

Codeship
Queued Builds on Codeship Basic

Jun 19, 11:42 UTC Identified - We traced the issue back to capacity issues with AWS and are looking at possible workarounds. We're further spawning more instances and see a slight decrease in queue times for builds.Jun 19, 11:16 UTC Update - We're starting additional instances to deal with the queued builds and are further looking into this.Jun 19, 10:54 UTC Investigating - We're looking into an issue with longer than usual queue times for builds on Codeship Basic.

Codeship
Queued Builds on Codeship Basic

Jun 19, 11:16 UTC Update - We're starting additional instances to deal with the queued builds and are further looking into this.Jun 19, 10:54 UTC Investigating - We're looking into an issue with longer than usual queue times for builds on Codeship Basic.

Codeship
Queued Builds on Codeship Basic

Jun 19, 10:54 UTC Investigating - We're looking into an issue with longer than usual queue times for builds on Codeship Basic.

Codeship
Heroku Deployment Issues

Jun 9, 21:45 UTC Resolved - Heroku deployments are now working normally again. Note that some builds may require a restart before passing.Jun 9, 15:53 UTC Update - Due to PECL package manager service issues (https://pecl.php.net) it is taking longer than expected to prepare and deploy the fix for Heroku build failures. We hope to have this resolved as soon as possible.Jun 9, 15:16 UTC Identified - We have identified the issue and are preparing a fix to roll out shortly.Jun 9, 13:17 UTC Investigating - We are seeing issues with builds deploying to Heroku failing. We are investigating now.

Codeship
Heroku Deployment Issues

Jun 9, 15:53 UTC Update - Due to PECL package manager service issues (https://pecl.php.net) it is taking longer than expected to prepare and deploy the fix for Heroku build failures. We hope to have this resolved as soon as possible.Jun 9, 15:16 UTC Identified - We have identified the issue and are preparing a fix to roll out shortly.Jun 9, 13:17 UTC Investigating - We are seeing issues with builds deploying to Heroku failing. We are investigating now.

Codeship
Heroku Deployment Issues

Jun 9, 15:16 UTC Identified - We have identified the issue and are preparing a fix to roll out shortly.Jun 9, 13:17 UTC Investigating - We are seeing issues with builds deploying to Heroku failing. We are investigating now.

Codeship
Heroku Deployment Issues

Jun 9, 13:17 UTC Investigating - We are seeing issues with builds deploying to Heroku failing. We are investigating now.

Codeship
Intermittent page load errors

Jun 7, 16:35 UTC Resolved - The application should now be resolving as normal.Jun 7, 14:49 UTC Identified - We are seeing intermittent page load errors, resulting in a 500 response page or an application failing to load error message. We are working to resolve this currently.

Codeship
Intermittent page load errors

Jun 7, 14:49 UTC Identified - We are seeing intermittent page load errors, resulting in a 500 response page or an application failing to load error message. We are working to resolve this currently.

Codeship
Stuck Builds on Codeship Basic

Jun 5, 21:31 UTC Resolved - Builds are processing as planned. We keep monitoring the build queues and will make further changes to improve stability.Jun 5, 18:16 UTC Monitoring - We've resolved the issue and are monitoring our systems. There still might be builds affected by this and we are identifying those and taking steps to unstuck those builds.Jun 5, 15:58 UTC Update - We are taking necessary steps to keep impact on users as minimal as possible and are further looking into the root cause for this.Jun 5, 14:01 UTC Investigating - We are seeing builds stuck in queue erroneously and are investigating further.

Codeship
Stuck Builds on Codeship Basic

Jun 5, 18:16 UTC Monitoring - We've resolved the issue and are monitoring our systems. There still might be builds affected by this and we are identifying those and taking steps to unstuck those builds.Jun 5, 15:58 UTC Update - We are taking necessary steps to keep impact on users as minimal as possible and are further looking into the root cause for this.Jun 5, 14:01 UTC Investigating - We are seeing builds stuck in queue erroneously and are investigating further.

Codeship
Stuck Builds on Codeship Basic

Jun 5, 15:58 UTC Update - We are taking necessary steps to keep impact on users as minimal as possible and are further looking into the root cause for this.Jun 5, 14:01 UTC Investigating - We are seeing builds stuck in queue erroneously and are investigating further.

Codeship
Builds stuck in queue

Jun 5, 14:01 UTC Investigating - We are seeing builds stuck in queue erroneously and are investigating further.

Codeship
Stuck builds on Codeship Basic

Jun 3, 12:17 UTC Resolved - The issue is now resolved and Codeship Basic builds are running as expected.Jun 3, 11:23 UTC Monitoring - The build VM pool for Codeship Basic is back in a healthy state and builds are running normally. We'll monitor behavior closely and will update this incident if necessary.Jun 3, 11:03 UTC Identified - We've identified a problem with scaling build VMs and taken first steps to remedy the issue. Looking into the root cause next.Jun 3, 10:54 UTC Investigating - We're looking into stuck and delayed builds on Codeship Basic

Codeship
Stuck builds on Codeship Basic

Jun 3, 11:23 UTC Monitoring - The build VM pool for Codeship Basic is back in a healthy state and builds are running normally. We'll monitor behavior closely and will update this incident if necessary.Jun 3, 11:03 UTC Identified - We've identified a problem with scaling build VMs and taken first steps to remedy the issue. Looking into the root cause next.Jun 3, 10:54 UTC Investigating - We're looking into stuck and delayed builds on Codeship Basic

Codeship
Stuck builds on Codeship Basic

Jun 3, 11:03 UTC Identified - We've identified a problem with scaling build VMs and taken first steps to remedy the issue. Looking into the root cause next.Jun 3, 10:54 UTC Investigating - We're looking into stuck and delayed builds on Codeship Basic

Codeship
Stuck builds on Codeship Basic

Jun 3, 10:54 UTC Investigating - We're looking into stuck and delayed builds on Codeship Basic

Codeship
Builds intermittently stuck or queued

Jun 3, 07:53 UTC Resolved - This incident has been resolved.Jun 3, 06:37 UTC Monitoring - We are keeping our eyes on the build machines for additional stuck or queued builds.Jun 3, 06:03 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Builds intermittently stuck or queued

Jun 3, 06:37 UTC Monitoring - We are keeping our eyes on the build machines for additional stuck or queued builds.Jun 3, 06:03 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Builds intermittently stuck or queued

Jun 3, 06:03 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Builds intermittently stuck or queued

Jun 3, 06:03 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Builds intermittently stuck or queued

Jun 2, 22:14 UTC Resolved - This incident has been resolved.Jun 2, 20:31 UTC Monitoring - We are keeping our eyes on the build machines for additional stuck or queued builds.Jun 2, 20:02 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Builds intermittently stuck or queued

Jun 2, 20:31 UTC Monitoring - We are keeping our eyes on the build machines for additional stuck or queued builds.Jun 2, 20:02 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Codeship Basic, build

Jun 2, 20:31 UTC Monitoring - We are keeping our eyes on the build machines for additional stuck or queued builds.Jun 2, 20:02 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Codeship Basic, build

Jun 2, 20:02 UTC Investigating - We're looking into an issue with stuck builds on Codeship Basic.

Codeship
Builds intermittently stuck or queued

Jun 1, 15:31 UTC Resolved - This incident has been resolved.Jun 1, 13:08 UTC Monitoring - We are keeping our eyes on the build machines for additional stuck or queued builds.Jun 1, 12:18 UTC Identified - We are seeing builds in some projects intermittently stuck or queuing beyond normal times. We are working to resolve the issue now.

Codeship
Builds intermittently stuck or queued

Jun 1, 13:08 UTC Monitoring - We are keeping our eyes on the build machines for additional stuck or queued builds.Jun 1, 12:18 UTC Identified - We are seeing builds in some projects intermittently stuck or queuing beyond normal times. We are working to resolve the issue now.

Codeship
Builds intermittently stuck or queued

Jun 1, 12:18 UTC Identified - We are seeing builds in some projects intermittently stuck or queuing beyond normal times. We are working to resolve the issue now.

Codeship
GitHub, Major Service Outage

May 31, 13:28 UTC Resolved - This incident has been resolved.May 31, 12:57 UTC Update - According to GitHub the issue is fixed and their service is operating normally again. We'll keep monitoring the situation.May 31, 12:40 UTC Monitoring - GitHub is reporting a major service outage, please see https://status.github.com/ for details. Projects hosted on Bitbucket or Gitlab are not affected by this issue.

Codeship
GitHub, Major Service Outage

May 31, 12:57 UTC Update - According to GitHub the issue is fixed and their service is operating normally again. We'll keep monitoring the situation.May 31, 12:40 UTC Monitoring - GitHub is reporting a major service outage, please see https://status.github.com/ for details. Projects hosted on Bitbucket or Gitlab are not affected by this issue.

Codeship
GitHub, Major Service Outage

May 31, 12:40 UTC Monitoring - GitHub is reporting a major service outage, please see https://status.github.com/ for details. Projects hosted on Bitbucket or Gitlab are not affected by this issue.

Codeship
Build issues installing gems

May 24, 20:02 UTC Resolved - This incident has been resolved.May 24, 19:23 UTC Monitoring - We've just rolled out a change that should continue propagating over the next 30-45 minutes. If you see any additional issues past that point, please let us know. You may need to clear your cache and restart your builds, as well.May 24, 18:04 UTC Investigating - We are seeing intermittent build failures related to downloading and installing Ruby gems. We are investigating now.

Codeship
Build issues installing gems

May 24, 19:23 UTC Monitoring - We've just rolled out a change that should continue propagating over the next 30-45 minutes. If you see any additional issues past that point, please let us know. You may need to clear your cache and restart your builds, as well.May 24, 18:04 UTC Investigating - We are seeing intermittent build failures related to downloading and installing Ruby gems. We are investigating now.

Codeship
Build issues installing gems

May 24, 18:04 UTC Investigating - We are seeing intermittent build failures related to downloading and installing Ruby gems. We are investigating now.

Codeship
Intermittent Composer issues

May 23, 21:19 UTC Resolved - We believe Composer is again working as normal.May 23, 18:14 UTC Monitoring - We are seeing intermittent timeouts and failures in some builds when installing Compose packages. The issues seem to be related to sporadic Composer outages throughout the day and we are monitoring the situation. In some cases, clearing your cache and restarting your builds may help, but as the issue is related to an external service this is unlikely to resolve it in most cases.

Codeship
Intermittent Composer issues

May 23, 18:14 UTC Monitoring - We are seeing intermittent timeouts and failures in some builds when installing Compose packages. The issues seem to be related to sporadic Composer outages throughout the day and we are monitoring the situation. In some cases, clearing your cache and restarting your builds may help, but as the issue is related to an external service this is unlikely to resolve it in most cases.

Codeship
Issues for PHP based projects using Composer

May 23, 15:09 UTC Resolved - This incident has been resolved.May 23, 13:23 UTC Update - Packagist deployed a possible fix and will monitor and investigate further, see https://twitter.com/packagist/status/867007744318083073May 23, 10:00 UTC Update - See https://twitter.com/packagist/status/866952618349920257 for an official updated from PackagistMay 23, 09:24 UTC Update - There is a GitHub issue with ongoing communication from the maintainers & community available at https://github.com/composer/composer/issues/6342May 23, 09:22 UTC Monitoring - Composer/Packagist is looking into an issue with their infrastructure, which can cause calls to the Composer CLI to fail. We're in contact with them and will keep you updated.

Codeship
Issues for PHP based projects using Composer

May 23, 13:23 UTC Update - Packagist deployed a possible fix and will monitor and investigate further, see https://twitter.com/packagist/status/867007744318083073May 23, 10:00 UTC Update - See https://twitter.com/packagist/status/866952618349920257 for an official updated from PackagistMay 23, 09:24 UTC Update - There is a GitHub issue with ongoing communication from the maintainers & community available at https://github.com/composer/composer/issues/6342May 23, 09:22 UTC Monitoring - Composer/Packagist is looking into an issue with their infrastructure, which can cause calls to the Composer CLI to fail. We're in contact with them and will keep you updated.

Codeship
Issues for PHP based projects using Composer

May 23, 10:00 UTC Update - See https://twitter.com/packagist/status/866952618349920257 for an official updated from PackagistMay 23, 09:24 UTC Update - There is a GitHub issue with ongoing communication from the maintainers & community available at https://github.com/composer/composer/issues/6342May 23, 09:22 UTC Monitoring - Composer/Packagist is looking into an issue with their infrastructure, which can cause calls to the Composer CLI to fail. We're in contact with them and will keep you updated.

Codeship
Issues for PHP based projects using Composer

May 23, 09:24 UTC Update - There is a GitHub issue with ongoing communication from the maintainers & community available at https://github.com/composer/composer/issues/6342May 23, 09:22 UTC Monitoring - Composer/Packagist is looking into an issue with their infrastructure, which can cause calls to the Composer CLI to fail. We're in contact with them and will keep you updated.

Codeship
Issues for PHP based projects using Composer

May 23, 09:22 UTC Monitoring - Composer/Packagist is looking into an issue with their infrastructure, which can cause calls to the Composer CLI to fail. We're in contact with them and will keep you updated.

Codeship
Codeship Pro, potential issues with builds

May 16, 14:34 UTC Resolved - We've implemented a fix and are adding tests to make sure this isn't triggered againMay 16, 13:15 UTC Update - We've narrowed this down to an issue affecting a single account. We'll look into this further and update this incident accordinglyMay 16, 13:01 UTC Investigating - We're investigating errors in accessing builds on Codeship Pro.

Codeship
Codeship Pro, potential issues with builds

May 16, 13:15 UTC Update - We've narrowed this down to an issue affecting a single account. We'll look into this further and update this incident accordinglyMay 16, 13:01 UTC Investigating - We're investigating errors in accessing builds on Codeship Pro.

Codeship
Codeship Pro, potential issues with builds

May 16, 13:01 UTC Investigating - We're investigating errors in accessing builds on Codeship Pro.

Codeship
Codeship Basic UI may indicate hanging builds

May 15, 18:36 UTC Resolved - Codeship Basic builds should be displaying correctly in the UI again.May 15, 17:20 UTC Identified - Builds on Codeship Basic may appear to be running indefinitely in the UI. If you encounter this, the build is fine and refreshing the page should clear the issue.

Codeship
Codeship Basic UI may indicate hanging builds

May 15, 17:20 UTC Identified - Builds on Codeship Basic may appear to be running indefinitely in the UI. If you encounter this, the build is fine and refreshing the page should clear the issue.

Codeship
Build failures with Python-based builds

May 9, 16:52 UTC Resolved - This incident has been resolved.May 9, 15:47 UTC Monitoring - Builds should now be running normally, although they may require a cache reset in some cases. We will continue to monitor the situation to look for additional issues.May 9, 15:13 UTC Investigating - Some users are experiencing intermittent failures with Python-based builds. We are investigating the cause now. In some cases, clearing your dependency cache and restarting the build may resolve the issue.

Codeship
Build failures with Python-based builds

May 9, 15:47 UTC Monitoring - Builds should now be running normally, although they may require a cache reset in some cases. We will continue to monitor the situation to look for additional issues.May 9, 15:13 UTC Investigating - Some users are experiencing intermittent failures with Python-based builds. We are investigating the cause now. In some cases, clearing your dependency cache and restarting the build may resolve the issue.

Codeship
Build failures with Python-based builds

May 9, 15:13 UTC Investigating - Some users are experiencing intermittent failures with Python-based builds. We are investigating the cause now. In some cases, clearing your dependency cache and restarting the build may resolve the issue.

Codeship
Hanging builds

May 5, 22:04 UTC Resolved - This incident has been resolved.May 5, 19:30 UTC Monitoring - We are continuing to observe the build queue catch up after the earlier issues and will keep our eyes on it for further problems.May 5, 17:24 UTC Update - The ongoing build issues are specific to Codeship Basic.May 5, 17:22 UTC Identified - We have observed and resolved an issue with builds hanging in the system. Stuck builds will need to be stopped and restarted but should then run as normal. Queued builds should now begin processing as normal.

Codeship
Hanging builds

May 5, 19:30 UTC Monitoring - We are continuing to observe the build queue catch up after the earlier issues and will keep our eyes on it for further problems.May 5, 17:24 UTC Update - The ongoing build issues are specific to Codeship Basic.May 5, 17:22 UTC Identified - We have observed and resolved an issue with builds hanging in the system. Stuck builds will need to be stopped and restarted but should then run as normal. Queued builds should now begin processing as normal.

Codeship
Hanging builds

May 5, 17:24 UTC Update - The ongoing build issues are specific to Codeship Basic.May 5, 17:22 UTC Identified - We have observed and resolved an issue with builds hanging in the system. Stuck builds will need to be stopped and restarted but should then run as normal. Queued builds should now begin processing as normal.

Codeship
Hanging builds

May 5, 17:22 UTC Identified - We have observed and resolved an issue with builds hanging in the system. Stuck builds will need to be stopped and restarted but should then run as normal. Queued builds should now begin processing as normal.

Codeship
Hanging builds on Codeship Basic

Apr 5, 02:16 UTC Resolved - All builds should be back to normal.Apr 4, 23:51 UTC Monitoring - We are monitoring the Node.js outage and during this time Basic builds may have issues during the "Preparing Virtual Machine" step or when using any nvm commands.Apr 4, 23:48 UTC Identified - Some Codeship Basic builds are impacted by a Node.js outage reported here: https://github.com/nodejs/nodejs.org/issues/1191.Apr 4, 23:07 UTC Investigating - We're currently looking into hanging builds on Codeship Basic, related to build VM startup.

Codeship
Hanging builds on Codeship Basic

Apr 4, 23:51 UTC Monitoring - We are monitoring the Node.js outage and during this time Basic builds may have issues during the "Preparing Virtual Machine" step or when using any nvm commands.Apr 4, 23:48 UTC Identified - Some Codeship Basic builds are impacted by a Node.js outage reported here: https://github.com/nodejs/nodejs.org/issues/1191.Apr 4, 23:07 UTC Investigating - We're currently looking into hanging builds on Codeship Basic, related to build VM startup.

Codeship
Hanging builds on Codeship Basic

Apr 4, 23:48 UTC Identified - Some Codeship Basic builds are impacted by a Node.js outage reported here: https://github.com/nodejs/nodejs.org/issues/1191.Apr 4, 23:07 UTC Investigating - We're currently looking into hanging builds on Codeship Basic, related to build VM startup.

Codeship
Hanging builds on Codeship Basic

Apr 4, 23:07 UTC Investigating - We're currently looking into hanging builds on Codeship Basic, related to build VM startup.

Codeship
Hanging builds on Codeship Basic

Apr 4, 23:07 UTC Investigating - We're currently looking into hanging builds on Codeship Basic, related to build VM startup.

Codeship
Some Bitbucket builds not triggers

Mar 28, 16:45 UTC Resolved - Bitbucket builds are fully restored.Mar 28, 15:47 UTC Update - Atlassian reports their webhook performance has recovered and they are currently working through the backlog.Mar 28, 14:11 UTC Monitoring - Atlassian updated their status page, they are experiencing delays with webhooks, see https://status.bitbucket.org/incidents/ysrx3vwmwxn5 for more information.Mar 28, 13:39 UTC Investigating - We are seeing some Bitbucket-based projects not receiving commits to trigger new builds, and are investigating now.

Codeship
Hanging builds on Codeship Basic

Mar 27, 08:47 UTC Resolved - This incident has been resolved.Mar 27, 08:28 UTC Monitoring - A fix has been implemented and we are monitoring the results.Mar 27, 07:49 UTC Identified - We identified the root cause and terminated those instances. We're looking at all other currently running instances as well as monitoring builds very closely.Mar 27, 07:41 UTC Investigating - We're currently looking into hanging builds on Codeship Basic, related to build VM startup.

Codeship
app.codeship.com is down

Mar 24, 10:14 UTC Resolved - This issue has now been resolved.Mar 24, 09:36 UTC Monitoring - A database issue was causing issues with the website and running builds. We put the app in maintenance mode to resolve the issue and are now back up. We're monitoring performance closely and will follow up.Mar 24, 09:17 UTC Investigating - Our website is currently offline and we're looking into it. No new builds are currently run, they are stored and will be triggered once we recovered.

Codeship
Performance Degradation on Codeship Pro

Mar 23, 19:11 UTC Resolved - All builds should now be back to normal.Mar 23, 18:34 UTC Monitoring - Builds are now beginning to return to normal capacity and we are continuing to monitor the service for additional issues.Mar 23, 18:25 UTC Update - Initial database maintenance is completed, we'll keep monitoring performance.Mar 23, 17:48 UTC Update - We'll have to do some emergency database maintenance, the web application will be unreachable for a couple minutes.Mar 23, 17:38 UTC Identified - We have identified a problem delayed log updates and implemented initial improvements. We're still investigating and working on a proper fix.Mar 23, 16:51 UTC Investigating - We're looking into issues with performance on Codeship Pro being lower than usually.

Codeship
Stuck builds on Codeship Basic

Mar 23, 12:24 UTC Resolved - This incident has been resolved.Mar 23, 10:20 UTC Monitoring - The issue has been identified and the machine responsible shut down. We're still looking into stuck builds and will monitor running builds closely.Mar 23, 08:53 UTC Investigating - We're looking into a small percentage of builds getting stuck on Codeship Basic

Codeship
Codeship not reachable

Mar 21, 11:09 UTC Resolved - This issue is now fully resolved.Mar 21, 10:54 UTC Monitoring - The issue has been resolved, https://app.codeship.com is loading again. We'll monitor performance.Mar 21, 10:45 UTC Identified - We are experiencing issues with our database. Our engineering team is looking into them and working on a fix.Mar 21, 10:31 UTC Investigating - We're currently looking into an issue with https://app.codeship.com not being reachable.

Codeship
Codeship not reachable

Mar 21, 10:54 UTC Monitoring - The issue has been resolved, https://app.codeship.com is loading again. We'll monitor performance.Mar 21, 10:45 UTC Identified - We are experiencing issues with our database. Our engineering team is looking into them and working on a fix.Mar 21, 10:31 UTC Investigating - We're currently looking into an issue with https://app.codeship.com not being reachable.

Codeship
Codeship not reachable

Mar 21, 10:45 UTC Identified - We are experiencing issues with our database. Our engineering team is looking into them and working on a fix.Mar 21, 10:31 UTC Investigating - We're currently looking into an issue with https://app.codeship.com not being reachable.

Codeship
Codeship not reachable

Mar 21, 10:31 UTC Investigating - We're currently looking into an issue with https://app.codeship.com not being reachable.

Codeship
web codeship not reachable

Mar 21, 10:31 UTC Investigating - We're currently looking into an issue with https://app.codeship.com not being reachable.

Codeship
Codeship Basic Intermittent Build Issues

Mar 16, 18:01 UTC Resolved - We have fixed this morning's build issues and apologize for the inconvenience.Mar 16, 14:45 UTC Investigating - We have received reports about intermittent build failures and hanging/stuck builds, running on Codeship Basic. We are investigating now.

Codeship
Codeship Basic Intermittent Build Issues

Mar 16, 14:45 UTC Investigating - We have received reports about intermittent build failures and hanging/stuck builds, running on Codeship Basic. We are investigating now.

Codeship
Build issues

Mar 1, 02:43 UTC Resolved - Build queues have returned to normal and builds are processing as expected.Mar 1, 00:48 UTC Monitoring - Build processing has returned to normal, but elevated queues may take some time to process. We're monitoring build processing.Feb 28, 22:50 UTC Update - The AWS S3 issue is resolved, but other AWS services are still experiencing increased error rates. We're monitoring https://status.aws.amazon.com/ and will keep you updated.Feb 28, 19:50 UTC Update - AWS continues to experience high error rates with S3 in US-EAST-1, which is impacting various other AWS services, among them EC2, which we rely on for build VMs. They believe they understand the root cause and are working on fixing the issue. See https://status.aws.amazon.com/ for details.Feb 28, 18:38 UTC Update - An AWS service issue is impacting Codeship builds, as well as many other tools such as Github, Heroku, and Quay.io. We are monitoring AWS' status and will respond and update accordingly.Feb 28, 18:33 UTC Identified - The issue has been identified and a fix is being implemented.Feb 28, 18:05 UTC Update - We are now also seeing webhook-related issues, as reported by Github: https://status.github.com/Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Build issues

Mar 1, 00:48 UTC Monitoring - Build processing has returned to normal, but elevated queues may take some time to process. We're monitoring build processing.Feb 28, 22:50 UTC Update - The AWS S3 issue is resolved, but other AWS services are still experiencing increased error rates. We're monitoring https://status.aws.amazon.com/ and will keep you updated.Feb 28, 19:50 UTC Update - AWS continues to experience high error rates with S3 in US-EAST-1, which is impacting various other AWS services, among them EC2, which we rely on for build VMs. They believe they understand the root cause and are working on fixing the issue. See https://status.aws.amazon.com/ for details.Feb 28, 18:38 UTC Update - An AWS service issue is impacting Codeship builds, as well as many other tools such as Github, Heroku, and Quay.io. We are monitoring AWS' status and will respond and update accordingly.Feb 28, 18:33 UTC Identified - The issue has been identified and a fix is being implemented.Feb 28, 18:05 UTC Update - We are now also seeing webhook-related issues, as reported by Github: https://status.github.com/Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Build issues

Feb 28, 22:50 UTC Update - The AWS S3 issue is resolved, but other AWS services are still experiencing increased error rates. We're monitoring https://status.aws.amazon.com/ and will keep you updated.Feb 28, 19:50 UTC Update - AWS continues to experience high error rates with S3 in US-EAST-1, which is impacting various other AWS services, among them EC2, which we rely on for build VMs. They believe they understand the root cause and are working on fixing the issue. See https://status.aws.amazon.com/ for details.Feb 28, 18:38 UTC Update - An AWS service issue is impacting Codeship builds, as well as many other tools such as Github, Heroku, and Quay.io. We are monitoring AWS' status and will respond and update accordingly.Feb 28, 18:33 UTC Identified - The issue has been identified and a fix is being implemented.Feb 28, 18:05 UTC Update - We are now also seeing webhook-related issues, as reported by Github: https://status.github.com/Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Build issues

Feb 28, 19:50 UTC Update - AWS continues to experience high error rates with S3 in US-EAST-1, which is impacting various other AWS services, among them EC2, which we rely on for build VMs. They believe they understand the root cause and are working on fixing the issue. See https://status.aws.amazon.com/ for details.Feb 28, 18:38 UTC Update - An AWS service issue is impacting Codeship builds, as well as many other tools such as Github, Heroku, and Quay.io. We are monitoring AWS' status and will respond and update accordingly.Feb 28, 18:33 UTC Identified - The issue has been identified and a fix is being implemented.Feb 28, 18:05 UTC Update - We are now also seeing webhook-related issues, as reported by Github: https://status.github.com/Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Build issues

Feb 28, 18:38 UTC Update - An AWS service issue is impacting Codeship builds, as well as many other tools such as Github, Heroku, and Quay.io. We are monitoring AWS' status and will respond and update accordingly.Feb 28, 18:33 UTC Identified - The issue has been identified and a fix is being implemented.Feb 28, 18:05 UTC Update - We are now also seeing webhook-related issues, as reported by Github: https://status.github.com/Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Build issues

Feb 28, 18:33 UTC Identified - The issue has been identified and a fix is being implemented.Feb 28, 18:05 UTC Update - We are now also seeing webhook-related issues, as reported by Github: https://status.github.com/Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Build issues

Feb 28, 18:05 UTC Update - We are now also seeing webhook-related issues, as reported by Github: https://status.github.com/Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Build issues

Feb 28, 18:02 UTC Investigating - We are investigating build issues related to an AWS service interruption.

Codeship
Codeship is experiencing build delays

Feb 2, 18:54 UTC Resolved - This incident has been resolved. Build pipelines on the Codeship Basic platform momentarily dropped, delaying build runs. All builds are now running as normally expected.Feb 2, 18:22 UTC Investigating - We are currently investigating an issue impacting build runs. More info to come.

Codeship
Codeship is experiencing build delays

Feb 2, 18:22 UTC Investigating - We are currently investigating an issue impacting build runs. More info to come.

Codeship
Build Notifications on Codeship Pro Are Being Delayed

Jan 18, 15:21 UTC Resolved - This issue has been resolved.Jan 18, 14:43 UTC Investigating - We are currently seeing an issue where build notifications are delayed. We are investigating build status and build notifications delay in delivery up to an hour.

Codeship
Build Notifications on Codeship Pro Are Being Delayed

Jan 18, 14:43 UTC Investigating - We are currently seeing an issue where build notifications are delayed. We are investigating build status and build notifications delay in delivery up to an hour.

Codeship
Build Notifications Are Being Delayed

Jan 18, 14:43 UTC Investigating - We are currently seeing an issue where build notifications are delayed. We are investigating build status and build notifications delay in delivery up to an hour.

Codeship
Codeship Pro build performance issues.

Jan 11, 00:15 UTC Resolved - Codeship Pro builds are now working as normal.Jan 11, 00:05 UTC Identified - We are seeing Codeship Pro build issues and failures.

Codeship
Codeship Pro build performance issues.

Jan 11, 00:05 UTC Identified - We are seeing Codeship Pro build issues and failures.

Codeship
Potential Build Allocation And Deployment Issues

Jan 10, 22:05 UTC Resolved - The AWS issue has now resolved but there may be sporadic issues over the next several minutes as the service recovers.Jan 10, 21:51 UTC Update - We have identified an issue with build allocation failures with Codeship Pro and deployment failures with Codeship Basic to AWS services due to upstream issues with AWS. Latest status available at http://status.aws.amazon.com/Jan 10, 20:53 UTC Identified - We are investigating build allocation failures with Codeship Pro and deployment failures with Codeship Basic.Jan 10, 20:47 UTC Investigating - We are currently investigating an issue causing possible deployments to Heroku and AWS to fail. More info to come.

Codeship
Potential Build Allocation And Deployment Issues

Jan 10, 21:51 UTC Update - We have identified an issue with build allocation failures with Codeship Pro and deployment failures with Codeship Basic to AWS services due to upstream issues with AWS. Latest status available at http://status.aws.amazon.com/Jan 10, 20:53 UTC Identified - We are investigating build allocation failures with Codeship Pro and deployment failures with Codeship Basic.Jan 10, 20:47 UTC Investigating - We are currently investigating an issue causing possible deployments to Heroku and AWS to fail. More info to come.

Codeship
Potential Build Allocation And Deployment Issues

Jan 10, 20:53 UTC Identified - We are investigating build allocation failures with Codeship Pro and deployment failures with Codeship Basic.Jan 10, 20:47 UTC Investigating - We are currently investigating an issue causing possible deployments to Heroku and AWS to fail. More info to come.

Codeship
Potential Build Allocation And Deployment Issues

Jan 10, 20:47 UTC Investigating - We are currently investigating an issue causing possible deployments to Heroku and AWS to fail. More info to come.

Codeship
Potential Deployment Issues

Jan 10, 20:47 UTC Investigating - We are currently investigating an issue causing possible deployments to Heroku and AWS to fail. More info to come.

Codeship
Potential Deployment Issues

Jan 10, 20:47 UTC Investigating - We are currently investigating an issue causing possible deployments to Heroku and AWS to fail. More info to come.

Codeship
Potential build allocation failures

Jan 9, 21:02 UTC Resolved - This incident has been resolved.Jan 9, 20:41 UTC Monitoring - A fix has been implemented and we are monitoring the results.Jan 9, 20:26 UTC Identified - We have confirmed an AWS build instance provisioning failure impacting new Codeship Pro builds.Jan 9, 20:19 UTC Investigating - We are investigating an issue provisioning AWS instances for Codeship Pro builds.

Codeship
Potential build allocation failures

Jan 9, 20:41 UTC Monitoring - A fix has been implemented and we are monitoring the results.Jan 9, 20:26 UTC Identified - We have confirmed an AWS build instance provisioning failure impacting new Codeship Pro builds.Jan 9, 20:19 UTC Investigating - We are investigating an issue provisioning AWS instances for Codeship Pro builds.

Codeship
Potential build allocation failures

Jan 9, 20:26 UTC Identified - We have confirmed an AWS build instance provisioning failure impacting new Codeship Pro builds.Jan 9, 20:19 UTC Investigating - We are investigating an issue provisioning AWS instances for Codeship Pro builds.

Codeship
Potential build allocation failures

Jan 9, 20:26 UTC Identified - We have confirmed an AWS build instance provisioning failure impacting new Codeship Pro builds.Jan 9, 20:19 UTC Investigating - We are investigating an issue provisioning AWS instances for Codeship Pro builds.

Codeship
Potential build allocation failures

Jan 9, 20:19 UTC Investigating - We are investigating an issue provisioning AWS instances for Codeship Pro builds.

Codeship
Build Delays And Timeouts

Dec 11, 18:57 UTC Resolved - This incident has been resolved.Dec 11, 18:20 UTC Monitoring - A fix has been implemented and we are monitoring the results.Dec 11, 17:42 UTC Investigating - We are aware of an issue impacting build allocation and build log timeouts, we are looking into this further.

Codeship
Build Delays And Timeouts

Dec 11, 18:20 UTC Monitoring - A fix has been implemented and we are monitoring the results.Dec 11, 17:42 UTC Investigating - We are aware of an issue impacting build allocation and build log timeouts, we are looking into this further.

Codeship
Build Delays And Timeouts

Dec 11, 17:42 UTC Investigating - We are aware of an issue impacting build allocation and build log timeouts, we are looking into this further.

Codeship
NodeJS binary installation slow

Dec 9, 17:18 UTC Resolved - This issue has been resolved.Dec 9, 16:23 UTC Monitoring - NodeJS is currently experiencing issues with slow downloads of their binaries, which may cause some node version installs to fail https://twitter.com/nodejs/status/807241179238699009. Please see https://github.com/nodejs/build/issues/562#issuecomment-266024258 for updates.

Codeship
NodeJS binary installation slow

Dec 9, 16:23 UTC Monitoring - NodeJS is currently experiencing issues with slow downloads of their binaries, which may cause some node version installs to fail https://twitter.com/nodejs/status/807241179238699009. Please see https://github.com/nodejs/build/issues/562#issuecomment-266024258 for updates.

Codeship
Docker platform build delays

Nov 30, 18:38 UTC Resolved - This issue has been resolved.Nov 30, 18:23 UTC Monitoring - Builds are now running as normally expected. We will continue to further monitor build allocation state.Nov 30, 17:28 UTC Investigating - We are currently experiencing an issue with Docker platform builds taking longer time than usual to allocate. Update to come.

Codeship
Docker platform build delays

Nov 30, 18:23 UTC Monitoring - Builds are now running as normally expected. We will continue to further monitor build allocation state.Nov 30, 17:28 UTC Investigating - We are currently experiencing an issue with Docker platform builds taking longer time than usual to allocate. Update to come.

Codeship
Docker platform build delays

Nov 30, 17:28 UTC Investigating - We are currently experiencing an issue with Docker platform builds taking longer time than usual to allocate. Update to come.

Codeship
Login issues

Nov 30, 15:54 UTC Resolved - This issue has been resolved. If you are still experiencing issues, please clear cookies from Codeship.com in your browser.Nov 30, 15:43 UTC Investigating - We're currently looking into login issues with the application

Codeship
Login issues

Nov 30, 15:43 UTC Investigating - We're currently looking into login issues with the application

Codeship
Docker platform are experiencing delayed builds

Nov 21, 17:53 UTC Resolved - This incident has been resolved.Nov 21, 17:42 UTC Monitoring - Our Docker platform is back to a normal provisioning state. We will continue to monitor the state of our build resource allocation until a complete stable state is confirmed.Nov 21, 16:55 UTC Investigating - Our engineering team is investigating an issue where builds on the Docker platform are delayed in running in the Allocating state

Codeship
Docker platform are experiencing delayed builds

Nov 21, 17:42 UTC Monitoring - Our Docker platform is back to a normal provisioning state. We will continue to monitor the state of our build resource allocation until a complete stable state is confirmed.Nov 21, 16:55 UTC Investigating - Our engineering team is investigating an issue where builds on the Docker platform are delayed in running in the Allocating state

Codeship
Docker platform are experiencing delayed builds

Nov 21, 16:55 UTC Investigating - Our engineering team is investigating an issue where builds on the Docker platform are delayed in running in the Allocating state

Codeship
Docker platform are experiencing delayed builds

Nov 21, 16:55 UTC Investigating - Oue engineering team is investigating an issue where builds on the Jet/Docker platform are delayed in running in the Allocating state

Codeship
codeship.com temporarily unavailable

Nov 21, 08:59 UTC Resolved - This incident has been resolved.Nov 21, 07:33 UTC Monitoring - The Codeship content pages at codeship.com are temporarily unavailable due to a CMS provider outage. You can access your projects and builds via app.codeship.com.

Codeship
codeship.com temporarily unavailable

Nov 21, 07:33 UTC Monitoring - The Codeship content pages at codeship.com are temporarily unavailable due to a CMS provider outage. You can access your projects and builds via app.codeship.com.

Codeship
Intermittent timeouts

Nov 7, 13:16 UTC Resolved - This incident has been resolved.Nov 7, 12:58 UTC Investigating - We are aware of a potential issue with intermittent page loading timeouts and build responsiveness issues. We are investigating now to learn more.

Codeship
Intermittent timeouts

Nov 7, 12:58 UTC Investigating - We are aware of a potential issue with intermittent page loading timeouts and build responsiveness issues. We are investigating now to learn more.

Codeship
Issue with GitHub webhook delivery

Oct 29, 07:17 UTC Resolved - Builds for GitHub based projects are getting triggered as expected again.Oct 29, 06:56 UTC Monitoring - We put a fix in place, and hooks seem to come in like they should.Oct 29, 06:51 UTC Identified - We identified the issue, and working on fixing it.Oct 29, 05:47 UTC Update - We see problems with hooks getting sent from GitHub not triggering builds correctly.Oct 29, 05:36 UTC Investigating - We have received reports of builds not triggering and are investigation a possible issue

Codeship
Issue with GitHub webhook delivery

Oct 29, 06:56 UTC Monitoring - We put a fix in place, and hooks seem to come in like they should.Oct 29, 06:51 UTC Identified - We identified the issue, and working on fixing it.Oct 29, 05:47 UTC Update - We see problems with hooks getting sent from GitHub not triggering builds correctly.Oct 29, 05:36 UTC Investigating - We have received reports of builds not triggering and are investigation a possible issue

Codeship
Issues with builds not triggering

Oct 29, 06:56 UTC Monitoring - We put a fix in place, and hooks seem to come in like they should.Oct 29, 06:51 UTC Identified - We identified the issue, and working on fixing it.Oct 29, 05:47 UTC Update - We see problems with hooks getting sent from GitHub not triggering builds correctly.Oct 29, 05:36 UTC Investigating - We have received reports of builds not triggering and are investigation a possible issue

Codeship
Issues with builds not triggering

Oct 29, 06:51 UTC Identified - We identified the issue, and working on fixing it.Oct 29, 05:47 UTC Update - We see problems with hooks getting sent from GitHub not triggering builds correctly.Oct 29, 05:36 UTC Investigating - We have received reports of builds not triggering and are investigation a possible issue

Codeship
Issues with builds not triggering

Oct 29, 05:47 UTC Update - We see problems with hooks getting sent from GitHub not triggering builds correctly.Oct 29, 05:36 UTC Investigating - We have received reports of builds not triggering and are investigation a possible issue

Codeship
Issues with builds not triggering

Oct 29, 05:36 UTC Investigating - We have received reports of builds not triggering and are investigation a possible issue

Codeship
Issues with .io DNS resolution

Oct 28, 13:25 UTC Resolved - The issue has been resolved.Oct 28, 11:58 UTC Monitoring - The issue is now limited to receiving webhooks from GitHub and Bitbucket, all other parts of the application are working as expected.Oct 28, 09:33 UTC Update - Docker added an update regarding .io DNS lookup to their status page at https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5813095738015eaf19001ce2Oct 28, 09:06 UTC Update - The issue with .io DNS resolution is also affecting triggering builds on Codeship via webhooks from GitHub and BitbucketOct 28, 08:34 UTC Update - Engineers from StatusPage narrowed down the issue to the .io top level domain with domains no longer delegating out to authoritative name servers. See http://metastatuspage.com/incidents/gmjyb3r6dmx3 for their incident report.Oct 28, 08:27 UTC Identified - We're seeing DNS resolution problems with the .io TLD, which blocks access to the quay.io registry we use to store required Docker images.Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
Issues with .io DNS resolution

Oct 28, 11:58 UTC Monitoring - The issue is now limited to receiving webhooks from GitHub and Bitbucket, all other parts of the application are working as expected.Oct 28, 09:33 UTC Update - Docker added an update regarding .io DNS lookup to their status page at https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5813095738015eaf19001ce2Oct 28, 09:06 UTC Update - The issue with .io DNS resolution is also affecting triggering builds on Codeship via webhooks from GitHub and BitbucketOct 28, 08:34 UTC Update - Engineers from StatusPage narrowed down the issue to the .io top level domain with domains no longer delegating out to authoritative name servers. See http://metastatuspage.com/incidents/gmjyb3r6dmx3 for their incident report.Oct 28, 08:27 UTC Identified - We're seeing DNS resolution problems with the .io TLD, which blocks access to the quay.io registry we use to store required Docker images.Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
Issues with .io DNS resolution

Oct 28, 09:33 UTC Update - Docker added an update regarding .io DNS lookup to their status page at https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5813095738015eaf19001ce2Oct 28, 09:06 UTC Update - The issue with .io DNS resolution is also affecting triggering builds on Codeship via webhooks from GitHub and BitbucketOct 28, 08:34 UTC Update - Engineers from StatusPage narrowed down the issue to the .io top level domain with domains no longer delegating out to authoritative name servers. See http://metastatuspage.com/incidents/gmjyb3r6dmx3 for their incident report.Oct 28, 08:27 UTC Identified - We're seeing DNS resolution problems with the .io TLD, which blocks access to the quay.io registry we use to store required Docker images.Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
Issues with .io DNS resolution

Oct 28, 09:06 UTC Update - The issue with .io DNS resolution is also affecting triggering builds on Codeship via webhooks from GitHub and BitbucketOct 28, 08:34 UTC Update - Engineers from StatusPage narrowed down the issue to the .io top level domain with domains no longer delegating out to authoritative name servers. See http://metastatuspage.com/incidents/gmjyb3r6dmx3 for their incident report.Oct 28, 08:27 UTC Identified - We're seeing DNS resolution problems with the .io TLD, which blocks access to the quay.io registry we use to store required Docker images.Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
Possible network issues

Oct 28, 09:06 UTC Update - The issue with .io DNS resolution is also affecting triggering builds on Codeship via webhooks from GitHub and BitbucketOct 28, 08:34 UTC Update - Engineers from StatusPage narrowed down the issue to the .io top level domain with domains no longer delegating out to authoritative name servers. See http://metastatuspage.com/incidents/gmjyb3r6dmx3 for their incident report.Oct 28, 08:27 UTC Identified - We're seeing DNS resolution problems with the .io TLD, which blocks access to the quay.io registry we use to store required Docker images.Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
Possible network issues

Oct 28, 08:34 UTC Update - Engineers from StatusPage narrowed down the issue to the .io top level domain with domains no longer delegating out to authoritative name servers. See http://metastatuspage.com/incidents/gmjyb3r6dmx3 for their incident report.Oct 28, 08:27 UTC Identified - We're seeing DNS resolution problems with the .io TLD, which blocks access to the quay.io registry we use to store required Docker images.Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
Possible network issues

Oct 28, 08:27 UTC Identified - We're seeing DNS resolution problems with the .io TLD, which blocks access to the quay.io registry we use to store required Docker images.Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
Possible network issues

Oct 28, 08:22 UTC Investigating - Heroku is reporting possible network issues with DNS lookups for the .io TLD. Please see https://status.heroku.com/incidents/967 for details.

Codeship
3rd Party services affected by DNS issues

Oct 21, 22:57 UTC Resolved - All 3rd party services are operating as normally expected. See https://www.dynstatus.com/incidents/5r9mppc1kb77 for Dyn's report of the incident.Oct 21, 16:26 UTC Update - Other services such as GitHub is affected by DNS issues as well. See https://www.dynstatus.com/incidents/nlr4yrr162t8 for more information and updates.Oct 21, 16:21 UTC Monitoring - A fix has been implemented and we are monitoring the results.Oct 21, 16:21 UTC Investigating - Heroku is current experiencing an outage with their DNS provider. Codeship services may be affected during this outage. More information to follow.

Codeship
3rd Party services affected by DNS issues

Oct 21, 16:26 UTC Update - Other services such as GitHub is affected by DNS issues as well. See https://www.dynstatus.com/incidents/nlr4yrr162t8 for more information and updates.Oct 21, 16:21 UTC Monitoring - A fix has been implemented and we are monitoring the results.Oct 21, 16:21 UTC Investigating - Heroku is current experiencing an outage with their DNS provider. Codeship services may be affected during this outage. More information to follow.

Codeship
Multiple 3rd party services affected by DNS issues

Oct 21, 14:33 UTC Resolved - All affected 3rd party services report the issue is resolved.Oct 21, 14:06 UTC Update - GitHub is working through the backlog of webhooks, missing builds should be triggered soon. https://twitter.com/githubstatus/status/789466733962752001Oct 21, 13:47 UTC Update - Dyn, the upstream DNS provider, reports performance is restored on their DNS service. 3rd party service performance should improve. https://www.dynstatus.com/incidents/nlr4yrr162t8Oct 21, 13:09 UTC Update - Zendesk, our ticket system is affected by the DNS issues as well, please see https://twitter.com/ZendeskOps/status/789440421516742656 for details.Oct 21, 12:43 UTC Update - Our payment provider is affected by the DNS issue as well, please see https://status.braintreepayments.com/incidents/rbrw2wdmpmts for more details.Oct 21, 12:19 UTC Monitoring - GitHub is experiencing issues with their upstream DNS provider, which is affecting (among others) webhook performance. Builds on Codeship might not be triggered or delayed. Please see https://twitter.com/githubstatus/status/789433336083001344 for more information.

Codeship
Multiple 3rd party services affected by DNS issues

Oct 21, 14:06 UTC Update - GitHub is working through the backlog of webhooks, missing builds should be triggered soon. https://twitter.com/githubstatus/status/789466733962752001Oct 21, 13:47 UTC Update - Dyn, the upstream DNS provider, reports performance is restored on their DNS service. 3rd party service performance should improve. https://www.dynstatus.com/incidents/nlr4yrr162t8Oct 21, 13:09 UTC Update - Zendesk, our ticket system is affected by the DNS issues as well, please see https://twitter.com/ZendeskOps/status/789440421516742656 for details.Oct 21, 12:43 UTC Update - Our payment provider is affected by the DNS issue as well, please see https://status.braintreepayments.com/incidents/rbrw2wdmpmts for more details.Oct 21, 12:19 UTC Monitoring - GitHub is experiencing issues with their upstream DNS provider, which is affecting (among others) webhook performance. Builds on Codeship might not be triggered or delayed. Please see https://twitter.com/githubstatus/status/789433336083001344 for more information.

Codeship
Multiple 3rd party services affected by DNS issues

Oct 21, 13:47 UTC Update - Dyn, the upstream DNS provider, reports performance is restored on their DNS service. 3rd party service performance should improve. https://www.dynstatus.com/incidents/nlr4yrr162t8Oct 21, 13:09 UTC Update - Zendesk, our ticket system is affected by the DNS issues as well, please see https://twitter.com/ZendeskOps/status/789440421516742656 for details.Oct 21, 12:43 UTC Update - Our payment provider is affected by the DNS issue as well, please see https://status.braintreepayments.com/incidents/rbrw2wdmpmts for more details.Oct 21, 12:19 UTC Monitoring - GitHub is experiencing issues with their upstream DNS provider, which is affecting (among others) webhook performance. Builds on Codeship might not be triggered or delayed. Please see https://twitter.com/githubstatus/status/789433336083001344 for more information.

Codeship
GitHub DNS issues

Oct 21, 13:47 UTC Update - Dyn, the upstream DNS provider, reports performance is restored on their DNS service. 3rd party service performance should improve. https://www.dynstatus.com/incidents/nlr4yrr162t8Oct 21, 13:09 UTC Update - Zendesk, our ticket system is affected by the DNS issues as well, please see https://twitter.com/ZendeskOps/status/789440421516742656 for details.Oct 21, 12:43 UTC Update - Our payment provider is affected by the DNS issue as well, please see https://status.braintreepayments.com/incidents/rbrw2wdmpmts for more details.Oct 21, 12:19 UTC Monitoring - GitHub is experiencing issues with their upstream DNS provider, which is affecting (among others) webhook performance. Builds on Codeship might not be triggered or delayed. Please see https://twitter.com/githubstatus/status/789433336083001344 for more information.

Codeship
GitHub DNS issues

Oct 21, 13:09 UTC Update - Zendesk, our ticket system is affected by the DNS issues as well, please see https://twitter.com/ZendeskOps/status/789440421516742656 for details.Oct 21, 12:43 UTC Update - Our payment provider is affected by the DNS issue as well, please see https://status.braintreepayments.com/incidents/rbrw2wdmpmts for more details.Oct 21, 12:19 UTC Monitoring - GitHub is experiencing issues with their upstream DNS provider, which is affecting (among others) webhook performance. Builds on Codeship might not be triggered or delayed. Please see https://twitter.com/githubstatus/status/789433336083001344 for more information.

Codeship
GitHub DNS issues

Oct 21, 12:43 UTC Update - Our payment provider is affected by the DNS issue as well, please see https://status.braintreepayments.com/incidents/rbrw2wdmpmts for more details.Oct 21, 12:19 UTC Monitoring - GitHub is experiencing issues with their upstream DNS provider, which is affecting (among others) webhook performance. Builds on Codeship might not be triggered or delayed. Please see https://twitter.com/githubstatus/status/789433336083001344 for more information.

Codeship
GitHub DNS issues

Oct 21, 12:19 UTC Monitoring - GitHub is experiencing issues with their upstream DNS provider, which is affecting (among others) webhook performance. Builds on Codeship might not be triggered or delayed. Please see https://twitter.com/githubstatus/status/789433336083001344 for more information.

Codeship
Investigating degraded Github webhook issue

Oct 20, 22:25 UTC Resolved - Issue has been resolved. All GitHub triggered builds are now running as expected.Oct 20, 21:44 UTC Monitoring - Builds should now begin to build as expected. Some builds may take a few minutes to trigger while our Build jobs queue catches up.Oct 20, 21:31 UTC Investigating - We are currently investigating Github's webhook service issue with Codeship. More info to come.

Codeship
Investigating degraded Github webhook issue

Oct 20, 21:44 UTC Monitoring - Builds should now begin to build as expected. Some builds may take a few minutes to trigger while our Build jobs queue catches up.Oct 20, 21:31 UTC Investigating - We are currently investigating Github's webhook service issue with Codeship. More info to come.

Codeship
Investigating degraded Github webhook issue

Oct 20, 21:31 UTC Investigating - We are currently investigating Github's webhook service issue with Codeship. More info to come.

Codeship
Codeship web and builds temporarily unavailable

Oct 17, 03:17 UTC Resolved - This incident has been resolved.Oct 17, 01:53 UTC Monitoring - The problem has been resolved. Codeship Engineering is monitoring for any continued instability with our upstream provider.Oct 17, 01:50 UTC Investigating - Codeship Engineering is investigating a problem with our upstream cloud provider and have identified an issue interrupting web and build availability

Codeship
Codeship web and builds temporarily unavailable

Oct 17, 01:53 UTC Monitoring - The problem has been resolved. Codeship Engineering is monitoring for any continued instability with our upstream provider.Oct 17, 01:50 UTC Investigating - Codeship Engineering is investigating a problem with our upstream cloud provider and have identified an issue interrupting web and build availability

Codeship
Codeship web and builds temporarily unavailable

Oct 17, 01:50 UTC Investigating - Codeship Engineering is investigating a problem with our upstream cloud provider and have identified an issue interrupting web and build availability

Codeship
Codeship.com and build processing are temporarily unavailable

Oct 15, 12:09 UTC Resolved - This incident has been resolved.Oct 15, 11:41 UTC Monitoring - Codeship.com is available and build processing is returning to normal.Oct 15, 11:36 UTC Identified - Due to an upstream infrastructure outage, app.codeship.com is unavailble. Our engineers are actively working on the problem.

Codeship
Codeship.com and build processing are temporarily unavailable

Oct 15, 11:41 UTC Monitoring - Codeship.com is available and build processing is returning to normal.Oct 15, 11:36 UTC Identified - Due to an upstream infrastructure outage, app.codeship.com is unavailble. Our engineers are actively working on the problem.

Codeship
Codeship.com and build processing are temporarily unavailable

Oct 15, 11:36 UTC Identified - Due to an upstream infrastructure outage, app.codeship.com is unavailble. Our engineers are actively working on the problem.

Codeship
500 Errors on Codeship

Oct 7, 12:41 UTC Resolved - This incident has been resolved.Oct 7, 12:35 UTC Monitoring - A fix has been implemented. Codeship Engineering is monitoring for resolution.Oct 7, 12:33 UTC Identified - We are aware of an issue causing errors in the Codeship application and are working to resolve the issue.

Codeship
500 Errors on Codeship

Oct 7, 12:35 UTC Monitoring - A fix has been implemented. Codeship Engineering is monitoring for resolution.Oct 7, 12:33 UTC Identified - We are aware of an issue causing errors in the Codeship application and are working to resolve the issue.

Codeship
500 Errors on Codeship

Oct 7, 12:33 UTC Identified - We are aware of an issue causing errors in the Codeship application and are working to resolve the issue.

Codeship
some builds are experiencing delays

Oct 6, 20:42 UTC Resolved - This issue has been resolved. Codeship is now operating as normally expected.Oct 6, 20:06 UTC Monitoring - We have successfully deployed a fix. Builds should now operate as expected. We will continue to monitor for any other delay instances.Oct 6, 19:52 UTC Identified - We have identified the issue and are working on implementing a fix.Oct 6, 19:24 UTC Update - Our engineering team is investigating an issue causing some builds being delayed and instances of degraded performance on the web application.Oct 6, 17:42 UTC Investigating - We are currently investigating the cause for some builds being delayed. We will provide an update as soon as more information is available.

Codeship
some builds are experiencing delays

Oct 6, 20:06 UTC Monitoring - We have successfully deployed a fix. Builds should now operate as expected. We will continue to monitor for any other delay instances.Oct 6, 19:52 UTC Identified - We have identified the issue and are working on implementing a fix.Oct 6, 19:24 UTC Update - Our engineering team is investigating an issue causing some builds being delayed and instances of degraded performance on the web application.Oct 6, 17:42 UTC Investigating - We are currently investigating the cause for some builds being delayed. We will provide an update as soon as more information is available.

Codeship
some builds are experiencing delays

Oct 6, 19:52 UTC Identified - We have identified the issue and are working on implementing a fix.Oct 6, 19:24 UTC Update - Our engineering team is investigating an issue causing some builds being delayed and instances of degraded performance on the web application.Oct 6, 17:42 UTC Investigating - We are currently investigating the cause for some builds being delayed. We will provide an update as soon as more information is available.

Codeship
some builds are experiencing delays

Oct 6, 19:24 UTC Update - Our engineering team is investigating an issue causing some builds being delayed and instances of degraded performance on the web application.Oct 6, 17:42 UTC Investigating - We are currently investigating the cause for some builds being delayed. We will provide an update as soon as more information is available.

Codeship
some builds are experiencing delays

Oct 6, 17:42 UTC Investigating - We are currently investigating the cause for some builds being delayed. We will provide an update as soon as more information is available.

Codeship
some builds are experiencing delays

Oct 6, 17:42 UTC Investigating - We are currently investigating the cause for some builds being delayed. We will provide an update as soon as more information is available.

Codeship
Codeship Outage

Sep 23, 02:24 UTC Resolved - This incident has been resolved.Sep 23, 01:22 UTC Update - Codeship application performance has returned to normal range. Engineering continues to monitor.Sep 23, 00:23 UTC Monitoring - We are beginning to see our database load slowly decrease. We will continue monitoring the processess.Sep 22, 22:55 UTC Update - Codeship is still experiencing a major outage and undergoing emergency maintenance. We will provide an update as soon as possibleSep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 23, 01:22 UTC Update - Codeship application performance has returned to normal range. Engineering continues to monitor.Sep 23, 00:23 UTC Monitoring - We are beginning to see our database load slowly decrease. We will continue monitoring the processess.Sep 22, 22:55 UTC Update - Codeship is still experiencing a major outage and undergoing emergency maintenance. We will provide an update as soon as possibleSep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 23, 01:22 UTC Update - Codeship application performance has returned to normal range. Engineering continues to monitor.Sep 23, 00:23 UTC Monitoring - We are beginning to see our database load slowly decrease. We will continue monitoring the processess.Sep 22, 22:55 UTC Update - Codeship is still experiencing a major outage and undergoing emergency maintenance. We will provide an update as soon as possibleSep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 23, 00:23 UTC Monitoring - We are beginning to see our database load slowly decrease. We will continue monitoring the processess.Sep 22, 22:55 UTC Update - Codeship is still experiencing a major outage and undergoing emergency maintenance. We will provide an update as soon as possibleSep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 23, 00:23 UTC Monitoring - We are beginning to see our database load slowly decrease. We will continue monitoring the processess.Sep 22, 22:55 UTC Update - Codeship is still experiencing a major outage and undergoing emergency maintenance. We will provide an update as soon as possibleSep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 22, 22:55 UTC Update - Codeship is still experiencing a major outage and undergoing emergency maintenance. We will provide an update as soon as possibleSep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 22, 22:55 UTC Update - Codeship is undergoing emergency maintenance. We will provide an update as soon as possibleSep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 22, 22:19 UTC Update - Maintenance has completed. We are still experiencing a partial outage which may delay triggering builds and experience slow build times.Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 22, 21:46 UTC Identified - We have identified the issue and will need to perform emergency maintenance on our database.Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Codeship Outage

Sep 22, 21:40 UTC Investigating - Our engineers are currently investigating the cause of this outage. We will provide an update as new information is discovered. We apologize for the inconvenience.

Codeship
Intermittent errors on the Docker platform

Sep 9, 11:52 UTC Resolved - This incident has been resolved.Sep 8, 14:54 UTC Monitoring - Docker Platform - The affected systems have recovered. We're monitoring metrics closely.Sep 8, 11:43 UTC Investigating - Docker Platform - We're currently looking into intermittent issues with Docker based builds.

Codeship
Intermittent errors on the Docker platform

Sep 8, 14:54 UTC Monitoring - Docker Platform - The affected systems have recovered. We're monitoring metrics closely.Sep 8, 11:43 UTC Investigating - Docker Platform - We're currently looking into intermittent issues with Docker based builds.

Codeship
Intermittent errors on the Docker platform

Sep 8, 11:43 UTC Investigating - Docker Platform - We're currently looking into intermittent issues with Docker based builds.

Codeship
Intermittent errors on the Docker platform

Sep 8, 11:43 UTC Investigating - Docker Platform - We're currently looking into intermittent issues with Docker based builds.

Codeship
Delays in builds triggering on Jet platform

Sep 8, 07:49 UTC Resolved - Codeship Jet - the incident is now resolved.Sep 8, 06:25 UTC Update - Codeship Jet - we're further looking into the issue. Builds are not impacted by the issue.Sep 8, 05:11 UTC Investigating - Codeship's Jet platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating.

Codeship
Delays in builds triggering on Jet platform

Sep 8, 06:25 UTC Update - Codeship Jet - we're further looking into the issue. Builds are not impacted by the issue.Sep 8, 05:11 UTC Investigating - Codeship's Jet platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating.

Codeship
Delays in builds triggering on Jet platform

Sep 8, 06:25 UTC Update - Codeship Jet - we're further looking into the issue. Builds are not impacted by the issue.Sep 8, 05:11 UTC Investigating - Codeship's Jet platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating.

Codeship
Delays in builds triggering on Jet platform

Sep 8, 05:11 UTC Investigating - Codeship's Jet platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating.

Codeship
DNS Updates

Sep 2, 01:42 UTC Completed - The scheduled maintenance has been completed.Sep 2, 01:39 UTC Verifying - Verification is currently underway for the maintenance items.Sep 2, 01:31 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Aug 31, 19:54 UTC Scheduled - We need to perform DNS updates for codeship.com The maintenance will affect all Codeship services. No service interruption is expected.

Codeship
DNS Updates

Sep 2, 01:39 UTC Verifying - Verification is currently underway for the maintenance items.Sep 2, 01:31 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Aug 31, 19:54 UTC Scheduled - We need to perform DNS updates for codeship.com The maintenance will affect all Codeship services. No service interruption is expected.

Codeship
DNS Updates

Sep 2, 01:31 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Aug 31, 19:54 UTC Scheduled - We need to perform DNS updates for codeship.com The maintenance will affect all Codeship services. No service interruption is expected.

Codeship
DNS Updates

Aug 31, 19:54 UTC Scheduled - We need to perform DNS updates for codeship.com The maintenance will affect all Codeship services. No service interruption is expected.

Codeship
codeship.com not reachable

Sep 1, 14:18 UTC Resolved - codeship.com is fully operational again and the database issues are resolved.Sep 1, 13:01 UTC Monitoring - Database load is at normal levels again. Our engineering team is monitoring the systems and we'll keep you updated.Sep 1, 10:11 UTC Investigating - We're seeing database connection errors again. We're looking into the root cause.Sep 1, 09:43 UTC Monitoring - DB connections are stable again, we're monitoring performanceSep 1, 09:39 UTC Identified - We tracked the error to an issue with our database servers. We're currently working on fixing those.Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
codeship.com not reachable

Sep 1, 13:01 UTC Monitoring - Database load is at normal levels again. Our engineering team is monitoring the systems and we'll keep you updated.Sep 1, 10:11 UTC Investigating - We're seeing database connection errors again. We're looking into the root cause.Sep 1, 09:43 UTC Monitoring - DB connections are stable again, we're monitoring performanceSep 1, 09:39 UTC Identified - We tracked the error to an issue with our database servers. We're currently working on fixing those.Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
codeship.com not reachable

Sep 1, 10:11 UTC Investigating - We're seeing database connection errors again. We're looking into the root cause.Sep 1, 09:43 UTC Monitoring - DB connections are stable again, we're monitoring performanceSep 1, 09:39 UTC Identified - We tracked the error to an issue with our database servers. We're currently working on fixing those.Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
codeship.com not reachable

Sep 1, 09:43 UTC Monitoring - DB connections are stable again, we're monitoring performanceSep 1, 09:39 UTC Identified - We tracked the error to an issue with our database servers. We're currently working on fixing those.Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
codeship.com not reachable

Sep 1, 09:43 UTC Monitoring - DB connections are stable again, we're monitoring performanceSep 1, 09:39 UTC Identified - We tracked the error to an issue with our database servers. We're currently working on fixing those.Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
codeship.com not reachable

Sep 1, 09:39 UTC Identified - We tracked the error to an issue with our database servers. We're currently working on fixing those.Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
codeship.com not reachable

Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
codeship.com not reachable

Sep 1, 09:33 UTC Investigating - We're currently looking into an issue with connectivity for codeship.com

Codeship
DNS and Loadbalancer Updates

Sep 1, 08:55 UTC Completed - The scheduled maintenance has been completed.Aug 10, 22:45 UTC Scheduled - The maintenance will affect all Codeship services. We will pause build processing for around 30min.

Codeship
Docker platform experiencing delayed builds

Aug 25, 16:56 UTC Resolved - This incident has been resolved.Aug 25, 16:21 UTC Monitoring - Builds have caught up. We're monitoring the build queues and keep an eye on things.Aug 25, 16:08 UTC Investigating - Codeship's Docker-based platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating to mitigate the issue.

Codeship
Docker platform experiencing delayed builds

Aug 25, 16:21 UTC Monitoring - Builds have caught up. We're monitoring the build queues and keep an eye on things.Aug 25, 16:08 UTC Investigating - Codeship's Docker-based platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating to mitigate the issue.

Codeship
Docker platform experiencing delayed builds

Aug 25, 16:08 UTC Investigating - Codeship's Docker-based platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating to mitigate the issue.

Codeship
Docker platform experiencing delayed builds

Aug 25, 16:08 UTC Investigating - Codeship's Docker-based platform is current experiencing an issue which may be creating intermittent delays in new builds triggering. We are currently investigating to mitigate the issue.

Codeship
Emergency Database Maintenance - Jet build processing paused

Aug 19, 02:38 UTC Completed - Scheduled maintenance was completed successfullyAug 19, 02:01 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Aug 18, 17:57 UTC Scheduled - Codeship will be performing routine maintenance to ensure the ongoing health and security of our database. During this timeframe, processing of Jet (Docker Infrastructure) builds will be paused. Builds will resume running when the maintenance window concludes.

Codeship
Emergency Database Maintenance - Jet build processing paused

Aug 19, 02:01 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Aug 18, 17:57 UTC Scheduled - Codeship will be performing routine maintenance to ensure the ongoing health and security of our database. During this timeframe, processing of Jet (Docker Infrastructure) builds will be paused. Builds will resume running when the maintenance window concludes.

Codeship
Emergency Database Maintenance - Jet build processing paused

Aug 18, 17:57 UTC Scheduled - Codeship will be performing routine maintenance to ensure the ongoing health and security of our database. During this timeframe, processing of Jet (Docker Infrastructure) builds will be paused. Builds will resume running when the maintenance window concludes.

Codeship
DNS and Loadbalancer Updates

Aug 10, 22:45 UTC Scheduled - The maintenance will affect all Codeship services. We will pause build processing for around 30min.

Codeship
DNS and Loadbalancer Updates

Aug 9, 00:47 UTC Scheduled - The maintenance will affect all Codeship services. We will pause build processing during the maintenance window.

Codeship
Delayed Build Updates

Aug 9, 19:42 UTC Resolved - This incident has been resolved.Aug 9, 17:59 UTC Investigating - We are seeing delayed build log updates.

Codeship
Delayed Build Updates

Aug 9, 17:59 UTC Investigating - We are seeing delayed build log updates.

Codeship
Builds not completing on Jet (Docker Platform)

Aug 9, 11:22 UTC Resolved - This incident has been resolved.Aug 9, 11:01 UTC Monitoring - We have identified a problem and deployed a fix. We will continue to monitor the Jet build platform.Aug 9, 10:40 UTC Investigating - Some users have reported an intermittent failure where a step completes successfully, but the step is not marked as completed. Therefore, the build will not continue. We are investigating this issue.

Codeship
Builds not completing on Jet (Docker Platform)

Aug 9, 11:01 UTC Monitoring - We have identified a problem and deployed a fix. We will continue to monitor the Jet build platform.Aug 9, 10:40 UTC Investigating - Some users have reported an intermittent failure where a step completes successfully, but the step is not marked as completed. Therefore, the build will not continue. We are investigating this issue.

Codeship
Builds not completing on Jet (Docker Platform)

Aug 9, 10:40 UTC Investigating - Some users have reported an intermittent failure where a step completes successfully, but the step is not marked as completed. Therefore, the build will not continue. We are investigating this issue.

Codeship
Builds not completing on Jet (Docker Platform)

Aug 9, 10:40 UTC Investigating - Some users have reported an intermittent failure where a step completes successfully, but the step is not marked as completed. Therefore, the build will not continue. We are investigating this issue.

Codeship
DNS and Loadbalancer Updates

Aug 9, 00:47 UTC Scheduled - The maintenance will affect all Codeship services. We will pause build processing during the maintenance window.

Codeship
Bitbucket Builds Not Triggering

Aug 6, 17:29 UTC Resolved - Bitbucket's API is now fully operational and all builds are triggering as normal.Aug 6, 16:11 UTC Identified - The BitBucket status API is experiencing issues.Aug 6, 15:10 UTC Investigating - We are aware that some Bitbucket-based projects are not triggering new builds and we are investigating currently.

Codeship
Bitbucket Builds Not Triggering

Aug 6, 16:11 UTC Identified - The BitBucket status API is experiencing issues.Aug 6, 15:10 UTC Investigating - We are aware that some Bitbucket-based projects are not triggering new builds and we are investigating currently.

Codeship
Bitbucket Builds Not Triggering

Aug 6, 15:10 UTC Investigating - We are aware that some Bitbucket-based projects are not triggering new builds and we are investigating currently.

Codeship
Bitbucket Builds Not Triggering

Aug 6, 15:10 UTC Investigating - We are aware that some Bitbucket-based projects are not triggering new builds and we are investigating currently.

Codeship
Intermittent api failures on Jet, Docker platform.

Aug 4, 22:16 UTC Resolved - Incident has been resolved. All builds are running as normally expected.Aug 4, 20:43 UTC Monitoring - We identified and pushed a fix for intermitted API failures for Jet, Docker platform. We will continue to monitor the status of this issue. Classic infrastructure users are not affected.

Codeship
Intermittent api failures on Jet, Docker platform.

Aug 4, 20:43 UTC Monitoring - We identified and pushed a fix for intermitted API failures for Jet, Docker platform. We will continue to monitor the status of this issue. Classic infrastructure users are not affected.

Codeship
Codeship application currently unavailable

Jul 26, 02:10 UTC Resolved - This incident has been resolved.Jul 26, 01:34 UTC Monitoring - Service has been restored and the database consistency is being monitoredJul 26, 01:27 UTC Investigating - Codeship engineering is currently investigating a database issue causing system availability issues

Codeship
Codeship application currently unavailable

Jul 26, 01:34 UTC Monitoring - Service has been restored and the database consistency is being monitoredJul 26, 01:27 UTC Investigating - Codeship engineering is currently investigating a database issue causing system availability issues

Codeship
Codeship application currently unavailable

Jul 26, 01:27 UTC Investigating - Codeship engineering is currently investigating a database issue causing system availability issues

Codeship
Codeship application currently unavailable

Jul 26, 01:27 UTC Investigating - Codeship engineering is currently investigating a database issue causing system availability issues

Codeship
Build impact from current AWS outage

Jul 19, 22:22 UTC Resolved - Build allocation issue has been resolved. All builds running on the Jet platform should run as normally expected.Jul 19, 22:05 UTC Investigating - We are currently investigating build allocation issues affecting projects on the Docker platform.Jul 19, 19:19 UTC Monitoring - AWS is currently experiencing an outage with their API service (see http://status.aws.amazon.com/ for more info.) This may cause builds to be stuck allocating, ignored, or run into a system error. We will continue to monitor any progress released from AWS regarding this issue.

Codeship
Build impact from current AWS outage

Jul 19, 22:05 UTC Investigating - We are currently investigating build allocation issues affecting projects on the Docker platform.Jul 19, 19:19 UTC Monitoring - AWS is currently experiencing an outage with their API service (see http://status.aws.amazon.com/ for more info.) This may cause builds to be stuck allocating, ignored, or run into a system error. We will continue to monitor any progress released from AWS regarding this issue.

Codeship
Build impact from current AWS outage

Jul 19, 19:19 UTC Monitoring - AWS is currently experiencing an outage with their API service (see http://status.aws.amazon.com/ for more info.) This may cause builds to be stuck allocating, ignored, or run into a system error. We will continue to monitor any progress released from AWS regarding this issue.

Codeship
npm installs failing with 502

Jul 6, 17:19 UTC Resolved - This incident has been resolved.Jul 6, 14:20 UTC Update - Some users are receiving 502 errors during npm install due to an issue with npm. If you use npm, you may experience degraded performance. NPM is aware of the issue and working toward a fix. They are tracking this incident on [their status site] http://status.npmjs.org/incidents/nl2mpgltpnvjJul 6, 10:06 UTC Monitoring - Some users are receiving 502 errors during npm install due to an issue with npm. If you use npm, you may experience degraded performance. There are several issues open on GitHub to address the issue, and we recommend following the issues while npm works to resolve it. https://github.com/npm/npm/issues/13284

Codeship
npm installs failing with 502

Jul 6, 14:20 UTC Update - Some users are receiving 502 errors during npm install due to an issue with npm. If you use npm, you may experience degraded performance. NPM is aware of the issue and working toward a fix. They are tracking this incident on [their status site] http://status.npmjs.org/incidents/nl2mpgltpnvjJul 6, 10:06 UTC Monitoring - Some users are receiving 502 errors during npm install due to an issue with npm. If you use npm, you may experience degraded performance. There are several issues open on GitHub to address the issue, and we recommend following the issues while npm works to resolve it. https://github.com/npm/npm/issues/13284

Codeship
npm installs failing with 502

Jul 6, 10:06 UTC Monitoring - Some users are receiving 502 errors during npm install due to an issue with npm. If you use npm, you may experience degraded performance. There are several issues open on GitHub to address the issue, and we recommend following the issues while npm works to resolve it. https://github.com/npm/npm/issues/13284

Codeship
npm installs failing with 502

Jul 6, 10:06 UTC Monitoring - Some users are receiving 502 errors during npm install due to an issue with npm. If you use npm, you may experience degraded performance. There are several issues open on GitHub to address the issue, and we recommend following the issues while npm works to resolve it. https://github.com/npm/npm/issues/13284

Codeship
Issues Creating New Bitbucket Projects

Jun 29, 21:43 UTC Resolved - This issue has been resolved. Bitbucket users can now create new projects for their repositories.Jun 29, 20:21 UTC Identified - Bitbucket's current authentication issue is currently preventing Codeship from connecting to any BitBucket repository. Please visit http://status.bitbucket.org/ for status updates.Jun 29, 19:54 UTC Investigating - We are currently investigating an issue with the Bitbucket API that may cause problems for some Bitbucket users creating new projects.

Codeship
Issues Creating New Bitbucket Projects

Jun 29, 20:21 UTC Identified - Bitbucket's current authentication issue is currently preventing Codeship from connecting to any BitBucket repository. Please visit http://status.bitbucket.org/ for status updates.Jun 29, 19:54 UTC Investigating - We are currently investigating an issue with the Bitbucket API that may cause problems for some Bitbucket users creating new projects.

Codeship
Issues Creating New Bitbucket Projects

Jun 29, 20:21 UTC Identified - Bitbucket's current authentication issue is currently preventing Codeship from connecting to any BitBucket repository. Please visit http://status.bitbucket.org/ for status updates.Jun 29, 19:54 UTC Investigating - We are currently investigating an issue with the Bitbucket API that may cause problems for some Bitbucket users creating new projects.

Codeship
Issues Creating New Bitbucket Projects

Jun 29, 19:54 UTC Investigating - We are currently investigating an issue with the Bitbucket API that may cause problems for some Bitbucket users creating new projects.

Codeship
Issues Creating New Bitbucket Projects

Jun 29, 19:54 UTC Investigating - We are currently investigating an issue with the Bitbucket API that may cause problems for some Bitbucket users creating new projects.

Codeship
Latency or errors with new builds

Jun 9, 21:06 UTC Resolved - This incident has been resolved.Jun 9, 20:46 UTC Monitoring - Our cloud provider is reporting and we are observing some recovery in service level. We will continue to monitor and watch for updated reports on the situation.Jun 9, 19:49 UTC Investigating - Codeship's cloud provider is reporting increased error rates in creating new compute instances which may cause problems in scaling our infrastructure to meet demand resulting in latency or errors for new builds. We are following the status here: http://status.aws.amazon.com/

Codeship
Latency or errors with new builds

Jun 9, 20:46 UTC Monitoring - Our cloud provider is reporting and we are observing some recovery in service level. We will continue to monitor and watch for updated reports on the situation.Jun 9, 19:49 UTC Investigating - Codeship's cloud provider is reporting increased error rates in creating new compute instances which may cause problems in scaling our infrastructure to meet demand resulting in latency or errors for new builds. We are following the status here: http://status.aws.amazon.com/

Codeship
Latency or errors with new builds

Jun 9, 19:49 UTC Investigating - Codeship's cloud provider is reporting increased error rates in creating new compute instances which may cause problems in scaling our infrastructure to meet demand resulting in latency or errors for new builds. We are following the status here: http://status.aws.amazon.com/

Codeship
Hanging builds

Jun 6, 15:33 UTC Resolved - All builds are now running as expected.Jun 6, 14:43 UTC Identified - We have identified an issue that is causing some builds to hang. We are currently implementing a fix to the problem. We apologize for the inconvenience.

Codeship
Hanging builds

Jun 6, 14:43 UTC Identified - We have identified an issue that is causing some builds to hang. We are currently implementing a fix to the problem. We apologize for the inconvenience.

Codeship
Issues with Application Performance

Jun 2, 13:04 UTC Resolved - The Heroku incident is resolved and our metrics are back to their usual levels as well.Jun 2, 12:15 UTC Update - Heroku (our hosting provider) is currently investigating an issue with their network, see https://status.heroku.com/incidents/900 for more information. We're monitoring our application status and will update this incident with any new information we get.Jun 2, 12:10 UTC Monitoring - Internal metrics show performance issue is resolved again. We're monitoring performance closely.Jun 2, 12:03 UTC Investigating - We're currently looking into issues with the performance of our application.

Codeship
Issues with Application Performance

Jun 2, 12:15 UTC Update - Heroku (our hosting provider) is currently investigating an issue with their network, see https://status.heroku.com/incidents/900 for more information. We're monitoring our application status and will update this incident with any new information we get.Jun 2, 12:10 UTC Monitoring - Internal metrics show performance issue is resolved again. We're monitoring performance closely.Jun 2, 12:03 UTC Investigating - We're currently looking into issues with the performance of our application.

Codeship
Issues with Application Performance

Jun 2, 12:10 UTC Monitoring - Internal metrics show performance issue is resolved again. We're monitoring performance closely.Jun 2, 12:03 UTC Investigating - We're currently looking into issues with the performance of our application.

Codeship
Issues with Application Performance

Jun 2, 12:03 UTC Investigating - We're currently looking into issues with the performance of our application.

Codeship
Docker Platform, degraded build logs

May 30, 15:30 UTC Resolved - This incident has been resolved.May 30, 15:03 UTC Monitoring - We have identified the issue and taken first steps to resolve it. We're currently working through a backlog of build logs and monitoring performance.May 30, 14:53 UTC Investigating - We're seeing issues with build logs on our Docker platform. We're currently looking into this and will provide an update shortly.

Codeship
Docker Platform, degraded build logs

May 30, 15:03 UTC Monitoring - We have identified the issue and taken first steps to resolve it. We're currently working through a backlog of build logs and monitoring performance.May 30, 14:53 UTC Investigating - We're seeing issues with build logs on our Docker platform. We're currently looking into this and will provide an update shortly.

Codeship
Docker Platform, degraded build logs

May 30, 14:53 UTC Investigating - We're seeing issues with build logs on our Docker platform. We're currently looking into this and will provide an update shortly.

Codeship
Database Maintenance

May 28, 14:47 UTC Completed - The scheduled maintenance has been completed.May 28, 14:40 UTC Verifying - Verification is currently underway for the maintenance items.May 28, 13:54 UTC In progress - the maintenance will require some additional time, everything is running smoothly but taking longer than expected.May 27, 14:18 UTC Scheduled - The Codeship platform will be unavailable as we perform necessary maintenance on our database servers from 8AM - 10AM EDT on Saturday, May 28, 2016.

Codeship
Database Maintenance

May 28, 14:40 UTC Verifying - Verification is currently underway for the maintenance items.May 28, 13:54 UTC In progress - the maintenance will require some additional time, everything is running smoothly but taking longer than expected.May 27, 14:18 UTC Scheduled - The Codeship platform will be unavailable as we perform necessary maintenance on our database servers from 8AM - 10AM EDT on Saturday, May 28, 2016.

Codeship
Database Maintenance

May 28, 13:54 UTC In progress - the maintenance will require some additional time, everything is running smoothly but taking longer than expected.May 27, 14:18 UTC Scheduled - The Codeship platform will be unavailable as we perform necessary maintenance on our database servers from 8AM - 10AM EDT on Saturday, May 28, 2016.

Codeship
Database Maintenance - Jet Platform

May 27, 14:18 UTC Scheduled - The Codeship Jet platform will be unavailable as we perform necessary maintenance on our database servers from 8AM - 10AM EDT on Saturday, May 28, 2016.

Codeship
Database Maintenance - Jet Platform

May 27, 08:48 UTC Resolved - This incident has been resolved.May 27, 08:08 UTC Identified - The Codeship Jet platform will be unavailable as we perform necessary maintenance on our database servers

Codeship
Database Maintenance - Jet Platform

May 27, 08:08 UTC Identified - The Codeship Jet platform will be unavailable as we perform necessary maintenance on our database servers

Codeship
Builds on Jet / Docker infrastructure failing with database error

May 23, 18:45 UTC Resolved - This incident has been resolved.

Codeship
Builds on Jet - Docker Infrastructure not running

Apr 28, 20:28 UTC Resolved - At approximately 4:00PM EST, we discovered intermitted issues with prevented some builds on our Docker infrastructure from being triggered. All builds are now running as normally expected.

Codeship
Builds on Jet / Docker infrastructure failing with volume errors

Apr 1, 15:34 UTC Resolved - This incident has been resolved.Apr 1, 15:30 UTC Monitoring - An update has been deployed to resolve the problems with Docker volumes. Codeship Engineering is monitoring the situation to confirm the issues are fixed.Apr 1, 14:39 UTC Identified - The issue is still not resolved. Codeship Engineering has identified the problem and is working to resolve.Apr 1, 14:22 UTC Monitoring - A fix had been deployed to resolve the problems with Docker volumes. Codeship Engineering is monitoring the situation to confirm the issues are resolved.Apr 1, 14:02 UTC Identified - Codeship Engineering has identified the issue causing build failures and is working to deploy a fixApr 1, 13:47 UTC Investigating - We are investigating reports of builds running on our Jet / Docker infrastructure failing with volume errors

Codeship
Builds on Jet / Docker infrastructure failing with volume errors

Apr 1, 15:30 UTC Monitoring - An update has been deployed to resolve the problems with Docker volumes. Codeship Engineering is monitoring the situation to confirm the issues are fixed.Apr 1, 14:39 UTC Identified - The issue is still not resolved. Codeship Engineering has identified the problem and is working to resolve.Apr 1, 14:22 UTC Monitoring - A fix had been deployed to resolve the problems with Docker volumes. Codeship Engineering is monitoring the situation to confirm the issues are resolved.Apr 1, 14:02 UTC Identified - Codeship Engineering has identified the issue causing build failures and is working to deploy a fixApr 1, 13:47 UTC Investigating - We are investigating reports of builds running on our Jet / Docker infrastructure failing with volume errors

Codeship
Builds on Jet / Docker infrastructure failing with volume errors

Apr 1, 14:39 UTC Identified - The issue is still not resolved. Codeship Engineering has identified the problem and is working to resolve.Apr 1, 14:22 UTC Monitoring - A fix had been deployed to resolve the problems with Docker volumes. Codeship Engineering is monitoring the situation to confirm the issues are resolved.Apr 1, 14:02 UTC Identified - Codeship Engineering has identified the issue causing build failures and is working to deploy a fixApr 1, 13:47 UTC Investigating - We are investigating reports of builds running on our Jet / Docker infrastructure failing with volume errors

Codeship
Builds on Jet / Docker infrastructure failing with volume errors

Apr 1, 14:22 UTC Monitoring - A fix had been deployed to resolve the problems with Docker volumes. Codeship Engineering is monitoring the situation to confirm the issues are resolved.Apr 1, 14:02 UTC Identified - Codeship Engineering has identified the issue causing build failures and is working to deploy a fixApr 1, 13:47 UTC Investigating - We are investigating reports of builds running on our Jet / Docker infrastructure failing with volume errors

Codeship
Builds on Jet / Docker infrastructure failing with volume errors

Apr 1, 14:02 UTC Identified - Codeship Engineering has identified the issue causing build failures and is working to deploy a fixApr 1, 13:47 UTC Investigating - We are investigating reports of builds running on our Jet / Docker infrastructure failing with volume errors

Codeship
Builds on Jet / Docker infrastructure failing with volume errors

Apr 1, 13:47 UTC Investigating - We are investigating reports of builds running on our Jet / Docker infrastructure failing with volume errors

Codeship
Errors with builds running on Jet / Docker Infrastructure

Mar 29, 18:10 UTC Resolved - This incident has been resolved.Mar 29, 18:08 UTC Investigating - Engineering is currently investigating an issue with builds running on Jet / Docker Infrastructure

Codeship
Errors with builds running on Jet / Docker Infrastructure

Mar 29, 18:08 UTC Investigating - Engineering is currently investigating an issue with builds running on Jet / Docker Infrastructure

Codeship
Builds not triggered

Mar 28, 19:13 UTC Resolved - The bulk of backlogged builds is either currently running or already finished. New builds are processes as they come in. There might be a couple more builds not being triggered yet and waiting on GitHub to re-submit the webhook. We'll look into the issue in more detail with our engineering team and make changes to our monitoring process to make sure we get notified earlier. Sorry for the inconvenience.Mar 28, 18:50 UTC Monitoring - An issue with an updated certificate caused problems for GitHub based webhooks. We switched back to the old certificate (for now, there is still plenty of time before it expires) and GitHub webhooks are triggering builds again. It might take a bit for webhooks to catch up though.Mar 28, 18:42 UTC Investigating - We're looking into an issue with builds not being triggered for some projects at the moment.

Codeship
Builds not triggered

Mar 28, 18:50 UTC Monitoring - An issue with an updated certificate caused problems for GitHub based webhooks. We switched back to the old certificate (for now, there is still plenty of time before it expires) and GitHub webhooks are triggering builds again. It might take a bit for webhooks to catch up though.Mar 28, 18:42 UTC Investigating - We're looking into an issue with builds not being triggered for some projects at the moment.

Codeship
Builds not triggered

Mar 28, 18:42 UTC Investigating - We're looking into an issue with builds not being triggered for some projects at the moment.

Codeship
Heroku Deployments may fail

Mar 15, 19:35 UTC Resolved - This incident has been resolved.Mar 15, 18:58 UTC Monitoring - Some deployments to Heroku are failing due to an issue with the 'heroku run' command. Please see https://status.heroku.com/incidents/868 for updates.

Codeship
Docker Caching Broken

Mar 8, 21:57 UTC Resolved - We reverted back to 1.9.1 to restore cache functionality. We'll investigate updating to Docker 1.10 at a later time. See https://community.codeship.com/t/docker-caching-broken-on-our-docker-based-infrastructure/62 for more information.Mar 8, 14:12 UTC Identified - A change to the way Docker addresses content in the latest release (1.10.x) broke the caching mechanism we implemented. You will see full rebuilds of the containers on your Docker builds at the moment and an increase in build time because of this. Please see https://github.com/docker/docker/issues/20316 for the original report on GitHub as well as the ongoing discussion. We’re currently looking into alternative ways to provide build cache during container builds.

Codeship
Builds not being triggered

Mar 2, 01:47 UTC Resolved - Builds are now triggering as expected.Mar 2, 00:16 UTC Monitoring - Our engineering team has pushed a change to fix the issue preventing builds from triggering. We will continue to monitor builds to ensure all issues are resolved.Mar 1, 23:37 UTC Identified - Current issue for builds not triggering is only affecting builds on our Docker infrastructure. Our engineering team is investigating the root cause.Mar 1, 23:29 UTC Investigating - We are currently investigating the issue preventing builds to trigger.

Codeship
Heroku deploy problem

Feb 16, 01:16 UTC Resolved - Heroku has resolved the issue with their heroku-cli tool: https://status.heroku.com/incidents/851Feb 16, 00:46 UTC Identified - Heroku has identified a problem with their heroku-cli tool and have pushed an update in attempt to resolve. Status updates available here: http://status.heroku.com/incidents/851Feb 15, 23:53 UTC Investigating - We are currently investigating reports of problems with Heroku deploys caused by a problem with heroku-cli reported here: https://github.com/heroku/heroku-cli/issues/203

Codeship
Degraded log availability on the Docker platform

Jan 25, 18:09 UTC Resolved - This incident has been resolved.Jan 25, 18:09 UTC Update - Issue has been resolved. Build logs on the Docker Infrastructure are now showing as expected.Jan 25, 13:46 UTC Identified - We have identified the issue and are working on a solution.Jan 25, 13:12 UTC Investigating - Some users are experiencing problems viewing logs for builds on the Docker infrastructure. Our engineering team is investigating the problem.

Codeship
Docker Infrastructure: Intermittent Build Failures

Jan 14, 20:47 UTC Resolved - The issue with resource limitation was resolved with our cloud provider and builds are running normally at this timeJan 14, 17:49 UTC Identified - Engineering has identified the issue with resource limits in place from our cloud provider. We are working to resolve with high priority. Only a small number of builds are affected at this time and can be retried.Jan 14, 16:31 UTC Investigating - Our monitoring is showing intermittent build failures for projects running on the Docker based infrastructure. Engineering is investigating.

Codeship
Builds stuck or delayed

Jan 12, 17:31 UTC Resolved - Builds are processing normallyJan 12, 17:11 UTC Investigating - We are currently investigating a problem with some builds stalled or delayed. We are investigating and will post an update shortly.

Codeship
Database Maintenance

Jan 1, 15:20 UTC Resolved - This incident has been resolved.Jan 1, 14:35 UTC Investigating - We will run a database maintenance.

Codeship
Bitbucket based builds not triggered

Dec 15, 15:57 UTC Resolved - BitBucket's builds are now running as normally expected.Dec 15, 15:16 UTC Monitoring - Bitbucket reports the issue is resolved on their end. We'll keep monitoring incoming hooks and update this story.Dec 15, 13:16 UTC Identified - Bitbucket is investigating an issue with dealys for the webhooks. Please see http://status.bitbucket.org/incidents/d1y55cg8kwgq for details.Dec 15, 12:11 UTC Investigating - We're currently looking into the reason for Bitbucket based projects not triggering builds.

Codeship
Delayed builds for Bitbucket based projects

Dec 3, 10:45 UTC Resolved - The issue as since been resolved by Atlassian. All builds are again triggered in a timely manner.Dec 2, 17:23 UTC Monitoring - BitBucket users may experience severe delays with triggering builds. BitBucket is currently having a major outage with their HTTP and webhook services. Please see http://status.bitbucket.org/incidents/g0qdhstfyw13 for real time updates of BitBucket's progress.Dec 2, 13:29 UTC Investigating - We're looking into an issue with delayed builds for Bitbucket based projects.

Codeship
Some build pages are showing errors

Nov 20, 14:21 UTC Resolved - This incident has been resolved.Nov 20, 11:40 UTC Monitoring - Our cloud provider identified networking issues on their end that were the root cause of the identified issue.Nov 20, 10:06 UTC Identified - Our cloud provider identified networking issues on their end that were the root cause of the identified issue.Nov 20, 09:40 UTC Investigating - Our cloud provider identified networking issues on their end that were the root cause of the identified issue.Nov 20, 02:54 UTC Identified - We have identified an intermittent communication issue between some of our systems. This is causing intermittent errors on some build pages, as well as affecting some build notifications.Nov 20, 02:00 UTC Investigating - We are investigating some communication errors between our services. This is causing intermittent errors when viewing the build status and logs of some projects, but is not affecting builds themselves.

Codeship
Degraded Webhook Performance

Nov 17, 17:59 UTC Resolved - This incident has been resolved.Nov 17, 14:55 UTC Monitoring - Github webook performance has returned to normal, we will continue to monitor webhook performance.Nov 17, 14:55 UTC Identified - We have identified a performance issue with Github webhooks. Builds are starting, but delayed.Nov 17, 14:46 UTC Investigating - We are investigating degraded webhook performance for new builds

Codeship
Bitbucket builds not triggering

Oct 30, 19:34 UTC Resolved - This incident has been resolved.Oct 30, 17:50 UTC Update - Bitbucket marked the issue as resolved, we'll keep monitoring incoming hooks for a little while longer.Oct 30, 14:51 UTC Monitoring - Bitbucket solved the underlying issue and are working through the backlog of hooks. We're starting additional build instances to deal with the increased load. Builds might be in waiting state a little while longer than usual.Oct 30, 14:00 UTC Update - Bitbucket acknowledged the webhook issue on their end and are looking into it. http://status.bitbucket.org/incidents/2h4pd9rrhvxvOct 30, 13:27 UTC Identified - We ruled out an issue on our end and are getting in touch with the team at Bitbucket.Oct 30, 13:05 UTC Investigating - We currently don't see any incoming Bitbucket webhooks, we're looking into this and will update with any new information we receive.

Codeship
Delayed builds for Bitbucket based projects

Oct 27, 01:02 UTC Resolved - BitButcket builds are now being triggered as expected.Oct 26, 18:42 UTC Identified - We're currently experiencing build delays for Bitbucket based projects. We're monitoring the status and will keep you updated.

Codeship
Degraded performance for BitBucket users

Oct 23, 17:18 UTC Resolved - BitBucket is fully operational.Oct 23, 14:43 UTC Update - BitBucket builds should now be operating as expected. There may still be some delays due to BitBucket's current network status.Oct 22, 18:34 UTC Update - BitBucket is currently working on fixing physical routing issues that is causing their networking issues. BitBucket builds will remain unstable until this gets resolved. Please see http://status.bitbucket.org/incidents/m3nns204x7rc for updates on their progress.Oct 22, 17:09 UTC Monitoring - BitBucket is currently having an issue with their memcached servers. This may cause for some builds to not trigger. Please visit http://status.bitbucket.org/ for updates.

Codeship
Docker Based Infrastructure - Builds not starting

Oct 22, 17:12 UTC Resolved - This incident has been resolved.Oct 22, 17:06 UTC Monitoring - Quay.io reports that their issue is resolved, and they are monitoring the situation.Oct 22, 17:01 UTC Investigating - A problem with a third-party Docker registry provider (quay.io) is preventing builds from starting on Codeship's Docker-based build infrastructure. Codeship Engineering is investigating and monitoring the situation. status.quay.io has more information.

Codeship
Builds may not trigger due to BitBucket's current network issue

Oct 22, 03:24 UTC Resolved - This incident has been resolved.Oct 21, 15:33 UTC Update - Bitbucket is reporting delays with webhooks, please see http://status.bitbucket.org/incidents/q11zrrgksyb9 for details.Oct 21, 13:45 UTC Monitoring - BitBucket is currently experiencing problems with their network which may impact builds from being triggered. Please visit http://status.bitbucket.org/ for status updates.

Codeship
Intermittent Availability Issues

Oct 20, 20:48 UTC Resolved - This incident has been resolved.Oct 20, 19:55 UTC Monitoring - A fix has been implemented and we are monitoring the results. Availability is fully restored.Oct 20, 19:26 UTC Identified - Availability to the Codeship web application has been restored. Engineering has identified the problem and is working to bring the problem to full resolution.Oct 20, 19:14 UTC Investigating - The Codeship web application is experiencing intermittent availability issues. Engineering is investigating.

Codeship
Docker Hub service disruption

Oct 15, 22:33 UTC Resolved - This incident has been resolved.Oct 15, 21:39 UTC Identified - The Docker Registry Hub API is experiencing a service disruption. See https://status.docker.com/ for details.

Codeship
Bitbucket Outage

Oct 15, 21:57 UTC Resolved - Bitbucket services are now available and are no longer disrupted. We will continue to monitor the situation.Oct 15, 20:52 UTC Identified - Bitbucket services are currently disrupted. Users may be unable to use Bitbucket services and will not see Codeship builds triggered. See http://status.bitbucket.org/incidents/64xvfxtjg9pk for more details.

Codeship
Builds not triggering from Github Repositories

Oct 20, 14:17 UTC Postmortem - We are sincerely sorry for the disruption to our service that occurred on October 14, 2015. We know you rely on Codeship to build, test and ship your software, and we strive for extremely high availability. Below are the details on what went wrong and what we are doing to prevent occurrence in the future. Events **14Oct2015 08:30 UTC **: Codeship Support team in the Vienna, Austria office identified that customer builds were not being triggered by changes to connected repositories. **14Oct2015 10:00 UTC **: Triage revealed payloads from Github and Bitbucket were being received and enqueued but not processed. After attempts to troubleshoot and resolve failed and queues continued to grow, the team escalated to Engineering in Boston. **14Oct2015 11:00 UTC **: Diagnosis showed that the service responsible for dequeuing incoming payloads from Github and Bitbucket lost its database connection and was not able to recover gracefully. Once identified, the worker was restarted allowing the queue of builds to process. **14Oct2015 11:33 UTC **: Queued builds were completely drained and service level returned to normal. ##What went wrong? Analysis of logs and exceptions showed that a software defect in the Codeship service that is responsible for taking queued incoming payloads from Github and Bitbucket and creating the jobs that run on the elastic build infrastructure. The defect prevented the service from operating in a resilient manner when there was a momentary, transient database connection failure. The problem took far too long to find and resolve, and a deficiency in monitoring and alerting was identified. The size of the queue was being monitored with proper alerting, however the threshold at which the alert would fire was too high to notify of the problem soon enough. ##What are we changing? In addition to resolving product quality issues around service resiliency, our monitoring and alerting is being audited and improved to better inform on-call staff of issues, when they occur. Issue resolution and triage procedures have been strengthened to more quickly identify and resolve service issues. ##Conclusion At Codeship, we take pride in providing software developers a service they can rely on, and we are very sorry for the inconvenience this incident has caused. We take these incidents very seriously and take a rigid approach to studying and learning from what happened.Oct 14, 12:00 UTC Resolved - Everything is back to normal. Builds are triggering and running again. We are very sorry for any troubles this has caused. Your Codeship CrewOct 14, 11:34 UTC Monitoring - Things are getting back to normal. It's possible that builds take a while to really run after showing up, because of a big queue. We'll keep an eye on this.Oct 14, 11:09 UTC Identified - The issue has been identified and builds are running. There will be some delay as the queue of builds is processed.Oct 14, 10:34 UTC Update - Builds are apparently also not triggering for BitBucket.Oct 14, 09:56 UTC Investigating - We are investigating an issue with GitHub repositories not triggering builds. We will update on our ongoing efforts to mitigate this issue.

Codeship
Builds not triggering from Bitbucket repositories

Oct 9, 18:59 UTC Resolved - Codeship has confirmed that builds triggered from updates to Bitbucket repositories are operating normallyOct 9, 18:50 UTC Update - Atlassian reports that Bitbucket Webhooks are once again operational. Codeship engineering is performing verification tasks to ensure successful operation on our end.Oct 9, 15:39 UTC Monitoring - Bitbucket is reporting degraded performance with their webhooks service, and builds are not being triggered from changes to Bitbucket repositories, as result. Codeship is monitoring the issue and will update as soon as more information is available. http://status.bitbucket.org/incidents/2vj22gg6n2h6

Codeship
Builds not triggering for BitBucket repositories

Oct 9, 04:04 UTC Resolved - Bitbucket Builds are running fine again. Sorry for the inconvenience.Oct 9, 03:33 UTC Monitoring - We have fixed the issue and are monitoring Bitbucket builds.Oct 9, 02:22 UTC Update - We've identified the issue and are working on a fix.Oct 9, 00:58 UTC Identified - We are investigating an issue with BitBucket repositories not triggering builds. We will update on our ongoing efforts to mitigate this issue.

Codeship
Builds not triggering for BitBucket repositories

Oct 8, 21:00 UTC Resolved - Builds are now triggering again for BitBucket repositories.Oct 8, 20:16 UTC Update - BitBucket's webhooks are now back up and operational. Builds may still take some time to trigger as BitBucket's background jobs will need to catch up.Oct 8, 17:45 UTC Monitoring - BitBucket is currently experiencing a major outage with their webhooks. Please visit http://status.bitbucket.org/incidents/sq556ckjq418 for more information and updates.Oct 8, 17:17 UTC Investigating - We are investigating an issue with BitBucket repositories not triggering builds. We will update on our ongoing efforts to mitigate this issue.

Codeship
Database Maintenance for Docker Infrastructre

Sep 24, 15:34 UTC Resolved - The maintenance was finished successfully. Codeship on Docker is sailing fine again!Sep 24, 15:24 UTC Identified - We are performing a database maintenance for our Docker based infrastructure, which can result in errors Codeship.com for about 2 to 3 minutes. Your Docker based builds will not be affected by this, neither will be any projects not using the Docker based platform.