Formatics.
Formatics.
We are receiving reports of the inability to build node applications. Our engineers are investigating for further details.
This issue has been resolved.
Since 08:00 UTC (3:00 AM Eastern), notifications in the Dublin and Virginia regions of Heroku Connect are no longer being delivered. Notifications have not been lost and will be delivered once this issue has been resolved, our engineers are working on a resolution. All Sync operations are continuing as normal.
This issue has been resolved.
We are currently investigating a dip in our routing availability in the EU Common Runtime.
This issue is now resolved
This service update has been completed. All platform functionality has returned to normal.
The scheduled service update is now underway.
This issue has since been resolved. Data services (Heroku Postgres, Kafka and Redis) in the Tokyo region were unavailable between 7:23 - 08:16 UTC and 08:46 - 09:04 UTC.
The incident is resolved.
These issues are now resolved.
This issue has been resolved.
The scheduled maintenance has been completed. All platform functionality has returned to normal.
The scheduled maintenance has been completed. All platform functionality has returned to normal.
We can confirm that this issue affected Pipelines between 09:24 UTC and 10:27 UTC. Also the Heroku Dashboard was unavailable between 09:37 UTC and 10:20 UTC.
This issue is now resolved.
This issue is now resolved.
This incident has been resolved.
This issue is now resolved.
On October 13, 2017, from 15:32 UTC to 16:27 UTC, and on October 17, 2017, from 09:57 UTC to 09:59 UTC, the Platform API experienced a rate limiting failure resulting in nearly complete downtime. Numerous Heroku services dependent on the API were affected. We accept full responsibility for this issue, and apologize for the negative effects our customers experienced.
Who was affected?
During both incidents, a large percentage of Platform API requests failed, returning the message 503 Service Unavailable. Customers were unable to access the Heroku Dashboard, deploy applications, manage add-ons and perform other actions dependent upon the Platform API.
What Happened?
A combination of rate limiting issues failed to control traffic load to the Platform API. These outages were due to a flaw in the rate limiting logic, preventing containment of the traffic. This flaw led to API failures for both customers and other internal services.
In each case, we identified the source of the incoming requests, and blocked continued actions which restored availability. Learning from the first incident, we were able to identify the pattern and take appropriate action much more quickly during the second incident.
What will we do to mitigate problems like this in the future?
In the short term, we will update our internal playbooks to help us identify, contain, and resolve incidents like this more rapidly. We are also reviewing our rate limiting logic, and will continue to invest in the resilience of response automation managing edge cases like this event.
This incident has been resolved.
This incident is now resolved
This incident has been resolved.
This issue has been resolved.
Once routing capacity was increased request latency returned to normal levels.
We have identified the issue and our engineers are putting a fix in place.
Applications that are deployed using the container registry are seeing issues with booting up processes (e.g. web dynos are not booting up, heroku run bash fails). This includes cycling dynos for running apps. Our Engineers are investigating.
This incident is now resolved.
The issue is resolved.
We are currently seeing latency or issues with Heroku Postgres and Redis CLI commands (heroku pg:wait, heroku pg:info, etc.). Our Engineers are investigating.
Requests to the EU region are experiencing an increase in latency. Our Engineers are investigating.
Latency has returned to normal.
The issue is resolved.
We have received reports of increased failure rates during Postgres provisions. We are investigating the issue.
The issue is resolved.
The issue is resolved.
Latency on Heroku API is back to normal.
Engineers are looking into possible latency in Heroku API.
Logging services have been restored and the issue is resolved.
Between July 28th 23:50 UTC and 02:30 UTC (2 hours 40 minutes) some application logs may not have been delivered, or may appear out-of-order, in logging add-ons and custom drains.
We are investigating losses and delays in application logs. Customers may be seeing delayed, repeated or lost logs from heroku logs, logging add-ons or the Heroku Dashboard "View logs" page.
We are investigating the possibility of missing application log lines.
A capacity issue with Heroku logplex may have caused delayed application logs between between 10:00 and 20:10 UTC today. The issue is resolved.
We are investigating an increase in latency and errors for API requests.
This incident is now resolved and the affected services are operating normally.
Everything appears to be back to normal.
The issue is resolved.
The issue is resolved.
This incident is now resolved.
We're resolving this incident since Node.js app build error rates have returned to normal levels. Please refer to the NPM issue for further details.
Once identified our engineers took steps to reduce the load and request latency has returned to normal levels.
Once identified our engineering team immediately began work increasing capacity. Both US and EU regions are now at full capacity and dyno start times have returned to normal.
This issue is resolved.
This maintenance is complete. All platform functionality has returned to normal.
This issue has been resolved.
The issue is resolved.
This incident is now resolved.
On June 13th, 2016 between 20:58 UTC and 23:00 UTC, a limited number of customers experienced application instability and loss of Runtime Metrics. We sincerely apologize for the problems this caused.
Here is some additional detail about what happened and steps we are taking to mitigate future outages of a similar nature.
Who was affected?
Customers who had the lab feature Runtime Metrics enabled would have experienced an application restart and a loss of Runtime Metrics.
What Happened?
A deployment of our runtime system (which generates Runtime Metrics) had a defect which wrongly interpreted the state of the lab feature. As a result, Runtime Metric events were not generated during this incident for customers who had the lab feature enabled.
During the resolution of the issue we restarted the impacted customer’s applications to start generating Runtime Metrics once again. During this restart customer’s applications may have been unstable, as the applications were transitioning from being stopped to started.
What will we do to mitigate problems like this in the future?
We have improved the testing and monitoring for Runtime Metrics to detect this failure prior to production and refactoring the Runtime Metrics feature itself to prevent this failure in the future. We also updated our playbooks to ensure that in the future if we need to restart apps dyno restart will be staggered and not all dynos from an app restarted at the same time.
The issue is resolved.
All portions of this maintenance are now complete and service has returned to normal. If you created an application during the maintenance and it isn't showing logs as expected, restarting your dynos with heroku restart will solve that issue. If you added add-ons requiring log drains during this maintenance, you may need to delete it and re-add it.
The issue is now resolved.
We are investigating issues with the heroku run command.
This issue is resolved. We'll post our post-mortem for the incident here once we've completed it.
Engineers are investigating issues with the application status. Some applications are seeing H10 errors and are not reachable.
GitHub sync and Review apps are currently unavailable. Our engineers are working on the fix.
The issue has been resolved. GitHub sync and Review apps were unavailable from 11:34pm PST (07:34 UTC) to 11:57pm PST (07:57 UTC). GitHub sync and Review apps requests during this time were delayed.
Our engineers are investigating issues with Heroku Connect. Customers may be seeing the delay for syncing data in the US region. Add-on provisioning for Heroku Connect is currently disabled for both US and EU regions.
The issue has been resolved.
Heroku Connect was unavailable from 4:44pm PST (00:44 UTC) to 5:37pm PST (01:37 UTC).
We are investigating increased error rates in Heroku Postgres management functions.
We are investigating increased error rates in Heroku Postgres management functions.
This issue was resolved retroactively. Please refer to the Retroactive text for details.
We're aware of the OpenSSL DROWN bug. Our security team and engineers are diligently working to assess and remediate any exposure as quickly as possible. We'll keep you updated as things progress. Updates to follow.
This issue is resolved.
The service update has completed successfully.
Heroku engineers are performing an emergency service update on the Heroku build system. For the next 15 minutes the following services will be unavailable:
- Application builds through
git push heroku
- Automated deployment via GitHub pull requests
- Application deployment via GitHub and Dropbox integration
We will provide an update when the service update has completed.
Engineers are investigating the cause of slow responses and errors from the Heroku API.
This issue is resolved.
This issue is now resolved.
Engineers are investigating the cause of slow responses and errors from the Heroku API.
The issue is now resolved.
The issue is now resolved.
The scheduled logging service update is now underway. To reiterate, for the duration you may see the following impact:
- API access will be limited or unavailable
- App scaling events will be unavailable, including new dyno provisioning, restarts, and idling/unidling
- Deploys will be suspended during this maintenance
We'll post more updates as necessary or when the maintenance is complete.
This maintenance is complete. All platform functionality has returned to normal regarding this maintenance. We continue to work on the API availability issues with https://status.heroku.com/incidents/860.
This issue has been resolved. The database having issues was replaced and applications tied to the old database were switched over to the new database. Affected applications will not have any metrics showing for the duration of this incident.
This service update is now complete. All logging functionality has returned to normal. Any applications created during the update should start seeing logs after their first daily cycling or manual restart.
The scheduled logging service update is now underway. To reiterate, for the duration you may see the following impact:
- Failures while creating new log drains.
- Failures adding add-ons that require log drain creation.
- Log tails may fail intermittently and will lose their 1500 line buffer.
- Newly created applications won't have logging until their first daily cycling post-service update.
- Log delivery for existing applications may be lossy for the duration.
We'll post more updates as necessary or when the service update is complete.
We are investigating reports of errors surrounding our billing and invoices.
This issue is resolved.
Invoice data is now accurate and any discrepancies have been corrected. Please review your current usage and open a support ticket if you see anything unexpected.
Again we’re sorry for this issue. We’re conducting a full post-mortem and will be posting it to this status incident once it’s completed.
We are currently investigating an increase in GitHub Sync build failures.
The incident is now resolved.
This morning (Feb 16th PST), several vendors disclosed a vulnerability (CVE-2015-7547) in the getaddrinfo functions in libc which allows an attacker to potentially execute remote code via DNS lookups returning a specially-crafted payload. You can find more details about this vulnerability in this excellent writeup by Google Security.
Updated stack images for our Cedar-14 stack have been released, and will propagate to your dynos within the next 24 hours.
Heroku Postgres and Heroku Redis data services are being automatically updated to include the recommended patches to protect against this issue. We expect this update process to be complete within the next few hours.
Users of Standard & Premium data plans for Heroku Postgres wishing to receive these updates sooner can make use of the follower changeover process, otherwise no further action is required.
As per our stack update policy, any applications running on deprecated stacks, e.g. Cedar-10, have not received these updates.
This issue has been resolved.
Earlier today a revision of access control for Heroku CLI updates caused an unexpected permission failure in the Heroku CLI plugin repository. This failure propagated throughout the CDN we use to publish Heroku CLI plugins. Our engineers have reverted the change and invalidated the faulty data from our CDN.
If customers continue to see issues with the Heroku CLI install please try running the heroku update command. If it continues to fail after that please log a support ticket at https://help.heroku.com/
Our engineers are investigating issues with the Heroku CLI. Customers may be seeing failures referencing "Error reading plugin: heroku-pipelines".
Builds are proceeding as normal. The issue is resolved.
Planned maintenance caused an unexpected increase in build delay, resulting in hanging builds. We have identified the cause of the delay.
We are looking into a likely cause of this issue along with steps toward mitigation.
We are seeing no further issues.
Our automated systems have detected potential platform errors. We are investigating.
Engineers are investigating increased error rates provisioning new Postgres and Redis databases. You may experience issues creating new databases and High availability databases may not recover after a failure.
This issue is now resolved.
The issue has been resolved. Application Metrics were unavailable from 18:06 UTC to 18:43 UTC, Feb 3 due to this issue.
The fix has been applied and we are seeing metrics on the dashboard now.
We are investigating an issue with the app that manages Heroku Data resources. Some heroku pg and heroku redis commands may be unavailable. Existing Postgres and Redis databases are unaffected.
This issue has been resolved.
This service update has completed successfully.
This service update is starting now.
GitHub sync and Review apps are working normally.
We are seeing elevated errors around the apps configured with GitHub sync or Review apps. This may include builds and the deploy tab of the app dashboard. https://status.github.com/ may have updated information regarding this issue.
All database services have been restored and the platform is operating normally.
From 22:27 to 22:37 UTC, we noticed increase latency and network errors across the platform. This caused minor instability across our systems as well as an increase in network-related errors in applications running on the platform. At this time, latency has returned to normal and systems are recovering. We are continuing to monitor for any further issues.
This issue has been resolved.
Engineers are investigating increased load on the app that manages Heroku Postgres and Redis resources. You may experience issues creating new databases or viewing information on existing databases, but running databases are unaffected.
This issue is resolved.
We are investigating a drop in overall routing availability. This might result in closed connections or other errors.
Build failures on Node.js apps have returned to normal operating level.
This incident is now resolved.
This issue is resolved.
This issue is resolved.
Engineers are manually migrating all the affected Redis databases to new servers. We estimate this will take a few more hours to complete.
If your app is still affected by this, you can bring it back online by creating a new hobby-dev Redis instance. To do that, run heroku addons:create heroku-redis:hobby-dev, then promote the new instance by running heroku redis:promote.
This issue is resolved.
API maintenance is complete. All systems operational.
This issue is now resolved.
This incident is now resolved.
The issue has been resolved.
This issue has been resolved.
This incident is now resolved.
On November 1st, between 11:06 and 13:51 UTC, a subset of apps (described below) experienced sporadic unavailability. Our service provider identified and fixed an infrastructure problem responsible for this incident and has taken further steps to prevent similar situations in the future.
We sincerely apologize for the problems this incident caused.
Here is some additional detail about what happened and steps we are taking to mitigate the impact of future outages of a similar nature.
What was affected?
Elastic Load Balancers (ELBs) associated with SSL Endpoints lost contact with some or all Heroku routing layer servers, resulting in sporadic impact to apps using the SSL Endpoint add-on. At any given time, at least 8% of apps using SSL Endpoint were affected, and the set of affected apps may have changed throughout the incident window. HTTP requests to impacted apps were met with an abruptly closed connection which would appear as a blank page in a browser. HTTPS requests would be abruptly closed as well, potentially during the TLS handshake phase. This may have appeared as a “TLS Handshake Failure” error in a browser and/or a blank page.
What Happened?
An AWS infrastructure problem caused a delay in propagation of state to the firewalls running on EC2’s underlying hosts, which prevented ELBs from being able to reach Heroku’s routing layer. AWS resolved the propagation delay, restoring communication with the Heroku routing layer and thus fully restoring service for impacted apps.
What will we do to mitigate problems like this in the future?
The root cause has now been corrected by AWS, and AWS has instituted more sensitive alarming. While this was a problem internal to AWS, it is our goal to limit or prevent impact in such situations whenever possible. We are working closely with AWS to improve incident response for problems like this and to prevent similar incidents in the future.
This maintenance is complete.
This maintenance is complete.
A fix for this issue has been identified and can either be executed manually or will eventually be applied to local clients.
We're investigating problems with build output during git pushes.
This issue is now resolved.
The GitHub API is back up and we have enabled GitHub sync. We continue to monitor.
The update of the Heroku build service has begun. For the next 30 minutes Heroku Git repositories, the Heroku button, Github Sync, Dropbox Sync, and the Heroku Builds API will not be available.
This service update is resolved.
The service update was completed successfully.
The Platform API has been restored and is now available.
The maintenance is now complete and our engineers are monitoring the logging infrastructure to ensure it's operating as expected.
The maintenance completed successfully.
We are investigating issues with GitHub Sync. Platform API and GIT builds are not impacted.
This issue is now resolved.
This service update is resolved.
Part of the service update could not be completed within the scheduled update period. This work will be performed in a future service update. The future update will be announced at least three business days in advance, in accordance with the Heroku platform update guidelines.
The update of the Heroku build service has begun. For the next 30 minutes Heroku Git repositories, the Heroku button, Github Sync, Dropbox Sync, and the Heroku Builds API will not be available.
Heroku engineers are investigating reports of errors accessing application logs.
This issue is resolved.
The service update completed successfully.
The Platform API service update has begun.
During this time the Heroku Platform API will not be available and the build system is disabled. Scheduled jobs in the Heroku Scheduler will not run. The Heroku Dashboard and Heroku Help systems will also not be available. Idling apps will not unidle during the update, but apps already running on dynos will continue to run.
While the API is being updated you will be unable to log into your Heroku account. If you need to communicate with our support staff during this time, please email outage-support@heroku.com. Support staff will not be able to perform API calls on your behalf during the service update.
We will post an update when the work is completed.
This issue has been resolved. This afternoon's change has been fixed and users can now access their "Deploy" tab.
We're beginning the scheduled service update to the production database management system now. During this process, customers will be unable to provision new production databases, and most heroku pg commands will be unavailable.
This service update has completed successfully.
This issue has been resolved. The backlog has cleared and git push deploys are working as expected again.
The operational update completed successfully.
The maintenance for status.heroku.com is starting.
While status.heroku.com is undergoing maintenance, please refer to https://twitter.com/herokustatus for the latest updates.
We will provide an update when the operational update is complete.
