| Sitecore via Azure Marketplace | Service is operating normally |
| Sitecore via AppSource | Service is operating normally |
| Device Detection | Service is operating normally |
| IP Geolocation | Service is operating normally |
Jul 29, 2021
Email Cloud 5xx API error rates and increased latency
As part of cloud infrastructure monitoring activities, Sitecore Cloud Operations team identified an event which we would like to bring to your attention. In case you have any follow up question regarding the event, please do not hesitate to raise a support case on that matter from the home page of Sitecore Support & Self-Service Portal.
Event description: Sparkpost has identified some elevated levels of 5xx API errors for the US-hosted customers. The issue is currently under investigation.
Impact: Various US-hosted APIs are failing.
Additional references: https://status.sparkpost.com/incidents/crgz59qmrc9t
Planned further actions: N/A
Who performs further actions: Sparkpost
Next update planned at: As soon as we receive an update
Any additional updates of the event status from Sitecore side may be published below.
Mitigated: Sparkpost team has implemented a fix and the issue is now mitigated.
Update: Sparkpost identified that the issue is only affecting Suppression List API in the US region. No other APIs are affected at the moment.
Jul 19, 2021
Email Cloud 5xx API error rates and increased latency
Elevated 5xx API error rates and increased latency for some SparkPost and SparkPost Enterprise customers (US hosted only) were detected.
Sparkpost engineers have implemented a fix.
Now the incident is resolved.
Resolved:
Jul 16, 2021
Email Cloud SparkPost connection failures
As part of cloud infrastructure monitoring activities, Sitecore Cloud Operations team identified an event which we would like to bring to your attention. In case you have any follow up question regarding the event, please do not hesitate to raise a support case on that matter from the home page of Sitecore Support & Self-Service Portal.
Event description: Some Email Cloud (Sparkpost) customers may get connection errors while trying to connect to the service. Sitecore team has already identified the root cause of the issue and working on access restoration.
Next update: in 1 hour
Resolved: Service access has been restored. If you have any issues please contact Sitecore Support.
Jul 12, 2021
Azure Front Door and CDN issue
As part of cloud infrastructure monitoring activities, Sitecore Cloud Operations team identified an event which we would like to bring to your attention. In case you have any follow up question regarding the event, please do not hesitate to raise a support case on that matter from the home page of Sitecore Support & Self-Service Portal.
Event description: We have received an update from Microsoft stating customers may have experienced connection issue to Azure front door services and Azure CDN environments, The issue started on July 12, 2021 at approximately 18:00 UTC.
Microsoft Engineers have determined that the backend queue balancer is reaching its operational threshold.
Impact: Customer might have experienced connection issues with Azure front door.
Planned further actions: Microsoft are working on re-routing the flow of traffic.
Refer to https://status.azure.com/en-us/status for more updates.
Who performs further actions: Microsoft
Next update planned at: July 12, 2021 @21:00 UTC
Any additional updates of the event status from Sitecore side may be published below.
Update:
The issue now pertains to Azure front door ONLY, primarily in the US & Canada regions. Other regions, including Europe, Asia Pacific and the Middle East, might experience the same impact. Microsoft team is currently applying mitigation.
For more information: https://app.azure.com/h/0TYG-DPG/b82fbb
Next update planned at: As soon as the mitigation is in place.
Mitigation: Microsoft has re-balanced incoming traffic to other nearby locations to alleviate the increased CPU utilization, and reduced the CPU utilization limit. These have successfully mitigated the incident.
Jul 08, 2021
Service management operation failures - North Europe
As part of cloud infrastructure monitoring activities, Sitecore Cloud Operations team identified an event that we would like to bring to your attention. In case you have any follow-up questions regarding the event, please do not hesitate to raise a support case on that matter from the home page of Sitecore Support & Self-Service Portal.
Event description: Starting at 21:19 UTC on 07 Jul 2021, a subset of customers in North Europe may intermittently receive error notifications when performing service management operations for services that rely on compute resources in this region. Retries may be successful for some customers.
Impact: May intermittently receive error notifications when performing service management operations, also starting at 01:21 UTC on 08 Jul 2021, you have been identified as a customer using Virtual Machines in North Europe who may receive error notifications when performing service management operations - such as create, delete, update, scaling, start, stop - for resources hosted in this region.
Current Status: The HTTP request queue limit on some of the gateway nodes reached thresholds that are suspected to have caused these intermittent failures. Previous attempts to mitigate this issue by upgrading Gateway components did not succeed and we are currently verifying an alternative fix for this issue.
Additional references:
https://app.azure.com/h/4_6G-39G/c95567
Planned further actions: The next update will be provided in 2 hours, or as events warrant.
Who performs further actions: Sitecore will monitor the status of this issue
Resolved: Azure has mitigated the issue. More details on https://status.azure.com/en-us/status/history/