Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Microsoft's Incident Report for PennO365

Following is a data extract from the PennO365 administrative console of incident reports sent by Microsoft within the last 30 days. We pull out incidents that are relevant to the current PennO365 service offerings and we refresh this data every 15 minutes. The last extract was taken on June 22, 2017 12:15.

--------------------------------------------------------------------------------------------------
Title EX105382
ServiceAffected Exchange Online
Status Service restored
Time 6/8/2017 5:50:00 PM UTC
LastUpdated 6/9/2017 2:23:42 AM UTC
Message Title: Delays sending and receiving email

EX105382 Details

==========
User Impact: Users may be experiencing delays of up to two hours when sending and receiving email messages.

Current status: We're investigating an issue in which some users may be experiencing delays of up to two hours when sending and receiving email messages. We have identified a fix and are preparing the fix for deployment to restore service.

Scope of impact: A few customers have reported this issue, and our analysis indicates that impact is specific to a subset of users who are served through the affected infrastructure.

Next update by: Friday, June 9, 2017, at 1:00 AM UTC, Title: Delays sending and receiving email

==========
User Impact: Users may be experiencing delays of up to two hours when sending and receiving email messages.

Current status: We're deploying the fix to remediate impact. Users will begin to experience relief as the fix is deployed throughout the affected environment.

Scope of impact: A few customers have reported this issue, and our analysis indicates that impact is specific to a subset of users who are served through the affected infrastructure.

Preliminary root cause: A service feature became degraded, which created message deferrals. These message deferrals caused delays in sending and receiving email messages.

Next update by: Friday, June 9, 2017, at 3:00 AM UTC, Title: Delays sending and receiving email

==========
User Impact: Users may have experienced delays of up to two hours when sending and receiving email messages.

Final status: We've completed the deployment of the fix and confirmed that service was restored.

Scope of impact: A few customers reported this issue, and our analysis indicated that impact was specific to a subset of users who were served through the affected infrastructure.

Start time: Thursday, June 8, 2017, at 5:50 PM UTC

End time: Friday, June 9, 2017, at 1:15 AM UTC

Preliminary root cause: A service feature became degraded, which created message deferrals. These message deferrals caused delays in sending and receiving email messages.

Next steps: - We're reviewing our monitoring services to find ways to reduce detection time and to improve our automated recovery processes.

This is the final update for the event.}
--------------------------------------------------------------------------------------------------
Title EX105149
ServiceAffected Exchange Online
Status Service restored
Time 6/7/2017 7:30:00 AM UTC
LastUpdated 6/7/2017 4:31:57 PM UTC
Message Title: Email delays

EX105149 Details

==========
User Impact: Users may have experienced delays when receiving email messages.

Final status: During peak impact, we removed affected infrastructure from active service rotation and shifted traffic to healthy infrastructure. Additionally, we've implemented configuration changes within the infrastructure to prevent impact from occurring while we revert the update that caused mailflow components to perform below acceptable thresholds.

Scope of impact: While we weren't aware of any customer reports of impact, our monitoring indicated that this issue could have potentially affected any of your users if they were routed through the affected infrastructure.

Start time: Wednesday, June 7, 2017, at 7:00 AM UTC

End time: Wednesday, June 7, 2017, at 1:30 PM UTC

Preliminary root cause: A recent update caused some mailflow components to perform below acceptable thresholds, creating a backlog of queued messages.

Next steps: - We're reviewing our update procedures to help catch this kind of problem during our testing cycle. - We're reviewing our code for optimizations and automated recovery options to prevent this kind of problem in the future.

This is the final update for the event., Title: Email delays
--------------------------------------------------------------------------------------------------
Title EX103663
ServiceAffected Exchange Online
Status Post-incident report published
Time 5/30/2017 9:30:00 PM UTC
LastUpdated 6/6/2017 9:06:26 PM UTC
Message A post-incident report has been published., Title: Can't access email

EX103663 Details

==========
User Impact: Users may have been unable to access their mailbox from Outlook, Outlook on the web, mobile devices, or other protocols.

Final status: We completed the configuration change and confirmed that affected systems have recovered.

Scope of impact: Many customers reported this issue, and our analysis indicated that the most significant impact occurred for users connecting from the Asia-Pacific region. This issue may have affected any of your users if they were routed through the affected infrastructure.

Start time: Tuesday, May 30, 2017, at 9:30 PM UTC

End time: Wednesday, May 31, 2017, at 2:30 AM UTC

Preliminary root cause: A code issue in a regularly scheduled deployment resulted in a configuration problem on a subset of front-end and back-end systems that facilitate connectivity to the service. This configuration problem caused incoming requests to be processed inefficiently, resulting in impact to service availability.

Next steps: - We're reviewing our update procedures to help catch this kind of problem during our testing cycle.

We'll publish a post-incident report within five business days., A post-incident report has been published., Current Status: We're investigating an issue in which some users may be unable to access or use Exchange Online services or features. We'll provide an update shortly....}
--------------------------------------------------------------------------------------------------
Title EX103413
ServiceAffected Exchange Online
Status Service restored
Time 5/28/2017 9:40:00 PM UTC
LastUpdated 6/1/2017 5:12:47 AM UTC
Message Title: Can't connect to Outlook on the web via Mozilla Firefox

EX103413 Details

==========
User Impact: Users may have been unable to connect to Outlook on the web via the Mozilla Firefox browser.

More info: This is an incremental update to provide the latest status on the issue.

Current status: We've completed restarting over 70 percent of the affected systems and we anticipate that all systems will have the updated certificate by 3:00 AM UTC on June 1, 2017. Users will continue to experience service restoration as the certificate data is refreshed across the environment. For complete details on this event, please reference the Restoring service post from 12:55 AM UTC on May 30th, 2017.

Estimated restoration time: By Thursday, June 1, 2017, at 3:00 AM UTC

Next update by: Wednesday, May 31, 2017, at 12:00 AM UTC, Title: Can't load EOP via Firefox

==========
User Impact: Users may be intermittently unable to connect to Exchange Online Protection (EOP) features via the Firefox browser.

More info: This issue is affecting access to safelinks via Firefox, however, users are able to connect to the affected sites via other browsers, such as Internet Explorer, Edge, Chrome, or Safari.

Current status: We're investigating an issue in which some users may be unable to access or use Exchange Online Protection (EOP) services or features via the Firefox browser. We've identified that a certificate may be causing the issue and we're analyzing systems logs to confirm certificate status and identify the source of the issue.

Scope of impact: Customer reports indicate that many users will likely experience impact related to this event. Our analysis indicates that this affects any users connecting to EOP sites via the Firefox browser.

Next update by: Monday, May 29, 2017, at 12:00 PM UTC, Title: Can't load EOP via Firefox

==========
User Impact: Users may be intermittently unable to connect to Exchange Online Protection (EOP) features via the Firefox browser.

More info: This issue is affecting access to safelinks via Firefox, however, users are able to connect to the affected sites via other browsers, such as Internet Explorer, Edge, Chrome, or Safari.

Current status: We've determined that some Online Certification Status Protocol (OCSP) stale certificate data was preventing access to the EOP services or features. We're restarting services in an attempt to clear the stale data from the affected environment. .

Scope of impact: Customer reports indicate that many users will likely experience impact related to this event. Our analysis indicates that this affects any users connecting to EOP sites via the Firefox browser.

Next update by: Monday, May 29, 2017, at 6:30 PM UTC, Title: Can't connect to Outlook on the web via Mozilla Firefox

==========
User Impact: Users may be unable to connect to Outlook on the web via the Mozilla Firefox browser.

More info: Affected users may see the following error when using Mozilla Firefox to connect to Office 365 services: "Invalid OCSP signing certificate in OCSP response." While we're focused on resolving the issue, users may connect to the affected sites via other browsers, such as Internet Explorer, Edge, Chrome, or Safari.

Current status: Our investigation determined that an updated certificate was already deployed across the service, but the old certificate data was still cached. We're continuing to restart systems across the affected environment to force them to install the updated certificate, which should resolve the underlying issue. We anticipate that restarting all affected systems will take several hours, but we are working to expedite this as much as possible. Users will continue to experience service restoration as the affected systems are restarted.

Scope of impact: A few customers have reported this issue, and our analysis indicates that this affects any users connecting to Outlook on the web via the Firefox browser.

Start time: Sunday, May 28, 2017, at 9:40 PM UTC

Next update by: Tuesday, May 30, 2017, at 1:00 AM UTC...}
top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


University of Pennsylvania Penn Computing University of Pennsylvania Information Systems & Computing (ISC)
Information Systems and Computing, University of Pennsylvania