Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Microsoft's Incident Report for PennO365

Following is a data extract from the PennO365 administrative console of incident reports sent by Microsoft within the last 30 days. We pull out incidents that are relevant to the current PennO365 service offerings and we refresh this data every 15 minutes. The last extract was taken on August 17, 2017 15:15.

--------------------------------------------------------------------------------------------------
Title EX113001
ServiceAffected Exchange Online
Status Service restored
Time 8/6/2017 10:00:00 PM UTC
LastUpdated 8/8/2017 12:28:58 PM UTC
Message Title: Can't connect from OWA mobile app

EX113001 Details

==========
User Impact: Users may be unable to connect to Exchange Online from the OWA for iPhone or Android app.

More info: Users can connect from other protocols such as the Outlook client, the Outlook mobile app, Outlook on the web, and Outlook on the web in their mobile browser.

Current status: We're analyzing system logs to determine the source of the issue.

Scope of impact: This issue may potentially affect any of your users attempting to connect to the OWA app on iOS or Android devices.

Next update by: Tuesday, August 8, 2017, at 4:00 AM UTC, Title: Can't connect from OWA mobile app

==========
User Impact: Users may be unable to connect to Exchange Online from the OWA for iPhone or Android app.

More info: Users can connect from other protocols such as the Outlook client, the Outlook mobile app, Outlook on the web, and Outlook on the web in their mobile browser.

Current status: We're investigating recent code changes and system logs to determine the source of the issue.

Scope of impact: This issue may potentially affect any of your users attempting to connect to the OWA app on iOS or Android devices.

Next update by: Tuesday, August 8, 2017, at 6:00 AM UTC, Title: Can't connect from OWA mobile app

==========
User Impact: Users may be unable to connect to Exchange Online from the OWA for iPhone or Android app.

More info: Users can connect from other protocols such as the Outlook client, the Outlook mobile app, Outlook on the web, and Outlook on the web in their mobile browser.

Current status: We've identified a recent update that introduced a code issue that prevents users from connecting to Exchange Online from the OWA for iPhone or Android app. We're working to develop a fix for this issue.

Scope of impact: This issue may potentially affect any of your users attempting to connect to the OWA app on iOS or Android devices.

Start time: Sunday, August 6, 2017, at 10:00 PM UTC

Preliminary root cause: A recent update introduced a code issue that prevents users from connecting to Exchange Online from the OWA for iPhone or Android app.

Next update by: Tuesday, August 8, 2017, at 8:00 AM UTC, Title: Can't connect from OWA mobile app

==========
User Impact: Users may be unable to connect to Exchange Online from the OWA for iPhone or Android app.

More info: Users can connect from other protocols such as the Outlook client, the Outlook mobile app, Outlook on the web, and Outlook on the web in their mobile browser.

Current status: We've developed and tested a fix, which will be deployed across the affected infrastructure. Users will experience service restoration as soon as the fix reaches their environment.

Scope of impact: This issue may potentially affect any of your users attempting to connect to the OWA app on iOS or Android devices.

Start time: Sunday, August 6, 2017, at 10:00 PM UTC

Estimated time to resolve: Within the next four to eight hours.

Preliminary root cause: A recent update introduced a code issue that prevents users from connecting to Exchange Online from the OWA for iPhone or Android app.

Next update by: Tuesday, August 8, 2017, at 1:00 PM UTC...}
--------------------------------------------------------------------------------------------------
Title MO112471
ServiceAffected Office 365 Portal
Status Service restored
Time 8/3/2017 11:30:00 PM UTC
LastUpdated 8/4/2017 4:06:45 AM UTC
Message Title: Usage reports issue

MO112471 Details

==========
User Impact: Admins may be unable to access usage reports from the Office 365 admin center.

Current status: We're investigating an issue where usage reports may not be functioning properly. We'll provide an update within the next hour.

Scope of impact: This issue may potentially affect any of your admins attempting to view usage reports.

Start time: Thursday, August 3, 2017, at 11:30 PM UTC

Next update by: Friday, August 4, 2017, at 12:30 AM UTC, Title: Usage reports issue

==========
User Impact: Admins may be unable to access usage reports from the Office 365 admin center.

Current status: We're continuing to investigate system logs to determine the source of the issue and the next troubleshooting steps.

Scope of impact: This issue may potentially affect any of your admins attempting to view usage reports.

Start time: Thursday, August 3, 2017, at 11:30 PM UTC

Next update by: Friday, August 4, 2017, at 1:30 AM UTC, Title: Usage reports issue

==========
User Impact: Admins may be unable to access usage reports from the Office 365 admin center.

Current status: We're investigating a recent code update that may have inadvertently caused impact to usage reports.

Scope of impact: This issue may potentially affect any of your admins attempting to view usage reports.

Start time: Thursday, August 3, 2017, at 11:30 PM UTC

Next update by: Friday, August 4, 2017, at 3:00 AM UTC, Title: Usage reports issue

==========
User Impact: Admins may be unable to access usage reports from the Office 365 admin center.

Current status: We're continuing to review a recent code update to confirm whether it is the source of impact.

Scope of impact: This issue may potentially affect any of your admins attempting to view usage reports.

Start time: Thursday, August 3, 2017, at 11:30 PM UTC

Next update by: Friday, August 4, 2017, at 4:30 AM UTC...}
--------------------------------------------------------------------------------------------------
Title MO112255
ServiceAffected Office 365 Portal
Status Service restored
Time 8/2/2017 5:49:00 PM UTC
LastUpdated 8/2/2017 10:37:50 PM UTC
Message Title: Can't sign in

MO112255 Details

==========
User Impact: Users may be unable to sign in to the Office 365 service.

Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes., Title: Can't sign in

==========
User Impact: Users may be unable to sign in to the Office 365 service.

More info: This issue appears to be isolated to a subset of users and not affecting all applications.

Current status: We identified a potential issue related to Content Delivery Network (CDN) configuration that may be related to a recent update. As a precaution, we're working to revert the update while we continue our investigation into the CDN.

Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure.

Next update by: Wednesday, August 2, 2017, at 8:30 PM UTC, Title: Can't sign in

==========
User Impact: Users may be unable to sign in to the Office 365 service.

More info: This issue appears to be isolated to a subset of users and not affecting all applications.

Current status: We confirmed that impact is being caused by a Content Delivery Network (CDN) issue, but it is not related to the update mentioned previously. We did revert the update as a precaution, but this didn't mitigate impact completely. We're working to further isolate the problem within the CDN to determine our next remediation steps.

Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure.

Start time: Wednesday, August 2, 2017, at 6:00 PM UTC

Preliminary root cause: A subset of CDN infrastructure involved in the authentication process is degraded, which is causing intermittent sign-in failures for Office 365 services and applications.

Next update by: Wednesday, August 2, 2017, at 9:30 PM UTC, Title: Can't sign in

==========
User Impact: Users may have been unable to sign in to the Office 365 service.

More info: This issue was isolated to a subset of customers and users with a specific firewall configuration.

Final status: Upon further investigation we determined that a third-party listing service incorrectly classified a critical CDN URL as a malicious site. This issue only affected customers and users that leverage firewall solutions that rely on the hpHosts listing service. Once the URL was whitelisted by the listing service, users reported that impact was mitigated. Any admins seeing continued impact should ensure that the following URL is whitelisted explicitly: secure.aadcdn.microsoftonline-p.com

Scope of impact: Impact was specific to a subset of customers and users whose firewall solution relies on the hpHosts listing service.

Start time: Wednesday, August 2, 2017, at 5:49 PM UTC

End time: Wednesday, August 2, 2017, at 7:30 PM UTC

Preliminary root cause: A critical Content Delivery Network (CDN) URL was classified as a malicious site by a third-party listing service, which impacted customers and users whose firewall solution relied on the hpHosts listing service. This led to intermittent sign-in failures for Office 365 services and applications for those customer and users.

Next steps: - We're reviewing our monitoring services to look for ways to reduce detection time and more quickly restore service.

This is the final update for the event.}
--------------------------------------------------------------------------------------------------
Title EX112205
ServiceAffected Exchange Online
Status False positive
Time 8/2/2017 4:36:54 PM UTC
LastUpdated 8/3/2017 12:14:41 AM UTC
Message Current Status: We're investigating an issue in which some users may be unable to access or use Exchange Online services or features. We'll provide an update shortly., Title: Outlook Connectivity

EX112205 Details

==========
User Impact: There wasn't any end-user impact.

Final status: The investigation is complete and we've determined the service is healthy. A service incident did not actually occur. This message will be removed from your Service Health Dashboard within the next 12 hours., Title: Outlook Connectivity

==========
User Impact: There wasn't any end-user impact.

Final status: The investigation is complete and we've determined the service is healthy. A service incident did not actually occur.}
--------------------------------------------------------------------------------------------------
Title EX112430
ServiceAffected Exchange Online
Status Service restored
Time 7/31/2017 2:55:00 PM UTC
LastUpdated 8/10/2017 7:50:22 PM UTC
Message Title: Mail sync issue

EX112430 Details

==========
User Impact: Users are unable to sync emails using Apple Mail on their Mac computers.

More info: While we're focused on remediation, users may be able to access their mail with the Outlook for Mac client or Outlook on the web as an alternative method.

Current status: We're working to develop a fix for this issue. After deploying a potential solution, we determined that a more comprehensive fix is needed to address the underlying issue causing impact. We're continuing our work to test and develop the fix, which will then be deployed across the affected environment.

Scope of impact: Impact is specific to users attempting to sync emails using the default mail client on Mac machines.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Preliminary root cause: A recent update contained a code issue that prevents users from syncing emails using Apple Mail on Mac computers.

Next update by: Friday, August 4, 2017, at 8:00 PM UTC, Title: Apple Mail sync issue

==========
User Impact: Users are unable to sync email, calendar, and contacts using Apple Mail on their Mac computers.

More info: While we're focused on remediation, users may be able to access their data with the Outlook for Mac client or Outlook on the web as an alternative method.

Current status: We've completed developing and testing a new fix, and we're preparing to deploy it across the affected environment. This process will begin over the weekend, and usually takes around 72 hours to reach the majority of systems. As the deployment progresses, users will begin to experience service restoration. In the interim, we encourage affected users to use the workaround described above.

Scope of impact: Impact is specific to users attempting to sync email, calendar, and contacts using the default mail client on Mac machines.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Estimated time to resolve: We expect this issue to be completely resolved for all affected users within the next 3-4 days as the fix is safely implemented through our change management process.

Preliminary root cause: A recent update contained a code issue that prevents users from syncing email, calendar, and contacts using Apple Mail on Mac computers.

Next update by: Monday, August 7, 2017, at 10:00 PM UTC, Title: Apple Mail sync issue

==========
User Impact: Users are unable to sync email, calendar, and contacts using Apple Mail on their Mac computers.

More info: While we're focused on remediation, users may be able to access their data with the Outlook for Mac client or Outlook on the web as alternative methods.

Current status: We've initiated the deployment of the fix, and we're continuing to monitor the environment as the deployment progress. Users will begin to experience service restoration as the deployment reaches their infrastructure. In the interim, we encourage affected users to use the workaround described above.

Scope of impact: Impact is specific to users attempting to sync email, calendar, and contacts using the default mail client on Mac machines.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Estimated time to resolve: We expect this issue to be completely resolved for all affected users within the next 2-3 days as the fix is safely implemented through our change management process.

Preliminary root cause: A recent update contained a code issue that prevents users from syncing email, calendar, and contacts using Apple Mail on Mac computers.

Next update by: Tuesday, August 8, 2017, at 10:00 PM UTC, Title: Apple Mail sync issue

==========
User Impact: Users are unable to sync email, calendar, and contacts using Apple Mail on their Mac computers.

More info: While we're focused on remediation, users may be able to access their data with the Outlook for Mac client or Outlook on the web as alternative methods.

Current status: Deployment of the fix is approximately 75 percent complete.

Scope of impact: Impact is specific to users attempting to sync email, calendar, and contacts with the default mail client on Mac machines using the Exchange Web Services protocol. Other devices such as mobile iOS devices using the Exchange ActiveSync protocol are not affected by this issue.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Estimated time to resolve: We expect this issue to be completely resolved for all affected users within the next 1-2 days as the fix is safely implemented through our change management process.

Preliminary root cause: A recent feature update for calendaring services contained a code issue that prevents users from syncing email, calendar, and contacts using Apple Mail on Mac computers.

Next update by: Wednesday, August 9, 2017, at 10:00 PM UTC...}
--------------------------------------------------------------------------------------------------
Title EX112199
ServiceAffected Exchange Online
Status False positive
Time 7/31/2017 2:55:00 PM UTC
LastUpdated 8/3/2017 7:05:52 PM UTC
Message Title: Mail sync issue

EX112199 Details

==========
User Impact: Users are unable to sync emails using Apple Mail on their Mac computers.

Current status: We're analyzing diagnostic data in an effort to determine the remediation steps.

Scope of impact: Your organization is affected by this event, and impact is specific to users attempting to sync emails using default mail client on their Mac machines.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Next update by: Wednesday, August 2, 2017, at 7:00 PM UTC, Title: Mail sync issue

==========
User Impact: Users are unable to sync emails using Apple Mail on their Mac computers.

Current status: We've reviewed diagnostic data and suspect that a configuration issue with mailbox components may be causing the problem. We've made a configuration change and are working with a test user to determine if impact persists.

Scope of impact: Your organization is affected by this event, and impact is specific to users attempting to sync emails using default mail client on their Mac machines.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Next update by: Wednesday, August 2, 2017, at 9:00 PM UTC, Title: Mail sync issue

==========
User Impact: Users are unable to sync emails using Apple Mail on their Mac computers.

Current status: We've confirmed that the configuration change has not fixed the issue as we anticipated. We suspect that a recent update may be causing the problem and we're working to confirm our findings.

Scope of impact: Your organization is affected by this event, and impact is specific to users attempting to sync emails using default mail client on their Mac machines.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Preliminary root cause: A recent update may be preventing users from syncing emails using Apple Mail on their Mac computers.

Next update by: Wednesday, August 2, 2017, at 11:00 PM UTC, Title: Mail sync issue

==========
User Impact: Users are unable to sync emails using Apple Mail on their Mac computers.

Current status: We've determined that a recent update is causing the issue. We've created and are deploying a fix, which we are verifying with test users.

Scope of impact: Your organization is affected by this event, and impact is specific to users attempting to sync emails using default mail client on their Mac machines.

Start time: Monday, July 31, 2017, at 2:55 PM UTC

Preliminary root cause: A recent update may be preventing users from syncing emails using Apple Mail on their Mac computers.

Next update by: Thursday, August 3, 2017, at 1:00 AM UTC...}
--------------------------------------------------------------------------------------------------
Title EX111121
ServiceAffected Exchange Online
Status Service restored
Time 7/26/2017 2:15:00 PM UTC
LastUpdated 7/28/2017 3:39:09 AM UTC
Message Current Status: We're investigating an issue in which some users may be unable to access or use Exchange Online services or features. We'll provide an update shortly., Title: Exchange Online connectivity

EX111121 Details

==========
User Impact: Users may be unable to connect to the Exchange Online service.

Current status: We're analyzing the affected infrastructure in order to isolate the cause of impact.

Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure.

Start time: Wednesday, July 26, 2017, at 2:15 PM UTC

Next update by: Wednesday, July 26, 2017, at 4:30 PM UTC, Title: Exchange Online connectivity

==========
User Impact: Users may be unable to connect to the Exchange Online service.

Current status: Our investigation determined that system services related to authentication became degraded and are performing below acceptable thresholds. We're restarting services across the affected infrastructure to remediate impact.

Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure. This issue is likely affecting a very small percentage of users and requests.

Start time: Wednesday, July 26, 2017, at 2:15 PM UTC

Preliminary root cause: System services related to authentication became degraded and are performing below acceptable thresholds within a limited subset of mailbox databases.

Next update by: Wednesday, July 26, 2017, at 5:00 PM UTC, Title: Exchange Online connectivity

==========
User Impact: Users may be unable to connect to the Exchange Online service.

Current status: We're continuing to restart services across the affected infrastructure to remediate impact.

Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure. This issue is likely affecting a very small percentage of users and requests.

Start time: Wednesday, July 26, 2017, at 2:15 PM UTC

Preliminary root cause: System services related to authentication became degraded and are performing below acceptable thresholds within a limited subset of mailbox databases.

Next update by: Wednesday, July 26, 2017, at 6:00 PM UTC...}
top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


University of Pennsylvania Penn Computing University of Pennsylvania Information Systems & Computing (ISC)
Information Systems and Computing, University of Pennsylvania