Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Microsoft's Incident Report for PennO365

Following is a data extract from the PennO365 administrative console of incident reports sent by Microsoft within the last 30 days. We pull out incidents that are relevant to the current PennO365 service offerings and we refresh this data every 15 minutes. The last extract was taken on December 16, 2017 20:00.

--------------------------------------------------------------------------------------------------
Title EX126015
ServiceAffected Exchange Online
Status Service restored
Time 12/14/2017 4:04:00 AM UTC
LastUpdated 12/15/2017 4:35:26 AM UTC
Message Title: Unable to download search data

EX126015 Details

==========
User Impact: Admins may be unable to retrieve reporting data from the Exchange admin center.

More info: This issue affects reporting data that contains SharePoint Online-specific data. Admins may receive the following error when they attempt to download affected reports: "500 Unexpected Error :( An error occurred and your request couldn't be completed. Please try again."

Current status: Our investigation determined that a recent update contained a configuration issue which interrupts the flow of data between SharePoint Online and our reporting feature in the admin center. We've identified a potential fix, and we're deploying it to a subset of infrastructure to ensure that it resolves the issue. Once testing is completed successfully, we'll deploy the fix to the remaining infrastructure.

Scope of impact: This issue may potentially affect any of your admins attempting to retrieve reporting data in Exchange admin center when the reports contain SharePoint-specific data.

Start time: Thursday, December 14, 2017, at 4:04 AM UTC

Preliminary root cause: A recent update contained a configuration issue which interrupts the flow of data between SharePoint Online and our reporting feature in the admin center.

Next update by: Thursday, December 14, 2017, at 11:00 PM UTC, Title: Unable to download reporting data

==========
User Impact: Admins may be unable to retrieve reporting data from the Exchange admin center.

More info: This issue affects reporting data that contains SharePoint Online-specific data. Admins may receive the following error when they attempt to download affected reports: "500 Unexpected Error :( An error occurred and your request couldn't be completed. Please try again."

Current status: We're deploying the fix to the affected infrastructure, and the deployment is approximately 90 percent complete. Admins will begin to experience service restoration as the deployment progresses.

Scope of impact: This issue may potentially affect any of your admins attempting to retrieve reporting data in Exchange admin center when the reports contain SharePoint-specific data.

Start time: Thursday, December 14, 2017, at 4:04 AM UTC

Preliminary root cause: A recent update contained a configuration issue which interrupts the flow of data between SharePoint Online and our reporting feature in the admin center.

Next update by: Friday, December 15, 2017, at 3:00 AM UTC, Title: Unable to download reporting data

==========
User Impact: Admins were unable to retrieve reporting data from the Exchange admin center.

More info: This issue affected reporting data that contains SharePoint Online-specific data. Admins may have received the following error when they attempted to download affected reports: "500 Unexpected Error :( An error occurred and your request couldn't be completed. Please try again."

Final status: We deployed a fix and confirmed that the issue has been resolved.

Scope of impact: This issue may have potentially affected any of your admins attempting to retrieve reporting data in the Exchange admin center when the reports contained SharePoint-specific data.

Start time: Thursday, December 14, 2017, at 4:04 AM UTC

End time: Friday, December 15, 2017, at 1:45 AM UTC

Preliminary root cause: A recent update contained a configuration issue which interrupted the flow of data between SharePoint Online and our reporting feature in the admin center.

Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles.

This is the final update for the event.}
--------------------------------------------------------------------------------------------------
Title MO125986
ServiceAffected Office 365 Portal
Status Service restored
Time 12/13/2017 11:12:34 PM UTC
LastUpdated 12/13/2017 11:21:42 PM UTC
Message Title: Can't provision tenants or users

MO125986 Details

User impact: Admins may have been unable to provision new tenants or users.

More info: Provisioning requests made during the impact duration had a 50% chance of failing. Any failed requests should be resubmitted now that service is restored.

Final status: We determined that a subset of systems that handle provisioning requests encountered a failure during regularly scheduled maintenance, which impacted their ability to process requests to add new tenants or users. We corrected the issue and ensured that new requests were processing as expected.

Scope of impact: Impact was limited to approximately 50% of provisioning requests made during the impact time frame.

Start time: Wednesday, December 13, 2017, at 8:32 PM UTC

End time: Wednesday, December 13, 2017, at 11:00 PM UTC

Preliminary root cause: An unexpected problem caused impact to occur during a standard service update.

Next steps: - We're reviewing our standard service update procedures to avoid similar impact in the future.

This is the final update for the event.
--------------------------------------------------------------------------------------------------
Title MO125908
ServiceAffected Office 365 Portal
Status Restoring service
Time 12/6/2017 12:00:00 AM UTC
LastUpdated 12/16/2017 12:46:41 AM UTC
Message Title: Office 365 usage report data freshness issue

MO125908 Details

==========
User Impact: Admins may notice that usage reports do not contain data from the last few days.

Current status: We discovered an issue that was impacting the ability of our systems to process usage reporting data. We've mitigated the issue that was causing impact, and our systems are now processing the backlog of data. We expect that reporting data will gradually catch up and be current by December 22, 2017.

Scope of impact: The impact is specific to Exchange Online and Skype for Business usage data from December 6, 2017 to present; however, the freshness of the data will improve as systems continue to process the backlog.

Start time: Wednesday, December 6, 2017, at 12:00 AM UTC

Estimated time to resolve: Based on the current rate of processing, we expect this problem to be resolved by December 22, 2017.

Preliminary root cause: A configuration problem related to a recent system update caused impact to the efficiency of processing usage report data, leading to a delay in data freshness.

Next update by: Friday, December 22, 2017, at 11:00 AM UTC, Title: Office 365 usage report data freshness issue

==========
User Impact: Admins may notice that usage reports do not contain data from the last few days.

Current status: Usage report data for Exchange Online and Skype for Business is now current; however, SharePoint Online and OneDrive for Business data was also affected by a processing issue and will be delayed until the backlog has cleared.

Scope of impact: Exchange Online and Skype for Business usage data was delayed between December 6, 2017 to December 15, 2017. SharePoint Online and OneDrive for Business usage data may be delayed starting December 12, 2017. The freshness of the data will improve as systems continue to process the backlog.

Start time: Wednesday, December 6, 2017, at 12:00 AM UTC

Estimated time to resolve: Based on the current rate of processing, we expect this problem to be resolved by December 22, 2017.

Preliminary root cause: A configuration problem related to a recent system update caused impact to the efficiency of processing usage report data, leading to a delay in data freshness.

Next update by: Friday, December 22, 2017, at 11:00 AM UTC}
--------------------------------------------------------------------------------------------------
Title MO125731
ServiceAffected Office 365 Portal
Status Service restored
Time 12/6/2017 12:00:00 AM UTC
LastUpdated 12/7/2017 12:23:47 AM UTC
Message Title: Office 365 domain management issue

MO125731 Details

==========
User Impact: Admins who are set as global admins may be unable to access the domain section of the Office 365 admin center.

Current status: Our investigation determined that a recent update contained a misconfiguration which prevented access to the domain section of the Office 365 admin center. We're implementing a configuration update to restore service.

Scope of impact: The impact is specific to global admins of organizations who have purchased their Office 365 subscriptions from a reseller.

Start time: Wednesday, December 6, 2017, at 12:00 AM UTC

Preliminary root cause: A recent update contained a misconfiguration which prevented access to the domain section of the Office 365 admin center.

Next update by: Wednesday, December 6, 2017, at 10:00 PM UTC, Title: Office 365 domain management issue

==========
User Impact: Admins who are set as global admins may be unable to access the domain section of the Office 365 admin center.

Current status: We've implemented the configuration update on a subset of infrastructure to validate that it successfully restores service. Once we've completed the validation, we'll implement the change to the remaining affected infrastructure.

Scope of impact: The impact is specific to global admins of organizations who have purchased their Office 365 subscriptions from a reseller.

Start time: Wednesday, December 6, 2017, at 12:00 AM UTC

Preliminary root cause: A recent update contained a misconfiguration which prevented access to the domain section of the Office 365 admin center.

Next update by: Thursday, December 7, 2017, at 12:00 AM UTC, Title: Office 365 domain management issue

==========
User Impact: Admins who are set as global admins may be unable to access the domain section of the Office 365 admin center.

Current status: The configuration update did not completely resolve the issue. We're reverting the update which caused the issue in an effort to restore service.

Scope of impact: The impact is specific to global admins of organizations who have purchased their Office 365 subscriptions from a reseller.

Start time: Wednesday, December 6, 2017, at 12:00 AM UTC

Preliminary root cause: A recent update contained a misconfiguration which prevented access to the domain section of the Office 365 admin center.

Next update by: Thursday, December 7, 2017, at 2:00 AM UTC, Title: Office 365 domain management issue

==========
User Impact: Admins who were set as global admins may have been unable to access the domain section of the Office 365 admin center.

Final status: We've reverted the update and monitored the environment to verify service was restored.

Scope of impact: The impact was specific to global admins of organizations who purchased their Office 365 subscriptions from a reseller.

Start time: Wednesday, December 6, 2017, at 12:00 AM UTC

End time: Wednesday, December 6, 2017, at 11:15 PM UTC

Preliminary root cause: A recent update contained a misconfiguration which prevented access to the domain section of the Office 365 admin center.

Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles.

This is the final update for the event.}
--------------------------------------------------------------------------------------------------
Title EX125826
ServiceAffected Exchange Online
Status Service restored
Time 12/4/2017 8:00:00 AM UTC
LastUpdated 12/11/2017 6:23:21 PM UTC
Message Title: Can't receive non-RFC compliant messages

EX125826 Details

==========
User Impact: External senders may receive an NDR if they send emails that contain bare linefeeds, which are not RFC compliant.

More info: External users sending this type of message may receive the following error message: "LED=554 5.6.0 Invalid message content, contains bare linefeeds."

Current status: We've identified a code issue that was recently introduced during our standard deployment process. We're preparing to deploy a fix to mitigate impact.

Scope of impact: This issue may affect any inbound emails that contain bare linefeeds, which are not RFC compliant. In most cases, this can occur if a sender is using a legacy application or device to send messages. If senders to your organization rely heavily on applications that are sending these message types, you may have users reporting impact related to this issue. While we work to remediate impact, we recommended that you advise any external senders affected by this issue to update their messaging systems to be RFC compliant, which will mitigate impact.

Start time: Monday, December 4, 2017, at 8:00 AM UTC

Preliminary root cause: A code change was recently introduced which affects processing of malformed messages containing bare linefeeds. We highly recommend any affected senders ensure they are sending RFC-compliant line endings - carriage return (CR) followed by a line feed (LF).

Next update by: Saturday, December 9, 2017, at 3:00 AM UTC, Title: Can't receive non-RFC compliant messages

==========
User Impact: External senders may receive an NDR if they send emails that contain bare linefeeds, which are not RFC compliant.

More info: External users sending this type of message may receive the following error message: "LED=554 5.6.0 Invalid message content, contains bare linefeeds."

Current status: We're deploying a fix into the affected environment and we'll continue to monitor the progress of the deployment until it is completed.

Scope of impact: This issue may affect any inbound emails that contain bare linefeeds, which are not RFC compliant. In most cases, this can occur if a sender is using a legacy application or device to send messages. If senders to your organization rely heavily on applications that are sending these message types, you may have users reporting impact related to this issue. While we work to remediate impact, we recommended that you advise any external senders affected by this issue to update their messaging systems to be RFC compliant, which will mitigate impact.

Start time: Monday, December 4, 2017, at 8:00 AM UTC

Estimated time to resolve: Based on current deployment progress, we expect this problem to be resolved within the next three days.

Preliminary root cause: A code change was recently introduced which affects processing of malformed messages containing bare linefeeds. We highly recommend any affected senders ensure they are sending RFC-compliant line endings - carriage return (CR) followed by a line feed (LF).

Next update by: Monday, December 11, 2017, at 7:00 PM UTC, Title: Couldn't receive non-RFC compliant messages

==========
User Impact: External senders may have received an NDR if they sent emails that contain bare line feeds, which are not RFC compliant.

More info: External users sending this type of message may have received the following error message: "LED=554 5.6.0 Invalid message content, contains bare line feeds."

Final status: We completed deployment of the fix to the affected systems.

Scope of impact: This issue may have affected any inbound emails that contained bare line feeds, which are not RFC compliant. In most cases, this could occur if a sender was using a legacy application or device to send messages. If senders to your organization rely heavily on applications that send these message types, you may have had users reporting impact related to this issue. While we worked to remediate impact, we recommended that you advise any external senders affected by this issue to update their messaging systems to be RFC compliant, which would have mitigated impact.

Start time: Monday, December 4, 2017, at 8:00 AM UTC

End time: Monday, December 11, 2017, at 4:00 AM UTC

Preliminary root cause: A code change was recently introduced which affected processing of malformed messages containing bare line feeds.

Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. - We highly recommend any affected senders ensure they are sending RFC-compliant line endings - carriage return (CR) followed by a line feed (LF).

This is the final update for the event.}
--------------------------------------------------------------------------------------------------
Title EX125578
ServiceAffected Exchange Online
Status Investigation suspended
Time 11/29/2017 9:47:00 PM UTC
LastUpdated 12/2/2017 1:59:14 PM UTC
Message Title: Can't see free/busy information for Federated users

EX125578 Details

==========
User Impact: Users may be unable to see the free/busy information of federated users in their calendars.

Current status: We're continuing to analyze data associated with a Microsoft Federation Gateway and Internet Information Services (IIS) logs provided by your representatives, to determine root cause and next steps to remediate impact.

Scope of impact: Your organization is affected by this event, and this could potentially impact any of your users.

Start time: Wednesday, November 29, 2017, at 9:47 PM UTC

Next update by: Saturday, December 2, 2017, at 12:00 PM UTC, Title: Can't see free/busy information for Federated users

==========
User Impact: Users may be unable to see the free/busy information of federated users in their calendars.

Current status: We're identified a potential Active Directory Federation Services (ADFS) configuration issue that may be causing impact. We're conducting validation tests to verify that this is the root cause of the issue.

Scope of impact: Your organization is affected by this event, and this could potentially impact any of your users.

Start time: Wednesday, November 29, 2017, at 9:47 PM UTC

Next update by: Saturday, December 2, 2017, at 2:00 PM UTC, Title: Can't see free/busy information for Federated users

==========
User Impact: Users may be unable to see the free/busy information of federated users in their calendars.

Final status: We've determined that a recently added domain in your environment was not configured correctly with ADFS. We recommend working with the Premier Identity Support team so that the configuration issue can be resolved.

Scope of impact: Your organization is affected by this event, and this could potentially impact any of your users.

Start time: Wednesday, November 29, 2017, at 9:47 PM UTC

End time: Saturday, December 2, 2017, at 1:00 PM UTC

Preliminary root cause: A recently added domain in your environment was not configured correctly with ADFS.

This is the final update for the event., Title: Can't see
top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


University of Pennsylvania Penn Computing University of Pennsylvania Information Systems & Computing (ISC)
Information Systems and Computing, University of Pennsylvania