Back to blog

Exploring the Office 365 Security & Compliance Center Part Two: Reports

May 10, 2016 by Emma Robinson

If you are involved in managing Office 365 for your organisation, you will know how important it is to be able to see what’s actually happening in your environment, and be able to use that data to inform decisions around protection, optimisation and adoption of the platform.
In this second part of our blog series on the new Security and Compliance Center in Office 365, we will look at the Reports that are now available natively, consider what they can tell you, and their limitations. (Read the full blog series here First ImpressionsReportsPermissionsData Loss PreventionData Management and Service AssuranceSearch and Investigation, and Alerts)
No time to read the full series now? Download our white paper ‘Getting Started with the Security and Compliance Center’ which includes all the blogs on Office 365 Security and Compliance. Get your copy here 
There are three areas of reporting that can be found here currently: Auditing, Supervisory Review, and Data Loss prevention (DLP). The reports are also available under the ‘Reports’ section of the new Office 365 Admin Center
S+C in Admin Center
But in the Security and Compliance Center, they are listed as follows:
Reports list
Many organisations have very strict requirements when it comes to auditing, and Office 365 has been developed over time so that it increasingly attempts to meet these needs natively. The new reports enable admins to:

  • ‘view activity in SharePoint Online and OneDrive for Business sites’; and
  • see sign-in activity and ‘mail related activity in Exchange Online’.

The reports available at the time of writing are Office 365 audit log report, Azure AD reports, Exchange audit reports.
Audit pop out
The ‘Audit Log’ allows you to drill down into user activity. If you need to see whether a certain individual viewed, copied, deleted or altered an item, it is possible to do this here. It is a ‘unified audit log’, which means that you can search for:

  • User activity in SharePoint Online and OneDrive for Business
  • User activity in Exchange Online (Exchange mailbox audit logging)
  • Admin activity in SharePoint Online
  • Admin activity in Azure Active Directory (the directory service for Office 365)
  • Admin activity in Exchange Online (Exchange admin audit logging)

The search functionality and the filtering is easy to use, with the capability to search within specific timeframes, and for a wide number of activities.
You are able to export the results to a CSV. file, but it is not possible to download more than 50,000 entries from a single search. In order to get around this limit you can run multiple searches with smaller data ranges and collate the information once it has been exported.
Things to note before you begin audit logging:

  1. You must turn on audit logging otherwise you won’t be able to use the feature. You will not be able to gather any information on the period before this is enabled, so it is important to be proactive if you wish to use audit information for a specific task or timeframe.
  2. To turn it on, click ‘Start recording user and admin activity’ on the Audit log search page in the Security and Compliance Center. Once you enable this, you cannot search immediately, and will need to wait a couple of hours while the data is being prepared.
  3. If this link is not visible on this page, it means that logging has already been enabled for your organisation.
  4. In order to search the Office 365 audit log, you must be assigned the ‘View-Only Audit Logs’ or the ‘Audit Logs’ role in Exchange Online. Microsoft’s Support page states that:

‘To be assigned one of these roles, a user must have an Exchange Online license. By default, these roles are assigned to the Compliance Management and Organization Management role groups on the Permissions page in the Exchange Admin Center. To give a user the ability to search the Office 365 audit log with the minimum level of privileges, you can create a custom role group in Exchange Online, add the View-Only Audit Logs or Audit Logs role, and then add the user as a member of the new role group.’

  1. Permission has to be assigned in Exchange Online, not just through the Permissions page in the Security and Compliance Center. This is because the underlying cmdlet used to search the audit log is an Exchange Online cmdlet.
  2. The current restriction on search data is restricted to activities performed within the last 90 days.
  3. It can also take up to 15 minutes for the audit log entry to appear after the event has occurred (in SharePoint Online and OneDrive for Business). In Exchange Online and Azure Active Directory it can take up to 12 hours for the event to be logged.

The main consideration when using auditing features within Office 365 is evidently one of timing. You need to ensure that logging is enabled for when you need it, and that the information you want from it will be available at the point at which it is required.
Supervisory Review:
The next report available in the center is for supervisory review. Many organisations need to have a supervisory review policy, which allows you to capture employee communications for examination by internal or external review. Industries that operate under strict regulation, such as financial or legal services, may require this policy, and it is possible to set it up to review any communications that contain certain phrasing.
The supervisory review report can be used to get the status of all supervisory review policies in your organisation.
Data Loss Protection:
Data loss protection policies take time and effort to set up, so it is important to verify that they are working, and that they contribute towards ensuring that your organisation remains compliant. There are two DLP reports available, enabling you to view information about the SharePoint Online and OneDrive for Business items in your organization, and review any DLP matches, overrides, or false positives for your configured DLP policies and rules.
Currently this section has two reports. The ‘DLP policy and rule matches’ report allows you to filter activity using date, location (SharePoint Online, OneDrive for Business and Exchange Online), and you can drill down into the different policies in place, and then click into the specific incidents.
DLP policy rules and matches
The DLP false positives and overrides report shows where the DLP policy has flagged sensitive data incorrectly, or where a user has overridden the organisation’s policy.
If the user has included ‘sensitive information’ in an email, such as financial details or company information, they would be alerted using a ‘Policy Tip’ (as seen below), warning about sensitive content. A false positive is where the user has clicked ‘Report’ because they do not think that it is applicable for the content that they are sending.
DLP tip
An override is where a similar Policy Tip notifies the sender that:
‘(recipient) is not authorised to receive this email. To send this message, you must override your organisation’s policy.’
Depending on the way in which Policy Tips are configured, they can merely warn workers, block their messages, or even allow them to override the block with a written justification, that is then sent to the Admin.
The DLP report allows you review how your users are interacting with your policies, and to check whether this is working effectively to achieve compliance. The reports make it possible to identify any areas for improvement or refinement to existing policies.
FP and OR
The reports section within the Security and Compliance Center is a great place to start reviewing your policies, and monitor user activity within the platform. As Microsoft have mentioned throughout the new center – it is not finished yet, and this sense of incompletion is visible in some of the reports and features. The center itself is promising, identifying many of the common compliance needs that an organisation may have, but in its current form there is not the breadth and depth available to provide the detail that admins often need when it comes to such important areas of Office 365 management. The other main issue is timing. When it comes to these reports, if you’re not careful you may not realise what you need until it’s too late to get it. If the features are not enabled in time, or if you fail to capture data within the center’s time constraints you might lose valuable information. In order to get the most out of the current Security and Compliance Center, it is important to have a proactive and forward-thinking approach, and while most people aim for this, it is not always achievable in reality – so it’s nice to have a margin of error.
There is definitely space for further refinements within what is currently a relatively limited feature-set, so it will be interesting to see the finished result – particularly as Microsoft have announced their aim to make this space a ‘one stop shop’. We look forward to seeing the shelves once they’re a little more filled.
Still interested in Office 365 Security and Compliance? Why not download our white paper and find out more about how to get started with the new center?
Cogmotive is the leading global provider of enterprise level reporting and analytics applications for Office 365. Find out more now.