In the third part of my Office 365 Secure Score series, we'll dive into various reports that need monitoring.

Paul Schnackenburg

Paul Schnackenburg works part time as an IT teacher as well as running his own business in Australia. He has MCSE, MCT, MCTS and MCITP certifications. Follow his blog TellITasITis.

Reporting in Office 365 ^

The first report is successful sign-ins after multiple failures. This could be a sign of a successful brute-force password-hacking attempt or a very forgetful user. To qualify for the 15-point score increase, you'll need to review this report weekly. Log in at, pick Learn more for the signs-ins after multiple failures report, and then click the Review button. This takes you (at the time of writing) to the old Azure portal and a list of reports.

Another report that'll give you 10 points if you look at it weekly is the sign-ins from unknown sources report. This one looks at logins from anonymous proxy IP addresses, such as the Tor network. It's highly unlikely that your users have a valid reason to access Office 365 anonymously, so this could give you an indication of a breached account.

Next on the list for 10 points is the signs-ins from multiple geographies report. This one covers "impossible travel" situations. In these, logins from a single user account occur from two separate locations such that it would be impossible for a user to travel between the locations in that time. Note that if a user has shared a password with a co-worker, is using a VPN, or if the IP geography assignment isn't accurate, this could provide false positives in the report. But in general, it's a good sign that one of the logins is by an attacker.

Continuing with the theme of reports, this time for 5 points, have a look at mailbox access by non-owners. This one is good for spotting malicious access to user mailboxes. Note however that users often have legitimate access to other people's mailboxes, so make sure you know what's legitimate and what's not.

Mailbox access by non owners in Office 365

Mailbox access by non owners in Office 365

Another 5-point report is the malware detections report. This shows how many times Microsoft blocked a malware attachment before it got to your users. The thinking here is that if you see an increase in attacks, you might want to warn your users about a particular type of malware.

Malware detections in Office 365

Malware detections in Office 365

Note that by default you probably don't have access to this report. Head over to, click on permissions, admin roles, and add your account to the Hygiene Management role.

Adding a user to Hygiene Management in Exchange Online

Adding a user to Hygiene Management in Exchange Online

Tracking user accounts and devices in Office 365 ^

Another thing to review weekly for 10 points is check role group changes. You could do this by checking the membership manually in the portal Groups. You can also filter the audit log search for role administration activity. The point is to catch suspect role group changes where attackers may be elevating their privileges.

For 10 points, you should also have a look at the device sign-in report weekly. This tracks devices possibly infected with malware or new device sign-ins from a possibly compromised account.

For 5 points, have a look at the account provisioning activity report. If you use a third-party application or service for user provisioning, this report will show expected traffic. But if you don’t, consider any records in this report malicious.

Note that for Secure Score to record the review of a report, you have to open it from the Secure Score portal. If you just go to the AAD portal and open them, it won't record the views.

These are lots of reports to look at, especially on a weekly basis. But Microsoft's point here I think is to show the line between your responsibility for security as a consumer of a cloud service and Microsoft's responsibility. There's a misconception that "if it's in the cloud, security is the provider's responsibility."

Yes, Microsoft manages some areas such as physical access to data centers, server and network protection, and governance over their IT management engineers. But endpoint security, good configuration, and most importantly, monitoring of what's going on in your tenant is your responsibility. And these reports are a good way to start. Another option for monitoring is to use the System Center Operations Manager (SCOM) Office 365 management pack, which is a bit limited in the value it provides; there's more information here.

There's also an Operations Management Suite (OMS) solution for Office 365—read more here. This solution provides similar information to the audit logs we looked at in part 2, but the power of OMS of course is the log analytics part. This enables you to correlate SharePoint and Exchange usage data along with user and group changes in AAD.

In the fourth and final piece of this series, we'll look at other Secure Score actions along with what's required if you want to do all possible actions. In addition, we'll explore email tracking and some other recommendations around security.

Win the monthly 4sysops member prize for IT pros



Leave a reply

Your email address will not be published. Required fields are marked *



Please ask IT administration questions in the forum. Any other messages are welcome.

© 4sysops 2006 - 2017

Log in with your credentials


Forgot your details?

Create Account