- Removing a corrupted Canon print driver - Fri, Apr 8 2016
- VMware vSphere licensing update 2016 – No love for the little guy - Fri, Mar 25 2016
- Veeam releases free Endpoint Backup 1.5 - Fri, Mar 18 2016
I had the pleasure of a fun issue lately where the account that Cisco Unified Communications Manager (CUCM) uses to connect to Active Directory kept getting locked out. As a result, nobody was able to log on reliably to either Jabber or Unified Cisco Unified Contact Center Express (UCCX), making for sad users. Frankly, it’s been a while since I’ve had to track these types of events down; not only did I have to remember where to look, but I also noted that the event codes have changed.
Enable auditing
Step one in getting any real information is to enable auditing at the domain level. For me, step one for setting up a new Active Directory domain is to enable both success and failure of auditing account logon events, either in the Default Domain Policy or the Default Domain Controllers Policy. These settings are set in Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > Audit Policy, as shown. You may want to choose to audit other events as well. After this is done, a number of events will be generated on the domain controllers any time a user logs on or off, both interactively or not.
Audit Policy
What to look for
Tracking both successes and failures of logons in an environment of any size is going to generate a large number of audit events in a very short period. So, in this case, we are going to have to use some good filters, either in Event Viewer or in your preferred logging utility. Active Directory draws a fine line in the sand regarding Event IDs; in this case, in the Server 2003 R2 release, you have one set for 2003 R2 and older and a completely different set for 2008 and newer.
For 2008 and newer, I like to filter for the following:
- § 4740: A user account was locked out.
- § 4625: An account failed to log on.
- § 6273: Network Policy Server denied access to a user.
- § 6279: Network Policy Server locked the user account due to repeated failed authentication attempts.
These last two are specific to an environment where you are using NPS to handle 802.1x-type activity for things such as Enterprise wireless access. A full listing of Server 2008 and newer security audit events is available in a KB.
Server 2003 R2 and earlier uses a different set of events; however, in the interest of not feeding bad habits, if you need those you’ll have to Google for yourself. It is time to get out of 2003 and join us in the future. 🙂
Tracking down the relevant information
Now that you have information to find what you’re looking for, the issue with a distributed Active Directory environment is how to find it. Good old Event Viewer is the source of what you’re going to be looking for, but that will get tedious when you have even five Active Directory controllers, as my environment does, let alone what a major corporation might have.
On the grand scale, a good answer might be a centralized logging system such as Splunk, Graylog, or VMware’s Log Insight, but for this case the fine folks at Microsoft provide a nice set of utilities called the Account Lockout and Management Tools (AL Tools), which are freely downloadable from their website.
Lockout status
When extracted, AL Tools are a collection of portable executables that allow you to find and target account lockout issues. Your first step is using the LockoutStatus.exe utility to target the offending account and see which domain controller in AD is unsuccessfully trying to authenticate, as well as when the last bad logon attempt was and the time/date of the latest lockout.
Simply open the application, go to File > Select Target, enter the account and domain, and let it do the work. In a highly distributed environment, this may also help you figure out what is causing the lockout if the logon attempt is application-based, because it identifies the domain controllers to which the account has unsuccessfully attempted to authenticate. This utility will also give you the ability to unlock accounts as needed.
EventCombMT
Next, you can use the EventCombMT utility also included in AL Tools. EventCombMT allows you to search one or more computers for a given set of parameters and then dump the output to a text file you can go through and analyze.
A number of built-in searches exist, including one for account lockouts. However, because this is an older utility, it still contains the Event IDs for 2003 and before; you’ll simply have to replace these with the newer codes above. Choosing this search will automatically add all DCs in your environment to the search.
Lockout event
Finally, if you’ve been able to nail down the domain controller to which the user is trying to authenticate, you can use Event Viewer to have a nice and pretty view of what the failed logon event looks like. Most important, after you track down the event either through Event Viewer or EventCombMT, the record will identify the computer the authentication attempt started from; so, even if you are looking at a situation such as mine where it is a service account and not one interactively used by a user, you can find out where the account is trying to log in.
In my case, a malformed Jabber configuration was causing the event. All I had to do was clear the config and rebuild, and the issue went away. In other situations, you may want to check out the services administrative tool and see if you are using user authentication on any of them.
In any case, good luck and good hunting!
