Fast event log search in PowerShell with the FilterHashtable parameter

In my last post, I showed you how to display and find specific events with the Get-WinEvent cmdlet. Whereas you can filter event messages easily with the Where-Object cmdlet, using the Data key from the FilterHashtable parameter is much faster.

Measuring the speed difference ^

Let's see an example by searching all security events related to cscript.exe with both methods. We will measure the speed execution of each method with the Measure-Command cmdlet.

First, I will filter a big Security log with the Where-Object cmdlet.

Where Object filtering speed

Where Object filtering speed

Now I will filter the same log with the Data key and the FilterHashtable parameter.

Data key filtering speed

Data key filtering speed

As you can see, filtering with the Where-Object cmdlet took 23 minutes, while using the Data key and the FilterHashtable parameter took only 33 seconds.

However, before you will be able to use the FilterHashtable parameter to filter events by Message, we first need to look at the raw data structure of an event.

Raw data structure of an event ^

Let's open an event with the Event Viewer console.

Though usually we stay on the General tab, this time we will select the Details tab and the XML view.

Notice there are two main elements named System (the green rectangle) and EventData (the blue rectangle), which is the data of the message itself.

XML structure view of an event

XML structure view of an event

To display this information in a user-friendly manner, Windows uses a DLL or an EXE file registered in the HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Application registry key.

The same data then appears in the General tab like you usually see it. This also translates the EventData (blue rectangle) to a human-readable paragraph.

And the same happened to the System data (green rectangle) where you can see for example that Windows has translated the Operation Code (OpCode) from the number 13 to "Installation."

User friendly view of an event

User friendly view of an event

Now let's come back to the XML view. In the System element, you will probably recognize some sub-elements matching to keys we used with the FilterHashtable parameter in my last post.

Here is the list of all keys currently implemented and available with the Get-WinEvent cmdlet:

  • LogName=<String[]>
  • ProviderName=<String[]>
  • Path=<String[]>
  • Keywords=<Long[]>
  • ID=<Int32[]>
  • Level=<Int32[]>
  • StartTime=<DateTime>
  • EndTime=<DataTime>
  • UserID=<SID>

EventData, the second main XML element, is a string array where you can find metadata about the error message itself.

To find one of these strings, you can use the Data key.

However, there are some restrictions:

  • You must provide the whole string you are searching for.
  • The search is case sensitive.
  • It does not permit wildcards or regular expressions.

For example, let's say you want to find all system events matching a planned operating system upgrade.

User friendly view of an upgrade event

User friendly view of an upgrade event

If you have a look at the XML metadata, here is what you see:

XML view of an upgrade event

XML view of an upgrade event

You have two options:

  • You can use any word or expression of the user-friendly General tab view (for example the word "Upgrade") combined with the Where-Object cmdlet, but this is the slowest method.
  • Or you can use the Data key combined with one of these two strings:
    • "Operating System: Upgrade (Planned)"
    • "0x80020003"

If you choose the second option, it will give you for example the following command line:

My advice: if you have an example of event you are searching for, it's really worth it to try and look to the metadata if there is something easy to search for. Otherwise, fall back to the Where-Object cmdlet and be patient if you have a big log file to parse.

Displaying only events for a specific account ^

A common mistake is to use the UserID key. However, this usually does not work because:

  • The UserID key is part of the System element and contains the ID of the account that has written the event. Most of the time, it is Local System (S-1-5-18) or NT Authority (S-1-5-19).
  • The UserID is only present in some events of the System and the Application log.
  • The account you are probably searching for is in fact in the event's message section (EventData), which you can query with the Data key.

The following command will display all events for the account matching the specified security identifier (SID).

Another feature of the Data key is that it accepts an array of strings as input. However, the system will process all strings with the OR operator.

This is interesting in our case because we can search an event for a specific user by his Security Account Manager (SAM) account name and by his account SID at the same time.

Displaying only events for a specific IP address and a specific port ^

Because the Data key processes all strings input with the OR operator, we must search for one argument with the Data key and search for the second one by piping the result to the Where-Object cmdlet.

For example, if we want to find events matching IP 10.11.22.33 and port 3389, we can use the following command line:

Join the 4sysops PowerShell group!

Your question was not answered? Ask in the forum!

1+
Share
8 Comments
  1. scott 3 years ago

    Your write ups are great, very interesting, but dammed hard to read with all the in line adverts in your web page,  they are too distracting.   4 of them, with no or small differences to the article text cause a headache when trying to see the actual content and getting snapped off in other directions.  Keep up the good articles but please do something about all the adverts.

    4+

  2. scott, thanks for your comment. We are considering to offer ad free content for our members. How much would you be willing to pay for this service?

    0

  3. Scott 3 years ago

    maybe a Patreon subscription,  ?  couple of bucks a month ?

    0

  4. impossible game 3 years ago

    wow what a great post is this… thanks for sharing boss, keep share these valuable content.
    I’m really impressed with your article, such great & usefull knowledge you mentioned here

    0

  5. Paul 3 months ago

    I know this article's been out for almost 3 years, but it REALLY helped me. I was using Where-Object as I usually do and wasn't happy with the wait times, hoping there was a faster/more efficient way.

     

    Thanks so much Luc!

    0

    • Paul 3 months ago

      An interesting item to note is that in testing it appears that the Data field search within -FilterHashTable isn't the most efficient. In this case ? -Property -Match actually wins out. I think a hybrid approach might be best, where you use -FilterHashTable with the structured data (all properties minus Data) with the Filter Left best practice and then leave unstructured string searching to Where-Object.

      PS C:\Users\generic.user> Measure-Command -Expression {Get-WinEvent -FilterHashTable @{LogName='Security';ID=4625,4740;Data='generic.user'}}

      TotalSeconds      : 36.0055711
       

      PS C:\Users\generic.user> Measure-Command -Expression {Get-WinEvent -FilterHashTable @{LogName='Security';ID=4625,4740;} | ? -Property Message -Match 'generic.user'}

      TotalSeconds      : 20.4539246

      I'm open to anyone chiming in on what I'm seeing.

      0

Leave a reply

Your email address will not be published. Required fields are marked *

*

© 4sysops 2006 - 2020

CONTACT US

Please ask IT administration questions in the forums. Any other messages are welcome.

Sending

Log in with your credentials

or    

Forgot your details?

Create Account