- Azure Sentinel—A real-world example - Tue, Oct 12 2021
- Deploying Windows Hello for Business - Wed, Aug 4 2021
- Azure Purview: Data governance for on-premises, multicloud, and SaaS data - Wed, Feb 17 2021
One of the issues with modern IT infrastructures is that they’re silent. If someone breaks into an office, with luck they will leave muddy footprints or, at least, broken locks or windows; however, IT systems are much more opaque. IDS and IPS systems attempt to fix this problem, but they can’t be everywhere and see everything. System logs do record most things, however, whether it’s a security breach or a malfunctioning website. The problem is correlating all the information from all these distributed systems.
Splunk offers an interesting solution to this problem (and many others) by being able to ingest almost any type of “machine data” (logs generated by any system), keeping it in its native format and letting you search through it easily. You can then set up alerts based on searches as well as create reports and dashboards.
Installation
Head over to the download page and grab the installer for Splunk. Supported platforms include Windows, Linux, Solaris, OSX, FreeBSD, AIX, and HP-UX. The Windows installation prompts you whether you want to run Splunk as a local system or other user (you can change your mind later). You can gather the data that you want Splunk to analyze in two ways: with an agent on each system (called a forwarder) or via WMI. The former is recommended in most cases unless you can’t install software on production servers or you don’t have administrative access (more information here). If you want to monitor other systems in an AD environment, make sure to install Splunk as a domain user with the right permissions. You can use Managed Service Accounts (MSAs) if your environment is Server 2008 or later. The installation is quick and, when complete, presents you with a login page in your browser.
The main page of Splunk gives you quick access to any apps you have installed, as well as access to the search functionality.
Apps
While the basic Splunk installation gives you the option to configure and search through many different data sources, apps are one thing that will make your life easier. Think of these as templates that are already set up and configured to work with a particular type of log source, such as Windows, Box, iptables, JBoss, Django, AWS Billing, Exchange, Cisco IOS, AD, CheckPoint, VMware, MobileIron, UNIX/Linux, NetFlow, Oracle, Android, and Hyper-V, among many, many others. Some of these are free, whereas others require licensing.
The types of machine data that Splunk can handle out of the box cover all the usual suspects.
With the Windows app installed, it was easy for me to access event logs and performance data with preconfigured lists, dashboards, and graphs.
Usage
Splunk stores your data in its original format and only applies a schema to it at indexing time. To start using Splunk, you simply enter terms in the search window. For more complex searches, there’s Splunk Search Processing Language (SPL), with shortcuts to help you find exactly what you’re looking for. Once you get results back, you can narrow your search further. Note that Splunk helps by providing interesting fields that can help you find what you’re looking for, as well as statistics for each of those fields. Everything is indexed by time, which is where the real power of Splunk comes into play.
Searching using SPL is very powerful, although it takes a little while to get used to the syntax.
Although it’s interesting to be able to mine through large log files easily, the real key comes when you can correlate data between different sources (based on time indexing), such as finding every user account across your entire network that failed to log in, or finding a particular page that is causing errors in a web-based shopping application. Once you’ve found the information that’s crucial for you, it’s easy to create alerts for monitoring, as well as reports and dashboards to visualize this information over time. Splunk excels at correlating data for business intelligence, security and compliance, and operational management requirements.
The Windows app really makes it quick to get value from Splunk for OS event and performance logging.
If you want to integrate Splunk into other tools, there are SDKs for Java, JavaScript, C#, Python, PHP, and Ruby. If you’re looking at a larger implementation of Splunk, you can cluster several servers together for high availability. You can also have servers in several locations correlate their data to a central Splunk system for visibility across your entire infrastructure. If you don’t want to plan, deploy, and maintain your own Splunk infrastructure in a larger environment, the option to use it as a service is available through Splunk Cloud.
Licensing
When you download Splunk you get (almost) the full functionality of the product for 60 days, with the major limitation that you can only index 500 MB of data per day. The paid Enterprise license adds larger data indexing, monitoring, clustering, and support. See here for a full breakdown. Note that, for smaller setups, you can convert to a free license and keep using Splunk forever, as long as you stay under the 500 MB/day limit (or don’t go over it more than three times in a month); see here.
Summary
Splunk borrows heavily from Hadoop and Map Reduce technologies to provide insight into large amounts of data. As a systems administrator, I see clear use cases for Splunk to really derive value from all the machine data that’s already gathered in our IT systems by correlating the information. And it’s fun to play with!
