Using Amazon SQS with PowerShell

• Author
• Recent Posts

Josh Rickard

Josh's primary focus is in Windows security and PowerShell automation. He is a GIAC Certified Windows Security Administrator (GCWN) and GIAC Certified Forensic Analyst (GCFA). You can reach Josh at MSAdministrator.com or on Twitter at @MS_dministrator.

Latest posts by Josh Rickard (see all)

• Search Event Logs and error codes with Netikus.net System32 - Thu, Jan 10 2019
• Netikus.net IPMon+ – GUI for ipmon.exe - Tue, Nov 20 2018
• Understanding PowerShell Begin, Process, and End blocks - Mon, Oct 15 2018

Amazon SQS is a flexible queue service that has many uses cases, from scalable web applications and processing systems to inter-application communications. This is useful if you need to process lots of data at once or have threads or workers to do a specific action to a large pool of data.

With the explosion of use cases for PowerShell and Docker in production systems, you can easily scale using queue systems like Amazon SQS. SQS is beneficial when your service or application has job workers that need to perform actions based on a large pool of data.

For example, let's say you had a web application to manage your Active Directory accounts. You would have a self-service web application that manages those accounts to make changes as needed. With this example, a queue is a perfect example of how to scale this application. Let's break this down further:

1. The user submits a requested change to an Active Directory account's properties via a webform.
2. The web application would place this request into an Amazon SQS queue.
3. You would have workers (or Docker containers) responsible for pulling data off the queues to complete the requested change.

Web application with a queue and worker backend

Using Amazon SQS in this situation can scale your workers in and out depending on the current size of your SQS queue. This allows your application service to be flexible but also have a specific purpose.

Using PowerShell in this situation is a no-brainer. Active Directory and PowerShell (or .NET) go hand in hand. There are definitely other cases for using PowerShell to interact with Amazon SQS. But to keep this post simple, I will continue with this example.

The first thing you need to do is install the AWS Tools for PowerShell or PowerShell Core:

1. PowerShell on Windows:

   Install-Package -Name AWSPowerShell

2. PowerShell Core:

   Install-Module -Name AWSPowerShell.NetCore

After you have installed AWSPowerShell, you will have access to all the available AWS functions, including the following list of SQS functions:

List of Amazon SQS functions

To continue, you will need all the necessary tokens from your AWS account. Accessing and creating these tokens and keys is out of the scope of this post.

Let's first create a new SQS queue. As you might guess, we will be using the New-SQSQueue cmdlet to create a new queue. This is how you create a new queue using PowerShell:

New-SQSQueue -QueueName 'my-new-powershell-queue' -AccesKey $AccessKey ‑SecretKey $SecretKey -Region 'us-east-1'

This will create a basic SQS queue. If you want or need to, you can specify certain attributes for your queue. These attributes range from encryption to VisibilityTimeout to setting the queue to be first in, first out (FIFO). The VisibilityTimeout attribute sets the duration to hide a queue message temporarily when a worker pulls an item from the queue. The neat thing about this attribute is that if the worker does not tell the queue it has processed the data successfully, the queue will automatically mark the data as available, as if the worker had never received the request originally.

Now that we've set up our new SQS queue, we now add data to it. To do this, we will use the Send-SQSMessage function. In our example, let's say a web application has a form that allows a user to change the DisplayName attribute of an Active Directory account. Once the user clicks Submit, the webform would add the following ($jsonData) to an Amazon SQS queue:

	{
  		"DisplayName": "Mr. Bob Smith",
  		"UserName": "bob.smith"
}

To mimic the webform, we will go ahead and add this data to our new queue:

Send-SQSMessage -QueueUrl 'https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/my-new-powershell-queue' -AccessKey $AccessKey  SecretKey $SecretKey  Region 'us-east-1'  MessageBody $jsonData

Now that we have data in our SQS queue, we could have our worker or Docker container continually check the queue for new data to process. Our example could have a worker take the information submitted to our Amazon SQS queue and run some PowerShell code to process the data. In our example, our worker is just responsible for changing the DisplayName on a given Active Directory account. Our worker would first have to check the SQS queue for any tasks or jobs. Here is a small snippet of code that checks the SQS queue for a single request. This code then creates a new PowerShell object of the requested data:

$SQSMessage = Receive-SQSMessage -QueueUrl $QueueUrl -WaitTimeInSeconds 10  MessageCount 1 -AccessKey $AccessKey -SecretKey $SecretKey -Region $Region

        if ($null -eq $SQSMessage)
        {
            return 'EMPTY'
        }
        else
        {
            $MessageBodyObject = $($SQSMessage.Body) | ConvertFrom-Json

            $props = @{
                MessageId = $SQSMessage.MessageId
                ReceiptHandle = $SQSMessage.ReceiptHandle
                UserName = $MessageBodyObject.UserName
                DisplayName = $MessageBodyObject.DisplayName
                DateTimeStamp = (Get-Date -Format yyyy-MM-dd) + " " + (Get-Date  Format HH:mm:ss)
            }

            $ReturnObject = New-Object PSObject -Property $props
        }

Now that we have our data from our SQS queue, we can perform the action requested:

Get-ADUser $ReturnObject.UserName-Properties Surname, GivenName, Initials | % {Set-ADUser -Identity $_ -DisplayName $ReturnObject.DisplayName}

If the task completes successfully, we may have to tell the original queue that the message we received is complete, and the SQS queue should remove it. To do this, we would call the Remove-SQSMessage function like this:

Remove-SQSMessage -QueueUrl $QueueUrl -ReceiptHandle $($ReturnObject.ReceiptHandle) -PassThru -Force -AccessKey $AccessKey ‑SecretKey $SecretKey -Region $Region

That's it! Amazon SQS queues allow you to design your applications or service infrastructure in a scalable manner. Whether you need to create an internal service, like a self-service Active Directory management tool, or a customer-facing scalable application, combining SQS and PowerShell is a possibility. This means developers and IT professionals can create scalable infrastructure and services using their favorite language, PowerShell!