- Scalability and availability for Azure Web Apps - Wed, Jun 28 2017
- Managing Azure Virtual Machine scale sets - Wed, May 31 2017
- Azure virtual machine scale sets - Mon, May 8 2017
Before digging into the details of Azure Queue storage, let me give you a basic overview of why queue data structure is important in the development world.
Queue data structure
In programming languages, stacks and queues are two different data structures to manipulate large collections of objects. Think of these data structures as a collection of items. The main differences between these collections are the following principles.
Stack data structure follows the LIFO (last in, first out) principle. Using push operation, each new member or item will be placed on the top of the list and the pop operation will remove the most recent item from the list. As you may guess, items are removed from the list in reverse order from the order they were added.
FIFO principle – stack
Queues use the FIFO (first in, first out) principle. The first item added to the queue will be the first item removed from the queue. That means if you want to remove the most recent item, you need to remove all other elements that were added before it.
FIFO principle – queue
In queue data structure, there are two principle operations – enqueue and dequeue.
When you add a new member to the end of the queue, we call this an enqueue operation. When you remove an element from the head, we call this a dequeue operation.
Enqueue and dequeue operations
Why are queues important?
In today’s world, we have complex applications that may have many different tightly integrated components that exist on different endpoints on-premises, in the cloud or at other geo-locations. But your application may require some of these components to exchange data.
Let’s say we have an application in our hybrid infrastructure, and some parts of the application are on-premises while another front-end part of the application exists in the cloud. In such a heterogeneous environment, you need to find a way to exchange your data between different endpoints. Your data may be stored on an SQL Server, FTP or HTTP endpoint but at the end, the endpoints need to speak the same language.
Queues help in this scenario by acting as a shared layer to provide connectivity between different components. Having a queue as the “man in the middle” for different application components has many advantages:
- Queues decouple application components. Tightly integrated architecture makes it harder to scale up if you need to present additional components in the future. Queues act as a shared layer between components to exchange data so that it’s a lot easier to scale up each component without changing the actual configuration.
- In a typical architecture, if one of your application’s components fail, then you lose communication between the components. But a queue keeps exchanged data until it has been processed at the destination. That also provides delivery guarantee for messages.
- Queue storage also delivers asynchronous messaging for different endpoints, which allows you to keep messages in the queue.
Azure Queue storage architecture
As I mentioned numerous times in this series, an Azure storage account is the main logical group to host different storage services.
Azure Queue storage architecture
Under the storage account, you can have queues and messages. A queue is a logical group to host the message and can be retrieved by client applications. Messages are 64 KB UTF-8 encoded texts. Each queue can host an unlimited number of messages, and any client application that has access to the storage account can access those messages.
Accessing the queue and messages
A message in a queue is nothing but a small XML file that can be accessed through a REST API. You can simply follow the API URI structure to access the queue or create/update messages within the queue.
In order to access the resource itself, you need to use a base URI.
Queue endpoint URI
https://storageaccountanil1.queue.core.windows.net is the base URI under the storage account. In order to access the queue itself, you need to use Base URI + QueueName. All letters in a queue must be lowercase and 3–63 characters long:
https://storageaccountanil1.queue.core.windows.net/myfirstqueue
To access the messages in a particular queue, use the following URI syntax:
https://storageaccountanil1.queue.core.windows.net/myfirstqueue/mymessage01
You can also use GET, PUT, HEAD and DELETE HTTP operations in order to update or delete individual messages in a queue.
Using PowerShell to manage queues and messages
There are two different ways that you can interact with queues using PowerShell. One is to use native Azure Storage commands, and the second is to invoke a REST API through the PowerShell Invoke-RestMethod cmdlet.
Native PowerShell commands
As you may remember from my previous article, we can create a storage account context and then use this context as a parameter for storage-related cmdlets.
$context = New-AzureStorageContext -StorageAccountName "storageaccountanil1" -StorageAccountKey "6nVoPw3yzEEbAEmboxtZ4L4zhg7/ADeZ3N/” New-AzureStorageQueue –Name "myfirstqueue" -Context $context
The New-AzureStorageQueue command creates a new queue named “myfirstqueue” under the specified storage account. To retrieve existing queues, Get-AzureStorageQueue will help you:
Get-AzureStorageQueue –Name "myfirstqueue" –Context $context
Retrieving queues
When it comes to inserting or removing messages from queues, there is no native PowerShell cmdlet and you need to rely on existing .NET classes just like we did before for Azure Table storage. There are a couple of important classes to be aware of:
Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage: This class represents a particular message in Queue Service, and you can use the AddMessage method to add a new message to a queue.
Adding a message to a queue
Using the Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage class, I added a new message (NewMessage01) to my queue. You can see the message count in the ApproximateMessageCount property in the output.
To retrieve a message from the queue, you can use the CloudQueue.GetMessage method. This method also includes a parameter called visibilityTimeout. That’s a timespan that specifies the visibility timeout interval. When you use the CloudQueue.GetMessage method to retrieve a message, that message will be marked as invisible for the amount of time you specified.
Retrieving messages
I specified “5” as the visibilityTimout, which makes GetMessage invisible to code. Therefore, if I run the same code within the 5 seconds repeatedly, I will get no response. After the timeout I can retrieve the message again.
visibilityTimeout parameter
The next thing would be to dequeue a message. The DequeueCount value will be incremented each time a message is dequeued (removed). After using the CloudQueue.GetMessage method to receive existing messages, you can use the CloudQueue.DeleteMessage method to remove them.
Removing a message from a queue
REST API
You can also use the Invoke-RestMethod cmdlet to call HTTP REST APIs. For more information about REST API URI and syntax, please refer to Microsoft's documentation.
A Put Message operation adds a new message to the queue. Just like we did with PowerShell and .NET classes, you can also specify a visibility timeout to make the message invisible for a period of time.
The URI syntax for a Put Message operation is as follows:
https://storageaccountanil1.queue.core.windows.net/myfirstqueue/messages?visibilitytimeout=&messagettl=
Two mandatory headers should be specified in this HTTP PUT request: Authorization and Date.
For the Authorization header you need to create a Shared Access Signature for delegated access to authenticate the request. Also, the Date Header should be no older than 15 minutes by the time it reaches the service.
You can create a variable in PowerShell to keep authorization and date headers.
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", 'xxxxxxxx')
$headers.Add("x-ms-date", 'xxxxxxxx')
Then you need to specify the -header parameter for Invoke-RestMethod.
We also need to use a -Body parameter to specify body in JSON format. For the body, you can build structure as follow:
$body=@{
QueueMessage = @(
@{
MessageText = "MyCustomPSMessage01"
}
)
}
And then you need convert it to JSON in order to use it as a value for the -body parameter.
$json = $body | ConvertTo-Json
PowerShell is not an application development language. Thus, you probably need to use tools like Visual Studio and frameworks like .NET to play with queues. In this article I just wanted to show you currently available PowerShell options that allow you to interact with the Azure Queue service.
In the next part of my Azure Storage series we are going to discuss Premium Storage benefits.
