- Using AWS Lambda functions with Docker containers - Fri, May 12 2023
- A Go AWS SDK example - Fri, Nov 11 2022
- Getting started with Jenkins - Tue, Aug 16 2022
Creating a lifecycle rule is a good way to automate the housekeeping of your S3 bucket. By using lifecycle rules, you can remove objects after so many days, by a certain date, or use Amazon's S3 Glacier for archiving. It's worth noting that you can set lifecycle rules by bucket, prefix, or tags, and you can also set them for current and non-current (previous) versions. I will demonstrate a simple example of removing all objects by a specified date.
Creating a bucket and configuring the lifecycle rule
I will be creating the lifecycle rule with PowerShell; however, it is possible to do this through Amazon's GUI interface. I will start by creating a new S3 bucket with PowerShell:
New-S3Bucket -BucketName demo-life-cycle
Building the rule with PowerShell requires you to make use of AWS classes. In this example, I won't be looking to move any objects to S3 Glacier. To do this, you would need to use the Amazon.S3.Model.LifecycleTransition class. If this is an area you want to look at, please review Amazon's documentation.
Set an expiration date through the Amazon.S3.Model.LifecycleRuleExpiration class. To do this, create an instance of the class:
$Expiration = [Amazon.S3.Model.LifecycleRuleExpiration]::new()
To mark a deletion by a number of days, use the day property. Use the Date property to set a date for objects to remove:
$Expiration.Date = [DateTime]::new(2018,12,28)
After setting the expiration part, initiate an instance of the Amazon.S3.Model.LifecycleRule class:
$LifecycleRule = [Amazon.S3.Model.LifecycleRule]::new()
Using Get-Member allows us to view the properties available to us to configure our rules:
$LifecycleRule | Get-Member -MemberType Properties
To get more details on these properties and their uses, look at the AWS documentation.
Now we can attach the expiration rule:
$LifecycleRule.Expiration = $Expiration
To complete the creation of the lifecycle rule, change the status to Enabled. Also, if you don't add anything to the Prefix rule, it will fail.
$LifecycleRule.Prefix = $null$LifecycleRule.Status = 'Enabled'
Applying a lifecycle rule to an S3 bucket
With the configuration lifecycle rule complete, it's time to add to our S3 bucket. We do this with the Write-S3LifecycleConfiguration cmdlet:
Get-S3Bucket -BucketName demo-life-cycle | Write-S3LifecycleConfiguration -Configuration_Rules $LifecycleRule
We can pipe Get-S3Bucket to Write-S3LifecycleConfiguration, and with the Configuration_Rules parameter, we can add our object containing the lifecycle rules. The S3 bucket now has a lifecycle rule set up and enabled.
Viewing lifecycle rules on S3 buckets
We've configured and added the rules to our S3 bucket. But how can we check the bucket has a rule configured or see what rules are set? Amazon provides the Get-S3LifecycleConfiguration cmdlet to do this. Get-S3LifecycleConfiguration will return a property named Rules, which you will need to expand to view lifecycle rules:
Get-S3Bucket -BucketName demo-life-cycle | Get-S3LifecycleConfiguration | Select-Object -ExpandProperty Rules
With additional rules defined (in this case, we have the expiration rule), we must expand this property also. To help view all rules defined, I have created a function called Get-BucketLifeCycle. The function will display a custom object output, like so:
You can pipe or add the bucket name as a parameter to view any lifecycle rules quickly.
Summary
Amazon provides an excellent way to manage your objects with lifecycle rules. Should you wish to have a monthly removal process or move to S3 Glacier to archive and keep costs down, you have a way to automate this. You can do this programmatically with PowerShell or through the GUI.
All of my code in this article, including the Get-BucketLifeCycle function, is on my GitHub page.
Subscribe to 4sysops newsletter!
function Get-BucketLifeCycle { [cmdletbinding(DefaultParameterSetName = 'FromPipeline')] param ( [Parameter(Mandatory, Position = 0, ValueFromPipeline, ParameterSetName = 'FromPipeline')] [Amazon.S3.Model.S3Bucket] $S3Bucket, [Parameter(Mandatory, Position = 0, ParameterSetName = 'ByBucketString')] [String] $BucketByString ) begin { if ($pscmdlet.ParameterSetName -eq 'ByBucketString') { $S3Bucket = Get-S3Bucket -BucketName $BucketByString } } process { $bucketRules = $S3Bucket | Get-S3LifecycleConfiguration foreach ($bucketRule in $bucketRules.Rules) { [PSCustomObject]@{ RuleName = $bucketRule.Id PrefixPath = $bucketRule.Filter.LifecycleFilterPredicate.Prefix DateTimeExpire = $bucketRule.Expiration.Date DaysUntilArchieve = $bucketRule.Expiration.Days NoncurrentExpire = $bucketRule.NoncurrentVersionExpiration.NoncurrentDays Transitions = $bucketRule.Transitions.Transitions StatusOfRule = $bucketRule.Status.Value } } } }