A Go AWS SDK example

• Author
• Recent Posts

Graham Beer

Graham is an experienced system engineer with a passion for PowerShell and automation. You can follow his blog or reach him on Twitter at @GKBeer.

Latest posts by Graham Beer (see all)

• How to create and use Adaptive Cards - Thu, Oct 5 2023
• Using AWS Lambda functions with Docker containers - Fri, May 12 2023
• A Go AWS SDK example - Fri, Nov 11 2022

What is the AWS SDK?

The Go AWS SDK, which is now on version 2, provides APIs and utilities to help build your Go applications within AWS Services. The SDK removes low-level complexity by providing error handling and authentication with your AWS account. The SDK lists all the services available along with their methods and parameters.

The SDK provides access to the AWS services, which you can use with a CI/CD pipeline or within your own applications. Although the AWS CLI is available, you will find the use of SDK a stronger alternative because it gives you a way to document your code. The code can then be checked into version control, such as git, allowing you to check changes. Using the SDK also gives you the flexibility to write your own custom utilities.

Install the AWS SDK for Go v2

The AWS SDK for Go v2 requires Go version 1.15 or higher. If you have an installed version of Go on your system, you can check the version below:

go version

You can download Go from the official site. Once Go is up and running, we are ready to initiate our local project. Open a terminal, and create a new directory:

mkdir new_s3_bucket && cd new_s3_bucket

We can now initialize our Go project:

go mod init AwsBucket

We are ready to retrieve the SDK now that these dependencies can be recorded in our go.mod file. We will start by adding a standard set of SDK modules to our application. Type in the following to your terminal:

go get github.com/aws/aws-sdk-go-v2
go get github.com/aws/aws-sdk-go-v2/config

The first module is the core of the AWS SDK, while the second is used for the AWS shared configuration. The next modules you would install would be the services you wish to interact with. These could be EC2, S3, Lambda, or many more. If you look at the AWS SDK GitHub resource, you will see the list of services available.

Access to AWS

To access AWS programmatically from your local machine, you are required to use access keys. If this is not something you've done before, you will need to head over to your AWS console. If you already know how to do this, feel free to skip to the Building with the SDK section.

Navigate to the Identity and Access Management (IAM) dashboard from within your console. Under Users, locate and click your user name. To generate your access keys, open the Security Credentials tab. The screen visible to you will look like the one shown below.

IAM user security credentials

Under Access Keys, click the Create access key button. This will generate your access keys, but make sure you copy the secret key. This is the only time you will be able to view the secret keys from the console. Should you lose the secret key, you will be required to generate a new access key.

To configure your access keys, you will need to install the AWS CLI. If you don't have it, follow these instructions from the AWS documentation.

With the AWS CLI installed, type aws configure. You will be prompted for your access and secret keys along with your region. The information is then stored in /.aws/credentials on the file system. The credentials file can be edited and items added to it.

You are now ready to get started.

Creating an S3 bucket

We are going to create an S3 bucket using Go SDK version 2. As we develop our S3 bucket, we will go through the code and explain the steps taken.

From the new_s3_bucket directory we created above, we need to import the S3 SDK module. This is done in the same way in which we imported the standard SDK modules. From the terminal, type:

go get github.com/aws/aws-sdk-go-v2/service/s3

With the modules in place, we can start building out our code. Create an empty main.go file. Start by adding the following code to your main.go file:

package main

import (
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
}

You will start each new project by using the config package to load the configuration from local environment variables, the AWS shared configuration file (~/.aws/config), and the AWS shared credentials file (~/.aws/credentials).

Our first line will call the LoadDefaultConfig method from the config module.

cfg, err := config.LoadDefaultConfig(context.TODO())

The method takes two parameter values: Go's context module from the Go standard library and something called a variadic function, which in computer programming stands for a function that accepts a variable number of arguments. These additional arguments include various options, such as defining the region and credentials. For details on the options, see here.

Passing context.TODO passes the function with a blank context. If you have populated your ~/.aws/config file with the region and have a default profile in the ~/.aws/credentials file, you don't need to add anything for the second parameter value.

However, let's say you have two profiles in your ~/.aws/credentials file and it looks something like this:

[default]
aws_access_key_id = <YOUR_DEFAULT_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_DEFAULT_SECRET_ACCESS_KEY>

[test-account]
aws_access_key_id = <YOUR_TEST_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_TEST_SECRET_ACCESS_KEY>

If you want to specify the test-account profile and a different region, then you can write the LoadDefaultConfig method like this:

cfg, err := config.LoadDefaultConfig(
	context.TODO(),
	config.WithRegion("eu-west-1"),
	config.WithSharedConfigProfile("test-account"),
)

After setting up our AWS environment in our Go file, a connection to the S3 API needs to be made to create the S3 bucket. Earlier, we imported the SDK s3 module. Using this, a connection to the API client can be made with the NewFromConfig function. The function returns a new client from the provided config.

This is done by passing the output of the LoadDefaultConfig method to the NewFromConfig method:

client := s3.NewFromConfig(cfg)

Now, we have a connection to make operations calls to Amazon's Simple Storage Service.

By referring to the Go AWS SDK v2 documentation on S3, we can see that the CreateBucket function is what we require:

AWS Go SDK CreateBucket function header

Clicking on the params type *CreateBucketInput provides a list of arguments that we can use to create our S3 bucket.

Now that we know the method call and the type of arguments to build our S3 bucket, we can start putting it all together. In our example, we will pass three input values to create the bucket: a name for the bucket, an access control list (ACL), and a configuration that takes a location constraint.

The ACL uses another external module, named types. This allows us to make our bucket private. Other types are available. To install, add "github.com/aws/aws-sdk-go-v2/service/s3/types" to your list of imports, and run go get github.com/aws/aws-sdk-go-v2/service/s3/types to make it available.

bucket, err := client.CreateBucket(context.TODO(), &s3.CreateBucketInput{
		Bucket: aws.String("aws-s3-bucket-test-001"),
		ACL:    types.BucketCannedACLPrivate,
		CreateBucketConfiguration: &types.CreateBucketConfiguration{
			LocationConstraint: types.BucketLocationConstraintEuWest1,
		},
	})

The bucket creation is wrapped into a function named News3Bucket.
To make this little project a bit more interesting, we will add to our bucket creation by creating a text file and uploading it to our newly created bucket.

Keeping with the S3 theme, we will create a function to upload the file. The function will use the Go standard library by using the OS package. The OS package uses the Open method to open the named file for reading. The file is then passed to the PutObject method from the Go AWS SDK v2 S3 module. The snippet of code looks like this:

f, err := os.Open(localfile.Name())
if err != nil {
	panic("Couldn't open local file")
}
client := s3.NewFromConfig(cfg)

upload, err := client.PutObject(context.TODO(), &s3.PutObjectInput{
	Bucket: aws.String(bn),
	Key:    aws.String(localfile.Name()),
	Body:   f,
}) 

The local file opened has its name passed to the key property, while the actual file is passed to the Body property. The function, named UploadFileToS3, takes the bucket name created from our News3Bucket function.

Finally, our project contains a function, CreateLocalFile, to create the local file to upload to the S3 bucket. The function again uses the Go OS package to create a file and write some text to it before closing.

Before running the program, here is our main.go file:

const file = "testfile.txt"
const content = "This is our content file for 4sysops!"
const bucketName = "aws-s3-bucket-test-001"

func main() {
	// Create local file to upload to newly created bucket
	localFile, err := CreateLocalFile(file, content)
	if err != nil {
		log.Fatal(err)
	}

	// Load the Shared AWS Configuration (~/.aws/config)
	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithRegion("eu-west-1"),
		config.WithSharedConfigProfile("sgnadmin"))
	if err != nil {
		log.Fatal(err)
	}

	News3Bucket(cfg, bucketName)
	UploadFileToS3(cfg, bucketName, localFile)
}

At the top, we define three constants: the file name, the content to add to our file, and the bucket name. Notice we then go on to call our three functions: CreateLocalFile, News3Bucket, and UploadFileToS3. The two files containing our functions are located at the root of the project.

From the terminal, type go run.

Creating the bucket from the terminal

Open the AWS console and navigate to S3. Locate the bucket name and look at its contents.

S3 bucket from the Amazon console

We can now access our newly created bucket and see that our file has been uploaded. Let's open the file and check its contents.

Test file created by the CreateLocalFile function

The full code is available in the GitHub repository.

Summary

You should now have a good understanding of what the AWS SDK is and how to use version 2 in the Go language. I've included various references and links to the Go documentation to highlight where to get more detailed information when it comes to building your own programs in the Go SDK. All the SDKs, which are available in many languages, give you another option to build within AWS. I hope I have inspired you all to take a look and try building something new in AWS.