Go by Example: Download File from S3

Go 1.23

Download files from Amazon S3 using the AWS SDK for Go v2. This example shows how to use the `s3/manager` Downloader to efficiently download objects to a local file.

Code

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    // Load AWS config
    cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-east-1"))
    if err != nil {
        log.Fatalf("unable to load SDK config, %v", err)
    }

    // Create an S3 client
    client := s3.NewFromConfig(cfg)

    // Create a downloader
    downloader := manager.NewDownloader(client)

    // Create a file to write the downloaded content to
    file, err := os.Create("downloaded.txt")
    if err != nil {
        log.Fatalf("failed to create file, %v", err)
    }
    defer file.Close()

    // Download the file
    numBytes, err := downloader.Download(context.TODO(), file, &s3.GetObjectInput{
        Bucket: aws.String("my-bucket"),
        Key:    aws.String("uploads/test.txt"),
    })
    if err != nil {
        log.Fatalf("failed to download file, %v", err)
    }

    fmt.Printf("Downloaded %d bytes\n", numBytes)
}

Explanation

Downloading objects from Amazon S3 can be optimized using the feature/s3/manager package's Downloader utility. Similar to the Uploader, the Downloader is designed to handle large files efficiently by downloading multiple parts of the file concurrently. This significantly reduces the total download time and improves resilience against network interruptions.

The Downloader works by taking an io.WriterAt interface as the destination for the data. This allows it to write different chunks of the file to the correct positions in the output stream as they arrive, regardless of the order. The standard os.File type in Go implements io.WriterAt, making it straightforward to download directly to a file on disk.

To implement a download, you initialize the S3 client and create a Downloader instance. You then call the Download method, passing the destination file and an s3.GetObjectInput struct specifying the bucket and object key. The method returns the total number of bytes downloaded upon completion, or an error if the operation fails.

Concurrent Downloads: Splits large objects into chunks and downloads them in parallel.
WriterAt Interface: Writes data chunks to specific offsets, enabling out-of-order downloading.
Simplified API: Abstracts away the complexity of Range headers and multipart assembly.

Code Breakdown

26

manager.NewDownloader(client) initializes the Downloader. Like the Uploader, it can be configured with options for part size and concurrency.

29

os.Create creates (or truncates) the local file. The returned *os.File implements io.WriterAt, which is required by the Downloader.

36

downloader.Download executes the download. The first argument is the context, the second is the destination writer (file), and the third is the S3 input parameters.

44

The function returns the number of bytes written. This is useful for verifying that the download completed successfully and matches the expected size.

Samplebadu

Go by Example: Download File from S3

Code

Explanation

Code Breakdown