Configuring APIGateway VTL to store JSON object directly on S3 —With Lambda authorizer enabled

Published in

AWS in Plain English

5 min readJan 20, 2022

When building a serverless application, it is often a misconception to use lambda as a middle layer for almost all the use-cases. Read when not to use a Lambda function guide by AWS to check out some examples. In this article, we focus on one of those not to use configurations called Velocity Template Language (VTL).

Remember to always keep in mind the words of wisdom from the AWS Serverless Product manager before we move forward:

Use Lambda function to transform the data, NOT to transport it

This phrase completely transformed the way our serverless application at Propellor.ai works. We leveraged API gateway’s VTL operations in as many use-cases as possible to reduce clutter and completely avoid lambda function.

Instead of going theoretical, let me explain one of our use-cases and walk you through the end-to-end configuration with screenshots in the next section. Here is a flow diagram of the usecase:

TL;DR We have a deeply nested large JSON object which is a combination of customers choice of analytics segment. This JSON is populated by various selections made by the customer on the dataset, filters applied, properties configured, schema types, title and description, identity details, metadata about the cohort, etc. Since the JSON is clientside rendered with numerous bits and pieces joined together the size and depth of data significantly increases as we support multiple versions and aggregate data results.

A typical JSON at the time of saving would be around

✓ ~20KB in size

✓ 9 parent keys with at least 3 levels of nesting in each

✓ have aggregated data points results at each level

Using VTL, JSON can be transferred directly to S3, with no middle man.

In a nutshell, Its a POST method API with body saved as is in a S3 bucket (with prefix) retrieved from the JSON Object

Prerequisites

Before we start, make sure your policy with the Trust relationship is configured as below.

create an IAM policy called with the following policy attached. In our case, we called it S3ReadPut-APIGW-Role and the policy attached to this role is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*"
            ],
            "Resource": "*"
        }
    ]
}

2. When creating role via console, APIGateway may not allow to find the policy to attach directly (might be AWS UI issue). In that case, create a Lambda role and make sure to change the trust relationship to the role as mentioned in the screenshot below:

Click on “Edit trust relationship” and update with the following policy.

NOTE: This step is NOT required if you are able to create APIGateway Role directly with the S3ReadPut-APIGW-Policy attached

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

3. S3 bucket should be in the same region where your API is deployed.

Step 1: Configure API and Lambda Authorizer

Create a new resource with the desired name and create a POST method. Although, we want to PUT the JSON as an object on S3 our requirement here is to share the body of an API via the POST method → fetch fields from it → use them to store the body as a JSON object. Refer to the screenshot below to configure the same

Step 2: Setup the API Resource and Method

In “Path override” I am using two keys that I receive from the frontend as prefix paths to store my final result.

Step 3: Configure the mapping template

Configure requestOverride to capture the specific event from the body to use them as parameters for your use-case. in our case, it's the S3 prefix and name of the JSON object.

$Util is a very useful package to parse JSON files and other expression related operations provided by APIGateway.

This version 1 of our VTL mapping templates serves the job well, hence we used it as is. If anyone has suggestions to improve this VTL, please comment below.

#set($body = $util.parseJson($input.body))
#set($context.requestOverride.path.id = $util.parseJson($input.json('$.id')))
#set($context.requestOverride.path.second_id = $util.parseJson($input.json('$.second_id')))
{
"body": $input.json('$')
}

VTL can be used with both GET and POST methods:

## use parseJson to get events from the RequestBody of a POST request#set($body = $util.parseJson($input.body))
#set($context.requestOverride.path.id = $util.parseJson($input.json('$.id')))## use input.params to get events from the query params of a GET request#set($context.requestOverride.path.id = $input.params("id"))
#set($context.requestOverride.path.second_id = $input.params("second_id"))

To learn more about API Gateway mapping templates, check out the link below.

API Gateway mapping template and access logging variable reference

context.authorizer.claims.property A property of the claims returned from the Amazon Cognito user pool after the method…

docs.aws.amazon.com

Bonus: API response messages

Testing results

In the screenshot below, you can find a 1086 lines JSON as body posted to API. This body is converted to a JSON file and dumped into an S3 bucket with the filename and prefix fetched from the body key-value pair. (S3 screenshot is not attached, as it will be all blurred 😬)