The Limitless CloudFormation Stack with Lambda-backed Resources

Photo by Lex Photography from Pexels

Just recently, I started experimenting with AWS Chatbot and its capabilities. Integration was pretty simple and I quickly managed to track my pipelines. The next thing I wanted to achieve is getting Slack notifications if one of my serverless application’s regions is unavailable, because that’s something which requires immediate action.

So I needed to create a CloudWatch alarm that monitors my Route53 health checks which have actions to send SNS notification if the region’s health changed.

As I did that, I intentionally bricked one of my regions just to notice that the CloudWatch alert simply did nothing —the metric at all didn’t even show any gathered metrics. StackOverflow quickly helped me to identify that I needed to create my alarm in us-east-1, because that’s the only region that is able to access Route53 metrics.

As I’m using the Serverless Framework and set up my Route53 health checks via CloudFormation resources, I was stuck now, because CloudFormation is not able to create resources in another region. Yeah, I could just do this with Terraform, but I was also sure, that there must be a way to get this integrated into my Serverless-managed CloudFormation stack. I’m not giving up easily.

There we go: AWS Lambda-backed custom CloudFormation resources! 🎉

Me, finding out about Lambda-backed CloudFormation resources

Being Limitless

Reading through the documentation, for me it felt like finding some holy grail. This basically frees you from all of CloudFormation’s limits like maximum template size or the fact that you can’t create resources in another region.

The quintessence: Lambda functions which are creating your custom resources in behalf of CloudFormation. That resolves to: you can create any resource, anywhere because you’ve got full access to the AWS API.

How does it work?

• CloudFormation triggers your Lambda function with parameters you define and a specific event (Create, Update or Delete) and a signed S3 URL.
• Your Lambda function creates (or updates or deletes) your custom resource(s).
• You notify CloudFormation about the successful (or unsuccessful) resource operation(s) by posting your results to the S3 URL.

Example for Lambda-backed resource creation with CloudFormation

Integrating it

Let’s recall our initial target: creating a CloudWatch Alarm in us-east-1, even though the stack resides in eu-central-1.

Let’s add the infra for our Lambda function that manages our Alarm:

functions:
  # [...]
  route53Alarms:
    package:
      include:
        - create-route-53-alarms.js
    handler: create-route-53-alarms.handler
    name: create-route-53-alarms
    reservedConcurrency: 1
    runtime: nodejs12.x
    memorySize: 1024
    timeout: 120

Then we add the custom CloudFormation resource:

resources:
  Resources:
    StatusHealthCheckAlarm:
      Type: 'Custom::LambdaBackedRoute53Alert'
      Properties:
        ServiceToken: !GetAtt Route53AlarmsLambdaFunction.Arn
        topicArn: 
        healthCheckId: !Ref RegionHealthCheck

In my example I’m passing the identifier of the Route53 health check I want to track via the Alarm. The ServiceToken has to be the ARN of the Lambda function that should be invoked for managing the resource. We can reference this via !GettAtt.

And that’s all for the infrastructure side already.

The last step is to write the actual code for our function.

💡 There’s a dependency for Lambda-backed resources with Python, where you only need to define methods for all event types and annotate them accordingly. As I love working with Node.js and I wanted to really understand whats going on, I didn’t make us of this.

const AWS   = require('aws-sdk')
const https = require('https')
const url   = require('url')

const cloudwatch = new AWS.CloudWatch({ region: 'us-east-1' })

exports.handler = async (event, context) => {
  try {
    switch (event.RequestType) {
      case 'Create': await createHealthCheck(event); break
      case 'Update': await updateHealthCheck(event); break
      case 'Delete': await deleteHealthCheck(event);
    }
    await sendResponse(event, context, 'SUCCESS')
  } catch (e) {
    console.log(e)
    await sendResponse(event, context, 'FAILURE')
  }
}

async function createHealthCheck(event) {
  await cloudwatch.putMetricAlarm({
    AlarmName: `region-unavailable`,
    ActionsEnabled: true,
    AlarmActions: [event.ResourceProperties.topicArn],
    OKActions: [event.ResourceProperties.topicArn],
    ComparisonOperator: 'LessThanThreshold',
    EvaluationPeriods: 1,
    MetricName: `HealthCheckStatus`,
    Namespace: 'AWS/Route53',
    Period: 60,
    Statistic: 'Minimum',
    Threshold: 1.0,
    Dimensions: [{
      Name: 'HealthCheckId',
      Value: event.ResourceProperties.healthCheckId,
    }]
  }).promise()
}

async function updateHealthCheck(event) {
  await deleteHealthCheck(event)
  await createHealthCheck(event)
}

async function deleteHealthCheck(event) {
  await cloudwatch.deleteAlarms({
    AlarmNames: [`region-unavailable}`]
  }).promise()
}

const sendResponse = (event, context, responseStatus) => {
  return new Promise((resolve, reject) => {
    const responseBody = JSON.stringify({
      Status: responseStatus,
      PhysicalResourceId: `${event.StackId}-${event.LogicalResourceId}`,
      StackId: event.StackId,
      RequestId: event.RequestId,
      LogicalResourceId: event.LogicalResourceId,
      Data: {},
    })

    const parsedUrl = url.parse(event.ResponseURL)
    const options = {
      hostname: parsedUrl.hostname,
      port: 443,
      path: parsedUrl.path,
      method: 'PUT',
      headers: {
        'content-type': '',
        'content-length': responseBody.length,
      },
    }
    const request = https.request(options, function(response) {
      console.log(response.statusCode)
      console.log(JSON.stringify(response.headers))
      context.done()
      resolve()
    })
    request.on('error', function(error) {
      console.error(error)
      context.done()
      reject()
    })
    request.write(responseBody)
    request.end()
  })
}

And that’s all we have to do. The next deployment via Serverless will trigger the CloudFormation stack update and therefore our function with the event type Create. If the CloudWatch Alarm can be successfully created, we’re passing the resource details to S3 including the SUCCESS status, which will be picked up by CloudFormation.

💡 Be sure to always submit a response to the signed S3 URL! CloudFormation has a really long timeout for waiting that you’ve submitted your results of the custom resource management. That’s why I’m always putting everything into a try/catch clause. If something goes horribly wrong, at least the status update with FAILURE will be submitted. Else you have to wait for the timeout of CloudFormation which is 1 hour!

If you’re changing any parameters for the custom resource, CloudFormation will notice and send your Lambda function the Update event. And finally — when you’re deleting your stack — CloudFormation will send the Delete event.

There’s more…

I’m writing a book on AWS fundamentals to help others learn about the core building blocks. The focus is on learning for the real world, not only for certifications! 📚
You can subscribe to the newsletter at awsfundamentals.com to get bite-sized preview content & other lessons free to your inbox every two weeks.

More content at plainenglish.io