I wanted to build an atomic counter to keep metrics of site visits and actions. Originally I started storing all the log messages and I was going to write some queries to summarise it. However, I decided that I didn’t need all the underlying data, at least not in this format, it’s stored elsewhere, so I wanted something which would just store the daily totals.
DynamoDB is great place to store this information. I decided to have a row per day with the date as the key. The counters would be attributes and as DynamoDB doesn’t tie you to particular attribute names I would be able to add new attributes whenever I wanted.
Lambda with an APIGateway front end was the obvious preference to do the work. The function would only trigger when I needed it to and there would be no extra services to maintain on my servers.
First of all the DynamoDB table to store the information. We’ll keep this simple and just create a table called AtomicCounter with a key called “Day” of type Number.
It can be hard to know where to start with building Lambda functions especially if you’re using CloudFormation (which you should be). I normally create a skeleton Lambda file, then build the CloudFormation template, edit the Lambda function inside Lambda, copy it back to the template file. It sounds a bit convoluted but it works.
First off an empty lambda function. Use this index.js file and place in a suitable folder.
'use strict';
/**
Lambda skeleton file
*
*/
exports.handler = function(event, context, callback) {
var response = {
"statusCode": 200,
"body": JSON.stringify("status": "success"),
"isBase64Encoded": false
};
callback(null, response);
}
Next we’ll build a CloudFormation template. The template below will commission an APIGateway and Lambda function together with an appropriate Role.
Note that we don’t include the DynamoDB table in the CloudFormation stack. This is because if you delete the stack this would also delete the database and we would lose all your data.
To run the template you first need to parse it to convert the serverless functionality into pure CloudFormation. This is what the second line in the template Transform: ‘AWS::Serverless-2016-10-31’ means. To do this run the command below.
aws cloudformation package \ --template-file LambdaAtomicCounter.yaml \ --output-template-file build/LambdaAtomicCounter-output.yaml \ --s3-bucket <NAME OF YOUR S3 BUCKET>
You will need to have installed and set up the AWS Command Line Interface (CLI). If you haven’t done that yet you can follow these steps (XXXX).
Once the template is complete you need to deploy it. This can be done using the command below:
aws --region eu-west-1 cloudformation deploy \ --template-file LambdaAtomicCounter-output.yaml \ --stack-name LambdaAtomicCounter \ --capabilities CAPABILITY_IAM
It can take a little bit of time for the stack to deploy but when it does you should see the resources available in whatever region you selected.
Obviously the function won’t do anything yet but you can now edit it either in Lambda itself or in your local system.
The Lambda function I’ve created is below.
'use strict';
const doc = require('dynamodb-doc');
const dynamo = new doc.DynamoDB();
/**
HTTP endpoint using API Gateway. Pass in a parameter e
which is the event name and a DynamoDB attribute.
The date can be overridden using an optional parameter c
which is the date in the format yyyyMMdd
*
*/
exports.handler = function(event, context, callback) {
var eventName;
var countersKey;
var responseBody;
var statusCode = 200
var requestPath = event.requestContext.path;
if (event.queryStringParameters && event.queryStringParameters.e && event.queryStringParameters.e !== "") {
console.log(event.queryStringParameters.e);
eventName = event.queryStringParameters.e.toUpperCase();
responseBody = {
"status": "success",
"eventkey": eventName
};
} else {
responseBody = {
"status": "Failure",
"eventkey": "No event parameter specified."
};
var response = {
"statusCode": 400,
"body": JSON.stringify(responseBody),
"isBase64Encoded": false
};
callback(null, response);
return;
}
if (event.queryStringParameters && event.queryStringParameters.m && event.queryStringParameters.c !== "") {
countersKey = event.queryStringParameters.c;
} else {
var dateObj = new Date();
countersKey = (dateObj.getUTCFullYear()*10000) + (dateObj.getUTCMonth() + 1) * 100 + dateObj.getDate();
}
console.log(countersKey);
var AWS = require("aws-sdk");
AWS.config.update({
region: "eu-west-1",
});
var dynamodb = new AWS.DynamoDB();
var tablePrefix = "TEST";
//Change to startsWith
if (requestPath == "/Prod/TheseChains") {
tablePrefix = "LIVE";
}
var countersTable = tablePrefix + "_cactCounters";
var docClient = new AWS.DynamoDB.DocumentClient();
var paramStr = "";
var params;
paramStr = "{\"TableName\":\"" + countersTable + "\"";
paramStr = paramStr + ",\"Key\":{\"Day\":" + countersKey + "}";
paramStr = paramStr + ",\"UpdateExpression\": \"ADD " + eventName + " :val\"";
paramStr = paramStr + ",\"ExpressionAttributeValues\":{\":val\":1}"
paramStr = paramStr + ",\"ReturnValues\":\"UPDATED_NEW\"";
paramStr = paramStr + "}";
console.log(paramStr);
params = JSON.parse(paramStr);
docClient.update(params, function(err, data) {
if (err) {
console.error("Unable to add item. Error JSON:", JSON.stringify(err, null, 2));
statusCode = 400;
responseBody = {
"status": "Failure",
"eventkey": "Unable to add item." + JSON.stringify(err, null, 2)
};
} else {
console.log("Added item:", JSON.stringify(data, null, 2));
}
});
var response = {
"statusCode": statusCode,
"body": JSON.stringify(responseBody),
"isBase64Encoded": false
};
callback(null, response);
}
The things to note are:
- We’re using the DynamoDB ADD function to store the record. This is a fantastic function as one command will update the item if it exists creating it if it doesn’t. It will do the same thing with the attribute as well, updating it if it exists or adding it if it doesn’t.
- The key is a date in the format yyyyMMdd. We’re only using one key so there’s no need for a SortKey
- You can send in an optional key using the parameter c
- The event name is read from a parameter e. This will increment the value in the corresponding attribute.
You can test the function using the built-in lambda tests or by triggering the API Gateway endpoint. When you deploy the CloudFormation template it will create two API Gateway endpoints, Prod and Stage. To find the endpoints
- Login to the API Gateway area in the AWS console
- Click on the LambdaAtomicCounter API
- Drill down to Stages and select the GET endpoint under Stage
- This will give you the url to invoke
The the url by pasting it in a browser and then appending a url parameter with the event name e.g. e=TEST
Once the url has been submitted you can go to DynamoDB and check that the event has been logged.
You can download the source code from GitHub at: https://github.com/HamishBuchanan/LambdaAtomicCounter