DynamoDB at 360°: Key Concepts and Best Practices for Modeling NoSQL Data on AWS

Where to start modeling non-SQL data in DynamoDB. Understand what it is, how it works and how to adapt it to your projects to take advantage of AWS.

Much of my daily work is done in the cloud, my favorite platform is Amazon Web Services and I have experimented with a lot of cloud resources. But one of the ones I use the most is DynamoDB.

And I would like to share with you the considerations that I believe should be taken before you get down to business.

Well, to start from the most basic, let's see what dynamo is and what it offers us.

DynamoDB is a NoSQL engine (not a database) owned by Amazon that focuses on performance, since by default it replicates the data of a table in at least 3 availability zones in the region where it is created. This is where the high availability it is known for comes from. If you're not familiar with concepts, regions, and availability zones, I recommend this documentation: https://aws.amazon.com/es/about-aws/global-infrastructure

The type of replication that Dynamo uses and that solves the problem of data update latency is known as leaderless replication . To learn more about this concept: https://distributed-computing-musings.com/2022/01/replication-introducing-leaderless-replication

This is also known as the data inconsistency problem, in dynamo we have 2 concepts to talk about consistency:

Consistent Eventually Reads.

Between writing and reading a data, there will be a slight latency, so consulting a recently written data could fall into an availability zone that did not have the last update. The official documentation does not provide an exact delay time, but it is almost imperceptible unless it is - for example - the score of a game in real time.

Strongly Consistent Reads

This consistency is what will always respond to queries with the latest updated data, without exception. But... under certain conditions, and that's their disadvantage.

• A strongly consistent reading may not be available if there is a delay or network interruption. In this case, DynamoDB may return a server error (HTTP 500).
• Strongly consistent readings are not compatible with global secondary indices.
• Strongly consistent readings use more performance capacity that the eventually consistent readings. This last fact leads us to our following concept: the read/write ability.

⚠️ DynamoDB uses Consistent Eventually Reads, unless otherwise specified. Reading operations(such as GetItem, Query and Scan) provide a parameter ConsistentRead. If you set this parameter to true, DynamoDB uses Strongly Consistent Reads.

A common mistake is to use Scan instead of Query to filter. What should be avoided whenever possible, since Scan you have to go through the entire table, when you have few elements it's irrelevant, but be careful when you're working with hundreds of elements.

‍

Among the questions you should always ask yourself before creating a table in DynamoDB are the following:

• What is the approach to data?
• What is the average size of the data?
• How often do I have to read the data? Do I need more ability, reading or writing? Both equally?
• How fast do I want to read that data?
• How fast do I want to write data?
• What kind of Dynamo board do I need? This is easily answered, since there are two options, so are we going to access the data very or infrequently?

Regarding the size of the data and what speed/ability to read and write you need. You must understand how they work capacity units and how to choose them correctly. This will define how quickly you can write and read data from a table.

To define the reading and writing ability you'll need, you must first define that reading/reading ability mode you need.

El modus is the one that defines how read/write performance will be charged.

There are two modes available:

• On Demand: It means that dynamo will automatically scale every time the traffic reading and/or writing of the table fluctuates.
• Use this mode when you don't know what workloads you're facing or when they're unpredictable.
• Provisioned: It means that you will specify the exact capacity units, or that by activating the Auto ScalingYou can choose a limit of Minimum and maximum capacity units in writing and reading.
• Use this mode when you know the size of the data and when you know if it varies and how much it varies. Also when you have metrics about how often you will write or read that data.
• If you want to use this mode, but you're not sure what capacity units, You can in the first instance put in mode On Demand or use the Auto Scaling Of the way Provisioned and use a measurement metric in Cloudwatch to find out how many units of reading and writing you need. Example:

{
    "MyTableReadCapacityUnitsLimitAlarm": {
      "Type": "AWS::CloudWatch::Alarm",
      "Properties": {
        "AlarmName": "myTable-ConsumedReadCapacity",
        "AlarmDescription": "Alarm when read capacity reaches 80% of my provisioned read capacity",
        "AlarmActions": [{ "Ref": "AlarmEmailNotificationSnsTopic" }],
        "Namespace": "AWS/DynamoDB",
        "MetricName": "ConsumedReadCapacityUnits",
        "Statistic": "Sum",
        "Period": "60",
        "Dimensions": [
          {
            "Name": "MyTableName",
            "Value": "arn:aws:dynamodb:{region}:{IdAccount}:table/MyTable"
          }
        ],
        "EvaluationPeriods": "1",
        "ComparisonOperator": "GreaterThanOrEqualToThreshold",
        "Threshold": "240",
        "Unit" : "Count"
      }
    }
}

Para mas casos y ejemplos: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/creating-alarms.html

If you don't know how to implement metrics, alarms and monitoring in Cloudwatch, take a look at this post: https://www.kranio.io/blog/metricas-y-alarmas-de-recursos-aws

Dato: You can change the reading/reading ability mode every 24hrs on a board.

Okay, but up to this point we don't know how to define a unit of capacity, so let's get to it.

For the tables on demand, reading units are defined as follows:

• A reading request Strongly Consistent From to 4 KB requires a unity of reading request.
• A reading request Consistent eventually From to 4 KB Requires average unity of reading request.
• A reading request transactional From to 4 KB Requires two units of reading request.

For the provisional tables, reading units are defined as follows:

RCU [reading capacity unit]

• 1 RCU is equivalent to a high consistency reading per second for an element up to 4 KB
• 1 RCU is equivalent to two possibly consistent reads per second for an element of up to 4 KB
• 2 RCU is equivalent to a reading transactional per second for items up to 4 KB

For example: Imagine that you have an 8 KB .xml file

If you want a strongly consistent, you need two RCUs.

If you want a consistent eventually you need an RCU.

If you want a transactional you need 4 RCUs.

WCU [write capacity unit]

• 1 WCU is equivalent to one write per second for an element up to 1 KB
• 2 WCU is equivalent to a writing transactional per second for items up to 1 KB.

For example: You have your 8 KB .xml file

If you want to hold a writing request standard per second, you need 8 WCUs.

If you want to hold a writing request transactional, you need 32 WCUs.

💡 Learn more about RCU and WCU:

https://n9.cl/ar38i

https://n9.cl/gr0fi

The default configuration provided by dynamo for when you're not sure about any of the above is as follows:

For tests, this is very good, but for real projects, you need to know your data and processes, to achieve efficiency and take advantage of all the possibilities that dynamo offers, adapting it 100% to you.

Basic anatomy of a dynamo board

Primary Key: It must be unique and can be of two types: Partition Key or Composity primary key

Partition Key: If the table has only one partition key, there cannot be two items with the same partition key value.

Composity primary key: It is a combination of a partition key and an ordering key. If the table has a composite primary key, two elements can have the same partition key value. However, those elements must have different sort key values.

Sort Key: It must be unique, also known as an ordering key.

Items: Each table contains zero or more elements. An element is a group of attributes that can be uniquely identified among all other elements.

Attributes: Each element is comprised of one or more attributes. An attribute is a fundamental piece of data, something that doesn't need to be broken down any further.

DynamoDB DAX

Another concept that must be taken into account when working with data focused on consistency and high availability is data caching, although there is another aws service called Elasticache for this purpose, DynamoDB has its own sub-service focused on cache, this is Dynamo accelerator (DAX). DAX consists of two caches located between the dynamo data table and the client, one is the element cache and the other the query cache.

The Flow when using DAX is as follows:

For a [key-value] query from the client application, the Cache [DAX] memory is searched. If it exists, the value is returned. Otherwise, the DynamoDB table is searched, returned to the client and stored in the cache [DAX]. If the DAX cluster has more than one node, the element is replicated to all nodes to ensure consistency.

For error handling in Dynamo

Correct error handling in any development is always fundamental to the quality of that development and says a lot about the developer's control over the code, or in this case, over an AWS resource.

Luckily, aws has good documentation on errors and their usefulness. Even so, let's look at some general points.

If dynamo is being implemented with SDK you have http responses of 3 types:

• 2xx for a successful operation.
• 4xx for an operation that was not successful. This type of error will have 3 components
• HttpCode
• The name that dynamo gives to the exception or problem you identified.
• An error message that more clearly explains the cause of the error.
• The available 4xx errors are:
• AccessDeniedException, ConditionalCheckFailedException, IncompleteSignatureException, ItemCollectionSizeLimitExceeded Exception, LimitExceededException, MissingAuthenticationTokenException, ProvisionedThroughputExceeded, ProvisionedThroughputExceeded, RequestLimitExceeded, ResourceInUseException, ResourceNotFoundException, ThrottlingException, UnrecognizedClientException, ValidationException
• 5xx for an aws-specific problem. It can also be a transitory error (of operational unavailability for example), which can be resolved with a retry. The available 5xx errors are:
• Internal Server Error (HTTP 500)
• Service Unavailable (HTTP 503)

To make good use of these errors, it is recommended to insert them into try-catch blocks.

💡 You can read more about error handling here: https://n9.cl/ugvbo

💡 Data: dynamo has special error handling due to the use of DynamoDB Transactions, which consists of performing multiple reads/writes in a single request. The most common mistake is conflicts, these are:

• A request PUTItem, updateItem or DeleteItem of an element conflicts with a request TransactWriteItems in progress that includes the same element.
• An item included in a request TransactWriteItems is part of another request TransactWriteItems in progress.
• An item included in a request TransactGetItems is part of another request TransactWriteItems, BatchWriteItem, PUTItem, updateItem or DeleteItem in progress.

To better understand what DynamoDB Transactions are: https://n9.cl/b9p0x

Well, now you have knowledge about what dynamo is, how it works, and some special considerations for the good management of your data. Remember to follow this advice: analyze the data before starting to model, understand your objective and concurrency of reading and writing, with that in mind you will have more control and therefore more efficiency.

Ready to optimize your projects with DynamoDB on AWS?

At Kranio, we have experts in data solutions who will help you implement good practices and efficient strategies using DynamoDB, ensuring the scalability and performance of your applications. Contact us and discover how we can promote the digital transformation of your company.

‍