Dynamodb Read Capacity for Query Result Larger Than Limit

In this lesson, we'll talk nearly using Scans with DynamoDB. The Scan call is the bluntest instrument in the DynamoDB toolset. By way of analogy, the GetItem call is like a pair of tweezers, deftly selecting the exact Item you want. The Query call is like a shovel -- grabbing a larger amount of Items but however small-scale plenty to avoid grabbing everything.

The Scan operation is like a payloader, grabbing everything in its path:

The Scan telephone call, reporting for duty.

Before nosotros dive too deeply into the Scan call, I want you to say the post-obit words out loud:

I will never use the Scan operation unless I know what I am doing.

The Scan operation operates on your entire table. For tables of existent size, this tin can quickly use up all of your Read Capacity. If yous're using it in your application's critical path, it will be very slow in returning a response to your users.

The Browse operation generally makes sense only in the following situations:

• you accept a very small-scale table;
• you're exporting all of your table'due south data to some other storage organization; or
• you apply global secondary indexes in a special way to fix a piece of work queue (very avant-garde).

With these caveats out of the way, let's explore the Browse phone call.

Scan basics

The Browse telephone call is probable the easiest of all DynamoDB calls. Merely provide a tabular array name, and it volition return all Items in the table (upwards to a 1MB limit):

            $ aws dynamodb scan \     --table-name UserOrdersTable \              $LOCAL                      

The response (truncated for brevity):

                          {              "Count"              :              25,              "Items"              :              [              {              "OrderId"              :              {              "Due south"              :              "20160630-28176"              },              "Username"              :              {              "S"              :              "daffyduck"              },              "Corporeality"              :              {              "N"              :              "88.3"              }              },              ...              {              "OrderId"              :              {              "S"              :              "20171129-28042"              },              "Username"              :              {              "S"              :              "alexdebrie"              },              "Amount"              :              {              "N"              :              "83.12"              }              }              ],              "ScannedCount"              :              25,              "ConsumedCapacity"              :              null              }                      

Every bit you lot can come across, it returned all of our Items dorsum to us.

Like the GetItem and Query calls, y'all can use a --project-expression to specify the particular attributes you want returned to y'all. I'll skip the case hither as it's similar to the previously given examples.

DynamoDB has a 1MB limit on the amount of data it will retrieve in a single request. Scans will frequently hitting this 1MB limit if you're using your table for real use cases, which means you'll need to paginate through results.

If you hit the 1MB limit with a Browse, it volition return a "NextToken" key in the response. Y'all can employ the value given with the --starting-token option to go on scanning from the location you lot previously ended.

You can test this behavior by passing a --max-items limit in our table. Let's make a Scan request with a max items limit of 1:

            $ aws dynamodb scan \     --table-name UserOrdersTable \     --max-items 1 \              $LOCAL                      

The response includes a unmarried Item, plus a NextToken to continue our Scan:

                          {              "Count"              :              25,              "Items"              :              [              {              "OrderId"              :              {              "S"              :              "20160630-28176"              },              "Username"              :              {              "S"              :              "daffyduck"              },              "Amount"              :              {              "North"              :              "88.3"              }              }              ],              "NextToken"              :              "eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDF9",              "ScannedCount"              :              25,              "ConsumedCapacity"              :              zippo              }                      

Parallel Scans

I utilise case for Scans is to export the data into cold storage or for data analysis. If you have a large amount of data, scanning through a table with a single procedure can take quite a while.

To alleviate this, DynamoDB has the notion of Segments which permit for parallel scans. When making a Scan, a request can say how many Segments to divide the tabular array into and which Segment number is claimed past the particular request. This allows you to spin up multiple threads or processes to scan the data in parallel.

Even with our pocket-size amount of data, we tin can test this out. Let's say we want to segment our table into three segments to be processed separately. Ane process could say there are three total segments and that it wants the items for segment "1":

            $ aws dynamodb scan \     --table-name UserOrdersTable \     --total-segments 3 \     --segment one \              $LOCAL                      

You tin run into the response just has xi items, rather than the total 25:

                          {              "Count"              :              11,              "Items"              :              [              {              "OrderId"              :              {              "S"              :              "20160630-28176"              },              "Username"              :              {              "Due south"              :              "daffyduck"              },              "Amount"              :              {              "Northward"              :              "88.3"              }              },              ...              {              "OrderId"              :              {              "S"              :              "20170609-9476"              },              "Username"              :              {              "S"              :              "yosemitesam"              },              "Amount"              :              {              "N"              :              "nineteen.41"              }              }              ],              "ScannedCount"              :              11,              "ConsumedCapacity"              :              cipher              }                      

Segments are nix-indexed, though I had trouble when trying to utilise Segment "0" with DynamoDB Local -- it kept returning 0 elements.

In the side by side section, we'll acquire near filtering your Query and Browse operations.

taylorfroustommer.blogspot.com

Source: https://www.dynamodbguide.com/scans/

Share this post