arundhaj

regression towards the datascience

AWS DynamoDB full table scan

 

Amazon DynamoDB is a NoSQL database service hosted on AWS. It is a fully managed and scalable document store database. It is quite similar to MongoDB.

As the data grows, scan operation on full table would return parts of the data with the LastEvaluatedKey. The application should initiate scan again from the LastEvaluatedKey. Below code snippet shows how to do it.

import boto3

dynamodb_client = boto3.resource('dynamodb', region_name='us-east-1')

my_table = dynamodb_client.Table('MyTableName')

filter_expression = Key('id').eq(record_id) &
                     Key('timestamp').between(long(start_date), long(end_date))

result_item = []

result_data = my_table.scan(
     FilterExpression=filter_expression
)

result_item.extend(result_data['Items'])

while 'LastEvaluatedKey' in result_data:
    result_data = my_table.scan(
        FilterExpression=filter_expression,
        ExclusiveStartKey=result_data['LastEvaluatedKey']
    )

    result_item.extend(result_data['Items'])

Assuming you have the credentials appropriately configured.

Hope this helps!

Comments