Amazon DynamoDB Persistence

This service allows you to persist state updates using the Amazon DynamoDB (opens new window) database. Query functionality is also fully supported.

Features:

• Writing/reading information to relational database systems
• Configurable database table names
• Automatic table creation

Disclaimer

This service is provided "AS IS", and the user takes full responsibility of any charges or damage to Amazon data.

Table of Contents

• Disclaimer
• Table of Contents
• Prerequisites
  • Setting Up an Amazon Account
• Configuration
  • Table schema
  • Credentials Configuration Using Access Key and Secret Key
  • Credentials Configuration Using Credentials File
  • Advanced Configuration
• Details
  • Caveats
• Developer Notes
  • Updating Amazon SDK
  • Running integration tests

Prerequisites

You must first set up an Amazon account as described below.

Users are recommended to familiarize themselves with AWS pricing before using this service. Please note that there might be charges from Amazon when using this service to query/store data to DynamoDB. See Amazon DynamoDB pricing pages (opens new window) for more details. Please also note possible Free Tier (opens new window) benefits.

Setting Up an Amazon Account

Login to AWS web console

• Sign up (opens new window) for Amazon AWS.
• Select the AWS region in the AWS console (opens new window) using these instructions (opens new window). Note the region identifier in the URL (e.g. https://eu-west-1.console.aws.amazon.com/console/home?region=eu-west-1 means that region id is eu-west-1).

Create policy controlling permissions for AWS user

1. Open Services -> IAM -> Policies
2. Click Create policy
3. Open JSON tab and input the below policy code, describing the permissions needed

Note: The below policy assumes that eu-west-1 region is used, the new table schema is used, and the default table name of openhab is used. Modify the policy accordingly if needed.

Note 2: As a more simple alternative, one can use pre-existing policy AmazonDynamoDBFullAccess, although the policy grants the openHAB user wider-than-necessary permissions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:UpdateTimeToLive",
                "dynamodb:ConditionCheckItem",
                "dynamodb:PutItem",
                "dynamodb:DeleteItem",
                "dynamodb:Scan",
                "dynamodb:Query",
                "dynamodb:UpdateItem",
                "dynamodb:DescribeTimeToLive",
                "dynamodb:DeleteTable",
                "dynamodb:CreateTable",
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:UpdateTable"
            ],
            "Resource": [
                "arn:aws:dynamodb:eu-west-1:084669220525:table/openhab",
                "arn:aws:dynamodb:eu-west-1:084669220525:table/openhab/index/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "dynamodb:ListTables",
                "dynamodb:DescribeReservedCapacity",
                "dynamodb:DescribeLimits"
            ],
            "Resource": "*"
        }
    ]
}

4. Click Next: Tags
5. Click Next: Review
6. Enter openhab-dynamodb-policy as the _Name_ol-prefix -->
7. Click Create policy to finish policy creation

Create user for openHAB with IAM

1. Open Services -> IAM -> Users -> Add users. Enter openhab as User name, and tick Programmatic access
2. Click Next: Permissions
3. Select Attach existing policies directly, and search policies with openhab-dynamodb-policy. Tick the openhab-dynamodb-policy and proceed with Next: Tags
4. Click Next: review
5. Click Create user
6. Record the Access key ID and Secret access key

Configuration

This service can be configured using the MainUI or using persistence configuration file services/dynamodb.cfg.

In order to configure the persistence service, you need to configure two things:

1. Table schema revision to use
2. AWS credentials to access DynamoDB

Table schema

The DynamoDB persistence addon provides two different table schemas: "new" and "legacy". As the name implies, "legacy" is offered for backwards-compatibility purpose for old users who like to access the data that is already stored in DynamoDB. All users are advised to transition to "new" table schema, which is more optimized.

At this moment there is no supported way to migrate data from old format to new.

New table schema

Configure the addon to use new schema by setting table parameter (name of the table).

Only one table will be created for all data. The table will have the following fields

Attribute	Type	Data type	Description
i	String	Yes	Item name
t	Number	Yes	Timestamp in milliepoch
s	String	Yes	State of the item, stored as DynamoDB string.
n	Number	Yes	State of the item, stored as DynamoDB number.
exp	Number	Yes	Expiry date for item, in epoch seconds

Other notes

• i and t forms the composite primary key (partition key, sort key) for the table
• Only one of s or n attributes are specified, not both. Most items are converted to number type for most compact representation.
• Compared to legacy format, data overhead is minimizing by using short attribute names, number timestamps and having only single table.
• exp attribute is used with DynamoDB Time To Live (TTL) feature to automatically delete old data

Legacy schema

Configure the addon to use legacy schema by setting tablePrefix parameter.

• When an item is persisted via this service, a table is created (if necessary).
• The service will create at most two tables for different item types.
• The tables will be named <tablePrefix><item-type>, where the <item-type> is either bigdecimal (numeric items) or string (string and complex items).
• Each table will have three columns: itemname (item name), timeutc (in ISO 8601 format with millisecond accuracy), and itemstate (either a number or string representing item state).

Credentials Configuration Using Access Key and Secret Key

Property	Default	Required	Description
accessKey		Yes	access key as shown in Setting up Amazon account.
secretKey		Yes	secret key as shown in Setting up Amazon account.
region		Yes	AWS region ID as described in Setting up Amazon account. The region needs to match the region that was used to create the user.

Credentials Configuration Using Credentials File

Alternatively, instead of specifying accessKey and secretKey, one can configure a configuration profile file.

Property	Default	Required	Description
profilesConfigFile		Yes	path to the credentials file. For example, /etc/openhab2/aws_creds. Please note that the user that runs openHAB must have approriate read rights to the credential file. For more details on the Amazon credential file format, see Amazon documentation (opens new window).
profile		Yes	name of the profile to use
region		Yes	AWS region ID as described in Step 2 in Setting up Amazon account. The region needs to match the region that was used to create the user.

Example of service configuration file (services/dynamodb.cfg):

profilesConfigFile=/etc/openhab2/aws_creds
profile=fooprofile
region=eu-west-1

Example of credentials file (/etc/openhab2/aws_creds):

[fooprofile]
aws_access_key_id=testAccessKey
aws_secret_access_key=testSecretKey

Advanced Configuration

In addition to the configuration properties above, the following are also available:

Property	Default	Required	Description
expireDays	(null)	No	Expire time for data in days (relative to stored timestamp)
readCapacityUnits	1	No	read capacity for the created tables
writeCapacityUnits	1	No	write capacity for the created tables

Refer to Amazon documentation on provisioned throughput (opens new window) for details on read/write capacity. In case you have not reserved enough capacity for write and/or read, you will notice error messages in openHAB logs. DynamoDB Time to Live (TTL) setting is configured using expireDays.

All item- and event-related configuration is done in the file persistence/dynamodb.persist.

Details

Caveats

When the tables are created, the read/write capacity is configured according to configuration. However, the service does not modify the capacity of existing tables. As a workaround, you can modify the read/write capacity of existing tables using the Amazon console (opens new window).

Similar caveat applies for DynamoDB Time to Live (TTL) setting expireDays.

Developer Notes

Updating Amazon SDK

1. Clean lib/*
2. Update SDK version in scripts/fetch_sdk_pom.xml. You can use the maven online repository browser (opens new window) to find the latest version available online.
3. scripts/fetch_sdk.sh
4. Copy printed dependencies to pom.xml

After these changes, it's good practice to run integration tests (against live AWS DynamoDB) in org.openhab.persistence.dynamodb.test bundle. See README.md (opens new window) in the test bundle for more information how to execute the tests.

Running integration tests

When running integration tests, local temporary DynamoDB server is used, emulating the real AWS DynamoDB API. One can configure AWS credentials to run the test against real AWS DynamoDB for most realistic tests.

Eclipse instructions

1. Run all tests (in package org.openhab.persistence.dynamodb.internal) as JUnit Tests
2. Configure the run configuration, and open Arguments sheet
3. In VM arguments, provide the credentials for AWS

-DDYNAMODBTEST_REGION=REGION-ID
-DDYNAMODBTEST_ACCESS=ACCESS-KEY
-DDYNAMODBTEST_SECRET=SECRET

Caught a mistake or want to contribute to the documentation? Edit this page on GitHub (opens new window)