Part-4: Working with multiple values and streams
Tutorial overview
- Part 1 - Introduction to key-value store
- Part 2 - Basic operation get, set and delete key-values
- Part 3 - Increment and decrement operations
- Part 4 - Working with multiple values and streams
- Part 5 - Managing data with TTL options
- Part 6 - Multiple key spaces
- Part 7 - Interaction with the key-val store from the CLI
Welcome to this part-4 of the key-value database tutorial for codehooks.io serverless Node.js backend. In this part we'll focus on how to work with multiple key-value pairs in a key-value database.
The Codehooks.io key-value database stores all keys as a sorted string index. This is a very useful feature if you need to store and retrieve multiple values together, as a group or as a segment of key-values.
IoT device - time series data example
To illustrate this feature we'll create an example use-case for capturing data from multiple IoT devices. In this example, each IoT device sends its location, identification and observations to a specific REST API network endpoint.
For example, a temperature sensor device in the "Alpha region West" reports its value each minute. And a wind force sensor in the "Beta region West" reports its values hourly, and so on.
To ensure that we can access and process the observation data sets effectiviely, we'll create a key string combination that captures all necessary information in one unique key-value pair.
To illustrate this concept lets imagine a series of IoT device observations as shown in the example below.
'/Alpha/West/density/2023-01-09T10:14:00.495Z' => '4.4191'
'/Alpha/West/humidity/2023-01-09T17:01:00.821Z' => '0.7539'
'/Alpha/West/temp/2023-01-09T21:15:00.450Z' => '87.100'
'/Beta/West/wind/2023-01-19T21:57:00.316Z' => '5.9477'
...
Now that we understand the concept for our IoT database, lets begin the development of an example backend API for IoT sensors.
We'll create a serverless REST API endpoint to receive observations from many IoT devices. The API will store each observation with a logic key that effectively creates a time series database with data points that we can explore and use in our application.
Lets get started with creating the first API to receive data observations from a device.
REST API endpoint to POST a single observation
The REST API route takes in 3 parameters for a given device, region, location and device. And the POST body must contain the actual device observation data point. For example:
POST /observation/North/Plaza/humid03
{"observation": "4.02"}
The route parameters are combined into a unique key with a time stamp which gives a correct time series for the data points. The following table shows some example device observations.
IoT devices time series | Data points |
---|---|
/East/BuilingA/temp01/2023-02-22T18:30:14.441Z | 21.4 |
/North/Plaza01/humid3/2023-02-23T05:23:11.306Z | 12.5 |
/East/BuilingB/temp02/2023-02-22T18:32:38.991Z | 19.0 |
The serverless function below implements a REST API endpoint route that can receive data from a device and insert it to the key-value database.
app.post('/observation/:region/:location/:device', async (req, res) => {
const { region, location, device } = req.params;
const { observation } = req.body;
const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
// string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
const conn = await Datastore.open();
const result = await conn.set(key, observation);
console.log(result);
res.status(201).end('Data received ok');
});
curl -X POST \
'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observation/East/BuilingA/temp01' \
--header 'x-apikey: <YOUR_API_TOKEN>' \
--header 'Content-Type: application/json' \
--data-raw '{
"observation": "21.4"
}'
Use the CLI command coho info --examples
to inspect your project API endpoint and Curl examples.
Inspecting the key-value database with the CLI
The Codehooks CLI can be used to manage the key-value database basic operations, get
, set
and del
. For example, to retrieve all observations for the East
locations you can run the following command:
coho get '/East*'
This will list all matching keys in the key-value database as the example output shows below.
{
key: '/East/BuilingA/temp01/2023-02-23T05:48:19.646Z',
value: '21.4'
}
{
key: '/East/BuilingA/temp01/2023-02-23T05:49:06.614Z',
value: '20.0'
}
2 keys
Endpoint to GET observations subkey
Similiar to the CLI example above, we can use the database API getAll
method to retrieve all key-value pairs that match a start segment of a particular key. In this example we are producing CSV data format for easy spreadsheet analysis etc.
app.get('/observations/:prefix', async (req, res) => {
const conn = await Datastore.open();
const { prefix } = req.params;
let count = 0;
res.set('content-type', 'text/csv');
res.write('Device, Data\n');
const stream = conn.getAll(`/${prefix}`);
stream
.on('data', (data) => {
const outString = `"${data.key}", ${data.val}\n`;
count++;
res.write(outString);
})
.on('end', () => {
res.end();
});
});
Testing the API with curl again.
curl -X GET \
'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observations/Alpha' \
--header 'x-apikey: <YOUR_API_TOKEN>' \
Device, Data
"/Alpha/Center/decibel/2023-01-09T11:11:31.218Z", 2.3428
"/Alpha/Center/decibel/2023-01-09T11:12:01.050Z", 6.0632
"/Alpha/Center/decibel/2023-01-09T13:13:30.541Z", 0.7196
"/Alpha/Center/decibel/2023-01-09T15:23:00.589Z", 9.7232
"/Alpha/Center/decibel/2023-01-09T15:34:00.520Z", 5.0089
"/Alpha/Center/decibel/2023-01-09T17:04:00.942Z", 9.1861
Complete source code
The complete source code for our IoT backend application is shown below.
import { app, Datastore } from 'codehooks-js'; // Standard JS lib for express style code
// list a serie of IoT device observations
app.get('/observations/:prefix', async (req, res) => {
const conn = await Datastore.open();
const { prefix } = req.params;
let count = 0;
res.set('content-type', 'text/csv');
res.write('Device, Data\n');
const stream = conn.getAll(`/${prefix}`);
stream
.on('data', (data) => {
const outString = `"${data.key}", ${data.val}\n`;
count++;
res.write(outString);
})
.on('end', () => {
res.write(`Result: ${count}`);
res.end();
});
});
// register a new IoT device observation
app.post('/observation/:region/:location/:device', async (req, res) => {
const { region, location, device } = req.params;
const { observation } = req.body;
// string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
const conn = await Datastore.open();
const result = await conn.set(key, observation);
console.log(result);
res.status(201).end('Data received ok');
});
export default app.init(); // Bind functions to the serverless cloud
This part-4 of the key-value store tutorial has shown how you can combine smart keys with the database streaming capabilities to create kind of IoT time series database with codehooks.io.
You can find the documentation for the key-value store API here.