Introduction

Blecon is enabling a new class of physical data sources and sensors based on Bluetooth that allows developers and application builders to quickly integrate real-world data into their products and applications. 

Sensors and other data sources that use Blecon can use nearby phones or gateways as hotspots, require zero configuration or coding to use, and deliver data as WebHooks; enabling them to work with any cloud, any web framework, and any language.

In this article, I will show you an example of how you can use Blecon and AWS to create a real-world streaming data ingestion pipeline in a few minutes, using Amazon Kinesis.  Amazon Kinesis is a hyper-scale message broker service by AWS, which allows you to process and react to incoming data in real-time as well as route it to multiple consumers.

Once you have created a stream of events in Kinesis, it can be consumed by multiple consumers. In this example, I will show how to route it to an Amazon Redshift cluster. Amazon Redshift is a petabyte-scale SQL database typically used for data warehouse applications.

This architecture is suitable for an environment where you have many hundreds or thousands of Blecon sensors all regularly reporting real-world conditions. 

This tutorial assumes intermediate-level knowledge of AWS.

Architecture overview

Let’s have a look at what we’ll build. 

Because Blecon presents Bluetooth sensor readings as HTTP requests, we can use a Lambda function with Function URL to ingest data to the Kinesis stream. Once the data is in the stream we use a delivery stream to write it to Redshift.

After we have created a stream of sensor readings in Amazon Kinesis, you can then analyze and consume this stream in multiple ways. 

Create Kinesis Data Stream

Let’s start with creating our Kinesis data stream. Kinesis is a big data streaming service designed for processing large-scale data streams. We're going to push sensor data into this stream.

To start, you’ll need an AWS account. If you’ve never used AWS before, this AWS getting started tutorial may be useful.

Note: I have used the us-east-1 region in my examples.

1. Open the Kinesis Data Streams page in the AWS Console either using the search or the menu
2. Click Create new stream
3. Choose a name such as sensorstream
4. Choose On-Demand
5. Click Create

Create ingestion Lambda function

The ingestion lambda function will receive HTTP requests from your Blecon network, and insert sensor events onto your Kinesis stream, where they can be consumed later.

1. From the AWS Console, click Lambda, and then Create function.
2. In the Create Lambda screen:
1. Choose Author from scratch.
2. Choose a name for the function such as BleconToDynamo.
3. Choose a runtime environment. For this example we will be using Python 3.9.
4. Click Advanced Settings and then select Function URL with auth type ‘NONE’.
5. Click Create function.
3. In the editor window, paste the following code:

import json, boto3
STREAM_NAME = 'sensorstream'
client = boto3.client('kinesis', region_name='us-east-1')

def lambda_handler(event, context):
   body = json.loads(event['body'])

    for record in body['request_data'].get('records', []):
       payload = {
           'sensor_value': record['event_value'],
           'device_id': body['network_headers']['device_id'],
           'timestamp': record['timestamp']
        }

       put_response = client.put_record(
           StreamName=STREAM_NAME,
           Data=json.dumps(payload),
           PartitionKey=body['network_headers']['device_id'])

   return {
       'statusCode': 200,
       'body': '{}',
    }

Note: Make sure STREAM_NAME matches what you chose earlier.

What does this do? It extracts the sensor events from the Blecon request and puts a subset of them onto the Kinesis stream. Specifically, it un-batches the events and sends a simplified event to the stream containing only the sensor value, the device ID, and the timestamp.

Your application will vary but keep in mind that Blecon devices batch sensor readings, so you will usually want to un-batch them for further processing.

Set permissions on Lambda Function

Your Lambda function needs permission to post onto the Kinesis stream you created.

1. Open the function in the AWS Console and go to Configuration, the Permissions
2. Click on the role name to open it in IAM
3. Click Add Permissions, then Attach Policy
4. Search for AmazonKinesisFullAccess and tick the checkbox
5. Click Attach policies at the bottom

Create the Redshift Cluster

1. Open Amazon Redshift from the AWS Console menu
2. Click Create Cluster
3. Choose a free trial cluster. Note that this provides you 1 free month of Redshift cluster usage, and you will need to delete or pause the cluster after this time to avoid incurring charges.
4. Set an admin password and make a note of this

Create Redshift Table

Once your Redshift cluster has been created, we need to create a table to store your data.

1. Open the Redshift Console, and then the cluster you created
2. Click Query data at the top and choose Query editor v2
3. In the left menu, choose Create then Table
4. Set the database as the default database called dev
5. Set the table name to sensor-table and schema Public
6. Create table schema in the same screen, and enter:

1. device_id type varchar

2. sensor_value type integer

3. timestamp type varchar

7. Click Create table

Create the Kinesis Delivery Stream

A Kinesis Delivery Stream batches up records and saves them to a location such as S3 or Redshift.

We need to use this to write our data to Redshift.

1. Open Kinesis in the AWS Console
2. Go to Delivery Streams and click Create delivery stream

As Source, choose Amazon Kinesis Data Streams

As Destination, choose Amazon Redshift

1. Click Create, then you will move onto more detailed configuration.
1. Choose the sensor data stream you already created in Source settings
2. Choose the Redshift cluster you created in the previous steps
3. Enter admin user and password you created earlier
4. Enter dev for the database name
5. Enter sensors for the table name
6. Enter sensor_value,device_id,timestamp in the Columns field
7. Kinesis needs an S3 bucket to collect records before inserting into Redshift. Create it now.
8. Kinesis performs a COPY command to insert data into Redshift..In the COPY command options, enter json 'auto'
9. Click Create delivery stream to finish

Allow access to Redshift from Kinesis

We need to ensure that public access is enabled, as Kinesis operates outside of your VPC.

1. Open your cluster in the AWS console. 
2. At the top click Actions and then Modify publicly accessible setting and ensure this is enabled, then click Save Changes

As a final step we need to allow IP access to your Redshift cluster from the Kinesis IP range. 

1. Still on the cluster, open the Properties tab
2. Scroll down to the VPC Security Group and open it
3. Click Edit inbound rules
4. Click Add rule
5. Create a rule that allows all traffic from a source of 52.70.63.192/27
6. Click Save rules

Results

1. Ensure data is flowing from your device(s)
2. Open the Query Editor V2 from the cluster main page in the AWS Console
3. Run the query 

   SELECT * FROM “sensors”;

   as below, from the “dev” database.

You should start to see a list of sensor values appearing in your data warehouse.

Conclusion

In this tutorial, I’ve shown you one way to connect Blecon-enabled Bluetooth devices to a data warehouse. 

This architecture is scalable to thousands of sensor readings per second and petabytes of storage. Applications of this pattern include

• Building a service that allows your customers to track physical metrics over time
• You want to train an AI model and need to amass a large amount of real-world physical data first
• Gathering physical analytics across releases, process changes or batches to support product decision-making or compliance

This is just one example; Blecon enables low-cost generic sensors or custom hardware to integrate with any cloud-native system efficiently and securely. See https://blecon.net for more.