A Pattern For Secure Uploads And Downloads In AWS IoT
Suppose you have a large fleet of connected sensors, all gathering data and triggering events based on conditional logic within the device. It’s one thing to provide event notifications, but what about all the continuous readings and status data? This data is likely to be much larger than what is selected for triggering an event or notification. This data may be able to provide valuable insights and is where analytics can provide incredible value.
If we want to leverage this data we first must consider how to ship it to the cloud, which forces us to think about security. Uploading data is not the only consideration when it comes to IoT. We must also consider the secure delivery of large amounts of configuration or firmware data to and from a connected device.
How to secure file transfers in AWS IoT: A Use Case
In the AWS ecosystem where we have a device connected using AWS IoT, you could download or upload a large amount of data by partitioning the data into individual MQTT messages and re-assembling after receipt. Assuming the packetization process also encrypts the data, this would certainly be secure. However, that’s not an efficient use of AWS IoT costs (paid by MQTT Message), and unnecessarily complicated since it requires partitioning and reassembling, and likely some form of acknowledgment or QoS increase to ensure all the data arrives successfully.
If you are familiar with storage options in AWS, you’re probably thinking:
“Can’t we just use S3?”
Yes, AWS S3 is a great solution for storing this kind of data based on cost, scalability, level of availability, and level of security. There is an added challenge here when we consider the fact that devices registered with AWS IoT don’t have the needed AWS credentials to make an authenticated request with S3 out of the box. Assuming this data needs to be well secured, we don’t want to leave the S3 bucket upen to unauthenticated requests.
For the purposes of this example, we will consider a pattern presented by iRobot at AWS re:Invent for secure file uploads (see video). The most pertinent information for the purposes of this post can be found starting on slide 60, but the entire pattern is highly recommended. We did not invent this approach, but have encountered similar problems, and now endorse the presented pattern because of its elegance, security, and reliance upon AWS best practices.
Consider These Use Cases For Secure Data Transfer In AWS IoT
For this particular problem, we’ve identified the following example use cases:
- IoT Device, buffering up sensor readings data and shipping nightly for non-urgent analytic processing
- IoT Device, downloading a firmware image update or other media (Although, if your device is running Amazon FreeRTOS, you have some really nice features for Over The Air (OTA) updates.
Secure Delivery:
The device can’t make an unauthenticated request out of the box, but with some setup in AWS IoT, it’s possible to request what’s called a pre-signed S3 URL. A pre-signed S3 URL contains AWS credentials in the URL itself, which are scoped to just the AWS API call and parameters we specify, and is only accessible for a limited duration. This means the URL allows access to a specific operation (GET or PUT) for a specific S3 Bucket, for a specific Key (filename in our case), only for a limited amount of time.
Here is how we could built this capability:
Step 1: Create a way to ask for a pre-signed URL
For this step, we need to set up a AWS IoT Rule on a dedicated MQTT topic to call a Lambda function that will make the request and return the URL. For this topic, it’s ok to use a topic that is not specific to the device, but something generic across the device fleet. Let’s say something like the topic below:
things/pre-signed/s3-url-requests
Configure the AWS IoT Rule with the request topic and select “Invoke a Lambda function passing the message data.”. When prompted for the Lambda function, select the option to create a new resource.
Step 2: Respond with the pre-signed URL
Next, the Lambda function needs to be implemented to request a presigned url with the parameters defined by our request. The request itself could look something like this:
{
'operation': 'putObject',
'bucket': 'spindance.blogs.secure-uploads-bucket',
'key': 'test.zip',
'replyTo': 'things/VerySmartThing/pre-signed-s3-url-responses'
}
The pre-signed URL needs to take into account the operation (Get or Put), S3 bucket name, and key (used as the destination name). In this example, we allow all of these parameters to be defined in the request. There is an additional parameter needed to generate the url, the number of seconds the url will be available before expiration. For this example, we forced the expiration duration to be controlled by the Lambda handler, but it could be specified in the request as well.
The replyTo field specifies the device-specific response topic that only the specific device is permitted to listen to, and that the Lambda function shall use to respond to a given request. In this case VerySmartThing is the “Thing Name” as the device was registered in AWS IoT. This ensures only the device will receive the presigned url. Permissions for MQTT topics like this are controlled by policy documents used by AWS IoT. For more information, see this approach for configuring secure, device-specific, communications AWS IoT Core with replacement variables in AWS IoT policies.
The AWS S3 SDK gives us an easy way to request the pre-signed URLs, so this is the approach we’ll use. Below is an example Lambda handler for that will generate the URL and reply back to the user.
var AWS = require('aws-sdk');
var iotdata = new AWS.IotData({endpoint: 'my-iot-endpoint-id.iot.us-east-2.amazonaws.com'});
exports.handler = (event, context, callback) => {
var s3 = new AWS.S3();
var params = {Bucket: event.bucket, Key: event.key, Expires: 60};
var url = s3.getSignedUrl(event.operation, params);
console.log('The URL is', url);
var params = {
topic: event.replyTo,
payload: url,
qos: 1
};
iotdata.publish(params, function(err, data){
if(err){
console.log(err);
callback(1, err);
}
else{
callback(null, url);
}
});
};
Having done the steps above, the device has the ability using an HTTPS call to upload or download a file to/from S3 using the pre-signed S3 URL. The device is delivered the URL on a secured channel that only the device can access.
Encrypting The Payload
Now, let’s consider the fact that we need to secure this file from something beyond server-side encryption. The S3 bucket can be secured by limiting access to specific IAM roles through IAM policies, and server-side encryption can be enabled on the S3 bucket itself, but suppose we want to encrypt the payload. Payload encryption keeps the data protected, but it can be done in such a way that the cloud and the device can not only decrypt the data provided from their counterpart, but also authenticate the payloads received from its counterpart. Suppose we also want to ensure that any files uploaded indeed came from the device itself. Conversely, suppose we need to ensure that only the device can decrypt a given payload. For this, let’s talk about some options for payload encryption.
Asymmetric encryption will enforce what we want. Only the owner of the private key can decrypt data encoded with the public key, and only the owner of the private key could have encrypted the data that can be successfully decrypted using the public key. AWS IoT Core requires the registration of a certificate as a part of Thing Registration, which contains the device public key. Therefore, by securing the private key on the device itself, we have the ability for the device to sign/encrypt data on the device and we can be sure it came from the device by successfully decrypting with the public device key. We can also be sure that only the device can decrypt data encoded with the public key.
Asymmetric encryption is a workable solution here, but is much slower and resource intensive than symmetric encryption. For more detail on this, see some example performance metrics and this thread discussing why asymmetric encryption is not a great fit for large amounts of data.
In terms of symmetric encryption, the AWS Key Management Service (KMS) provides a secure service for requesting symmetric keys. The problem is that the device doesn’t have the needed AWS credentials to request a Key from KMS. Does this sound familiar? The solution here is to create a secure way for the device to securely obtain the symmetric key from KMS for payload encryption.
This can be accomplished by encrypting the symmetric key itself with asymmetric encryption (using the device public key as described above) ensuring only the device can decrypt the symmetric key. This key can then be used for encrypting and decrypting the data, and the key itself can only be interpreted by the device.
Here is how we can build this capability:
Step 3: Create a way for the device to ask for an encrypted KMS Key
To accomplish this, we first need to set up an AWS IoT Rule on a dedicated MQTT topic to call a Lambda function that respond with an encrypted KMS key specific to a device. Following a similar pattern as defined above, we can use a generic topic for all requests and specify the response topic using a replyTo field in the request.
KMS provides different types of keys, including symmetric “data encryption keys”. For further details on KMS, see the developer guide for additional information. Request the creation of a new symmetric data encryption key to be used only for payloads to and from this device from KMS using the AWS SDK.
Next, well need the public key from the device certificate to encrypt the data encryption key itself. We could do this a few different ways. One option is to lookup the device certificate itself in AWS IoT, then extract the public key using something like pkijs or another library. This could even be done as a part of the registration and onboarding process for the device so it’s only done once. This is definitely the best approach since all that information is already available on the cloud, but for the sake of keeping this example somewhat simple, let’s just send it in the request since it’s a public key after all. This won’t put the symmetric key at risk since the only way to decrypt the encrypted symmetric key is to have the private key, which is secured on the device itself. We are simply telling the cloud how to encrypt it, so only the device may decrypt.
{
'certificateId': '1234abcd-12ab-34cd-56ef-1234567890ab',
'publicKey: 'MIIBIjANBgk...TBkxf+4pB1wIDAQAB',
'replyTo': 'things/VerySmartThing/payload-key-responses'
}
Step 4: Respond with a new encrypted key.
The final step is to use the public key provided in the request to encrypt the data encryption key and deliver it by responding on the device specific replyTo topic. This can be done with crypto or another similar library.
Step 5: Encrypt and Decrypt Payloads
On the device, store the encrypted KMS key as a configuration value to be utilized later. When the device is ready to upload data, decrypt the key in memory and use for data encryption. This follows the AWS recommendations for using these keys, as opposed to storing them unsecured. When the device is ready to unpack downloaded data, decrypt the key and use for data decryption. Depending on the use case, this key should be refreshed periodically.
Step 6: Use the KMS key in other AWS services
Services consuming the data uploaded from the device can be given access to the KMS key to encrypt or decrypt the KMS key and used to encrypt files to be uploaded to S3 via the established pre-signed S3 URL approach from the first section.
It is important to note that KMS doesn’t track data encryption keys by default, so you’ll need to keep track them yourself. The encryption keys come pre-encrypted themselves when delivered from KMS using a master key. This means you can store the encryption keys in a secure manner on something like S3, where they are only decryptable using a master key in KMS, which does have managed access.
Services consuming the data uploaded from the device, or producing data to be downloaded by the device can be given access to the KMS master key and therefore the symmetric data encryption key. This access is all auditable, revocable, and controlled through AWS IAM policies.
Putting It All Together
Using this pattern, AWS IoT devices can benefit from additional AWS services like S3 for file transfers, and KMS for providing payload encryption keys securely. Enabling secure file transfers for your device can save your operations costs and development complexity by using the services AWS intended for those concepts, while not forcing the device to manage an additional identity beyond the device certificate required for connectivity to AWS IoT.
Thank you to the iRobot team for presenting this pattern. We felt motivated to promote and advocate, as well as highlight some of the additional advantages we observed from our experience.
If you have similar, or an entirely different, set of challenges on your next IoT project, we can help. Let us know!