Article, IoT Security

Securing IoT Devices – Manufacturing and Provisioning

June 23, 2023

One of the most vulnerable stages of an IoT product is the manufacturing process.  Ensuring that a device leaves the assembly line secured and ready to be connected requires some unique infrastructure and forethought. This article provides an overview of the primary security goals and best practices.  The example is based on an ESP32 being provisioned to AWS, but the principles can be applied to any cloud infrastructure or hardware platform.

SpinDance has developed patterns that make it easier to securely manufacture devices, including CI/CD pipelines, secrets management, and specialized software.

Securing IoT devices can be broken into two primary objectives, authenticity and privacy. 

Goals of Authenticity

  1. Prevent users from exposing sensitive information like WiFi credentials with un-authenticated devices
  2. Prevent counterfeit devices from provisioning themselves to your IoT endpoint
  3. Prevent an attacker from executing unauthorized code on the device

Registration

Before we address how IoT devices can be authenticated, we should understand the two main methods used to authenticate and register a device to an IoT endpoint such as AWS IoT Core. 

Bulk Registration

  • Bulk Registration means that a unique key pair is generated for every device manufactured.  A manifest is stored in the Cloud that enables it to authenticate each device.  The Cloud maintains a certificate for each device and can individually validate them as they come online.
  • The downside to bulk registration is that it requires a manifest to be transported between the manufacturing facility and the Cloud.  The cloud can pre-generate this or it can be uploaded later, but any loss or corruption of information in the manifest results in devices in production that can’t be registered to an IoT endpoint.  Not good.

Just in Time Registration (JITR)

  • JITR removes the need for the cloud to know any information about each specific device waiting to connect, it can authenticate devices when they attempt to connect for the first time by ensuring the device is able to present a certificate signed by a Certificate Authority pre-registered and authorized by the Cloud environment.
  • The downside of JITR is a slightly more complex configuration up front.  Certificate Authorities need to be registered and secured.  A security breach of the Certificate Authority doesn’t allow isolating specific devices.

In either case the device needs to store sensitive information at the time of manufacturing.  For Bulk Registration it needs a certificate the cloud can validate against a manifest.  Think of this as a list of serial numbers that get claimed each time a device connects.  For JITR it needs an x.509 certificate signed by a Certificate Authority(CA) registered in AWS.  Think of this as a signature on a check, it’s the signature that guarantees the check was written by an authorized user but they can write an infinite number of checks.

Manufacturing and Provisioning devices for JITR

We are going to focus on setting up the JITR process when the device is being manufactured.  This is SpinDance’s preferred method for device registration because it eliminates the risk of orphaned devices if the manifest is mishandled. 

We can ensure the device’s origin for end users and the IoT endpoint by validating the device has an x.509 certificate signed by a CA that is registered in our AWS account.  We use some tools to automate this process and support a Sub CA to be configured on a hardware device such as a Yubikey.  This allows the CA to be offline and controlled via physical access to a device.  It’s similar to having a key to a safe.

In order to provision the device during manufacturing we must be able to sign x.509 certs with our registered Sub CA. This involves the following highly sensitive information:

  • A private/public key pair, unique to each device, used to create a Certificate Signing Request (CSR)
  • Access to the registered Certificate Authority so that the CSR can be fulfilled and return a valid x.509 certificate. This is the only source of truth to verify device authenticity. 

Secure Boot:

Protecting unauthorized software from being executed on the ESP32 can be accomplished with Espressif’s Secure Boot V2.

Espressif’s docs explain the secure boot process in detail but in summary:

  • A private key is stored securely in the cloud and used to create a boot loader and sign application binaries
  • The device is provisioned with a secure boot loader that cannot be modified utilizing eFuse.
  • Before booting into any second stage boot loader or application binary the secure boot loader uses a public key in secure memory to verify the software is signed by the same private key used to create the secure boot loader: Verifying an Image
  • Any binaries delivered via OTA updates will also need to be signed by the secure boot private key or they will not be booted.

Goals of Privacy

  1. Protect information in transport
  2. Protect sensitive information even if someone has physical access to the device. 

TLS Encryption:

We can protect information in transport using standard TLS protocol between the device and the cloud. The device’s x.509 cert is used authenticate the device to the server and establish a secure communication channel with AWS using TLS.

Flash Encryption

Protecting software on the device can be accomplished with flash encryption. 

  • A private symmetric flash encryption key is stored on the device in secure memory utilizing eFuse
  • Before an application image is first booted during manufacturing it is encrypted in place with the flash encryption key. Subsequent OTA’s are encrypted as the image is downloaded to the device.
  • Any information stored in flash can only be deciphered with the flash encryption key protecting it even if an attacker copied the device’s flash memory.

NVS Encryption

Non Volatile Storage enables the device to persist information across boot cycles on the ESP32. NVS encryption is enabled by default when Flash Encryption is enabled. The NVS encryption keys are generated on the device and never exposed so we don’t generally have to be concerned about them in manufacturing.  Espressif’s doc’s on NVS Encryption

Sensitive Data Recap

Let’s catalog the sensitive information that is generally required to provision a device during manufacturing:

  • Certificate Authorities (root CA and Sub CA’s)
  • Device Public/Private key pair used to generate CSR and x.509 cert
  • Secure Boot signing key
  • Flash Encryption Key
  • Proof of Possession Keys (ie. QR codes or Bluetooth passkeys)

We refer to the collection of all these elements as the “Device Stamp”.  The Device Stamp contains immutable and secret information that needs to be placed on the device once and never altered.  It is akin to physically “stamping” the device.

Two factors make protecting information in the stamp a unique challenge:

  • Manufacturing is global

      • Manufacturing is often outsourced overseas to facilities that should not be trusted with access to sensitive data.  Breaches can happen even in trusted facilities.  The consequence of a breach can result in:
        • Loss of devices already in the field
        • Counterfeit devices
        • Malicious code on seemingly secure devices
        • Reverse engineering to exploit vulnerabilities
        • Theft of intellectual property

  • Time is Money

    • Every second required on a manufacturing line will incur significant costs. For every 100,000 devices, each additional second per device equates to 28 hours of additional time on the manufacturing line. Reducing the time by 30 seconds can save a month of manufacturing uptime.

The ideal method to protect sensitive data is to never expose it in the first place.  This can be accomplished by generating private keys on the device.

SpinDance has developed specialized software and utilities to protect sensitive information during manufacturing. The device receives instructions from a Manufacturing Utility to generate keys without exposing them to the manufacturer.

Example Manufacturing Flow

  • Cert generation – 30s
  • Flashing binaries – 1 min (optional)
  • Encrypting in place on first boot – 1 min

This flow minimizes transporting or exposing sensitive information but is still optimized for speed and cost. It protects sensitive information by generating all private keys on the device.  Certificates are signed by a Sub CA utilizing a hardware device such as Yubikey. 

 

 

 

  • Signed Application binary and Mfg binary are pre-flashed by chip supplier
    1. If this is performed by the Mfg Utility it will depend on the size of the binary but expect this to take ~1 minute
    2. One partition has manufacturing software
    3. Second partition has signed application software
    4. Both are plain text binaries
  • Device boots into the Mfg application binary (not a secure bootloader)
  • Mfg Utility checks that Mfg binary and signed application binary meet minimum version requirements
    1. If not, an updated version of the Mfg binary and/or Application binary can be flashed by the Mfg Utility
  • Mfg Utility instructs device to generate RSA key pair
    1. This can take 7-12 seconds using Mbed-TLS
  • Device returns a CSR to the Mfg Utility
    1. Only the public key is returned to the Mfg Utility as part of the CSR
  • Mfg Application Creates x.509 certificate signed by Sub CA (Yubikey).  
    1. CA will need to be already registered in AWS or registered before the device attempts to come online.
    2. The passkey required to access the Sub CA on the Yubikey can be stored in an encrypted configuration file.  This way the Yubikey can only be used by that specific Mfg Utility configuration.
  • Mfg Utility writes x.509 certificate to device
  • Mfg Utility sends additional information (serial #, IoT endpoint) to be stored in NVS
  • Mfg Utility overwrites bootloader partition with secure bootloader and instructs it to boot into Application binary partition on the next boot
    1. Secure boot loader will need to be signed by the same key used to sign the Application binary
  • On initial boot: this can take ~60s
    1. Secure bootloader validates signature of Application binary
    2. Application is encrypted in place using a flash encryption key generated on the device
    3. Unencrypted device stamp stored in NVS during manufacturing is moved to Encrypted NVS and the unencrypted NVS is erased
  • At this point the device is fully secured and can only be updated with a signed application binary via Over the Air updates.

 

This example is just one of many possible flows. It’s possible to generate information needed from the stamp on the device, on the machine running the manufacturing utility, or remotely in the cloud. The ideal flow will depend on:

  • The number of devices
  • Product specific considerations
  • The location and trustworthiness of the manufacturing facility
  • Institutional experience securing sensitive data

The utilities and workflows we’ve developed at Spindance are based on more than a decade of experience with IoT devices. Connected products have unique security requirements during manufacturing that are often overlooked.  While it’s impossible to completely eliminate vulnerabilities, a well planned manufacturing strategy can significantly reduce the cost per device without compromising security.

About the author, Jonathan Rand:

Currently a Product Owner at SpinDance. Jonathan has always had an innate desire to know how things work; not just devices but also people and personalities. He believes the best outcomes come from a team that is empowered and invested in a shared vision. The Product Owner role allows him to balance compassion for people, a desire to create, and an interest in all things tech.

At SpinDance, Jonathan helps organizations bridge the gap between ambitious product ideas and the technical skillset required to make them a reality. He has led the product development for multiple custom IoT platforms that support thousands of connected products. He is passionate about developing strategies that will continue to deliver value over the long term.