Delta Firmware Updates over Cellular LPWAN
Since the October release of Pelion Device Management Services, the platform introduced support for updating a device firmware using a feature called delta updates. For some time now, it was on my list of things to reserve time and play with it and write down my thoughts. This blog is the result of my experiment.
Delta updates are a critical feature when trying to update IoT devices in the field cause in the majority of situations if not all, the network bandwidth is limited and costly. Any mechanism that can be employed to reduce network usage has a huge effect on savings, both in terms of power usage of the device while receiving the firmware and staying awake and the associated bandwidth cost.
In essence, a delta update is a mechanism to push a delta (the binary difference) of the firmware currently running in the device with the newly updated one. Pelion employs the well-established and popular bsdiff algorithm (with some Arm extensions) to generate the binary delta, resulting in considerable size savings as we’ll see later on.
For the interested reader who wants to read more about bsdiff internals can have a look at the paper ‘Naive differences of executable code’ from the original author of bsdiff, Colin Percival.
To demonstrate those size savings we will take the official mbed-cloud-client example that is used as our reference code to connect to Pelion cloud, produce delta firmware updates and exercise them under different networks types, Wi-Fi, Cat-M1 and NB-IoT. We will mark down the delta size reductions as well as the time spent to deliver those firmware’s to devices using across the different network types.
But first, let’s describe the steps required to produce a delta firmware image:
NOTE: We showcase the delta firmware generation using our command-line tools to shed a light on the “behind-the-scenes” process. That said, our integrated development environment Mbed Studio provides built-in support for the generation and publishing of firmware with delta generation support currently in development (scheduled for the upcoming v1.0)
Producing the Delta image — In a nutshell
In our documentation, you can find a step-by-step tutorial on how to prepare and launch firmware (and delta) updates with explanations of the different concepts, but in summary, it involves the following:
NOTE: The same steps to produce a delta firmware are applicable on any board supported by Pelion Device Management Client library as long as is ‘Pelion Device Ready’ certified and able to connect to Pelion Device Management Service with some form of network connectivity.
Let’s start by cloning the mbed-os-example-pelion repository:
$ mbed import https://github.com/ARMmbed/mbed-os-example-pelion
[mbed] Working path "/Users/chrvas02/Projects" (directory)
[mbed] Importing program "mbed-os-example-pelion" from "https://github.com/ARMmbed/mbed-os-example-pelion" at latest revision in the current branch
[mbed] Adding library "mbed-cloud-client" from "https://github.com/ARMmbed/mbed-cloud-client" at rev #7b583acf30ca
[mbed] Adding library "mbed-os" from "https://github.com/ARMmbed/mbed-os" at rev #e642a7d8b360
[mbed] Adding library "drivers/COMPONENT_WIFI_ISM43362" from "https://github.com/ARMmbed/wifi-ism43362" at rev #1978369b2310
[mbed] Adding library "drivers/COMPONENT_WIFI_ESP32" from "https://github.com/ARMmbed/esp32-driver" at rev #d6bd83f8710b
[mbed] Adding library "drivers/storage/COMPONENT_NUSD" from "https://github.com/OpenNuvoton/NuMaker-mbed-SD-driver" at rev #56406071a446
[mbed] Adding library "TESTS/pelion-e2e-python-test-library" from "https://github.com/ARMmbed/pelion-e2e-python-test-library" at rev #4a3a533f8b21Prior to building and flashing on the device, we need to create both a device certificate and an authentication certificate. The former is used by the device to be able to connect to Pelion Device Management Service while the latter is used to ensure the updates come from a trusted source.
For development purposes, we can utilize the ‘mbed dm init’ to create them automatically for us, passing only our desired device metadata:
NOTE: We are using developer certificates in our example but in a production setting you should really follow the guidelines as described in our Factory Provisioning documentation.
$ mbed dm init -a "<API_KEY>" -d "letsmakecoffe.com" --model-name "awesome-coffe-machine" -q --force
[mbed] Working path "/Users/chrvas02/Projects/mbed-os-example-pelion" (program)
[INFO] 2020-03-13 15:54:32 - __main__ - Found developer certificate named Dev-Certificate
[INFO] 2020-03-13 15:54:32 - __main__ - Writing developer certificate Dev-Certificate into c file mbed_cloud_dev_credentials.c
[WARNING]: Certificates generated with this tool are self-signed and for testing only
[WARNING]: This certificate is valid for 90 days. For production,use certificates with at least 10 years validity.
[INFO] 2020-03-13 15:54:32 - manifesttool.init - Certificate written to .update-certificates/default.der
[INFO] 2020-03-13 15:54:32 - manifesttool.init - Private key written to .update-certificates/default.key.pem
[INFO] 2020-03-13 15:54:32 - manifesttool.init - Default settings written to .manifest_tool.json
[INFO] 2020-03-13 15:54:32 - manifesttool.init - Wrote default resource values to update_default_resources.cThe tool generated all the required developer certificates, the private key to sign our future updates and the manifest file ‘.manifest_tool.json’ file which contains the metadata of our chosen configuration. This file will be used later in the tutorial when we create our delta image:
NOTE: Manifests play a crucial role in a device firmware update. That said, in a future blog post we’ll describe the background behind the manifest and offer some tips and advice when generating them. Stay tuned!
$ cat .manifest_tool.json | jq{
"classId": "b5b0afb3-06e9-5bc6-81b6-a413ddaa4645",
"default-certificates": [
{
"file": ".update-certificates/default.der"
}
],
"deviceURNs": [],
"modelName": "awesome-coffe-machine",
"private-key": ".update-certificates/default.key.pem",
"vendorDomain": "letsmakecoffe.com",
"vendorId": "0e56af43-f31f-574c-aa45-ad54915ca9fe"
}
We now need to set up network connectivity and configure accordingly the project’s configuration “mbed_app.json” file with the appropriate parameters.
Cellular
For Cellular connectivity, mandatory parameters are the following:
Wi-Fi
For Wi-Fi connectivity, wireless credentials must be entered:
NOTE: Depending on the type of board and the wireless/cellular module installed, a number of extra parameters might also be required and you should consult the documentation of your board for more information.
Now we are ready to compile and flash the initial firmware on the device:
$ mbed compile -m auto --flash[mbed] Working path "/Users/chrvas02/Projects/mbed-os-example-pelion" (program)
[mbed] Detected "DISCO_L475VG_IOT01A" connected to "/Volumes/DIS_L4IOT" and using com port "/dev/tty.usbmodem141203"
Building project mbed-os-example-pelion (DISCO_L475VG_IOT01A, GCC_ARM)
Scan: mbed-os-example-pelion
Using ROM regions bootloader, header, application in this build.
Region bootloader: size 0x9000, offset 0x8000000
Region header: size 0x70, offset 0x8011000
Region application: size 0xeec00, offset 0x8011400
Compile [ 0.1%]: CertificateEnrollmentClient.cpp
Compile [ 0.2%]: BufferedPrint.c
Compile [ 0.3%]: CertificateEnrollmentClientCommon.cpp...
...
...
Update Image: ./BUILD/DISCO_L475VG_IOT01A/GCC_ARM/mbed-os-example-pelion_update.bin. [1]
Image: ./BUILD/DISCO_L475VG_IOT01A/GCC_ARM/mbed-os-example-pelion.bin. [2]
Notice at the end of the compilation process, the compiler produces two images:
- The update image [1], which contains only the application code and is used for updating the device.
- The full image [2], which combines the application with the bootloader and metadata. This image is initially flashed on the device.
Let’s open a serial terminal to the board to verify that indeed the board has connected to Pelion Device Management Service:
I am using CoolTerm app on a Mac to access serial, if you are on some other platform take a look at our documentation for a list of terminal applications you can use.
And Pelion Portal should display the device as connected:
Let’s save this initial image in a ‘firmwares/’ directory on the project and give it a new name.
$ mkdir firmwares/
$ cp ./BUILD/DISCO_L475VG_IOT01A/GCC_ARM/mbed-os-example-pelion_update.bin firmwares/mbed-os-example-
pelion_original.binModify the code to simulate an update of the running firmware and produce a new image. Open ‘main.cpp’ in the project’s root directory and add a ‘printf()’ statement:
And compile again:
$ mbed compile -m autoThis time we don’t flash on the device (no ‘–flash’ argument passed) since we are only interested in the compiler artifacts.
Let’s copy the resulted firmware to our ‘firmwares/’ directory and give it a new name:
$ cp ./BUILD/DISCO_L475VG_IOT01A/GCC_ARM/mbed-os-example-pelion_update.bin firmwares/mbed-os-example-pelion_update.binNow, we have two firmwares, the original firmware that was flashed on the device ‘mbed-os-example-pelion_original.bin’ and the new one ‘mbed-os-example-pelion_update.bin’ with our simulated change:
$ ls -l firmwares/total 1504
-rw-r--r-- 1 chrvas02 staff 381408 25 Mar 13:59 mbed-os-example-pelion_original.bin
-rw-r--r-- 1 chrvas02 staff 381432 25 Mar 14:05 mbed-os-example-pelion_update.bin
We are now ready to create the delta firmware image using the delta-tool.
To do so we need to pass both firmwares and the ‘manifest_tool.json’ configuration file generated by the ‘manifest-tool’ in our previous step. Further, we need to specify the filename of the configuration file that which will contain the metadata of the delta update (’-o delta-tool-generated-manifest.json’)
$ python3 delta-tool/tools/delta-tool.py \
-b delta-tool/bsdiff/bsdiff \
firmwares/mbed-os-example-pelion_original.bin \
firmwares/mbed-os-example-pelion_update.bin \
-i .manifest_tool.json \
-d firmwares/delta_image.bin \
-o delta-tool-generated-manifest.jsonAfter executing the command, our delta firmware ‘delta_image.bin’ file is produced:
$ ls -l firmwares/total 1552
-rw-r--r-- 1 chrvas02 staff 23984 25 Mar 14:08 delta_image.bin
-rw-r--r-- 1 chrvas02 staff 381408 25 Mar 13:59 mbed-os-example-pelion_original.bin
-rw-r--r-- 1 chrvas02 staff 381432 25 Mar 14:05 mbed-os-example-pelion_update.bin
Notice that the delta firmware is reduced to ~24K from the ~381K of the updated firmware, a considerable improvement!
And the generated ‘delta-tool-generated-manifest.json’ with the delta update metadata:
$ cat delta-tool-generated-manifest.json | jq{
"classId": "b5b0afb3-06e9-5bc6-81b6-a413ddaa4645",
"default-certificates": [
{
"file": ".update-certificates/default.der"
}
],
"deviceURNs": [],
"modelName": "awesome-coffe-machine",
"private-key": ".update-certificates/default.key.pem",
"vendorDomain": "letsmakecoffe.com",
"vendorId": "0e56af43-f31f-574c-aa45-ad54915ca9fe",
"payloadHash": "aa147139f2d5dc37d84883771af356745471250420243f07e098790fcc008e6f",
"payloadSize": 23984,
"payloadFile": "firmwares/delta_image.bin",
"payloadFormat": "bsdiff-stream",
"installedFile": "firmwares/mbed-os-example-pelion_update.bin",
"deltaFile": "firmwares/delta_image.bin",
"precursorFile": "firmwares/mbed-os-example-pelion_original.bin"
}
We are now ready to upload the update manifest and the delta firmware image to Pelion Device Management Service and prepare for the update campaign. For this, we utilize the ‘manifest-tool update prepare’ command to perform it for us:
$ manifest-tool update prepare -a <API_KEY> -i delta-tool-generated-manifest.json[INFO] 2020-03-25 14:13:45 - manifesttool.prepare - Using delta_image.bin-2020-03-25T14:13:45 as payload name.
[INFO] 2020-03-25 14:13:45 - manifesttool.prepare - Using delta_image.bin-2020-03-25T14:13:45-manifest as manifest name.
[INFO] 2020-03-25 14:13:46 - manifesttool.prepare - Created new firmware at http://firmware-catalog-media-ca57.s3.dualstack.us-east-1.amazonaws.com/delta_image_b76d958802a64f32a484e77207a79339.bin
[INFO] 2020-03-25 14:13:46 - manifesttool.prepare - Created new manifest at http://firmware-catalog-media-ca57.s3.dualstack.us-east-1.amazonaws.com/manifest_dd62b06f53e641dea190795a0ef5c0db
[INFO] 2020-03-25 14:13:46 - manifesttool.prepare - Manifest ID: 017135e055c70000000000010010017c
Now that both the firmware and the manifest file are uploaded to Pelion Device Management Service, we are ready to start the campaign to update our device. Although it’s possible to use the command line to perform the operation using the Pelion service APIs, instead we will utilize the Pelion portal this time. Take a look at the video below that demonstrates creating and starting the update campaign:
Analysis
Now that we’ve described how delta firmware updates work and the size reduction advantage they bring when producing the image, let’s exercise them across wireless and cellular and mark down the time spent to deliver those updates. A quick search online will reveal plenty of material that describe the technical merits of each cellular technology and areas where it excels (or fails) in IoT uses cases, so we won’t spend time repeating the same information here. For the interesting reader though who would like to learn more, we recommend watching the webinar comparing NB-IOT and Cat-M1 created by the Pelion Connectivity team here at Arm, that goes into full details on the differences between these two cellular technologies. Further, it includes a lengthy QA section of many common questions asked from our users and developers and is worth watching.
For reference, here is a summary table that highlights the differences:
The following tables are the result of our experiment running a firmware update across the different connectivity types. For each type, we record the time taken to update the firmware using both a full and a delta image (top and bottom respectively):
If we collect the results and attempt to create a graph, we get the following picture:
Clearly and regardless of the chosen connectivity type, the impact that the delta update has on limiting the transmission time is significant, over ~90% in all cases. Although configurations may vary across different use cases and environments, employing delta firmware updates can prove a significant aid in the never-ending quest to save bandwidth costs and preserve the battery of the device.
You can find the sheet with the data in our shared document plus a description of the hardware and mobile operators used to conduct the testing as well as the logs from the devices.
Conclusion
Updating the firmware of the device is a critical feature of any IoT platform that ensures the device runs the latest code with any security fixes released after its deployment in the production field. Further, optimizing the delivery of the firmware using delta updates aids in saving network bandwidth costs and preserving the device battery to expand its lifespan. Keep in mind though, regardless of how far these optimizations can go, firmware updates remains an energy-heavy operation and as noted in our “Going to Production guide”:
“You should understand the energy requirements of both firmware update and your device’s normal operations, and plan update frequencies that don’t drastically reduce the time between battery changes. For this, remember that the accessibility of the battery is important — a device placed in plain sight (for example, mounted on a wall in a normal house) can have a shorter battery life than a device buried in the ground along a pipeline.”
Pelion Device Management Service provides robust support for firmware and delta updates and already is used by many of our customers to implement their production update strategy. We provide free-tier access to the platform for our users and developers so they can easily get started so why not creating an account and exercise the delta update mechanism we discussed here.
Are you next?
Acknowledgements
I would like to sincerely thank my colleague Daniel Lee for his valuable help during the conducting of tests and in the writing of this post.
