Getting Started with Flutter for Embedded Linux
For a few years now at SpinDance, we’ve been looking for a great solution for creating nice looking UIs for embedded systems that have a screen on them. Until now, we haven’t really been happy with the options out there. The tide is now turning on that with the rise of Flutter. Read on to see why Flutter on embedded Linux is becoming our GOTO solution for this space.
Understanding Dart and Flutter
Flutter’s Rise in UI Development
Flutter is an open-source UI toolkit from Google. Dart is the actual programming language behind Flutter. While Flutter is best known for building cross platform mobile apps, it is capable of allowing developers to build natively compiled applications for various platforms, including mobile, web, desktop, and even embedded, from a unified codebase. Flutter’s popularity has been on the rise over the past few years and our developers here at SpinDance really like it as an alternative to cross platform development frameworks like React Native.
Source: https://www.statista.com/statistics/869224/worldwide-software-developer-working-hours
Embedder Layer: The Key to Flexibility
Flutter’s embedder layer is particularly significant for embedded Linux systems. This layer interfaces directly with the OS, allowing for optimized performance on embedded devices. For a deeper understanding, refer to Flutter’s [Architectural Layers](https://docs.flutter.dev/resources/architectural-overview#architectural-layers).
Embedded Linux Explained
The Role of Embedded Linux
Embedded Linux brings the power of Linux to embedded systems like IoT devices. It offers customization, a supportive open-source community, and is ideal for systems with moderate to high resource availability. While many embedded devices can run perfectly fine on a smaller microcontroller with bare metal or an RTOS (real time operating system), sometimes you just need more horsepower. This is particularly true if you are doing higher end processing like machine learning or if your embedded system is supporting a higher-end user interface such as a full-color touchscreen.
Embedded Linux also brings the benefits of a full operating system such as thread management, memory management, drivers for peripherals, and application management. This can significantly speed up development time in certain applications.
Yocto and Build Systems
We won’t go deep into the build system for this post, but it is worth mentioning that there are a number of ways to build a custom linux OS for your particular application hardware and application stack that gets rid of all the fluff in standard distributions like Ubuntu. This is a great way to optimize your OS for hardware limitations like CPU power, memory, and disk space.
For this application, we chose to use the Yocto Project (https://www.yoctoproject.org/) to create our custom OS.
UX in Embedded Devices: Evolving Trends
You can put a lot of effort into building an amazing embedded device for your users, but at the end of the day, if it has a poor user experience, people aren’t going to want to use it. In many traditional embedded systems, the user experience is often a simple display and a handful of buttons or dials. A microwave is a perfect example of this. There is a simple LCD display with limited functionality. There are a handful of buttons for setting the time and performing various functions. For many microwaves, there are extra features like defrosting by weight that require the right incantation and a fair bit of luck to get it working right.
For generations that have grown up with dynamic touchscreen interfaces their whole lives, many people are becoming less tolerant of these less intuitive experiences. Having a touchscreen that you can update to display exactly what the user needs at the right time can significantly improve someone’s level of satisfaction with your device. I have a sneaking suspicion that this trend is going to continue.
Is the right response “Let’s just slap a touchscreen UI on everything!”? Definitely not, but more things are likely to move in that direction. Personally, I don’t really want a touchscreen on my toaster.
So let’s say you’ve made the decision to build a nice UI for your fancy new touchscreen device, now comes the tough part of choosing what technology to use to build that UI.
Existing UI Tooling in Embedded Systems
User interfaces for embedded devices aren’t new. We’ve been building them for years, and there are a number of ways that people have done it.
Qt/QML
From my experience, Qt has been the leader in the embedded UI space for quite some time. It supports building applications in both Python and C++. It has a rich set of UI components and a fairly comprehensive styling system. However, I have yet to meet a developer who actually likes working with it. Additionally, licensing for a commercial application of it can be messy.
Electron/Web
If you can run a browser or render HTML, then another great alternative is to leverage all of the existing UI frameworks that have been built for the Web. This could include things like React, Angular, etc. This has a couple of advantages. The developer pool and experience for these technologies run deep, which is great. UI/UX designers are also well versed in designing applications based on web frameworks. However, with any web framework, you get to deal with all of the complications that come with it. This can include complex dependency management, frequent updates, a fragile and complex rendering stack, and a likely need for having a backend and frontend application both running on your device. In our experience, this works, but is overly complex and not very efficient.
Low Level Rendering
You can swing the other direction and also use more direct pixel-by-pixel rendering to the frame buffer. This is highly efficient, but I would be hard-pressed to recommend doing this for an embedded linux application unless you are really resource constrained. The developer experience is painful and it is going to take a long time to build out some of the basic functionality that you would expect in a modern application.
Flutter
Enter Flutter. For us, Flutter checks all the boxes for making it a great choice for building UIs for embedded systems. The embedder layer provides a lot of flexibility for optimization. The developer experience is on par with developing for mobile applications. UI/UX designers know how to work within the confines of this framework. You can also get all of your frontend and backend functionality packaged within a single application if that makes sense for your use case. Since Flutter is open-source with no complex licensing, it also makes it very easy to use for commercial applications. On top of this, the memory and CPU performance of Flutter is pretty phenomenal.
The biggest drawback of Flutter for embedded Linux is that it is still pretty new, so community resources are still light in this area and not all packages for Flutter and Dart support Linux out of the box. So far, this hasn’t been a major issue for us. But what does it look like to actually get started with this?
Starting with Embedded Flutter
Selecting the Right Hardware
Key considerations include board architecture (Aarch64, ARMv8), balance between RAM, processing power, storage needs, and display/input characteristics. We would recommend the following:
- x64 or ARM64 CPU architecture
- At least 512MB of RAM
- A multi-core processor with at least 1GHz clock speed
For storage, using something like a micro SD card works, but for a production build, we would recommend using eMMC if possible.
Choosing the Right Flutter Embedder
Unfortunately, embedded Linux embedders are not officially supported by Flutter yet. However, several custom embedders have been created by and supported by Sony. Many community versions exist as well and all of them are open source, which is great for any customizations you might need. The main difference between the different embedders is how rendering to displays is handled. At a high level, X11, Wayland, and DRM are the top options.
- X11: Great for multi-display applications and lots of community resources. However, this is the most resource intensive renderer and comes with a handful of security concerns that aren’t ideal.
- Wayland: This is a more modern alternative to X11. It also provides great support for multi-display applications while being less resource intensive.
- DRM: This is a more direct-to-screen rendering approach which is perfect for single screen applications. It is the best for its low resource utilization, but takes a bit more work from the developer to get it set up and working correctly. There is also a specific embedder created and tuned for the Raspberry Pi that we tested.
Demo and Debugging
To demonstrate and test running a Flutter app on an embedded device, we used the following hardware:
- Raspberry Pi Zero 2W
- 512MB RAM
- 1GHz Quad-Core ARM64
- uSD card
- Waveshare 7” Capacitive Touch LCD
- Refresh rate = 60Hz
- FPS = 60
The Linux OS built with Yocto used the following components:
- Built with Yocto Project (Poky)
- Base image with minimal software
- Added:
- Weston (Wayland compositor)
- Flutter engine
- Flutter Wayland (embedder / backend)
- Flutter DRM GBM (embedder / backend)
- Flutter-pi (embedder / backend)
- Flutter Gallery App
Once the image was built and flashed to the Raspberry Pi Zero, the Flutter Gallery app was booted up using the 3 different embedders listed above.
Below is a summary of the performance while running a full screen animation.
From our testing, both the DRM embedder and the Flutter-pi embedder performed really well. There was a little bit of lag noticed on the touch screen when using dragging interactions, but overall the app is amazingly responsive for the limited hardware it is on.
Beyond just the performance, we are still working on best practices for developing and debugging Flutter on embedded devices, so stay tuned.
A link to the repo we used for this demo can be found here:
https://github.com/lapumb-spindance/devfest-demo-pi
Conclusion: Flutter’s Role in Embedded Applications
Flutter isn’t a universal solution for every embedded application. However, it excels in scenarios where a higher end UI is needed. Keeping designs simple and focusing on user experience are paramount in leveraging Flutter’s full potential in embedded systems. We are pretty excited about this for future projects…especially if we can avoid having to write anymore embedded UIs in Qt or React.
Feel free to reach out if you have any questions or want to talk more.
For another example, check out how Toyota leveraged Flutter for their infotainment systems. Link is here: https://flutter.dev/showcase/toyota