Add an LVGL GUI to Lego Mindstorms using LMS-ESP32

Where the screen of the Mindstorms EV3 had a resolution of 178 x 128 pixels, the screens of the Lego Mindstorms Inventor and Lego SPIKE Prime hubs only have a resolution of 5×5 pixels. Wouldn’t it be nice to add a touch-sensitive TFT screen to the Lego Hubs? Keep reading if you are curious how we use the Wifi LMS-ESP32 Micrpython board to achieve this.

Overview of the LVGL (Light and Versatile Graphics Library) Library

You will find many graphics libraries for driving TFT screens, but most of them are limited to basic graphics functions such as drawing lines, rectangles, and text. Building a usable GUI from these basic components is quite a hassle. It becomes even more complex if you would like to add interaction with a touch-sensitive screen.

We propose to use the LVGL (Light and Versatile Graphics Library) library, which not only comes in a C-version but also in a MicroPython version that integrates nicely with our MicroPython eco-system on both the LMS-ESP32 as well as the Lego hubs.

Using LVGL, you can choose from a large variety of widgets. For instance, a meter widget shown to the left or buttons, labels, sliders, etc. LVGL takes care of handling the touch events on the different widgets. You decide what should happen when a button is clicked by using call-back functions in your code.

To get an idea of what widgets are available in LVGL, you can look at the LVGL examples.

Basic Micropython LVGL setup with LMS-ESP32

The default firmware that comes with the LMS-ESP32 already has the LVGL Micropython library integrated. There is no need to load additional software libraries on the LMS-ESP32.

The code below initiates the LVGL library for the ILI9341 TFT screen and touch panel. Because the LVGL library uses the 4MByte PSRAM present in the ESP32-WROVER module, running this code twice results in an error. We advise you to reset the LMS-ESP32 before running the code below.

import espidf as esp
import lvgl as lv
from ili9XXX import ili9341,LANDSCAPE
from xpt2046 import xpt2046

#init TFT display
disp = ili9341( miso=12, mosi=13, clk=14, cs=15, dc=23, rst=25,
                backlight=-1,power=-1,width=320, height=240, rot=LANDSCAPE)
# use same SPI as display, init touch
touch = xpt2046(spihost=esp.HSPI_HOST,cs=26,transpose=False,
                cal_x0=3865, cal_y0=329, cal_x1=399, cal_y1=3870)

Once the screen is initialized, generating a GUI from the widgets in the LVGL library is very easy. For example, drawing a button in the middle of the screen is as easy as follows:

scr =  lv.scr_act()
btn = lv.btn(scr)
label = lv.label(btn)
label.set_text("Button")
btn.center()

Demo

In the tutorial video below, the LMS-ESP32 drives a TFT screen using LVGL with a linear slider and a meter widget. The values of the widgets are being sent from the Lego hub using the UartRemote library.

Code on the LMS-ESP32

All the code running on the LMS-ESP32 is shown below. The first few lines of the code initialize the TFT screen (which has an ILI9341 controller) and the touch display (with an XPT2046 controller). The calibration values for the touch panel are just taken from an example and should be good to go.

from uartremote import *
import espidf as esp
import lvgl as lv
from ili9XXX import ili9341,LANDSCAPE
from xpt2046 import xpt2046

#init display
disp = ili9341(miso=12, mosi=13, clk=14, cs=15, dc=23, rst=25,
               backlight=-1,power=-1,
               width=320, height=240, rot=LANDSCAPE)
# use same SPI as display, init touch
touch = xpt2046(spihost=esp.HSPI_HOST,cs=26,transpose=False,
                cal_x0=3865, cal_y0=329, cal_x1=399, cal_y1=3870)


arc = lv.arc(lv.scr_act())
arc.set_range(0,360)
arc.set_bg_angles(0,360)
arc.set_size(150,150)
arc.align(lv.ALIGN.CENTER,0,0)


slider = lv.slider(lv.scr_act())
slider.set_width(200)
slider.align(lv.ALIGN.CENTER,0,-100)
slider.set_range(0,360)

label1 = lv.label(lv.scr_act())
label1.set_long_mode(lv.label.LONG.SCROLL_CIRCULAR)         # Circular scroll
label1.set_width(150)
label1.set_text("Demo using LVGL library. ")
label1.align(lv.ALIGN.CENTER, 0, 100)
ur=UartRemote()

def show_angle(angle):
    arc.set_value(angle%360)
    slider.set_value(angle%360,1)
    
ur.add_command(show_angle)

ur.loop()

Next, three different widgets are initialized on the TFT screen.:

• arc-widget
  We set the arc range between 0 and 360. Furthermore, by changing the start and stop angles to 0 and 360, respectively, we obtain a full 360 arc, After setting the size to 150, the arc widget is moved to the middle of the screen.
• slider-widget
  We set the slider’s width to 200, center it horizontally, move it to the top of the screen, and finally, set the range between 0 and 360.
• label-widget
  This widget is not necessary for the demo, but just shows a scrolling text

Next, the function show_angle is defined that sets the values of both the arc as well as the slider to the values angle. We add this function to the UartRemote commands, and finally, we call the UartRemote `ur.loop()`, which is waiting for any command to be received through the UartRemote link.

Code on the Lego Mindstorms

Below you find the code running on the Lego Mindstorms Inventor.

from projects.mpy_robot_tools.uartremote import *
import hub

ur=UartRemote('A')
motor = hub.port.B.motor


motor.mode([(2,0)]) # absolute position, raw units

motor.preset(0)
while (True):
    m=motor.get()[0]
    ack,val=ur.call('show_angle','repr',m)
    

We first import the UartRemote library. In this example, we connect the motor to port B. By setting the motor.mode to [(2,0]), we use absolute positions in raw units. Then we can use the motor.get() method to obtain the absolute angle of the motor stored in the first element of the returning array. That angle is used to call the show_angle function remotely on the LMS-ESP32 using a UartRemote library call command

Hardware

In this tutorial, we use a TFT screen with a resistant touch panel with a resolution of 340×240 full-color pixels. We deploy a single SPI (Serial Peripheral Interface) interface for both the TFT screen and the touch panel, each having its own CS (Chip Select). This limits the number of cables needed to connect the display. We connect the panel with a 1mm pitched 10-wire cable to the LMS-ESP32’s GPIO port. This cable takes much less space than 10 DuPont cables. We designed break-out boards for both the panel and the GPIO connection that accommodate the 1mm pitched connectors. For applications where you do not need touch capabilities, you could use the solder pads for changing the spare wire to switch the TFT backlight.

For prototyping purposes, you can also hook up a TFT display directly. When using the TFT panel with the Touch panel, you need to connect the MOSI, MISO, and SCK signals of the TFT SPI bus with the similar signals of the Touch controller, namely T_DI, T_DO, and T_CLK, respectively. The chip select of the touch panel, T_CS, is still needed as a separate signal. This is shown in the schematic below.

The TFT_CS and TS_CS remain separate signals. The PCB board we provide takes care of these connections and break out on a 10-pin 1mm wire. The other side of this 10-wire cable is connected to the GPIO bus of the LMS-ESP32. The VCC and GND pins are directly connected to the 3V3 and GND pins on the GPIO header of the LMS-ESP32. The mapping of the signals from the display and touch panel to the GPIO pins on the LMS-ESP32 board is shown below:

Things can be simplified when you deploy only the TFT screen without the touch panel. Then you do not need to make the connection with the T_CS, T_DI, T_DO, and T_SLK pins.

Shopping list

What should you order to get started? You should collect a touch-sensitive TFT screen with an SPI interface at your favorite store. We can provide you with the break-out PCBs, the connectors, and the cables. If you have a stable hand, you can solder the connectors yourselves or order the PCBs pre-soldered.

Advanced usage of this TFT board

Mr Jos uses our TFT PCBs and an LMS-ESP32 hooked up to lego Mindstorms EV3 to create beautiful GUIs for his projects. Look at his latest creation: a Lego Pin sort machine where you can follow the sorting progress on the TFT screen.

He created icons of all the different Legopins and shows them in the GUI.