Context - Experience at Groundlight

During my internship at Groundlight, one of the projects I worked on was an ESP32 Camera device for taking pictures to be input into the Groundlight Vision pipeline. I started by working on the firmware for the ESP32 Camera boards, which is open-source, and you can find here.

I had previously build a WebSerial tool for programming AeroSense devices previously, and knew that a similar tool for programming device settings would enhance the user workflow.

I then took this one step further, and developed a web tool that could not only setup a flashed device, but could also flash firmware onto a new device. This was crucial to improving our new user pipeline, making it far easier to try the Groundlight Vision pipeline.

My crucial mistake

Despite my best efforts to recall how I made the system at Groundlight, I totally forgot. Unfortunately, I didn’t forget that it was possible, leaving me in the sad state of knowing that I could reimplement the firmware flashing feature but without knowing how.

… but now I can share my experience with you guys!

Attempt #1

The first thing I tried was using the npm library esptool.ts. This library is marketed as the only javascript esptool implementation with proper types and in a proper library format in the readme.

After implementing a small wrapper around the esptool.ts flashing functionality, I tried to flash my ESP32 board. And it didn’t work. I got the following console error: Unknown chip family. Wtf.

The board I was using housed an ESP32-S3 chip, which has been released for some time now. I looked through the library source code and found the following supported chips:

export enum ChipFamily {
  ESP32 = "esp32",
  ESP8266 = "esp8266",
  ESP32S2 = "esp32S2",
}

And when was the library last updated? Last Week?? Oh COME ON.

Attempt #2

It turns out that espressif (the company behind ESP32 chips) makes an esptool-js library, and it IS actually formatted as a proper library, includes types, and is available on npm.

After implementing another small wrapper around the flashing functionality, I tried to flash my device, and… no dice. Hmm.

A mystery

The first thing I tried was using the esptool-js official online tool. And it didn’t work either? Weird. I then put my device into bootloader mode, like I do before I flash it normally… wait a second!

Why do I have to put my device into bootloader mode to be able to flash it? I know for a fact that this is not necessary for some of my ESP32 boards.

Why do I have to put this board into bootloader mode for programming?

I took another board out of my box of parts, and it connected to the esptool-js online tool, and it worked without any intervention! I tried to flash it again, and it didn’t work?? Aaahhhfuck

It must be a firmware issue. To confirm my suspicions I flashed a blink program onto the initial board, and it connected to the esptool-js official tool just fine.

More mysterious

I narrowed the firmware differences for a successful upload without being put in bootloader mode down to a single line of code:

Serial.begin(115200);

Seriously?? This couldn’t possibly be a bug.

Automatic Bootloader

aka a lesson in reading the datasheet first

After reading the datasheet for the ESP32-S3 chip, there is a specific circuit required for the chip to automatically enter bootloader mode with the execution of the esptool program. This circuit requires a serial converter chip (i.e. FTDI bridge), and connecting RTS and DTR with EN (RESET) and GPIO0 (BOOT) respectively, and is described here.

Therefore it is actually a bug (not a feature) that the automatic bootloader function works on the boards I was using, because they do not have this circuit implemented.

Getting everything else working

I may not have an interesting story about getting the rest of it working, but I can provide a working example for esptool-js version 0.4.3 for anyone else who is struggling with the same problems I was.

export const flash = async (
    port: SerialPort,
    partitions: { data: Uint8Array, address: number }[],
    getProgress?: (fileIndex: number, written: number, total: number) => void
) => {
    // if the serial port is open, it must be closed
    // await port.close();

    const transport = new Transport(port);
    const loaderOptions = {
        transport,
        baudrate: 1500000,
        romBaudrate: 115200,
    } as LoaderOptions;
    
    const esploader = new ESPLoader(loaderOptions);
    const chip = await esploader.main();

    const flashOptions: FlashOptions = {
        fileArray: partitions.map(p => ({
            data: esploader.ui8ToBstr(p.data),
            address: p.address,
        })),
        flashSize: "keep",
        flashMode: "keep",
        flashFreq: "keep",
        eraseAll: false,
        compress: true,
        reportProgress: (fileIndex, written, total) => {
            getProgress?.(fileIndex, written, total);
        },
    } as FlashOptions;
    await esploader.writeFlash(flashOptions);
}

Remember that there are multiple failure points in this function, so it’s a good idea to wrap the whole thing in a try/catch. Cheers!