(AKA, remember to flush your buffers, USB ZLP edition)

Introduction

In USB, zero-length packets (ZLPs) have an important role in some types of data transfers. Implementation errors related to ZLPs cause many problems that are subtle and difficult to debug, especially in embedded devices. ZLP refers only to the data payload size: there are additional bytes of protocol framing around the payload.

The most common implementation errors are in the Communication Device Class (CDC), often used to emulate legacy serial protocols such as RS-232. CDC interfaces are common on Arduino and related microcontroller (MCU) boards that are designed to be accessible to inexperienced developers.

I’m going to be mostly talking about full-speed (12Mbits/sec) USB, because that’s what most cheap USB-capable microcontrollers can manage.

Emulating a stream

USB CDC emulates a serial protocol stream using bulk transfer packets. A transfer can consist of multiple packets. In Full-speed USB, these packets have a maximum payload size of 64 bytes.

The USB 2.0 specification standardizes some software abstractions on the host side. One of these is the I/O Request Packet (IRP), which is how client software requests service from the USB Driver (USBD). The IRP includes a buffer that the client uses to send or receive data.

When a bulk IRP involves more data than can fit in one maximum-sized data payload, all data payloads are required to be maximum size except for the last data payload, which will contain the remaining data. A bulk transfer is complete when the endpoint does one of the following:

  • Has transferred exactly the amount of data expected
  • Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet

There is no standard buffer size for CDC, so hosts and devices can use any buffer size that is appropriate for their purposes. This means that when a bulk transfer to the host ends with a maximum-sized packet, it must be followed by a ZLP in order for the host to retire the IRP and notify the client of completion. (This assumes that the transfer doesn’t completely fill the buffer provided with the IRP.) Likewise, the host may send a ZLP following a transfer that is a multiple of the maximum packet size.

Receiving ZLPs

Some MCU USB firmware stacks have bugs related to receiving ZLPs. Often, the APIs don’t have a way to distinguish receiving a ZLP from not receiving a packet at all. If the USB stack does flow control of incoming data by sending NAK to the host when the application hasn’t read the data, this means that a ZLP can cause that endpoint to stop receiving data. The arrival of a ZLP causes the USB stack to start sending NAK in response to new data, but the application doesn’t know that the ZLP has arrived, so it can’t signal to the USB stack to resume receiving. This is still the case for the official Arduino core for AVR. (It also has gone unfixed for more than six years.)

macOS is one host OS that is known to send ZLPs for USB CDC. This means that sometimes, macOS sending USB CDC data that is a multiple of 64 bytes can lock up a MCU’s USB stack (at least on that endpoint).

Sending ZLPs

In the opposite direction, many MCU USB stacks fail to send ZLPs when required. For macOS, Linux, and other Unix-like OSes with a POSIX serial terminal driver abstraction, this often isn’t a problem, because the POSIX terminal driver allows reads of a serial device to return a partial buffer if a certain amount of time has passed since the last reception.

Windows programs often don’t have that behavior, and sometimes use buffers as large as 4096 bytes. This means that a MCU sending USB CDC that is a multiple of 64 bytes might not have that data be read by Windows unless the Windows receive buffer has filled.

One way to fix this is for the USB stack to send a ZLP when the application flushes a write that is a multiple of the maximum packet size. The official Arduino core for AVR overcorrects and always sends a ZLP after any maximum-length packet. In practice, this doesn’t seem to cause problems, though it theoretically could.

Testing

Testing these situations isn’t easy. You need to send transfers that are a multiple of the maximum packet size, and this isn’t easy to confirm without some sort of USB protocol analyzer. Hardware protocol analyzers for USB usually aren’t cheap. Wireshark can do USB capture on some platforms, but isn’t always easy to work with.

Conclusion

Zero-length packets (ZLPs) on USB can cause all kinds of subtle problems that can be difficult to diagnose. If you’re working on an embedded device that speaks USB CDC, it might be a good idea to test transfers that are a multiple of the maximum packet size, in both directions.