A fast USB storage gadget on the Pine A64

While iterating on some code that runs on my Pine A64 I got tired of constantly moving the SD card between the Pine A64 and my laptop to flash a new image. It’s cumbersome to move the SD card back and forth, and also prevents remote development.

It turns out that it is possible turn the Pine64¹ into a USB device which exposes the SD card² as a mass storage device over USB. This allows the SD card to be accessed over USB from another computer without removing it from the Pine A64. There are two approaches to doing this, one relies on u-boot only and another which relies using a Linux kernel module. This post covers how to use both.

FEL Mode

The key technology to this approach while keeping the SD card untouched is the FEL mode that all Allwinner CPUs have. When there is no bootable media attached, the boot ROM will turn the OTG USB port into a peripheral that has a special protocol. This protocol allows the host to read and write to arbitrary memory and execute code in memory.

The combination of these primitives means it’s possible to do the following:

• Load the u-boot SPL and execute it
• Load a full u-boot and execute it
• Optionally load a Linux kernel image and initrd for u-boot to load

Since the Pine A64 can only boot from the SD card, entering FEL mode is pretty easy since the boot ROM looks for a header on sector 16 of the SD card, it’s easy to wipe out the boot header with the following command:

$ dd if=/dev/zero of=/dev/mmcblk0 bs=1024 count=1 seek=8

After running this and rebooting, the device will enter FEL mode. Assuming the OTG port³ is connected to another computer, that computer should see a device with Allwinner’s USB vendor id 0x1f3a.

$ lsusb -d 1f3a:
Bus 005 Device 066: ID 1f3a:efe8 Allwinner Technology sunxi SoC OTG connector in FEL/flashing mode

To interact the device in FEL mode, the primary tool is from the sunxi-tools repository. At the time of this post, building from latest master would provide a sunxi-fel binary that can list the Pine A64 in FEL mode.

$ ./sunxi-fel -l
USB device 005:066   Allwinner A64     92c001ba:44004620:78918300:4023030f

sunxi-fel provides a uboot command that can load u-boot into memory and boot into it.

$ ./sunxi-fel uboot u-boot-sunxi-with-spl.bin

The serial console should print out output from the SPL and then eventually drop into the u-boot prompt.

U-Boot SPL 2021.10 (Jan 01 1970 - 00:00:01 +0000)
DRAM: 1024 MiB
Trying to boot from FEL
...

Using U-Boot

With u-boot it’s possible to expose the SD card over USB with the ums command once it is loaded via FEL mode. sunx-fel also supports injecting u-boot a script into the environment so long as the file start with a special header of #=uEnv and is written to a special memory location.

#=uEnv
ums_and_reboot=echo 'entering ums...'; ums 0 mmc 0; echo 'about to reboot..'; reset;
bootcmd=run ums_and_reboot;

With the u-boot script it’s possbile to load uboot and automatically run the ums command.

$ ./sunxi-fel uboot u-boot-sunxi-with-spl.bin write-with-progress 0x43100000 uenv

The above command will cause a mass storage USB device to appear.

$ lsusb -d 1f3a:
Bus 005 Device 079: ID 1f3a:1010 Allwinner Technology Android device in fastboot mode

This USB device can be written to or read from with dd. Zeroing out the card entirely can be done with the following command.

$ dd if=/dev/zero of=/dev/disk/by-id/usb-Linux_UMS_disk_0_92c001baea519a49-0:0 status=progress

The ums command will keep running until a Ctrl-C is dispatched to u-boot over the serial console. Then u-boot will trigger a reboot. If a new image was written to the SD card, then rebooting will boot into the new image.

Using Linux

The second option is to use u-boot to load a kernel image and an initrd which prepares the g_mass_storage kernel module and triggers a reboot once the USB device has been ejected. The g_mass_storage module needs the following kernel configuration options enabled.

CONFIG_USB_GADGET=y¬
CONFIG_USB_MASS_STORAGE=y
CONFIG_USB_ROLE_SWITCH=y

For performance reasons, we should be using the performance governor so the USB stack will be running at the maximum speed. This will make read and write operations much faster.

CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y

A small busybox initrd is sufficient to prepare the g_mass_storage module.

Creating a busybox based initrd

Assuming that busybox has been compiled in to a single static executable, creating an initrd is pretty straight forward. Simply create all of the basic directories of the root fileystem and place the busybox executable at /bin/busybox.

$ mkdir -p root/bin/
$ mkdir -p root/dev/
$ mkdir -p root/etc/
$ mkdir -p root/lib/
$ mkdir -p root/mnt/
$ mkdir -p root/proc/
$ mkdir -p root/sbin/
$ mkdir -p root/sys/
$ mkdir -p root/tmp/
$ mkdir -p root/usr/bin/
$ mkdir -p root/usr/sbin/
$ mkdir -p root/var
$ cp busybox /root/bin/busybox

Even though busybox comes with init functionality, I think it’s easier just to create a shell script at /root/init that sets up busybox, passes the SD card block device to the g_mass_storage module and polls for ejection.

#!/bin/busybox sh

/bin/busybox --install -s # Install's busybox symlinks

# Mount all required filesystems

mount -t devtmpfs  devtmpfs  /dev
mount -t proc      proc      /proc
mount -t sysfs     sysfs     /sys
mount -t tmpfs     tmpfs     /tmp
mount -t tmpfs     mdev      /dev

mount -t configfs none  /sys/kernel/config
mount -t debugfs none /sys/kernel/debug

echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

# Switch the Pine A64 into peripheral mode
echo 'peripheral' > /sys/devices/platform/soc/1c19000.usb/musb-hdrc.2.auto/mode

# Maximize the clock of the MMC
echo 150000000 > /sys/kernel/debug/mmc0/clock

# Open the SD card as a file over USB
echo /dev/mmcblk0 > /sys/class/udc/musb-hdrc.2.auto/device/gadget/lun0/file

# Poll for ejection
echo 'Polling for ejection...'
while [ ! -z $(cat  /sys/class/udc/musb-hdrc.2.auto/device/gadget/lun0/file) ]
do
  sleep 1
done

echo 'Ejected, rebooting...'

reboot -f

Finally the entire root directory can be placed into a CPIO archive, compressed and have the u-boot header attached.

$ cd root
$ find . | cpio -ov --format=newc | lzma -9 >../initramfz
$ cd ..
$ mkimage -A arm64 -O linux -T ramdisk -C lzma -d initramfz initramfs.uImage

Enabling the gadget

With u-boot, a kernel with the g_mass_storage module and the above initrd it’s possible to use FEL mode to load all of them and boot into a Linux kernel which exposes the SD card over USB. The last thing that is needed is a u-boot script which sets the kernel arguments and immediately boots into the kernel.

#=uEnv
set_boot_args=setenv bootargs quiet console=ttyS0,115200 g_mass_storage.luns=1 g_mass_storage.removable=1
bootcmd=run set_boot_args; booti $kernel_addr_r $ramdisk_addr_r $fdtcontroladdr;

All of these files can be loaded and executed

$ sunxi-fel uboot u-boot-sunxi-with-spl.bin write 0x40080000 Image.lzma write 0x4FF00000 initramfs.uImage writes 0x43100000 uenv

The above command will cause a mass storage USB device to appear.

$ lsusb -d 0525:
Bus 005 Device 089: ID 0525:a4a5 Netchip Technology, Inc. Pocketbook Pro 903 / Mobius 2 Action Cam / xDuoo X3 / PocketBook Pro 602

By default the USB vendor id is from Netchip. It can be changed by specifying the idVendor module parameter. Like the mass storage device created by u-boot this can be written to or read from with dd. Since our initrd polls for ejection, ejecting the drive from the host will trigger a reboot on the device. If a new image was written, the reboot will trigger a boot of the new image.

$ dd if=/dev/zero of=/dev/disk/by-id/usb-Linux_File-Stor_Gadget-0:0 status=progress
$ eject -s /dev/disk/by-id/usb-Linux_File-Stor_Gadget-0:0

Performance Metrics

Regardless of how the gadget is implemented in u-boot or in Linux I have found that the optimal block size is 4MB for most SD cards. I have also found adding the oflag=direct flag when writing or the iflag=direct when reading ensures the throughput is higher.

Since u-boot does not increase the clock speeds to maximum and since u-boot does not initialize multiple cores, the implementation is slower.

Conclusion

Due to Allwinner CPUs design, it is possible to load in u-boot via USB directly to RAM. Once loaded, u-boot can expose internal storage over USB and the internal storage can be changed as needed. Alternatively, a full kernel and initrd can be loaded over USB and booted into, which can also expose the internal storage over USB. The Linux approach allows for better performance which may matter depending on the size of the internal storage.