Setting the Cameras to use External Trigger

Eye Tracking on the Raspberry Pi 5

This is a record of my efforts to implement eye tracking using two cameras mounted inside a Mandalorian helmet that I 3D printed. I based my choice of cameras to use on the specs for the Project Aria glasses. According to the paper that Meta Reality Labs Research published about the first-generation Project Aria glasses, they are using "two monochrome, global-shutter inward-facing cameras for eye-tracking with a diagonal field of view of 80 degrees. They typically operate at a 320x240 pixels resolution." With the intention of using whatever I selected with a Raspberry Pi, I found this listing on Amazon: innomaker GS Camera OV7251 Sensor with Global Shutter External Trigger Stream Mode 0.3MP Frame Rate up tp 158fps 8bit 10bit rawdata Format for Raspberry Pi 5 4B 3B+ 3B 3A+ CM3+ CM3 Pi Zero W At 14 bucks a piece, I ordered two and am currently waiting for them to be delivered. In the meantime, I'm making sure my Raspberry Pi is properly configured so I'll just be able to plug in the cameras when they arrive.

The OV7251 has a global shutter, just like the Project Aria cameras. There are monochrome, outputting 8- or 10-bit RAW BW, just like the Project Aria cameras. Well, I don't know what bit-depth the Project Aria cameras actually are. However, the OV7251 is actually higher resolution, at 640x480 compared to Aria's 320x240. I wasn't trying to get higher resolution cameras; that's just what I was able to find. The final reason I selected these cameras was the "external trigger" feature. The Project Aria paper spends a bit of time explaining that they are careful to synchronize all the onboard sensors, and for eye tracking it is important to ensure each frame from the camera is synchronized with the other camera. There are single board systems with two camera modules that do this, as well as providing a single output feed, but I wanted the flexibilty to place the cameras wherever necessary inside the helmet. There are also boards made by Arducam that will synchronize the cameras connected to them, but those add even more cost, require even more space inside the helmet, and the smallest boards in stock were for up to four cameras. All the cameras connected to a board like that must be identical, so even if I wanted external cameras, I wouldn't want them to also be low-resolution global shutter black and white cameras. Expansion boards just weren't the right choice for this project. Thankfully, the OV7251 cameras can be triggered externally, or by software. So my plan is to connect both trigger signals to a single 3.3V pin on the Pi.

What's new with the Pi 5?

I am specifically using a Pi 5 I because I already had a 5 sitting around. I could have used a 4 instead, but the Pi 5 has something that was previously only available on the Raspberry Computer Modules: two camera connector ports!

In the picture above, it is labeled as "2 x 4-lane MIPI DSI/CSI connectors". However, these aren't the same 15-pin connectors as previous full-sized Raspberry Pi boards. These are the same connectors that the Pi Zero boards use, which means the 15-pin ribbon cable on the OV7251 cameras won't fit directly into the Pi 5's ports. So I ordered the Adafruit CSI or DSI Cable Adapter Thingy for Raspberry Pi, which will convert the 15-pin cable from the camera into a 22-pin CSI signal that will actually interface with the Pi. While I was ordering from Adafruit, I ordered a couple of 20cm long CSI cables to reach from the adapters, one for each camera, to the ports on the Pi 5. Remember, I'm going to try to fit all of this inside a 3D-printed Mandalorian helmet...

Configuring the Pi 5

If I were using official camera modules for the Raspberry Pi, all I would have to do is plug the cameras into the CSI connectors and the Pi would automatically detect them and have the proper drivers to use them immediately. Thankfully for an Amazon listing, the product description included a link to the GitHub repository with driver code and the user manual. Looking through the user manual, I found instructions for downloading the GitHub code, updating drivers and libraries for the Pi, and configuring the boot config file to recognize the cameras. What follows is a list of the commands that I ran.

The Code

$ sudo git clone https://github.com/INNO-MAKER/cam-mipiov7251-trigger.git

This is supposed to provide the code that will trigger the OV7251 cameras.

$ sudo apt-get install raspberrypi-kernel-headers

Since the Pi 5 was already running the latest version of Raspberry Pi OS (version "bookworm" at the time of writing), all I needed to do was install the Linux kernel headers, which are used in one of the library files for the camera module.

$cd cam-mipiov7251-trigger

$sudo chmod -R a+rwx *

$cd ov7251_driver_source_code/source_code

$sudo make && sudo make install

These next commands open the directory that was just downloaded from GitHub, make all the files inside executable, then runs the make command to build the files. The user manual notes that the 8-bit stream is the default mode, so at some point I'll have to change to an 8- or 10-bit stream with the external trigger.

$sudo cp ov7251_driver_source_code/inno_mipi_ov7251_cm4_dual.dtbo /boot/overlays

Next, I copied the dtbo file to the /boot/overlays directory. This is based on the instructions for the Compute Modules and ignoring the instructions for the older RPi models that only support one camera connection.

$sudo nano /boot/firmware/config.txt

All the instructions I saw said to open /boot/config.txt, but that information is deprecated. The file still exists, but if you actually open it, the file just contains this warning:

DO NOT EDIT THIS FILE

The file you are looking for has moved to /boot/firmware/config.txt

So instead, I went to /boot/firmware/config.txt like I was told and found the line camera_auto_detect=1 and commented it out like so:

#camera_auto_detect=1

This is based on information in the article How to use Two Camera Modules with Raspberry Pi 5, specifically the section "How to use two unofficial camera modules with Raspberry Pi 5". When I had finished running the `make install` earlier, there was this output:

--------------------------------------

ADD 'dtparam=i2c_vc=on' and 'dtoverlay=inno_mipi_ov7251' to your /boot/config.txt

ADD 'disable_touchscreen=1' to your /boot/config.txt if a touchscreen is attached

ADD 'cma=128M' to your /boot/cmdline.txt

--------------------------------------

I don't have a touchscreen installed, so I'm ignoring that. I mostly followed the rest of those instructions, and instead put everything in `/boot/firmware/config.txt`:

dtparam=i2c_vc=on

dtoverlay=inno_mipi_ov7251,cam0

dtoverlay=inno_mipi_ov7251,cam1

cma=128M

Setting the Cameras to use External Trigger

According to the user manual for the cameras I'm using, there are four "working modes": 10- and 8-bit stream modes that operate at 158 fps, then 10- and 8-bit external trigger modes. The modes are numbered in that order, from 0 to 3. So for my purposes, I'm going to use the 10-bit external trigger mode, so there is some extra color depth when trying to pick out the user's pupils from the image. The user manual says to run these commands to set the work mode:

$cd cam-mipiov7251-trigger/inno_mipi_ov7251_driver_pi_lattice_linux5.15

$sudo make setmode2

This will set the work mode to option 3 from the list. If I wanted to use the 8-bit external trigger mode, I would run sudo make setmode3. However, the file path specified in the user manual doesn't actually exist, so instead, I ran the sudo make setmode2 command in ~/cam-mipiov7251-trigger/ov7251_driver_source_code/source_code, which actually ran. Now I just have to wait until I get my cameras, adapters, and extension ribbons in the mail to try this out! Once I have the cameras connected, they should show up when I run the command libcamera-hello --list-cameras. A 3.3V GPIO rising edge signal should trigger the output frame, if properly configured.

Update

It didn't work. I connected the cameras using the ribbon cables, used a breadboard to connect both External Trigger pins on the cameras to the same 3.3V GPIO pin on the Pi, connect both GND pins to a GND GPIO, then ran libcamera-hello --list-cameras, and no cameras were found. I've been messing around with trying different drivers and even wrote a script which is supposed to activate the GPIO signal, then check for a response from the cameras, but it hasn't worked.