ユニファ開発者ブログ

ユニファ株式会社システム開発部メンバーによるブログです。

The Pi with Eyes

By Matthew Millar Research Scientist at ユニファ

Purpose:

This blog will cover the progress of an IoT project for computer vision. The final goal will be to locate an individual and to track this individual to see where they are in a room and to tell if they go near a marked area in the room. This will involve; person detection, tracking, and distance measurements from a Raspberry Pi.
Last time we set up the Pi and got things ready to go.
My 2nd Favorite Pi - ユニファ開発者ブログ

This blog post will look at setting up the Pi so it can stream the video feed from a camera. We will cover how to install OpenCV on the Pi and set up a synchronized camera feed from a special Camera Hat which will be discussed later!

IoT:

The IoT is mainly about connecting everything around us to make a holistic data acquisition system. This means all the data that you create is gathered by sensors and sent to the mother node! Let’s break down IoT.
In a nutshell, IoT is a collection of Node or edge devices that gather data by sampling the world around them. This data can range to a large amount because of the different data that can be collected. It can be temperature and weather data, camera feeds, IR sensors, tablets, refrigerators and appliances, and wearable devices to name a few. This is not a comprehensive list as this list is endless and getting longer every day.

Cameras are one of the most common methods for gathering data in an environment. They can be considered the first edge device that was implemented. The world's first security camera came from a chicken farmer who wanted to see who stole his eggs back in 1933. So, the farmer rigged up a camera (like that 35mm film camera professional (use) and hipster have) and set a trap so when the door to the chicken hut was opened the camera would take a picture. He even went so far to put a tin can to make a noise and have the thief look up at the camera [1]. This was the first recorded instance of a security camera. (Yes, the thief was captured and was found guilty).

OpenCV on Pi:

We will need at least one library for using cameras on the Raspberry Pi. That one will be the most common one to use in computer vision projects, OpenCV. This can be complex to install as the older method

pip install opencv-contrib-python

It does not work well with Opencv4 on some Pis. (I tried and failed). So I had to install it from the source. We will go through the steps one by one to get started.
First things first let us get things ready for OpenCV
Update and upgrade the OS first (just like Linux)

sudo apt-get update && sudo apt-get upgrade

Next, we need cmake installed to build the files later.

sudo apt-get install build-essential cmake pkg-config

And then the I/O packages for images

sudo apt-get install libjpeg-dev libtiff5-dev libjasper-dev libpng-dev

And I/O packages for videos

sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev

We need the GTX development libraries for OpenCV so let’s get them too

sudo apt-get install libfontconfig1-dev libcairo2-dev
sudo apt-get install libgdk-pixbuf2.0-dev libpango1.0-dev
sudo apt-get install libgtk2.0-dev libgtk-3-dev

And a library for aiding in Matrix Manipulation

sudo apt-get install libatlas-base-dev gfortran

And finally the HDF5 and QT GUI

sudo apt-get install libhdf5-dev libhdf5-serial-dev libhdf5-103
sudo apt-get install libqtgui4 libqtwebkit4 libqt4-test python3-pyqt5

Now with all that installed and ready to go let's make our environment to use OpenCV in

Step 1 get and install pip (if you don’t have it already)

wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo python3 get-pip.py
sudo rm -rf ~/.cache/pip

Now the virtual environment manager

sudo pip install virtualenv virtualenvwrapper

We will update the bashrc file now with the needed information

vim ~/.bashrc

and then add this to the bottom of the file (adjust as needed but if you followed the above exactly then there should be no adjustments)

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

Exit out of the bashrc file and run source to apply all the changes to the bashrc file

source ~/.bashrc

Finally, we can create a new virtual environment

mkvirtualenv myenv_name -p python3

Activate the environment using

Source /path/to/env/myenv_name/bion/activate

And install some needed dependencies.

pip install "picamera[array]"
pip install numpy

Now we can finally get to installing OpenCV from the source

cd ~
wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.1.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.1.1.zip
unzip opencv.zip
unzip opencv_contrib.zip
mv opencv-4.1.1 opencv
mv opencv_contrib-4.1.1 opencv_contrib

This will get all the code you will need to build and install OpenCV
The next step is to make the swap file larger as if you don’t you will not be able to install OpenCV as it takes up too much room

sudo vim /etc/dphys-swapfile

and change the swap size from 100 to 2048. But later on, you will change it back to 100 as using 2048 can burn out your SD card quickly.

# set size to absolute value, leaving empty (default) then uses computed value
#   you most likely don't want this, unless you have an special disk situation
# CONF_SWAPSIZE=100
CONF_SWAPSIZE=2048

After the install, you will change it back to 100 Stop and start the swap service

sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start

Now go back to your virtual environment to work on it. And then we will start to build and configure OpenCV

cd ~/opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
    -D ENABLE_NEON=ON \
    -D ENABLE_VFPV3=ON \
    -D BUILD_TESTS=OFF \
    -D INSTALL_PYTHON_EXAMPLES=OFF \
    -D OPENCV_ENABLE_NONFREE=ON \
    -D CMAKE_SHARED_LINKER_FLAGS=-latomic \
    -D BUILD_EXAMPLES=OFF ..

What makes this special is that ENABLE_NEON is on so that OpenCV is optimized for the ARM processor.
Now you can run

cmake 

But make sure you are inside the build file use pwd to make sure it is in the directory opencv/build/
Next step after the build is done using the

make -j4

Command to compile the code. The -j4 will make it faster as it will use all 4 cores of the Pi. You can leave off the -j4 to avoid possible race conditions if it freezes during install. This will take a hot minute so get lunch and possibly dinner and come back.
Now you can run

sudo make install
sudo ldconfig

The next step is to change the swap size back in the above steps.

Now we need to make a sym-link from the Opencv to python.

cd /usr/local/lib/python3.7/site-packages/cv2/python-3.7
sudo mv cv2.cpython-37m-arm-linux-gnueabihf.so cv2.so
cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
ln -s /usr/local/lib/python3.7/site-packages/cv2/python-3.7/cv2.so cv2.so

Now if all that worked then you can check to see if it actually worked or not (took me 3 tries so don’t get disheartened).
Activate your virtual environment and run the following commands

Python
>>>Import cv2
>>>Cv2.__version__

The Camera Man:

Now we can turn our attention to building the camera system for the project. Looking at the final goal of the project of distance estimation we will need a synchronized set of cameras. Using two cameras that are calibrated and synchronized can greatly improve the accuracy of distance/depth predictions [2]. The camera kit that I will be using will be the Arducam 5MP Synchronized Stereo Camera Bundle Kit for Raspberry Pi that can be bought from Uctronics [3]. Using this chip will handle all the synchronization issues that come with dual camera feeds.

The Setup:

The setup is pretty straight forward. The first step is to connect the ribbon cable from the ArduCam chip to the camera input on the Pi.
f:id:unifa_tech:20191129113459j:plain
f:id:unifa_tech:20191129113515j:plain
Then place the pins into the right connectors on the HAT this will be the topmost pins on the Pi.
f:id:unifa_tech:20191129113532j:plain
And that's it you are set up and ready to begin installing more packages and dependencies! Let's plug all out stuff in the pi and your setup should look something like this
f:id:unifa_tech:20191129113643j:plain
Now even MORE setup and setting changes!
The first step is to enable the camera. On the Pi Desktop go to

Preferences->Raspberry Pi Configuration->Interfaces.

Then Enable the Camera, SSH, VNC, and I2C by clicking the enable radio button. The next step is to download the SKD for the ArduCam chip from GitHub

git clone https://github.com/ArduCAM/MIPI_Camera.git

Go into the RPI folder in the repo and run this command

chmod +x ./enable_i2c_vc.sh
./enable_i2c_vc.sh

This will enable the i2c_vc.
Guess what we need even more Packages again! So run these commands to install them both

sudo apt-get update && sudo apt-get install libzbar-dev libopencv-dev

This will set you up well so you are finally finished installing packages.
The next step is to make the install the code from the repos so do this

cd MIPI_Camera/RPI
make install

Next, compile the examples for testing

make clean && make

And finally, run it in preview mode (a C program from the chips creators).

./preview_setMode 0

To actually test out your code. If everything worked out ok you should start to see an image being streamed from the device to your Desktop like this
f:id:unifa_tech:20191129113834j:plain
or this
f:id:unifa_tech:20191129113857j:plain

And there you have it you now have an RPi 3B+ with a stereo camera that is synchronized and ready for the next steps.
Next time I will install TensorFlow and start with a simple object detection AI.

Sources:

[1] https://innovativesecurity.com/the-worlds-first-security-camera/

[2] Peleg, S., & Ben-Ezra, M. (n.d.). Stereo panorama with a single camera. Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149). doi: 10.1109/cvpr.1999.786969

[3] https://www.uctronics.com/index.php/arducam-synchronized-stereo-camera-bundle-kit-5mp-for-raspberry-pi-2173.html