Using OpenCV with the Raspberry Pi camera

If you’re a Raspberry Pi enthusiast like me, you probably love the Raspberry Pi camera. It delivers pictures in such high quality and speed that it’s practically begging to be used in computer vision contexts.

Unfortunately, that’s not as easy as it sounds. Although the userland library offers some nice utilities for working with the Pi’s camera from the command line, and OpenCV, as a great computer vision library, can easily be installed on the Pi using apt-get, that alone doesn’t solve the problem. The userland programs save encoded pictures to disk, so that if you wanted to work with the pictures using them directly you’d have to take the picture, encode it as a JPG, save it to disk, and then read it from disk with OpenCV and decode it back into a picture. Quite unnecessary.

But there’s good news! And, once again, it shows the power of open source in action. There is now a library that allows you to take pictures with the Pi Cam and get them back as an OpenCV matrix. Let’s take a look at how.

As I was looking for ways of using the Pi Cam with OpenCV, I was daunted at first. It’s great to be able to read the sources for the camera, but that means working a lot with semaphores and doing very, very low-level things. I was grimly prepared to try to use the userland libraries as includes in my own code and make a new library that would return pictures as OpenCV objects when I discovered Pierre Raufast’s blog. He wrote up a way of exposing the camera’s pictures as IplImages, which are OpenCV’s structs in C.

The code was great, but it was buried in executable programs. I was planning on extracting its logic into a library that I could use in my own programs, when I found another post by him saying that it’d already been done by Emil Valkov, based on Pierre’s previous work. Well,  what do you know. I went straight to his blog and checked it out.

As it turns out, he had his code on Github so I forked it and got to work on it. It’s super easy to compile and I was able to very quickly put together a small program that either shows you pictures from the camera or video using the OpenCV highgui modules. But one thing bothered me a bit: The resolution was set really low, to 640×480. As it turns out, that was hard coded into the code. So I modified Emil’s sources, leaving the same interface in place and exposing a second function that allows the user to set the camera capture’s dimensions. The default function just delegates work with default resolution to that function. And now I’ve sent Emil a pull request about it.

What am I trying to say here? Open source is awesome. Three enthusiasts from very different parts of the globe were able to collaborate on a pretty complex technical topic, and it happened completely ad-hoc. Plus, other people can profit from our work (which is mainly Pierre’s and Emil’s work, I just changed a teeny tiny bit of it).

So if you’re wanting to use the Raspberry Pi’s awesome camera with OpenCV and would rather write C++ than C, you can use Emil’s library like this to capture and show a single picture:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include "RaspiCamCV.h"

int main(int argc, char** argv) {
RaspiCamCvCapture * camera = raspiCamCvCreateCameraCapture(0);
cv::Mat image(raspiCamCvQueryFrame(camera));
cv::namedWindow("Display", CV_WINDOW_AUTOSIZE);
cv::imshow(window_name, image);
cv::waitKey(0);
return 0;
}

 

It’s as simple as that!

Advertisements
About

My name’s Daniel Lee. I’m an enthusiast for open source and sharing. I grew up in the United States and did my doctorate in Germany. I've founded a company for planning solar power. I've worked on analog space suit interfaces, drones and a bunch of other things in my free time. I'm also involved in standards work for meteorological data. I worked for a while German Weather Service on improving forecasts for weather and renewable power production. I later led the team for data ingest there before I started my current job, engineering software and data formats at EUMETSAT.

Tagged with: , ,
Posted in Uncategorized
32 comments on “Using OpenCV with the Raspberry Pi camera
  1. Poola says:

    Hi Erget,
    Thanks for the post. I was trying to to use PiCam with OpenCV and this blog and Emi’s Library came as a boon. I followed the steps in Emils site . Iam having problem linking the libraries. Both gcc and Cmake .txt are giving me errors. The libraries from Emils are in home/pi/…./build/lib. but the linker is giving me the error that usr/bin/ld -l cant find the libraries. Iam failing to link the libraries. Could you please guide me where am I making mistake and help me to resolve this error.

    Thanks.
    And Happy New Year
    Poola

    Like

    • erget says:

      Hi Poola,

      I’m assuming you’re using cmake to get the compilation ready and then make to actually compile. If that’s the case, everything should work right as long as you have the right libraries configured. In my version of the library I use make to compile it. You can see a copy here which shows what part I had to adapt to my profile in order for it to work – it’s just the path to the Userland root directory.

      Otherwise ld often makes problems because the path of your shared libraries isn’t in its search path. It’s governed by $LD_LIBRARY_PATH, so check that and see what it contains. If you need, add the path to the shared objects to that. But if you’re working with the robidouille project I don’t see a reason why that wouldn’t work, unless you have some problems that are much further down the chain. Hope that helps!

      Cheers,
      Daniel

      Like

  2. Poola says:

    Hi Erget
    Thank u for the replay
    When I use the Cmake for compilation I get the same error.
    And with gcc also Iam getting the same error. Just not able to figure out what is going wrong!!
    main.cpp:(.text+0*68):undefined reference to ‘raspiCamCvCreateCameraCapture’
    main.cpp.text+0768):undefined reference to ‘raspiCamCvQueryFrame
    collect2: ld returned 1 exit status

    -gcc `pkg-config –cflags opencv` `pkg-config –libs opencv` -I/home/pi/git/robidouille/raspicam_cv -L/home/pi/git/robidouille/raspicam_cv -llibraspicamcv L/home/pi/git/raspberrypi/userland/build/lib -lmmal_core -l mmal_util -lvcos -lbcm_host -o main main.cpp

    the error is
    /usr/bin/ld: cannot find -lraspicamcv
    collect2: ld returned 1 exit status

    i changed
    -L/home/pi/git/robidouille/raspicam_cv -llibraspicamcv to -L/home/pi/git/robidouille/raspicam_cv/llibraspicamcv.a
    this gives me
    main.cpp:(.text+0*68):undefined reference to ‘raspiCamCvCreateCameraCapture’
    main.cpp.text+0768):undefined reference to ‘raspiCamCvQueryFrame
    collect2: ld returned 1 exit status

    This is the present status.
    I will try again and check LD_LIBRARY_PATH as you suggested.
    And let you know my progress.

    Thank u 🙂
    Poola

    Like

    • erget says:

      Hi Poola,

      I’m having a hard time figuring out exactly what the problem is because I don’t know exactly what you’re compiling and in what context. Only seeing the output without the command is a bit tough.

      1. Did the source for raspicam_cv compile alright? That would be the first step.
      2. If you’re looking for an example of a file that uses the library, see https://github.com/erget/opencv-experiments/blob/master/src/display-camera.cpp. Take a look at that project’s root folder for a simple, working CMakeLists.txt. This should compile without any problems as long as you make sure that any machine-specific paths are correct.

      Like

  3. Poola says:

    Hi Daniel,
    With any program I am writing with OpenCV and RaspiCam I am facing this problem.
    1.raspicam_cv compiled perfectly.
    2.build worked perfectly well.
    3.I tried the program You have mentioned(above in the blog) excatly, and try cmake or gcc it give me error while linking.
    4. Even the program by Emil also gives me the error while linking.
    5. raspicam_cv is in home/pi/git/robidouille/raspicam_cv
    so when i link the same using cmake or
    -L/home/…/raspicam_cv -llibraspicamcv
    the error i get is
    /usr/bin/ld: cannot find -lraspicamcv
    collect2: ld returned 1 exit status

    6. So i tried the command
    -L/home/…/raspicam_cv /libraspicamcv
    then the error is as follows
    main.cpp(text+068):undefined reference to ‘raspiCamCvCreateCameraCapture’
    main.cpp.(text+0768):undefined reference to raspiCamCvQueryFrame
    collect2: ld returned 1 exit status.

    I am able to compile other C++ programs. So i am not able to figure out the problem, is it the version or the command I am using . i dont know.
    7. I tried using the src file you have mentioned and the cmaketxt file as it is. still iam getting the same error. 😦

    Thanks mate for your patience and time
    Regards
    Poola

    Like

  4. erget says:

    Hey Poola,

    Okay, so you cloned the opencv-experiments repository, that’s good. I go about it like this:

    git clone https://github.com/erget/opencv-experiments.git
    cd opencv-experiments/
    # Check that CMakeLists.txt has the right paths for the variables
    # RASPICAM_CV and USERLAND, since they’re set for my system
    # using other paths
    mkdir build
    cmake .
    make

    That should do it.

    The only other things I can think of is that somehow userland didn’t build correctly. You ran the “buildme“ script in userland’s root directory, right?

    I’m not the compilation expert, I normally just use cmake and make to build my stuff and try not to worry about setting the linker stuff myself with -L and all that, but this works fine for me.

    I don’t have a “main.cpp“ in my “src“ directory, so the error that I see from you above definitely isn’t from my repository. The compiler definitely isn’t finding the shared object to link to though. You listed the link *after* you listed your source file, right? Take a look here for some syntax on gcc, maybe this is relevant for you:
    http://stackoverflow.com/questions/1517138/trying-to-include-a-library-but-keep-getting-undefined-reference-to-messages

    Like

  5. Poola says:

    Hi there again
    main.cpp was the name given to the cature an image.the same program in this blog.
    I shall try once more again follwing the steps. hopefully it works.
    Any other C or C++ program is working just fine. With raspicam I am making mistake either in including libraries or liking the libraries or in the method using them. Let me try!!

    Thanks
    Was in Germany last month. Luved Berlin,Hamburg,Bremen and traveled mostly in north Germany.
    Regards
    Poola

    Like

  6. Poola says:

    Hi Daniel,
    The problem is solved and thanks for helping me solve it.
    U directed me to your program, and told me to prefer using cmake for compiling, well that helped.
    cmake is a better option when trying to link libraries from different sources to the module. gcc can be used but only in a particular method, libraries need to be linked individually and in a proper order. And also module and libraries need to be placed in correct order,which sometimes might be time consuming and prone to errors.
    well the problem I was facing was
    main.cpp (text+068):undefined reference to raspiCamCvCreateCameraCapture’
    main.cpp.(text+0768):undefined reference to raspiCamCvQueryFrame
    collect2: ld returned 1 exit status.

    My module was not linked to raspicam_cv.

    And these lines from your Cmake txt helped me
    1) SET(DISPLAY_CAMERA_LIBS
    ${RASPICAM_CV}/libraspicamcv.a
    ${MMAL_LIBS} ${BCM_LIBS} ${OpenCV_LIBS})

    2) SET(DISPLAY_CAMERA build/display-camera)
    3)add_executable(${DISPLAY_CAMERA} src/display-camera.cpp)
    4) target_link_libraries(${DISPLAY_CAMERA} ${DISPLAY_CAMERA_LIBS})

    As seen in point 4 cmake links the executable build/display-camera with the RASPICAM_CV and Userland libraries, as required.
    So that solves the problem:-)

    Thank you
    Poola

    Like

  7. you’re right Daniel, open source is awesome 😉
    let’s continue !

    Like

    • erget says:

      Yeah! Although I’ve still got to do some work on my pull request – haven’t gotten to it yet though because of the holidays 😉 It’ll get done though.

      Like

  8. Javier says:

    Hi Daniel
    Really awesome work ( all together with Pierre and Emil of course). I am developing an aplication using raspicam and opencv and currently using the library and works perfectly!. Thanks for that.
    Just one question, my C/C++ knowlodge is a bit rusty (quite a long time since i do not use it) i would like to be able to access other parameters of the camera ( such as exposure mode or shutter speed). Could you please give some indications on where to focus to be able to achieve that goal?
    Are these parameters currently available on the still_port = camera->output[MMAL_CAMERA_CAPTURE_PORT] structure?
    Thanks in advanced and congrats again!
    Javi

    Like

    • erget says:

      Hey Javi,
      I did a little digging and I see that the exposure mode is set as part of the struct RASPICAM_CAMERA_PARAMETERS, which is included from RaspiCamControl.h. An instance of this struct is stored as camera_parameters in RASPIVID_STATE, which is in RaspiCamCV.c.

      If you’ll notice, one of these RASPIVID_STATEs is used to control the camera parameters. Since a struct’s parameters are public, you could do something like this (as an example):

      RaspiCamCvCapture * capture = raspiCamCvCreateCameraCapture(0);
      capture->pState->camera_parameters->exposureCompensation = -10;

      in order to set the exposure compensation to -10.

      See the typedef for RASPICAM_CAMERA_PARAMETERS in RaspiCamControl.h for more info on how this works.

      If you’re interested in making the library capable of setting that with a OpenCV-like API, as Emil has told me he likes, you could fork my repository and do the following:

      1. Add a property ID to the enum in RaspiCamCV.h (i.e. for setting the exposure mode, etc.)
      2. Extend raspiCamCvSetCaptureProperty in RaspiCamCV.c to accept that property and set it accordingly in the capture. I’ve got an example for height and width in the function.

      Sadly, my Pi’s out of commission right now, so I can’t test it, otherwise I’d see if I could just extend it quickly myself 🙂 Good luck!

      Like

  9. eelay says:

    Hi Daniel,
    thanks for your work on this. I would gladly use our optimization with user set resolution. But I run into a problem while compiling.
    I edited the makefile to fit my path of userland and opencv and the make command works when I try to build the original file. But when I’m trying your fork I’ll get this error:
    RaspiCamCV.c:419:6: error: conflicting types for ‘raspiCamCvSetCaptureProperty’
    RaspiCamCV.h:34:5: note: previous declaration of ‘raspiCamCvSetCaptureProperty’ was here
    make: *** [objs/RaspiCamCV.o] Error 1
    Any suggestions?

    Like

    • erget says:

      It sounds like you might be redefining raspiCamCvSetCaptureProperty. If you look in my sources, you’ll see a declararion (that you may or may not want to extend for your own purposes because it doesn’t cover all camera properties) of this function in RaspiCamCv.h. It’s defined in the corresponding C file. If you are redefining it in order to set your variables, this will cause a conflict which you can resolve by instead extending the current function. That would seem to me to be the likeliest source of your error.

      Like

  10. eelay says:

    I just wanted to let you know that there are officially new drivers for the RPi Cam
    http://www.raspberrypi.org/phpBB3/viewtopic.php?f=43&t=62364
    With this the standard opencv functions work!! No more mixture of c and c++. We all can now use the new c++ functions without changing anything.
    Good news

    Liked by 1 person

    • erget says:

      Wow, that is great news 🙂 So this means I can address the Raspicam just like any other camera in OpenCV now?

      Like

      • eelay says:

        Yes, I just ran sudo rpi-update to update the firmware. After that, VideoCapture recognises the camera
        #include
        #include
        using namespace cv;
        int main(int, char**)
        {
        VideoCapture cap(0); // open the default camera
        for(;;)
        {
        Mat frame;
        cap >> frame; // get a new frame from camera
        imshow(“cam”, frame);
        if(waitKey(30) >= 0) break;
        }
        return 0;
        }

        Like

      • erget says:

        That’s really great news. Thanks for sharing it!

        Like

  11. SANDIPAN says:

    which firmware?

    Like

  12. […] moyen simple pour installer tout ça. Et j’y suis arrivé (enfin j’espère) ici, ici et ici […]

    Like

  13. franken says:

    i have firstly build raspberry pi userland libraries after that i have download the raspicam library but when i do “make” in raspicam_cv folder i get the following error :

    make: * No rule to make target ‘objs/RaspiCamControl.o’, needed by ‘libraspicamcv.a’. Stop.

    plz how can i solve this thhanks

    Like

  14. Hi !

    Im trying to use geany in my raspberry pi, im not good with linking libraires and including paths.
    Can you help me ?

    Like

    • erget says:

      Sorry, I don’t know anything about that library.

      Like

      • im geting stuck in this step.

        Add the include path

        -I$(HOME)/git/robidouille/raspicam_cv

        Link with the raspicamcv library

        -L$(HOME)/git/robidouille/raspicam_cv -lraspicamcv

        Link with the userland libraries:

        -L$(HOME)/git/raspberrypi/userland/build/lib -lmmal_core -lmmal -l mmal_util -lvcos -lbcm_host

        I dont know where to put this.

        Like

  15. How I can calibrate Rassberry pi cam using OpenCV

    Like

  16. Nadav B says:

    Your fork link leads to nowhere 😦

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

From the archive