OpenCV(c++) 使用记录

[TOC]
OpenCV(C++)的基础学习使用记录,主要是fressspace的形态学处理.

使用背景

自动驾驶场景中比较复杂的一类是非结构化场景,这种场景下应用fressspace进行规划是常见的思路,尤其是高质量激光雷达量产后freespace的精度/广度/深度能提供很多信息供决策规划.目前参与的项目中对非结构化道路/寻库等场景,使用的是激光雷达与摄像头融合后的freespace,在此基础上结合其他感知信息进行规划.

目前传过来的freespace为500x500,分辨率为0.1m的灰度图,由于噪声/无用的灰度值等信息需要预处理一下,因此使用了OpenCV进行了二值化以及形态学的处理.这里记录一下学习使用过程.事后整理有些粗糙,万事不可拖延症,越拖越低效…

安装部署

ROS1中默认安装了OpenCV,版本查询见下.

pkg-config --modversion opencv

如果ROS中安装的版本不满足需求,可以自行下载源码进行安装.安装参考.
同时在catkin_make中指定自行安装的OpenCV版本,例如一种方式:

set(OpenCV_DIR   /usr/local/share/OpenCV)　　＃必须指定到包含 .cmake的上一层
find_package(OpenCV REQUIRED)

The Core Functionality (core module)

下面是从官网的文档进行学习后的记录.

1. Mat介绍,image container

早期使用C structure called IplImage存储image data, 问题是user is responsible for taking care of memory allocation and deallocation. 因此OpenCV 2.0 introduced a new C++ interface-Mat,unless you are targeting embedded platforms除了嵌入式平台都适用.

mat类两个数据部分:

1. the matrix header:containing information such as the size of the matrix, the method used for storing.
2. a pointer to the matrix containing the pixel values.
   空间效率:copy operators will only copy the headers and the pointer to the large matrix, not the data itself.改一影响全部.可以读一段内存数据的一部分构建object,即create a region of interest (ROI) in an image.

Mat D (A, Rect(10, 10, 100, 100) ); // using a rectangle
Mat E = A(Range::all(), Range(1,3)); // using row and column boundaries

共用的如何删除?reference counting mechanism->last one used it.
如何copy数据? cv::Mat::clone() and cv::Mat::copyTo() functions.

2. Storing methods

store the pixel values: color space and the data type.

color space:

1. grayscale
2. colorful ways
   • RGB+alpha (A)transparency: most common as our eyes use something similar,butOpenCV standard display system composes colors using the BGR.
   • HSV and HLS decompose colors into their hue, saturation and value/luminance components
   • YCrCb is used by the popular JPEG image format.
   • CIE Lab :measure the distance of a given color to another color.

data type:

1. char 1byte
2. unsigned or signed 2byte
3. float 4byte
4. double 8byte

3. Creating a Mat object explicitly

0. <<operator of Mat: only works for two dimensional matrices, 重载可用于 cout 等.

1. cv::Mat::Mat Constructor

   Mat M(2,2, CV_8UC3, Scalar(0,0,255));

   参数:(rows,cols,data type,cv::Scalar)
   data type详解:
   CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number]
cv::Scalar:初始化默认值.initialize all matrix points with a custom value.

2. C/C++ arrays

   int sz[3] = {2,2,2};
   Mat L(3,sz, CV_8UC(1), Scalar::all(0));

   参数:(dimension,pointer containing the size for each dimension, 其余参考上面)

3. cv::Mat::create function

   M.create(4,4, CV_8UC(2));

   cannot initialize the matrix values.

4. cv::Mat::zeros, cv::Mat::ones, cv::Mat::eye

   Mat E = Mat::eye(4, 4, CV_64F); #对角阵
   Mat O = Mat::ones(2, 2, CV_32F); #单位阵
   Mat Z = Mat::zeros(3,3, CV_8UC1); #零阵

5. small matrices, 逗号初始化

   Mat C = (Mat_<double>(3,3) << 0, -1, 0, -1, 5, -1, 0, -1, 0);
   C = (Mat_<double>({0, -1, 0, -1, 5, -1, 0, -1, 0})).reshape(3); //C++11

6. cv::randu() function

   Mat R = Mat(3, 2, CV_8UC3);
   randu(R, Scalar::all(0), Scalar::all(255)); // lower and upper limit

4. Output formatting

1. Default

   cout << "R (default) = " << endl <<        R           << endl << endl;

2. Python

   cout << "R (python)  = " << endl << format(R, Formatter::FMT_PYTHON) << endl << endl;

3. CSV

   cout << "R (csv)     = " << endl << format(R, Formatter::FMT_CSV   ) << endl << endl;

4. Numpy

   cout << "R (numpy)   = " << endl << format(R, Formatter::FMT_NUMPY ) << endl << endl;

5. C

   cout << "R (c)       = " << endl << format(R, Formatter::FMT_C     ) << endl << endl;

5. Output of other common items

1. 2D Point

   Point2f P(5, 1);
    cout << "Point (2D) = " << P << endl << endl;

2. 3D Point

   Point3f P3f(2, 6, 7);
   cout << "Point (3D) = " << P3f << endl << endl;

3. std::vector via cv::Mat

   vector<float> v;
    v.push_back( (float)CV_PI);
    v.push_back(2);
    v.push_back(3.01f);
    cout << "Vector of floats via Mat = " << Mat(v) << endl << endl;

4. std::vector of points

   vector<Point2f> vPoints(20);
   for (size_t i = 0; i < vPoints.size(); ++i)
       vPoints[i] = Point2f((float)(i * 5), (float)(i % 7));
   cout << "A vector of 2D Points = " << vPoints << endl << endl;

6. 元素的访问/修改

1. at 操作

   uchar pixel_value = Mat.at<uchar>(row, col);
   Mat.at<uchar>(row, col) = pixel_value; 

   但是效率很低, 个别像素的使用可以考虑.

2. ptr 操作
   通过指针偏移的方式进行像素的查找、遍历和修改的, 因此效率相对较高.

   uchar pixel_value = Mat.ptr<uchar>(row)[col];
   Mat.ptr<uchar>(row)[col] = pixel_value; 

3. 迭代器访问

   #include "opencv2/opencv.hpp"
   using namespace cv;
   void main() {
       Mat img = imread("1.jpg");
       Mat dst = img.clone();
       imshow("src", img);
       Mat_<Vec3b>::iterator it = dst.begin<Vec3b>();//初始位置
       Mat_<Vec3b>::iterator itend = dst.end<Vec3b>();//终止位置
       for (; it != itend; it++) {
           (*it)[0] = 0;
           (*it)[1] = 255;
           (*it)[2] = 0;
       }
       imshow("dst", dst);
       waitKey(0);
       destroyAllWindows();
   }

Image Processing

Morphological Operations 形态学处理

基本操作腐蚀与膨胀, 可以实现如下功能:

• Removing noise 除噪.
• Isolation of individual elements and joining disparate elements in an image, 隔离/连通.
• Finding of intensity bumps or holes in an image 去孔.

1. Dilation 膨胀
   基本单元: kernel, usually a square or circle, has a defined anchor point, usually being the center of the kernel. 需要指定内核, 通常是锚定点为中心的指定大小的方形/圆形/椭圆等, 内核的选定合适与否决定了算法运行的快慢/处理后边界的粗糙度等.
   原理: maximal pixel value overlapped by kernel and replace the image pixel in the anchor point position with that maximal value.

2. Erosion 腐蚀
   原理: minimal pixel value overlapped by kernel and replace the image pixel under the anchor point with that minimal value.

示例代码:

#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"
#include <iostream>
using namespace cv;
using namespace std;
Mat src, erosion_dst, dilation_dst;
int erosion_elem = 0;
int erosion_size = 0;
int dilation_elem = 0;
int dilation_size = 0;
int const max_elem = 2;
int const max_kernel_size = 21;
void Erosion( int, void* );
void Dilation( int, void* );
int main( int argc, char** argv )
{
  CommandLineParser parser( argc, argv, "{@input | LinuxLogo.jpg | input image}" );
  src = imread( samples::findFile( parser.get<String>( "@input" ) ), IMREAD_COLOR ); 
  //Load an image (can be BGR or grayscale)
  if( src.empty() )
  {
    cout << "Could not open or find the image!\n" << endl;
    cout << "Usage: " << argv[0] << " <Input image>" << endl;
    return -1;
  }
  namedWindow( "Erosion Demo", WINDOW_AUTOSIZE );
  namedWindow( "Dilation Demo", WINDOW_AUTOSIZE ); 
  //Create two windows (one for dilation output, the other for erosion)
  moveWindow( "Dilation Demo", src.cols, 0 );
  createTrackbar( "Element:\n 0: Rect \n 1: Cross \n 2: Ellipse", "Erosion Demo",
          &erosion_elem, max_elem,
          Erosion );
  createTrackbar( "Kernel size:\n 2n +1", "Erosion Demo",
          &erosion_size, max_kernel_size,
          Erosion );
  createTrackbar( "Element:\n 0: Rect \n 1: Cross \n 2: Ellipse", "Dilation Demo",
          &dilation_elem, max_elem,
          Dilation );
  createTrackbar( "Kernel size:\n 2n +1", "Dilation Demo",
          &dilation_size, max_kernel_size,
          Dilation );
  //Create a set of two Trackbars for each operation
  //The first trackbar "Element" returns either erosion_elem or dilation_elem
  //The second trackbar "Kernel size" return erosion_size or dilation_size for the corresponding operation.
  Erosion( 0, 0 );
  Dilation( 0, 0 );
  waitKey(0);
  return 0;
}
void Erosion( int, void* )
{
  int erosion_type = 0;
  if( erosion_elem == 0 ){ erosion_type = MORPH_RECT; }
  else if( erosion_elem == 1 ){ erosion_type = MORPH_CROSS; }
  else if( erosion_elem == 2) { erosion_type = MORPH_ELLIPSE; }
  Mat element = getStructuringElement( erosion_type,
                       Size( 2*erosion_size + 1, 2*erosion_size+1 ),
                       Point( erosion_size, erosion_size ) );
  //cv::getStructuringElement,specify kernel:(three shapes for our kernel, size of our kernel ,anchor point)
  erode( src, erosion_dst, element );
  //cv::erode,three parameters:(source image, output image, kernel the default is a simple 3x3 matrix)
  imshow( "Erosion Demo", erosion_dst );
}
void Dilation( int, void* )
{
  int dilation_type = 0;
  if( dilation_elem == 0 ){ dilation_type = MORPH_RECT; }
  else if( dilation_elem == 1 ){ dilation_type = MORPH_CROSS; }
  else if( dilation_elem == 2) { dilation_type = MORPH_ELLIPSE; }
  Mat element = getStructuringElement( dilation_type,
                       Size( 2*dilation_size + 1, 2*dilation_size+1 ),
                       Point( dilation_size, dilation_size ) );
  dilate( src, dilation_dst, element );
  imshow( "Dilation Demo", dilation_dst );
}

3. Opening

$$
dst=open(src,element)=dilate(erode(src,element))
$$

适用于:

4. Closing

$$
dst=close(src,element)=erode(dilate(src,element))
$$

适用于:

4. Morphological Gradient

$$
dst=morphgrad(src,element)=dilate(src,element)−erode(src,element)
$$

适用于:找到外轮廓

5. Top Hat

$$
dst=tophat(src,element)=src−open(src,element)
$$

适用于:

6. Black Hat

$$
dst=blackhat(src,element)=close(src,element)−src
$$

适用于:

示例代码:

#include "opencv2/imgproc.hpp"
#include "opencv2/imgcodecs.hpp"
#include "opencv2/highgui.hpp"
#include <iostream>
using namespace cv;
Mat src, dst;
int morph_elem = 0;
int morph_size = 0;
int morph_operator = 0;
int const max_operator = 4;
int const max_elem = 2;
int const max_kernel_size = 21;
const char* window_name = "Morphology Transformations Demo";
void Morphology_Operations( int, void* );
int main( int argc, char** argv )
{
  CommandLineParser parser( argc, argv, "{@input | baboon.jpg | input image}" );
  src = imread( samples::findFile( parser.get<String>( "@input" ) ), IMREAD_COLOR );
  if (src.empty())
  {
    std::cout << "Could not open or find the image!\n" << std::endl;
    std::cout << "Usage: " << argv[0] << " <Input image>" << std::endl;
    return EXIT_FAILURE;
  }
  namedWindow( window_name, WINDOW_AUTOSIZE ); // Create window
  createTrackbar("Operator:\n 0: Opening - 1: Closing  \n 2: Gradient - 3: Top Hat \n 4: Black Hat", window_name, &morph_operator, max_operator, Morphology_Operations );
  createTrackbar( "Element:\n 0: Rect - 1: Cross - 2: Ellipse", window_name,
                  &morph_elem, max_elem,
                  Morphology_Operations );
  createTrackbar( "Kernel size:\n 2n +1", window_name,
                  &morph_size, max_kernel_size,
                  Morphology_Operations );
  Morphology_Operations( 0, 0 );
  waitKey(0);
  return 0;
}
void Morphology_Operations( int, void* )
{
  // Since MORPH_X : 2,3,4,5 and 6
  int operation = morph_operator + 2;
  Mat element = getStructuringElement( morph_elem, Size( 2*morph_size + 1, 2*morph_size+1 ), Point( morph_size, morph_size ) );
  morphologyEx( src, dst, operation, element );
  imshow( window_name, dst );
}

其他算法

为了找到实现 Matlab 里 imfill(BW.'hole') 的算法, 探索了如下 opencv 算法.

floodfill() 泛洪算法

类似与画图软件里面的画圈填充颜色.
参数:

• img: 为待使用泛洪算法的图像.
• mask: 为掩码层, 使用掩码可以规定是在哪个区域使用该算法, 如果是对于完整图像都要使用, 则掩码层大小为原图行数+2, 列数+2. 是一个二维的 0 矩阵, 边缘一圈会在使用算法是置为1. 而只有对于掩码层上对应为 0 的位置才能泛洪, 所以掩码层初始化为 0 矩阵.
• seed: 为泛洪算法的种子点, 也是根据该点的像素判断决定和其相近颜色的像素点, 是否被泛洪处理.
• newvalue: 是对于泛洪区域新赋的值(B,G,R).
• (loDiff1, loDiff2, loDiff3): 是相对于 seed 种子点像素可以往下的像素值, 即 seed(B0,G0,R0), 泛洪区域下界为(B0-loDiff1,G0-loDiff2,R0-loDiff3).
• (upDiff1,upDiff2,upDiff3): 是相对于 seed 种子点像素可以往上的像素值, 即 seed(B0,G0,R0), 泛洪区域上界为 (B0+upDiff1,G0+upDiff2,R0+upDiff3).
• flag: 为泛洪算法的处理模式.
  示例代码:https://docs.opencv.org/4.5.2/d1/d17/samples_2cpp_2ffilldemo_8cpp-example.html#a12
  解读参考:https://blog.csdn.net/qq_37385726/article/details/82313004

drawContours()

思路是找到每个空洞的轮廓, 依据轮廓间的包含关系(hierarchy), 对小的被包含的轮廓内的形状填充.

参数:

• image: Destination image.
• contours: All the input contours. Each contour is stored as a point vector.
• contourIdx: Parameter indicating a contour to draw. If it is negative, all the contours are drawn.
• color: Color of the contours.
• thickness: Thickness of lines the contours are drawn with. If it is negative (for example, thickness=FILLED ), the contour interiors are drawn.
• lineType: Line connectivity.
• hierarchy: Optional information about hierarchy. It is only needed if you want to draw only some of the contours (see maxLevel ).
• maxLevel: Maximal level for drawn contours.
  • If it is 0, only the specified contour is drawn.
  • If it is 1, the function draws the contour(s) and all the nested contours.
  • If it is 2, the function draws the contours, all the nested contours, all the nested-to-nested contours, and so on.
    This parameter is only taken into account when there is hierarchy available.
• offset: Optional contour shift parameter. Shift all the drawn contours by the specified 𝚘𝚏𝚏𝚜𝚎𝚝=(dx,dy).

findContours() 找到边界

示例代码:https://docs.opencv.org/4.5.2/da/d32/samples_2cpp_2contours2_8cpp-example.html#a21