How to Implement OpenCV Computer Vision Algorithms in IoT Home Automation

OpenCV: Overview of the Computer Vision Library

Computer vision (CV) is a subset of artificial intelligence that deals with detecting, processing, and distinguishing objects in digital images and videos. Building computer vision applications comes with a variety of solutions and technologies, including libraries, frameworks, and platforms. OpenCV is one of these solutions. It’s a set of libraries with over 2500 tools that vary from classical machine learning (ML) algorithms, such as linear regression, support-vector machines (SVMs), and decision trees, to deep learning and neural networks.

OpenCV is an open-source solution that can be freely used, modified, and distributed under the Apache license. The library was initially introduced by Intel in 1998, and the corporation has been supporting OpenCV and contributing to it ever since. For example, one of the latest contributions is the Deep Neural Network (DNN) module added to OpenCV 3.3. It allows users to train neural networks on deep learning frameworks and then port these networks to OpenCV for further actions.

The library is compatible with an array of operating systems, including Windows, Linux, macOS, FreeBSD, Android, iOS, BlackBerry 10, and supports software written in C/C++, Python, and Java. It has strong cross-platform capability and compatibility with other frameworks. For example, you can easily port and run TensorFlow, Caffe, PyTorch, and other models in OpenCV with almost no adjustments.

OpenCV comprises a broad range of modules intended to process images, detect and track objects, describe features, and perform many other tasks. Here are some of the modules:

• core. Core functionality
• imgproc. Image Processing
• videoio. Video I/O
• video. Video Analysis
• objdetect. Object Detection
• ml. Machine Learning
• stitching. Images stitching

The OpenCV library is equipped with the GPU module that provides high computational power to capture videos, process images, and handle other operations in real-time. The OpenCV GPU module enables developers to create advanced algorithms for high-performance computer vision applications.

OpenCV has a large worldwide community. Dozens of thousands of AI scientists, researchers, and engineers have been providing valuable insights into the library for more than 20 years.

You can also read our Guide to the OpenCV Library in Infographics.

OpenCV Applications

OpenCV is a multifunctional and multipurpose solution found in a wide spectrum of applications. For example, it helps robots and self-driving cars navigate and avoid obstacles using object detection and tracking features. Implementing OpenCV in industrial solutions automates security systems and quality control procedures. Let’s consider some more OpenCV applications in various industries.

Healthcare

OpenCV detection and classification algorithms can identify a disease or potential health threat by analyzing patients’ organs and tissue pictures. We used the OpenCV library to build an iOS app for identifying melanoma skin cancer. The SkinView project algorithms detect the edges of a mole, calculate its color, shape, and other parameters, and decide whether or not the mole is malignant. After that, the app sends the results to a doctor. In this project, we reduced data processing time to less than 0.1 seconds and increased diagnostic accuracy to 80%.

Autonomous Vehicles

Using computer vision along with multiple sensors and cameras can help vehicles navigate either in the air (unmanned aerial vehicles) or in city traffic (self-driving cars). OpenCV offers plenty of algorithms to detect, classify, and track people and objects.  These algorithms allow a vehicle to identify traffic signs, lane lines, pedestrians, and other cars, thus guiding it safely to its destination.

Augmented Reality

Computer vision integrated into mobile software can power augmented reality (AR) apps, enabling users to combine virtual and real-world objects. AR solutions based on OpenCV algorithms can visualize architecture and design projects, improve the learning process at schools and universities, and assist businesses in promoting their products and optimizing customer experience.

Facial Recognition

A facial recognition system detects, classifies, and recognizes people by their facial features. This technology has widespread use in different applications, including photo and video editing apps and social media. OpenCV human face detection algorithms can help authenticate smartphone users, verify identities in entry systems, and keep people safe in public gathering places.

OpenCV in IoT Home Automation

Smart homes are the Internet of Things systems that assist people in running household functions. The networks of IoT devices can control lights, regulate indoor temperature, water plants, and turn on the TV. Computer vision is one of the technologies that make homes intelligent—for example, a smart fridge can use cameras to check food supplies. Developers can leverage a large variety of computer vision algorithms and tools available in the OpenCV library to create smart home projects.

Providing security is an integral part of IoT home automation. Thus, smart security solutions can assist parents in looking after their kids. Deploying computer vision applications for people detecting improves safety in many alarm and video intercom systems. Implementing OpenCV face recognition can prevent strangers from entering a house or apartment.

In addition to protecting homes from intruders, it is necessary to ensure the safety of people who live alone and cannot always take care of themselves. Computer vision people detection systems based on OpenCV algorithms and neural networks can remotely monitor elderly people and people with health problems and disabilities. In case of an emergency, they can alert relatives or caregivers. Here, we’ll share our personal experience in building a remote monitoring system for real-time human detection with OpenCV.

How to Use Computer Vision to Detect People in Smart Homes with the OpenCV Library

The primary task of the Algodroid project was to develop computer vision-based algorithms for people detecting. Our client wanted to integrate a CV system into an IoT solution to recognize life-threatening situations and provide safety for the elderly in their homes. The system had to:

• detect a human;
• identify a posture;
• detect a fall;
• distinguish between a fall and sleep or rest;
• warn of the dangerous condition.

As a multi-feature cross-platform library with high processing speed, OpenCV was a suitable option to complete the above tasks.

Our project employed OpenCV human detection algorithms and neural networks to track and analyze people’s activities throughout the day. We used datasets provided by the client and created our videos to help the model differentiate between human postures. However, the amount of data was insufficient for effective training and understanding the context. So, we broke the task down into 2 steps.

First, we used OpenCV to implement computer vision for people detecting and skeleton visualization. We built neural networks and trained them to detect humans using a variety of data freely available online. To segment a human skeleton, we used TensorFlow-based BodyPix. This is an open-source ML model that allows for real-time body part segmentation. The model segments a human body into 24 parts and visualizes each part as a set of pixels of the same color.

After segmenting a body, our human detection system determined its biomechanical data, such as body geometry and movements. These parameters were calculated and classified by OpenCV motion tracking algorithms.

To estimate the posture of a person, we turned to synthetic data generation. By using simulation libraries, we created a physical model of a human body based on real proportions, biometric and biomechanical data. We placed the model in a virtual environment and generated probable scenarios of human actions. For example, a person could walk some distance, then stagger and fall—all in all, there were about a hundred scenarios. Based on them, the algorithms learned to estimate the posture.

For the second step of the task, we built decision trees to match the estimated posture with the target state. The algorithms compared the posture with the patterns from the simulated scenarios and predicted falls.

We developed a communication system that collected data from all of the cameras that were installed in the house. After identifying a fall, it could send the picture and notify an emergency medical service for further assistance. We were also planning to design a custom Android tablet for the elderly to contact their relatives and get an online medical consultation.

While working on this project, we tried different types of cameras—from simple RGB cameras to Orbbec Astra and Intel RealSense depth cameras. They tracked people during the day and sent images to a single-board computer (SBC) for processing. We used NVIDIA Jetson Xavier NX as the SBC.

Intel RealSense provided high-quality RGB-D images that simplified the work of computer vision algorithms. Depth cameras used infrared light and could detect events even in the dark, but they were too expensive and required more time to process the images. The cheapest solution was an RGB camera. Combined with BodyPix, it showed the same results as depth cameras but saved time on image processing.

The biggest advantage of using the OpenCV library for this project was the opportunity to combine different methods and approaches within one platform. Our fall detection system comprised both classical ML algorithms and deep learning. As a result, the system could detect 10 target states out of 10 activities. The DNN module allowed us to train neural networks on other frameworks and then successfully run them on OpenCV.

To protect personal data, minimize response time, and ensure the stability of our IoT network, we implemented edge computing. That way, we could process data and run algorithms directly on the SBC offline.

Main Challenges of Implementing People Detection in OpenCV-based Projects

The biggest challenge of any project related to real-time human detection in computer vision is reducing false positives and achieving high detection accuracy. We faced this challenge too.

Initially, we used simple OpenCV machine learning algorithms to differentiate postures, whether a person walks, lies down, or falls. Implementing Hidden Markov Models (HMMs) helped us predict a fall based on the previous behavior of a person. We detected body contours, defined the center of mass of the body, and analyzed the deviations from this data frame by frame. The more explicit the deviations, the higher the probability of falls.

At first sight, this approach seemed to work, but finally, it proved too inefficient—the system detected a fall when a person was sitting down or just leaning forward. In addition, contour detection depended on the lighting in the room: we couldn’t detect people using computer vision in poor lighting. As a consequence, the number of false positives was too high, so we had to give up this idea despite the inexpensiveness and simplicity of use.

To achieve better accuracy in human body detection, we decided to build neural networks based on biometric parameters and biomechanical data. To train the models, we generated synthetic datasets with the help of simulation services. Unity, Gazebo, and other simulator platforms provide test environments that simplify and speed up the training process.

Implementing people detection using computer vision can get more complicated in indoor spaces, such as homes. A person can be hidden behind a piece of furniture, so the camera will not capture the entire body and therefore the system will not get the complete biometric data. After trying different approaches, we combined neural networks with decision trees—classical machine learning algorithms included in OpenCV motion detection. These algorithms have significant advantages: they are fast and easy to understand, they can learn from small amounts of data or when some data is missing.

Conclusion

Smart home IoT networks make people’s lives easier in many respects. They automate daily routines, control energy, and ensure the safety of the household. Modern smart homes often rely on machine learning and artificial intelligence technologies. For example, remote monitoring systems can be based on computer vision algorithms for people detection.

A distant monitoring system integrated into a home automation solution can look after elderly and disabled people who live independently. Cameras track people’s activities, and algorithms analyze their behaviors, identifying emergencies.

OpenCV is a popular choice for IoT home automation projects with computer vision. It is an open-source ecosystem of tools, algorithms, and neural networks that perform many functions. OpenCV is one of the libraries we use for building CV applications. For example, OpenCV motion detect algorithms can identify and track people in real-time. If you want to create a smart solution using the latest advances in computer vision or AI, drop us a line and share your thoughts.