WP1 Development, implementation and validation of machine learning algorithms
D1.1 Report requirements from users, work scenarios and performance indicators
The current activity was dedicated to collecting relevant requirements from current and potential customers. Based on them, the existing video data sources that can be used as sources for the developed solution and AI algorithms were studied, starting from analog video data to state-of-the-art IP systems. Communication and data protocols in the field, especially ONVIF, were also analyzed. Particular attention was paid to the synthesis and aggregation of data collected from RETAIL beneficiaries, in order to identify the needs and expected results of the proposed solution as well as to identify the minimum performance that the system must achieve in order to be used in their activity. The following scenarios of interest were identified and defined in general: Shelf monitoring and emptyness detection, Store area monitoring, Vegetable-fruit area monitoring and spoilage detection and Area monitoring with expensive products, to name a few. Last but not least, the GDPR aspect was analyzed - of great interest to potential beneficiaries.
D1.2 Report standards of interest, legislative framework and compliance issues
The ONVIF standard, which was used in this system, was described. The purpose of this standard is to establish compatibility between multiple network devices. Creating a system in which you want to configure devices belonging to different manufacturers may cause problems with compatibility between cameras and IP recorders. The role of the ONVIF standard is to standardize the IP interface. Currently, this standard is an association of several hundred companies, being supported by the top manufacturers in the CCTV industry. Later, ONVIF began to develop and introduce standard profiles, which are used to improve device compatibility. Within this system, 4 of the ONVIF profiles were used: S, T, G, and M. Secondly, the purpose of GDPR in data protection in general and the way it defines personal data and sensitive personal data was reviewed. It has also been detailed how this applies to videos taken from surveillance cameras, and what regulations and principles should remain. Another important topic is how anonymization is defined by the GDPR, why it is necessary, and how it could be done. It has also been briefly shown how high-level machine learning can learn human prejudice and how this can lead to discrimination.
D1.3 Report techniques for implementing algorithms based on neural networks
During this activity, the key deep neural networks used in the modules developed for this project were presented. The techniques by which these networks were developed and integrated were also discussed, as well as the unified framework for the development of deep networks, and the way in which training experiments can be defined.
D1.4 Validated algorithm for product quality recognition using deep neural networks
This activity presents an analysis of the implementation of product quality recognition algorithms using deep neural networks. The state-of-the-art recognition methods were presented and the proposed solution was analyzed, based on two architectures that collaborate for quality recognition. The first has the role of detecting and segmenting the displayed products, and the second has the role of classifying the detected products according to their quality.
D1.5 Validated algorithm for identifying the degree of shelf filling using deep neural networks
The importance of automatic detection of unavailability of products on the shelf was presented. Also, two types of approaches were examined: automatic detection of objects on the shelf, respectively automatic detection of emptyness in the shelf. In the case of automatic detection of objects on the shelf, an additional step in post-processing is required, namely the detection of emptyness already having the information on the areas where products are present. This post-processing can be implemented based on the second type of algorithms presented. The combination of the two types of algorithms seems to offer the most advantages, because their functionalities are complementary and, in this way, one can benefit from the advantages of each type, in part.
D1.6 Validated algorithm for detecting suspicious behavior using deep neural networks
Numerous methods have been proposed in the literature for the detection of suspicious behavior in video streams. A peculiarity of this analysis is the complexity, requiring a multi-expert decision in both space and time. The candidate architectures described are complementary, computationally optimized and with state-of-the-art results on public databases.
WP2 Design, development and implementation of the experimental models of the proposed solution
Functional dedicated Retail interface software module
The proposed systems, namely the integrated platform and the data interconnection / delivery module, have been developed with the main purpose of receiving video data streams, as well as generating alarms and data aggregation. To these is added the SMARTRetail module which brings new functionality dedicated to the retail environment. The interface, in its entirety, is extremely easy to use and intuitive, this being very useful for the beneficiary who can focus on the results of the platform and their exploitation for the necessary purpose, not on how to use it.
D2.2 Integrated and functional module for recognizing product quality using deep neural networks
The proposed systems, respectively the algorithms for recognizing the quality of products using deep neural networks have been developed and integrated in the SMARTRetail module, bringing new functionality and dedicated to the retail environment. The algorithms are of the latest generation and offer reliable results, both networks having performances around the threshold of 80% and 90% respectively.
D2.3 Integrated and functional module for identifying the degree of shelf filling using deep neural networks
The proposed systems, respectively the algorithms for identifying the degree of shelf filling using deep neural networks have been developed and integrated in the SMARTRetail module, bringing new functionality and dedicated to the retail environment. The algorithms are state-of-the-art and provide reliable results.
D2.4 Integrated and functional module for detecting suspicious behavior using deep neural networks
The proposed systems, respectively the algorithms for detecting suspicious behavior using deep neural networks have been developed and integrated in the SMARTRetail module, bringing new functionality and dedicated to the retail environment. The algorithms are state-of-the-art and provide reliable results.
D2.5 Integrated and functional mododules: Data Fusion and SMART Retail Dashboard
During this activity, the dedicated Data Fusion module was developed and implemented, as well as the dedicated SMARTRetail menus. I followed: Maintaining the unit of the graphical setting and configuration interface; Reporting and viewing events in a centralized and unitary way; Its solution includes event filtering capabilities for quick and easy identification of events of interest; The solution provides “Dispatcher” tools to ensure the operator’s attention, visual and audible, in case of detection of an event that falls under the “alarm / alert”; The developed software modules were tested internally using the existing infrastructure and data relevant to the actual usage scenarios.
D2.6 Laboratory integrated test report
The combination of the KVision platform with the latest generation algorithms developed in this project leads to the generation of extremely useful information, presented in an intuitive way, for the retail market. The current activity focused on testing the solution developed within the project, integrated testing in the sense that it started from the tested version of the software platform in which the newly developed AI algorithms were integrated, also tested. We aimed to validate a solution that could be installed, as a pilot or directly as a product, in the field, at one of the potential identified beneficiaries. Cyclic testing, in several steps, interspersed with activities to fix the identified malfunctions and optimizations of code and algorithms, led to the validation of the version obtained and the confirmation of the status of major version “release”.
WP3 Pilot model development, testing and final validation
D3.1 Complete documentation for Retail dedicated algorithms validated by real-time testing
The project aimed to develop, test and integrate a series of algorithms with the goal of improving the quality of services and consumer satisfaction in supermarket stores. To this end, we have developed three solutions covering the following scenarios: (i) recognizing the quality of the products on the shelf and warning in case of detection of spoiled products, (ii) detecting the degree of filling of the shelves and warning in case of emptiness, (iii) detection and warning in case of suspicious behavior of people in the store. The developed algorithms have been tested, validated and integrated in the SMARTRetail type platform, being high performance algorithms that can help the retail field by increasing the quality of services and improving the consumer experience.
D3.2 Validated modules for dedicated processing
The product quality recognition algorithm aims to receive images captured from supermarket store shelves, analyze them, and warn in case of detection of altered products. This algorithm uses two interconnected processing networks. The first network is of the Mask R-CNN type and its role is to identify all the products on the shelf. The second network is represented by a ResNeXt architecture and has the role of classifying the detections found in the previous step into two distinct classes: quality product or altered product. The role of the shelf filling algorithm is to receive captured images from supermarket store shelves, analyze them, and alert operators if portions of shelves without products are detected. This algorithm uses a ResNet-101 network capable of distinguishing a very large number of different product classes. The suspicious behavior detection algorithm aims to receive images captured from stores, analyze them temporarily and warn operators if people whose behavior can be classified as suspicious are detected. This algorithm uses three processing structures: (i) a motion estimator followed by the extraction of features with ResNet-101 networks, (ii) the spatio-temporal components are analyzed through a ConvLSTM network in order to encode the actions of persons, (iii) finally, a dense architecture is used to classify actions into normal actions and suspicious actions.
D3.3 Dashboard mode and report generation validated
The KVision platform interface has been updated in the project and specific features have been added. The current stage focused on testing and validating the functionality with fixing the identified problems. When monitoring live, in real time, the operator has quick and easy access to the data necessary for the proper treatment of the event: the icon, the type of event, the name of the devices associated with the event, the date and time of the event declaration, the itinerary description. Project-specific search functions for finding events of interest were included, including video source, period of interest, algorithm of interest, object that triggered the alarm.
D3.4 Integrated platform that includes validated Retail dedicated processing modules
The development team wanted the system interface to be extremely simple to use and completely intuitive. This is 100% configurable, with the user having access to algorithms, media sources, processing instances, notifications. Full setting of algorithms is allowed, with characteristic settings for each of them. The user also has the necessary tools to add or delete processing instances. This ensures resource management - both hardware, the active instances being those supported by the hardware platform, but also software, at the licensing level being activated a number of processing instances that the user will be able to activate and allocate according to his needs, needs that can change dynamically over time.
D3.5 Documentation of hardware processing platforms and instances supported processing competitors
The algorithms developed and integrated in this project can process information in real time, from several high-resolution processing instances. By running several algorithms simultaneously, a robust solution is obtained, with a great utility for supervising supermarket type stores, thus increasing the quality of their services and improving the consumer experience in them. During the activity, testing sessions were carried out for different hardware platforms in order to identify the platforms with the best cost-benefit ratio and also to be able to recommend to potential customers the necessary hardware platform depending on the number of processing instances that will be activated. For reference, the minimum hardware architecture required is Intel (R) Core (TM) i7-8700K CPU with 16 GB RAM and NVidia GForce GTX 1080 GPU.
D4.6 Report on exploitation strategy and sustainability
Within the consortium in general and in the company in particular, the basic directions specific to good practices regarding the exploitation of the results of R&D projects were analyzed, namely: (i) Exploitation of the results Directly through their use in further research and development; developing, creating or marketing a product or process; creating and providing services based on them; their use in standardization activities. (ii) Exploitation of results Indirectly through the transfer of results; licensing; spin-offs.